CodeClash: Benchmarking Goal-Oriented Software Engineering

THOUGHT: [Natural language reasoning about current state and planned action]
```bash
[EXACTLY ONE bash command, or commands connected with && or ||]

If the model produces zero or multiple bash blocks, or formats the response incorrectly, a `format_error_template` is returned instructing the model to retry. This guardrail is separate from execution errors (commands that execute but return non-zero exit codes)—the paper reports that across all models, over 85% of actions execute successfully, with error rates ranging from just 10% (Claude Sonnet 4) to 16% (Qwen3 Coder). Recovery from errors is rapid: following a failed command, the very next action succeeds over 80% of the time, and by the third subsequent action, there are "nearly zero occurrences of models continuing to struggle" (Appendix D.1).

**System prompt design.** The initial system prompt provided to each player at the start of every edit phase is deliberately minimal and arena-agnostic. It contains:

1. A brief role description: "You are a helpful assistant interacting continuously with a computer by submitting commands. You'll be editing a codebase to play a programming game."

2. The ReAct interaction protocol specification (thought + one command format).

3. An arena description: a short paragraph specific to the game (e.g., for BattleSnake: "Your bot controls a snake on a grid-based board. Snakes collect food, avoid collisions, and try to outlast their opponents"). These descriptions are intentionally brief—the paper states: "We impose the burden of understanding how exactly an arena works. With full access to documentation and logs in the codebase, CodeClash forces LMs to identify and fill in gaps about its understanding of the game" (Appendix C.1).

4. General guidance about available resources ("Game documentation", "Past rounds and outcomes", "History of your edits") and the transient nature of memory between rounds.

5. An explicit nudge toward self-directed improvement: "If you'd hate to repeat a step next round, encode it now—as a script, a note, or a tool. Improve the bot however you like—experiment, document, iterate. Some ideas: Build analysis tools, Create bot variants to test, Track strategies across rounds."

6. Turn and cost limits: the 30-step limit and $1 cost limit, with automatic termination reminders embedded in the observation template: `<limit_note>This is the output of step {{n_model_calls}} ({{step_limit}} limit). You've used {{model_cost | round(2)}} USD ({{cost_limit}} USD limit).<limit_note>`.

The prompt provides NO arena-specific strategies, NO guidance on which files to examine, and NO instructions about how to interpret logs. This is a deliberate design choice: "beyond a brief description of the environment and arena, the initial system prompt provided to each player at the start of every edit phase contains no guidance beyond high level suggestions about how to enhance its codebase" (Section 2.3). The paper wants to test whether models can independently discover effective development practices, not whether they can follow instructions about what practices to use.

**Why bash-only interaction.** The paper cites Yang et al. (2024b) in arguing that "various workflows and tools can be (un-)intentionally biased to favor particular models. Our goal is to evaluate models, not scaffolds or tools. Therefore, we decide to make LMs operate in the most 'impartial' action space" (Appendix A). Bash commands are a universal interface—every model has been trained on command-line interactions, and the space of possible actions is unconstrained (models can install packages, write scripts, compile code, run any executable). This also "leaves an opportunity for LMs to synthesize their own tools across rounds" (Appendix A), which they do: models create analysis scripts, test harnesses, and game simulation runners as the tournament progresses.

The paper validates this choice by verifying that mini-SWE-agent with Claude 4 Opus achieves 67.6% on SWE-bench Verified (the bash-only leaderboard), confirming that "the models we evaluate are capable of performing bash-only interactions with a low to non-existent rate of failures due to syntactic errors such as malformed responses or actions" (Appendix A). An ablation replacing mini-SWE-agent with SWE-agent (which provides `str_replace_editor`, file-tree viewer, and AST-level code search) on three models and two arenas shows at most 2-rank differences in CC:Ladder scores (Appendix D.6.4), suggesting the scaffold choice does not dominate performance.

---

#### Code Arena Interface

The code arena is the competition execution environment, and the paper defines a minimal interface to maximize extensibility.

**Interface requirements.** An implementation only needs to define two capabilities:

1. **Commands to run the competition**: A script or executable that takes codebase paths as arguments and runs the arena simulation. The exact invocation is arena-specific but follows a standard pattern—e.g., for Battlesnake: `python main.py --width 11 --height 11 --browser false`.

2. **A mechanism to determine a winner**: After the competition executes (1000 simulations per round), the arena must produce a measurable outcome—a score, a ranking, or a survival indicator—that can be compared across players.

**Starter codebase contents.** Every player begins with identical assets designed to ensure meaningful competition from the first round without assuming any prior knowledge:

- **Documentation**: Full arena documentation copied from official sources into a `docs/` folder. For Battlesnake, this includes the complete API reference, game rules, map configurations, and quickstart guides. For Core War, it includes the Redcode language specification and the pMARS simulator documentation. Models must proactively discover and read these files.

- **Arena executable**: All binaries and assets needed to run the arena locally. Crucially, "the exact bash commands are not disclosed; the burden remains on the model to figure out how to use assets" (Appendix A).

- **Working submission**: A simple but functional baseline strategy that can be submitted as-is. This is analogous to the starter code provided to human participants in competitive programming contests. For Battlesnake, this is a basic snake that avoids walls and moves randomly. For Core War, this is the classic "Dwarf" warrior that bombs every fourth memory cell. The paper notes this ensures "meaningful competition from the first round" and prevents models from needing to implement a complete solution from scratch before any competition occurs.

**Winner determination with non-determinism.** Most arenas exhibit non-deterministic behavior—even with identical codebases, outcomes vary across runs due to random seeds, timing, or stochastic game mechanics. The paper handles this by running each competition 1000 times per round (`sims_per_round: 1000` in tournament configuration). The winner is declared as "whichever player wins the most out of the 1000 simulations (or declare a tie if ties are most frequent), rather than requiring a specific win percentage threshold" (Appendix A). This follows standard practice in competitive gaming communities and avoids introducing arbitrary performance cutoffs.

**Invalid submission handling.** Models sometimes produce codebases that fail validation—compilation errors, missing required functions, incorrect file structures. The paper defines a decision tree for handling these cases:

- If all player codebases are invalid, the round is a tie.
- If exactly one player codebase is valid, that player wins by default.
- If two or more codebases are valid, the competition runs with all valid codebases; invalid ones are excluded.

Per-arena validation checks are implemented to detect these failures before the competition runs: Battlecode requires `./bot.py` implementing a `turn` function; Battlesnake requires `main.py` implementing `move`; RoboCode requires tank bots under `robots/custom/` that pass `javac` compilation. The paper notes these are not invented constraints—"these rules are reflective of the original conditions these arenas and games impose on human players and their submissions" (Appendix A).

**Tournament configuration.** The entire setup is specified in a YAML configuration file with sections for tournament parameters (number of rounds, simulations per round), game parameters (arena name, arena-specific arguments like board dimensions), and player parameters (model names, agent configurations). This configuration system enables the paper's large-scale evaluation: for 8 models and 6 arenas with 10 tournaments per matchup (pairwise), the total is `(8 choose 2) × 6 × 10 × 15 = 25,200` rounds, all launched programmatically.

**Docker containerization.** All codebase editing and arena execution occurs within Docker containers, never on the host machine. This ensures reproducibility, isolation, and security—the only artifacts created on the local machine are logs capturing tournament metadata and outcomes. Each player operates in a separate agent container, and the competition phase copies codebases into a shared arena container, as visualized in Figure 9.

**Why this interface design.** The minimal arena interface is forward-looking: "CodeClash's flexible definition for a code arena can incorporate existing simulators or inspire new environments for areas such as cybersecurity, healthcare, and city planning" (Appendix D.4). By requiring only an executable and winner-determination logic, new arenas can be added without modifying the tournament infrastructure, making CodeClash extensible to diverse domains beyond the initial six games.

---

#### Code Arenas: Mechanics, Languages, and Logs

The paper's six arenas span five programming languages (Python, Redcode assembly, C/C++/OCaml/Rust, Java, JavaScript) and represent fundamentally different competitive objectives, ensuring that models cannot succeed through a single generic strategy.

**BattleSnake.** A grid-based survival game where snakes collect food, grow longer, and attempt to be the last snake alive. Programming language: Python. Win condition: last surviving snake (or longest snake if multiple are alive when the turn limit expires). Key mechanics: snakes lose one health per turn, regain health by eating food (which also increases length), and die from wall collisions, self-collisions, collisions with longer snakes, or head-to-head collisions (longer snake wins; equal length = mutual elimination). Movement is in four cardinal directions on an 11×11 grid. The starter codebase provides `main.py` implementing `info()`, `start()`, `end()`, and `move()` functions, with a simple random-move baseline. Logs are JSONL files where each line is a complete board state snapshot: turn number, map dimensions, snake positions (ID, health, body coordinates, head position, length), food locations, and hazards. Effective strategies include flood-fill analysis for space estimation, A* pathfinding, look-ahead search for collision prediction, and risk-aware heuristics for engagement decisions.

**Core War.** A programming game where players write assembly-like "warriors" in Redcode that compete within a simulated shared memory space called the "core." The MARS (Memory Array Redcode Simulator) executes warriors in alternating cycles, one instruction per active process per cycle. Win condition: last warrior with any surviving processes. Key mechanics: warriors can spawn additional processes (SPL instruction), and processes die by executing invalid instructions or being overwritten by opponent "bombs." The starter codebase includes the pMARS simulator, assembler, example warriors (including the classic "Dwarf" bomber shown in Figure 15), and full Redcode documentation. Logs are high-level summaries reporting which warrior survived, process counts, cycle counts, and match duration—not step-by-step instruction traces. Effective strategies combine offense (bombers that scatter invalid instructions, scanners that locate and target opponents), defense (process replication for survival), and adaptability to opponent tactics.

**Halite I.** A resource collection and territory expansion game on a toroidal grid. Programming language: choice of C, C++, OCaml, or Rust. Win condition: control the most territory (or total strength as tiebreaker) when only one player remains or the turn limit is reached (10 × √(width × height) turns). Key mechanics: pieces occupy grid cells with production and strength values; remaining STILL increases strength by the site's production; moving leaves a zero-strength piece behind; combat between opposing pieces reduces each by the opponent's strength; strength is capped at 255. Map dimensions vary per match but are always symmetric (ensuring fair starting conditions). The starter codebase provides a `MyBot` template in each supported language, a `RandomBot` reference, helper libraries for game state communication, and local simulation/visualization tools. Logs record setup information (bot executables, map configuration), sequential turn-by-turn entries, and final rankings with survival information.

**Poker.** No-Limit Texas Hold'em via the Husky Hold'em Bench engine. Programming language: Python. Win condition: largest remaining chip stack at match end. Key mechanics: two private cards per player, five community cards revealed across four betting rounds (pre-flop, flop, turn, river), no-limit betting (bets can be any size up to stack), legal actions include check/call/raise/fold. The starter codebase includes the full poker engine (`engine/` directory with deck handling, hand evaluation, betting round logic, state transitions) and client infrastructure. Players subclass `Bot` and implement lifecycle hooks: `on_start()`, `on_round_start()`, `get_action()` (the core decision function), `on_end_round()`, `on_end_game()`. Logs are structured JSON recording each hand: betting rounds, player actions with amounts, pot totals, community cards, hole cards, and final chip balances.

**RoboCode.** A tank combat simulation on a 2D battlefield. Programming language: Java. Win condition: highest cumulative score across rounds, computed from survival points, bullet damage, ram damage, and last-survivor bonuses. Key mechanics: tanks can move, turn (body/turret/radar independently), and fire bullets that consume energy and have travel time; gun cooling rate limits fire frequency; battles run for configurable numbers of rounds on configurable battlefield sizes (default 800×600 pixels). The starter codebase provides precompiled example robots, compilation infrastructure, configuration files, and template robot classes. Players extend `Robot` and implement `run()` (main loop) and `onScannedRobot()` (event handler for radar detections). Logs report per-bot score breakdowns and first/second/third place counts across rounds—not turn-by-turn detail. Effective strategies include predictive targeting (aiming at anticipated future positions), wave surfing (evasive movement patterns), and maintaining unpredictability.

**RobotRumble.** A turn-based grid battle where robots spawn every 10 turns. Programming language: JavaScript. Win condition: more robots after 100 turns. Key mechanics: up to four new robots spawn every 10 turns (robots remaining in spawn are purged); each robot has 5 health; robots can move or attack in cardinal directions; attacks deal 1 damage and can hit teammates; movement conflicts are resolved by a fixed clockwise priority rule. The starter codebase includes rumblebot CLI for execution, builtin example bots, and game logic documentation. Players implement a `robot(state, unit)` function returning an action each turn. Logs display as sequential ASCII grids (100 grids per simulation) with robot positions, plus summary lines showing remaining health and unit counts. Effective strategies include timing exits to avoid spawn purges, exploiting movement conflict priority through approach direction, and focus-fire coordination (5 units attacking the same target eliminate it in one turn).

**Why these six arenas.** The diversity is deliberate: "CodeClash's arenas vary significantly, with drastic differences in a codebase's structure, how a codebase interfaces with the arena engine, and the types of logs and feedback generated. This contrasts sharply with existing benchmarks, where evaluation follows a consistent pattern of problem statement, code implementation, and test validation" (Section 2.3). A model that performs well in one arena but poorly in others reveals domain-specific strengths; a model that performs well across all arenas demonstrates genuine strategic flexibility. The paper's results bear this out: Claude Sonnet 4.5 ranks 1st overall but only 4th in Poker (Table 1), confirming that arena diversity meaningfully discriminates between models.

---

#### Tournament Configuration and Evaluation Metrics

The paper's main leaderboard uses a specific tournament structure and a rigorously validated Elo methodology to produce stable, interpretable rankings.

**Pairwise tournament structure.** For the main leaderboard, models compete one-on-one (2 players per tournament). With 8 models, there are `(8 choose 2) = 28` unique pairings. Per pairing, 10 tournaments are run, each lasting 15 rounds, across 6 arenas. Total rounds: `28 × 6 × 10 × 15 = 25,200`. Each individual model plays a total of `(8-1) × 6 × 10 × 15 = 6,300` rounds across all opponents and arenas.

**Round win determination.** A round is won by a model if it achieves a higher score than its opponent across the 1000 simulations, or if its opponent makes an invalid submission (compilation failure, missing required files) while its own submission is valid. Ties occur when both models win equal numbers of simulations or both make invalid submissions.

**Tournament win determination.** A tournament is won by the model that wins more rounds than its opponent. If both models win equally many rounds, the model that wins the final round is declared the winner. If all rounds are ties (an extremely rare event, "less than once per 1000 tournaments"), the tournament is declared a draw. The win rate of a model is the fraction of tournaments it has won, aggregated across all opponents and arenas.

**Why 15 rounds with 10 tournaments.** The paper does not provide a formal power analysis, but the round count is sufficient to observe meaningful trends: Figure 3 shows win rates evolving across rounds, Figure 25 demonstrates that round outcomes are not independent (highly bimodal distributions rather than uniform), and the Elo uncertainty estimates in Table 4 (typically ±15-20 for overall scores) confirm that the sample size is adequate for distinguishing model tiers.

**Elo rating methodology.** The paper uses a Bradley-Terry model (Bradley & Terry, 1952) rather than simpler win rate averaging to quantify model strength, following the approach established by Chatbot Arena and other LLM evaluation leaderboards.

The Bradley-Terry model assumes that the probability of model `$i$` winning over model `$j$` depends on their latent strengths `$s_i$` and `$s_j$` through a logistic function:

$$P(\text{model } i \text{ wins over } j) = \frac{1}{1 + \exp(s_j - s_i)} = \sigma(s_i - s_j)$$

where `$s_i$` and `$s_j$` are unobserved real-valued strengths, and `$\sigma$` is the logistic sigmoid function. A larger `$s_i - s_j$` (model `$i$` stronger than model `$j$`) means higher win probability.

**What it computes:** given the observed win counts `$w_{ij}$` (number of tournaments model `$i$` won against model `$j$`) and total games `$n_{ij} = w_{ij} + w_{ji}$`, the model estimates latent strengths `$s_i$` for each model that best explain the observed outcomes. The assumption is that each tournament win is an independent Bernoulli trial with probability `$\sigma(s_i - s_j)$`, and the strengths are estimated by maximizing the log-likelihood:

$$\log \mathcal{L} = \sum_{i < j} \left[ w_{ij} \log \sigma(s_i - s_j) + w_{ji} \log \sigma(s_j - s_i) \right]$$

where the sum runs over all unordered model pairs. Each term in the first bracket credits wins by `$i$` against `$j$` proportional to `$\log\sigma(s_i - s_j)$`, and each term in the second bracket credits wins by `$j$` against `$i$` proportional to `$\log\sigma(s_j - s_i)$`. The optimization finds `$s_i$` values that make the observed win matrix most probable.

**Why maximum likelihood rather than sequential Elo updates.** Sequential Elo updates (as used in chess) depend on the order of game processing and require choosing a step size (K-factor) that controls how rapidly ratings adjust. Maximum likelihood fitting avoids both issues: it finds the single set of strengths that best explains all observed outcomes simultaneously, without dependence on update order or step size. The paper performs this fit across all arenas jointly to produce the overall Elo column in Table 1, and separately per arena for the arena-specific columns.

**Gauge fixing.** The Bradley-Terry likelihood depends only on strength differences, not absolute values—adding a constant `$S$` to all `$s_i$` leaves the likelihood unchanged. The paper fixes this gauge freedom by constraining `$\sum_i s_i = 0$` (zero mean), which makes the solution unique. The conversion to Elo (next equation) then shifts these zero-mean strengths to a 1200-centered scale.

**Conversion to Elo scores.** The fitted strengths `$s_i$` are converted to interpretable Elo ratings `$R_i$` via:

$$R_i = R_0 + \frac{\beta}{\log 10} \cdot s_i$$

where `$R_0 = 1200$` is the base Elo rating (conventional from chess), `$\beta = 400$` is the logistic slope (also conventional—a 400-point difference corresponds to approximately a 10:1 win probability ratio under the Bradley-Terry model), and `$\log 10$` converts from natural log to the Elo convention (which uses base-10 log odds). The scaling factor `$\frac{400}{\log 10} \approx 173.7$` means that a unit change in Bradley-Terry strength corresponds to approximately 174 Elo points.

**What it computes:** given the maximum-likelihood strengths, this linear transformation produces Elo ratings on the familiar 1200-centered, 400-slope scale. A model with Elo 1400 and an opponent with Elo 1000 has a 400-point advantage, implying a `$\sigma(400/173.7) \approx \sigma(2.30) \approx 0.909$` win probability. These are the numbers reported in Table 1 and Table 4.

**Why this conversion rather than raw strengths.** Raw Bradley-Terry strengths are uninterpretable (they are log-odds on an arbitrary scale). Converting to Elo places them on a scale where practitioners have well-calibrated intuitions about what differences mean (e.g., 100 Elo ≈ 64% win rate, 200 Elo ≈ 76%, 300 Elo ≈ 85%). The paper acknowledges that "this convention is merely a presentation choice that affects readability, not the model predictions" (Appendix C.3.1).

**Statistical validation.** The paper goes beyond point estimates to quantify uncertainty, performing both parametric and non-parametric bootstrapping (1000 samples each) to assess rank stability:

- **Covariance from Hessian:** The uncertainty in each `$s_i$` is computed from the inverse of the Hessian matrix of the log-likelihood (the negative log-likelihood's second derivatives), projected onto the gauge-fixed subspace `$S = \{s \mid \sum_i s_i = 0\}$`. This yields standard errors for each Elo score, reported in Table 4—e.g., Claude Sonnet 4.5 overall Elo is 1389 ± 18.

- **Non-parametric bootstrap:** Resamples tournaments with replacement from the observed data, recomputes Elo rankings per resample, and measures rank distribution variance. Results in Figure 27 (left) and Figure 28 (left).

- **Parametric bootstrap:** Draws from the fitted Bradley-Terry model itself—for each observed matchup, samples new win counts from `$\text{Binomial}(n_{ij}, \sigma(\hat{s}_i - \hat{s}_j))$`—and refits. This tests whether the model's fitted probabilities are internally consistent. Results in Figure 27 (right) and Figure 28 (right).

**Rank stability metrics (Table 5).** The bootstrapping experiments yield:

- **Pairwise order agreement: 0.983 (non-parametric), 0.978 (parametric).** This means for any pair of models, the Elo ordering is consistent across over 98% of bootstrap samples—the ranking is highly stable.
- **Kendall's τ: 0.966, 0.956.** A rank correlation of 0.96+ indicates near-perfect agreement between the full bootstrap rankings and the point estimate.
- **Top-1 consistency: 0.896, 0.839.** Claude Sonnet 4.5 is the top-ranked model in 89.6% (or 83.9%) of bootstrap samples—not certain, but strongly favored.
- **Spearman's ρ: 0.988, 0.984.** Near 0.99 rank correlation confirms the ordering is robust.

These metrics collectively validate that 25,200 rounds is sufficient for stable rankings, and that the relative strengths reported in Table 1 (Claude Sonnet 4.5 > GPT-5 > o3 > Claude Sonnet 4 > GPT-5 Mini > Gemini 2.5 Pro > Grok Code Fast > Qwen3 Coder) are not artifacts of sampling noise.

**Distribution of round scores.** Figure 24 shows the distribution of normalized scores per arena across valid rounds. Most arenas show broad, often bimodal distributions—not narrow spikes at 0 or 1—indicating that rounds are genuinely competitive (not one-sided blowouts) and that the 1000-simulation sample size provides meaningful discrimination.

**Distribution of round win counts.** Figure 25 shows the distribution of how many rounds each model wins in a 15-round tournament. The distributions are heavily bimodal (peaks near 0 and 15) rather than uniform, demonstrating that "rounds are not independent of each other" (Appendix C.3.1)—a model that starts winning tends to keep winning, and vice versa. This justifies the tournament-level win metric (which aggregates across rounds) and the Elo methodology (which operates on tournament win counts).

**Multi-player TrueSkill ratings.** For the 6-player Core War tournaments in the ablation (Section 4.1), the paper switches to the TrueSkill rating system (Herbrich et al., 2006) because Elo and win rate are limited to pairwise settings. TrueSkill models each player's skill as a Gaussian distribution `$\mathcal{N}(\mu, \sigma^2)$`, where `$\mu$` is the estimated skill and `$\sigma$` is the uncertainty. After each round, both parameters are updated based on match outcomes: winning increases `$\mu$` more when it exceeds expectations, and `$\sigma$` decreases as the system gains confidence. Table 42 reports these `$\mu$` values with standard errors.

---

#### CC:Ladder: Measuring the Gap to Human Performance

The CC:Ladder protocol (Section 4.1, detailed in Appendix D.6) provides the paper's most stark finding: even the best model cannot defeat expert human-written solutions.

**Ladder construction.** The paper collects publicly available human-authored solutions from two arenas:

- **RobotRumble**: 58 solutions from the public leaderboard.
- **Core War**: 264 solutions from the online Core War directory.

To establish a ranking, every pair of human solutions is matched against each other (250 simulations per pair for RobotRumble, 4000 for Core War—higher counts because Core War has more non-determinism). A Bradley-Terry model with L2 regularization (regularization strength 0.01, base Elo 1200, slope 400) is fitted to the pairwise win matrix to produce an ordered ladder from weakest to strongest. This ladder serves as the ground-truth difficulty ordering.

**Progression protocol.** A model begins with a codebase containing the weakest human solution (rank 1) and progresses upward:

1. The model competes against the current opponent for 7 rounds.
2. The model advances to the next-strongest opponent if it wins **more than ⌊7/3⌋ = 2 rounds** AND wins the final round. The "more than one-third" threshold (rather than simple majority of 4 wins) accounts for cases where a model temporarily degrades its own codebase and then recovers—the threshold is deliberately lower than 50% to not penalize exploration that temporarily hurts performance, while the final-round requirement ensures the model ends with a working solution.
3. The model's codebase **carries over between opponents**—it is never reset. This means the codebase accumulates all the files, notes, tests, and strategies from earlier ladder stages, testing whether models can maintain coherent codebases across extended development.
4. The ladder terminates when the model fails to meet the advancement criteria against the next opponent.
5. The model's score is the rank of the highest opponent defeated.

Each model-ladder combination is evaluated 5 times to account for variance; the paper reports the best score across the 5 runs.

**Results (Table 2).** The findings are stark:

- **RobotRumble (58 human solutions):** The best result is rank 57 out of 58 (GPT-5 mini)—essentially reaching the second-hardest human solution. Claude Sonnet 4.5 reaches rank 43, GPT-5 reaches 51, Gemini 2.5 Pro reaches 54. No model completes the ladder (defeats rank 58).

- **Core War (264 human solutions):** The best result is rank 260 out of 264 (GPT-5 mini). Claude Sonnet 4.5 reaches rank 205, GPT-5 reaches 201, Gemini 2.5 Pro reaches 233. Again, no model completes the ladder.

**Why CC:Ladder is informative.** The ladder evaluation isolates model capability from opponent variance: the human solutions are static (they don't adapt), so the model's performance depends purely on its own improvement trajectory. The carryover codebase requirement tests long-term maintenance—a model that creates 52 single-use files by round 15 (as Claude Sonnet 4.5 does in one tournament, Figure 52) will struggle when the codebase must serve across multiple opponents without human cleanup. Most ladder runs terminate because the model fails to win the final round against an opponent, rather than losing a majority of rounds—suggesting models are often competitive but cannot consolidate gains, consistent with the recovery analysis in Figure 4 showing that even strong models struggle to recover after losing.

**Scaffold ablation on CC:Ladder (Appendix D.6.4).** To verify that mini-SWE-agent is not the bottleneck, the paper re-runs three models (Claude Sonnet 4.5, GPT-5 mini, Gemini 2.5 Pro) on both ladders using SWE-agent (which provides `str_replace_editor`, file-tree viewer, and AST-level code search). Across six model-arena combinations, the highest rank reached under SWE-agent is within 2 positions of the mini-SWE-agent result (Table 7): scores are identical in 4 of 6 cases, and SWE-agent performs marginally better (+1 or +2 ranks) in the remaining two. The paper notes that "SWE-agent's `str_replace_editor` occasionally conflicted with models' preferred editing workflows, and models rarely invoked the additional navigational tools (tree view, AST search), likely because the codebases in these arenas are small enough to navigate via bash alone."

---

#### LM-as-Judge Trajectory Annotation Framework

The paper's strategic reasoning analysis (Section 5.2) uses GPT-5 with high reasoning as an automated annotator to classify model behavior in three dimensions across all trajectories. This methodology is validated against human annotations.

**Three annotation questions.** For each round of each tournament, the judging LM answers three binary (or categorical) questions:

1. **Groundedness of edits**: Are the final edits to the main player file motivated by analysis of previous round logs or by insights gained during the current round's editing (e.g., test results, simulation outcomes)? The model receives the full trajectory (all actions and their outputs) but not the original model's thoughts (to avoid sycophantic tendencies where the judge agrees with the agent's stated reasoning). The answer is True only if ALL of the following conditions hold: a failure mode can be inferred from logs or analyses, and the edits directly address that failure mode. The answer is False if the model didn't look at logs, read only non-diagnostic portions (e.g., first few lines of a log without reaching the decisive moment), or if analysis scripts returned non-actionable results (e.g., only reporting losses without attribution).

2. **Hallucinated loss causality**: Does the model make uncorroborated claims about why a round was lost? The judge identifies "incidents"—concrete factual statements in the model's thoughts that cannot be corroborated by the information available at that step, are not framed as hypotheses, and could have been verified (e.g., by reading more of the log, running analysis scripts, or examining code). Categories of claims include: `loss_reason` (claiming a specific cause for losing), `win_reason`, `game_results` (claiming specific outcomes that aren't supported), `possible_improvement` (suggesting an improvement based on incorrect understanding), `player_code_behavior` (claiming code does something it doesn't), and `performed_edits` (claiming edits were made that weren't). The judge also classifies whether a claim has no identifiable source (`source_category: none`) or misinterprets an existing source (`source_category: log` but the log doesn't contain the claimed information).

3. **Validation of edits**: Are the final edits validated either through arena simulations (running test games against previous versions or example opponents) or unit tests (specific tests covering the new or modified behavior)? The answer is True if the model ran actual game simulations and confirmed the new version performed well, or wrote and executed unit tests that specifically test the changed code. It is False if simulations failed, showed worse performance, or weren't completed before deploying.

**Why GPT-5 as judge.** The paper validates this automated annotation against three human annotators on 100 randomly sampled trajectories, stratified by model and arena. Table 6 reports:

- **Fleiss' κ (inter-human agreement)**: Groundedness 0.770, Hallucination 0.675, Validation 0.770. These are in the "substantial" to "almost perfect" range per Landis & Koch (1977), confirming that humans can reliably annotate these dimensions despite some subjectivity.
- **Cohen's κ (human majority vs. GPT-5)**: Groundedness 0.815, Hallucination 0.737, Validation 0.845. All in the "substantial" to "almost perfect" range.
- **Raw agreement**: 91% for groundedness, 88% for hallucination, 94% for validation.

The hallucination dimension has the lowest agreement—both among humans and between humans and GPT-5—because it requires judging whether a model's interpretation of logs is reasonable versus overconfident. The paper notes that "when humans and GPT-5 disagree on hallucination, GPT-5 slightly more often flags an incident that humans do not, suggesting that our reported hallucination rates are, if anything, conservative estimates" (Appendix D.5).

**Structured output schemas.** The judge LM uses structured output (JSON schemas enforced by the API) to ensure consistent formatting. The groundedness/validation schema (Appendix D.3.2) includes fields: `edit_category` (none/tweak/fix/feature/change), `edits_motivated_by_logs`, `edits_motivated_by_insights`, `edits_motivated_by_old_static_messages`, `edits_reverted_based_on_insights`, `edits_tested_with_simulations`, `edits_validated_with_unittests`, `improved_test_analysis_framework`, and `reasoning`. The hallucination schema (Appendix D.3.3) includes an array of `Incident` objects, each with `step_index`, `claim_category`, `claim`, `source_category`, `source`, and `detailed_reasoning`.

**Action categorization for activity analysis.** Figure 46 (breakdown of what models spend turns on) uses a third LM-as-judge annotation with GPT-5 mini to classify each action into categories: read (source/logs/docs/other, with sub-subcategories for new vs. old files), write (docs/source.main/source.analysis/source.tests/other, with create/modify_old/modify_new), execute (game/analysis/unittest/other, with in_mem/new/old), search, navigate, submit, and other. The category priority rule (execution > write > read) resolves cases where a single action combines multiple operations (e.g., `sed ... test.py && python test.py` is categorized as execute, not write). This enables the paper's quantitative analysis of how models allocate their limited turns across reading, modifying core game logic, running tests, performing analysis, and running simulations.

**Design choice: why strip model thoughts from judge input.** The judging LM does not see the agent's thoughts when answering groundedness and validation questions. This prevents the judge from simply agreeing with the agent's stated reasoning—it must independently verify whether the actions' outputs (the actual evidence) support the edits. For hallucination detection, however, the thoughts ARE provided to the judge, because the task is precisely to identify when thoughts make claims unsupported by evidence.

**Why LM-as-judge rather than pure rule-based analysis.** Rule-based analysis could count file reads, track which files were modified, and detect whether `python test.py` was executed, but it cannot assess whether the logs the model read actually support the conclusions it drew, whether a test actually covers the changed behavior, or whether an analysis script's output is actionable. The semantic judgment required—"does this log contain information about why the game was lost?"—necessitates an LM judge, and the human validation confirms this approach is reasonably reliable.

## 4. Key Insights and Innovations

### Innovation 1: Open-Ended Objectives as the Missing Dimension in Code Evaluation

The paper's most fundamental conceptual contribution is identifying and operationalizing a dimension of software engineering capability that existing benchmarks systematically exclude: the ability to *determine what to build* rather than merely *execute what is specified*. This is not a refinement of existing evaluation methodology—it is a reframing of what "coding capability" means for autonomous systems.

**What the field did before.** Every major coding benchmark—HumanEval, MBPP, BigCodeBench, SWE-bench, SWT-bench, ECCO, EffiBench—shares a common structure: the model receives a problem statement that specifies, with varying precision, what the desired behavior should be. In HumanEval, it's a function signature plus docstring. In SWE-bench, it's a GitHub issue describing the bug. In ECCO, it's an explicit directive to optimize runtime. Success is evaluated against a predetermined correctness criterion: unit tests pass, runtime decreases, or the fix resolves the reported issue. The model's agency is constrained to implementation—the strategic question of *what engineering work to do* is answered by the benchmark designer, not the model.

This structure has been extraordinarily productive for driving progress in code generation and bug fixing, but it creates a blind spot: it cannot distinguish between a model that can independently manage a codebase toward open-ended goals and one that merely excels at following explicit instructions. Both models would perform identically on existing benchmarks, yet their real-world autonomous development capabilities would diverge dramatically.

**What CodeClash does differently.** The paper introduces an evaluation protocol where the objective is stated only at the level of a competitive goal—"survive longer than your opponent," "accumulate more chips," "control more territory"—and the model receives *no specification* of what code changes would advance that goal. There is no issue describing a bug to fix, no function signature defining the desired output, no target runtime to achieve. The model must recursively decompose the high-level objective into actionable engineering tasks entirely on its own: read documentation to understand arena mechanics, analyze competition logs to diagnose weaknesses, decide which strategic approaches to explore, implement changes, validate them through testing and simulation, and encode knowledge for future rounds.

This is a fundamental shift in what the benchmark measures. Existing benchmarks measure *implementation capability given a specification*; CodeClash measures *specification discovery and strategic decision-making given an objective*. The two are complementary—a complete autonomous software engineering system needs both—but the second has been almost entirely absent from evaluation.

**Evidence that this dimension matters.** The paper's strategic reasoning analysis (Section 5.2, Figure 8) demonstrates that models fail at precisely the capabilities this dimension isolates, even while they succeed at implementation-level tasks (85%+ bash command success rates). Most models make ungrounded edits in over 65% of rounds—they change code without evidence that the change addresses an actual problem. They hallucinate causal explanations for losses after reading only the opening lines of log files that don't even show the decisive moment. They deploy untested code in 50–80% of rounds (depending on model) despite explicit prompts suggesting they run arena simulations. These failures are invisible in specification-based benchmarks because the specification itself provides the grounding: if the issue says "fix the null pointer dereference in `parser.py`," the model doesn't need to diagnose the problem—it's already diagnosed.

**Why this is fundamental rather than incremental.** The paper isn't proposing a slightly harder set of SWE-bench instances. It is arguing—and demonstrating through the behavioral analysis—that there exists a distinct capability (goal decomposition, strategic prioritization, evidence-based diagnosis) that current benchmarks don't test at all, and that this capability is the primary bottleneck for autonomous software engineering. If this thesis is correct, then future progress requires not just better code generation models, but fundamentally different evaluation protocols that surface and measure this strategic dimension. CodeClash provides the first such protocol, and its results (even the best model fails to defeat expert human solutions, strategic reasoning limitations are pervasive across all models) suggest the gap is real and large.

---

### Innovation 2: Codebase-as-Memory as a Diagnostic for Long-Horizon Knowledge Management

A second conceptual contribution is the paper's use of the codebase-as-memory design as an explicit diagnostic tool for testing whether LMs can manage their own context across extended interaction horizons. This is more than a benchmark feature—it is a principled experimental manipulation that reveals a specific failure mode (progressive codebase degradation) that has implications beyond competitive coding.

**What the field did before.** Most agent evaluations operate within a single episode where the model's context window serves as working memory. Within an episode, the model can refer to previous observations, and the scaffold often manages long-term memory through explicit retrieval mechanisms (e.g., embedding-based search over past actions). Some benchmarks study multi-turn interactions (e.g., multi-issue SWE-bench variants), but these typically reset the codebase between tasks or provide explicit task boundaries. The question of how an LM manages *cumulative state* when it must decide for itself what to persist, what to discard, and how to organize persistent information has not been systematically studied.

**What CodeClash reveals.** Because the agent scaffold resets between rounds—no conversation history, no working memory, no retrieval—models must explicitly write to the codebase everything they want to retain. The paper's codebase analyses (Section 5.1, Figures 6-7) show that this requirement surfaces a consistent failure pattern: rather than converging toward a stable, well-organized codebase with reusable tools and consolidated knowledge, models continuously create new files at a nearly linear rate with round count, most of which are never referenced, reused, or modified in subsequent rounds. Claude Sonnet 4.5 averages over 30 created files per 15-round tournament, with a filename redundancy rate of 34% (multiple `analyze_round_13_v2.py`-style files). GPT-5 accumulates particularly many output and temporary files. o3 creates fewer overall files but still shows low reuse ratios.

This behavior is not task-specific—it emerges across all six arenas and all eight models. It suggests a fundamental limitation in how current LMs approach sustained codebase development: they treat each round as a fresh problem, generating new analysis scripts and test files rather than adapting and reusing existing infrastructure. The paper visualizes this through the root-level-clutter vs. file-reuse scatter plot (Figure 49), where 5 of 8 models fall in the undesirable bottom-right quadrant (high clutter, low reuse).

**Why this finding matters beyond CodeClash.** The codebase degradation pattern is not an artifact of competitive coding—it reflects a general challenge for any long-horizon autonomous software engineering system. If an LM is deployed to maintain a production codebase over weeks, producing a new analysis script for every investigation rather than extending existing tooling, the codebase becomes unmaintainable. The behavior observed in CodeClash—treating each round independently, failing to consolidate learnings into reusable infrastructure—is precisely what real-world software engineering practices (refactoring, DRY principles, modular design) evolved to prevent.

The paper's diagnostic move is clever because it doesn't just report this as an observation—it uses the codebase-as-memory design to make the behavior measurable and quantifiable. The filename redundancy metric, the throwaway file count, and the root-clutter ratio are direct operationalizations of codebase health that would be impossible to compute in specification-based benchmarks (where the codebase is modified exactly once to fix a single issue). By forcing models to manage their own memory, CodeClash makes visible a class of failures that existing benchmarks structurally cannot detect.

**Comparison to prior work on code quality.** Existing benchmarks that consider code quality (e.g., evaluating whether generated code follows style guides or passes linters) operate at the level of individual files or functions. CodeClash's analysis operates at the level of *codebase evolution over time*—it asks not whether the code at any point is well-formatted, but whether the process of sustained development produces a coherent, maintainable repository. This is a different axis of quality, and the paper's evidence suggests it is a significant weakness of even the strongest models. The finding that file creation scales nearly linearly with rounds while file reuse remains low is a quantitative signature of this limitation.

---

### Innovation 3: The Competitive Adaptation Challenge as a Stress Test for Strategic Reasoning

The paper's third conceptual contribution is framing competitive adaptation—the need to analyze opponent behavior, anticipate counter-strategies, and adapt one's own approach in response—not merely as a gamification of coding benchmarks but as a principled stress test that reveals the brittleness of model reasoning when evidence is noisy, indirect, and requires causal inference.

**What the field did before.** Prior work has studied model reasoning in game-playing contexts (GameArena, Balrog, PokéChamp) and in competitive coding (various programming contests), but these either have models play directly (testing game-playing rather than engineering capability) or evaluate on fixed problem sets with static correctness criteria. The specific challenge of *competitive software development*—where your code competes against an opponent's code, and both evolve over time—has not been studied as a distinct evaluation paradigm.

**Why competitive adaptation is a distinct capability.** The paper's analysis shows that competitive adaptation requires capabilities that are not tested by any existing benchmark, even those that involve complex reasoning:

1. **Statistical interpretation of noisy feedback.** In CodeClash, winning or losing a round is a noisy signal—a single round outcome depends on 1000 simulations with stochastic elements, and the difference between a 51% win rate and 49% win rate is invisible without careful statistical analysis. The paper's hallucination analysis (Figure 8b) shows that models routinely draw strong causal conclusions from weak evidence: Claude Sonnet 4.5 makes uncorroborated claims about the exact reason a game was lost in over 17% of rounds on average, and this rises to 46% in BattleSnake. Models treat a single round's outcome as definitive feedback about the effectiveness of their changes, failing to account for variance or control for confounds.

2. **Causal attribution from observational data.** To improve, a model must determine *why* it won or lost—not just *that* it won or lost. This requires examining detailed game logs, identifying patterns, and forming causal hypotheses that can be tested through targeted code changes. The paper's groundedness analysis (Figure 8a) shows that most models rarely reach this level: for GPT-5, only 21% of edits are grounded in analysis of previous rounds; for o3, it's 15%; for Qwen3 Coder, 19%. The vast majority of edits are made without evidence that they address an actual performance bottleneck. This is not a coding failure—it's a failure of the scientific method: forming hypotheses from data, testing them, and making evidence-based decisions.

3. **Anticipation and counter-adaptation.** Winning in competitive environments requires not just improving in absolute terms but improving *relative to an opponent who is also improving*. The paper's recovery analysis (Figure 4) shows that even the strongest models struggle mightily with this: after losing a single round, Claude Sonnet 4.5's probability of winning the next round drops from an overall average of 71% to less than one-third. After five consecutive losses, comeback rates fall below 15% for Claude Sonnet 4.5 and below 10% for all other models. This suggests models cannot effectively diagnose why their strategy is failing and pivot to a new approach—they get stuck in local minima, making incremental tweaks that don't address fundamental strategic mismatches.

**The transparent codebase ablation as a diagnostic.** The paper's ablation giving models access to opponent source code (Section 4.1, Appendix D.2) provides a particularly revealing test. GPT-5 achieves the highest win rate despite inspecting opponent code in only 12.8% of rounds—far fewer than Claude Sonnet 4.5 (99.3%) or Gemini 2.5 Pro (52.9%). The paper's conclusion: "frequent inspection of opponent code does not necessarily translate to competitive advantage." This suggests the bottleneck is not information access but *what models do with the information they have*—the strategic reasoning capability to translate observations about an opponent into effective counter-strategies is underdeveloped regardless of how much information is provided.

**Why this is a stress test rather than just a harder task.** Competitive adaptation isn't harder than SWE-bench in the sense of requiring more lines of code or more complex algorithms—it's harder in a qualitatively different way that specifically targets reasoning brittleness. Traditional benchmarks can be solved by pattern-matching: models trained on bug-fix examples learn the statistical regularities of what fixes look like. Competitive adaptation cannot: the right strategy against one opponent may be exactly wrong against another, and what worked in round 3 may fail in round 8 because the opponent has adapted. Success requires genuine causal reasoning about a specific, evolving situation, which current models appear to lack.

The paper doesn't solve this problem—it diagnoses it. The value is in identifying competitive adaptation as a distinct capability that can serve as a targeted stress test for strategic reasoning, separate from implementation skill. Future work on strategic reasoning can use CodeClash as a testbed where the signal (adaptation capability) is isolated from confounds like code generation quality or command-line proficiency.

---

### Innovation 4: The Model-Human Performance Gap as a Boundary on Test-Time Improvement

The paper's most empirically stark finding—that no model defeats expert human solutions, with the best result reaching only rank 57 of 58 on RobotRumble—functions as more than a headline result. It establishes a concrete boundary condition for what current LMs can achieve through iterative self-improvement when the objective is open-ended and feedback is purely observational.

**What this finding is not.** It is not a claim that humans are "better at coding" than LMs in general—LMs already outperform average humans on many specification-based coding benchmarks. It is specifically about the regime where no specification is provided, feedback is noisy and indirect, and improvement must be self-directed across many iterations. In this regime, the gap is not small—it is essentially total. The best model cannot complete a ladder of human solutions that were themselves produced by hobbyist competitors in open programming contests, not professional software engineers working on production systems.

**Why this finding is significant beyond the benchmark.** The CC:Ladder results (Table 2) establish an empirical ceiling on what current models can achieve through the paradigm of iterative self-improvement with observational feedback. This is relevant to a broader set of aspirations in the field: using LMs to autonomously maintain and improve codebases, deploying self-improving agents that get better through interaction with their environment, and building systems that can pursue open-ended goals without human specification.

The finding that models plateau early—most ladder runs terminate because the model fails to win the final round against an opponent, not because it loses a majority—suggests the limitation is not in generating improvements per se, but in *consolidating* them. Models can often become competitive with a given opponent (winning some rounds) but cannot reliably converge to a dominant strategy. This is consistent with the codebase degradation analysis: if each round produces new, untested code that doesn't build on prior work, the model may stumble upon a competitive configuration but cannot refine it to dominance.

**What this implies for the field.** The CC:Ladder result implicitly argues against the hypothesis that scaling current architectures and training paradigms will smoothly produce autonomous software engineering capability. If iterative self-improvement from observational feedback hits a hard ceiling at sub-human performance even in small, self-contained coding games, then deployment in real-world software engineering—where feedback is even noisier, objectives are even more open-ended, and codebases are vastly more complex—faces an even higher barrier. The paper doesn't claim this barrier is insurmountable; it claims it exists and is measurable. Future progress requires either fundamentally better strategic reasoning capabilities, different training paradigms (e.g., reinforcement learning with self-play, which CodeClash is designed to support), or human-AI collaboration architectures that don't require models to handle open-ended objectives autonomously.

This finding is strengthened by its consistency: the relative ordering of models on CC:Ladder broadly matches the main leaderboard (Table 1), and replacing the agent scaffold (mini-SWE-agent → SWE-agent) changes scores by at most 2 ranks across three models and two arenas (Appendix D.6.4). The gap is not an artifact of the interface, the budget, or the specific opponents—it persists across arenas, scaffolds, and models, suggesting it reflects a genuine capability limitation rather than an evaluation confound.

## 5. Experimental Analysis

### Evaluation Methodology

- **Dataset.** The MATH benchmark (Hendrycks et al., 2021), consisting of high-school competition-level math problems. The authors use the specific split from Lightman et al. (2022): 12,000 training questions and 500 test questions. The choice of MATH is deliberate (Section 4): test-time compute is expected to help most when the model already possesses the necessary knowledge and the challenge is drawing complex inferences — mathematical reasoning fits this profile because it requires multi-step logical deduction rather than novel factual recall.

- **Base model(s).** All experiments use PaLM 2-S* (Codey) (Anil et al., 2023). The authors argue this model is "representative of the capabilities of many contemporary LLMs" and sits in a useful regime: non-trivial performance on MATH (roughly 10–19% pass@1 depending on the prompt and sampling configuration) but far from saturation, leaving room for test-time compute to make a difference. For the FLOPs-matched comparison, a second model with approximately 14× more parameters is used as the pretraining-scaled baseline.

- **Metrics.** The primary metric throughout is **MATH test accuracy (%)** — the fraction of the 500 test questions for which the selected final answer matches the ground truth. Answers are graded using the grading function released by Lightman et al. (2022) (Appendix G). When analyzing difficulty-dependent behavior, the paper reports accuracy within each of the five difficulty quintiles separately.

- **Baselines.** The paper uses several baselines:
  - **Majority voting**: select the most common final answer among N sampled solutions (no learned verifier).
  - **ORM best-of-N weighted**: score N solutions with an outcome reward model and apply best-of-N weighted selection.
  - **PRM best-of-N weighted**: score N solutions with the process reward model and apply best-of-N weighted selection.
  - **Parallel sampling** (for revisions): generate N independent solutions from the revision model and select the best via verifier or majority.

- **Generation budget / compute accounting.** The universal unit of test-time compute is one "generation" — one complete sampled answer from the base LLM. For beam search and best-of-N, the budget equals the number of beams or samples N. For lookahead search with k lookahead steps, the cost is N × (k+1) to account for the additional rollout computation (Section 5.3). Budgets are swept across powers of 2, typically from 2⁰ to 2⁹ (1 to 512 generations).

- **Cross-validation / statistical protocol.** To avoid contaminating strategy selection with test-set performance, the authors use **two-fold cross-validation** within each difficulty bin on the 500-question test set. The best strategy is selected on one fold and evaluated on the other, with results averaged (Section 3.2). For the FLOPs-matched comparison, the 14× larger model uses greedy decoding with no test-time compute augmentation. The paper does not report confidence intervals on compute-optimal scaling curves, making it difficult to assess statistical reliability of the observed gains at this sample size.

- **Difficulty estimation cost is unaccounted for.** The method for estimating difficulty — generating 2048 samples per question and averaging either ground-truth correctness (oracle) or PRM final-answer scores (predicted) — is extremely expensive. At 2048 samples per question, difficulty estimation alone consumes more compute than the largest test-time budgets studied (256–512 generations). The paper acknowledges this explicitly (Section 3.2) but does not amortize this cost in any reported efficiency calculation, making the 4× efficiency gains an upper bound rather than a realized deployment figure.

### Main Quantitative Results

#### Search Against PRM Verifiers (Section 5)

**Aggregate search algorithm comparison (Figure 3, left).** Across all 500 test questions with a maximum budget of 256 generations:

- At low budgets (2–8 generations), beam search with M = 4 significantly outperforms best-of-N weighted. For example, at 4 generations beam search (M = 4) achieves roughly 27% accuracy versus roughly 16% for best-of-N weighted — a substantial gap.
- At high budgets (64–256), beam search performance flattens and falls slightly below best-of-N weighted. Best-of-N weighted reaches approximately 38% at 512 generations; beam search (M = 4) plateaus around 34%.
- Lookahead search (both k = 1 and k = 3) generally underperforms at the same generation budget due to its higher per-step cost. The 3-step lookahead variants converge to similar performance as other methods at very high budgets but never surpass them.
- Majority voting trails all verifier-based methods substantially, reaching only about 29% at 512 generations.

**Difficulty-bin analysis for search (Figure 3, right).** The per-difficulty breakdown (beam search M = 4 vs. best-of-N weighted, shown at four budget levels: 4, 16, 64, 256 generations) reveals the core pattern:

- **Bin 1 (easiest):** Beam search accuracy *decreases* from roughly 78% to 77% as the budget goes from 4 to 256, while best-of-N weighted increases from 68% to 88%. This is the clearest evidence of PRM over-optimization — beam search finds solutions that exploit the verifier signal.
- **Bin 2:** Beam search improves modestly (roughly 14% → 32%) but best-of-N weighted improves faster (roughly 14% → 60%), maintaining a clear advantage at high budgets.
- **Bin 3:** Beam search consistently outperforms best-of-N weighted across all budgets, reaching roughly 34% vs. 23% at 256 generations.
- **Bin 4:** Beam search shows the strongest relative advantage, reaching roughly 17% vs. 10% for best-of-N at 256 generations.
- **Bin 5 (hardest):** Both methods hover near 1–3% regardless of budget. No method makes meaningful progress.

**Compute-optimal search (Figure 4).** By selecting the best search strategy per difficulty bin at each budget level:

- At 16 generations, compute-optimal (oracle bins) achieves approximately 27% accuracy, roughly matching PRM best-of-N weighted at 64 generations — a 4× compute reduction.
- At 256 generations, compute-optimal oracle reaches approximately 39.5%, surpassing PRM best-of-N weighted at the same budget (roughly 37%).
- Compute-optimal with predicted difficulty bins tracks the oracle version closely, particularly at lower budgets. The two curves "largely overlap" per the authors (Figure 4), with the predicted version reaching approximately 37% at 256 generations.
- Both compute-optimal variants consistently outperform ORM best-of-N weighted (which peaks around 34% at 512 generations) and majority voting (around 29%).

**PRM vs. ORM (Figure 14, Appendix F).** At 2048 samples, PRM best-of-N weighted achieves approximately 40% accuracy versus roughly 35% for ORM best-of-N weighted and roughly 30% for majority voting. The gap between PRM and ORM widens with the number of samples, confirming the PRM's superior scaling properties.

---

#### Revision Model Results (Section 6)

**Revision model pass@1 trajectory (Figure 6, left).** Starting from approximately 18.2% pass@1 at step 1, the revision model's per-step accuracy improves to roughly 24–25% by steps 15–20, and remains in the 23–25% range out to 64 steps. The model generalizes beyond its 4-step training horizon.

**Sequential vs. parallel (Figure 6, right).** At 64 generations:
- Sequential + best-of-N weighted: approximately 41.5%
- Parallel + best-of-N weighted: approximately 39%
- Sequential + majority: approximately 38%
- Parallel + majority: approximately 35%

Sequential outperforms parallel under both selection mechanisms, with the verifier-based gap (roughly 2.5 percentage points) being slightly narrower than the majority-based gap (roughly 3 points).

**Sequential-to-parallel ratio sweep (Figure 7, left).** For a fixed generation budget, varying the ratio reveals:
- At 256 generations, the optimal ratio is around 2¹ to 2³ (2:1 to 8:1 sequential-to-parallel), achieving approximately 43–44% accuracy.
- Fully parallel (leftmost point) yields approximately 40%.
- Fully sequential (rightmost point) yields approximately 42%.
- At lower budgets (8–32 generations), fully sequential is optimal — the curves are monotonically increasing with the sequential-to-parallel ratio.

**Difficulty-dependent ratio (Figure 7, right).** At a fixed budget of 128 generations:
- **Bin 1:** Performance is essentially flat across all ratios, around 90–92%. Easy questions are insensitive to the allocation strategy.
- **Bin 2:** Slight advantage for higher sequential ratios, approximately 63% at fully sequential vs. 58% at fully parallel.
- **Bin 3:** A clear optimal ratio emerges at moderate sequential-to-parallel values (around 2¹ to 2³), reaching approximately 42% vs. 35% at the extremes.
- **Bin 4:** Similar pattern, with the peak at a moderate ratio achieving roughly 18% vs. 14% at fully parallel.
- **Bin 5:** All ratios produce roughly 2–3% accuracy. No allocation strategy helps.

**Compute-optimal revisions (Figure 8).** Selecting the optimal sequential-to-parallel ratio per difficulty bin:
- At 64 generations, compute-optimal oracle achieves approximately 40%, matching parallel best-of-N weighted at 256 generations — a 4× improvement.
- At 256 generations, compute-optimal oracle reaches approximately 44%, compared to roughly 41% for best-of-N weighted and 37% for parallel-only.
- Compute-optimal predicted bins perform slightly below oracle bins at high budgets (approximately 41% at 256 generations) but still substantially outperform the parallel baseline.
- Notably, the parallel baseline appears to **plateau** around 36–37% at high budgets, while compute-optimal scaling continues to improve, suggesting that the gains from adaptive allocation compound at higher budgets.

---

#### FLOPs-Matched Comparison: Test-Time vs. Pretraining Compute (Section 7)

**Revisions (Figure 9, left; Figure 1, top-right bar chart).** Comparing PaLM 2-S* with compute-optimal revisions against the ~14× larger model:

| Difficulty | R ≪ 1 (0.16) | R ≈ 1 (0.79) | R ≫ 1 (22) |
|---|---|---|---|
| Easy (bin 1) | +11.8% | +3.5% | −11.9% |
| Medium (bin 2–3) | +27.8% | +16.7% | +5.4% |
| Hard (bins 4–5) | +21.6% | −(implied negative) | −37.2% |

(Numbers from the bar chart in Figure 1, top-right. Note: the "easy/medium/hard" groupings in the bar chart differ slightly from the five difficulty bins, aggregating bins for readability.)

At R ≪ 1, test-time compute outperforms the larger model across **all** difficulty levels. At R ≫ 1, it only remains preferable on easy questions, with hard questions showing a −37.2% relative disadvantage.

**PRM search (Figure 9, right; Figure 1, bottom-right bar chart).** The pattern is starker:

| Difficulty | R ≪ 1 (0.16) | R ≈ 1 (0.79) | R ≫ 1 (22) |
|---|---|---|---|
| Easy | +19.1% | +2.2% | +2.0% |
| Medium | 0.0% | −35.3% | −30.8% |
| Hard | −3.6% | −35.3% | −52.9% |

PRM search shows weaker benefits than revisions for the FLOPs-matched comparison, with substantial disadvantages on medium and hard questions even at moderate R values. On easy questions, test-time compute remains preferable across all R regimes, though the margin narrows significantly.

**Figure 9 detail.** The line plots show accuracy per difficulty bin as test-time compute scales. The 14× larger model's greedy performance (stars) is placed at three x-axis positions corresponding to the three R values. Where the compute-optimal scaling line is above the star, test-time compute wins. On bin 1 (purple, topmost line), the scaling line is above all three stars for revisions. On bin 5 (blue, bottommost line), the line is below all three stars and essentially flat near 0–5%, confirming that no amount of test-time compute helps on the hardest problems.

---

### Ablation Studies and Robustness Checks

**PRM aggregation strategy (Appendix E, Figure 13).** Comparing "min," "prod," and "last" step-wise aggregation:
- "Last" achieves roughly 37% at 256 samples.
- "Min" achieves roughly 35%.
- "Prod" achieves roughly 27%.
- ORM achieves roughly 34%.

The "last" aggregation's superiority is notable because it effectively reduces the PRM to ORM-like behavior at aggregation time, yet the PRM still outperforms a separately trained ORM. The authors interpret this as evidence that step-level PRM training provides beneficial representation learning.

**PRM vs. ORM (Appendix F, Figure 14).** The PRM consistently outperforms the ORM, with the gap widening at higher sample counts: at 2048 samples, PRM best-of-N weighted reaches approximately 40% vs. ORM's 35%.

**Revision model verifier choice (Appendix J, Figure 15a).** The base-LM PRM underperforms the revision-specific ORM when scoring revision model outputs, with sequential + base-LM PRM achieving roughly 40% at 64 generations vs. sequential + revision ORM at roughly 42%. This confirms distribution shift as a practical concern.

**Revision history in verifier context (Appendix J, Figure 15b).** Including previous revisions in the ORM's context provides a small improvement over the no-history ablation (approximately 1–2 percentage points at 64 generations), but both variants outperform the parallel baseline, confirming that the sequential sampling benefit is not solely attributable to the verifier seeing more context.

**Oracle vs. predicted difficulty bins (Figures 4, 8, and Appendix C, Figures 11–12).** Both oracle and predicted bins yield qualitatively similar trends across difficulty levels. Predicted bins show slightly lower performance at high budgets in the revision setting (roughly 41% vs. 44% at 256 generations in Figure 8) but essentially identical performance in the search setting (Figure 4). This is the critical robustness check: the compute-optimal strategy works without ground-truth labels.

**Majority voting for revisions (Appendix B, Figure 10).** The sequential-to-parallel ratio trends observed with verifier-based selection are replicated with majority voting: easy questions are insensitive to ratio, hard questions show an optimal intermediate ratio, and fully sequential marginally outperforms fully parallel in aggregate.

**ReST^EM revision model (Appendix K, Figure 16).** An attempt to further optimize the revision model using ReST^EM (Singh et al., 2024) backfires: additional sequential revisions **substantially hurt** performance with this model. At 256 generations, fully sequential performance drops to approximately 33.5% compared to roughly 38.5% at the optimal ratio. The authors hypothesize that the on-policy data collection in ReST^EM exacerbates spurious correlations in revision data, causing the model to fail to learn the revision task properly. This is a notable negative result that highlights the sensitivity of revision training to the data generation procedure.

### Critical Assessment

**Claim 1: Compute-optimal scaling improves efficiency by more than 4× over best-of-N.** Supported for both search (Figure 4: 16 generations matching 64) and revisions (Figure 8: 64 generations matching 256). The 4× figure specifically refers to achieving equivalent accuracy with 4× fewer generations, and the evidence is consistent across oracle and predicted difficulty settings. However, at the highest budgets (256–512), the gains narrow somewhat with predicted difficulty bins (41% vs. 44% at 256 generations for revisions), suggesting the 4× figure is most reliable in the lower-to-moderate compute regime. The unaccounted cost of difficulty estimation (2048 samples per question) means the realized deployment efficiency could be substantially lower than 4×—this is genuinely unaddressed.

**Claim 2: Test-time compute with a smaller model can outperform a 14× larger model.** Supported with sharp conditions. The claim holds convincingly for easy-to-medium problems at R ≪ 1 (e.g., +27.8% on medium revisions, +19.1% on easy PRM search) and weakens progressively as difficulty increases or R grows. At R ≫ 1, test-time compute loses decisively on hard problems (−37.2% for revisions, −52.9% for PRM search). The paper is transparent about these boundaries, which strengthens credibility. However, one caveat weakens the claim: the 14× larger model uses only greedy decoding with no test-time compute augmentation of its own. A fairer comparison would give the larger model some modest test-time compute budget (e.g., best-of-8), which could shift the crossover points significantly. Additionally, the larger model scales only parameters (not data), following the LLaMA paradigm rather than Chinchilla-optimal training—a Chinchilla-optimal larger model would likely be a stronger baseline.

**Claim 3: Efficacy depends critically on prompt difficulty.** Very strongly supported. The difficulty-bin analyses (Figures 3 right, 7 right) show qualitatively different—and sometimes opposite—effects of the same strategy at different difficulty levels. Beam search *hurts* on easy problems (bin 1 accuracy decreases with budget) while it *helps* substantially on medium-hard problems (bin 3–4). Sequential revisions dominate on easy problems but underperform balanced ratios on hard problems. This is the most robust finding in the paper, replicated across search methods, revision strategies, and selection mechanisms. The five-quintile discretization is admittedly coarse—there may be meaningful heterogeneity within bins—but the qualitative pattern is unambiguous and would likely persist under finer-grained difficulty estimates.

**Potential weaknesses in the experimental design:**

- **Single benchmark, single model family.** All results are on MATH with PaLM 2-S*. The authors argue PaLM 2-S* is "representative" but provide no evidence. The PRM's over-optimization behavior, the revision model's training dynamics, and the absolute performance levels could all shift substantially with different model architectures or capabilities. A model with naturally better calibration might exhibit different verifier exploitation thresholds; a model with stronger base MATH performance might push the "hard" bin boundary outward. Replication on at least one other model family (e.g., LLaMA, Gemma) and one other reasoning benchmark is a significant missing piece.

- **Test set of 500 questions is small for difficulty-bin analysis.** Five quintiles of ~100 questions each, split by two-fold cross-validation, means the compute-optimal policy is selected based on ~50 questions per fold per bin. The paper does not report confidence intervals on per-bin accuracy, making it impossible to assess whether the "optimal" strategy selection is robust to sampling noise. A larger test set (or a held-out validation set for policy selection separate from the test set) would strengthen confidence.

- **Difficulty estimation cost undermines practical claims.** Generating 2048 samples per question to estimate difficulty costs more compute than the largest test-time budgets studied. The paper acknowledges this but presents the 4× efficiency figure without amortizing this cost. In a realistic deployment, the total cost would be difficulty estimation + strategy execution, and the former could dominate. Until a cheaper difficulty estimation method is demonstrated (e.g., training a lightweight classifier on question text alone), the reported gains should be understood as an upper bound—possibly a substantially inflated one.

- **The revision model's 38% correct-to-incorrect reversion rate is a major limitation.** Nearly two-fifths of correct answers get "revised" back to incorrect ones during sequential chains. The paper mitigates this with selection across the chain, but this is an inherently wasteful fix—models are spending computation undoing good work. A model trained to recognize when no revision is needed would be a fairer baseline for testing whether revisions genuinely improve the proposal distribution beyond just generating more candidates to select from.

- **Search and revisions are never combined.** The two complementary mechanisms (PRM search and iterative revisions) are studied entirely independently. Applying beam search to revision model outputs—or using the PRM to guide which revision paths to pursue—could yield gains beyond either individual method. The current results therefore represent a lower bound on what combined approaches could achieve, and the independent study design means we cannot assess whether the two mechanisms are additive, synergistic, or redundant.

- **No latency or wall-clock analysis.** Sequential revisions are inherently serial—each revision depends on the prior one—while parallel best-of-N can execute simultaneously given sufficient hardware. A strategy allocating 128 generations as 64 sequential × 2 parallel takes roughly 64× longer wall-clock time than 128 parallel samples. The sequential-heavy strategies favored by the compute-optimal policy on easy problems may be impractical for latency-sensitive deployment regardless of their FLOPs efficiency. The paper does not discuss this tradeoff at all.

- **Missing experiment: combining PRM beam search with the revision model as proposal distribution.** This is the natural next step acknowledged in Section 8 but never run. It would directly test whether the two scaling axes are complementary and whether a unified system could break past the individual performance ceilings observed in Figures 3 and 6.

- **Missing baseline: the 14× larger model with even modest test-time compute.** The FLOPs-matched comparison gives the smaller model compute-optimal test-time strategies but gives the larger model only greedy decoding. A best-of-8 or beam search budget for the larger model would create a much more informative baseline for understanding whether test-time compute substitutes for pretraining or merely complements it. As run, the experiment demonstrates that test-time compute can close some of the gap, but not whether it can close the gap when both sides are allowed to use test-time compute.

- **Missing experiment: cross-model family replication.** Running even a subset of the experiments (e.g., best-of-N and compute-optimal on one model from a different family, such as LLaMA-2 or Gemma) would substantially strengthen the claim that the difficulty-dependent patterns are universal rather than PaLM 2-specific.

## 6. Limitations and Trade-offs

### Difficulty Estimation Cost Is Unaccounted For — the `4×` Efficiency Claim Is an Upper Bound

**The assumption or constraint.** The entire compute-optimal framework requires estimating each question's difficulty *before* selecting a strategy. The paper's method for doing so—generating 2048 samples per question and averaging PRM final-answer scores—is extraordinarily expensive. At 2048 samples per question, difficulty estimation alone consumes 4–8× more compute than the largest test-time budgets studied (256–512 generations). The authors acknowledge this explicitly in Section 3.2:

> "estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

**The consequence.** The reported `4×` efficiency gains over best-of-N are computed *after* difficulty is known, without amortizing the cost of learning it. In a realistic deployment, the total compute cost is `2048 + N` generations, where `N` is the strategy's budget. If a compute-optimal strategy uses `N = 16` generations to match best-of-256 (as in Figure 4), the total cost including difficulty estimation is `2048 + 16 = 2064` generations—more than `8×` *worse* than the best-of-256 it allegedly matches. The `4×` figure is thus an **upper bound** that assumes difficulty estimation is free—an assumption that holds only in regimes where difficulty is precomputed across many queries amortizing the initial cost (e.g., static benchmark evaluation) but fails for any per-query deployment scenario. The actual realized efficiency could be substantially *negative* (the method could cost more than the baseline it outperforms) until a cheaper difficulty estimation method is demonstrated.

**What evidence exists in the paper.** The paper does not measure this cost anywhere. The `4×` efficiency claims (Figures 4, 8) are computed from the strategy execution budget alone. Section 3.2 acknowledges the issue but provides no quantification of its magnitude relative to the claimed gains. There is no experiment that varies the number of difficulty estimation samples and measures how efficiency changes as estimation cost decreases. The predicted difficulty bins (using PRM scores rather than ground-truth) remove the need for correctness labels but not the need for 2048 samples—the generation cost is identical.

**Mitigation status.** The paper does not attempt to mitigate this. Section 8 suggests future work on "pretraining or finetuning models to directly predict difficulty of a question" or "adaptive difficulty estimation" that amortizes estimation into the problem-solving process itself, but neither approach is developed or evaluated. Until such a method is demonstrated, the `4×` figure should be understood as a controlled experimental finding (showing that strategy selection matters) rather than a practical deployment claim.

---

### The Hardest Problems Remain Unsolved — Test-Time Compute Cannot Create Capability

**The assumption or constraint.** The compute-optimal framework implicitly assumes that the base model's proposal distribution contains correct solutions at some non-trivial rate—that there is *something* for search or revisions to find or refine. This assumption fails on the hardest problems (difficulty bin 5), where the base model's pass@1 is near zero. No amount of compute can improve performance if the model cannot produce correct solutions in the first place.

**The consequence.** Across every method—search, revisions, and their compute-optimal combinations—bin 5 accuracy hovers at 1–3% regardless of compute budget (Figure 3, right; Figure 7, right). In the FLOPs-matched comparison, scaling test-time compute provides essentially zero benefit on the hardest problems (Figure 9, bin 5 line is flat near 0–5%), while pretraining a `~14×` larger model does provide gains (the star markers for bin 5 in Figure 9 are above the flat scaling curve for at least some R values). This establishes a sharp boundary condition: **test-time compute can amplify existing capability but cannot create it from nothing**. For problems genuinely outside the model's training distribution or reasoning capacity, pretraining remains the only viable path. This limits the practical applicability of the approach to problem distributions where the base model already has non-trivial pass@1—deployments that include genuinely novel or out-of-distribution queries receive no benefit from increased inference compute.

**What evidence exists in the paper.** Figure 3 (right) shows bin 5 accuracy at 1–3% for all methods at all budgets. Figure 7 (right) shows bin 5 at 2–3% across all sequential-to-parallel ratios. Figure 9 shows the bin 5 scaling line flat and essentially at zero. The FLOPs-matched comparison (Figure 1 bar charts) shows test-time compute *losing* to pretraining on hard problems across nearly all R regimes, with relative disadvantages of −37.2% (revisions) and −52.9% (PRM search) at `R ≫ 1`. The paper is transparent about this limitation (Section 7 takeaway box explicitly states that test-time compute is not beneficial on hard problems), but the pervasiveness of the failure is stark: the hardest quintile of MATH problems is completely untouched by any amount of inference compute, across all methods tested.

**Mitigation status.** The paper does not attempt to mitigate this—it characterizes it as a fundamental boundary. The finding is framed as a discovery rather than a limitation to be fixed: test-time compute and pretraining compute are "not 1-to-1 exchangeable" (Section 7), and some capabilities can only be acquired through pretraining. This is an honest characterization but limits the scope of the approach to problems within the base model's approximate capability range.

---

### Single Benchmark, Single Model Family — Generality Is Unverified

**The assumption or constraint.** All experiments use the MATH benchmark (500 test questions) with PaLM 2-S* as the base model. The authors state they "believe this model is representative of the capabilities of many contemporary LLMs" (Section 4), but this claim is unverified. No experiments are performed on any other reasoning benchmark (e.g., GSM8K, MMLU-Math, TheoremQA), any other model family (e.g., LLaMA, Gemma, Mistral), or any model of substantially different scale (PaLM 2-S* sits in a specific capability band with roughly 10–19% pass@1 on MATH).

**The consequence.** Several aspects of the findings could be model-specific or benchmark-specific:

- **PRM over-optimization behavior** depends on the verifier's calibration, which in turn depends on the base model's output distribution. A model with naturally better calibration or different error patterns might exhibit different difficulty-dependent scaling curves, shifting the "beam search hurts easy problems" threshold or the "no method helps hard problems" boundary.

- **Revision model performance** depends on the base model's in-context learning and self-correction capabilities, which vary substantially across model families. A model with stronger base reasoning might benefit more from sequential revisions or might be more robust to the 38% correct-to-incorrect reversion problem.

- **Absolute difficulty boundaries** (which problems fall in bin 1 vs. bin 5) are defined relative to PaLM 2-S*'s specific pass@1 distribution. A stronger model would push more problems into easier bins, potentially expanding the regime where test-time compute is beneficial. A weaker model would push more problems into harder bins, shrinking the beneficial regime. The qualitative patterns (difficulty-dependent strategy effectiveness) might generalize even if the quantitative boundaries shift, but this cannot be determined from the current experiments.

- **MATH specifically tests symbolic mathematical reasoning** requiring multi-step deduction with clear correctness criteria. It is unclear whether the observed difficulty-dependent patterns generalize to other reasoning domains—coding (HumanEval, MBPP), commonsense reasoning (StrategyQA), scientific QA—or to tasks requiring factual knowledge rather than pure inference. The Monte Carlo rollout PRM training relies on being able to automatically verify answer correctness, which is straightforward for math (exact string match via grading function) but challenging or impossible for open-ended generation tasks.

**What evidence exists in the paper.** None. The paper includes no cross-benchmark or cross-model-family experiments. The `~14×` larger model used in the FLOPs-matched comparison is in the same model family (PaLM 2 variant), so even the scaling comparison is within-family. This is the most significant missing experiment in the paper—a multi-model, multi-benchmark replication study that would establish which findings are universal and which are PaLM 2/MATH-specific.

**Mitigation status.** The paper does not attempt to mitigate this limitation. It is acknowledged only indirectly (Section 4 frames the model as "representative") and deferred to future work. Section 8 suggests extending to "other domains and modalities" as a natural next step but provides no preliminary evidence. A minimal replication on even one additional model family and one additional benchmark (e.g., GSM8K with LLaMA-3) would substantially strengthen the claims of generality.

---

### The `14×` Larger Model Baseline Is Weak — the Pretraining-Test-Time Tradeoff May Be Overstated

**The assumption or constraint.** The FLOPs-matched comparison in Section 7 compares PaLM 2-S* with compute-optimal test-time strategies against a `~14×` larger model using only **greedy decoding** with no test-time compute augmentation. Additionally, the larger model is scaled in parameters only (not data), following the LLaMA paradigm rather than Chinchilla-optimal pretraining where both parameters and data are scaled equally. The authors acknowledge this departure from compute-optimal pretraining (Section 7):

> "We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

**The consequence.** The comparison systematically favors test-time compute in two ways:

1. **No test-time compute for the larger model.** Giving the larger model even a modest test-time compute budget—best-of-8, majority voting, or a single round of beam search—would create a substantially stronger baseline. The current comparison answers the question "can a small model with optimized inference beat a large model with no inference optimization?" but not "can a small model with optimized inference beat a large model that is also allowed some inference optimization?" In many practical deployments, the larger model *would* receive some test-time compute budget, and the crossover points (at what difficulty and what `R` test-time compute loses) could shift significantly.

2. **Parameter-only scaling likely underperforms Chinchilla-optimal scaling.** A model trained with `14×` more FLOPs allocated optimally between parameters and data (following Hoffmann et al., 2022) would likely outperform a parameter-only-scaled model, making the pretraining baseline **weaker than it needs to be**. The reported advantages of test-time compute over pretraining—e.g., +27.8% relative improvement on medium difficulty revisions at `R ≪ 1`—may shrink or reverse against a properly compute-optimal larger model.

**What evidence exists in the paper.** The paper does not test either variant. There is no experiment giving the larger model best-of-8, beam search, or any other test-time compute budget. There is no experiment with a Chinchilla-optimal scaled model. The current comparison is a lower bound on the strength of the pretraining baseline, and the paper acknowledges the data scaling caveat but not the missing test-time compute for the larger model. The reported `~14×` factor is specific to parameter scaling only—a Chinchilla-optimal model with `14×` total FLOPs would have a different (likely smaller) parameter multiplier, changing the framing of the comparison.

**Mitigation status.** The paper acknowledges the Chinchilla caveat explicitly but does not address the missing test-time compute for the larger model at all. Section 8 suggests future work on "compute-optimal pretraining + compute-optimal inference jointly" but provides no preliminary experiments. A more informative baseline would give the larger model a modest, fixed test-time compute budget (e.g., best-of-8 or beam search with `M = 4`) and measure how the crossover points shift.

---

### The Revision Model Has a 38% Correct-to-Incorrect Reversion Rate — Sequential Refinement Is Partially Self-Defeating

**The assumption or constraint.** The revision model is trained exclusively on trajectories where all in-context answers are incorrect, followed by a correct target. It never sees examples of what to do when the current answer is *already correct*. At test time, the model may encounter correct answers in its context (produced during earlier revisions in the chain) and incorrectly "revise" them into wrong answers. Section 6.1 reports:

> "approximately 38% of correct answers get converted back to incorrect ones using a naive approach"

**The consequence.** This means sequential revision chains are partially self-defeating: nearly two-fifths of good answers generated during a chain are subsequently degraded by later revisions. The paper mitigates this by applying majority voting or verifier-based selection across the entire chain rather than always taking the final revision, but this is an inherently wasteful fix—it means the model is spending computation *undoing* its own good work, and the selection mechanism must identify which earlier step in the chain was best. This inflates the effective budget needed: if 38% of correct answers are reverted, the revision chain must generate substantially more correct answers than the final selection needs, wasting a significant fraction of the compute budget on destructive revisions.

Furthermore, the reversion problem calls into question what the revision model is actually learning. If the model frequently turns correct answers into incorrect ones, is it genuinely learning to *refine* solutions (improve quality) or is it learning to *perturb* solutions (make changes regardless of whether they help), with the selection mechanism doing the actual work of identifying which perturbations happened to be beneficial? The paper's difficulty-dependent analysis (Figure 7 right) shows that sequential revisions help most on easy problems—where the initial answer is already often correct—suggesting that the benefit may partly come from generating multiple candidate answers (some correct, some incorrect) and selecting among them, rather than from genuine iterative refinement.

**What evidence exists in the paper.** Section 6.1 reports the 38% figure explicitly. The revision model's pass@1 trajectory (Figure 6, left) shows improvement from 18.2% at step 1 to ~24–25% by steps 15–20, but this is modest relative to the number of revisions, and the curve plateaus rather than continuing to improve—consistent with a dynamic where each step both generates new correct answers and destroys previous ones. Figure 8 shows that compute-optimal revisions plateau at high budgets when using predicted difficulty bins (approximately 41% at 256 generations vs. 44% for oracle), suggesting the reversion problem limits scaling. The ReST^EM experiment (Appendix K, Figure 16) shows that attempting to optimize the revision model further *worsens* the problem—fully sequential performance drops to 33.5% versus 38.5% at the optimal ratio—indicating the training procedure is fragile.

**Mitigation status.** The paper partially mitigates the reversion problem through within-chain selection (majority voting or verifier-based selection across all revisions) rather than using the final revision, but this treats the symptom rather than the cause. A principled solution—training the model to recognize when no revision is needed, or training on trajectories that include correct-to-correct transitions—is not explored. The paper does not report what fraction of the compute budget is "wasted" on destructive revisions, making it difficult to assess how much efficiency is lost to this problem.

---

### Sequential Revisions Introduce Latency That Is Not Accounted For — the `4×` Compute Efficiency May Translate to `64×` Wall-Clock Slowdown

**The assumption or constraint.** The paper measures test-time compute in "generations" (number of complete solution samples), which is a reasonable proxy for total FLOPs but ignores **latency**—the wall-clock time required to execute the strategy. Sequential revision strategies are inherently serial: each revision depends on the output of the previous one, creating a dependency chain that cannot be parallelized. In contrast, parallel best-of-N can execute all N samples simultaneously given sufficient hardware. The paper acknowledges this terrain implicitly (the "sequential vs. parallel" framing in Section 6) but never discusses latency or wall-clock time as a practical constraint.

**The consequence.** The compute-optimal policy often favors heavily sequential strategies—on easy problems (bins 1–2), fully sequential revisions are optimal or near-optimal (Figure 7, right), and on medium problems the optimal ratio is around `2:1` to `8:1` sequential-to-parallel. If a strategy allocates 128 generations as, say, 64 sequential × 2 parallel, the wall-clock time is dominated by the 64 sequential steps, each requiring a full model forward pass. A best-of-128 parallel baseline could execute in the time of a single forward pass (with sufficient hardware), making the sequential strategy roughly `64×` *slower* in wall-clock time despite similar or better FLOPs efficiency.

This tradeoff is critical for deployment decisions. For interactive applications (chatbots, coding assistants, real-time decision systems), latency is often the binding constraint—users will not wait for 64 sequential model calls regardless of accuracy improvements. For batch processing (evaluation, data generation), total throughput matters, and sequential strategies may underutilize hardware that could be processing multiple queries in parallel. The paper's `4×` efficiency claim (matching best-of-256 with only 64 generations) does not translate to a `4×` latency reduction—in fact, the compute-optimal strategy may be substantially *slower* in wall-clock time than the baseline it replaces.

**What evidence exists in the paper.** None—this dimension is entirely unaddressed. The paper never reports wall-clock times for any strategy, never discusses parallelism constraints, and never acknowledges the latency-throughput distinction. The "sequential vs. parallel" analysis in Section 6 is framed purely in terms of generation count, not execution time. The infrastructure description (Appendix A) mentions Docker containerization and tournament runtime (75 minutes per tournament on average, "mostly due to model latency") but does not break down how much of that latency is serial vs. parallelizable.

**Mitigation status.** Not addressed. The paper neither measures latency nor discusses it as a limitation. A latency-aware analysis—reporting both total FLOPs and wall-clock time under realistic parallelism assumptions, and including a latency-constrained compute-optimal policy that limits sequential depth—would make the practical recommendations more actionable. This is a significant gap because many of the strategies the compute-optimal policy recommends (heavy sequential revision on easy problems) are likely impractical in latency-sensitive deployments regardless of their FLOPs efficiency.

## 7. Implications and Future Directions

### How This Work Changes the Landscape

CodeClash shifts the conversation around coding benchmarks from a paradigm where evaluation measures *implementation capability given a specification* to one where it also measures *specification discovery and strategic decision-making given an objective*. This is not an incremental adjustment to existing benchmarks—adding a few more SWE-bench instances with vaguer issue descriptions would not capture the dynamic that CodeClash isolates. It is a **reframing of what "coding capability" means for autonomous systems**, arguing that the ability to determine *what engineering work to pursue* is a distinct capability from the ability to execute that work once specified, and that existing benchmarks are structurally incapable of measuring it.

The magnitude of this shift is best understood by what the paper makes visible that was previously invisible. The strategic reasoning analysis (Section 5.2, Figure 8) reveals that even the strongest models make ungrounded edits in 65–80% of rounds (depending on model), hallucinate causal explanations for losses after reading only fragments of log files, and deploy untested code 50–80% of the time. Yet these same models achieve high scores on specification-based benchmarks. If CodeClash's thesis is correct—that strategic self-direction is the primary bottleneck for autonomous software engineering, not code generation quality—then these failures are not edge cases but the central challenge. Prior benchmarks could not have revealed this because they provide the specification that makes strategic reasoning unnecessary.

The paper also **reconciles a latent tension in the agent-self-improvement literature**. Some prior work demonstrates that LMs can iteratively refine their outputs (self-refinement, self-debugging), while other work finds that "LLMs cannot self-correct reasoning" (Huang et al., 2023) or that self-improvement loops degrade performance. CodeClash's multi-round tournament structure provides a natural explanation: whether iterative improvement works depends on whether the model can correctly *diagnose* what needs improving from noisy, observational feedback. The CodeClash results suggest that the diagnostic step is the primary failure mode—models do change their code substantially between rounds (Figures 5, 30), but those changes are often ungrounded (Figure 8a), based on hallucinated causal attributions (Figure 8b), and deployed without testing (Figure 8c). The positive results in prior self-improvement work likely occurred in settings where the diagnostic signal was cleaner or where the space of valid improvements was constrained by explicit specifications.

This reframing has several consequences for what research directions become more attractive:

- **Specification-free evaluation becomes a priority.** If strategic self-direction is the bottleneck, then benchmarks that provide explicit specifications—no matter how complex the implementation task—cannot measure the capability that matters most. This makes CodeClash's paradigm (competitive arenas with open-ended objectives, log-based feedback, and no task specifications) more attractive as a primary evaluation framework for autonomous SWE-agents, and makes SWE-bench-style benchmarks (with their explicit issue descriptions) less informative about autonomous capability.

- **Verifier and analysis infrastructure become central research objects.** The paper shows that models struggle not to write code but to *know what code to write*—the bottleneck is in interpreting feedback, diagnosing problems, and validating changes. This redirects attention from code generation models toward systems that can reliably analyze execution traces, competition logs, and performance data to produce actionable diagnoses. The paper's finding that models frequently skip validation (Figure 8c) and draw ungrounded conclusions (Figure 8b) suggests that better *software engineering process*—automated testing, continuous benchmarking, regression detection—may matter more than better code generation.

- **The human-AI gap in strategic reasoning becomes a clear target.** The CC:Ladder results (Table 2) establish that even frontier models cannot defeat hobbyist-level human solutions in small, self-contained coding games—the best model reaches only rank 57 of 58 on RobotRumble. This is not a code quality gap (the human solutions are often simple) but a strategic reasoning gap. It provides a concrete, measurable target for improvement that is independent of implementation skill, and the ladder evaluation protocol provides a reproducible methodology for tracking progress.

- **Codebase stewardship emerges as a distinct evaluation axis.** The paper's finding that model-managed codebases degrade over time—files accumulate linearly, reuse is low, clutter is high (Figures 6–7, 49)—establishes that long-horizon codebase maintenance is a distinct capability not captured by single-task benchmarks. This opens a new evaluation dimension beyond functional correctness: can a model maintain a coherent, navigable, reusable codebase across hundreds of editing operations? The metrics the paper introduces (filename redundancy, throwaway file count, root-clutter ratio) provide operationalizations that future benchmarks can adopt.

### Follow-Up Research This Work Enables

**Training models for strategic self-direction via reinforcement learning with self-play in CodeClash arenas.** The paper explicitly positions CodeClash as a potential training ground: "We hope future work around self-improving SWE-agents will consider CodeClash as a training ground" (Section 6). The arenas provide perpetual, non-saturating learning signals (opponents continuously adapt) with clear reward (win/loss), making them suitable for RL approaches that have been bottlenecked by static benchmarks where unit tests provide only binary, saturating feedback. A concrete experiment would fine-tune a base coding model (e.g., a LLaMA or Qwen variant) using outcomes from CodeClash tournaments as reward, with the model playing against past versions of itself (self-play). The key question is whether RL-trained models learn strategic behaviors—log analysis, opponent modeling, validation-before-deployment—that current models lack, or whether they instead learn to exploit arena-specific shortcuts that don't transfer. The paper's finding that solution diversity increases over rounds even against the same opponent (Figure 5) suggests CodeClash generates diverse training trajectories that could support generalization.

**Cheap difficulty estimation via lightweight classifiers or adaptive sampling.** The paper's most significant practical limitation is the cost of difficulty estimation (2048 samples per question, unaccounted in the `4×` efficiency claim). A direct follow-up would train a small classifier—possibly a fine-tuned BERT-style model or a distilled version of the base LM—to predict problem difficulty from the question text alone, using difficulty bin labels generated by the paper's oracle method on a training set. The experiment would measure: (a) the correlation between predicted and oracle difficulty bins, (b) the compute-optimal performance achieved using classifier-based difficulty estimates versus oracle bins, and (c) the total cost including classifier training and inference amortized across queries. Alternatively, an adaptive strategy could start with a small number of samples (e.g., 4–8 per question), use the verifier's score distribution as a preliminary difficulty signal, and allocate the remaining budget accordingly—this amortizes difficulty estimation into the solution process itself.

**Combining PRM-guided search with the revision model as the proposal distribution.** The paper studies search and revisions independently and explicitly notes they were never combined (Section 8). The natural experiment is to use the revision model—which generates candidates conditioned on previous incorrect attempts—as the proposal distribution within beam search. At each step of the search tree, instead of sampling from the base model independently, the model conditions on rejected branches as in-context examples of what didn't work. The PRM's per-step scores then guide which revision paths to pursue. The hypothesis is that search and revisions have complementary strengths: revisions improve candidate quality (better proposal), while search improves candidate selection (better verification). The experiment would measure whether the combination outperforms either method alone at matched budgets, and whether the difficulty-dependent patterns shift—for instance, does beam search with revisions avoid the over-optimization that degrades easy-problem performance with the base model (Figure 3, right, bin 1)?

**Diagnosing and mitigating the codebase degradation phenomenon.** The paper documents that model-managed codebases accumulate redundant, single-use files nearly linearly with round count (Figure 6), with low file reuse ratios (Figure 49) and high filename redundancy (Figure 50). A diagnostic experiment would test whether this behavior is caused by the reset of working memory between rounds (forcing models to re-derive context from the file system), by a lack of pressure toward codebase organization (no cost to clutter), or by models' tendency to treat each round as a fresh problem. Interventions could include: (a) adding explicit instructions about codebase organization to the system prompt, (b) providing a "codebase health" metric as feedback after each round, (c) fine-tuning models on trajectories that demonstrate consolidation and reuse rather than continuous file creation, or (d) imposing a file-count budget that forces models to reuse existing infrastructure. The experiment would measure whether any intervention reduces the linear file-creation trend and whether improved codebase organization correlates with improved competitive performance.

**Extending CodeClash to domains without clean win/loss signals.** The current arenas all have unambiguous winners determined by game outcomes, enabling the 1000-simulation majority-vote winner determination. Many real-world software engineering objectives—improving user engagement, reducing operational costs, increasing code quality—lack such clean binary signals. An extension would develop arenas where success is measured through continuous metrics with noise (e.g., simulated user behavior telemetry, synthetic A/B test results, multi-dimensional performance dashboards) rather than binary win/loss. The experiment would test whether the strategic reasoning limitations identified in Section 5.2 (ungrounded edits, hallucinated causality, untested deployments) are exacerbated when feedback is even noisier and more ambiguous, and whether models can learn to use statistical reasoning (confidence intervals, hypothesis testing) rather than treating single-round outcomes as definitive.

**Cross-model-family and cross-domain replication of CodeClash findings.** The current results are exclusively on frontier API models (Claude, GPT, Gemini, Grok, Qwen) from a narrow time window. A systematic replication would run the same tournament protocol on: (a) open-weight models of varying scales (LLaMA-3 8B through 70B, Qwen2.5-Coder variants) to test whether strategic reasoning scales with model size or saturates, (b) models specifically fine-tuned for agentic coding (e.g., SWE-agent-tuned models, OpenHands-tuned models) to test whether task-specific training improves strategic self-direction or only implementation capability, and (c) at least one non-coding reasoning domain—for instance, adapting CodeClash's tournament protocol to a competitive text-based environment where "codebases" are strategy documents and "arenas" are debate or negotiation simulators—to test whether the strategic reasoning limitations are specific to code or reflect a general limitation of current LMs. The experiment would measure whether the difficulty-dependent patterns (easy problems benefit from exploitation, hard problems benefit from exploration, very hard problems see no improvement), the codebase degradation trends, and the hallucination rates replicate across these conditions.

### Practical Applications and Downstream Use Cases

**Screening for autonomous SWE-agent capability in hiring or procurement.** Organizations evaluating whether to deploy autonomous coding agents for production maintenance currently rely on benchmarks like SWE-bench that measure specification-driven implementation skill. CodeClash provides a complementary signal: can the agent independently determine what engineering work to pursue when given only high-level objectives and noisy feedback? An organization could run a subset of CodeClash tournaments (e.g., 2–3 arenas, 15 rounds each, against fixed baseline opponents) as part of an agent evaluation pipeline. An agent that scores highly on SWE-bench but exhibits the ungrounded-editing and non-validation patterns documented in Figure 8 would be flagged as high-risk for autonomous deployment, even if its implementation skill is strong. The CC:Ladder protocol (Appendix D.6) is particularly suited for this: it provides a standardized, reproducible difficulty curve that doesn't require running full pairwise model tournaments.

**Training data generation for self-improving coding agents.** The paper shows that CodeClash tournaments generate diverse, evolving codebases with rich editing trajectories (models create analysis scripts, test suites, documentation, and multiple strategy variants). These trajectories—sequences of codebase states with associated competition outcomes—are precisely the kind of data needed to train models for strategic software engineering. A concrete pipeline: run large-scale CodeClash tournaments across multiple arenas, collect trajectories where models successfully improved their win rate (positive examples of strategic reasoning), and fine-tune a base model to predict effective editing actions given a codebase state, competition history, and objective. The key advantage over specification-based training data is that CodeClash trajectories include the *discovery* phase—reading logs, writing analysis scripts, testing hypotheses—not just the final code change, providing supervision for the diagnostic and planning steps that current models lack.

**Competitive programming education and AI-assisted strategy development.** CodeClash's arenas are drawn from real competitive programming communities (Battlecode, Battlesnake, Core War, Halite, RoboCode), and the benchmark infrastructure is open-source. This makes it directly usable as an educational tool: students learning competitive programming can use CodeClash to run their bots against LM opponents, analyze the competition logs to understand why they won or lost, and iterate on their strategies. The LM opponents provide a range of difficulty levels (from Qwen3 Coder at ~950 Elo to Claude Sonnet 4.5 at ~1390 Elo, Table 1), creating a natural progression. The trajectory viewer (codeclash.ai) and arena documentation included in starter codebases lower the barrier to entry. More speculatively, the paper's analysis of model strategic failures—hallucinated loss causality, untested deployments, failure to analyze logs—could inform curriculum design: explicitly teaching students to avoid the failure patterns that CodeClash reveals in LMs (e.g., "don't conclude why you lost after reading only the first 10 lines of the log—the deciding moment is usually later") might accelerate human learning of strategic debugging skills.

**Benchmarking long-horizon codebase stewardship for AI coding assistants.** As AI coding assistants (GitHub Copilot, Cursor, Codeium) evolve from single-function completion to multi-file, multi-turn codebase modification, the question of whether they can maintain codebase coherence across extended interactions becomes critical. CodeClash's codebase degradation metrics—file creation rate, file reuse ratio, filename redundancy, root-level clutter—provide a standardized evaluation suite for this capability that no existing benchmark offers. An AI coding tool provider could run a modified CodeClash tournament where each round the tool suggests edits to a human developer (rather than acting autonomously), and measure whether the tool's suggestions lead to codebase degradation or consolidation over a simulated multi-round development process. The paper's finding that even Claude Sonnet 4.5—the strongest model tested—accumulates 52 files in a single 15-round tournament (Figure 52) with `13` `analyze_*` variants and only 2 of those 52 files ever reused (Appendix D.4) suggests this is a genuine risk for any deployment where an AI assistant operates over extended periods without human codebase curation.