Large Language Monkeys: Scaling Inference Compute with Repeated Sampling

🎯 Pitch

Generating hundreds of candidate solutions from weaker LLMs lets them beat state-of-the-art single-shot performance—DeepSeek-Coder-V2-Instruct’s solve rate on SWE-bench Lite jumps from 15.9% to 56%. This coverage scales log-linearly over four orders of magnitude, revealing inference-time scaling laws, but common verification methods plateau without automatic verifiers, leaving untapped potential.

1. Executive Summary

This paper studies how repeated sampling — generating many independent candidate solutions from an LLM and selecting a correct one via a verifier — scales inference compute to improve reasoning performance across multiple tasks and models. The core named concept is coverage (the fraction of problems solved by any generated sample), which the paper observes scales log-linearly with the number of samples over four orders of magnitude, modeled by an exponentiated power law, suggesting inference-time scaling laws analogous to pretraining scaling laws. On SWE-bench Lite, repeated sampling with DeepSeek-Coder-V2-Instruct raises the issue solve rate from 15.9% with one sample to 56% with 250 samples, surpassing the single-sample state-of-the-art of 43% and establishing that weaker models can outperform stronger single-sample baselines when paired with sufficient inference compute. In domains without automatic verifiers — math word problems from GSM8K and MATH — common selection methods (majority voting, reward model best-of-N, weighted majority voting) plateau beyond approximately 100 samples, establishing that coverage gains are actionable only when verification precision scales alongside the sample budget.

2. Context and Motivation

The Core Problem: Inference Compute as an Untapped Scaling Axis

The fundamental question this paper tackles is deceptively simple: what happens if we let LLMs try a problem many times instead of just once? For years, the dominant strategy for improving LLM capabilities has been scaling training compute — building bigger models, training on larger datasets for longer durations. The scaling laws pioneered by Kaplan et al. (2020) and Hoffmann et al. (2022) provide principled frameworks for predicting how performance improves as pretraining compute increases, giving model developers confidence that their investments in training infrastructure will yield predictable returns.

But when it comes to inference — the moment when a trained model actually solves problems — the field has largely defaulted to a single-attempt paradigm. Users prompt a model, it generates one response, and that response is taken as the answer. This paper argues that this single-attempt constraint represents a massive, underexploited opportunity: inference compute is an independent scaling axis that has been left almost entirely uncharacterized.

The paper frames this gap through two interconnected properties that determine whether repeated sampling is useful:

Coverage: As we increase the number of independent samples, what fraction of problems can we solve using any of the generated samples? This measures the model's ability to eventually stumble upon a correct answer if given enough attempts.
Precision: Given a collection of generated samples, can we reliably identify which ones are correct? This measures our verification capability.

Both properties must hold for repeated sampling to translate into practical gains. Without coverage, no amount of selection can produce correct answers from incorrect candidates. Without precision, we cannot distinguish the correct solution from the haystack of incorrect ones — the "needle in a haystack" problem.

The paper doesn't invent new model architectures or training procedures. Instead, it provides the first systematic characterization of how repeated sampling behaves across tasks, models, and sample budgets, establishing inference-time scaling as a first-class object of study.

Why This Problem Matters: A New Degree of Freedom in LLM Deployment

The practical and theoretical implications of understanding inference-time scaling are substantial. The paper highlights several dimensions where this understanding changes the landscape:

Weaker models can outperform stronger ones. The current LLM market is structured around a performance hierarchy: frontier models (GPT-4o, Claude 3.5 Sonnet) sit at the top and command premium pricing, while open-source models (Llama, DeepSeek, Gemma) trade off capability for lower cost. Repeated sampling disrupts this hierarchy. As the paper demonstrates, a weaker model — when allowed many attempts — can exceed the single-sample performance of a much stronger one. On SWE-bench Lite, DeepSeek-Coder-V2-Instruct (an open-source model) achieves 56% issue resolution with 250 samples, exceeding the single-sample state-of-the-art of 43% achieved by a mixture of GPT-4o and Claude 3.5 Sonnet (Figure 2, Table 1). This is not an incremental improvement — it's a paradigm shift in how capabilities are accessed.

Cost-performance tradeoffs become multi-dimensional. When deciding which model to deploy, practitioners have traditionally balanced two levers: model quality and per-token cost. Repeated sampling introduces a third lever: the sample budget. As Table 1 demonstrates, sampling DeepSeek-Coder-V2-Instruct five times ( $10.80 total cost) solves more SWE-bench issues than a single attempt from GPT-4o ($ 39 total cost, 3.6× more expensive) or Claude 3.5 Sonnet ($51 total cost, 4.7× more expensive). This creates an entirely new optimization landscape where the cheapest route to a target accuracy may involve a weaker model with many samples rather than a stronger model with one.

Inference systems become a distinct workload. The paper notes that repeated sampling is fundamentally different from interactive chatbot serving (Section 5). Production chatbot deployments prioritize low latency, which often forces low per-device batch sizes and reduces hardware utilization. In contrast, repeated sampling — generating many completions for the same prompt — can emphasize throughput over latency, exploit high batch sizes, and benefit from specialized attention optimizations that share prefix computations across sequences. This means that the inference cost of repeated sampling can be lower than naively making many parallel API requests, further strengthening the economic case for sampling-based approaches.

A bridge to training-inference tradeoffs. The paper situates inference-time scaling within the broader conversation about total compute allocation. Just as Hoffmann et al. (2022) established that pretraining compute should be optimally split between model size and data quantity, this work opens the question: given a total compute budget (training + inference), what fraction should be spent on each phase? If inference-time compute can compensate for reduced training compute on certain problems, the optimal allocation may look very different from current practice. The paper's observation that coverage scales log-linearly with the sample budget (Section 3) provides the first functional form for reasoning about this tradeoff.

Where Prior Approaches Fall Short

The paper identifies several specific gaps in prior understanding that motivate its systematic study.

Repeated sampling has been demonstrated but never characterized. Prior work has shown that repeated sampling works in specific domains. AlphaCode (Li et al., 2022) famously used up to a million samples per problem for competitive programming, establishing that coding performance continues to scale with massive sample budgets. CodeLlama (Rozière et al., 2023) and other code-focused models have adopted pass@k as a standard evaluation metric, implicitly acknowledging the value of multiple attempts. Greenblatt (2023) observed log-linear scaling of ARC puzzle performance as sample count increased.

However, these findings were domain-specific, model-specific, and anecdotal. No prior work asked the systematic questions: Does this scaling behavior generalize across tasks and model families? What is the functional form of the relationship between coverage and sample budget? How small a model can benefit? How many samples are needed to see substantial gains? The field lacked a unified empirical picture, making it impossible to draw general conclusions or make principled decisions about inference compute allocation.

The precision problem is understood separately from coverage. The paper distinguishes itself by treating coverage and precision as co-equal, interacting properties. Prior work on verification — through majority voting (Wang et al., 2023), reward models (Cobbe et al., 2021; Lightman et al., 2023), or outcome-based verifiers — largely studied verification in isolation, without examining how verifier performance changes as the sample budget increases and correct answers become increasingly rare within the candidate pool. The paper's key insight in Section 4 is that common verifiers saturate: majority voting and reward models plateau around 100 samples even though coverage continues improving to 10,000 samples (Figure 7). This means that prior work, which typically operated in low-sample regimes (tens of samples), never encountered the scaling bottleneck that becomes the dominant limitation at larger budgets. The gap between coverage and verifier performance widens with additional samples — a finding that reframes verification as the critical bottleneck for scaling inference compute.

Inference-time scaling laws are absent. Training scaling laws (Hestness et al., 2017; Kaplan et al., 2020; Hoffmann et al., 2022) are cornerstones of modern LLM development. They provide predictive relationships between compute, model size, data quantity, and loss that guide investment decisions and architecture choices. The paper draws a direct parallel: just as these laws characterize training compute scaling, the field needs analogous laws for inference compute. The paper's exponentiated power law model (Equation 3) — $c \approx \exp(a k^b)$ — is a first attempt at such a characterization. This matters practically because without a functional form, it's impossible to extrapolate: if a model achieves 30% coverage with 100 samples, what budget is needed for 90%? The paper's scaling analysis answers this question for specific model-task pairs and provides a template for answering it in general.

The tradeoff between model size and sample count is unexplored. A practitioner choosing between Llama-3-8B and Llama-3-70B faces a cost-quality tradeoff. But with repeated sampling, this becomes a three-way tradeoff: model size, sample count, and total cost. The paper's FLOPs-matched analysis (Section 2.3, Figure 4) reveals that the optimal choice depends on the task and budget. On MATH, the smaller 8B model always achieves higher coverage than the 70B model at fixed total FLOPs because its lower per-token cost allows more samples. On CodeContests, the larger model dominates because the capability gap is too wide for extra samples to bridge. Prior work provided no framework for making this determination on new tasks.

How This Paper Positions Itself

The paper does not claim to invent a new method. It explicitly frames itself as a characterization study — an empirical investigation that establishes baselines, identifies patterns, and raises questions for future work. This is a deliberate positioning choice. Rather than adding another technique to the rapidly growing portfolio of test-time compute methods (tree search, self-verification, iterative refinement), the paper steps back and asks: what is the simplest possible approach — independent repeated sampling with a verifier — capable of, and where does it break?

This positioning is important because it makes the paper's contributions foundational rather than competitive. The exponentiated power law model, the observation that coverage curves from different model sizes are horizontally shifted S-curves with similar slopes (Section 3.2, Figure 6), and the characterization of verifier saturation (Section 4.1) are findings that apply regardless of which specific technique is used to generate or verify samples. They describe properties of the underlying probability distributions from which models sample, making them relevant to any method that operates on those distributions.

The paper also positions itself as a counterpoint to the single-attempt mindset that dominates both LLM evaluation benchmarks and production deployments. By showing consistent, large gains across five diverse tasks — competitive programming (CodeContests), formal proof verification (MiniF2F-MATH), software engineering (SWE-bench Lite), and math word problems (GSM8K, MATH) — the paper makes the case that single-attempt evaluation fundamentally underestimates what models can do. A model scoring 15.9% on SWE-bench Lite with one attempt appears weak, but the same model scoring 56% with 250 attempts is state-of-the-art. This reframes how the community should think about model capabilities: not as fixed quantities, but as functions of the inference budget.

The paper explicitly acknowledges the limitations of its scope. It studies only independent repeated sampling — each attempt is generated from the same prompt with no feedback from previous attempts, no iterative refinement, and no adaptive allocation of samples across problems. The authors flag this as a deliberate simplification (Section 5) and suggest that richer strategies — conditioning on previous solutions, using execution feedback, or adaptively allocating samples — could yield further gains. By establishing the baseline, the paper creates a reference point against which future, more sophisticated methods can be compared.

Finally, the paper positions itself relative to the verification literature by identifying a specific, measurable bottleneck: the saturation of majority voting and reward models around 100 samples. Rather than simply observing that verification is hard, the paper quantifies how hard it is as a function of sample budget (Figure 7) and connects this to the distribution of correct sample frequencies across problems (Figure 8). This transforms verification from a binary "solved or not" question into a scaling problem with its own characteristic curve, suggesting that verification, like generation, may benefit from its own scaling laws.

3. Technical Approach

3.1 Reader Orientation

This paper is an empirical characterization study, not a new method proposal. The "system" being studied is the simplest possible inference-time scaling strategy: generate many independent candidate solutions from a frozen LLM by sampling with non-zero temperature, then use a verifier to identify and select correct solutions from the generated collection. The core idea is that inference compute — measured as the number of generated samples — is an independent scaling axis that can be systematically characterized, and that the relationship between sample count and problem-solving coverage often follows a log-linear (specifically, exponentiated power law) pattern across multiple tasks and models.

3.2 Big-Picture Architecture (Diagram in Words)

The experimental pipeline has four major stages, but it is useful to think of it as two parallel tracks depending on whether the task provides automatic verification:

Track 1 — Tasks with automatic verifiers (CodeContests, MiniF2F-MATH, SWE-bench Lite):

Prompt construction: A problem description is combined with few-shot examples (task-specific, see Appendices A.1–A.4) and any required scaffolding (e.g., Moatless Tools for SWE-bench) to form a complete prompt.
Independent sampling: The same prompt is fed to the LLM $N$ times with a fixed positive temperature (0.5–1.8 depending on task). Each of the $N$ forward passes produces one complete candidate solution. The $N$ samples are generated independently — no sample conditions on or receives feedback from any other sample.
Automatic verification: Each candidate solution is checked for correctness using task-specific verifiers: unit test suites for SWE-bench Lite, hidden input-output test cases for CodeContests, and the Lean4 proof checker for MiniF2F-MATH. A solution is scored as pass (1) or fail (0).
Coverage computation (pass@k): For each problem, the fraction of correct solutions among all $N$ samples is computed. The unbiased estimator from Chen et al. (2021) is used to estimate coverage (pass@k) for any $k \leq N$ . If at least one sample passes verification, the problem is considered "covered" under that budget.

Track 2 — Tasks without automatic verifiers (GSM8K, MATH):

Stages 1 and 2 are identical, but stages 3–4 differ because no ground-truth verifier exists at inference time:

Answer extraction: For each generated chain-of-thought solution, a regular expression extracts the final answer (the string following "####" for GSM8K; the output of the minerva_math extraction functions from LM-Eval for MATH). These extracted answers are strings that can be compared.
Oracle verification (for coverage bounds): To compute coverage, the extracted answer is compared against the ground-truth answer using exact string matching and the is_equiv function from LM-Eval. This uses the ground-truth label and is therefore an upper bound — it tells us what fraction of problems could be solved if we had a perfect verifier.
Practical verification (for precision measurement): Three methods are tested on the same sample collections:
- Majority voting: Select the most common final answer across the $k$ samples.
- Reward model + Best-of-N: Score each sample using ArmoRM-Llama3-8B-v0.1, select the answer from the highest-scoring sample.
- Reward model + Majority vote: Sum reward model scores per unique answer, select the answer with the highest total score.
  The success rate with each method is computed for increasing $k$ , revealing where practical verification saturates relative to the oracle coverage bound.

The two tracks share the same core question: as we increase the number of independent samples, how does the fraction of problems with at least one correct solution (coverage) change, and can we reliably identify those correct solutions (precision)?

3.3 Roadmap for the Deep Dive

First, the definition and estimation of coverage (pass@k) — what it mathematically is, why it is the core metric, how the unbiased estimator works, and why the paper uses this formulation rather than simpler alternatives. This is the foundation that all subsequent analysis builds on.
Second, the experimental setup and sampling configuration across all five tasks — prompt construction, temperature selection, few-shot example choices, token limits, and the reasoning behind these choices. Understanding the sampling apparatus is essential because coverage behavior depends on the sampling distribution.
Third, the scaling law formulation — how the paper models the relationship between coverage and sample count using an exponentiated power law, the fitting procedure, the interpretation of parameters $a$ and $b$ , and what this functional form implies about the nature of inference-time scaling.
Fourth, the FLOPs-matched comparison framework — how inference FLOPs are approximated for dense transformer models, how the tradeoff between model size and sample count is analyzed, and what the FLOPs formula reveals about when smaller models with more samples are cost-effective.
Fifth, the verification methods for non-automatable tasks — the mechanics of majority voting, reward model best-of-N, and reward model weighted majority voting, how they are evaluated across random subsets, and the statistical protocol for measuring precision scaling.

3.4 Detailed, Sentence-Based Technical Breakdown

This is an empirical characterization paper whose core idea is that inference compute — operationalized as the number of independent samples drawn from a frozen LLM — is a scaling axis that can be systematically measured, modeled, and compared across tasks and models.

Coverage (pass@k): Definition, Estimation, and Rationale

The central metric of the paper is coverage: the fraction of problems for which at least one generated sample among $k$ attempts is correct. In coding settings, this is identical to the established pass@k metric, where $k$ denotes the number of samples generated per problem. The paper uses the notation "coverage" and "pass@k" interchangeably across all five tasks, extending the pass@k concept to math word problems and formal proofs where a "pass" means that the extracted final answer matches the ground truth (verified by the oracle) or that the generated proof is accepted by the Lean4 checker.

Why coverage is the right metric for this study. Coverage isolates the model's generative capability from the orthogonal problem of verification. It answers the question: "If we had a perfect verifier, what fraction of problems could the model solve by generating $k$ independent attempts?" This decomposition is essential because it lets the paper separately characterize (1) how generation quality scales with samples and (2) how verification quality limits the translation of coverage into real performance. Without this separation, poor verification would confound the measurement of generation scaling, making it impossible to determine whether scaling failures are due to the model not producing correct solutions or the verifier failing to identify them.

The paper adopts the unbiased estimation formula from Chen et al. (2021) to compute pass@k from a finite number of generated samples $N$ , where $N \geq k$ . For each problem index $i$ in the test set, let $C_i$ be the number of correct samples among the $N$ total generated samples. The pass@k estimate is:

$\text{pass@k} = \frac{1}{\text{\# of problems}} \sum_{i=1}^{\text{\# of problems}} \left(1 - \frac{\binom{N - C_i}{k}}{\binom{N}{k}}\right)$

where $\binom{N - C_i}{k}$ is the number of ways to choose $k$ samples from the $N - C_i$ incorrect samples, $\binom{N}{k}$ is the number of ways to choose any $k$ samples from $N$ , and the fraction $\frac{\binom{N - C_i}{k}}{\binom{N}{k}}$ is therefore the probability that all $k$ chosen samples are incorrect.

What the formula computes. For a single problem, the term $\left(1 - \frac{\binom{N - C_i}{k}}{\binom{N}{k}}\right)$ is the probability that, when drawing $k$ samples without replacement from the $N$ available samples, at least one of the $k$ is correct. This is exactly the definition of pass@k for that problem. The outer summation averages this probability across all problems in the test set, producing the expected fraction of problems solved under a budget of $k$ samples. The formula is computed for each $k$ of interest, where $k \leq N$ .

Why this form — the unbiased estimator and without-replacement correction. A naive approach would compute pass@k by repeatedly drawing random subsets of size $k$ from the $N$ samples and reporting the empirical fraction where at least one correct sample appears. This is an unbiased but high-variance estimator. The formula above is the exact analytical expectation of that process, eliminating sampling variance entirely for a given set of generated samples. The use of binomial coefficients $\binom{\cdot}{\cdot}$ rather than powers $(1 - C_i/N)^k$ correctly accounts for sampling without replacement — if $N$ is small relative to $k$ , the without-replacement correction becomes important because the probability of never drawing a correct sample decreases faster than the with-replacement approximation would suggest. The paper mentions using the "numerically stable implementation" from Chen et al. (2021), which avoids computing large binomial coefficients directly (likely using log-space computation or iterative product formulations) to handle the large $N = 10{,}000$ regime.

What $C_i$ captures and why $N$ must be large. The quantity $C_i$ is the empirical number of correct samples for problem $i$ out of $N$ total generations. $C_i / N$ is therefore a Monte Carlo estimate of the model's probability of generating a correct solution for that problem under the given sampling configuration (temperature, prompt, etc.). When $N = 10{,}000$ , this empirical probability is estimated with standard error proportional to $\sqrt{p(1-p)/N}$ , which is at most $\sqrt{0.25/10000} = 0.005$ — giving reasonable precision even for very rare correct solutions (e.g., $p = 0.001$ has standard error $\approx 0.0003$ ). This large $N$ is what enables the paper to estimate coverage deep into the tail of the distribution, where correct solutions are generated for only a fraction of a percent of samples.

The extrapolation property. Because the formula computes pass@k analytically from the $N$ generated samples, the paper can report pass@k for any $k \leq N$ without generating additional samples. For example, from $N = 10{,}000$ samples per problem, the paper reports coverage at $k = 1, 2, 4, 8, \dots, 10{,}000$ . This is economically essential — generating 10,000 separate experiments for each $k$ would be computationally prohibitive. The unbiased estimator makes the entire scaling curve derivable from a single large generation run per model per task.

Relationship to the success rate with a verifier. When a perfect verifier is available (automatic verification tasks), the success rate equals coverage because every correct sample is identified. When only imperfect verifiers are available (GSM8K, MATH), coverage is an upper bound on achievable success rate. The gap between the two — coverage minus practical verifier success rate — measures the verifier's precision loss and is the paper's primary tool for diagnosing verification bottlenecks (Section 4).

Experimental Setup: Tasks, Models, and Sampling Configurations

The paper evaluates repeated sampling across five tasks and multiple model families. Each task-model pair has a specific sampling configuration that is held constant across all sample budgets. Understanding these configurations is essential because coverage behavior depends on the sampling distribution induced by temperature, prompt format, and few-shot examples.

The five tasks and their verification mechanisms:

GSM8K: Grade-school math word problems. The paper evaluates on a random subset of 128 problems from the GSM8K test set. There is no automatic verifier — correctness is determined by comparing the extracted numerical answer to the ground truth using exact string matching and the is_equiv function from LM-Eval. Coverage uses this ground-truth comparison as an oracle verifier; practical verification uses majority voting and reward model methods.
MATH: Competition-level math problems (harder than GSM8K). Evaluated on 128 random test-set problems. Same verification setup as GSM8K: oracle ground-truth comparison for coverage, majority voting and reward models for practical verification. The minerva_math functions from LM-Eval are used for answer extraction.
MiniF2F-MATH: Mathematical theorems from the MATH dataset that have been formalized into Lean4, a proof-checking language. Evaluated on the 130 test-set problems derived from MATH. The Lean4 proof checker serves as an automatic verifier — any proof that the checker accepts is correct. Coverage equals success rate.
CodeContests: Competitive programming problems from the AlphaCode dataset. Each problem has a natural language description and hidden input-output test cases. Evaluated on the 140 test-set problems without image tags. The hidden test cases serve as an automatic verifier — solutions are executed against concatenated public, private, and generated tests. Coverage equals success rate. Models are required to write solutions in Python3.
SWE-bench Lite: Real-world GitHub issues from 12 repositories, subset to 300 problems. Each problem requires editing files in a codebase snapshot. The repository's unit test suite serves as an automatic verifier (though with documented flakiness, see Section 4.2.1). Evaluated with DeepSeek-Coder-V2-Instruct equipped with the Moatless Tools agent framework. One "sample" for SWE-bench is an entire multi-turn trajectory where the model interacts with the codebase through tools for file navigation and editing.

Model families and sizes evaluated:

Llama 3: Llama-3-8B (base), Llama-3-8B-Instruct, Llama-3-70B-Instruct. Used for CodeContests, MiniF2F-MATH, GSM8K, and MATH with $N = 10{,}000$ samples per problem.
Gemma: Gemma-2B and Gemma-7B. Used for MATH and CodeContests with $N = 10{,}000$ samples.
Pythia: Eight sizes from 70M to 12B (Pythia-70M, 160M, 410M, 1B, 1.4B, 2.8B, 6.9B, 12B). Used for MATH and CodeContests with $N = 10{,}000$ samples.
DeepSeek-Coder-V2-Instruct: Used exclusively for SWE-bench Lite with $N = 250$ samples per problem (restricted due to the high cost of multi-turn agent trajectories).

Task-specific sampling configurations:

For CodeContests, the paper uses temperature 0.6, top-p 0.95, max generation length 1024 tokens, and two few-shot examples randomly sampled per problem from the training set. These settings follow CodeLlama (Rozière et al., 2023), which the paper cites as precedent. The prompt format wraps the problem description and few-shot examples in a standard "Q: [problem] A: [solution]" template. Solutions must parse input from stdin and write output to stdout.

For MiniF2F-MATH, the paper uses temperature 0.5 with no nucleus sampling (no top-p), max generation length 200 tokens, and five fixed few-shot examples from the validation set. The few-shot theorems are mathd_algebra_116, amc12_2000_p5, mathd_algebra_132, mathd_algebra_11, and mathd_numbertheory_84. The prompt includes Lean4 header imports, the few-shot theorems (with names replaced to theorem1 through theorem5), and the target theorem (named theorem6). Theorem names are anonymized to prevent the model from exploiting information leaked by the theorem name. The Lean4 proof checker (lean-dojo 1.1.2 with Lean 4.3.0-rc2) is used with a 10-second timeout per tactic step for verification.

For MATH, the paper uses temperature 0.6 with no nucleus sampling, max generation length 512 tokens, and five fixed few-shot examples from Lewkowycz et al. (2022). The prompt format is "Problem: [text] Solution: [chain-of-thought]". Answer extraction uses minerva_math functions from LM-Eval to identify the final boxed or stated answer.

For GSM8K, the paper uses temperature 0.6 with no nucleus sampling, max generation length 512 tokens, and five few-shot examples randomly sampled per problem from the GSM8K training set. The prompt format follows the standard "Question: [text] Answer: [chain-of-thought] #### [number]" pattern. Answer extraction uses a regular expression that captures the string following "####". The extracted answer is compared to ground truth using exact string matching supplemented by the is_equiv function from LM-Eval.

For SWE-bench Lite, the paper uses DeepSeek-Coder-V2-Instruct with the Moatless Tools agent framework at commit a1017b78e3e69e7d205b1a3faa83a7d19fce3fa6, Voyage AI embeddings for retrieval, and a temperature of 1.6 — selected via a sweep over temperatures {1.0, 1.4, 1.6, 1.8} on a random 50-problem subset of the test set. Each "sample" is one complete agent trajectory where the model navigates the codebase, reads files, and writes patches. All 250 samples per problem are independent trajectories. The paper notes that 34 of the 300 SWE-bench Lite problems have flaky test suites (listed in Appendix B, Table 3); for these problems, each candidate solution's test suite is run 11 times and majority voting across runs determines pass/fail.

Why these specific temperatures and configurations. The paper does not perform extensive hyperparameter optimization per budget — a fixed temperature is used for all sample counts. This is a deliberate choice that reflects the paper's characterization goals: the question is not "what is the optimal temperature for each budget?" but "given a reasonable sampling setup, how does coverage scale with budget?" The temperatures are chosen to be positive (to ensure diversity among samples) and roughly consistent with prior work in each domain (0.6 for code and math following prior literature, 1.6 for SWE-bench where the multi-turn agent trajectory benefits from higher stochasticity). The paper uses no nucleus sampling (no top-p filtering) for GSM8K, MATH, and MiniF2F-MATH, while using top-p=0.95 for CodeContests following the CodeLlama setup. These differences mean that cross-task comparisons of absolute coverage numbers should account for different sampling distributions, but within-task scaling trends are self-consistent.

The Exponentiated Power Law Model for Inference-Time Scaling

The paper proposes that the relationship between coverage $c$ (the pass@k value) and the number of samples $k$ can be modeled using an exponentiated power law:

$\log(c) \approx a k^b$

which, when exponentiated, becomes:

$c \approx \exp(a k^b)$

where $a \in \mathbb{R}$ is a scale parameter (always negative in practice, since coverage is bounded above by 1), $b \in \mathbb{R}$ is a shape parameter controlling the curvature of the scaling, $k$ is the number of samples, and $c \in [0, 1]$ is the predicted coverage (pass@k) at that sample count.

What this equation computes. Given two fitted parameters $a$ and $b$ , the model predicts the expected coverage at any sample count $k$ . On a log-coverage scale, the relationship is $a k^b$ , which is a power law in $k$ . When $b < 0$ , $\log(c)$ becomes less negative (i.e., coverage increases) as $k$ grows, but the rate of increase slows — this is the "diminishing returns" pattern characteristic of scaling laws. When $b = 0$ or $b > 0$ , the model would predict coverage decreasing or increasing without bound, respectively, which are degenerate cases not observed in the paper's experiments.

Why this functional form. The paper cites the GPT-4 technical report (OpenAI et al., 2024), which observed that the mean log pass rate on coding problems follows a power law relationship with training compute. The inference-time analog replaces "training compute" with "number of samples" $k$ and models the log of coverage rather than the log of loss. A power law in log-space is a natural starting point because training scaling laws consistently find power-law relationships between compute and performance metrics (Hestness et al., 2017; Kaplan et al., 2020; Hoffmann et al., 2022). The exponentiated form $\exp(a k^b)$ ensures that predicted coverage is always positive; the parameter $a$ (being negative) ensures coverage is bounded above by 1 as $k \to \infty$ .

An alternative functional form would be a sigmoid (logistic) curve, since coverage is naturally bounded between 0 and 1 and can exhibit S-shaped saturation. However, the paper's observed coverage curves (Figure 3, Figure 5) do not show pronounced sigmoid saturation within four orders of magnitude for most model-task pairs — coverage continues to increase log-linearly. The exponentiated power law is simpler (two parameters vs. three or four for a sigmoid), captures the dominant log-linear trend, and connects conceptually to training scaling laws.

Fitting procedure. The paper fits the parameters $a$ and $b$ by:

Sampling 40 points evenly spaced on a logarithmic scale from $k = 1$ to $k = 10{,}000$ (for tasks where $N = 10{,}000$ ) and removing any duplicate $k$ values.
Computing the empirical coverage at each of these $k$ using the unbiased pass@k estimator (Equation 1).
Using SciPy's curve_fit function — which performs non-linear least squares optimization — to find the $(a, b)$ pair that minimizes the squared error between the model $\exp(a k^b)$ and the empirical coverage at each of the 40 sampled points.

The paper reports fitted curves for multiple model-task pairs (Figure 5, Appendix C.2, Figure 10) along with the mean absolute error between the fitted curve and the empirical coverage across 100 evenly-spaced points on the log scale. The reported errors are typically small (0.002–0.03 are typical values), indicating that the power law provides a reasonable fit, though the paper acknowledges exceptions — most notably Llama-3-8B-Instruct on MiniF2F-MATH, where the fit is visibly poorer (Figure 5, bottom-right panel, error 0.0297 ± 0.0157).

Interpretation of parameters $a$ and $b$ . The parameter $a$ controls the intercept in log-space: more negative $a$ values mean lower coverage at small sample counts. For example, Gemma-2B on CodeContests has $a = -8.54$ (Figure 10), reflecting its near-zero pass@1 of 0.02%. Llama-3-8B-Instruct on GSM8K has $a = -0.27$ , reflecting high pass@1. The parameter $b$ controls the curvature of scaling: more negative $b$ (more negative than, say, -0.5) means that log-coverage increases quickly at small $k$ and then saturates; $b$ close to 0 means that log-coverage increases very slowly (nearly flat). On CodeContests, $b \approx -0.11$ to $-0.14$ for most models, indicating slow but steady improvement. On MATH, $b \approx -0.35$ to $-0.46$ for instruction-tuned models, indicating faster initial gains that gradually diminish.

What "inference-time scaling laws" means in this context. The paper does not claim that the exponentiated power law is a universal law of nature. Rather, it observes that for many model-task pairs, this simple two-parameter function fits the empirical coverage curves well over four orders of magnitude in sample count. This is similar in spirit to how training scaling laws are empirical regularities — not derived from first principles, but consistently observed and practically useful for extrapolation. The paper explicitly notes that the fit is not perfect (MiniF2F-MATH is a notable exception) and that more complex forms may be needed for universal characterization. The key contribution is the observation that coverage often follows this pattern, not the claim that the two-parameter model is optimal or complete.

Practical implication of the power law. If the exponentiated power law holds, then the log-coverage is proportional to $k^b$ . This means that to increase log-coverage by a fixed amount, the required multiplicative increase in $k$ depends on the current coverage level (because of the $k^b$ non-linearity). For $b = -0.4$ , doubling $k$ from 1 to 2 changes log-coverage by a factor of $2^{-0.4} \approx 0.76$ , while doubling from 1000 to 2000 changes log-coverage by a factor of $(2000^{-0.4} - 1000^{-0.4})$ , which is a much smaller absolute change. This diminishing-returns property means that the marginal benefit of additional samples decreases as the budget grows — a hallmark of scaling laws across domains.

FLOPs-Matched Comparison: Model Size vs. Sample Count

Section 2.3 of the paper reframes the coverage results from a cost perspective: given a fixed total inference FLOP budget, is it better to use a larger model with fewer samples or a smaller model with more samples? This analysis requires a FLOP accounting model for dense transformer LLMs.

The FLOPs-per-token approximation. The paper uses the standard formula for dense transformers where the majority of parameters participate in matrix multiplications:

$\text{FLOPsPerToken}(\text{ContextLen}) \approx 2 \cdot \left(\text{NumParameters} + 2 \cdot \text{NumLayers} \cdot \text{TokenDim} \cdot \text{ContextLen}\right)$

where $\text{NumParameters}$ is the total number of model parameters, $\text{NumLayers}$ is the number of transformer layers, $\text{TokenDim}$ is the hidden dimension size, and $\text{ContextLen}$ is the current sequence length (number of tokens processed so far).

What this formula computes. The factor of 2 accounts for the multiply-accumulate operations in both forward and backward passes of matrix multiplications — for inference (forward pass only), a single multiply-accumulate counts as one FLOP. The $2 \cdot \text{NumParameters}$ term approximates the FLOPs for the feedforward and attention projection layers (linear transformations whose cost is proportional to parameter count). The $2 \cdot 2 \cdot \text{NumLayers} \cdot \text{TokenDim} \cdot \text{ContextLen}$ term accounts for the attention mechanism itself, whose cost scales with sequence length because the attention scores are computed between all pairs of tokens in the context.

Total inference FLOPs computation. The total FLOPs for a generation run is:

$\text{TotalInferenceFLOPs} \approx \left(\sum_{t=1}^{\text{NumPromptTokens}} \text{FLOPsPerToken}(t)\right) + \left(\sum_{t=1}^{\text{NumDecodeTokens}} \text{FLOPsPerToken}(t + \text{NumPromptTokens}) \cdot \text{NumCompletions}\right)$

where $\text{NumPromptTokens}$ is the length of the input prompt (including few-shot examples), $\text{NumDecodeTokens}$ is the average number of tokens generated per completion, $\text{NumCompletions}$ is the number of independent samples generated, and $t$ indexes token positions.

Why this form — the two-term structure. The first summation computes the one-time cost of processing the prompt. For all $\text{NumCompletions}$ samples, the prompt is identical, so its processing cost is incurred only once if key-value caches are shared across samples — this is the "shared prefix" optimization exploited by systems like Hydragen (Juravsky et al., 2024) and SGLang (Zheng et al., 2024). The second summation computes the per-sample cost of generating $\text{NumDecodeTokens}$ new tokens. The context length grows by $\text{NumPromptTokens}$ in the generation phase (since the model attends to the prompt plus previously generated tokens), causing the FLOPsPerToken to increase slightly with each generated token. The multiplication by $\text{NumCompletions}$ accounts for the fact that generation is repeated independently for each sample.

What this means for the model-size tradeoff. For a fixed total FLOP budget, a smaller model has lower FLOPsPerToken (both the $\text{NumParameters}$ and attention terms are smaller), allowing more completions. The tradeoff depends on two factors: (1) how much worse the smaller model's per-sample accuracy is, and (2) how much coverage improves with additional samples. If coverage for the smaller model improves rapidly with sample count — as it does on MATH for Llama-3-8B-Instruct (Figure 3, left) — then the smaller model can overtake the larger model at high sample counts despite lower per-sample accuracy. If coverage for the smaller model is essentially zero (as on CodeContests for Pythia models), no number of samples compensates for the capability gap.

Application to Llama-3 results (Figure 4). The paper re-plots the coverage curves from Figure 2, replacing the x-axis (number of samples $k$ ) with total inference FLOPs. For Llama-3-8B-Instruct and Llama-3-70B-Instruct on each task, the coverage at each $k$ is mapped to a point on the FLOP axis using the formula above. The resulting curves (Figure 4) show the FLOP-efficiency frontier: which model provides higher coverage at each FLOP budget? On MiniF2F-MATH, GSM8K, and MATH, the 8B model dominates across all budgets — its per-token cost advantage (roughly $8.75\times$ fewer parameters) allows so many more samples that even its lower per-sample accuracy is overcome. On CodeContests, the 70B model dominates — its per-sample accuracy advantage is too large for the 8B model's sample-count advantage to overcome.

Caveats of FLOPs as a cost metric. The paper acknowledges that FLOPs alone is a "crude cost metric that ignores other aspects of system efficiency" (Section 2.3). In particular, repeated sampling workloads benefit from high batch sizes and specialized optimizations (shared prefix attention) that improve throughput relative to single-attempt inference. This means that the FLOPs comparison may understate the cost advantage of repeated sampling — the actual dollar cost or latency advantage of many samples from a small model could be larger than FLOP counts suggest. The paper does not incorporate batch size, memory bandwidth, or hardware utilization into its cost model, leaving these factors to qualitative discussion in Section 5.

Verification Methods for Non-Automatable Tasks

For GSM8K and MATH, the paper tests three verification methods to understand the precision side of the coverage-precision decomposition. All three methods operate on the same collections of $N = 10{,}000$ generated samples per problem and are evaluated across multiple random subsets to measure how performance scales with the number of samples available for verification.

Majority voting (self-consistency). For a given subset of $k$ samples for a problem, the extracted final answer from each sample is collected. Answers are considered equivalent if they are exact string matches or if the is_equiv function from LM-Eval returns True (this handles mathematically equivalent expressions like "1/2" and "0.5"). The most frequent answer (the plurality) is selected as the model's final answer. The success rate for majority voting at sample count $k$ is the fraction of problems where the plurality answer matches the ground truth.

Why majority voting saturates. Majority voting can only select the most common answer. If a correct answer appears in only 1% of samples for a problem, it will never be selected by majority voting regardless of how many samples are drawn — the incorrect majority will always dominate. As the sample count increases and coverage improves through the discovery of rare correct solutions (e.g., problems where the model's correct-sample probability is 0.1% or 1%), these rare correct solutions do not shift the plurality. This is why the majority voting curve in Figure 7 flattens around 100 samples even as coverage continues to rise: coverage gains beyond that point come from problems where correct solutions are too rare to affect the vote.

Reward model + Best-of-N. The paper uses ArmoRM-Llama3-8B-v0.1 (Wang et al., 2024), a reward model that scores highly on the reasoning section of the RewardBench leaderboard (Lambert et al., 2024). For each sample in a subset of size $k$ , the reward model assigns a scalar score. The sample with the highest score is selected, and its extracted answer is taken as the final answer. The success rate is the fraction of problems where this top-scoring answer matches the ground truth.

Reward model + Majority vote (weighted voting). This method combines the reward model's scoring with consensus. For each unique answer in the subset, the reward model scores of all samples giving that answer are summed. The answer with the highest total score is selected. This is a form of weighted majority voting where the "votes" are reward model scores rather than count-based.

Why the reward model also saturates. The reward model is trained to distinguish correct from incorrect reasoning, but it is imperfect. If a correct solution appears with frequency 0.1% among 10,000 samples, there are roughly 10 correct samples and 9,990 incorrect ones. Even if the reward model assigns higher scores to correct samples on average, the presence of a small number of incorrect samples that the reward model mis-scores as excellent (false positives) can dominate the top-N selection. As the sample count grows, the expected number of such false positives grows, making it increasingly likely that a highly-scored incorrect sample out-competes the rare correct one. This is the "needle in a haystack" problem: the verifier must discriminate one correct solution from thousands of incorrect ones, and its finite accuracy creates a ceiling.

Statistical protocol for measuring verification scaling. To compute the success rate for each verification method at sample count $k$ , the paper does not use the unbiased estimator (which only applies to coverage). Instead:

For each problem, 100 random subsets of size $k$ are drawn from the pool of $N = 10{,}000$ samples.
The verification method is applied to each subset, producing a final answer.
The success rate is the fraction of the 100 subsets (averaged across problems) where the selected answer matches ground truth.
The mean and standard deviation across the 100 subsets per problem (and across problems) are reported (Figure 7).

This protocol directly measures the expected success rate of using $k$ random samples with the given verification method, without the analytical extrapolation of the pass@k formula. The standard deviation across subsets captures the variance due to which specific $k$ samples happen to be drawn — this is particularly important for rare correct solutions, where some subsets will contain zero correct samples and others will contain one or more.

The gap between coverage and verification success (Figure 7). The paper plots coverage (the oracle verifier pass@k) alongside the three practical verification methods for Llama-3-8B-Instruct and Llama-3-70B-Instruct on GSM8K and MATH. The key pattern: coverage continues to increase smoothly from 100 to 10,000 samples (reaching >95% on GSM8K and >98% on MATH for Llama-3-8B-Instruct), while all three verification methods plateau around 100 samples and show minimal improvement thereafter. The gap between the oracle curve and the verification curves widens with additional samples — at 100 samples the gap might be 20–30 percentage points; at 10,000 samples the gap exceeds 50 percentage points. This demonstrates that verification, not generation, is the bottleneck for scaling repeated sampling in non-automatable domains.

The Coverage Curve Overlay Analysis: Horizontal Shifts and S-Curves

Section 3.2 presents an interesting empirical observation: when the coverage curves of different model sizes from the same family are plotted with a logarithmic x-axis (sample count), they appear to have the same shape (same slope) but different horizontal offsets. This suggests that within a model family, the multiplicative increase in sample budget needed to improve coverage from $c$ to $c'$ is approximately constant across model sizes.

The overlay procedure. The paper selects an anchor coverage value $c$ and shifts each model's coverage curve leftward (in log-space) so that all curves pass through the point $(1, c)$ . Mathematically, for each model, the paper finds the closest $k$ such that $\text{pass@k} = c$ (denoted $\text{pass@k}^{-1}(c)$ ), then divides every $k$ value on that model's curve by $\text{pass@k}^{-1}(c)$ . This is equivalent to a horizontal shift by $\log(\text{pass@k}^{-1}(c))$ on a logarithmic x-axis. The anchor coverage $c$ is chosen as the maximum pass@1 score among all models in the family, so that the rightmost curve (the strongest model) requires no shift, and weaker models are shifted leftward.

What this reveals about within-family scaling. If the shifted curves collapse onto a single curve (i.e., they overlap), it means that the shape of the scaling function is invariant across model sizes within a family. The only difference between a 2B and a 70B model from the same family is how many samples are needed to reach a given coverage level — the 2B model might need 100 times as many samples as the 70B model, but both follow the same S-curve trajectory. This is visible in Figure 6 for Llama, Gemma, and Pythia models on MATH and CodeContests. The similarity is particularly striking for the Pythia family on MATH, where eight model sizes from 70M to 12B collapse onto nearly the same curve after horizontal shifting.

Why this matters. If the within-family scaling shape is invariant, then the coverage scaling behavior of all models in a family can be characterized by two numbers: the shape parameters $(a, b)$ of the common curve, plus a model-specific horizontal offset (or equivalently, the pass@1 value). This would dramatically reduce the dimensionality of inference-time scaling characterization — instead of fitting separate power laws for every model size, one could fit a single master curve and estimate offsets from cheap pass@1 measurements. The paper does not develop this into a predictive model, but the observation provides a conceptual foundation for such an extension.

Caveats. The curve collapse is not perfect. On CodeContests for Llama-3, the 8B, 8B-Instruct, and 70B-Instruct curves do not overlap perfectly after shifting (Figure 6, bottom panel). The instruction-tuned variants show different scaling behavior than the base model, suggesting that post-training (instruction tuning, RLHF) can change not just the intercept (pass@1) but the shape of the scaling curve itself. The paper does not investigate this quantitatively beyond showing the shifted plots.

Chain-of-Thought Faithfulness Analysis

The paper includes a small but illuminating human evaluation to address a potential objection: if correct final answers are generated through nonsensical reasoning chains, then verification based on the reasoning (as reward models attempt to do) would be impossible — the correct answer would be indistinguishable from lucky guesses with flawed reasoning.

Evaluation protocol. For Llama-3-8B-Instruct on GSM8K, the authors manually graded 105 chain-of-thought explanations from correct samples across problems binned by pass@1 frequency. The bins were: 0–10% pass@1 (5 problems, 15 CoTs graded), 10–25% (10 problems, 30 CoTs), 25–75% (29 problems, 30 CoTs), and 75–100% (84 problems, 30 CoTs). For each graded chain-of-thought, the evaluator judged whether the reasoning steps were logically valid and led correctly to the final answer, or whether the reasoning was faulty despite arriving at the correct answer.

Result. Over 90% of the graded chains-of-thought were faithful — the reasoning was correct and led properly to the final answer. Even in the hardest bin (problems solved in ≤10% of attempts), 11 out of 15 CoTs were correct, 1 was incorrect, and 3 came from a single problem where the ground-truth answer itself was incorrect (see Appendix E of the paper). This demonstrates that when the model does produce a correct final answer, it is usually doing so through genuine reasoning, not lucky guessing. The implication is that there is signal in the chain-of-thought that a verifier could exploit — the precision bottleneck is not due to an absence of verifiable reasoning, but due to the difficulty of discriminating correct reasoning from incorrect reasoning at scale.

The mislabeled GSM8K problem. During the human evaluation, the authors identified one GSM8K test-set problem (HuggingFace index 1042) with an incorrect ground truth answer. The problem involves computing average losses across three horse races. The ground-truth solution incorrectly computes the third-race loss as $15 (10 × 1.5) instead of$ 16.50 (11 × 1.5), leading to a final answer of $3 average loss instead of the correct$ 3.50. This mislabeled problem is the only one where Llama-3-70B-Instruct failed to generate a "correct" sample across 10,000 attempts — because the model consistently produced the mathematically correct answer rather than the dataset's erroneous ground truth. This datum is interesting but does not materially affect the paper's scaling results because it represents 1 out of 128 problems (0.78% of the test subset).

The Precision Problem Formalized: Rare Correct Solutions and the Growing Coverage-Verification Gap

The paper's Section 4 frames the precision problem through a concrete statistical lens: as the sample budget increases, coverage improves primarily through the discovery of rare correct solutions — problems where the model's probability of generating a correct answer is very low (e.g., <1%). These rare correct solutions are the primary contributors to coverage growth at moderate-to-high sample counts.

Evidence from sample frequency distributions (Figure 8). Figure 8 presents a bar chart for Llama-3-8B-Instruct on GSM8K and MATH, showing the empirical correct-sample frequency $\hat{p}_i = C_i / N$ for each problem $i$ , with $N = 10{,}000$ . The bars are colored green if majority voting (self-consistency) correctly identified the answer for that problem, and red otherwise. The distribution shows a long tail of problems with very low $\hat{p}_i$ — many problems where the correct answer appears in only a handful of the 10,000 samples (e.g., 0.1% to 1% frequency). These low-frequency problems are predominantly colored red, meaning majority voting fails to identify the correct answer even though it exists in the sample pool. In contrast, problems with high $\hat{p}_i$ (where the correct answer is the plurality) are colored green.

Why this creates a diverging gap. At $k = 100$ , most problems with $\hat{p}_i > 0.01$ (1% frequency) will have at least one correct sample in the pool (expected count $\geq 1$ ), contributing to coverage. However, majority voting will only succeed on problems where $\hat{p}_i > 0.5$ (the correct answer is the strict plurality). Reward model methods may succeed on problems with somewhat lower $\hat{p}_i$ if the reward model assigns systematically higher scores to correct solutions, but as $\hat{p}_i$ becomes very small, the expected number of incorrect samples with spuriously high reward-model scores outnumbers the few correct samples, causing reward-model-based selection to fail. As $k$ grows from 100 to 10,000, coverage increases by incorporating problems with $\hat{p}_i$ between roughly 0.0001 and 0.01 — but these are precisely the problems where no practical verifier can identify the correct answer from the large pool of distractors. Hence the gap widens with $k$ .

Implications for verifier design. The paper's framing implies that the key challenge for scaling verification is not improving accuracy on average-difficulty problems (where the correct answer is common), but improving discrimination on low-frequency problems. A verifier that achieves 99% accuracy in identifying correct-vs-incorrect samples still fails on a problem with $\hat{p}_i = 0.001$ and $k = 10{,}000$ because the expected 10 false positives among 9,990 incorrect samples can out-compete the 10 true positives. The paper does not propose solutions to this challenge, but the characterization itself establishes the metric and the benchmark against which future verifier improvements should be measured.

The SWE-bench Lite Analysis: Repeated Sampling with Agent Trajectories

The SWE-bench Lite experiments (Section 2.1, Figure 2, and Appendix B) demonstrate that repeated sampling scales to complex, multi-turn agent tasks, not just single-turn generation. This is a distinct experimental setup from the other four tasks.

What constitutes a "sample" on SWE-bench. Unlike CodeContests (where one sample is one Python file) or MATH (where one sample is one chain-of-thought final-answer pair), one sample on SWE-bench Lite is an entire agent trajectory. The model interacts with the Moatless Tools framework through multiple turns: it receives the issue description and codebase snapshot, uses tools to search for relevant files, reads file contents, determines which lines to edit, writes a patch, and may iterate based on execution results. Each sample is a complete, independent trajectory — the model does not see or learn from previous trajectories for the same issue. The 250 samples per problem are generated independently with temperature 1.6.

Why the sample budget is limited to 250. The paper generates only 250 samples per SWE-bench problem (compared to 10,000 for other tasks) because each trajectory is computationally expensive — it involves multiple LLM calls, tool invocations, and context management. At 250 samples, the SWE-bench experiments already demonstrate that coverage continues to increase (the curve in Figure 2, top-left, is steep and shows no sign of saturation at $k = 250$ ), and the achieved coverage of 56% exceeds the single-attempt state-of-the-art. The paper does not claim that 250 is sufficient to reach the coverage ceiling — it is a practical budget chosen to demonstrate the scaling trend and achieve a new SOTA within feasible cost.

The temperature sweep. The paper conducted a sweep over temperatures {1.0, 1.4, 1.6, 1.8} on a random 50-problem subset of the test set to select temperature 1.6. This is notable because 1.6 is substantially higher than the temperatures used for other tasks (0.5–0.6). The higher temperature reflects the greater need for exploration in the SWE-bench setting: the agent must discover which files to edit, what changes to make, and how to structure the patch — a much larger action space than writing a single Python function or proof. Higher temperature increases diversity across trajectories, which is essential for coverage scaling when the space of possible solutions is large.

Test suite flakiness handling (Section 4.2.1, Appendix B.2). The paper identifies 34 of 300 SWE-bench Lite problems (11.3%) with flaky test suites — running the same candidate solution through the test suite multiple times can yield inconsistent pass/fail outcomes. For 30 of these 34 problems, even the dataset's ground-truth solutions exhibit flakiness. To handle this, the paper runs each candidate solution's test suite 11 times on flaky problems and uses majority voting across the 11 runs to determine the final pass/fail verdict. The paper also reports results with all 34 flaky problems removed (Figure 9 in Appendix B) and confirms that the scaling trend is essentially unchanged: coverage increases from 23.31% at $k=1$ to 54.14% at $k=250$ on the clean subset, compared to 24.67% to 56.00% on the full set. The flaky tests do not create a false impression of scaling — the trend is robust.

Cost analysis (Table 1). The paper reports the dollar cost of solving SWE-bench issues using API pricing for DeepSeek-Coder-V2-Instruct ( $0.0072 per attempt), GPT-4o ($ 0.13 per attempt), and Claude 3.5 Sonnet ( $0.17 per attempt). The costs are computed per attempt (one full agent trajectory), not per token, because the Moatless Tools framework involves variable-length trajectories with multiple API calls. Sampling DeepSeek five times costs$ 5 \times $0.0072 \times 300 = $10.80 $total and solves 29.62% of issues, while single attempts from GPT-4o ($ 39 total) and Claude ($51 total) solve 24.00% and 26.70%, respectively. This is the paper's most direct demonstration that repeated sampling from a cheaper model can simultaneously improve performance and reduce cost compared to single attempts from expensive frontier models.

CodeContests False Negative Analysis (Section 4.2.2)

The paper identifies a subtle limitation of the CodeContests verification setup that illustrates the gap between "passing all test cases" and "being a correct solution." This is not a flaw in the paper's methodology but a cautionary observation about the reliability of test-case-based verification — important context for interpreting the CodeContests coverage results.

Source of false negatives. The CodeContests test suites are composed of public test cases (visible in the problem description), private test cases (hidden from contestants), and generated test cases (programmatically created by mutating original test inputs). Two issues create false negatives:

Multiple valid outputs: Some problems allow multiple distinct correct outputs for a given input (e.g., any valid topological ordering of a graph), but the test cases expect one specific output string. A solution producing a different but equally valid output fails the test.
Malformed generated test cases: The mutation process can produce inputs that violate the problem's stated input constraints (e.g., generating a zero when the problem promises a positive integer). Different correct solutions may handle these out-of-spec inputs differently — some may crash, others may produce valid output that doesn't match the expected test output.

Quantifying the prevalence. The paper tests the provided "correct" solutions for each CodeContests problem against the test suites. Of the 122 problems in the test set that have Python3 solutions, 35 problems (28.7%) have at least one "correct" solution (as judged by dataset authors) that nevertheless fails the corresponding test suite. This means that for these 35 problems, a model could generate a genuinely correct solution and still be scored as incorrect.

Implication for coverage interpretation. For these 35 problems, the measured coverage is a lower bound on true coverage — some truly correct model outputs are classified as failures. This adds noise to the scaling curves but does not invalidate them, because the false negative rate is fixed per problem and applies equally at all sample counts. The scaling trend (coverage increasing with samples) remains valid because it reflects the model's ability to generate solutions that pass the specific test harness, even if that harness is imperfect. The paper raises this issue as a caution for practitioners: test-case-based verification is not as definitive as proof-checker-based verification, and repeated sampling amplifies the need for comprehensive, well-specified test suites.

Summary of Design Choices and Their Justifications

Independent sampling with fixed temperature rather than adaptive temperature scheduling: Simplifies the experimental design to isolate the effect of sample count alone, without conflating it with per-sample quality optimization. Future work can explore temperature annealing or adaptive sampling strategies.
Unbiased pass@k estimator rather than empirical resampling: Eliminates sampling variance from the coverage measurement, enabling precise characterization of scaling behavior even at extreme $k$ values where direct resampling would be impractical.
Exponentiated power law rather than sigmoid or other saturating functions: Connects to training scaling law literature and fits the observed log-linear scaling over four orders of magnitude for most model-task pairs. The two-parameter form is minimal while capturing the dominant trend.
Temperature 1.6 for SWE-bench (selected via sweep) rather than using the same temperature as other tasks: The multi-turn agent trajectory has a much larger action space than single-turn code generation, requiring higher stochasticity for effective exploration.
ArmoRM-Llama3-8B-v0.1 as the reward model rather than training task-specific verifiers: Reflects the paper's goal of characterizing off-the-shelf verification methods. The paper acknowledges that specialized verifiers might perform better but deliberately focuses on characterizing the limitations of widely available, general-purpose verification tools.
100 random subsets for verification evaluation rather than a single draw: Provides confidence intervals on the success rate, revealing the variance due to which specific samples happen to be in the subset — essential for understanding the reliability of verification at different sample budgets.
No combination of search strategies or iterative refinement: The paper deliberately studies only independent, non-interacting samples to establish a clean baseline. The authors explicitly flag more sophisticated strategies (solution diversity mechanisms, multi-turn feedback, learning from previous attempts) as future work in Section 5.

4. Key Insights and Innovations

Innovation 1: Inference Compute as an Independent Scaling Axis — Reframing Model Capability as a Function of Budget, Not a Fixed Property

The most fundamental conceptual move this paper makes is to decouple model capability from the single-attempt evaluation paradigm. Before this work, the standard way to measure what an LLM "can do" was to give it one shot at each problem and report the fraction it solves. This produces a single number — 15.9% on SWE-bench Lite, 40% on MATH, etc. — that implicitly treats model capability as a fixed property of the trained model.

The paper's reframing is deceptively simple but profound: capability is not a point estimate; it is a function of the inference compute budget. A model that solves 15.9% of problems with one attempt is the same model that solves 56% of problems with 250 attempts (Figure 2, top-left panel). The model hasn't changed — the budget has. This transforms "how good is this model?" from a static question into a dynamic one: "how good can this model be, given a specified compute budget?"

What makes this a genuine innovation rather than an obvious observation is that the paper quantifies the functional form of this relationship. The exponentiated power law $c \approx \exp(a k^b)$ (Section 3.1, Equation 3) provides a two-parameter model that fits coverage curves over four orders of magnitude in sample count. This is directly analogous to the training scaling laws that revolutionized how the field thinks about pretraining — before Kaplan et al. (2020) and Hoffmann et al. (2022), model developers had heuristics but no predictive framework for how training compute translated into performance. The paper establishes the first analogous framework for inference compute, making it possible to ask questions like: "If I need 90% coverage, and my model has a measured $(a, b)$ on this task, how many samples do I need?" Answering such questions requires the functional form — without it, scaling is just anecdata.

The comparison to prior work highlights the shift. AlphaCode (Li et al., 2022) demonstrated that sampling up to a million times improves competitive programming performance, but treated it as an engineering trick specific to code generation. Greenblatt (2023) observed log-linear scaling on ARC puzzles with GPT-4o, but as a one-off observation. Neither established a general framework. This paper does: across five tasks, three model families, and model sizes from 70M to 70B, coverage scales log-linearly with sample count (Figure 3). The universality of the pattern — even if the exponentiated power law fit is imperfect on some tasks like MiniF2F-MATH (Figure 5, bottom-right) — establishes inference-time scaling as a first-class scaling axis alongside model size and training data, not a niche trick for coding competitions.

This is a fundamental conceptual shift, not an incremental improvement. It changes how researchers should evaluate models (pass@1 is incomplete; pass@k curves are the relevant object), how practitioners should select models (the cheapest route to a target accuracy may involve a weaker model with many samples), and how the field should think about the training-inference compute tradeoff (the two are partially substitutable, as Section 2.3 demonstrates). The paper doesn't just report numbers — it provides a new vocabulary and measurement framework for reasoning about inference compute.

Evidence: Figure 2 (five tasks, all show monotonic coverage improvement), Figure 3 (thirteen models, all show log-linear scaling), Figure 5 (fitted power laws with low error for most model-task pairs).

Innovation 2: The Coverage-Precision Decomposition — Identifying Verification as the Scaling Bottleneck, Not Generation

The paper's second major conceptual contribution is to decompose the repeated sampling problem into two orthogonal, independently measurable properties — coverage and precision — and to demonstrate empirically that precision, not coverage, is the bottleneck for scaling.

Before this work, the literature treated verification as a binary problem: either you have a verifier (unit tests, proof checker) and repeated sampling works, or you don't and you're stuck with approximate methods like majority voting whose limitations were understood qualitatively but never characterized as a function of sample budget. The key studies on self-consistency (Wang et al., 2023) and reward model verification (Cobbe et al., 2021; Lightman et al., 2023) operated in low-sample regimes (typically up to 40–100 samples) where majority voting and reward models show improving trends. They never pushed far enough to observe the saturation.

This paper makes the diagnostic move of plotting coverage and verification success on the same axes as functions of sample budget (Figure 7). This single visualization reveals a pattern that was invisible in prior work: coverage continues to increase log-linearly from 100 to 10,000 samples (reaching >95% on GSM8K, >98% on MATH for Llama-3-8B-Instruct), while all three practical verification methods — majority voting, reward model best-of-N, and reward model weighted majority voting — plateau around 100 samples and show negligible improvement thereafter. The gap between the oracle curve and the verification curves widens with additional samples, not narrows.

This is not just an empirical observation — it's a diagnostic reframing that changes what the field should optimize. If verification were the minor partner and generation the bottleneck, research effort should focus on better sampling strategies, diversity mechanisms, or iterative refinement. The paper's decomposition shows the opposite: for tasks without automatic verifiers, generation ceases to be the binding constraint around 100 samples, and verification becomes the sole obstacle to further gains. Every additional dollar of inference compute spent beyond that point buys improved coverage (more problems have correct solutions somewhere in the candidate pool) but delivers zero improvement in practical success rate because the verifier can't find them.

The paper deepens this diagnosis with Figure 8 — showing that problems where the correct answer is generated at low frequency (<1% of samples) are precisely those where majority voting fails. This connects the abstract "precision bottleneck" to a concrete statistical mechanism: rare correct solutions cannot shift the plurality, and they are statistically indistinguishable from high-scoring false positives for a reward model with imperfect accuracy. The bottleneck is not that verifiers are "bad" — it's that the task of identifying a 0.1%-frequency correct solution from 9,999 distractors is fundamentally harder than any current verifier can handle.

This is a fundamental diagnostic contribution, not an incremental measurement. It identifies where the field should invest effort (verifier robustness, particularly for low-frequency correct solutions) and where additional investment yields diminishing returns (generation scaling beyond ~100 samples, given current verifiers). The paper's manual chain-of-thought faithfulness analysis (Table 2) further strengthens the diagnosis by ruling out a potential confound: it shows that >90% of correct solutions arrive via valid reasoning, meaning there is genuine signal for verifiers to exploit if they can be made sensitive enough.

Evidence: Figure 7 (four panels showing diverging oracle-vs-practical curves for GSM8K and MATH with both Llama-3 model sizes), Figure 8 (low-frequency correct solutions are predominantly red/verifier-failure), Table 2 (CoT faithfulness >90% across difficulty bins).

Innovation 3: Within-Family Scaling Invariance — The Shape of Coverage Curves Is Largely Independent of Model Size

The paper's third conceptual contribution is the observation that within a model family, coverage curves from different model sizes have approximately the same shape when plotted on a logarithmic x-axis, differing primarily by a horizontal shift (Section 3.2, Figure 6). This is a scaling-law-level empirical regularity with significant implications for how inference-time scaling should be characterized and predicted.

The mechanism is the curve-overlay analysis described in Section 3: for a given model family (Llama, Gemma, Pythia) and task (MATH, CodeContests), each model's coverage curve passes through an anchor point $(1, c)$ after being shifted leftward by $\log(\text{pass@k}^{-1}(c))$ , and the shifted curves approximately collapse onto a single S-curve. This is most striking for the Pythia family on MATH (Figure 6, third panel), where eight model sizes from 70M to 12B — a 171× range in parameter count — trace essentially the same curve after shifting.

What makes this observation novel is that it is non-obvious and has no analog in training scaling laws. Training scaling laws characterize how loss scales with compute, but they don't predict that models of different sizes have the same-shaped learning curves — indeed, they usually don't. The finding that inference-time scaling shape is approximately invariant within a family suggests a separation of concerns: the model's per-sample accuracy (captured by pass@1 or the horizontal offset) determines where the scaling curve starts, while the shape of how additional samples improve coverage is determined by something more fundamental — perhaps the entropy of the model's output distribution, the structure of the problem space, or the geometry of correct-solution clusters in the model's sampling manifold.

The practical upshot is significant: if within-family scaling invariance holds, then characterizing inference-time scaling for an entire model family requires only two numbers per model — the pass@1 (or equivalent horizontal offset) plus the common shape parameters $(a, b)$ of the master curve. This dramatically reduces the measurement burden. Instead of generating 10,000 samples for every model size, one could fit the master curve from a few representative models and estimate offsets from cheap pass@1 measurements on new sizes.

The paper does not develop this into a predictive framework — it presents the observation and moves on. But the observation itself is a fundamental empirical regularity that, if validated across more families and tasks, could become as useful for inference-time scaling as the Chinchilla laws are for training. The fact that instruction-tuned variants (Llama-3-8B-Instruct vs. Llama-3-8B) don't collapse perfectly (Figure 6, bottom panel, CodeContests) is itself informative: it suggests that post-training changes not just the intercept but the shape of the scaling curve, perhaps because instruction tuning alters the diversity or calibration of the output distribution.

This is a descriptive innovation rather than a methodological one — it identifies a pattern, not a technique. Its significance lies in the conceptual simplification it offers: instead of treating each model-task-budget combination as a distinct data point, it reveals an underlying low-dimensional structure that could guide resource allocation and model selection.

Evidence: Figure 6 (four-panel overlay showing curve collapse for Llama, Gemma, and Pythia on MATH, plus Llama on CodeContests), Figure 3 (raw coverage curves whose similar slopes motivate the overlay analysis).

Innovation 4: The Economic Reframing — Weaker Models with Many Samples Can Be Both Better and Cheaper Than Stronger Models with One Sample

The paper's fourth innovation is to reframe the model selection problem from a quality-per-dollar perspective into a multi-dimensional optimization over model strength, sample budget, and total cost. This is more than a cost analysis — it's a demonstration that the "obvious" strategy of using the strongest model you can afford is economically suboptimal under the inference-time scaling paradigm.

Prior work on cost-performance tradeoffs (e.g., model routing systems like RouteLLM, Ong et al., 2024; model ensembling) operated within the single-attempt paradigm: the question was "which model should I call for this query?" The answer was always a single model. This paper shows that the question should be: "which model should I call, and how many times?" — and the answer is sometimes a weaker model called many times rather than a stronger model called once.

The evidence is compelling because it cuts across both FLOP-based and dollar-based cost metrics. On SWE-bench Lite (Table 1), sampling DeepSeek-Coder-V2-Instruct five times (total cost $10.80) solves 29.62% of issues — more than single attempts from GPT-4o (24.00%,$ 39 total, 3.6× more expensive) or Claude 3.5 Sonnet (26.70%, $51 total, 4.7× more expensive). This is not a marginal cost-quality tradeoff — it's a Pareto improvement: the cheaper strategy is simultaneously the higher-performing one. The ability to achieve better results at lower cost by switching from a frontier model to an open-source model with repeated sampling disrupts the premise that access to the strongest models is necessary for state-of-the-art results.

The FLOPs-matched analysis (Figure 4) generalizes this finding beyond API pricing to a hardware-independent cost metric. It reveals that the optimal model size is task-dependent: on MATH, GSM8K, and MiniF2F-MATH, the 8B model with many samples always achieves higher coverage than the 70B model with fewer samples at equal total FLOPs, because the 8B model's ~9× per-token cost advantage buys enough extra samples to overcome its lower per-sample accuracy. On CodeContests, the 70B model dominates — the capability gap is too wide for sample count to bridge. This task-dependence means there is no universal answer to "should I use a smaller model with more samples?" — the answer emerges from the specific scaling curves.

What makes this a conceptual innovation rather than just a cost calculation is that it introduces a new degree of freedom into the model deployment decision process. Before this work, a practitioner choosing a model for a task had two knobs: model quality and per-token price. The paper adds a third knob — sample budget — and shows that it interacts with the other two in non-trivial ways. This reframes the deployment problem as a three-dimensional optimization rather than a two-dimensional tradeoff, and provides the empirical methodology (FLOPs-matched coverage curves, API cost comparisons) for making that optimization concrete.

This is an applied innovation — it doesn't change how models work, but it changes how they should be used. Its significance is practical: it gives practitioners a principled way to consider inference budget as a first-class resource, and it undermines the assumption that frontier models are always the right choice for high-stakes applications.

Evidence: Table 1 (DeepSeek 5× cheaper and better than GPT-4o/Claude single attempts on SWE-bench), Figure 4 (FLOPs-matched curves showing task-dependent optimal model size), Figure 2 (weaker models with many samples exceed GPT-4o single-attempt performance on all five tasks).

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper evaluates on five distinct tasks: GSM8K (128 randomly-sampled test-set problems from the grade-school math dataset of Cobbe et al., 2021), MATH (128 randomly-sampled test-set problems from the competition-level math dataset of Hendrycks et al., 2021), MiniF2F-MATH (130 test-set problems formalized from MATH into Lean4, from Zheng et al., 2021), CodeContests (140 test-set problems from the AlphaCode competitive programming dataset of Li et al., 2022, filtering out problems with image tags), and SWE-bench Lite (300 real-world GitHub issues from Jimenez et al., 2024, subset to the "Lite" split requiring single-file edits across 12 repositories). The random subsets for GSM8K and MATH are used to keep generation costs tractable at $N = 10{,}000$ samples per problem; the paper explicitly states these subsets were randomly sampled from the full test sets.
Base model(s). The primary models are Llama-3-8B-Instruct, Llama-3-70B-Instruct, and Llama-3-8B (base) from Meta (2024), evaluated on CodeContests, MiniF2F-MATH, GSM8K, and MATH. Gemma-2B and Gemma-7B (Team et al., 2024) and the full Pythia scaling suite (eight sizes from 70M to 12B; Biderman et al., 2023) are evaluated on MATH and CodeContests to test scaling across model sizes and families. DeepSeek-Coder-V2-Instruct (DeepSeek-AI et al., 2024) is used exclusively for SWE-bench Lite because its context length supports the multi-turn agent trajectories required by that task. The paper argues that Llama-3 models are "representative of the capabilities of many contemporary LLMs" and that the range from 70M to 70B covers multiple orders of magnitude in scale, making the coverage trends broadly informative. GPT-4o (OpenAI, 2024) and Claude 3.5 Sonnet (Anthropic, 2024) appear only as single-attempt baselines for cost comparisons in Table 1.
Metrics. The central metric is coverage (equivalently pass@k), defined as the fraction of problems for which at least one of $k$ generated samples is correct. Coverage is estimated using the unbiased estimator from Chen et al. (2021): for each problem $i$ with $C_i$ correct samples out of $N$ total generated, $\text{pass@k} = \frac{1}{\text{\#problems}} \sum_i (1 - \binom{N-C_i}{k} / \binom{N}{k})$ . For tasks with automatic verifiers (CodeContests, MiniF2F-MATH, SWE-bench Lite), coverage equals the achievable success rate. For tasks without automatic verifiers (GSM8K, MATH), additional metrics are computed: majority vote success rate (fraction of problems where the plurality answer among $k$ samples matches ground truth), reward model best-of-N success rate (using ArmoRM-Llama3-8B-v0.1 to select the highest-scoring sample), and reward model weighted majority vote success rate (summing reward scores per unique answer). For these verification metrics, 100 random subsets of size $k$ are drawn per problem, and the mean and standard deviation across subsets are reported.
Baselines. The paper uses several points of comparison rather than formal algorithmic baselines in the traditional sense. Single-attempt GPT-4o serves as an external capability baseline on all tasks — the paper compares coverage for the weaker Llama/DeepSeek models against GPT-4o's pass@1 (Figure 2, horizontal dashed lines). Single-attempt state-of-the-art on SWE-bench Lite (CodeStory Aide using a mixture of GPT-4o and Claude 3.5 Sonnet, achieving 43%) provides the target to beat (Figure 2, top-left annotated point). Single-attempt Moatless Tools with GPT-4o (24.67%) and single-attempt Moatless Tools with Claude 3.5 Sonnet (26.67%) serve as more directly comparable baselines since they share the same agent framework (Figure 2, top-left; Table 1). For the verification analysis, oracle coverage (pass@k with ground-truth answer checking) serves as the theoretical upper bound against which practical verification methods are compared (Figure 7). Within the FLOPs-matched analysis, the comparison is between Llama-3-8B-Instruct and Llama-3-70B-Instruct, with each serving as the baseline for the other depending on which achieves higher coverage at a given FLOP budget.
Generation budget / compute accounting. The generation budget is measured in number of independent samples per problem, denoted $k$ . For all tasks except SWE-bench Lite, the paper generates $N = 10{,}000$ samples per problem per model, then uses the unbiased estimator to report pass@k for any $k \leq N$ . For SWE-bench Lite, the budget is limited to $N = 250$ due to the high computational cost of multi-turn agent trajectories. The paper considers budgets from $k = 1$ to $k = 10{,}000$ (or $k = 250$ for SWE-bench). For the FLOPs-matched comparison (Section 2.3, Figure 4), the generation budget is translated into inference FLOPs using the formula $\text{TotalInferenceFLOPs} \approx \sum_{t=1}^{\text{NumPromptTokens}} \text{FLOPsPerToken}(t) + \sum_{t=1}^{\text{NumDecodeTokens}} \text{FLOPsPerToken}(t + \text{NumPromptTokens}) \times \text{NumCompletions}$ , where FLOPsPerToken is approximated as $2 \times (\text{NumParameters} + 2 \times \text{NumLayers} \times \text{TokenDim} \times \text{ContextLen})$ . For the API cost comparison (Table 1), the budget is measured in US dollars using per-attempt pricing for each model (DeepSeek-Coder-V2-Instruct: $0.0072/attempt, GPT-4o:$ 0.13/attempt, Claude 3.5 Sonnet: $0.17/attempt), with costs computed as per-attempt cost × number of samples × 300 issues.
Cross-validation / statistical protocol. There is no cross-validation because no model training or hyperparameter selection is performed on the test data. The paper generates samples once per model-task pair, then computes pass@k analytically for all $k$ using the unbiased estimator — this is a deterministic function of the generated samples. For the verification metrics (majority voting, reward model methods), 100 random subsets of size $k$ are drawn per problem, the verification method is applied to each subset, and the mean and standard deviation of the success rate across these 100 subsets (and across all problems) are reported. This protocol measures both the expected success rate and its variability due to random subset selection. The SWE-bench temperature selection (1.6) was conducted via a sweep on a random 50-problem subset of the test set, separate from the full 300-problem evaluation. The paper also reports results with and without 34 flaky-test problems (Figure 9) to assess robustness. For CodeContests false negative analysis, the paper runs each problem's test suite on the dataset-provided correct solutions to quantify the false negative rate — this is a diagnostic measurement, not a cross-validation procedure.

Main Quantitative Results

Coverage Scaling Across Tasks (Section 2.1)

The headline result appears in Figure 2: coverage increases smoothly with sample count across all five tasks for Llama-3 models and DeepSeek-Coder-V2-Instruct. The paper generates $N = 10{,}000$ samples per problem for the four single-turn tasks and $N = 250$ for the multi-turn SWE-bench Lite, then reports pass@k across the full range.

On SWE-bench Lite (Figure 2, top-left), DeepSeek-Coder-V2-Instruct with Moatless Tools achieves 15.9% pass@1 (annotated on the curve). As the sample count increases to $k = 250$ , coverage rises to 56%, which exceeds the single-attempt state-of-the-art of 43% (CodeStory Aide with mixed GPT-4o and Claude 3.5 Sonnet, shown as a horizontal dashed line) and substantially exceeds single-attempt GPT-4o with the same Moatless Tools framework (24.67%, also annotated). The curve shows no sign of saturation at $k = 250$ — it is still rising steeply, suggesting that larger budgets would yield further gains if cost constraints permitted.

On CodeContests (Figure 2, top-right), Llama-3-70B-Instruct achieves pass@1 of roughly 20% (read from the curve at $k=1$ ). Coverage reaches approximately 55% at $k = 10{,}000$ . Llama-3-8B-Instruct starts lower (pass@1 roughly 5%) and reaches roughly 30% at $k = 10{,}000$ . Both models exceed the single-attempt GPT-4o performance (dashed horizontal line at approximately 35–40%) at high sample counts — the 70B model crosses this threshold around $k = 100$ , the 8B model around $k = 1{,}000$ – $2{,}000$ . The gap between the two Llama models narrows as $k$ increases, but the 70B model maintains a clear advantage at all budgets.

On MiniF2F-MATH (Figure 2, middle-left), Llama-3-70B-Instruct achieves pass@1 of roughly 25% and reaches approximately 55% at $k = 10{,}000$ . Llama-3-8B-Instruct achieves pass@1 of roughly 15% and also reaches approximately 55% at $k = 10{,}000$ , nearly closing the gap entirely with the larger model at high sample counts. Both exceed the single-attempt GPT-4o line (approximately 38%) by $k \approx 200$ – $500$ .

On MATH with oracle verifier (Figure 2, middle-right), Llama-3-8B-Instruct starts at pass@1 of approximately 48% (read from the curve) and reaches approximately 98.44% at $k = 10{,}000$ (exact number quoted in Section 4.1 text). Llama-3-70B-Instruct starts at approximately 65% pass@1 and reaches near-perfect coverage as well. Both models surpass the single-attempt GPT-4o line (approximately 60–65%) — the 8B model crosses around $k = 4$ – $8$ , the 70B model is already above it at $k=1$ .

On GSM8K with oracle verifier (Figure 2, bottom-left), both models achieve very high coverage. Llama-3-8B-Instruct reaches approximately 95%+ coverage by $k = 100$ and continues to improve toward near-100% at $k = 10{,}000$ (the 70B model does not reach 100% due to a single mislabeled ground-truth answer, documented in Appendix E). Both substantially exceed GPT-4o's single-attempt performance at even modest $k$ .

Key cross-task pattern: The paper's most important finding from this set of experiments is not the absolute numbers but the universality of the trend: every model on every task shows monotonic coverage improvement with sample count. The weaker models (Llama-3-8B-Instruct, DeepSeek-Coder-V2-Instruct) all exceed the single-attempt performance of GPT-4o at sufficient sample budgets, establishing that inference compute can substitute for model capability. The exact crossover point varies by task and model, but the qualitative pattern is invariant.

Coverage Scaling Across Model Sizes and Families (Section 2.2)

Figure 3 extends the coverage analysis to a broader set of models on MATH and CodeContests, revealing that coverage gains from repeated sampling hold across model sizes from 70M to 70B parameters and across Llama, Gemma, and Pythia families.

On MATH with oracle verifier (Figure 3, left panel), coverage increases for every model tested. The most dramatic relative gains occur for the smallest models: Pythia-160M increases from pass@1 of 0.27% to pass@10k of 57% — a more than 200× relative improvement. Pythia-70M increases from near-zero pass@1 to approximately 25% at $k = 10{,}000$ . The larger models (Llama-3-70B-Instruct, Llama-3-8B-Instruct) start from higher pass@1 values and converge toward near-100% coverage. The instruction-tuned Llama models consistently achieve higher coverage than their base-model counterparts at equivalent parameter counts. The Pythia models show a clear monotonic ordering: larger models achieve higher coverage at all budgets.

On CodeContests (Figure 3, right panel), the pattern is similar but with one critical exception: all Pythia models achieve zero coverage on this dataset, even at $k = 10{,}000$ . No Pythia model (from 70M to 12B) ever generates a correct solution that passes the test cases. The paper attributes this to Pythia being trained on less coding-specific data than Llama and Gemma (Section 2.2). For the other model families: Gemma-2B achieves the most striking relative gain, increasing from pass@1 of 0.02% to pass@10k of 7.1% — a more than 300× relative increase. Gemma-7B increases from pass@1 of roughly 0.5% to pass@10k of roughly 12%. The Llama-3 models (8B base, 8B-Instruct, 70B-Instruct) all achieve non-trivial coverage, with the 70B-Instruct model dominating.

The Pythia-on-CodeContests zero-coverage result is a key negative finding. It establishes a boundary condition for repeated sampling: if the model's per-sample probability of correctness is essentially zero (no successful generation observed across 10,000 attempts), then no amount of repeated sampling helps. The paper does not determine the minimum pass@1 threshold below which coverage gains become practically negligible, but the Pythia result suggests that models with near-zero single-attempt performance on a task are unlikely to benefit from repeated sampling without additional training or prompting improvements.

FLOPs-Matched Comparison (Section 2.3)

Figure 4 re-plots the coverage curves from Figure 2 after replacing the x-axis (number of samples $k$ ) with total inference FLOPs, enabling a direct comparison of Llama-3-8B-Instruct and Llama-3-70B-Instruct at equal compute cost. The key insight is that the optimal model size depends on the task.

On MiniF2F-MATH (Figure 4, top-left), the 8B model achieves higher coverage than the 70B model at all FLOP budgets shown. At roughly $10^{15}$ total inference FLOPs, the 8B model achieves approximately 35% coverage while the 70B model achieves approximately 25%. At $10^{16}$ FLOPs, the gap narrows but the 8B model maintains its lead (roughly 55% vs. 48%). The 8B model's per-token cost advantage (roughly 9× fewer parameters) buys enough additional samples to more than compensate for its lower per-sample accuracy.

On CodeContests (Figure 4, top-right), the pattern reverses: the 70B model dominates across all FLOP budgets. At roughly $4 \times 10^{15}$ FLOPs, the 70B model achieves approximately 25% coverage while the 8B model achieves approximately 5%. At $10^{16}$ FLOPs, the 70B model reaches approximately 45% vs. the 8B model's 20%. The capability gap between the two models on competitive programming is too large for additional samples to bridge — the 8B model's per-sample accuracy is so much lower that even with many more samples, it cannot match the 70B model at equal FLOPs.

On MATH with oracle verifier (Figure 4, bottom-left), the 8B model achieves higher coverage at all FLOP budgets. At roughly $5 \times 10^{13}$ FLOPs, the 8B model achieves approximately 55% coverage vs. the 70B model's 70%, but as the FLOP budget increases, the 8B model's curve rises faster and overtakes the 70B model. At higher FLOPs, both models converge toward near-100% coverage. The paper notes that the 8B model is more FLOP-efficient for MATH despite its lower per-sample accuracy.

On GSM8K with oracle verifier (Figure 4, bottom-right), both models achieve very high coverage across the FLOP range. The 8B model is consistently above the 70B model, though the difference is small (both exceed 90% coverage at the lowest FLOP budgets shown, around $3 \times 10^{13}$ ).

Key takeaway from FLOPs analysis: The fact that the optimal model size depends on the task means there is no universal answer to "should I use a bigger model or more samples?" — practitioners must measure the scaling curves for their specific task and model family. The paper establishes the methodology for making this determination but does not provide a predictive framework for extrapolating from one task to another.

API Cost Comparison on SWE-bench Lite (Table 1)

Table 1 presents the dollar-cost analysis for SWE-bench Lite. Keeping the agent framework (Moatless Tools) constant:

DeepSeek-Coder-V2-Instruct at $0.0072 per attempt, with **5 samples** per issue, solves **29.62%** of issues at a total cost of **$ 10.80** (300 issues × $0.0072 × 5). This is the baseline (relative cost 1×).
GPT-4o at $0.13 per attempt, with **1 sample** per issue, solves **24.00%** at a total cost of **$ 39** (relative cost 3.6×).
Claude 3.5 Sonnet at $0.17 per attempt, with **1 sample** per issue, solves **26.70%** at a total cost of **$ 51** (relative cost 4.7×).

The DeepSeek strategy is simultaneously cheaper and more effective — it achieves a higher solve rate (29.62% vs. 24.00% and 26.70%) at 3.6–4.7× lower total cost. The paper notes that the DeepSeek model is "over 10x cheaper" per attempt than the frontier models, and that the cost advantage of repeated sampling is further amplified by throughput optimizations not reflected in API pricing (Section 5 discussion of shared-prefix attention and high-batch-size inference).

The Exponentiated Power Law Fits (Section 3.1)

Figure 5 and Appendix C.2 (Figure 10) present the exponentiated power law fits $c \approx \exp(a k^b)$ for multiple model-task pairs. The paper reports the fitted parameters $(a, b)$ and the mean absolute error between the fitted curve and empirical coverage across 100 evenly-spaced log-scale points.

For Llama-3-8B-Instruct on MATH (Figure 5, top-left): $a = -1.33$ , $b = -0.43$ , with error $0.003 \pm 0.0027$ . The fit is excellent — the curve closely tracks the empirical data across all $k$ .

For Llama-3-70B-Instruct on MATH (Figure 5, top-center): $a = -0.75$ , $b = -0.46$ , error $0.0056 \pm 0.0036$ . The more negative $b$ (compared to $-0.43$ for the 8B model) indicates faster initial coverage gains that then saturate slightly more quickly.

For Llama-3-8B-Instruct on CodeContests (Figure 5, top-right): $a = -3.88$ , $b = -0.11$ , error $0.002 \pm 0.0015$ . The very shallow $b = -0.11$ indicates slow but steady coverage improvement over four orders of magnitude — the curve is almost linear on a log-log plot ("log-linear" in the paper's terminology). The large negative $a$ reflects the low starting coverage.

For Llama-3-8B-Instruct on MiniF2F-MATH (Figure 5, bottom-right): $a = -1.33$ , $b = -0.08$ , error $0.0297 \pm 0.0157$ . This is the poorest fit among the main results, with the highest error and visible deviation between the fitted curve and the empirical data. The paper explicitly flags this (Section 3.1): "We highlight that some curves, such as Llama-3-8B-Instruct on MiniF2F-MATH, do not follow this trend closely."

For Pythia-70M on MATH (Figure 5, bottom-left): $a = -7.59$ , $b = -0.18$ , error $0.0052 \pm 0.0071$ . The deeply negative $a$ reflects the model's extremely low pass@1, while the moderate $b$ indicates noticeable curvature.

For Pythia-12B on MATH (Figure 5, bottom-center): $a = -3.92$ , $b = -0.35$ , error $0.0189 \pm 0.0118$ . The higher error compared to Llama models suggests that the power law fit is less precise for Pythia models, though still reasonable.

Appendix C.2 (Figure 10) extends the fitting to additional models, including DeepSeek-Coder-V2-Instruct on SWE-bench Lite ( $a = -1.74$ , $b = -0.21$ ), Gemma models on CodeContests and MATH, and all Pythia models on MATH. The consistent pattern is that $b$ (the shape parameter) is more negative for MATH (typically $-0.3$ to $-0.5$ ) than for CodeContests (typically $-0.1$ to $-0.15$ ), indicating that coverage on MATH improves more rapidly at small $k$ and then saturates, while coverage on CodeContests improves more steadily across the entire range.

Within-Family Coverage Curve Overlay (Section 3.2)

Figure 6 presents the curve-overlay analysis. For each model family and task, all models' coverage curves are shifted horizontally (in log-space) to pass through a common anchor point $(1, c_{\text{anchor}})$ , where $c_{\text{anchor}}$ is the maximum pass@1 among models in that family. The paper observes that the shifted curves approximately collapse onto a single S-curve, indicating that within a family, the scaling shape is largely invariant to model size.

For Llama-3 on MATH (Figure 6, top-left), the three curves (8B base, 8B-Instruct, 70B-Instruct) collapse well after shifting. The 70B-Instruct curve (shifted least, since it has the highest pass@1 at the anchor point $c = 0.48$ ) and the 8B-Instruct curve overlap almost perfectly. The 8B base curve (which has lower pass@1 and receives a larger leftward shift) overlaps reasonably but shows some deviation at intermediate $k$ values.

For Gemma on MATH (Figure 6, top-right), the 2B and 7B curves collapse onto each other after shifting, with the anchor coverage $c = 0.2$ (the 7B model's pass@1).

For Pythia on MATH (Figure 6, bottom-left), this is the most striking example: eight model sizes from 70M to 12B (a 171× range in parameter count) collapse onto essentially the same curve after shifting. The anchor coverage is $c = 0.02$ (the 1.4B model's pass@1). This is the strongest evidence for within-family scaling invariance.

For Llama-3 on CodeContests (Figure 6, bottom-right), the collapse is noticeably poorer. The 8B-Instruct and 70B-Instruct curves overlap reasonably, but the 8B base curve shows a different shape — it rises more slowly at small $k$ and saturates at a lower maximum coverage. The instruction-tuned models appear to have a qualitatively different scaling shape from the base model on this task.

Interpretation: The approximate curve collapse implies that the multiplicative increase in sample budget needed to go from coverage $c$ to $c'$ is roughly constant across model sizes within a family. For example, if going from 20% to 40% coverage requires 10× more samples for a 70M model, it also requires roughly 10× more samples for a 12B model from the same family — the difference is that the 12B model starts from a higher pass@1 and therefore hits any given coverage target at a lower absolute sample count. The paper does not develop this into a predictive model (it remains a qualitative observation), but it provides the conceptual foundation for more parsimonious scaling characterizations.

Verification Scaling and the Precision Bottleneck (Section 4.1)

Figure 7 is the paper's central result on verification. It plots coverage (oracle verifier) alongside three practical verification methods — majority voting, reward model + best-of-N, and reward model + majority vote — as functions of the number of samples $k$ , for both Llama-3 model sizes on GSM8K and MATH.

On Llama-3-8B-Instruct, GSM8K (Figure 7, top-left): Coverage increases from roughly 82% at $k = 100$ to over 95% at $k = 10{,}000$ . Majority voting increases from roughly 40% at $k = 10$ to roughly 43% at $k = 10{,}000$ , essentially flat after $k \approx 100$ . Reward model + best-of-N increases from roughly 38% to roughly 42% over the same range, also plateauing. Reward model + majority vote performs similarly, reaching roughly 43%. The gap between coverage and all practical methods widens dramatically — from roughly 40 percentage points at $k = 100$ to over 50 percentage points at $k = 10{,}000$ .

On Llama-3-70B-Instruct, GSM8K (Figure 7, top-right): Coverage reaches near-perfect levels (>97% at $k = 1{,}000$ , approaching 100% at $k = 10{,}000$ ). Practical methods again plateau: majority voting saturates around 55–58%, reward model methods around 55–60%, all showing minimal improvement beyond $k = 100$ . The gap reaches roughly 40+ percentage points.

On Llama-3-8B-Instruct, MATH (Figure 7, bottom-left): The paper reports exact figures in the text (Section 4.1): coverage increases from 82.9% at $k = 100$ to 98.44% at $k = 10{,}000$ . However, practical methods show "the biggest performance increase is only from 40.50% to 41.41% over the same sample range." The gap between oracle and practical performance grows from roughly 42 points at $k = 100$ to roughly 57 points at $k = 10{,}000$ .

On Llama-3-70B-Instruct, MATH (Figure 7, bottom-right): The pattern is identical — coverage increases while practical methods plateau, with a widening gap.

The paper's central numerical claim about verification: "As the number of samples increases, the gap between coverage (i.e. performance with a perfect verifier) and the performance of these methods increases as well" (Section 4.1, final sentence). Figure 7 provides the visual evidence; the exact MATH numbers (82.9% → 98.44% coverage vs. 40.50% → 41.41% practical) are quoted in the abstract and Section 4.1 text.

Chain-of-Thought Faithfulness (Table 2)

Table 2 reports the human evaluation of 105 chain-of-thought explanations from correct Llama-3-8B-Instruct samples on GSM8K. The problems are binned by pass@1 frequency:

0–10% pass@1 (5 problems): 15 CoTs graded, 11 judged correct, 1 incorrect, 3 from a mislabeled problem (incorrect ground truth).
10–25% pass@1 (10 problems): 30 CoTs graded, 27 correct, 3 incorrect.
25–75% pass@1 (29 problems): 30 CoTs graded, 28 correct, 2 incorrect.
75–100% pass@1 (84 problems): 30 CoTs graded, all 30 correct.

Overall, over 90% of the graded chains-of-thought were faithful — the reasoning steps were logically valid and led correctly to the final answer. This holds even for problems in the hardest bin (≤10% pass@1), where 11 of 15 CoTs were correct. The paper argues that this demonstrates genuine reasoning signal in the chain-of-thought, which a sufficiently capable verifier could exploit. The single mislabeled GSM8K problem (Appendix E) accounted for 3 of the 4 problematic CoTs in the hardest bin — the model was generating correct reasoning for a problem with an erroneous ground-truth answer.

Sample Frequency Distributions (Section 4.2, Figure 8)

Figure 8 presents bar charts showing, for each problem in the GSM8K and MATH subsets, the fraction of the $N = 10{,}000$ samples that arrive at the correct final answer. Bars are colored green if majority voting (self-consistency) correctly identifies the answer for that problem, and red otherwise. The figure reveals a long tail of problems where the correct answer frequency is very low (many problems with <1% correct-sample frequency) and where these low-frequency problems are predominantly red — majority voting fails to identify the correct answer even though it exists in the sample pool. In contrast, high-frequency problems (where the correct answer is the plurality) are predominantly green. The paper uses this to explain why majority voting saturates: as the sample budget increases, coverage improves by generating correct solutions for problems where those solutions are too rare to affect the plurality vote. The figure caption emphasizes: "there are many problems with correct solutions, where the correct solutions are sampled infrequently."

Ablation Studies and Robustness Checks

SWE-bench Lite test suite flakiness (Appendix B.2, Figure 9): The paper identifies 34 of 300 SWE-bench Lite problems (11.3%) with flaky test suites — running the same candidate solution through the test suite multiple times yields inconsistent pass/fail verdicts. For 30 of these 34 problems, even the dataset's ground-truth solutions exhibit flakiness (Table 3 lists all affected problem IDs). The paper handles this by running each candidate solution's test suite 11 times on flaky problems and using majority voting across runs to determine the final verdict. As a robustness check, the paper reports results both with all 300 problems (Figure 9, right panel) and with the 34 flaky problems removed (Figure 9, left panel, 266 problems remaining). The scaling trends are essentially identical: coverage increases from 23.31% to 54.14% at $k=250$ on the clean subset vs. 24.67% to 56.00% on the full set. The single-attempt baselines shift slightly: CodeStory Aide goes from 43.00% to 41.35%, Claude 3.5 Sonnet + Moatless Tools from 26.67% to 24.81%. The paper's conclusion that DeepSeek-Coder-V2-Instruct with 250 samples exceeds the single-attempt SOTA holds in both cases (54.14% vs. 41.35% on the clean subset). The flaky tests do not create the scaling trend — the improvement is genuine.

CodeContests false negative rate (Section 4.2.2): The paper runs each CodeContests problem's test suite on the dataset-provided "correct" reference solutions. Of the 122 problems in the test set with Python3 solutions, 35 (28.7%) have at least one reference solution that fails the corresponding test suite. The root causes are: (1) multiple valid outputs being possible for a given input, but the test expecting one specific output string, and (2) programmatically generated test inputs that violate the problem's stated input constraints. The paper raises this as a caution: the measured CodeContests coverage is a lower bound on true coverage because some genuinely correct model outputs are falsely classified as failures. This does not invalidate the scaling trends (the false negative rate is fixed per problem) but it does mean that the absolute coverage numbers for CodeContests understate the models' true capabilities. The paper does not attempt to re-grade CodeContests solutions with a more permissive checker — it reports results against the standard test harness and flags the limitation for practitioners.

Temperature sweep for SWE-bench Lite (Appendix B.1): The paper conducted a sweep over sampling temperatures {1.0, 1.4, 1.6, 1.8} on a random 50-problem subset of the SWE-bench Lite test set. Temperature 1.6 was selected based on this sweep. The paper does not report the full sweep results (pass@k curves across temperatures), only stating that 1.6 was chosen. This is a missing ablation — it would be interesting to know how sensitive coverage is to temperature in the SWE-bench setting, and whether the optimal temperature changes with the sample budget (higher temperatures might provide more diversity and benefit larger budgets more). The high temperature (1.6 vs. 0.5–0.6 for other tasks) is notable and likely reflects the much larger action space of multi-turn agent trajectories.

Oracle vs. practical verification gap robustness (Section 4.1, Figure 7 error bars): The verification metrics (majority voting, reward model methods) are computed by drawing 100 random subsets of size $k$ per problem and reporting the mean and standard deviation. The standard deviation bands in Figure 7 are narrow (typically ±1–2 percentage points for each method), indicating that the reported plateauing behavior is not an artifact of subset sampling variance — the verification methods consistently saturate, with low variability across different random draws of $k$ samples.

Model and temperature consistency across tasks: The paper uses different temperatures for different tasks (0.5 for MiniF2F, 0.6 for MATH/GSM8K/CodeContests, 1.6 for SWE-bench). It does not perform an ablation comparing coverage at different temperatures for the same task (except the SWE-bench sweep). This means that the cross-task comparison of absolute coverage numbers confounds task difficulty with sampling temperature. However, within-task scaling trends are internally consistent since temperature is held fixed for all $k$ .

Single vs. multiple model families: The paper evaluates Llama, Gemma, and Pythia on MATH and CodeContests (Figure 3), but only Llama on GSM8K and MiniF2F-MATH, and only DeepSeek on SWE-bench Lite. The cross-family comparison is therefore limited to MATH and CodeContests. For GSM8K, MiniF2F, and SWE-bench, the paper cannot claim that the scaling pattern generalizes beyond Llama and DeepSeek — though the consistency across two task-model-family pairs (plus the Pythia results on MATH) is suggestive.

No ablation on sampling diversity mechanisms: The paper uses only temperature-based sampling to create diversity among samples. It does not compare this against alternative diversity mechanisms such as nucleus sampling with different top-p values, prompt variation, metadata conditioning (as in AlphaCode), or other methods mentioned in the discussion (Section 5). The paper acknowledges this as a limitation and future work direction, but it means the reported coverage curves are specific to the chosen temperature settings — lower or higher temperatures could shift the curves.

No multi-turn or feedback ablation: The paper generates all samples independently with no feedback from previous attempts. For tasks with automatic verifiers (CodeContests, MiniF2F), the model never sees execution results — it generates solutions in a single turn. The paper does not compare this single-turn approach against a setup where the model receives execution feedback and can iterate (as is standard in many code generation systems and SWE-bench agents). This is a deliberate simplification to establish a baseline, but it means the coverage numbers are not directly comparable to systems that use multi-turn refinement. The SWE-bench setup does use multi-turn agent trajectories, but each trajectory is still independent of other trajectories for the same issue.

Reward model choice (Section 4.1): The paper uses ArmoRM-Llama3-8B-v0.1 (Wang et al., 2024) as its sole reward model for verification experiments. This model scores highly on the reasoning section of RewardBench (Lambert et al., 2024) but is a general-purpose reward model not specifically trained on GSM8K or MATH. The paper does not compare against task-specific verifiers (e.g., the verifiers from Cobbe et al., 2021 for GSM8K, or process reward models from Lightman et al., 2023 for MATH). This is consistent with the paper's goal of characterizing off-the-shelf verification methods, but it means the paper cannot claim that all reward models saturate — only that this particular (strong, general-purpose) reward model does. A task-specialized reward model might scale further.

Critical Assessment

Claim: "Inference compute is an independent scaling axis that produces large, consistent gains." This claim is strongly supported for the specific models and tasks tested. Figures 2 and 3 demonstrate that coverage increases smoothly with sample count for every model-task pair that achieves non-zero pass@1. The scaling spans four orders of magnitude ( $k = 1$ to $k = 10{,}000$ ) for most experiments, which is a substantial range. The consistency across five diverse tasks (from formal proofs to GitHub issues) and three model families (Llama, Gemma, Pythia) strengthens the generality claim.

However, the "independent" part deserves scrutiny. The paper studies only independent repeated sampling with fixed temperature — it does not compare this against, say, scaling temperature at fixed sample count, or using a different decoding strategy. The claim that inference compute in the form of repeated sampling is a scaling axis is well-supported; the claim that it is a scaling axis independent of other inference-time choices (temperature, prompting, verification strategy) is not directly tested, since those choices are held fixed.

Additionally, the Pythia-on-CodeContests zero-coverage result establishes an important boundary: the scaling axis only exists conditional on the model having non-zero per-sample probability of correctness. If pass@1 is effectively zero, inference compute provides no benefit. The paper does not quantify the minimum pass@1 threshold below which scaling becomes ineffective, which limits the practical applicability of the finding — a practitioner with a model that achieves 0.1% pass@1 on their task does not know whether 10,000 samples will yield meaningful gains or be wasted compute.

Claim: "The relationship between coverage and the number of samples is often log-linear and can be modeled with an exponentiated power law." This claim is supported with reasonable evidence but with notable exceptions. The exponentiated power law fits well for most Llama-3 × MATH combinations (errors of 0.003–0.006) and for CodeContests (errors of 0.002–0.006), but it fits poorly for MiniF2F-MATH (error 0.0297, visible deviation in Figure 5). The Pythia models on MATH show higher errors (0.005–0.022) than Llama models. The paper acknowledges the MiniF2F exception but does not investigate why the power law fails there — is it a property of the task (formal proofs may have different scaling characteristics than code or math word problems), the model (Llama-3 may not be well-suited to Lean proof generation), or the limited number of samples (perhaps the curve would eventually become log-linear at even higher $k$ )?

The "inference-time scaling laws" framing is aspirational. Training scaling laws (Kaplan et al., 2020; Hoffmann et al., 2022) have held over many orders of magnitude, model families, and datasets, and have been validated by multiple independent groups. The paper's inference-time power law fits come from a single generation run per model-task pair (no replication across random seeds), use only three model families, and span at most four orders of magnitude. The paper does not test whether the fitted parameters $(a, b)$ predict coverage at $k > 10{,}000$ (out-of-distribution extrapolation), which is the acid test for a scaling law. The paper also does not investigate whether the power law parameters transfer across tasks — e.g., whether $(a, b)$ measured on MATH predicts anything about CodeContests scaling. These are not fatal weaknesses for an initial characterization study, but they mean the "scaling law" claim should be understood as "we observe a recurrent functional form" rather than "we have discovered a predictive law."

Claim: "Coverage curves from different model sizes within the same family have similar shapes and differ primarily by horizontal offset." This claim is supported by the overlay plots in Figure 6, with the Pythia-on-MATH case being particularly compelling (eight model sizes collapse onto one curve). However, the evidence is mixed: the Llama-3-on-CodeContests overlay (Figure 6, bottom-right) shows the base model (8B) diverging in shape from the instruction-tuned models. This suggests that the shape invariance may only hold for models that have undergone similar post-training — instruction tuning appears to change not just the intercept (pass@1) but the scaling shape. The paper does not investigate whether this is due to instruction tuning reducing the diversity of outputs (making scaling more efficient) or some other mechanism.

The overlay analysis is also purely descriptive — the paper does not provide a statistical test for curve similarity (e.g., comparing the fitted $b$ parameters across models and testing whether they are significantly different). The visual collapse is compelling but qualitative. A quantitative analysis (e.g., measuring the residual variance after shifting and testing whether it is significantly larger than sampling noise) would strengthen the claim.

Furthermore, the overlay is performed post-hoc — the anchor coverage $c$ and the horizontal shift are chosen to maximize visual alignment. This is not a predictive procedure; it cannot be used prospectively to estimate a new model's scaling curve from its pass@1. For the observation to become practically useful, one would need to demonstrate that the shift amount (log of the k-value at the anchor coverage) is predictable from measurable model properties (e.g., pass@1, or training compute, or benchmark performance on related tasks).

Claim: "Common verification methods plateau beyond approximately 100 samples, creating a growing gap with coverage." This claim is strongly supported by Figure 7 and the exact MATH numbers (coverage from 82.9% to 98.44% vs. practical methods from 40.50% to 41.41%). The evidence is clean, consistent across both Llama-3 model sizes and both GSM8K and MATH, and the statistical protocol (100 random subsets, error bars in Figure 7) is appropriate.

However, the paper tests only three verification methods, and only one reward model (ArmoRM-Llama3-8B-v0.1). The conclusion that "common verification methods don't scale" could be strengthened by evaluating additional verifiers: process reward models (Lightman et al., 2023), LLM-as-judge prompting, or ensemble verifiers. The paper also does not test whether fine-tuning the reward model on in-distribution data (GSM8K/MATH training sets, or even on the base model's own outputs) would improve scaling — it uses a general-purpose reward model off-the-shelf. It is possible that a task-specialized reward model would scale further, and the paper cannot rule this out.

More importantly, the paper establishes that verification fails to scale but does not deeply investigate why the reward model saturates. Is the reward model failing because it assigns high scores to incorrect solutions that happen to look convincing (false positives)? Or is it failing because correct solutions with rare answers look unusual and receive low scores (false negatives)? Figure 8 suggests the former (rare correct answers are drowned out by high-scoring false positives), but the paper does not present a confusion matrix or calibration analysis for the reward model. Such an analysis would not only strengthen the diagnosis but would also suggest which direction of verifier improvement (reducing false positives vs. improving sensitivity to rare correct answers) is most critical.

What experiments would have strengthened the paper:

Out-of-distribution extrapolation test for the power law. Generate $N = 50{,}000$ samples for one model-task pair and test whether the power law fitted on the first 10,000 samples accurately predicts coverage at 20,000, 30,000, and 50,000. This would validate (or falsify) the "scaling law" interpretation.
Replication across random seeds. All coverage curves come from one generation run per model-task pair (one set of $N = 10{,}000$ samples). The sampling variance of the pass@k estimator is zero (given the samples) because the formula is exact, but the samples themselves are random. Generating a second independent set of 10,000 samples and comparing the resulting coverage curves would reveal the variance due to sampling and test whether the fitted power law parameters $(a, b)$ are stable across seeds.
Ablation over temperature for a non-SWE-bench task. The paper uses different temperatures for different tasks but does not systematically vary temperature on any single task (except the SWE-bench sweep). Showing coverage curves for temperature $\in \{0.3, 0.6, 0.9, 1.2\}$ on MATH or CodeContests would reveal whether the scaling shape (the $b$ parameter) is temperature-invariant or whether higher temperatures produce steeper or shallower scaling.
Comparison with non-sampling test-time compute methods. The paper studies only independent repeated sampling. How does this compare against beam search, best-of-N with a trained verifier, or iterative refinement at the same total inference FLOPs? The paper positions itself as a characterization study rather than a bakeoff, but including one such comparison (e.g., "for MATH, does beam search with a process reward model achieve better coverage at equal FLOPs than independent sampling?") would help practitioners choose between inference-time strategies.
Difficulty-stratified scaling analysis. The paper mentions (in Figure 8) that problems where correct solutions are rare drive the coverage-verification gap, but it does not stratify the coverage curves by problem difficulty (e.g., pass@1 bins). Such stratification would reveal whether coverage scaling is dominated by easy problems (where a few samples suffice) or hard problems (where many samples are needed), and whether the scaling shape differs by difficulty. This would connect the paper's findings to the "compute-optimal test-time scaling" literature that conditions on problem difficulty.
A predictive model for the horizontal offset in the overlay analysis. If within-family scaling invariance holds, the horizontal offset should be predictable from, say, the model's training compute or its performance on a related benchmark. The paper observes the invariance but does not attempt to model it. A regression of offset vs. parameter count or training FLOPs (even with only the available data points) would test whether the invariance can be made practically useful.

Data quality caveats: The GSM8K and MATH evaluations use subsets of only 128 randomly-sampled problems from the test sets. This is a pragmatic choice to keep generation costs manageable at $N = 10{,}000$ samples per problem, but it means the coverage estimates have non-trivial binomial uncertainty: for a true coverage of 50%, 128 problems yields a 95% confidence interval of approximately ±8.7 percentage points. The verification experiments (100 random subsets) account for subset-sampling variance of the selection method but not for problem-sampling variance of the test set itself. The paper does not report confidence intervals for the coverage curves that account for the finite problem count. This makes precise comparisons (e.g., is the 56% SWE-bench coverage statistically significantly above the 43% SOTA?) difficult without additional assumptions.

The SWE-bench flaky test handling: Running each solution's test suite 11 times and majority voting is a reasonable ad-hoc solution, but it does not address the root cause — some problems have inherently non-deterministic test outcomes. The paper correctly reports results both with and without flaky problems (Figure 9) and shows the trend is robust. However, removing 34 of 300 problems (11.3%) changes the absolute coverage numbers by 1–2 percentage points, which is within the noise introduced by the 300-problem sample size. The flaky test issue is more a caution for SWE-bench users than a threat to the paper's conclusions.

Conclusion on experimental support: The paper's central empirical claims — that coverage scales log-linearly with sample count across diverse tasks and models, that this scaling can be approximated by an exponentiated power law, and that practical verification methods saturate around 100 samples — are supported by the presented experiments. The evidence is strongest for the universality of coverage scaling (Figures 2 and 3) and for the verification saturation (Figure 7). The evidence is moderate for the exponentiated power law as a "scaling law" (the fits are reasonable but unvalidated out-of-distribution, and MiniF2F is a counterexample) and for the within-family shape invariance (the overlay analysis is qualitative and post-hoc, and fails for instruction-tuned vs. base Llama models on CodeContests). The FLOPs-matched and API cost comparisons (Figure 4, Table 1) are convincing demonstrations that repeated sampling from weaker models can be cost-effective, but they are specific to the tested model-task pairs and do not provide a general framework for predicting when this will be the case. The paper's contributions are strongest as a systematic empirical characterization that establishes baselines and identifies patterns; its weakest points are the lack of out-of-distribution validation, missing ablations, and the qualitative nature of some secondary claims.

6. Limitations and Trade-offs

Difficulty Estimation Is Implicit, Not Explicit

The paper’s entire framework — measuring coverage as a function of sample budget — treats all problems as benefiting equally from additional samples. The coverage metric averages over all problems in the test set, producing a single curve that implicitly assumes homogeneity across problems. In reality, some problems achieve high coverage with only a few samples (easy problems where the model has high per-sample accuracy), while others remain unsolved even at $k = 10{,}000$ (hard problems where the model never generates a correct solution, as demonstrated by the Pythia zero-coverage result on CodeContests in Section 2.2). The paper does not stratify coverage curves by problem difficulty (e.g., binned by pass@1), nor does it estimate how many samples are needed to achieve a given coverage level for a specific problem. The only difficulty-adjacent analysis appears in Figure 8, which shows per-problem correct-sample frequencies but does not connect this to scaling curves.

The consequence is that a practitioner cannot use the paper’s results to make per-problem budget allocation decisions. The uncovered scaling curves tell you that, on average, 1,000 samples yield 80% coverage on Task X, but they do not tell you whether your specific problem — which might be unusually hard or easy — needs 10 samples or 50,000. This matters because in many deployment scenarios (customer support, code review, tutoring), the cost of each problem matters individually, and a uniform budget is wasteful: easy problems get more samples than needed, while hard problems get fewer than required to solve them. The paper provides no mechanism for estimating problem difficulty before sampling, no adaptive allocation strategy, and no analysis of how coverage gains are distributed across the difficulty spectrum. This contrasts with work like Brown et al. (2024), which explicitly conditions test-time compute strategies on estimated difficulty.

Evidence in the paper: The paper does not measure or report difficulty-stratified coverage curves. The closest evidence is Figure 8 (Section 4.2), which shows the distribution of per-problem correct-sample frequency, and Table 2 (Section 4.1), which bins Chain-of-Thought faithfulness by pass@1 frequency — but neither presents how coverage scales separately by difficulty bin. The Pythia zero-coverage result on CodeContests (Section 2.2, Figure 3) demonstrates that coverage scaling can fail entirely for models below a capability threshold, but the paper does not generalize this observation to per-problem difficulty within a single model’s output distribution.

Mitigation status: The paper does not attempt to address this limitation. It mentions in Section 5 that “access to existing samples, particularly if verification tools can provide feedback on them, may be helpful when generating future attempts,” which gestures at adaptive strategies, but provides no implementation or analysis. Future work on difficulty-adaptive sampling would require a difficulty estimation mechanism, which the paper does not develop.

Coverage Is Measured With an Oracle Verifier, But the Precision Bottleneck Is Only Diagnosed, Not Solved

The paper’s headline coverage gains — 56% on SWE-bench Lite, 98.44% on MATH, 7.1% on CodeContests with Gemma-2B — are all measured using perfect (oracle or automatic) verifiers. For tasks with automatic verifiers (CodeContests, MiniF2F-MATH, SWE-bench Lite), this is reasonable: the verifier exists and works. For tasks without automatic verifiers (GSM8K, MATH), the coverage numbers represent an upper bound that is unreachable with current verification methods. The paper demonstrates this starkly in Figure 7: coverage on MATH with Llama-3-8B-Instruct rises from 82.9% to 98.44% as $k$ increases from 100 to 10,000, while practical verification methods (majority voting, reward model best-of-N, reward model weighted voting) improve only from 40.50% to 41.41% over the same range.

The consequence is that for a large class of important tasks — any domain without automatic verification, which includes most natural language reasoning, question answering, summarization, and creative generation — the paper’s coverage numbers are aspirational, not actionable. A practitioner reading that 98.44% coverage is achievable on MATH with 10,000 samples cannot realize that number in practice. They will get roughly 41%, and spending more on inference compute beyond ~100 samples yields negligible improvement. The paper identifies this as the central bottleneck but offers no solution and only minimal diagnosis of why the tested verification methods saturate. It tests only one reward model (ArmoRM-Llama3-8B-v0.1) and does not analyze whether the saturation is due to false positives (high-scoring incorrect answers), false negatives (low-scoring correct answers), or both. Without this diagnostic, practitioners cannot determine whether investing in better verifiers, different verification strategies, or alternative approaches (e.g., formalizing problems into verifiable languages) is the right path forward.

Evidence in the paper: Figure 7 (Section 4.1) provides the direct evidence — the growing gap between oracle coverage and practical verification success on GSM8K and MATH across sample budgets. Figure 8 (Section 4.2) shows that low-frequency correct solutions (those sampled in <1% of attempts) are predominantly missed by majority voting, establishing that the verification bottleneck is concentrated on problems where correct answers are rare. Table 2 (Section 4.1) demonstrates that >90% of correct chains-of-thought are faithful, ruling out the hypothesis that correct answers are generated via nonsense reasoning that a verifier could not possibly recognize. However, the paper does not present a confusion matrix, calibration curve, or error analysis for the reward model, leaving the precise failure mode uncharacterized.

Mitigation status: The paper explicitly frames verification as an open problem — “our results from Section 4 highlight the importance of improving sample verification methods when tools for automatically doing so are unavailable” (Section 5). It suggests “equipping models with the ability to assess their own outputs” and “designing converters that can make an unstructured task verifiable, for example by formalizing an informal math statement into a language like Lean” as future directions. These are useful pointers but are not implemented or evaluated in the paper. The limitation stands as a barrier that the paper identifies but does not lower.

The “Scaling Law” Is Fitted Post-Hoc on a Single Run Per Model-Task Pair, Without Out-of-Distribution Validation or Cross-Seed Replication

The paper proposes that coverage scales with sample count according to an exponentiated power law $c \approx \exp(a k^b)$ and presents this as an “inference-time scaling law” analogous to training scaling laws (Kaplan et al., 2020; Hoffmann et al., 2022). However, the evidence for this claim is substantially weaker than for training scaling laws. Training scaling laws have been validated by training multiple models at different compute budgets and showing that the fitted parameters predict held-out budgets. The paper’s inference-time power law parameters are fitted to a single generation run of $N = 10{,}000$ samples per model-task pair, with no held-out validation at higher sample counts, no replication across independent generation runs (different random seeds), and no test of whether the fitted $(a, b)$ predict coverage at $k > N$ or on different model sizes or tasks.

The consequence is that the “scaling law” framing is descriptive, not predictive. The power law fit tells you that the observed coverage curve can be summarized by two parameters — a useful compression, but not a law. A practitioner cannot take a fitted $(a, b)$ from, say, Llama-3-8B-Instruct on MATH and reliably predict what coverage Llama-3-8B-Instruct would achieve with 50,000 samples, or what coverage a hypothetical Llama-3-8B-Base would achieve. The paper does not demonstrate that the power law parameters transfer across model sizes (the within-family overlay analysis in Figure 6 suggests the shape may be invariant, but the paper does not test whether $(a, b)$ fitted on one model size predicts another’s curve), across tasks, or across sampling hyperparameters (temperature, top-p). A scaling law that cannot extrapolate is not a law — it is a curve fit.

Additionally, the paper acknowledges that the power law fit fails on MiniF2F-MATH (Figure 5, bottom-right: error 0.0297 ± 0.0157, with visible deviation between fit and data), and fits are poorer for Pythia models than for Llama models (Figure 10). These exceptions are noted but not investigated — the paper does not hypothesize why some tasks or model families produce poorer fits, which limits the generality of the claim. The single-run nature of the data means that sampling variance in the generated solutions (which are stochastic) could affect the fitted parameters — a different set of 10,000 samples might yield different $(a, b)$ values, but the paper provides no estimate of this variance.

Evidence in the paper: Section 3.1 states the power law model and reports fitting errors. The paper explicitly notes that “these laws are not as exact as training scaling laws (most strikingly on MiniF2F-MATH)” (Section 3.1). The fitting procedure (Appendix C.1) uses 40 evenly-spaced log-scale points from 0 to 10,000, with SciPy’s curve_fit — a standard but purely interpolative procedure. No extrapolation experiment is reported.

Mitigation status: The paper is transparent that the fits are imperfect and does not claim universality. The language is appropriately hedged: “they provide encouraging early evidence that the benefits of inference scaling can be characterized” (Section 3.1). However, the paper does not propose a validation protocol (e.g., fitting on the first 5,000 samples and predicting coverage at 10,000; or generating 50,000 samples for one model-task pair and testing the prediction), nor does it discuss the statistical uncertainty of the fitted parameters. This is a limitation that future work must address to elevate the observation from a descriptive curve fit to a predictive scaling law.

The Generation Budget Is Pure Sample Count — Latency, Memory, and System-Level Costs Are Unaccounted For

The paper operationalizes inference compute as the number of independent samples $k$ . When translating to cost, it uses either inference FLOPs (Section 2.3) or API pricing (Table 1). Both metrics implicitly assume that generating $k$ samples costs $k$ times the cost of generating 1 sample — that the marginal cost per sample is constant and that total cost is linear in $k$ . In practice, this is not true for two reasons: (1) shared-prefix attention optimizations (Hydragen, SGLang, bifurcated attention) reduce the per-sample cost when many samples share the same prompt, making repeated sampling cheaper than the linear-cost assumption; (2) sequential dependencies in generation mean that generating $k$ samples in parallel requires $k \times$ the memory and may hit hardware limits (GPU memory, batch size ceilings) that force serial execution, making repeated sampling more expensive in wall-clock time than the linear-cost assumption.

The consequence is that the paper’s cost comparisons are approximate and may not reflect real deployment economics. The FLOPs-matched comparison (Figure 4) uses a per-token FLOP formula that does not account for the batch-size-dependent throughput gains that repeated sampling workloads can exploit on high-throughput inference systems. The paper acknowledges this in Section 5: “Repeated sampling inference can therefore be accomplished at a lower cost than naively making many parallel requests to a chatbot-oriented API.” This means the FLOPs-based comparison may understate the cost advantage of repeated sampling — the actual dollar or time savings could be larger. Conversely, the API cost comparison (Table 1) uses per-attempt pricing for DeepSeek-Coder-V2-Instruct at $0.0072 per attempt, which may not reflect the bulk pricing or throughput discounts available when generating 250 attempts per problem across 300 problems. Practitioners running their own infrastructure would face different cost structures.

More critically, the paper’s focus on sample count ignores latency. Repeated sampling is embarrassingly parallel: all $k$ samples can be generated simultaneously if sufficient hardware is available. But in latency-constrained settings (interactive applications, real-time systems), the wall-clock time to generate $k$ samples is bounded by the slowest sample, not the average. If generating one solution takes 10 seconds and the practitioner needs an answer within 15 seconds, $k$ is capped at roughly 2–3 samples regardless of FLOP budget. The paper does not discuss latency constraints, throughput-latency tradeoffs, or the hardware requirements (number of GPUs, memory per GPU) to achieve a given level of parallelism. This is a significant gap for practitioners deploying repeated sampling in production.

Evidence in the paper: Section 5 discusses inference systems and notes that “Repeated sampling is a distinct LLM inference workload from serving chatbot requests” and that “repeated sampling can benefit from specialized attention optimizations that exploit overlaps in prompts.” However, this discussion is qualitative — no experiments measure the actual throughput or latency of repeated sampling on real hardware, and no comparison of batching strategies or parallelism configurations is provided. The paper’s cost numbers (FLOPs and API pricing) are the only quantitative cost metrics reported.

Mitigation status: The paper acknowledges the system-level differences (Section 5) and mentions that these can make repeated sampling “more cost-effective than naively making many parallel requests,” but does not quantify these effects. Future work would need to benchmark repeated sampling throughput and latency on specific hardware configurations, with attention to batch size, memory pressure, and the price-performance of different parallelism strategies (e.g., tensor parallelism vs. data parallelism for serving many copies of the same prompt).

Automatic Verifiers Are Assumed to Be Deterministic and Perfect — But the Paper Shows They Are Not

The paper’s central thesis is that in domains with automatic verifiers, coverage gains translate directly into improved task performance. However, the paper’s own analysis reveals that automatic verifiers in two of the three “automatable” domains have significant imperfections. On CodeContests, 35 of 122 problems (28.7%) with Python3 reference solutions have at least one “correct” solution that fails the test suite (Section 4.2.2) — the verifier produces false negatives. On SWE-bench Lite, 34 of 300 problems (11.3%) have flaky test suites that produce inconsistent results on the same candidate solution, requiring the paper to implement a workaround (11 runs + majority voting) to approximate a deterministic verdict (Section 4.2.1, Appendix B.2).

The consequence is that the paper’s coverage numbers for CodeContests and SWE-bench Lite are noisy measurements, not ground truth. On CodeContests, the measured coverage is a lower bound — some fraction of genuinely correct model outputs are classified as incorrect, meaning the model’s true capability is higher than reported. The paper does not estimate the magnitude of this underestimation. On SWE-bench Lite, the flaky test workaround (11 runs + majority voting) adds measurement noise: a solution that passes 6/11 runs is counted as passing, while a solution that passes 5/11 is counted as failing, even though both might be “correct” under a more robust test suite. The paper shows (Figure 9) that removing the 34 flaky problems produces similar scaling trends, but this changes the absolute coverage numbers (56% on all 300 problems vs. 54.14% on 266 clean problems at $k=250$ ) and removes over 11% of the benchmark — a non-trivial subset.

More broadly, this limitation exposes a tension in the paper’s framing: the paper treats “tasks with automatic verifiers” as a clean category where coverage equals success rate, but two of the three such tasks have known verification imperfections. The only task with truly reliable verification is MiniF2F-MATH (the Lean4 proof checker is sound — if it accepts a proof, the proof is correct), and that is also the task where the power law fit is poorest (Figure 5). This suggests that the simplest, most principled verification regime (formal proofs) may have different scaling characteristics than the pragmatic, imperfect verification regimes (unit tests, test cases) that dominate practical applications.

Evidence in the paper: Section 4.2.1 documents the SWE-bench flaky tests and lists all 34 affected problem IDs in Table 3 (Appendix B). Figure 9 (Appendix B.2) shows coverage curves with and without flaky problems. Section 4.2.2 quantifies the CodeContests false negative rate (35/122 problems with failing reference solutions) and explains the root causes (multiple valid outputs, malformed generated test inputs). The paper explicitly flags these as “cautionary tales” and notes that “applying repeated sampling to these problems contains an element of ‘rolling the dice’ to generate a solution that is not only correct, but emits the particular outputs that pass the tests” (Section 4.2.2).

Mitigation status: The paper handles SWE-bench flakiness by running each solution’s test suite 11 times and majority voting (Section 4.2.1). This is a reasonable ad-hoc fix but does not address the root cause — the test suites contain non-deterministic or incorrectly-specified tests. For CodeContests, no mitigation is attempted; the paper reports results against the standard test harness and flags the false negative issue for practitioners. The paper does not propose improved verification infrastructure or discuss how much the false negative rate would need to be reduced for repeated sampling to become reliably actionable. This is a pragmatic acknowledgment of real-world messiness, but it means the “automatic verifier” category is less clean than the paper’s framework implies.

The Paper Does Not Compare Repeated Sampling Against Other Test-Time Compute Strategies

The paper studies exactly one approach to scaling inference compute: generate $k$ independent samples from the same model with the same prompt and fixed temperature, then use a verifier to select the best one. This is the simplest possible test-time compute strategy, which the authors explicitly choose as a deliberate simplification to establish a baseline (Section 5). However, the paper does not compare this approach against any alternative — not beam search, not iterative refinement with execution feedback, not adaptive temperature scheduling, not majority voting with a process reward model, not best-of-N with a trained outcome verifier, and not any of the strategies studied in contemporaneous work on compute-optimal test-time scaling (Brown et al., 2024; Snell et al., 2024).

The consequence is that the paper cannot answer the question a practitioner most wants answered: given a fixed compute budget, should I use repeated sampling or some other test-time strategy? The paper demonstrates that repeated sampling works, but provides no evidence that it works better than alternatives at equal cost. For tasks with automatic verifiers, beam search guided by execution feedback might achieve higher coverage per sample than independent sampling. For tasks with verification bottlenecks, majority voting with a process reward model might saturate at a higher level than the general-purpose reward model the paper tests. The paper’s contribution is showing that repeated sampling scales — not that it scales optimally. By not including any comparative baselines, the paper leaves open the question of whether the observed coverage curves represent the best achievable inference-time performance or merely a lower bound that more sophisticated strategies could exceed.

This limitation is particularly acute for the FLOPs-matched comparison (Figure 4), which answers “Llama-3-8B with many samples vs. Llama-3-70B with few samples” but does not answer “Llama-3-8B with repeated sampling vs. Llama-3-8B with beam search” or “Llama-3-8B with repeated sampling vs. Llama-3-8B with iterative refinement.” The cost analysis in Table 1 compares DeepSeek with 5 repeated samples against GPT-4o/Claude with 1 sample — but does not compare DeepSeek with 5 independent samples against DeepSeek with 2 multi-turn refinement trajectories (which might provide similar or better performance at equal or lower cost). The paper’s strong claims about inference compute as a scaling axis are specific to independent repeated sampling — they do not generalize to inference compute spent on other strategies without additional evidence.

Evidence in the paper: Section 5 explicitly lists “improving repeated sampling” as a future direction, mentioning solution diversity mechanisms (AlphaCode-style metadata conditioning), multi-turn interactions with execution feedback, and learning from previous attempts as promising avenues. These are acknowledgments that the current approach is a baseline, not an optimized strategy. However, the paper provides no experiments comparing any of these alternatives against the baseline, even on a single task.

Mitigation status: The paper does not mitigate this limitation experimentally. It frames its contribution as characterization rather than optimization: “our goal is to systematically characterize these benefits across a range of tasks, models, and sample budgets” (Section 1). This is a valid scope, and the baseline characterization is valuable. However, for a practitioner deciding how to spend inference compute, the absence of comparative baselines means the paper’s results should be interpreted as a lower bound on what inference-time scaling can achieve, not a prescription that independent repeated sampling is the best approach. Future work that compares repeated sampling against search, refinement, and adaptive strategies at equal cost would be needed to guide deployment decisions.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not introduce a new model architecture, training procedure, or inference algorithm. It makes a simpler and arguably more foundational contribution: it converts inference-time compute from an anecdotal trick into a measurable, characterizable, and actionable scaling axis. Before this work, the field knew that repeated sampling sometimes helped — AlphaCode used a million samples for competitive programming, CodeLlama reported pass@k, and Greenblatt observed log-linear scaling on ARC — but these were isolated demonstrations, not a systematic framework. The paper's contribution is the framework itself: the coverage-precision decomposition, the exponentiated power law functional form, and the across-task, across-model-family empirical sweep that establishes inference-time scaling as a general phenomenon rather than a domain-specific curiosity.

The magnitude of this shift is best understood as a reframing rather than a paradigm shift. The paper does not overturn the pretraining-centric view of LLM capability — training bigger models on more data remains the primary driver of progress. Instead, it introduces a second, independent dimension of capability that was hiding in plain sight. The implication is that model evaluation and model deployment must now account for this dimension. A single-number pass@1 score — the standard metric on most leaderboards — systematically understates what a model can do. The "capability" of a model is not a point; it is a curve. This reframing is analogous to the shift from "classification accuracy" to "accuracy-vs-compute Pareto curves" in the neural architecture search literature or from "single-seed results" to "mean-and-variance across training runs" in the reinforcement learning community. It adds a dimension that was previously collapsed.

The paper resolves a latent tension in the LLM deployment community. Practitioners have long faced a choice: pay for a frontier model (GPT-4o, Claude) and accept high per-query costs, or use an open-source model (Llama, DeepSeek) and accept lower quality. The implicit assumption was that this is a quality-cost tradeoff on a single curve — you get what you pay for. The paper demonstrates that this tradeoff curve is not one-dimensional. The combination of a weaker model with many samples can sit above and to the left of the frontier-model single-sample point on a quality-cost plot — simultaneously better and cheaper (Table 1: DeepSeek 5× samples at 3.6× lower cost than single-sample GPT-4o, while achieving higher accuracy). This does not mean cheap models always beat expensive ones — the Pythia zero-coverage result on CodeContests (Figure 3) is a hard counterexample — but it means the decision space is a two-dimensional surface (model strength × sample count) rather than a one-dimensional curve. Practitioners must now consider both dimensions when optimizing for cost-quality tradeoffs.

This paper redirects research attention toward verifier scaling. Before this work, the primary obstacle to scaling inference compute was presumed to be generation diversity — can models produce enough different solutions that at least one is correct? The paper's most striking finding is that this is not the bottleneck in practice. For tasks with automatic verifiers, generation scaling works smoothly across four orders of magnitude. The bottleneck is verification: on GSM8K and MATH, coverage reaches >95% with 10,000 samples, but practical verifiers stall at ~41% after ~100 samples (Figure 7). This reframes the research agenda. Instead of asking "how can we make models generate better diverse solutions?" — a direction that has attracted significant effort (tree search, temperature annealing, prompt ensembling) — the priority should be "how can we build verifiers that scale to identify rare correct solutions in large candidate pools?" The paper shows that the gap between oracle coverage and practical verification widens with additional samples (Figure 7). Every dollar spent on generation beyond ~100 samples is wasted if verification doesn't improve. This makes verifier robustness — particularly sensitivity to low-frequency correct solutions and resistance to high-scoring false positives — the critical path for scaling inference compute in non-automatable domains.

The paper makes certain previously attractive research directions less compelling. The observation that independent repeated sampling produces smooth, log-linear coverage gains over four orders of magnitude suggests that elaborate search strategies may be unnecessary in many settings. Beam search, Monte-Carlo tree search, and iterative self-refinement add complexity and cost but may not substantially improve coverage per FLOP compared to independent sampling — or at minimum, the burden of proof now lies on more complex methods to demonstrate gains over the simple repeated-sampling baseline on a FLOPs-matched basis. Similarly, the verification saturation result (Figure 7) suggests that majority voting and off-the-shelf reward models are dead ends for scaling beyond ~100 samples — improving these methods incrementally (e.g., better reward model architectures) is unlikely to close the 50+ percentage point gap between oracle and practical performance. What is needed is a fundamentally different verification paradigm — perhaps task-specific verifier fine-tuning on model-generated outputs, process-level verification that scores reasoning steps rather than final answers, or formalization-to-verifiable-language pipelines — and the paper provides the quantitative benchmark against which such paradigms should be measured.

This work also elevates inference systems from an infrastructure afterthought to a research domain. The paper notes that repeated sampling is a fundamentally different workload from chatbot serving — it prioritizes throughput over latency, benefits from shared-prefix attention optimizations, and can exploit high batch sizes (Section 5). As the community begins to treat inference compute as a first-class resource to be optimized, inference systems that can generate 10,000 samples cost-effectively become as important as training systems that can process trillions of tokens. This connects the paper's findings to the growing literature on inference optimizations (Hydragen, SGLang, bifurcated attention), reframing these not as engineering niceties but as enabling technologies for a new scaling paradigm.

Follow-Up Research This Work Enables

Per-difficulty and adaptive sampling allocation. The paper's coverage curves aggregate over all problems in a test set, but the per-problem correct-sample frequencies (Figure 8) reveal massive heterogeneity: some problems are solved on 80% of attempts, others on 0.1%. A natural extension is to stratify coverage curves by problem difficulty (binned by pass@1) and measure whether the exponentiated power law parameters $(a, b)$ differ systematically across difficulty levels. Does coverage improve primarily through easy problems (where a few samples suffice to achieve near-100% coverage) or hard problems (where many samples are needed for small gains)? The answer has direct practical implications: if easy problems saturate quickly, the compute-optimal strategy would allocate few samples to them and redirect the budget to medium-hard problems where coverage is still rising. A strong follow-up would: (a) bin MATH or CodeContests problems into difficulty quintiles based on pass@1, (b) fit separate power laws per bin, (c) simulate an adaptive policy that estimates difficulty from the first 10 samples and allocates the remaining budget accordingly, and (d) measure whether adaptive allocation achieves higher aggregate coverage at fixed total budget than uniform allocation. This connects directly to the "compute-optimal test-time scaling" framework of Snell et al. (2024) but with the paper's simpler independent-sampling baseline.

Out-of-distribution validation of the exponentiated power law. The paper fits power laws to coverage curves spanning $k = 1$ to $k = 10{,}000$ but never tests whether the fitted parameters extrapolate. A definitive stress test would: (a) generate $N = 50{,}000$ or $100{,}000$ samples per problem for one model-task pair (e.g., Llama-3-8B-Instruct on MATH), (b) fit the power law on the first 10,000 samples, (c) use the fitted $(a, b)$ to predict coverage at $k = 20{,}000$ , $30{,}000$ , $50{,}000$ , and (d) compare the predictions against the actual coverage computed from the full sample set. If the power law holds out-of-distribution, it graduates from a descriptive curve fit to a predictive scaling law and could be used by practitioners to estimate required sample budgets for target coverage levels. If it breaks (e.g., coverage saturates more sharply than the power law predicts), that reveals the functional form's limitations and motivates richer models (e.g., sigmoid or broken power law). The paper's own evidence hints at saturation on some tasks — Llama-3-8B-Instruct on MATH reaches >98% coverage at 10,000 samples, leaving only ~2 percentage points of headroom, making the shape of the approach to 100% a critical open question.

Training task-specific verifiers on model-generated outputs and measuring their scaling behavior. The paper's verification analysis (Section 4.1) uses a single off-the-shelf reward model (ArmoRM-Llama3-8B-v0.1) and finds it saturates around 100 samples. But the paper also demonstrates that the chain-of-thought reasoning in correct solutions is >90% faithful (Table 2), suggesting signal exists for a verifier to exploit. A critical follow-up would: (a) fine-tune a process reward model (PRM) or outcome reward model (ORM) on the base model's own MATH/GSM8K outputs, using the Monte Carlo rollout or best-of-N labeling procedures from Lightman et al. (2023) or Wang et al. (2024), (b) evaluate its verification scaling curve using the same 100-random-subset protocol as Figure 7, and (c) measure whether the task-specific verifier saturates at a higher sample count or accuracy level than the off-the-shelf reward model. The hypothesis is that distribution shift between the reward model's training data and the base model's output distribution causes the saturation — a verifier trained specifically on Llama-3-8B-Instruct's MATH outputs might continue improving beyond 100 samples, narrowing the gap with oracle coverage. A negative result (even a task-specific verifier saturates) would be equally informative: it would suggest that the bottleneck is fundamental to scoring-based verification when correct answers are very rare, not an artifact of distribution shift, and would motivate alternative verification paradigms.

Repeated sampling with multi-turn execution feedback on code and proof tasks. The paper's CodeContests and MiniF2F experiments use single-turn generation — the model produces a complete solution without seeing execution results or type-checking feedback. This contrasts with how humans solve these problems (iteratively writing, testing, and debugging) and with state-of-the-art coding agents that incorporate execution feedback. A natural extension would: (a) allow the model to see test execution results (CodeContests) or Lean4 error messages (MiniF2F) after each generation attempt, (b) let it produce a revised solution conditioned on this feedback, and (c) measure whether multi-turn trajectories achieve higher coverage at equal total generation budget than independent single-turn samples. The tradeoff is that each multi-turn trajectory costs more tokens (multiple LLM calls) but may have a higher per-trajectory success probability. The paper's repeated sampling framework provides a clean baseline for this comparison: at a fixed FLOP budget, does $N$ independent single-turn samples beat $N/5$ five-turn trajectories? The SWE-bench experiments already use multi-turn agent trajectories (each "sample" is an entire agent loop), suggesting the framework naturally extends — the question is whether interleaving verification feedback into the sampling loop improves sample efficiency.

Cross-model-family and cross-task transfer of scaling law parameters. The paper observes that within a model family (Pythia, Gemma, Llama), coverage curves from different model sizes have similar shapes after horizontal shifting (Figure 6). This is purely descriptive. A predictive extension would: (a) measure whether the shape parameter $b$ is approximately constant across model sizes within a family (by fitting separate power laws and testing whether $b$ differs significantly), (b) if so, develop a model that predicts the horizontal offset (equivalently, the pass@1) from measurable model properties — parameter count, training FLOPs, or benchmark performance on related tasks — and (c) test whether this predictive model generalizes to held-out model sizes (e.g., fit $(a, b)$ on Pythia-1B through Pythia-12B, predict the coverage curve for Pythia-160M, and compare against its empirical curve). If successful, this would allow practitioners to estimate the full coverage curve for a new model from a single pass@1 measurement, dramatically reducing the cost of characterizing inference-time scaling. A negative result — offsets are not predictable from simple model properties — would itself be informative, suggesting that coverage scaling depends on subtle aspects of the model's output distribution (e.g., diversity, calibration) not captured by parameter count or training compute.

Verifier failure mode analysis: false positives vs. false negatives at scale. The paper shows that verification saturates (Figure 7) and that low-frequency correct solutions are primarily missed (Figure 8), but does not dissect why the reward model fails. A mechanistic follow-up would: (a) take the Llama-3-8B-Instruct MATH sample collection at $k = 1{,}000$ or $k = 10{,}000$ , (b) for each problem where the reward model + best-of-N selects an incorrect answer, compute the histogram of reward scores for correct vs. incorrect samples, (c) measure whether the failure is due to (i) false positives: one or more incorrect samples receiving higher scores than any correct sample, pointing to reward model miscalibration, or (ii) false negatives: correct samples receiving systematically lower scores than they should, pointing to reward model bias against the distribution of correct answers, and (d) compute the expected number of false positives as a function of sample count given the reward model's empirical score distribution, to test whether the observed saturation point (around 100 samples) is predicted by a simple model of random false-positive occurrence. This would diagnose whether improving the reward model's precision (reducing false positives) or its recall (increasing sensitivity to correct answers) is the higher-priority goal for scaling verification.

Practical Applications and Downstream Use Cases

Cost-efficient inference for high-stakes code generation. The SWE-bench Lite result — 56% issue resolution at $10.80 total cost vs. 24.67% at$ 39 for single-attempt GPT-4o (Table 1) — has immediate implications for software engineering tools. A team maintaining a CI/CD pipeline or an automated bug-fixing system can deploy DeepSeek-Coder-V2-Instruct with a sample budget of 5–10 attempts per issue and achieve better results than calling a frontier model once, at a fraction of the cost. The key practical insight is that the cost savings are not marginal — they represent a 3–4× reduction in API spend while simultaneously improving outcomes. For a startup processing 10,000 issues per month, the difference between $390,000 (GPT-4o single-sample) and$ 108,000 (DeepSeek 5× sampling) is a meaningful fraction of engineering budget. The paper provides sufficient detail (Moatless Tools version, Voyage AI embeddings, temperature 1.6) for teams to replicate this deployment with off-the-shelf components.

Model evaluation beyond pass@1. The paper demonstrates that single-attempt evaluation systematically understates model capabilities. For model developers (Meta, Google, Anthropic) and benchmark maintainers (HELM, LM-Eval, Open LLM Leaderboard), the implication is that pass@k curves should become a standard part of model reporting. A model card showing that Llama-3-8B-Instruct achieves 48% pass@1 on MATH but >98% pass@10k tells a fundamentally different story about the model's mathematical reasoning than the single number 48%. Practically, reporting pass@k curves for a small set of k values (e.g., k = 1, 10, 100, 1000, 10000) requires generating 10,000 samples per problem, which is feasible for a small evaluation set (the paper uses 128 problems) and can be amortized across many models if the sample collections are released publicly (as the paper does). The LM-Eval framework already supports pass@k estimation; the paper provides the motivation for benchmark maintainers to make it a first-class metric alongside pass@1.

Verifier-as-a-service for non-automatable tasks. The verification saturation result (Figure 7) — that off-the-shelf verifiers plateau around 41% accuracy on MATH while oracle coverage exceeds 98% — defines a clear product opportunity. A team that builds a specialized verification service for math reasoning — trained on the base model's own MATH outputs using process-level annotation or Monte Carlo rollout labels — could unlock the 57 percentage points of latent performance that current verifiers leave on the table. The business model is straightforward: the generation phase (producing 10,000 cheap samples from an open-source model) costs far less than calling a frontier model, and the verification phase (scoring those samples with a specialized verifier) adds value by selecting the correct answer from the candidate pool. The paper's numbers suggest that even if the specialized verifier only closes half the gap (reaching ~70% accuracy instead of 41%), it would outperform single-attempt GPT-4o at lower cost. The paper also demonstrates that the chain-of-thought signal exists for verifiers to exploit (Table 2: >90% reasoning faithfulness), de-risking the verifier training problem.

Inference budget as a deployment configuration knob. For teams deploying LLMs in production, the paper provides a methodology for jointly optimizing model size and sample count against a cost constraint. Rather than asking "Llama-3-8B or Llama-3-70B?" as a binary choice, a team can generate coverage curves for each model on their task-specific evaluation set, translate them to FLOPs or dollar cost (using the FLOPs formula from Section 2.3 or API pricing), and select the (model, sample_count) pair that maximizes coverage subject to a per-query budget. The paper's FLOPs-matched analysis (Figure 4) shows this is not a trivial decision: on MATH, the 8B model with many samples beats the 70B model at all FLOP budgets; on CodeContests, the reverse holds. A team deploying a coding assistant would therefore default to the 70B model, while a team deploying a math tutor would use the 8B model with high sample counts. The paper gives practitioners the conceptual framework and the empirical methodology to make these decisions systematically rather than by intuition or vendor marketing.

When to Prefer This Method

The paper does not articulate an explicit decision rule comparing repeated sampling against named alternative inference-time strategies (beam search, iterative refinement, compute-optimal test-time scaling). It positions repeated sampling as the simplest baseline and explicitly acknowledges that more sophisticated strategies may outperform it (Section 5). The paper's contribution is characterizing what this baseline achieves, not arguing that practitioners should prefer it over all alternatives. A "when to prefer" matrix would therefore be an imposition — the paper does not provide the comparative data needed to construct one.

Instead, the paper implicitly defines the conditions under which repeated sampling is worth considering at all:

The task must have a verifiable correctness signal — either automatic (unit tests, proof checkers, test cases) or through a verification method that scales to the intended sample budget. If the best available verifier saturates at 100 samples (as on MATH with current reward models), spending budget beyond that point on generation is wasteful regardless of the sampling strategy.
The model must have non-zero per-sample probability of correctness on the task. The Pythia-on-CodeContests zero-coverage result (Figure 3) establishes that if pass@1 is effectively zero, repeated sampling provides no benefit — there are no correct solutions to find. The paper does not quantify the minimum pass@1 threshold, but practitioners can estimate it by generating a modest number of samples (e.g., 100) and checking whether any are correct.
The cost structure must make additional samples economically viable relative to using a stronger model. The FLOPs-matched analysis (Figure 4) and API cost comparison (Table 1) show that this depends on the task: on MATH and GSM8K, more samples from a smaller model beat fewer samples from a larger one; on CodeContests, the opposite holds. Practitioners should measure this on their specific task rather than extrapolating from the paper's task set.