ArXiv: 2410.13639

🎯 Pitch

OpenAI’s o1 doesn’t just sample more—it thinks differently, using internal patterns like Divide and Conquer and Self-Refinement to bypass the search-space ceilings that stall standard inference-time methods. Human reward models can boost Best-of-N from 15% to 33%, but o1 still dominates because it plans and revises rather than relying on output diversity alone.

1. Executive Summary

This paper empirically analyzes the reasoning patterns of OpenAI's o1 model by comparing it against four Test-time Compute methods—Best-of-N (generating multiple outputs and selecting the best), Step-wise BoN (breaking problems into subproblems with per-step selection), Self-Refine (iterative feedback and refinement), and Agent Workflow (domain-specific system prompts with structured task decomposition)—using GPT-4o as the backbone across three domains: math (AIME), coding (USACO), and commonsense reasoning (HotpotQA, Collie). The o1 model achieves the best performance on most benchmarks, with particularly large gains in math and coding, while the Agent Workflow emerges as the strongest Test-time Compute baseline—substantially outperforming Step-wise BoN and Self-Refine—though it still trails o1 on code and math tasks. The paper identifies six reasoning patterns of o1—Systematic Analysis (SA), Method Reuse (MR), Divide and Conquer (DC), Self-Refinement (SR), Context Identification (CI), and Emphasizing Constraints (EC)—with DC and SR appearing most frequently, suggesting these are the key drivers of o1's reasoning capability. The analysis also demonstrates that reward model quality and search space jointly bound the performance of sampling-based methods, with a human reward model lifting BoN on HotpotQA from under 15% to 33% accuracy, establishing that test-time search methods approach diminishing returns once the backbone model's output diversity plateaus—while o1's internal reasoning patterns circumvent this bottleneck entirely.

2. Context and Motivation

What Gap Does This Paper Address?

The paper tackles a surprisingly under-explored question: what are the actual reasoning mechanisms that make the o1 model work? By the time this paper was written (late 2024), OpenAI's o1 had already generated substantial excitement for its ability to "think before responding," demonstrating dramatic improvements over previous models on math, coding, and multi-step reasoning tasks. However, the model's internal reasoning process was effectively a black box — OpenAI released the model as a product but revealed essentially nothing about its architecture, training methodology, or inference mechanism. The field was left to speculate: was o1 simply running a sophisticated test-time compute strategy like Best-of-N or tree search, but at massive scale? Did it use some form of reinforcement learning to learn an internal "thinking" policy? Did its improvements come from new training data, a novel architecture, or something else entirely?

The paper identifies this gap directly in the abstract:

"the mechanisms behind these methods are still unexplored."

This is not hyperbole. At the time of writing, no systematic analysis had been published that compared o1's performance against explicit test-time compute baselines to understand what o1 was actually doing — what reasoning patterns it deployed, how those patterns varied across task types, and whether existing test-time compute methods could approximate o1's behavior. The paper positions itself as a first step toward filling this gap through careful empirical comparison and qualitative analysis of o1's reasoning traces.

But the paper also addresses a second gap, which is arguably more actionable for the broader research community: the lack of a systematic comparison among test-time compute methods themselves. When this paper was written, the test-time compute landscape was fragmented. Best-of-N (BoN) sampling, step-wise decomposition, self-refinement, and agent-based workflows had each been proposed and evaluated independently, but no study had placed them on equal footing — same backbone model, same benchmarks, same evaluation protocol — to understand their relative strengths and weaknesses. This matters because researchers and practitioners trying to improve their own models needed guidance on which test-time strategy to invest in, and under what conditions each strategy was effective. The paper fills this gap by running all four methods against GPT-4o on the same filtered benchmark set, producing directly comparable results.

The two gaps — understanding o1 and comparing test-time methods — are connected: by establishing the performance ceiling and failure modes of existing test-time compute approaches, the paper can determine whether o1's advantages come from doing similar things better, or from doing something qualitatively different that the community hasn't replicated yet.

Why Is This Problem Important?

The importance has both practical and theoretical dimensions.

Practical importance: cost-efficient deployment. The paper opens by referencing the observation from Snell et al. (2024) that scaling test-time compute "can be more effective than scaling model parameters." This is not just an academic curiosity — it has direct economic implications. Training larger models is enormously expensive (requiring clusters of thousands of GPUs, months of time, and megawatts of power), and the returns from additional parameters are diminishing. If test-time compute can substitute for some of that pretraining investment — as the companion paper by Snell et al. had argued — then organizations making deployment decisions need to understand which test-time strategies work and when. A company deciding between buying a larger model API subscription versus engineering a custom test-time compute pipeline for a smaller model needs empirical guidance. This paper provides some of that guidance by showing, for example, that Agent Workflow with domain-specific prompts can achieve roughly 46% accuracy on Collie compared to o1's ~53% (minivariant) and ~34% (preview), closing much of the gap without requiring access to o1's proprietary internals.

Theoretical importance: understanding what "reasoning" means in LLMs. The question "what makes o1 good at reasoning?" is really a question about what reasoning means for current-generation language models. Is reasoning fundamentally about search — expanding the number of candidate solutions and selecting the best one (the BoN paradigm)? Is it about decomposition — breaking complex problems into simpler sub-problems and solving them sequentially (the Step-wise BoN and Agent Workflow paradigms)? Is it about self-critique — iteratively identifying and fixing errors (the Self-Refine paradigm)? Or is it something more integrated — a learned internal process that combines all of these in a way that no single test-time method captures?

By identifying six distinct reasoning patterns in o1's outputs (SA, MR, DC, SR, CI, EC) and showing that DC and SR appear most frequently, the paper provides empirical evidence that the most effective reasoning involves a combination of decomposition and iterative refinement, where the model internally plans by breaking problems down and then checks its own work. This is significant because it suggests that simple test-time methods that do only one thing — pure BoN (diversity without refinement) or pure Self-Refine (refinement without decomposition) — are inherently limited. The best test-time baseline, Agent Workflow, outperforms the others precisely because it combines decomposition (through domain-specific prompts) with structured execution, approaching a form of internal planning.

Real-world impact on model development. The paper explicitly states its goal is to "guide developers and researchers in understanding the principle of o1 and facilitate the growth of foundation models." This is not modest: the authors are positioning their analysis as a roadmap for the open-source community to replicate or approximate o1's capabilities without access to OpenAI's proprietary technology. The six reasoning patterns they identify could directly inform the design of training data, system prompts, or reinforcement learning rewards for next-generation models. If DC and SR are indeed the key patterns, then training data that demonstrates these patterns — examples showing step-by-step decomposition followed by self-checking — might be disproportionately valuable.

What Prior Approaches Existed and Where Do They Fall Short?

The paper's related work section (Section 2) identifies two main categories of prior approaches, each with specific limitations:

1. Scaling model parameters and training data. This is the dominant paradigm that produced models like GPT-4, Llama 3, Qwen, Yi, and DeepSeek. The approach is straightforward: more parameters, more data, more pretraining compute. The paper acknowledges the successes of this approach but identifies a clear limitation:

"the approach of enhancing model performance by increasing model parameters and collecting more data is facing a bottleneck."

This bottleneck is both economic (diminishing returns per dollar of compute invested) and practical (the largest models are becoming increasingly difficult to deploy at scale). The paper cites Snell et al. (2024) as evidence that test-time compute can be more efficient than parameter scaling in matched-FLOP comparisons, making this bottleneck not just a practical inconvenience but a principled limit on the parameter-scaling paradigm.

2. Existing test-time compute methods. Four distinct approaches had been proposed prior to this paper, each with documented limitations:

  • Best-of-N (BoN): Generate NN candidate outputs and select the best one using a scoring function. The limitation, as the paper demonstrates (Section 4.1), is that "the performance of BoN tends to stabilize" as NN increases — beyond a certain point, adding more samples doesn't help because the backbone model's output distribution has limited diversity. The paper shows in Figure 5 that for GPT-4o on HotpotQA, performance plateaus around N=8N = 8 and even declines slightly at N=16N = 16. This is a fundamental ceiling: no amount of selection can surface a correct answer if the model never generates it.

  • Step-wise BoN: Decompose the problem into steps, generate NN candidates per step, and select the best at each step. This approach addresses the limited-diversity problem by confining search to individual steps rather than full solutions — the probability of generating a correct step is higher than generating a correct full solution. However, the paper identifies a critical failure mode: error accumulation across steps. "An error in any intermediate step can lead to error accumulation, which significantly affects the final output" (Section 4.5). This is compounded by the long-context problem (Section 4.3): as the model generates more intermediate steps, the context grows, and the model's ability to follow the original problem constraints degrades. The paper shows in Table 2 that Step-wise BoN produces an average of 274 tokens of intermediate reasoning for HotpotQA and 450 for Collie, and notes that this long context "can prevent the model from following the original input text." This is a structural limitation: step-wise methods trade breadth (many independent attempts) for depth (building on previous steps), but the depth introduces compounding errors.

  • Self-Refine: Have the model critique and revise its own output iteratively. The paper finds this method's performance "is not significant" (Section 4.1), with minimal improvement over the base GPT-4o and even degraded performance on Collie. The authors hypothesize that "LLMs may generate responses that slightly deviate from the required format during the refinement iterations." This is consistent with findings from Huang et al. (2023), who showed that prompting LLMs to self-correct reasoning is largely ineffective — the model lacks the introspection capability to reliably identify its own errors, and repeated revisions can drift away from the problem constraints.

  • Agent Workflow: Use a structured agent framework with domain-specific system prompts and tool access. This is the strongest baseline in the paper (24.7% overall accuracy, reaching 46% on Collie). However, the paper notes that "there is still a gap between the Agent Workflow and the o1 model, which may be because Agent Workflow explores a less diverse space of responses" (Section 4.1). The Agent Workflow's prompts and workflows are pre-specified by human designers — they reflect human priors about how to decompose problems but may miss solution strategies that a learned system like o1 can discover.

The shared limitation: no internal planning. A theme running through all four test-time methods is that they are external scaffolding — they modify how the model's outputs are generated, evaluated, and combined, but they don't change the model's internal computation per token. The model still generates text autoregressively, one token at a time, with no explicit internal deliberation. BoN generates more tokens in parallel; Step-wise BoN generates more tokens sequentially; Self-Refine generates more tokens iteratively; Agent Workflow generates more tokens with better prompts. But none of them enable the model to plan internally before committing to output tokens — which is precisely what o1 appears to do, given its explicit "thinking" phase before responding.

This distinction is important because it suggests a qualitative gap, not just a quantitative one. If o1's advantage came purely from doing test-time compute better (e.g., more samples, better verifier, better prompts), then scaling up existing methods should eventually close the gap. But the paper's results suggest otherwise: even with unlimited NN, BoN plateaus; even with careful step-by-step prompts, Step-wise BoN suffers from error accumulation; even with sophisticated agent architectures, Agent Workflow trails o1 significantly on math (15.6% vs. 44–62%) and code (22.2% vs. 44.6–12.2%). This pattern implies that o1 has some capability that existing test-time compute methods fundamentally lack.

How Does the Paper Position Itself?

The paper occupies a distinctive position at the intersection of two research threads: test-time compute optimization and reasoning analysis of frontier models. It positions itself not as advancing either thread alone, but as connecting them — using the lens of test-time compute methods to reverse-engineer what makes o1 effective.

As a comparative study, not a method proposal. The paper is explicit that it doesn't propose a new technique. It compares o1 against existing methods "to investigate the reasoning patterns of o1" (Section 1). This is an analytical contribution: the value is in the insights derived from systematic comparison, not in a new algorithm or training recipe. This positions the paper differently from work like Snell et al. (2024), which proposed a framework for compute-optimal test-time scaling, or Wang et al. (2023), which proposed a method for hierarchical hypothesis search. This paper's contribution is empirical characterization rather than technical innovation.

As a bridge between black-box and white-box analysis. The paper can't look inside o1's architecture — OpenAI hasn't disclosed it — but it can look at o1's observable behavior (its reasoning traces, accuracy patterns, token counts) and compare it against the observable behavior of known methods. By controlling for the backbone model (GPT-4o in all test-time baselines) and the benchmarks, the paper isolates the methodological difference between o1 and existing approaches. This is a form of behavioral reverse-engineering: if o1 performs better than BoN on a task, and the gap persists even as BoN's NN increases, then whatever o1 is doing is not equivalent to "generate more candidates and pick the best one." If o1 performs better than Step-wise BoN, then whatever o1 is doing is not equivalent to "sequentially decompose and select per-step."

As a contributor of taxonomies and empirical patterns. The paper's most original contribution is the six-category reasoning pattern taxonomy (SA, MR, DC, SR, CI, EC). This taxonomy is developed bottom-up from inspecting o1's reasoning traces, not borrowed from prior literature. It provides a vocabulary for describing what o1 does that other methods don't: while BoN, Step-wise BoN, and Self-Refine each implement one of these patterns (BoN ≈ parallel search, Self-Refine ≈ SR, Agent Workflow ≈ DC), o1 implements all six, switching between them adaptively based on the task (Figure 1 shows CI and EC dominating in commonsense reasoning, DC and MR dominating in math and coding). This explains why no single test-time method matches o1 across the board — each captures only a subset of o1's reasoning repertoire.

As a practical guide for the open-source community. The paper explicitly frames its contribution as enabling the development of new models: the findings "can guide developers and researchers" (Section 6), the code and datasets are released on GitHub, and the reasoning pattern analysis is intended to "facilitate the growth of foundation models." This is a positioning choice: rather than treating o1 as a competitor to be benchmarked against, the paper treats it as a system to be learned from. The implicit argument is that understanding what o1 does — even without knowing how it's built — can inform the next generation of open-source reasoning models.

Acknowledged limitations of this positioning. The paper is appropriately cautious about what its analysis can and cannot establish. It can observe correlations (e.g., "DC and SR appear most frequently in o1's reasoning traces") but cannot establish causation (e.g., "DC and SR cause o1's superior performance"). It can compare o1 against GPT-4o-based baselines but cannot factor out the effect of o1 potentially being a fundamentally different model architecture or size. The paper doesn't attempt to claim that the six reasoning patterns constitute a complete account of o1's mechanism — they are patterns observable in the output, which may or may not reflect internal computation. The positioning is therefore one of informed hypothesis generation: the paper provides the best evidence currently available about what o1 is doing, while acknowledging that definitive answers await either disclosure from OpenAI or further research.

3. Technical Approach

3.1 Reader Orientation

This paper is not building a new system but performing a controlled empirical comparison between OpenAI's proprietary o1 model and four representative test-time compute strategies (all using GPT-4o as the backbone), combined with a qualitative behavioral analysis of o1's reasoning traces to identify recurring patterns. The core problem it solves is the absence of systematic knowledge about how o1 achieves its reasoning improvements — what inference-time mechanisms it employs, how they compare to existing methods, and what reasoning patterns characterize its outputs across different task domains. The solution is a multi-method benchmarking study augmented with manual reasoning pattern annotation and diagnostic experiments (varying reward models, search budgets NN, and data difficulty filtering) that collectively characterize o1's performance surface and failure modes relative to explicit test-time compute baselines.

3.2 Big-Picture Architecture (Diagram in Words)

The paper's analytical architecture has five interconnected components:

  1. Benchmark Selection and Filtering Module — Selects four datasets spanning three reasoning domains (math, code, commonsense) and applies a model-based difficulty filter to remove samples that current LLMs solve trivially, enabling performance discrimination between strong methods.

  2. Model Evaluation Targets — Three closed-source models (o1-preview, o1-mini, GPT-4o) that provide direct generation baselines without test-time augmentation.

  3. Test-Time Compute Methods (all based on GPT-4o) — Four distinct inference augmentation strategies: Best-of-N (parallel generation + selection), Step-wise BoN (sequential decomposition + per-step selection), Self-Refine (iterative self-critique + revision), and Agent Workflow (domain-specific system prompts + structured task decomposition).

  4. Reward Model Comparison Module — Evaluates how different verifier qualities (Skywork-Reward-Gemma-2-27B, URM-LLaMa-3.1-8B, GPT-4o as judge, Human) affect the performance ceiling of selection-based test-time methods.

  5. Reasoning Pattern Annotation Pipeline — A manual qualitative analysis that reads o1's "thinking" traces, categorizes each reasoning step into one of six pattern types (SA, MR, DC, SR, CI, EC), and computes pattern frequency distributions per benchmark.

Information flows as follows: each benchmark → optional difficulty filtering (Section 3.1) → parallel evaluation across o1 variants, GPT-4o direct, and four test-time methods with GPT-4o backbone → accuracy computation per benchmark → reward model comparison for BoN methods (Section 4.5) → qualitative annotation of o1 reasoning traces for pattern analysis (Section 4.2) → synthesis of findings.

3.3 Roadmap for the Deep Dive

  • First, the benchmark selection and difficulty filtering mechanism (Section 3.1), since this determines what problems the models are evaluated on and is the foundation for all reported results.
  • Second, the four test-time compute methods (Best-of-N, Step-wise BoN, Self-Refine, Agent Workflow) — their operational mechanics, generation budgets, and selection procedures — since these are the baselines against which o1 is compared.
  • Third, the reward model evaluation setup (Section 4.5), which is a diagnostic experiment revealing the sensitivity of BoN methods to verifier quality and establishing an upper bound through human selection.
  • Fourth, the reasoning pattern taxonomy and annotation methodology (Section 4.2), including how patterns were defined, annotated, and counted across benchmarks — since this is the paper's most original analytical contribution.
  • Fifth, the evaluation metrics and grading procedures per dataset, since accuracy is computed differently for different answer formats (exact match, code execution, constraint satisfaction).

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical analysis and behavioral characterization paper whose core idea is that by systematically comparing o1 against explicit test-time compute methods on controlled-difficulty benchmarks and qualitatively analyzing o1's reasoning traces, we can infer what reasoning mechanisms o1 employs and why existing test-time methods fall short.

Benchmark Selection and Difficulty Filtering

The paper evaluates on four benchmarks selected to cover three reasoning domains: commonsense reasoning (HotpotQA for multi-hop document-based QA, Collie for constrained text generation), code (USACO bronze-level algorithmic problems), and math (AIME competition problems). The selection is explicitly motivated by o1's reported strengths: "According to the OpenAI o1 report, the model demonstrates exceptional improvements in areas such as mathematics and coding" (Section 1). This is not a random sample of NLP benchmarks — it targets exactly the domains where o1's improvements are claimed to be largest, making it a maximally informative comparison.

Why these four benchmarks? Each serves a distinct analytical purpose. HotpotQA tests multi-hop reasoning with external context — the model must synthesize information from multiple documents, which requires identifying relevant context (a reasoning pattern the paper later labels "Context Identification"). Collie tests constraint satisfaction in generation — the model must produce text adhering to lexical and structural constraints, which requires maintaining constraint awareness throughout generation (labeled "Emphasizing Constraints"). USACO tests algorithmic problem-solving with code generation and execution. AIME tests pure mathematical reasoning with objective numeric answers. Together, they cover the spectrum from open-ended constrained generation to closed-form symbolic reasoning.

The difficulty filtering mechanism. The paper acknowledges a practical problem: current LLMs (including GPT-4o) already perform well on many existing benchmark samples, creating ceiling effects that mask performance differences between methods. To address this, the paper implements a model-based difficulty filter inspired by LIME (Zhu et al., 2024). The procedure works as follows:

  1. For HotpotQA and Collie (the commonsense reasoning benchmarks deemed potentially too easy), four different LLMs are used as evaluators: Llama3-72B (Dubey et al., 2024), Qwen-72B (Bai et al., 2023), Claude (via Anthropic's API), and an unspecified fourth model (the text mentions "using four different LLMs (i.e., Llama3-72B, Qwen-72B, Claude" but lists only three by name; the fourth may be Yi as mentioned in the Related Work).

  2. Each of the four models answers every sample in the original benchmark.

  3. A sample is retained if and only if two or fewer of the four models can answer it correctly. That is, the filter keeps only questions that the majority of these strong models get wrong — the "hard" subset.

  4. For USACO and AIME, the paper states that samples are filtered similarly: "We employ LLMs like Llama3-72B, Qwen-72B, and Claude to solve these problems, selecting only those that prove challenging across multiple models" (Section 3.1). However, the specific threshold (≥2 models correct → discard) may vary — the text is less precise for these benchmarks.

What the filter produces. Table 3 reports the filtered dataset sizes: HotpotQA retains 274 questions, Collie retains 226, USACO retains 139, and AIME retains 90. The paper demonstrates the filter's effect in Figure 8, showing that on the raw HotpotQA benchmark, different LLMs (GPT-4o, Qwen2.5, etc.) have similarly high scores with little separation, while on the filtered benchmark the scores are "relatively lower and show greater distinction." This validates the filter's purpose: it creates a benchmark where the performance gap between strong and weak methods is measurable rather than compressed into a few percentage points at the ceiling.

Why this filtering approach? The alternative — using the entire benchmark — would produce results where most methods achieve similar high accuracy, making it impossible to determine which reasoning patterns or test-time strategies are actually effective. The filter converts the evaluation from "can any decent model solve this?" to "which methods can handle problems that genuinely challenge current LLMs?" This is methodologically sound for a comparative study, though it does mean results on filtered benchmarks are not directly comparable to results reported in other papers on the full benchmarks.

A subtle design choice: the filter uses four different model families (Llama, Qwen, Claude, and possibly Yi) rather than, say, four variants of the same model. This increases the diversity of the "judge panel" and reduces the risk that the filter simply selects for problems that a particular model architecture struggles with. If a problem is hard for Llama, Qwen, Claude, and the fourth model simultaneously, it's more likely to be genuinely difficult rather than an artifact of one model's specific weakness.

Important caveat on USACO and AIME filtering. The paper is less explicit about the filtering procedure for these benchmarks. AIME is described as containing "90 problems from AIME 22, AIME 23, and AIME 24, and have been extracted directly from the AOPS wiki page" — this may be the full set of available problems from those years, not a filtered subset. USACO bronze problems are already competition-level, so the inherent difficulty may be sufficient without additional filtering. The paper's Table 3 shows 139 USACO problems and 90 AIME problems, but doesn't specify whether these counts represent filtered or full sets.

Test-Time Compute Methods: Best-of-N (BoN)

Best-of-N is the simplest test-time compute method evaluated and serves as the parallel sampling baseline. The procedure is:

  1. For a given input prompt, the backbone model (GPT-4o) generates NN independent complete outputs. The paper evaluates N=1,4,8,16N = 1, 4, 8, 16 in most experiments, with N=4N = 4 being the primary setting for the main comparison in Table 1.

  2. Each output is scored by a reward model that estimates output quality. The paper's default reward model is GPT-4o itself acting as a judge (Section 4.5, Figure 4 notation "GPT4o" as reward model).

  3. The output with the highest reward model score is selected as the final answer.

What "most suitable response" means. The paper doesn't specify the exact scoring rubric used by GPT-4o as the reward model for BoN. In the context of HotpotQA and AIME, where answers are compared against ground truth for accuracy scoring, the reward model presumably evaluates whether the generated response contains a plausible correct answer. For Collie, it would need to check constraint satisfaction. For USACO, it would need to evaluate code correctness. The paper's treatment of the reward model selection mechanism is relatively high-level, leaving implementation details unspecified.

Generation budget and cost model. The cost of BoN is exactly NN — NN independent forward passes through GPT-4o, each producing one complete output. For the main results (Table 1), N=4N = 4 is the primary setting, costing 4 generations per question. The paper also sweeps NN to study scaling behavior (Section 4.6), going up to N=16N = 16 for GPT-4o-based BoN and also evaluating Qwen2.5-72B and Llama3-70B as backbone models at the same NN values.

Why BoN is a necessary baseline. BoN represents the "naive" test-time compute strategy: more samples, better odds of hitting a correct answer. Its performance ceiling is determined entirely by two factors: (a) the backbone model's probability of generating a correct answer on any single attempt (its pass@1), and (b) the reward model's ability to recognize the correct answer when it appears. If o1 outperforms BoN even at high NN, then whatever o1 is doing is not equivalent to "generate more candidates from the same distribution" — it must be either improving the candidate generation process itself (raising pass@1) or using a fundamentally different selection mechanism.

Test-Time Compute Methods: Step-wise BoN

Step-wise BoN extends the BoN idea from whole-solution selection to per-step selection within a sequential decomposition. The procedure is:

  1. The model first analyzes the problem and decomposes it into a sequence of sub-problems or reasoning steps.

  2. For the first step, the model generates NN candidate responses (e.g., NN different proposed first steps).

  3. A reward model scores all NN step-candidates and selects the best one. This becomes the "accepted" first step.

  4. The accepted first step is provided as context, and the model generates NN candidate responses for the second step, conditioned on the accepted first step.

  5. The reward model selects the best second step, which becomes context for the third step, and so on.

  6. This process continues iteratively until a complete solution is produced.

Generation budget. For a problem decomposed into KK steps, Step-wise BoN requires N×KN \times K total generations — NN candidates per step, KK steps. The paper reports results with N=1N = 1 (essentially greedy step-by-step decomposition) and N=4N = 4 (per-step selection among 4 candidates). The same GPT-4o reward model is used for per-step selection as for BoN.

What the paper means by "analysis and decomposition." The initial decomposition step is crucial but underspecified. The paper states that Step-wise BoN "enables LLMs to analyze a problem and break it down into several sub-problems" (Section 3.2), but doesn't detail the prompting strategy that produces this decomposition. This is important because the quality of the decomposition — whether the sub-problems are logically independent, whether they cover all necessary reasoning, whether their order is correct — determines the upper bound on what per-step selection can achieve. A bad decomposition creates a structural bottleneck that no amount of per-step search can circumvent.

Why Step-wise BoN is theoretically motivated. The motivation stems from a simple probabilistic observation: the probability of generating a correct complete solution in one shot is the product of conditional probabilities across all reasoning steps. If the model's pass@1 for a full solution is pp, and the solution has KK roughly independent reasoning steps each with correctness probability qq, then p≈qKp \approx q^K. Even with q=0.8q = 0.8 (each step is 80% likely to be right) and K=5K = 5 steps, p≈0.33p \approx 0.33 — the full solution is unlikely despite each step being individually likely. Step-wise BoN addresses this by independently searching for each step: generate NN candidates per step, increasing the chance of getting each step right, and then assemble the best per-step selections into a complete solution. For N=4N = 4 and q=0.8q = 0.8, the probability of selecting a correct step (assuming the reward model can identify it) is approximately 1−(1−0.8)4=0.99841 - (1-0.8)^4 = 0.9984, which, across 5 steps, yields (0.9984)5≈0.992(0.9984)^5 \approx 0.992 — a dramatic improvement over 33%.

The critical failure mode: long-context instruction following. The paper identifies that Step-wise BoN generates very long intermediate contexts, and that these long contexts degrade the model's ability to follow the original problem constraints. Table 2 reports the average reasoning token lengths for Step-wise BoN (N=4N = 4): 274 tokens for HotpotQA, 450 tokens for Collie, 440 tokens for USACO, and 263 tokens for AIME. The paper argues that "excessively long context information can prevent the model from following the original input text to generate the correct response" (Section 4.3). This is a specific instantiation of the well-documented "lost in the middle" phenomenon: as context length grows, models attend less effectively to earlier parts of the context, which in Step-wise BoN includes the original problem specification. When later steps are conditioned on many preceding steps plus the original prompt, the model may effectively "forget" what the original question was asking.

Error accumulation as a compounding problem. Even without the long-context problem, Step-wise BoN faces error accumulation: if the reward model accepts a subtly wrong step at step 3, all subsequent steps (4, 5, ...) are conditioned on that error. The reward model's per-step accuracy needs to be very high, and its failures are multiplicative in their impact. The paper explicitly notes: "an error in any intermediate step can lead to error accumulation, which significantly affects the final output of the model" (Section 4.5). This is not solved by increasing NN — with a fixed reward model, increasing NN only helps if the reward model can reliably distinguish correct from incorrect steps, and if the correct step exists among the NN candidates for that step.

Test-Time Compute Methods: Self-Refine

Self-Refine implements an iterative self-improvement loop where the model generates an initial output, critiques it, and revises it based on the critique. The paper uses the implementation from Madaan et al. (2024) directly: "We directly use the code from the GitHub of the Self-Refine" (Section 3.2).

Procedure. The standard Self-Refine loop (Madaan et al., 2024) consists of:

  1. Initial generation: The model produces a first-draft output for the given input.

  2. Feedback generation: The model is prompted to provide feedback on its own output — identifying weaknesses, errors, or areas for improvement.

  3. Refinement: The model generates a revised output, conditioned on the original input, its initial output, and its self-generated feedback.

  4. Steps 2–3 can be repeated for multiple refinement iterations. The paper uses 3 refinement iterations (as indicated by the "33" in the Self-Refine row of Table 1).

  5. The final refined output after the last iteration is used as the answer.

Why Self-Refine is included despite known limitations. The paper acknowledges in Section 2 that "a number of related works" have explored self-critique and refinement approaches, but also notes the contradictory literature showing "LLMs cannot self-correct reasoning yet" (Huang et al., 2023). Including Self-Refine serves as a boundary condition test: if Self-Refine provides any benefit on these benchmarks, it suggests that iterative refinement has some utility; if it fails (as it largely does), it confirms that naive self-critique without learned refinement capabilities is insufficient for complex reasoning.

The reported failure cases. The paper finds that Self-Refine's performance is "not significant" (Section 4.1), offering only slight improvement over GPT-4o on some tasks and actually degrading on Collie (the constrained generation task). The hypothesized mechanism for Collie degradation is that "LLMs may generate responses that slightly deviate from the required format during the refinement iterations." Specifically, each refinement iteration risks drifting further from the original format constraints — for example, a refinement that adds a clarifying phrase might inadvertently introduce a forbidden word (like "be" or "of" in Collie's constraints), or change the sentence count.

Test-Time Compute Methods: Agent Workflow

The Agent Workflow represents the most sophisticated test-time compute baseline, and it's the only one that approaches o1's performance on some benchmarks. Unlike the other methods, Agent Workflow is not a single algorithm but a meta-framework that uses domain-specific system prompts, structured task decomposition, and potentially tool access, all implemented through existing agent frameworks.

Implementation per domain. The paper uses different agent implementations for different task types:

  • For commonsense reasoning (HotpotQA, Collie): The paper leverages "the existing state-of-the-art agent framework (Zhou et al., 2023; 2024)" — referring to the Agents framework and its symbolic learning extension. These frameworks provide structured agent architectures with planning, tool use, and memory components.

  • For code (USACO) and math (AIME): The paper uses "the top-picked agents from GPTs, specifically code copilot and math solver, respectively" (Section 3.2). GPTs are OpenAI's customizable chatbot configurations that include bespoke system prompts and optional tool access. The "code copilot" GPT is presumably configured for algorithmic problem-solving with code generation and execution capabilities; the "math solver" GPT is configured for mathematical reasoning.

Key design characteristic: domain-specific system prompts. The paper argues that what makes Agent Workflow effective is its use of "a series of domain-specific system prompts" that "reduce unnecessary reasoning steps" and "carefully align with the reasoning problems" (Section 4.1). This is fundamentally different from generic prompting or generic test-time compute — the agent's behavior is shaped by human-crafted prompts that encode domain knowledge about how to approach problems in that domain. For example, a coding agent might have a system prompt that instructs it to first identify the algorithmic paradigm (greedy, DP, graph traversal), then design data structures, then implement, then test — essentially encoding a meta-reasoning strategy.

What "Agent Workflow" computes that BoN doesn't. The critical distinction is that Agent Workflow changes the process of generation, not just the selection among generations. BoN generates N independent complete solutions and picks the best; Agent Workflow generates a structured sequence of sub-actions, where each sub-action is informed by the results of previous sub-actions. The workflow creates task-specific decomposition paths that are pre-specified by the agent's system prompt, reducing the search space from "all possible solution approaches" to "approaches consistent with this problem-solving methodology."

Why it still trails o1. The paper hypothesizes that the remaining gap "may be because Agent Workflow explores a less diverse space of responses" (Section 4.1). The agent's system prompt constrains its approach to a particular methodology; if that methodology isn't the optimal one for a given problem — or if the problem requires a creative insight that doesn't fit the pre-specified workflow — the agent will miss solutions that o1's learned internal reasoning might discover. This is the classic flexibility-efficiency tradeoff: the workflow's structure makes it efficient (fewer wasted exploration steps) but less flexible (cannot discover novel solution strategies outside its prescribed paths).

Reward Model Comparison Setup (Diagnostic Experiment)

The paper conducts a targeted diagnostic experiment (Section 4.5, Figure 4) to measure how reward model quality affects the performance of BoN-based methods. This experiment is crucial for understanding whether the limitations of search-based test-time compute are due to the generation process or the selection process.

Reward models compared. Four selection mechanisms are evaluated for the same set of GPT-4o-generated BoN outputs (with N=4N = 4):

  1. Skywork-Reward-Gemma-2-27B (abbreviated SRG in Figure 4) — an open-source reward model based on Gemma-2-27B, trained by Skywork (Liu & Zeng, 2024) and available on HuggingFace. This model appears on the RewardBench leaderboard (Lambert et al., 2024), which benchmarks reward model quality.

  2. URM-LLaMa-3.1-8B — the Uncertainty-aware Reward Model by Lou et al. (2024), based on Llama-3.1-8B. This model is designed to express uncertainty about its judgments when appropriate, making it potentially more calibrated than standard reward models.

  3. GPT-4o as judge — the default reward model used in the main experiments. The paper uses GPT-4o to score candidate outputs and select the best one. The exact prompting strategy for GPT-4o-as-judge is not specified.

  4. Human as reward model — human annotators manually select the best response from the N=4N = 4 candidates. This represents the Oracle upper bound for any reward-model-based selection, since a human (with sufficient domain expertise) should be able to identify the correct answer if it exists among the candidates, assuming the criteria for correctness are objective and checkable.

What the comparison reveals. On HotpotQA, the reward model quality gradient is dramatic: specialized reward models (SRG, URM-LLaMa) achieve under 15% accuracy, GPT-4o achieves the accuracy shown in Table 1 (which for BoN at N=4N=4 is 13.50%), while human selection reaches 33% — more than doubling the best automated result. This demonstrates that BoN's performance is not limited by generation quality (the correct answer exists among the 4 candidates often enough) but by the automated reward model's inability to reliably identify it. On Collie, the gap is smaller — automated reward models achieve results closer to human performance — suggesting that Collie's constraint-satisfaction nature makes answer quality more objectively evaluable by automated methods.

Why this experiment matters for understanding o1. If o1 outperforms BoN even when BoN uses human selection (which it does: o1-mini achieves 62% on AIME vs. BoN's implied ceiling from human selection on HotpotQA at 33%), then o1's advantage cannot be explained solely by having a better internal verifier. It must also be generating different (better) candidates than the GPT-4o backbone — changing the proposal distribution, not just the selection mechanism.

Search Space Scaling Experiment (Section 4.6)

A second diagnostic experiment (Figure 5) measures how BoN performance scales with NN across three backbone models: GPT-4o, Qwen2.5-72B, and Llama3-70B. The NN values tested are 1, 4, 8, and 16, all on the filtered HotpotQA benchmark.

What "search space" means here. The paper uses "search space" to refer to the number of candidate outputs NN that BoN generates and evaluates. A larger NN means a larger search space — more candidates from which to select. The term is slightly nonstandard (usually "search space" refers to the space of possible solutions, not the number explored), but the meaning is clear: NN controls how broadly the model explores its output distribution.

The plateau effect. Figure 5 shows that all three backbone models' performance plateaus or declines at N>8N > 8. For GPT-4o, performance at N=1N = 1 is approximately 13%, rising to roughly 17% at N=4N = 4, roughly 16% at N=8N = 8, and roughly 16% at N=16N = 16. Qwen2.5-72B starts higher (around 18% at N=1N = 1) and reaches roughly 40% at N=4N = 4, 40% at N=8N = 8, and 40% at N=16N = 16 (estimated from Figure 5). The paper interprets this as: "the performance of the search methods is jointly related to the reward model and searching space" (Section 4.6). Specifically, beyond a certain NN, the model has exhausted its distribution of meaningfully different correct answers — additional samples are variants of the same (mostly incorrect) approaches, and the reward model cannot distinguish among them.

Implication for o1 comparison. This plateau demonstrates that simply scaling up parallel sampling (NN) cannot match o1's performance, even with a perfect reward model. There is a fundamental generation bottleneck: the backbone model's pass@1 rate and the diversity of its output distribution impose an upper limit on what selection-based methods can achieve, regardless of selection quality. o1 must therefore be doing something that either increases pass@1 (generating better initial candidates) or fundamentally alters the generation process beyond simple parallel sampling.

Reasoning Pattern Taxonomy and Annotation Methodology

This is the paper's most original analytical contribution and represents a qualitative, exploratory analysis rather than a quantitative method.

How the patterns were defined. The paper does not specify an a priori taxonomy development process. The six patterns (SA, MR, DC, SR, CI, EC) are presented as findings — patterns that emerged from inspecting o1's reasoning traces. The implication is that the authors read through o1's "thinking" outputs, observed recurring structural elements, and grouped them into these six categories. The taxonomy is therefore descriptive (derived from observation) rather than prescriptive (defined in advance and applied).

The six pattern types, defined operationally:

  1. Systematic Analysis (SA): The model begins by examining the overall structure of the problem — inputs, outputs, constraints, applicable algorithmic paradigms or data structures — before diving into specific solution steps. The paper's description: "Starting from the overall structure of the problem, o1 first analyzes the inputs and outputs, as well as the constraints, and then decides on the choice of algorithm and the use of data structures" (Section 4.2). This is a top-down structural analysis pattern.

  2. Method Reuse (MR): The model recognizes that the problem is structurally equivalent to a known problem class (shortest path, knapsack, etc.) and directly applies the standard solution method. "For some problems that can be transformed into classic problems... o1 can quickly reuse existing methods to solve them" (Section 4.2). This is a pattern-matching to known solutions behavior.

  3. Divide and Conquer (DC): The model breaks a complex problem into simpler sub-problems, solves each sub-problem independently or sequentially, and then combines the solutions. "It breaks down a complex problem into subproblems and constructs the overall solution by solving the subproblems" (Section 4.2). This is the classic problem decomposition pattern.

  4. Self-Refinement (SR): The model evaluates its own intermediate reasoning, identifies potential issues, and corrects errors. "o1 assesses its reasoning process during inference to determine if there are any issues and correct any errors" (Section 4.2). This is internal self-critique and correction, distinct from the Self-Refine test-time method which does this externally through separate API calls.

  5. Context Identification (CI): For tasks with external context (documents, passages), the model first summarizes relevant information from the context, organizes it in relation to the query, and then synthesizes an answer. "o1 first summarizes different aspects of the context related to the query, and then gives the response for the corresponding query" (Section 4.2). This is a context-processing and relevance-filtering pattern.

  6. Emphasizing Constraints (EC): For tasks with explicit constraints on the output format or content, the model repeatedly references and reinforces those constraints during its reasoning process to ensure compliance. "o1 usually emphasizes the corresponding constraints during the reasoning process" (Section 4.2). This is a constraint-maintenance pattern.

Annotation procedure. The paper states: "We randomly selected 20 to 30 samples of each benchmark to count the number of different reasoning patterns" (Section 4.2). This is a manual annotation on a relatively small sample (80–120 samples total across four benchmarks, representing roughly 11–15% of the total filtered dataset of 729 questions). The paper does not report inter-annotator agreement, annotation guidelines, or whether multiple annotators were involved. The counts in Figures 1 and 2 are absolute frequencies — how many times each pattern was observed across the annotated samples.

Why the counts in Figure 2 sum to more than the number of samples. A single o1 reasoning trace can contain multiple reasoning patterns — for example, a solution might start with Systematic Analysis (SA), then apply Divide and Conquer (DC) on the decomposed sub-problems, and include a Self-Refinement (SR) check at the end. The frequency counts therefore represent pattern occurrences, not sample classifications. A single sample contributes to multiple pattern counts if o1's reasoning trace exhibits multiple patterns.

Pattern distribution across tasks (Figure 1). The paper reports that different tasks elicit different pattern distributions:

  • HotpotQA: CI appears most frequently (~70 occurrences in the annotated sample), followed by DC (~50), with SA, MR, SR, and EC appearing less frequently (~10–20 each). This makes sense: HotpotQA requires processing multiple documents (hence CI) and synthesizing information across them (hence DC).

  • Collie: EC dominates (~65 occurrences), with other patterns appearing much less frequently (~5–10 each). This also makes sense: Collie's primary challenge is satisfying output constraints, so EC (repeatedly checking and reinforcing constraints) is the natural strategy.

  • USACO: DC and SA are most frequent (~50–60 occurrences each), with MR also prominent (~35). Coding problems benefit from systematic structural analysis (SA), algorithmic decomposition (DC), and recognizing standard problem patterns (MR).

  • AIME: DC dominates (~35 occurrences), with MR also notable (~25), and SR, SA appearing less frequently (~15 each). Math problems require decomposition into sub-problems and application of known mathematical techniques (MR).

The key claim about DC and SR. Figure 2 aggregates across all benchmarks and shows that DC (Divide and Conquer) and SR (Self-Refinement) are the two most frequently observed patterns overall, each with roughly 60–80 occurrences in the total annotated sample (compared to SA at ~65, MR at ~45, CI at ~75 but concentrated in HotpotQA, EC at ~70 but concentrated in Collie). The paper interprets this as evidence that DC and SR "might be the key to o1's success" and that "the combination of SR and DC plays a crucial role in enhancing the performance of o1" (Section 4.2).

Important caveat on this claim. The frequency of a pattern in o1's output traces does not necessarily indicate its causal contribution to performance. DC might appear frequently because complex problems naturally elicit decomposition, not because decomposition causes the solution to be correct. SR might appear in both correct and incorrect traces. The paper does not separate pattern frequencies for correct vs. incorrect samples, which would strengthen the causal claim. As presented, the pattern frequency analysis establishes what o1 does (descriptive) but not what makes it work (causal).

Evaluation Metrics and Answer Grading Procedures

The paper uses accuracy as the primary metric across all benchmarks, but the grading procedure differs by task type because answer formats differ.

HotpotQA and AIME (rule-based answer extraction). The paper states: "we design a rule to determine whether the model-generated response contains the correct answer and use the accuracy of the model's responses as the final score" (Section 3.3). This is a pattern-matching approach: the model's full output is searched for a substring matching the ground-truth answer. For HotpotQA, answers are typically short spans extracted from the context documents; for AIME, answers are integers (the AIME answer format is always an integer between 0 and 999). The rule-based extraction avoids penalizing models for verbose outputs that contain the correct answer embedded within explanatory text.

Collie (constraint satisfaction check). The paper states: "we directly determine whether the model-generated response is correct" (Section 3.3). For Collie, the "correctness" criteria are the constraints specified in the prompt (e.g., exactly 3 sentences, no word "be," no word "of," no word "is"). The evaluation presumably checks each constraint individually and marks the response as correct only if all constraints are satisfied. The paper mentions that Self-Refine degrades on Collie because refinement iterations cause outputs that "slightly deviate from the required format" — this suggests the evaluation is strict about constraint violations.

USACO (execution-based evaluation). The paper states: "we manually run the LLMs-generated code on the test examples, and regard the code passing the test cases as right" (Section 3.3). This is the standard execution-based evaluation for code generation: the model's code is executed against hidden test cases, and the output is considered correct if it passes all test cases. "Manually run" suggests a human performed the execution rather than an automated sandbox, though the difference is procedural rather than methodological.

Why different grading procedures are necessary. The diversity of answer formats across benchmarks precludes a single evaluation metric. HotpotQA answers are text spans (evaluated by substring match), AIME answers are integers (evaluated by exact numeric match), Collie outputs are paragraphs (evaluated by constraint checking), and USACO outputs are programs (evaluated by execution). Using a uniform evaluation method (e.g., always exact match) would unfairly penalize verbose models on HotpotQA or fail to capture the functional correctness of code on USACO. The paper's approach of tailoring the grading procedure to each benchmark is standard practice.

Reasoning Token Extraction for o1 (Section 4.4)

To analyze the relationship between reasoning length and accuracy, the paper extracts o1's internal reasoning tokens from its API responses.

How reasoning tokens are extracted. The paper states: "we developed a rule to extract o1's reasoning tokens and computed their count across different tasks" (Section 4.4). At the time of the paper, OpenAI's o1 API (if available) or the ChatGPT interface with o1 displayed the model's "thinking" process as a collapsible section separate from the final answer. The extraction rule likely parses the API response or interface output to separate the thinking/tokens from the answer tokens, possibly using delimiters (e.g., > Thinking and > Input markers visible in the paper's case study figures). The paper does not specify the exact extraction method, but the case study figures (Figures 7, 9, 10, 11) show structured thinking traces with labeled sections.

What is measured. For each task, the paper computes: (a) the average number of reasoning tokens across all samples ("ALL"), (b) the average for correctly answered samples ("True"), (c) the average for incorrectly answered samples ("False"), and (d) the average input prompt length ("Input"). These are plotted in Figure 3.

Key observation from Figure 3. The number of reasoning tokens varies dramatically by task: HotpotQA and Collie (commonsense reasoning) have shorter reasoning traces, while USACO (code) and AIME (math) have substantially longer traces. Within each task, the reasoning token counts for correct and incorrect samples are similar — there is "no clear correlation" between reasoning length and correctness within a task. The input prompt length also does not predict reasoning length. The paper concludes: "for more difficult tasks like Code and Math, the model often requires a longer reasoning process to obtain the correct answer" (Section 4.4). The reasoning length is task-determined, not performance-determined; harder tasks elicit longer thinking regardless of whether the final answer is correct.

Why this matters. If reasoning length were strongly correlated with correctness — say, correct answers consistently used longer reasoning — that would suggest that simply "thinking longer" (more test-time compute) improves accuracy, consistent with a search or iterative refinement mechanism. The finding that correct and incorrect traces have similar lengths within a task suggests that o1's reasoning length is determined by task complexity (how many steps the problem intrinsically requires) rather than by correctness (longer thinking doesn't guarantee correctness). This is more consistent with a structured reasoning process that follows a plan determined by problem analysis, rather than an open-ended search that keeps going until it finds a good answer.

Overall Design Philosophy and Justifications

Why compare against GPT-4o rather than a more capable backbone? The paper uses GPT-4o as the backbone for all test-time compute methods because it is the most capable publicly available model with an API at the time of writing, and it represents the state of the art for "standard" (non-reasoning-enhanced) LLMs. Comparing against GPT-4o with test-time compute isolates the effect of the test-time strategies: any gap between BoN/Step-wise BoN/Self-Refine/Agent Workflow and o1 reflects what the test-time methods cannot achieve even when applied to a very strong base model. If the paper had used a weaker backbone, the gap might reflect backbone capability rather than test-time method limitations.

Why use o1-preview AND o1-mini? The paper evaluates both o1 variants. o1-preview is the earlier, more capable but slower version; o1-mini is a smaller, faster variant optimized for STEM reasoning. Comparing them reveals that o1-mini sometimes outperforms o1-preview (e.g., 62% vs. 44% on AIME, 53.5% vs. 34.1% on Collie), suggesting that "the reasoning process of o1 does not always lead to better improvements" and that the relationship between model scale, reasoning depth, and accuracy is not monotonic (Section 4.1). This is an important empirical observation: more reasoning isn't always better, and the o1-mini variant may have been optimized differently than o1-preview.

Why 3 refinement iterations for Self-Refine? The paper uses 3 iterations without explaining this choice. The Madaan et al. (2024) paper experimented with varying numbers of iterations; 3 is a common default in that literature, balancing the potential for improvement against the risk of drift and the cost of additional API calls.

Why N=4N = 4 as the primary BoN setting? The paper uses N=4N = 4 for the main comparison (Table 1) and sweeps up to N=16N = 16 in the scaling analysis (Figure 5). N=4N = 4 represents a modest test-time budget — 4x the inference cost of a single generation — and is likely chosen because it offers a meaningful improvement over N=1N = 1 without being prohibitively expensive. The scaling analysis shows that performance plateaus around N=8N = 8, so N=4N = 4 captures most of the BoN benefit.

Why 20–30 samples for reasoning pattern annotation? The paper annotates "20 to 30 samples of each benchmark" for reasoning patterns (Section 4.2). This is a practical choice reflecting the manual effort required to read and categorize o1's often-lengthy reasoning traces. It provides enough samples to estimate relative pattern frequencies (with error bars that are not reported) without requiring annotation of all 729 filtered problems. The tradeoff is that pattern frequency estimates have significant sampling uncertainty, particularly for rarer patterns.

What the paper does NOT do (important for understanding scope). The paper does not: (a) train any models or propose any new architectures; (b) modify o1's behavior — it observes o1 as a fixed black box; (c) perform controlled ablation studies on o1 itself (it cannot, since o1 is a proprietary model with no exposed knobs); (d) measure latency or wall-clock time — all comparisons are in terms of accuracy per method, not speed or cost; (e) combine test-time methods (e.g., Agent Workflow + BoN) to see if hybrid approaches close the gap with o1. The paper is purely observational and comparative, establishing empirical patterns that can inform future model development but not testing causal hypotheses about why o1 works.

4. Key Insights and Innovations

Innovation 1: A Behavioral Taxonomy for Reverse-Engineering Black-Box Reasoning

The paper's most original conceptual contribution is the six-category reasoning pattern taxonomy (Systematic Analysis, Method Reuse, Divide and Conquer, Self-Refinement, Context Identification, Emphasizing Constraints) — not as a theoretical framework, but as an empirically grounded diagnostic instrument for characterizing what a reasoning model actually does in its thinking traces. This matters because the field had no shared vocabulary for describing the internal deliberation behaviors that differentiate models like o1 from standard LLMs.

What the field did before. Prior characterizations of model reasoning fell into two camps. The first camp studied external reasoning methods — Chain-of-Thought prompting, tree search, self-consistency — focusing on what the system designer adds, not what the model does internally. The second camp made coarse distinctions (e.g., "the model reasons step-by-step") without differentiating types of reasoning steps or how they combine. There was no taxonomy for saying "this model uses decomposition-then-verification" versus "this model pattern-matches to known solution templates" as distinct, observable behavioral strategies. The paper fills this gap by developing categories bottom-up from reading o1's traces, producing a vocabulary that is simultaneously descriptive (capturing observed diversity) and diagnostic (enabling comparison across tasks and models).

Why this is a conceptual advance rather than just labeling. The taxonomy doesn't merely name behaviors — it reveals non-obvious structural properties of o1's reasoning. Figure 1 shows that pattern distributions are strongly task-dependent: CI and EC dominate commonsense reasoning, while DC and MR dominate math and coding. This task-pattern alignment is not something one would predict a priori — it suggests o1 has learned (or been trained to deploy) different reasoning strategies for different problem types, adapting its internal deliberation to the demands of the task. This is a fundamentally different capability from generic "chain-of-thought," which applies the same "think step by step" heuristic regardless of problem type. The taxonomy makes this adaptivity visible and measurable.

Distinguishing incremental from fundamental. The taxonomy itself is a fundamental conceptual tool, not an incremental refinement. Prior work had no way to ask "what reasoning patterns does this model use, and how do they vary across tasks?" — the question wasn't even well-posed without the categories. The annotation methodology (manual inspection of 20–30 traces per benchmark) is admittedly preliminary and lacks inter-annotator agreement statistics, limiting replication. But the core intellectual move — that reasoning patterns are categorizable, task-dependent, and observable in output traces — reframes the problem from "how good is this model?" to "what reasoning strategies does this model employ, and which ones matter?" This is a qualitative shift in analytical approach, not just a new metric.

Evidence anchor. The pattern frequency distributions in Figures 1 and 2, showing DC and SR as most prevalent overall but CI/EC dominating in commonsense tasks and DC/MR in math/coding tasks, substantiate the claim that the taxonomy captures meaningful behavioral variation rather than arbitrary categories.

Innovation 2: Difficulty Filtering as a Principled Diagnostic for Comparative Studies

The paper introduces a model-based difficulty filtering procedure that transforms saturated benchmarks (where all strong models score similarly high) into discriminating diagnostic instruments. This is not presented as a methodological contribution, but it functions as one: without it, the comparative analysis would be largely uninformative.

What the field did before. Standard practice in LLM evaluation treats benchmarks as fixed — you report accuracy on the full test set and hope there's enough headroom to differentiate models. When benchmarks saturate (as many have with GPT-4-class models), researchers either move to harder benchmarks (which may not exist for the target capability) or report near-identical scores that mask genuine performance differences. The LIME framework (Zhu et al., 2024) proposed filtering with multiple model judges, which this paper adapts, but the practice was not widespread in reasoning evaluation.

Why this is intellectually distinctive for this paper's goals. The filtering procedure does more than create headroom — it changes the nature of the comparison. By retaining only samples that a majority of strong models (Llama3-72B, Qwen-72B, Claude) get wrong, the filter selects for problems where correct answering requires capabilities beyond those of current standard-scale LLMs. This means the remaining benchmark tests not "can the model answer this?" but "does the model possess capabilities that distinguish it from strong but non-reasoning-enhanced baselines?" The filter transforms the evaluation from a generic accuracy contest into a targeted probe for reasoning-specific improvements. This is a conceptually elegant move: rather than searching for a "hard enough" benchmark (which may not exist or may not target the right capabilities), you construct one from existing data by using other models as difficulty oracles. The resulting benchmark is definitionally hard for current models and therefore maximally informative about what makes o1 different.

Figure 8 as evidence of necessity. The comparison between raw and filtered HotpotQA/Collie scores (Figure 8) makes the case: on raw benchmarks, model scores cluster together with small separations; on filtered benchmarks, separations are larger and more informative. Without this filtering, many of the paper's comparative claims — e.g., that Agent Workflow substantially outperforms BoN on Collie (46.07% vs. 38.50% in Table 1) — would be compressed into a few percentage points or lost entirely.

Distinguishing incremental from fundamental. The filtering approach is an incremental refinement of existing evaluation methodology (building on LIME), but it has fundamental implications for how the field conducts comparative studies of frontier models. As models saturate existing benchmarks, the choice is between developing ever-harder benchmarks (slow, expensive, domain-specific) and constructing difficulty-filtered variants of existing benchmarks (fast, cheap, reusable). This paper demonstrates the latter approach at a scale and with a rigor that makes it a template for future work.

Innovation 3: The Generation-Selection Decomposition as a Lens for Understanding Test-Time Compute Limits

The paper's diagnostic experiments — varying reward model quality (Section 4.5) and search budget NN (Section 4.6) — collectively demonstrate a finding that, while not entirely novel in concept, is empirically characterized with unusual clarity: that the performance of search-based test-time compute methods is bounded by the product of two independent factors — the backbone model's output diversity and the reward model's discriminative accuracy — and that scaling either factor alone yields diminishing returns.

What the field did before. Prior work on Best-of-N and verifier-guided search (Cobbe et al., 2021; Lightman et al., 2023; Snell et al., 2024) studied scaling behavior but typically varied NN while holding the verifier fixed, or compared verifiers at a single NN. The decomposition into orthogonal bottlenecks — "can the model generate a correct answer?" and "can the verifier recognize it?" — was understood theoretically but rarely demonstrated empirically with clean factor separation. This paper achieves that separation through two experiments: (1) holding generation fixed (same GPT-4o outputs) while varying the reward model from weak (specialized RM scoring ~10%) to strong (human scoring ~33%), establishing the selection bottleneck; and (2) holding the reward model fixed while varying NN from 1 to 16 across three backbone models, establishing the generation bottleneck via the plateau at N>8N > 8.

Why the combination of these experiments is illuminating. Neither experiment alone tells the full story. The reward model experiment (Figure 4) shows that on HotpotQA, a perfect selector (human) achieves 33% — more than double the best automated reward model. This establishes that the correct answer exists in the candidate set often enough to achieve much higher accuracy if selection were perfect. The fact that o1 achieves substantially higher accuracy than this human-selection upper bound (o1-mini: ~62% on AIME; the HotpotQA human BoN ceiling is 33%) implies o1 is not just selecting better from the same distribution — it's generating from a different, better distribution. The search space experiment (Figure 5) reinforces this: even when NN grows, performance plateaus, meaning the backbone model's distribution has limited diversity and o1 must be doing something beyond exploring that same distribution more thoroughly.

The conceptual implication: test-time compute is not a unitary resource. This decomposition establishes that "spending more test-time compute" is ambiguous — you can spend it on generation (more samples, higher NN) or on selection (better reward models, more expensive verification per sample), and the returns to each are independent and saturating. This is analogous to the precision-recall tradeoff in information retrieval, but applied to reasoning: more generation is like increasing recall (more candidates includes more correct answers), while better selection is like increasing precision (fewer incorrect answers slip through). The plateau in Figure 5 shows that recall saturates (the model runs out of new correct answers to generate), while Figure 4 shows that precision can be far from saturation (better reward models could substantially improve accuracy from the same candidates).

Evidence anchor. The joint implication of Figure 4 (reward model ceiling) and Figure 5 (NN plateau) is that o1's advantage cannot be explained by either factor alone — it must involve a qualitatively different generation process, consistent with the paper's later finding that o1 deploys structured reasoning patterns (DC, SR) rather than independent parallel sampling.

Distinguishing incremental from fundamental. The generation-selection decomposition itself is incremental — these concepts exist in prior work. The empirical demonstration of their independent saturation, and the use of that demonstration to argue that o1 does something qualitatively different from BoN, is a fundamental diagnostic contribution that should inform how the community designs and evaluates future test-time methods. It changes the question from "how much test-time compute?" to "which bottleneck are we addressing with this test-time compute?"

Innovation 4: Task-Dependent Reasoning Strategy as a Learned Capability, Not Prompt Engineering

The finding that o1's reasoning patterns vary systematically by task domain (Figure 1) — CI and EC for commonsense reasoning, DC and MR for math and coding — is more than an observation about o1. It constitutes evidence for a capability that has no clear analog in prior systems: autonomous strategy selection based on problem type, implemented as part of the model's internal generation process rather than as externally imposed prompt structure.

What the field did before. Prior test-time compute methods achieve task-appropriate behavior through external mechanisms: Agent Workflow uses domain-specific system prompts hand-crafted by humans; Step-wise BoN relies on the initial decomposition step being appropriate for the problem type; Self-Refine uses a generic "critique and revise" loop regardless of domain. In all cases, the strategy — how to approach the problem, whether to decompose or search or verify — is either hard-coded by the system designer or induced by a generic prompt ("think step by step") that applies uniformly. The model itself doesn't choose a reasoning strategy; it follows the one it's given.

o1's pattern distributions suggest something different: the model adapts its reasoning approach to the problem type without external prompting to do so. When given a commonsense QA problem (HotpotQA), it deploys CI to process documents and EC to track constraints. When given a math problem (AIME), it deploys DC to decompose and MR to apply known techniques. When given a coding problem (USACO), it deploys SA for structural analysis and DC for algorithmic decomposition. The same model, without task-specific system prompts, shifts its reasoning strategy based on what the problem demands.

Why this matters for the field's understanding of reasoning. This capability — if real — implies that o1 has learned, during training, to perform meta-reasoning: assessing the problem type and selecting an appropriate reasoning strategy. This is a fundamentally different capability from executing a reasoning strategy that was provided in the prompt. It suggests that o1's training process taught it not just how to reason (the individual patterns) but when to reason which way (the strategy selection). This is a higher-order skill that existing test-time compute methods cannot replicate because they hard-code the strategy choice through prompts, rather than learning it from data.

The caveat and why the claim is appropriately tentative. The paper doesn't prove that o1 learned this adaptation — it could be an emergent property of a system that always deploys all patterns and the observed distributional differences reflect which patterns produce useful outputs for different tasks (i.e., on Collie, EC-heavy traces survive while DC-heavy traces produce constraint violations and get pruned internally). The paper's black-box methodology cannot distinguish "the model chose this strategy" from "this strategy survived internal filtering." But the empirical pattern — task-dependent strategy distributions — is robust and unlikely to be an artifact of the annotation procedure, since the pattern-task alignments are intuitively sensible and consistent.

Evidence anchor. Figure 1 shows CI at ~70 occurrences for HotpotQA but near-zero for AIME and USACO; EC at ~65 for Collie but near-zero elsewhere; DC and SA at 50–60 for USACO. These are not subtle differences — they are categorical shifts in reasoning behavior across tasks.

Distinguishing incremental from fundamental. If confirmed through further research (e.g., controlled experiments varying problem type while holding difficulty constant), this finding would be fundamental: it identifies a capability — learned meta-reasoning — that represents a qualitative advance over prompt-based strategy specification and would need to be explicitly targeted in future model training. The paper's contribution is identifying this as an empirical pattern and framing it as a capability worth building toward, not proving its mechanism.

5. Experimental Analysis

Evaluation Methodology

  • Dataset. The paper evaluates on four benchmarks spanning three domains: HotpotQA (Yang et al., 2018) — a multi-hop commonsense QA dataset requiring synthesis across multiple documents; Collie (Yao et al., 2023) — a constrained text generation dataset requiring outputs to satisfy lexical and structural constraints; USACO (Shi et al., 2024) — bronze-level algorithmic coding competition problems; and AIME — 90 problems drawn from AIME 22, AIME 23, and AIME 24 competitions (sourced from the AOPS wiki). For HotpotQA and Collie, a difficulty filter is applied: four models (Llama3-72B, Qwen-72B, Claude, and an unspecified fourth — likely Yi) answer each sample, and samples answered correctly by more than two of the four models are discarded. This retains 274 HotpotQA questions and 226 Collie questions. USACO and AIME are also filtered, with the same principle of retaining only problems that prove challenging across multiple models, yielding 139 USACO problems and 90 AIME problems (Table 3). The filtering is designed to create a benchmark where strong models' performance differences are measurable rather than compressed at the ceiling.

  • Base model(s). All test-time compute baselines use GPT-4o as the backbone — selected because it is the most capable publicly available model with an API at the time of writing, representing the state of the art for standard (non-reasoning-enhanced) LLMs. The paper also evaluates o1-preview and o1-mini as direct generation baselines, representing OpenAI's proprietary reasoning-enhanced models. For the search space scaling experiment (Section 4.6), two additional backbone models are tested with BoN: Qwen2.5-72B and Llama3-70B. The paper does not report whether GPT-4o, o1-preview, and o1-mini share the same underlying architecture or parameter count — this is unknown for proprietary models.

  • Metrics. The primary metric is accuracy (% of questions answered correctly), but grading procedures differ by benchmark. For HotpotQA and AIME, a rule-based extraction checks whether the model's output contains the correct answer (substring match for HotpotQA text spans; integer match for AIME numeric answers). For Collie, correctness is determined by whether the generated paragraph satisfies all specified constraints (e.g., exactly 3 sentences, no word "be," no word "of," no word "is"). For USACO, the model's generated code is manually executed against test cases, and a sample is scored correct only if all test cases pass. The paper reports accuracy per benchmark and also an "Overall" score in Table 1, though the aggregation method for "Overall" (simple average? weighted?) is not specified.

  • Baselines. The paper evaluates four test-time compute methods, all using GPT-4o as the backbone, plus direct generation from three models:

    • GPT-4o (Direct): Single forward pass, no test-time augmentation.
    • o1-preview (Direct): OpenAI's reasoning model, "designed to spend more time reasoning before they respond."
    • o1-mini (Direct): A smaller, faster variant of o1 optimized for STEM reasoning.
    • Best-of-N (BoN) (N=4): Generates NN independent outputs from GPT-4o, scores each with a reward model (GPT-4o itself acting as judge), and selects the highest-scoring output. Also evaluated at N=1N=1, N=8N=8, and N=16N=16 in scaling experiments.
    • Step-wise BoN (N=4): Decomposes the problem into sub-problems, generates NN candidates at each step, selects the best per step via the reward model, and conditions subsequent steps on accepted previous steps. Also evaluated at N=1N=1.
    • Self-Refine: Implements the iterative feedback-and-revision loop from Madaan et al. (2024), using 3 refinement iterations. The paper uses the published Self-Refine code directly.
    • Agent Workflow: Uses domain-specific agent frameworks: the Agents framework (Zhou et al., 2023; 2024) for commonsense reasoning (HotpotQA, Collie), and GPTs (OpenAI's customizable chatbot configurations) — specifically "code copilot" for USACO and "math solver" for AIME. These frameworks provide structured system prompts and, where applicable, tool access.

  • Generation budget / compute accounting. The paper does not define a unified compute metric. For BoN, the budget is NN (number of complete generations); for Step-wise BoN, it is N×KN \times K where KK is the number of decomposition steps; for Self-Refine, it is 4 generations (1 initial + 3 refinement iterations); for Agent Workflow, the cost is not quantified because the number of internal agent steps depends on the framework's operation. There is no FLOPs or token-based accounting, and latency/wall-clock time is not measured. Comparisons are therefore in terms of accuracy per method configuration, not accuracy per unit of compute. This is a significant methodological choice: different methods consume vastly different amounts of inference compute, and the paper does not control for this in its primary comparison (Table 1).

  • Cross-validation / statistical protocol. The paper does not report confidence intervals, standard deviations, or statistical significance tests for any of its accuracy measurements. For the reasoning pattern annotation (Section 4.2), 20–30 samples per benchmark are randomly selected and manually annotated; inter-annotator agreement is not reported. The data filtering procedure uses a fixed threshold (more than two models correct → discard), with no sensitivity analysis on the threshold choice. All accuracy numbers in Table 1 are point estimates without error bounds.

Main Quantitative Results

Overall Benchmark Comparison (Table 1)

The headline result from Table 1 is that o1-mini achieves the highest accuracy on 3 of 4 benchmarks (Collie: 53.53%, USACO: 12.23%, AIME: 62.00%), while o1-preview achieves the highest on HotpotQA (14.59%) and the best overall score (34.32% vs. o1-mini's 35.77% — it's unclear which metric produces "best overall" since o1-mini leads on 3 of 4 tasks). The o1 models' advantage is largest in math and coding: on AIME, o1-mini achieves 62.00% vs. GPT-4o's 12.22% — a roughly 5× improvement — while on USACO, o1-preview achieves 44.60% vs. GPT-4o's 5.04% (nearly 9× improvement). In commonsense reasoning, the advantage is smaller but still substantial: on Collie, o1-mini leads GPT-4o by 10.17 percentage points (53.53% vs. 43.36%).

BoN results (Table 1). BoN with N=4N=4 provides marginal improvement over GPT-4o Direct on HotpotQA (13.50% vs. 13.14%) but degrades on Collie (39.82% vs. 43.36%) and shows no improvement on USACO (5.04% for both) or AIME (12.22% for both). When NN increases to 8, BoN improves further on HotpotQA (16.42%) and slightly on USACO (7.91%) and AIME (13.33%), but continues to underperform GPT-4o Direct on Collie (38.50%). The paper attributes the Collie degradation to "Collie's strict format requirements, which limit the effectiveness of diverse outputs from LLMs" (Section 4.1) — generating multiple diverse candidates increases the chance that some candidates violate format constraints, and the reward model may select a format-violating candidate that appears otherwise high-quality.

Step-wise BoN results (Table 1). Step-wise BoN (N=4N=4) achieves the best BoN-family result on HotpotQA (15.69% vs. GPT-4o's 13.14%), but performance collapses on other benchmarks: Collie drops from 43.36% (GPT-4o Direct) to 19.55% (N=4N=4) and 5.31% (N=1N=1); USACO scores 0.00% for both N=1N=1 and N=4N=4; AIME drops to 7.78% (N=4N=4) from 12.22% (GPT-4o Direct). The paper explains this as a combination of long-context instruction-following failure (Table 2: average reasoning token lengths of 274–450 for Step-wise BoN) and the specific challenge of Collie's and AIME's "highly complex" output format and reasoning requirements (Section 4.3).

Self-Refine results (Table 1). Self-Refine with 3 iterations scores 13.25% on HotpotQA (compared to 13.14% for GPT-4o Direct), 0.00% on Collie (vs. 43.36%), 0.00% on USACO (vs. 5.04%), and 9.23% on AIME (vs. 12.22%). The overall score of 5.62% is the lowest of any method. The paper notes that the performance improvement is "not significant" and attributes failures to "responses that slightly deviate from the required format during the refinement iterations" (Section 4.1), particularly for Collie where constraint adherence degrades with each revision.

Agent Workflow results (Table 1). Agent Workflow achieves the best performance among test-time compute methods on all four benchmarks: HotpotQA 14.96%, Collie 46.07%, USACO 22.22%, AIME 15.56%, with an overall score of 24.70%. On Collie, Agent Workflow (46.07%) outperforms GPT-4o Direct (43.36%) and is the only test-time method to do so; on USACO, its 22.22% far exceeds all other test-time methods (which score 0–7.9%); on AIME, it achieves 15.56% vs. 12.22% for GPT-4o Direct. However, there remains a substantial gap to o1 models: on USACO, o1-preview's 44.60% is 2× Agent Workflow's 22.22%; on AIME, o1-mini's 62.00% is 4× Agent Workflow's 15.56%.

The o1-preview vs. o1-mini inversion. An intriguing pattern in Table 1: o1-mini outperforms o1-preview on Collie (53.53% vs. 34.07%), AIME (62.00% vs. 44.00%), and overall (35.77% vs. 34.32%), but o1-preview dominates on USACO (44.60% vs. 12.23%). This non-monotonic relationship between model variant and task suggests that "the reasoning process of o1 does not always lead to better improvements" (Section 4.1) — the smaller, STEM-optimized variant can outperform the larger preview variant on math and constrained generation, while the preview variant is dramatically better at coding. The paper does not provide an explanation for this pattern beyond noting its existence.

Search Space Scaling Analysis (Figure 5)

BoN performance is evaluated on filtered HotpotQA with N∈{1,4,8,16}N \in \{1, 4, 8, 16\} for three backbone models: GPT-4o, Qwen2.5-72B, and Llama3-70B. The key finding is that performance plateaus or declines beyond N=8N=8 for all three backbones. Qwen2.5-72B shows the strongest BoN scaling, rising from approximately 18% at N=1N=1 to roughly 40% at N=4N=4, then flattening at roughly 40% for N=8N=8 and N=16N=16. GPT-4o rises from approximately 13% at N=1N=1 to roughly 17% at N=4N=4, then plateaus at roughly 16% for N=8N=8 and N=16N=16. Llama3-70B follows a similar pattern at lower absolute values. The paper interprets this plateau as evidence that "the performance of the search methods is jointly related to the reward model and searching space" (Section 4.6) — beyond a certain NN, the backbone model has exhausted its distribution of meaningfully different correct answers, and additional samples are variants of the same (mostly incorrect) approaches that the reward model cannot usefully distinguish.

Reward Model Comparison (Figure 4)

For BoN (N=4N=4, GPT-4o backbone) on HotpotQA and Collie, four reward models are compared: Skywork-Reward-Gemma-2-27B (SRG), URM-LLaMa-3.1-8B, GPT-4o as judge, and human selection. On HotpotQA, the quality gradient is steep: SRG and URM-LLaMa achieve under 15% accuracy, GPT-4o achieves approximately 13.5% (consistent with Table 1), while human selection reaches approximately 33% — more than doubling the best automated result. This establishes that the correct answer exists in the candidate set substantially more often than automated reward models can identify it, demonstrating a selection bottleneck. On Collie, the gap is narrower: automated reward models produce results closer to human performance, suggesting Collie's constraint-based correctness is more objectively evaluable by automated methods. The paper concludes: "the reward model is crucial for the search methods" (Section 4.5), and improving the reward model could unlock substantial gains from existing generation distributions.

Reasoning Pattern Frequency Analysis (Figures 1 and 2)

Manual annotation of o1's reasoning traces (20–30 samples per benchmark, 80–120 total) yields the following pattern frequencies (Figure 1):

  • HotpotQA: Context Identification (CI) dominates at approximately 70 occurrences in the annotated sample, followed by Divide and Conquer (DC) at approximately 50. Other patterns (SA, MR, SR, EC) appear less frequently at roughly 10–20 occurrences each.
  • Collie: Emphasizing Constraints (EC) dominates at approximately 65 occurrences, with all other patterns appearing infrequently (roughly 5–10 each).
  • USACO: DC and Systematic Analysis (SA) are most frequent at approximately 50–60 occurrences each, with Method Reuse (MR) at approximately 35, and SR at roughly 25.
  • AIME: DC dominates at approximately 35 occurrences, with MR at approximately 25, and SR/SA at roughly 15 each.

Aggregating across all benchmarks (Figure 2), DC and SR appear most frequently overall (roughly 60–80 occurrences each), followed by SA (~65), CI (~75, concentrated in HotpotQA), EC (~70, concentrated in Collie), and MR (~45). The paper interprets this as evidence that "the combination of SR and DC plays a crucial role in enhancing the performance of o1" (Section 4.2).

Reasoning Token Analysis (Figure 3 and Table 2)

The paper extracts o1's reasoning tokens and computes average token counts per task, separately for all samples, correct samples, and incorrect samples (Figure 3). The key observations:

  • Reasoning length varies dramatically by task domain: USACO produces the longest reasoning traces (average token counts for "ALL" are highest), followed by AIME, then Collie and HotpotQA (commonsense reasoning tasks have substantially shorter traces).
  • Within a task, correct and incorrect samples have similar reasoning lengths: there is "no clear correlation between the input prompt length and the length of the reasoning tokens" (Section 4.4). The reasoning length is task-determined (harder tasks elicit longer thinking) rather than performance-determined (longer thinking doesn't distinguish correct from incorrect outputs within a task).
  • Input prompt length does not predict reasoning length: the "Input" bar in Figure 3 is substantially smaller than reasoning token bars for all tasks and does not covary with reasoning length.

Table 2 reports the average reasoning token length of Step-wise BoN (N=4N=4), not o1: 273.59 for HotpotQA, 450.31 for Collie, 439.90 for USACO, and 262.51 for AIME. The paper uses these to support the claim that Step-wise BoN's long intermediate contexts degrade instruction-following: "excessively long context information can prevent the model from following the original input text to generate the correct response" (Section 4.3).

Data Filter Effectiveness (Figure 8)

Figure 8 compares LLM scores on raw vs. filtered versions of HotpotQA (left subfigure) and Collie (right subfigure). On raw HotpotQA, multiple models cluster at similar high scores; on filtered HotpotQA, scores are lower and show greater separation between models. On Collie, the pattern is similar: the filtered benchmark produces larger performance gaps between models. The paper concludes that the filtering "demonstrates the effect of our data filter strategy" (Section 4.7), noting that "on HotpotQA, the differences between Qwen2.5 and GPT-4o become evident on our filtered benchmark." Specific numeric values are not quoted in the text for Figure 8.

AIME Year-by-Year Analysis (Figure 6)

o1-preview, o1-web, and o1-mini are evaluated separately on AIME 22, AIME 23, and AIME 24 (Figure 6). o1-mini achieves the best performance across all three years, reaching approximately 60% on AIME 24, roughly 55–60% on AIME 22, and roughly 55% on AIME 23. o1-preview's performance fluctuates significantly: approximately 57% on AIME 24 but around 40% on AIME 22 and AIME 23. o1-web (which the paper mentions only in Figure 6 and does not describe elsewhere) tracks between the two, at roughly 45–50% across all three years. The paper notes that "o1-mini demonstrates the best performance (around 60%) across these three datasets" while "the performance of the o1-preview fluctuates significantly across different datasets" (Section 4.8). No explanation is offered for the fluctuation.

Ablation Studies and Robustness Checks

Reward model quality (Figure 4): Varying the reward model used for BoN selection from weak (specialized reward models scoring under 15% on HotpotQA) to strong (human selection scoring 33%) reveals that selection quality, not generation diversity, is the binding constraint on HotpotQA — the correct answer exists in the candidate set substantially more often than automated reward models can identify it. On Collie, where automated reward models perform closer to human-level, the selection bottleneck is less severe, suggesting that constraint-satisfaction tasks are more objectively evaluable by automated methods.

Search budget NN (Figure 5): Increasing NN from 1 to 16 for BoN on HotpotQA across three backbone models (GPT-4o, Qwen2.5-72B, Llama3-70B) shows that performance plateaus around N=8N=8 for all models. This is a robustness check confirming that the plateau is not specific to GPT-4o — even a stronger backbone (Qwen2.5-72B achieves roughly 40% at N=4N=4 vs. GPT-4o's ~17%) hits the same saturating pattern. The invariance of the plateau point across backbones of different capabilities suggests it reflects a structural property of BoN rather than a particular model's limitation.

Step-wise BoN NN variation (Table 1): For Step-wise BoN, increasing NN from 1 to 4 improves performance on HotpotQA (13.50% → 15.69%) and AIME (5.56% → 7.78%) but degrades on Collie (5.31% → 19.55% — wait, this is an improvement, but from an already-collapsed base) and has no effect on USACO (0.00% for both). This is a partial negative result: per-step search provides some benefit in multi-hop QA but cannot rescue performance on tasks where the decomposition itself fails (USACO, Collie).

Self-Refine iteration count (Table 1, negative result): Using 3 refinement iterations (the Madaan et al., 2024 default) produces minimal or negative improvement over GPT-4o Direct across all benchmarks. This is a negative result confirming that iterative self-critique without learned refinement capabilities is insufficient for the reasoning tasks tested, consistent with Huang et al. (2023).

Difficulty filtering (Figure 8): Comparing model scores on raw vs. filtered benchmarks confirms that the filtering procedure successfully increases discrimination between models. This is a robustness check for the benchmark construction methodology: the filter achieves its intended effect of creating headroom for comparing strong models.

o1 variant comparison (Table 1, Figure 6, implicit ablation): Comparing o1-preview and o1-mini across benchmarks reveals that the relationship between model variant and performance is task-dependent and non-monotonic. On Collie, o1-mini outperforms o1-preview by 19.46 percentage points (53.53% vs. 34.07%); on AIME, o1-mini leads by 18.00 points (62.00% vs. 44.00%); but on USACO, o1-preview leads o1-mini by 32.37 points (44.60% vs. 12.23%). This is not explicitly framed as an ablation but functions as one: it shows that "more reasoning" (presumably o1-preview does more internal deliberation than o1-mini) does not uniformly improve performance.

Backbone model for BoN (Figure 5): Evaluating BoN with three different backbone models (GPT-4o, Qwen2.5-72B, Llama3-70B) confirms that the NN plateau is not an artifact of GPT-4o specifically. All three backbones saturate around N=8N=8, though at different absolute accuracy levels. This is a robustness check demonstrating that the plateau is a property of the BoN method, not the specific backbone.

Critical Assessment

Claim: OpenAI's o1 model achieves the best performance on most benchmarks

This claim is supported by Table 1, with o1-mini leading on 3 of 4 benchmarks and o1-preview leading on HotpotQA. However, the claim requires qualification. First, "best performance on most benchmarks" is true for the filtered benchmarks — the paper does not report o1 performance on the full, unfiltered benchmarks, so we cannot determine whether o1's advantage is specific to hard problems or generalizes to easier ones. Second, the comparison is against GPT-4o-based test-time methods specifically — the paper does not compare against other closed-source reasoning models (e.g., Claude 3.5 with extended thinking, if available) or against GPT-4o with more sophisticated test-time strategies (e.g., combining Agent Workflow with BoN, or using o1's own outputs to train a verifier for GPT-4o). The claim that o1 is "best" is therefore relative to a specific, non-exhaustive set of baselines.

A deeper concern: the paper does not control for the possibility that o1 and GPT-4o are different models with different base capabilities. If o1 is a fundamentally different architecture, trained on different data, or substantially larger than GPT-4o, then o1's superior performance may reflect pretraining differences rather than the effectiveness of o1's reasoning mechanism. The paper implicitly assumes that comparing o1 against GPT-4o + test-time compute isolates the effect of o1's reasoning approach, but without knowing o1's base model specifications, this assumption is unverifiable. If o1 is, say, a 10× larger model than GPT-4o with additional reasoning-specific training, its advantage on math and coding may partly reflect scale and training data rather than reasoning mechanism.

Claim: The Agent Workflow method achieves better performance than Step-wise BoN due to domain-specific system prompts

This claim is supported by Table 1 (Agent Workflow: 24.70% overall vs. Step-wise BoN: 9.79% at N=4N=4) and the observation that Agent Workflow uses "domain-specific system prompts" while Step-wise BoN uses a generic decomposition. However, there are important confounds. Agent Workflow uses different frameworks for different domains — the Agents framework for commonsense reasoning, GPTs code copilot for coding, GPTs math solver for math — while Step-wise BoN uses the same generic decomposition procedure across all tasks. The performance difference could therefore reflect either (a) the benefit of domain-specific prompting, (b) the benefit of the specific agent frameworks used (which may have additional capabilities beyond prompting, such as tool use or memory), or (c) the fact that Step-wise BoN suffers from specific failure modes (long-context degradation, error accumulation) that are not inherent to decomposition-based approaches but are artifacts of the particular Step-wise BoN implementation.

The paper does not run an ablation where Agent Workflow uses a generic (non-domain-specific) prompt, nor one where Step-wise BoN uses domain-specific decomposition prompts. Without these, the causal attribution to "domain-specific system prompts" is speculative. It could equally be that Agent Workflow's structured execution framework (with planning, tool use, and error recovery) matters more than the domain-specificity of the prompts.

Claim: Six reasoning patterns are identified in o1, and DC and SR are the most important

The pattern taxonomy is a genuine empirical contribution, but the evidence for "DC and SR are the most important" is weak. Figure 2 shows DC and SR as the most frequent patterns, but frequency does not establish importance. A pattern could appear frequently in both correct and incorrect traces — the paper does not separate frequencies by correctness, so we cannot determine whether DC or SR actually predict successful reasoning. A pattern could be frequent because complex problems naturally elicit it (e.g., hard problems require decomposition regardless of whether the model ultimately succeeds), making frequency a marker of problem complexity rather than solution effectiveness.

More fundamentally, the annotation is performed on a small sample (20–30 traces per benchmark, or roughly 80–120 total traces) by an unspecified number of annotators with no reported inter-annotator agreement. The pattern definitions (SA, MR, DC, SR, CI, EC) are described in a few sentences each in Section 4.2, which may not be sufficient for reliable annotation — distinguishing "Systematic Analysis" from "Divide and Conquer" or "Method Reuse" from "Divide and Conquer" could be ambiguous in practice. The paper's claim that these patterns "might be the key to o1's success" (Section 4.2) is appropriately hedged, but the hedging should be stronger given the methodological limitations.

A missing experiment: the paper could have annotated reasoning patterns in the incorrect o1 traces separately and compared pattern distributions for correct vs. incorrect samples. If DC and SR appear with similar frequency in both, their role as "key to success" is questionable. If they appear significantly more often in correct traces, the causal claim is strengthened.

Claim: The number of reasoning tokens varies across tasks, and there is no clear correlation with correctness within a task

This claim is supported by Figure 3 and seems robust. However, the paper's interpretation — that "for more difficult tasks like Code and Math, the model often requires a longer reasoning process" — conflates task difficulty with task domain. USACO and AIME are harder benchmarks than HotpotQA and Collie, but they also involve fundamentally different types of reasoning (algorithmic design vs. document synthesis vs. constraint satisfaction). The longer reasoning traces could reflect the nature of the reasoning (code requires step-by-step algorithm development) rather than difficulty per se. A within-domain difficulty comparison (easy vs. hard math problems, easy vs. hard coding problems) would distinguish these explanations but is not performed.

Additionally, the paper's observation that correct and incorrect traces have similar lengths within a task is based on averages (the "True" and "False" bars in Figure 3). This doesn't rule out distributional differences — for example, very short traces might be uniformly incorrect (insufficient reasoning) while medium-length traces might have the highest accuracy, and the average could be similar. A histogram or distribution plot would be more informative than mean comparisons.

Claim: The reward model's capability and the search space both limit the upper boundary of searching methods

This claim is well-supported by the combination of Figure 4 (reward model quality dramatically affects BoN accuracy — 33% human vs. under 15% for specialized RMs — on HotpotQA) and Figure 5 (NN plateau around N=8N=8 for all three backbones). The two experiments together demonstrate independent saturation of selection quality and generation diversity. However, the claim's quantitative precision is limited by the fact that the reward model experiment (Figure 4) is run only on HotpotQA and Collie, not on USACO or AIME. We don't know whether the reward model bottleneck is equally severe for code and math tasks, where answer correctness may be more objectively verifiable (code passes tests or doesn't; math answers are right or wrong). The human-selection ceiling might be higher for those tasks, or the automated reward models might perform better.

A missing experiment: evaluating BoN with human selection on AIME and USACO. If human selection on AIME achieves, say, 40% while o1 achieves 62%, the claim that o1 does something qualitatively different from BoN is strengthened. If human selection achieves close to 62%, then o1's advantage might be largely attributable to having a better verifier (or more computation for verification) rather than a qualitatively different reasoning process.

Claim: Step-wise BoN is limited by long-context instruction-following

This claim is supported by Table 2 (Step-wise BoN reasoning token lengths of 263–450) and the observation that Step-wise BoN performs worst on tasks with complex output formatting (Collie: 5.31% at N=1N=1, 19.55% at N=4N=4; USACO: 0.00% for both). However, the paper does not directly test the long-context hypothesis. It doesn't, for example, truncate Step-wise BoN's context to see if shorter contexts improve performance, or compare Step-wise BoN performance on the same problems with different context lengths. The attribution of Step-wise BoN's failure to long-context degradation is therefore plausible but untested — it could equally be that the decomposition quality is poor (the initial breakdown into sub-problems is wrong), that the per-step reward model is ineffective, or that error accumulation (not context length) is the primary failure mode. These mechanisms are not experimentally separated.

Missing experiments that would strengthen the paper

  1. Combining test-time methods. Agent Workflow + BoN, or Agent Workflow + Self-Refine, would test whether hybrid approaches can close the gap with o1. The paper's conclusion that o1 does something qualitatively different is weakened if a simple combination of existing methods significantly improves performance.

  2. Within-task difficulty stratification. Does o1's advantage over test-time methods persist on easier problems, or is it specific to the hard problems selected by the filter? Repeating the comparison on the filtered-out (easy) samples would reveal whether o1's reasoning patterns are uniformly beneficial or only activate on challenging problems.

  3. Controlled reward model training. Training a reward model specifically on GPT-4o's outputs for each benchmark (rather than using off-the-shelf models) would strengthen the BoN results, since the specialized RMs used in Figure 4 may not be well-calibrated for these specific tasks.

  4. Inter-annotator agreement for reasoning patterns. Without it, the reliability of the six-category taxonomy is unknown. Two independent annotators applying the same category definitions to the same traces would reveal whether the patterns are objectively identifiable or subjective.

  5. Correctness-conditioned pattern analysis. Reporting pattern frequencies separately for correct and incorrect o1 traces would transform the pattern analysis from descriptive to potentially diagnostic, addressing the "frequency ≠ importance" concern.

  6. Compute-matched comparison. Controlling for total inference FLOPs or tokens across methods (Agent Workflow vs. BoN at NN chosen to equalize compute cost) would make the efficiency claims more precise. Currently, Agent Workflow may use more or less compute than BoN (N=4N=4), and we cannot determine which is more efficient.

  7. Statistical uncertainty. Confidence intervals or standard errors for all accuracy measurements would allow readers to assess whether reported differences (e.g., o1-mini 62.00% vs. o1-preview 44.00% on AIME) are statistically significant given the test set sizes (90 AIME problems).

  8. Ablation of domain-specific prompts in Agent Workflow. Running Agent Workflow with a generic system prompt (e.g., "solve this problem step by step") vs. the domain-specific prompts would isolate the contribution of domain knowledge in the prompt from the contribution of the structured workflow.

Conditions on the claims

The paper's central finding — o1 outperforms existing test-time compute methods — holds on the filtered, hard subset of four benchmarks, with the o1 advantage being largest on math and coding. The finding that DC and SR are the most frequent o1 reasoning patterns holds across the 80–120 annotated traces but hasn't been replicated on larger samples or by independent annotators. The finding that test-time search methods are bounded by reward model quality and search space holds on HotpotQA specifically (Figure 4, Figure 5) with suggestive extension to Collie, but the quantitative bounds may differ substantially for code and math tasks where verification is more objective. The paper's claims about why o1 works (DC and SR as key mechanisms, internal planning as the distinguishing factor) are interpretive hypotheses supported by observed correlations, not experimentally demonstrated causal mechanisms.

6. Limitations and Trade-offs

6.1 The Black-Box Nature of o1 Prevents Causal Claims About Reasoning Mechanisms

The assumption or constraint. The paper's central analytical move — identifying six reasoning patterns in o1's traces and claiming that DC and SR "might be the key to o1's success" (Section 4.2) — rests on the assumption that observable output patterns reflect internal reasoning mechanisms. The paper acknowledges this limitation implicitly by positioning itself as an observational study rather than a causal analysis, but the distinction matters for practical use. The paper can see what o1 writes in its thinking traces, but it cannot see how those traces are generated — whether through learned internal planning, through a search process that prunes unpromising branches before output, through a verifier that scores and selects reasoning paths, or through some combination. The paper states in the abstract that "the mechanisms behind these methods are still unexplored," and its contribution is to provide empirical characterization, not mechanistic explanation.

The consequence. A developer reading this paper cannot determine whether to invest in (a) training data that demonstrates DC and SR patterns, (b) reinforcement learning that rewards models for exhibiting these patterns, (c) inference-time search architectures that implement DC-like decomposition, or (d) something else entirely. The observed patterns could be epiphenomenal — things o1 says while reasoning that correlate with but do not cause successful problem-solving. For example, o1 might always produce decomposition-like language when tackling complex problems, regardless of whether the decomposition actually structures its computation. If so, training models to mimic the surface form of DC and SR would not replicate o1's capabilities. The paper's pattern analysis provides descriptive vocabulary but not causal design principles, and conflating the two would lead to misplaced engineering effort.

What evidence exists in the paper. The paper does not measure the causal contribution of individual reasoning patterns. It reports pattern frequencies (Figures 1 and 2) but does not separate frequencies for correct vs. incorrect traces — so we cannot determine whether DC appears more often when o1 succeeds (suggesting causal importance) or appears uniformly regardless of outcome (suggesting it is a generic feature of o1's generation style). The annotation sample is small (20–30 traces per benchmark, Section 4.2), inter-annotator agreement is not reported, and the pattern definitions (SA, MR, DC, SR, CI, EC) are described in single sentences each, leaving substantial room for annotation subjectivity. The paper does not run the critical ablation: applying the same annotation methodology to incorrect o1 traces and comparing pattern distributions. Without this, the claim that "SR and DC are the main reasoning patterns of o1" and are "crucial for improving reasoning performance" (Section 6) is an interpretive leap beyond what the data support.

Mitigation status. The paper does not attempt to establish causality. It acknowledges the limitation implicitly through cautious language ("might be the key," Section 4.2) but does not discuss the gap between observed patterns and underlying mechanisms. No future work is proposed to bridge this gap (e.g., training a model to exhibit specific patterns and measuring the causal effect, or comparing pattern frequencies in correct vs. incorrect traces). The limitation is not addressed — the paper treats pattern identification as the endpoint of analysis rather than as hypothesis generation requiring further testing.

6.2 The Difficulty Filtering Procedure Removes Generalizability to Standard Benchmarks

The assumption or constraint. The paper applies a model-based difficulty filter that retains only samples "that more than two models can answer correctly" are discarded (Section 3.1 — the filter keeps samples where ≥2 of 4 judge models get the answer wrong). This produces benchmarks where accuracy rates are substantially lower than on the original datasets: Figure 8 shows that models score much lower on filtered HotpotQA and Collie than on raw versions. The paper's headline results (Table 1) are reported exclusively on these filtered benchmarks, with the raw-benchmark performance shown only in Figure 8 for comparison purposes. The paper does not evaluate o1 on the full, unfiltered benchmarks.

The consequence. The reported accuracy numbers are not comparable to any other published results on these benchmarks, since other papers report on the full datasets. A practitioner deciding whether to adopt o1 or Agent Workflow for a production task cannot use Table 1 to estimate expected accuracy on their problem distribution unless that distribution happens to match the filtered-hard subset. If their real-world problems span a range of difficulties — including easy problems that the filter removed — the reported numbers may substantially underestimate actual performance (since easy problems would raise accuracy) or overestimate the relative advantage of o1 over baselines (if o1's advantage is largest on hard problems and smaller or nonexistent on easy ones). The paper demonstrates that the filter successfully creates performance separation between models (Figure 8), but this very success means the results are specific to the hard subset and do not characterize model behavior on the full difficulty spectrum.

What evidence exists in the paper. Figure 8 directly shows that model scores are substantially lower and more differentiated on filtered vs. raw benchmarks. The paper presents this as evidence that the filter works ("demonstrates the effect of our data filter strategy," Section 4.7), but it also constitutes evidence of a generalizability limitation: the quantitative results in Table 1 are conditional on the filtering threshold (≥2 of 4 models wrong → keep) and would change if the threshold were adjusted. The paper does not report sensitivity to the filtering threshold — what happens if the threshold is 3 of 4 models wrong? 1 of 4? No sensitivity analysis is performed. The paper also does not report o1's performance on the discarded (easy) samples, leaving unknown whether o1's reasoning patterns provide any benefit on problems that current LLMs already handle well.

Mitigation status. The paper does not address this limitation. It does not report results on unfiltered benchmarks for o1, does not analyze performance as a function of problem difficulty within the filtered set, and does not discuss the external validity implications of the filtering procedure. No future work is proposed on generalizing the findings to broader difficulty distributions. The limitation is not addressed, and readers should treat the reported accuracies as specific to a hard-problem subset rather than as estimates of expected performance on the original benchmarks.

6.3 Compute and Latency Costs Are Not Measured, Making Efficiency Comparisons Impossible

The assumption or constraint. The paper compares methods solely on accuracy, with no measurement, estimation, or control of the computational cost required to achieve that accuracy. The test-time compute methods consume vastly different amounts of inference compute: BoN (N=4N=4) generates 4 complete outputs per question; Step-wise BoN (N=4N=4) generates 4 candidates per decomposition step, with total cost depending on the number of steps; Self-Refine generates 1 initial output plus 3 refinements = 4 outputs per question; Agent Workflow's cost is unquantified (it depends on the internal operation of the agent frameworks and GPTs used). The o1 model's cost is also unmeasured — the paper can count o1's reasoning tokens (Figure 3) but cannot compare this to the GPT-4o baselines' token costs because o1's per-token pricing, model size, and architecture are unknown. The paper acknowledges none of this: there is no discussion of FLOPs, tokens, API cost, latency, or any other compute metric.

The consequence. A practitioner cannot determine which method offers the best accuracy per unit of cost or time. For example, Agent Workflow achieves 24.70% overall accuracy vs. BoN (N=4N=4)'s 17.65% (Table 1), but if Agent Workflow consumes 50× more tokens per question (plausible given the structured multi-step agent execution), then BoN might be more cost-effective for applications with budget constraints. Conversely, if o1 achieves 44.00% on AIME but costs 100× more per query than GPT-4o + Agent Workflow at 15.56%, the practical superiority depends on the value of accuracy vs. the cost of compute. The paper's finding that BoN plateaus at N>8N>8 (Figure 5) shows that additional compute yields diminishing returns, but doesn't establish whether the initial compute investment in BoN is more or less efficient than the compute investment in Agent Workflow or o1. Without cost measurement, the paper's comparisons establish a performance ranking but not a resource-efficiency ranking, which is what deployment decisions require.

What evidence exists in the paper. The paper measures some proxy quantities: Figure 3 shows o1's reasoning token counts (varying from short for commonsense tasks to long for math/coding), and Table 2 shows Step-wise BoN's average reasoning token length (263–450 tokens). But these are not compared across methods — we don't know how many tokens Agent Workflow or Self-Refine consume per question. Figure 5 sweeps NN for BoN (showing the compute-accuracy tradeoff for that one method) but doesn't place other methods on the same cost axis. The paper reports generation budgets for some methods (N=4N=4 for BoN, 3 iterations for Self-Refine) but these are not commensurable: a BoN generation and an Agent Workflow step consume different amounts of computation. The paper's "Overall" score in Table 1 aggregates accuracy across benchmarks without any weighting by cost, latency, or benchmark size.

Mitigation status. The paper does not address this limitation. It does not mention compute cost, latency, or efficiency as relevant dimensions. No future work is proposed on cost-controlled comparisons. The limitation is not addressed, and all comparative claims in the paper should be interpreted as accuracy comparisons at unspecified and potentially very different compute costs.

6.4 Single Backbone Model (GPT-4o) and Single Reasoning-Enhanced Model (o1) Limit Generalizability

The assumption or constraint. All test-time compute baselines use GPT-4o as the sole backbone model. The reasoning pattern analysis examines only o1 (and only the specific o1-preview and o1-mini versions available at the time). The paper does not test whether its findings — about which test-time methods work best, about the NN plateau in BoN, about the reward model bottleneck — generalize to other model families (Claude, Gemini, Llama, Qwen) or to open-weight models where internal mechanisms could be studied more directly. The paper also does not test whether reasoning patterns similar to o1's emerge in other models when they are prompted with chain-of-thought or given test-time compute budgets. The paper states that it selected GPT-4o because it is "a multimodal model that integrates text, vision, and audio processing capabilities" (Section 3.2), but the choice appears motivated by availability rather than representativeness.

The consequence. The paper's comparative findings may be specific to GPT-4o's particular strengths and weaknesses. GPT-4o might be unusually bad at self-refinement (explaining Self-Refine's poor performance) while other model families might benefit more from iterative revision. GPT-4o's output diversity may saturate at a particular NN (Figure 5 shows plateau at N=8N=8), but a model with higher sampling temperature or different training might scale further. The reward model bottleneck (Figure 4: human selection reaches 33% vs. GPT-4o's ~13.5% on HotpotQA) may be specific to GPT-4o's output distribution — a model with more calibrated confidence might produce outputs that are easier for automated reward models to evaluate. More fundamentally, if o1 is a substantially different underlying model from GPT-4o (different architecture, different scale, different training data), then comparing o1's accuracy against GPT-4o + test-time compute confounds the effect of model capability with the effect of reasoning mechanism. o1's 5× improvement on AIME (62% vs. 12.22%) might partly reflect a stronger base model rather than reasoning-specific enhancements. The paper has no way to separate these, since o1's specifications are undisclosed.

What evidence exists in the paper. The search space scaling experiment (Figure 5) does evaluate BoN with three different backbones (GPT-4o, Qwen2.5-72B, Llama3-70B) on HotpotQA, finding that all three plateau at similar NN values. This provides some evidence that the plateau effect generalizes across model families for BoN specifically. However, this cross-model test is conducted only for BoN and only on HotpotQA — we don't know whether Step-wise BoN's failure on code/math tasks is GPT-4o-specific, or whether Agent Workflow's strong performance on Collie would hold with a different backbone. The reasoning pattern analysis is conducted on o1 only, with no comparison to patterns in other models' chain-of-thought outputs. The paper also evaluates o1-preview and o1-mini separately (Table 1), revealing substantial performance differences between the two variants (e.g., USACO: o1-preview 44.60% vs. o1-mini 12.23%), which demonstrates that o1 is not a monolithic entity — findings about one variant may not apply to the other.

Mitigation status. The paper partially addresses this for BoN scaling behavior (Figure 5 tests 3 backbones) but does not extend the multi-backbone analysis to other test-time methods, does not test whether reasoning patterns generalize beyond o1, and does not control for the possibility that o1's base model differs substantially from GPT-4o. No future work is proposed on cross-model replication. The limitation is partially addressed for one specific finding (BoN plateau) and unaddressed for all other claims.

6.5 The Reasoning Pattern Taxonomy Has Unknown Reliability and No Demonstrated Predictive Power

The assumption or constraint. The six-category reasoning pattern taxonomy (SA, MR, DC, SR, CI, EC) is developed through manual inspection of 20–30 o1 reasoning traces per benchmark (Section 4.2). The paper does not report: (a) whether multiple annotators independently categorized the same traces and achieved agreement, (b) whether the category definitions are sufficiently precise to yield consistent annotations across different annotators or different samples, (c) whether annotators were blind to the research hypotheses when categorizing, or (d) whether the patterns can be reliably identified by someone not involved in developing the taxonomy. The pattern definitions in Section 4.2 are brief (1–2 sentences each) and leave substantial room for interpretation — for example, the boundary between "Systematic Analysis" (analyzing inputs, outputs, constraints before solving) and "Divide and Conquer" (breaking a problem into subproblems) could be ambiguous when o1 both analyzes the problem structure and decomposes it in the same reasoning segment.

The consequence. The paper's core qualitative finding — that DC and SR are o1's most frequent and important reasoning patterns — depends on the reliability of the annotation. If two independent annotators would assign different pattern labels to the same reasoning traces, the frequency counts in Figures 1 and 2 are noisy and the ranking of patterns by frequency could change substantially. More importantly, frequency does not establish importance: DC might appear frequently because complex problems inherently require decomposition-like language, not because DC causes successful solutions. The paper does not report pattern frequencies separately for correct vs. incorrect o1 outputs. If DC appears equally often in traces that lead to wrong answers vs. right answers, its role as "key to o1's success" is unsupported. Without correctness-conditioned analysis, the pattern taxonomy is a descriptive catalog of what o1 says rather than a diagnostic of what makes o1 effective. A developer trying to replicate o1's capabilities would not know whether to invest in eliciting DC-like behavior (if it's causal) or to ignore surface-level patterns and focus on other mechanisms (if they're epiphenomenal).

What evidence exists in the paper. The paper provides frequency counts (Figures 1 and 2) and qualitative case studies (Figures 7, 9, 10, 11) illustrating how patterns manifest in specific examples. The case studies are annotated with the paper's interpretation of o1's reasoning structure (e.g., Figure 7 shows o1 "Navigating the evolution" for HotpotQA, Figure 9 shows a structured approach of "Identify Key Concepts" → "Analyze Constraints" → "Apply Mathematical Formulas" → "Construct Logical Reasoning" for AIME). These illustrations demonstrate that the patterns can be identified in individual traces, but do not demonstrate that they can be identified reliably across traces by independent annotators. The paper does not report inter-annotator agreement, correctness-conditioned pattern frequencies, or any quantitative validation of the taxonomy.

Mitigation status. The paper does not address this limitation. It does not discuss annotation methodology beyond stating the sample size, does not acknowledge the subjectivity inherent in the pattern definitions, and does not propose validation procedures. The taxonomy is presented as a finding rather than as a preliminary framework requiring validation. No future work is proposed on validating or refining the pattern categories. The limitation is not addressed, and the reasoning pattern analysis should be treated as hypothesis-generating qualitative observation rather than as a validated diagnostic instrument.

6.6 The Evaluation Scope Excludes Open-Ended Generation, Real-World Applications, and Practical Deployment Constraints

The assumption or constraint. All four benchmarks require closed-form answers with objective correctness criteria: HotpotQA answers are text spans extractable from provided documents; AIME answers are integers between 0 and 999; USACO solutions are programs that pass test cases; Collie outputs are paragraphs that satisfy explicitly enumerated constraints (word counts, forbidden words, sentence counts). The evaluation is entirely accuracy-based — did the model produce the correct answer or satisfy the constraints? This excludes a wide range of practically important reasoning tasks: open-ended analysis (e.g., writing a legal brief, analyzing a business case), creative problem-solving with multiple valid solutions, tasks where correctness is subjective or multi-dimensional (e.g., "write a good essay"), and interactive tasks requiring multi-turn reasoning with a user or environment. The paper also does not evaluate any deployment-relevant metrics: latency (how long does o1 take to respond compared to GPT-4o?), cost (what is the API pricing for o1's reasoning tokens vs. GPT-4o's output tokens?), robustness (does o1's performance degrade under input perturbations, adversarial examples, or distribution shift?), or failure modes (when o1 gets a problem wrong, does it fail gracefully with an identifiable error, or does it produce confident-sounding incorrect reasoning?).

The consequence. A practitioner deciding whether to use o1 for a production reasoning task cannot extrapolate from this paper's results unless their task closely resembles the evaluated benchmarks — objective-answer math, coding, and constrained QA problems. For a task like "analyze this contract for risky clauses and suggest revisions," there is no evidence about whether o1's reasoning patterns (DC, SR, etc.) transfer, whether o1 outperforms Agent Workflow, or whether the reward model bottleneck identified for BoN applies. The paper's claim that o1 "demonstrates exceptional improvements in areas such as mathematics and coding" (Section 1) is supported by the AIME and USACO results, but the paper evaluates no tasks outside these domains, so the claim's scope is precisely the domains tested. Additionally, the absence of latency and cost data means that even for the evaluated tasks, a practitioner cannot make deployment decisions. If o1-mini achieves 62% on AIME but takes 30 seconds and costs 0.50perquerywhileGPT−4o+BoN(0.50 per query while GPT-4o + BoN (N=8)achieves13.33) achieves 13.33% in 2 seconds and costs 0.02, the choice depends on the application's accuracy-latency-cost tradeoff — and the paper provides no data to inform this tradeoff.

What evidence exists in the paper. The paper's evaluation is entirely accuracy-focused, on four benchmarks spanning three domains. Figure 3 reports o1's reasoning token counts (showing that USACO and AIME elicit longer reasoning than HotpotQA and Collie) but does not translate token counts into latency or cost. The paper does not discuss failure modes qualitatively beyond noting that Self-Refine produces format deviations on Collie (Section 4.1) and that Step-wise BoN suffers from long-context instruction-following degradation (Section 4.3). There is no analysis of how o1 fails when it produces incorrect answers — whether it makes arithmetic errors, logical leaps, assumption violations, or something else — which would be valuable for understanding the boundaries of its capability.

Mitigation status. The paper does not address these scope limitations. It does not discuss the restricted nature of the benchmarks, does not propose evaluation on open-ended or subjective reasoning tasks, does not measure latency or cost, and does not analyze failure modes. The Conclusion (Section 6) states that the paper "hope[s] our study on the reasoning patterns of o1 can guide developers and researchers," but does not acknowledge that this guidance is currently limited to objective-answer tasks in math, coding, and commonsense QA. No future work is proposed on broadening the evaluation scope. The limitation is not addressed.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not propose a new method or a new model — it provides a diagnostic lens and a behavioral taxonomy for understanding what reasoning-enhanced models like o1 actually do when they think. This reframes the conversation around test-time compute from "how much?" to "what kind?", and from performance benchmarking to mechanism characterization. The shift is not a paradigm overturn but a reorientation of analytical priorities: rather than treating reasoning models as black-box score producers to be ranked on leaderboards, the paper demonstrates that it is possible — and productive — to open the box partially by reading the thinking traces, categorizing the reasoning moves, and comparing the resulting behavioral profile against known test-time strategies.

What changes concretely. Before this paper, the dominant question about o1 was "how well does it perform?" — answered by benchmark scores. This paper adds a second question: "what reasoning patterns does it deploy, and how do they vary by task?" The answer — that o1 uses Divide and Conquer and Self-Refinement most frequently overall, but shifts to Context Identification and Emphasizing Constraints for commonsense tasks, and to Method Reuse and Systematic Analysis for coding — provides a vocabulary for describing reasoning behavior that did not previously exist in a systematic form. This vocabulary enables researchers to ask sharper questions: not "is my model as good as o1?" but "does my model exhibit DC and SR patterns on math tasks, and if not, what would it take to elicit them?"

The paper also shifts the burden of evidence for test-time compute claims. The finding that BoN plateaus at modest NN (around 8, Figure 5) and that the reward model bottleneck can be severe (human selection more than doubles automated BoN accuracy on HotpotQA, Figure 4) establishes that naive scaling of parallel sampling is not a viable path to o1-level reasoning. Any future method claiming test-time compute improvements must now demonstrate that it addresses either the generation bottleneck (producing more diverse, higher-quality candidates) or the selection bottleneck (verifying candidates more accurately), or that it operates through a qualitatively different mechanism (like learned internal planning). The paper does not close any research directions, but it makes the "just scale up BoN" argument much harder to sustain by providing clear empirical saturation curves.

The work also reconciles conflicting intuitions about the role of self-refinement. The literature was split between papers showing self-critique helps (Madaan et al., 2024) and those showing it doesn't (Huang et al., 2023). This paper's finding that the Self-Refine method (external iterative prompting) provides essentially no benefit on these benchmarks, while Self-Refinement appears as one of o1's most frequent internal reasoning patterns (SR, Figure 2), suggests a resolution: self-refinement works when it is a learned internal capability, not when it is prompted externally. This reframes the self-critique debate from "does it work?" to "how do we train models to do it internally rather than prompting them to simulate it?" — a more productive question that the paper's pattern taxonomy helps operationalize.

What becomes less attractive as a research direction. The paper's demonstration that Step-wise BoN degrades on complex tasks due to long-context instruction-following failures (Table 2, Section 4.3) and that Self-Refine provides negligible gains, combined with the BoN plateau at N=8N = 8, collectively suggest that incrementally improving existing external test-time scaffolding is unlikely to close the gap with o1. The remaining performance delta between Agent Workflow (the strongest baseline at 24.7% overall) and o1-mini (35.8% overall) is large, and the fact that even human-level verification (Figure 4) cannot close the gap on HotpotQA implies that better selection alone is insufficient — the generation process itself must change. This redirects research attention away from search-over-outputs methods and toward methods that modify the model's internal generation process: training for learned reasoning strategies, reinforcement learning from reasoning traces, or architectures that explicitly support planning before generation.

What becomes more attractive. The paper's reasoning pattern taxonomy makes training data engineering for reasoning a more concrete and testable research program. If DC and SR are the most frequent patterns in o1's reasoning (Figure 2), then constructing training data that demonstrates these patterns — problems solved via explicit decomposition with self-checks at each stage — becomes a directly motivated intervention. The task-dependence of patterns (Figure 1: CI for QA, EC for constrained generation, DC/MR for math and coding) further suggests that domain-adaptive reasoning training is more promising than generic "think step by step" prompting: train the model to deploy different reasoning strategies for different problem types, as o1 appears to do. The paper provides both the behavioral target (what patterns to elicit) and the diagnostic tool (the pattern taxonomy) to evaluate whether training interventions actually change reasoning behavior in the intended direction.

Follow-Up Research This Work Enables

Training a model to explicitly exhibit o1's six reasoning patterns and measuring the causal effect on accuracy. The paper identifies what o1 does but cannot establish causation. A direct follow-up would fine-tune a base model (e.g., Llama-3-70B or Qwen2.5-72B) on reasoning traces annotated with the six pattern categories — where each training example demonstrates one or more patterns in sequence — and then measure both (a) whether the fine-tuned model's reasoning traces show increased pattern frequency (a manipulation check using the paper's annotation methodology) and (b) whether accuracy on the filtered benchmarks improves relative to a control fine-tuned on unannotated reasoning traces. The key measurement is the within-task pattern-accuracy correlation: does increasing DC frequency on AIME actually improve AIME accuracy? This transforms the paper's observational finding into a causal test.

Correctness-conditioned pattern analysis on a larger sample with inter-annotator agreement. The paper's pattern frequency analysis (Section 4.2, Figures 1 and 2) does not separate correct from incorrect traces and uses only 20–30 samples per benchmark with no reliability metrics. A direct replication-and-extension would annotate all ~729 filtered questions (or a substantially larger random sample of ~200) with two or more independent annotators, compute Cohen's kappa for each pattern category, and then compare pattern frequency distributions for correct vs. incorrect o1 outputs within each benchmark. This addresses the paper's most significant methodological gap and would either validate or refute the claim that DC and SR are "key to o1's success." If patterns appear with equal frequency in correct and incorrect traces, the paper's central interpretive claim collapses; if patterns are significantly enriched in correct traces, the claim is strengthened and the enriched patterns become high-priority targets for training data construction.

Combining Agent Workflow with BoN to test whether hybrid test-time methods close the gap with o1. The paper evaluates test-time methods independently but never combines them. Agent Workflow (24.7% overall) and BoN (N=4N=4, 17.7% overall) address different bottlenecks — Agent Workflow provides structured decomposition, BoN provides diversity through parallel sampling. A natural experiment: run Agent Workflow with its domain-specific prompts, but at each agent decision point, generate NN candidate next actions and use a reward model (GPT-4o as judge, or a trained verifier) to select the best one, then continue. This is essentially "Agent Workflow + Step-wise BoN at the action level." Measuring the combined accuracy on the filtered benchmarks and comparing to o1 would establish an empirical upper bound for what external test-time scaffolding can achieve with GPT-4o as the backbone. If the combination approaches o1's performance (e.g., reaching 10%+ improvement over Agent Workflow alone), then o1's advantage is primarily about orchestrating known techniques; if the gap remains large, the "qualitatively different mechanism" hypothesis is strengthened.

Benchmarking o1 against test-time methods on the discarded (easy) samples to test difficulty-dependence of reasoning benefit. The paper's difficulty filter removes samples that ≥2 of 4 judge models answer correctly, creating a hard-problem benchmark. But o1's reasoning patterns may provide little or no benefit on easier problems — or, as Snell et al. (2024) found for test-time compute more generally, the benefit may be largest on medium-difficulty problems and diminish on both very easy and very hard problems. A direct extension: evaluate o1-preview, o1-mini, GPT-4o Direct, and Agent Workflow on the discarded (easy) samples from HotpotQA and Collie. If o1's advantage over GPT-4o is small or zero on the easy subset, this establishes that o1's reasoning capability selectively activates for or selectively benefits hard problems — a boundary condition with practical implications for routing (use cheaper models for easy queries). If o1 substantially outperforms on easy problems too, that suggests its reasoning is uniformly beneficial and the difficulty filter was unnecessary for demonstrating o1's advantage.

Measuring whether reasoning pattern distributions change with o1's sampling temperature or system prompt. The paper treats o1 as a fixed black box, but if the o1 API exposes any controllable parameters (temperature, top-p, system prompt), a simple experiment would vary these and measure how the pattern frequency distribution shifts. For example, if lowering temperature increases Method Reuse frequency (less exploration, more reliance on known patterns) while raising temperature increases Divide and Conquer frequency (more decomposition into novel sub-problems), this would provide evidence that the patterns reflect controllable generation parameters rather than fixed model properties. Even without API access, comparing pattern distributions between o1-preview and o1-mini (already benchmarked in Table 1 but not analyzed for pattern differences) would test whether the variant difference in accuracy (e.g., o1-mini 62% vs. o1-preview 44% on AIME) corresponds to differences in reasoning strategy.

Extending the pattern taxonomy to open-weight models fine-tuned on reasoning traces. The paper's taxonomy was developed on o1, which may have unique properties. A replication-and-extension would apply the same annotation methodology to the chain-of-thought outputs of other models — GPT-4o with standard CoT prompting, Claude 3.5 with extended thinking (if available), and open-weight models like DeepSeek or Llama-3 fine-tuned on reasoning data. This tests whether the six patterns are universal reasoning strategies that any sufficiently capable model can exhibit, or are specific to o1's training and architecture. If open-weight models fine-tuned on math reasoning data spontaneously exhibit DC and MR patterns (even without explicit pattern-labeled training data), that suggests these patterns emerge naturally from reasoning training; if they don't, it suggests o1's training involved something more specific. This would inform whether the open-source community should focus on replicating o1's training recipe or on engineering explicit pattern elicitation.

Practical Applications and Downstream Use Cases

Cost-efficient routing in production reasoning systems. The paper shows that o1's advantage is largest on hard problems (the filtered benchmarks), while Figure 5 shows that BoN with a strong backbone (Qwen2.5-72B) achieves ~40% on filtered HotpotQA — potentially competitive with o1 on a cost-adjusted basis for medium-difficulty problems. A production system handling a stream of reasoning queries could implement a two-tier routing architecture: use a fast, cheap model (GPT-4o or Qwen2.5-72B) with a modest test-time compute budget (BoN N=4N=4, or Agent Workflow with domain prompts) as the default handler, and escalate only the queries where the initial model's confidence or verifier score is low to the more expensive o1 API. The paper's difficulty filtering methodology (using multiple judge models to estimate problem hardness) provides a blueprint for the escalation criterion. The potential cost savings are substantial: if o1 API pricing is ~10× higher than GPT-4o (a reasonable estimate for reasoning-heavy models), and 60–70% of queries can be handled adequately by the cheaper tier, the blended cost per query could be 3–4× lower than routing everything to o1, while maintaining most of the accuracy gain on the queries that genuinely need it.

Training data generation for reasoning-specific fine-tuning. The paper identifies that o1's reasoning traces exhibit structured patterns (DC, SR, SA, MR) that are absent from standard chain-of-thought outputs. For teams building open-weight reasoning models, this suggests a concrete data generation pipeline: (1) collect a diverse set of hard reasoning problems (math, code, QA); (2) use o1 (via API) to generate reasoning traces for these problems; (3) apply the paper's annotation methodology (or an automated classifier trained on the paper's annotated samples) to label trace segments with pattern categories; (4) fine-tune a base model on these structured traces, potentially with a loss that explicitly encourages pattern diversity across problem types (matching the task-pattern alignment in Figure 1). The paper's finding that DC and SR are o1's most frequent patterns provides a prioritization: training data should disproportionately feature examples where decomposition is followed by explicit self-checking. The 4× improvement on AIME from GPT-4o (12.2%) to o1-mini (62.0%) in Table 1 provides a quantitative target for what such fine-tuning might achieve — even capturing half that gain would be transformative for open-weight models.

Difficulty-aware evaluation benchmark construction. The paper's filtering methodology — using a panel of strong models to identify hard problems — is directly reusable for constructing evaluation suites that remain discriminating as models improve. Rather than waiting for benchmarks to saturate and then scrambling to create new ones, evaluation designers can periodically re-filter existing benchmarks using the latest strong models as judges, maintaining a "frontier-hard" subset that always stresses the best available systems. The paper demonstrates this on HotpotQA and Collie (Figure 8), but the approach generalizes: any benchmark with objective correctness criteria can be filtered through a panel of current models, with the threshold (e.g., "keep samples where ≥K of M judge models fail") tuned to achieve desired difficulty. This is immediately applicable to benchmarks like MATH, GSM8K, HumanEval, and MMLU, producing "MATH-hard," "HumanEval-hard," etc. that remain challenging as base model capabilities improve.

When to Prefer This Approach

The paper does not propose a new method or system against which alternatives should be weighed — it is a comparative analysis and behavioral characterization. The paper's contribution is in providing evidence about which approaches work under what conditions, not in introducing a deployable method. The relevant decision guidance is therefore when to use o1 vs. test-time compute methods, based on the paper's findings:

  • Prefer o1 (or future reasoning-enhanced models) when: (1) The task involves complex multi-step reasoning in math, code, or constrained generation where the paper shows o1's advantage is largest (AIME: o1-mini 62.0% vs. GPT-4o 12.2%; USACO: o1-preview 44.6% vs. GPT-4o 5.0%); (2) The cost of an incorrect answer is high relative to the inference cost, justifying the presumably more expensive o1 API; (3) The problem distribution is skewed toward the "hard" subset that the paper's filter identifies — problems where multiple strong standard models fail.

  • Prefer Agent Workflow with GPT-4o (or a strong backbone) when: (1) The task is commonsense reasoning or constrained generation, where Agent Workflow achieves 46.1% on Collie (approaching o1-mini's 53.5%) and 15.0% on HotpotQA (exceeding o1-preview's 14.6%); (2) Latency or cost constraints preclude o1 usage but allow multi-step agent execution; (3) The domain has clear procedural workflows that can be encoded in system prompts, as the paper's domain-specific agent prompts demonstrate.

  • Prefer BoN with a strong backbone and careful reward model selection when: (1) The task allows parallel sampling and has an objective verifiability criterion (so the reward model bottleneck is less severe than on HotpotQA, Figure 4); (2) The backbone model already has non-trivial pass@1 (since BoN plateaus at N=8N = 8 and cannot create capability that isn't already in the distribution); (3) Simplicity of implementation is prioritized over maximizing accuracy — BoN requires only repeated API calls and a scoring function, with no agent framework integration.

  • Avoid Self-Refine and Step-wise BoN (in their current form) for complex reasoning tasks — the paper's evidence shows Self-Refine provides negligible or negative improvement (5.6% overall vs. GPT-4o's 18.4%), and Step-wise BoN degrades sharply on tasks with complex output formatting or reasoning requirements (0.0% on USACO, 5.3–19.6% on Collie). These methods' failure modes (format drift during refinement, long-context instruction-following degradation) are structural and not resolved by increasing the iteration count or NN.