Competitive Programming with Large Reasoning Models

🎯 Pitch

Scaling up general-purpose reinforcement learning allows an AI to invent its own sophisticated test-time reasoning tricks—like writing brute-force checks to validate its code—and trounce hand-engineered Olympiad strategies to achieve elite human-level coding performance. After witnessing o3’s self-taught validation behaviors, the need for the painstakingly designed domain-specific pipelines that defined earlier systems simply disappears.

1. Executive Summary

This paper studies how reinforcement learning applied to large language models improves performance on complex coding and reasoning tasks, evaluating OpenAI's o1, o1-ioi, and early o3 checkpoints on the CodeForces competitive programming platform and the 2024 International Olympiad in Informatics (IOI). The work compares two fundamentally different approaches to test-time reasoning: hand-engineered test-time strategies (clustering- and reranking-based selection from 10,000 candidate solutions per subtask, as used in o1-ioi) versus learned test-time reasoning strategies that emerge naturally from end-to-end RL (o3 writing brute-force solutions to cross-validate its optimized implementations without human-crafted pipelines). The general-purpose o3 model achieves a CodeForces rating of 2724 (99.8th percentile) and scores 395.64 points at IOI 2024 with only 50 submissions per problem—outperforming the hand-crafted o1-ioi system that scored 213 points under the same constraints, and establishing that scaling general-purpose reinforcement learning surpasses domain-specific test-time heuristics. This substitution holds robustly for competitive programming benchmarks, but the paper demonstrates through software engineering evaluations (HackerRank Astra at 63.92% pass@1 for o1, SWE-bench Verified at 71.7% for o3) that these reasoning gains transfer to real-world coding tasks only when the model has been trained with sufficient RL compute to independently develop sophisticated verification behaviors rather than relying on externally imposed test-time pipelines.

2. Context and Motivation

The Core Problem: Hand-Engineered vs. Learned Test-Time Reasoning in Competitive Programming

The central question this paper investigates is: when AI systems tackle complex reasoning tasks like competitive programming, should we rely on hand-crafted, domain-specific test-time strategies, or can general-purpose reinforcement learning produce superior reasoning behaviors that emerge naturally from the model itself?

This question has deep practical and theoretical implications. On the practical side, building competitive programming AI systems has historically required enormous human effort — teams of engineers designing specialized search algorithms, clustering pipelines, and selection heuristics tailored to the specific structure of programming competitions. AlphaCode [7] generated up to a million candidate solutions per problem and used elaborate filtering and clustering, while AlphaCode 2 [6] refined this pipeline to achieve 85th-percentile CodeForces performance. The development of these systems was as much an exercise in human engineering as in machine learning. If general-purpose RL training can produce models that autonomously develop equally sophisticated reasoning strategies, the need for such labor-intensive domain-specific engineering largely disappears.

The theoretical implication is even more fundamental: it speaks to whether reasoning strategies are better discovered through search over human-designed heuristics or through optimization over model parameters. A model that learns to write brute-force solutions, execute them, and cross-validate its optimized implementations — as o3 does (Figure 6) — is effectively discovering a test-time reasoning algorithm through gradient-based optimization rather than having that algorithm specified by a human. This connects to broader questions in AI about the relationship between search-based reasoning (exploring many explicit alternatives at inference time) and learned reasoning (internalizing the search process into model weights during training).

Why Competitive Programming Is an Ideal Testbed

The paper's choice of competitive programming as the primary evaluation domain is deliberate and well-motivated, though the paper itself does not spell out all the reasons explicitly. Let me articulate them:

Objective, verifiable correctness. Competitive programming problems have a definitive right answer — a program either passes all test cases or it does not. This provides the clean reward signal that RL requires, without the ambiguity inherent in tasks like summarization or dialogue where "goodness" is subjective. The presence of test cases also enables a natural form of self-verification: a model can write code, execute it against available test cases, observe failures, and iterate.

Multi-step reasoning under constraints. Solving a CodeForces Division 1 problem typically requires: (1) parsing a complex problem statement, (2) recognizing which algorithmic technique applies (dynamic programming, graph theory, number theory, etc.), (3) designing an algorithm with the correct time and memory complexity, (4) implementing it correctly in code, and (5) handling edge cases. Errors at any step propagate to failure. This multi-stage nature makes competitive programming an excellent stress test for chain-of-thought reasoning — the model must maintain logical coherence across potentially hundreds or thousands of tokens of internal deliberation.

Difficulty calibration by rating. CodeForces uses an Elo-based rating system that precisely ranks both human competitors and problems. The paper reports that human participants at the 89th percentile (o1's level) solve a characteristic subset of problems, while the 99.8th percentile (o3's level) solves a much harder subset. This provides a principled, continuous metric for tracking progress that generalizes beyond any single benchmark.

The substitution-of-strategies dynamic. Because IOI has a specific submission format (50 submissions per problem, subtask-based scoring) and CodeForces has its own format (pretests during the competition, full tests afterward), there is a natural experiment: do models that develop their own reasoning strategies (o3) adapt to these constraints autonomously, or do you need to hard-code knowledge of the submission format into the system (as o1-ioi does)?

The Gap in Prior Work

The paper identifies several specific limitations in existing approaches to AI for competitive programming:

1. Large-scale sampling with hand-crafted selection pipelines (the AlphaCode paradigm). Both AlphaCode [7] and AlphaCode 2 [6] achieved impressive results by generating massive numbers of candidate solutions (up to 10^6 per problem) and then filtering them through multi-stage human-designed pipelines involving clustering by output behavior, reranking by learned scoring functions, and selection based on test case performance. While effective, this approach has fundamental limitations:

The strategies are static and non-adaptive. The filtering pipeline is designed once by humans and applied uniformly to all problems, regardless of problem type, difficulty, or the model's specific strengths and weaknesses on that problem.
The strategies do not improve with training. The human-designed heuristics operate at inference time only — they cannot be refined through gradient-based optimization or RL. Any improvement to the pipeline requires human re-engineering.
The strategies are task-specific. A pipeline designed for CodeForces-style problems does not automatically transfer to IOI (with its different submission format and subtask structure) or to real-world software engineering tasks (HackerRank Astra, SWE-bench). Each new domain requires a new round of human engineering.
The strategies exploit problem structure that may not generalize. AlphaCode's reliance on clustering by test case outputs assumes that programs with identical behavior on generated test inputs are likely to have the same correctness status — an assumption that holds for many competitive programming problems but may not hold for open-ended coding tasks.

2. Early code-focused LLMs without explicit reasoning. Codex [2] and early code-LLMs [1] demonstrated that large language models can generate correct code from natural language descriptions, with performance improving log-linearly with model size. However, these models operated in a "generate once and hope" paradigm — they produced a single solution (or at best sampled a few solutions independently) without any iterative refinement, self-verification, or chain-of-thought reasoning about the problem structure. Their success on competitive programming was correspondingly limited: the paper reports that gpt-4o, a strong non-reasoning model, achieves only an 808 CodeForces rating (11th percentile), indicating it can solve only the easiest problems in Division 1 contests.

The key insight is that competitive programming — unlike simpler code generation tasks — requires debugging. A solution that is 90% correct but has an off-by-one error, a missed edge case, or a subtle algorithmic inefficiency will fail the full test suite. Without the ability to test code, observe failures, and iteratively refine, even a model with strong algorithmic knowledge will underperform dramatically.

3. The limitation of prompting-based reasoning. Prior to o1, the dominant approach to improving LLM reasoning was chain-of-thought prompting [16] — encouraging the model to "think step by step" by providing examples of step-by-step reasoning in the prompt. While this produces improvement on many reasoning benchmarks, it has important limitations:

The quality of reasoning is bounded by what the model's pretraining and supervised fine-tuning have already internalized. Prompting does not teach the model new reasoning strategies — it only elicits existing capabilities.
The reasoning is generated in a single forward pass, without the opportunity for backtracking, trying alternative approaches, or verifying intermediate results through external tools.
The model cannot learn from its own reasoning errors. If it produces a flawed chain of thought, there is no training signal to correct that behavior — at best, the flawed output can be filtered out by an external verifier.

4. The gap between RL-trained reasoning models and hand-engineered systems. OpenAI's o1 [4, 12] represented a breakthrough: an LLM trained with RL to produce extended internal chains of thought, identify and correct its own errors, and use external tools (code execution) to verify its outputs. The o1-preview achieved a CodeForces rating of 1258 (62nd percentile), demonstrating that RL-trained reasoning dramatically outperforms prompting-based approaches (gpt-4o's 808). However, o1 itself achieved 1673 (89th percentile), and the paper does not specify whether o1 used any hand-crafted test-time strategies beyond its learned chain-of-thought. This leaves open the question: how much further can domain-specific engineering push performance beyond what general-purpose RL achieves?

The o1-ioi system was specifically designed to answer this question. By taking the o1 checkpoint, continuing RL training focused specifically on coding tasks, and adding the AlphaCode-style test-time pipeline (10,000 solutions per subtask, clustering, reranking, round-robin submission), o1-ioi reached 2214 (98th percentile) on CodeForces. This represents a gain of 541 rating points from the hand-crafted strategy alone — a substantial improvement that demonstrates the power of domain-specific engineering.

But the deeper question remained: was this gain necessary, or could further RL training achieve the same result without the hand-crafted pipeline? The development of o3, trained with "significantly greater compute resources" (Section 4.1), provided the answer: o3 reached 2724 (99.8th percentile) — a 510-point gain over o1-ioi without any of the hand-crafted test-time strategies. This is the paper's central empirical finding.

Conflicting Methodologies in the Literature

The paper implicitly addresses a methodological tension in the AI-for-reasoning literature:

The search-over-heuristics paradigm (AlphaCode, AlphaCode 2) treats test-time compute as a resource to be spent on generating many candidate solutions and filtering them through human-designed pipelines. The model generates raw candidates; the intelligence lies in the selection process. This approach separates "generation quality" from "selection quality" and optimizes the selection pipeline independently.

The learned-reasoning paradigm (o1, DeepSeek-R1 [3], Kimi k1.5 [15]) treats test-time compute as a resource to be spent on thinking — producing longer, more careful chains of thought that include self-critique, verification, and revision. The model internalizes the selection and refinement process into its generation. There is no separate selection pipeline; the model itself decides which outputs to pursue and which to discard.

The paper's contribution is not just an empirical comparison showing that learned reasoning wins, but a demonstration of mechanism: Figure 6 shows o3 spontaneously developing a test-time strategy that partially replicates the hand-crafted pipeline. Specifically, for problems where verification is nontrivial, o3 writes simple brute-force solutions — trading efficiency for correctness — and then cross-checks the outputs against its optimized algorithmic implementations. This is exactly the kind of behavior that a human engineer might design into a pipeline: generate a slow-but-correct reference implementation, then use it to validate the fast-but-complex solution. The crucial difference is that o3 discovered this strategy through RL training rather than having it explicitly programmed.

How This Paper Positions Itself

The paper positions itself at the convergence of several research threads:

1. The scaling of reinforcement learning for reasoning. Recent work from DeepSeek-R1 [3] and Kimi k1.5 [15] independently demonstrated that RL training with chain-of-thought reasoning dramatically improves performance on mathematical and programming challenges. This paper extends that finding into the competitive programming domain, showing that the gains are not only in accuracy but in the sophistication of the reasoning strategies themselves — the model evolves from single-pass code generation to multi-step verify-and-refine loops.

2. The obsolescence of domain-specific test-time engineering. The paper's narrative arc is: o1 (general RL) → o1-ioi (RL + hand-crafted strategies, strong results) → o3 (more RL, no hand-crafted strategies, even stronger results). The hand-crafted pipeline was a stepping stone, not a destination. The paper is making the case that the research community should invest in scaling RL training compute rather than designing ever-more-elaborate test-time pipelines, because the RL approach not only matches but exceeds human-designed strategies, and it does so with models that generalize across domains (CodeForces, IOI, SWE-bench, HackerRank Astra) without per-domain customization.

3. A bridge to software engineering. Sections 5.1 and 5.2 demonstrate that the reasoning capabilities are not limited to algorithmic problem-solving. o1 achieves 63.92% pass@1 on HackerRank Astra (project-oriented, multi-file, framework-specific coding challenges) and o3 achieves 71.7% on SWE-bench Verified (real-world GitHub issue resolution). These results are presented not as the main contribution but as evidence that reasoning transfer is real — the model's ability to plan, verify, and refine its outputs generalizes beyond the narrow domain of competitive programming.

4. The "emergence" of test-time strategies through RL. Perhaps the paper's most provocative positioning claim is that complex test-time reasoning strategies "emerged naturally from end-to-end RL" (Section 1). This language of "emergence" positions the work within a broader discourse about whether sophisticated behaviors in large models are explicitly programmed (by humans designing architectures, objectives, or pipelines) or spontaneously arise from sufficiently scaled optimization of simple objectives. The paper takes a clear stance: scaling RL with a correctness reward signal is sufficient to produce behaviors (brute-force verification, iterative refinement, tool use) that previously required explicit human design.

Where This Paper Leaves Gaps

Even as it answers the question of hand-crafted vs. learned strategies, the paper opens new questions:

What is the mechanism of transfer? How does RL training on code correctness generalize to the meta-cognitive skill of "write a brute-force solution to verify my optimized solution"? The paper demonstrates that it happens but does not investigate how — what in the training signal, model architecture, or chain-of-thought format enables this generalization.
Where are the limits? The paper does not report the RL training compute budget for o3 (it simply says "significantly greater" than o1), nor does it present scaling curves showing how performance improves with RL compute. Without these, it is impossible to assess whether further scaling would continue to yield gains or whether performance is asymptoting.
What is the cost-benefit ratio? Hand-crafted strategies like o1-ioi's require human engineering effort but are computationally inexpensive at inference (relative to model size). Learned strategies like o3's require enormous RL training compute but are "free" at inference (the strategies are baked into the model's weights). The paper does not quantify the total compute cost of either approach, making it impossible to determine which is more cost-effective for a given performance target.

3. Technical Approach

3.1 Reader Orientation

This paper is primarily an empirical comparison of two fundamentally different approaches to test-time reasoning in competitive programming and software engineering: hand-engineered, domain-specific inference strategies (AlphaCode-style pipelines with clustering and reranking) versus learned reasoning behaviors that emerge from scaling general-purpose reinforcement learning. The core idea is that while human-crafted test-time strategies can yield substantial improvements over base models, further scaling of RL training produces models that autonomously develop equally or more sophisticated reasoning strategies—writing brute-force validators, cross-checking outputs, and iteratively debugging—without requiring any domain-specific engineering at inference time.

The problem being solved is: given a difficult coding or reasoning task, how should we allocate computational and engineering effort to maximize correctness—by designing specialized search and selection algorithms that operate at test time, or by investing those resources into training a model that internalizes effective reasoning strategies into its weights? The "shape" of the solution is a progression through three systems (o1 → o1-ioi → o3) that increasingly shift the balance from test-time engineering to training-time RL, with the final system eliminating hand-crafted strategies entirely while achieving superior performance.

3.2 Big-Picture Architecture (Diagram in Words)

The paper describes three distinct systems that share a common foundation but differ in how they deploy reasoning at inference time. Here is the high-level architecture of each:

OpenAI o1 (general-purpose reasoning model):

Base LLM trained with reinforcement learning to produce extended internal chains of thought before answering.
Code execution tool that allows the model to write code, run it in a secure sandbox, and observe outputs (compilation errors, test case results).
Chain-of-thought process where the model plans solutions, writes code, executes it, observes failures, and iteratively refines—all within a single coherent chain of reasoning.
No external selection pipeline: the model's own reasoning (in its chain of thought) determines which solutions to pursue and which to discard. The final answer is the output the model converges on after its internal deliberation.

OpenAI o1-ioi (domain-specific competition system):

Fine-tuned o1 checkpoint produced by resuming RL training from o1 with additional emphasis on challenging coding tasks, C++ generation, and IOI submission format requirements.
Parallel solution generation: samples 10,000 candidate solutions per subtask from the fine-tuned model (each solution is a complete, independent program).
Test input generation and validation: prompts the model to write C++ programs that generate random test inputs satisfying subtask constraints, then writes validators to filter out invalid inputs, keeping 256 validated random test cases per subtask.
Clustering by output behavior: runs all 10,000 solutions on the 256 model-generated test inputs and groups solutions that produce identical outputs on all test cases into the same cluster. Two programs that agree on 256 random test inputs are likely implementing the same (correct or incorrect) algorithmic approach.
Learned scoring function: scores each solution based on a trained quality estimator (details not specified in the paper, but conceptually similar to AlphaCode's approach).
Reranking and selection: assigns each cluster a score defined as the average score of its member solutions minus a penalty for each previous submission attempt from that cluster. Penalty weights are tuned by random search on previous years' IOI problems via direct simulation of the submission process.
Round-robin submission strategy: submits up to 50 solutions in round-robin fashion across subtasks, starting from the hardest subtask. For each subtask, picks the top-ranked solution in the top-ranked cluster. When a subtask is solved (maximum score attained), stops sampling for that subtask. When submitting to a subtask that is a strict superset of an already-solved subtask, filters out solutions whose outputs on the solved subtask's test inputs don't match the correct outputs (the "subtask inheritance" pruning mechanism).

OpenAI o3 (scaled general-purpose reasoning model):

Base LLM trained with significantly more RL compute than o1, without any coding-specific fine-tuning or domain-specific test-time strategy design by humans.
Code execution tool (same as o1) allowing write-execute-observe-refine cycles.
Self-developed test-time strategies: the model autonomously discovers sophisticated verification behaviors during its chain of thought—most notably, for problems where verification is nontrivial, it writes simple brute-force solutions (trading efficiency for guaranteed correctness) and cross-checks their outputs against its optimized algorithmic implementations (Figure 6).
Simple selection: at evaluation time, for IOI 2024, the top 50 solutions with the highest test-time compute from 1,024 samples per problem are submitted. No subtask-specific prompts, no manual problem partitioning, no clustering or reranking pipelines—the model's own reasoning is the sole selection mechanism.
Unified problem solving: unlike o1-ioi which breaks each IOI problem into subtasks and solves them separately, o3 receives the complete problem statement in a single prompt and produces solutions that naturally cover multiple or all subtasks simultaneously.

Information flows through these systems in distinctly different patterns. In o1 and o3, information flows in a tight loop: model generates reasoning → writes code → executes code → observes results → generates more reasoning → refines code → executes again, all within one continuous chain of thought. The model's internal deliberation is the control logic. In o1-ioi, information flows through a human-designed pipeline: model generates 10,000 independent solutions → test input generators produce 256 test cases → solutions are clustered → solutions are scored → a selection algorithm picks which to submit → the IOI grader provides feedback → the submission strategy adapts for subsequent attempts.

3.3 Roadmap for the Deep Dive

First, the o1 training approach and its chain-of-thought mechanism, because this is the foundation that both subsequent systems build upon—understanding how RL-trained reasoning works in o1 is prerequisite to understanding what o1-ioi adds and what o3 scales.
Second, the o1-ioi training modifications (the "Coding RL Fine-tuning" phase), since these represent the first attempt to push beyond general-purpose reasoning into domain-specific optimization, and they establish the baseline that o3 must surpass.
Third, the o1-ioi test-time strategy in full detail—the clustering mechanism, the scoring and reranking pipeline, the submission algorithm—because this is the hand-crafted system that the paper's central claim argues is made obsolete by o3's learned strategies.
Fourth, the o3 training and inference approach, focusing on what is different from o1 and o1-ioi (the scale of RL training, the absence of hand-crafted strategies, the emergence of verification behaviors).
Fifth, the FLOPs and compute accounting—what we know and what the paper leaves unspecified—because the comparison between o1-ioi and o3 is fundamentally a comparison of different compute allocation strategies (test-time search vs. training-time RL), and understanding this tradeoff requires being explicit about where the compute went.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical comparison paper whose core technical contributions are (1) the o1-ioi hand-crafted test-time pipeline, described in sufficient detail to serve as a strong domain-specific baseline, and (2) the demonstration that o3, trained with scaled-up RL and no domain-specific inference engineering, develops qualitatively more sophisticated reasoning strategies that outperform the hand-crafted approach. The paper does not introduce novel training algorithms; rather, it applies existing RL-for-reasoning techniques at increased scale and carefully documents the behavioral consequences.

OpenAI o1: Foundation via RL-Trained Chain-of-Thought Reasoning

What o1 is. OpenAI o1 is a large language model trained with reinforcement learning to produce extended internal chains of thought before outputting a final answer. The model is not described in architectural detail in this paper (which is not primarily about o1), but its key properties are established in prior work [4, 12, 16] and referenced in Section 2.

How the reasoning process works. The model generates an internal chain of thought—a sequence of tokens that represents its step-by-step deliberation about the problem—before producing the externally visible answer. During this chain of thought, the model can engage in several reasoning behaviors that are directly enabled by the RL training:

Error identification and correction: the model spots logical flaws, off-by-one errors, or algorithmic inefficiencies in its own partial solutions and revises them before finalizing.
Task decomposition: complex problems are broken into smaller, more manageable sub-problems, each addressed sequentially.
Alternative exploration: when an initial approach fails (as evidenced by the model's own reasoning or by code execution results), the model backtracks and tries a different method.

Tool use for code execution. Crucially, o1 is trained to use external tools, specifically a secure code execution environment (referenced as the OpenAI Code Interpreter tool in Section 2). This means that during its chain of thought, o1 can:

Write a candidate solution in code.
Submit that code for execution against available test cases.
Observe the execution output—compilation errors, runtime failures, test case pass/fail results.
Incorporate that feedback into subsequent reasoning: if a test fails, the model can analyze why it failed and adjust the algorithm accordingly.

This creates a verify-and-refine loop that operates within a single chain of thought. The model is not merely generating code; it is debugging code, with the execution environment providing ground-truth feedback on correctness. Section 2 states: "By testing and refining its outputs, o1 iteratively improves its solutions over the course of a single sample."

The RL training signal. The paper does not detail o1's exact RL objective, but the system card [4] and independent work like DeepSeek-R1 [3] suggest that the model is trained to maximize correctness on reasoning tasks where correct answers can be automatically verified. For coding tasks, this means the reward signal is binary (or near-binary): does the submitted code pass all test cases? The chain of thought itself is not directly supervised—the model discovers effective reasoning strategies through RL, guided solely by the final correctness signal.

This is a crucial point for understanding the paper's contribution. The chain-of-thought behaviors that o1 exhibits (error correction, decomposition, tool use) are not explicitly programmed by providing examples of these behaviors in training data. They are learned strategies that maximize the probability of correct final answers under the constraints of the chain-of-thought format. This is why the paper frames o1 as a "reasoning model"—the reasoning process is the means by which the model achieves correctness, and the specific form that reasoning takes is discovered through optimization, not specified by humans.

The role of test-time compute. o1 uses test-time compute for two purposes that are often conflated in the literature: (1) generating longer, more careful chains of thought (which consumes more tokens per problem), and (2) potentially sampling multiple distinct chains of thought and selecting among them (a form of majority voting or best-of-N, though the paper is not explicit about whether o1 uses this). The key distinction from the AlphaCode paradigm is that in o1, the primary mechanism for using test-time compute is to think longer and more carefully, not to generate many independent solutions and filter them externally. The model's own reasoning is the filter.

Performance baseline. On CodeForces Division 1 contests from late 2023 and 2024 (problems unseen during training), o1 achieved a CodeForces rating of 1673, placing it at the 89th percentile of human competitive programmers. This is up from o1-preview's 1258 (62nd percentile) and gpt-4o's 808 (11th percentile), as shown in Figure 1. The gap between gpt-4o and o1-preview (450 rating points) represents the gain from adding RL-trained chain-of-thought reasoning to a strong base model; the gap from o1-preview to o1 (415 rating points) represents the gain from further RL training.

o1-ioi Training: Coding-Specific RL Fine-Tuning

Motivation. The transition from o1 to o1-ioi was motivated by the observation (Section 3, Figure 2) that "increasing both the amount of reinforcement learning (RL) compute and test-time inference compute consistently improved model performance" on competitive mathematics and coding tasks. Figure 2 shows scaling curves where both axes—RL training compute and test-time inference compute—independently contribute to improved performance. This observation suggested two complementary paths to better competitive programming performance: (1) continue scaling RL training, now focused specifically on coding, and (2) design more sophisticated test-time strategies to make better use of the model's improved capabilities.

The fine-tuning procedure. Section 3.1 describes the o1-ioi training in three steps:

Resume RL training from the OpenAI o1 checkpoint. This means o1-ioi is not trained from scratch—it starts from a model that already possesses general-purpose chain-of-thought reasoning capabilities and then receives additional RL training targeted at coding.
Emphasize challenging programming problems in the RL training data. While the paper does not specify the exact dataset or filtering criteria, this step tilts the training distribution away from the general mixture of math, science, and coding problems that o1 was trained on, and toward the specific distribution of algorithmic problems that appear in competitive programming. The intended effect is to strengthen the model's ability to plan, implement, and debug complex algorithmic solutions—skills that are tested heavily at high CodeForces ratings but may be less prevalent in general coding tasks.
Guide the model to produce outputs in the IOI submission format. The IOI has specific requirements for how solutions must be formatted (the #include directives, function signatures, input/output handling conventions). By including these formatting constraints in the RL training signal, the model learns to produce solutions that can be directly submitted to the IOI grading system without post-processing. This is an example of domain adaptation through fine-tuning rather than through post-hoc formatting scripts.

What changed relative to o1. The paper notes two specific improvements in o1-ioi's behavior after this fine-tuning (Section 3.1):

Improved C++ generation and runtime checks. The model became more proficient at writing C++ (the language used for IOI submissions) and at understanding and handling runtime errors that emerge during code execution.
Iterative execution and refinement of solutions. The model strengthened its ability to write code, execute it, observe failures, and iteratively improve—the verify-and-refine loop that o1 already possessed, now tuned specifically for the types of errors common in algorithmic programming (time limit exceeded, memory limit exceeded, wrong answer on specific test cases, etc.).

Performance impact without test-time strategies. After this fine-tuning but without the full hand-crafted test-time strategy, o1-ioi achieved a CodeForces rating of 1807 (93rd percentile), up from o1's 1673 (89th percentile) as shown in Figure 3. This 134-point gain represents the pure benefit of coding-specific RL fine-tuning on top of general-purpose reasoning training. When a simple post-hoc filter was applied—rejecting any solution that failed the provided public test cases—the rating rose to 2092 (96th percentile). This 285-point gain from filtering demonstrates that the model was generating correct solutions among its candidates, but that without selection, incorrect solutions were sometimes being chosen. The selection problem—picking the right solution from a set of candidates—is exactly what the full test-time strategy addresses.

o1-ioi Test-Time Strategy: The Full Hand-Crafted Pipeline

This is the most technically detailed part of the paper and represents the "hand-crafted" approach that o3 is compared against. Understanding this pipeline in detail is essential for appreciating what o3 learned to do autonomously.

Problem formulation: subtask-based decomposition. IOI problems are scored on a subtask-by-subtask basis. A typical IOI problem might have 5-8 subtasks, each with different constraints (smaller input sizes, special structural properties) and different point values. Competitors receive the maximum score they achieve across all their submissions for each subtask independently. The o1-ioi system exploits this structure by solving each subtask separately. For each IOI problem, the system creates separate versions of the problem statement for each subtask, with the information about other subtasks removed. This is done "simply by creating one version of the document for each subtask with the information about the other subtasks removed" (Section 3.2).

This decision has both advantages and disadvantages:

Advantage: the model can focus on the specific constraints of one subtask at a time, potentially finding simpler solutions that work for limited input sizes or special cases. The scoring system rewards this—you don't need a solution that handles all subtasks simultaneously.
Disadvantage: it precludes solutions that naturally cover multiple subtasks with a single algorithm. A general solution that handles all subtasks might be discoverable by a model that thinks about the full problem, but the subtask decomposition prevents the model from seeing the full context. This is precisely the limitation that o3 overcomes by receiving the complete problem statement.

Solution generation: 10,000 candidates per subtask. For each subtask, o1-ioi generates 10,000 complete, independent solutions. Each solution is a full C++ program that reads from standard input, writes to standard output, and is intended to solve that specific subtask. The 10,000 number is not arbitrary—it is inherited from AlphaCode [6], which used up to a million solutions per problem, and represents a tradeoff between coverage (sampling enough solutions that at least a few are correct) and computational cost. For o1-ioi, with 6 IOI problems and roughly 5-8 subtasks per problem, this amounts to on the order of 300,000–480,000 solutions generated across the full competition.

The paper does not specify the sampling temperature or other generation hyperparameters for o1-ioi, but the fact that the 10,000 solutions are being generated for the purpose of filtering (not all will be submitted) suggests that diversity is prioritized over per-sample correctness—high-temperature sampling to explore the space of possible algorithms.

Test input generation: 256 validated random test cases per subtask. Before solutions can be clustered by their outputs, the system needs test inputs on which to evaluate them. o1-ioi generates these test inputs programmatically:

Prompt the model to write C++ test input generators given the problem specification and subtask constraints. These are programs that, when executed, produce random test inputs that satisfy the subtask's constraints (e.g., for a graph problem, generate random graphs with the specified number of nodes and edges).
Use these generators to produce 256 random test inputs per subtask. The choice of 256 (a power of 2) is likely for implementation convenience and because the clustering step requires binary agreement (match or don't match) on all test cases.
Prompt the model to write test input validators in C++ that check whether a given test input satisfies the subtask constraints. These are separate programs from the generators.
Filter the generated test inputs: each generated test input is checked against all validators. A test input is accepted only if it passes at least 75% of the validators. The 75% threshold (rather than 100%) suggests that the validators themselves may have false-positive rejection behavior—perhaps due to edge cases in the constraint definitions—and requiring unanimity would discard too many valid inputs.

The generation-validation-filter pipeline ensures that the 256 test inputs are both diverse (randomly generated) and legitimate (satisfy the subtask constraints). This is critical because clustering solutions based on their outputs uses these test inputs as a proxy for behavioral equivalence.

Clustering by output behavior. Once the 256 test inputs are available, each of the 10,000 candidate solutions is executed on each of the 256 test inputs, and its output for each test case is recorded. For each solution, this produces a 256-element vector of outputs. Two solutions are placed in the same cluster if and only if their outputs match exactly on all 256 test cases.

The rationale for this clustering approach is: if two programs produce identical outputs on 256 diverse, randomly-generated test inputs, they are very likely implementing the same algorithmic logic—either both are correct (producing the right answer for all possible inputs), or both share the same bug (producing the same wrong answers). If a program has a bug that affects only certain edge cases, there is a reasonable chance that at least one of the 256 random inputs triggers that edge case, causing a mismatch with correct programs. However, this is not guaranteed—subtle bugs that only manifest on very specific inputs might be missed by 256 random test cases.

The clustering serves a dual purpose in the submission strategy:

Efficiency: instead of submitting individual solutions one at a time—which would waste the limited 50 submissions on multiple variants of the same algorithm—the system submits one representative from each cluster. If the representative fails, the entire cluster is penalized, making it less likely that another solution from the same cluster will be submitted.
Signal aggregation: solutions that arrive at the same answer through different implementations are separated into different clusters, providing independent evidence for that answer. If multiple distinct clusters produce the same output, that output is more likely to be correct.

Scoring and reranking. Each solution is scored based on a combination of factors (Section 3.2):

A learned scoring function that estimates solution quality. The paper does not describe this function's architecture or training procedure, but it likely follows the approach of AlphaCode [6, 7], where a separate model or head is trained to predict whether a solution will pass the full test suite given the problem statement and the solution code.
Errors on model-generated test inputs. If a solution crashes, times out, or produces an error (rather than an actual output) on any of the 256 generated test inputs, this is a strong negative signal. The paper's phrasing "errors on model-generated test inputs" suggests a penalty for each such failure.
Failing the provided public test cases. IOI problems include a small number of public test cases (typically 1-5 per subtask) that competitors can see and test against before submitting. If a solution fails any of these, it is almost certainly buggy and should receive a heavy penalty.

Each cluster is then assigned a score defined as:

$\text{cluster\_score} = \frac{1}{|C|} \sum_{s \in C} \text{score}(s) - \lambda \cdot \text{attempts}(C)$

where $C$ is the set of solutions in the cluster, $\text{score}(s)$ is the individual solution score from the learned scoring function (adjusted for test case failures), $\text{attempts}(C)$ is the number of times a solution from this cluster has already been submitted (and failed), and $\lambda$ is a penalty weight.

What it computes: the average quality of solutions in the cluster (higher is better, since the cluster likely represents a valid algorithm), minus a penalty for each previous failed submission from that cluster (to avoid repeatedly submitting the same wrong algorithm).

Why this form: simply taking the maximum score in the cluster would be vulnerable to outliers—a single highly-scored but actually buggy solution could cause the system to repeatedly submit from a cluster that consistently fails. The average provides robustness, and the penalty provides exploration: once a cluster has been tried and failed, the system is pushed to explore other clusters before returning to it. The penalty weight $\lambda$ is "tuned by random search on solutions to previous years' IOI problems, by directly simulating the submission process" (Section 3.2). This is an important detail: the parameters of the human-designed pipeline are themselves optimized through automated search, not set arbitrarily by intuition.

Submission strategy. The system submits at most 50 solutions per problem (the IOI limit for human competitors). The submission algorithm operates as follows, quoted and paraphrased from Section 3.2:

Round-robin over subtasks, starting from the hardest. The system maintains a priority order over subtasks (hardest first) and rotates through them, allocating one submission at a time. Starting from the hardest subtask means that if a solution solves a hard subtask, it likely also handles easier ones (since harder subtasks typically have looser constraints), potentially earning points on multiple subtasks simultaneously.
For each subtask, select the top-ranked solution from the top-ranked cluster. "Top-ranked" refers to the cluster scoring formula above. The highest-scoring cluster that has not been exhausted (all its solutions already submitted) donates its highest-scoring individual solution.
Cease sampling when a subtask is solved. If the IOI grader reports that the subtask has received its maximum possible score (meaning at least one submitted solution was correct), the system stops allocating further submissions to that subtask. The 50-submission budget is reallocated to the remaining unsolved subtasks.
Subtask inheritance pruning. When submitting to a subtask that is a strict superset of an already-solved subtask (meaning the harder subtask has all the constraints of the easier one plus additional ones), the system filters out any candidate solutions whose outputs on the solved subtask's test inputs do not match the correct outputs. The reasoning is: any solution that would have passed the harder subtask must also pass the easier subtask. If a candidate solution produces outputs that are known to be wrong on the easier subtask (because we've already established the correct outputs through solving the easier subtask), it cannot possibly be correct for the harder subtask. This pruning mechanism reduces the candidate pool for harder subtasks as the competition progresses and more subtasks are solved.

This submission strategy embodies several design principles that are characteristic of hand-crafted competition systems:

Budget awareness: the 50-submission limit is a hard constraint, and the strategy explicitly manages this budget across subtasks.
Information reuse: solving a simpler subtask provides information (the correct outputs on its test cases) that is used to prune candidates for harder subtasks, creating a virtuous cycle where early successes accelerate later ones.
Exploration-exploitation balance: the cluster penalty $\lambda$ trades off between trying the algorithm that looks best on paper (exploitation) and trying alternative algorithms when the best-looking one fails (exploration).
Domain knowledge embedding: the structure of IOI—subtask hierarchy, independent scoring per subtask, the 50-submission limit—is hard-coded into the strategy. The system does not learn these rules; they are given by the competition format and encoded by human engineers.

Performance impact. Figure 3 shows that the full o1-ioi test-time strategy (RL fine-tuning + clustering + reranking + round-robin submission) achieved a CodeForces rating of 2214 (98th percentile), up from 2092 (96th percentile) with only public test filtering, and 1807 (93rd percentile) with the raw model. At IOI 2024 under the official 50-submission limit, o1-ioi scored 213 points out of 600, placing it in the 49th percentile (Figure 4). As a reference point, randomly selecting 50 submissions from the 10,000 candidates would have yielded an average of 156 points, meaning the test-time strategy contributed approximately 57 additional points—a 36% improvement over random selection.

When the IOI submission limit was relaxed to 10,000 per problem (effectively allowing all generated solutions to be submitted), o1-ioi's score rose to 362.14, just above the gold medal threshold of approximately 360 points (Figure 4). This demonstrates that the model was generating gold-medal-worthy solutions; the gap between 213 (50 submissions) and 362.14 (10,000 submissions) represents the selection difficulty—picking the right 50 out of 60,000 total candidates (10,000 per subtask × 6 problems, roughly).

OpenAI o3: Scaling RL to Replace Hand-Crafted Strategies

What o3 is. o3 is described as a "later version" of the o-series models, "trained with significantly greater compute resources than o1" (Section 4.1). The paper does not provide architectural details, parameter counts, or FLOP budgets for o3 training—these are presumably in the o3 system card [13], which is cited but not summarized in this paper. For the purposes of this analysis, o3 should be understood as a scaled-up version of the o1 approach: same basic framework (LLM with RL-trained chain-of-thought reasoning and tool use for code execution), but with more training compute, likely a larger model, and possibly improved RL algorithms.

The critical absence: no hand-crafted test-time strategy. The paper is explicit and emphatic on this point (Section 4): "unlike o1-ioi or AlphaCode, o3 does not depend on coding-specific test-time strategies defined by humans. Instead, we found that complex test-time reasoning strategies emerged naturally from end-to-end RL." At evaluation time, o3 receives a simple prompt containing the problem statement, constraints, and available sample test cases. It generates its chain of thought, writes code, executes it, and produces a final answer—all without any external clustering, reranking, subtask decomposition, or submission schedule designed by humans.

The sampling and selection approach for IOI 2024. For the IOI 2024 retrospective evaluation (Section 4.2), a slightly different protocol was used than for CodeForces:

1,024 solutions sampled per problem—significantly fewer than o1-ioi's 10,000 per subtask (which, aggregated across subtasks, would be roughly 50,000–80,000 per problem). The reduction from 10,000 to 1,024 (approximately a factor of 10–80×) suggests that o3's per-solution quality is substantially higher, requiring less brute-force coverage.
Single prompt containing the original problem statement—no subtask decomposition. This means o3 sees the full problem, with all its constraints and subtask structure, in one prompt. The model must determine for itself which subtasks it can solve, whether to aim for partial-credit solutions on hard subtasks, and how to structure its solution to maximize score.
Top 50 solutions selected by test-time compute—the paper states "we selected the top 50 solutions with the highest test-time compute from 1,024 samples per problem." This phrasing "highest test-time compute" likely means the solutions where the model spent the most tokens in its chain of thought—the most deliberate, carefully-reasoned solutions—rather than the solutions that scored highest on an external scoring function. The selection criterion is based on the model's own investment of effort, not on a human-designed quality heuristic.

The emergence of verification strategies. The most striking qualitative result in the paper is Figure 6, which shows a segment of o3's chain of thought for a CodeForces problem. In this excerpt, o3:

Writes an optimized algorithmic solution to the problem.
Recognizes that verifying this solution against only the provided public test cases is insufficient because "verification is nontrivial."
Independently decides to write a brute-force solution—a simpler, slower algorithm that is guaranteed to be correct for small inputs because it exhaustively enumerates possibilities.
Executes both the optimized solution and the brute-force solution on the same test inputs.
Cross-checks their outputs—if they disagree, the optimized solution has a bug.
Uses this cross-validation to catch and fix errors in the optimized solution.

This is a sophisticated debugging strategy that a competitive programmer would employ: when you're unsure about your optimized algorithm, write a slow-but-correct reference implementation and use it as a test oracle. The crucial point is that no human told o3 to do this. The behavior was not specified in a system prompt, encoded in a test-time pipeline, or provided as an example in a few-shot prompt. It emerged from RL training as a strategy that maximizes the probability of producing correct final answers.

The paper frames this discovery explicitly (Section 4.2): "the sophisticated test-time techniques that emerged during o3 training, such as generating brute-force solutions to verify outputs, served as a more than adequate replacement and eliminated the need for the hand-engineered clustering and selection pipelines required by o1-ioi."

What o3's behavior implies about the RL training. Although the paper does not specify o3's training methodology in detail, the emergence of verification behaviors like brute-force cross-checking suggests several properties of the RL process:

The reward signal must reward correctness strongly enough that verification is worth the computational cost. Writing a brute-force solution, executing it, and comparing outputs consumes tokens and time. The RL training must have created a selection pressure where this cost is repaid by increased probability of being correct—meaning the model has learned that verification improves its reward in expectation.
The chain-of-thought format must be long enough to accommodate multi-step verification. o3's verification strategy requires: reasoning about the need for verification, implementing a brute-force solution, executing both programs, comparing outputs, diagnosing discrepancies, and fixing bugs. This is a much longer chain of thought than simply writing a single solution and hoping it's correct. The RL training must support chains of thought of sufficient length to accommodate this complexity.
The model must have access to code execution during training, not just at evaluation time. The verification strategy depends on being able to run code and observe outputs. If code execution were only available at test time, the model could not learn to use it strategically—it would have to discover effective tool-use behaviors from scratch at inference, which is unlikely.
The training distribution must include problems where verification is useful. If all training problems were trivially verifiable by running against provided test cases, there would be no pressure to develop alternative verification strategies. The emergence of brute-force cross-checking suggests that the training distribution included problems where:
1. The provided test cases were insufficient to catch bugs (verification was genuinely "nontrivial").
2. A brute-force solution was feasible (small enough input constraints that an exhaustive algorithm would terminate).
3. The optimized solution was error-prone (bugs were common enough that verification paid off).

This diversity in training problems is what allows the model to learn conditional strategies: "when verification is easy (test cases suffice), just test; when verification is hard but brute-force is feasible, write a brute-force checker."

Performance on CodeForces. Figure 5 shows o3 achieving a CodeForces rating of 2724 (99.8th percentile), up from o1-ioi's 2214 (98th percentile) with the full hand-crafted pipeline. This 510-point gain is remarkable because it represents an improvement on top of a system that was already heavily optimized for competitive programming. The 99.8th percentile means that o3 would rank among the top 200 active CodeForces competitors worldwide (Figure 10).

Table 1 in Appendix B provides a per-problem breakdown of o3's CodeForces performance across 12 Division 1 contests. Several patterns are notable:

Near-perfect pass@1 on easy problems: For problems rated 800–1400 (the easiest third of Division 1 problems), o3's pass@1 (correct on first attempt, before any ranking or selection) is essentially 1.00—the model reliably solves these on its first try.
Strong pass@1 on medium problems: For problems rated 1700–2400 (the middle range), pass@1 ranges from roughly 0.20 to 1.00, with most in the 0.80–1.00 range. The model sometimes needs a few attempts (the "# failed submissions" column shows 0–3 failures before a successful submission) but generally converges quickly.
Variable performance on hard problems: For problems rated 2600 and above, pass@1 drops substantially—some problems have pass@1 near zero (e.g., Contest 1909 G at 3000 rating: 3/1162 pass@1, or 0.26%) while others are surprisingly high (e.g., Contest 1919 F1 at 2300 rating: 1090/1162 pass@1, or 93.8%). This variability suggests that problem difficulty for o3 is not perfectly captured by the human-assigned rating; some problems that humans find hard may align with o3's strengths, and vice versa.
The ranking step is essential for hard problems: For the hardest problems that o3 does solve, the pass@10 (after ranking 1162 samples) is dramatically higher than pass@1 (without ranking). For example, Contest 1916 F (2900 rating): pass@1 is 369/1162 (31.7%), while pass@10 after ranking is 0.98 (98%). This means that o3 generates a correct solution somewhere in its 1,162 samples for almost all attempts, but finding it requires ranking the samples by some quality metric. However, unlike o1-ioi's external scoring function, o3's ranking is based on "test-time compute"—the model's own investment of deliberation, not a separately trained verifier.

Performance on IOI 2024. Figure 7 shows the IOI 2024 results under different submission strategies. With the strict 50-submission limit, o3 scored 395.64 points out of 600, well above the gold medal threshold of approximately 360 points. This is nearly double o1-ioi's 213 points under the same constraints, and it exceeds even o1-ioi's 362.14 points under the relaxed 10,000-submission limit.

The o3 IOI results were produced by "a later version of o3 than the CodeForces results" and "included additional fresher training data" (Section 4.2). This means the IOI and CodeForces evaluations are not exactly the same model—the IOI version had further RL training and possibly updated training data. The paper confirms that IOI 2024 occurred after the training data cut-off for this model and that a contamination check using embedding search verified the test problems were not in the training set.

The sampling protocol for IOI (1,024 solutions per problem, top 50 selected by test-time compute) is simpler than o1-ioi's in every dimension: fewer total samples (1,024 vs. ~50,000+), no subtask decomposition, no clustering, no test input generation and validation, no learned scoring function, no round-robin submission scheduling. The fact that this simpler approach achieves nearly double the score is the paper's central empirical argument for the superiority of learned reasoning over hand-crafted strategies.

Compute Accounting: What We Know and What's Missing

The paper's approach to compute comparison. The paper does not provide a formal FLOPs-matched comparison between o1-ioi and o3. Instead, it makes a qualitative argument: scaling RL training compute produced a model whose learned strategies (brute-force cross-checking, iterative debugging) replicate and exceed the functionality of the hand-crafted pipeline. The comparison is behavioral rather than quantitative.

What compute went into o1-ioi. The paper specifies:

RL training: "additional training compute" focused on coding tasks, starting from the o1 checkpoint. The magnitude of this additional compute is not quantified. The total o1 training compute is not disclosed in this paper.
Test-time compute: 10,000 solutions per subtask, with roughly 5-8 subtasks per problem and 6 IOI problems, totaling approximately 300,000–480,000 solution generations for the IOI competition. Each solution involves a chain of thought + code generation + code execution. For CodeForces, 10 submissions are made per problem (following the AlphaCode convention), with presumably a similar per-problem sampling budget.
Additional inference compute for the test-time pipeline: 256 test input generator and validator generations, 256 test input validations, 10,000 solution executions on 256 test cases (2.56 million program executions per subtask), clustering computation, scoring computation, and round-robin scheduling. These are not model inference costs but are non-trivial computational overhead.

What compute went into o3. The paper specifies:

RL training: "significantly greater compute resources than o1" (Section 4.1). No quantification is provided. The o3 system card [13] may contain details.
Test-time compute: for IOI, 1,024 solutions per problem × 6 problems = 6,144 total solution generations. For CodeForces, 1,162 samples per problem (the per-contest participant count) with 10 submissions per problem, across 12 contests with roughly 5-10 problems each, totaling on the order of 100,000–140,000 solution generations.
No additional pipeline compute: no clustering, no scoring function, no test input generation—just model inference and code execution.

What's missing for a fair comparison. To determine whether o3 truly represents a more efficient use of total compute (training + inference), one would need:

Total FLOPs for o1-ioi RL fine-tuning (the additional beyond o1) versus total FLOPs for o3 RL training (the additional beyond o1). If o3 is 10× more expensive to train than o1-ioi but saves 50× at inference time, it might still be compute-optimal for high-throughput deployments. If o3 is 100× more expensive and saves only 10× at inference, the hand-crafted pipeline remains cost-effective for low-throughput, high-stakes applications like IOI.
Per-solution inference cost: o3's solutions include longer, more sophisticated chains of thought (brute-force generation, cross-validation, debugging). Each o3 solution may cost more FLOPs than an o1-ioi solution. The reduction from 10,000 to 1,024 solutions might be partially offset by increased per-solution cost.
Amortization across problems: o3's RL training cost is a one-time investment that benefits all future problems. o1-ioi's test-time pipeline cost is incurred per-problem. For a system that will solve millions of problems, the training cost is amortized to near-zero per problem, making o3's higher training cost potentially irrelevant. For a system that will solve only 6 problems (like the IOI competition), training cost dominates.

The paper does not provide any of these numbers, making it impossible to draw conclusions about compute efficiency. The paper's argument is exclusively about capability (o3 achieves higher scores) and simplicity (o3 requires no human engineering of inference strategies), not about cost-effectiveness.

Software Engineering Evaluations: Testing Transfer

Why these evaluations matter. Sections 5.1 and 5.2 evaluate o1-preview and o1 on HackerRank Astra and SWE-bench Verified, respectively, with o3 results reported for SWE-bench. These evaluations serve a specific purpose in the paper's argument: they test whether the reasoning capabilities that drive competitive programming performance transfer to real-world software engineering tasks. If o1's reasoning only helped on algorithmic puzzles but not on practical coding, the approach would be of limited interest.

HackerRank Astra (Section 5.1). The Astra dataset consists of 65 "project-oriented coding challenges" that simulate real-world development. Key properties that distinguish it from competitive programming:

Multi-file, long-context scenarios: tasks involve building features across multiple files and frameworks (React.js, Django, Node.js), requiring the model to maintain coherence across a much larger context than a single-file algorithmic solution.
No public test cases provided: unlike CodeForces or IOI, Astra does not give the model example inputs and expected outputs to test against. This prevents the model from using test-case-based verification—it must determine correctness through other means (or simply generate code that it believes is correct on the first try).
Framework-specific knowledge: tasks require familiarity with specific web frameworks, their APIs, and idiomatic usage patterns—knowledge that is distinct from algorithmic reasoning.

Figure 8 shows that gpt-4o achieves 50.91% pass@1 and 69.52% average score. o1-preview improves this to 60.89% pass@1 (+9.98 percentage points) and 75.55% average score (+6.03 points). o1 further improves to 63.92% pass@1 (+3.03 points) and 75.80% average score.

The fact that reasoning improves performance on Astra (despite the absence of test cases to verify against) suggests that the benefits of chain-of-thought reasoning are not solely due to the verify-and-refine loop with code execution. Instead, the model's ability to plan, break down tasks, and reason about framework interactions—even without external verification—contributes to correctness. This aligns with the paper's broader thesis: reasoning is a general capability, not a domain-specific trick.

SWE-bench Verified (Section 5.2). SWE-bench Verified is OpenAI's human-validated subset of 500 tasks from the original SWE-bench [5], designed to evaluate AI models on resolving real-world GitHub issues. Each task provides a repository, an issue description, and the expectation that the model produces a patch that fixes the issue. Key properties:

Real-world codebases: tasks involve actual open-source projects with thousands of lines of code, complex dependency structures, and project-specific conventions.
No explicit test cases in the prompt: like Astra, SWE-bench does not provide example test cases—though the repository itself typically contains a test suite that the model could potentially discover and run.
Tool use is via open-source scaffold: for o1-preview, which "was not trained to use code execution or file editing tools" (Section 5.2), the evaluation used the "best-performing open-source scaffold at the time of initial implementation, Agentless." This scaffold provides the model with the ability to search the codebase, read files, and propose edits, but it is an external framework, not a capability the model learned through RL. For o1 and o3, the paper does not specify whether a scaffold was used or whether these models were trained with direct repository interaction capabilities.

Figure 9 shows: gpt-4o at 33.2%, o1-preview at 41.3% (+8.1 points), o1 at 48.9% (+7.6 points), o3 at 71.7% (+22.8 points over o1). The evaluation protocol: "All models are given 5 tries to generate a candidate patch. If the model fails after 5 attempts, it is considered an incorrect attempt. All evaluations are averaged over 3 trials. We do not penalize the model for system failures (e.g., container hangs or grading failures), and we retry these rollouts until we can record a valid attempt."

The SWE-bench results are significant for two reasons:

The absolute performance of o3 (71.7%) represents a dramatic improvement over o1 (48.9%)—a 22.8 percentage point gain that far exceeds the typical year-over-year progress on this benchmark. This suggests that scaling RL training produces capabilities that genuinely transfer to software engineering, not just competitive programming.
The progression across model versions mirrors the competitive programming results: gpt-4o → o1-preview → o1 → o3 shows monotonic improvement at each step, with o3's gain being the largest. This parallel scaling behavior across three very different domains (CodeForces, IOI, SWE-bench) strengthens the argument that reasoning is a unified capability that improves with RL training compute.

Design Choices and Their Justifications

The paper's design choices reflect its status as an empirical comparison rather than a proposal of new methods. Nevertheless, several choices are worth examining:

Why 10,000 solutions per subtask for o1-ioi? The number is inherited from AlphaCode [6, 7], which used even larger numbers (up to 10^6). The justification is coverage: competitive programming problems often require specific algorithmic insights, and even a strong model may only produce that insight in a fraction of its samples. Generating many samples and filtering intelligently is more reliable than hoping the model discovers the insight in a single try. The paper does not report the pass@1 of o1-ioi on IOI problems (what fraction of the 10,000 samples would be correct if submitted), but the gap between 213 points (50 selected submissions) and 362.14 points (all 10,000 submissions per subtask) suggests that a significant fraction of the 10,000 samples are incorrect—otherwise, 50 carefully-selected submissions would have scored much closer to 362.

Why 256 test inputs for clustering? The choice of 256 is a power of 2 convenient for implementation, but the reasoning behind the specific value is likely: (1) 256 is large enough that two different algorithms have high probability of producing different outputs on at least one input (making false-positive merges unlikely), (2) 256 is small enough that generating and validating the inputs, plus executing 10,000 solutions on all of them (2.56 million executions), is computationally feasible within the 10-hour competition window, and (3) 256 provides enough inputs that a bug missed by all of them is genuinely rare—a bug that survives 256 random tests without triggering is a subtle edge case that would likely also survive the IOI test suite.

Why no subtask decomposition for o3? The paper's rationale, though not stated explicitly as a design choice, is that o3's reasoning is sophisticated enough that it can handle the full problem statement and determine for itself the appropriate subtask strategy. Giving o3 the full problem statement preserves the possibility of solutions that elegantly cover multiple subtasks with a single algorithm—a capability that o1-ioi's subtask decomposition precludes. The paper notes that o3 "produced robust solutions capable of covering many, if not all, subtasks—without the need for subtask-specific prompts, manual partitioning, or intricate submission strategies" (Section 4.2).

Why 1,024 samples per problem for o3 (vs. 10,000 for o1-ioi)? The reduction in sample count reflects a fundamental difference in the quality of individual solutions. o3's chain-of-thought reasoning (including brute-force verification, iterative debugging) means that each sample is the product of much more internal deliberation than an o1-ioi sample. Generating 10,000 such high-effort samples would be extremely expensive and, as the results show, unnecessary—1,024 is sufficient to achieve gold medal performance. The paper does not explicitly state this as a design choice, but the reduction from 10,000 to 1,024 (a 10× reduction in sample count) is one of the implicit efficiency benefits of learned reasoning strategies.

Why select by "test-time compute" for o3? The selection criterion for o3's IOI submissions—"the top 50 solutions with the highest test-time compute from 1,024 samples per problem"—is unusual. Typically, one would select by a learned scoring function (like o1-ioi) or by majority voting among answers. Selecting by "test-time compute" (likely the number of tokens in the chain of thought, or some measure of deliberation effort) implies that the model's own investment of reasoning effort is a proxy for solution quality. The intuition is: if a problem is hard and the model recognizes that, it will naturally spend more tokens reasoning about it; the solutions that received the most deliberation are likely the ones where the model was most uncertain and therefore most careful. This is a heuristic, and the paper does not validate it against alternative selection criteria, but the results (395.64 points, well above gold) suggest it works in practice.

Why 5 attempts for SWE-bench? The SWE-bench evaluation protocol gives each model 5 tries per task. This is a different philosophy from CodeForces (10 tries) and IOI (50 submissions). The 5-attempt limit is likely inherited from the original SWE-bench evaluation framework and reflects the practical constraint that a developer using an AI coding assistant would not wait for dozens of attempts—after a few failed tries, they would take over manually. The paper averages over 3 trials to reduce variance from the stochastic nature of LLM patching.

4. Key Insights and Innovations

Innovation 1: Learned Test-Time Reasoning Strategies Can Fully Replace Hand-Engineered Inference Pipelines Without Performance Loss—And Actually Exceed Them

The paper's most consequential intellectual move is not demonstrating that o3 outperforms o1-ioi—it's establishing that the entire category of hand-crafted test-time strategies (clustering, reranking, subtask decomposition, round-robin submission scheduling) is rendered unnecessary by sufficient RL training. This is a fundamental reframing of the relationship between training and inference in reasoning systems.

What the field assumed before this work. The dominant paradigm for competitive programming AI, exemplified by AlphaCode (Li et al., 2022), AlphaCode 2 (Leblond et al., 2023), and the o1-ioi system itself, treated test-time strategy design as an essential engineering component. The assumption was that even a strong model generates a noisy distribution of solutions—some correct, most incorrect—and that the path to high performance lies in generating vast numbers of candidates (10^6 in AlphaCode, 10,000 per subtask in o1-ioi) and then filtering them through sophisticated, domain-specific selection pipelines. These pipelines encoded human knowledge about competition structure (subtask scoring rules, the 50-submission limit, test case constraints) that the model itself did not possess. The implicit model of intelligence was: generation is learned; selection is engineered.

What this paper demonstrates instead. The o3 results invert this model. The paper shows that when RL training is scaled sufficiently, the model internalizes the selection and verification logic into its generation process. Rather than generating 10,000 independent solutions and relying on an external clustering algorithm to group them by behavior, o3 generates a much smaller number of solutions (1,024 per problem for IOI) where each solution is the product of an extended chain of thought that includes its own verification—writing brute-force reference implementations, executing them, cross-checking outputs, and debugging discrepancies (Figure 6). The selection is not post-hoc; it happens during generation, through the model's own deliberation about which approaches to pursue and which to abandon.

The specific evidence for replacement, not just improvement. Figure 3 shows that o1-ioi's hand-crafted test-time strategy contributed approximately 367 CodeForces rating points (from 1807 to 2214). Figure 5 shows that o3, without any hand-crafted strategy, achieved 2724—a 510-point gain over o1-ioi's full system. Critically, the o1-ioi-to-o3 improvement (510 points) exceeds the contribution of the hand-crafted strategy itself (367 points). This means that the RL training that produced o3's learned verification behaviors contributed more to performance than the entire human-designed pipeline—even starting from a model (o1-ioi base) that was already fine-tuned for coding.

The IOI results tell the same story with different numbers: o1-ioi with hand-crafted strategy scored 213 points under the 50-submission limit; o3 with no hand-crafted strategy scored 395.64 points (Figure 7). The hand-crafted strategy was worth approximately 57 points over random selection (213 vs. 156 for o1-ioi), but the learned strategies were effectively worth 182 additional points beyond that (395.64 - 213). The learned approach didn't just match the hand-crafted approach—it nearly doubled its effectiveness.

Why this is fundamental rather than incremental. This finding changes what it means to "engineer" a competitive programming system. Before o3, engineering effort was split between training (improving the model's code generation capabilities) and inference (designing selection algorithms). After o3, the evidence suggests that inference-time engineering can be eliminated entirely for this class of problems—the model handles both generation and selection through learned reasoning. This is not a better clustering algorithm or a more clever scoring function; it's the elimination of the need for such external mechanisms. The intellectual shift is from "how do we best filter noisy model outputs?" to "how do we train models that produce less noisy outputs in the first place, and verify their own work?"

A critical nuance about generality. The paper is careful not to claim that learned strategies always dominate hand-crafted ones for every possible domain or budget level. The o1-ioi hand-crafted strategy was itself a stepping stone—it pushed performance to 2214 on CodeForces and 213 on IOI, which was state-of-the-art. The claim is that with sufficient scaling, learned strategies surpass hand-crafted ones. The paper does not quantify "sufficient"—training o3 required "significantly greater compute resources than o1" (Section 4.1), but no FLOP comparison is provided. A pragmatic reading is: if you have the budget to scale RL training to o3-like levels, invest there rather than in test-time engineering; if you are resource-constrained, hand-crafted strategies remain valuable.

Innovation 2: The Substitution Relationship Between Training-Time RL and Test-Time Search Is Empirical, Not Axiomatic—And This Paper Provides the First Demonstration in a Real-World Competitive Domain

The idea that training compute can substitute for test-time search is not new in machine learning—it's the principle behind distillation, amortized inference, and model-based planning. But demonstrating this substitution in a domain as complex as competitive programming, where the search space is combinatorial and the correctness criterion is unforgiving, required both (a) building the hand-crafted search system to serve as a strong baseline and (b) scaling RL training to the point where it overtook that baseline. This paper is the first to do both.

What existed before. Prior work in the LLM reasoning space studied these axes separately. The AlphaCode line (Li et al., 2022; Leblond et al., 2023) focused on test-time search with fixed models, showing that more sophisticated search (larger sample counts, better clustering, learned scoring) improved performance at inference time. The o1 line (OpenAI, 2024) and concurrent efforts like DeepSeek-R1 (DeepSeek-AI, 2025) and Kimi k1.5 (Kimi Team, 2025) focused on improving model capabilities through RL training, showing that chain-of-thought reasoning boosted performance across math and coding benchmarks. But no prior work ran the head-to-head experiment: take a model trained with extensive RL on coding (o1-ioi base), add the best available hand-crafted test-time pipeline (the clustering and submission strategy), observe the performance ceiling (2214 CodeForces, 213 IOI), then scale RL further and remove the hand-crafted pipeline to see whether the learned strategies alone could exceed that ceiling (2724 CodeForces, 395.64 IOI).

The conceptual contribution. The paper operationalizes what was previously a theoretical tradeoff—"invest in training vs. invest in inference"—into a concrete empirical comparison with a clear outcome. This matters because the substitution relationship is not guaranteed. One could imagine a world where hand-crafted test-time strategies capture irreducible domain knowledge (the IOI submission format, the subtask scoring rules, the typical structure of competitive programming problems) that no amount of general-purpose RL training could internalize. The paper shows that this is not the case: o3's chain of thought, which emerges purely from RL with a correctness reward, develops behaviors that functionally replicate the key components of the hand-crafted pipeline—verification against reference implementations (analogous to the clustering step's comparison of outputs) and iterative debugging (analogous to the reranking step's selection of the most promising candidates).

Evidence that this is not merely a scaling curve. If o3 were simply a larger, better-trained version of o1 that happened to outperform o1-ioi through raw capability, the paper's argument would be weaker. But Figure 6 provides qualitative evidence of behavioral change, not just quantitative improvement. The model is not just better at writing correct code on the first try—it is doing something categorically different from o1 and o1-ioi. It is writing a completely separate brute-force implementation, executing it, and using the outputs to validate its optimized solution. This is a strategy that o1 and o1-ioi did not exhibit (or at least not reliably). The strategy itself is what substitutes for the external pipeline—the model performs verification internally rather than relying on an external process to cluster and filter.

The implication for future system design. This finding suggests that the marginal return on engineering effort for test-time pipelines may be zero (or negative, counting opportunity cost) once RL training reaches a certain scale. Every hour spent designing clustering algorithms, tuning penalty weights via random search on previous IOI problems, or implementing subtask inheritance pruning is an hour not spent improving the RL training recipe. For organizations with sufficient compute budgets, the paper argues implicitly for a reallocation of engineering effort from inference-time heuristics to training-time optimization. This is a strategic insight for AI development resource allocation, not just a technical finding about model performance.

Innovation 3: Verification Strategies Emerge from End-to-End RL Training Without Explicit Supervision on Verification Behavior

The emergence of brute-force verification in o3 (Figure 6) is the paper's most striking qualitative finding, and it carries implications that extend beyond competitive programming into the broader study of how complex behaviors arise in trained models.

The standard approach to verification in AI systems. Prior work on verification in code generation typically took one of two approaches. The first was external verification: train a separate model to predict correctness (the learned scoring function in both AlphaCode systems and o1-ioi), or use test case execution as an external oracle (the public test filtering used in o1-ioi). The second was prompted verification: provide few-shot examples showing a model verifying its own work, or use system prompts instructing the model to "check your answer" (Wei et al., 2022; Madaan et al., 2023). Both approaches have the same limitation: the verification behavior is specified by humans, either through training a separate system or through explicit instruction in the prompt.

What o3 does differently. The brute-force verification strategy in Figure 6 was not trained through supervised examples of correct verification behavior, nor was it prompted with instructions to cross-check against brute-force solutions. It emerged from RL training where the only signal was whether the final submitted code passed the test cases. The model discovered that writing a slow-but-correct reference implementation and comparing outputs is an effective way to increase the probability that its final answer is correct. This is a learned meta-cognitive strategy—the model has learned not just how to solve coding problems, but a strategy for determining whether its solution to a coding problem is correct, and it applies this strategy selectively to problems where simple test-case checking is insufficient.

Why this is intellectually significant beyond the performance numbers. The emergence of verification behaviors challenges a common assumption in the literature: that complex, multi-step reasoning strategies need to be explicitly taught through imitation learning (supervised fine-tuning on examples of good reasoning) or through structured search (tree-of-thought, graph-of-thought). The o3 results suggest that with sufficient RL training and a clean reward signal, models can invent effective reasoning strategies that humans would recognize as sophisticated (brute-force cross-checking, iterative debugging) but that were never demonstrated in the training data. This has implications for how we think about training reasoning models: it may be more effective to invest in better reward signals and more RL compute than to curate datasets of "good reasoning" examples, because the model may discover strategies that humans didn't think to demonstrate.

A critical caveat about the training data. The paper does not establish that the brute-force verification strategy was absent from o3's training data. o3 was trained on a large corpus of coding problems and solutions; it's possible that some training examples included brute-force solutions alongside optimized ones, or that the training data contained comments discussing verification strategies. The paper's claim of "emergence" would be stronger if it could demonstrate that such examples were systematically excluded from training. However, even if the training data contained instances of brute-force verification, the key point is that the model learned to deploy this strategy conditionally—only when verification is nontrivial, only when a brute-force solution is feasible—without explicit conditioning on problem features or being told when to verify. This conditional deployment is the genuinely emergent behavior, even if the individual components (writing code, executing it, comparing outputs) were present in training.

Connection to the broader "emergence" discourse. This finding contributes to the ongoing debate about whether sophisticated behaviors in large models are "truly emergent" or merely interpolating patterns present in training data. The paper takes a clear stance: "complex test-time reasoning strategies emerged naturally from end-to-end RL" (Section 1). The strength of this evidence depends on what one considers sufficient to establish emergence. The paper demonstrates that the behavior exists, that it was not explicitly programmed (no hand-crafted pipeline), and that it was not prompted (o3 receives only the problem statement). Whether this constitutes "emergence" or "sophisticated interpolation" depends on one's prior about what patterns exist in the training data, which the paper does not resolve.

Innovation 4: The Unified Evaluation Across Competitive Programming and Software Engineering Establishes That RL-Trained Reasoning Is a Transferable Capability, Not a Domain-Specific Optimization

The paper's final conceptual contribution is its demonstration that reasoning capabilities trained primarily on algorithmic correctness transfer to qualitatively different software engineering tasks (Figures 8 and 9), providing evidence against the hypothesis that competitive programming success reflects narrow, benchmark-specific optimization.

The "overfitting to benchmarks" concern. A persistent worry in the LLM evaluation literature is that improvements on a specific benchmark reflect contamination of the training data with benchmark problems, or narrow optimization on the benchmark's particular format and difficulty distribution. Competitive programming is especially susceptible to this concern because (a) CodeForces and IOI problems are publicly available, (b) solutions are widely posted online, and (c) the format (problem statement → code file → test suite) is highly stereotyped. One could imagine that o-series models succeed on competitive programming because they've memorized problem-solution pairs from their training data, or because they've learned superficial patterns of the competition format without developing genuine reasoning capabilities.

The paper's multi-domain evidence structure. The paper mitigates this concern through deliberate evaluation design:

Temporal separation: CodeForces evaluations use Division 1 contests from late 2023 and 2024, all occurring after o3's training data cut-off, with embedding-based contamination checks as a secondary safeguard (Appendix B.1).
Format diversity: The four evaluation domains (CodeForces, IOI, HackerRank Astra, SWE-bench Verified) differ substantially in format, constraints, and success criteria. CodeForces is a contest platform with public pretests and hidden full tests. IOI is a multi-day Olympiad with subtask-based scoring and a 50-submission limit. HackerRank Astra involves multi-file, framework-specific projects without public test cases. SWE-bench involves patching real open-source repositories based on GitHub issue descriptions.
Parallel scaling curves: The performance progression across model versions (gpt-4o → o1-preview → o1 → o3) follows a consistent upward trajectory across all four domains. On CodeForces: 808 → 1258 → 1673 → 2724. On SWE-bench: 33.2% → 41.3% → 48.9% → 71.7%. The fact that each incremental investment in RL training yields improvements across all domains—rather than, say, competitive programming improving while SWE-bench stagnates—suggests a shared underlying capability being strengthened.

The specific evidence for transfer. On HackerRank Astra, o1 achieves 63.92% pass@1 (Figure 8), compared to gpt-4o's 50.91%. This domain is particularly interesting as a transfer test because it explicitly lacks public test cases, meaning the verify-and-refine loop that the model uses in competitive programming cannot operate via simple test-case checking. The model's improvement must come from other aspects of reasoning—better planning, better code organization, better anticipation of potential bugs—that transfer from training on algorithmic problems. The paper does not provide o3 results on Astra, which is a notable gap in the transfer evidence.

On SWE-bench Verified, o3 achieves 71.7% (Figure 9), more than double gpt-4o's 33.2%. SWE-bench tasks require understanding large codebases, identifying the right files to modify, and producing patches that interact correctly with existing code—skills that are not directly trained in competitive programming but apparently benefit from the same RL training. The paper specifies that o1-preview used the Agentless scaffold for repository interaction because it was not trained with code execution or file editing tools, but does not clarify whether o1 and o3 had native repository interaction capabilities or also relied on scaffolds. This is a meaningful gap: if o3's SWE-bench performance relied on an external scaffold rather than learned tool use, the transfer evidence is weaker.

Why this matters for the paper's thesis. The transfer results are essential to the paper's argument that RL-trained reasoning is a general capability that replaces domain-specific engineering, not just a better way to do competitive programming. If o3 only excelled at CodeForces and IOI while performing similarly to o1 on SWE-bench, the paper's claim would be narrower: RL training helps with algorithmic reasoning specifically, but domain-specific engineering remains necessary for real-world tasks. The SWE-bench result (71.7%, a 22.8-point improvement over o1) strengthens the claim that learned reasoning strategies generalize, reducing the need for task-specific inference pipelines across multiple domains, not just competitive programming.

The limits of the transfer evidence. The paper evaluates on only two software engineering benchmarks (Astra and SWE-bench), both of which are code-generation tasks with objective correctness criteria. The transfer claim would be stronger with evidence from domains that are less code-centric—scientific reasoning, mathematical proof, or strategic planning—where the reasoning strategies learned from coding might not directly apply. The paper acknowledges these limits implicitly by focusing its claims on "coding tasks" and "reasoning domains" rather than making universal claims about all AI applications.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper evaluates on four distinct datasets: (1) CodeForces Division 1 contests from late 2023 and 2024, comprising 12 simulated contests with a total test set of problems occurring after o3's training data cut-off, verified via embedding-based contamination checks (Appendix B.1); (2) the 2024 International Olympiad in Informatics (IOI), consisting of 6 algorithmic problems with subtask-based scoring and a maximum total of 600 points; (3) the HackerRank Astra dataset of 65 project-oriented coding challenges spanning frameworks like React.js, Django, and Node.js (Section 5.1); and (4) SWE-bench Verified, OpenAI's human-validated subset of 500 tasks from the original SWE-bench designed to evaluate real-world GitHub issue resolution (Section 5.2).
Base model(s). Four model versions are evaluated in progression: gpt-4o (a non-reasoning baseline), o1-preview (an early reasoning model with RL-trained chain-of-thought), OpenAI o1 (a more extensively RL-trained reasoning model with tool use for code execution), o1-ioi (o1 further fine-tuned on coding tasks with hand-crafted test-time strategies), and an early checkpoint of o3 (a scaled-up reasoning model with "significantly greater compute resources than o1" and no hand-crafted inference strategies). The o3 version used for IOI 2024 is described as "a later version of o3 than the CodeForces results" with "additional fresher training data" (Section 4.2).
Metrics. CodeForces performance is measured by Elo-based rating, calculated by simulating contest participation—solving problems under full test suites with time and memory constraints, allowing 10 independent submissions per problem (following AlphaCode [6]), and estimating the model's rating by directly maximizing the likelihood of observed rankings relative to human participants using the formula $\frac{1}{10^{(R_B - R_A)/400} + 1}$ for win probability (Appendix B.4). Thinking time is normalized by assigning the model "the median of the scores of the human participants that solved that problem in the contest with the same number of failed attempts" to reduce parallelism advantages (Appendix B.3). Percentiles are relative to CodeForces' global leaderboard of active participants. IOI performance is measured by total points scored out of 600 under the official subtask-based grading system. HackerRank Astra reports pass@1 (probability of completing a task on the first attempt) and average score (mean proportion of test cases passed across all tasks). SWE-bench Verified reports the percentage of 500 tasks for which the model produces a correct patch within 5 attempts, averaged over 3 trials, with system failures (container hangs, grading failures) excluded and retried (Section 5.2).
Baselines. The evaluations use progressive baselines rather than a single fixed reference: gpt-4o serves as the non-reasoning baseline for all domains; o1-preview provides the first reasoning-model baseline; o1 provides the mature reasoning baseline; o1-ioi with its full hand-crafted test-time strategy (clustering, reranking, round-robin submission) provides the domain-specific optimization baseline against which o3's learned strategies are compared. At IOI 2024, an additional baseline of "random 50 submissions" from o1-ioi's candidate pool (without the clustering and reranking strategy) establishes the contribution of the selection pipeline: random selection yielded 156 points versus 213 points with the hand-crafted strategy (Figure 4).
Generation budget / compute accounting. The paper measures test-time compute in terms of number of candidate solutions generated per problem: o1-ioi generates 10,000 solutions per subtask for IOI (approximately 300,000–480,000 total across 6 problems with ~5–8 subtasks each) and makes up to 10 submissions per problem for CodeForces; o3 generates 1,024 solutions per problem for IOI (selected to the top 50 by test-time compute) and up to 1,162 samples per CodeForces contest. The paper does not provide a formal FLOPs comparison between o1-ioi and o3 training, stating only that o3 used "significantly greater compute resources" during training (Section 4.1). Inference-time cost accounting is incomplete: the computing overhead of o1-ioi's test-time pipeline (256 test input generations, 256 validations, 2.56 million solution executions across 10,000 solutions for clustering, scoring function evaluation) is not quantified against o3's simpler inference costs. For SWE-bench, each model gets exactly 5 attempts per task, averaged over 3 trials, with no penalty for system failures.
Cross-validation / statistical protocol. No cross-validation is described in the paper—the evaluations are direct, single-pass measurements on held-out test sets. For CodeForces, temporal separation ensures all contests occurred after the training data cut-off, with an "embedding search to confirm test problems have not been seen" during training serving as a secondary contamination check (Appendix B.1). For IOI 2024, the paper states the same: "IOI 2024 occurred after the training cut-off for this model, and we additionally confirmed with search that the IOI test problems are not contaminated with the training set" (Section 4.2). For the hand-crafted o1-ioi pipeline, penalty weights in the cluster scoring formula were "tuned by random search on solutions to previous years' IOI problems, by directly simulating the submission process" (Section 3.2), representing a form of hyperparameter optimization on a held-out temporal validation set. SWE-bench evaluations are averaged over 3 trials to reduce variance from stochastic generation, but no confidence intervals are reported.

Main Quantitative Results

CodeForces Progression: From Non-Reasoning to Reasoning Models

Figure 1 establishes the baseline progression: gpt-4o (non-reasoning) achieves a CodeForces rating of 808 (11th percentile); o1-preview (early RL-trained reasoning) jumps to 1258 (62nd percentile); and o1 (further RL training) reaches 1673 (89th percentile). The jump from gpt-4o to o1-preview represents a 450-point gain attributable to the introduction of RL-trained chain-of-thought reasoning with code execution capabilities. The further jump from o1-preview to o1 adds 415 points through additional RL training.

Figure 3 shows the o1-ioi progression on CodeForces. The base o1-ioi model (coding-specific RL fine-tuning only, no test-time strategies) achieves 1807 (93rd percentile)—a 134-point gain over o1's 1673, representing the pure benefit of additional coding-focused RL training. Adding a simple post-hoc filter that rejects any solution failing public test cases raises performance to 2092 (96th percentile), a 285-point gain that is attributed not to improved generation but to better selection from among already-generated candidates. The full o1-ioi test-time strategy (clustering, reranking, round-robin submission, subtask inheritance pruning) yields 2214 (98th percentile), adding another 122 points beyond the simple filter. Cumulatively, the hand-crafted test-time components contribute 407 points over the raw o1-ioi model (from 1807 to 2214).

Figure 5 presents the o3 result on the same CodeForces benchmark: 2724 rating (99.8th percentile). This represents a 510-point gain over the full o1-ioi system (2214) and a 917-point gain over o1-ioi's raw model without test-time strategies (1807). In percentile terms, o3 would "rank among the top 200 active competitors worldwide" (Appendix B.5). Figure 10 visualizes o3's position relative to top human competitors, showing that while "the very best human competitors remain much stronger than o3, with solve rates in excess of 85%," o3 falls within the top 200 globally.

CodeForces Per-Problem Analysis

Table 1 in Appendix B provides a granular breakdown of o3's performance across all 12 CodeForces Division 1 contests. The table reports, for each problem: problem rating (human-calibrated difficulty), pass@1 (solved on first attempt without any ranking), pass@10 after no ranking (ability to solve within 10 attempts without ranking), and pass@10 after ranking 1,162 samples. Several patterns emerge:

Easy problems (rated 800–1400): Near-perfect pass@1 across the board. For example, Contest 1909 problems A–C, rated 800–1400, all show pass@1 within 1066–1156 out of 1162 participants (91.7–99.5% solve rate). These are reliably solved on the first attempt without any ranking or selection.
Medium problems (rated 1700–2400): Pass@1 varies substantially. Some problems are solved easily (e.g., Contest 1909 D at 1900 rating: 1099/1162 pass@1, 94.6%), while others require multiple attempts (e.g., Contest 1984 D at 2000 rating: 193/1162 pass@1, 16.6%, but rising to 0.84 pass@10 after ranking). The "# failed submissions" column shows that when problems aren't solved on the first try, o3 typically requires only 1–3 failures before converging to a correct solution—indicating that the debugging loop (write, execute, observe failure, revise) is effective but not instant.
Hard problems (rated 2600+): Performance is highly variable. Some hard problems are solved with surprising reliability (e.g., Contest 1919 F2 at 2800 rating: 227/1162 pass@1, 19.5%, reaching 0.89 pass@10 after ranking). Others remain essentially unsolved (e.g., Contest 1909 G at 3000 rating: 3/1162 pass@1, 0.26%—solved by only 3 of 1,162 participants). The "# failed submissions" column for successfully solved hard problems typically shows more failures: Contest 2002 F1 at 2600 rating required 8 failed submissions before succeeding, and Contest 1984 H at 3300 rating required 3.
The ranking step matters most for hard problems. For problems where pass@1 is very low, the pass@10 after ranking is dramatically higher: Contest 1916 F at 2900 rating goes from 369/1162 pass@1 (31.7%) to 0.98 pass@10 after ranking—meaning that among 1,162 samples, a correct solution exists for 98% of attempts, but finding it without ranking would require many more than 10 submissions. The ranking mechanism (selecting by "test-time compute") is essential for converting the model's raw capability into actual contest performance on hard problems.

IOI 2024 Results Under Progressive Constraints

Figure 4 presents o1-ioi's IOI 2024 live competition results under the official 50-submission limit. The hand-crafted test-time strategy yielded 213 points, placing the system in the 49th percentile. The contribution of the selection strategy is isolated by comparison: selecting 50 random submissions from the 10,000 candidates per subtask would have yielded an average of only 156 points. The hand-crafted strategy thus contributed approximately 57 points (213 minus 156), a 36.5% improvement over random selection within the tight submission budget.

When the submission limit is relaxed to 10,000 per problem—effectively allowing all generated candidates to be submitted—o1-ioi's score rises to 362.14 points, just above the gold medal threshold of approximately 360 points (Figure 4). The gap between 213 (50 selected submissions) and 362.14 (all 10,000 submissions per subtask) demonstrates that o1-ioi was generating gold-medal-worthy solutions among its candidates, but the hand-crafted selection strategy was unable to identify enough of them within the 50-submission constraint. The selection strategy successfully identifies approximately 59% of the maximum achievable score (213/362.14).

Figure 7 presents the comparative IOI 2024 results. o3, using only 1,024 samples per problem with the top 50 selected by test-time compute (no hand-crafted pipeline, no subtask decomposition, no clustering or reranking), achieves 395.64 points under the official 50-submission limit. This is:

182.64 points above o1-ioi's 213 under the same 50-submission constraint—nearly double the performance.
33.5 points above o1-ioi's 362.14 achieved with 10,000 submissions per subtask—meaning o3 with 50 carefully-selected submissions outperforms o1-ioi with essentially unlimited submissions.
35.64 points above the gold medal threshold of approximately 360—exceeding the highest honor by a comfortable margin.

The paper notes that "o3 produced robust solutions capable of covering many, if not all, subtasks—without the need for subtask-specific prompts, manual partitioning, or intricate submission strategies" (Section 4.2).

HackerRank Astra: Reasoning Transfer to Project-Oriented Coding

Figure 8 presents HackerRank Astra results for a subset of the model progression (o3 is not evaluated on Astra, which represents a gap in the full transfer evidence). gpt-4o achieves 50.91% pass@1 and 69.52% average score. o1-preview improves to 60.89% pass@1 (+9.98 percentage points) and 75.55% average score (+6.03 points). o1 further improves to 63.92% pass@1 (+3.03 points over o1-preview) and 75.80% average score (+0.25 points).

The HackerRank Astra domain is notable because it "does not provide public test cases, which prevents us from relying on hand-crafted test-time tactics" (Section 5.1). This means the verify-and-refine loop that o1 uses on CodeForces and IOI—where it can execute code against provided test cases and iterate—cannot operate via simple test-case checking. The improvements over gpt-4o must therefore derive from other reasoning capabilities developed through RL training: better task decomposition, more careful planning of multi-file interactions, or more robust code generation that anticipates edge cases without needing external verification feedback.

The paper notes that the pass@1 improvement from o1-preview to o1 (63.92% vs. 60.89%) "demonstrates o1's enhanced reasoning and adaptability" and that these results show reasoning "extend to more practical, industry-related coding tasks" (Section 5.1). However, the relatively small incremental gain from o1-preview to o1 (3.03 percentage points) compared to the leap from gpt-4o to o1-preview (9.98 points) suggests that reasoning benefits for framework-specific, project-oriented coding may saturate earlier than for algorithmic problem-solving—or that Astra's difficulty distribution is insufficiently challenging for the ceiling effect to be informative.

SWE-Bench Verified: Scaling to Real-World Software Engineering

Figure 9 presents the full model progression on SWE-bench Verified. gpt-4o achieves 33.2%. o1-preview improves to 41.3% (+8.1 percentage points). o1 reaches 48.9% (+7.6 points over o1-preview, +15.7 points over gpt-4o). o3 achieves 71.7% (+22.8 points over o1, +38.5 points over gpt-4o). The o3 gain of 22.8 percentage points over o1 is the single largest improvement across any domain reported in the paper and represents o3 more than doubling the gpt-4o baseline (33.2% → 71.7%).

The evaluation protocol is specified: "All models are given 5 tries to generate a candidate patch. If the model fails after 5 attempts, it is considered an incorrect attempt. All evaluations are averaged over 3 trials" (Section 5.2). The paper does not report variance across the 3 trials, so it is unclear how stable these numbers are. The o3 SWE-bench evaluation used an early checkpoint, while the CodeForces and IOI evaluations used potentially different o3 versions (the IOI version was "a later version... with additional fresher training data"). This means the SWE-bench result is not strictly from the same model that achieved the CodeForces 2724 rating.

An important architectural detail affects how these results should be interpreted: "Because o1-preview was not trained to use code execution or file editing tools, the best-performing open-source scaffold at the time of initial implementation, Agentless, was used" (Section 5.2). The paper does not clarify whether o1 and o3 used the same scaffold or whether they had native repository interaction capabilities (direct file reading, searching, editing) learned through RL. If o3's 71.7% was achieved using an external scaffold, the transfer claim is about reasoning capability applied through a generic interface, not about learned tool use transferring from competitive programming to software engineering. If o3 had native repository interaction capabilities, the 71.7% reflects a combination of learned reasoning and learned tool use. The paper's silence on this distinction is a meaningful gap in the evaluation methodology.

Ablation Studies and Robustness Checks

The paper does not contain formal ablation studies in the traditional sense—there are no controlled experiments where specific components of the training or inference pipeline are removed to isolate their contributions. Instead, the progressive model comparison across o1 → o1-ioi → o3 serves as a de facto ablation of the hand-crafted test-time strategy, and the selection strategy comparison at IOI serves as an ablation of the test-time pipeline components. I will describe these as ablations, since that is how the paper implicitly uses them.

Ablation of hand-crafted test-time strategy via model progression (Figures 1, 3, 5): The CodeForces rating progression—o1 at 1673 (general RL, no hand-crafted strategy) → o1-ioi base at 1807 (coding-specific RL, no hand-crafted strategy) → o1-ioi full at 2214 (coding-specific RL + full hand-crafted strategy) → o3 at 2724 (scaled RL, no hand-crafted strategy)—constitutes an implicit ablation where the hand-crafted strategy is first added (o1-ioi base to o1-ioi full, contributing 407 points) and then removed while simultaneously scaling RL (o1-ioi full to o3, gaining 510 points net). The fact that o3 with no hand-crafted strategy outperforms o1-ioi with the full strategy demonstrates that scaling RL can compensate for and exceed the removal of domain-specific inference heuristics. However, this is not a clean ablation because RL training scale and the hand-crafted strategy are not varied independently—we cannot separate how much of o3's gain comes from the scaling of RL versus the removal of the hand-crafted strategy (which might have actually limited o1-ioi by constraining it to subtask-specific solutions rather than allowing unified solutions).

Ablation of subtask decomposition and clustering via IOI submission comparisons (Figure 7): The IOI results compare three configurations: (a) o1-ioi with 50 submissions and full hand-crafted strategy (subtask decomposition, clustering, reranking, round-robin), scoring 213; (b) o1-ioi with 10,000 submissions and no selection strategy (effectively removing the selection components while retaining subtask decomposition), scoring 362.14; and (c) o3 with 50 submissions and no hand-crafted strategy at all (removing subtask decomposition entirely, no clustering, no reranking), scoring 395.64. The comparison between (b) and (c) shows that removing subtask decomposition (o3 receives the full problem statement) while simultaneously scaling RL training actually improves performance over having subtask decomposition with a weaker model. However, this is again not a clean ablation—the model and the strategy are changed simultaneously.

Ablation of selection mechanism within o1-ioi (Figures 3 and 4): Within the o1-ioi system, three levels of selection are tested. Raw model output without any filtering yields 1807 on CodeForces. Adding only public test filtering (reject solutions that fail the provided test cases) yields 2092—the 285-point gain isolates the contribution of a very simple selection mechanism that any model could use. Adding the full test-time strategy (clustering + reranking + learned scoring) on top yields 2214—the additional 122 points isolates the marginal contribution of the sophisticated human-designed components beyond simple test-case checking. At IOI, the random-vs-strategy comparison (50 random submissions scoring 156 vs. hand-crafted selection scoring 213) isolates the 57-point contribution of the selection strategy under the tight submission budget, confirming that the selection mechanism provides substantial value when submissions are scarce.

Ablation of submission budget via relaxed constraints (Figure 4, rightmost bar): When o1-ioi is allowed 10,000 submissions per problem instead of 50, performance jumps from 213 to 362.14. This demonstrates that the bottleneck in the 50-submission setting is not generation quality (the model can produce gold-medal solutions) but selection quality (the strategy fails to identify enough correct solutions within 50 tries). The fact that o3 achieves 395.64 with only 50 submissions (no relaxed constraint) indicates that o3's per-solution quality is high enough—due to its internal verification and debugging—that selection from 1,024 samples with a simple criterion ("highest test-time compute") is sufficient. The relaxed-constraint ablation reveals that o1-ioi and o3 face different bottlenecks: o1-ioi is limited by selection accuracy; o3 is limited by raw generation quality (and even that limit is above gold).

Ablation of problem formulation via full-problem vs. subtask prompting (Section 4.2): The paper explicitly contrasts o1-ioi's subtask-decomposed prompting ("we divided each IOI problem into its composite subtasks... creating one version of the document for each subtask") with o3's unified prompting ("we adopted a different approach when evaluating o3: sampling from a single prompt containing the original problem statement"). The results (o3 scoring 395.64 vs. o1-ioi's 213 under 50 submissions) suggest that receiving the full problem statement enables o3 to produce solutions that cover multiple subtasks simultaneously—a capability that subtask decomposition explicitly precludes. However, this comparison again conflates model quality with prompting strategy.

Negative result: o1-ioi's selection strategy leaves substantial performance on the table. The gap between o1-ioi's 50-submission score (213) and its unlimited-submission score (362.14), shown in Figure 4, is a negative result for the hand-crafted selection pipeline. Despite extensive engineering—clustering by output behavior on 256 generated test inputs, learned scoring, cluster penalty tuning via random search on previous IOI problems, subtask inheritance pruning—the pipeline recovered only 59% of the achievable score. Over 40% of the points that the model could have earned were lost to selection failures. This suggests fundamental limitations in the clustering-by-output approach: bugs that manifest only on specific inputs not included in the 256 generated test cases would produce identical output clusters to correct solutions, causing the system to submit buggy solutions while believing them correct. The paper does not quantify the false-positive rate of the clustering—how many clusters that appeared correct (consistent outputs on all 256 test inputs) actually contained only buggy solutions that would fail the hidden IOI test suite.

Negative result: public test filtering provides most of the easy selection gains. On CodeForces (Figure 3), the jump from raw o1-ioi (1807) to public-test-filtered o1-ioi (2092) accounts for 285 of the 407 total points gained by the hand-crafted pipeline—70% of the total test-time strategy benefit comes from a trivial filter that any system can implement. The sophisticated components (clustering, reranking, learned scoring, round-robin scheduling) contribute only the remaining 122 points (30%). This suggests that much of the hand-crafted pipeline's value could be captured by simply generating code, running it against the provided test cases, and rejecting any that fail—a strategy that requires zero domain-specific engineering.

Critical Assessment

Claim: "Complex test-time reasoning strategies emerged naturally from end-to-end RL."

The paper's central empirical claim is that o3 spontaneously develops sophisticated verification behaviors (brute-force cross-checking, iterative debugging) through RL training alone, without explicit supervision on these strategies. The evidence for this claim is qualitative rather than quantitative—Figure 6 provides a single illustrative example of o3's chain of thought showing brute-force verification. The paper does not report what fraction of o3's solutions exhibit this behavior, whether it appears selectively on problems where verification is genuinely nontrivial, whether it correlates with problem difficulty or solution correctness, or whether o1 and o1-ioi ever exhibit similar behaviors (if o1 also occasionally writes brute-force verifiers, the "emergence" claim would be about frequency rather than capability). A single cherry-picked example of impressive behavior is not sufficient to establish that the behavior "emerged" as a systematic strategy—the paper would need to quantify the prevalence of brute-force verification, its correlation with successful problem-solving, and its absence (or lower prevalence) in earlier models.

More fundamentally, the paper cannot rule out the possibility that o3's training data contained explicit examples of brute-force verification strategies. CodeForces editorials, forum discussions, and solution writeups frequently describe the "write a slow solution to verify your fast solution" technique—it is a standard competitive programming practice. If such descriptions appeared in o3's training corpus, the "emergence" would be better characterized as the model learning to retrieve and deploy an existing strategy at appropriate moments, rather than inventing it de novo. The paper's embedding-based contamination check only verifies that the specific test problems were not in training, not that verification strategies were absent from the training distribution.

The claim would be strengthened by: (1) quantifying the frequency of brute-force verification in o3's solutions across all 12 CodeForces contests, (2) comparing this frequency to o1 and o1-ioi on the same problems, (3) demonstrating that the frequency correlates with problem characteristics where verification is known to be nontrivial (e.g., geometry problems where floating-point precision matters, or graph problems with complex constraints that make manual verification difficult), and (4) providing a training data analysis showing that brute-force verification examples were systematically excluded or that the strategy emerged even in problem domains where standard training data would not contain it.

Claim: "o3 achieves gold without hand-crafted domain-specific strategies or relaxed constraints."

This claim is strongly supported by the IOI 2024 results (Figure 7): o3 scores 395.64 under the official 50-submission limit with no hand-crafted test-time strategy, exceeding the gold medal threshold of approximately 360 points. The evidence is direct and unambiguous—the model participated under the same constraints as human competitors (10 hours, 50 submissions per problem) and achieved a gold-medal score. The claim that no hand-crafted strategies were used is supported by the description of o3's evaluation protocol: single prompt with full problem statement, 1,024 samples, top 50 selected by test-time compute, no clustering, no subtask decomposition, no reranking.

However, the claim applies specifically to this evaluation of this model on this dataset. The paper acknowledges that the o3 IOI evaluation used "a later version of o3 than the CodeForces results" that "included additional fresher training data" (Section 4.2). This means the result is not from the exact same model that achieved 2724 on CodeForces—it is from a further-trained version. The claim of "without hand-crafted strategies" is accurate for the evaluation protocol, but the model itself benefited from additional training that may have included further coding-specific optimization. The "fresher training data" in particular could have included data that improved IOI-specific capabilities beyond what general scaling would predict.

A deeper concern: the selection mechanism for o3's IOI submissions—"the top 50 solutions with the highest test-time compute from 1,024 samples per problem"—is itself a form of hand-crafted strategy, albeit a very simple one. The decision to select by test-time compute rather than random sampling, majority voting, or best-of-N weighted by a verifier is a human design choice that encodes the prior belief that "more thinking = better solutions." While this is far simpler than o1-ioi's pipeline, it is not zero human design. The claim "without hand-crafted domain-specific strategies" would be more precise as "without the elaborate clustering, reranking, and subtask-specific engineering that characterized o1-ioi."

Claim: "Scaling general-purpose reinforcement learning, rather than relying on domain-specific techniques, offers a robust path toward state-of-the-art AI in reasoning domains."

This claim is the paper's thesis statement, and it is supported with qualifications. The evidence shows that for competitive programming (CodeForces, IOI), scaling RL training produced a model (o3) that outperforms the best available domain-specific system (o1-ioi with its full pipeline). For software engineering (SWE-bench), the same scaling produced substantial improvements (71.7% for o3 vs. 48.9% for o1). These are two distinct domains where scaled RL outperforms domain-specific engineering.

The qualifications are substantial:

The claim depends on the scale of RL training that is achievable. The paper provides no quantification of the RL training compute for o3, stating only that it is "significantly greater" than o1. Without knowing the magnitude of "significantly greater," we cannot assess whether the approach is practical or merely possible. If o3 requires 100× the training compute of o1 to achieve a 1.6× improvement in CodeForces rating (1673 → 2724), organizations with limited compute budgets might rationally prefer the domain-specific approach, which requires less training but more engineering.
The claim is tested on only two reasoning domains (competitive programming and software engineering), both of which are code-generation tasks with objective correctness criteria. The paper does not demonstrate that scaling RL general-purpose training outperforms domain-specific techniques on mathematical reasoning, scientific discovery, strategic planning, legal analysis, medical diagnosis, or any other reasoning domain. The phrase "reasoning domains" in the claim overstates the evidence, which is specific to code-centric tasks.
The domain-specific system (o1-ioi) was limited by its architecture, not necessarily by the inherent ceiling of domain-specific approaches. o1-ioi's pipeline enforced subtask decomposition, which prevented the model from discovering unified solutions that cover multiple subtasks. A domain-specific system that did NOT enforce subtask decomposition might have performed better. Similarly, o1-ioi's clustering required 256 generated test inputs per subtask—a more sophisticated test generation strategy (e.g., using the model to generate adversarial test cases targeting specific edge cases) might have improved selection accuracy. The paper compares o3 against one particular domain-specific system, not against the best possible domain-specific system.
o3's performance, while impressive, still falls short of the best humans. Figure 10 shows that the top human competitors solve over 85% of problems while o3 solves substantially fewer (its solve rate across the 12 CodeForces contests can be estimated from Table 1 at roughly 60–70%, depending on how "solve" is defined). The gap between o3 and the world's best competitive programmers is real and significant, meaning that the claim of "state-of-the-art AI" is qualified by "but not yet state-of-the-art among humans in this domain."

Missing Experiments and Evaluations

Several experiments would substantially strengthen the paper's claims but are absent:

Training compute scaling curves: The paper mentions Figure 2 as showing that "scaling RL training and extending test-time inference led to marked gains" on competitive mathematics, but does not provide analogous curves for competitive programming or software engineering. Without scaling curves showing how CodeForces rating or SWE-bench accuracy improve as a function of RL training FLOPs, we cannot assess whether o3's performance represents a predictable outcome of scaling or a fortunate inflection point.
o3 on HackerRank Astra: The paper reports o1 and o1-preview on Astra, but not o3. Given that Astra lacks public test cases (preventing the verify-and-refine loop), o3's performance would directly test whether the learned verification strategies transfer to domains where code execution feedback is unavailable. If o3 shows substantial improvement on Astra over o1, it strengthens the transfer claim; if o3 plateaus, it suggests the verification strategy is domain-specific.
Controlled ablation of RL training scale vs. hand-crafted strategy: An experiment where the base o1-ioi model (without the hand-crafted pipeline) is evaluated with increasing amounts of additional RL training would show the crossover point where learned strategies overtake hand-crafted ones. Without this, we cannot distinguish between "o3's RL training was sufficient" and "o3's RL training was far more than necessary"—perhaps the crossover happens at much lower training budgets, making the approach more practical than the paper implies.
Comparison against an improved domain-specific system: An o3-base model with a hand-crafted strategy added (the reverse of the paper's experiment) would test whether domain-specific techniques are made obsolete by learned reasoning or whether they remain additive. If o3 + hand-crafted strategy outperforms o3 alone, domain-specific engineering retains value; if the hand-crafted strategy adds nothing (or hurts), the case for obsolescence is stronger.
Variance estimates and statistical significance: The paper reports single-point estimates for all metrics. For SWE-bench, it averages over 3 trials but does not report standard deviations, confidence intervals, or test-retest reliability. For CodeForces, the rating is estimated from 12 contests, but uncertainty in the rating estimation (e.g., bootstrap confidence intervals) is not reported. Given that the paper's central argument rests on comparing point estimates across models (e.g., 2214 vs. 2724), knowing whether these differences are statistically significant matters for interpreting the results.
Failure analysis on IOI: The paper provides full code submissions for o1-ioi's IOI solutions (Appendix C) but does not analyze o3's IOI failures. Understanding which subtasks o3 failed on, and why (e.g., time limit exceeded, wrong answer, algorithm fundamentally incapable of solving the subtask), would illuminate the ceiling of the learned-reasoning approach. If o3's failures are on problems requiring novel algorithmic insights rather than careful implementation, it suggests that RL-trained reasoning still struggles with genuine creativity.

Conditions Where the Claims Hold

Based on the evidence presented, the paper's claims about learned reasoning replacing domain-specific engineering hold under the following conditions:

The reward signal is clean and objective. Competitive programming and software engineering benchmarks provide binary or near-binary correctness signals (code passes tests or doesn't). The paper's claims do not extend to domains where correctness is ambiguous, multi-dimensional, or subjective (e.g., creative writing, dialogue generation, strategic planning with delayed rewards).
The training distribution contains problems where verification is both feasible and nontrivial. The emergence of brute-force verification in o3 likely depends on training problems where (a) the provided test cases sometimes miss bugs, creating pressure to develop alternative verification, and (b) a brute-force reference solution is feasible within the model's execution environment, enabling the verification strategy to be discovered through RL. In domains where no such reference solution exists (e.g., proving mathematical theorems where a simpler-but-correct proof is not available), learned verification may not emerge.
Sufficient RL training compute is available. The paper provides no evidence about the minimum RL training budget required for learned strategies to overtake hand-crafted ones, but the fact that o3 required "significantly greater compute resources than o1" suggests the threshold is high. Organizations without access to o3-scale training infrastructure may find hand-crafted strategies more cost-effective.
The model has access to code execution at both training and inference time. The verify-and-refine loop fundamental to o3's reasoning depends on being able to execute code and observe outputs. In domains without executable outputs—or where execution is expensive, slow, or unsafe—the learned strategies that drive o3's performance may not be applicable.
The problems are within the model's fundamental capability range. The paper does not claim that learned reasoning can solve problems that are categorically beyond the model's knowledge—only that it improves the model's ability to produce correct solutions to problems it has the underlying knowledge to solve. For entirely novel algorithmic paradigms or mathematical structures not represented in training, neither hand-crafted nor learned strategies would be expected to help.

6. Limitations and Trade-offs

6.1 The Hand-Crafted vs. Learned Comparison Confounds Model Scale, RL Training Budget, and Inference Strategy into a Single Uncontrolled Variable

The assumption or constraint. The paper's central empirical argument—that learned reasoning strategies (o3) surpass hand-crafted test-time pipelines (o1-ioi)—rests on comparing two systems that differ along multiple axes simultaneously: o3 was trained with "significantly greater compute resources than o1" (Section 4.1), is a later model version with potentially different architecture or scale, received "additional fresher training data" for the IOI evaluation (Section 4.2), and operates without subtask decomposition or clustering. The paper provides no quantification of the RL training FLOPs for either o1-ioi or o3, making it impossible to determine whether the performance difference is attributable to the removal of hand-crafted strategies, the scaling of RL training, or the improvement in base model quality from additional pretraining or architectural changes in the later o3 checkpoint.

The consequence. The core comparison driving the paper's thesis—"scaling general-purpose reinforcement learning, rather than relying on domain-specific techniques, offers a robust path toward state-of-the-art AI" (Section 6)—is underidentified. We do not know whether hand-crafted strategies become obsolete at some RL training budget threshold, or whether they would be additive at any scale if combined with an o3-quality base model. An equally plausible interpretation of the results is: better models benefit less from hand-crafted pipelines because their raw output quality is higher, but the pipelines themselves remain useful. The paper cannot distinguish between "hand-crafted strategies are categorically superseded by learned reasoning" and "hand-crafted strategies add value at all scales, but o3's base capability is so high that the pipeline's marginal benefit is smaller and was not tested."

What evidence exists in the paper. The paper provides no controlled experiment where model quality and inference strategy are varied independently. The comparison is strictly o1-ioi (with pipeline) vs. o3 (without pipeline), where o3 has both more RL training and no pipeline. There is no evaluation of o1-ioi's base model with o3-scale RL training (would it match o3?), no evaluation of o3 with hand-crafted strategies added (would it exceed 395.64 on IOI?), and no scaling curves showing how the benefit of the hand-crafted pipeline changes as a function of model capability or RL training budget. Figure 2, which shows scaling behavior, is described only for "competitive mathematics" and is not reproduced for competitive programming or coding tasks. The paper does not report the pass@1 of raw o1-ioi or raw o3 on the IOI problems—we see only the results after the full pipeline (o1-ioi) or after simple test-time compute selection (o3), making it impossible to isolate how much of o3's advantage comes from better generation vs. better implicit selection through its chain-of-thought.

Mitigation status. Not addressed. The paper acknowledges the asymmetry obliquely by noting that o3 used "significantly greater compute resources" and a "later version" with "additional fresher training data," but treats these as features of the approach rather than confounds that weaken the comparison. No suggestion is made for future work to disentangle the contributions. Practitioners reading this paper cannot determine whether their resource-constrained setting (where o3-scale training is infeasible) would benefit more from investing in hand-crafted pipelines or from maximizing whatever RL training they can afford.

6.2 The Difficulty Estimation and Selection Mechanism That o3 Uses Internally Is a Black Box, with No Characterization of When It Fails or How It Compares to External Alternatives

The assumption or constraint. For the IOI 2024 evaluation, o3's solutions are selected by taking "the top 50 solutions with the highest test-time compute from 1,024 samples per problem" (Section 4.2). The paper provides no definition of "test-time compute" beyond this phrase—it could mean number of tokens in the chain of thought, total FLOPs consumed during generation, wall-clock time, or some learned scoring of deliberation effort. The paper also provides no analysis of whether this selection criterion is optimal, how it compares to alternative selection methods (majority voting, PRM-based scoring as used in o1-ioi, or even random selection), or in what regimes it succeeds versus fails. For the CodeForces evaluation, "pass@10 (ranking 1162)" appears in Table 1 without specifying what the ranking criterion is—again, presumably test-time compute, but this is not stated.

The consequence. The paper's claim that o3 achieves gold-medal performance "without hand-crafted domain-specific strategies" (Section 4.2) is technically true but misleading in one important respect: the selection of which 50 out of 1,024 solutions to submit is itself a hand-crafted strategy. It is a simpler one than o1-ioi's pipeline, but it encodes a human prior ("solutions where the model thought harder are better") that may not hold universally. Without knowing how sensitive o3's IOI score is to the selection criterion, we cannot assess whether the result depends on this specific human choice. If random selection from the 1,024 samples would score substantially lower—as it did for o1-ioi, where random 50 submissions scored 156 vs. 213 with the pipeline (Figure 4)—then o3's success partly relies on a human-designed selection heuristic, just a much simpler one. If random selection would score similarly (suggesting o3's per-solution quality is uniformly high), then the claimed obsolescence of selection pipelines is fully supported—but this is not tested.

Furthermore, the "test-time compute" criterion is opaque in a way that matters for deployment. A practitioner cannot reproduce the selection process without knowing exactly how "test-time compute" is measured. If it involves internal model metrics (token counts, probability scores) that are accessible to OpenAI but not available through a standard API, the reported results are not independently verifiable or deployable by third parties.

What evidence exists in the paper. None. The paper does not report:

The definition of "test-time compute" or how it is quantified.
The IOI score that would result from random selection of 50 solutions, majority voting, or any alternative selection method.
The distribution of test-time compute values across the 1,024 samples—are there clear outliers, or is the distribution continuous?
The correlation between test-time compute and actual correctness on problems where ground-truth correctness is known (e.g., the CodeForces contests where the full test suite is available).
Whether "highest test-time compute" was chosen because it performed best in retrospective analysis, or because it was the a priori design choice.

Mitigation status. Not addressed. The paper treats "top 50 solutions with the highest test-time compute" as a natural default rather than a methodological choice that requires justification. No ablation of selection criteria is performed for o3, and no comparison is made between o3's implicit selection (through chain-of-thought) and the external selection pipelines of o1-ioi. Future work would need to characterize whether test-time compute is a reliable proxy for solution quality across problem types, difficulties, and model versions, and whether it generalizes to domains without code execution feedback.

6.3 The Paper's Claims Depend on a Single Benchmark Family (Competitive Programming) with No Evidence of Transfer to Non-Code Reasoning Domains

The assumption or constraint. The paper's conclusion—that "scaling general-purpose reinforcement learning offers a robust path toward state-of-the-art AI in reasoning domains" (Section 6)—extrapolates from two evaluation domains (competitive programming and software engineering) to "reasoning domains" broadly. Both domains are code generation tasks with objective, automatically-verifiable correctness criteria (test cases, GitHub issue resolution). The paper does not evaluate o3 on mathematical reasoning (beyond the passing mention of Figure 2 for competitive mathematics, which shows a different model), scientific reasoning, strategic planning, logical deduction, medical diagnosis, legal analysis, or any non-code reasoning task.

The consequence. The transfer claim rests on an untested assumption: that the reasoning strategies which emerge from RL training on code correctness (brute-force verification, iterative debugging) are instances of a general reasoning capability that will manifest in any reasoning domain given sufficient RL training, rather than being specific to the structure of programming tasks. Programming is unusual among reasoning domains in having executable ground truth—you can run the code and see if it works. In mathematical proof, legal argumentation, or strategic analysis, there is no analogous "execute and check" operation. The brute-force verification strategy that o3 discovered (write a slow reference implementation, execute both, compare outputs) depends fundamentally on code executability. It is unclear what the analog would be in, say, mathematical theorem proving (write a simpler proof of a special case? That doesn't verify the general case). If the learned strategies driving o3's performance are code-specific, the paper's claim about "reasoning domains" overstates the evidence.

The SWE-bench and HackerRank Astra results (Figures 8, 9) provide partial transfer evidence, but only to other code-centric tasks. SWE-bench still involves writing code that passes tests (the repository's test suite). HackerRank Astra involves building features that are evaluated against hidden test cases. Neither establishes transfer to non-code reasoning.

What evidence exists in the paper. The paper evaluates on four datasets (CodeForces, IOI, HackerRank Astra, SWE-bench Verified), all of which are code generation or code repair tasks. The qualitative evidence for reasoning strategy emergence (Figure 6) is a code verification behavior. The paper cites related work on mathematical reasoning (DeepSeek-R1, Kimi k1.5) to establish that RL-trained chain-of-thought improves math performance, but does not present its own results on mathematical benchmarks. The paper's most expansive claim—"o-series large reasoning models will unlock many new use cases for AI in science, coding, math, and many other fields" (Section 6)—lists "science" and "math" but provides no evaluation in those domains.

Mitigation status. Not addressed. The paper does not acknowledge the domain specificity of its evidence or qualify its claims to code-centric reasoning. The discussion of future applications in "science, coding, math, and many other fields" (Section 6) is presented as a natural extrapolation rather than a hypothesis requiring validation. A more careful treatment would explicitly note that the demonstrated reasoning capabilities are specific to tasks with executable verification, and that transfer to non-executable reasoning domains remains to be tested.

6.4 Training Compute and Inference Cost Are Not Quantified, Making Cost-Effectiveness Comparisons Impossible

The assumption or constraint. The paper makes a comparative claim about two approaches to achieving competitive programming performance—hand-crafted test-time strategies vs. scaled RL training—but provides no quantification of the resources required by either approach. For o1-ioi, we know it generates 10,000 solutions per subtask and runs a clustering and reranking pipeline, but the computational cost of this pipeline (FLOPs, GPU-hours, wall-clock time) is not estimated. For o3, we know it was trained with "significantly greater compute resources than o1" (Section 4.1) and generates 1,024 solutions per IOI problem, but neither the training FLOPs nor the per-solution inference cost are disclosed. The o3 system card [13] is cited but not summarized, and no numbers are extracted from it.

The consequence. A practitioner deciding whether to adopt the "scaled RL" approach or the "hand-crafted pipeline" approach cannot make a cost-informed decision. Consider three plausible scenarios, all consistent with the paper's data:

Scenario A: o3 training cost is 100× o1-ioi training cost. For a one-time competition like IOI (6 problems), the total cost is dominated by training, making o3 far more expensive per competition. For a high-throughput deployment solving millions of CodeForces-style problems, the training cost is amortized to near-zero per problem, and o3's cheaper inference (1,024 samples vs. 10,000 per subtask) might make it more cost-effective in the long run.
Scenario B: o3 training cost is "only" 5× o1-ioi training, but o3's per-solution inference cost is much higher because its chains of thought include brute-force verification and iterative debugging, consuming many more tokens per solution than o1-ioi's relatively shallow samples. The reduction from 10,000 to 1,024 solutions might be offset or reversed by per-solution cost increases.
Scenario C: o3 is strictly more expensive in both training and inference, and the only advantage is capability (it solves problems no cost-effective hand-crafted system can solve). This would reframe the paper's contribution from "learned reasoning is the efficient path" to "learned reasoning is the only known path to gold-medal performance, regardless of cost."

Without knowing which scenario is true, the paper's implicit recommendation—invest in scaling RL rather than in test-time engineering—cannot be evaluated on economic grounds.

What evidence exists in the paper. The paper reports sample counts (10,000 for o1-ioi per subtask, 1,024 for o3 per problem) and submission counts (50 for both systems at IOI, 10 for CodeForces), but no FLOP counts, GPU-hour estimates, or dollar costs. The phrase "significantly greater compute resources" (Section 4.1) is the only quantification of o3's training cost. The paper does not report chain-of-thought length distributions, token counts per solution, or inference latency for any model. For the hand-crafted pipeline, the computational cost of generating and validating 256 test inputs, executing 10,000 solutions on each of 256 test cases (2.56 million executions per subtask), and running the clustering algorithm is not estimated.

Mitigation status. Not addressed. The paper does not acknowledge the absence of cost quantification as a limitation. The word "cost" appears only in the context of the IOI submission penalty tuning ("simulating the submission process") and in the context of brute-force solutions "trading efficiency for correctness." No resource budget comparison is attempted, and no suggestion is made for future cost analysis. This is a significant gap given that the paper's practical recommendation—shift investment from test-time engineering to RL training—is fundamentally a resource allocation decision.

6.5 The Hardest CodeForces Problems Remain Essentially Unsolved by o3, with No Analysis of the Failure Modes Limiting Further Progress

The assumption or constraint. The paper presents o3's CodeForces rating of 2724 (99.8th percentile) and its gold medal at IOI 2024 as evidence that scaled RL training produces state-of-the-art competitive programming AI. However, the per-problem breakdown in Table 1 reveals that o3's performance is concentrated on easier problems, with the hardest problems in each contest remaining largely unsolved. For example: Contest 1909 G (rating 3000): 3/1162 pass@1, 0.03 pass@10—essentially never solved. Contest 1943 E2 (rating 3300): 0/1162 pass@1. Contest 1965 E (rating 3100): 0/1162. Contest 1951 I (rating 3200): 0/1162. Across the 12 contests, problems rated above approximately 3000 are almost never solved by o3, even with ranking of 1,162 samples. The paper does not analyze why these problems are unsolved—whether the model fails to understand the problem, cannot design a correct algorithm, implements the algorithm with subtle bugs that even its verification strategies miss, or times out due to computational complexity.

The consequence. The paper's narrative of steady progress (gpt-4o → o1-preview → o1 → o1-ioi → o3) implies that further scaling of RL training will continue to improve performance, eventually closing the gap with top human competitors who solve over 85% of problems (Figure 10). But the near-zero pass@1 on the hardest problems suggests a qualitative capability ceiling, not a quantitative scaling curve. If the hardest problems require algorithmic insights that o3 fundamentally cannot generate—novel applications of advanced techniques like heavy-light decomposition, centroid decomposition, fast Fourier transform on graphs, or sophisticated dynamic programming optimizations—then further RL training on existing problem distributions may not help, because the training signal never rewards the model for discovering these insights (it never produces them to be rewarded). This is the classic exploration problem in RL: if the model never randomly generates a correct solution for a problem class, RL provides no learning signal for that class.

The paper's claim about "state-of-the-art AI in reasoning domains" (Section 6) is accurate for the problems o3 can solve, but the unsolved hardest problems represent a potentially different kind of reasoning—requiring creative algorithmic design rather than careful implementation and verification—that the current RL approach may not address. The gap between o3 and top humans is not just quantitative (o3 solves 60-70% of problems, top humans solve 85%+) but qualitative: the problems o3 cannot solve are systematically the hardest ones requiring the most creative algorithmic insight.

What evidence exists in the paper. Table 1 in Appendix B provides the per-problem breakdown showing near-zero pass@1 on the hardest problems. Figure 10 shows that "the very best human competitors remain much stronger than o3, with solve rates in excess of 85%." However, the paper provides no failure analysis: no categorization of why the hardest problems are unsolved, no examples of o3's chain of thought on unsolved problems (does it recognize its own inability and give up? Does it generate confident but incorrect solutions? Does it time out?), and no discussion of whether the failure mode is insufficient exploration (never generating a correct solution to receive reward) or insufficient capability (correctly generating an algorithmic approach but implementing it with unfixable bugs).

Mitigation status. Partially acknowledged. The paper notes that "the very best human competitors remain much stronger than o3" (Appendix B.5) and that the gap exists. However, it does not treat this as a limitation of the learned-reasoning approach specifically—the same gap would presumably exist for any AI system—and does not analyze whether the failure modes on hard problems are addressable through further RL scaling or require fundamentally different training approaches (e.g., incorporating formal verification, training on proof-like reasoning traces, or using search-based exploration that goes beyond the model's spontaneous generation abilities). Future work would need to characterize the capability ceiling of RL-trained reasoning and determine whether creative algorithmic design can emerge from RL on correctness signals or requires different training paradigms.

6.6 The Evidence for Emergent Verification Strategies Is a Single Qualitative Example, Not a Systematic Behavioral Analysis

The assumption or constraint. The paper's most striking qualitative claim—that o3 spontaneously develops sophisticated test-time strategies like writing brute-force solutions to verify optimized implementations—is supported by exactly one example (Figure 6). The paper does not report: what fraction of o3's solutions exhibit brute-force verification, whether the frequency of this behavior correlates with problem difficulty or problem domain (e.g., more common in geometry vs. graph problems), whether it appears selectively on problems where the provided test cases are insufficient (the "verification is nontrivial" condition), whether o1 or o1-ioi ever exhibit similar behavior (and at what frequency), or whether solutions that employ brute-force verification are more likely to be correct than those that don't. The paper describes this as "an advanced test-time strategy discovered by o3" (Section 4.1) and frames it as evidence of emergence, but the evidence is anecdotal.

The consequence. A central pillar of the paper's argument—that learned reasoning strategies replace hand-crafted pipelines because the model internalizes the verification and selection logic—rests on the claim that o3's behavior is fundamentally different from o1 and o1-ioi in a way that accounts for its superior performance. But without quantifying this behavioral difference, the paper cannot rule out a simpler explanation: o3 is a better model that writes more correct code on the first try, and the brute-force verification example is an interesting but rare behavior that contributes negligibly to overall performance. If brute-force verification occurs in only 5% of o3's solutions, and those solutions are only marginally more likely to be correct, the "emergence of verification strategies" is a fascinating curiosity rather than the mechanism driving the 510-point CodeForces improvement.

Furthermore, the single example may be cherry-picked—selected because it is the most impressive instance of verification behavior, not because it is representative. The paper does not describe its methodology for selecting Figure 6 or state whether other examples were considered. In competitive programming, writing a brute-force solution to verify an optimized one is a well-known technique discussed in editorials, tutorials, and forum posts. If such discussions were in o3's training data, the behavior in Figure 6 could reflect the model retrieving and applying a known strategy rather than inventing it. The contamination check described in Appendix B verifies only that the test problems were not in training, not that discussions of verification strategies were absent.

What evidence exists in the paper. Figure 6 and the surrounding paragraph in Section 4.1. No quantitative analysis of verification behavior frequency, correlation with correctness, or comparison across model versions is provided. The paper does not report whether o3's chain of thought on other problems shows similar verification patterns (e.g., writing test generators, testing edge cases, comparing against alternative implementations) or whether the brute-force verification strategy is unique to the specific problem shown. The paper does not analyze the conditions under which o3 chooses to verify vs. trust its initial solution—a crucial aspect of the claimed "emergence" of conditional strategy deployment.

Mitigation status. Not addressed. The paper treats the single example as sufficient evidence for the claim that "complex test-time reasoning strategies emerged naturally from end-to-end RL" (Section 1) and that these strategies "served as a more than adequate replacement and eliminated the need for the hand-engineered clustering and selection pipelines" (Section 4.2). No suggestion is made for future behavioral analysis. A systematic study would need to: (1) define and measure verification behaviors (brute-force writing, test case generation, output comparison, edge-case analysis) across a large sample of o3's solutions, (2) establish baseline frequencies for o1 and o1-ioi to determine what is genuinely "emergent," (3) correlate verification behavior with problem characteristics and solution correctness, and (4) demonstrate that the performance gap between o3 and earlier models is partly explained by increased verification behavior rather than merely illustrated by a single example.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper forces a categorical reframing of the relationship between training-time optimization and test-time strategy design in AI reasoning systems. Before this work, the dominant paradigm for competitive programming AI—established by AlphaCode and refined by o1-ioi—treated test-time strategy engineering as an essential, non-negotiable component of the system. The assumption was that even strong models generate noisy distributions of solutions, and the path to high performance lies in generating vast numbers of candidates and filtering them through human-designed selection pipelines. The paper's core finding—that o3, with no hand-crafted test-time strategy beyond selecting by test-time compute, achieves a CodeForces rating of 2724 (510 points above o1-ioi's full pipeline) and an IOI score of 395.64 (nearly double o1-ioi's 213 under identical constraints)—inverts this assumption. The model internalizes the verification and selection logic into its generation process through RL training, rendering external pipelines unnecessary at sufficient scale.

The magnitude of this shift is paradigm-level for the subfield of AI for competitive programming but more incremental for the broader field of reasoning systems. For competitive programming specifically, the paper effectively closes the book on the AlphaCode paradigm of massive sampling plus human-engineered filtering. The demonstration that o3 achieves gold-medal IOI performance with 1,024 samples per problem (vs. o1-ioi's ~50,000+) and no subtask decomposition means that the primary engineering challenge shifts from "how do we design better filtering algorithms?" to "how do we train models that generate better solutions in the first place, and verify their own work?" This is a reallocation of research effort from inference-time heuristics to training-time optimization—a change in where intelligence is encoded rather than a change in what intelligence is possible.

For the broader reasoning field, the paper's contribution is a strong empirical data point in an ongoing debate rather than a settled conclusion. The debate is: can end-to-end RL with a correctness reward produce sophisticated, multi-step reasoning strategies that rival or exceed human-designed reasoning procedures? The paper provides compelling evidence that the answer is "yes" for code-generation tasks with executable verification. The brute-force cross-checking behavior in Figure 6—where o3 writes a slow-but-correct reference implementation, executes both programs, and compares outputs—is exactly the kind of meta-cognitive strategy that prior work attempted to elicit through prompting, tree search, or explicit verification modules. That it emerges from RL with only a correctness signal challenges the assumption that complex reasoning strategies must be explicitly taught.

The paper also resolves a latent tension in the literature between two competing narratives. The AlphaCode line (2022-2023) demonstrated that sophisticated test-time search could push model performance far beyond raw generation quality. The o1/DeepSeek-R1/Kimi k1.5 line (2024-2025) demonstrated that RL-trained chain-of-thought reasoning could dramatically improve raw generation quality. These narratives pointed in different directions—invest in inference engineering vs. invest in training—and the field lacked a direct comparison. This paper provides that comparison and delivers a clear verdict: at the scale of o3's training, the training investment dominates. Hand-crafted test-time strategies that contributed 407 CodeForces rating points to o1-ioi (from 1807 to 2214) are exceeded by the 510-point gain from further RL training that produces o3 (2214 → 2724). The test-time strategies are not worthless—they pushed o1-ioi to state-of-the-art—but they are a stepping stone, not a destination.

Several research directions become more attractive as a result of this work:

Scaling RL training for reasoning becomes the primary lever, validated by the direct comparison against hand-crafted alternatives. Work on improving RL algorithms, reward signal quality, and training efficiency for reasoning tasks is directly supported.
Understanding and inducing emergent verification behaviors becomes a central scientific question. If o3 discovered brute-force verification through RL, what other reasoning strategies might emerge with different reward structures, training distributions, or chain-of-thought formats? This opens a research program on the "natural history" of learned reasoning strategies.
Transfer of learned reasoning to non-code domains is validated as a tractable goal by the SWE-bench result (71.7% for o3, up from 33.2% for gpt-4o). The improvement suggests that reasoning capabilities developed on code correctness generalize, motivating work on domains where correctness is harder to define automatically.

Conversely, some research directions become less attractive:

Designing increasingly elaborate test-time pipelines for competitive programming is a diminishing-returns endeavor. The paper shows that the marginal benefit of sophisticated clustering, reranking, and submission scheduling—beyond simple public test filtering—was only 122 CodeForces rating points for o1-ioi, and that o3's learned strategies exceed the full pipeline's benefit. Future engineering effort is better spent on training than on inference heuristics.
Prompting-based approaches to reasoning (few-shot chain-of-thought, self-critique prompting) are further marginalized. The gulf between gpt-4o (808 CodeForces, no RL reasoning training) and o1-preview (1258, with RL reasoning training) demonstrates that prompting elicits only a fraction of the reasoning capability that RL training develops. Prompting remains useful for quick deployment but is not a path to state-of-the-art.
Training separate verifier models (the learned scoring function in o1-ioi) is called into question. If the generation model can learn to verify its own outputs through its chain of thought—as o3 does with brute-force cross-checking—the additional complexity of training, maintaining, and running a separate verifier may not be justified.

The paper also introduces a new diagnostic: the gap between a model's performance under constrained selection (50 submissions) and unlimited selection (10,000 submissions) reveals whether the bottleneck is generation quality or selection accuracy. For o1-ioi, this gap was 149 points (213 vs. 362), indicating a severe selection bottleneck—the model could generate gold-medal solutions but the pipeline couldn't identify them within 50 tries. For o3, this diagnostic is not directly measured (o3 was not evaluated with unlimited submissions), but the fact that 50 selected submissions achieved 395.64—above o1-ioi's 10,000-submission score of 362.14—implies that o3's bottleneck is much tighter: its generation quality is high enough, and its implicit selection via test-time compute is accurate enough, that the constrained-submission penalty is small. This diagnostic provides a principled way to assess where future systems need improvement.

Follow-Up Research This Work Enables

Quantifying the emergence and prevalence of learned verification strategies across model scales and problem types. The paper provides a single qualitative example of brute-force verification (Figure 6) but no systematic analysis of how frequently o3 employs this strategy, under what conditions, and whether it causally improves correctness. A strong follow-up would instrument o3 (or an open reproduction) to detect specific verification behaviors—writing a second implementation, generating additional test cases, comparing outputs across implementations, explicitly stating verification intent in the chain of thought—and measure their frequency across the 12 CodeForces contests in Table 1. The key analyses would be: (a) whether verification behavior frequency increases with problem difficulty (rating) or with problem characteristics where verification is known to be nontrivial (e.g., geometry, constructive algorithms), (b) whether solutions that exhibit verification behaviors are more likely to be correct than those that don't, controlling for problem difficulty, and (c) whether the frequency of verification behaviors increases monotonically from o1 to o1-ioi to o3, or appears suddenly at some scale threshold. If verification behavior is rare and uncorrelated with correctness, the "emergence of strategies" narrative collapses and the performance gains must be attributed to other factors (better raw code generation, longer deliberation). If verification is prevalent and causally effective, it validates the paper's central mechanism.

Determining the RL training compute threshold at which learned strategies overtake hand-crafted pipelines. The paper compares o1-ioi (with pipeline) to o3 (without pipeline) without quantifying the RL training budgets of either system, making it impossible to determine at what scale the crossover occurs. A critical follow-up would train a series of models with increasing RL compute—starting from o1 as a shared base, adding coding-specific RL in controlled increments—and evaluate each with and without the full o1-ioi hand-crafted pipeline on the CodeForces and IOI benchmarks. The key output would be a crossover plot: on the x-axis, RL training FLOPs (relative to o1 baseline); on the y-axis, CodeForces rating; with two curves (with pipeline, without pipeline) that cross at some training budget. This experiment would answer: (a) is the crossover at a training budget achievable by smaller labs, or does it require o3-scale resources?, (b) do hand-crafted strategies remain additive at all scales (the curves never converge), or do they become irrelevant beyond some threshold (the without-pipeline curve overtakes and maintains its lead)? Without this experiment, the paper's practical recommendation—invest in RL training, not test-time engineering—is untethered from any specific resource requirement.

Stress-testing learned reasoning on problems that require genuinely novel algorithmic synthesis rather than careful implementation of known techniques. The per-problem breakdown in Table 1 reveals that o3 solves essentially zero problems rated above 3000 on CodeForces, even with ranking of 1,162 samples. A rigorous follow-up would categorize the unsolved problems by the type of reasoning they require: (a) problems where the model's chain of thought indicates understanding of the problem but failure to design a correct algorithm, (b) problems where the model generates a plausible algorithm but implements it with bugs that even its verification strategies miss, (c) problems where the model recognizes its inability and explicitly gives up, and (d) problems where the model produces confident but completely incorrect solutions (hallucinating algorithmic properties). This categorization would reveal whether the ceiling is a limitation of algorithmic creativity (the RL training only reinforces patterns from training data and cannot synthesize genuinely novel algorithms), implementation reliability (the model knows what to do but can't execute it perfectly), or metacognitive awareness (the model doesn't know what it doesn't know). If the ceiling is primarily type (a)—creative algorithmic design—then further RL scaling on existing problem distributions may not help, and the field needs fundamentally different approaches (e.g., RL with search-based exploration that goes beyond the model's spontaneous generation, or training on formal proof traces that teach algorithmic reasoning rather than pattern matching). If the ceiling is type (b) or (c), further scaling of the current approach may eventually close the gap with top humans.

Evaluating whether learned reasoning strategies transfer to non-executable reasoning domains, and characterizing what analogs of "verification" emerge. The paper's transfer evidence is limited to code-generation tasks (SWE-bench, HackerRank Astra), which share the property of executable ground truth with competitive programming. A critical extension would evaluate o3 on reasoning benchmarks where correctness cannot be verified by executing code: formal mathematical proof (e.g., miniF2F, ProofNet), multi-step logical deduction (e.g., PrOntoQA, ProofWriter), and strategic planning (e.g., TravelPlanner, AlfWorld). For each domain, the evaluation would measure not just accuracy but also the presence and nature of self-verification behaviors in the chain of thought. The key question is: does o3 develop domain-appropriate verification strategies, or does it attempt to apply code-style verification (looking for an executable check) and fail? If o3 exhibits domain-appropriate verification—for instance, in mathematical proof, checking special cases of a claimed theorem, attempting to construct counterexamples, or breaking the proof into lemmas and verifying each—this would be strong evidence that RL-trained reasoning is a general capability. If o3's chain of thought in non-code domains shows confusion, repetition, or code-style verification attempts that are irrelevant, it would indicate that the learned strategies are code-specific and that general reasoning requires different training paradigms (e.g., training on multiple reasoning modalities simultaneously, or providing domain-specific verification tools beyond code execution).

Measuring the sensitivity of o3's IOI performance to the "highest test-time compute" selection criterion. The paper uses a specific, human-chosen selection heuristic—pick the 50 solutions where the model spent the most deliberation effort—but provides no ablation of this choice. A follow-up would evaluate o3 on the IOI 2024 problems under alternative selection criteria: random selection of 50 solutions, majority voting among the 1,024 solutions, selection by a simple verifier (e.g., public test case pass rate), and selection by an o1-ioi-style learned scoring function. If random selection achieves a score close to 395.64 (say, within 20 points), it means o3's per-solution quality is uniformly high and the "highest test-time compute" criterion is not load-bearing—the result is robust to any reasonable selection method. If random selection scores substantially lower (say, 300), it means the selection criterion matters and the paper's result depends on this specific human design choice, weakening the claim that o3 eliminates the need for human-crafted selection strategies. This experiment also directly tests the paper's implicit claim that "test-time compute" (deliberation effort) is a reliable proxy for solution quality—a claim that, if validated, would have implications beyond competitive programming for how we select among candidate solutions from reasoning models.

Open reproduction with a model that reports training compute, chain-of-thought traces, and per-problem solution quality. The paper's core claims are difficult to evaluate independently because o3 is a proprietary model with undisclosed training details, and the paper does not release chain-of-thought traces (beyond the single excerpt in Figure 6), per-solution correctness labels, or compute budgets. A critically important follow-up would be an open reproduction using a model where training is fully documented: start with an open-weight base model (e.g., DeepSeek-V3 or Llama 3), apply RL training for code correctness following the DeepSeek-R1 recipe, and evaluate on the same CodeForces contests and IOI problems used in this paper (or a comparable held-out set). The open reproduction would report: total RL training FLOPs, chain-of-thought length distributions, frequency of verification behaviors, per-problem pass@1 before and after RL training, and the contribution of any test-time selection strategies. This would allow the community to determine: (a) whether the paper's qualitative findings (emergent verification, superiority of learned over hand-crafted strategies) replicate in a transparent setting, (b) at what training budget the crossover occurs, and (c) whether the findings are specific to OpenAI's training infrastructure or generalize to the open-source RL-for-reasoning paradigm. The CodeForces contest format and IOI problem archive are publicly available, making this reproduction feasible for any lab with sufficient GPU resources.

Practical Applications and Downstream Use Cases

Automated competitive programming training and contest participation. The most direct application is deploying o3 (or similarly trained models) as an AI competitor in programming contests. The paper demonstrates that o3 achieves gold-medal IOI performance (395.64 points, above the ~360 threshold) and a 2724 CodeForces rating (99.8th percentile) without any contest-specific engineering. For contest organizers, this means AI systems can now participate in programming Olympiads under the same rules as human competitors—50 submissions per problem, 10-hour time limits, standard computing environments—and achieve elite results. For competitive programming training platforms (CodeForces, AtCoder, LeetCode), o3's ability to generate correct solutions with internal verification means it could serve as an automated tutor: given a problem, it produces not just a solution but a chain of thought that includes verification, explaining why the solution is correct. The per-problem breakdown in Table 1 shows that o3's pass@1 is near-perfect for problems up to rating ~2000, meaning it could reliably generate correct, verified solutions for the vast majority of practice problems that students encounter.

Automated code review and bug detection in software engineering. The verification strategy that o3 discovered—writing a simpler reference implementation and cross-checking outputs—generalizes beyond competitive programming to any software engineering task where a specification can be implemented in multiple ways. For a production codebase, a system could: (1) take a complex, optimized function, (2) prompt o3 to write a simpler, less efficient version of the same function (trading performance for obvious correctness), (3) generate a comprehensive test suite for both implementations, (4) execute both and flag any input where outputs diverge. This directly mirrors o3's behavior in Figure 6 but applied to real-world code. The SWE-bench result (71.7% for o3 on resolving GitHub issues) provides supporting evidence that the model can operate effectively on real codebases. This application is particularly valuable for safety-critical software (aerospace, medical devices, financial systems) where correctness is paramount and the cost of a bug far exceeds the computational cost of verification. The key enabler over prior approaches is that o3 autonomously decides when and how to verify—it doesn't require a human to write the reference implementation or specify verification procedures.

Data generation and filtering for self-improvement of code models. The paper demonstrates that o3 generates higher-quality solutions with fewer samples than o1-ioi (1,024 vs. ~50,000+ per IOI problem) and that its solutions include internal verification traces. This makes o3 an excellent source of training data for improving smaller, more efficient code models through distillation. A practical pipeline would: (1) collect a large set of competitive programming and software engineering problems, (2) use o3 to generate solutions with full chain-of-thought verification traces, (3) filter to solutions that pass all available test cases (ensuring correctness), (4) fine-tune a smaller model on these traces, teaching it both the solution and the verification strategy. The resulting model could approach o3's quality at a fraction of the inference cost—crucially, the verification behavior would be learned through imitation of o3's traces rather than requiring the enormous RL training budget that originally produced the behavior. This is the standard distillation pipeline, but the paper's key contribution is demonstrating that the teacher model's outputs include sophisticated reasoning strategies (not just correct answers) that can serve as rich training targets.

Competitive programming as an RL training environment for general reasoning. The paper's results establish competitive programming as a uniquely valuable training domain for reasoning models because it combines: (a) an objective, automatically-evaluable reward signal (does the code pass all tests?), (b) an environment that supports open-ended exploration (the model can write, execute, and debug code), (c) a natural curriculum of increasing difficulty (CodeForces problem ratings from 800 to 3500+), and (d) a direct comparison to human expert performance through the rating system. Organizations training reasoning models can use competitive programming performance—specifically, the CodeForces rating on temporally held-out Division 1 contests as used in this paper—as a leading indicator of reasoning capability that is cheaper and faster to evaluate than downstream tasks like mathematical theorem proving or scientific research. The paper shows that improvements on CodeForces correlate with improvements on SWE-bench and HackerRank Astra, suggesting competitive programming performance is a reasonable proxy for general coding and reasoning ability. The 12-contest Division 1 benchmark used in this paper (Table 1) provides a ready-made evaluation suite that other labs can adopt.

When to Prefer This Method

The paper articulates a clear tradeoff between two approaches to achieving strong competitive programming performance: scaling general-purpose RL training (the o3 approach) versus augmenting a weaker model with hand-crafted test-time strategies (the o1-ioi approach). The evidence supports the following decision framework:

Prefer scaling RL training when: (1) you have access to sufficient compute to train at a scale where learned verification strategies emerge—the paper shows this is achievable at o3's scale but does not quantify the minimum budget; (2) the model will be deployed across many problems, amortizing the high training cost over a large inference volume (the SWE-bench and CodeForces results demonstrate this amortization); (3) you need the system to generalize across problem formats without per-domain re-engineering (o3 handled CodeForces, IOI, and SWE-bench with no format-specific modifications); (4) the problems are within the model's capability range—o3's near-zero pass@1 on 3000+ rated CodeForces problems (Table 1) indicates that even scaled RL cannot solve problems requiring algorithmic creativity beyond the training distribution.
Prefer hand-crafted test-time strategies when: (1) RL training compute is severely constrained and you must maximize performance from a fixed model (o1-ioi's pipeline added 407 CodeForces rating points over its base model, a substantial gain without additional training); (2) the problem domain has specific structural constraints that can be exploited through engineering—o1-ioi's subtask decomposition and inheritance pruning exploited IOI's specific scoring format; (3) you are competing in a one-shot, high-stakes event (like a single IOI competition) where the training cost of a scaled model cannot be amortized, and the engineering cost of a hand-crafted pipeline is acceptable.

The paper's key insight is that this is not a permanent tradeoff: as RL training capabilities continue to scale, the crossover point moves downward in training budget, making the learned-reasoning approach accessible to more users. The paper does not quantify where this crossover lies, which is its main practical gap.