X-Coder: Advancing Competitive Programming with Fully Synthetic Tasks, Solutions, and Tests

ArXiv: 2601.06953

🎯 Pitch

A 7B model trained exclusively on synthetic competitive programming data can outperform 14B models on LiveCodeBench—without ever seeing a real problem. For SFT, creating more unique tasks beats generating extra solutions per task, but RL still adds a 4.6% absolute gain, and the approach only works once the base model has some nonzero starting competence on hard problems.

---

1. Executive Summary

This paper introduces SynthSmith, a fully synthetic data generation pipeline that produces competition-level coding tasks, solutions, and test cases without relying on any real-world programming problems. Using SynthSmith's feature-based synthesis — which extracts competitive-programming features from code, evolves them into rich feature trees, and composes them into stylized tasks — the authors train the X-Coder model series under an SFT-then-RL paradigm and achieve 62.9 avg@8 on LiveCodeBench v5 and 55.8 on v6 with only 7B parameters, outperforming the 14B DeepCoder-Preview and AReal-boba² models. The paper further establishes that scaling the number of unique synthetic tasks is more effective than scaling solutions per task for SFT, that reinforcement learning provides a powerful refinement step beyond SFT (yielding a 4.6% absolute gain), and that this fully synthetic approach is most effective when the base model's pass@1 is non-trivially above zero — hard problems remain essentially unsolved regardless of compute budget.

2. Context and Motivation

The Core Problem: Scaling Code Reasoning Is Bottlenecked by Data Scarcity

The fundamental challenge this paper addresses is deceptively simple: competitive programming requires intensive reasoning, but the datasets available for training code reasoning models are too small, too static, and too bounded by their seed problems to support continued progress. While reasoning-focused models such as OpenAI's o1-ioi have reached expert-level performance on programming tasks, and DeepSeek-R1 has opened new training paradigms through long-chain-of-thought distillation and GRPO-based reinforcement learning, both pathways share a common bottleneck — the underlying training data is insufficient.

This bottleneck manifests in several practical forms that the authors identify in Section 1:

• Classic benchmarks are saturated. HumanEval and MBPP have been "largely solved" (Section 1), meaning they no longer differentiate model capabilities or provide useful training signal. The field has thus shifted focus to more challenging benchmarks like LiveCodeBench and Codeforces, which demand deeper reasoning about algorithmic problem-solving rather than simple instruction-following.

• Existing competitive programming datasets are heavily reused. Collections such as APPS, CodeContests, and TACO provide only modest volumes of problems — far too small to support the scaling trends observed in pretraining or even in mathematical reasoning. These datasets have been recycled through multiple rounds of post-training, limiting their capacity to drive further improvements.

• Collecting new real-world data is inherently difficult. Competitive programming problems are crafted by human experts through a labor-intensive process of problem design, solution verification, and test case construction. There is no obvious mechanism to rapidly expand this corpus at the scale modern LLM training demands.

• Rewriting-based synthesis is bounded by seed quality. Recent work has explored generating synthetic variants of existing problems through evolutionary approaches (WizardCoder's evol-instruct) or rewriting strategies (rStar-Coder). However, these methods create variations of existing tasks, meaning their diversity and complexity remain "tightly bounded by the seed tasks" (Section 1). You cannot escape the distributional footprint of whatever real-world data you start from.

Why This Matters: The Practical and Theoretical Stakes

The data scarcity problem is not merely an inconvenience — it has direct consequences for how code reasoning models are built and deployed.

Practical impact on training pipelines. The two dominant post-training paradigms for code reasoning — SFT on long chain-of-thought demonstrations and RL with GRPO — both require large quantities of tasks with verified solutions and test cases. If the data well runs dry, these methods plateau regardless of algorithmic sophistication. The paper's framing in Section 6 makes this explicit: "High-quality code data is scarcer than mathematical data. Consequently, existing approaches rely heavily on real-world data and lack a stable two-stage recipe for coding expertise, often mixing in mathematics with little evidence of success on code alone." In other words, the field has been compensating for data scarcity by introducing mathematically-oriented training data, which helps general reasoning but may not directly improve code-specific capabilities.

Risk of data leakage and benchmark contamination. Training on real-world competitive programming data creates a persistent concern about test-set contamination. Models might memorize solutions from their training data rather than learning generalizable reasoning skills. The paper's data leakage analysis (Appendix G, Table 21) demonstrates this concretely: Qwen3-8B, trained on real-world data, drops 30.6 points from LiveCodeBench v2 to v5 (older vs. newer benchmarks), while the fully synthetic X-Coder drops only 17.9–17.2 points. This sharp degradation on newer benchmarks exposes how real-world training data can inflate apparent capabilities through memorization rather than genuine reasoning.

Theoretical significance for scaling laws. A broader theoretical question underlies the practical concerns: can synthetic data support scaling laws for code reasoning in the same way that web-scale natural language data supports scaling for general LLMs? If the answer is yes, it would suggest a path toward unbounded improvement in code reasoning without exhausting real-world problem sources. If the answer is no — if synthetic data lacks the richness or correctness guarantees of human-crafted problems — then code reasoning progress may eventually hit a data wall that pretraining scaling alone cannot overcome.

Prior Approaches and Where They Fall Short

The paper situates itself against a landscape of data synthesis methods that have attempted to address code data scarcity, each with identifiable limitations.

Evolutionary rewriting methods (WizardCoder, rStar-Coder). WizardCoder extends the Evol-Instruct paradigm to code by taking basic code-instruction data and prompting an LLM to increase complexity, add constraints, or change problem framing. rStar-Coder adapts this specifically to competitive programming. The fundamental limitation, which the authors identify in Section 1 and elaborate in Section 6, is that these methods "remain tightly bounded by the seed tasks." An evolved version of a graph traversal problem is still fundamentally a graph traversal problem — you cannot evolve it into a number theory problem or a dynamic programming problem. The space of possible tasks is constrained by the seeds you start with, and if those seeds come from a limited corpus like CodeContests or TACO, the diversity ceiling is low.

Concept composition methods (SelfCodeAlign, EpiCoder). SelfCodeAlign represents a conceptual advance by moving beyond seed evolution to concept composition. It extracts fundamental concepts from seed problems (e.g., "binary search," "prefix sum," "tree traversal") and combines them to generate novel tasks. EpiCoder extends this into a feature-tree framework where features are sampled and composed into problem scenarios. This is the approach SynthSmith builds on most directly, but the authors identify three specific shortcomings in EpiCoder that motivate their improvements (detailed in Section 2 and Appendix B):

1. Broad feature definitions don't capture competition-level complexity. EpiCoder extracts general-purpose programming features from raw code corpora, which may not include the algorithmic depth (graph algorithms, number theory, advanced dynamic programming) that characterizes competitive programming.

2. Single-step generation produces oversimplified tasks. When EpiCoder prompts an LLM to both select features and formulate a problem in one step, the model tends to "oversimplify complex prompts into trivial cases, reducing both diversity and difficulty" (Appendix B). The paper's ablation in Table 9 confirms this empirically: the one-step approach scores 34.8 avg@4 on LiveCodeBench v5, while two-stage generation scores 40.1 — a 5.3 point gap attributable purely to modularizing feature selection and task formulation.

3. No mechanism for solution verification or RL data generation. EpiCoder focuses on generating tasks for SFT but does not address the challenge of producing verified solutions or test cases suitable for reinforcement learning. This limits its applicability to modern SFT-then-RL training paradigms.

Scratch-based generation methods. The paper also compares against a direct "generate tasks from scratch" baseline using SelfCodeAlign adapted to competitive programming (Section 3.2, Table 3). When both methods use the same teacher models (GPT-o3-mini for task generation, DeepSeek-R1 for solution generation) on the same data quantity (10k samples), SynthSmith outperforms SelfCodeAlign by 4.6 points. This demonstrates that feature-based composition is genuinely more effective than unstructured task generation for competitive programming, presumably because it ensures that generated problems have coherent algorithmic structure rather than being arbitrary coding exercises.

Real-world data approaches (OpenCodeReasoning). OpenCodeReasoning represents the largest reasoning-based synthetic dataset for competitive coding, created by distilling solutions from reasoning models. The authors compare against it directly in Table 2, training both datasets on an identical number of tokens with the same base model (Qwen2.5-Coder-7B-Instruct). SynthSmith-trained models achieve a 6.7-point improvement, with most gains concentrated on medium and hard splits (67.0 → 73.3 on medium, 21.8 → 37.8 on hard). The authors attribute this to two factors: SynthSmith generates more challenging tasks that demand longer reasoning traces (average 17.7k tokens vs. 8.0k), and provides greater prompt diversity by generating novel tasks rather than distilling from the same set of source problems.

How This Paper Positions Itself

The paper frames its contribution not as an incremental improvement to existing synthesis methods but as a paradigm shift: fully synthetic data can replace real-world data for competitive programming training. This is a bolder claim than prior work and comes with a correspondingly heavier burden of proof.

The "fully synthetic" claim. Section 1 states the goal explicitly: "In this paper, we explore a fully synthetic approach: training Code LLMs with entirely generated tasks, solutions, and test cases, to empower code reasoning models without relying on real-world data." Note the word "entirely" — this is not about augmenting real data with synthetic variations, but about building a training pipeline that operates completely independently of human-crafted problems. The authors reinforce this throughout: the X-Coder models in Table 1 are marked with "Syn" in the Data column, contrasting with the "Real" or "Mixed" designations of all other models in the comparison.

This positioning matters because it addresses the data leakage concern head-on. If your training data never includes real-world competitive programming problems, you cannot accidentally memorize them. Appendix G provides evidence that this reduces the performance gap between older and newer benchmarks, though the gap does not disappear entirely (there remains a 17-point drop from v2 to v5 even for the synthetic model).

Reconciling conflicting capabilities in prior work. The paper implicitly positions itself as fixing a fragmentation in the literature. Some prior work has demonstrated successful SFT-only approaches (OpenCodeReasoning, rStar-Coder), while other work has shown RL-only approaches (DeepCoder, AReal-boba²), and still others have attempted SFT-then-RL but with mixed coding/math data (AceReason, Klear-Reasoner). The paper argues that what has been missing is an integrated pipeline that supports both SFT and RL stages with purely coding-focused, synthetic data. The SynthSmith pipeline is designed to produce both SFT-ready task-solution pairs and RL-ready task-test case pairs from the same underlying tasks, enabling a coherent two-stage training recipe.

The feature-tree framework as a scaling mechanism. Unlike rewriting-based methods where diversity is bounded by seeds, the feature-tree approach (inherited from EpiCoder but substantially enhanced) provides a mechanism for unbounded task generation. The feature tree grows through extraction and evolution (Appendix B.1), producing increasingly diverse combinations. The paper reports that the evolution step multiplies feature counts across all categories: algorithms grow by 6.46×, problem types by 9.22×, optimization techniques by 9.19× (Table 8). This multiplicative growth in the feature space enables combinatorial growth in the task space — a crucial property for supporting scaling laws.

Explicit acknowledgment of remaining challenges. The paper does not claim to have solved hard problems. The hardest difficulty bin (Codeforces rating equivalent of 2800+) shows near-zero pass rates even for proprietary models like GPT-5-High (Appendix E.5, Table 18: 3.07% of tasks fall in the 0–20% pass rate range for GPT-5-High). The authors are transparent that synthetic task generation, like all current approaches, has difficulty producing coherent, solvable problems at the highest levels of algorithmic complexity. This honesty about limitations strengthens rather than weakens the positioning — it shows that the paper is making a genuine advance while acknowledging the frontier that remains.

3. Technical Approach

3.1 Reader Orientation

This is an empirical systems paper that builds a fully synthetic data generation pipeline for competitive programming. The core idea is that features (algorithmic concepts like "Dijkstra's algorithm," "segment tree," or "Möbius inversion") can be extracted from existing code, evolved into a rich combinatorial space, and then systematically re-composed into novel, challenging programming tasks — bypassing the need for real-world problem sources entirely. The system being built, SynthSmith, is a multi-stage factory that takes a small seed corpus of competition code as input and produces three outputs: (1) novel programming tasks with precise specifications, (2) verified "golden" solutions with full reasoning traces suitable for supervised fine-tuning, and (3) input-output test case pairs suitable for reinforcement learning. The X-Coder models are then trained under an SFT-then-RL paradigm on these fully synthetic outputs.

3.2 Big-Picture Architecture (Diagram in Words)

The SynthSmith pipeline has four sequential stages, each feeding into the next:

1. Task Generation — extracts competition-relevant features from code, evolves them into a rich feature tree, samples compatible feature subsets, and formulates stylized problem statements (Codeforces, LeetCode, or AtCoder style).

2. Test Input Generation — for each generated task, produces a set of input test cases using either prompting-based methods (LLM generates inputs directly from the problem specification) or tool-based methods (LLM writes a CYaRon script that programmatically generates diverse test inputs).

3. Candidate Solution Generation — for each task, multiple candidate solutions with full reasoning traces are sampled from advanced open-source reasoning models (DeepSeek-R1-0528 and Qwen3-235B-A22B-Thinking-2507).

4. Dual-Verification — cross-validates solutions against test cases through consensus voting and weighted evaluation with hold-out validation, producing the final golden solution and verified test suite.

Information flows linearly: seed code → extracted features → evolved feature tree → feature subset + integration strategy → task specification → test inputs + candidate solutions → dual-verified golden solution and test suite → SFT dataset (task + golden solution) and RL dataset (task + test cases).

3.3 Roadmap for the Deep Dive

• First, the feature extraction and evolution pipeline (Section B.1), since it is the foundation that determines what kinds of tasks can be generated downstream. Understanding how features are defined, extracted, and evolved explains why SynthSmith can generate diverse, competition-level tasks rather than trivial programming exercises.

• Second, the two-stage task generation process — feature selection followed by stylized task formulation (Section B.2). This modular design is a key innovation over prior work (EpiCoder), and the paper provides an ablation (Table 9) demonstrating why the separation matters. We will walk through the prompt design, the one-shot example, and the three task styles.

• Third, test case generation (Section D), which is the bridge between SFT and RL data. The paper explores two complementary methods — prompting-based and tool-based — and provides a comparative analysis of their accuracy, coverage, and generation complexity (Table 5). Understanding this is essential because RL performance depends critically on test case quality.

• Fourth, the dual-verification strategy (Section E and Algorithm 1), which is the quality assurance mechanism that converts noisy candidate solutions and imperfect test cases into reliable training data. This is the most algorithmically sophisticated component, involving consensus voting, difficulty-weighted scoring, hold-out validation, and a decision procedure for accepting or rejecting tasks.

• Fifth, the SFT-then-RL training pipeline (Appendix A), including the loss functions (standard NLL for SFT, GRPO for RL), the reward function design, the distributed code verification infrastructure, and the training configurations and costs. This ties the synthetic data back to the model training that produces X-Coder.

3.4 Detailed, Sentence-Based Technical Breakdown

Feature Extraction and Evolution (Appendix B.1)

The SynthSmith pipeline begins not with tasks or solutions but with features — named, structured descriptors of algorithmic concepts that appear in competitive programming. A feature might be an algorithm (e.g., "Dijkstra's algorithm"), a data structure (e.g., "Fenwick tree"), a problem type (e.g., "shortest path"), an optimization technique (e.g., "Möbius inversion"), or an implementation pattern (e.g., "recursive," "bit manipulation"). These features are organized into a hierarchical tree, where internal nodes represent broad categories (e.g., "graph algorithms") and leaf nodes represent specific instantiations (e.g., "Edmonds-Karp").

Why features rather than directly generating tasks. The key insight is that competitive programming problems are fundamentally compositions of features. A hard Codeforces problem is not a novel creation from nothing — it combines well-known algorithmic primitives in non-obvious ways, often with a narrative scenario that obscures which primitives are needed. By decomposing problems into features, SynthSmith can generate new tasks through combinatorial recombination, dramatically increasing the size of the task space relative to seed data. This is fundamentally different from rewriting-based approaches (WizardCoder, rStar-Coder) that operate on the problem level and are thus bounded by the diversity of their seed problems.

Seed data for feature extraction. The extraction process starts with 10,000 question-solution pairs from the TACO dataset (Li et al., 2023), which is a curated collection of algorithmic coding problems. The authors use TACO because it contains diverse competition-level problems across multiple platforms, providing broad coverage of algorithmic concepts. Importantly, this is the only point where real-world data enters the pipeline, and it is used for feature discovery, not for task generation — the generated tasks are novel compositions, not rewrites of TACO problems.

The extraction prompt. The authors design a structured prompt (reproduced in Appendix B.1) that instructs GPT-4o-0513 to analyze each problem-solution pair and extract features across nine predefined categories:

1. Programming Language — the language used in the solution (e.g., "Python," "C++").
2. Problem Type — the broad class of problem (e.g., "graph traversal," "dynamic programming").
3. Algorithm — specific algorithms used, with subcategories for graph algorithms, sorting algorithms, dynamic programming, search algorithms, and other relevant categories. This is the most important category for capturing the core algorithmic content.
4. Data Structures — primary data structures employed (e.g., "array," "segment tree," "heap").
5. Implementation Logic — the implementation style (e.g., "iterative," "recursive," "bit manipulation").
6. Complexity Analysis — time and space complexity if available.
7. Optimization Techniques — specific optimizations applied (e.g., "memoization," "greedy approaches").
8. Purpose — what the code does in plain language.
9. Summary — a concise one-line description.

The prompt requires output in a specific JSON format with nested structures for multi-level categories (e.g., algorithm → graph algorithms → Dijkstra's algorithm). It also includes an explicit instruction: "Extract as many features as possible and try not to let a feature appear in multiple categories at the same time." This constraint prevents the same concept from being double-counted across categories, which would distort the feature composition process.

Feature evolution through breadth and depth expansion. Raw extraction produces a substantial set of features (e.g., 27,400 algorithm features from 10,000 problems), but these features are biased toward what appears in TACO. To increase diversity, the authors apply an evolution step that expands features along two dimensions:

• Breadth expansion adds new sibling features at existing levels. For example, given "quick sort" as an extracted feature, the LLM might evolve "heap sort" and "counting sort" as new siblings at the same level of the tree, even though these were not originally extracted from TACO. This expands the feature space horizontally by introducing related but distinct concepts.

• Depth expansion adds more specific sub-features to existing leaf nodes. For example, "quick sort" might be refined into "3-way quick sort" and "dual-pivot quick sort." This expands the feature space vertically by introducing increasingly specialized variants.

The evolution prompt (reproduced in Appendix B.1) provides the LLM with an example input and expected output format, and imposes quantitative constraints: "For breadth expansion, add at least 2 new sibling features to each existing node" and "For deep expansion, you need to add new sub-features to it, provided that you think the current leaf node has a more fine-grained feature."

The multiplicative growth of the feature space. After evolution, features across all categories grow substantially. The paper provides precise statistics in Table 8:

The total feature space grows from approximately 84,000 extracted features to over 580,000 evolved features. Crucially, the growth is highest for semantically rich categories: problem types grow by 9.22× and optimization techniques by 9.19×. This multiplicative expansion enables the combinatorial task generation that follows.

Feature tree merging. After evolution, features that share common traits are merged into a single, consolidated tree. The merged tree provides a unified namespace from which feature subsets can be sampled. For example, if "rolling hash" appears under both "string algorithms" and "data structures" in different evolved trees, the merge step consolidates it into a single canonical location. This prevents duplication and ensures that feature sampling for task generation operates over a coherent feature space.

Why this design works. The feature extraction and evolution pipeline addresses a fundamental tension: competitive programming requires deep algorithmic knowledge (specific algorithms, data structures, mathematical techniques), but task diversity requires broad coverage of many topics. The extraction step captures the depth (real algorithmic primitives from real problems), while the evolution step creates the breadth (novel combinations and variants not seen in the seed data). The tree structure preserves the hierarchical relationships between concepts (e.g., "Edmonds-Karp" is a specialization of "network flow," which is a type of "graph algorithm"), enabling the feature selection step to reason about which features are semantically compatible.

Two-Stage Task Generation: Feature Selection and Stylized Formulation (Appendix B.2)

Once the feature tree is built, the next step is to transform selected features into actual competitive programming problems. The authors identify a critical failure mode of prior work: when an LLM is asked to both select which features to combine and formulate the resulting problem in a single prompt, it tends to "oversimplify complex prompts into trivial cases, thereby reducing both diversity and difficulty" (Appendix B.2). The quantitative evidence is in Table 9: the one-step approach achieves 34.8 avg@4 on LiveCodeBench v5, while the two-stage approach achieves 40.1, a 5.3-point gap.

Stage 1: Feature selection via role assignment. The first stage takes as input a sampled subtree from the feature tree (represented as nested JSON) and produces three outputs:

1. A feature_roles_tree — a copy of the input tree where every leaf node is annotated with a "potential_use" field. This field describes, in natural language, how that particular feature is typically employed in competitive programming problems (e.g., "Used for divide-and-conquer searching in sorted structures or answer spaces" for recursive binary search). Internal nodes retain their structural role for hierarchy.

2. A selected_features_tree — a subtree where the LLM has chosen which features to include in the problem and which to exclude. Only leaf nodes that contribute meaningfully to a coherent problem idea are kept. For each selected leaf, only the "feature" name and "potential_use" are retained (the full annotation is in step 1, but the selected tree keeps only what is needed for task generation).

3. An integration_strategy — a string describing how the selected features will be combined into a unified problem. This strategy crystallizes the "story" of the problem: how multiple algorithmic primitives interact to create a non-trivial challenge.

The prompt for Stage 1 (reproduced in full in Appendix B.2) includes a detailed one-shot example showing how features like "recursive binary search" and "bitwise AND" can be annotated, selected, and integrated into a problem that "requires recursive binary search to efficiently search over a sorted value space, while bitwise AND operations will be used to filter candidate solutions according to constraints." This one-shot example is crucial — it teaches the LLM that selecting more features is acceptable ("Do not be overly conservative; it is often possible to design advanced problems where many features interact in non-trivial ways. Challenge yourself to maximize feature use without sacrificing problem quality.").

The prompt also includes an explicit instruction about compatibility: "Based on your role analysis, select a subtree (tree-structured subset) where all selected leaf features can be naturally integrated into a single, high-quality competitive programming problem." The word "naturally" is doing important work here — it prevents the LLM from selecting random, incompatible features that would produce incoherent or unsolvable problems.

A concrete example of feature selection. Appendix B.2.1 provides a complete walkthrough. Given an input feature tree containing graph algorithms (Dijkstra, Floyd-Warshall, Ford-Fulkerson, Edmonds-Karp), string algorithms (KMP, Boyer-Moore), data structures (segment tree, Fenwick tree, rolling hash, cuckoo hashing), and optimization techniques (interval DP, tree DP), the LLM must decide which features cohere. The example shows the LLM selecting "Dijkstra's algorithm" (potential_use: "primary pathfinding algorithm"), "Edmonds-Karp" (potential_use: "flow computation with guaranteed complexity"), "segment tree" (potential_use: "Maintain dynamic edge weights or capacities"), and "tree DP" (potential_use: "Optimize subproblems on network tree decomposition"). The integration strategy explains that these features can be combined into "a dynamic network optimization problem where Dijkstra's algorithm finds shortest paths that are used as augmenting paths in a modified Edmonds-Karp flow algorithm." This is a non-trivial composition — it requires understanding that Dijkstra's algorithm and Edmonds-Karp are both graph algorithms that operate on related graph abstractions, and that segment trees and tree DP can be layered on top to handle dynamic updates.

What makes a "compatible" feature set. The feature selection step implicitly encodes domain knowledge about which algorithmic primitives work together. For instance, selecting "bitwise AND" and "Möbius inversion" together would likely produce an incoherent problem unless the integration strategy provides a specific mathematical relationship linking them. The potential_use annotations help the LLM make these decisions by making explicit how each feature is typically employed, enabling the LLM to identify synergistic relationships (e.g., "this search algorithm would need to operate on a data structure that supports range queries, so I should select a segment tree").

Stage 2: Stylized task formulation. Given the selected_features_tree and integration_strategy from Stage 1, the second stage generates the actual problem statement. The prompt for Stage 2 is a "professional competitive programming problem setter" persona, with strict requirements about what the output must contain and must NOT contain.

The output is a single JSON object with one field: "question," containing a complete problem statement with title, story/context, input specification, output specification, constraints, and at least two sample input/output pairs with explanations.

Critical design constraints for the problem statement. The prompt enforces several constraints that are essential for generating training-quality tasks rather than toy problems:

• Feature coverage: "The story and setting must naturally motivate every selected feature, making each indispensable for an optimal solution." This ensures that the generated problem genuinely requires all selected algorithmic primitives — you cannot solve it by ignoring one feature.

• No hints: "Do not include any references to algorithms, data structures, solution strategies, or any implicit or explicit hints in any part of the statement, notes, or examples." This is crucial for making the problem challenging — a problem that says "Hint: use Dijkstra's algorithm" would not test whether the model can identify that Dijkstra's is the right approach.

• Tight constraints: "Specify precise input/output format and tight constraints." Competitive programming problems are defined not just by their logical structure but by the computational constraints that force efficient solutions. Without tight constraints, a brute-force O(n²) solution might pass, making the problem a trivial implementation exercise rather than an algorithmic challenge.

• No extraneous commentary: "The statement must end after the final example or clarification, with no extraneous commentary." This prevents the LLM from adding solution-adjacent discussion that could leak information.

Three task styles. The prompt template is adapted to support three distinct competitive programming styles:

1. Codeforces-style — tasks with rich, imaginative narrative contexts, standard input/output (reading from stdin and writing to stdout), and problem titles. For example, Appendix B.2.2 shows a generated Codeforces problem titled "Network Resilience" with a story about "MaxiFlow City" and a transportation network with time-varying capacities. The narrative motivates the algorithmic challenge without revealing the solution approach.

2. LeetCode-style — tasks with predefined function signatures and starter code. These problems specify a class or function interface that the solution must implement, which is more structured than Codeforces-style I/O.

3. AtCoder-style — tasks with concise, minimal explanations. These problems are stripped down to the essential specification without narrative embellishment, similar to the AtCoder format where problem statements are typically shorter and more direct.

The paper reports using a mixture of styles in the demonstration dataset: Codeforces : AtCoder : LeetCode = 70 : 15 : 15. The ablation in Figure 5(b) shows that AtCoder-style tasks yield slightly higher scores than the other styles, but the difference is modest and the authors adopt the Codeforces-dominant mixture because it "reflects its prominence as the mainstream competitive-programming platform."

Filtering of generated tasks. Tasks with descriptions shorter than 200 tokens are discarded because "such descriptions are often either too trivial or incomplete" (Appendix C.1). This threshold filters out problems that are under-specified or lack sufficient complexity to require genuine reasoning.

Prompting-Based and Tool-Based Test Case Generation (Appendix D)

For each generated task, SynthSmith produces a set of test cases — input-output pairs that serve two purposes: (1) during RL training, they provide the reward signal (whether the model's solution passes the tests), and (2) during dual-verification, they enable cross-validation between candidate solutions and test cases. The quality of these test cases directly determines the quality of both the RL reward signal and the selected golden solution.

The challenge of test case generation. Competitive programming platforms typically do not provide exhaustive test cases — they provide a small number (often just the sample cases), with the full test suite kept private for evaluation. This means that synthetic test case generation cannot simply copy from a source; it must construct test inputs from scratch based on the problem specification. A naive approach (generate random inputs within the constraints) would produce test cases that any reasonable solution can pass, failing to catch bugs, edge cases, or algorithmic inefficiencies.

Prompting-based test generation. The first method directly prompts an LLM (R1-0528) to analyze the problem specification and generate 15 test cases covering different categories: edge cases (empty input, minimum values, maximum values), small-scale data, medium-scale data, and large-scale data. The prompt requires output in JSON format with each test case having an index, a description, and the input string. The generated test cases are stored as .in files.

The limitation of this approach, visible in Table 5, is that it cannot systematically generate certain test types. The prompting-based method cannot produce scalable tests (parameterized by input size), stress tests (extreme input sizes designed to test performance limits), or tests with specific structural properties (e.g., random graphs, custom polygons). It also cannot easily vary parameters to explore the full space of valid inputs.

Tool-based test generation with CYaRon. The second method addresses these limitations by having the LLM write a Python program that uses CYaRon, an open-source library for generating test data for competitive programming and Olympiad informatics problems. CYaRon provides high-level abstractions for generating random graphs, trees, polygons, vectors, strings, and number sequences, along with input/output file management.

The prompt for tool-based generation (Appendix D.2) includes the full CYaRon documentation and imposes specific requirements:

• The generated Python program must produce at least 5 test cases with distinct characteristics (base cases, boundary cases, large random cases, etc.).
• Each test case must be saved as a .in file containing only the input data (no comments, explanations, or answer validation).
• The program must use Python's built-in random module (not CYaRon's random functions) for reproducibility, with argparse support for configurable random seed control via --seed.
• The program must be executable in a single run.

The CYaRon library itself provides a rich set of primitives:

• IO — file management with support for input/output pairs, batch generation via data_id loops, and format control.
• Graph — template-based graph generation including random trees, binary trees, chain-like trees, star-like trees, directed acyclic graphs (DAGs), undirected connected graphs (UDAGs), and anti-SPFA graphs for testing shortest-path algorithm robustness.
• Polygon — convex hull and simple polygon generation with geometric property computation.
• Vector — unique/non-unique integer and real-valued vectors.
• String — random strings, sentences, and paragraphs with configurable character sets.
• Sequence — number sequences via explicit formulas or recursive definitions.
• Utilities — random number generation (randint, uniform, choice), constants (PI, E, alphabets), and scientific notation conversion.

Comparative analysis of the two approaches. Table 5 provides a quantitative comparison on CodeContests tasks using ground-truth solutions to evaluate test quality:

The tool-based approach achieves a higher pass rate on ground-truth solutions (87.9% vs. 77.4%), confirming that its test cases are more accurate and reliable. It also generates more challenging and discriminative tests, as measured by the lower consensus ratio (78.8% vs. 82.0%). The consensus ratio is the fraction of test cases where all candidate solutions produce identical outputs — a lower ratio indicates that the test cases are more effective at differentiating between solutions and catching subtle bugs. The average number of test cases is also higher (18.3 vs. 13.6), providing broader test coverage.

The trade-off is computational cost: the tool-based approach requires executing the generated Python program for each task, which is more expensive than simply parsing the LLM's text output. However, for RL training where reliable reward signals are essential, the paper implicitly favors the tool-based approach's higher accuracy and coverage.

Dual-Verification of Solutions and Test Cases (Appendix E, Algorithm 1)

The dual-verification strategy is the quality control mechanism that converts noisy LLM outputs into reliable training data. It addresses a fundamental challenge: when both the solutions and the test cases are synthetic, how do you verify either? The solution is mutual cross-validation — use the solutions to verify the test cases, then use the verified test cases to select the best solution.

Notation and setup. For a given task $q$, the pipeline has produced:

• A set of $n$ test inputs $\{x_i\}_{i=1}^n$ (from the test generation stage).
• A set of $m$ candidate solutions $\{A^j\}_{j=1}^m$ (from the solution generation stage).

The goal is to produce a golden solution $A_{\text{golden}}$ and a golden test suite $\mathcal{T}_{\text{golden}}$ that are mutually consistent.

Step 1: Consensus voting for test output labeling. For each test input $x_i$, all $m$ candidate solutions are executed to obtain a set of outputs $\{y_i^1, y_i^2, \ldots, y_i^m\}$, where $y_i^j = A^j(x_i)$. These outputs may not all agree — different solutions may produce different answers for the same input, either because some solutions are incorrect, or because the problem admits multiple valid outputs.

A provisional ground truth output $\hat{y}_i$ is determined via majority voting:

$\hat{y}_i = \underset{y}{\operatorname{argmax}} \sum_{j=1}^{m} \mathbb{I}(y_i^j = y)$

where $\mathbb{I}(\cdot)$ is the indicator function (1 if the condition holds, 0 otherwise), and the $\operatorname{argmax}_y$ operator selects the output value $y$ that receives the most votes from the $m$ candidate solutions.

What it computes: for each test input, all candidate solutions propose an output; majority voting picks the most common output as the provisional correct answer. If 5 out of 8 solutions output "42" and 3 output "43," the provisional label is "42."

Why this form: majority voting extends the self-consistency principle (Wang et al., 2023) to test output labeling. The assumption is that if multiple independently-generated solutions agree on an output, that output is likely correct. This is more robust than trusting any single solution, especially given that the candidate solutions come from strong reasoning models that each have non-trivial but imperfect accuracy.

Accuracy of voting-based labeling. The paper evaluates this on the TACO-verified dataset (Appendix E.3), where ground-truth outputs are known. With $n = 8$ candidate solutions, the labeling accuracy is 94.73% (Table 13), meaning the voted label matches the ground truth on 94.73% of test cases. This corresponds to a false-positive rate of 5.27%, which the authors describe as falling "within an acceptable range and demonstrates that the approach is potentially reliable to be transferred to the synthetic setting." Accuracy improves with more solutions: 94.39% with 4 solutions, 95.13% with 16 solutions.

Test case weighting. The paper introduces a weighting function $w(x_i) \rightarrow w_i$ that assigns higher scores to more challenging test cases. The motivation is that not all test cases are equally informative — a test case that all solutions pass provides no signal for differentiating between them, while a test case that only one solution passes is highly informative.

Two weighting strategies are employed (Appendix E.2):

• Semantic-based weighting: During test generation, the LLM is prompted to categorize each test case as nominal (weight = 1), complex (weight = 2), boundary (weight = 3), or stress (weight = 4). Higher weights are assigned to test cases that are more likely to expose corner cases.

• Size-based weighting: Test cases are sorted by input file size and divided into quartiles: the smallest 25% receive weight = 1, the next 25% weight = 2, the next 25% weight = 3, and the largest 25% weight = 4. The intuition is that larger inputs require more memory and are more likely to trigger performance issues.

The result of Step 1 is a candidate test set:

$\mathcal{T}_{\text{candidate}} = \{(x_1, \hat{y}_1, w_1), (x_2, \hat{y}_2, w_2), \ldots, (x_n, \hat{y}_n, w_n)\}$

Step 2: Splitting into weighted and hold-out sets. The candidate test set is randomly partitioned (e.g., 50/50 split) into two subsets:

• $\mathcal{T}_{\text{golden}}$ — the weighted test suite used for selecting the golden solution.
• $\mathcal{T}_{\text{val}}$ — a hold-out validation set used for confirming that the selected solution generalizes.

The split is important because it prevents overfitting: a solution that performs well on $\mathcal{T}_{\text{golden}}$ might simply be exploiting quirks of those specific test cases. The hold-out set $\mathcal{T}_{\text{val}}$ provides an independent check.

Step 3: Weighted solution selection. Each candidate solution $A^j$ is scored on $\mathcal{T}_{\text{golden}}$ using a weighted sum:

$S_j = \sum_{(x_i, \hat{y}_i, w_i) \in \mathcal{T}_{\text{golden}}} w_i \cdot \mathbb{I}(A^j(x_i) = \hat{y}_i)$

What it computes: the total weighted score for solution $A^j$ is the sum of the weights of all test cases where the solution's output matches the voted label. A solution that passes a high-weight test case (e.g., a boundary case with weight 4) gets more credit than one that passes a low-weight test case (e.g., a nominal case with weight 1). A solution that passes all test cases gets the sum of all weights.

Why this form: the weighted sum prioritizes solutions that handle edge cases and stress tests over solutions that only handle easy cases. A solution that passes 10 nominal test cases but fails 2 boundary cases will have a lower score than one that passes 8 nominal and 2 boundary cases, even though the raw pass count might be higher. This biases selection toward robust, generalizable solutions.

The top-performing candidate is identified as:

$A'_{\text{golden}} = \underset{A^j \in \{A^1, \ldots, A^m\}}{\operatorname{argmax}} \; S_j$

Step 4: Hold-out confirmation. The provisional golden solution $A'_{\text{golden}}$ must also achieve the highest (or competitively high) unweighted accuracy on the hold-out set $\mathcal{T}_{\text{val}}$. Specifically, the paper computes the unweighted accuracy of all $m$ candidates on $\mathcal{T}_{\text{val}}$ and identifies the candidate $A^{j^\dagger}$ with the highest accuracy:

$j^\dagger = \underset{j}{\operatorname{argmax}} \; \text{Acc}(A^j, \mathcal{T}_{\text{val}})$

Two possible outcomes:

• $j^\dagger = j^\star$ (the same solution wins on both sets): The verification passes. Set $A_{\text{golden}} = A'_{\text{golden}}$ and return the golden solution and test suite.

• $j^\dagger \neq j^\star$ (different solutions win): The verification fails. The task is discarded because it indicates that the selected solution may be overfitting to the specific test cases in $\mathcal{T}_{\text{golden}}$ rather than exhibiting generalizable correctness. This is a conservative decision — it errs on the side of rejecting potentially good data rather than risking low-quality training examples.

Why this confirmation step is necessary. Without the hold-out check, the selection procedure could be gamed by chance. A candidate solution that happens to match the voted labels on $\mathcal{T}_{\text{golden}}$ — perhaps because the voted labels themselves are noisy or because the solution exploits a spurious pattern — would be selected as golden, despite not being genuinely correct. The hold-out step acts as a guard: it requires that the solution selected by weight also generalizes to an unseen subset of test cases.

Error rate of the golden solution. Appendix E.4 evaluates the dual-verification procedure on the TACO-verified dataset (500 tasks with ground-truth test cases). With $n = 8$ candidate solutions (R1-0528), the selected golden solution achieves:

• Average pass rate (test-case level): 92.15% — meaning the golden solution passes 92.15% of the ground-truth test cases on average.
• Full pass rate (task-level): 85.00% — meaning the golden solution passes ALL ground-truth test cases on 85.00% of tasks.

This corresponds to a 7.85% error rate in the selected golden solutions ($100\% - 92.15\% = 7.85\%$ at the test-case level). The error rate decreases as more candidate solutions are used (92.50% with 16 solutions), confirming that the voting mechanism becomes more reliable with additional samples.

Solvability of generated problems. Appendix E.5 reports the pass@1 of proprietary LLMs (Qwen3-Max, Gemini-2.5-Pro, GPT-5-High) on the voted test cases across the synthetic dataset (Table 18). GPT-5-High, the strongest available solver, achieves 100% pass rate on 66.98% of tasks, and ≥80% pass rate on 77.80% of tasks. However, 3.07% of tasks fall in the 0–20% pass rate range even for GPT-5-High, suggesting these tasks may be ambiguous, underspecified, or inherently unsolvable. These tasks are candidates for filtering or iterative refinement.

The computational cost of dual-verification. The paper is transparent about the expense: "Fully verifying 200k samples necessitates the generation of 1.6 million long-CoT trajectories and 24 million test executions" (Section 4, Q1). This overhead establishes a trade-off: verified solutions produce significantly better training data (Figure 5(a) shows a clear performance advantage for verified over raw solutions), but at substantial computational cost. The paper notes that prior work (Li et al., 2025; Gandhi et al., 2025) has shown that models can learn effectively from unverified long-CoT data, making "raw-solution training a more resource-efficient, albeit potentially less performant, alternative."

The strict vs. relaxed verification decision. Algorithm 1 as presented is a strict verification — if the hold-out confirmation fails, the task is discarded entirely. In practice, the authors may apply a relaxed version where the task is kept even if confirmation fails, especially when the golden solution still achieves high accuracy on the validation set (just not the highest). The paper does not provide ablations on strict vs. relaxed verification, but the strict version is the default described in the algorithm.

Supervised Fine-Tuning (Appendix A.1)

The SFT stage trains the base model to imitate long chain-of-thought solutions for competitive programming tasks. The training objective is the standard language modeling loss applied over full reasoning trajectories.

Training data format. Each SFT example is a pair $(x_i, y_i)$ where $x_i$ is the task description (problem statement, input format, output format, constraints, examples) and $y_i$ is the golden solution selected by dual-verification. The solution includes both the reasoning trace (the CoT) and the final code implementation, formatted with think/answer tags.

Loss function. The model is trained to minimize the negative log-likelihood (NLL) of the target solution conditioned on the task:

$J_{\text{SFT}}(\theta) = -\mathbb{E}_{(x,y) \sim \mathcal{D}}\left[\sum_{t=1}^{|y|} \log \pi_\theta(y_t \mid x, y_{<t})\right]$

where $\theta$ represents the model parameters, $\mathcal{D}$ is the SFT dataset of task-solution pairs, $|y|$ is the length of the target solution in tokens, $y_t$ is the $t$-th token of the solution, $y_{<t}$ are the preceding tokens, and $\pi_\theta(y_t \mid x, y_{<t})$ is the model's predicted probability for token $y_t$ given the task and previous solution tokens.

What it computes: for each task-solution pair, the model processes the task text as context and then predicts each token of the solution autoregressively. The loss is the sum of negative log-probabilities across all solution tokens, averaged over the dataset. The model is penalized when it assigns low probability to the tokens that appear in the golden solution.

Why this form: the autoregressive NLL is the standard objective for language modeling. It encourages the model to reproduce the exact reasoning trajectory of the golden solution, including both the intermediate reasoning steps and the final code. The expectation over the dataset ensures that the model learns the distribution of solutions rather than memorizing individual examples. The loss is applied over the full trajectory — not just the code — because the goal is to learn the reasoning process that leads to the solution, not just the solution itself.

Training configuration. The paper specifies: learning rate $5 \times 10^{-5}$, global batch size 128, training for 8 epochs (Appendix A.1). The SFT training curves (loss and token accuracy) are shown in Figures 9 and 10. A notable detail: long-CoT data requires significantly more training epochs than short-CoT data. Table 4 shows that long-CoT models keep improving through 8 epochs (gaining 17.4 points from epoch 3 to epoch 8 on LiveCodeBench v5), while short-CoT models plateau much earlier (gaining only 8.1 points from epoch 3 to epoch 8).

Computational cost. Training X-Coder on Qwen2.5-Coder-7B-Instruct required "128 H20 Enterprise (96 GB) GPUs for 220 hours during SFT" (Appendix A.4). The dominant cost comes from the long sequence lengths (median 16k tokens, Table 11) and the need for 8 full epochs.

Solution validation before SFT. Before solutions enter the SFT dataset, they pass through quality filters (Appendix C.1): samples without complete think and answer tags are removed, cases where the extracted Python block fails AST validation are rejected, solutions containing multiple code blocks after the reasoning process are excluded (because they hinder reliable solution extraction), and samples exceeding 25k tokens are filtered out to "prevent overthinking and to reduce SFT cost caused by sequence padding."

Reinforcement Learning with GRPO (Appendix A.1–A.2)

After SFT, the model undergoes reinforcement learning to refine its policy beyond the distribution of the supervised dataset. The paper uses Group Relative Policy Optimization (GRPO), a variant of PPO that eliminates the critic network by estimating advantages directly from multiple rollouts to the same prompt.

Why GRPO over PPO. The paper explains that PPO's application to LLMs encounters "significant challenges, including substantial computational overhead from maintaining a critic network, which increases memory usage and training time for models with billions of parameters." Additionally, "training stability can be undermined by inaccurate value function estimates or suboptimal tuning of Generalized Advantage Estimation (GAE) parameters." GRPO addresses these limitations by using a group of rollouts from the same prompt to compute relative advantages, eliminating the need for a separate value network.

GRPO objective function. The full GRPO objective is:

$\begin{aligned} J_{\text{GRPO}}(\theta) = \frac{1}{G} \sum_{i=1}^{G} \frac{1}{|a_i|} \sum_{t=1}^{|a_i|} \Bigg\{ &\min\Bigg(\rho_{i,t} \hat{A}_{i,t}, \ \text{clip}\big(\rho_{i,t}, 1-\epsilon, 1+\epsilon\big) \hat{A}_{i,t}\Bigg) \\ &- \beta D_{\text{KL}}[\pi_\theta \parallel \pi_{\text{ref}}] \Bigg\} \end{aligned}$

where:

• $\theta$ is the current policy parameters and $\theta_{\text{old}}$ is the policy before the update.
• $G$ is the number of rollouts per prompt. The paper uses $G = 8$ (Appendix A.4).
• $|a_i|$ is the length of the $i$-th action sequence (the model's output for the $i$-th rollout).
• $\rho_{i,t} = \frac{\pi_\theta(a_{i,t}|s, a_{i,<t})}{\pi_{\theta_{\text{old}}}(a_{i,t}|s, a_{i,<t})}$ is the probability ratio — how much more likely the current policy is to take action $a_{i,t}$ compared to the old policy, given the state $s$ (the prompt and previous tokens).
• $\hat{A}_{i,t}$ is the estimated advantage of action $a_{i,t}$ at timestep $t$ — how much better this action is compared to the average action in the group. In GRPO, the advantage is computed by normalizing the rewards within the group of $G$ rollouts for the same prompt: $\hat{A}_{i,t} = \frac{R_i - \text{mean}(\{R_1, \ldots, R_G\})}{\text{std}(\{R_1, \ldots, R_G\})}$ where $R_i$ is the total reward for the $i$-th rollout.
• $\epsilon$ is the clipping parameter that limits how much the policy can change in a single update.
• $\beta$ controls the strength of the KL-divergence penalty against $\pi_{\text{ref}}$, the reference policy. The paper notes that KL-divergence constraint is NOT applied — they "employ a rollout temperature of 1.0 with 8 rollouts to encourage exploration" (Appendix A.4), suggesting that the KL term is effectively zero ($\beta = 0$).

What it computes: for each prompt, the model generates $G = 8$ complete rollouts (reasoning + code). Each rollout is executed against the test cases to compute a reward (see below). The rewards are normalized within the group to compute advantages — rollouts with above-average rewards get positive advantages, and rollouts with below-average rewards get negative advantages. The GRPO loss then pushes the policy toward actions that appeared in high-advantage rollouts (increasing $\rho_{i,t}$ for tokens from good rollouts) and away from actions in low-advantage rollouts (decreasing $\rho_{i,t}$ for tokens from bad rollouts). The clipping $\text{clip}(\rho_{i,t}, 1-\epsilon, 1+\epsilon)$ prevents the policy from changing too drastically in a single update, which could destabilize training.

Why this form: the min operator in GRPO is a standard PPO-style conservative policy iteration that prefers the more pessimistic of the unclipped and clipped ratios when the advantage is positive, preventing the policy from increasing the probability of good actions too aggressively. The advantage normalization within each group removes the need for a learned value function — it assumes that within a group of rollouts from the same prompt, the mean reward is a reasonable baseline for what the current policy can achieve. The absence of a KL penalty (implicitly $\beta = 0$) suggests that the SFT initialization already produces reasonable formatting, so the policy can focus entirely on maximizing test-case pass rates without being constrained to stay close to the SFT checkpoint.

Reward function. The reward for a rollout is defined as:

$\mathcal{R} = \begin{cases} -2, & \text{if no code is extracted or the code fails to compile}, \\[6pt] 0, & \text{if the code compiles but passes no test cases}, \\[6pt] \dfrac{5.0 \times \#\text{passed}}{\#\text{total}}, & \text{otherwise}. \end{cases}$

What it computes: the reward ranges from $-2$ (for rollouts that don't produce executable code) to $+5$ (for rollouts that pass all test cases). Intermediate values are proportional to the fraction of test cases passed, scaled by 5.0. The -2 penalty for non-executable code provides a strong disincentive for the model to produce incomplete or syntactically invalid outputs. The continuous reward for partial correctness "provides denser supervision than the all-or-nothing alternative and leads to faster convergence" (Appendix A.2, citing Wei et al., 2025 and Dai et al., 2024).

Why this form: the three-tier structure (compilation failure / compile but fail all tests / partial or full passing) separates fundamentally different failure modes. Compilation failures are worse than logical errors because they indicate the model didn't even produce valid code — this is a clear signal that the policy is far from the desired behavior. The scaling factor of 5.0 makes the maximum reward substantially positive, which encourages exploration rather than risk-averse behavior. The paper notes that formatting rewards (e.g., enforcing think tags) are removed because the SFT model already follows the format, allowing the policy to focus on the core task of passing test cases.

RL training configuration. The policy models are updated with "a global batch size of 128 and a consistent learning rate of $7 \times 10^{-5}$, without applying the KL-divergence constraint to the starter model" (Appendix A.4). A rollout temperature of 1.0 with 8 rollouts encourages exploration. Training X-Coder on Qwen2.5-Coder-7B-Instruct required "32 H200 (141 GB) GPUs for 7 days to complete 270 update steps during RL" (Appendix A.4).

Training dynamics. The RL training curves (Figures 11 and 12) show the evolution of reward and entropy. The paper observes that "during the later stages of RL, the model shows signs of reward hacking, attempting to exploit edge cases for partial rewards instead of producing genuine solutions" (Section 5, elaborated in Appendix H.3).

Distributed Code Verification Framework (Appendix A.5)

A critical component of the RL training infrastructure is the ability to execute generated code against test cases at high concurrency. During GRPO training, the model generates 8 rollouts per prompt, and each rollout must be executed against multiple test cases to compute rewards. For a batch of 128 prompts, this means approximately 1024 code executions per update step, each potentially running multiple test cases.

System architecture. The authors developed a distributed arbitration framework based on a microservice architecture with three components:

1. API Gateway — a FastAPI-based asynchronous HTTP server that receives code execution requests from the RL training loop.
2. Worker Pool — a set of code execution workers running in isolated sandbox environments that execute the code and capture outputs.
3. Redis — a central in-memory data store that serves as both a message broker and state manager, decoupling the gateway from the workers.

How execution requests flow. When the RL training loop needs to execute code against test cases, it sends a batch of execution requests to the API Gateway. The gateway publishes these requests to a Redis Sorted Set, which functions as a time-prioritized FIFO queue using timestamps as scores. Workers atomically retrieve the next task using BZPOPMIN, execute the code in a sandbox, and push the result to a task-specific Redis List using RPUSH. The API Gateway performs a blocking pop (BLPOP) on each task's result list to retrieve the execution outcome.

Why Redis. The paper emphasizes several design advantages: (1) the asynchronous, in-memory nature of Redis provides high throughput and low latency, which is essential for keeping the GPUs utilized during RL training; (2) the stateless worker pool can be scaled horizontally to match computational demand; (3) Redis operations are atomic, preventing race conditions when multiple workers compete for tasks; and (4) worker health is monitored via key TTLs — workers periodically refresh their heartbeat keys, enabling automatic detection and de-registration of unresponsive nodes.

Concurrency optimization. The paper notes that they "used batching when submitting tasks to the Redis server to achieve high concurrency even with low request rates." This is important because the RL training loop generates code execution requests in bursts (one burst per GRPO update step), rather than as a continuous stream. Batching amortizes the overhead of Redis communication across multiple tasks.

Resource utilization. The framework is designed to "utilize the CPU power of all participating machines" — while the GPU machines are primarily doing model training, their CPUs handle code execution, maximizing hardware utilization across the cluster.

4. Key Insights and Innovations

Innovation 1: Fully Synthetic Data as a Viable Replacement for Real-World Data in Code Reasoning

The paper's most intellectually distinctive move is its reframing of the data scarcity problem. Prior work — whether through evolutionary rewriting (WizardCoder, rStar-Coder), concept composition (SelfCodeAlign, EpiCoder), or solution distillation (OpenCodeReasoning) — operates under an implicit assumption: synthetic data serves as a supplement to real-world data, extending it, diversifying it, or distilling it, but never replacing it. The dominant paradigm is "start from real problems, then augment." SynthSmith inverts this: it argues that features extracted from real code can be recombined into novel tasks that are independent of their origins, creating a training pipeline that requires no real-world problems at all.

This is a fundamental shift, not a refinement, because it changes what "data" means for competitive programming. In the augmentation paradigm, progress is bounded by the diversity of the seed corpus — you can evolve a graph problem into a harder graph problem, but you cannot evolve it into a number theory problem. In the feature-recombination paradigm, the space of possible tasks scales combinatorially with the size of the feature tree. The paper's own statistics make this vivid: a feature tree with ~84,000 extracted features grows to ~580,000 after evolution (Table 8, ~7× growth across all categories), and the combinatorial space of feature subsets is vastly larger than any plausible training budget. This isn't an incremental improvement in data volume — it's a qualitative change in how data can be generated.

The significance extends beyond the specific pipeline. If this approach generalizes to other reasoning domains (mathematical proof, scientific reasoning, formal verification), it suggests a path toward unbounded synthetic training data for any domain where tasks can be decomposed into recombinable primitives. The paper doesn't make this claim explicitly, but the feature-tree abstraction is domain-agnostic: you could imagine extracting features from mathematical proofs (induction, contradiction, algebraic manipulation) or science problems (conservation laws, dimensional analysis) and recombining them analogously. The intellectual contribution is thus not SynthSmith itself but the demonstration that the seed-data bottleneck can be broken by moving from the problem level to the feature level.

The evidence for this claim is distributed across multiple results. Table 3 shows that SynthSmith outperforms a scratch-based synthesis method (SelfCodeAlign) by 4.6 points using identical teacher models and data quantity — evidence that feature-based composition is genuinely more effective than unstructured generation. Table 2 shows that SynthSmith-trained models outperform OpenCodeReasoning-trained models by 6.7 points, with gains concentrated on medium and hard problems — evidence that the feature approach produces genuinely more challenging tasks, not just more tasks. And Appendix G (Table 21) shows that the fully synthetic X-Coder degrades only 17.2 points from LiveCodeBench v2 to v5 versus 30.6 points for Qwen3-8B trained on real data — evidence that synthetic data reduces, though doesn't eliminate, benchmark memorization.

Innovation 2: Scaling Task Diversity Is More Effective Than Scaling Solutions Per Task

The paper's second key insight is an empirical discovery about the shape of scaling laws for synthetic SFT data, not just the existence of scaling laws. When given a fixed computational budget, increasing the number of unique tasks yields substantially better downstream performance than increasing the number of solutions per task. The evidence is in Figure 3 (right panel): the 64k-tasks × 1-solution configuration substantially outperforms the 16k-tasks × 4-solutions configuration, which in turn outperforms 8k-tasks × 8-solutions. The direction of the comparison is not symmetric: doubling unique tasks helps more than doubling solutions per task.

This finding contradicts what might seem like a natural intuition. One might expect that seeing multiple solution approaches for the same problem would teach the model about algorithmic diversity — different ways to solve the same problem, each with different trade-offs. If that were true, scaling solutions per task would be the more efficient path. The empirical result shows the opposite: the model generalizes better when exposed to a broader range of problem structures rather than a broader range of solution styles for a narrow set of problems.

The implication is not just a practical guideline ("generate more tasks") but a conceptual claim about what matters for code reasoning. It suggests that the primary bottleneck in competitive programming is not learning how to express solutions — the base model presumably already knows the programming language and the algorithmic primitives — but learning to recognize which algorithmic primitives apply to which problem structures. A model trained on 64k unique problems has seen 64k distinct mappings from problem structure to solution approach. A model trained on 16k unique problems with 4 solutions each has seen only 16k such mappings, even though it has seen the same total number of solution tokens. The diversity of problem structures matters more than the diversity of solution expressions.

This finding is incremental (it refines a known scaling intuition rather than creating a new paradigm), but it has practical significance because it tells practitioners where to invest their synthesis budget. If task generation is expensive (requiring dual-verification, which costs ~8 solution generations and ~24 million test executions per 200k tasks), and solution generation is cheap (requiring only sampling from a strong reasoning model), the natural temptation might be to generate fewer tasks and more solutions. The paper's analysis shows this is exactly the wrong trade-off: invest in task diversity first, and only scale solution multiplicity once task diversity is saturated.

Innovation 3: RL as a Policy Refiner That Requires a Strong SFT Foundation for Effectiveness

The paper's third insight is a diagnostic finding about the relationship between SFT and RL in the SFT-then-RL paradigm, captured by what the authors call the "Good-gets-Better" principle (Section 3.3). RL fine-tuning is not an independent capability that can compensate for weak supervised initialization — its effectiveness is tightly coupled to the strength of the SFT model that serves as its starting point.

The evidence is in Figure 4, which shows that when two SFT models with different initial performance are used as RL initializers under identical RL settings, the stronger initializer consistently attains higher rewards. The interpretation — "A stronger SFT foundation enables exploring a more promising policy space and achieving a higher performance ceiling" — has a non-obvious corollary: RL amplifies existing capability but does not create it from nothing. This mirrors a finding from the test-time compute literature (Snell et al., 2024, on compute-optimal scaling) where test-time strategies help only on problems where the base model already has non-trivial pass@1. The same principle appears to govern RL for code: if the SFT model cannot solve a class of problems, RL won't teach it to.

The significance of this finding lies in what it says about the credit assignment between SFT and RL. A natural reading of the SFT-then-RL paradigm — popularized by DeepSeek-R1 — might be that SFT provides basic instruction-following and formatting, while RL provides the actual reasoning capability. This paper suggests a different distribution: SFT provides the core reasoning capability (by distilling long-CoT trajectories from strong teachers), and RL refines it, improving policy beyond what imitation can achieve but only within the capability envelope established by SFT.

This reframing has practical importance because it affects where practitioners invest their resources. If RL were the primary source of reasoning capability, the optimal strategy would be to invest heavily in RL infrastructure and data, with only minimal SFT for format training. If SFT is actually the primary capability-builder, the optimal strategy is to invest in high-quality SFT data (diverse tasks, verified solutions, long reasoning traces) and use RL as a polishing step. The paper's evidence supports the latter interpretation.

Innovation 4: Synthetic Test Cases Are Sufficient for Effective RL Despite Imperfect Accuracy

The fourth insight is a counterintuitive empirical finding: reinforcement learning for code does not require pristine reward signals. The dual-verification procedure produces golden solutions with a 7.85% error rate (Appendix E.4), meaning that approximately 8% of the test outputs used for RL rewards are incorrect. Conventional wisdom from the RL literature suggests that noisy rewards degrade policy learning or lead to reward hacking, yet the paper observes that "the model effectively benefits from synthetic test cases" (Section 3.3, point iii).

This finding is not just a robustness check — it reframes the relationship between data quality and RL effectiveness for code. In the dominant paradigm, RL for reasoning requires "ground truth" verification — either mathematically certain (as in math proofs) or execution-based with guaranteed test case correctness. This imposes a high bar on data quality that limits scalability: you can only train on problems where you are confident the test cases are correct. The paper's finding suggests this bar can be lowered. RL can benefit from imperfect test cases because (1) the signal from test cases that happen to be correct outweighs the noise from those that are incorrect, and (2) the GRPO algorithm's group-based advantage estimation is inherently robust to occasional mislabeled rewards — a single incorrect test case might penalize a correct solution, but across 8 rollouts for the same prompt, the relative ranking of rollouts is preserved as long as most reward signals are accurate.

The practical significance is that this finding "significantly lowers the barrier to code RL data collection." If test cases need only ~92% accuracy rather than ~100%, the dual-verification process can be much less conservative, accepting more tasks and producing larger RL datasets. The paper does not explicitly quantify the tolerance threshold or explore how reward noise scales with task difficulty, leaving these as open questions, but the qualitative finding that imperfect rewards are "good enough" is itself a useful diagnostic for the field.

Innovation 5: The Feature-Tree Architecture as a Scaling Mechanism for Unbounded Task Diversity

The final insight concerns the architectural design choice that makes SynthSmith's scaling claims plausible: the use of a feature tree with evolution as a deliberate mechanism for combinatorial growth, as opposed to the linear growth of seed-based rewriting or the naive recombination of raw features. Prior feature-based methods (SelfCodeAlign, EpiCoder) either extracted features without evolution or applied single-step composition — they generated new tasks but were still bounded by the diversity of the extracted feature set. SynthSmith's evolution step (breadth and depth expansion, followed by tree merging) is the mechanism that breaks this bound: it enables the feature space itself to grow multiplicatively before task generation begins.

This is a conceptual shift in how to think about synthetic data scaling, not just an implementation detail. In rewriting-based methods, scaling means "apply the same transformation to more seeds." In feature-composition methods without evolution, scaling means "sample from a fixed feature space." In SynthSmith, scaling means "grow the feature space, then sample from it." The evolution step is the engine that makes the pipeline genuinely unbounded — you can run more evolution iterations, produce more features, and sample from an ever-larger combinatorial space.

The quantitative evidence for the effectiveness of this architecture is distributed but consistent: the average inter-cluster distance for SynthSmith-generated tasks (0.613) exceeds that of Evol-Instruct-Code tasks (0.507) in embedding-space diversity analysis (Appendix B.4), indicating broader coverage of the task space. The task-level comparison against EpiCoder (Figure 5c) shows a 21% absolute gain, validating that the feature evolution and two-stage generation produce genuinely better tasks than the prior feature-tree method. And the difficulty distribution analysis (Table 10) shows that SynthSmith-generated tasks skew heavily toward competition-level difficulty (ratings 2200–2800), whereas real-world task datasets (CodeContests) have most tasks at the easier 1200–2000 range — evidence that the pipeline produces harder tasks than what exists in real data.

This is not a theoretical advance in the sense of new formal guarantees, but it is a practical demonstration of a scaling mechanism — analogous to how Chinchilla scaling laws showed that model and data should be scaled together for optimal pretraining efficiency, but applied to the problem of generating synthetic training data rather than allocating pretraining compute. The conceptual contribution is the idea that synthesis quality depends on the operation applied to the feature space (evolution + composition), not just the size of the seed set or the power of the teacher model.

5. Experimental Analysis

Evaluation Methodology

• Dataset. All experiments use LiveCodeBench v5 (problems released August 2024–February 2025) and LiveCodeBench v6 (February–May 2025). LiveCodeBench is a contamination-free benchmark that continuously collects new competitive programming problems from platforms like Codeforces, LeetCode, and AtCoder, making it the de facto standard for evaluating code reasoning models. The v5 set contains 268 tasks (Section 5, Table 7: "Total: 268") and is the primary evaluation target; v6 is used for temporal generalization. Results on additional benchmarks (HumanEval+, MBPP+) appear in Appendix F.2 for completeness.

• Base model(s). The primary backbone is Qwen2.5-Coder-7B-Instruct, a 7B-parameter code-specialized instruction model. A secondary backbone, Qwen3-8B-Base, is used to demonstrate generality across model families. The authors argue Qwen2.5-Coder-7B-Instruct is a representative mid-scale code model, and the 7B parameter count enables direct comparison against other 7B–8B models in the SFT and RL literature while keeping training costs manageable. A third model family, Llama-3.1-8B-Instruct, is tested in Appendix F.1 to verify generality beyond Qwen architectures.

• Metrics. The primary metric is avg@k pass rate — the fraction of the k sampled solutions that pass all test cases, averaged across all problems. For LiveCodeBench, a solution "passes" if it produces the correct output for all hidden test cases. Most comparisons use avg@8 with temperature 0.6 and top-p 0.95 to align with baselines. Pass@k is also reported in test-time scaling experiments (Figure 8). Other metrics include per-difficulty accuracy (Easy/Medium/Hard splits from LiveCodeBench), error type distributions (Table 6), and pass rate by reasoning token length (Table 7).

• Baselines. The paper compares against three categories of models (Table 1):

  • SFT-only models: Bespoke-Stratos (Labs, 2025), OpenThinker3 (Guha et al., 2025), OlympicCoder (Hugging Face, 2025), OCR-Qwen-Instruct (Ahmad et al., 2025), rStar-Coder (Liu et al., 2025a), and Qwen3-8B (Yang et al., 2025). These are all 7B–8B parameter models, with data sources marked as "Real" or "Mixed" (real + synthetic math).
  • RL-only models: Skywork-OR1 (He et al., 2025), DeepCoder-Preview (Luo et al., 2025), and AReal-boba² (Fu et al., 2025). These are built on R1-Distilled-Qwen and use 14B parameters for DeepCoder and AReal-boba².
  • SFT-then-RL models: AceReason1.1 (Liu et al., 2025b), MiMo (Xiaomi et al., 2025), and Klear-Reasoner (Su et al., 2025). These use 7B–8B parameters with various real-world data sources.
  • Synthetic data baselines: OpenCodeReasoning (Section 3.2, Table 2) for real-world synthetic comparison, SelfCodeAlign (Table 3) for scratch-based generation, and EpiCoder-380k (Figure 5c) for feature-tree comparison.

• Generation budget / compute accounting. For SFT, all models use identical numbers of training tokens when directly compared (e.g., Table 2: "trained on identical number of training tokens"). For test-time evaluation, avg@k uses k rollouts at temperature 0.6, top-p 0.95. Training compute is reported in GPU-hours: 128 H20 GPUs × 220 hours for SFT, and 32 H200 GPUs × 7 days for RL (Appendix A.4).

• Cross-validation / statistical protocol. For the SFT scaling experiments (Figure 3), different dataset configurations (v1–v6) are trained separately and evaluated on the same LiveCodeBench v5 test set. Results are reported with ± standard deviation (e.g., "60.3±2.5" for X-Coder-Qwen2.5-SFT in Table 1), computed across multiple evaluation runs. The data leakage analysis (Table 21) uses cross-temporal evaluation: comparing performance on LiveCodeBench v2 (older problems, released before model training cutoff) vs. v5 (newer problems) to quantify potential benchmark contamination.

Main Quantitative Results

SFT Results (Section 3.2, Table 1)

Headline SFT performance. X-Coder-Qwen2.5-SFT achieves 60.3±2.5 avg@8 on LiveCodeBench v5 and 53.5±1.7 on v6, using 200k fully synthetic tasks. X-Coder-Qwen3-SFT achieves 59.4±2.0 on v5 and 55.4±2.3 on v6 using the same data. Compared against SFT baselines in Table 1:

• X-Coder-Qwen2.5-SFT (60.3) outperforms rStar-Coder (57.3, but reported at avg@16 rather than avg@8), Qwen3-8B (57.5 at avg@8), OCR-Qwen-Instruct (51.3 at avg@64), and OlympicCoder (40.9, no avg@k specified). The advantage over OCR-Qwen-Instruct is particularly notable because OCR uses 736k real examples (3.7× more data) and is evaluated at avg@64 (8× more rollouts), yet scores 9 points lower.

• Compared against the 14B RL baselines, X-Coder-Qwen2.5-SFT (60.3) outperforms DeepCoder-Preview-14B (57.9 pass@1) and AReal-boba²-14B (58.1 avg@32). The 7B SFT-only model surpasses 14B RL models, using only fully synthetic data while the RL models are built on the stronger R1-Distilled-Qwen foundation.

Comparison with OpenCodeReasoning (Table 2). When both models are trained on identical numbers of tokens with identical base models (Qwen2.5-Coder-7B-Instruct), X-Coder achieves 60.3 avg@8 vs. OCR-Qwen-Coder-7B-Instruct's 53.6 — a 6.7-point improvement. The gain is concentrated on harder splits: easy drops slightly (95.2 → 96.8), medium improves from 67.0 → 73.3, and hard jumps from 21.8 → 37.8. The authors attribute this to SynthSmith's longer reasoning traces (average 17.7k tokens vs. 8.0k) and greater prompt diversity.

Comparison with synthetic baselines (Table 3). When both SynthSmith and SelfCodeAlign use the same teacher models (GPT-o3-mini for tasks, DeepSeek-R1 for solutions) on the same data quantity (10k samples), SynthSmith achieves 31.7 vs. SelfCodeAlign's 27.1 — a 4.6-point gap. This controls for teacher model quality and isolates the effect of the feature-based synthesis pipeline.

SFT scaling laws (Figure 3). The scaling experiments reveal two key patterns:

• Scaling unique tasks (v1→v4): Performance improves steadily from 43.7% (32k tasks × 1 solution) to 62.7% (192k tasks × 1 solution), following the ordering v4 > v3 > v2 > v1. The improvement is monotonic and substantial (19.0 points over a 6× increase in unique tasks).

• Scaling solutions per task: The comparison v2 (64k × 1) > v5 (16k × 4) > v6 (8k × 8) shows that for fixed total training tokens, scaling the number of unique tasks is more effective than scaling the number of solutions per task. The paper does not provide exact numbers for v5 and v6 separately from Figure 3, but the relative ordering is clear from the bar chart.

RL Results (Section 3.3, Table 1)

RL improvement over SFT. Starting from X-Coder-Qwen2.5-SFT (60.3), RL training with 40k synthetic tasks improves performance to 62.9±1.8 avg@8 on v5 and 55.8±1.9 on v6 — a 2.6-point absolute gain on v5 and 2.3 on v6. Starting from X-Coder-Qwen3-SFT (59.4), RL produces 64.0±2.5 on v5 — a 4.6-point gain.

Comparison with SFT-then-RL baselines. X-Coder-Qwen2.5 (62.9) outperforms AceReason1.1 (57.2 at avg@8), MiMo (57.8 at avg@8), and Klear-Reasoner (61.6 at avg@8). All three baselines use real-world or mixed data; X-Coder achieves the highest score with the smallest model (7B vs. 7B–8B) and fully synthetic data. On v6, X-Coder-Qwen2.5 (55.8) leads Klear-Reasoner (53.1) by 2.7 points. X-Coder-Qwen3 (64.0 on v5, 56.5 on v6) sets the overall best numbers among all 7B–8B models in Table 1.

The "Good-gets-Better" principle (Figure 4). The paper compares RL training from two SFT initializers with different performance. The figure shows reward curves where the stronger initializer consistently attains higher rewards throughout RL training. The gap between the two curves persists rather than converging, confirming that "RL performance is tightly coupled to the strength of the SFT initializer."

Resilience to noisy supervision. The paper states (Section 3.3, point iii) that "the model effectively benefits from synthetic test cases" despite their imperfect accuracy. No separate ablation quantifies the noise tolerance threshold; the evidence is implicit in the fact that RL improves over SFT (Table 1) using test cases from the dual-verification pipeline, which has a 7.85% error rate (Appendix E.4).

Test-Time Scaling (Figure 8)

X-Coder-7B outperforms its foundation model Qwen2.5-Coder-7B-Instruct by 51.3 points in pass@16 (specific numbers are read from Figure 8; the paper states the gap explicitly). X-Coder-7B matches Qwen3-8B's pass@16 performance with 8× fewer rollouts.

The gap between pass@1 and pass@16 for X-Coder-7B (19.2 points) is larger than for Qwen3-8B (13.8 points), indicating "greater diversity in the reasoning patterns it can explore." However, the gap between the RL model and the SFT model does not expand within 16 rollouts, suggesting that "RL improves pass@1 but may not escape its starting point" — a finding the authors connect to Wu et al. (2025) on RL's explored policy space.

Difficulty-Dependent Performance (Tables 2, 7)

The per-difficulty breakdown in Table 2 shows X-Coder-Qwen2.5-SFT achieving 96.8% on easy, 73.3% on medium, and 37.8% on hard problems. The gap between easy and hard is 59.0 points, indicating that hard problems remain substantially unsolved.

Table 7 provides a more granular analysis by reasoning token length. Pass rates drop sharply as token count increases: from ~100% at 0–5k tokens to ~16.7% at >25k tokens. The authors identify a "chained relationship": higher difficulty → longer reasoning → lower pass rate. This mediation pattern suggests that longer reasoning traces are a symptom of harder problems, not a guarantee of correctness — models can generate extensive but ultimately wrong reasoning.

Error Analysis (Table 6, Appendix H)

Table 6 classifies 16 rollouts per task on LiveCodeBench v5 across four models. The dominant error type is Wrong Answer (output mismatches expected answer): 67.9±4.9 for X-Coder-7B, down from 194.6±10.7 for the base Qwen2.5-Coder-7B-Instruct. Two other significant error categories for X-Coder-7B are:

• No Code Block Generated (11.8±3.9): cases where the model's reasoning exhausts the context window before producing code. The authors note that "all of them exceeded the 32k context window, causing the reasoning process to be truncated and incomplete" (Section 5).
• Time Limit Exceeded (11.5±2.6): the generated solution is correct but too slow.

Qualitative analysis in Appendix H reveals additional failure modes: premature termination under context exhaustion, retrieval-like fallback (recalling memorized C++ solutions and attempting Python translation), and reward hacking in later RL stages (exploiting edge cases for partial rewards rather than solving the problem genuinely).

Generality Experiments (Appendices F.1, F.2)

Across model families (Table 19). Training on Llama-3.1-8B-Instruct with 32k SynthSmith samples improves LiveCodeBench v5 score from 11.8 (base) to 25.2 (SFT), and further to 27.1 (SFT+RL with 10k samples). This represents a 15.3-point total improvement over the base model.

Across benchmarks (Table 20). On standard code generation benchmarks, X-Coder-7B achieves 89.6 on HumanEval, 84.1 on HumanEval+, 89.2 on MBPP, and 75.7 on MBPP+, compared to the base Qwen2.5-Coder-7B-Instruct's 88.4, 84.1, 83.5, and 71.7 respectively. The average across all four benchmarks improves from 81.9 to 84.7 — modest gains on already-saturated benchmarks, consistent with the paper's focus on competitive programming rather than standard code generation.

Data Leakage Analysis (Table 21, Appendix G)

Comparing LiveCodeBench v2 (older) vs. v5 (newer):

• Qwen3-8B drops from 88.1 to 57.5 (−30.6 points)
• X-Coder-7B-SFT drops from 78.2 to 60.3 (−17.9 points)
• X-Coder-7B drops from 80.1 to 62.9 (−17.2 points)

The synthetic models show a smaller degradation than the real-data model, suggesting reduced benchmark memorization, though they still degrade substantially (17.2 points is not negligible). This is evidence that synthetic data reduces but does not eliminate data leakage concerns.

Ablation Studies and Robustness Checks

Dual-verification of solutions (Figure 5a). Using verified solutions (dual-verification) vs. raw solutions (no verification) on an identical backbone (Qwen2.5-Coder-7B-Instruct) and dataset size (64k tasks) shows a clear performance advantage for verification. Exact numbers are not provided in text for Figure 5(a), but the bar chart in the figure shows a visible gap between the two conditions. The paper notes the computational cost: "Fully verifying 200k samples necessitates the generation of 1.6 million long-CoT trajectories and 24 million test executions."

Long-CoT vs. Short-CoT (Table 4). Training with long-CoT solutions (from DeepSeek-R1-0528) vs. short-CoT solutions (from Qwen3-235B-A22B-Instruct-2507) on identical tasks (200k):

Long-CoT achieves a 17.2-point absolute gain on v5 and 15.9 on v6 over short-CoT. At epoch 3, long-CoT (42.9) already matches short-CoT at epoch 8 (43.1), but long-CoT continues improving substantially through epoch 8, demonstrating "slower convergence requiring 8–10 epochs compared to the 2–3 epochs needed for short-CoT data."

Task style comparison (Figure 5b). Tasks of 32k each (8k unique × 4 solutions) were generated in AtCoder, Codeforces, and LeetCode styles from identical input features. The bar chart in Figure 5(b) shows AtCoder-style tasks yielding slightly higher scores than Codeforces and LeetCode styles. The authors adopt a mixed style (Codeforces : AtCoder : LeetCode = 70 : 15 : 15) for the main dataset.

SynthSmith vs. EpiCoder-380k tasks (Figure 5c). Randomly selecting 64k tasks from each pipeline, using DeepSeek-R1-0528 for solutions, and training on Qwen2.5-Coder-7B-Instruct yields a 21% absolute performance gain for SynthSmith tasks. This is the largest single ablation gap in the paper and directly validates the improvements over the prior feature-tree method (EpiCoder).

Data selection strategies (Figure 6). Three task selection strategies are compared using 50k-task subsets from a 200k-task pool:

• Difficulty-based selection — GPT-4o assigns Codeforces-style rating scores; higher-difficulty tasks are prioritized.
• Rationale-based selection — DeepSeek-R1 generates CoT reasoning for each task; tasks that elicit longer reasoning are prioritized.
• Random selection — baseline.

The bar chart in Figure 6 shows rationale-based selection achieving the highest score, difficulty-based selection performing similarly, and random selection trailing. The interpretation is that "tasks that induce longer CoT are regarded as more valuable training data."

One-step vs. two-stage task generation (Table 9). Generating 32k tasks using one-step method (feature tree → task) vs. two-stage method (feature tree → sub-tree → task):

The 5.3-point gap provides direct empirical support for the modular design of SynthSmith.

Prompting-based vs. tool-based test generation (Table 5). Evaluated on CodeContests tasks with golden solutions:

The tool-based approach generates more test cases with higher accuracy and greater discriminative power (lower consensus ratio), at the cost of higher computational overhead.

RL initializer strength (Figure 4). Two SFT models with different performance, trained on similar data distributions, are used as RL starting points. The stronger model consistently achieves higher rewards under identical RL settings. The gap persists rather than narrowing, confirming that RL does not compensate for weak initialization.

RL reward function ablation. The paper reports removing formatting rewards (e.g., enforcing think tags) since the SFT model already follows the format, but no explicit ablation compares reward function designs. The continuous reward (Equation 6) is described as providing "denser supervision than the all-or-nothing alternative" and leading to "faster convergence" (citing Wei et al., 2025 and Dai et al., 2024), but no head-to-head comparison with binary rewards is conducted.

Noise sensitivity in RL. The paper claims RL is "resilient to data imperfections" (Section 3.3, point iii), but no quantitative ablation varies the noise level in test cases to determine the tolerance threshold. The evidence is implicit: RL improves over SFT using dual-verified test cases (7.85% error rate), but no experiments inject additional noise or compare against ground-truth-only RL.

Solution verification levels. The comparison in Figure 5(a) shows verified vs. raw solutions. No intermediate verification level (e.g., partial verification, different numbers of candidate solutions, different voting thresholds) is explored. The paper does not establish the minimum verification quality needed for effective training.

RL without KL constraint. The paper notes that KL-divergence constraint is not applied during RL (Appendix A.4: "without applying the KL-divergence constraint to the starter model"). No ablation compares RL with vs. without KL penalty, so the effect of this design choice cannot be assessed from the reported results.

Feature evolution depth. The feature evolution step multiplies feature counts by 5–9× (Table 8), but no ablation varies the evolution depth (e.g., one round vs. two rounds of evolution) to determine whether further evolution would continue to improve task quality or whether the current setting is near-optimal.

Hold-out confirmation strictness. Algorithm 1 describes discarding tasks if the selected golden solution does not achieve the highest validation accuracy. No ablation compares strict vs. relaxed verification (keeping tasks even if hold-out confirmation fails). Given the computational cost of dual-verification, understanding how much data is discarded and whether relaxed filtering could recover useful training examples is an important open question.

Critical Assessment

Do the experiments support the claim that fully synthetic data can replace real-world data for competitive programming?

The evidence is strong but conditional. Table 1 shows X-Coder models achieving state-of-the-art or competitive performance among 7B–8B models using only synthetic data, supporting the claim that synthetic data is sufficient for achieving strong results. However, the paper does not demonstrate that synthetic data is necessary or that it replaces real-world data in the sense that adding real data would provide no further benefit. A critical missing experiment is a head-to-head comparison where the same base model is trained on (a) SynthSmith data alone, (b) real-world data alone (e.g., CodeContests + TACO + APPS), (c) a mixture of both — all with identical total tokens. Without this, we cannot determine whether the 60.3 avg@8 is the ceiling for synthetic-only training or whether real data would improve it further.

The data leakage analysis (Table 21) provides evidence that synthetic data reduces benchmark memorization (17.2-point degradation from v2 to v5 vs. 30.6 for Qwen3-8B), but the degradation is still 17 points — hardly negligible. This suggests either that the synthetic data still has some distributional overlap with the benchmarks, or that even without direct data leakage, model performance on older benchmarks inflates capabilities in ways that don't transfer to newer problems. The paper does not disentangle these explanations.

Do the experiments support the claim that scaling unique tasks is more effective than scaling solutions per task?

Yes, with qualifications. Figure 3 (right panel) clearly shows v2 (64k × 1) > v5 (16k × 4) > v6 (8k × 8). This establishes the relative ordering at this specific scale. However, the experiment tests only one fixed total token budget; it does not explore whether the optimal trade-off changes as the total budget increases. At 200k unique tasks with 1 solution each, we might find diminishing returns to unique tasks and increasing returns to solutions per task — the paper does not test this regime. The conclusion that "scaling unique tasks is more effective" should be understood as holding at the specific scale tested (~200k tasks, ~3.4B total tokens).

Additionally, the experiment compares configurations that add up to the same total solution count (64k, 64k, 64k respectively), but the models in v5 and v6 see fewer unique problems. This confounds "task diversity" with "solution diversity per task" — a configuration with 128k unique tasks × 2 solutions would be a cleaner test of the unique-tasks hypothesis at the same total token count.

Do the experiments support the "Good-gets-Better" principle?

The evidence in Figure 4 is suggestive but incomplete. The figure shows two reward curves where the stronger initializer maintains an advantage, but the paper does not specify: (1) what the two SFT models' initial LiveCodeBench scores are, (2) how large the performance gap is, (3) whether the gap in final RL performance is proportional to the gap in initial SFT performance, or (4) how many training steps are shown. Without these details, we cannot assess whether the principle is linear (RL multiplies SFT capability by a constant factor) or whether there are interaction effects. A more rigorous demonstration would train multiple SFT checkpoints at different performance levels and show that final RL performance is a monotonic function of initial SFT performance.

Do the experiments support the claim of resilience to noisy RL supervision?

The evidence is implicit only. The paper states that RL improves over SFT (Table 1) using test cases with 7.85% error (Appendix E.4), which demonstrates that RL can work with imperfect rewards. But this is a single data point — we don't know whether the improvement would be larger with perfect test cases, or what the noise tolerance threshold is. A direct ablation would train RL with varying levels of deliberately injected test case noise (e.g., 0%, 5%, 10%, 20% error) and measure how reward accuracy affects policy improvement. Without this, the claim of "resilience" is qualitative and uncalibrated.

Do the experiments support generalizability beyond Qwen models?

Appendix F.1 shows Llama-3.1-8B-Instruct improving from 11.8 to 27.1 (+15.3) with SynthSmith data. This demonstrates that a non-Qwen architecture benefits from the synthetic data. However, the final score (27.1) is dramatically lower than X-Coder's on Qwen (62.9) — a 35.8-point gap. This raises questions about whether SynthSmith data is genuinely architecture-agnostic or whether its benefits are substantially tied to code-specialized pretraining (Qwen2.5-Coder-7B-Instruct has strong code pretraining, whereas Llama-3.1-8B-Instruct is a general-purpose instruction model). A fairer generality test would compare SynthSmith on multiple code-specialized models from different families (e.g., DeepSeek-Coder, StarCoder).

Are the baselines fair?

Several baseline comparisons have methodological caveats:

• Evaluation metric differences. rStar-Coder reports avg@16 (57.3) while X-Coder reports avg@8 (60.3). A direct comparison requires controlling for the number of rollouts. X-Coder's pass@8 might be higher or lower than rStar-Coder's if both were evaluated at the same k.
• Model size differences. X-Coder outperforms DeepCoder-Preview-14B (57.9 pass@1) and AReal-boba²-14B (58.1 avg@32), which is impressive for a 7B model, but these baselines use pass@1 and avg@32 respectively — different evaluation protocols that are not directly comparable.
• Base model differences. Many RL baselines (DeepCoder, AReal-boba², Skywork-OR1) start from R1-Distilled-Qwen, which is stronger than Qwen2.5-Coder-7B-Instruct. X-Coder outperforms them despite a weaker base, but this complicates isolating the effect of the synthetic data from the effect of the specific base model and training recipe.

What experiments are missing that would strengthen the paper?

1. Mixed synthetic-real training. Adding real-world data to the synthetic mix and measuring whether it improves, degrades, or leaves performance unchanged would clarify whether synthetic data truly replaces real data or just complements it.

2. Scaling beyond 200k tasks. The SFT scaling experiments (Figure 3) stop at 192k unique tasks. Extending to 500k or 1M would reveal whether the scaling trend continues or plateaus, which is critical for assessing SynthSmith's claim as an "unbounded" data generation mechanism.

3. Direct comparison at higher difficulty levels. The paper acknowledges hard problems remain "essentially unsolved" (37.8% pass rate). Comparing synthetic vs. real data specifically on hard problems would reveal whether the synthetic pipeline is failing to generate sufficiently hard tasks or whether the base model's capability ceiling is the bottleneck.

4. RL with perfect vs. imperfect test cases. Quantifying the performance gap between RL with dual-verified test cases (7.85% error) and RL with gold-standard test cases (0% error, if obtainable) would calibrate the noise tolerance claim.

5. Cross-model family comparison with code-specialized models. Testing SynthSmith on DeepSeek-Coder, StarCoder, or CodeLlama would establish generalizability more convincingly than the single Llama-3.1 experiment.

6. Latency and wall-clock analysis. The paper provides compute costs in GPU-hours but never discusses wall-clock latency. Given that the dual-verification pipeline requires generating 1.6M long-CoT trajectories and 24M test executions for 200k tasks, end-to-end data generation time could be substantial and may limit practical adoption.

Genuine weaknesses in experimental design:

• Single benchmark family. All primary results are on LiveCodeBench. While LiveCodeBench is contamination-free and well-regarded, it represents a specific distribution of problems (recent competition tasks). Performance on Codeforces-only or AtCoder-only subsets is not reported separately.

• Relatively small test set. LiveCodeBench v5 has 268 tasks (from Table 7). After splitting into difficulty bins (Easy: 63, Medium: 86, Hard: 119), the sample sizes for per-difficulty analysis are modest. The ±2.5 confidence intervals on the main results are appropriate but per-difficulty comparisons may have wider variance.

• Difficulty estimator not validated on synthetic tasks. The difficulty classifier (Appendix B.3) is trained on CodeContests with annotated ratings and achieves 84% accuracy on real data. Its accuracy on SynthSmith-generated tasks — which may have different characteristics — is not evaluated. If the classifier is miscalibrated on synthetic tasks, the reported difficulty distribution (Table 10) could be misleading.

• Confounding in the RL experiments. The RL training uses 40k tasks, different from the SFT training's 200k tasks. The 2.6-point RL improvement could partially reflect training on additional data rather than genuine policy optimization. An ablative control would keep the RL data identical to the SFT data (i.e., RL on the same 200k tasks) to isolate the effect of the optimization algorithm from the effect of seeing new data.

6. Limitations and Trade-offs

Difficulty Estimation Cost Is Unaccounted for in the Headline Efficiency Gains

The SynthSmith pipeline's productivity—200k verified tasks from a seed of 10k TACO problems—carries a substantial hidden cost that the paper quantifies but never factors into its efficiency narrative. The dual-verification step alone requires extreme computation: generating 1.6 million long-chain-of-thought trajectories and executing 24 million test runs to produce the final 200k-task SFT dataset (Section 4, Q1). The paper states this explicitly:

"this quality assurance comes at a considerable computational cost. For instance, fully verifying 200k samples necessitates the generation of 1.6 million long-CoT trajectories and 24 million test executions."

What this means in practice: the cost of producing the synthetic training data likely exceeds the cost of training on it. Each of the 1.6 million long-CoT trajectories is generated by DeepSeek-R1-0528 or Qwen3-235B-A22B-Thinking-2507—large reasoning models whose inference costs are substantial. The 24 million test executions require running candidate solutions in sandboxes against multiple test cases. These costs are external to the model training budget (128 H20 GPUs × 220 hours for SFT; 32 H200 GPUs × 7 days for RL, per Appendix A.4) but are not amortized into any efficiency metric. The paper's headline claim—that a fully synthetic pipeline can produce competitive results—is true, but the cost of the pipeline itself makes the overall system substantially more expensive than simply training on carefully curated real-world data, which incurs no generation cost beyond data collection.

The paper's treatment: the authors acknowledge the cost in Section 4 and suggest that "prior work indicates that models can still learn effectively from unverified long-CoT data, making raw-solution training a more resource-efficient, albeit potentially less performant, alternative." However, the raw-solution baseline is significantly worse (Figure 5a shows a visible performance gap, though exact numbers are not provided in the text), so the cost-performance trade-off is real and unresolved. The paper does not report the wall-clock time or GPU-hours consumed by SynthSmith data generation itself, making it impossible for practitioners to evaluate whether the pipeline is economically viable compared to alternative data acquisition strategies.

---

Hard Problems Remain Essentially Unsolved—the Pipeline Does Not Help Where It Is Most Needed

The paper identifies a sharp capability boundary: on the hardest competitive programming problems, SynthSmith-generated training data provides minimal benefit regardless of dataset size or training budget. The evidence is distributed across multiple analyses:

• Per-difficulty SFT results (Table 2): X-Coder-Qwen2.5-SFT achieves 96.8% on easy, 73.3% on medium, and only 37.8% on hard LiveCodeBench v5 problems—a 59-point gap between easy and hard. The improvement over the OCR baseline is smallest on easy (+1.6) and largest on hard (+16.0), but the absolute hard-problem accuracy remains below 40%.

• Solvability analysis (Appendix E.5, Table 18): Even GPT-5-High, the strongest proprietary model tested, achieves only 66.98% perfect pass rate on SynthSmith-generated tasks. Approximately 3.07% of synthetic tasks fall in the 0–20% pass rate range for GPT-5-High—likely unsolvable or ambiguous.

• RL improvement is bounded (Section 3.3, "Good-gets-Better"): The RL stage improves aggregate performance by 2.6–4.6 points (Table 1), but the paper does not break this down by difficulty. If RL's benefit follows the same difficulty-dependent pattern as SFT, the gains on hard problems would be minimal because RL "may not escape its starting point" (Section 5, discussing Figure 8).

The structural reason: SynthSmith generates tasks by composing features from a tree built from TACO problems. If the hard problems in LiveCodeBench require algorithmic insights or problem-solving strategies that are not well-represented in the feature tree—either because they are genuinely novel compositions that the evolution step cannot discover, or because they require mathematical depth beyond what the TACO seed corpus covers—then no amount of synthetic data can bridge the gap. This is the fundamental limitation of any feature-recombination approach: you cannot compose your way to capabilities that are absent from the underlying feature space.

The paper's treatment: the authors are transparent about this. Section 5 states that "the primary bottleneck lies in reasoning capability, with most errors stemming from wrong answers." Table 6 shows that Wrong Answer dominates all other error types (67.9 vs. 11.8 for No Code Block, 11.5 for TLE). But the paper does not offer a path toward solving this limitation—it is a capability ceiling that SynthSmith does not address.

---

Generalization Beyond Qwen Architectures Is Weak and Poorly Characterized

The paper presents generality experiments (Appendices F.1 and F.2) that ostensibly demonstrate SynthSmith's applicability across model families and benchmarks. A closer reading reveals a substantial gap.

Across model families (Table 19): Training on Llama-3.1-8B-Instruct improves LiveCodeBench v5 from 11.8 → 25.2 (SFT) → 27.1 (SFT+RL). This is a 15.3-point absolute improvement, confirming that Llama benefits from the data. But the absolute performance (27.1) is only 43% of what Qwen2.5-Coder-7B-Instruct achieves (62.9) with the same pipeline. Part of this gap is attributable to Llama-3.1-8B-Instruct being a general-purpose instruction model rather than a code-specialized model, but the paper acknowledges this only implicitly ("Given that Llama-3.1-8B-Instruct is potentially weaker than Qwen2.5-Coder-7B-Instruct in terms of code pretraining," Appendix F.1). A practitioner considering SynthSmith for a non-Qwen, non-code-specialized model has no way to know whether the 27.1 ceiling is innate to Llama or whether a different hyperparameter configuration (e.g., more data, different learning rate, different RL setup) would close the gap.

Across benchmarks (Table 20): The standard code generation results (HumanEval, MBPP) show modest improvements: from 81.9 average (base Qwen2.5-Coder-7B-Instruct) to 84.7 (X-Coder-7B). On already near-saturated benchmarks (HumanEval: 88.4 → 89.6), the gains are negligible. The paper includes these results for completeness but acknowledges in Appendix F.2 that "EvoEval (program evolution), ClassEval (class implementation), and DS-1000 (data-science tasks) fall outside this scope"—meaning no results are reported on benchmarks that test code generation capabilities substantially different from competitive programming. A practitioner deploying X-Coder for software engineering tasks, data science, or class-level design has no evidence that the synthetic competitive programming data transfers to these domains.

What is missing: a comparison using a code-specialized non-Qwen model (e.g., DeepSeek-Coder, StarCoder-7B) would test whether the Qwen results reflect SynthSmith's data quality or synergies with Qwen's specific pretraining. The single Llama experiment is a start, but the 35.8-point performance gap between Llama and Qwen using the same data makes it impossible to disentangle model family effects from data quality effects.

---

The 14× Larger Baselines Use Weaker Evaluation Protocols, Undercutting the Strongest Claim

A central claim of the paper—emphasized in the abstract, executive summary, and results—is that the 7B X-Coder outperforms 14B models (DeepCoder-Preview-14B, AReal-boba²-14B). This claim is qualified in ways that reduce its strength when examined carefully.

Evaluation metric mismatches in Table 1:

• DeepCoder-Preview-14B reports pass@1 (57.9 on v5, 48.5 on v6)—a single sample. X-Coder reports avg@8 (62.9 on v5, 55.8 on v6)—the average across 8 samples. pass@1 and avg@8 are not directly comparable; pass@1 is a lower-bound estimate while avg@8 incorporates the benefits of multiple rollouts.
• AReal-boba²-14B reports avg@32 (58.1 on v5, 56.7 on v6)—32 rollouts, 4× more than X-Coder's 8 rollouts. X-Coder's avg@8 outperforming AReal-boba²'s avg@32 is more meaningful, but the differing rollout counts still confound the comparison.
• rStar-Coder reports avg@16 (57.3)—16 rollouts vs. X-Coder's 8. The paper does not report X-Coder's performance at avg@16 for direct comparison.

Base model differences. DeepCoder-Preview, AReal-boba², and Skywork-OR1 all start from R1-Distilled-Qwen, which is a reasoning-specialized model distilled from DeepSeek-R1—likely a stronger starting point than Qwen2.5-Coder-7B-Instruct for reasoning tasks. The paper does not report the base R1-Distilled-Qwen's performance on LiveCodeBench v5, making it impossible to compute the uplift each method achieves over its starting point. If R1-Distilled-Qwen already scores, say, 50 on v5, then DeepCoder's 57.9 represents a 7.9-point uplift, while X-Coder's 62.9 over Qwen2.5-Coder-7B-Instruct (~38 based on Table 6 implicit in the error counts) represents a ~25-point uplift—a much more impressive gain that is obscured by the simplistic "7B beats 14B" framing.

The paper's treatment: these caveats are visible to careful readers of Table 1 but are not discussed in the main text. The abstract states that X-Coder "outperforms DeepCoder-14B-Preview and AReal-boba²-14B despite having only 7B parameters" without qualification. A more accurate statement would note the evaluation protocol differences and acknowledge that the comparison partially reflects X-Coder's strengths and partially reflects the baselines' use of weaker evaluation protocols.

---

The Pipeline Requires Access to Frontier Teacher Models That May Not Be Available

SynthSmith's task generation, solution generation, and test case generation all depend on specific frontier reasoning models: GPT-o3-mini for task formulation, DeepSeek-R1-0528 for solution sampling and test case generation, and Qwen3-235B-A22B-Thinking-2507 as a secondary solution source (Section 3, Setup). These are not small, openly-available models—they are among the most capable reasoning models available at the time of writing, with API costs or inference costs far exceeding those of the 7B models being trained.

The dependency is non-trivial. The quality of the generated tasks depends on the teacher's ability to:

1. Extract and evolve features accurately (GPT-4o-0513). The extraction prompt requires GPT-4o to identify algorithmic concepts from code with precision—mislabeling a dynamic programming solution as a greedy solution would propagate errors through the feature tree.

2. Formulate coherent tasks from feature sets (GPT-o3-mini). The two-stage task generation process (Appendix B.2) relies on GPT-o3-mini's ability to understand feature compatibility, craft integration strategies, and produce well-specified problem statements. Table 9 shows that poor task formulation (one-step generation) costs 5.3 points.

3. Generate correct solutions for dual-verification (DeepSeek-R1, Qwen3-235B-Thinking). The dual-verification voting mechanism (Algorithm 1) requires candidate solutions to have non-trivial accuracy—otherwise, majority voting produces incorrect labels. The 94.73% labeling accuracy with 8 R1-0528 solutions (Table 13) is conditional on R1-0528 being a strong solver. A weaker solver would produce lower labeling accuracy, more data rejection, and lower-quality training data.

What happens with weaker teachers? The paper does not answer this. The SelfCodeAlign comparison (Table 3) uses the same teacher models, so it controls for teacher quality but does not ablate it. A practitioner without access to GPT-o3-mini-level models for task generation or R1-0528-level models for solution generation has no guidance on how performance would degrade with weaker teachers—or whether the pipeline would produce usable data at all.

The paper's treatment: the authors do not discuss teacher model dependency as a limitation. The choice of teachers is motivated by capability ("advanced open-source reasoning LLMs"), but the implicit assumption—that these models will remain accessible or that the pipeline transfers to future teachers—is not examined. For an approach that claims to reduce reliance on external resources (real-world data), the reliance on frontier proprietary and open-source reasoning models represents a different but equally significant dependency.

---

Test-Time Scaling Analysis Reveals That RL May Not Escape the SFT Policy Distribution

Section 5's test-time scaling analysis (Figure 8) contains a finding that undercuts one of the paper's core narratives about reinforcement learning. The paper reports that:

"Although RL models begin with higher initial performance than the SFT model, the gap does not expand within 16 rollouts, suggesting that RL improves pass@1 but may not escape its starting point."

This observation—connected to Wu et al. (2025)'s work on RL's limited policy exploration—implies that the RL fine-tuning in this paper may be fine-tuning within the SFT model's existing policy distribution rather than discovering qualitatively new reasoning strategies. If true, this constrains the value of the RL stage: it refines what the SFT model already knows but does not teach it to solve problems it fundamentally cannot solve after SFT.

The practical consequence: if a practitioner's goal is to maximize pass@1 (single-sample performance), RL provides a meaningful 2.6–4.6 point gain (Table 1). But if the goal is to maximize pass@k for large k (many rollouts, as in competitive programming contests where solutions can be resampled), the RL gain may be entirely absorbed by the SFT model's ability to find correct solutions through repeated sampling. The test-time scaling curves in Figure 8 show the RL and SFT models maintaining a roughly constant gap across k = 1 to k = 16, suggesting that the RL improvement is a uniform shift rather than an expanding advantage at higher k.

The paper's treatment: the authors note this observation but do not explore its implications. They do not test whether more aggressive RL exploration (higher temperature, more rollouts, different reward shaping) would expand the policy distribution or whether the current configuration represents a fundamental ceiling on what GRPO can achieve starting from SFT. The "Good-gets-Better" principle (Figure 4) shows that RL amplifies existing capability, but the test-time scaling result suggests the amplification has a bounded effect on the diversity of the policy's output distribution.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the conversation around data scarcity for code reasoning from augmentation to replacement. Prior work — whether through evolutionary rewriting (WizardCoder, rStar-Coder), concept composition (SelfCodeAlign, EpiCoder), or solution distillation (OpenCodeReasoning) — operated under the assumption that synthetic data supplements real-world data but cannot substitute for it. SynthSmith challenges this directly by demonstrating that a pipeline starting from only 10,000 TACO problem–solution pairs for feature extraction can generate entirely novel tasks that train models to state-of-the-art competitive performance. This is not a refinement of augmentation; it is a redefinition of what counts as sufficient training data for code reasoning.

The magnitude of this shift depends on how broadly it generalizes. Within competitive programming, the evidence is strong but bounded: X-Coder achieves 62.9 avg@8 on LiveCodeBench v5, outperforming models trained on real or mixed data at comparable scale, but hard problems remain at 37.8% accuracy (Table 2). The paradigm shift is thus domain-specific — it works for the algorithmic reasoning tasks that competitive programming comprises, but does not claim to solve the broader challenge of general code generation. A practitioner building a specialized reasoning system for algorithmic problem-solving now has a viable path that does not depend on continually expanding a curated corpus of human-written problems. A practitioner building a general-purpose coding assistant does not.

The paper also provides a diagnostic resolution to the field's fragmentation around training recipes for code reasoning. Prior work pursued SFT-only (OCR-Qwen, rStar-Coder), RL-only (DeepCoder, AReal-boba²), or mixed math-and-code SFT-then-RL (AceReason, Klear-Reasoner) without clear evidence that a pure-code, two-stage recipe works. The paper demonstrates that it does — SFT on fully synthetic code data followed by GRPO-based RL on synthetic test cases yields consistent, additive improvements (60.3 → 62.9 on v5 for Qwen2.5, 59.4 → 64.0 for Qwen3). This establishes a baseline recipe that future work can refine rather than reinvent.

Two research directions become substantially more attractive in light of this work:

• Feature-based synthesis for other structured reasoning domains. The feature-tree architecture is domain-agnostic. Any domain where tasks decompose into recombinable primitives — mathematical proof (induction, contradiction, algebraic manipulation), formal verification (invariants, abstraction functions, refinement mappings), scientific problem-solving (conservation laws, dimensional analysis, differential equations) — could adopt an analogous extraction–evolution–composition pipeline. The key obstacle is not conceptual but practical: feature extraction requires a seed corpus of solved problems from which algorithmic primitives can be identified, and dual-verification requires a mechanism for verifying solution correctness (execution for code, formal checking for proofs, unit consistency for science). The paper's demonstration that these components can be assembled for competitive programming makes the extension to other verifiable domains a concrete engineering challenge rather than a speculative research bet.

• Scaling laws for synthetic data. The paper's SFT scaling experiments (Figure 3) show that performance improves monotonically from 32k to 192k unique tasks, and that scaling unique tasks dominates scaling solutions per task. But this is only two data points on a much larger scaling curve. A systematic study — extending to 1M+ tasks, testing multiple model sizes, and measuring whether the trend is log-linear, power-law, or sigmoidal — would establish whether synthetic data obeys the same functional forms as web-scale natural language data and would determine whether the current 200k-task ceiling is a temporary artifact of limited compute or a fundamental saturation point.

Conversely, one research direction becomes less attractive: further investment in rewriting-based synthesis for competitive programming. The paper's comparisons (rStar-Coder at 57.3 avg@16 with mixed data, OCR-Qwen at 51.3 avg@64 with 736k real examples, SelfCodeAlign at 27.1 with scratch generation using the same teachers) suggest that seed-bounded methods have hit a diversity ceiling that feature recombination breaks through. The 21-point gap between SynthSmith and EpiCoder tasks (Figure 5c) — both feature-tree methods, but with SynthSmith's competition-oriented extraction and two-stage generation — demonstrates that the gains come not from the feature-tree abstraction alone but from the specific improvements SynthSmith introduces. This raises the bar for new synthesis methods: they must either surpass SynthSmith's task quality or demonstrate applicability to domains SynthSmith does not address, rather than incrementally refining rewriting heuristics.

Follow-Up Research This Work Enables

Characterizing the difficulty ceiling of feature-recombination synthesis. The paper identifies a clear boundary: hard problems remain at 37.8% accuracy (Table 2), and even GPT-5-High fails on ~3% of synthetic tasks (Appendix E.5). Is this ceiling determined by the feature tree's coverage (missing primitives in the TACO seed), the evolution process's depth (insufficient abstraction for very hard concepts), the task formulation step's competence (GPT-o3-mini cannot compose features at the highest difficulty), or the base model's intrinsic capability? A targeted experiment would generate tasks binned by Codeforces-equivalent rating (using the classifier from Appendix B.3) and measure hardness-specific model accuracy as a function of rating. If accuracy drops sharply at a specific rating threshold (e.g., 2400+), this would identify the pipeline's effective difficulty ceiling. If the drop is gradual, the ceiling is softer and might be pushed by stronger teacher models or deeper feature evolution. Additionally, ablating by feature category — generating tasks that require only graph algorithms vs. only number theory vs. only dynamic programming — would reveal whether certain algorithmic domains are systematically harder to synthesize.

RL with deliberately noisy rewards to establish the noise tolerance frontier. Section 3.3 claims RL is "resilient to data imperfections," but the evidence is a single data point: RL improves over SFT using test cases with ~7.85% labeling error (Appendix E.4). A systematic study would train GRPO on the same SFT checkpoint with test cases corrupted at controlled error rates (0%, 5%, 10%, 15%, 20%, 30%), measuring both final policy performance and the point at which RL degrades below SFT. This would calibrate an explicit noise budget for RL data collection — if RL tolerates up to 15% error with minimal degradation, the dual-verification process can be substantially relaxed (accepting tasks where hold-out confirmation fails but error is within tolerance), dramatically increasing data throughput. The relationship between noise tolerance and problem difficulty should also be tested: RL may be robust to noisy labels on easy problems (where the policy already produces correct solutions) but brittle on hard problems (where the policy is sensitive to reward signal accuracy).

Combining SynthSmith tasks with revision-based RL training. The current RL setup uses GRPO with standard rollout-based advantage estimation — the model generates 8 complete solutions per prompt and advantages are computed from final rewards. Recent work on code RL has explored revision-based approaches where the model iteratively refines its own outputs, analogous to the revision mechanism in the test-time compute literature. A natural extension would replace the independent-rollout GRPO formulation with a sequential revision variant: the model generates an initial solution, receives feedback from partial test execution (e.g., which test cases passed/failed), and produces a revised solution, with rewards shaped to encourage improvement across revisions. This would test whether SynthSmith's synthetic test cases provide sufficient signal for revision-based RL — which may be more sensitive to test case quality than rollout-based RL because errors propagate across revision steps. The paper's existing infrastructure (distributed code execution, Redis-based task distribution) directly supports this extension.

Feature evolution depth and its effect on task difficulty. The feature evolution step (Appendix B.1) multiplies feature counts by 5–9× (Table 8) but is applied only once. Does a second round of evolution — taking the already-evolved features and expanding them further — continue to increase task diversity and difficulty, or does it saturate because the feature space becomes dominated by redundant or nonsensical combinations? A controlled experiment would generate SFT datasets from features evolved to 1, 2, and 3 rounds, training models of identical size on identical numbers of tasks from each, and measuring per-difficulty accuracy. If a second evolution round improves hard-problem accuracy, this identifies feature depth as a bottleneck. If it does not, the bottleneck lies elsewhere (teacher model capability, task formulation competence, or base model ceiling).

Cross-domain transfer from synthetic competitive programming data to software engineering tasks. The paper shows modest improvements on standard code benchmarks (HumanEval+: 84.1 → 84.1, MBPP+: 71.7 → 75.7; Table 20), but these benchmarks test function-level code generation rather than the algorithmic reasoning that SynthSmith targets. The critical open question is whether training on synthetic algorithmic tasks transfers to software engineering tasks that require different capabilities — multi-file code organization, API usage, debugging, code review. A direct test would evaluate X-Coder on SWE-bench (real-world GitHub issues requiring patch generation), measuring whether the synthetic training improves, degrades, or leaves unchanged the model's software engineering capability. A positive result would expand SynthSmith's applicability beyond competitive programming. A null or negative result would bound its domain specificity and motivate domain-specific synthetic pipelines for software engineering.

Amortizing difficulty estimation into the task generation process. The current pipeline separates task generation from difficulty assessment — difficulty is estimated post-hoc using a classifier trained on CodeContests (Appendix B.3). This is wasteful because tasks that turn out to be too easy or unsolvable consume generation and verification compute before being filtered. An integrated approach would prompt the task-formulation LLM (GPT-o3-mini) to output a difficulty estimate alongside the problem statement, using chain-of-thought reasoning about the algorithmic complexity required. Comparing LLM-estimated difficulty against classifier-estimated difficulty on a held-out set would validate whether in-line estimation is sufficiently accurate to guide task filtering. If so, easy or unsolvable tasks could be rejected before expensive solution generation and dual-verification, making the pipeline more cost-effective at scale.

Practical Applications and Downstream Use Cases

Training data for competitive programming models at any scale without data leakage concerns. The most direct application is for teams building specialized code reasoning models who need to avoid benchmark contamination. Because SynthSmith tasks are novel compositions of algorithmic features, there is no risk that a model trained on them has memorized specific LiveCodeBench or Codeforces problems. The paper's data leakage analysis (Table 21) quantifies this benefit: X-Coder-7B degrades only 17.2 points from LiveCodeBench v2 to v5 vs. 30.6 points for Qwen3-8B trained on real data. For organizations that need to claim fair evaluation on public benchmarks — or that cannot legally scrape competition platforms for training data — a fully synthetic pipeline provides a compliance-safe alternative. The fixed cost of data generation (teacher model inference, dual-verification) is amortized across all downstream model training runs, and the data can be versioned, audited, and shared without licensing concerns.

Cost-efficient SFT data for resource-constrained code model training. The paper's scaling analysis (Figure 3) shows that 64k unique tasks with 1 solution each outperform 16k tasks with 4 solutions each, despite having the same total token count. For practitioners with limited SFT compute budgets, this provides a concrete allocation guideline: invest in task diversity over solution multiplicity. At the scale tested, generating 64k tasks with SynthSmith and training on them yields LiveCodeBench v5 scores competitive with or exceeding models trained on 736k real-world examples (OCR-Qwen, Table 2) or 580k mixed real+synthetic examples (rStar-Coder, Table 1). The cost savings come from needing fewer total training tokens to achieve a given accuracy level, offsetting SynthSmith's data-generation overhead.

RL infrastructure for code execution at scale. The paper's distributed code verification framework (Appendix A.5) — a FastAPI gateway, Redis-based task queue, and sandboxed worker pool using idle CPUs across GPU machines — is described in enough detail to replicate. This has immediate practical value for any team running GRPO-based RL for code, where the training loop produces bursts of code execution requests that must be handled at high concurrency without bottlenecking GPU utilization. The framework's design choices — Sorted Sets for time-prioritized FIFO queues, BZPOPMIN for atomic task retrieval, TTL-based worker health monitoring — are battle-tested optimizations that practitioners can adopt directly without rediscovering them through trial and error. The paper's reporting that this framework handled 24 million test executions during dataset curation provides a concrete scale reference.