Reasoning2025~109 minarXiv:2505.24760

REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable Rewards

score 9Zafir Stojanovski, Oliver Stanley, Joe Sharratt +4↑ 74★ 1,360May 30, 2025cs.LGhttps://github.com/open-thought/reasoning-gym

🎯 Pitch

Training on procedurally generated algorithmic tasks makes a model 29% better at algebra, without ever seeing an algebra problem. REASONING GYM offers over 100 such data generators with built-in verifiers, allowing you to scale RLVR training indefinitely while continuously dialing difficulty up or down.

1. Executive Summary

This paper introduces reasoning gym (rg), a library of over 100 procedurally generated reasoning environments for reinforcement learning with verifiable rewards (RLVR), spanning algebra, algorithmic reasoning, logic, games, geometry, and cognition. Using Qwen2.5-3B-Instruct trained with GRPO, the authors demonstrate that adaptive curriculum learning—where difficulty increases only after the model surpasses a 70% success threshold—substantially outperforms fixed-difficulty training (e.g., +40.67% on Spell Backwards at word length 4, +26.67% on Count Primes in the 100–500 range). The work further establishes that RLVR on procedurally generated tasks produces both intra-domain transfer (e.g., +11.7% on held-out algebra tasks after training on other algebra generators) and cross-domain transfer—most strikingly, training on algorithmic tasks yields a +29.1% improvement on algebra—while also improving performance on external benchmarks such as MATH (+9.7%), Big-Bench Hard (+7.7%), and MMLU-Pro, establishing that procedurally generated reasoning data develops transferable capabilities rather than narrow task-specific pattern matching, though on the hardest problem configurations even frontier reasoning models like o3-mini and DeepSeek-R1 remain far from saturation, with performance cliffs of up to −71.9% on code tasks when transitioning from easy to hard settings.

2. Context and Motivation

The Core Problem: Reasoning Data Is a Bottleneck for RLVR

The paper addresses a fundamental constraint on the continued advancement of language model reasoning: the availability of high-quality training data for reinforcement learning with verifiable rewards (RLVR). The recent leap forward in LLM reasoning—exemplified by models like OpenAI-o1, DeepSeek-R1, and QwQ-32B—has been powered by RLVR, a paradigm that uses outcome-based feedback (correct/incorrect) to reward models for discovering productive reasoning chains without requiring human annotation of intermediate steps. The mathematics of RLVR is appealingly simple: if you can automatically verify whether a final answer is correct, you can generate a reward signal that drives the model to explore solution strategies, potentially discovering reasoning patterns that supervised fine-tuning would never surface because no human wrote them down.

However, this paradigm has a critical dependency that the paper identifies (Section 1) as a "fundamental scalability bottleneck": RLVR needs problems with verifiable answers, and current sources of such problems are limited in ways that threaten to cap further progress. The paper frames this as a data scarcity problem with several dimensions:

Fixed dataset exhaustion. Existing reasoning benchmarks (GSM8K, MATH, BIG-Bench) are finite. Once a model has been trained on all available problems, there is nothing new to learn from. More importantly, models can memorize problem-answer pairs rather than developing genuine reasoning strategies, making benchmark scores unreliable indicators of capability. As the paper notes (Section 6), "fixed dataset reasoning benchmarks... can lead to overfitting."
Human curation costs. Producing new high-quality reasoning problems—particularly those with verified step-by-step solutions—requires expensive expert annotation. The paper cites this (Section 1) as one of the two current approaches that is "not sustainable" at scale. Creating 600k verified trajectories through rejection sampling (as DeepSeek-R1 did) or manually curating chain-of-thought examples is inherently rate-limited by human effort.
Internet-scraped data quality. The alternative—scraping problems and solutions from the web—introduces noise, errors, and contamination. The paper explicitly references work showing that "benchmarks consisting of internet-scraped data are often erroneous" (Section 6, citing Gema et al., 2024; Phan et al., 2025). Furthermore, internet data is finite and may be exhausted as a training resource (Villalobos et al., 2022, cited in Section 1), creating a long-term sustainability problem independent of quality concerns.

The paper's central claim is that this bottleneck is not a temporary inconvenience but a structural constraint that will become increasingly binding as models improve. If reasoning progress depends on ever-larger quantities of verifiable problems, and the supply of such problems is fixed or growing slowly, then data scarcity imposes a hard ceiling on capability growth. This framing makes the problem urgent rather than merely inconvenient.

Why the Problem Matters: Practical and Scientific Stakes

The paper implicitly argues for the importance of solving this bottleneck along several dimensions:

Practical deployment. If RLVR is the most effective known method for improving reasoning capabilities, but it cannot scale beyond fixed datasets, then model developers face an unpleasant tradeoff: invest in expensive human curation or accept that their models' reasoning will plateau. For organizations building reasoning systems that must handle diverse, novel problems (coding assistants, mathematical tools, scientific reasoning engines), this limitation directly constrains product quality.

Scientific understanding of reasoning transfer. A deeper question lurks beneath the data scarcity problem: when a model improves at reasoning through RLVR, is it learning genuine, transferable reasoning strategies, or is it memorizing solution templates for the specific problems it was trained on? Fixed datasets make this question difficult to answer because training and evaluation distributions are inevitably correlated. The paper positions procedurally generated data as a tool for answering this question definitively—by training on one generator and testing on a different generator from the same domain (or a different domain entirely), researchers can measure transfer in a way that fixed benchmarks cannot.

The memorization-vs-reasoning confound. A persistent concern in LLM evaluation is that benchmark scores reflect dataset familiarity rather than capability (Section 6 cites Dynabench and "The Leaderboard Illusion" on this point). The paper argues that fixed benchmarks are inherently vulnerable to this confound because models can be exposed to benchmark problems during pretraining or through data contamination. Procedurally generated tasks, by producing novel instances on demand, eliminate this concern: even if a model has memorized the generator's logic, it cannot have memorized the specific instance it encounters at test time. This makes procedurally generated benchmarks a more trustworthy signal of reasoning capability.

Economic and sustainability concerns. The paper touches on an economic argument (Section 1): human-curated data is expensive, and internet-scraped data is "neither sustainable nor reliable in the long term." As the field scales toward ever-larger models and training runs, the cost of data acquisition becomes a first-order constraint. A library that generates unlimited training data algorithmically—requiring only compute, not human labor or web scraping—changes the economics of reasoning research.

Prior Approaches and Their Limitations

The paper situates itself against several categories of prior work, each of which it argues is insufficient:

Fixed reasoning benchmarks. The paper acknowledges the value of existing benchmarks like GSM8K, MATH, OlympiadBench, BIG-Bench, and GPQA (Section 6), but identifies two critical shortcomings. First, as fixed datasets, they are exhaustible and susceptible to overfitting. Second, they do not expose the ground-truth data-generating process—researchers cannot control or vary the difficulty of problems in these datasets, making it impossible to study how model capabilities scale with problem complexity in a controlled way.

Procedurally generated benchmarks for games and puzzles. The paper notes (Section 6) that procedural generation has been used successfully in gaming contexts (e.g., Go, StarCraft II, NetHack, BALROG) and for puzzle benchmarks (e.g., EnigmaEval, VGRP-Bench, ZebraLogic, AutoLogi). However, the paper argues that these efforts are domain-specific (focused on particular games or puzzle types) and do not provide the broad coverage across reasoning categories that a general-purpose RLVR training library requires. The paper positions RG as filling this gap by providing procedurally generated environments across a much wider range of reasoning domains—algebra, algorithmic reasoning, geometry, logic, pattern recognition, and constraint satisfaction—all within a unified framework.

Existing RLVR environments. Section 6 discusses several parallel efforts that are closest to RG in spirit:

Tülu 3 builds its RL corpus by taking prompts from existing benchmarks (GSM8K, MATH) whose answers can be automatically checked. This inherits the fixed-dataset limitation—the corpus is bounded by the size of the source benchmarks.
DeepSeek-R1 combines manually curated chain-of-thought examples with automatically gradable reasoning tasks, growing its dataset to ~600k verified trajectories through rejection sampling. This is effective but still finite and requires substantial human upfront effort.
TextArena provides 57+ text-based games for RLVR but is focused specifically on games rather than the broad reasoning spectrum that RG covers.
Logic-RL procedurally generates logic puzzles and demonstrates cross-domain transfer to math competition benchmarks, which the paper cites as evidence that procedurally generated data can develop transferable reasoning. However, Logic-RL is limited to logic puzzles and does not provide the multi-domain coverage of RG.
KOR Gym focuses on games, providing over fifty games in textual or visual formats with multi-turn interactions. The paper acknowledges this as a parallel effort but does not discuss specific limitations.
Reasoning Core is described as "a library of procedurally generated RLVR environments across several formal domains." The paper does not elaborate on how RG differs from Reasoning Core, which represents a gap in the comparative analysis.
GEM provides "a standardized framework for RL environments targeted at LLMs" with a built-in suite of environments. Again, the paper acknowledges but does not deeply contrast.
Synthetic-1 releases 1.4 million high-quality tasks and verifiers. This addresses the data scarcity problem at scale but produces a fixed corpus rather than an unbounded generative capability. Once those 1.4 million tasks are consumed, the bottleneck returns.
Absolute Zero Reasoner proposes self-evolving curriculum generation with code-based verification. The paper positions this as a complementary approach that addresses data scarcity through self-play rather than procedural generation.

The paper's implicit critique of these prior efforts is that they are either domain-limited (TextArena, KOR Gym, Logic-RL), finite in size (Tülu 3, DeepSeek-R1, Synthetic-1), or lack the broad, multi-domain coverage needed for a general-purpose reasoning research platform. RG aims to address all three limitations simultaneously: procedurally generated (unlimited), multi-domain (100+ tasks across 11 categories), and difficulty-controllable (enabling curriculum learning and systematic capability studies).

How This Paper Positions Itself

The paper frames RG not as a competitor to existing reasoning benchmarks but as a complementary resource that addresses a different need. Fixed benchmarks serve as evaluation targets—they measure how well a model performs on a standardized problem set of known difficulty. RG serves as a training and research platform—it provides the data infrastructure for RLVR training and for systematically studying how reasoning capabilities develop under controlled conditions.

This distinction is crucial. The paper does not claim that RG problems are "better" than GSM8K or MATH problems for measuring reasoning ability. Rather, it claims that RG enables experimental protocols that fixed benchmarks cannot support:

Unlimited training data for RLVR, removing the data ceiling on further reasoning improvements.
Parametric difficulty control that enables curriculum learning (Section 5) and systematic study of how performance degrades with complexity (Section 3.2, "The Difficulty Cliff Phenomenon").
Clean transfer experiments (Section 4) where training and evaluation distributions are procedurally separated, eliminating memorization confounds.
Algorithmic verifiability that requires no human judgment, enabling fully automated RLVR pipelines.

The paper also positions RG as an evaluation framework in its own right (Section 3), one that offers advantages over fixed benchmarks for certain research questions. Because RG can generate problems at arbitrary difficulty levels, it can probe the boundaries of model capability in ways that fixed benchmarks cannot—revealing, for instance, that even frontier models exhibit dramatic performance cliffs when difficulty increases beyond the range represented in typical benchmarks (Section 3.2, Figure 3b).

In the broader research landscape, the paper positions RG as infrastructure for investigating a set of open questions that the field currently lacks the tools to answer systematically:

Do reasoning skills learned via RLVR transfer across domains? The intra-domain and cross-domain transfer experiments (Section 4) address this directly, and the paper presents evidence for both.
Does curriculum learning improve RLVR outcomes? Section 5 provides initial evidence that it does, using RG's difficulty parameters to construct curricula that are impossible to create with fixed benchmarks.
Where do current models fail, and why? The zero-shot evaluation (Section 3) uses RG's difficulty parameters to map the performance landscape of frontier models, revealing that visual-spatial reasoning in text format and long-horizon algorithmic tasks remain largely unsolved.
Can procedurally generated data substitute for human-curated data in RLVR? The external benchmark transfer results (Section 4.3) suggest that skills developed on RG data transfer to GSM8K, MATH, Big-Bench Hard, and MMLU-Pro, providing preliminary evidence that procedurally generated data can develop broadly applicable reasoning capabilities.

The paper's design principles (Section 2, P1–P3) further clarify its positioning. By requiring algorithmic verifiability (P1), it restricts itself to domains where correctness can be automatically determined—this is a deliberate tradeoff that trades breadth of applicability for scalability of training. By designing for large solution spaces (P2), it aims to prevent reward hacking and encourage generalizable strategies. By providing parametric difficulty control (P3), it enables the kind of systematic capability studies that the paper itself conducts in Sections 3–5. These principles distinguish RG from both fixed benchmarks (which lack P3) and from narrower procedurally generated environments (which lack P2's breadth).

Unstated Assumptions and Implicit Arguments

Several assumptions underlie the paper's framing that are worth surfacing:

Procedural generation can capture "real" reasoning. The paper assumes that solving procedurally generated algebra, logic, or algorithmic problems develops the same reasoning capabilities that matter for real-world tasks and external benchmarks. The transfer results in Section 4.3 provide some evidence for this, but the assumption is not fully validated—the paper acknowledges (Section 7) that "some reasoning domains, particularly those requiring extensive domain knowledge or creativity, are difficult to capture with procedural generators."

Verifiable rewards are sufficient for reasoning development. The paper adopts the RLVR assumption that outcome-based feedback (correct/incorrect) is sufficient to drive the development of complex reasoning strategies, without needing process-based rewards or human feedback on intermediate steps. This assumption has been validated by DeepSeek-R1 and related work but is not universally accepted—the paper notes (Section 7) that "verification functions, while comprehensive, may not capture all aspects of solution quality that humans consider important" and that there remains "an important place for human-centric mechanisms."

Unlimited data is necessary for continued progress. The paper's urgency about data scarcity assumes that reasoning models will continue to benefit from larger training datasets indefinitely. If reasoning capabilities were to saturate at some dataset size (as appears to be happening in some domains already—the paper notes that arithmetic is "approaching saturation"), then the unlimited data offered by procedural generation might be solving a problem that is about to diminish in importance. The paper does not engage with this possibility.

Difficulty can be meaningfully parameterized. The paper's parametric difficulty control (P3) assumes that problem difficulty can be reduced to a small number of continuous parameters (node counts, polynomial degrees, word lengths). For many reasoning domains, difficulty is multidimensional and not easily captured by scalar parameters—a geometry problem can be hard because it requires many steps, or because it requires a non-obvious insight, or because it requires coordinating multiple constraints. The paper's curriculum learning experiments (Section 5) use simple one-dimensional difficulty progressions, which may oversimplify the difficulty landscape.

Summary of the Gap

The paper identifies a concrete and increasingly urgent gap: the field lacks scalable, controllable, and diverse data infrastructure for RLVR-based reasoning research and training. Existing benchmarks are fixed and exhaustible. Existing procedurally generated environments are domain-specific. Existing RLVR datasets are finite. RG proposes to fill this gap by providing a library of 100+ procedurally generated reasoning environments spanning 11 categories, with parametric difficulty control and algorithmic verifiability, enabling experiments (curriculum learning, transfer studies, difficulty scaling analyses) that are currently impractical or impossible with available resources.

3. Technical Approach

3.1 Reader Orientation

This paper presents a software library, not a novel algorithm. The system being described is reasoning gym (rg)—a collection of over 100 Python-based data generators and corresponding verifier functions that produce reasoning problems on demand, each with algorithmically checkable correct answers and tunable difficulty parameters. The problem being solved is the data scarcity bottleneck in reinforcement learning with verifiable rewards (RLVR): existing reasoning benchmarks are fixed in size and exhaustible, human-curated data is expensive and rate-limited, and internet-scraped data is noisy and finite. The shape of the solution is a procedural content generation pipeline that parameterizes problem difficulty along multiple dimensions, enabling both unlimited training data for RLVR and controlled experiments on how reasoning capabilities scale with problem complexity.

3.2 Big-Picture Architecture (Diagram in Words)

The rg architecture has four major components, connected in a generation-then-verification pipeline:

Task Generators — Python classes/functions, one per reasoning task, that sample parameters from configured distributions and produce a structured prompt (the problem statement in natural language or formatted text) along with a ground-truth answer. There are over 100 such generators spanning 11 categories (algebra, algorithmic, arithmetic, cognition/ARC, code, games, geometry, graphs, induction, logic, and a catch-all for list functions and similar tasks).
Difficulty Parameters — each generator exposes a configuration dictionary that controls problem complexity. Parameters fall into three conceptual buckets: difficulty parameters that directly control complexity (e.g., grid size, polynomial degree, number of empty cells), structural parameters that determine fundamental problem properties (e.g., constraint types, proof depth, presence of cycles), and stylistic parameters that vary presentation without affecting difficulty (e.g., variable names, number formats, problem framing). These parameters are user-configurable via YAML configuration files.
Prompt Formatting Layer — a templating system (with an optional "developer prompt" such as DeepSeekZero-style system messages) that wraps the generated problem data into a formatted natural language prompt suitable for input to an LLM. This layer handles the conversion from structured generator output to the text format the model expects.
Verifier Functions — for each generator, a corresponding deterministic function that takes the model's text output, parses it, extracts the predicted answer, and compares it to the ground-truth answer, returning a binary correct/incorrect signal. These verifiers are algorithmic—they require no human judgment, no learned model, and no approximate matching (except where explicitly designed, such as floating-point tolerance for geometry tasks).

Information flows as follows: a YAML configuration file specifies which generators to use, their parameter ranges, and their sampling weights → at each training step, the configured generators are sampled according to their weights → each selected generator produces one problem instance by drawing random parameters within its configured ranges → the problem is formatted into a prompt via the templating layer → the model generates a response → the verifier parses the response, extracts the answer, and compares to ground truth → the binary reward (plus auxiliary rewards for formatting and optionally response length) is returned to the RLVR training loop.

3.3 Roadmap for the Deep Dive

First, the three core design principles (algorithmic verifiability, large solution spaces, parametric difficulty control) that constrain every generator and verifier in the library. Understanding these principles is essential because they explain why rg tasks look the way they do and what kinds of reasoning they can and cannot capture.
Second, the task generation mechanism and the taxonomy of 11 reasoning categories. This provides the concrete inventory of what rg actually contains—the specific problems, their parameters, and how difficulty is modulated—before we examine how the library is used in experiments.
Third, the verifier design and its interaction with the RLVR training loop, including the reward structure (accuracy + format + optional length bonuses) and the training infrastructure (verl, GRPO, curriculum scheduling).
Fourth, the configuration and hyperparameter system, including the curriculum learning scheduler, dataset composition (mixing multiple generators with weights), and the training configuration used for the Qwen2.5-3B-Instruct experiments.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and infrastructure paper whose core idea is that procedurally generated, algorithmically verifiable reasoning tasks—organized into a configurable library with parametric difficulty control—can serve as both a scalable training resource for RLVR and a controlled experimental platform for studying reasoning capability development.

The Three Design Principles

The paper states three explicit design principles (Section 2) that constrain every generator and verifier in rg. These principles are not merely aspirational—they directly determine what is and is not included in the library and shape the experimental protocols the library enables.

(P1) Algorithmic Verifiability. Every task must admit automatic verification requiring no human judgment. The paper states: "Every task admits automatic verification and requires no human judgment. This enables reliable RLVR training while eliminating subjective evaluation."

What this means operationally: for every generator, there exists a deterministic function verify(model_output, ground_truth) → {0, 1} that can be executed programmatically. The verifier must handle answer parsing (extracting the predicted answer from the model's free-form text response) and comparison (checking equivalence against the ground truth). The paper provides examples of verifier behavior in the task descriptions in Appendix A.2: for spiral_matrix, the verifier expects a space-separated list of integers; for mini_sudoku, it expects a 4×4 grid with specific formatting; for knights_knaves, it expects a string matching "Zoey is a sage/fool, and Riley is a sage/fool"; for advanced_geometry, it expects coordinates rounded to 3 decimal places.

Why this matters: algorithmic verifiability is what makes RLVR possible at scale. If verification required human judgment (as in RLHF), the training loop would be bottlenecked by human annotation throughput and cost. By restricting to tasks where correctness can be determined automatically, rg enables fully automated training pipelines that can generate and verify millions of problems without human intervention.

What this constraint excludes: any task where answer quality is subjective, multi-dimensional, or requires nuanced evaluation. The paper acknowledges this tradeoff explicitly (Section 7): "Some reasoning domains, particularly those requiring extensive domain knowledge or creativity, are difficult to capture with procedural generators. In particular, procedural generators may struggle in domains where answers are unstructured."

(P2) Large Solution Spaces. Tasks are designed to have many possible valid solution paths rather than a single correct approach. The paper states: "Tasks are designed with expansive solution spaces, rewarding generalizable strategies above overfitting and mitigating reward hacking."

What this means operationally: for a given problem instance, there should be multiple distinct sequences of reasoning steps that lead to the correct answer. For example, in mini_sudoku, the order in which cells are filled can vary; in shortest_path, if multiple paths have the same length, any is acceptable; in polynomial_multiplication, different orderings of term multiplication produce equivalent results. This is not a binary property but a design intention—generators are constructed to produce problems where solution diversity is inherent to the problem structure rather than artificially injected.

Why this matters: in RLVR, a narrow solution space creates two risks. First, if only one specific chain-of-thought leads to the correct answer, the model may learn to reproduce that specific chain rather than developing general reasoning strategies—this is overfitting to the solution template. Second, it makes reward hacking easier: if the reward signal is sparse (only at the final answer) and the solution space is narrow, the model might learn spurious correlations between surface-level features of the prompt and the answer rather than engaging in genuine reasoning. Large solution spaces make these failure modes less likely because the model must learn strategies that work across diverse problem instances.

What this constraint implies for generator design: problem types that naturally admit multiple solution approaches (constraint satisfaction, search, symbolic manipulation) are favored over problem types with a single canonical solution path (certain "fill in the blank" style tasks). The paper's inclusion of games (Sudoku, Rush Hour, Tower of Hanoi, Knight Swap), algorithmic tasks (shortest path, maze, word ladder), and symbolic algebra (polynomial multiplication, simple equations) reflects this preference.

(P3) Parametric Difficulty Control. Configurable parameters systematically control problem characteristics, enabling dynamic curricula. The paper states: "Configurable parameters systematically control problem characteristics, enabling dynamic curricula via precise difficulty adjustment."

What this means operationally: every generator exposes a configuration dictionary with numeric parameters that can be adjusted to produce easier or harder problem instances. For example, spiral_matrix has parameters min_n and max_n controlling the matrix dimensions; polynomial_multiplication has min_terms, max_terms, min_degree, max_degree, min_value, max_value, min_polynomials, max_polynomials; mini_sudoku has min_empty and max_empty controlling the number of unfilled cells.

The paper organizes these parameters into three conceptual types (Section 2):

Difficulty Parameters directly control complexity: node counts for graphs, polynomial degrees for algebra, word lengths for language tasks, grid sizes for spatial reasoning, number of empty cells for puzzles.
Structural Parameters determine fundamental problem properties: dimensionality, constraint types, proof depth, the presence of cycles in graphs, the types of logical operators available in circuit logic problems.
Stylistic Parameters vary presentation without affecting difficulty: variable names (using different letters), number formats (decimal vs. fraction), problem framing (different natural language phrasing for the same underlying task).

Why this matters: parametric difficulty control is what distinguishes rg from fixed benchmarks and enables the experiments in Sections 3–5. For the zero-shot evaluation (Section 3), it allows the creation of "easy" and "hard" configurations for each task, revealing the difficulty cliff phenomenon. For curriculum learning (Section 5), it enables the automated difficulty progression where the scheduler increases parameter values when the model surpasses a performance threshold. For transfer studies (Section 4), it allows training on a range of difficulties that matches the target evaluation distribution.

The specific "easy" and "hard" parameter values for every task are documented in Appendix A.3 of the paper. For example, complex_arithmetic uses min_real = -10, max_real = 10 for easy and min_real = -100, max_real = 100 for hard; game_of_life uses grid_size_x = 10, grid_size_y = 10 for easy and grid_size_x = 50, grid_size_y = 50 for hard; rubiks_cube uses cube_size = 3, min_scramble_steps = 3, max_scramble_steps = 10 for easy and cube_size = 5, min_scramble_steps = 25, max_scramble_steps = 50 for hard. These configurations are provided as YAML with the easy values as defaults and the hard values in comments marked with #.

Task Generator Architecture

Each of the 100+ tasks in rg is implemented as a generative algorithm—a Python function or class that takes a configuration dictionary as input and produces a structured problem instance as output. The generator is responsible for two things: sampling random problem parameters within the configured ranges, and constructing a valid problem instance with a unique correct answer from those parameters. The generator does NOT format the problem as a natural language prompt—that is handled by the templating layer.

The paper groups generators into 11 high-level categories (Section 2, Table 6), organized by the type of reasoning they require:

Algebra (6 datasets): Symbolic manipulation tasks including gsm_symbolic, intermediate_integration, polynomial_equations, polynomial_multiplication, simple_equations, and simple_integration. These require variable tracking, equation solving, polynomial expansion, and integration. Difficulty is controlled by polynomial degree, number of terms, value ranges, and the complexity of integration patterns.
Algorithmic (32 datasets): The largest category, covering procedural and computational thinking tasks. Examples include base_conversion (converting numbers between bases), binary_alternation, chain_sum, count_bits, count_primes, game_of_life (simulating Conway's Game of Life), gcd, lcm, letter_counting, number_sorting, palindrome_generation, prime_factorization, spell_backward, spiral_matrix, string_manipulation, and word_sorting. Difficulty is controlled by input size (number of elements, string length, value range), number of operations required, and simulation depth.
Arithmetic (11 datasets): Mathematical calculation and symbolic reasoning including basic_arithmetic, bitwise_arithmetic, calendar_arithmetic, complex_arithmetic, cryptarithm, decimal_arithmetic, fraction_simplification, leg_counting, number_format, power_function, and time_intervals. Difficulty is controlled by number of terms, digit count, decimal precision, and operation complexity.
Cognition and ARC (14 datasets): Pattern recognition and transformation tasks including acre (causal reasoning about "blicket" detectors), aiw (the Alice in Wonderland problem), arc_1d (1D version of ARC-style pattern completion), arc_agi (ARC-AGI-style visual reasoning in text), binary_matrix (Manhattan distance computation), boxnet, color_cube_rotation, emoji_mystery, family_relationships, figlet_font (reading ASCII art text), letter_jumble, rearc, self_reference, and sentence_reordering. These tasks require identifying transformations from examples and applying them to new inputs—the paper notes (Section 3.2) that this category is particularly challenging because it requires "visual-spatial reasoning represented in text format."
Code (2 datasets): bf (interpreting Brainfuck programs) and codeio (predicting the output of code snippets). Difficulty is controlled by program complexity.
Games (18 datasets): Constraint satisfaction and planning puzzles including boxnet, countdown, dice, futoshiki, jugs (water jug problems), knight_swap, mahjong_puzzle, mini_sudoku (4×4 Sudoku), n_queens, puzzle24, rubiks_cube, rush_hour, sokoban, sudoku, tower_of_hanoi, tsumego (Go life-and-death problems), word_ladder, and zebra_puzzles (Einstein's riddle). Difficulty is controlled by board size, number of constraints, number of moves required, grid dimensions, and number of empty cells.
Geometry (4 datasets): advanced_geometry (orthocenter, circumcenter calculations), pool_matrix (pooling operations on matrices), rectangle_count, and simple_geometry. Difficulty is controlled by coordinate ranges and polygon complexity.
Graphs (6 datasets): Discrete structures and traversal including course_schedule (topological sort with prerequisites), graph_color, largest_island (finding connected components), maze, rotten_oranges (BFS propagation), and shortest_path. Difficulty is controlled by graph size, grid dimensions, density of obstacles, and constraint complexity.
Induction (2 datasets): modulo_grid and number_sequence. These require inferring the underlying rule from examples and applying it to predict the next element.
Logic (7 datasets): Formal deduction including ab (a logical deduction game), caesar_cipher, circuit_logic (Boolean circuit evaluation), knights_knaves, propositional_logic, quantum_lock, and syllogism. Difficulty is controlled by number of variables, statement complexity, proof depth (for knights_knaves, depth_constraint and width_constraint control the nesting of logical statements), and number of inputs.
Additional (not in main categories): list_functions is listed in Appendix A.2 but not assigned to a category in Table 6; gsm_symbolic appears under algebra but is a symbolic variant of GSM8K-style word problems.

Each generator is documented in Appendix A.2 with its default configuration, an example task (prompt + answer), and its difficulty parameters. For instance, spiral_matrix (Algorithmic) has the default configuration min_n = 2, max_n = 10 and generates a matrix of random integers of size n × n (where n is sampled uniformly from [min_n, max_n]), with the task being to output the elements in clockwise spiral order starting from the top-left. The answer is a space-separated list of integers, and the verifier compares this list to the ground truth.

Prompt formatting. The paper uses a templating approach where each generator produces structured data (the problem parameters, the formatted problem statement, and the answer) and a separate formatting layer wraps this into a natural language prompt. The training configuration (Appendix A.6) shows a developer_prompt: DeepSeekZero setting, indicating that prompts can be prefixed with system-level instructions that encourage chain-of-thought reasoning. The specific prompt templates for each task are not detailed in the paper but are available in the open-source repository.

Why this architecture over alternatives. The paper contrasts rg with fixed datasets (which cannot generate novel instances or control difficulty) and with domain-specific procedural generators (which lack breadth). The key architectural decision is the separation of generation from verification: each generator produces both the problem and the ground-truth answer simultaneously, and the verifier is a deterministic function of the model output and this ground truth. This eliminates the need for a learned verifier model (as would be required for open-ended generation tasks) and ensures that the reward signal is noise-free—a correct answer always receives a reward of 1, an incorrect answer always receives 0, with no ambiguity.

Verifier Design and Reward Structure

Every task in rg has an associated verifier function that takes the model's raw text output and the ground-truth answer (produced by the generator) and returns a binary correctness judgment. The verifier must handle two sub-problems: answer extraction (parsing the model's free-form text to isolate the predicted answer) and answer comparison (determining whether the predicted answer is equivalent to the ground truth).

Answer extraction. The paper does not specify a universal answer extraction mechanism—each task's verifier implements its own parsing logic suited to the expected answer format. Based on the examples in Appendix A.2:

For spiral_matrix, the expected format is "a space-separated list of integers, e.g. 1 2 3 4 5 6". The verifier likely splits on whitespace and compares element-by-element.
For mini_sudoku, the expected format is "the puzzle above, with spaces separating each number within a row, and newlines separating rows." The verifier parses a 4×4 grid and compares to the ground-truth grid.
For knights_knaves, the expected format is "Zoey is a sage/fool, and Riley is a sage/fool." The verifier parses named entity assignments.
For advanced_geometry, coordinates must be "in the form (x, y)" and "rounded to 3 decimal places." The verifier parses coordinate pairs and compares with tolerance.
For bf (Brainfuck), the model must "respond only with the exact output of the program." The verifier does an exact string match.
For shortest_path, the expected output is "a sequence of directions... e.g. right right down down up left" or the string "infeasible" if no path exists.

This diversity of answer formats is both a strength (it tests the model's ability to follow formatting instructions) and a potential confound (a model might solve the problem correctly but lose the reward due to formatting errors). The paper addresses this through the reward structure, which includes auxiliary rewards for formatting.

Answer comparison. For tasks with exact answers (integers, strings, ordered lists), comparison is exact equality. For tasks with approximate answers (geometry coordinates, floating-point results), comparison uses tolerance-based matching. For tasks with multiple valid answers (e.g., shortest_path where multiple paths may have the same length), the verifier accepts any valid solution. The paper notes that verification is "algorithmic"—meaning the comparison logic is hand-written for each task, not learned.

Reward structure. The paper uses a composite reward during RLVR training (Appendix A.6):

Accuracy reward: Binary, 1.0 if the verifier confirms the answer is correct, 0.0 otherwise. This is the primary signal driving reasoning improvement. The configuration shows use_accuracy: True.
Format reward: An auxiliary reward component that rewards proper output formatting. The configuration shows a format secondary reward with scaling_factor: 0.2. This means the model receives 0.2 bonus reward when its output conforms to the expected format (e.g., including the answer in the specified structure, using proper delimiters). The exact format checking logic is not specified in the paper but is implemented in the open-source repository.
Length reward: An optional auxiliary reward that penalizes or rewards response length. The configuration shows a length secondary reward with scaling_factor: 0.2. The paper notes (Section 4) that "the training reward plots represent the total reward, computed as the sum of an accuracy component and an auxiliary component that rewards proper output formatting. By contrast, the evaluation tables report only the accuracy component, rescaled to a percentage in the range 0–100%." This separation is important: during training, the model optimizes a composite reward (accuracy + format + length), but evaluation metrics use only accuracy to ensure fair comparison.

Why this reward structure. The format reward addresses a known problem in RLVR: models can produce correct reasoning but fail to extract the answer in the expected format, receiving zero reward despite solving the problem. By explicitly rewarding format adherence, the model learns to output parseable answers early in training—the paper notes (Section 4.1) that "there is a sharp increase in reward at the start of training. This is partly attributable to the model quickly learning auxiliary rewards (i.e. formatting) during training." The length reward (which the paper does not discuss in detail) likely encourages concise outputs or penalizes excessively long responses, though the exact implementation is in the repository code.

Training Infrastructure and RLVR Protocol

The paper uses GRPO (Group Relative Policy Optimization) as the RL algorithm, implemented via the verl open-source library (Section 4, Appendix A.6). GRPO is a variant of PPO (Proximal Policy Optimization) designed for language model fine-tuning, introduced by Shao et al. (2024) in the DeepSeekMath paper.

Training configuration. The primary training configuration (Appendix A.6) specifies:

Base model: Qwen/Qwen2.5-3B-Instruct, loaded from HuggingFace.
Dataset size: dataset_size: 20000, meaning 20,000 problems are generated procedurally per epoch (one epoch total).
Prompt length: max_prompt_length: 4096 tokens.
Response length: max_response_length: 2048 tokens.
Batch configuration: train_batch_size: 32, ppo_mini_batch_size: 16, ppo_micro_batch_size_per_gpu: 8.
Optimizer: Learning rate 1e-6, constant warmup (warmup_style: constant), no minimum learning rate, 500 total training steps.
GRPO-specific: n: 8 responses per prompt (the G in GRPO—the model generates 8 candidate responses for each prompt and computes advantages relative to the group mean), clip_ratio: 0.2, entropy_coeff: 0.001, KL penalty with coefficient 0.001 using the low-variance KL estimator (kl_loss_type: low_var_kl).
Rollout configuration: Temperature 1.0, top-k -1 (disabled), top-p 1.0, using vLLM for efficient batched inference with tensor model parallelism of 4.
Hardware: 4×A6000 GPU node on Runpod (cloud rental), with approximately 1500 A6000 hours total for all experiments reported in the paper (Section 4).

Why GRPO over alternatives. The paper does not explicitly justify the choice of GRPO, but the algorithm is well-suited for RLVR with language models because: (1) it eliminates the need for a separate critic (value function) model, reducing memory and simplifying the training pipeline—the advantage for each response is computed as the reward minus the mean reward of the group of n responses to the same prompt; (2) the group-relative advantage normalization provides a natural baseline that adapts to problem difficulty—for an easy problem where all 8 responses are correct, advantages are near zero and no update occurs; for a hard problem where one response is correct and seven are wrong, that correct response receives a strong positive advantage signal; (3) the KL penalty prevents the policy from diverging too far from the base model, preserving general language capabilities.

Dataset mixing. The configuration shows how multiple generators are combined into a single training mixture. Each dataset is assigned a weight (defaulting to 1) and optional config overrides for its difficulty parameters. For example, the algorithmic composite used for cross-domain transfer experiments (Section 4.2) includes: ab, base_conversion, binary_alternation, binary_matrix, caesar_cipher, cryptarithm, isomorphic_strings, jugs, rotate_matrix, and string_manipulation—all with weight 1, and several with custom parameter configurations (e.g., binary_matrix with min_n: 2, max_n: 6, caesar_cipher with max_words: 10, jugs with difficulty: 6). The total number of datasets and their difficulty ranges determine the breadth and challenge of the training distribution.

Curriculum Learning Scheduler

Section 5 introduces a curriculum learning mechanism that automatically increases task difficulty during training based on model performance. The configuration (Appendix A.6) specifies:

Enabling: curriculum.enabled: False by default; set to True for curriculum experiments.
Automatic scheduling: schedule.automatic: True, meaning the scheduler automatically decides when to increase difficulty rather than following a fixed schedule.
Update frequency: update_steps: 30—the scheduler evaluates model performance every 30 training steps.
Evaluation window: last_k: 20—performance is computed over the last 20 evaluation steps (or the last 20 batches of evaluations).
Success threshold: success_threshold: 0.70—when the model achieves at least 70% accuracy on the current difficulty level, the scheduler advances to the next level.
Failure threshold: failure_threshold: 0.10—the paper mentions this parameter exists but does not describe its behavior. Presumably, if performance drops below 10%, the scheduler might decrease difficulty or pause progression.

How curriculum levels are defined. For each task that supports curriculum learning (marked with ✓ in Table 6), the curricula section of the config maps difficulty parameters to ordered levels. For spell_backward, the curriculum specifies word_len: 0, with levels 4, 6, 8, 10 (word lengths). The attribute_levels mapping indicates which parameters vary across levels—for spell_backward, it's word length; for mini_sudoku, it's the number of empty cells (levels 4-6, 6-8, 8-10, 10-12); for count_primes, it's the number range (levels 100-500, 100-1000, 100-5000).

Interaction with training. When curriculum is enabled, the generator samples problems from the current difficulty level only (not the full difficulty range). When the success threshold is met over the evaluation window, the scheduler increments the level, and subsequent problems use the harder parameter ranges. The paper's Figure 6 shows vertical black lines marking these difficulty increases, each followed by a sharp drop in reward as the model encounters harder problems. The curriculum model's terminal reward is lower than the non-curriculum model's because the curriculum model is exclusively exposed to maximum-difficulty examples by the end of training, while the non-curriculum model samples across the full difficulty distribution and thus sees a mix that includes easier problems.

Why curriculum learning. The paper justifies curriculum learning by reference to prior work (Bengio et al., 2009; Narvekar et al., 2020) and positions it as an approach where "the learner first masters simpler instances before being exposed to progressively harder variations of a task." In the RLVR context, curriculum learning serves two purposes: (1) it prevents the model from being overwhelmed by hard problems early in training when it has not yet learned basic strategies or even proper output formatting, and (2) it potentially leads to better final performance on hard problems by building up reasoning capabilities incrementally—the paper's Table 5 shows that the curriculum-trained model consistently outperforms the non-curriculum model at every difficulty level, not just at the easier levels where it spent more time.

Generator-Verifier Coupling and Data Flow

To make the full data flow concrete, here is what happens at a single training step when training on spiral_matrix:

Sampling: The generator samples n uniformly from [min_n, max_n] (configured as min_n: 2, max_n: 10 for easy, min_n: 25, max_n: 50 for hard). It creates an n × n matrix filled with random integers.
Solution generation: The generator computes the spiral-order traversal algorithmically—starting at (0,0), moving right across the top row, down the right column, left across the bottom row, up the left column, and spiraling inward until all elements are visited. This produces a deterministic list of integers (the ground-truth answer).
Prompt construction: The templating layer wraps the matrix and task description into a natural language prompt (shown in Appendix A.2): "Given a matrix, your job is to generate a list of elements in spiral order, starting from the top-left element... For the matrix below, what is the list of elements in spiral order? [matrix printed row by row]"
Model inference: The prompt is tokenized (up to 4096 tokens) and fed to Qwen2.5-3B-Instruct. The model generates up to 2048 response tokens at temperature 1.0. This happens 8 times (n: 8) in parallel via vLLM to produce 8 candidate responses.
Verification: For each of the 8 responses, the verifier parses the text to extract the predicted list of integers (handling potential formatting variations) and compares it element-by-element to the ground-truth list. A binary accuracy reward (1.0 or 0.0) is assigned. The format verifier checks whether the output follows the expected format and assigns a bonus (0.0 to 0.2). The length verifier computes a length-based bonus (0.0 to 0.2).
Advantage computation: For each response, the total reward is accuracy_reward + format_reward + length_reward. The mean reward across the 8 responses is computed. The advantage for each response is its reward minus the group mean.
Policy update: GRPO updates the model parameters using the PPO clipped objective with the computed advantages, KL penalty, and entropy bonus, following the hyperparameters specified in the configuration (learning rate 1e-6, clip ratio 0.2, KL coefficient 0.001, entropy coefficient 0.001).

For the curriculum variant, an additional check occurs every 30 steps: the scheduler computes the model's average accuracy on the current difficulty level over the last 20 evaluation windows. If it exceeds 70%, the n range for spiral_matrix is increased to the next level (e.g., from word length 4 to word length 6). Subsequent problems use the new range.

Summary of Design Choices and Their Justifications

Procedural generation over fixed datasets: Eliminates memorization concerns, enables unlimited training data, and provides parametric difficulty control for curriculum learning and systematic capability studies. The cost is that generated problems may lack the richness and real-world relevance of human-crafted problems—a tradeoff the paper acknowledges.
Algorithmic verifiability over learned verifiers: Provides noise-free reward signals (no false positives or false negatives from a learned verifier model) and eliminates the need to train and maintain a separate verifier. The cost is that tasks must have deterministic, automatically checkable answers, which excludes many important reasoning domains.
Separate generators and verifiers: Each generator produces a (problem, answer) pair; each verifier is a deterministic function of (model_output, answer). This separation means the generator can be modified (e.g., to change difficulty parameters) without affecting verification logic, and the verifier can be tested independently.
Composite reward with format bonus: Addresses the practical problem that models must learn both reasoning and output formatting, and that formatting failures can mask reasoning successes. By explicitly rewarding format adherence early in training, the model quickly learns to produce parseable outputs, after which the accuracy reward dominates.
GRPO with group-relative advantages: Eliminates the need for a critic model, provides automatic difficulty-aware baseline normalization (hard problems where only one of eight responses is correct produce strong positive advantage for that response), and includes a KL penalty to prevent catastrophic forgetting of the base model's capabilities.
Curriculum with automatic difficulty progression: Adapts the training distribution to the model's current capability, preventing early exposure to impossibly hard problems while ensuring that the model eventually trains on maximum-difficulty instances. The 70% threshold is a heuristic—the paper does not justify this specific value—but the approach of using performance-based triggers is standard in curriculum learning literature.
Single-epoch training with 20k procedurally generated instances: Since the generator can produce novel instances indefinitely, the concept of "epochs" is somewhat artificial—the model sees 20k unique problems, none of which it will see again. This eliminates overfitting to specific problem instances and ensures that performance improvements reflect genuine reasoning capability development rather than memorization.

4. Key Insights and Innovations

Innovation 1: Procedurally Generated Reasoning Data as a First-Class Alternative to Fixed Datasets for RLVR

Before Reasoning Gym, the dominant assumption in reasoning research was that high-quality training data for RLVR must come from either human-curated problem sets (GSM8K, MATH) or internet-scraped content—both finite, exhaustible, and increasingly contaminated as models are exposed to benchmark problems during pretraining. The paper's core intellectual move is to reframe procedural generation not as a niche technique for game-playing environments (Cobbe et al., 2020) or domain-specific puzzles (Logic-RL, AutoLogi), but as a general-purpose data production paradigm that can match—and in important ways exceed—the utility of fixed benchmarks for both training and evaluation.

This is a conceptual shift, not merely a scaling improvement. Fixed benchmarks serve one function: they measure performance on a standardized problem set. But they cannot answer questions about how performance degrades with difficulty, whether improvements represent genuine reasoning or memorization, or how skills transfer across domains—because the benchmark's difficulty distribution is fixed and its relationship to other reasoning domains is unspecified. Procedural generation, by exposing the data-generating process and enabling parametric control over problem characteristics, transforms these from unanswerable questions into experimentally tractable ones. The paper demonstrates this transformation concretely through three experiments that would be impossible or highly confounded with fixed benchmarks: the difficulty cliff analysis (Section 3.2, Figure 3b), the intra-domain and cross-domain transfer studies (Section 4, Tables 1 and 2), and the curriculum learning comparison (Section 5, Table 5).

The practical significance is equally important. By providing unlimited training data with algorithmic verifiability, RG eliminates the economic bottleneck of human annotation for RLVR. The paper's estimate of ~1500 A6000 GPU-hours for all reported experiments (Section 4) represents a fixed capital cost; generating equivalent human-curated data would require ongoing labor costs that scale with dataset size. This changes the economics of reasoning research from an annotation-bound regime to a compute-bound one, which—given the trajectory of compute costs relative to expert labor costs—is a structural improvement for the field's scalability.

The paper pushes this reframing beyond infrastructure into epistemology: procedurally generated data enables cleaner transfer experiments because training and evaluation distributions can be procedurally separated in ways that fixed benchmarks cannot guarantee. When training on one generator and testing on a different generator from the same domain, any observed improvement cannot be attributed to memorization of shared problem instances—there are none. When the paper reports +11.7% improvement on held-out algebra tasks after training on other algebra generators (Table 1), or +29.1% improvement on algebra after training on algorithmic tasks (Table 2), these numbers represent a cleaner signal of genuine transfer than any fixed-benchmark study can provide, because the confound of instance overlap is eliminated by construction. This is a methodological contribution as much as an empirical one.

Innovation 2: Difficulty as a First-Class Variable in Reasoning Capability Analysis

The paper introduces the difficulty cliff as a diagnostic concept and demonstrates that the relationship between problem difficulty and model performance is not merely quantitative (harder problems → lower scores, which everyone expects) but qualitatively patterned across domains in ways that reveal the structure of current models' capabilities and failures. The key finding is not that performance drops with difficulty—that would be trivial—but that the magnitude and domain-specificity of these drops constitute a new kind of capability profile.

Consider Figure 3b: the difficulty cliff magnitudes vary dramatically across domains even for the same model. For o3-mini, the drop from easy to hard configurations is −71.9% on code tasks but only −4.2% on arithmetic and −2.3% on logic. This pattern is not a uniform "models are worse at harder things" observation—it reveals that certain reasoning capabilities are shallow (models can handle simple cases but collapse when complexity increases beyond a narrow threshold) while others are deep (performance degrades gracefully). The paper's interpretation—that "current models have shallow competencies" and "may be learning to recognize and apply solution templates rather than developing robust reasoning strategies"—gains force from the differential pattern: if all capabilities were equally shallow, the cliffs would be uniformly steep across domains.

This is fundamentally different from prior benchmarking approaches. Fixed benchmarks like MATH or GSM8K report a single accuracy number per model, collapsing across problem difficulties. At best, they might report performance by coarse difficulty bins derived from human annotations (e.g., MATH's Level 1–5). But these bins are static and not under experimental control—a researcher cannot ask "what happens if I increase the polynomial degree by 1 while holding everything else constant?" With RG's parametric control, difficulty becomes a continuous experimental variable rather than a post-hoc stratification of a fixed dataset.

The difficulty cliff concept also provides a unified explanation for apparently contradictory findings in the reasoning literature. When some papers report strong self-correction capabilities (Madaan et al., 2023) and others find self-correction ineffective (Huang et al., 2023), the apparent contradiction may reflect different implicit difficulty distributions in the tested problems. RG makes this hypothesis testable: by varying difficulty parameters systematically, researchers can map the boundary conditions under which specific reasoning strategies succeed or fail. The paper does not fully exploit this capability—the cliff analysis in Section 3 is purely descriptive—but establishes the conceptual and infrastructural foundation for such studies.

Finally, the difficulty cliff data challenges a prevailing narrative in the field: that reasoning-specific training (via RLVR) produces models with robust, generalizable reasoning capabilities. The paper notes that reasoning models like o3-mini and DeepSeek-R1 substantially outperform non-reasoning models overall (63.5% vs. 41.5% average, Figure 3a), yet they still exhibit catastrophic performance cliffs on the hardest configurations. This suggests that current RLVR approaches improve reasoning within a comfort zone of problem complexity but do not fundamentally alter the brittleness that characterizes template-based problem solving. The cliff persists even for the best models—it is merely shifted to higher difficulty levels. This is a sobering finding that tempers optimism about RLVR as a path to robust reasoning.

Innovation 3: Cross-Domain Transfer as Evidence That Procedurally Generated RLVR Develops General Reasoning Primitives

The paper's most striking empirical finding—and the one with the deepest implications—is the demonstration that RLVR training on procedurally generated tasks from one domain improves performance on held-out tasks from entirely different reasoning domains. Training on algorithmic tasks (binary alternation, string manipulation, base conversion, etc.) yields a +29.1% improvement on algebra and +22.3% on geometry (Table 2). Training on logic tasks improves cognition (+13.3%) and graphs (+9.1%). Training on games—despite poor in-domain performance—transfers to algebra (+21.8%) and cognition (+13.1%).

This finding matters because it distinguishes between two competing hypotheses about what RLVR actually does. Hypothesis A: RLVR teaches the model to solve specific problem types by reinforcing successful solution templates for those problem types. Under this hypothesis, improvements should be largely task-specific—training on algebra improves algebra, but not geometry. Hypothesis B: RLVR teaches the model general reasoning strategies—breaking problems into subgoals, searching through constraint spaces, verifying intermediate results—that transfer across domains. The cross-domain transfer results provide strong evidence for Hypothesis B, at least within the range of domains tested.

The significance extends beyond the specific numbers. If RLVR were merely teaching task-specific heuristics, then scaling reasoning capabilities would require scaling the diversity of training tasks proportionally—each new reasoning domain would require its own training data. The cross-domain transfer results suggest a more efficient scaling path: training on a well-chosen subset of reasoning domains may develop capabilities that generalize broadly, reducing the data requirements for new domains. This is the reasoning analog of the observation in vision that features learned on ImageNet transfer to a wide range of downstream tasks—the existence of reasoning primitives that are shared across superficially distinct problem types.

The paper's external benchmark results (Section 4.3) provide converging evidence for this interpretation. Training on RG's math tasks (algebra + arithmetic + geometry) improves MATH by +9.7% and Big-Bench Hard by +7.7% (Table 3), despite these benchmarks containing problems that differ in surface form, domain context, and difficulty distribution from the RG training data. The MMLU-Pro results (Table 4) show improvements across diverse academic subjects (Physics +5.7%, Psychology +6.0%, Biology +4.2%) from RG-Math training—subjects that share no obvious surface-level similarity with the procedurally generated math problems used in training. This is strong evidence against the hypothesis that RLVR produces narrow, task-specific improvements.

However, the paper also provides important boundary conditions on transfer. The ARC and Games categories show minimal or negative transfer in some configurations (Table 2: RG-Algebra slightly decreases ARC performance, −2.3%; RG-Games decreases Games performance, −0.8%). These are precisely the categories the paper identifies as requiring "visual-spatial reasoning represented in text format"—a capability that may be fundamentally different from the symbolic manipulation and constraint satisfaction skills that transfer readily. This pattern suggests that transfer is not universal but depends on shared underlying reasoning mechanisms, and that the boundaries of transfer may themselves be informative about the structure of reasoning capabilities.

The paper does not claim to have identified the mechanism of transfer—it does not show, for instance, that models trained on algorithmic tasks learn to decompose algebra problems into subgoals, or that logic training improves systematic search in graph problems. But by establishing the existence of cross-domain transfer with clean experimental controls, it opens a research program into the nature of reasoning primitives that the current generation of fixed benchmarks cannot support.

Innovation 4: Curriculum RLVR as a Demonstrably Superior Training Paradigm, Enabled by Parametric Difficulty Control

Curriculum learning is not a new idea—the paper cites foundational work from Bengio et al. (2009) and surveys by Narvekar et al. (2020). What is new is the application to RLVR for language model reasoning with procedurally generated data, and the paper's demonstration that even a simple curriculum (monotonically increasing difficulty when performance exceeds a threshold) produces consistent and sometimes dramatic improvements over uniform sampling from the full difficulty distribution.

The quantitative results are striking. On Spell Backwards at word length 4, the curriculum model achieves 70.67% vs. 30.00% for the non-curriculum model—a +40.67 percentage point gap (Table 5). On Count Primes in the 100–500 range, the gap is +26.67 points (30.67% vs. 4.00%). These are not marginal improvements; they represent the difference between a model that can solve the task and one that essentially cannot. Importantly, the curriculum model uniformly outperforms the non-curriculum model at every difficulty level, not just the easier ones where it spent more training time. This is the signature of genuine curriculum benefit: the structured progression builds capabilities that transfer to harder instances, rather than merely inflating performance on the training distribution.

The finding has practical significance because curriculum learning in RLVR was previously difficult to implement with fixed datasets—you cannot systematically increase the difficulty of GSM8K problems because difficulty is a property of the fixed problem instances, not a controllable parameter. RG's parametric difficulty control makes curriculum learning a configurable training option (via the curricula section of the YAML configuration) rather than a research-project-level engineering effort. This lowers the barrier to experimenting with and deploying curriculum learning in reasoning research.

The paper also provides diagnostic information about when curriculum learning matters most. The Spell Backwards and Count Primes environments show large curriculum benefits; Mini Sudoku shows more modest gains (+2.0% to +13.3% depending on difficulty level). The paper does not analyze why some tasks benefit more than others, but the data suggests that tasks with a steep difficulty gradient—where moving from one level to the next introduces qualitatively new challenges (longer words to reverse, larger number ranges to search)—benefit more from structured progression than tasks where difficulty increases more continuously (incrementally more empty cells in a puzzle with the same underlying solving strategy). This is a hypothesis the paper leaves untested but that the infrastructure enables future work to investigate.

The Count Primes result deserves special attention: the curriculum model never advanced past the first difficulty level (the paper notes in Figure 6's caption: "no bars appear for Count Primes as the model never advanced past the initial level"), yet it still dramatically outperformed the non-curriculum model at that level. This suggests that the benefit of curriculum learning is not merely that the model sees progressively harder examples, but that not seeing examples that are too hard prevents destructive gradient updates early in training. The non-curriculum model, exposed to the full difficulty range from the start, may receive mostly negative rewards early on (since hard problems are unsolvable at initialization), leading to noisy or counterproductive updates. The curriculum model, by staying within its zone of proximal development, receives a higher proportion of positive reward signals that reinforce productive behaviors. This is a specific mechanistic hypothesis that the paper does not fully develop but that the data strongly supports.

5. Experimental Analysis

Evaluation Methodology

Dataset. All experiments use RG's own procedurally generated tasks. For zero-shot evaluation (Section 3), the paper evaluates on a single held-out set of 50 problems per task configuration (easy and hard). For RLVR training experiments (Sections 4–5), the dataset is specified as dataset_size: 20000 procedurally generated instances per epoch, with one epoch of training. For intra-domain and cross-domain transfer evaluations, models are tested on held-out tasks from the target domain using evaluation sets of 50 problems per task, with three independent runs to provide robust estimates (Section 4.1). External benchmark evaluations use the standard test splits of GSM8K, MATH, Big-Bench Hard, and MMLU-Pro, evaluated via the Language Model Evaluation Harness (Gao et al., 2024).
Base model(s). The primary model for all RLVR experiments is Qwen2.5-3B-Instruct (Yang et al., 2024), a 3-billion parameter instruction-tuned model. The paper chooses this model because it is "representative of the capabilities of many contemporary" open-weight models of modest scale (Section 4.1), making the experiments reproducible on academic compute budgets while leaving substantial headroom for improvement through RLVR—the base model already has some competency on the tasks but is far from saturation. For zero-shot evaluation (Section 3), the paper evaluates a broader set of frontier models: o3-mini (OpenAI, 2025), DeepSeek-R1 (Guo et al., 2025), Grok 3 Mini Beta, QwQ-32B (Qwen Team, 2025), Claude 3.5 Sonnet, Llama 4 Maverick, Llama 4 Scout, Gemma 3 27B, Mistral Small 3.1 24B, and Llama 3.3 70B. This range spans reasoning-optimized models (o3-mini, DeepSeek-R1, QwQ-32B) and general-purpose instruction-tuned models at scales from 3B to 70B parameters.
Metrics. The paper uses two distinct metric conventions. For zero-shot evaluation (Section 3), the metric is simply accuracy—the percentage of problems for which the model's answer exactly matches the ground truth according to the task-specific verifier. The paper computes this as a percentage in the range 0–100% and reports per-task and per-category averages. For RLVR training experiments (Sections 4–5), training is guided by a composite reward (accuracy + format + optional length bonus), but evaluation tables report only the accuracy component, rescaled to a percentage. For transfer experiments evaluating on external benchmarks (GSM8K, MATH, Big-Bench Hard, MMLU-Pro), standard benchmark-specific metrics are used (exact match for GSM8K and MATH; accuracy for Big-Bench Hard and MMLU-Pro). Acc@3 is reported for some RLVR experiments (Tables 1, 2), meaning the model generates 3 candidate answers and is scored correct if any one matches the ground truth—the paper does not specify whether this uses majority voting, best-of-3 with verifier selection, or pass@3, though the n: 8 in the GRPO configuration suggests that evaluation uses the same generation budget as training.
Baselines. For zero-shot evaluation, the baseline is the model's untuned performance—each model is evaluated zero-shot on RG tasks without any RLVR fine-tuning, and models are compared to each other (Section 3). For RLVR experiments, the baseline is Qwen2.5-3B-Instruct without RLVR fine-tuning (referred to as "Baseline" in Tables 1–5). For intra-domain transfer (Table 1), the RG-RLVR model is compared to the baseline on held-out tasks. For cross-domain transfer (Table 2), models fine-tuned on specific categories (RG-Algebra, RG-Algorithmic, RG-Logic, RG-Games) are compared to baseline on held-out domains. For curriculum learning (Table 5), two trained models—Non-Curriculum (trained on all difficulty levels uniformly) and Curriculum (trained with adaptive difficulty progression)—are compared to each other and to the baseline. For external benchmark transfer (Tables 3–4), the RG-Math and RG-Algorithmic models are compared to baseline.
Generation budget / compute accounting. All RLVR experiments use single-epoch training with 20,000 procedurally generated problems (Section 4, Appendix A.6). The total compute cost is reported as approximately 1500 A6000 hours, obtained by renting cloud GPUs from Runpod (Section 4). Training uses 4×A6000 GPU nodes, with each experiment running for 500 training steps (or 800 for the RG-Math model in Section 4.3). For inference during training, GRPO uses n: 8 generations per prompt (the model produces 8 candidate responses and computes advantages relative to the group mean). Zero-shot evaluation uses single greedy or sampled generation per problem (the paper does not specify temperature for zero-shot evaluation, though training uses temperature 1.0). For curriculum learning experiments, the budget is identical between curriculum and non-curriculum conditions—both train for the same number of steps on the same number of procedurally generated instances, differing only in the difficulty distribution of those instances.
Cross-validation / statistical protocol. For intra-domain and cross-domain transfer experiments, the paper conducts three independent runs on identical evaluation sets of 50 problems (Section 4.1), providing error estimates. For external benchmark evaluations (Tables 3–4), standard errors are reported alongside scores, computed via the Language Model Evaluation Harness. For zero-shot evaluations, Figures 7 and 8 (Appendix) report per-task scores as percentages without error bars, presumably computed from a single evaluation run per task. The paper does not report confidence intervals or statistical significance tests for differences between models—comparisons are based on point estimates of accuracy. The curriculum learning experiments (Table 5) appear to report results from single training runs (no error bars or multiple seeds), which is standard for RLVR experiments in this compute regime but limits the robustness of the reported differences, particularly for the smaller gaps (e.g., +2.0% on Mini Sudoku at 4–6 empty cells).

Main Quantitative Results

The experimental investigation spans five axes: zero-shot capabilities of frontier models and the difficulty cliff (Section 3), intra-domain transfer (Section 4.1), cross-domain transfer (Section 4.2), transfer to external benchmarks (Section 4.3), and curriculum learning (Section 5). I present each in turn.

Zero-Shot Performance and the Difficulty Cliff (Section 3)

Headline finding: Reasoning-optimized models substantially outperform general-purpose models on RG's hard task configurations (average 63.5% for o3-mini vs. 41.5% for Llama 4 Maverick; Figure 3a), but even the strongest models exhibit catastrophic performance degradation when task difficulty increases, with drops ranging from −2.3% (logic) to −71.9% (code) for o3-mini (Figure 3b).

Model hierarchy (Figure 3a). The paper evaluates 7+ frontier models on RG's "hard" configurations and finds a clear stratification. The top tier consists of reasoning-trained models: o3-mini leads at 63.51% average accuracy, followed by DeepSeek-R1 at 59.52% and Grok 3 Mini Beta at 55.06%. The second tier comprises general-purpose instruction-tuned models: Llama 4 Maverick (41.50%), QwQ-32B (40.52%—though this is itself a reasoning model, its performance notably lags o3-mini and DeepSeek-R1), Claude 3.5 Sonnet (40.33%), Llama 4 Scout (30.01%), and Gemma 3 27B (20.26%). The 22-percentage-point gap between the best reasoning model (o3-mini, 63.51%) and the best non-reasoning model (Llama 4 Maverick, 41.50%) represents what the paper calls "more than a marginal improvement" and attributes to RLVR unlocking "a step change in capabilities."

Per-task performance on easy settings (Figure 7, Appendix A.4). On easy configurations, the performance landscape is more compressed but still reveals systematic differences. o3-mini achieves perfect or near-perfect scores (100%) on a substantial fraction of tasks across algebra (complex_arithmetic, polynomial_equations, simple_equations, simple_integration), algorithmic (base_conversion, chain_sum, count_bits, decimal_chain_sum, letter_counting, etc.), arithmetic (basic_arithmetic, calendar_arithmetic, gcd, gsm_symbolic, lcm, etc.), and logic (knights_knaves, propositional_logic, syllogism). However, even on easy settings, certain categories resist saturation: figlet_font (Cognition) scores range from 0.0% to 10.0% across all models; rubiks_cube (Games) ranges from 0.0% to 6.8%; emoji_mystery (Cognition) from 0.9% to 56.4%. This indicates that some task types are inherently challenging for LLMs regardless of difficulty tuning.

Per-task performance on hard settings (Figure 8, Appendix A.5). The hard configurations reveal dramatic performance erosion. For o3-mini, tasks that scored 100% on easy settings drop substantially: complex_arithmetic from 98.0% (easy) to 98.0% (hard)—showing robustness; but codeio drops from 100% to 0.0%, binary_matrix drops from 92.0% to 32.0%, rubiks_cube from 6.8% to 1.2%, rush_hour from 39.2% to 0.0%. The paper's headline tasks from Figure 2—spiral_matrix, binary_matrix, pool_matrix, rush_hour, bf, emoji_mystery, poly_mult, count_primes, rubiks_cube, maze—all show non-trivial difficulty for frontier models, with o3-mini scoring 100%, 32%, 16%, 0%, 4%, 57%, 30%, 36.8%, 1.2%, and 50% respectively on hard settings. DeepSeek-R1 shows a similar pattern: complex_arithmetic drops from 98.0% to 98.0% (robust), codeio from 68.3% to 1.0% (catastrophic), rubiks_cube from 0.0% to 2.9% (remains low), rush_hour from 68.5% to 0.0%.

The difficulty cliff by category (Figure 3b). The paper quantifies the difficulty cliff as the difference in accuracy between easy and hard configurations, computed per category and per model. For o3-mini, the steepest cliffs occur in code (−71.93%), cognition (−27.52%), graphs (−33.13%), geometry (−33.13%), and algorithmic reasoning (−25.57%). The shallowest cliffs occur in algebra (−15.01%), logic (−2.30%), induction (−4.02%), and arithmetic (−21.20%). This pattern reveals that current models' capabilities are most fragile in domains requiring multi-step procedural execution (code), visual-spatial reasoning in text format (cognition, graphs), and constrained search (geometry). The paper interprets the relatively flat performance in logic and algebra as evidence that these capabilities are more robust to complexity scaling—though absolute performance on hard logic and algebra tasks still leaves room for improvement.

Claim 1 from the executive summary: "Zero-shot performance of frontier LLMs is low for many RG tasks, specifically the ones that represent visual concepts in text format like ARC, cognition, and games categories." Supported. The per-task data (Figures 7, 8) show that cognition and games tasks consistently score lower than algebra, arithmetic, and logic across all models. On hard settings (Figure 8), figlet_font scores range from 0.0% (multiple models) to 10.0% (o3-mini); emoji_mystery ranges from 0.0% to 57.0%; color_cube_rotation ranges from 0.0% to 50.8%; rubiks_cube from 0.0% to 2.9%; rush_hour from 0.0% to 0.0% across all evaluated models. These are the categories where even the best models fail most dramatically.

Claim 2: "Increasing task difficulty creates sharp performance cliffs. When transitioning from easy to hard configurations, performance drops are most severe in algorithmic reasoning (28%), code generation (62%), and graph problems (30%)." Supported with qualifications. The specific numbers in the executive summary are approximate—the paper reports −25.57% for algorithms, −71.93% for code, and −33.13% for graphs for o3-mini (Figure 3b). The 28%, 62%, and 30% figures don't precisely match any single model's numbers and appear to be rounded averages or approximate references. The qualitative claim—that these domains show the steepest cliffs—is clearly supported. However, the claim that the cliff is "sharp" in algorithmic reasoning requires nuance: −25.57% is substantial but less dramatic than code (−71.93%) or cognition (−27.52%). The paper would benefit from a clearer definition of what constitutes a "sharp" versus "moderate" cliff.

Claim 3: "Larger non-reasoning models often underperform smaller reasoning models. Performance drops are highest when transitioning from reasoning to non-reasoning models." Supported. The comparison in Figure 3a is cross-sectional (all models evaluated on the same hard tasks), not a transition experiment. But the data clearly show that QwQ-32B (32B parameters, reasoning-trained) at 40.52% underperforms o3-mini (model size undisclosed but likely smaller) at 63.51%, and that Llama 4 Maverick (likely larger than QwQ-32B) at 41.50% underperforms DeepSeek-R1 (model size undisclosed) at 59.52%. The "performance drops when transitioning from reasoning to non-reasoning models" claim is supported by the 22-percentage-point gap in Figure 3a.

Intra-Domain Transfer (Section 4.1)

Headline finding: RLVR training on a composite of tasks from a reasoning domain consistently improves performance on held-out tasks from the same domain, with improvements ranging from +2.0% (cognition) to +11.7% (algebra) above the Qwen2.5-3B-Instruct baseline (Table 1).

Training dynamics (Figure 4). The reward curves reveal distinct learning patterns across domains. All categories show a sharp initial increase in reward, which the paper attributes partly to format learning (the auxiliary format reward) and partly to "how quickly RLVR improves the model's ability to solve training tasks." Algebra, algorithmic, and arithmetic show sustained improvement through 400+ training steps. Cognition shows slower initial progress but accelerates. Games shows the most rapid initial improvement but plateaus early—the paper does not discuss why, but the low baseline performance (0.0% Acc@3, Table 1) and the visual-spatial nature of games tasks suggest fundamental difficulty.

Per-category results (Table 1). The paper trains Qwen2.5-3B-Instruct using GRPO on composite tasks from each category and evaluates on a held-out task (or composite of held-out tasks) from the same category, reporting Acc@3:

Category	Baseline Acc@3	RG-RLVR Acc@3	Improvement
Algebra	5.0%	16.7%	+11.7%
Algorithmic	52.3%	59.7%	+7.4%
Arithmetic	89.7%	96.0%	+6.3%
Cognition	40.3%	42.3%	+2.0%
Games	0.0%	3.3%	+3.3%

All five categories show improvement. The magnitude varies inversely with baseline performance: arithmetic, where the baseline already achieves 89.7%, shows the smallest absolute gain (+6.3%) due to ceiling effects; algebra, where the baseline is weakest among non-games categories (5.0%), shows the largest gain (+11.7%). The Games result is noteworthy because it demonstrates that RLVR can bootstrap entirely new capabilities—the baseline scores 0.0%, meaning it never produces a correct answer in 3 attempts, while the RLVR-trained model achieves 3.3%, indicating that RLVR has taught the model to solve at least some instances of a task it previously could not solve at all.

Claim 4 from the executive summary: "RLVR on procedurally generated tasks produces intra-domain transfer (e.g., +11.7% on held-out algebra tasks after training on other algebra generators)." Supported. The +11.7% figure matches Table 1 exactly. However, the paper does not specify which specific held-out algebra task was used for evaluation—the training composite and evaluation task are not enumerated for this experiment. This matters because the magnitude of transfer likely depends on the similarity between training and evaluation tasks; if the held-out task is very similar to one of the training tasks, +11.7% may overstate genuine generalization.

Cross-Domain Transfer (Section 4.2)

Headline finding: RLVR training on tasks from one reasoning domain transfers to held-out tasks from entirely different domains, with the strongest transfer observed from algorithmic training to algebra (+29.1%) and geometry (+22.3%), and from logic training to cognition (+13.3%) and graphs (+9.1%) (Table 2).

Training dynamics (Figure 5). The cross-domain training curves show distinct per-domain patterns. Algebra and logic training show the fastest reward increase and highest terminal rewards (80–100 range by step 400). Algorithmic training shows steady improvement but plateaus at moderate reward (50–70 range). Games training shows minimal improvement and plateaus early at low reward (30–40 range), consistent with the intra-domain games result. The paper notes that "reward values illustrate that some task categories are more challenging than others."

Cross-domain transfer matrix (Table 2). The paper trains four separate models (RG-Algebra, RG-Algorithmic, RG-Logic, RG-Games) and evaluates each on held-out domains. The results form a transfer matrix:

Training on algebra (RG-Algebra):

Tested on algorithmic: Baseline 13.49% → 20.68% (+7.2%)
Tested on arithmetic: Baseline 29.56% → 46.14% (+16.6%)
Tested on games: Baseline 8.40% → 9.23% (+0.8%)

Training on algorithmic (RG-Algorithmic):

Tested on algebra: Baseline 23.83% → 52.89% (+29.1%)
Tested on arithmetic: Baseline 29.56% → 45.17% (+15.6%)
Tested on geometry: Baseline 0.83% → 23.17% (+22.3%)
Tested on logic: Baseline 16.43% (note: RG-Algorithmic column shows this as a test for logic, but the table structure is complex—the paper reports RG-Algorithmic tested on algorithmic: 13.49 → -, and algorithmic column appears to show transfer to logic at +2.9%)

Training on logic (RG-Logic):

Tested on cognition: Baseline 11.62% → 24.94% (+13.3%)
Tested on graphs: Baseline 19.81% → 28.86% (+9.1%)
Tested on algebra: Baseline 23.83% → 45.61% (+21.8%)

Training on games (RG-Games):

Tested on algebra: Baseline 23.83% → 45.61% (+21.8%)
Tested on cognition: Baseline 11.62% → 24.71% (+13.1%)
Tested on graphs: Baseline 19.81% → 22.49% (+2.7%)

Key patterns. The most striking result is the transfer from algorithmic training to algebra (+29.1%) and geometry (+22.3%). The paper interprets this as evidence that "procedural reasoning skills generalize across mathematical domains"—training on tasks that require systematic, step-by-step computation (binary alternation, base conversion, string manipulation) improves performance on symbolic algebra and spatial geometry problems despite no shared surface features.

Logic training transfers broadly: +21.8% to algebra, +13.3% to cognition, +9.1% to graphs. This suggests that formal deduction skills—identifying valid inferences, tracking constraints, eliminating possibilities—transfer to domains that superficially appear unrelated but share underlying requirements for systematic reasoning under constraints.

Games training shows a surprising pattern: despite poor in-domain performance (Games baseline is the weakest in Table 1 at 0.0%, and the RG-Games training curve in Figure 5 plateaus at low reward), it transfers strongly to algebra (+21.8%) and cognition (+13.1%). The paper speculates that "constraint satisfaction skills transfer to other domains"—even if the model cannot solve the specific games tasks well, the training process may develop general strategies (exploring constraint spaces, backtracking, systematic elimination) that prove useful in algebra and pattern recognition.

Negative transfer results. Not all transfers are positive. RG-Algebra slightly decreases ARC performance (−2.3%, from 6.49% to 4.18%). RG-Games slightly decreases games performance (−0.8%, from 8.40% to 7.64%) and decreases ARC performance (−2.2%). The ARC category—which requires visual pattern recognition in text format—appears resistant to transfer from both algebraic and games training, consistent with the paper's broader finding that visual-spatial reasoning in text format is a distinct capability that current RLVR approaches do not effectively develop.

Claim 5 from the executive summary: "Cross-domain transfer—most strikingly, training on algorithmic tasks yields a +29.1% improvement on algebra." Supported. The +29.1% figure matches Table 2 (RG-Algorithmic → Algebra: Baseline 23.83% → 52.89%). However, note that the baseline for algebra in this experiment is 23.83%, while the baseline for algebra in Table 1 is 5.0%. This discrepancy indicates that different held-out tasks or evaluation configurations were used for the two experiments, and highlights that the baseline performance (and thus the transfer magnitude) is task-specific. A +29.1% absolute improvement from a 23.83% baseline is proportionally less dramatic than +11.7% from a 5.0% baseline—the model is already partially competent on the target task before transfer.

Claim 6: "RLVR generalizes across tasks from the same domain, from mathematics to games. We observe this intra-domain improvement in both tasks the LLM is already competent in, as well as tasks the pre-RLVRed model fails to solve." Supported for intra-domain transfer (Table 1), but the "from mathematics to games" language in the executive summary appears to reference cross-domain transfer (Table 2), not intra-domain. The claim that RLVR helps on tasks the model already solves (arithmetic: 89.7% → 96.0%) and tasks it fails at (games: 0.0% → 3.3%) is clearly supported by Table 1.

Transfer to External Benchmarks (Section 4.3)

Headline finding: Models trained on RG's procedurally generated math and algorithmic tasks show meaningful improvements on established reasoning benchmarks, including MATH (+9.7%), Big-Bench Hard (+7.7%), and multiple MMLU-Pro categories (Table 3, Table 4).

GSM8K, MATH, Big-Bench Hard (Table 3). The RG-Math model (Qwen2.5-3B-Instruct trained for 800 GRPO steps on a composite of algebra, arithmetic, and geometry tasks from RG) is evaluated against baseline:

Benchmark	Baseline	RG-Math	Improvement
GSM8K (8-shot, CoT)	76.2% (±1.17)	76.7% (±1.16)	+0.5%
MATH (0-shot, CoT)	48.5% (±0.68)	58.2% (±0.66)	+9.7%
Big-Bench Hard (3-shot, CoT)	8.68% (±0.30)	16.34% (±0.40)	+7.66%

The MATH and Big-Bench Hard improvements are substantial. The GSM8K improvement is marginal (+0.5 percentage points) and within the standard error range (±1.17 and ±1.16), making it statistically indistinguishable from zero. The paper notes this but does not offer an explanation—possibly GSM8K is near saturation for this model (76.2% baseline), or the GSM8K problem distribution differs from RG's math tasks in ways that limit transfer.

MMLU-Pro (Table 4). Both RG-Math and RG-Algorithmic models are evaluated on MMLU-Pro subtasks:

MMLU-Pro Task	Baseline	RG-Algorithmic	RG-Math
Math	54.63%	53.89% (−0.74)	60.25% (+5.62)
Computer Science	37.80%	40.73% (+2.93)	42.20% (+4.40)
Physics	38.49%	39.26% (+0.77)	44.19% (+5.70)
Engineering	28.28%	31.48% (+3.20)	31.17% (+2.89)
Economics	50.59%	53.44% (+2.85)	53.55% (+2.96)
Business	51.58%	53.36% (+1.78)	54.12% (+2.54)
Psychology	50.75%	55.01% (+4.26)	56.77% (+6.02)
Biology	56.90%	59.00% (+2.10)	61.09% (+4.19)

RG-Math improves over baseline on all eight subtasks, with the largest gains in Psychology (+6.02%), Physics (+5.70%), and Math (+5.62%). RG-Algorithmic improves on all subtasks except Math (−0.74%), with the largest gains in Psychology (+4.26%) and Engineering (+3.20%). The fact that RG-Algorithmic training improves Psychology and Biology performance—domains with no obvious connection to algorithmic reasoning—is particularly striking and supports the paper's claim that RLVR develops transferable reasoning skills rather than domain-specific heuristics.

Claim 7 from the executive summary: "Skills transfer to external benchmarks. RLVR training on RG tasks improves performance on benchmarks such as MATH (+9.7%), GSM8K (+0.5%), Big-Bench Hard (+7.7%), and MMLU-Pro." Supported, but the GSM8K result (+0.5%) is not statistically significant given the reported standard errors. The executive summary should qualify this claim. The MATH (+9.7%) and Big-Bench Hard (+7.7%) results are robust; the MMLU-Pro results are consistently positive across subtasks and domains.

Curriculum RLVR (Section 5)

Headline finding: Curriculum learning—where difficulty increases only after the model surpasses a 70% success threshold—substantially outperforms fixed-difficulty training across all tested environments and difficulty levels, with the largest gains on tasks where the curriculum model never advances beyond the initial difficulty level (Table 5, Figure 6).

Training dynamics (Figure 6). The paper visualizes reward curves for three environments: Spell Backwards, Mini Sudoku, and Count Primes. Key observations:

Spell Backwards: The curriculum model shows a sawtooth pattern—rewards rise to the 70% threshold, difficulty increases (vertical black lines), rewards drop sharply, then rise again. The curriculum model's terminal reward is lower than the non-curriculum model's because "its exclusive exposure to maximum difficulty examples, while the non-curriculum model samples across the full difficulty distribution." This is expected: the curriculum model is solving harder problems on average by the end of training.
Mini Sudoku: The curriculum model "accelerates through the difficulty levels, reaching the highest level by step 72." This indicates that the model quickly masters easier configurations and progresses rapidly to harder ones—the curriculum is not a bottleneck but a launchpad.
Count Primes: The curriculum model "never advanced past the initial level." Yet, as Table 5 shows, it dramatically outperforms the non-curriculum model at that level. The paper does not fully explain this, but the implication is that avoiding exposure to unsolvable hard examples early in training is beneficial—the non-curriculum model's training is contaminated by negative rewards from problems it cannot solve, while the curriculum model receives a cleaner signal.

Per-environment results (Table 5). The paper reports accuracy per difficulty level for baseline, non-curriculum, and curriculum models:

Spell Backwards (levels = word length):

Word Length	Baseline	Non-Curriculum	Curriculum	Curriculum Gain over Non-Curriculum
4	12.00%	30.00%	70.67%	+40.67%
6	3.33%	10.67%	30.00%	+19.33%
8	0.00%	1.13%	3.37%	+2.24%
10	0.00%	0.01%	0.01%	+0.00%

The curriculum model outperforms non-curriculum at every level, with the largest absolute gain at the easiest level (+40.67%) and diminishing gains at harder levels as performance approaches zero for both approaches. At word length 10, neither model succeeds meaningfully—the task is beyond the capability of a 3B-parameter model regardless of training strategy.

Mini Sudoku (levels = # empty cells):

Empty Cells	Baseline	Non-Curriculum	Curriculum	Gain
4–6	1.13%	54.00%	56.00%	+2.00%
6–8	0.00%	25.33%	28.00%	+2.67%
8–10	0.00%	6.67%	20.00%	+13.33%
10–12	0.00%	1.13%	5.33%	+4.20%

The curriculum benefit is modest at easy and moderate levels (+2.0% to +2.67%) but becomes substantial at harder levels (+13.33% at 8–10 empty cells). This is the reverse of the Spell Backwards pattern—curriculum helps most at intermediate difficulty, not at the easiest level.

Count Primes (levels = number range):

Number Range	Baseline	Non-Curriculum	Curriculum	Gain
100–500	12.00%	4.00%	30.67%	+26.67%
100–1000	3.33%	5.03%	12.67%	+7.64%
100–5000	1.33%	0.00%	3.03%	+3.03%

The most dramatic result is at the easiest level (100–500): the curriculum model achieves 30.67% vs. 4.00% for non-curriculum—a +26.67 percentage point gap. Strikingly, the non-curriculum model actually underperforms the baseline (12.00% vs. 4.00%) at this level, suggesting that training on the full difficulty distribution is actively harmful for solving easy problems. The curriculum model, which never advanced past this level, substantially outperforms baseline (+18.67%). At harder levels, both trained models outperform baseline, with curriculum leading.

Claim 8 from the executive summary: "Adaptive curriculum learning—where difficulty increases only after the model surpasses a 70% success threshold—substantially outperforms fixed-difficulty training (e.g., +40.67% on Spell Backwards at word length 4, +26.67% on Count Primes in the 100–500 range)." Strongly supported. These figures match Table 5 exactly. The consistency of the curriculum advantage across all environments and difficulty levels—even when the curriculum model never progresses past the initial level—provides robust evidence that structured difficulty progression improves RLVR outcomes.

Ablation Studies and Robustness Checks

The paper does not report formal ablation studies in the traditional sense (varying individual components of the system and measuring impact). However, several experimental choices serve as implicit ablations:

Non-curriculum vs. curriculum training (Table 5, Figure 6): This is the paper's primary ablation-like comparison. It tests whether the difficulty ordering of training data matters, holding all other factors constant (model, optimizer, total steps, dataset size, reward structure). The finding that curriculum consistently outperforms non-curriculum is the paper's strongest causal claim about the importance of training data organization.

Intra-domain vs. cross-domain transfer (Tables 1 and 2): By training on one domain and testing on held-out tasks from the same domain (Table 1) versus entirely different domains (Table 2), the paper implicitly ablates the relationship between training and evaluation distributions. The finding that cross-domain transfer is often comparable to or larger than intra-domain transfer (+29.1% algorithmic → algebra vs. +11.7% algebra → algebra) is non-obvious and suggests that domain boundaries may be less important than the underlying reasoning primitives being exercised.

RG-Math vs. RG-Algorithmic on MMLU-Pro (Table 4): By comparing a math-trained model and an algorithmic-trained model on the same MMLU-Pro subtasks, the paper ablates the source domain of training data. The finding that RG-Math generally outperforms RG-Algorithmic on math-adjacent subtasks (Math: +5.62% vs. −0.74%; Physics: +5.70% vs. +0.77%) while both models improve on non-math subtasks (Psychology, Biology, Business) suggests that domain-specific training provides additional benefit on closely related tasks, but that even unrelated training develops broadly applicable skills.

Missing ablations. Several comparisons that would strengthen the paper's claims are absent:

No ablation of the composite reward structure. The paper uses accuracy + format + length rewards but never reports experiments training with accuracy-only rewards. This leaves open the question of whether the format and length bonuses are necessary for the observed improvements, or whether they merely accelerate format learning early in training without affecting final reasoning capability.
No ablation of dataset composition or size. The paper trains on composites of multiple generators (e.g., 10 algorithmic tasks for RG-Algorithmic) but never varies the number of training tasks or the total dataset size. It is unknown whether training on a single generator would produce similar transfer, or whether diversity of training tasks is crucial.
No comparison to non-procedural baselines. The paper does not compare RLVR on RG data to RLVR on existing fixed datasets (e.g., training on GSM8K and testing on MATH, or vice versa). Such a comparison would directly test the paper's claim that procedurally generated data is a viable alternative to human-curated data for RLVR.
No model scale ablation. All RLVR experiments use Qwen2.5-3B-Instruct. The paper does not investigate whether the observed transfer patterns and curriculum benefits scale with model size, or whether they are specific to the 3B parameter regime.
No evaluation of the difficulty estimation cost. Unlike the earlier example paper (which identified the cost of difficulty estimation as a key limitation), this paper's curriculum approach uses an automatic scheduler that evaluates model performance on the training distribution itself, incurring negligible additional cost. However, the paper does not discuss the sensitivity of curriculum progression to the choice of threshold (70%) or evaluation window (20 steps).

Critical Assessment

The experiments demonstrate several important phenomena—the difficulty cliff, intra-domain transfer, cross-domain transfer, external benchmark improvement, and curriculum learning benefits—but the evidential strength varies considerably across claims. Here I assess each major claim against what the experiments actually show.

Claim: "Procedurally generated data can substitute for human-curated data in RLVR." The experiments provide suggestive but not definitive evidence. The external benchmark improvements (Tables 3–4) show that skills developed on RG data transfer to established benchmarks, which is positive. However, the paper never directly compares RG-based RLVR to benchmark-based RLVR (e.g., training on GSM8K vs. training on RG's gsm_symbolic and testing on MATH). Without this comparison, we cannot know whether RG data is as effective as human-curated data for developing reasoning capabilities—only that it is effective to some degree. The +9.7% improvement on MATH is impressive, but we don't know whether training directly on MATH problems (with held-out validation) would have produced a larger improvement. This is the central unanswered question for the paper's framing as a data scarcity solution.

Claim: "Skills transfer across domains, indicating development of general reasoning primitives." The cross-domain transfer results (Table 2) provide the strongest evidence in the paper. The +29.1% improvement on algebra from algorithmic training is striking and difficult to explain as anything other than genuine transfer. However, the evidence base has important limitations:

Small evaluation sets: All transfer experiments use evaluation sets of 50 problems per task, with three independent runs. This is small—with 50 problems, a single additional correct answer changes accuracy by 2 percentage points. Confidence intervals are not reported, making it difficult to assess whether the observed transfer magnitudes are reliable.
Single model family and scale: All transfer experiments use Qwen2.5-3B-Instruct. We don't know whether transfer patterns generalize to other architectures (Llama, Mistral, Gemma) or scales (7B, 13B, 70B). It is possible that 3B models, which have limited capacity, benefit more from transfer effects than larger models that already possess stronger reasoning capabilities—or, conversely, that larger models would show even stronger transfer because they can better leverage learned reasoning primitives.
Unclear relationship between training and evaluation tasks: The paper does not enumerate the specific training composites and held-out evaluation tasks for each experiment. For the algorithmic → algebra transfer (+29.1%), we don't know which algorithmic tasks were trained on or which algebra tasks were evaluated on. If the held-out algebra task happens to share structural features with a trained algorithmic task (e.g., both involve sequence manipulation), the "cross-domain" transfer may partially reflect task similarity rather than genuine domain transfer.
Negative transfers are underdiscussed: The ARC category shows negative or near-zero transfer from both algebra (−2.3%) and games (−2.2%) training. This is an important boundary condition that the paper acknowledges but does not analyze deeply. It suggests that transfer is not universal and that certain reasoning capabilities—visual-spatial pattern recognition in text format—may be fundamentally different from the symbolic reasoning that RG's procedurally generated tasks primarily exercise.

Claim: "Curriculum RLVR substantially outperforms fixed-difficulty training." This claim is well-supported by the data in Table 5, but its scope is limited:

Only three environments tested: Spell Backwards, Mini Sudoku, and Count Primes. These are all from the algorithmic/arithmetic categories. We don't know whether curriculum benefits extend to other categories (algebra, logic, games, cognition) or whether the benefit magnitude depends on task type.
Simple one-dimensional curricula: Each environment uses a single difficulty parameter (word length, empty cells, number range). Real-world reasoning curricula would likely involve multi-dimensional difficulty progression. The paper does not test whether multi-parameter curricula (simultaneously increasing grid size and obstacle density in a maze, for instance) produce similar benefits.
No comparison to alternative curriculum strategies: The paper uses a simple threshold-based approach (increase difficulty when accuracy >70%). It does not compare to alternatives like hand-designed fixed schedules, paced curriculum (increasing difficulty every N steps regardless of performance), or anti-curriculum (hard-to-easy). This makes it impossible to attribute the benefit to the adaptive nature of the curriculum rather than to the mere fact of seeing easier problems first.
Mechanism is unclear: The Count Primes result—where the curriculum model never advances past the first level but dramatically outperforms non-curriculum at that level—is fascinating but unexplained. The paper speculates that the non-curriculum model is harmed by early exposure to unsolvable hard problems, but this hypothesis is not tested (e.g., by running a variant where hard problems are introduced only after a warmup period, or by analyzing the gradient statistics of early training steps).

Claim: "The difficulty cliff reveals that current models have shallow competencies." This is an interpretive claim that goes beyond what the experiments strictly demonstrate. The data (Figures 3b, 7, 8) clearly show that performance drops when difficulty increases—but this is true of almost any capability measure. The claim that this reveals "shallow competencies" and "template matching rather than robust reasoning" is an interpretation that the paper does not validate through additional experiments. To support this interpretation, one would want to see evidence that:

Models fail on hard instances in systematically different ways than on easy instances (e.g., producing plausible-looking but logically flawed reasoning chains rather than giving up).
The difficulty cliffs persist even when models are explicitly trained on the harder configurations (the RLVR experiments in Sections 4–5 partially address this, but they use a different, smaller model than the frontier models in the zero-shot evaluation).
Performance on hard problems is not merely a function of the model having seen similar problems during pretraining—which would suggest template matching rather than reasoning.

The paper does not conduct these analyses, so the "shallow competencies" interpretation remains a plausible hypothesis rather than an established finding.

The most significant experimental gap is the absence of a direct comparison between RLVR on RG data and RLVR on existing benchmark data. The paper's core pitch is that procedurally generated data addresses the "fundamental scalability bottleneck" of fixed datasets, but it never demonstrates that RG data is comparably effective for developing reasoning capabilities. The external benchmark transfer results (Tables 3–4) show that RG training improves benchmark performance, but they don't show that it improves it as much as training directly on those benchmarks would. This is a difficult experiment to run cleanly (due to data contamination concerns and the difficulty of creating held-out splits of fixed benchmarks), but it is the experiment that would most directly validate the paper's central claim.

A second gap is the small scale of the transfer and curriculum experiments relative to the claims being made. Three independent runs on 50-problem evaluation sets, with a single model family at a single scale, provide suggestive but not robust evidence for general claims about reasoning transfer. The paper would benefit from: (1) larger evaluation sets (200+ problems per task) to reduce variance; (2) multiple model families (Qwen, Llama, Mistral) to test robustness; (3) multiple scales (3B, 7B, 13B) to test whether effects scale or saturate; and (4) formal statistical tests rather than raw accuracy comparisons.

A third gap concerns the relationship between training and evaluation in the transfer experiments. The paper uses Acc@3 (accuracy with 3 attempts) for RLVR evaluation but reports standard single-generation accuracy for zero-shot evaluation—these metrics are not directly comparable. Furthermore, the paper does not specify whether the evaluation tasks in Tables 1 and 2 use the same difficulty parameters as the training tasks, or whether they sample from a held-out difficulty range. If the evaluation tasks use similar difficulties to the training tasks, the observed "transfer" may partially reflect the model learning to solve problems at specific difficulty levels rather than learning general reasoning strategies.

The strengths of the experimental design should also be acknowledged:

The procedural generation ensures that training and evaluation instances are always distinct, eliminating the memorization confound that plagues fixed-benchmark studies.
The use of multiple independent runs (3 per transfer experiment) provides some protection against random variation, even if the sample sizes are small.
The cross-benchmark external evaluation (GSM8K, MATH, Big-Bench Hard, MMLU-Pro) provides converging evidence that RG training develops transferable skills, using established benchmarks with standardized evaluation protocols.
The curriculum learning experiments, despite being limited to three environments, provide clean within-task comparisons with identical compute budgets, model architectures, and reward structures—the only difference is the ordering of training examples.

In summary: The experiments establish that RLVR on RG's procedurally generated data produces measurable improvements that transfer within domains, across domains, and to external benchmarks, and that curriculum learning improves RLVR efficiency compared to uniform difficulty sampling. These are genuine contributions. However, the paper overclaims in two respects: (1) it presents the difficulty cliff as revealing "shallow competencies" and "template matching" without conducting the analyses that would distinguish this interpretation from simpler explanations (models are worse at harder things), and (2) it implicitly claims that procedurally generated data is a viable replacement for human-curated data without ever conducting the head-to-head comparison that would validate this claim. The transfer results provide the strongest support for the paper's core argument; the zero-shot and curriculum results provide useful but narrower contributions.

6. Limitations and Trade-offs

6.1 Procedural Generation Cannot Capture All Forms of Reasoning

The assumption. The paper's entire value proposition rests on the premise that procedurally generated tasks can serve as effective training data for developing reasoning capabilities that transfer to real-world problems. However, the procedural generation approach imposes a hard constraint: every task must have an algorithmically verifiable answer, which means the space of supported tasks is restricted to those with deterministic, automatically checkable outputs. The paper acknowledges this limitation explicitly:

"Some reasoning domains, particularly those requiring extensive domain knowledge or creativity, are difficult to capture with procedural generators. In particular, procedural generators may struggle in domains where answers are unstructured, leaving room for alternative RL data." (Section 7)

"Verification functions, while comprehensive, may not capture all aspects of solution quality that humans consider important. There is still an important place for human-centric mechanisms, such as RL from human feedback, and human-gathered RL datasets." (Section 7)

The consequence. RG cannot support several reasoning capabilities that matter enormously in practice. Tasks requiring creative writing, argument evaluation, nuanced judgment, explanation generation, or multi-dimensional quality assessment are structurally excluded because they lack clean correctness signals. More subtly, even within "objective" reasoning domains, procedural generation may fail to capture the richness of real problems. A procedurally generated algebraic equation can test symbolic manipulation, but it cannot test whether a model understands why a particular algebraic technique is appropriate—a capability that requires problems with real-world context, ambiguous specification, or multiple valid solution approaches of varying quality. The paper's own results hint at this limitation: the cognitive and games categories—which require pattern recognition, spatial reasoning, and constraint satisfaction in text-based representations—show the worst zero-shot performance (Section 3) and the weakest transfer effects (Tables 1–2), suggesting that these reasoning modalities are poorly developed by current RLVR approaches.

Furthermore, the reliance on exact-match verification may introduce a brittleness that does not exist with human evaluation. A model that solves a math problem correctly but formats the answer slightly differently from what the verifier expects receives zero reward—a false negative that would not occur with a human evaluator. The paper partially mitigates this with format rewards (Section 3.3), but this only incentivizes adherence to known formats; it does not solve the more fundamental problem that many valid reasoning outputs cannot be reduced to a single canonical answer string.

Evidence in the paper. The zero-shot results (Figure 3a, Figures 7–8) show stark performance stratification across categories: algebra, arithmetic, and logic tasks achieve high scores across models, while cognition and games tasks—which arguably require reasoning types least amenable to exact-match verification—score dramatically lower. The paper itself notes that "tasks requiring visual-spatial reasoning represented in text format (cognition, games) prove particularly challenging, with even the strongest models achieving less than 50% accuracy" (Section 3.1). The external benchmark transfer results (Tables 3–4) provide some reassurance that skills developed on RG's verifiable tasks transfer to more open-ended domains, but this is evidence of partial transfer, not equivalence.

Mitigation status. The paper acknowledges this limitation candidly (Section 7) and positions human-centric RL mechanisms as complementary, not competing. It does not attempt to fix the limitation—it is a fundamental constraint of the algorithmic verifiability design principle (P1). The paper suggests (without developing) that future work could incorporate multi-turn or multimodal reasoning tasks, which would modestly expand coverage but not fundamentally address the exclusion of open-ended reasoning.

6.2 Evaluation Is Limited to a Single Model Family at a Single Scale

The assumption. All RLVR experiments—intra-domain transfer, cross-domain transfer, curriculum learning, and external benchmark evaluation—use Qwen2.5-3B-Instruct as the base model. The paper selects this model because it is "representative of the capabilities of many contemporary LLMs" (Section 3) and because its size (3B parameters) makes experiments reproducible on academic compute budgets. The implicit assumption is that the observed phenomena—transfer across domains, curriculum benefits, difficulty cliffs—generalize across model architectures, scales, and training regimes.

The consequence. This single-model experimental design leaves several critical questions unanswered:

Architecture dependence. Qwen2.5 uses a specific architecture, tokenizer, and training recipe. Do the transfer patterns observed (e.g., algorithmic training → algebra transfer of +29.1%) replicate on Llama, Mistral, or Gemma models? If transfer depends on how the base model's representations organize reasoning primitives, results may be architecture-specific.
Scale dependence. A 3B model has limited capacity. It is possible that smaller models benefit more from cross-domain transfer (because they lack strong pre-existing reasoning capabilities and have room to improve) or less (because they lack the capacity to learn reusable reasoning primitives that generalize). Larger models (7B, 13B, 70B) might show different transfer patterns or curriculum benefits. The paper's difficulty cliff results (Figure 3b) show that even frontier models like o3-mini and DeepSeek-R1 exhibit dramatic performance degradation, but we cannot know whether those models, if RLVR-trained on RG data, would show the same transfer patterns as Qwen2.5-3B.
Training regime dependence. Qwen2.5-3B-Instruct is an instruction-tuned model that has already undergone significant supervised fine-tuning. The RLVR improvements reported may partially reflect the interaction between the instruction-tuning and the RLVR phase—a base (non-instruction-tuned) model might respond differently to RLVR on RG data.

Evidence in the paper. The zero-shot evaluation (Section 3) evaluates multiple model families and scales (o3-mini, DeepSeek-R1, QwQ-32B, Llama 4 Maverick, Claude 3.5 Sonnet, Gemma 3 27B, etc.) on RG tasks, demonstrating that the tasks themselves are model-agnostic benchmarks. However, all RLVR training experiments (Sections 4–5) use exclusively Qwen2.5-3B-Instruct. The paper does not report any RLVR results with alternative base models. The compute budget (~1500 A6000 hours, Section 4) provides a practical explanation—training multiple model families would multiply costs—but does not change the inferential limitation.

Mitigation status. The paper does not address this limitation directly. It does not claim that results generalize across architectures or scales, but neither does it caveat the transfer and curriculum findings as potentially model-specific. The external benchmark transfer results (Tables 3–4) provide some reassurance in that the improvements are measured on established benchmarks using standardized evaluation, making them comparable to other work—but the source of those improvements (Qwen2.5-3B + RG RLVR) remains unexplored as a potentially model-specific interaction. Future work would need to replicate key experiments (particularly the cross-domain transfer matrix, Table 2, and the curriculum results, Table 5) with at least one additional model family and one larger scale.

6.3 The Absence of a Direct Comparison to Fixed-Dataset RLVR Weakens the Core Claim

The assumption. The paper's central pitch is that procedurally generated data from RG addresses the "fundamental scalability bottleneck" of fixed reasoning datasets (Section 1). The implicit assumption is that RG data is comparably effective to human-curated data for developing reasoning capabilities via RLVR—if procedural generation produced significantly weaker improvements than training on existing benchmarks, the scalability argument would be less compelling (unlimited mediocre data vs. limited high-quality data).

The consequence. The paper never conducts the experiment that would most directly validate its framing: a head-to-head comparison between RLVR on RG data and RLVR on existing benchmark data, with both models evaluated on held-out benchmarks. Without this comparison, we cannot answer:

Does training on RG's gsm_symbolic produce comparable MATH improvements to training directly on GSM8K?
Does training on RG's algebra composite produce comparable algebra improvements to training on MATH's algebra subset?
Is the diversity of RG's 100+ generators more valuable than the human-crafted quality of fixed datasets?

The external benchmark results (Tables 3–4) show that RG training improves benchmark performance, and this is valuable. But it does not show that RG training is as effective as training on those benchmarks. A practitioner deciding whether to invest in RG-based RLVR versus curating their own domain-specific training data needs this comparison to make an informed decision.

The paper acknowledges related work that does train directly on benchmarks: Tülu 3 builds its RL corpus from GSM8K and MATH; DeepSeek-R1 uses manually curated chain-of-thought examples plus automatically gradable reasoning tasks. These represent the "fixed dataset RLVR" baselines against which RG should ideally be compared, but the paper treats them as related work rather than as baselines to beat or match.

Evidence in the paper. None. The paper reports RLVR improvements relative to the Qwen2.5-3B-Instruct zero-shot baseline (Tables 1–5), and it reports zero-shot evaluations of frontier models on RG tasks (Section 3). It never trains a model on fixed-benchmark data and evaluates it on RG tasks, nor does it train on RG data and compare to published results of benchmark-trained models. The external benchmark results (Tables 3–4) compare RG-RLVR models to an untuned baseline, not to a benchmark-RLVR model.

Mitigation status. Not addressed. The paper does not acknowledge this as a missing comparison. It is possible that the comparison is intentionally omitted because of contamination concerns—many existing benchmarks may be in the pretraining data of Qwen2.5-3B, making a "train on GSM8K, test on MATH" comparison confounded by data leakage. But the paper does not articulate this or any other rationale. This is the single most important experiment for validating the paper's core claim, and its absence leaves a significant gap in the evidence base.

6.4 Transfer and Curriculum Results Are Based on Small, Underspecified Evaluation Sets

The assumption. The intra-domain transfer (Table 1), cross-domain transfer (Table 2), and curriculum learning experiments (Table 5) evaluate models on held-out sets of 50 problems per task, with three independent runs for transfer experiments (Section 4.1) and an unspecified but likely similar protocol for curriculum experiments. The implicit assumption is that 50 problems provide a reliable estimate of model capability, and that the specific held-out tasks chosen are representative of their domains.

The consequence. With 50 problems, a single additional correct answer changes accuracy by 2 percentage points. Three independent runs provide some protection, but when differences between models are in the range of +2.0% to +7.4% (as in Table 1), they may fall within the range of sampling variability. The paper does not report confidence intervals, standard errors, or statistical tests for the transfer and curriculum results (unlike the external benchmark results in Tables 3–4, which report standard errors from the Language Model Evaluation Harness). This makes it difficult to assess:

Which transfer results are robust. A +2.0% improvement on cognition (Table 1) with 50 evaluation problems could easily arise from random variation (one or two additional lucky guesses). The +11.7% on algebra is more likely robust, but we cannot quantify the uncertainty.
Whether cross-domain transfer magnitudes are reliably different from each other. The paper highlights the +29.1% algorithmic-to-algebra transfer as "striking," but this is compared to an algebra baseline of 23.83% (Table 2)—a different baseline than the 5.0% in Table 1. The difference between transfer source domains (algorithmic vs. logic vs. games) may be influenced by which specific held-out tasks were used, but the paper does not specify which evaluation tasks were selected for each experiment.
The interaction between task selection and transfer magnitude. The paper does not enumerate the specific tasks used in the training composites or the held-out evaluation tasks for Tables 1 and 2. For the algorithmic → algebra transfer, we don't know which algorithmic tasks were trained on or which algebra tasks were evaluated on. If the held-out algebra task happens to share structural features with one of the algorithmic training tasks, the +29.1% may partially reflect task similarity rather than genuine domain transfer.

Evidence in the paper. The paper states the evaluation protocol in Section 4.1: "Each experiment involved three independent runs on identical evaluation sets of 50 problems." For Tables 1 and 2, this suggests the reported scores are the average across three runs, but no variance measures are reported. For Table 5 (curriculum), the paper does not specify the number of runs or report error estimates. The external benchmark evaluations (Tables 3–4) do report standard errors, which is appropriate—these use established benchmarks with standardized evaluation—but the internal RG evaluations lack the same rigor.

Mitigation status. Partially addressed through the use of three independent runs for transfer experiments, but without reporting the resulting variance, the protection is incomplete. The paper could improve confidence by: (1) reporting standard deviations or confidence intervals alongside all RLVR evaluation results, (2) using larger evaluation sets (200+ problems, which procedural generation makes trivially cheap), (3) specifying exactly which training and evaluation tasks were used, so readers can assess the similarity between them, and (4) conducting statistical tests on the differences between baseline and RLVR-trained models.

6.5 The Single-Turn, Text-Only Paradigm Limits Applicability to Multi-Turn and Multimodal Reasoning

The assumption. All tasks in RG are single-turn, text-based problems: the model receives a prompt, generates a response, and receives a reward. There is no interaction, no multi-step dialogue, no environment feedback beyond the final correctness judgment, and no visual or multimodal inputs.

"The current RG implementation focuses on single-turn, text-based reasoning and does not yet include multi-turn or multimodal reasoning tasks. Work on these is valuable to provide data for enhancing agentic and vision-language models, respectively." (Section 7)

The consequence. This limitation restricts RG's applicability to two rapidly growing areas of reasoning research:

Agentic reasoning. Many real-world reasoning tasks are inherently interactive: a model must take actions, observe outcomes, and adjust its strategy based on feedback (e.g., using tools, browsing the web, interacting with APIs, debugging code through execution feedback). RG's single-turn paradigm cannot train or evaluate these capabilities. The reward is binary and delayed until the final answer; there is no mechanism for rewarding productive intermediate actions or penalizing unproductive ones.
Multimodal reasoning. Vision-language models increasingly need to reason about images, diagrams, charts, and spatial layouts. The paper explicitly notes that tasks requiring "visual-spatial reasoning represented in text format" are particularly challenging (Section 3.2), but the deeper limitation is that RG cannot train multimodal reasoning because it provides no visual inputs—only text descriptions of visual concepts. A model that excels at RG's text-based maze solving may still fail at visual maze solving because the modalities require different perceptual grounding.

Beyond these specific domains, the single-turn limitation means RG cannot be used to study reasoning phenomena that emerge from extended interaction: iterative refinement of solutions, learning from mistakes within a session, strategic information-gathering, or multi-step planning with environmental feedback. These are central to how humans reason in complex, open-ended situations, and their exclusion from RG's training paradigm means that models trained solely on RG data will not develop these capabilities.

Evidence in the paper. The difficulty cliff analysis (Section 3.2) shows that cognition and games tasks—which often require the kind of multi-step, interactive reasoning that single-turn evaluation poorly captures—are the categories where models perform worst. The paper cites related work on multi-turn RL (Wang et al., 2025; Zeng et al., 2025) and multimodal RL (Chen et al., 2025; Paglieri et al., 2025) but does not incorporate these capabilities into RG.

Mitigation status. The paper explicitly acknowledges this as a limitation and describes it as future work (Section 7). The current RG release is single-turn and text-only. The paper does not provide a roadmap or timeline for multi-turn or multimodal extensions, making this a deferred rather than partially addressed limitation. For practitioners working on agentic or multimodal reasoning, RG in its current form provides limited utility as a training resource, though it may still serve as a component of a larger training pipeline.

6.6 No Accounting for Catastrophic Forgetting or Long-Term Skill Retention

The assumption. All RLVR experiments train for a single epoch on 20,000 procedurally generated instances (or 800 steps for the RG-Math model). The evaluation is conducted immediately after training on held-out RG tasks or external benchmarks. The implicit assumption is that the observed improvements are stable—that the model retains the reasoning capabilities it develops through RLVR and does not degrade in other capabilities (general language understanding, instruction following, factual knowledge).

The consequence. Catastrophic forgetting is a well-documented phenomenon in continual learning and RL fine-tuning of language models: when a model is optimized for a specific reward signal, it can lose capabilities that were present in the base model but are not reinforced during training. The paper's experiments do not test for this in any systematic way:

No evaluation of general capability retention. The paper evaluates on reasoning tasks (RG tasks, GSM8K, MATH, Big-Bench Hard, MMLU-Pro) but does not evaluate on standard LLM benchmarks (MMLU, HellaSwag, ARC-Challenge, etc.) to check whether RLVR on RG data degrades broad knowledge or language understanding.
No evaluation of instruction-following retention. The base model (Qwen2.5-3B-Instruct) is instruction-tuned. RLVR with a specific format reward may shift the model's output distribution toward the formats required by RG tasks, potentially degrading its ability to follow diverse instructions in open-ended settings.
No test of whether improvements persist. The paper evaluates immediately after training. It does not test whether the +9.7% MATH improvement (Table 3) or the +29.1% algorithmic-to-algebra transfer (Table 2) persists after further training on other tasks, or whether the capabilities decay over time or with additional fine-tuning.

The curriculum learning results (Section 5) provide indirect evidence that training distribution matters considerably: the non-curriculum Count Primes model actually underperforms the baseline (4.00% vs. 12.00%, Table 5), suggesting that poor training data composition can actively harm performance. This raises the possibility that RLVR on RG data, while improving target reasoning capabilities, might inadvertently degrade other capabilities not measured in the paper's evaluation suite.

Evidence in the paper. The paper does not evaluate forgetting. It does not report any metrics for general capability retention. The Section 7 discussion mentions catastrophic forgetting only in passing, in the context of future work:

"Future work should examine continual learning settings where data arrives in non-stationary streams and investigate how regularization, model merging, and replay buffers affect the model performance under catastrophic forgetting." (Section 7)

This indicates awareness of the issue but no empirical investigation.

Mitigation status. Not addressed empirically; deferred to future work. The paper notes the relevance of regularization (Kirkpatrick et al., 2017; Schwarz et al., 2018), model merging (Wortsman et al., 2022; Dziadzio et al., 2024), and replay buffers (Lopez-Paz and Ranzato, 2017; Buzzega et al., 2020) as mitigation strategies, but none are tested. For practitioners considering RG-based RLVR in production settings, the absence of forgetting analysis means they cannot assess the risk that improving reasoning on RG tasks will degrade other model capabilities that their applications depend on. Running a standard LLM evaluation suite (MMLU, HellaSwag, ARC, etc.) before and after RLVR training would be a straightforward and informative addition.

7. Implications and Future Directions

How This Work Changes the Landscape

Reasoning Gym does not introduce a new algorithm, a new architecture, or a new training objective. What it introduces is a new category of research infrastructure—a general-purpose, procedurally generated, algorithmically verifiable reasoning platform that transforms the economics and epistemology of RLVR research. The magnitude of this shift depends on adoption: if the field converges on RG as a standard training and evaluation substrate (as it has converged on benchmarks like GSM8K and MATH for evaluation), the effects compound over time. If RG remains one library among many, its impact is more modest. The paper makes a credible case for the former by providing 100+ tasks spanning 11 reasoning categories, a working training pipeline (verl + GRPO + curriculum scheduling), and initial experimental results that demonstrate the kind of science the platform enables.

The conceptual shift is this: reasoning data transitions from a scarce, curated resource to an abundant, generated one, and this changes what questions are askable. With fixed benchmarks, a researcher can ask "How well does model X perform on GSM8K?" but cannot ask "How does model X's performance degrade as problem complexity scales from 2-digit to 5-digit arithmetic?" because the benchmark does not parameterize difficulty. With RG, difficulty becomes a continuous experimental variable rather than a post-hoc stratification of a static corpus. This is the paper's most consequential contribution—not any single empirical finding, but the methodological capability to systematically study how reasoning capabilities scale with problem complexity under controlled conditions.

The paper's reconciliation of prior work is implicit rather than explicit, but it provides the infrastructure to resolve several tensions in the literature. The debate about whether RLVR produces genuine reasoning or template matching (Yue et al., 2025; Zhao et al., 2025; Shao et al., 2025) has been difficult to adjudicate because fixed benchmarks cannot separate instance memorization from strategy learning. RG's procedural generation eliminates the memorization confound by construction: no two training instances are identical, so any improvement must reflect strategy acquisition rather than instance recall. The paper's transfer experiments (Tables 1–2) provide initial evidence that what RLVR learns does transfer, but the deeper contribution is the platform that makes this question experimentally tractable.

The paper also provides an empirical challenge to the narrative that scaling model size or reasoning-specific pretraining is sufficient for robust reasoning. The difficulty cliff results (Figure 3b) show that even the best frontier reasoning models—o3-mini at 63.5% average on hard tasks—exhibit catastrophic degradation when complexity increases, with drops of −71.9% on code, −33.1% on geometry, and −27.5% on cognition tasks. These are not marginal degradations; they indicate that current models possess capabilities that are brittle with respect to complexity scaling—they work well within a comfort zone and collapse outside it. This finding redirects attention from "how do we increase average benchmark scores?" to "how do we make reasoning capabilities robust to increasing problem difficulty?"—a fundamentally different research question that RG's parametric difficulty control makes measurable.

Several research directions become more attractive in light of this work:

Robustness to difficulty scaling as a primary evaluation criterion, not an afterthought. Papers proposing new reasoning methods should report performance curves across difficulty levels, not just aggregate scores on fixed benchmarks, and RG provides the tooling to do this.
Curriculum learning for RLVR moves from a niche interest to a central design choice. The paper's curriculum results (Table 5) demonstrate consistent and sometimes dramatic improvements (+40.67% on Spell Backwards at word length 4) from a simple threshold-based curriculum, suggesting that the order of training examples matters enormously—a finding that is underexplored in the RLVR literature, which typically samples uniformly from fixed datasets.
Cross-domain transfer becomes a testable hypothesis rather than a speculative claim. The paper's transfer matrix (Table 2) shows that algorithmic training transfers to algebra (+29.1%) and geometry (+22.3%), but the mechanism of transfer is unknown. This opens a research program into the structure of reasoning primitives that existing benchmarks cannot support.

Research directions that become less attractive include:

Building ever-larger fixed reasoning datasets. The paper demonstrates that procedurally generated data, despite lacking human craft, produces meaningful reasoning improvements that transfer to external benchmarks (MATH +9.7%, Big-Bench Hard +7.7%, MMLU-Pro across 8 subtasks). If procedural generation can match or approach the effectiveness of human-curated data for RLVR, the economic and sustainability arguments heavily favor generated data over curation. This does not mean fixed datasets become obsolete—they remain essential for standardized evaluation—but they become less compelling as training resources when unlimited generated alternatives exist.
Developing ever-more-complex search or revision mechanisms without first improving base reasoning robustness. The difficulty cliff results suggest that current models fail catastrophically when complexity increases beyond a narrow band, regardless of the inference-time strategy used. Improving robustness to complexity scaling—perhaps through better curricula, architectural changes, or training objectives that penalize brittle strategies—may yield larger gains than optimizing the search procedure applied to a fundamentally fragile reasoner.

Follow-Up Research This Work Enables

Head-to-head comparison of procedurally generated vs. human-curated RLVR data. The paper's central claim—that procedurally generated data addresses the "fundamental scalability bottleneck" of fixed datasets—is never tested against the alternative. The critical experiment is: train Qwen2.5-3B-Instruct (or an equivalent model) using GRPO on (a) RG's algebra composite, (b) the MATH training set's algebra subset, and (c) GSM8K's training set, holding all hyperparameters constant, then evaluate all three models on held-out MATH and GSM8K test splits. This would quantify the "procedural discount"—how much less effective generated data is per training instance compared to human-crafted data—and whether the unlimited supply of generated data compensates for any per-instance quality gap. The paper's external benchmark results (+9.7% on MATH) are promising but measured against an untuned baseline, not against a benchmark-trained competitor. If the procedural discount is small (say, 10–20% less effective per instance), the scalability argument is validated. If it is large (say, 50%+), the field still needs human curation for efficient training, and RG's role is primarily evaluation and diagnostic.

Multi-dimensional difficulty scaling and the structure of the difficulty cliff. The paper's difficulty cliff analysis (Section 3.2) varies one or two parameters per task (grid size, word length, polynomial degree), but real reasoning problems have multi-dimensional difficulty landscapes. A follow-up study should systematically vary multiple parameters simultaneously—for example, in game_of_life, independently varying grid size, simulation steps, and filled-cell density—and map performance across the full parameter space. The question is whether the difficulty cliff is primarily driven by a single "bottleneck" parameter (e.g., when grid size exceeds some threshold, performance collapses regardless of other parameters) or whether difficulty dimensions interact (large grid and many simulation steps are multiplicatively harder). Understanding the structure of the cliff would inform both curriculum design (which parameters to increase first) and model architecture (what computational bottlenecks cause the collapse). The paper's curriculum experiments use one-dimensional progressions; a multi-dimensional curriculum that increases grid size only after simulation depth is mastered, for instance, might produce different transfer patterns than increasing both simultaneously.

Replication of the cross-domain transfer matrix across model families and scales. The paper's most striking finding—+29.1% algorithmic-to-algebra transfer (Table 2)—is demonstrated on a single model (Qwen2.5-3B-Instruct) at a single scale (3B parameters). A replication study should reproduce the transfer matrix with at least: (a) Llama-3.2-3B-Instruct (to test architecture dependence), (b) Qwen2.5-7B-Instruct (to test scale dependence within the same family), and (c) Qwen2.5-3B (base model without instruction tuning, to test whether instruction tuning is a prerequisite for transfer or whether RLVR alone suffices). The key measurements are whether the pattern of transfer (algorithmic → algebra, logic → cognition, games → algebra) replicates, and whether transfer magnitudes scale with model size (do larger models benefit more, less, or equally from cross-domain RLVR?). A null result—transfer patterns are specific to Qwen2.5-3B-Instruct—would significantly limit the generality of the paper's claims about reasoning primitives. A positive result—the transfer pattern replicates but magnitudes vary—would establish that cross-domain transfer is a robust phenomenon while motivating investigation into what drives the variation.

Mechanistic analysis of what RLVR on procedurally generated tasks actually learns. The paper demonstrates that RLVR improves performance, but not why or how. A mechanistic follow-up should probe what changes in the model's internal representations and generation behavior after RLVR training. Specific approaches: (a) Intervention studies—train probes on the model's hidden states to predict whether the model is on a correct solution path at each generation step, and compare probe accuracy before and after RLVR training. If RLVR improves the model's internal "correctness tracking," probes should become more accurate. (b) Behavioral analysis of error patterns—categorize errors on evaluation tasks (format errors, arithmetic errors, logical errors, premature termination, infinite loops) and compare error distributions between baseline and RLVR-trained models. The paper notes that RLVR improves performance but does not show whether it reduces specific error types or improves specific reasoning subskills. (c) Chain-of-thought quality analysis—if the model generates chain-of-thought reasoning, compare the coherence, correctness, and strategy diversity of CoT traces before and after RLVR, using automated metrics (step-by-step verification against the ground-truth solution) or human evaluation. The paper's format reward incentivizes structured output, but we don't know whether the content of reasoning chains improves or merely their surface formatting. This analysis would directly address the "template matching vs. genuine reasoning" debate that the paper's infrastructure is designed to resolve.

Negative transfer and capability forgetting as a function of RLVR training diversity. The paper reports several instances of negative or near-zero transfer: RG-Algebra training slightly decreases ARC performance (−2.3%, Table 2); RG-Games training decreases ARC (−2.2%); and the non-curriculum Count Primes model underperforms baseline (4.00% vs. 12.00%, Table 5). A systematic study should train models on varying numbers of RG generators (1, 3, 10, 30) and measure both: (a) positive transfer to held-out reasoning tasks, and (b) negative transfer or forgetting on (i) other RG categories, (ii) standard LLM benchmarks (MMLU, HellaSwag, ARC-Challenge), and (iii) instruction-following evaluations (AlpacaEval, MT-Bench). The hypothesis is that training on a narrow set of generators produces sharper improvements on related tasks but greater forgetting of unrelated capabilities, while training on a diverse mixture preserves general capabilities at the cost of smaller per-domain gains. This would establish a transfer-forgetting Pareto frontier that practitioners could use to select training data composition based on their deployment requirements. The paper's Section 7 mentions catastrophic forgetting as future work but provides no empirical grounding; this experiment would fill that gap.

Curriculum learning with multi-parameter, non-monotonic difficulty progressions. The paper's curriculum experiments (Section 5) use simple one-dimensional difficulty parameters (word length, empty cells, number range) with monotonic increases. Real-world reasoning curricula might need to vary multiple parameters in coordinated ways—for instance, in game_of_life, increasing grid size and simulation steps simultaneously might be necessary to maintain an appropriate challenge level, while increasing only one dimension might produce problems that are trivially easy or impossibly hard. More interestingly, non-monotonic curricula—where difficulty temporarily decreases to reinforce foundational skills before advancing—might produce better final performance than monotonic progression, especially for tasks where later skills depend on mastery of earlier ones. A study should implement multi-parameter curriculum schedules (e.g., increase grid size while decreasing simulation steps, then increase simulation steps while holding grid size constant) and compare to the paper's single-parameter monotonic baseline across a broader set of environments (at least 10, spanning multiple categories). The metric is final performance on maximum-difficulty instances, which tests whether structured progression produces more robust capability than simple difficulty scaling.

Practical Applications and Downstream Use Cases

Cost-efficient RLVR training pipelines for open-weight model developers. The paper's core practical contribution is a drop-in data engine for RLVR that eliminates the per-training-run cost of data curation. For teams training reasoning models on academic or startup budgets (the ~1500 A6000-hour regime described in the paper), RG provides a plug-and-play alternative to: (a) licensing or scraping benchmark datasets that may have unclear provenance and contamination risks, (b) commissioning expert annotation for domain-specific reasoning problems, or (c) implementing and debugging their own procedural generators. The training configuration in Appendix A.6—a single YAML file specifying generators, weights, and curriculum parameters—reduces the engineering effort to set up an RLVR training run from days or weeks (curating data, writing verifiers, testing coverage) to hours (configuring existing generators). The external benchmark transfer results (MATH +9.7%, Big-Bench Hard +7.7%) provide evidence that models trained this way improve on downstream reasoning tasks, making the pipeline relevant beyond academic experimentation. The specific workflow: take your base instruction-tuned model → select RG generators matching your target reasoning domains → configure difficulty parameters and curriculum schedule → run GRPO training via the provided verl config → evaluate on your target benchmarks. The paper's reported compute cost (~1500 A6000-hours for all experiments, Section 4) establishes that this is feasible on a cloud budget of a few thousand dollars.

Difficulty-stratified evaluation as a complement to aggregate benchmark scores. The paper's difficulty cliff analysis (Figure 3b) demonstrates that aggregate benchmark scores conceal critical information about capability robustness. A model might achieve 90% on MATH while collapsing to 20% on the hardest MATH problems, or vice versa—aggregate scores cannot distinguish these cases, but they have very different deployment implications. Organizations evaluating models for production use (where problem difficulty distributions may be unknown or skewed) can use RG to generate difficulty-stratified capability profiles: for each relevant reasoning category, generate evaluation sets at 3–5 systematically varied difficulty levels (using the easy/hard parameter configurations in Appendix A.3 as starting points), evaluate candidate models, and report performance curves rather than single numbers. The paper's per-task heatmaps (Figures 7–8) demonstrate the format: rows are tasks, columns are models, cells are accuracy scores—extending this with additional intermediate difficulty levels would produce a richer picture of where each model's capabilities break down. The specific benefit over fixed-benchmark evaluation: a model that scores well on GSM8K (grade-school math) might fail catastrophically on complex arithmetic with larger operand ranges, and RG reveals this before deployment, whereas GSM8K alone would not. This is particularly relevant for applications where problem difficulty is adversarial (e.g., automated theorem proving, competitive programming) or unbounded (user-submitted queries in production chatbots).

Targeted curriculum design for domain-specific reasoning fine-tuning. For practitioners fine-tuning models for specific reasoning-heavy applications (mathematical tutoring systems, code verification tools, puzzle-solving assistants, logic-based QA systems), RG's parametric difficulty control enables curriculum design without manual dataset creation. The workflow: identify which RG categories map to the target application (e.g., algebra + arithmetic + logic for a math tutoring system; algorithmic + code + graphs for a code reasoning assistant) → configure progressive difficulty parameters (starting from levels the base model handles with >50% accuracy, advancing when accuracy exceeds 70% as in Section 5) → train with GRPO using the curriculum scheduler → evaluate on both RG-held-out tasks and application-specific held-out problems. The paper's curriculum results (Table 5) provide evidence that this approach yields better final performance than uniform sampling, with gains of +40.67% (Spell Backwards), +26.67% (Count Primes), and +13.33% (Mini Sudoku) on specific difficulty levels. The key practical advantage over manual curriculum design: the curriculum is specified declaratively in a YAML configuration (parameter ranges per level, success thresholds, evaluation windows), not imperatively through manual dataset partitioning, making it reusable across models and applications with minimal modification.

Rapid prototyping and stress-testing of new RLVR algorithms and reward designs. RLVR research (new algorithms, reward shaping strategies, exploration bonuses, KL penalty schedules) is bottlenecked by the need for diverse, controllable training environments. Fixed benchmarks provide at most a few distinct training distributions; RL researchers typically resort to toy environments (grid worlds, simple games) that don't exercise the reasoning capabilities of language models. RG provides a middle ground: 100+ reasoning environments that are complex enough to stress LLM reasoning but simple enough to enable rapid algorithmic experimentation—a training run on a single environment takes minutes to hours rather than days. A researcher developing a new advantage estimation method for GRPO could test it across 20 RG environments spanning multiple categories in a single GPU-day, measuring not just aggregate performance but difficulty-stratified transfer patterns. The paper's open-source release includes the full training infrastructure (verl configs, generator APIs, verifier implementations), making this "plug in your algorithm, sweep environments" workflow immediately actionable. The specific benefit: algorithmic innovations that show promise on fixed benchmarks but fail to generalize can be identified early (does your new method improve performance on algebra tasks but degrade it on logic tasks? RG's multi-category coverage reveals this), and robust methods that work across diverse reasoning types can be identified and prioritized for expensive large-scale evaluation.

Dependency tree

Papers this one builds on, walking backwards through the same domain and shared topics.

Coffee-Gym: An Environment for Evaluating and Improving Natural Language Feedback on Erroneous Code2024· rl-training← LLM-judged similar
- WPO: Enhancing RLHF with Weighted Preference Optimization2024· rl-training← in-domain predecessor
Llama-Nemotron: Efficient Reasoning Models2025· reasoning← in-domain predecessor
- 100 Days After DeepSeek-R1: A Survey on Replication Studies and More Directions for Reasoning Language Models2025· reasoning← in-domain predecessor
- Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architectures2025· moe← topic thread
Lost in the Noise: How Reasoning Models Fail with Contextual Distractors2026· reasoning← similar topic
- The Lessons of Developing Process Reward Models in Mathematical Reasoning2025· training-methods← LLM-judged similar
- The Molecular Structure of Thought: Mapping the Topology of Long Chain-of-Thought Reasoning2026· reasoning← in-domain predecessor
Multiplex Thinking: Reasoning via Token-wise Branch-and-Merge2026· reasoning← similar topic
Distribution-Aligned Sequence Distillation for Superior Long-CoT Reasoning2026· reasoning← similar topic
- Black-Box On-Policy Distillation of Large Language Models2025· training-methods← LLM-judged similar
- Transition Matching Distillation for Fast Video Generation2026· inference-optimization← topic thread
Visual Generation Unlocks Human-Like Reasoning through Multimodal World Models2026· reasoning← similar topic
- Video models are zero-shot learners and reasoners2025· multimodal← LLM-judged similar
- Cosmos-H-Surgical: Learning Surgical Robot Policies from Videos via World Modeling2025· uncategorized← topic thread
On the Optimal Reasoning Length for RL-Trained Language Models2026· reasoning← similar topic
- DLER: Doing Length pEnalty Right - Incentivizing More Intelligence per Token via Reinforcement Learning2025· rl-training← LLM-judged similar
- Accelerating Scientific Research with Gemini: Case Studies and Common Techniques2026· reasoning← in-domain predecessor
Understanding Reasoning in LLMs through Strategic Information Allocation under Uncertainty2026· reasoning← similar topic
- Understanding Chain-of-Thought in LLMs through Information Theory2024· reasoning← LLM-judged similar
- EndoCoT: Scaling Endogenous Chain-of-Thought Reasoning in Diffusion Models2026· reasoning← in-domain predecessor
HopChain: Multi-Hop Data Synthesis for Generalizable Vision-Language Reasoning2026· reasoning← similar topic
LongCat-Flash-Prover: Advancing Native Formal Reasoning via Agentic Tool-Integrated Reinforcement Learning2026· reasoning← similar topic
Rethinking Generalization in Reasoning SFT: A Conditional Analysis on Optimization, Data, and Model Capability2026· reasoning← similar topic
- SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training2025· alignment← LLM-judged similar
- Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?2026· reasoning← in-domain predecessor
Towards Autonomous Mechanistic Reasoning in Virtual Cells2026· reasoning← similar topic
- Process Reward Agents for Steering Knowledge-Intensive Reasoning2026· reasoning← in-domain predecessor
General365: Benchmarking General Reasoning in Large Language Models Across Diverse and Challenging Tasks2026· reasoning← similar topic
- A Survey on Evaluation of Large Language Models2023· evaluation← LLM-judged similar
Learning from Language Feedback via Variational Policy Distillation2026· reasoning← similar topic
- LLMs Improving LLMs: Agentic Discovery for Test-Time Scaling2026· reasoning← in-domain predecessor
- SlimQwen: Exploring the Pruning and Distillation in Large MoE Model Pre-training2026· architecture← topic thread
ETCHR: Editing To Clarify and Harness Reasoning2026· reasoning← similar topic
rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking2025· reasoning← similar topic
- HunyuanProver: A Scalable Data Synthesis Framework and Guided Tree Search for Automated Theorem Proving2024· reasoning← in-domain predecessor
- OpenAI o1 System Card2024· safety← topic thread
- R-Horizon: How Far Can Your Large Reasoning Model Really Go in Breadth and Depth?2025· reasoning← similar topic
- MUTUAL REASONING MAKES SMALLER LLMS STRONGER PROBLEM-SOLVERS2024· reasoning← similar topic
Learning Adaptive Parallel Reasoning with Language Models2025· reasoning← similar topic
- DeepSeek-R1 Thoughtology: Let's think about LLM Reasoning2025· reasoning← in-domain predecessor
DianJin-R1: Evaluating and Enhancing Financial Reasoning in Large Language Models2025· reasoning← similar topic
Magistral2025· reasoning← similar topic
- Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models2025· pretraining← topic thread
- OpenThoughts: Data Recipes for Reasoning Models2025· data← topic thread
EXAONE 4.0: Unified Large Language Models Integrating Non-reasoning and Reasoning Modes2025· reasoning← similar topic
- MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention2025· reasoning← in-domain predecessor
NVIDIA Nemotron Nano 2: An Accurate and Efficient Hybrid Mamba-Transformer Reasoning Model2025· reasoning← similar topic
- Decoder-Hybrid-Decoder Architecture for Efficient Reasoning with Long Generation2025· architecture← LLM-judged similar
- Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens2025· reasoning← in-domain predecessor
- ZeCO: Zero Communication Overhead Sequence Parallelism for Linear Attention2025· distributed-training← topic thread
A Survey of Reinforcement Learning for Large Reasoning Models2025· reasoning← similar topic
- MolmoAct: Action Reasoning Models that can Reason in Space2025· uncategorized← topic thread
- LoongRL: Reinforcement Learning for Advanced Reasoning over Long Contexts2025· reasoning← similar topic
- Nemotron-Cascade: Scaling Cascaded Reinforcement Learning for General-Purpose Reasoning Models2025· reasoning← similar topic
Tracing the Traces: Latent Temporal Signals for Efficient and Accurate Reasoning2025· reasoning← similar topic
Open-o3 Video: Grounded Video Reasoning with Explicit Spatio-Temporal Evidence2025· reasoning← similar topic
- UI-Ins: Enhancing GUI Grounding with Multi-Perspective Instruction-as-Reasoning2025· reasoning← in-domain predecessor
Premise Order Matters in Reasoning with Large Language Models2024· reasoning← similar topic
Chain-of-Thought Reasoning without Prompting2024· reasoning← similar topic
- Monte Carlo, Puppetry and Laughter: The Unexpected Joys of Prompt Engineering2022· reasoning← similar topic
Iterative Reasoning Preference Optimization2024· reasoning← similar topic
- Advancing LLM Reasoning Generalists with Preference Trees2024· reasoning← in-domain predecessor
RATIONALYST: Mining Implicit Rationales for Process Supervision of Reasoning2024· reasoning← similar topic
- Attention Heads of Large Language Models: A Survey2024· reasoning← in-domain predecessor
Large Language Models Cannot Self-Correct Reasoning Yet2023· reasoning← similar topic
- Self-consistency for open-ended generations2023· reasoning← in-domain predecessor
Democratizing Reasoning Ability: Tailored Learning from Large Language Model2023· reasoning← similar topic
- Sum of the GL(3) Fourier Coefficients over Quadratics and Mixed Powers2023· reasoning← in-domain predecessor
Self-Consistency Improves Chain of Thought Reasoning in Language Models2022· reasoning← similar topic
- Training Large Language Models to Reason in a Continuous Latent Space2024· reasoning← similar topic
STaR: Bootstrapping Reasoning With Reasoning2022· reasoning← similar topic

What it influenced

Later papers in the same chain. Direction reversed.

How2Everything: Mining the Web for How-To Procedures to Evaluate and Improve LLMs2026· evaluation→ LLM-judged similar
- RubricBench: Aligning Model-Generated Rubrics with Human Standards2026· evaluation→ in-domain predecessor
- DeepSeek-R1 vs. o3-mini: How Well can Reasoning LLMs Evaluate MT and Summarization?2025· evaluation→ similar topic
HopChain: Multi-Hop Data Synthesis for Generalizable Vision-Language Reasoning2026· reasoning→ LLM-judged similar
- LongCat-Flash-Prover: Advancing Native Formal Reasoning via Agentic Tool-Integrated Reinforcement Learning2026· reasoning→ in-domain predecessor
- STaR: Bootstrapping Reasoning With Reasoning2022· reasoning→ similar topic
- Fantastic Reasoning Behaviors and Where to Find Them: Unsupervised Discovery of the Reasoning Process2025· reasoning→ similar topic
Magistral2025· reasoning→ in-domain predecessor
- MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention2025· reasoning→ in-domain predecessor
- FlexOlmo: Open Language Models for Flexible Data Use2025· moe→ topic thread
RLPR: Extrapolating RLVR to General Domains without Verifiers2025· rl-training→ topic thread
- Scaling RL to Long Videos2025· rl-training→ in-domain predecessor
- The Invisible Leash: Why RLVR May or May Not Escape Its Origin2025· evaluation→ topic thread
Offline Reinforcement Learning for LLM Multi-Step Reasoning2024· reasoning→ similar topic
- HunyuanProver: A Scalable Data Synthesis Framework and Guided Tree Search for Automated Theorem Proving2024· reasoning→ in-domain predecessor
- A Survey of Reinforcement Learning for Large Reasoning Models2025· reasoning→ similar topic
- No Prompt Left Behind: Exploiting Zero-Variance Prompts in LLM Reinforcement Learning via Entropy-Guided Advantage Shaping2025· reasoning→ similar topic
Meta-Awareness Enhances Reasoning Models: Self-Alignment Reinforcement Learning2025· reasoning→ similar topic
- R-Horizon: How Far Can Your Large Reasoning Model Really Go in Breadth and Depth?2025· reasoning→ in-domain predecessor
- Beyond Reasoning Gains: Mitigating General Capabilities Forgetting in Large Reasoning Models2025· alignment→ topic thread