Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model

ArXiv: 2503.24290

🎯 Pitch

A minimalist RL recipe—vanilla PPO with no KL regularization and a learned critic—achieves superior reasoning benchmarks using only 10% of the training steps of DeepSeek-R1-Zero, debunking the need for complex reward designs. Crucially, the critic automatically suppresses repetitive failure modes that cause training collapse in GRPO-based approaches.

1. Executive Summary

This paper introduces Open-Reasoner-Zero (ORZ), the first fully open-source implementation of large-scale reasoning-oriented reinforcement learning applied directly to a base language model—a paradigm the authors term Reasoner-Zero training—and demonstrates that a minimalist recipe of vanilla PPO with GAE (λ=1, γ=1), simple rule-based binary rewards for answer correctness, and no KL regularization is sufficient to achieve steady scaling of both benchmark accuracy and response length across the Qwen2.5 model family (0.5B to 32B). Using the same Qwen2.5-32B base model as DeepSeek-R1-Zero-Qwen-32B, ORZ attains superior performance on AIME2024 (48.1%), MATH500 (92.2%), and GPQA Diamond (55.5%) while requiring only 1/10 of the training steps, establishing that a learned critic—which the authors show quantitatively assigns more negative advantages to repetitive token patterns than GRPO's group-based estimator—provides the robust credit assignment necessary for stable training at scale, even as response length exhibits large fluctuations during optimization.

2. Context and Motivation

The Core Problem: Scaling Reasoning-Oriented RL on Base Models Without Training Instability

The fundamental challenge this paper addresses is how to apply large-scale reinforcement learning directly to base language models for reasoning tasks without encountering training instability, while keeping the recipe simple enough for broad adoption. Recent breakthroughs—most notably OpenAI's o1 and DeepSeek's R1-Zero—demonstrated a remarkable phenomenon: as you scale up RL training on reasoning tasks, both benchmark performance and response length increase steadily without saturation. This "training-time scaling" suggests a new axis for improving LLM capabilities beyond pretraining scaling laws. However, the practical know-how for reproducing these results remained locked behind proprietary implementations (OpenAI) or sparse technical descriptions (DeepSeek's R1-Zero paper, which the authors note "outlined their training pipeline briefly").

The gap is not merely about missing code. It is about missing understanding. Several concurrent open-source efforts (SimpleRL-Zoo, DAPO, VAPO, Logic-RL) attempted to replicate Reasoner-Zero-style training, but they either underperformed, required substantially more compute, or encountered training instability—particularly the phenomenon where model outputs degenerate into repetitive or incoherent text midway through training. The paper is motivated by a specific failure mode observed in community implementations of GRPO (the algorithm used in DeepSeek-R1-Zero): sudden training collapse where the model starts generating repetitive loops and reward destabilizes. This is documented explicitly in Section 3.3 (Figure 7) and attributed to GRPO's lack of a learned value function for credit assignment.

The problem matters for several reasons the paper raises explicitly or implies:

• Democratization of reasoning research. If Reasoner-Zero training requires proprietary infrastructure or undisclosed tricks, the broader research community cannot systematically study the scaling properties of reasoning-oriented RL. The paper's title and abstract emphasize "open source," "accessibility," and "encouraging further exploration"—this is not just a technical contribution but a deliberate effort to lower the barrier to entry.

• Efficiency in training compute. Even when Reasoner-Zero training works, prior approaches (DeepSeek-R1-Zero, DAPO) required large numbers of training steps. ORZ achieves competitive or superior results with 1/10 the steps, which translates directly to lower compute cost and faster research iteration.

• Understanding why training succeeds or fails. The paper positions itself as going beyond simply releasing a working implementation. Sections 3.3 and Appendix C provide quantitative analysis of how the learned critic assigns negative advantages to repetitive patterns—a mechanistic explanation for why PPO stabilizes training while GRPO does not. This understanding is valuable for diagnosing failures and designing future algorithms.

Prior Approaches and Their Shortcomings

The paper identifies several strands of prior work, each with limitations that motivate ORZ:

DeepSeek-R1-Zero [2] defined the paradigm but left gaps in reproduction. DeepSeek's paper demonstrated Reasoner-Zero training at scale using GRPO (Group Relative Policy Optimization), achieving strong results on reasoning benchmarks. However, the paper identifies three critical gaps: (1) the training recipe was described only "briefly," leaving implementers to guess at critical hyperparameters and design choices; (2) GRPO—the core algorithm—lacks a learned value function, which the paper argues (and demonstrates empirically) is a fundamental limitation for credit assignment; and (3) the implementation was not open-source, preventing direct replication, ablation, and improvement.

Concurrent open-source Reasoner-Zero efforts showed instability or inefficiency. The paper explicitly engages with DAPO [5], VAPO [18], Logic-RL [15], SimpleRL-Zoo [17], and other concurrent works. The key criticisms: DAPO "matches ORZ's AIME performance, but uses roughly fivefold more training iterations and underperforms on other benchmarks, potentially due to its data processing strategies" (Section 4). VAPO "reports stronger AIME2024 accuracy... but scales less efficiently compared to ORZ, reaching only about 60% of ORZ's score at the same iteration budget" and "faces value function learning challenges" that ORZ's simpler formulation avoids (Section 4). The paper positions these as evidence that getting the details right—algorithm choice, GAE parameters, KL handling, data curation, reward design—matters enormously for both efficiency and stability.

The RLHF community's standard toolbox may not be optimal for reasoning. The de facto standard in RLHF (InstructGPT, Ouyang et al., 2022) includes KL regularization—either as a penalty term in the reward or as a loss term—to prevent the policy from drifting too far from the base model. DeepSeek-R1-Zero and other reasoning models adopted this practice. The paper challenges this orthodoxy directly (Section 2.2, Figure 3 mid): "We achieve stable training without relying on any KL-based regularization techniques." The motivation is both practical (reducing hyperparameter tuning, lowering memory overhead from not loading a reference model) and conceptual—KL regularization potentially "limit[s] exploration during policy optimization," which may be especially harmful when training from a base model that needs to learn entirely new reasoning behaviors rather than just aligning to human preferences.

Limited understanding of the learned critic's role in reasoning RL. Prior work on GRPO for reasoning (DeepSeek-R1-Zero, concurrent replications) used group-based advantage estimation—comparing a response's reward to the average reward within its group—as a simpler alternative to learning a value function. The paper argues this is a false simplicity because it sacrifices the ability to perform token-level credit assignment. Without a critic, the algorithm cannot distinguish between "this token is part of a correct reasoning chain" and "this token appears in a response that happens to score well on average but contains degenerate patterns." The paper provides both qualitative visualizations (Figure 5, right) and quantitative evidence (Figure 5, left; Figure 7) that the critic learns to identify and penalize repetitive patterns, and that this is the mechanism behind PPO's superior stability. This is a non-obvious finding: the value of a critic is not just in reducing variance (the standard argument) but in preventing a specific failure mode (repetition collapse) that GRPO suffers from.

How This Paper Positions Itself

The paper positions itself not as proposing a fundamentally new algorithm, but as identifying and validating a minimal set of design choices that make Reasoner-Zero training work reliably at scale, and providing the first comprehensive open-source implementation. This is an empirical engineering contribution with scientific analysis: the goal is to show that vanilla PPO with specific GAE parameters (λ=1, γ=1), no KL regularization, simple binary rewards, and carefully curated diverse training data is sufficient, and to explain why it is sufficient through analysis of the critic's behavior.

The positioning is explicitly contrastive along several axes:

Against DeepSeek-R1-Zero: ORZ uses PPO instead of GRPO, removes format rewards (using only answer correctness), removes KL regularization entirely, uses a different training data mixture, and achieves better results with 1/10 the training steps. The paper does not claim to have discovered something DeepSeek didn't know—only to have developed a simpler, more efficient, and fully open recipe that produces better results.

Against the RLHF orthodoxy: ORZ challenges the assumed necessity of KL regularization and complex reward shaping. The paper argues that for reasoning tasks with verifiable binary rewards, these complications are not just unnecessary but potentially harmful, as they constrain exploration and add hyperparameter tuning burden. Figure 3 (mid) shows that adding KL loss or KL reward shaping slows down training and reduces final performance—a concrete demonstration that the standard RLHF recipe does not transfer directly to reasoner training.

Against GRPO-based approaches: The central technical argument is that a learned critic enables superior credit assignment, and that this is not a minor implementation detail but the key mechanism preventing training collapse. The paper goes beyond anecdotal reports of GRPO instability to provide quantitative evidence: Figure 5 shows that PPO's advantage estimates are consistently more negative for repetitive tokens than GRPO's would be, and Figure 7 shows GRPO's reward crashing and repetition scores spiking around training step 240 while PPO remains stable.

The "minimalist" framing is itself a contribution. By showing that a stripped-down recipe works, the paper simplifies the design space for future research. Researchers extending ORZ do not need to tune KL coefficients, design format rewards, or implement complex reward shaping—they can focus on data scaling, model scaling, and test-time compute, which the paper identifies as the key directions for future work (Section 5). This is analogous to how the Chinchilla scaling laws simplified pretraining by showing that the relationship between parameters, data, and compute follows a predictable power law—ORZ aims to provide a similarly clean baseline for reasoning-oriented RL.

The Specific Gap: Training Stability as the Primary Bottleneck

A deeper reading reveals that the paper's central preoccupation is not performance per se, but training stability at scale. The abstract highlights "training stability," the introduction promises "in-depth insights into overcoming training instability from value estimation perspectives," Section 3.3 is entirely devoted to critic analysis, and Appendix C.1 explicitly compares PPO vs. GRPO stability curves. The paper argues—implicitly—that the reason prior open-source efforts struggled to replicate DeepSeek-R1-Zero's results is not algorithmic sophistication but fragility: GRPO-based training tends to collapse, and without understanding why, practitioners cannot fix it. ORZ's contribution is identifying that the critic is the mechanism that prevents collapse, and that the specific GAE configuration (λ=1, γ=1) creates a particularly clean signal for the critic to learn from (advantage = terminal reward - value, value target = terminal reward).

This focus on stability explains several design choices that might otherwise seem arbitrary: PPO over GRPO (the critic stabilizes training), λ=1 over λ=0.95 (prevents length collapse, as shown in Figure 3 left), no KL regularization (removes a potential source of instability from improperly tuned coefficients), and careful data curation to ensure clean reward signals (excluding proof problems, balancing difficulty). The paper is not just saying "this recipe works"—it is saying "this recipe works reliably, even when response length fluctuates dramatically (Figure 2, ORZ-32B curve), and here is the mechanistic reason why."

3. Technical Approach

3.1 Reader Orientation

Open-Reasoner-Zero is a training pipeline that takes a base language model (Qwen2.5, any size from 0.5B to 32B) with no prior instruction tuning or reasoning specialization, and through reinforcement learning alone—using only binary rewards for correct final answers—teaches it to solve competition-level math problems, logical puzzles, and scientific reasoning questions by generating long, structured chain-of-thought responses that incorporate self-reflection and error correction. The system solves the problem of training instability in reasoning-oriented RL: prior approaches using GRPO (Group Relative Policy Optimization) suffered from sudden collapse where models degenerated into repetitive text loops, and the paper's core insight is that replacing GRPO's group-based advantage estimator with a learned critic network trained via vanilla PPO—configured with the specific GAE parameters λ=1, γ=1—provides token-level credit assignment that actively penalizes repetitive patterns, enabling stable scaling of both accuracy and response length across hundreds of training steps without any KL regularization, entropy bonuses, or format rewards.

3.2 Big-Picture Architecture (Diagram in Words)

The system has six major components organized into two phases (rollout and update) that repeat for hundreds of iterations:

1. Training Dataset (ORZ 57k) — A curated collection of ~57,000 reasoning questions with verified ground-truth answers, spanning math competitions (AIME through 2023, MATH, Numina-Math, Tulu3 MATH, OpenR1-Math-220k, AoPS forum) and programmatically synthesized general reasoning tasks (logical puzzles, multi-step problems, counterfactual scenarios). Each question has a reference answer that can be checked for exact match. Questions that cannot be reliably auto-graded (proofs) are excluded.

2. Policy Model (π_θ) — The base Qwen2.5 model being trained, initialized from pretrained weights with no SFT or distillation. It generates 64 complete response trajectories per prompt at temperature=1.0, top-p=1.0, following a specific prompt template that instructs it to think inside thinking... response tags and output its final answer inside <answer>...</answer> tags.

3. Rule-Based Reward Function — A deterministic binary scorer that extracts the content between <answer> and </answer> tags, compares it to the reference answer via exact string match, and returns reward R=1 if they match and R=0 otherwise. There is no format reward, no partial credit, no reward shaping. This is the sole training signal.

4. Critic Model (V_φ) — A separately initialized Qwen2.5 model of the same architecture as the policy but with a randomly initialized value head (a linear layer outputting a single scalar, initialized from U(-√5, √5) with no bias). The critic takes a state s_t (prompt + partial response up to token t) and predicts the expected future return V_φ(s_t). The policy and critic do NOT share weights. The critic is trained to minimize the squared error between its prediction and the terminal reward R (the value target when λ=1, γ=1).

5. PPO with GAE(λ=1, γ=1) Update Mechanism — The core algorithm that converts the critic's value estimates into per-token advantage estimates Â_t = R - V_φ(s_t), then updates the policy to increase the probability of tokens with positive advantage and decrease those with negative advantage, using PPO's clipped surrogate objective (ε=0.2). The critic is updated in 12 mini-batches per iteration; the policy is updated exactly once per generation (strict on-policy).

6. Annealing Stage (32B only) — A final 100-step training phase using only the hardest 13k prompts (where the model achieved fewer than 4 correct answers out of 64 attempts during the first 1100 steps). Uses a linear learning rate decay from the base rate down to 3×10⁻⁷ for the policy.

Information flows as follows: a batch of 128 prompts is sampled from the dataset → the policy generates 64 completions per prompt (8,192 trajectories total) → the rule-based reward labels each trajectory as 1 or 0 → the critic computes V_φ(s_t) for every token position in every trajectory → advantage is computed as Â_t = R - V_φ(s_t) and batch-normalized → the policy is updated once using the PPO clipped objective on all trajectories → the critic is updated 12 times on minibatches from the same trajectories → the next iteration begins with the updated policy.

3.3 Roadmap for the Deep Dive

• First, the formal RL algorithm (PPO + GAE) and why the specific parameter choice λ=1, γ=1 matters—this is the mathematical core that enables the simplified advantage formula Â_t = R - V(s_t) and value target V_target = R, which dramatically simplifies implementation and improves stability.

• Second, the five key design principles (PPO over GRPO, GAE parameters, KL regularization removal, minimal rewards, data scaling) as a coherent package—understanding why each choice was made and how they interact, since the paper's claim is that this specific combination works while alternatives fail or underperform.

• Third, the data curation pipeline, prompt template design, and reward function implementation—the engineering details that ensure the RL training sees diverse, cleanly-gradable problems and that the model learns to format its outputs correctly without explicit format supervision.

• Fourth, the training hyperparameters and compute configuration—the concrete numerical settings (learning rates, batch sizes, generation budgets, annealing schedule) that make the system reproducible and that distinguish ORZ's efficiency from prior work.

• Fifth, the critic and advantage estimation analysis—not just "what" the system does but "why" the critic is the key mechanism enabling stable training, including the quantitative comparison showing PPO assigns more negative advantages to repetitive tokens than GRPO would.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical systems paper whose core idea is that a specific configuration of vanilla PPO—with GAE parameters λ=1 and γ=1, no KL regularization, and simple binary rewards—applied to carefully curated diverse training data is sufficient for stable, efficient large-scale reasoning-oriented RL training from a base model, and that the learned critic is the mechanism that prevents the training collapse observed in GRPO-based approaches.

The RL Algorithm: PPO with GAE(λ=1, γ=1)

The paper adopts Proximal Policy Optimization (PPO; Schulman et al., 2017) as the core RL algorithm, deliberately choosing it over the GRPO (Group Relative Policy Optimization) used in DeepSeek-R1-Zero. To understand this choice, we need to understand what both algorithms do and where they differ.

The RL framing for language generation. For each input question q (the prompt), the policy model π_θ generates a group of n responses {o_1, o_2, ..., o_n}, where n=64 is the number of sampled responses per prompt (the rollout size). Each response o_i is a sequence of tokens—a trajectory τ_i = (s_0, a_0, s_1, a_1, ..., s_{T_i-1}, a_{T_i-1})—where s_t is the state at step t (the prompt concatenated with all previously generated tokens) and a_t is the token generated at step t. The reward function assigns a single terminal reward R_i ∈ {0, 1} at the end of the sequence. All intermediate rewards are zero: r_t = 0 for t < T_i-1, and r_{T_i-1} = R_i.

What advantage estimation does and why it matters. The central challenge in RL is credit assignment: given that the final answer was correct (R=1) or incorrect (R=0), how much should each individual token in the response be rewarded or penalized? The advantage Â_t at token position t answers this question. If Â_t is positive, the token was "better than expected" and the policy should increase its probability in similar contexts. If Â_t is negative, the token was "worse than expected" and the policy should decrease its probability. The key difference between PPO and GRPO is how they compute this advantage.

How GRPO computes advantage (and why it fails). GRPO computes advantage by comparing a response's reward to the average reward of all responses generated for the same prompt within the same batch. Formally, for a group of n responses to the same prompt, GRPO normalizes rewards within the group: each response's advantage is proportional to (R_i - mean(R_1, ..., R_n)) / std(R_1, ..., R_n). Crucially, this assigns the same scalar advantage to every token in a given response. If a response is correct (R=1) and the group average is 0.4, every token in that response gets the same positive advantage—even if the response contains repetitive loops at the end that happened to appear after the correct answer was already stated. This is the mechanism behind training collapse: the model gets reinforced for generating repetitive tokens because they appear in high-reward trajectories, and over time, this positive feedback loop causes the model to produce more and more repetition until the output degenerates entirely (Figure 7 shows GRPO's truncation rate and repeat score spiking to 1.0 around step 240 while reward crashes).

How PPO with a learned critic computes advantage (and why it works). PPO learns a separate value function V_φ (the critic) that predicts, for any partial state s_t, the expected future return—the reward the model expects to eventually receive if it continues generating from this point. The advantage for a token is then computed as the difference between what actually happened (the terminal reward R) and what was expected (the critic's prediction V_φ(s_t)). Because the critic is a neural network that processes the actual content of s_t, it can learn to recognize that states containing repetitive patterns are unlikely to lead to correct answers, and assign them low values. This means that when a correct response contains some repetitive tokens, those tokens get negative advantage (because V_φ(s_t) is low for those states even though the final reward is high), correctly penalizing the repetition even though the overall answer was correct. Figure 5 provides direct evidence: the average advantage assigned by PPO to tokens after the onset of repetitive patterns is consistently lower (more negative) than what GRPO would assign to the same tokens.

The GAE(λ=1, γ=1) simplification. Generalized Advantage Estimation (GAE; Schulman et al., 2015) is a method for computing advantage estimates that balances bias and variance using two parameters: γ (the discount factor) and λ (the bias-variance trade-off parameter). The general formula is:

$$\hat{A}_t^{\text{GAE}(\gamma,\lambda)} = \sum_{k=0}^{T-t-1} (\gamma\lambda)^k \delta_{t+k}$$

where $\delta_{t+k} = r_{t+k} + \gamma V_\phi(s_{t+k+1}) - V_\phi(s_{t+k})$ is the temporal difference (TD) error at step t+k, $r_{t+k}$ is the reward at that step, $V_\phi(s_{t+k})$ is the critic's value estimate for the state at that step, $\gamma$ is the discount factor controlling how much future rewards matter, and $\lambda$ controls the bias-variance trade-off (λ=0 gives low-variance high-bias TD(0) estimates; λ=1 gives high-variance low-bias Monte Carlo estimates).

What it computes: For each token position t in a trajectory of length T, GAE computes a weighted sum of future TD errors, where the weight decays by a factor of γλ per step into the future. The TD error δ at each future step measures the difference between the observed reward-plus-next-value and the current value estimate—if the critic underestimated the value at step t+k (δ is positive), this adds positive advantage to all earlier tokens, telling them "this path turned out better than you thought." The γλ decay means that errors in the near future matter more than errors in the far future, which reduces variance at the cost of some bias.

Why this form: The key insight is that the sum telescopes when γ=1 and λ=1. With these parameters:

$\hat{A}_t^{\text{GAE}(1,1)} = \sum_{k=0}^{T-t-1} \delta_{t+k} = \sum_{k=0}^{T-t-1} (r_{t+k} + V_\phi(s_{t+k+1}) - V_\phi(s_{t+k}))$

Because intermediate rewards r_t are all zero and the sum of value differences telescopes ($\sum_{k=0}^{T-t-1} (V_\phi(s_{t+k+1}) - V_\phi(s_{t+k})) = V_\phi(s_T) - V_\phi(s_t)$, where $V_\phi(s_T) = 0$ for the terminal state), this collapses to:

$\hat{A}_t^{\text{GAE}(1,1)} = R - V_\phi(s_t)$

This is an extraordinarily simple formula: the advantage of each token is simply the terminal reward minus the critic's prediction at that token position. There are no intermediate computations, no discounting, no hyperparameter interpolation—just a subtraction.

The value target simplifies correspondingly. In standard PPO, the critic is trained to predict $V_\phi(s_t)^{\text{target}} = \hat{A}_t + V_\phi(s_t)$ (the advantage plus the current value, which equals the estimated return). With GAE(1,1):

$V_\phi(s_t)^{\text{target}} = (R - V_\phi(s_t)) + V_\phi(s_t) = R$

The value target for every token in a trajectory is simply the terminal reward R. The critic's job reduces to: for every token position, predict whether this trajectory will eventually produce a correct answer. The value loss becomes:

$\mathcal{J}_{\text{value}}(\phi) = \frac{1}{2} \mathbb{E}_{\tau \sim \pi_{\theta_{\text{old}}}} \left[ \sum_{t=0}^{T-1} (V_\phi(s_t) - R)^2 \right]$

What this means operationally: For a trajectory that ends with a correct answer (R=1), every token position gets a value target of 1—the critic should learn to predict "this state leads to success." For an incorrect trajectory (R=0), every token gets a value target of 0. The critic must learn to distinguish, at any point mid-generation, whether the current partial response is on a path to success or failure. Because the critic processes the actual token content, it can learn patterns like "the state where the model starts repeating '52 × 26 / 51' over and over is unlikely to lead to a correct answer" and assign low values to such states, even if those states appear in trajectories that ultimately get R=1 (because the correct answer was stated before the repetition began).

Why γ=1 specifically matters for reasoning tasks. The discount factor γ controls the effective time horizon: a lower γ (e.g., 0.95) means that rewards in the far future are exponentially discounted, encouraging the policy to obtain rewards as quickly as possible. The paper argues (Section 2.2) that for reasoning tasks requiring long chain-of-thought, γ=1 is critical because "a lower γ assigns exponentially decreasing weights to future rewards, inducing the model to prematurely terminate generation in order to more immediately obtain rewards." With γ=1, every token in the trajectory matters equally—there is no penalty for thinking longer, which allows response length to grow naturally as the model learns to use more extensive reasoning (Figure 2 shows average response length increasing from roughly 1k to over 8k tokens during training for ORZ-32B).

Why λ=1 specifically. The GAE parameter λ controls the bias-variance trade-off. λ=1 gives Monte Carlo estimates (unbiased but high variance); λ=0 gives TD(0) estimates (biased but low variance). The paper argues (Section 2.2) that "in large-scale training scenarios, the substantial data volume naturally mitigates variance concerns, encouraging us to adopt a bias-free configuration." With 128 prompts × 64 responses = 8,192 trajectories per iteration, the law of large numbers makes variance manageable, and eliminating bias becomes the priority.

What happens with λ=0.95 (the ablation result). Figure 3 (left) shows that training with GAE λ=0.95 produces a much slower reward progression and leads to "collapsed length dynamics"—the response length curve degrades rather than growing. The bias introduced by λ<1 means that advantage estimates are systematically skewed, which appears to create a feedback loop where the model learns to produce shorter, less thoughtful responses because the truncated advantage estimates fail to properly credit early reasoning tokens for eventual success.

The PPO objective functions. With advantages computed, the policy is updated to maximize:

$\mathcal{J}_{\text{PPO}}(\theta) = \mathbb{E}_{\tau \sim \pi_{\theta_{\text{old}}}} \left[ \sum_{t=0}^{T-1} \min\left( \rho_t(\theta) \hat{A}_t, \text{clip}(\rho_t(\theta), 1-\epsilon, 1+\epsilon) \hat{A}_t \right) \right]$

where $\rho_t(\theta) = \frac{\pi_\theta(a_t|s_t)}{\pi_{\theta_{\text{old}}}(a_t|s_t)}$ is the probability ratio—how much more (or less) likely the current policy is to generate token a_t in state s_t compared to the old policy that generated the training data. The clipping parameter $\epsilon = 0.2$ limits how far the ratio can deviate from 1, preventing destructive large policy updates.

What it computes: For each token, the objective takes the minimum of two terms: the unclipped product ρ_t(θ)Â_t and the clipped version clip(ρ_t(θ), 0.8, 1.2)Â_t. When the advantage is positive (Â_t > 0), the policy wants to increase ρ_t(θ) (make the token more likely), but the clip prevents it from going above 1.2—the policy cannot increase probability by more than 20% per update. When the advantage is negative (Â_t < 0), the policy wants to decrease ρ_t(θ) (make the token less likely), but the clip prevents it from going below 0.8—the policy cannot decrease probability by more than 20% per update. The min operation ensures that the clip is only applied when it would reduce the objective (i.e., when the policy is moving too far in the advantageous direction), not when the clip would artificially inflate the objective.

Why this form: The clipping mechanism is PPO's key innovation over vanilla policy gradient. Without clipping, a single large advantage estimate could cause the policy to radically change its behavior in one update, potentially destroying capabilities learned in previous iterations. The clip creates a trust region—the policy cannot change too much in any single update, which is essential for stable training when the reward signal is sparse (only at the end of potentially very long trajectories) and the advantage estimates, while unbiased, still have variance.

Design Principle 1: PPO Over GRPO — The Learned Critic

The paper's choice of PPO over GRPO is the central architectural decision, and it is justified by a specific mechanistic argument rather than just empirical comparison. Understanding this requires understanding what GRPO does without a critic.

GRPO's advantage formula. In GRPO, for a group of n responses {o_1, ..., o_n} to the same prompt, the advantage for response i is computed as:

$\hat{A}_i^{\text{GRPO}} = \frac{R_i - \text{mean}(R_1, ..., R_n)}{\text{std}(R_1, ..., R_n)}$

This is a group-based normalization: each response is scored relative to its peers in the same batch. The normalized advantage is then assigned uniformly to every token in the response. There is no learned value function, no per-token differentiation—every token in a above-average response gets the same positive advantage, and every token in a below-average response gets the same negative advantage.

What this misses. Consider a response that reaches the correct answer after extensive reasoning, but then continues generating and falls into a repetitive loop (the exact pattern shown in Figure 5's right panel, where the model repeats "52 × 26 / 51" dozens of times). With GRPO, if this response is correct (R=1) and the group average is 0.4, every token—including the dozens of repetitive tokens—gets an advantage of (1 - 0.4) / std ≈ positive value. The model is reinforced for repeating. Over many training iterations, this positive reinforcement accumulates, the probability of repetition increases, and the model eventually collapses into producing nothing but repetitive text (Figure 7 shows this collapse for GRPO around step 240).

What the critic learns to do. The paper demonstrates (Section 3.3, Figure 5 right) that the value function V_φ(s_t) learns to assign lower values to states containing repetitive patterns. Specifically, in the visualization, coherent reasoning text receives higher V_φ values while the repetitive "52 × 26 / 51 = ..." tokens receive progressively lower values. This means that for those repetitive tokens, the advantage Â_t = R - V_φ(s_t) becomes negative even though the final reward R=1—because the critic's low value estimate outweighs the positive terminal reward. The model is penalized for repeating, even in otherwise correct trajectories.

Quantitative evidence (Figure 5 left). The paper identifies all tokens that appear after the onset of the first noteworthy repetitive pattern within a generation, and computes the average advantage assigned to these tokens by PPO vs. what GRPO would have assigned to the same tokens. The result: "advantage assignments of our PPO configuration are consistently lower (i.e., more negative) to these tokens with repetitive patterns compared to GRPO across the majority of training iterations." This is not a small difference—it is a qualitative reversal from positive (GRPO reinforces repetition) to negative (PPO penalizes repetition), and it explains why PPO maintains stable training while GRPO collapses.

Why this is credit assignment, not just variance reduction. The standard argument for learning a value function in RL is variance reduction: Monte Carlo estimates (like GRPO's group normalization) have high variance because they depend on a single random outcome, and bootstrapping from a learned value function reduces this variance. But here, the benefit is more fundamental: the learned value function provides token-level differentiation that a group-based scalar cannot. This is credit assignment—identifying which parts of a trajectory contributed to success or failure—not just variance reduction.

Practical note on weight sharing. The paper explicitly states that the policy and critic "do not share weights during training." This is a departure from many RLHF implementations where the policy and value function share a base transformer to save memory. The separation means the critic can develop representations specialized for value prediction without constraining the policy's representational capacity, and vice versa.

Design Principle 2: GAE Parameters (γ=1, λ=1) — Preventing Premature Termination and Length Collapse

The paper's choice of γ=1 and λ=1 is justified by ablation (Figure 3 left), but the reasoning behind why these parameters matter specifically for reasoning tasks deserves careful attention.

The discount factor γ and the effective planning horizon. In standard RL, γ < 1 encodes a preference for earlier rewards over later rewards, which makes mathematical sense for continuous control tasks with infinite horizons (where undiscounted returns would diverge). For language generation, where every trajectory has a finite length T, γ < 1 serves a different purpose: it encourages the model to produce correct answers in fewer tokens. With γ=0.95, a reward obtained at token position 100 contributes only 0.95^100 ≈ 0.006 times as much to the advantage as a reward obtained at token position 1—a reduction of more than 150×. The paper argues this creates a perverse incentive: "inducing the model to prematurely terminate generation in order to more immediately obtain rewards." The model learns that shorter responses get higher effective rewards per token (because the discount hasn't decayed them as much), so it truncates its reasoning.

Why reasoning tasks need γ=1. Reasoning, especially of the type that emerges during ORZ training (with self-reflection, error checking, and revision), requires the model to explore long chains of thought. The reflection patterns identified in Figure 4 (right)—"wait," "recheck," "retry," "alternatively," "however"—are inherently self-interrupting: the model realizes it may have made an error and goes back to reconsider. If γ < 1, the model is penalized for this: every token spent reconsidering pushes the eventual reward further into the discounted future, reducing its effective value. With γ=1, there is no penalty for thinking longer—all that matters is whether the final answer is correct. This allows the natural emergence of longer reasoning chains as the model discovers that more extensive thinking improves accuracy (Figure 2 shows response length growing from ~1k to ~8k tokens for ORZ-32B).

The λ parameter and the bias-variance trade-off. Setting λ=1 gives purely Monte Carlo advantage estimates: Â_t = R - V_φ(s_t) uses only the actual terminal reward and the current value estimate, with no bootstrapping from future value estimates. Setting λ < 1 (e.g., 0.95) introduces some bootstrapping, which reduces variance but introduces bias from the (imperfect) value function. The paper's ablation shows that λ=0.95 leads to "collapsed length dynamics"—the model's response length stops growing and degrades. The likely mechanism: the value function, especially early in training when it is poorly calibrated, systematically underestimates the value of early reasoning tokens (because the connection between early reasoning and eventual correctness is hard to learn). When λ < 1, this underestimation propagates into the advantage estimates, making early reasoning tokens look worse than they should, which discourages the model from producing long reasoning chains—creating a feedback loop where shorter responses get reinforced, the value function never learns to value early reasoning, and length collapses.

Why large-scale training makes λ=1 viable. The standard concern with Monte Carlo estimates (λ=1) is high variance: a single binary outcome R ∈ {0,1} provides a noisy signal for every token in a potentially very long trajectory. But ORZ processes 128 prompts × 64 responses = 8,192 trajectories per iteration, and training runs for hundreds of iterations. With tens of millions of token-level updates, the variance averages out. The paper explicitly argues: "in large-scale training scenarios, the substantial data volume naturally mitigates variance concerns, encouraging us to adopt a bias-free configuration." This is a key insight: what would be a problem in small-scale RL (high variance from Monte Carlo estimates) becomes manageable at scale, and eliminating bias becomes more important than reducing variance.

The elegant algebraic simplification. Setting γ=1 and λ=1 makes the GAE formula collapse from a weighted sum of TD errors to a simple subtraction $R - V_\phi(s_t)$. The paper provides the full derivation in Appendix D (reproduced in Algorithm 1). This simplification has practical benefits beyond the statistical ones: it eliminates the need to store or compute intermediate TD errors, reduces the GAE computation to a single vector subtraction per token, and makes the value target simply $R$—the critic's job is to predict, at every token position, whether the trajectory will eventually succeed. This is conceptually clean and easy to implement correctly (the paper provides reference pseudocode).

Design Principle 3: Removing KL Regularization — Letting the Policy Explore Freely

KL regularization—penalizing the KL divergence between the current policy and a reference policy (typically the base model)—is a standard component of RLHF pipelines (InstructGPT, Ouyang et al., 2022) and was used in DeepSeek-R1-Zero. The paper removes it entirely. The ablation (Figure 3, mid) compares three conditions: no KL regularization, KL loss (adding a KL divergence term to the objective), and KL reward shaping (subtracting a KL penalty from the reward). "W/O. KL" achieves the highest reward on the training set and the longest response length.

What KL regularization does in standard RLHF. In RLHF, the reward model is trained on human preference data and can be exploited—the policy may learn to generate text that scores highly under the reward model but is nonsensical or low-quality by human standards. KL regularization keeps the policy close to the base model (which generates fluent text), preventing this reward hacking. It is essentially a constraint: "improve the reward, but don't stray too far from what the base model would say."

Why it may be harmful for reasoning RL. The paper argues (Section 2.2): "KL regularization constrains the policy model to remain close to the original base model distribution, potentially limiting exploration during policy optimization." For Reasoner-Zero training, the base model (Qwen2.5-32B base) has no reasoning specialization—its outputs on math problems might be short, incorrect, or unstructured. The entire point of training is to move the policy far away from the base distribution—to teach it to generate long structured chain-of-thought, to self-reflect, to verify its own answers. KL regularization would penalize this movement, effectively fighting against the training objective.

Why reward hacking via the reward function isn't a concern here. The paper uses a rule-based binary reward that checks exact string match against a ground-truth answer. There is no learned reward model to exploit—the reward function cannot be "hacked" because it is not a neural network with blind spots; it is a deterministic string comparison. The only way to get R=1 is to produce the correct answer in the correct format. This eliminates the primary motivation for KL regularization in standard RLHF.

Practical benefits of removing KL. The paper highlights two concrete advantages: (1) it "obviates the need to navigate the large and challenging-to-tune design space inherent to KL regularization, greatly simplifying the training procedure"—there is no KL coefficient to tune, no schedule to design, no need to balance the KL term against the PPO objective; and (2) it "lowers computational overhead and memory usage, eliminating the need to load the weight of a separate reference model and calculate log probabilities using it"—the reference model would require a full copy of the base model weights in GPU memory, doubling the memory footprint for the policy component.

The empirical result (Figure 3, mid). Both KL Loss and KL Penalty conditions show slower reward progression and shorter response lengths compared to W/O KL. This supports the exploration-limiting hypothesis: KL regularization is actively preventing the policy from learning the behaviors (long reasoning, self-reflection) that lead to higher rewards. The paper does not explore whether a very small KL coefficient might help, but the default values tested clearly hurt.

Design Principle 4: Minimal Rule-Based Reward — No Format Rewards, No Shaping

DeepSeek-R1-Zero used a composite reward: an accuracy reward for correct answers plus a format reward for properly structuring the response with thinking... response tags. The paper argues that even the format reward is unnecessary—the base model, given the right prompt template, naturally produces well-formatted outputs, and the accuracy reward alone is sufficient to reinforce correct formatting (because incorrectly formatted answers cannot be extracted and compared to the reference, so they never get R=1).

The reward function implementation. The reward function (Section 2.3) extracts the content between <answer> and </answer> tags from the model's generated response, then compares it (via exact string match) to the reference answer. The reward is 1 for an exact match and 0 for everything else—including responses that fail to include properly formatted <answer>...</answer> tags, responses with malformed tags, and responses with incorrect answers regardless of format quality.

Evidence that format self-corrects without explicit reward. Figure 4 (left) shows the "Correct Format Ratio" during training for ORZ-7B and ORZ-32B. The base model (step 0) already has a high correct format ratio—the prompt template is effective at eliciting the desired structure even before any RL training. Within roughly 25 training steps for ORZ-7B and 50 steps for ORZ-32B, the correct format ratio converges to nearly 1.0. The mechanism: unformatted responses always get R=0 (because the answer extraction fails), so the policy learns to always produce the correct format to have any chance of receiving a positive reward. No separate format reward signal is needed—the accuracy reward alone provides sufficient incentive.

Why this matters beyond simplicity. The paper frames format rewards as a potential source of reward hacking: "minimal design leaves no room for potential reward hacking." If a format reward were included, the model could potentially learn to produce perfect formatting with no substantive reasoning—exploiting the format reward while ignoring accuracy. By making format a prerequisite for accuracy (you can't get the accuracy reward without correct format) rather than a separate reward component, the training signal is cleaner and the optimization target is unambiguous: produce correctly formatted correct answers.

Handling of ungradable problems. The paper excludes "problems that are challenging to evaluate with our rule-based reward function, such as proof-oriented problems" from the training data. This is an important data curation choice: if a problem's answer cannot be reliably auto-graded, it provides a noisy reward signal that can mislead training. By restricting to problems with clean, verifiable answers (numeric answers, multiple choice, short text), the reward signal remains binary and unambiguous for every training example.

Design Principle 5: Scaling Training Data Quantity and Diversity

The paper's data ablation (Figure 3, right) compares training on the full ORZ 57k dataset against training on MATH train 7.5k (a standard academic benchmark dataset). The results are stark: MATH 7.5k leads to performance plateauing early (both reward and response length level off), while ORZ 57k produces sustained improvement throughout training. This is presented as evidence that data scale is "pivotal" and that "increasing training data quantity can effectively improve the model's reasoning capabilities."

Dataset composition. The ORZ 57k dataset (Section 2.3) is constructed from: AIME problems up to 2023, the MATH dataset [10], the Numina-Math collection [11], Tulu3 MATH [12], OpenR1-Math-220k [13], AoPS forum problems, and "programmatically synthesized" general reasoning tasks including "logical puzzles, multi-step reasoning problems, and counterfactual scenarios that require the model to apply structured thinking across diverse domains."

Data curation process. The paper describes a multi-stage filtering pipeline: (1) collect public data from the listed sources; (2) synthesize additional reasoning tasks programmatically; (3) exclude problems that cannot be reliably auto-graded (proofs); (4) apply "LLM-based filtering to evaluate problem difficulty, removing samples with extreme pass rates to maintain a balanced dataset." This last step is important: problems that are trivially easy (the base model always gets them right) provide no learning signal (all advantages are near zero because R=1 for all samples and V_φ learns to predict 1, so Â_t ≈ 0), while problems that are impossibly hard (the base model never gets them right) also provide no signal (all R=0, V_φ learns to predict 0, Â_t ≈ 0). The learning signal comes from problems where the base model sometimes succeeds and sometimes fails, creating variation in R that the critic can learn to predict.

Why diversity matters beyond quantity. The paper emphasizes "diversity" alongside "quantity." A dataset of 57k near-identical arithmetic problems would not produce the generalization seen in Table 2 (where ORZ-32B trained only on reasoning tasks improves MMLU and MMLU_PRO over the instruction-tuned baseline). The inclusion of logical puzzles, counterfactual scenarios, and problems from diverse sources (competition math, forum posts, synthetic generation) ensures the model encounters varied reasoning patterns, preventing overfitting to a narrow problem distribution.

Why plateauing happens with small datasets. With only 7.5k problems from MATH, the model quickly exhausts the learning signal: it masters the problems it can solve, and the remaining problems are too hard (R=0 for all samples, no gradient). The critic cannot learn to distinguish promising from unpromising partial solutions on these hard problems because all trajectories end in failure—every state gets V_φ(s_t) ≈ 0, and all advantages are near zero. The policy stops improving. With 57k diverse problems, there are always problems at the right difficulty level to provide a learning signal, sustaining improvement across hundreds of training steps.

Prompt Template Design

The prompt template (Appendix Table 5) is the mechanism by which the base model is instructed to produce structured reasoning outputs. The full template:

A conversation between User and Assistant. The user asks a question, and the Assistant solves it.
The assistant first thinks about the reasoning process in the mind and then provides the user
with the answer. The reasoning process and answer are enclosed within  thinking  response and
<answer> </answer> tags, respectively, i.e.,  thinking reasoning process here  response
<answer> answer here </answer>. User: You must put your answer inside <answer> </answer> tags, i.e.,
<answer> answer here </answer>. And your final answer will be extracted automatically by the \boxed{} tag.
{{prompt}}
Assistant:  thinking

Design rationale. The template serves multiple functions: (1) It establishes a two-phase structure—thinking (inside thinking... response) followed by answering (inside <answer>...</answer>)—that encourages the model to separate reasoning from final answer, making answer extraction reliable. (2) It explicitly tells the model its answer "will be extracted automatically by the \boxed{} tag," creating pressure to produce extractable answers. (3) It ends with Assistant: thinking, which primes the model to start generating thinking tokens immediately, without a preamble. (4) The template does NOT specify what the thinking should contain—there is no instruction to "verify your answer" or "check your work." Self-reflection and verification behaviors emerge naturally through RL because they improve accuracy, not because they are prompted.

Why this matters for training stability. If the model's outputs were inconsistently formatted, the reward extraction would fail unpredictably, creating noise in the binary reward signal. The template minimizes this by making the desired format explicit and by priming the model to start with thinking. The rapid convergence of the correct format ratio (Figure 4 left) confirms that the template + accuracy reward combination is sufficient to establish consistent formatting.

Training Hyperparameters and Configuration

The paper provides detailed hyperparameters in Appendix B. Understanding these numbers is essential for assessing the paper's efficiency claims and for reproduction.

Model initialization. Both the policy network (π_θ) and critic network (V_φ) are initialized from Qwen2.5 base model weights of the appropriate size (7B or 32B for the main experiments; 0.5B and 1.5B for scaling ablations). The critic gets an additional value head—a randomly initialized linear layer that outputs a scalar—with weights drawn from U(-√5, √5) and no bias term. The policy and critic "do not share weights during training," meaning there are two full copies of the base model in GPU memory, one for generating responses (policy) and one for predicting values (critic).

Optimizer settings. Both networks use AdamW with β = [0.9, 0.95] and no weight decay. The learning rates differ between policy and critic: the policy network uses lr = 1 × 10⁻⁶, the critic network uses lr = 5 × 10⁻⁶. Both use constant learning rate schedules with linear warmup over the first 50 optimizer steps. The higher critic learning rate likely reflects that the value head is randomly initialized and needs to learn quickly from scratch, while the policy starts from a strong pretrained initialization and should change more conservatively.

Generation configuration. Each training iteration samples 128 unique prompts from the dataset. The policy generates 64 responses per prompt, for a total of 128 × 64 = 8,192 trajectories per iteration. Generation uses temperature = 1.0 and top-p = 1.0—maximum stochasticity, with no nucleus sampling truncation. This maximally stochastic generation is important for exploration: the policy needs to try diverse reasoning approaches to discover which ones work.

Update configuration. The policy network performs exactly one optimization step per iteration (strict on-policy)—the generated trajectories are used once and then discarded for the policy update. The critic network processes the same experiences in 12 mini-batches, performing 12 optimization steps per iteration. This asymmetry reflects the different sensitivity to off-policy data: the policy objective's importance sampling correction breaks down if the policy changes too much (the probability ratio ρ_t becomes inaccurate), while the value function's supervised regression target (R) does not depend on the generating policy, so multiple updates on the same data are less problematic.

Advantage normalization. The paper applies "batch level advantage normalization in the training." This means that across all 8,192 trajectories in a batch, the advantages Â_t are normalized to have mean 0 and standard deviation 1 before being used in the PPO objective. Normalization ensures that the effective learning rate is consistent across iterations even as the reward distribution changes (e.g., as the model improves and more trajectories get R=1, raw advantages shift). It also prevents any single trajectory with an unusually large positive or negative advantage from dominating the update.

What is notably absent. The paper explicitly states that training operates "stably without any KL-related regularization terms or entropy bonuses, demonstrating that vanilla PPO can achieve stable training without these commonly used stabilization techniques." There is no KL penalty, no KL loss term, and no entropy bonus added to the objective. This is a significant departure from standard RLHF practice and from DeepSeek-R1-Zero (which used KL regularization), and it is one of the paper's key claims about minimalism.

Sample packing. The paper mentions "sample packing during training" without elaboration. In the context of LLM training, this typically means concatenating multiple short trajectories into a single sequence (separated by padding or EOS tokens) to reduce wasted computation on padding tokens, which is especially valuable when response lengths vary widely—some reasoning chains may be 500 tokens, others 8,000 tokens, and packing ensures near-100% GPU utilization.

Annealing stage (32B only). For the 32B model, after the main training phase, the paper adds a 100-step annealing stage. The process is: (1) identify difficult prompts during the first 1,100 training steps—prompts where the model achieves "fewer than 4 correct answers out of a total of 64 attempts"; (2) collect 13k such prompts; (3) train for 100 additional steps on ONLY these difficult prompts, with a linear learning rate decay schedule that reduces the policy learning rate to 3 × 10⁻⁷. The annealing stage is designed to "enhance the model's capability on more complex reasoning tasks" by focusing the final training budget on the problems where the model has the most room to improve. This is analogous to annealing practices in LLM pretraining where the final phase of training uses a curated high-quality dataset with a decaying learning rate to squeeze out additional performance.

Training duration and efficiency claims. The paper's headline efficiency claim is that ORZ-32B requires "only 1/10 of the training steps compared to the DeepSeek-R1-Zero pipeline." DeepSeek-R1-Zero does not report exact step counts, but based on the learning curves in Figure 1, ORZ-32B reaches its peak performance (or near-peak) at roughly 1,000 steps, while the DeepSeek-R1-Zero-Qwen-32B curve continues climbing past 10,000 steps. At 1,000 steps, ORZ-32B achieves AIME2024 accuracy of approximately 48% vs. DeepSeek-R1-Zero-Qwen-32B's approximately 33% at the same step count (and DeepSeek eventually reaches ~47% at ~10,000 steps). The 1/10 claim refers to total training duration to reach equivalent or better performance, not to per-step compute (which differs between the two systems due to different batch sizes, model sizes, and infrastructure).

Critic and Advantage Estimation Analysis

Section 3.3 and Appendix C.1 provide the paper's mechanistic explanation for why PPO stabilizes training, going beyond empirical comparison to analyze what the critic learns and how it affects advantage estimates.

Qualitative analysis: what the value function learns. The paper observes that the value function V_φ(s_t) "effectively identifies repetitive patterns (i.e., excessive repetition), which consistently occurs when a sudden collapse of vanilla GRPO." Figure 5 (right) visualizes this on a specific example: a trajectory where the model correctly solves a probability problem but then enters a loop of repeating "52 × 26 / 51" dozens of times. The value function assigns progressively lower values to each successive repetition of the fraction—the first occurrence gets a moderate value, the second gets a lower value, and by the twentieth repetition, the value has dropped dramatically. Conversely, the coherent reasoning text before the repetition receives higher values. This is credit assignment: the critic has learned that "getting stuck in a repetitive loop" is a pattern that predicts failure (or at least, does not contribute to success), and it penalizes these tokens accordingly even though they appear in a trajectory that ultimately receives R=1 (the correct answer was stated before the loop began).

Quantitative analysis: advantage comparison on repetitive tokens. The paper performs a controlled comparison (Figure 5, left): (1) identify all tokens that appear after the onset of the first repetitive pattern within a generation; (2) compute the average advantage assigned to these tokens by PPO (with batch-level normalization); (3) compute what the average advantage would be if GRPO were used instead (group-based normalization of terminal rewards, assigned uniformly to all tokens in the response). The result: PPO's advantage estimates for repetitive tokens are "consistently lower (i.e., more negative)" than GRPO's across most training iterations. The paper uses the word "penalizing"—PPO actively discourages the policy from generating these repetitive tokens, while GRPO either fails to discourage them or actively encourages them.

Why negative advantage for repetition matters. In policy gradient methods, tokens with negative advantage have their probability decreased. If the model generates a repetitive pattern and receives negative advantage for it, the model becomes less likely to generate that pattern in the future. Over many iterations, this shapes the policy away from repetition. With GRPO, repetitive tokens in correct trajectories receive positive advantage, so the model becomes MORE likely to repeat—a positive feedback loop that eventually causes the collapse shown in Figure 7.

Figure 7: direct stability comparison. The figure compares PPO and GRPO on three metrics during ORZ-7B training: reward, truncation rate, and average repeat score. GRPO shows sudden destabilization around step 240—reward crashes, truncation rate (fraction of generations that hit the maximum token limit) surges to 1.0, and average repeat score (a metric of repetitive content) also surges to 1.0. PPO maintains stable rewards throughout training with low truncation and repeat scores. This is the empirical smoking gun: GRPO-based training collapses due to runaway repetition, and PPO prevents it.

The mechanism chain. Putting the pieces together: (1) the critic learns to predict low values for states containing repetitive patterns (Figure 5 right); (2) this produces negative advantages for repetitive tokens, even in otherwise correct trajectories (Figure 5 left); (3) negative advantages cause the policy to reduce the probability of generating repetitive tokens; (4) this prevents the positive feedback loop where repetition gets reinforced; (5) training remains stable without the collapse observed in GRPO (Figure 7). The paper thus provides a causal mechanism for PPO's superiority, not just a correlation.

Methodological note on the GRPO comparison. The paper compares PPO's actual advantage estimates against what GRPO "would have assigned" to the same tokens. This is a hypothetical comparison—they did not actually run GRPO on the same trajectories and compare (which would be impossible since GRPO uses group-based normalization that depends on the full batch). Instead, they computed what GRPO's advantage would be by taking the terminal reward of each trajectory, normalizing within the batch, and assigning that normalized scalar to all tokens. This is a fair approximation of GRPO's behavior and makes the comparison interpretable.

Appendix C.1: more detail on the stability comparison. The appendix reiterates that "vanilla GRPO often suffers from significant training instability, a phenomenon also observed in many community implementations" and that "this instability typically manifests as a deterioration in generation quality midway through training, with models tending to produce repetitive or incoherent text." The paper positions ORZ's findings as corroborating these community observations and providing the mechanistic explanation.

4. Key Insights and Innovations

Innovation 1: The Learned Critic as a Failure-Mode Preventer, Not Just a Variance Reducer

The paper's most conceptually distinctive contribution is reframing what a value function does in reasoning-oriented RL. The standard argument for learning a critic—traceable through the RL literature from Sutton through GAE to modern RLHF—is that it reduces variance in advantage estimates by bootstrapping from learned values rather than relying on noisy Monte Carlo returns. The paper shows this framing is incomplete for the reasoning domain. The critic's primary value is not statistical (reducing variance) but diagnostic: it performs token-level credit assignment that identifies and penalizes specific degenerate behaviors that would otherwise cause training collapse.

This is a shift in what we think the critic is for. Before ORZ, the field's assumption—implicit in the widespread adoption of GRPO in DeepSeek-R1-Zero and its community replications—was that a group-based advantage estimator (comparing a response's reward to batch mates) was a reasonable simplification that traded some statistical precision for implementation simplicity. GRPO's formula is elegant: normalize rewards within a group, assign the same scalar advantage to every token in each response. But this elegance masks a fatal flaw: it cannot distinguish between tokens that contributed to a correct answer and tokens that happened to appear in the same response but are actively harmful (repetitive loops, degeneracies). Every token in a correct response gets positive reinforcement, regardless of its actual contribution.

The paper demonstrates that this isn't a minor imprecision—it is a feedback loop that causes training collapse. Figure 5 (left) shows the quantitative difference: PPO's advantage estimates for tokens after the onset of repetitive patterns are consistently more negative than GRPO's would be for the same tokens. This is not a small calibration difference; it is a qualitative reversal from positive advantage (GRPO reinforces repetition) to negative advantage (PPO penalizes it). Figure 7 shows the consequence: GRPO's reward crashes and repeat scores spike to 1.0 around step 240, while PPO remains stable throughout.

The significance goes beyond the PPO-vs-GRPO comparison. The paper is arguing that credit assignment is the binding constraint for scaling reasoning-oriented RL, not exploration, not reward design, not KL control. If you get credit assignment right—by learning a critic that can recognize and devalue degenerate patterns at the token level—everything else (stability, length scaling, benchmark performance) follows. If you get it wrong, no amount of reward engineering or hyperparameter tuning will save you from eventual collapse. This is a diagnostic insight that will shape future algorithm design: any new algorithm for reasoning RL should be evaluated not just on final accuracy but on whether its advantage estimates correctly assign negative values to known degenerate patterns.

The fact that the critic learns to identify repetition without being explicitly trained to do so—it emerges purely from the value prediction objective—is evidence that repetition is a naturally occurring failure mode that the critic discovers and suppresses. This is a more fundamental finding than a simple algorithm comparison: it suggests that learned critics in language model RL serve an implicit "anomaly detection" function that simpler estimators cannot replicate.

Innovation 2: The Sufficiency of Minimalism—Challenging the RLHF Orthodoxy for Reasoning

The paper makes a deliberate and empirically substantiated argument that the standard RLHF toolbox—KL regularization, format rewards, entropy bonuses, complex reward shaping—is not just unnecessary for reasoning-oriented RL but actively harmful. This is a reframing of what constitutes a "safe" training recipe.

The dominant assumption in the RLHF community, established by InstructGPT (Ouyang et al., 2022) and carried forward into reasoning models like DeepSeek-R1-Zero and Kimi k1.5, is that KL regularization is essential to prevent the policy from drifting too far from the base model and producing degenerate outputs. The reasoning is clear for alignment tasks: a learned reward model can be exploited, so constraining the policy to stay near the base distribution prevents reward hacking. The paper argues this logic does not transfer to reasoning tasks with rule-based verifiable rewards. A rule-based reward function checking exact string match against a ground-truth answer cannot be "hacked" in the same way a learned reward model can—there are no blind spots to exploit, no spurious correlations that inflate reward without genuine correctness. The only way to get the reward is to produce the correct answer in the correct format.

The paper's ablation (Figure 3, mid) shows that adding KL regularization—whether as a loss term or a reward penalty—slows down training and reduces final performance. This is a negative result with positive implications: it means the standard RLHF recipe is over-engineered for the reasoning domain, and the added complexity (tuning KL coefficients, loading a reference model, computing additional log probabilities) actively works against the training objective. The reference model is the base model—the very distribution the policy is trying to escape to learn structured reasoning, self-reflection, and error correction. KL regularization constrains escape.

Similarly, the paper shows that format rewards—a separate component in DeepSeek-R1-Zero's reward function for properly structuring responses with thinking tags—are unnecessary because the base model, given a well-designed prompt template, already produces correctly formatted outputs with high probability (Figure 4, left), and the accuracy reward alone is sufficient to drive format adherence to near 100% within the first ~50 training steps. Responses with incorrect formatting never receive R=1 (the answer extraction fails), so the policy rapidly learns to produce correct formatting as a prerequisite for any reward. No separate signal is needed.

The conceptual move here is from "safety through constraints" to "safety through simplicity." The paper is arguing that the safest training recipe is the one with the fewest moving parts—fewer hyperparameters to tune, fewer components that can interact in unexpected ways, fewer opportunities for reward hacking. This is a philosophical stance about how to design reliable ML systems, and it contrasts with the prevailing trend toward increasingly complex reward engineering in reasoning models (DeepSeek-R1's format rewards and language consistency rewards, Kimi k1.5's multi-component reward design).

The practical significance is substantial: researchers building on ORZ do not need to navigate the "large and challenging-to-tune design space inherent to KL regularization" or design format rewards. They can focus on scaling data, models, and test-time compute—which the paper identifies as the key directions for future work—without being distracted by stabilization techniques that the paper shows are counterproductive. This is analogous to how the Adam optimizer simplified deep learning training by eliminating the need for careful learning rate scheduling: ORZ simplifies reasoning RL by eliminating the need for KL tuning.

Innovation 3: GAE(γ=1, λ=1) as a Deliberately Bias-Free Advantage Configuration

The paper's specific choice of GAE parameters—γ=1 and λ=1—might appear as a minor implementation detail, but it represents a conceptual innovation in how to formulate the RL objective for reasoning tasks. The standard intuition in RL, carried over from continuous control and game-playing domains, is that γ < 1 is necessary for mathematical tractability (preventing infinite returns) and that λ < 1 is beneficial for reducing variance. The paper argues both intuitions are wrong for reasoning-oriented language model training.

The argument for γ=1 is that reasoning tasks have a natural finite horizon (the model eventually stops generating or hits a token limit), so the mathematical necessity of discounting for convergence does not apply. More importantly, γ < 1 creates a perverse incentive: by exponentially discounting future rewards, it encourages the model to produce shorter responses so that the reward arrives before the discount factor has attenuated it too severely. For a task where the key behaviors needed—self-reflection, error checking, multi-step verification—are inherently time-consuming, discounting future rewards is actively fighting the training objective. The ablation (Figure 3, left) confirms this: γ=0.95 leads to collapsed response length dynamics compared to the steady growth observed with γ=1.

The argument for λ=1 is more subtle. λ < 1 introduces bias into advantage estimates by bootstrapping from an imperfect value function. In small-scale settings, this bias is accepted as a trade-off for reduced variance. But the paper argues that at the scale of ORZ training—8,192 trajectories per iteration, hundreds of iterations, tens of millions of token-level updates—the variance from Monte Carlo estimates is naturally mitigated by data volume. The bias, however, remains. Specifically, the bias from λ < 1 systematically undervalues early reasoning tokens because the (initially poorly calibrated) value function underestimates how much early reasoning contributes to eventual success. This creates a feedback loop: early reasoning gets undervalued → the policy produces less early reasoning → the value function never learns to value it → length collapses. The ablation confirms that λ=0.95 produces exactly this pattern.

The conceptual innovation is not the specific parameter values but the deliberate prioritization of bias elimination over variance reduction justified by training scale. This inverts the standard RL wisdom, which typically favors some bias to control variance in small-data regimes. The insight is that large-scale RL training changes which statistical concerns are binding constraints: at sufficient scale, variance washes out and bias becomes the limiting factor. This has implications beyond GAE parameters—it suggests that other design choices in RL training (value function architecture, advantage normalization, update frequency) should also be re-evaluated through the lens of "what is the bias I am introducing, and can I afford it at scale?"

A practical consequence is the dramatic simplification of the GAE computation (Appendix D): the advantage formula collapses from a weighted sum of TD errors to a single subtraction R - V(s_t), and the value target becomes simply R. This is not just mathematically elegant—it means the critic's learning objective is maximally clear (predict, at every token position, whether the trajectory will succeed), and the advantage signal is free of the compounding approximation errors that plague multi-step bootstrapping. The paper provides reference pseudocode (Algorithm 1) that is remarkably short.

Innovation 4: Data Scaling as the Primary Driver of Sustained Improvement in Reasoning RL

While the observation that more training data improves performance is not novel in itself, the paper makes a specific argument that is underappreciated in the reasoning RL literature: data quantity and diversity, not algorithmic sophistication, is the primary bottleneck for sustained improvement during Reasoner-Zero training. This is a reframing of where research effort should be directed.

The evidence comes from the data ablation (Figure 3, right): training on the MATH 7.5k dataset (a standard academic benchmark) leads to both reward and response length plateauing early, while ORZ's 57k diverse dataset produces sustained improvement throughout training with no signs of saturation. The paper does not claim ORZ 57k is optimally sized or composed—it is a proof of concept that scaling data breaks through the plateau that limited prior attempts.

The mechanism for why small datasets plateau is specific to the Reasoner-Zero setting. The learning signal comes from problems where the base model sometimes succeeds and sometimes fails—these create variation in the terminal reward R that the critic can learn to predict and that the policy can learn to influence. Problems that are trivially easy (always R=1) provide no gradient because all advantages are near zero (the critic learns to predict 1, so Â_t ≈ 0). Problems that are impossibly hard (always R=0) also provide no gradient (the critic learns to predict 0). With a small dataset like MATH 7.5k, the model quickly masters the solvable problems and the remaining problems are too hard for any correct trajectories—the learning signal vanishes. A larger, more diverse dataset ensures a continuous supply of problems at the right difficulty level.

But the diversity claim goes beyond difficulty calibration. The paper's generalization results (Table 2) show that ORZ-32B—trained only on reasoning tasks with no instruction tuning—improves MMLU from 83.3% (base) to 84.9% and MMLU_PRO from 55.1% to 74.4%, surpassing the instruction-tuned Qwen2.5-32B-Instruct (69.2% on MMLU_PRO). This is striking: RL training on a dataset of math and logic problems transfers to general knowledge benchmarks. The paper attributes this to the diversity of reasoning patterns in the training data—logical puzzles, counterfactual scenarios, multi-step deductions across domains—which forces the model to learn generalizable reasoning strategies rather than narrow math-specific heuristics.

This insight reframes the scaling agenda for reasoning RL. The paper's future directions (Section 5) lead with "Data Scaling: We will investigate how to effectively scale up by increasing the quantity, quality and diversity of training data," and the open-source release of the ORZ 57k dataset is explicitly intended to "encourage the research community to contribute and share more training data." The paper is arguing that data—not algorithms, not reward design, not hyperparameter tuning—is the axis along which the next order-of-magnitude improvements will come, and that making data curation a community effort is the highest-leverage way to advance the field.

Innovation 5: The Diagnostic Framework for Training Stability—Making Collapse Explainable and Avoidable

The paper's final conceptual contribution is not a technique but a diagnostic methodology for understanding and preventing training collapse in reasoning RL. Prior to ORZ, reports of GRPO instability were anecdotal—community forum posts describing "sudden degeneration" or "repetition collapse" without systematic characterization. The paper provides three concrete, quantitative diagnostics that both explain why collapse happens and verify that the proposed solution prevents it.

The first diagnostic is the value function visualization (Figure 5, right). By examining what the critic assigns high vs. low values to, one can directly observe whether the model has learned to distinguish productive reasoning from degenerate patterns. This is a qualitative check but an informative one—it reveals what the critic considers predictive of success or failure, which may surface unexpected failure modes beyond repetition.

The second diagnostic is the comparative advantage analysis (Figure 5, left). By computing what advantage different algorithms would assign to the same tokens (specifically, tokens in repetitive regions), one can quantify whether an algorithm is penalizing or reinforcing degenerate behaviors. The paper's key finding—that PPO assigns consistently more negative advantages to repetitive tokens than GRPO—is a diagnostic result that generalizes beyond this specific comparison. Any new algorithm for reasoning RL should be evaluated on whether its advantage estimates correctly penalize known failure patterns.

The third diagnostic is the stability monitoring triplet (Figure 7): training reward, truncation rate (fraction of generations hitting the maximum token limit), and average repeat score. These three metrics together provide an early warning system for training collapse. The paper shows that GRPO's collapse is visible in all three metrics simultaneously around step 240—reward crashes, truncation rate jumps to 1.0, repeat score jumps to 1.0. In contrast, PPO maintains stable values for all three throughout training. This triplet provides a practical protocol for monitoring the health of reasoning RL training runs and catching instability before it becomes catastrophic.

This diagnostic framework is a contribution to research methodology, not just to algorithm design. It provides a way to study why training succeeds or fails that is grounded in measurable quantities rather than post-hoc speculation. Future work on reasoning RL can use these diagnostics to evaluate new algorithms, compare design choices, and identify failure modes—even if those algorithms are not PPO and even if they use different advantage estimators. The paper is providing both a recipe (how to train stably) and a set of tools (how to verify that training is stable and understand what is happening inside the model).

5. Experimental Analysis

Evaluation Methodology

• Dataset. The primary benchmarks are AIME 2024, AIME 2025, MATH500, and GPQA Diamond—standard competition-level math and science reasoning datasets. Training uses the ORZ 57k dataset (57,000 curated question-answer pairs from AIME through 2023, MATH, Numina-Math, Tulu3 MATH, OpenR1-Math-220k, AoPS forum, and programmatically synthesized general reasoning tasks). Generalization is evaluated on MMLU and MMLU_PRO. All evaluations use the same prompt template as training (Appendix Table 5).

• Base model(s). Primary experiments use Qwen2.5-32B and Qwen2.5-7B base models—deliberately chosen as the same base model family and size as DeepSeek-R1-Zero-Qwen-32B to enable direct comparison. Scaling ablations extend to Qwen2.5-1.5B and Qwen2.5-0.5B to demonstrate robustness across model scales. For the distillation experiment, ORZ-R1-Distill-Qwen-14B initializes from DeepSeek-R1-Distill-Qwen-14B.

• Metrics. All benchmark evaluations report accuracy (%) — the fraction of problems for which the model's extracted final answer matches the ground-truth answer via exact string match, averaged over 16 sampled responses per question (pass@1 with 16 attempts). During training, the paper tracks train reward (average binary reward on the 128-prompt training batch) and average response length (in tokens). The stability analysis (Figure 7) additionally tracks truncation rate (fraction of generations that hit the maximum token limit) and average repeat score (a metric of repetitive content). For the critic analysis (Figure 5), the paper computes average advantage on repetitive tokens under both PPO and a hypothetical GRPO calculation applied to the same trajectories.

• Baselines. The primary external baselines are DeepSeek-R1-Zero-Qwen-32B [2] and DAPO-Qwen-32B [5] — both Reasoner-Zero models trained from the same Qwen2.5-32B base model. DeepSeek-R1-Zero-Qwen-32B results are taken directly from the DeepSeek paper (reported at 47.0% on AIME2024, 91.6% on MATH500, 55.0% on GPQA Diamond). DAPO results are reported both from the original paper (50.0% on AIME2024) and from the authors' own evaluation of the released DAPO checkpoint using their evaluation metric (48.3% on AIME2024, 37.9% on AIME2025, 71.8% on MATH500, 16.0% on GPQA Diamond). For the generalization experiments, the baseline is Qwen2.5-32B-Instruct (the instruction-tuned variant without reasoning-specific RL). For the distillation experiment, the baselines are DeepSeek-R1-Distill-Qwen-14B and DeepSeek-R1-Distill-Qwen-32B. The ablation studies in Figure 3 use internal baselines: GAE λ=0.95 vs. λ=1.0, KL Loss and KL Penalty vs. no KL regularization, and MATH Train 7.5k vs. ORZ 57k. The PPO vs. GRPO stability comparison in Figure 7 is an internal head-to-head run.

• Generation budget / compute accounting. The paper does not measure compute in FLOPs or GPU-hours. The primary unit of comparison across methods is training steps (iterations), where one step processes 128 prompts × 64 responses = 8,192 trajectories. The headline efficiency claim—"only 1/10 of the training steps compared to the DeepSeek-R1-Zero pipeline"—refers to the number of iterations required to reach equivalent or superior benchmark performance, with ORZ-32B reaching ~48% on AIME2024 at roughly 1,000 steps while DeepSeek-R1-Zero-Qwen-32B requires approximately 10,000 steps to reach ~47% (Figure 1). However, the paper does NOT normalize for per-step compute differences—DeepSeek-R1-Zero likely used different batch sizes, generation budgets per prompt, and model configurations, so the 10× step reduction does not necessarily imply 10× total FLOPs reduction. For the DAPO comparison, the paper states DAPO uses "roughly fivefold more training iterations" without normalizing for per-iteration compute. All evaluation results are computed using 16 responses per question (majority voting is not used; the paper reports pass@1 averaged over 16 samples, which differs from standard pass@k metrics).

• Cross-validation / statistical protocol. The paper does not employ cross-validation or statistical significance testing. All benchmark evaluations are reported on the standard test sets. The training reward and response length curves (Figure 2) are reported on the training data itself, not a held-out validation set. There is no mention of confidence intervals, error bars, or multiple random seeds for any reported result. The annealing stage's 13k "difficult prompts" are identified based on the model's own performance during the first 1,100 training steps—this is a form of adaptive data selection but not cross-validation. The paper's approach is purely empirical: train once, evaluate on standard benchmarks, report point estimates. This is consistent with common practice in the LLM scaling literature but means the reported numbers should be interpreted as single-run results with unknown variance.

Main Quantitative Results

The paper organizes its quantitative results into four clusters: training dynamics (what happens during training), algorithm comparisons (PPO vs. GRPO and internal ablations), benchmark evaluations (final performance vs. prior work), and generalization (transfer to non-reasoning benchmarks).

Training Dynamics: Steady Scaling of Reward and Response Length

Figure 2 shows the primary training-time metrics—average training reward and average response length—across ORZ models at four scales (0.5B, 1.5B, 7B, 32B) over the first ~1,000 training steps. The key patterns:

• Train reward increases steadily at all scales. For ORZ-32B, training reward grows from approximately 0.2 at step 0 to approximately 0.6–0.7 by step 800, with no signs of plateausing. ORZ-7B similarly grows from ~0.15 to ~0.55. The smaller models (0.5B, 1.5B) show proportionally lower absolute rewards but the same monotonic upward trend, reaching approximately 0.3–0.4 by step 1,000. This is evidence that the minimalist RL recipe works across model sizes without requiring size-specific tuning.

• Response length grows substantially at all scales. ORZ-32B average response length increases from roughly 1,000 tokens at step 0 to roughly 8,000–10,000 tokens by step 800, though with notable fluctuations—the curve oscillates between ~6k and ~10k without any apparent periodicity. ORZ-7B shows smoother growth from ~1k to ~6k tokens. ORZ-1.5B grows from ~500 to ~2,000 tokens. ORZ-0.5B shows minimal growth (staying around 500–1,000 tokens), suggesting there may be a minimum model size below which length scaling does not emerge.

• Length fluctuations in ORZ-32B do not destabilize training. The paper explicitly notes that "the Response Length curve of the ORZ-32B model exhibits noticeable fluctuations, yet these fluctuations do not negatively impact training stability or the continuous growth of reward" (Section 3.4). This is presented as evidence of robustness: the training recipe tolerates substantial stochastic variation in generation length without collapsing, which is particularly notable given that GRPO-based training (Figure 7) shows length-related metrics (truncation rate) spiking to 1.0 during collapse.

• Reflection patterns emerge naturally and correlate with longer responses. Figure 4 (right) tracks two additional length metrics for ORZ-32B: average response length overall and average length of correct responses that contain reflection keywords ("wait," "recheck," "retry," "alternatively," "however"). The average correct reflection length consistently exceeds the overall average length, and both curves trend upward during training. The gap is substantial—at step 600, correct reflection responses average roughly 6,000 tokens while the overall average is roughly 3,500 tokens. This is quantitative evidence that self-reflection (checking work, reconsidering, retrying) is not just an emergent behavior but one specifically associated with correct answers and longer, more thorough reasoning.

• Format adherence converges rapidly without explicit supervision. Figure 4 (left) shows the "correct format ratio"—the fraction of generated responses that properly use thinking... response and <answer>...</answer> tags. ORZ-7B starts at approximately 0.85 correct format ratio and converges to ~1.0 within 25 training steps. ORZ-32B starts lower (approximately 0.65) but converges to ~0.95 by step 50 and remains stable. This confirms that the rule-based accuracy reward, which implicitly penalizes unformatted responses (they cannot be scored and always get R=0), is sufficient to drive format adherence without a dedicated format reward component.

Benchmark Evaluations: Superior Performance with 1/10 the Training Steps

Figure 1 (the paper's headline figure) and Table 1 present the central performance comparisons against DeepSeek-R1-Zero-Qwen-32B and DAPO-Qwen-32B.

AIME 2024 (Figure 1, top-left; Table 1):

• ORZ-32B reaches approximately 48.1% accuracy (averaged over 16 responses), surpassing DeepSeek-R1-Zero-Qwen-32B's reported 47.0% and DAPO-Qwen-32B's reported 50.0% (note: DAPO's number is from their paper; ORZ's own evaluation of the released DAPO checkpoint gives 48.3%, making the comparison essentially tied given unknown variance).
• The learning curves in Figure 1 show that ORZ-32B reaches ~40% at roughly 300 training steps, ~45% at ~600 steps, and plateaus near ~48% by ~1,000 steps. DeepSeek-R1-Zero-Qwen-32B, by contrast, climbs steadily from ~15% at step 10^1 to ~47% at step 10^4—requiring approximately 10× as many training steps to reach the same accuracy. ORZ-7B reaches ~17.9% (Table 4), demonstrating that meaningful AIME performance is achievable even at the 7B scale through RL alone.
• The efficiency claim is visible in the x-axis: ORZ-32B's curve is shifted left by roughly an order of magnitude relative to DeepSeek-R1-Zero-Qwen-32B's.

AIME 2025 (Figure 1, top-right; Table 1):

• ORZ-32B achieves 36.0%, with no DeepSeek-R1-Zero-Qwen-32B comparison available (DeepSeek did not report AIME2025 results). DAPO-Qwen-32B, evaluated by ORZ's own metric on the released checkpoint, achieves 37.9%—a small advantage for DAPO on this benchmark, though ORZ notes DAPO required approximately 5× more training iterations.
• The AIME2025 benchmark was released after the training data cutoff (AIME data up to 2023 was included in ORZ 57k), so this tests generalization to unseen competition problems. ORZ-7B reaches 15.6% (Table 4).

MATH500 (Figure 1, bottom-left; Table 1):

• ORZ-32B achieves 92.2%, outperforming DeepSeek-R1-Zero-Qwen-32B's 91.6% and substantially outperforming DAPO-Qwen-32B's 71.8% (as evaluated by ORZ). The learning curve shows steady improvement from ~80% at step 10^1 to ~92% by step 10^3, with a noticeably steep initial climb in the first 100 steps.
• This is a particularly strong result because DAPO's substantially lower MATH500 score (71.8% vs. ORZ's 92.2%) suggests a systematic difference in training approach. The paper hypothesizes this is "related to their data curation and formatting approach, which transforms every answer into an integer for verification disambiguation" (Section 3.4)—DAPO's format choices may cause it to perform poorly on MATH500 questions with non-integer answers.
• ORZ-7B reaches 81.4% (Table 4).

GPQA Diamond (Figure 1, bottom-right; Table 1):

• ORZ-32B achieves 55.5%, essentially matching DeepSeek-R1-Zero-Qwen-32B's 55.0% and dramatically outperforming DAPO-Qwen-32B's 16.0% (ORZ's evaluation of the DAPO checkpoint). The DAPO GPQA result is the most striking discrepancy: 16.0% vs. 55.5% represents a fundamental difference in capability on this benchmark.
• The learning curve for ORZ-32B shows improvement from ~26% at step 10^1 to ~55% by step 10^3, with continued upward trajectory. ORZ-7B reaches 36.6% (Table 4).

Cross-benchmark patterns in the learning curves (Figure 1):

• Across all four benchmarks, ORZ-32B's curves consistently show rapid improvement in the first 100–300 steps, then more gradual improvement through step ~1,000. There is no evidence of performance degradation at any point—unlike supervised fine-tuning where overfitting can cause test performance to decline, the RL training continues to improve or maintain benchmark accuracy.
• The 7B curves consistently track below the 32B curves but show the same qualitative pattern—steady improvement without saturation—at a proportionally lower absolute level. This is evidence of model scale as a consistent factor: the same training recipe produces better results with larger models, but it works at all tested scales.

Training curves for small models (Figure 6, Appendix A):

• ORZ-1.5B and ORZ-0.5B show measurable but much weaker performance: ORZ-1.5B reaches 3.5% on AIME2024, 1.0% on AIME2025, 58.0% on MATH500, and 16.8% on GPQA Diamond (Table 4). ORZ-0.5B reaches 1.0% on AIME2024, 0.2% on AIME2025, 31.0% on MATH500, and 12.1% on GPQA Diamond.
• The learning curves in Figure 6 show shallow upward slopes—the 0.5B model improves from approximately 0% to 1% on AIME2024 over 200 steps, and the 1.5B model improves from approximately 0% to 3.5%. These models are clearly not competitive on hard benchmarks, but the fact that any improvement occurs at all at these scales is notable—the paper frames this as evidence that "meaningful performance gains are observable even at the scale as small as 0.5B parameters" (Section 3.2).

Generalization to Non-Reasoning Benchmarks

Table 2 presents a striking result: RL training on reasoning tasks alone improves performance on general knowledge benchmarks.

• MMLU: ORZ-32B achieves 84.9%, up from Qwen2.5-32B-Base (83.3%) and Qwen2.5-32B-Instruct (83.2%). The improvement over the base model (+1.6 percentage points) is modest but real—it means the reasoning RL training did not degrade factual knowledge. The improvement over the instruction-tuned variant (+1.7 points) is more notable: a model trained only on math and logic problems, with no instruction tuning or human preference data, outperforms the instruction-tuned model on a general knowledge benchmark.

• MMLU_PRO: ORZ-32B achieves 74.4%, dramatically up from Qwen2.5-32B-Base (55.1%) and Qwen2.5-32B-Instruct (69.2%). The +19.3 point improvement over the base model and +5.2 point improvement over the instruction-tuned variant suggest that the reasoning skills learned during ORZ training—structured thinking, multi-step deduction, verification—transfer effectively to the more challenging MMLU_PRO questions, which require deeper reasoning than standard MMLU.

• DAPO comparison: DAPO-Qwen-32B achieves 79.7% on MMLU and 64.5% on MMLU_PRO—both substantially below ORZ-32B. This reinforces the hypothesis that ORZ's data curation and training approach produces more generalizable reasoning skills.

Distillation Model Enhancement

Table 3 shows that ORZ training applied to an already-reasoning-enhanced model yields further gains:

• ORZ-R1-Distill-Qwen-14B (initialized from DeepSeek-R1-Distill-Qwen-14B) achieves 75.2% on AIME2024 (up from 69.7%), 60.0% on AIME2025 (up from 49.1%), 95.6% on MATH500 (up from 93.9%), and 60.4% on GPQA Diamond (up from 59.1%).
• Notably, this 14B model after ORZ training surpasses the larger DeepSeek-R1-Distill-Qwen-32B on AIME2024 (75.2% vs. 72.6%), MATH500 (95.6% vs. 94.3%), and is competitive on GPQA Diamond (60.4% vs. 62.1%). On AIME2025, it ties the 32B model at 60.0%.
• The training used only 300 iterations on the 13k difficult prompts identified during ORZ-32B training, demonstrating that the ORZ recipe is effective even with limited data and steps when starting from a distilled model.

Ablation Studies and Robustness Checks

The paper conducts three primary ablations (Figure 3) and several supplementary analyses.

GAE λ parameter (Figure 3, left): Training ORZ-7B with GAE λ=1.0 vs. λ=0.95. λ=1.0 produces "reward that rapidly increases and then steadily grows," while λ=0.95 shows "much slower reward progression." On response length, λ=1.0 maintains "reasonable increasing speed," while λ=0.95 leads to "collapsed length dynamics"—the response length curve degrades rather than growing. The paper interprets this as evidence that the bias introduced by λ<1 systematically undervalues early reasoning tokens, discouraging the model from producing long reasoning chains and creating a feedback loop toward shorter responses. The ablation uses ORZ-7B training data rather than benchmark evaluation, running for approximately 200 steps.

KL regularization (Figure 3, mid): Three conditions tested on ORZ-7B: no KL regularization, KL Loss (adding a KL divergence penalty term to the PPO objective), and KL Reward Shaping (subtracting a KL penalty from the reward). "W/O. KL" achieves the highest reward and longest response length, with KL Loss and KL Reward Shaping both showing "slower training process" on both metrics. The gaps are visible but not enormous—at step 200, W/O KL reward is approximately 0.75 vs. approximately 0.60 for both KL variants; W/O KL response length is approximately 1,200 tokens vs. approximately 1,000 for KL variants. The paper's interpretation that KL regularization is "actively harmful" is supported directionally, but the differences are modest in absolute terms and the ablation is limited to ORZ-7B with 200 steps—longer training might amplify or reduce the gap.

Data scale (Figure 3, right): Training ORZ-7B on the full ORZ 57k dataset vs. MATH Train 7.5k (a sub-sample of the MATH training set). With ORZ 57k, both training reward and response length show sustained improvement through 600 training steps. With MATH 7.5k, both metrics plateau early—reward saturates around step 150 and response length plateaus by step 100. At step 600, ORZ 57k reward is approximately 0.80 vs. MATH 7.5k's approximately 0.70, and ORZ 57k response length is approximately 2,400 tokens vs. MATH 7.5k's approximately 600 tokens. This is the most dramatic ablation result in the paper: data scale is the primary driver of sustained improvement, and training on a small benchmark dataset (even a high-quality one like MATH) leads to early stagnation.

Data curation: English-only vs. English+Chinese (Figure 8, Appendix C.2): Adding Chinese data to the ORZ 57k dataset produces "inferior training stability and final model performance." The English-only dataset yields consistently higher reward (approximately 0.55 at step 100 vs. 0.45 for English+Chinese) and longer responses (approximately 900 tokens vs. 700 tokens). The paper does not deeply analyze why Chinese data degrades training, but references OpenR1's finding that "SFT performance degradation on Chinese subsets was due to simpler question patterns"—suggesting that Chinese data may introduce quality or difficulty distribution issues. This is a practical but somewhat unsatisfying result: the paper excludes Chinese data without investigating whether higher-quality Chinese data could be beneficial.

PPO vs. GRPO stability (Figure 7, Appendix C.1): This is not presented as an ablation in the main paper but as a direct head-to-head comparison. Training ORZ-7B with PPO vs. GRPO, monitoring reward, truncation rate, and average repeat score. GRPO experiences sudden destabilization around step 240: reward crashes (from approximately 0.6 to approximately 0.2), truncation rate surges from near-zero to 1.0, and average repeat score also surges to 1.0. PPO maintains stable reward (around 0.5–0.6), low truncation rate (near zero), and low repeat scores throughout. The GRPO run is discontinued after collapse; PPO continues stably. This is the paper's strongest evidence for the necessity of a learned critic—GRPO without a critic fails catastrophically, and the failure mode (repetition collapse) is exactly what the critic learns to penalize (Figure 5).

Annealing stage effectiveness: The paper does not provide an ablation comparing ORZ-32B with vs. without annealing. The annealing stage (100 additional steps on 13k difficult prompts with learning rate decay) is presented as part of the standard recipe but its marginal contribution is not isolated. The paper states it is "explicitly designed to enhance the model's capability on more complex reasoning tasks" but provides no evidence that training without annealing would produce worse results. Given that the learning curves in Figure 1 appear to be plateauing by step ~1,000 anyway, the annealing contribution may be modest relative to the main training phase.

Value head initialization: The critic's value head is initialized from U(-√5, √5) with no bias term (Appendix B). The paper does not ablate this choice—no comparison to alternative initializations (zero, normal distribution, learned initialization from the base model's hidden states). This is a minor point but relevant for reproduction.

Critic update frequency: The critic is updated 12 times per iteration (12 mini-batches) while the policy is updated once. The paper does not ablate this ratio, stating that the critic is "less sensitive to off-policy updates" because its target (R) does not depend on the generating policy. However, the specific choice of 12 mini-batches is not justified beyond this statement—numbers like 4 or 24 might work equally well or better, and the interaction with the 12× higher effective learning rate for the critic (5×10⁻⁶ vs. 1×10⁻⁶ for the policy) is not explored.

Prompt template: The paper uses a specific template (Appendix Table 5) throughout training and evaluation. No ablation is provided comparing alternative templates—for example, omitting the explicit instruction about answer extraction, or changing the tag names. The template's contribution to format adherence (Figure 4, left) is confounded with the training signal: the accuracy reward incentivizes formatting regardless of the specific template, so the template's marginal value is unclear.

Critical Assessment

Do the Experiments Support the Central Claims?

The paper makes several major claims. Let us evaluate each against the experimental evidence.

Claim: "Vanilla PPO with GAE (λ=1, γ=1) and simple rule-based rewards, without any KL regularization, is sufficient to scale up both benchmark performance and response length." The evidence strongly supports this for Qwen2.5 models at the tested scales. Figures 1 and 2 show that the recipe works—performance improves and response length grows across four model sizes on four benchmarks, with no training collapse. The ablation studies (Figure 3) demonstrate that specific components (λ=1.0, no KL, large diverse dataset) are individually important for the recipe's success. However, "sufficient" here means "sufficient for the Qwen2.5 model family on the tested benchmarks with the ORZ 57k dataset." The paper does not test whether the recipe transfers to other model families (Llama, Mistral, Gemma), other domains beyond math and logic, or other dataset compositions. The claim of sufficiency is bounded by these experimental conditions, and the paper does not explore failure modes—training runs that might fail with this recipe under different conditions (e.g., much larger models, different data distributions, much longer training).

Claim: "Open-Reasoner-Zero achieves superior performance across AIME2024, MATH500, and GPQA Diamond while requiring only 1/10 of the training steps compared to DeepSeek-R1-Zero." Supported in the narrow sense of "reaches the same or better benchmark numbers with fewer training iterations." ORZ-32B reaches ~48% on AIME2024 at ~1,000 steps vs. ~47% at ~10,000 steps for DeepSeek-R1-Zero-Qwen-32B (Table 1, Figure 1). MATH500 and GPQA Diamond comparisons are similar. However, there are important caveats. First, "training steps" is not a normalized unit of compute—the two systems likely used different batch sizes, different numbers of generated responses per prompt, different model parallelism strategies, and different hardware. A fair FLOPs comparison (like the one in the summarized paper's Section 7) is not attempted. Second, DeepSeek-R1-Zero-Qwen-32B was not evaluated by the ORZ authors on their setup—the numbers come from the DeepSeek paper and may use different evaluation protocols (e.g., different number of samples for pass@k, different grading functions). Third, the AIME2024 comparison is close (48.1% vs. 47.0%) and within a range where evaluation differences (prompt template, sampling temperature, number of samples, grading strictness) could account for the gap. The claim of "superior performance" is technically correct for the reported numbers but the practical difference from DeepSeek-R1-Zero is modest.

Claim: "The learned critic effectively identifies and devalues repetitive response patterns, yielding more robust advantage estimations and enhancing training stability." This is the paper's strongest mechanistic claim and is well-supported by the evidence. Figure 5 (right) shows the value function assigning lower values to repetitive tokens in a concrete example. Figure 5 (left) shows that PPO's advantage estimates for repetitive tokens are consistently more negative than GRPO's would be. Figure 7 shows GRPO collapsing with severe repetition while PPO remains stable. The causal chain is plausible and internally consistent: the critic learns that repetition predicts failure, this produces negative advantages for repetitive tokens, the policy reduces repetition probability, and training avoids collapse. However, there are two unverified links: (1) The paper does not show that the negative advantages cause reduced repetition—it shows correlation between negative advantages and training stability, but does not ablate the critic's ability to detect repetition (e.g., by training a critic on trajectories with repetition artificially removed) to show that stability depends specifically on repetition detection rather than some other property of learned value functions. (2) The GRPO comparison in Figure 5 (left) is hypothetical—it computes what GRPO "would have assigned" rather than running GRPO and PPO on identical trajectories. This is reasonable (PPO and GRPO with the same policy snapshot would generate different trajectories, making direct comparison impossible), but it means the advantage comparison compares PPO's actual advantages on PPO-generated trajectories against GRPO-approximated advantages on the same PPO-generated trajectories. The trajectories themselves were generated under PPO's optimization, which may have already suppressed some repetition that GRPO would have produced.

Claim: "We demonstrate that a minimalist approach... is sufficient to scale up both benchmark performance and response length." The "minimalist" framing is supported by the ablations showing that added components (KL regularization, format rewards, λ<1, smaller datasets) hurt or are unnecessary. However, "minimalist" is relative—the system still requires: two full copies of the base model in memory (policy + critic, no weight sharing), 64 response generations per prompt per iteration, 12 critic updates per policy update, a carefully curated 57k dataset with LLM-based difficulty filtering, a specific prompt template, an annealing stage for the 32B model, and batch-level advantage normalization. This is "minimalist" compared to DeepSeek-R1-Zero's multi-component reward and KL regularization, but it is still a complex distributed training system. The paper's framing as "minimalist" serves a rhetorical purpose (simplifying the design space) but should not be confused with "trivial to implement."

Claim: "We release comprehensive resources including code, data, and model to the community." Factually true—the GitHub and HuggingFace repositories are linked in the abstract. This is a contribution to reproducibility that goes beyond the experimental results themselves.

Genuine Weaknesses in the Experimental Design

No compute-normalized comparisons. The headline efficiency claim (1/10 the training steps) is based on iteration counts, not total FLOPs or GPU-hours. The paper does not report the total compute used for ORZ training or compare it to estimates of DeepSeek-R1-Zero's training compute. Given that ORZ might use different batch sizes, generation budgets, or model parallelism strategies, the 10× iteration reduction could correspond to anywhere from 2× to 20× total compute reduction—we cannot tell from the reported data. This is a significant omission for a paper that positions efficiency as a central contribution.

Single model family, single dataset domain. All results use Qwen2.5 models and math/logic reasoning tasks. The paper does not demonstrate that the recipe works for other model architectures (e.g., Llama, Mistral), other domains (code generation, formal theorem proving, scientific reasoning beyond multiple choice), or non-English languages. The generalization experiments (Table 2) show transfer from math/logic training to general knowledge benchmarks, which is encouraging, but the scope of the training data and evaluation remains narrow relative to the paper's ambition of "democratizing advanced RL training techniques."

No statistical characterization of variance. All results are point estimates from single training runs. There are no error bars, no confidence intervals, no multiple random seeds, no cross-validation. For benchmark evaluations with 16 responses per question, the reported accuracy has inherent sampling variance that is not characterized—a 48.1% on AIME2024 with 30 questions (AIME has 30 questions per year) and 16 samples per question could easily vary by ±2-3% due to sampling noise alone. The paper's "superior performance" claims relative to DeepSeek-R1-Zero-Qwen-32B (48.1% vs. 47.0%) fall within this noise range, making the claim of superiority uncertain. This is standard practice in the LLM literature but worth noting.

The annealing stage is not ablated. The 100-step annealing phase on 13k difficult prompts is described as part of the recipe but its contribution is never isolated. It is possible that the final performance gains attributed to the overall ORZ recipe are substantially due to this targeted fine-tuning phase, which is conceptually distinct from the "minimalist RL from base model" framing. An ablation comparing ORZ-32B with and without annealing would clarify whether the main RL training phase alone achieves the reported numbers or whether annealing is load-bearing.

No exploration of failure modes for the ORZ recipe itself. The paper demonstrates that ORZ works (no collapse, steady improvement), but does not explore when it might fail. What happens if training continues past 1,000–1,200 steps? Does performance plateau, degrade, or continue improving? What happens with even larger models (70B, 405B) where the dynamics might differ? What happens if the dataset is expanded to include noisier or more diverse problems (e.g., informal forum posts with ambiguous answers)? Understanding the failure modes of the recipe would strengthen the paper's contribution to training stability, since stability is most informative at its limits.

The DAPO comparison is confounded by evaluation differences. The paper evaluates the released DAPO checkpoint using ORZ's own evaluation metric and reports substantially different numbers than DAPO's paper for some benchmarks (especially GPQA Diamond: 16.0% vs. not reported; MATH500: 71.8% vs. not reported). The paper attributes this to DAPO's data formatting approach (converting answers to integers), but this is a hypothesis, not a demonstrated fact. The comparison is between ORZ training + ORZ evaluation vs. DAPO training + DAPO evaluation (from the DAPO paper) and also vs. DAPO training + ORZ evaluation. The multiple comparison baselines make it difficult to cleanly attribute performance differences to training methodology rather than evaluation artifacts.

The 7B and 32B models use different training data. The paper states that the 13k difficult prompts for annealing were identified based on ORZ-32B's training dynamics. The ORZ-7B model presumably does not use this annealing data (the paper does not specify whether a similar annealing stage was applied to 7B). This means the 7B and 32B results are not directly comparable—the 32B model benefits from an additional data curation step that the 7B model does not.

Missing Experiments That Would Have Strengthened the Paper

GRPO with the same data and compute budget as PPO. Figure 7 shows GRPO collapsing at step 240, but was this GRPO run with the same hyperparameters, same data, and same generation budget as the PPO run? The paper does not specify GRPO's configuration (advantage normalization, learning rate, clipping parameters). A controlled comparison where GRPO is given every advantage—tuned learning rate, optimal group size, perhaps a smaller generation budget to increase update frequency—would make the PPO superiority argument more convincing. It is possible that GRPO can be stabilized with appropriate tuning and that the comparison in Figure 7 reflects suboptimal GRPO configuration rather than a fundamental algorithmic limitation.

Ablation of the critic's contribution by degrading it. The paper argues that the critic's ability to detect repetition is the key to stability. An experiment that degrades this ability—for example, by training the critic on shorter trajectories where repetition is less likely, or by adding noise to the value targets—would test whether stability depends specifically on the critic being able to identify repetitive patterns, or whether any reasonable value function provides sufficient advantage estimation.

Training on the full 57k dataset with GRPO. The data ablation (Figure 3, right) shows that ORZ 57k sustains improvement while MATH 7.5k plateaus. Would GRPO trained on ORZ 57k also avoid collapse? Perhaps GRPO's collapse in Figure 7 is exacerbated by training on a smaller or less diverse dataset, and the primary driver of stability is data scale rather than algorithm choice. The paper does not disentangle these factors.

Comparison against a KL-regularized PPO baseline with γ<1 and λ<1. The paper's ablation compares λ=1.0 vs. λ=0.95 and no-KL vs. KL, but these comparisons are in isolation. What about the "standard" PPO configuration used in RLHF—γ=0.95, λ=0.95, KL regularization—applied to the same data and compute budget? This would test whether ORZ's specific parameter choices (γ=1, λ=1, no KL) are genuinely necessary or whether any well-tuned PPO configuration works at sufficient data scale.

Evaluation with different numbers of samples per question. All benchmark results use 16 responses per question. Reporting pass@1 (single sample), pass@4, and pass@64 would reveal how much of ORZ's performance comes from the model's ability to produce correct answers consistently vs. occasionally. DeepSeek-R1-Zero reports multiple pass@k metrics; ORZ's choice of "averaged on 16 responses" makes direct comparison difficult.

Conditions Under Which Claims Hold

• The 10× training step reduction claim holds only when comparing iteration counts to DeepSeek-R1-Zero-Qwen-32B's reported learning curves. It is not normalized for per-iteration compute, model size, or infrastructure differences. The claim should be interpreted as "ORZ's learning curves rise faster on a per-iteration basis" rather than "ORZ uses 10× less total compute."

• The "superior performance" claim holds for the specific benchmarks reported (AIME2024, MATH500, GPQA Diamond) with the specific evaluation protocol (16 responses averaged). On AIME2024, the margin over DeepSeek-R1-Zero is 1.1 percentage points; on GPQA Diamond, it is 0.5 percentage points. These margins are within plausible evaluation noise. The claim is better supported for MATH500 (92.2% vs. 91.6%) and is strongest relative to DAPO on MATH500 and GPQA Diamond, though the DAPO comparison is confounded by potential evaluation differences.

• The "training stability" claim is well-supported for PPO at the tested scales and durations (~1,000 steps for 32B). Whether stability would hold for much longer training, larger models, or different data distributions is unknown. The GRPO instability is demonstrated convincingly for ORZ-7B at ~240 steps with unreported GRPO hyperparameters.

• The "minimalist recipe works" claim holds for Qwen2.5 models on math/logic reasoning with the ORZ 57k dataset. Transfer to other model families, domains, or data compositions has not been demonstrated.

6. Limitations and Trade-offs

The 10× Training Step Reduction Claim Is Not Normalized for Per-Step Compute

The assumption or constraint. The paper's headline efficiency claim—that ORZ requires "only 1/10 of the training steps compared to the DeepSeek-R1-Zero pipeline" (Abstract, Section 1, Section 3.4)—measures training progress by iteration count, not by total FLOPs or GPU-hours. The paper does not report total compute used for ORZ training, does not estimate DeepSeek-R1-Zero's per-step compute, and makes no attempt to normalize the comparison for differences in batch size, generation budget per prompt, model parallelism strategy, or hardware efficiency. The comparison is purely based on reading DeepSeek-R1-Zero's published learning curves (Figure 1) and observing that ORZ reaches ~48% AIME2024 at ~1,000 steps while DeepSeek reaches ~47% at ~10,000 steps.

The consequence. The 10× figure is uninterpretable as a genuine efficiency gain. If ORZ uses substantially more compute per step than DeepSeek-R1-Zero—for example, larger batch sizes (128 prompts × 64 responses = 8,192 trajectories), two full model copies in memory (policy + critic with no weight sharing, Appendix B), and 12 critic updates per policy update—then the total FLOPs reduction could be anywhere from 2× to effectively zero. The paper explicitly notes that DAPO "uses roughly fivefold more training iterations" than ORZ (Section 4), framing this as an efficiency advantage, but without per-step compute normalization, the iteration count is a misleading metric. A practitioner trying to estimate the cost of reproducing ORZ cannot derive a GPU-hour budget from the reported data. This matters because efficiency is one of the paper's three main selling points alongside stability and open-source release.

What evidence exists in the paper. The paper reports the per-iteration configuration (128 prompts, 64 responses each, Section 2.3 and Appendix B) but never converts this to FLOPs or compares it to DeepSeek-R1-Zero's configuration, which is not publicly known in detail. The learning curves in Figure 1 show ORZ-32B saturating around step 1,000 while DeepSeek-R1-Zero-Qwen-32B continues climbing past step 10,000, establishing the raw step-count difference. However, the x-axis is "Training Steps" with no normalization factor. The paper acknowledges training DeepSeek-R1-Distill-Qwen-14B for only 300 iterations (Appendix B) but does not use this to bound how much compute ORZ-style training actually costs relative to alternatives.

Mitigation status. Not addressed. The paper does not discuss the limitation of iteration-based efficiency comparisons, does not provide FLOPs estimates, and does not caveat the 10× claim. The abstract states the claim without qualification. A reader unfamiliar with RL scaling might reasonably conclude ORZ uses 10× less total compute, which the paper provides no evidence to support.

Training Stability Is Demonstrated Only for ~1,000 Steps on a Single Model Family

The assumption or constraint. The paper's central argument is that PPO with a learned critic provides training stability that GRPO lacks, preventing the repetition collapse documented in Figure 7. However, all evidence for this claim comes from training runs of limited duration: ~1,000 steps for ORZ-32B (Figure 1, Figure 2), ~600 steps for ORZ-7B (Figure 3), and ~400 steps for the GRPO comparison run (Figure 7). All experiments use the Qwen2.5 model family. The paper does not explore whether PPO's stability holds for substantially longer training, for other model architectures (Llama, Mistral, Gemma), or for much larger models (70B, 405B) where value function learning dynamics may differ.

The consequence. The claim that the learned critic "prevents training collapse" is empirically supported only within the specific bounded regime tested. Three failure modes are left unexplored. First, the critic itself may eventually overfit or degrade with extended training—if the value function's accuracy deteriorates over time (as the policy distribution shifts and the critic's training data becomes increasingly stale between the 12 mini-batch updates), the advantages it produces could become misleading, potentially causing instability at step counts beyond those tested. Second, the response length fluctuations observed in ORZ-32B (Figure 2, where length oscillates between ~6k and ~10k tokens) are described as benign during the ~1,000-step window, but these oscillations might amplify with longer training, eventually triggering truncation or repetition issues even under PPO. Third, larger models may exhibit different value learning dynamics—the paper's own data shows that ORZ-32B's response length is substantially more volatile than ORZ-7B's (Figure 2), suggesting that stability properties are not scale-invariant. A practitioner training a 70B or 405B model following the ORZ recipe cannot assume stability will hold based on the presented evidence.

What evidence exists in the paper. The training curves in Figure 1 and Figure 2 all terminate around step 1,000–1,200 without showing what happens beyond. The GRPO collapse in Figure 7 occurs at step 240, which establishes that GRPO is unstable earlier than PPO but does not prove PPO would remain stable indefinitely. The paper notes that ORZ-32B's response length "exhibits noticeable fluctuations, yet these fluctuations do not negatively impact training stability" (Section 3.4), but this observation applies only to the observed window. The paper provides no theoretical argument or extrapolation for why stability should persist beyond the tested duration.

Mitigation status. Not addressed. The paper does not discuss the bounded nature of its stability evidence, does not report any runs that continued past the plateau region, and does not frame stability as a property demonstrated only within the tested regime. The claim is stated in absolute terms: "vanilla PPO... is sufficient to achieve steady scalability" (Abstract) and PPO enables "stable training" (Section 2.2) without qualification about the tested range.

All Results Depend on a Single Model Family; Transferability to Other Architectures Is Unknown

The assumption or constraint. Every experiment in the paper—training dynamics (Figure 2), benchmark evaluations (Table 1, Table 4), ablations (Figure 3), generalization analysis (Table 2), and distillation results (Table 3)—uses models from the Qwen2.5 family exclusively. The paper states it uses "Qwen2.5-{7B, 32B} base models as our main foundation" (Section 2.3) and extends to Qwen2.5-{0.5B, 1.5B} for scaling ablations. The distillation experiment initializes from DeepSeek-R1-Distill-Qwen-14B—still within the Qwen family. No Llama, Mistral, Gemma, or other architecture is tested. The paper explicitly justifies the Qwen2.5 choice as enabling direct comparison to DeepSeek-R1-Zero-Qwen-32B (which used the same base model), but makes no claim about transferability.

The consequence. The paper's core findings—that PPO with GAE(1,1) and no KL regularization scales stably, that the critic learns to penalize repetition, that data diversity prevents plateauing, that response length grows naturally—may be partially or entirely specific to the Qwen2.5 architecture's pretraining characteristics. Qwen2.5 models may have particular in-context learning properties, tokenizer characteristics, or pretraining data distributions that make them amenable to the ORZ recipe in ways that other model families are not. For example, the paper observes that the base model already produces correctly formatted responses with high probability (Figure 4, left: ~65% for 32B, ~85% for 7B at step 0), which likely depends on Qwen2.5's specific pretraining mixture. A different base model with lower initial format adherence might require explicit format rewards that the paper argues are unnecessary. Similarly, the learned critic's ability to detect repetition (Figure 5) may depend on the base model's tokenizer and the typical patterns of repetition in Qwen2.5-generated text—other architectures might produce repetition that is harder or easier for a critic to identify. Without experiments on other model families, the paper's claim to provide a "foundational open framework for large-scale RL research on LLMs" (Section 4) rests on an untested assumption of architectural generality.

What evidence exists in the paper. The paper provides systematic evidence across model sizes within Qwen2.5 (0.5B through 32B), demonstrating that the recipe scales across two orders of magnitude in parameter count. However, this is evidence for size transferability within a single architecture, not architectural transferability. The paper does not cite or discuss prior work that might support cross-architecture generalization of its findings, nor does it provide any ablation that isolates architecture-specific vs. architecture-agnostic components of the recipe. The acknowledgment that DAPO's formatting approach (converting answers to integers) "highlights the advantages of our data curation methodology" (Section 3.4) implicitly addresses data-wise transferability but not model-wise transferability.

Mitigation status. Not acknowledged as a limitation. The paper does not discuss the fact that all experiments use a single model family, does not suggest that results may not transfer, and does not include cross-architecture testing in its future work directions (Section 5), which focus on data scaling, model scaling (within the implicit assumption of similar architectures), test-time scaling, and scenario scaling. A reader might reasonably assume the recipe is architecture-agnostic given the absence of caveats.

The GRPO Instability Comparison May Reflect Suboptimal GRPO Configuration Rather Than a Fundamental Algorithmic Defect

The assumption or constraint. The paper's strongest mechanistic claim—that PPO's learned critic is necessary for training stability and that GRPO collapses due to inability to perform token-level credit assignment—rests on a single head-to-head comparison (Figure 7, Appendix C.1) where GRPO training destabilizes around step 240. However, the paper provides minimal details about the GRPO configuration used in this comparison. It does not specify GRPO's learning rate, advantage normalization scheme, group size, clipping parameters, or any stabilization techniques attempted. The paper states that "vanilla GRPO often suffers from significant training instability, a phenomenon also observed in many community implementations" (Appendix C.1), but community implementations may suffer from implementation errors or suboptimal tuning that are not inherent to the algorithm.

The consequence. There are at least three plausible alternative explanations for GRPO's collapse in Figure 7 that the paper does not rule out. First, GRPO may require a lower learning rate than PPO because its group-based advantages have different statistical properties (e.g., higher variance due to smaller effective sample size for normalization). The paper's GRPO run may simply have used an inappropriately high learning rate. Second, GRPO's stability may depend on having an appropriate group size—with 64 responses per prompt, the group-based normalization may be reliable, but the paper does not specify whether this group size was used for the GRPO comparison or whether a different batch configuration was employed. Third, GRPO may benefit from techniques not used in the comparison, such as clipping advantages, using a reference model for KL regularization (as DeepSeek-R1-Zero did), or applying repetition penalties during generation. The paper's claim that GRPO is inherently unstable is not supported by evidence that GRPO was given a fair chance with tuned hyperparameters. This matters because if GRPO can be stabilized through appropriate configuration, the paper's central argument for choosing PPO over GRPO—and the associated cost of maintaining a separate critic model—weakens substantially.

What evidence exists in the paper. Figure 7 shows three metrics (reward, truncation rate, average repeat score) for PPO and GRPO over ~400 training steps on ORZ-7B. The GRPO curves crash simultaneously at step 240. However, the paper does not report what hyperparameter sweep (if any) was conducted for GRPO before this run, whether the collapse is reproducible across multiple random seeds, or whether alternative GRPO configurations were tested and also failed. The only configuration detail provided is that this is "vanilla GRPO" (Appendix C.1), with no specification of learning rate, group size, advantage normalization, or clipping. The quantitative advantage comparison in Figure 5 (left) shows that PPO assigns more negative advantages to repetitive tokens than GRPO would on the same PPO-generated trajectories, but this is not evidence that GRPO would necessarily fail if it were generating its own trajectories under a properly tuned configuration.

Mitigation status. Partially mitigated by appeal to community experience. The paper invokes "many community implementations" and references OpenR1 discussions about GRPO reproduction difficulties (Section 2.2, footnote) as corroborating evidence. However, this appeal shifts the burden of proof to community anecdotes rather than controlled experiments. The paper does not conduct a systematic GRPO hyperparameter study, does not test whether GRPO with a lower learning rate or different batch configuration avoids collapse, and does not acknowledge the possibility that GRPO might be stabilizable.

The Generalization Results Are Evaluated on a Single Checkpoint Without Ablation of the Mechanism

The assumption or constraint. Table 2 reports that ORZ-32B improves on MMLU (83.3% → 84.9%) and MMLU_PRO (55.1% → 74.4%) relative to the Qwen2.5-32B base model, despite being trained exclusively on math and logic reasoning tasks with no instruction tuning. The paper presents this as evidence that "pure scaled-up RL training on reasoning-oriented tasks" produces generalizable reasoning skills (Section 3.4). However, the generalization evaluation is conducted on a single final checkpoint, with no analysis of when during training the generalization improvements emerge, whether they correlate with specific benchmarks (e.g., does MMLU_PRO improvement appear simultaneously with MATH500 improvement, or later?), or what mechanism transfers reasoning skills to general knowledge questions.

The consequence. The generalization result could be explained by factors other than transferable reasoning skills. The Qwen2.5-32B base model already achieves 83.3% on MMLU—near the instruction-tuned variant's 83.2%—suggesting that MMLU performance is largely determined by pretraining knowledge rather than reasoning format. The +1.6 point improvement on MMLU could be noise, a consequence of the model learning to format answers in a way that better matches MMLU's multiple-choice grading, or an effect of the RL training acting as a regularizer that incidentally preserves or slightly improves factual knowledge. The MMLU_PRO improvement (+19.3 points) is too large to be noise, but the mechanism is opaque: is the model genuinely reasoning better about MMLU_PRO questions, or has it learned to apply math-style structured thinking (breaking down problems, checking work) to any formatted question, regardless of domain? Without trajectory-level analysis of the model's MMLU_PRO responses (e.g., does it use thinking tags? Does it show reflection patterns?), the claim that "pure scaled-up RL training" causes genuine generalization of reasoning capabilities remains an interpretation, not an established fact. A practitioner hoping that ORZ training will improve their model's general reasoning capabilities cannot predict which capabilities will transfer or how much improvement to expect.

What evidence exists in the paper. Table 2 reports point estimates for MMLU and MMLU_PRO on the final ORZ-32B checkpoint and compares to base and instruct baselines, plus DAPO. There is no learning curve for MMLU or MMLU_PRO during training (in contrast to the reasoning benchmarks in Figure 1), no analysis of generated responses on these benchmarks, and no ablation testing whether the generalization benefit comes specifically from the RL training or from some other aspect of the pipeline (e.g., the prompt template, which is designed to elicit structured thinking and might independently improve performance on formatted multiple-choice questions). The paper does not evaluate ORZ-7B's generalization, which would provide a second data point on whether the effect scales with model size or training intensity.

Mitigation status. Not acknowledged as an open question. The paper presents the generalization results as a positive finding without discussing alternative explanations or the limitations of single-checkpoint evaluation. The future work section (Section 5) mentions "Scenario Scaling" as a direction for "generalizing reasoning capabilities to increasingly diverse tasks," which implicitly acknowledges that current generalization is limited, but does not interrogate the mechanism of the generalization already observed.

The Training Data, While Open-Sourced, Is Not Characterized for Contamination Relative to Evaluation Benchmarks

The assumption or constraint. The ORZ 57k training dataset includes AIME problems "up to 2023" (Section 2.3), MATH problems [10], and content from Numina-Math, Tulu3 MATH, OpenR1-Math-220k, and AoPS forum posts. The evaluation benchmarks include AIME 2024, AIME 2025, MATH500, and GPQA Diamond. The paper states that AIME problems through 2023 are included in training and that AIME 2024 and 2025 serve as held-out evaluation, but it provides no systematic analysis of overlap between training data and evaluation benchmarks. For MATH500 (a subset of the MATH dataset) and GPQA Diamond, the degree of training-evaluation overlap is not discussed.

The consequence. Three contamination risks exist. First, the MATH dataset [10] was used as a training data source and MATH500 is a subset of MATH—if any MATH500 problems appear in the training data (even reformulated or with different numbers), the 92.2% MATH500 result partially reflects memorization rather than reasoning. The paper does not state whether MATH500 problems were explicitly excluded from the ORZ 57k training set. Second, GPQA Diamond is a relatively small benchmark (198 questions in the Diamond subset) and its questions may have appeared in some of the aggregated training sources (OpenR1-Math-220k, AoPS forum), since GPQA is a publicly available dataset and these sources aggregate publicly available problems. Third, the synthetic general reasoning tasks generated programmatically (Section 2.3) are not described in sufficient detail to assess whether they might inadvertently resemble evaluation benchmark questions. A practitioner evaluating ORZ's reasoning capabilities needs to know whether the impressive benchmark numbers reflect genuine reasoning ability or partially reflect training on questions similar to those in the test set. This is particularly important for MATH500, where ORZ's 92.2% is the strongest reported result relative to baselines.

What evidence exists in the paper. The paper explicitly notes that AIME data was included only "up to 2023" and that AIME 2024 and 2025 evaluate generalization to unseen competition years. For MATH500 and GPQA Diamond, no contamination analysis is provided. The data curation section (Section 2.3) describes the sources but does not describe a decontamination process against standard benchmarks. The paper does not report any overlap statistics, n-gram similarity analyses, or canonical contamination checks. The fact that ORZ-32B substantially outperforms DAPO on MATH500 (92.2% vs. 71.8%) and GPQA Diamond (55.5% vs. 16.0%) could indicate better reasoning or could partially reflect differences in training data coverage of these benchmarks.

Mitigation status. Partially mitigated for AIME by the temporal split (training on AIME through 2023, evaluating on 2024 and 2025), which provides a clean test of genuine generalization to unseen competition problems. On AIME 2024, ORZ-32B achieves 48.1%, comparable to DeepSeek-R1-Zero-Qwen-32B's 47.0% and DAPO's 48.3%—the temporal split suggests that AIME 2024 performance is not substantially inflated by contamination relative to these baselines. However, for MATH500 and GPQA Diamond, no such split exists and no decontamination analysis is performed. The paper does not acknowledge contamination risk as a limitation or call for future work on benchmark decontamination in reasoning RL training data.

7. Implications and Future Directions

How This Work Changes the Landscape

Open-Reasoner-Zero does not introduce a fundamentally new algorithm, but it shifts the field's understanding of what matters for stable reasoning-oriented RL in a way that will redirect research effort and simplify the design space. The conceptual shift has four components.

First, a reframing of the learned critic's role: from variance reducer to failure-mode detector. Before ORZ, the standard justification for learning a value function in RL was statistical—bootstrapping from learned values reduces the variance of Monte Carlo advantage estimates. This framing, inherited from continuous control and game-playing domains, made GRPO seem like a reasonable simplification: if the critic is just a variance reduction tool, replacing it with group-based normalization is an acceptable trade of statistical precision for implementation simplicity. ORZ demonstrates that this framing is wrong for the reasoning domain. The critic's primary value is not reducing variance but performing diagnostic credit assignment—identifying and penalizing specific degenerate behaviors (repetitive loops) that group-based estimators reinforce. Figure 5 provides the smoking gun: the critic assigns progressively lower values to repetitive tokens (right panel), and PPO's advantage estimates for those tokens are consistently more negative than GRPO's would be (left panel). This is not variance reduction—it is anomaly detection. The implication is that any algorithm for reasoning RL must be evaluated not just on final accuracy but on whether its credit assignment mechanism correctly penalizes known failure modes. Researchers designing new algorithms should ask: "Does my advantage estimator assign negative values to repetition, even in trajectories that end with correct answers?" ORZ provides both the diagnostic methodology (comparative advantage analysis on repetitive tokens, value function visualization, the stability monitoring triplet of reward/truncation/repeat) and the baseline answer (PPO passes; GRPO fails).

Second, a challenge to the RLHF orthodoxy: KL regularization is not just unnecessary for reasoning but counterproductive. The paper's ablation (Figure 3, mid) shows that adding KL regularization—the de facto standard in instruction-tuning RLHF since InstructGPT—slows training and reduces final performance. This is a reversal of the conventional wisdom that KL constraints are essential for preventing policy collapse. The mechanism is intuitive in retrospect: for reasoning tasks with verifiable binary rewards, there is no learned reward model to exploit, so the primary motivation for KL (preventing reward hacking) does not apply. Worse, KL regularization constrains the policy to stay near the base model, which is exactly the distribution the training is trying to escape—the base model does not produce structured chain-of-thought, self-reflection, or multi-step verification. After ORZ, researchers building reasoning RL systems can confidently omit KL regularization and its associated hyperparameter tuning burden, memory overhead (no reference model needed), and implementation complexity. This simplifies the design space substantially: the paper identifies five components (PPO with learned critic, GAE λ=1 γ=1, no KL, binary rewards, diverse data) as a sufficient recipe, and three of those components represent removing things that were previously considered essential.

Third, a reconciliation of conflicting community experiences with GRPO. Before ORZ, the open-source RL community faced a confusing situation: DeepSeek-R1-Zero reported strong results using GRPO, but community attempts to replicate it encountered "sudden degeneration" and "repetition collapse." The paper provides the mechanistic explanation that reconciles these experiences: GRPO's group-based advantage normalization assigns the same scalar advantage to every token in a response, which means repetitive tokens in correct trajectories get positive reinforcement. This creates a feedback loop: repetition gets reinforced → model repeats more → repetition becomes more common in correct trajectories → reinforcement strengthens → collapse. The collapse documented in Figure 7 (reward crashes, truncation rate and repeat score spike to 1.0 around step 240) is not a random failure or an implementation bug—it is a predictable consequence of GRPO's credit assignment mechanism when applied to reasoning tasks where repetition is a common failure mode. This explanation converts an anecdotal, mysterious failure into a diagnosed, avoidable one, and establishes that the choice between PPO and GRPO is not a matter of minor implementation preference but of whether the training will remain stable at scale.

Fourth, a reorientation of the scaling agenda from algorithms to data. The data ablation (Figure 3, right) shows that training on MATH 7.5k plateaus early while ORZ 57k sustains improvement throughout training. This is not the standard "more data is better" observation—it reveals a specific mechanism: the learning signal in Reasoner-Zero training comes from problems at the right difficulty level where the model sometimes succeeds and sometimes fails. Small datasets quickly exhaust this signal as the model masters solvable problems and the remaining ones are too hard. Large, diverse datasets provide a continuous supply of problems at productive difficulty levels. This finding redirects research effort: rather than developing more sophisticated algorithms (the paper shows that vanilla PPO is sufficient and that lookahead-style search is unnecessary), the highest-leverage way to improve reasoning RL is to scale training data quantity and diversity. The paper's open-source release of ORZ 57k explicitly aims to "encourage the research community to contribute and share more training data" (Section 5), framing data curation as a community effort rather than a competitive advantage. This is a strategic bet: that the next order-of-magnitude improvements will come from data scaling (quantity, quality, diversity, domain coverage), not from algorithmic innovations that add complexity to the training recipe.

What becomes more attractive after ORZ. Research on learned value functions for language model RL—understanding what critics learn, how they represent task structure, and whether they can be used for test-time guidance or interpretability—becomes more attractive because ORZ demonstrates that the critic is doing something mechanically important (identifying degenerate patterns), not just statistically helpful. Research on data curation for reasoning RL—how to measure problem difficulty, how to balance diversity and difficulty, how to generate synthetic reasoning data—becomes central. Research on scaling model size in the ORZ paradigm becomes natural: the paper shows the recipe works from 0.5B to 32B, and extending to 70B, 405B, or mixture-of-experts architectures is a straightforward scaling experiment.

What becomes less attractive. Research on GRPO stabilization techniques—trying to patch GRPO with repetition penalties, adaptive group sizes, or hybrid advantage estimators—becomes less attractive because ORZ provides evidence that the fundamental problem (lack of token-level credit assignment) cannot be fixed without a learned value function. Research on complex reward engineering for reasoning (format rewards, language consistency rewards, step-level process rewards) becomes less urgent because ORZ shows that binary outcome rewards are sufficient and that adding reward components creates surfaces for reward hacking without providing necessary signal. Research on KL regularization tuning for reasoning RL becomes largely moot.

Follow-Up Research This Work Enables

Cross-architecture replication to establish whether ORZ's findings are Qwen-specific. The paper's exclusive use of Qwen2.5 models leaves open the question of whether the recipe transfers to other architectures (Llama, Mistral, Gemma) or whether Qwen2.5's specific pretraining characteristics—high initial format adherence (Figure 4, left), particular repetition patterns, tokenizer properties—make it unusually amenable to the ORZ recipe. A strong follow-up would replicate the ORZ-7B training on Llama-3-8B base and Mistral-7B base using the same ORZ 57k dataset, the same hyperparameters (learning rates, batch sizes, GAE configuration), and the same evaluation protocol (16 responses averaged on AIME2024, MATH500, GPQA Diamond). The key measurements: (a) does training remain stable (no repetition collapse, steady reward growth, growing response length) on each architecture; (b) does the critic learn to penalize repetition (replicate Figure 5 on each architecture); (c) do benchmark accuracies reach comparable levels relative to each architecture's base capability; (d) does format adherence emerge without explicit format rewards on architectures with different pretraining mixtures. A negative result—instability or plateauing on non-Qwen architectures—would reveal that the ORZ recipe depends on architectural properties not yet identified, and would motivate investigation into which specific pretraining characteristics enable or prevent stable reasoning RL.

Scaling the ORZ recipe to much larger models to test whether stability is scale-invariant. The paper demonstrates stability at 0.5B through 32B, but the ORZ-32B response length curve exhibits substantial fluctuations (oscillating between ~6k and ~10k tokens in Figure 2) that the ORZ-7B curve does not. This raises the question of whether the fluctuations amplify at larger scales, potentially crossing a threshold where they trigger truncation or repetition issues. A natural extension is to apply the ORZ recipe to Qwen2.5-72B or Llama-3-70B, monitoring not just final benchmark performance but the same stability diagnostics the paper introduces: value function behavior on repetitive patterns (replicate Figure 5), the reward/truncation/repeat stability triplet (replicate Figure 7 but for PPO at scale), and response length volatility as a function of model size. The experiment would distinguish between two hypotheses: (a) the fluctuations are benign at all scales because the critic's credit assignment continues to suppress degenerate behaviors regardless of length variance; or (b) there is a critical model size or training duration beyond which even PPO's critic cannot prevent collapse, defining a fundamental limit for Reasoner-Zero training that would redirect research toward understanding and mitigating that limit.

Ablating the critic's contribution by training with a degraded value function. The paper's central mechanistic claim—that the critic specifically prevents collapse by detecting and penalizing repetition—rests on correlational evidence (Figures 5 and 7) but does not isolate the causal role of repetition detection. A targeted ablation would train ORZ-7B under three critic conditions: (a) the standard critic; (b) a critic trained on trajectories where all repetitive spans have been artificially truncated (replace repetition with an EOS token during value training), which should eliminate the critic's ability to learn that repetition predicts failure; (c) a critic that receives the same loss but is deliberately handicapped by limiting its context window to a small number of preceding tokens, preventing it from detecting long-range repetition patterns. If condition (b) or (c) leads to training collapse similar to GRPO's (Figure 7), this would establish that repetition detection—not just any learned value function—is the necessary mechanism for stability. If training remains stable even with a degraded critic, it would suggest the benefit of PPO over GRPO comes from some other property (e.g., the functional form of the advantage, the supervised learning signal for the critic, the separation of policy and value networks), requiring a revision of the paper's mechanistic explanation.

Difficulty-conditioned data sampling to improve training data efficiency. The paper identifies that the productive learning signal comes from problems at intermediate difficulty—problems where the model's success rate is neither 0% nor 100% (Section 2.3, discussion of "extreme pass rates"). The annealing stage takes a step toward exploiting this by selecting the hardest 13k prompts for a final training phase. A more systematic experiment would replace uniform random sampling from the dataset with a difficulty-adaptive curriculum: maintain a running estimate of each problem's pass rate (based on the most recent N training iterations), and sample problems with probability proportional to some function of that pass rate (e.g., maximum variance at pass rate ~0.5, or a U-shaped curriculum that shifts from easy to hard). The experiment would compare curriculum-based sampling against uniform sampling on ORZ-7B training, measuring: (a) sample efficiency (reward and benchmark accuracy vs. total trajectories processed); (b) final performance ceiling; (c) whether curriculum sampling allows faster learning with smaller datasets (addressing the data scaling bottleneck). This would test whether the primary value of the ORZ 57k dataset is its size or its diversity of difficulty, and whether a smaller but carefully difficulty-curated dataset can match or approach the performance of uniform sampling from a larger dataset.

Combining PPO's critic with test-time compute: using the value function for adaptive generation. The paper demonstrates that the critic learns to predict, at any token position, whether the current trajectory is likely to succeed (value target = probability of correct final answer). This learned value function could be used at test time to guide generation: rather than generating 64 responses and selecting by terminal reward (which requires completing every trajectory), use the critic to prune unpromising partial trajectories early, allocating the generation budget to the most promising paths. This is essentially beam search or best-of-N with early stopping, but using a learned value function that was trained adversarially against the policy's failure modes (repetition, incoherence) rather than a separate verifier trained on static data. A concrete experiment: on AIME2024, compare standard parallel sampling (64 independent completions, select by terminal reward) against critic-guided generation (generate 64 responses but stop any that drop below a value threshold mid-generation, reallocating the compute budget to new trajectories). The metric would be accuracy vs. total tokens generated, testing whether the critic-learned credit assignment transfers to efficient test-time compute allocation. This connects the paper's training-time stability mechanism to the test-time compute scaling agenda the paper identifies as future work (Section 5).

Applying ORZ training to code generation with unit-test-based rewards. The paper restricts to math and logic problems with extractable answers because the binary reward function requires clean string matching. Code generation presents a natural extension: unit tests provide the same binary reward signal (all tests pass → 1; any test fails → 0) without requiring answer extraction heuristics. The key question is whether the emergent behaviors documented for math reasoning—self-reflection ("wait," "recheck," "retry"), length scaling, format adherence without explicit rewards—transfer to code. A concrete experiment: apply ORZ training to a base code model (e.g., Qwen2.5-Coder-7B base or DeepSeek-Coder-7B base) using a dataset of programming problems with unit tests (e.g., filtered Codeforces problems, MBPP, or APPS with verified test suites). Measure whether: (a) response length grows during training (do models learn to write more extensive code with comments, tests, and error handling?); (b) reflection-like patterns emerge (do models learn to read their own code and identify bugs before finalizing?); (c) the critic learns to identify code-specific degenerate patterns (infinite loops, syntax errors that would cause test failure) and penalize them. This would test whether the ORZ recipe generalizes beyond mathematical reasoning to a second domain with verifiable rewards, and whether the specific emergent behaviors are math-specific or reflect a more general "reasoning under verifiable outcome" capability.

Practical Applications and Downstream Use Cases

Training competitive reasoning models at substantially reduced computational cost. The paper's headline efficiency result—ORZ-32B reaches 48.1% AIME2024 and 92.2% MATH500 in ~1,000 training steps, matching or exceeding DeepSeek-R1-Zero-Qwen-32B's performance at ~10,000 steps—means that a small team with access to a modest GPU cluster can train a competitive reasoning model from a base checkpoint in days rather than weeks. The specific recipe eliminates expensive components: no SFT or distillation stage before RL (training starts directly from the base model), no KL regularization (no reference model to load, no KL coefficient to tune, saving ~50% memory for the policy component), no format rewards (no reward engineering), and no process reward model (no PRM training). Using the paper's reported configuration—128 prompts per iteration, 64 responses per prompt, 8,192 total trajectories—a team with 8× A100 GPUs could complete ORZ-7B training in under a day (estimated from the ~600-step curves in Figure 1) and ORZ-32B in under a week. The open-source release of code, training data, hyperparameters, and model weights means there is no hidden engineering required for reproduction. This dramatically lowers the barrier to entry for research on reasoning-oriented RL, enabling academic labs and smaller companies to participate in what was previously a resource-intensive endeavor dominated by a handful of industrial labs.

Bootstrapping reasoning data generation for self-improvement pipelines. The paper's demonstration that ORZ training applied to an already-distilled reasoning model (ORZ-R1-Distill-Qwen-14B, Table 3) yields further gains—improving from 69.7% to 75.2% on AIME2024 and surpassing the larger DeepSeek-R1-Distill-Qwen-32B on three of four benchmarks—provides a template for iterative self-improvement. The workflow: use an existing reasoning model (or an ORZ-trained model) to generate high-quality reasoning trajectories on unlabeled problems, filter for correctness using verifiable rewards (unit tests, math answer checking, execution results), and use these trajectories as training data for a subsequent round of ORZ training. The paper's key insight—that the critic provides robust credit assignment even on trajectories with mixed quality (correct answers with repetitive segments)—means that the generated data does not need to be perfectly clean; the critic will learn to distinguish productive reasoning from degenerate patterns in the training data itself. The 300-iteration distillation training on 13k difficult prompts (Section 3.4, Appendix B) provides a concrete budget: starting from a distilled model, a few hundred iterations on a targeted dataset of hard problems is sufficient for meaningful gains. This makes iterative self-improvement feasible without access to human-annotated reasoning data or larger teacher models.

Deploying reasoning capabilities in resource-constrained settings through small-model RL training. The paper's scaling experiments (Table 4, Figure 6) show that the ORZ recipe produces meaningful reasoning improvements even at small model sizes: ORZ-1.5B reaches 58.0% on MATH500 (up from whatever the Qwen2.5-1.5B base achieves) and ORZ-7B reaches 81.4% on MATH500 and 36.6% on GPQA Diamond. While these numbers do not compete with frontier models, they demonstrate that on-device or edge-deployed models can acquire structured reasoning capabilities through RL training alone, without distillation from larger models. A concrete deployment scenario: a mobile math tutoring application that runs a 1.5B ORZ-trained model locally on a phone, generating step-by-step solutions with self-verification on high-school math problems (MATH500 covers competition-level high-school math), without sending data to a cloud API. The model would have been trained on the ORZ 57k dataset using the paper's recipe, producing structured thinking... response outputs that the application can render as an interactive reasoning display. The absence of KL regularization during training means the model has genuinely learned new behaviors (reasoning structures, reflection) rather than staying close to the base distribution, making the small model's capabilities qualitatively different from simply prompting a small base model with few-shot examples. The paper's release of ORZ-0.5B, ORZ-1.5B, and ORZ-7B weights means this is immediately deployable today.

Improving general-purpose LLM benchmarks through reasoning-specific RL without instruction tuning. Table 2 shows that ORZ-32B—trained exclusively on math and logic problems with binary outcome rewards—outperforms Qwen2.5-32B-Instruct on MMLU (84.9% vs. 83.2%) and substantially outperforms it on MMLU_PRO (74.4% vs. 69.2%), despite having no instruction tuning, no human preference data, and no training on general knowledge tasks. This is a practically significant finding for organizations that maintain both instruction-tuned and reasoning-specialized models: ORZ training on a reasoning dataset can produce a single model that is competitive on general benchmarks while also possessing specialized reasoning capabilities, potentially eliminating the need to maintain separate model variants for different use cases. The improvement on MMLU_PRO (+19.3 points over the base model) is large enough to matter for benchmark leaderboards. A deployment scenario: a company serving both general-purpose chat (measured by MMLU/MMLU_PRO) and STEM tutoring (measured by MATH/AIME/GPQA) could train one ORZ model rather than maintaining an instruction-tuned model for chat and a separate reasoning model for STEM, simplifying deployment infrastructure and reducing serving costs. The paper does not guarantee this transfer—MMLU_PRO improvement might not replicate across all instruction-tuned base models or all training datasets—but the result is striking enough to warrant testing in any organization with both general and reasoning deployment needs.