Understanding R1-Zero-Like Training: A Critical Perspective

ArXiv: 2503.20783

🎯 Pitch

The 'Aha moment' in reasoning models isn't learned through RL—it already exists in the base model. By removing a hidden length bias in GRPO, the authors slash wasted token generation while achieving state-of-the-art math reasoning with a 7B model trained in just 27 hours.

1. Executive Summary

This paper critically analyzes the R1-Zero-like training paradigm by examining its two core components—base models and RL algorithms—using Qwen2.5, Llama-3.1, and DeepSeek-V3-Base models on the MATH benchmark with evaluation extending to AIME 2024, AMC, MATH500, Minerva Math, and OlympiadBench. The work identifies an optimization bias in Group Relative Policy Optimization, or GRPO (response-level length normalization that artificially inflates incorrect-response length, and question-level standard deviation normalization that overweights easy or hard questions), and proposes Dr. GRPO (removing both the per-response length division and per-question standard deviation normalization to recover an unbiased PPO objective), which reduces incorrect response length by ~1.4k tokens while preserving reasoning performance. The paper also reveals that Qwen2.5 base models achieve peak performance without any prompt template—a ~60% improvement over 4-shot prompting—and that self-reflection behaviors already appear in base models including DeepSeek-V3-Base prior to RL, establishing that the "Aha moment" is a pretraining artifact rather than an emergent RL phenomenon, and that RL gains are most pronounced only when there is a substantial model-template mismatch that RL must reconstruct.

2. Context and Motivation

The Core Problem: We Don't Understand Why R1-Zero-Style RL Training Works

The deep learning community was galvanized by DeepSeek-R1-Zero (Guo et al., 2025), which demonstrated a striking result: applying pure reinforcement learning directly to a base language model—with no supervised fine-tuning as an intermediate step—could substantially enhance mathematical reasoning capabilities. The model's performance on competition math benchmarks improved dramatically during RL training, accompanied by a characteristic "double-increase" phenomenon where both reward and response length grew together throughout training. Most dramatically, the model appeared to spontaneously develop sophisticated reasoning behaviors like self-reflection—the celebrated "Aha moment"—where the model would pause mid-solution, recognize an error, and backtrack to correct itself.

This result captured the community's imagination because it suggested a fundamentally simpler path to reasoning-capable LLMs: rather than the expensive pipeline of collecting human demonstrations, fine-tuning, reward modeling, and iterative refinement that characterized previous approaches, one might simply train a base model with RL using outcome-based rewards (e.g., whether the final answer is correct). The apparent emergence of complex reasoning strategies from a simple reward signal seemed to validate a minimal-intervention philosophy that resonated with the broader AI goal of learning complex behaviors from simple objectives.

However, the paper identifies a critical gap: the community has been racing to reproduce and extend DeepSeek-R1-Zero without adequately understanding why it works, which components are essential, and what observed phenomena are genuine versus artifactual. This gap matters because without mechanistic understanding, the field risks drawing incorrect conclusions about what RL contributes, which base models are suitable, and how to design training algorithms. The rapid proliferation of R1-Zero replications—SimpleRL-Zero (Zeng et al., 2025), Open-Reasoner-Zero (Hu et al., 2025), PRIME-Zero (Cui et al., 2025), TinyZero (Pan et al., 2025)—all report similar phenomena (reward increase, length increase, emergence of self-reflection), but they use different base models (predominantly Qwen2.5 variants), different RL algorithms, and different implementations. This creates a confounded landscape where the causes of observed behaviors cannot be isolated.

Why Understanding R1-Zero Matters: Scientific and Practical Stakes

The importance of understanding this training paradigm operates on multiple levels:

Scientific: What does RL actually contribute to reasoning? The most fundamental question is whether the improvements and behaviors observed during R1-Zero-like training represent genuine emergence of reasoning strategies through reinforcement learning, or whether they represent the amplification of capabilities already latent in the base model. If base models already possess reasoning abilities and self-reflection patterns before any RL is applied—as this paper demonstrates—then the narrative of pure RL creating reasoning from scratch is misleading. The "Aha moment" would be better understood as a pretraining artifact that RL makes more visible (by increasing response length, giving the model more opportunities to exhibit existing behaviors), rather than a genuinely new capability. Distinguishing between these interpretations has profound implications for how we allocate resources between pretraining and post-training, and for our fundamental understanding of what RL contributes to language model capabilities.

Algorithmic: Are we optimizing what we think we're optimizing? The R1-Zero recipe uses Group Relative Policy Optimization (GRPO), a variant of PPO proposed by Shao et al. (2024) that estimates advantages using grouped reward normalization rather than a learned value function. This paper identifies that GRPO—and, surprisingly, many widely-used open-source PPO implementations—contains an optimization bias that systematically favors longer incorrect responses. If the celebrated "double-increase" phenomenon (reward and length increasing together) is partially an artifact of this bias rather than genuine improvement in reasoning quality, then the field has been misinterpreting a key empirical signature of this training paradigm. Understanding whether the length increase is a feature (emergence of long chain-of-thought reasoning) or a bug (optimization bias inflating response length) is crucial for designing the next generation of RL algorithms.

Practical: How should practitioners choose base models and templates? The R1-Zero replication literature has converged on Qwen2.5 base models as the starting point of choice (Zeng et al., 2025; Hu et al., 2025; Cui et al., 2025), but this paper's analysis reveals that Qwen2.5 models are unusual: they achieve ~60% higher benchmark accuracy when no template is applied compared to standard 4-shot prompting, and they already function as near-chat-completion models rather than pure sentence-completion base models. If Qwen2.5 models were pretrained on concatenated question-answer texts—as the paper hypothesizes based on their behavior—then using them to "reproduce" R1-Zero conflates the effects of RL with the effects of domain-specific pretraining. Practitioners building reasoning systems need to know whether the impressive gains reported by RL-only tuning recipes generalize to genuinely "pure" base models that haven't been exposed to instruction-formatted data during pretraining.

Economic: What is the role of question set coverage and template design? The paper shows that the interaction between prompt template and training question set can dramatically alter RL dynamics—with the R1 template requiring broad question coverage to achieve good performance, while the Qwen-Math template can achieve strong performance even when trained on out-of-distribution, simpler questions (GSM-8K). This has direct implications for the cost and data requirements of R1-Zero-style training, and suggests that many reported results may be sensitive to seemingly minor design choices that haven't been systematically varied.

Where Prior Approaches Fall Short

The paper identifies several specific limitations in prior work that motivate its systematic analysis:

1. Lack of base model scrutiny in R1-Zero replications. Nearly all prior R1-Zero replications default to Qwen2.5 base models without investigating whether these models have special properties that make them particularly amenable to RL-based reasoning improvement. Zeng et al. (2025) report that SimpleRL-Zero achieves strong math reasoning using only 8K MATH examples, but don't examine whether the Qwen2.5-Math-7B base model already possesses reasoning capabilities that RL merely surfaces. Hu et al. (2025) similarly use Qwen2.5 base models and report emergence of long chain-of-thought behaviors, but don't control for the base model's existing question-answering format. Liu et al. (2025b) raised the possibility that the "Aha moment" may not exist in open-source replications, but critically, they did not test DeepSeek-V3-Base—the actual base model from which the original R1-Zero was trained. This paper completes that missing piece by hosting DeepSeek-V3-Base-685B and demonstrating that even this model, which the authors describe as a "nearly pure base model," already generates self-reflection keywords like "Aha" and "wait" prior to any RL.

2. Confounding between genuine algorithmic improvement and implementation artifacts. The paper makes a crucial observation that extends beyond GRPO: even PPO, which is unbiased by mathematical formulation, is implemented with a length bias in nearly all major open-source RLHF frameworks. The authors examined TRL (von Werra et al., 2020), OpenRLHF (Hu et al., 2024), and VeRL (Sheng et al., 2024) and found that all of them normalize the policy loss by response length (either per-response or per-batch), which diverges from the mathematically correct PPO objective (Equation 2). This formulation-implementation misalignment predates GRPO—the authors speculate it originated from pretraining conventions (Shoeybi et al., 2019) where packing tokens into fixed-length contexts makes length normalization natural for numerical stability—but its effect on RL dynamics has gone unexamined. The consequence is that two phenomena widely interpreted as evidence of successful RL training—consistently increasing response length and the divergence of correct vs. incorrect response lengths—may be partially attributable to this implementation choice rather than to the RL algorithm's ability to incentivize deeper reasoning.

3. Conflation of RL gains with template restoration. The paper's finding that Qwen2.5-Math base models perform dramatically better without templates reveals a subtle but important confound in the interpretation of RL training curves. When a base model is RL-tuned with a template that mismatches its pretraining format (e.g., applying the R1 template to a Qwen2.5-Math model), the initial performance is artificially depressed because the template disrupts the model's native question-answering ability. RL then "improves" the policy, but much of this improvement may simply be the model learning to work around the template—recovering capabilities it already had—rather than acquiring genuinely new reasoning skills. The paper's template ablation (Figure 6) directly demonstrates this: under the Qwen-Math template or no template, the base model starts at much higher accuracy, and RL provides smaller absolute gains. This reframes the interpretation of RL contributions: the gains are largest precisely when the starting point is artificially low due to model-template mismatch.

4. Treatment of self-reflection as an emergent RL phenomenon. The original DeepSeek-R1-Zero paper described self-reflection behaviors as emerging during the course of RL training—the "Aha moment" where the model learns to allocate more test-time compute by reconsidering its approach. Prior work by Liu et al. (2025b) and Yeo et al. (2025) questioned whether this emergence is genuine in open-source replications, showing that Qwen2.5 base models already produce self-reflection keywords. However, they did not test DeepSeek-V3-Base, leaving open the possibility that the original R1-Zero result represented a genuine emergence on a truly "pure" base model. This paper closes that gap by directly testing DeepSeek-V3-Base-685B and finding it too generates self-reflection patterns. Moreover, the paper goes further by analyzing DeepSeek-R1-Zero itself and finding that self-reflection behaviors are not positively correlated with higher accuracy during inference—responses containing self-reflection are approximately equally likely to be correct or incorrect compared to responses without self-reflection. This challenges a key pillar of the R1-Zero narrative and suggests that self-reflection may be better understood as a stylistic pattern that correlates with longer responses (which RL encourages) rather than a causal mechanism for improved reasoning.

5. Insufficient analysis of question set × template interactions. Prior replication efforts differ substantially in their training data: SimpleRL-Zero uses 8K MATH questions, Open-Reasoner-Zero uses a broader 57K mixture, and the original DeepSeek-R1 used a large but unspecified dataset. The community has generally assumed that more and more diverse data is better for RL training (Luo et al., 2025; Hu et al., 2025), but this paper's template-dependent results complicate that picture. When using the Qwen-Math template, training on GSM-8K (a simpler dataset with grade-school math problems) yields the best final performance—even outperforming training on harder, more diverse data. This inverts the conventional wisdom and suggests that the optimal data strategy depends on the alignment between the template and the base model's pretraining format, an interaction that prior work had not systematically explored.

How This Paper Positions Itself

This paper positions itself as a critical re-examination rather than a novel method proposal. While it does introduce Dr. GRPO as a concrete algorithmic contribution, the broader framing is analytical: the goal is to "understand R1-Zero-like training" by decomposing it into base model properties and RL algorithm behavior, then systematically studying each component in isolation.

The analytical framework is organized around two axes that map to the training pipeline:

Axis 1: Base model analysis (Section 2). Rather than treating the base model as a black-box starting point for RL, the paper develops a set of diagnostic measurements that characterize base models along three dimensions: question-answering ability (does the model attempt to answer, or merely complete text?), exploration ability (can the model generate correct solutions at some non-trivial rate, as measured by pass@8?), and self-reflection tendencies (does the model already produce reflection keywords before RL?). These diagnostics are applied across model families (Qwen2.5, Llama-3.1, DeepSeek) to establish the range of base model behaviors and identify which properties correlate with downstream RL success.

Axis 2: RL algorithm analysis (Section 3). Rather than accepting GRPO/PPO as given and focusing on reward design or data scale, the paper scrutinizes the optimization objective itself. The approach is to mathematically decompose GRPO's objective function (Equation 3) to identify where biases enter—specifically, the response-length normalization term $1/|o_i|$ and the question-level standard deviation normalization $1/\text{std}(R)$—then experimentally validate that removing these terms (Dr. GRPO) produces different training dynamics while maintaining or improving final performance.

The paper's stance toward existing work is both integrative and corrective:

• Integrative: It incorporates findings from prior replication efforts (SimpleRL-Zero, Open-Reasoner-Zero, TinyZero) and synthesizes them into a unified framework where template choice, base model properties, and question set coverage are all recognized as interacting factors that determine RL outcomes. This moves beyond the pattern of individual papers reporting results on single configurations.

• Corrective: It directly challenges several claims that have become conventional wisdom in the R1-Zero literature: (1) that the "Aha moment" emerges through RL, (2) that increasing response length during training indicates genuine reasoning development, (3) that Qwen2.5 models are representative base models for studying R1-Zero-style training, and (4) that PPO implementations faithfully implement the intended objective. Each challenge is supported by targeted experiments and direct inspection of model outputs.

The paper also positions itself within the broader landscape of LLM RL research by connecting to the literature on outcome-based reward optimization (Lambert et al., 2024), policy gradient methods for language (Ahmadian et al., 2024), and the relationship between pretraining and post-training (Yang et al., 2024a;b). It draws explicit links to REINFORCE Leave-One-Out (RLOO; Kool et al., 2019; Ahmadian et al., 2024) by showing that Dr. GRPO's advantage estimate is equivalent to RLOO's up to a scaling factor (Appendix A), situating the proposed fix within a well-understood variance reduction framework.

Finally, the paper's minimalist recipe—Qwen2.5-Math-7B + Dr. GRPO + MATH level 3-5 questions + Qwen-Math template, achieving 43.3% on AIME 2024 with 27 hours on 8 A100s—is positioned as a demonstration that understanding (not just scaling) enables efficiency. The recipe is notable not for its complexity but for its simplicity: by choosing the right template (one that matches the base model's pretraining format), using an unbiased optimizer, and training on appropriately selected data, strong results emerge without the compute scale typically associated with R1-Zero replications. This is consistent with the paper's broader message that careful analysis of fundamentals can substitute for brute-force scaling.

3. Technical Approach

3.1 Reader Orientation

This paper is primarily an analytical investigation and algorithmic correction, not a proposal of a new model architecture. The "system" being studied is the R1-Zero training pipeline—a recipe that takes a base language model, applies reinforcement learning with outcome-based rewards on math problems, and produces a model with enhanced reasoning capabilities. The core idea is that by decomposing this pipeline into its two essential components (base models and the GRPO optimization algorithm) and subjecting each to rigorous scrutiny, the authors can separate genuine algorithmic contributions from artifacts of pretraining data and implementation biases, culminating in a corrected optimizer (Dr. GRPO) and a minimalist training recipe that achieves state-of-the-art performance.

3.2 Big-Picture Architecture (Diagram in Words)

The R1-Zero-like training pipeline under analysis has five major components:

1. Base LLM (Qwen2.5, Llama-3.1, or DeepSeek-V3-Base variants): The pretrained language model that serves as the initial policy $\pi_\theta(\cdot|q)$—the distribution from which responses to math questions are sampled. This model has undergone no supervised fine-tuning, only pretraining on general or math-domain corpora.

2. Prompt Template (R1 template, Qwen-Math template, or none): A text wrapper that formats the raw math question into a prompt the base model can interpret as requesting an answer rather than text completion. The choice of template dramatically affects initial policy behavior and RL dynamics.

3. RL Optimizer (GRPO or Dr. GRPO): The algorithm that updates the model parameters based on outcome rewards. It samples groups of responses per question, computes per-token advantage estimates without a learned value function, and applies clipped policy updates. Dr. GRPO is the proposed correction that removes two normalization biases present in the original GRPO formulation and in common open-source PPO implementations.

4. Reward Function (Math-Verify, rule-based): A deterministic verifier that checks whether the model's generated text contains the correct final answer to the question. Returns binary reward: 1 if the answer string matches, 0 otherwise. There is no learned reward model—only outcome verification.

5. Training Question Sets (MATH, GSM-8K, ASDiv, ORZ-57K): Collections of math problems of varying difficulty and domain coverage from which questions are sampled during online RL training. The interaction between question set properties and template choice determines training dynamics.

Information flow: A math question is sampled from the training set → the template wraps it into a prompt → the current policy generates G=8 complete responses (each up to 3,000 tokens) → Math-Verify assigns each response a binary reward based on final-answer correctness → the RL optimizer (GRPO or Dr. GRPO) computes token-level advantage estimates from these group rewards and constructs a clipped surrogate loss → model parameters are updated → process repeats from step 1 with updated policy.

3.3 Roadmap for the Deep Dive

• First, the base model analysis framework and its three diagnostic dimensions (question-answering ability, exploration ability via pass@8, self-reflection detection), which establishes what properties base models have before any RL is applied and why these properties matter for interpreting RL outcomes.

• Second, the formal RL objective and the MDP formulation of language generation, since this defines the optimization target that GRPO and Dr. GRPO both attempt to maximize—understanding the objective is prerequisite to understanding why the algorithmic bias matters.

• Third, the detailed breakdown of GRPO's objective function and the identification of its two biases (response-level length bias and question-level difficulty bias), including the mathematical analysis of where each bias enters and how they distort the effective advantage signal.

• Fourth, the proposed fix—Dr. GRPO—including the specific modifications to the loss computation, the connection to REINFORCE Leave-One-Out (RLOO) that establishes its unbiasedness, and the corrected masked-mean implementation.

• Fifth, the experimental infrastructure for analyzing base models and RL dynamics, including the diagnostic suite (template comparison, pass@8 sweeps, self-reflection detection via keyword+LLM cross-validation), the training hyperparameters, and the evaluation benchmarks.

3.4 Detailed, Sentence-Based Technical Breakdown

This is an analytical paper with an algorithmic correction whose core idea is that the observed phenomena in R1-Zero-like training (increasing response length, emergence of self-reflection) are partially attributable to (a) pretraining artifacts in commonly-used base models and (b) optimization biases in the GRPO objective and its PPO implementations, rather than being pure emergent properties of RL—and that correcting these biases yields better token efficiency without sacrificing reasoning performance.

Base Model Analysis Framework: Three Diagnostic Dimensions

The paper develops a systematic diagnostic protocol applied to six base models: Qwen2.5-Math-1.5B, Qwen2.5-Math-7B, Qwen2.5-7B, Llama-3.1-8B, DeepSeek-Math-7B, and DeepSeek-V3-Base-685B. Each model is probed along three dimensions designed to characterize its readiness for R1-Zero-like training and to reveal pretraining artifacts that might be mistaken for RL-driven emergence.

Dimension 1: Question-Answering Ability. The first diagnostic asks: when presented with a math problem, does the model attempt to answer it, or does it merely complete the text in a sentence-continuation pattern? This matters because RL requires the model to operate as a question-answering policy $\pi_\theta(\cdot|q)$—the policy must produce responses that contain an answer that can be verified against ground truth. If the model primarily generates continuations (e.g., repeating the question, adding commentary, asking follow-up questions), RL has no reward signal to optimize.

The measurement procedure works as follows. For each model, the authors sample responses to the same 500 MATH training questions under three template conditions:

Template 1 (R1 template):

A conversation between User and Assistant. The User asks a question, and the Assistant solves it. 
The Assistant first thinks about the reasoning process in the mind and then provides the User 
with the answer. The reasoning process is enclosed within  thinking  response and answer is 
enclosed within <answer> </answer> tags, respectively, i.e.,  thinking reasoning process here 
 response <answer> answer here </answer>.
User: {question}
Assistant:  thinking

This is the template from the original DeepSeek-R1-Zero work (Guo et al., 2025). It explicitly instructs the model to separate reasoning from final answer and provides a conversational framing.

Template 2 (Qwen-Math template):

<|im_start|>system
Please reason step by step, and put your final answer within \boxed{}.
<|im_end|>
<|im_start|>user
{question}
<|im_end|>
<|im_start|>assistant

This uses the Qwen chat format tokens and provides a simpler instruction: reason step-by-step and box the final answer. It comes from Zeng et al. (2025).

Template 3 (No template):

{question}

The raw question text with no additional formatting, instructions, or special tokens.

After generating responses, GPT-4o-mini is used as a judge to classify each response as either answering the question (1) or exhibiting sentence-completion patterns (0), regardless of answer quality. The answering rate is the percentage of responses classified as 1.

The results (Figure 3, left panel) reveal a striking pattern: Llama-3.1-8B and all DeepSeek models have low answering rates without templates (~20-30%) but improve substantially with the appropriate template (R1 template, reaching ~80-100%). In contrast, all Qwen2.5 models achieve ~100% answering rate with no template—they natively operate as question-answering systems even when given raw text. This immediately suggests that Qwen2.5 base models have been exposed to instruction-formatted data or concatenated question-answer pairs during pretraining, a hypothesis the paper investigates further in Section 2.2. DeepSeek-V3-Base shows the lowest no-template answering rate, marking it as a "nearly pure base model" in the authors' assessment.

Dimension 2: Exploration Ability (pass@8). The second diagnostic asks: can the base model generate any correct solutions when sampling multiple responses, even at elevated temperatures that encourage diversity? This is measured by pass@8 accuracy—the fraction of questions for which at least one of eight sampled responses contains the correct final answer. This metric is critical because RL can only improve a policy if reward signals exist: if pass@8 is zero, the model never produces correct answers, and RL receives no positive feedback to amplify.

The measurement sweeps sampling temperature from 0.1 to 1.0, with higher temperatures generally improving exploration by increasing response diversity (the standard exploration-exploitation tradeoff). Each model uses its best template as determined in the first diagnostic dimension.

The results (Figure 3, middle panel) show that all tested models achieve non-trivial pass@8 rates, indicating they are "ready for RL." However, the magnitude varies dramatically: Qwen2.5 models (especially Qwen2.5-Math-7B and Qwen2.5-7B) achieve pass@8 rates of approximately 70-80% at higher temperatures, surpassing even DeepSeek-V3-Base-685B. This suggests that Qwen2.5 base models already have strong math-solving capabilities before any RL is applied. The paper notes that this may partially explain why most R1-Zero replications use Qwen2.5 models—they start from a higher baseline, making it easier to demonstrate improvement.

Dimension 3: Self-Reflection Detection. The third diagnostic asks: does the base model already produce self-reflection behaviors (keywords like "rethink," "reevaluate," "wait," "let me check again") prior to RL training? This directly tests the claim that the "Aha moment" emerges through RL.

The detection procedure combines two complementary methods and cross-validates them. Keyword-based detection maintains a manually curated pool of 15 self-reflection keywords: recheck, rethink, reassess, reevaluate, re-evaluate, reevaluation, re-examine, reexamine, reconsider, reanalyze, double-check, check again, think again, verify again, go over the steps. A response is flagged if it contains any of these strings. However, the authors note that naive keyword matching produces false positives—e.g., a response containing "wait" in a non-reflective context (such as "wait for the result")—so they maintain a small, highly selective pool to reduce noise.

LLM-based detection uses GPT-4o-mini with a detailed prompt (reproduced in Appendix H) that classifies responses into four categories: 0 (off-topic/nonsensical), 1 (direct answering without self-reflection), 2 (exhibits self-reflection at any level, including implicit patterns without explicit keywords), 3 (pure Python code without self-reflection). This catches implicit self-reflection that keywords miss but can itself produce false positives on lengthy, complex responses.

Cross-validation combines the two: a response is considered to contain genuine self-reflection only if both detectors agree. This filters out keyword false positives (where a word appears incidentally) and LLM false positives (where the judge over-interprets complex text). The analysis is conducted at the question level across 500 MATH questions, with 8 responses per question at each of 10 temperature settings, giving 40,000 total responses per model. A question is marked as exhibiting self-reflection if at least one of its eight responses at a given temperature is flagged.

The results (Figure 3, right panel; Figure 12 in Appendix D) show that all base models, including DeepSeek-V3-Base-685B, already generate self-reflection keywords prior to RL. DeepSeek-V3-Base produces a "decent amount of self-reflections" (Section 2.3), including explicit "Aha" and "wait" keywords (examples in Figure 13, Appendix E). Different model families favor different keywords: Qwen2.5 models frequently use "check again," "double-check," and "recheck"; DeepSeek models avoid "re-evaluate" and "verify again" but use other terms; Llama models favor "think again." The paper hypothesizes this reflects differences in pretraining data composition.

With these three diagnostics in hand, the paper establishes that the ingredients commonly attributed to RL-driven emergence—question-answering format, math-solving capability, and self-reflection patterns—are already present in base models to varying degrees. This reframes the analysis: the question is not "what does RL create from nothing?" but rather "how does RL interact with and amplify what is already latent in pretrained models?"

Formal RL Objective and MDP Formulation

The paper formulates language generation during RL training as a token-level Markov Decision Process (MDP). This formulation is standard in the RLHF literature but is crucial for understanding where the optimization bias originates.

The MDP $\mathcal{M} = (\mathcal{S}, \mathcal{A}, r, p_{\mathcal{Q}})$ is defined as follows. The state $s_t \in \mathcal{S}$ at generation step $t$ is the concatenation of the input question $q$ and all tokens generated so far: $s_t = [q_1, \ldots, q_M, o_1, \ldots, o_{t-1}]$. The action space $\mathcal{A}$ is the vocabulary—at each step, the policy $\pi_\theta$ selects a next token $o_t$ from the vocabulary. The transition is deterministic: selecting token $o_t$ from state $s_t$ leads to state $s_{t+1} = [s_t, o_t]$. An episode begins by sampling a question $q \sim p_{\mathcal{Q}}$ from the question distribution, and ends when the policy generates the end-of-sequence token or reaches a maximum generation budget (3,000 tokens in these experiments).

The optimization objective is to maximize expected return without KL regularization:

$J(\pi_\theta) = \mathbb{E}_{q \sim p_{\mathcal{Q}}} \left[ \mathbb{E}_{o \sim \pi_\theta(\cdot|q)} [R(q, o)] \right]$

where $R(q, o) = \sum_{t=1}^{|o|} r(s_t, o_t)$ is the cumulative return (sum of token-level rewards) for the complete response $o$ given question $q$, and $\pi_\theta(\cdot|q)$ is the autoregressive policy that produces response $o$ token by token.

What it computes: the expected total reward when the model generates responses to questions drawn from the training distribution. The outer expectation averages over the question distribution; the inner expectation averages over responses generated by the current policy. The return $R(q, o)$ is the sum of all token-level rewards along the trajectory.

Why this form: the standard RL formalism from Sutton and Barto (2018), adapted to language generation. Notably, this objective omits the KL divergence term $\beta \text{D}_{\text{KL}}[\pi_\theta \Vert \pi_{\text{ref}}]$ that is standard in RLHF (Christiano et al., 2017). The paper argues this omission is justified for reasoning because: (1) the reward function is a rule-based verifier that is equally reliable regardless of how far the policy deviates from its initial distribution—unlike a learned reward model, which is only accurate near the distribution of data it was trained on (Jaques et al., 2019; Stiennon et al., 2020); (2) removing the KL term "not only saves the memory and computation required by $\pi_{\text{ref}}$ during training, but also potentially leads to better performance for R1-Zero-like training" (citation to Hu et al., 2025). Throughout the paper, $\beta = 0$ is assumed.

GRPO Objective: Formulation and Bias Identification

Group Relative Policy Optimization (GRPO), proposed by Shao et al. (2024), is a variant of Proximal Policy Optimization (PPO) (Schulman et al., 2017b) designed to eliminate the need for a learned value function $V_\phi$ when estimating advantages. The standard PPO surrogate objective is:

$J_{\text{PPO}}(\pi_\theta) = \mathbb{E}_{q \sim p_{\mathcal{Q}}, o \sim \pi_{\theta_{\text{old}}}(\cdot|q)} \left[ \sum_{t=1}^{|o|} \min\left( \frac{\pi_\theta(o_t|q, o_{<t})}{\pi_{\theta_{\text{old}}}(o_t|q, o_{<t})} \hat{A}_t, \text{clip}\left( \frac{\pi_\theta(o_t|q, o_{<t})}{\pi_{\theta_{\text{old}}}(o_t|q, o_{<t})}, 1 - \epsilon, 1 + \epsilon \right) \hat{A}_t \right) \right]$

where $\pi_{\theta_{\text{old}}}$ is the behavior policy (the policy that generated the responses, frozen before the update), $\epsilon = 0.2$ is the clipping parameter (preventing the new policy from deviating too far from the old one in a single update), and $\hat{A}_t$ is the estimated advantage of the $t$-th token. In standard PPO, $\hat{A}_t$ is typically computed using Generalized Advantage Estimation (GAE) (Schulman et al., 2015) with a learned value network $V_\phi$, which estimates the expected future return from state $s_t$.

What it computes: a surrogate loss for the policy parameters $\theta$ that approximates the true policy gradient while constraining the policy update size. The minimum operation between the clipped and unclipped objectives creates a pessimistic bound—when the advantage is positive, we want to increase the probability of token $o_t$, but if the probability ratio $\pi_\theta/\pi_{\theta_{\text{old}}}$ already exceeds $1 + \epsilon$, the clip prevents further increase; when the advantage is negative, we want to decrease probability, but if the ratio drops below $1 - \epsilon$, the clip prevents further decrease.

Why this form: PPO's clipped surrogate is the dominant algorithm in LLM RL because it stabilizes training by preventing destructively large policy updates, which is especially important when the policy is a multi-billion parameter neural network with sensitive pretrained representations. However, training a separate value network $V_\phi$ is computationally expensive and introduces additional hyperparameters (value loss weight, value clipping, value network architecture). GRPO eliminates this cost.

GRPO replaces the value-function-based advantage estimator with a group-based Monte Carlo estimate. The procedure is:

1. For each question $q$, sample a group of $G$ complete responses $\{o_1, \ldots, o_G\}$ from the current policy $\pi_{\theta_{\text{old}}}$.
2. Compute the return (binary outcome reward) for each response: $R_i = R(q, o_i) \in \{0, 1\}$.
3. For each token in response $o_i$, set the advantage estimate to the normalized reward:

$\hat{A}_{i,t} = \frac{R(q, o_i) - \text{mean}(\{R(q, o_1), \ldots, R(q, o_G)\})}{\text{std}(\{R(q, o_1), \ldots, R(q, o_G)\})}$

The GRPO objective function is then:

$J_{\text{GRPO}}(\pi_\theta) = \mathbb{E}_{q \sim p_{\mathcal{Q}}, \{o_i\}_{i=1}^G \sim \pi_{\theta_{\text{old}}}(\cdot|q)} \left[ \frac{1}{G} \sum_{i=1}^{G} \frac{1}{|o_i|} \sum_{t=1}^{|o_i|} \min\left( \frac{\pi_\theta(o_{i,t}|q, o_{i,<t})}{\pi_{\theta_{\text{old}}}(o_{i,t}|q, o_{i,<t})} \hat{A}_{i,t}, \text{clip}\left( \frac{\pi_\theta(o_{i,t}|q, o_{i,<t})}{\pi_{\theta_{\text{old}}}(o_{i,t}|q, o_{i,<t})}, 1 - \epsilon, 1 + \epsilon \right) \hat{A}_{i,t} \right) \right]$

What it computes: the average of per-token clipped surrogate losses across all $G$ responses, where each token in response $o_i$ shares the same advantage estimate $\hat{A}_{i,t}$ computed from the group's reward statistics. Unlike PPO which estimates per-token advantages that vary within a response (e.g., early tokens might have higher advantage because they set up later success), GRPO assigns a single advantage to all tokens in a response based solely on whether that response's final answer was correct.

Why this form: it eliminates the need for a value network by using the empirical mean of group rewards as the baseline (the variance reduction term $B(q, o_{<t})$ in the policy gradient derivation). The standard deviation normalization is intended to provide adaptive scaling—responses to questions with high reward variance get smaller updates, while responses to questions with consistent rewards get larger updates. This follows the common practice of advantage normalization in RL (Andrychowicz et al., 2021).

However, the paper identifies two biases that this formulation introduces (illustrated conceptually in Figure 4):

Bias 1: Response-level length bias from $1/|o_i|$. Dividing the sum over tokens by the response length $|o_i|$ means that the per-token gradient contribution is weighted inversely by how long the response is. For a correct response ($\hat{A}_{i,t} > 0$), this division means that shorter responses receive larger gradient updates per token than longer ones—the policy is pushed more strongly toward brevity when it answers correctly. For an incorrect response ($\hat{A}_{i,t} < 0$), the opposite occurs: longer responses are penalized less per token because the negative advantage is diluted across more tokens—the policy receives weaker discouragement from lengthy incorrect answers. The net effect is an asymmetric pressure: the optimizer prefers short correct answers and long incorrect answers.

Bias 2: Question-level difficulty bias from $1/\text{std}(R)$. The standard deviation of rewards across the $G$ responses to question $q$ depends on question difficulty. For extremely easy questions, all $G$ responses are correct ($R_i = 1$ for all $i$), so $\text{std}(R) = 0$ and the advantage is undefined (or would require a division-by-zero fix). For extremely hard questions, all responses are incorrect ($R_i = 0$), again yielding $\text{std}(R) = 0$. For questions of intermediate difficulty where some responses are correct and others incorrect, $\text{std}(R)$ is positive. The consequence is that questions with low standard deviation (very easy or very hard) receive disproportionately large gradient updates relative to questions with higher standard deviation, because dividing by a smaller standard deviation inflates the advantage magnitude. The paper notes that while batch-level advantage normalization is standard in RL, GRPO does it per-question, which creates differential weighting across questions rather than uniform treatment.

The length bias is not unique to GRPO. The paper makes a striking observation: examining the source code of popular open-source RLHF frameworks reveals that even "vanilla PPO" implementations contain length normalization. The authors analyzed TRL (von Werra et al., 2020), OpenRLHF (Hu et al., 2024), and VeRL (Sheng et al., 2024)—three widely-used frameworks—and found that all compute the policy loss by dividing by response length rather than by a fixed constant. Listing 1 in the paper (Section 3.1) shows two common variants:

# Variant 1: per-response length normalization (e.g., OpenRLHF)
loss_variant1 = masked_mean(ppo_loss, response_mask, dim=-1).mean()

# Variant 2: per-batch length normalization (e.g., trl, verl)
loss_variant2 = masked_mean(ppo_loss, response_mask, dim=None).mean()

The masked_mean function (in these biased implementations) computes (tensor * mask).sum(axis=dim) / mask.sum(axis=dim), which divides by the number of response tokens (a variable quantity) rather than by a fixed maximum length constant. This creates the same length bias: shorter responses contribute more per token to the loss than longer ones.

The paper hypothesizes that this implementation choice originated in the pretraining stage (Shoeybi et al., 2019), where training examples are packed into fixed-length contexts and normalizing by the number of valid tokens in each example provides natural handling of varying sequence lengths. This convention carried over to RLHF implementations without recognition that, in RL, the response length is a policy output that varies systematically with training, making length normalization a source of bias rather than a numerical convenience.

Dr. GRPO: The Corrected Optimizer

The proposed fix—Dr. GRPO (Group Relative Policy Optimization Done Right)—consists of two modifications that eliminate the identified biases:

Modification 1: Remove response-length normalization. Instead of dividing the per-token loss sum by $|o_i|$, replace it with a constant. The corrected masked mean function changes the normalization denominator from the dynamic response length to MAX_TOKENS (the maximum generation budget, a global constant throughout training):

# Biased (red in Listing 1):
def masked_mean(tensor, mask, dim):
    return (tensor * mask).sum(axis=dim) / mask.sum(axis=dim)

# Unbiased (green in Listing 1):
def masked_mean(tensor, mask, dim):
    return (tensor * mask).sum(axis=-1) / MAX_TOKENS

The key change: division by a fixed constant MAX_TOKENS rather than mask.sum(axis=dim) (which equals the response length). This means that every token in every response contributes equally to the loss, regardless of response length. A token in a 100-token response has the same gradient weight as a token in a 1000-token response, eliminating the asymmetric pressure toward long incorrect responses and short correct responses.

Modification 2: Remove question-level standard deviation normalization. Drop the division by $\text{std}(\{R(q, o_1), \ldots, R(q, o_G)\})$ from the advantage computation. The corrected advantage estimate is:

$\tilde{A}_{i,t} = R(q, o_i) - \text{mean}(\{R(q, o_1), \ldots, R(q, o_G)\})$

The resulting Dr. GRPO objective function is:

$J_{\text{Dr.GRPO}}(\pi_\theta) = \mathbb{E}_{q \sim p_{\mathcal{Q}}, \{o_i\}_{i=1}^G \sim \pi_{\theta_{\text{old}}}(\cdot|q)} \left[ \frac{1}{G} \sum_{i=1}^{G} \sum_{t=1}^{|o_i|} \min\left( \frac{\pi_\theta(o_{i,t}|q, o_{i,<t})}{\pi_{\theta_{\text{old}}}(o_{i,t}|q, o_{i,<t})} \tilde{A}_{i,t}, \text{clip}\left( \frac{\pi_\theta(o_{i,t}|q, o_{i,<t})}{\pi_{\theta_{\text{old}}}(o_{i,t}|q, o_{i,<t})}, 1 - \epsilon, 1 + \epsilon \right) \tilde{A}_{i,t} \right) \right]$

What it computes: the PPO objective as defined in Equation (2), with the advantage estimated by Monte Carlo return using the group mean as an unbiased baseline (Sutton and Barto, 2018). The outer expectation is over questions and response groups; the inner sum runs over all tokens in all $G$ responses without length re-weighting, and each token's advantage is the raw centered reward (how much better or worse this response was than the group average, in reward units, without normalization by standard deviation).

Why this form: it recovers the original PPO objective from Equation (2). The paper provides a detailed derivation in Appendix A showing that the unbiased policy gradient is:

$\nabla_\theta J(\pi_\theta) = \mathbb{E}_{q \sim p_{\mathcal{Q}}} \left[ \mathbb{E}_{\{o_i\}_{i=1}^G \sim \pi_\theta(\cdot|q)} \left[ \frac{1}{G} \sum_{i=1}^{G} \sum_{t=1}^{|o|} \nabla_\theta \log \pi_\theta(o_{i,t}|q, o_{i,<t}) \tilde{A}_{i,t} \right] \right]$

where $\tilde{A}_{i,t} = R(q, o_i) - \text{mean}(\{R(q, o_1), \ldots, R(q, o_G)\})$ without any standard deviation normalization and without any per-response length normalization. The group mean serves as the baseline $B(q, o_{<t})$ in the policy gradient theorem (Equation 5 in the paper), which is guaranteed not to bias the gradient because $\mathbb{E}_{o_t \sim \pi_\theta(\cdot|q, o_{<t})}[\nabla_\theta \log \pi_\theta(o_t|q, o_{<t}) B(q, o_{<t})] = 0$ for any baseline independent of the action $o_t$.

Connection to RLOO. The paper notes that Dr. GRPO's advantage estimate is equivalent to REINFORCE Leave-One-Out (RLOO; Kool et al., 2019; Ahmadian et al., 2024) up to a constant scaling factor of $G/(G-1)$:

$\frac{G}{G-1} \tilde{A}_{i,t} = \frac{G}{G-1} R(q, o_i) - \frac{1}{G-1} \sum_{j=1, j \neq i}^{G} R(q, o_j) - \frac{1}{G-1} R(q, o_i) = \hat{A}_{i,t}^{\text{RLOO}}$

This scaling factor can be absorbed into the learning rate without affecting optimization dynamics, meaning Dr. GRPO is effectively RLOO implemented with PPO's clipped surrogate objective. This connection provides theoretical grounding: RLOO is known to provide unbiased advantage estimates (Ahmadian et al., 2024), and the PPO clip provides training stability that pure REINFORCE lacks.

Practical implementation. The hyperparameters for Dr. GRPO training (Table 6, Appendix G):

• Maximum response length: 3,000 tokens
• Sampling temperature: 1.0
• Number of responses per question: $G = 8$
• Optimizer: AdamW ($\beta_1 = 0.9, \beta_2 = 0.95$, weight decay 0.0)
• Learning rate: $1 \times 10^{-6}$ (constant schedule)
• Policy clipping parameter $\epsilon = 0.2$
• Inner proximal update epoch: 1 (single PPO epoch per batch of rollouts)
• KL loss coefficient: 0.0 (no KL penalty)
• KL penalty coefficient: 0.0
• Gradient norm clipping: 1.0
• Hardware: 8× A100 GPUs, roughly one day of training per experiment

All experiments are implemented using the Oat framework (Liu et al., 2025a), which supports actor-learner collocation for training efficiency.

Experimental Infrastructure for Base Model Analysis

The base model analysis infrastructure (Section 2) involves several specific experimental protocols:

Template-matching experiment (Section 2.1). For each base model, responses are first generated with no template to establish baseline answering rates (using GPT-4o-mini as judge, with the detailed classification prompt in Appendix H, first prompt). Then both the R1 template and Qwen-Math template are applied, and the template producing the higher answering rate is selected. Finally, pass@8 accuracy is measured for each model with its best template across 10 temperature settings (0.1 to 1.0), generating 8 responses per question per temperature on 500 MATH training questions. This establishes both whether the model can answer questions and whether it can explore successful trajectories.

No-template performance evaluation (Section 2.2). For Qwen2.5-Math models specifically, the paper evaluates performance with no template on five standard benchmarks: AIME 2024, AMC, MATH500, Minerva Math, and OlympiadBench. Greedy decoding is used with a 3,000-token generation budget, following common practice. The comparison includes the same model with 4-shot prompting and with R1 and Qwen-Math templates, all at the same generation budget. This directly quantifies the "template penalty" described in Section 2.2.

Self-reflection detection pipeline (Section 2.3 and Appendices D, E). The cross-validation procedure works as follows:

1. For each question, generate 8 responses from the base model at each of 10 temperatures (0.1 to 1.0), yielding 80 responses per question across the temperature sweep.
2. Apply keyword-based detection using the 15-keyword pool. A question is flagged if at least one of its 8 responses at a given temperature contains any keyword.
3. Apply LLM-based detection using GPT-4o-mini with the detailed prompt from Appendix H (second prompt). The LLM classifies each response as category 0 (off-topic), 1 (no self-reflection), 2 (self-reflection present), or 3 (pure Python code). A question is flagged if at least one response is category 2.
4. Cross-validate: a question is marked as exhibiting self-reflection only if both detectors flag it. This produces the counts shown in the right panel of Figure 3.

For DeepSeek-R1-Zero analysis (Section F), the paper takes the additional step of relating self-reflection to accuracy. For each question that elicits at least one self-reflective response from R1-Zero across 8 trials, 100 responses are sampled and divided into two groups (those containing self-reflection and those not). The accuracy difference between groups is computed per question. The distribution of these differences (Figure 15) shows that self-reflection is not positively correlated with higher accuracy—approximately half of questions show the self-reflection group performing better, and half show the reverse.

Experimental Infrastructure for RL Dynamics Analysis

Training datasets (Table 3). Four question sets are used for RL training, spanning different levels of difficulty and coverage:

• ORZ-57K: A diverse, large collection combining AIME, Numina-Math, and Tulu3 MATH problems. This represents broad coverage and is similar to what larger-scale replications use.
• MATH-12K: The MATH training set (Hendrycks et al., 2021) containing high-school competition math questions. This is the standard benchmark for math reasoning.
• GSM-8K: Grade-school math word problems that are substantially simpler than MATH. This tests whether training on easier questions can transfer to harder evaluations.
• ASDiv-2K: Basic algebra problems involving only addition, subtraction, multiplication, and division. This represents the simplest domain, testing extreme out-of-distribution transfer.

Evaluation benchmarks. Performance is evaluated on five standard math reasoning benchmarks:

• AIME 2024 (Li et al., 2024a): Competition problems from the American Invitational Mathematics Examination
• AMC (Li et al., 2024a): American Mathematics Competition problems
• MATH500 (Hendrycks et al., 2021): A 500-question subset of the MATH test set
• Minerva Math (Lewkowycz et al., 2022): A collection of quantitative reasoning problems
• OlympiadBench (He et al., 2024): Olympiad-level bilingual multimodal scientific problems

Reward function. The reward is computed using the Math-Verify library (HuggingFace), which implements a rule-based verifier:

$R(q, o) = \begin{cases} 1 & \text{if } o \text{ contains the correct final answer to } q \\ 0 & \text{otherwise} \end{cases}$

This is "minimalistic" (Section 3.2): there is no process reward, no partial credit, no reward shaping. The model receives binary feedback solely based on whether the final answer matches the ground truth. For math problems, this is feasible because answers are typically numeric or algebraic expressions that can be checked deterministically.

Training-monitoring metrics. The paper tracks five quantities during training (Figure 5):

1. Training reward: Average outcome reward across responses in the training batch, showing whether the policy's correctness rate is improving.
2. Output length: Average token count of generated responses, showing whether response length is increasing (either genuinely, through more elaborate reasoning, or spuriously, through optimization bias).
3. Evaluation output length (correct): Average length of correct responses on benchmark evaluations.
4. Evaluation output length (incorrect): Average length of incorrect responses on benchmark evaluations. The divergence between this and correct-response length is a key diagnostic for optimization bias.
5. Average benchmark score: Mean accuracy across the five evaluation benchmarks.

Dr. GRPO vs. GRPO comparison protocol (Section 3.2). The head-to-head comparison uses Qwen2.5-1.5B as the base model, the R1 template, and MATH-12K as the training set. Both algorithms use identical hyperparameters from Table 6, differing only in the two modifications (length normalization and std normalization). The experiment runs for approximately 150 policy iteration steps, with evaluations conducted periodically on all five benchmarks.

Template × Question Set interaction experiment (Section 3.3). Starting from Qwen2.5-Math-1.5B, Dr. GRPO is applied with each of three templates (R1, Qwen-Math, No template) crossed with each of four question sets (ORZ-57K, MATH-12K, GSM-8K, ASDiv-2K), producing 12 training runs. Each run uses identical Dr. GRPO hyperparameters and evaluation protocol. The goal is to isolate how template choice moderates the effect of question set coverage on RL performance.

Domain-specific pretraining experiment (Section 3.4). To test whether math pretraining improves the RL ceiling, three Llama-3.2-3B variants are compared:

• Llama-3.2-3B: The vanilla base model with no additional math pretraining.
• Llama-3.2-3B-FineMath: Continual pretrained on the FineMath dataset (Allal et al., 2025), which adds domain-specific math knowledge.
• Llama-3.2-3B-NuminaQA: Continual pretrained on concatentated question-answer texts from NuminaMath-1.5 (Li et al., 2024b) for 2 epochs at learning rate $1 \times 10^{-5}$. This simulates the hypothesized Qwen2.5 pretraining pattern where question-answer pairs are concatenated as plain text.

All three variants are then RL-tuned with Dr. GRPO using the R1 template, tracking benchmark accuracy and response length. The comparison between GRPO and Dr. GRPO is also replicated on the FineMath variant to verify the bias effect generalizes beyond Qwen2.5 models.

FLOPs and compute accounting. Each experiment runs on 8× A100 GPUs and completes in "about one day" (Appendix G, opening). The final minimalist recipe (Qwen2.5-Math-7B + Dr. GRPO + MATH level 3–5 questions + Qwen-Math template) requires exactly 27 hours on this hardware configuration, as stated in Section 1.

Design choices and their justifications:

• $G = 8$ responses per question: Balances variance reduction in advantage estimation against computational cost. Larger $G$ provides better group statistics (more accurate mean and standard deviation estimates) but increases per-step FLOPs linearly. The choice of 8 is common in RLOO literature (Ahmadian et al., 2024).

• Temperature 1.0 for sampling: High temperature encourages exploration, which is important during RL because the policy needs to discover diverse trajectories that lead to correct answers. The base model analysis (Figure 3, middle) shows that pass@8 is highest at elevated temperatures for most models.

• Constant learning rate $1 \times 10^{-6}$ with no decay: The paper uses a constant schedule, which is unusual compared to typical LLM fine-tuning (which often uses cosine or linear decay). This may be because RL training involves non-stationary data distributions (the policy itself changes), making it unclear when learning rate decay would be beneficial.

• No KL regularization ($\beta = 0$): Justified by the use of a rule-based verifier. Unlike learned reward models that are only reliable near their training distribution, Math-Verify is a deterministic function that is equally valid for any response format. Removing the KL term reduces memory usage (no need to store $\pi_{\text{ref}}$) and avoids constraining the policy to remain near its pretrained distribution, which could limit exploration of new reasoning strategies.

• Single PPO epoch per batch ($\text{inner proximal update epoch} = 1$): Standard practice in on-policy RL to prevent the policy from overfitting to the current batch of rollouts (which become increasingly off-policy with multiple gradient steps).

• Maximum response length of 3,000 tokens: Chosen because Qwen2.5-Math base models have a context length of 4K tokens. The 3,000-token generation budget leaves room for the prompt while ensuring responses don't exceed the model's context window. The paper explicitly notes (Appendix B) that this is shorter than the 8K budget used by some other methods (OpenReasoner-Zero, R1-Distill models), and provides separate results at 8K for comparison.

4. Key Insights and Innovations

Innovation 1: Reframing the "Aha Moment" as a Pretraining Artifact, Not an RL-Derived Emergence

The most intellectually disruptive contribution of this paper is not the algorithmic fix (Dr. GRPO) but the systematic demonstration that the celebrated "Aha moment"—the spontaneous emergence of self-reflection behaviors during R1-Zero-style RL training—is already present in base models before any RL is applied, including in DeepSeek-V3-Base-685B, the very model from which the original R1-Zero was trained.

Prior to this work, the dominant interpretation of the Aha moment, crystallized in Guo et al. (2025), treated self-reflection as an emergent capability that RL creates: the model learns to allocate additional test-time computation by recognizing its own errors and backtracking, and this behavior arises purely from optimizing outcome-based rewards. This narrative resonated deeply because it aligned with broader AI aspirations of complex behaviors emerging from simple objectives. The subsequent wave of R1-Zero replications (Zeng et al., 2025; Hu et al., 2025; Pan et al., 2025) largely accepted this framing, reporting the presence of self-reflection keywords as evidence that their reproductions had successfully captured the phenomenon—though none tested DeepSeek-V3-Base to verify whether the original result represented genuine emergence.

This paper fundamentally reframes the Aha moment through a simple but powerful diagnostic move: ask whether the behavior exists in the base model before RL, using the same detection methods that would later be applied to the RL-trained model. The answer is unambiguous. Every base model tested, across three different model families (Qwen2.5, Llama-3.1, DeepSeek), generates self-reflection keywords and patterns prior to any RL training (Figure 3, right panel). The cross-validation detection methodology (keyword matching plus LLM-based classification, requiring both to agree) rules out the possibility that these are merely incidental word occurrences. DeepSeek-V3-Base-685B—which the paper characterizes as a "nearly pure base model" based on its low no-template answering rate—produces responses containing explicit "Aha," "wait," and "let me try again" patterns (Figure 13, Appendix E).

This is a fundamental reframing, not an incremental correction of prior work. It changes the causal interpretation of what RL contributes. Rather than RL creating self-reflection from nothing, the more parsimonious explanation is that RL amplifies an existing behavioral pattern: by increasing response length (which it does, as shown by the training dynamics in Figure 5), RL gives the model more opportunities to exhibit self-reflection patterns that were already in its behavioral repertoire. The binary reward signal (1 for correct answer, 0 otherwise) provides no explicit incentive for self-reflection—there is no partial credit for recognizing an error—so there is no RL mechanism through which the behavior could be learned de novo. Instead, the model's pretraining distribution already contained examples of problem-solving with self-correction (e.g., in math forum discussions, textbooks with worked solutions that include error-checking, or synthetic data), and the model reproduces these patterns when generating long responses.

The paper goes further than simply demonstrating the behavior exists in base models. It also analyzes whether self-reflection during inference is actually causal in improving accuracy, using DeepSeek-R1-Zero itself. The result (Figure 15, Appendix F) is a null finding: responses containing self-reflection are approximately equally likely to be correct as responses without self-reflection, with the accuracy difference between the two groups centered near zero across questions. This challenges not just the origin of the Aha moment but its functional significance: if self-reflection doesn't make responses more likely to be correct, it may be better understood as a stylistic artifact that correlates with longer outputs (which RL encourages) rather than a mechanism for improved reasoning. The paper explicitly acknowledges that self-reflection could still be beneficial during training by improving exploration (a point outside the scope of their inference-stage analysis), but the burden of proof has shifted: proponents of the Aha moment must now demonstrate causal benefit, not merely document its presence.

The significance of this reframing extends beyond the specific case of self-reflection. It suggests a general principle: when RL training of base models appears to produce "emergent" behaviors, the first diagnostic step should be to check whether those behaviors exist in the base model's outputs at sufficient sampling budgets. Many capabilities that appear to arise during RL may be better understood as the model's pretrained behaviors becoming more visible as response length increases, rather than genuinely new skills acquired through reward optimization. This principle applies to long chain-of-thought reasoning, verification behaviors, and other patterns that the R1-Zero literature has attributed to RL-driven emergence.

Innovation 2: Identifying a Pervasive Implementation Bias That Masquerades as Algorithmic Progress

The paper's second major contribution is the identification that GRPO—and, critically, nearly all major open-source PPO implementations—contains an optimization bias that systematically favors longer incorrect responses, and that this bias is sufficient to produce the "double-increase" phenomenon (reward and length rising together) that the field has widely interpreted as evidence of genuine reasoning development.

This is an innovation at the level of diagnostic concept: the paper doesn't just propose a new algorithm (Dr. GRPO is a straightforward correction), but introduces a new category of analysis—implementation-auditing of RL objectives for LLMs—that reveals how the gap between mathematical specification and code can produce spurious empirical signatures. Prior work on R1-Zero replications (Zeng et al., 2025; Hu et al., 2025) reported the same length-increase phenomenon as DeepSeek-R1-Zero and interpreted it positively: the model is learning to "think longer" about harder problems, developing more elaborate reasoning chains. The possibility that this increase was partially an artifact of how the loss was computed was not considered.

The paper's analysis of this bias operates at multiple levels. First, it mathematically decomposes GRPO's objective (Equation 3) to identify two distinct sources: the response-level $1/|o_i|$ term that down-weights gradients for long responses (making the optimizer less effective at penalizing lengthy incorrect answers), and the question-level $1/\text{std}(R)$ term that gives disproportionate weight to very easy or very hard questions. The bias is not subtle—Figure 4 illustrates it conceptually, and the experimental results (Figure 5, bottom panel) show that Dr. GRPO reduces incorrect-response length by roughly 1.4K tokens while maintaining or improving benchmark performance. This is a large effect size.

Second, and more importantly, the paper discovers that this bias predates GRPO and exists in PPO implementations that are mathematically unbiased in their formulation. By auditing the source code of TRL (von Werra et al., 2020), OpenRLHF (Hu et al., 2024), and VeRL (Sheng et al., 2024)—frameworks that collectively underpin most open-source LLM RL research—the authors find that all normalize the policy loss by response length in their masked_mean computation (Listing 1, Table 2). This is not a bug in the sense of a coding error; it's a formulation-implementation misalignment where a practice that originated in pretraining (where length normalization handles variable-length packed sequences) carried over to RL without recognition that response length becomes a policy-dependent variable that the optimizer can exploit.

The significance of this finding is that it potentially invalidates a key piece of evidence that the R1-Zero community has used to validate their reproductions. If the length increase during training is partially a bias artifact (as Dr. GRPO's ablation demonstrates), then previous reports of "successful reproduction" based on observing this phenomenon are confounded. The paper is careful not to claim that all length increase is artifactual—Dr. GRPO still shows some length growth (Figure 5, Plot 2), consistent with the model genuinely developing longer reasoning chains—but it demonstrates that the magnitude of length increase has been overestimated, and that a substantial portion is attributable to optimization bias rather than reasoning improvement. This matters because response length is a proxy metric: if practitioners use it to decide when training is "working," they may be misled.

The practical implication is clear and actionable: existing open-source RLHF codebases should adopt the corrected masked_mean implementation (dividing by a constant MAX_TOKENS rather than by the dynamic number of response tokens). The paper provides the concrete code change (Listing 1, the green line) and validates it experimentally across model scales (1.5B, 3B, 7B) and model families (Qwen2.5, Llama-3.2). This is an incremental change in implementation complexity but a potentially large impact on the reliability of future RL training results.

Innovation 3: Demonstrating That Template Choice Determines Whether RL Gains Are Recovery or Genuine Improvement

The paper's third contribution is a diagnostic framework for decomposing RL training gains into template-recovery effects versus genuine capability improvement, enabled by the discovery that Qwen2.5 base models perform ~60% better with no template than with standard 4-shot prompting (Table 1). This finding reframes the interpretation of R1-Zero-like training results and provides a principled basis for template selection.

Prior R1-Zero replications (Zeng et al., 2025; Hu et al., 2025; Pan et al., 2025) all apply prompt templates (model-specific chat formats or the R1 template) to base models before RL training, as is standard practice to elicit question-answering behavior from sentence-completion models. The resulting training curves show the model starting at low accuracy and improving substantially—which has been interpreted as RL teaching the model to reason. This paper's template analysis reveals that for Qwen2.5 models specifically, the template itself depresses performance, and RL partially recovers what the template destroyed. The base Qwen2.5-Math-7B model achieves 38.2% average accuracy across five benchmarks with no template, but only 26.5% with the Qwen-Math template (the one the Qwen team built for these models), and near-zero with the R1 template (Table 1). The training curves in Figure 6 show that under the R1 template (left panel), the model starts near 0% and climbs to ~40%, creating the appearance of dramatic RL-driven improvement. Under the Qwen-Math template (middle panel), the starting point is higher and the gain is proportionally smaller. Under no template (right panel), the starting point is highest and RL provides relatively modest additional gains.

This is a conceptual reframing of how to evaluate RL contributions. The paper doesn't claim that RL provides no value—all configurations improve, and the final accuracies are comparable across templates given appropriate question sets—but it demonstrates that the magnitude of reported gains is heavily confounded by how much the template mismatches the model's pretraining format. The "pure RL" narrative (RL creating reasoning capability from a blank slate) is revealed to be most applicable precisely when the template is most mismatched—which is the case where the base model's actual capabilities are most obscured. Conversely, when the template matches the model's native format (no template for Qwen2.5 models), RL provides smaller but still real gains, and these gains can more reliably be attributed to genuine improvement rather than format adaptation.

The paper extends this insight to the interaction between templates and question set coverage (Section 3.3). The finding that GSM-8K (simple grade-school math) can produce better final performance than ORZ-57K (diverse competition math) when using the Qwen-Math template, but the reverse holds with the R1 template, inverts the conventional wisdom that broader question coverage is always better. The paper's explanation—that when the template matches the base model's pretraining format, even simple questions can reinforce useful reasoning behaviors already present in the model, whereas mismatched templates require the model to relearn basic formatting from diverse examples—provides a principled framework for data selection that prior work lacked.

The practical implication is that practitioners should measure their base model's performance without a template before choosing one for RL training. If the base model already answers questions in its raw pretraining format, applying a template may be counterproductive, suppressing capabilities that RL must then reconstruct. This is a simple diagnostic—generate a few hundred responses with no template and check whether they contain answers—that could substantially change how R1-Zero-like training is configured for a given base model. The paper's hypothesis that Qwen2.5 models were pretrained on concatenated question-answer texts suggests this pattern may be increasingly common as model developers incorporate instruction-formatted data into pretraining corpora (Yang et al., 2024b explicitly states that Qwen2.5-Math uses chat model data during pretraining). If so, the template-mismatch confound may affect an expanding fraction of future R1-Zero replications.

Innovation 4: Establishing That Domain-Specific Pretraining—Not RL—Is the Binding Constraint on Reasoning Capability

The paper's fourth contribution is an empirical demonstration that the ceiling of R1-Zero-like training is set primarily by the base model's domain knowledge, not by the RL algorithm or training duration. This is most clearly shown through the Llama-3.2-3B experiment (Section 3.4, Figure 7), but the finding generalizes across the paper's broader analysis.

The field's enthusiasm for R1-Zero-style training has been driven partly by the promise that RL alone—without supervised fine-tuning, without domain-specific pretraining, without curated instruction data—can produce strong reasoning models from generic base models. The paper's Llama experiment tests this promise directly by comparing RL training on three variants of the same architecture: the vanilla Llama-3.2-3B base model, a version continual-pretrained on FineMath (a math-domain corpus), and a version further continual-pretrained on concatenated question-answer texts from NuminaMath (mimicking the hypothesized Qwen2.5 pretraining pattern). The result: vanilla Llama improves only marginally with RL (from ~3.3% to ~6.8% average benchmark accuracy), while the math-pretrained variants reach ~14.8% (FineMath) and ~20.7% (NuminaQA). The RL algorithm (Dr. GRPO) is identical across all conditions; the difference is entirely in what knowledge the base model brings to the table.

This is a negative result with strong positive implications: it establishes a clear boundary condition for when R1-Zero-like training is likely to succeed. If the base model lacks domain knowledge (as measured by its pre-RL accuracy on the target benchmarks), RL cannot compensate—there are no correct trajectories to reward, and the exploration problem becomes intractable. This is consistent with the paper's earlier finding that all tested base models have non-trivial pass@8 rates (Figure 3, middle), indicating that successful RL requires at least some capability to generate correct answers. But the Llama experiment sharpens this observation: even when pass@8 is above zero, the magnitude of RL improvement depends strongly on how much domain knowledge the base model has internalized.

The significance of this finding is that it reframes the R1-Zero narrative from "RL can replace supervised fine-tuning" to "RL can amplify capabilities that pretraining has already installed." This is a more modest claim but a more accurate one, and it has direct implications for resource allocation: if the goal is to build a strong math reasoner, investing compute in domain-specific pretraining (e.g., FineMath, NuminaMath) yields compounding returns because it both raises the starting accuracy and increases the ceiling that RL can reach. The paper's own minimalist recipe reflects this insight: it uses Qwen2.5-Math-7B—a model already heavily pretrained on math—as the starting point, and achieves state-of-the-art results with relatively modest RL compute (27 hours on 8× A100s).

The finding also connects to broader scaling principles. Just as the Chinchilla scaling laws (Hoffmann et al., 2022) showed that pretraining compute should be allocated proportionally between model size and data quantity, this paper suggests an analogous principle for post-training: the compute allocated to RL (inference-time optimization through policy updates) should be proportional to the domain knowledge already embedded in the model through pretraining. Pouring RL compute into a domain-ignorant base model yields diminishing returns; investing that compute in better pretraining data yields compounding returns. This is not a law that the paper formalizes mathematically, but it emerges clearly from the empirical results—the 7B Qwen2.5-Math model with RL outperforms larger models without domain pretraining, and the 685B DeepSeek-V3-Base (which the paper describes as having relatively weak math-solving capability compared to Qwen2.5-Math-7B, Figure 3 middle panel) produced R1-Zero results that smaller, domain-specialized models can now match or exceed.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper uses multiple datasets across different experiments. For base model analysis (Section 2), responses are generated from 500 questions sampled from the MATH training set (Hendrycks et al., 2021). For RL training (Section 3), four question sets are used: ORZ-57K (a diverse mixture combining AIME, Numina-Math, and Tulu3 MATH problems), MATH-12K (the full MATH training set), GSM-8K (grade-school math word problems), and ASDiv-2K (basic algebra problems involving only +, −, ×, ÷). For evaluation, five standard benchmarks are used: AIME 2024, AMC (both from Li et al., 2024a), MATH500 (Hendrycks et al., 2021), Minerva Math (Lewkowycz et al., 2022), and OlympiadBench (He et al., 2024).

• Base model(s). The paper investigates six base models spanning three families: Qwen2.5-Math-1.5B, Qwen2.5-Math-7B, Qwen2.5-7B (Yang et al., 2024a;b), Llama-3.1-8B (Grattafiori et al., 2024), DeepSeek-Math-7B (Shao et al., 2024), and DeepSeek-V3-Base-685B (Liu et al., 2024). Qwen2.5 models were chosen because they dominate recent R1-Zero replications (Zeng et al., 2025; Hu et al., 2025; Cui et al., 2025). DeepSeek-V3-Base-685B is included specifically because it is the model from which the original DeepSeek-R1-Zero was RL-tuned—testing it fills a critical gap left by prior analyses (Liu et al., 2025b; Yeo et al., 2025) that only examined open-source replications. Llama-3.1-8B and DeepSeek-Math-7B provide cross-family comparison.

• Metrics. Three categories of metrics are tracked:

  • Answering rate: The percentage of model responses that attempt to answer the question (as opposed to exhibiting sentence-completion patterns), judged by GPT-4o-mini using a classification prompt reproduced in Appendix H (first prompt). This is used only in base model diagnostics.
  • pass@k accuracy: The fraction of questions for which at least one of k sampled responses contains the correct final answer. For base model analysis, pass@8 is measured at temperatures from 0.1 to 1.0. For RL evaluation, greedy accuracy (k=1) is reported on the five benchmarks.
  • Benchmark accuracy (%): Greedy decoding accuracy on AIME 2024, AMC, MATH500, Minerva Math, and OlympiadBench, with average across benchmarks as the primary aggregate metric. Answers are verified using Math-Verify (HuggingFace), which checks whether the generated text contains the correct final answer string.
  • Response length: Average token count of generated responses, tracked separately for correct and incorrect responses during RL evaluation.

• Baselines. The paper compares against multiple categories of prior work:

  • Base model with 4-shot prompting: Standard few-shot evaluation, reported for Qwen2.5-Math-1.5B and Qwen2.5-Math-7B in Table 1.
  • Base model with specific templates: R1 template, Qwen-Math template, and no template (Table 1, Figure 3 left panel).
  • GRPO (Shao et al., 2024): The original algorithm as specified in Equation (3), with response-level length normalization and question-level standard deviation normalization.
  • Instruct models: Qwen2.5-Math-1.5B-Instruct and Qwen2.5-Math-7B-Instruct (Yang et al., 2024b), providing SFT-based baselines at comparable scales.
  • R1-Distill models: R1-Distill-Qwen-1.5B and R1-Distill-Qwen-7B (Guo et al., 2025), using the distilled R1 reasoning data as an alternative to pure RL.
  • Prior R1-Zero replications: SimpleRL-Zero-7B (Zeng et al., 2025), PRIME-Zero-7B (Cui et al., 2025), and OpenReasoner-Zero-7B (Hu et al., 2025), all evaluated at both 3K and 8K generation budgets where applicable.

• Generation budget / compute accounting. A universal 3,000 token generation budget is used for all evaluations and training, determined by the Qwen2.5-Math models' base context length of 4K tokens (Appendix B). For RL training, each update samples G=8 responses per question (Table 6), with each response limited to 3,000 tokens. For baselines that support longer contexts (OpenReasoner-Zero and R1-Distill models), separate results at 8K generation budget are also reported for completeness. Training compute is measured in wall-clock time: all experiments run on 8× A100 GPUs for "about one day," with the final minimalist recipe (7B model) requiring exactly 27 hours (Section 1).

• Cross-validation / statistical protocol. For the self-reflection detection analysis (Section 2.3, Appendix D), the paper employs a cross-validation technique combining keyword-based detection (15 manually curated self-reflection keywords) and LLM-based detection (GPT-4o-mini classification into four categories) where a response is counted as exhibiting self-reflection only if both methods agree. This filters keyword false positives (incidental word occurrences) and LLM false positives (over-interpretation of complex text). For the Dr. GRPO vs. GRPO comparison (Appendix C, Figure 9), 3 independent RL runs with different random seeds are conducted, with mean curves and standard deviation bands reported, to verify that the improvements are statistically significant rather than seed-dependent. The ablation of bias terms (Appendix C, Figure 8) also reports curves across training iterations, with consistent trends across runs.

Main Quantitative Results

Base Model Diagnostics: Most Base Models Already Possess Question-Answering and Reasoning Capabilities

The paper's first major empirical finding is that commonly-used base models—particularly Qwen2.5 variants—exhibit properties that have been attributed to RL-driven emergence even before any RL training occurs. The evidence unfolds across the three diagnostic dimensions:

Question-answering ability (Figure 3, left panel). The answering rate metric reveals a stark dichotomy. Llama-3.1-8B and all DeepSeek models achieve low answering rates without templates (~20-30%), but improve to ~80-100% with the R1 template. However, all Qwen2.5 models achieve approximately 100% answering rate with no template at all—they natively produce answers rather than sentence completions. DeepSeek-V3-Base-685B shows the lowest no-template answering rate (~0.2), qualifying it as a "nearly pure base model" in the authors' assessment. The practical consequence is that applying a template to a Qwen2.5 base model is unnecessary for eliciting answers and may actually degrade performance, a point verified in Table 1.

Exploration ability: pass@8 (Figure 3, middle panel). All six base models achieve non-trivial pass@8 accuracy across temperatures 0.1 to 1.0, confirming that correct trajectories exist in the initial policy distribution for RL to amplify. However, the magnitudes differ substantially. Qwen2.5-Math-7B achieves approximately 70-80% pass@8 at higher temperatures, while DeepSeek-V3-Base-685B reaches approximately 60-70%. This means Qwen2.5-Math-7B starts from a substantially higher exploration ceiling. The paper notes this may partially explain why recent R1-Zero replications converge on Qwen2.5 models—they provide stronger initial policies that make RL improvement easier to demonstrate.

Self-reflection detection (Figure 3, right panel, and Figure 12, Appendix D). Across all models and temperatures, the cross-validated detection (requiring agreement between keyword-based and LLM-based methods) identifies hundreds of questions where base model responses contain self-reflection patterns. The count varies by model and temperature, but the key result is that DeepSeek-V3-Base-685B generates a "decent amount of self-reflections" (Section 2.3), including explicit examples with "Aha" and "wait" (Figure 13, Appendix E). The frequency increases with temperature, consistent with higher sampling temperatures producing more diverse (and thus more likely self-reflective) outputs. Figure 10 (Appendix D) breaks down keyword occurrences across model families, revealing that different models favor different reflection keywords—Qwen2.5 models emphasize "check again," "double-check," and "recheck," while DeepSeek models avoid "re-evaluate" and "verify again" but use other patterns, and Llama models frequently use "think again."

Qwen2.5-Math Models Suffer ~60% Performance Loss from Prompt Templates

Table 1 presents the benchmark performance of Qwen2.5-Math-1.5B and Qwen2.5-Math-7B under four configurations: 4-shot prompting, R1 template, Qwen-Math template, and no template. The headline result for Qwen2.5-Math-7B:

• No template: 38.2% average accuracy across five benchmarks
• Qwen-Math template: 26.5% average (a 30.6% relative reduction)
• R1 template: 0.0% average (near-complete destruction of capability)
• 4-shot prompting: 23.8% average (a 37.7% relative reduction)

The ~60% relative improvement from applying no template versus 4-shot prompting is the largest single effect size reported in the paper for a non-RL intervention. On individual benchmarks, the pattern holds: MATH500 accuracy is 69.0% with no template versus 61.6% with 4-shot prompting, AMC is 45.8% versus 22.5%, and AIME 2024 is 0.2% versus 3.3% (note: these are near-floor numbers for the hardest benchmark). For Qwen2.5-Math-1.5B, the effect is similar: 33.1% with no template versus 19.7% with 4-shot prompting.

This result directly supports the paper's hypothesis that Qwen2.5-Math models were pretrained on concatenated question-answer texts ($q; o$ pairs) and achieve peak performance when presented with questions in their native pretraining format—raw text without any template wrapping. Templates disrupt this format, degrading performance, and RL must then reconstruct what the template destroyed.

GRPO's Length Bias Produces Spurious Response-Length Growth; Dr. GRPO Mitigates It

Figure 5 provides the head-to-head comparison of GRPO and Dr. GRPO when training Qwen2.5-1.5B with the R1 template on MATH-12K over ~150 policy iteration steps. The key quantitative comparisons:

Training dynamics (Plots 1-2):

• Both algorithms show increasing reward (Plot 1), rising from approximately 0.3 to 0.55-0.6, following the "double-increase" pattern associated with DeepSeek-R1-Zero.
• Both show increasing response length during training (Plot 2). However, GRPO's response length diverges upward from Dr. GRPO's starting around iteration 50-75, with GRPO reaching approximately 900-1,000 tokens by iteration 150, compared to approximately 700-750 tokens for Dr. GRPO. The gap of ~200-250 tokens represents the length growth attributable to optimization bias.

Evaluation results (Plots 3-5):

• Correct response length (Plot 3): Dr. GRPO produces correct responses of ~800-900 tokens, trending slightly upward. GRPO's correct responses start shorter but grow to approximately 1,100-1,200 tokens—a divergence of ~300 tokens compared to Dr. GRPO.
• Incorrect response length (Plot 4): This is where the bias effect is most pronounced. Dr. GRPO's incorrect responses remain relatively stable at approximately 400-500 tokens throughout training. In contrast, GRPO's incorrect responses grow from approximately 600 tokens to approximately 1,600-1,800 tokens by iteration 150—a gap of approximately 1,200-1,400 tokens compared to Dr. GRPO. This is the direct manifestation of the response-level length bias: GRPO penalizes long incorrect responses less per token than short ones, creating an asymmetric pressure toward lengthier wrong answers.
• Average benchmark score (Plot 5): Both algorithms reach approximately 18-20% average accuracy by iteration 150, with the curves largely overlapping. Dr. GRPO occasionally shows slightly higher accuracy (by 1-2 percentage points), but the primary effect is token efficiency: achieving comparable reasoning performance with substantially shorter (and thus more computationally efficient) responses, particularly for incorrect generations.

The ablation of individual bias terms (Appendix C, Figure 8) confirms that the length normalization term is the primary driver of response-length inflation. The variant without length normalization (GRPO w/o Len Norm) tracks Dr. GRPO closely in terms of response length (middle subplot), while the variant without standard deviation normalization (GRPO w/o Std Norm) still shows length growth comparable to vanilla GRPO. All unbiased variants outperform vanilla GRPO in both training reward and evaluation accuracy, suggesting that removing biases not only improves token efficiency but also enables better policy learning.

Statistical Robustness Across 3 Independent Seeds

Appendix C, Figure 9 reports results from 3 independent RL runs comparing GRPO and Dr. GRPO. The mean curves (solid lines) with standard deviation bands (shaded areas) show:

• Dr. GRPO achieves average benchmark accuracy of approximately 20-22% across the 3 seeds, compared to approximately 16-18% for GRPO—a consistent and statistically meaningful gap.
• Dr. GRPO's response length stabilizes around 700-800 tokens, while GRPO's continues to grow to approximately 1,000-1,100 tokens.
• The standard deviation bands are relatively narrow and non-overlapping by the end of training, confirming that the effects are not seed-dependent.

Template × Question Set Interaction: Coverage Requirements Depend on Template Match

Figure 6 presents the 12-way experiment crossing three templates (R1, Qwen-Math, No template) with four question sets (ORZ-57K, MATH-12K, GSM-8K, ASDiv-2K), training Qwen2.5-Math-1.5B with Dr. GRPO. Key quantitative findings:

Under the R1 template (left panel):

• Training on ORZ-57K (broad, diverse coverage) achieves the highest final average accuracy of approximately 35-40%.
• MATH-12K reaches slightly lower performance of approximately 30-35%.
• GSM-8K plateaus at approximately 20-25%.
• ASDiv-2K plateaus around 15-20%.
• The ordering is monotonic with question set size and diversity: more and harder questions yield better performance. This aligns with the conventional wisdom that broad coverage is important for RL training (Luo et al., 2025; Hu et al., 2025).

Under the Qwen-Math template (middle panel):

• The ordering inverts. GSM-8K achieves the highest final accuracy of approximately 45-50%, substantially outperforming ORZ-57K (approximately 38-42%), MATH-12K (approximately 35-40%), and ASDiv-2K (approximately 30-35%).
• This means training on the simplest, most out-of-distribution question set (grade-school math) produces the best transfer to competition-level benchmarks—nearly doubling the initial accuracy from ~25% to ~45-50%.

Under no template (right panel):

• All question sets converge to similar final performance of approximately 40-42%, though ASDiv-2K shows more variance and slightly lower final accuracy.
• The initial accuracy is highest across all three template conditions, confirming that no template preserves Qwen2.5-Math's native capabilities.

The paper's interpretation is that when there is a "large mismatch between base models and templates" (R1 template + Qwen2.5-Math), the policy improvement mainly comes from RL actively teaching the model new behaviors, which requires diverse data with good coverage. When the template matches the model's pretraining format (Qwen-Math template or no template), RL is primarily reinforcing existing reasoning behaviors rather than teaching new ones, so even small, simple, and out-of-distribution question sets suffice.

Domain-Specific Pretraining Dramatically Improves RL Ceiling for Llama Models

Figure 7 (left) presents the RL training curves for three Llama-3.2-3B variants under identical Dr. GRPO training with the R1 template:

• Vanilla Llama-3.2-3B: Starts from approximately 6-8% average benchmark accuracy and reaches approximately 6-7% by iteration 300—essentially no improvement. The model lacks sufficient math domain knowledge to generate correct trajectories at rates that provide meaningful reward signal.
• Llama-3.2-3B-FineMath (continual pretrained on FineMath): Starts lower (~5%) but reaches approximately 14-15% by iteration 300. The math-specific pretraining provides enough domain knowledge for RL to amplify.
• Llama-3.2-3B-NuminaQA (concatenated QA continual pretraining): Starts near zero but improves steadily to approximately 20-21% by iteration 300, substantially outperforming FineMath-only pretraining.

The 3× gap between vanilla Llama and NuminaQA-pretrained Llama (6.8% vs. 20.7% from Table 4) at the same model scale, with the same RL algorithm and hyperparameters, demonstrates that pretraining domain knowledge is the binding constraint on RL ceiling, not model size or training duration.

The right panel of Figure 7 confirms that the GRPO length bias generalizes beyond Qwen2.5 models: when training Llama-3.2-3B-FineMath, GRPO produces average response lengths of approximately 1,100-1,200 tokens (dashed line) while Dr. GRPO produces approximately 700-800 tokens, with similar or slightly better accuracy for Dr. GRPO (solid lines, both reaching ~14-15%). This demonstrates that the bias is not specific to a particular model family or scale.

Final Benchmark Results: A Minimalist Recipe Achieves State-of-the-Art

Table 4 (Appendix B) and Figure 2 consolidate the benchmark performance of the paper's models against baselines. The headline results for the 7B scale:

Oat-Zero-7B (the paper's minimalist recipe: Qwen2.5-Math-7B + Dr. GRPO + MATH level 3-5 questions + Qwen-Math template, 27 hours on 8× A100s):

• AIME 2024: 43.3%
• AMC: 62.7%
• MATH500: 80.0%
• Minerva Math: 30.1%
• OlympiadBench: 41.0%
• Average: 51.4%

This establishes a new state-of-the-art among 7B models trained with pure RL (no SFT, no distillation), outperforming:

• SimpleRL-Zero-7B: 46.6% average (trained on 8K MATH examples)
• PRIME-Zero-7B: 48.0% average (uses implicit process rewards)
• OpenReasoner-Zero-7B @ 3K budget: 43.0% average (trained on 57K diverse questions)
• Qwen2.5-Math-7B-Instruct: 45.1% average (SFT-trained instruct model)

The margin over PRIME-Zero (51.4% vs. 48.0%, a 3.4 percentage point gap) is particularly notable because PRIME-Zero uses a more sophisticated reward mechanism (implicit process rewards from token-level rollout values), while Oat-Zero uses only binary outcome rewards—supporting the paper's claim that algorithmic simplicity, when combined with appropriate template selection and unbiased optimization, can match or exceed more complex approaches.

At the 1.5B scale, Oat-Zero-1.5B achieves 42.1% average, substantially outperforming R1-Distill-Qwen-1.5B at the same 3K budget (22.0%) and even exceeding it at 8K (41.5%). This demonstrates that pure RL can match or exceed distillation-based approaches even at small scales, given appropriate base model selection and unbiased optimization.

At the 3B scale, the Oat-Zero-3B model (Llama-3.2-3B-NuminaQA + Dr. GRPO) achieves 20.7% average, demonstrating that domain-specific pretraining can bootstrap a originally non-math model (vanilla Llama-3.2-3B: 3.3%) to competitive performance. This is substantially lower than the 7B results but establishes the principle that the base model's pretraining determines the RL ceiling.

Ablation Studies and Robustness Checks

Ablation of GRPO bias terms (Appendix C, Figure 8): The paper tests four variants on Qwen2.5-1.5B with a mixture of 3K diverse math questions from ASDiv, MATH, and AIME (pre-2023). Dr. GRPO (no length normalization, no std normalization) achieves the highest training reward and evaluation accuracy. GRPO without length normalization achieves similar response length to Dr. GRPO (~600-700 tokens) but slightly lower accuracy (~18% vs. ~20% at step 150), confirming that length normalization is the primary driver of length inflation but that removing std normalization also contributes to accuracy gains. Vanilla GRPO produces the longest responses (~900 tokens) and lowest accuracy (~16-17%), confirming both biases are detrimental. The ordering is Dr. GRPO ≈ GRPO w/o Len Norm > GRPO w/o Std Norm > Vanilla GRPO in terms of evaluation accuracy.

Self-reflection detection methodology validation (Appendix D): The paper compares three detection approaches across models and temperatures (Figure 12). Keyword-based detection alone produces the highest counts (up to ~180 questions flagged), LLM-based detection produces intermediate counts (up to ~80 questions), and cross-validated detection produces the most conservative estimates (up to ~50 questions). The cross-validation is necessary because Figure 11 provides qualitative examples of false positives: Case (a) shows a response containing the word "check" in a non-reflective context (simple repetition of a calculation), while Case (b) shows a response flagged by GPT-4o-mini as self-reflective that upon inspection is merely a lengthy correct solution without genuine error recognition or backtracking.

Self-reflection and accuracy correlation in DeepSeek-R1-Zero (Appendix F, Figure 15): For questions that elicit at least one self-reflective response from DeepSeek-R1-Zero across 8 trials, 100 responses are sampled and split into with/without self-reflection groups. The accuracy difference between groups per question shows a distribution approximately centered on zero (mean slightly negative at approximately -0.05, median near 0). The paper describes this as "nearly half [of] responses with self-reflection do not achieve higher accuracy than those without self-reflection." This is a robustness check on the claim that self-reflection is not causally linked to accuracy—it verifies that the pattern holds not just for base models but also for the fully RL-trained R1-Zero model.

Response category analysis for DeepSeek-V3-Base vs. R1-Zero (Appendix F, Figure 14 and Table 5): Analysis of 500 MATH questions across five difficulty levels reveals that R1-Zero substantially increases the proportion of correct responses compared to the base model across all difficulty levels. However, the average length of incorrect responses is longer than that of correct responses in both models: R1-Zero incorrect responses average 8,206.1 tokens versus 4,965.4 tokens for correct responses. The paper hypothesizes this is because harder questions produce longer responses and are also more likely to be answered incorrectly, creating a composition effect. This finding is consistent with the GRPO-length-bias narrative: if RL algorithms artificially encourage long incorrect responses, R1-Zero's incorrect responses would be expected to show disproportionate length growth. Table 5 also shows that "unformatted" responses (those failing to follow the specified output format) increase substantially from the base model (880.7 average tokens) to R1-Zero (7,870.3 average tokens), consistent with the observation in Liu et al. (2025b).

Generalization of bias findings beyond MATH-12K training: While the primary Dr. GRPO comparison (Section 3.2, Figure 5) uses MATH-12K as training data, the template × question set experiment (Section 3.3, Figure 6) uses Dr. GRPO across all configurations and confirms that the unbiased optimizer works effectively across diverse question sets and templates without requiring retuning. The Llama experiment (Section 3.4, Figure 7) further validates Dr. GRPO on a different model family and scale. The bias ablation (Appendix C, Figure 8) uses a different training mixture (ASDiv + MATH + AIME pre-2023), confirming robustness to training data composition.

Critical Assessment

Where the Evidence Strongly Supports the Claims

Claim: Base models, including DeepSeek-V3-Base, already exhibit self-reflection before RL. The evidence for this claim is direct and multi-method. The paper hosts DeepSeek-V3-Base-685B (filling the critical gap left by prior work), generates 500 × 8 × 10 = 40,000 responses, and applies a dual-detection methodology (keyword + LLM with cross-validation) that reduces false positives. The qualitative examples in Figure 13 (Appendix E) provide face validity—the model produces "Aha!" and "Wait, I'm overthinking. Let's try again" in its responses. This is not a subtle statistical finding requiring careful interpretation; the behaviors are present in the raw outputs. The cross-family replication (Qwen2.5, Llama-3.1, DeepSeek) shows the finding is not model-specific.

Claim: Qwen2.5 base models perform better without templates. Table 1 provides unambiguous evidence: 38.2% with no template vs. 23.8% with 4-shot prompting for the 7B model. The effect is replicated at 1.5B scale (33.1% vs. 19.7%) and holds across five diverse benchmarks. The paper's hypothesis about pretraining on concatenated QA text is circumstantial (Qwen2.5-Math's technical report, Yang et al., 2024b, mentions using chat model data during pretraining), but the empirical pattern is robust regardless of its cause.

Claim: GRPO's length bias artificially inflates incorrect-response length. Figure 5 (Plot 4) shows a stark effect: Dr. GRPO's incorrect responses stay at ~500 tokens while GRPO's grow to ~1,600-1,800 tokens—a ~1.2-1.4K token gap. The ablation in Figure 8 confirms that removing length normalization (but not std normalization) is sufficient to eliminate most of this growth. The three-seed replication (Figure 9) shows statistical robustness. However, one caveat: the comparison uses the R1 template with Qwen2.5-1.5B, and the bias magnitude might differ with other templates or base models. The Llama replication (Figure 7, right) shows the effect generalizes, but only to one additional model scale.

Claim: Dr. GRPO maintains or improves reasoning performance while improving token efficiency. Across Figures 5, 7, 8, and 9, Dr. GRPO matches or slightly exceeds GRPO's benchmark accuracy while producing substantially shorter responses. The performance gains are modest (1-3 percentage points) and within the range of seed variation, so the primary benefit is efficiency rather than accuracy. The paper's characterization of this as "better token efficiency while preserving reasoning performance" is well-supported; a stronger claim of "improving reasoning performance" would rely on small, potentially non-significant differences.

Where the Evidence Is Incomplete or Overstated

The "minimalist recipe achieves state-of-the-art" claim (43.3% AIME 2024, Figure 2, Table 4). This claim is technically correct but requires important qualifications. First, the comparison against other 7B models is confounded by generation budget: Oat-Zero-7B uses a 3K token budget (due to Qwen2.5-Math's 4K context length), while R1-Distill-Qwen-7B at 8K budget achieves 33.3% AIME (Table 4) and OpenReasoner-Zero at 8K achieves 13.3%—but both are reported as underperforming Oat-Zero at the 3K constraint. The paper reports the 8K numbers in Table 4 and is transparent about this, but the Figure 2 bar chart comparison does not clearly communicate the budget mismatch to a casual reader. Second, Oat-Zero-7B's superiority over PRIME-Zero-7B (51.4% vs. 48.0% average) is real but the methods are not isolated ablations—PRIME-Zero uses implicit process rewards and a different training setup, so it's unclear whether Dr. GRPO, the template choice, the question set selection, or some combination drives the difference. Third, the Qwen2.5-Math-7B base model already achieves 26.5% average with the Qwen-Math template and no RL (Table 4, "Qwen2.5-Math-7B" row), meaning RL contributes approximately 24.9 percentage points to reach 51.4%—substantial but far from the "RL from scratch" narrative that the R1-Zero phenomenon originally suggested.

The claim that RL gains are largest when model-template mismatch is greatest (Section 3.3). Figure 6 supports this qualitatively but the analysis is limited by the absence of a key control: training the base model with the R1 template using SFT before RL, to establish whether SFT alone can recover the template-mismatch performance gap. The paper shows that RL can close the gap (R1 template + ORZ-57K reaches ~40%, comparable to Qwen-Math template + GSM-8K at ~45-50%), but doesn't demonstrate that RL is necessary for this recovery—template adaptation might be achievable through simple behavioral cloning on correctly formatted examples, without RL.

The Llama-3.2-3B domain pretraining experiment (Section 3.4, Figure 7). While the result is directionally clear (more math pretraining → higher RL ceiling), the experiment conflates two variables: (1) the amount of math-specific pretraining data and (2) the format of that data (FineMath as standard pretraining vs. NuminaQA as concatenated QA text). The paper hypothesizes that the QA-format pretraining (NuminaQA) is what drives Qwen2.5's template-free performance, and the Llama experiment partially tests this by comparing FineMath (no QA format) with NuminaQA (QA format). However, the NuminaQA model also receives more total pretraining (FineMath + 2 epochs NuminaQA vs. FineMath alone), so the data quantity and format are confounded. An ablation where the same total tokens are allocated to FineMath-style vs. QA-style pretraining would be needed to isolate the format effect.

The cross-validation methodology for self-reflection detection is rigorous but has an unacknowledged blind spot. The keyword pool (Appendix D) consists of 15 terms, all of which are variations on explicit self-correction ("recheck," "reevaluate," etc.). This pool is "small" and "highly selective" by design, which reduces false positives at the cost of potentially missing implicit self-reflection that doesn't use these specific words—the LLM-based detector is supposed to catch these cases. However, the cross-validation requirement (both detectors must agree) might systematically undercount implicit self-reflection that uses more subtle language, since keyword detection will miss it entirely. If DeepSeek-V3-Base is more likely to produce implicit rather than explicit self-reflection (the paper notes it avoids certain keywords like "re-evaluate"), the counts in Figure 3 (right) might underestimate its true self-reflection rate. The paper acknowledges that "it may still be insufficient to identify some implicit behaviors," but this limitation affects the quantitative interpretation of "how much" self-reflection exists in each model, not the qualitative finding that it does exist.

The paper doesn't explore whether length bias in PPO implementations has affected published results. The finding that TRL, OpenRLHF, and VeRL all contain length bias (Table 2, Listing 1) is a major contribution suggesting that published RLHF results may have been affected by this artifact. However, the paper doesn't test this: it doesn't replicate any prior work with the corrected implementation to quantify how much of previously reported performance was due to the bias. This would be a natural follow-up but is beyond the paper's scope, which focuses on R1-Zero-like training specifically. The claim that "this formulation-implementation misalignment was present even before the publication of GRPO" (Section 3.1) is observational (based on code inspection) rather than experimentally validated.

Missing Experiments That Would Strengthen the Paper

• A direct comparison of PPO (with and without the corrected masked_mean) to Dr. GRPO at scale. The paper shows that Dr. GRPO fixes both GRPO-specific biases and general PPO length bias, but doesn't isolate whether the PPO bias alone (in a standard PPO-with-value-network setup) is sufficient to produce the length-growth phenomenon at the 7B scale. All main comparisons use GRPO/Dr. GRPO without value networks.

• Evaluation of the "minimalist recipe" at the 3K vs. 8K generation budget for the Oat-Zero models themselves. The paper reports Oat-Zero results only at 3K and compares against baselines at both 3K and 8K, but doesn't report Oat-Zero performance at 8K to establish whether the recipe benefits from longer generation or is saturated at 3K.

• Difficulty-stratified analysis of RL gains. The paper analyzes base model self-reflection by difficulty (Appendix F, Figure 14) but doesn't break down RL accuracy gains by question difficulty. This would reveal whether RL improves the model on all difficulty levels equally (consistent with genuine reasoning improvement) or primarily on easier questions (consistent with format adaptation).

• Training data ablation isolating question format vs. difficulty. The GSM-8K finding (simple questions with Qwen-Math template ≈ best performance) could be explained either by simplicity (easy questions are sufficient to reinforce existing reasoning) or by format (GSM-8K's word-problem format matches what Qwen2.5-Math saw during pretraining). An experiment training on the hardest MATH questions only would help distinguish these hypotheses.

• Comparison of the concatenated-QA continual pretraining approach against SFT. Llama-3.2-3B-NuminaQA is described as "continual pretraining" on concatenated QA texts, but the distinction from SFT (where the model is trained to predict answers given questions) is unclear when the data format is simply concatenated [question][answer]—the model is still learning to predict answer tokens conditioned on question tokens. An SFT baseline on the same data would clarify whether the "pretraining" framing (vs. "instruction tuning") matters for downstream RL performance.

Summary Assessment

The paper's core experimental contributions—that self-reflection exists in base models including DeepSeek-V3-Base, that Qwen2.5 models perform better without templates, and that GRPO contains a length bias that Dr. GRPO corrects—are well-supported by direct, multi-method evidence. The strength of the experimental design lies in its breadth (testing across model families, scales, templates, and question sets) rather than depth in any single configuration. The primary limitation is that the paper's interpretative claims (e.g., about what drives template-dependent RL dynamics, about the pretraining origins of Qwen2.5 behavior, about the causal role of self-reflection) rely on observational patterns rather than controlled mechanistic experiments. The paper is transparent about several of these limitations (e.g., the pretraining hypothesis is explicitly labeled as a hypothesis, Section 2.2), but readers should treat the explanatory narrative as suggestive rather than definitive. The experimental results are most convincing at the level of "what happens" (empirical regularities) and less so at the level of "why it happens" (causal mechanisms).

6. Limitations and Trade-offs

The Minimalist Recipe's State-of-the-Art Claim Depends on Generation-Budget Mismatches With Baselines

The assumption or constraint. The paper's headline result—Oat-Zero-7B achieving 43.3% on AIME 2024 and 51.4% average accuracy, "establishing a new state-of-the-art" (Section 1, Figure 2)—is evaluated at a 3,000-token generation budget because "we employ the Qwen2.5-Math base models, which have a context length of 4k" (Appendix B). However, several competing methods were designed for and achieve substantially better performance at 8K token budgets. The paper reports both 3K and 8K numbers for some baselines in Table 4 but the Figure 2 bar chart—the visual centerpiece for the SOTA claim—does not distinguish between budget levels or note the constraint. Specifically, R1-Distill-Qwen-7B achieves 54.7% average at 8K versus only 28.5% at 3K, and OpenReasoner-Zero-7B achieves 45.9% at 8K versus 43.0% at 3K.

The consequence. A practitioner selecting a model for deployment would need to know whether Oat-Zero-7B's advantage persists when all models are evaluated under their optimal generation budgets. If Oat-Zero-7B saturates near 3K tokens (as suggested by the Qwen2.5-Math base model's 4K context limit), then models capable of leveraging 8K contexts might match or exceed its performance when given their full budget. The paper does not report Oat-Zero-7B performance at 8K (e.g., by using a base model with longer context or by evaluating with extended generation), leaving this comparison incomplete. The SOTA claim is therefore qualified: it holds under a specific budget constraint that disadvantages some baselines more than others. A fairer comparison would report all models at their native optimal budgets, or would report Oat-Zero at 8K to establish whether the gains are budget-robust.

What evidence exists in the paper. Table 4 (Appendix B) acknowledges the budget discrepancy by including both 3K and 8K results for R1-Distill and OpenReasoner-Zero. The Oat-Zero results are reported only at 3K. The paper notes "we thus limit the generation budget at 3k for all baselines compared" and "for models that are trained for a longer context...we also report their performance at 8k generation budget." The asymmetry—reporting baselines at both budgets but their own model at only one—means the full picture requires readers to cross-reference Table 4 against Figure 2, which does not visually encode the budget distinction.

Mitigation status. The paper partially addresses this through transparency in Table 4, but the Figure 2 bar chart (the primary result visualization) does not reflect the budget caveat. No experiment tests whether Oat-Zero benefits from extended generation budgets comparable to its competitors. This is a scope limitation—the Qwen2.5-Math base model's 4K context is a hard constraint—but it means the SOTA claim should be understood as budget-conditional rather than absolute.

Difficulty Estimation Cost Is Not Accounted for in Template and Question-Set Selection

The assumption or constraint. The paper demonstrates that template selection—specifically, using no template for Qwen2.5-Math models—dramatically affects both initial and final RL performance (Table 1, Figure 6). The optimal template is identified by running a diagnostic experiment: generating responses with and without templates and measuring answering rate and pass@8. Similarly, the best question set for a given template is identified post-hoc by running the full 12-way template × question-set experiment (Figure 6). In a deployment context, a practitioner with a new base model would need to replicate this diagnostic to determine the optimal template and question set—but the paper does not account for the cost of this exploratory phase in its efficiency claims.

The consequence. The paper reports that the final minimalist recipe requires "only 27 hours compute on 8× A100 GPUs" (Section 1). This figure includes only the final RL training run, not the cost of determining that (a) no template is best for Qwen2.5-Math, (b) Dr. GRPO outperforms GRPO, (c) MATH level 3-5 questions are sufficient, and (d) the Qwen-Math template (rather than no template) produces the best final accuracy when combined with question-set selection. Each of the 12 curves in Figure 6 represents a separate RL training run, and the base model diagnostics (Figure 3) require generating 40,000+ responses across temperatures and templates. The total compute to arrive at the recipe likely exceeds the recipe's own cost by a large multiple, making the "27 hours" figure a lower bound that would not be replicable by someone starting from scratch with an uncharacterized base model.

What evidence exists in the paper. The paper reports individual experiment costs qualitatively ("all our experiments are performed on 8 × A100 GPUs and finished in about one day," Appendix G) but never aggregates the total exploration cost or amortizes it over the final recipe. The template × question-set experiment alone involves 12 training runs (Figure 6), each comparable in cost to the final 27-hour run. The base model diagnostics require generation at scale: 500 questions × 8 responses × 10 temperatures = 40,000 responses per model, plus LLM-based judging via GPT-4o-mini.

Mitigation status. Not addressed. The paper treats template and question-set selection as part of the analysis rather than part of the recipe cost. A practitioner might reduce this overhead by running smaller-scale diagnostics (e.g., evaluating on 50 questions rather than 500, sweeping fewer temperatures for pass@8), but the paper does not validate whether reduced diagnostics would reliably identify the same optimal configuration.

The Paper Does Not Demonstrate That Self-Reflection Behaviors Are Causally Irrelevant to RL Training—Only That They Do Not Correlate With Inference Accuracy

The assumption or constraint. One of the paper's central claims is that the "Aha moment" is a "pretraining artifact rather than an emergent RL phenomenon" (Section 1, Takeaways). The evidence for this claim has two components: (1) base models, including DeepSeek-V3-Base, already generate self-reflection patterns before RL (Figure 3, right panel; Figure 13), and (2) self-reflection in DeepSeek-R1-Zero is not positively correlated with higher accuracy during inference (Appendix F, Figure 15). However, these two findings address different causal questions. Finding (1) shows that self-reflection exists pre-RL. Finding (2) shows that self-reflection during inference is not predictive of correctness. Neither finding addresses whether self-reflection behaviors during RL training contribute to exploration and policy improvement, which is the mechanism through which the Aha moment is hypothesized to matter in the original DeepSeek-R1-Zero narrative (Guo et al., 2025).

The consequence. The paper's reframing—that the Aha moment is a pretraining artifact—could be interpreted as implying that self-reflection is epiphenomenal and irrelevant to RL success. This would be an overstatement of the evidence. It is possible that self-reflection behaviors, while already present in the base model, serve an important exploratory function during training: a model that sometimes backtracks and re-evaluates its solutions may discover correct reasoning paths that a purely forward-generating model would miss, even if those self-reflective trajectories are not more likely to be correct than non-reflective ones when sampled at inference time. If this is the case, then RL is not creating self-reflection but is amplifying its use in ways that improve exploration—which would still represent a meaningful (though more nuanced) emergent phenomenon. The paper acknowledges this possibility in passing: "acknowledging that self-reflection may improve exploration during training—a potential positive effect outside this section's scope" (Appendix F), but the main text and takeaways do not carry this qualification, instead emphasizing that "self-reflection does not necessarily imply higher accuracy" (Section 2.3, Takeaways).

What evidence exists in the paper. Figure 15 (Appendix F) shows that for DeepSeek-R1-Zero, the accuracy difference between responses with and without self-reflection is approximately centered on zero, with a mean near -0.05. This is compelling evidence that self-reflection during inference is not a reliable correctness signal. However, the paper does not conduct any experiment that tests whether self-reflection during training matters: for instance, comparing RL training with a model variant that is somehow prevented from generating self-reflective patterns (e.g., via constrained decoding or by training on a dataset where self-reflection tokens are masked), or analyzing whether questions that elicit self-reflection during early training show larger accuracy gains than those that do not.

Mitigation status. The paper partially mitigates this by including the caveat in Appendix F that the inference-stage analysis does not address training-stage effects. However, the strength of the language in the main text ("Aha Moment Already Appears in Base Models Including DeepSeek-V3-Base") and the Takeaways ("Nearly all base models already exhibit the 'Aha moment'") implies a stronger conclusion than the evidence supports regarding the functional role of self-reflection. A reader who does not carefully study Appendix F could reasonably conclude that self-reflection is irrelevant to R1-Zero's success, which is not established.

The Pretraining-Format Hypothesis for Qwen2.5 Behavior Is Compelling but Untested

The assumption or constraint. The paper's explanation for why Qwen2.5 base models achieve peak performance without templates rests on a specific causal hypothesis: "they might pretrain on concatenated question-answer texts when cooking the models" (Section 1, Takeaways), i.e., that Qwen2.5-Math models were exposed to instruction-formatted data during pretraining where questions and answers appear as contiguous text, making raw questions the optimal prompt format. The paper explicitly labels this as a hypothesis ("we hypothesize that..."; "If our hypothesis turns out true") but the strength of the downstream claims—that we should "be more careful about using Qwen2.5 models to reproduce DeepSeek-R1-Zero, since the base models are already SFT-like without templates" (Section 2.2)—depends on this hypothesis being correct.

The consequence. If the hypothesis is wrong—if Qwen2.5-Math's template-free performance arises from a different mechanism (e.g., the model's pretraining distribution happened to contain sufficient raw math problem text that it learned to answer questions without explicit instruction formatting)—then the paper's recommendation to avoid Qwen2.5 models for R1-Zero reproduction may be overly cautious. The models might still represent legitimate base models that just happen to have strong math priors, and RL on them might still demonstrate the same principles as RL on a "pure" base model, just starting from a higher baseline. More importantly, if the mechanism is different from the one hypothesized, the paper's suggestion that concatenated QA pretraining (as applied to Llama-3.2-3B-NuminaQA in Section 3.4) is the right way to improve RL ceilings for other models might target the wrong intervention.

What evidence exists in the paper. The evidence is entirely correlational: Qwen2.5-Math models perform best without templates (Table 1), they achieve 100% answering rates without templates (Figure 3, left panel), and the Qwen2.5-Math technical report (Yang et al., 2024b) "uses chat model's data (question-answer pairs) during the pretraining stage" (Section 2.2). The Llama experiment (Section 3.4, Figure 7) shows that concatenated QA pretraining (NuminaQA) improves the RL ceiling compared to standard math pretraining (FineMath), which is consistent with the hypothesis but confounds data quantity with data format (the NuminaQA model received FineMath pretraining plus 2 additional epochs of NuminaQA data). The paper does not conduct the direct test: comparing two Llama variants pretrained on the same total token count, one in standard pretraining format (math documents) and one in concatenated QA format, to isolate whether the QA format specifically drives the improvement.

Mitigation status. The paper is transparent that this is a hypothesis ("we hypothesize," Section 2.2; "If our hypothesis turns out true," Section 2.2; "likely to be pretrained," Section 3.4) and does not present it as established fact. However, the practical recommendations in the paper (using Qwen2.5 models with no template, applying concatenated QA pretraining to improve other models' RL ceilings) are built on this hypothesis without alternative explanations being seriously considered. The paper does not discuss, for instance, whether Qwen2.5-Math's template-free performance might arise from its architecture, tokenizer, or specific pretraining data mixture rather than from concatenated QA formatting specifically.

The Method Has Only Been Validated on Math Reasoning With Outcome-Verifiable Rewards

The assumption or constraint. All experiments in the paper use mathematical reasoning benchmarks (AIME, AMC, MATH500, Minerva Math, OlympiadBench) where correctness is determined by a deterministic rule-based verifier (Math-Verify) that checks whether the generated text contains the correct final answer string. The reward function is binary and outcome-based: $R(q, o) = 1$ if the final answer matches, 0 otherwise. This setup relies on two properties that do not generalize to many important LLM applications: (1) correctness is objectively verifiable through string matching against a known ground-truth answer, and (2) partial reasoning or intermediate steps are not rewarded—only the final answer matters. The paper does not test the approach on tasks where correctness is ambiguous (e.g., summarization quality, dialogue helpfulness), multi-dimensional (e.g., code generation where both correctness and style matter), or where intermediate reasoning steps carry value independent of the final output (e.g., scientific explanation, legal analysis).

The consequence. A practitioner considering Dr. GRPO for non-math domains cannot infer from this paper whether the approach will transfer. Several aspects of the method are specifically adapted to the math-reasoning-with-outcome-reward setting and may fail elsewhere:

• Dr. GRPO removes KL regularization ($\beta = 0$), justified because "RL-tuning reasoning models typically employs rule-based verifiers as $r$, eliminating the concerns of distributional shift" (Section 3). In domains where a learned reward model is necessary (because ground-truth labels don't exist at scale), removing KL regularization could cause the policy to drift into regions where the reward model is inaccurate, degrading performance (Jaques et al., 2019; Stiennon et al., 2020).
• The binary outcome reward provides no signal about response quality beyond final-answer correctness. In domains where outputs have continuous quality (e.g., translation adequacy, summarization coverage), binary rewards would be too sparse for effective RL, and the Dr. GRPO advantage estimator (which uses only group-mean centering of binary rewards) might not provide sufficient gradient signal.
• The finding that Qwen2.5-Math models perform best without templates is specific to models pretrained on math data with concatenated QA formatting. In other domains (creative writing, dialogue, code generation), base models almost certainly require templates to produce useful outputs, and the template-mismatch analysis would not directly apply.

What evidence exists in the paper. None beyond the math domain. The paper does not include experiments on code generation (e.g., HumanEval, MBPP), general reasoning (e.g., ARC, BigBench), or any non-math task. The evaluation benchmarks (Table 4) are exclusively math competitions and math problem sets. Even within math, the paper does not test on proof-based problems or problems requiring multi-modal inputs (despite OlympiadBench being described as "multimodal" in He et al., 2024; the paper's evaluation appears to use text-only questions).

Mitigation status. Not addressed. The paper's scope is explicitly mathematical reasoning ("R1-Zero-like training" as defined by DeepSeek-R1-Zero's original setting), and the authors do not claim generalization to other domains. However, a practitioner reading the paper might reasonably assume that Dr. GRPO and the minimalist recipe represent general improvements to RL tuning of LLMs rather than domain-specific optimizations. The paper does not discuss which properties of math reasoning (deterministic verification, binary correctness, the specific pretraining characteristics of Qwen2.5-Math) might be necessary for the approach to work, leaving the burden of assessing transferability entirely on the reader.

The 3,000-Token Generation Budget Constrains the Analysis of Length-Growth Phenomena

The assumption or constraint. All experiments enforce a maximum response length of 3,000 tokens (Table 6), motivated by the Qwen2.5-Math base models' 4K-token context window (which must also accommodate the prompt). The Dr. GRPO vs. GRPO comparison (Figure 5), the template × question-set experiment (Figure 6), and the Llama pretraining experiment (Figure 7) all operate under this constraint. However, the length-growth phenomenon that the paper critiques—the "double-increase" of reward and response length during R1-Zero training—is most dramatic in settings where the generation budget is larger, as in the original DeepSeek-R1-Zero work where responses could reach tens of thousands of tokens. The 3,000-token ceiling may truncate or mask behaviors that would appear at longer generation lengths.

The consequence. The paper's finding that Dr. GRPO reduces incorrect-response length from approximately 1,600-1,800 tokens to approximately 400-500 tokens (Figure 5, Plot 4) is measured with a hard ceiling at 3,000 tokens. This means the comparison captures length differences within the budget but cannot distinguish between (a) Dr. GRPO genuinely preventing length growth versus (b) Dr. GRPO simply causing length growth to saturate at a lower value while GRPO would eventually also saturate (or continue growing) at budgets beyond 3,000 tokens. If the optimization bias primarily affects the rate of length growth rather than the asymptotic length, then the absolute gap between GRPO and Dr. GRPO might narrow at larger budgets as both approach natural ceilings determined by problem complexity. Conversely, if GRPO's bias causes unbounded length growth (the optimizer continually exploits the length-normalization loophole), then the gap would widen at larger budgets, and the 3,000-token experiments actually understate the bias's severity. The paper cannot distinguish these scenarios because the budget ceiling is binding.

Additionally, the 3,000-token limit may affect the self-reflection analysis. Self-reflection behaviors are more likely in longer responses (the model has more tokens in which to reconsider its approach), so the observation that DeepSeek-V3-Base and R1-Zero both exhibit self-reflection might partly reflect that both models are given sufficient generation budget to produce these patterns, rather than indicating that self-reflection is equally prevalent in both. A systematic analysis of how self-reflection frequency varies with generation budget is not conducted.

What evidence exists in the paper. Figure 5, Plots 2-4, show length trajectories up to iteration 150 for GRPO and Dr. GRPO. GRPO's incorrect-response length appears to still be increasing at the end of training, suggesting it has not saturated at the 3,000-token ceiling. However, the training reward (Plot 1) has begun to plateau, and it is possible that length growth would decelerate in later iterations. The paper does not report what fraction of responses hit the 3,000-token ceiling in either condition, which would indicate whether the budget is actively constraining behavior. For R1-Zero analysis (Appendix F), Table 5 reports average response lengths of approximately 4,965 (correct) and 8,206 (incorrect) tokens for DeepSeek-R1-Zero—well above the 3,000-token budget used in the paper's own experiments—confirming that R1-Zero's behavior occurs at a scale the paper's experiments do not reach.

Mitigation status. The paper acknowledges the context-length constraint in Appendix B: "we thus limit the generation budget at 3k for all baselines compared." However, it does not discuss how this constraint might affect the interpretation of its length-growth findings or whether the Dr. GRPO advantage would be larger, smaller, or unchanged at longer generation budgets. The choice of 3,000 tokens is pragmatic (determined by the base model's context window) but limits the generality of conclusions about response-length dynamics, which are central to the paper's critique of R1-Zero-like training.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper represents a methodological correction and diagnostic reframing, not a paradigm shift in the Kuhnian sense. It does not propose a new training paradigm, a new model architecture, or a fundamentally different approach to reasoning. Rather, it changes the landscape by introducing a new category of analysis—implementation auditing of RL objectives for LLMs—that reveals how deeply the gap between mathematical specification and code-level implementation can shape the phenomena the field has been interpreting as evidence of algorithmic progress.

The magnitude of the shift is best understood along three dimensions:

First, the paper reframes what counts as "understanding" in the R1-Zero literature. Prior to this work, the community's approach to understanding R1-Zero was primarily through reproduction: implement the recipe with different base models, different datasets, different infrastructure, and verify that the same qualitative phenomena (reward increase, length increase, emergence of self-reflection) appear. The paper demonstrates that this form of understanding is inadequate because it confounds genuine algorithmic effects with implementation artifacts and pretraining biases. The key methodological contribution is showing that the right unit of analysis is not "does the training curve look right?" but rather "what would the training curve look like if we removed each component in turn?"—a decompositional approach that separates the contributions of base model properties, template choice, question set composition, and optimization bias. This reframing, if adopted, would change how future work evaluates R1-Zero-like training: rather than reporting a single training run with a single configuration, the expectation would become to report ablations that establish which components are necessary versus incidental.

Second, the paper partially resolves the tension between two competing narratives in the R1-Zero literature. On one side, the original DeepSeek-R1-Zero paper (Guo et al., 2025) and its replications (Zeng et al., 2025; Hu et al., 2025) presented the "Aha moment"—the emergence of self-reflection behaviors during RL—as evidence that pure RL can create sophisticated reasoning strategies from simple outcome rewards. On the other side, Liu et al. (2025b) and Yeo et al. (2025) argued that open-source replications did not exhibit genuine emergence because their base models already contained self-reflection patterns. The paper resolves this tension by completing the missing empirical link: it directly tests DeepSeek-V3-Base-685B (the model from which the original R1-Zero was trained, which prior critics had not examined) and finds that it too already generates self-reflection behaviors. This eliminates the possibility that the original R1-Zero represented a qualitatively different, "pure emergence" case. The resolution is not that self-reflection doesn't happen—it does, in both base models and RL-trained models—but that it is not created by RL, and its presence during RL training should not be interpreted as evidence of a capability that RL generated from nothing. The paper further refines this resolution by showing that even in the fully-trained R1-Zero model, self-reflection is not causally associated with higher accuracy during inference (Figure 15), challenging the functional significance of the behavior. This effectively closes the "does the Aha moment emerge?" debate and redirects the question toward "what, if anything, does self-reflection contribute to training dynamics?"—a more productive framing that the paper leaves open for future work.

Third, the paper makes certain research directions more attractive while reducing enthusiasm for others. On the "more attractive" side: (a) auditing and correcting implementation-level biases in RL objectives becomes a priority, since the paper shows that a simple fix (changing one line of the masked_mean function) produces measurably different training dynamics; (b) careful characterization of base models before RL training becomes essential, since the paper shows that base model properties—template sensitivity, native question-answering format, existing self-reflection patterns—determine the interpretation of RL gains; (c) domain-specific pretraining as a lever for improving RL ceilings becomes a clear research investment, since the Llama-3.2-3B experiment (Figure 7) demonstrates a 3× improvement in final RL performance from math-domain pretraining alone, with identical RL algorithm and hyperparameters. On the "less attractive" side: (a) engineering more complex RL algorithms for reasoning (e.g., process reward models, sophisticated exploration bonuses) becomes lower priority relative to fixing basic optimizer correctness—the paper shows that even GRPO and PPO, the simplest and most widely-used algorithms, contain unexamined biases; (b) reporting single-configuration RL training curves as evidence of successful reproduction becomes insufficient—the paper demonstrates that the same qualitative phenomena (length increase, reward increase) can arise from optimization bias, and that a proper demonstration of RL benefit must control for this confound; (c) the narrative that RL "creates" reasoning capabilities de novo loses credibility—the paper's evidence that reasoning patterns, question-answering format, and self-reflection all exist in base models shifts the burden of proof to anyone claiming emergent capabilities, requiring them to demonstrate that the behavior was absent pre-RL under adequate sampling budgets.

Follow-Up Research This Work Enables

Auditing and correcting length bias in published RLHF results. The paper's discovery that TRL, OpenRLHF, and VeRL all contain the same $1/|o_i|$ length bias in their PPO implementations (Table 2, Listing 1) raises a pressing question: how much of previously reported RLHF performance is attributable to this bias rather than to the algorithms themselves? A direct follow-up would replicate a sample of influential RLHF papers—spanning summarization (Stiennon et al., 2020), dialogue (Jaques et al., 2019), and instruction-following—using both the original biased masked_mean implementation and the corrected constant-normalization version, measuring whether response length, reward, and win-rate against baselines change. This is feasible because the correction requires changing a single line of code (Listing 1) and the original training configurations are publicly available in many cases (since they use TRL or OpenRLHF). The key question is not whether the bias exists (the code inspection is unambiguous) but whether its effect size is large enough to have materially affected conclusions. The paper's own experiments show large effects on response length (~1,200-token gap for incorrect responses, Figure 5) but modest effects on accuracy (1-3 percentage points). In RLHF settings where reward models are trained on human preference data, the length bias could have a different manifestation: human annotators often prefer longer, more detailed responses (a well-known confound in RLHF data collection), so the bias—which pushes the policy toward longer incorrect responses (or less-penalized incorrect responses) and shorter correct ones—might interact with annotation artifacts in ways not captured by the binary-reward math setting. A comprehensive audit would distinguish between settings where the bias is consequential and settings where it is negligible, providing guidance to practitioners about whether switching to the corrected implementation is worth the engineering cost.

Does self-reflection during training improve exploration, even if it doesn't predict inference accuracy? The paper acknowledges (Appendix F) that its analysis of self-reflection and accuracy is limited to the inference stage, and that self-reflection "may improve exploration during training—a potential positive effect outside this section's scope." This is a sharply defined follow-up question that the paper makes newly tractable by providing the detection methodology (keyword-LLM cross-validation) and by establishing that base models already produce self-reflection at non-trivial rates. A strong experiment would instrument the RL training loop to track, for each question in each training batch, whether the sampled responses contain self-reflection, and then analyze whether questions that elicit self-reflective trajectories during early training show larger accuracy improvements over the course of training compared to questions that elicit only forward-reasoning trajectories. If self-reflection during training improves exploration, the mechanism would likely be that backtracking and re-evaluating intermediate steps sometimes leads the model to correct paths it would otherwise miss—even if, after training, the model's inference-time self-reflection is no more accurate than its forward reasoning (because the exploration benefits have already been incorporated into the policy's default behavior). This would reconcile the paper's null finding on inference-time self-reflection with the possibility that self-reflection is nonetheless important for training, and would provide a more nuanced picture than either "the Aha moment is emergent" or "the Aha moment is irrelevant." An alternative hypothesis—that self-reflection is purely epiphenomenal and neither training nor inference causal—could be tested by training a variant where self-reflection tokens are masked from the policy gradient (so the model can still produce them but receives no RL signal for doing so), and comparing the final policy quality against the unmasked baseline.

Why does the Qwen-Math template enable out-of-distribution transfer (GSM-8K → AIME) while the R1 template does not? The paper's template × question-set experiment (Figure 6) reveals a striking and counterintuitive pattern: with the Qwen-Math template, training on GSM-8K (simple grade-school math) achieves the best final performance on competition benchmarks, substantially outperforming training on MATH-12K or ORZ-57K. With the R1 template, this pattern inverts—GSM-8K underperforms broader question sets. The paper's interpretation is that when the template matches the base model's pretraining format, RL is "reinforcing useful reasoning behaviors instead of infusing new knowledge," but the specific mechanism is unclear. A mechanistic follow-up would analyze the model's responses at different stages of training under both templates, categorizing the types of reasoning patterns that emerge (step-by-step decomposition, verification, alternative approaches) and whether these patterns differ qualitatively between the GSM-8K-trained and MATH-12K-trained policies. A strong test of the "reinforcement vs. infusion" hypothesis would be to measure whether the GSM-8K-trained policy generates substantially different reasoning structures (e.g., more systematic step-by-step breakdowns, more explicit checking) compared to the MATH-12K-trained policy when evaluated on the same competition problems, despite having never seen problems of that difficulty during training. If true, it would suggest that the Qwen-Math template + Qwen2.5-Math base model combination enables a form of reasoning-schema transfer that is independent of problem difficulty—a finding with practical implications for data curation (train on easy, diverse problems; evaluate on hard ones) and theoretical implications for what RL contributes (amplifying general reasoning patterns rather than problem-specific strategies).

Does the concatenated QA pretraining format cause the Qwen2.5 template-free performance, and can it be replicated for other model families? The paper hypothesizes that Qwen2.5-Math's template-free performance arises because it was pretrained on concatenated question-answer texts, but the Llama-3.2-3B experiment (Section 3.4) that tests this confounds data format with data quantity: the NuminaQA model received FineMath pretraining plus 2 additional epochs of NuminaQA data. A clean experiment would pretrain Llama-3.2-3B on a fixed token budget (e.g., 10B tokens) drawn either from standard math-domain documents (FineMath-style) or from concatenated question-answer pairs (NuminaQA-style), controlling for total tokens, and then evaluate both the template-free answering rate and the downstream RL ceiling. The prediction from the paper's hypothesis is that the QA-pretrained model should show higher template-free performance and a higher RL ceiling, because the QA format more closely matches the inference task. If this prediction holds, it would provide a recipe for improving RL readiness of any base model through data formatting alone, without changing total compute—a finding of substantial practical importance given the growing interest in post-training optimization. If the prediction fails (both formats produce similar RL ceilings), it would suggest that domain knowledge, not data format, is the binding constraint, and that Qwen2.5-Math's unusual behavior has alternative explanations (e.g., specific architectural choices, multi-stage training that included instruction tuning, or contamination with benchmark-formatted data).

Can the Dr. GRPO correction be extended to process-reward and learned-reward settings? The paper's Dr. GRPO removes KL regularization (β=0) because the reward function is a deterministic rule-based verifier for which distributional shift is not a concern (Section 3). However, if Dr. GRPO is to be applied in settings requiring learned reward models (e.g., summarization, dialogue, creative writing) or process rewards (e.g., PRIME's implicit process rewards from token-level rollout values, Cui et al., 2025), the interaction between the unbiased optimizer and KL regularization needs characterization. Specifically, if the length bias in standard PPO implementations was partially acting as an implicit regularizer—by down-weighting long responses, it might have accidentally prevented the policy from drifting too far into regions where the reward model is inaccurate—then removing the bias without adding KL regularization could degrade performance in learned-reward settings. A systematic experiment would train models with Dr. GRPO + learned reward model across a range of KL coefficients (β from 0 to typical RLHF values of 0.01-0.1), on a task where ground-truth quality can be evaluated independently of the learned reward (e.g., summarization with ROUGE as a proxy, or dialogue with human evaluation). The prediction is that Dr. GRPO should still improve token efficiency (shorter responses for comparable quality) but may require non-zero KL regularization to maintain reward model accuracy—finding the Pareto frontier of (β, response length, quality) would provide practical guidance for adoption beyond math reasoning.

Stress-test: does Dr. GRPO's advantage hold at generation budgets beyond 3,000 tokens? The paper's experiments are constrained by the Qwen2.5-Math base model's 4K context window, capping generation at 3,000 tokens. The length-growth phenomenon that the paper critiques—the "double-increase" of reward and response length—is most dramatic in settings with larger generation budgets (the original DeepSeek-R1-Zero produced responses of tens of thousands of tokens, and Table 5 of this paper reports R1-Zero incorrect responses averaging 8,206 tokens). A stress-test would apply Dr. GRPO to a base model with longer context (e.g., Llama-3.1-8B with 128K context, DeepSeek-V3-Base with its native context length) and train with a generation budget of 8K, 16K, or 32K tokens, measuring whether (a) Dr. GRPO's length-reduction effect scales proportionally with budget or saturates at some natural ceiling, (b) GRPO's bias causes length to grow without bound as budget increases (consistent with an exploitable optimization loophole), and (c) the accuracy gap between Dr. GRPO and GRPO widens, narrows, or stays constant with budget. This experiment would determine whether the paper's 3,000-token results underestimate or overestimate the practical importance of the bias correction in the long-generation regimes where R1-Zero-like training is typically deployed. The paper's own evidence (Figure 5, Plot 4, showing GRPO's incorrect-response length still increasing at iteration 150) suggests the bias effect has not saturated at 3,000 tokens, implying that longer-budget experiments would reveal larger absolute gaps.

Practical Applications and Downstream Use Cases

Correcting open-source RLHF infrastructure. The paper identifies a concrete, one-line bug in the masked_mean function used by TRL, OpenRLHF, and VeRL—the three most widely-used open-source RLHF frameworks. The fix (replacing mask.sum(axis=dim) with a constant MAX_TOKENS in the loss denominator, Listing 1) is trivial to implement and has been validated across model scales (1.5B, 3B, 7B), model families (Qwen2.5, Llama-3.2), and training configurations (GRPO and PPO variants). Any team using these frameworks for post-training—whether for reasoning, instruction-following, safety alignment, or preference optimization—can adopt this fix immediately with no additional computational cost and a clear expectation of reduced response-length inflation, particularly for incorrect outputs. The quantitative benefit from the paper: Dr. GRPO reduces incorrect-response length by approximately 1,200-1,400 tokens (Figure 5, Plot 4) while maintaining or slightly improving benchmark accuracy. For production deployments where inference cost scales with response length (token-based API pricing, GPU memory and latency constraints), this directly translates to cost savings. For a deployment processing millions of queries per day, reducing average response length by 30-50% (the gap between GRPO's ~1,600-token and Dr. GRPO's ~500-token incorrect responses) represents substantial infrastructure savings. The fix is backward-compatible with existing training pipelines—it requires changing only the loss computation, not the data, model architecture, or hyperparameter schedules.

Base model diagnostics before RL training investment. The paper's three-dimensional diagnostic protocol (answering rate, pass@8 exploration, self-reflection detection) provides a practical screening tool for teams deciding whether a given base model is a good candidate for R1-Zero-like training. The protocol can be executed in hours on modest hardware: generating responses to a few hundred questions at a few temperature settings, checking answer format and pass@K, and scanning for self-reflection keywords. The paper's results provide benchmarks for what "good" looks like: answering rates below ~60-70% without templates indicate a model that requires careful template engineering; pass@8 rates below ~30-40% suggest limited exploration capability that will constrain RL ceiling; and the presence or absence of self-reflection patterns can be documented pre-RL to avoid later misattribution. For a team evaluating a new base model (e.g., a domain-specific model pretrained on scientific literature, or a multilingual model for cross-lingual reasoning), running this diagnostic before committing to weeks of RL training could prevent investments in configurations where RL is unlikely to succeed (e.g., a model with near-zero pass@8 on the target domain, where no amount of RL can create correct trajectories from nothing). The paper also provides a practical heuristic: if the base model performs better without a template than with one (as Qwen2.5-Math does), the template should probably be omitted during RL training, since applying it will unnecessarily depress initial performance and create an artifactual "gain" when RL recovers baseline capabilities.

Template and question-set selection for cost-efficient RL training. The paper's finding that GSM-8K (simple math, 8K questions) can produce better final performance on competition benchmarks than ORZ-57K (diverse math, 57K questions) when using the Qwen-Math template (Figure 6, middle panel) has direct implications for training data curation. Training on 8K questions instead of 57K reduces per-iteration data generation cost by approximately 7×, and training on simpler questions reduces per-response generation length (simpler problems typically require shorter solutions), further reducing compute. For a team with a base model that already possesses domain knowledge (as Qwen2.5-Math does for math), this suggests a strategy of training on the simplest available dataset that still provides correct-answer signal, rather than the most diverse or most difficult dataset—counter to the prevailing intuition that broader coverage is always better. Conversely, for a base model that is mismatched to its template (e.g., applying the R1 template to a model not pretrained on conversational formats), the paper shows that broad question-set coverage is essential (Figure 6, left panel), and the training data budget should be allocated to diversity rather than difficulty. The practical decision rule emerging from the paper is: diagnose the model-template match first (via the answering-rate diagnostic), and choose question-set coverage based on the degree of mismatch. This is a concrete, actionable workflow that does not require the full 12-way experiment the paper conducted—the diagnostic can be run once, and the question-set strategy can be selected accordingly.