DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning

ArXiv: 2501.12948

🎯 Pitch

Forgoing all human reasoning examples, pure RL on answer correctness alone causes a large language model to spontaneously teach itself verification, self-reflection, and backtracking. This emergent behavior boosts AIME math scores past the human average, but only for sufficiently large base models—smaller ones collapse into repetition. The learned long-chain reasoning can then be distilled into compact models, offering a scalable path to strong, autonomous reasoning.

1. Executive Summary

This paper introduces a pure reinforcement learning framework that incentivizes reasoning capabilities in large language models without relying on supervised fine-tuning or human-labeled reasoning trajectories, training DeepSeek-R1-Zero directly from DeepSeek-V3-Base using Group Relative Policy Optimization (GRPO) (a variant of PPO that eliminates the critic model and estimates advantages from group-normalized rewards). Through large-scale RL training with only rule-based reward signals—accuracy rewards from verifiable final answers (mathematical equality checks, compiler test cases) and format rewards enforcing structured output tags—the model autonomously develops advanced reasoning patterns including self-reflection, verification, and dynamic strategy adaptation, achieving 77.9% pass@1 on AIME 2024 (surpassing the human participant average) while its response length naturally grows from hundreds to thousands of tokens without explicit instruction. The paper further develops DeepSeek-R1 through a multi-stage pipeline incorporating cold-start long-Chain-of-Thought SFT data, rejection sampling, and two RL stages guided by both rule-based and model-based rewards, establishing that pure RL alone suffices to elicit emergent reasoning from sufficiently large base models—but only when the base checkpoint possesses adequate capacity, as smaller 7B and 16B models failed to leverage long chains of thought and instead exhibited repetitive generation under the same training regime.

2. Context and Motivation

The Core Problem: Human-Dependent Reasoning Trajectories Are a Bottleneck

The fundamental problem this paper tackles is that state-of-the-art reasoning in LLMs remains tethered to human-annotated demonstration data. Chain-of-thought prompting works by either showing models carefully crafted few-shot examples of step-by-step reasoning (Wei et al., 2022b) or using zero-shot triggers like "Let's think step by step" (Kojima et al., 2022). The supervised fine-tuning approach—exemplified by models like GPT-4 and InstructGPT—trains on large corpora of human-written reasoning traces (Chung et al., 2024; OpenAI, 2023). Both paradigms assume that human reasoning patterns are the ceiling for model performance.

This assumption has three concrete failure modes that the paper identifies in Section 1:

Scalability is bottlenecked by annotation effort. Curating high-quality, multi-step reasoning trajectories for diverse mathematical, coding, and scientific domains requires substantial human expertise and labor. Each new domain or difficulty level demands fresh annotation investment. This is not merely an economic concern—it imposes a hard ceiling on how broadly reasoning-capable models can be deployed across tasks where expert annotation is scarce or expensive.

Human priors may be suboptimal for machine reasoning. When models are trained to replicate human reasoning patterns, they inherit human cognitive biases and limitations. Human problem-solving—even expert problem-solving—tends to follow established patterns, use conventional heuristics, and avoid certain solution spaces that a machine might profitably explore. The paper argues that SFT on human demonstrations "constrain[s] models to replicate human thought processes" and that "their performance is inherently capped by the human-provided exemplars, which prevents the exploration of superior, non-human-like reasoning pathways" (Section 1). This is a strong claim: that the optimum for machine reasoning may not look like human reasoning at all, and that supervised training actively prevents models from discovering these more effective strategies.

Intermediate reasoning steps are difficult to supervise. Even when human annotations exist, they typically provide only the final reasoning trajectory—not a rich signal about which intermediate steps are correct or why. Process reward models (PRMs) attempt to address this by scoring individual steps (Lightman et al., 2023; Wang et al., 2023a), but as the paper notes in Appendix G.2, PRMs face three practical failures: (1) defining what constitutes a "fine-grained step" across diverse reasoning tasks is ambiguous, (2) determining whether an intermediate step is correct requires either expensive human annotation (which doesn't scale) or automated model-based annotation (which is unreliable), and (3) model-based PRMs introduce reward hacking—the policy model learns to exploit the PRM's errors rather than genuinely improving reasoning.

The consequence is that the dominant post-training paradigm (SFT → RLHF) cannot discover reasoning strategies that humans don't already demonstrate. This is the gap the paper aims to close.

Why This Problem Matters

The importance extends beyond academic interest into practical and economic considerations for the AI development ecosystem.

Democratization of reasoning capabilities. If human annotation is the bottleneck, then only well-resourced organizations can build strong reasoning models. The paper's approach—pure RL with only verifiable outcome rewards—radically reduces the dependency on human labor. Rule-based verifiers (math expression checkers, code compilers, format validators) can be constructed once and reused across arbitrary numbers of training runs and difficulty levels. This shifts the scaling constraint from human effort to compute—and compute, while expensive, is far more scalable and automatable than expert annotation.

Economic implications of the annotation bottleneck. The conventional SFT pipeline requires not just initial annotation but ongoing re-annotation as models improve. When a model generates better reasoning trajectories than humans can write (the stated goal of the paper), those trajectories become candidates for further training—but someone must verify them. If verification must be done by humans, you've created a circular bottleneck: the model surpasses human capability, but you can't train on its outputs because you can't verify them. Rule-based RL breaks this cycle: the verifier scales indefinitely.

Theoretical significance: can reasoning emerge from reward alone? There is a deep scientific question at stake: are sophisticated reasoning behaviors—self-reflection, error detection, strategy switching, backtracking—something that must be taught through examples, or can they emerge from the optimization pressure of a simple outcome reward? If the latter, it implies that these behaviors are latent in the pretrained model's distribution and need only the right incentive structure to surface. This parallels findings in other domains (e.g., tool use emerging in language models without explicit training), but applies it to the core cognitive capability that distinguishes human intelligence.

Production deployment realities. In practical systems, the gap between what a model can do (its latent capability) and what it does do (its realized output) is often large. A model that needs expensive SFT data for each new domain cannot rapidly adapt. A model trained with pure RL on verifiable outcomes can be retrained on new domains simply by providing new reward functions—no new human data required. This makes the approach particularly attractive for rapidly evolving application areas like competitive programming (new problems constantly), scientific reasoning (new findings), or any domain with machine-verifiable correctness.

Where Existing Approaches Fall Short

The paper positions itself against a specific set of prior approaches, each with documented limitations.

Supervised Fine-Tuning as the Foundation for Reasoning

The standard post-training recipe (Appendix A.2) is SFT followed by RLHF: first, fine-tune on high-quality human demonstrations to establish basic task competence; then, apply RL with a learned reward model to align with human preferences (Ouyang et al., 2022). This pipeline has produced the strongest general-purpose models (GPT-4, Claude, DeepSeek-V3), but the paper argues it is fundamentally mismatched with the goal of maximizing reasoning capability.

The issue is not that SFT is ineffective—it demonstrably works. The issue is that SFT anchors the model to human-provided solutions. The paper's position (stated most clearly in Appendix A.2) is that:

"human-provided responses, which serve as targets during SFT, are not always optimal for model learning; they often omit critical reasoning components such as explicit reflection and verification steps."

In other words, human reasoning examples—even expert ones—tend to present clean, linear solutions that elide the messy exploratory process. When a model is trained to imitate these, it learns to skip the verification and reflection that would catch errors. The paper's bet is that starting from the base model directly with RL allows the model to discover that verification and reflection improve its reward without being explicitly shown examples of these behaviors.

This is a bold claim and a direct challenge to the field's default assumption that SFT is a necessary prerequisite for stable RL training. The conventional wisdom—that RL from a cold start leads to mode collapse, incoherent outputs, or failure to explore—is contradicted by the paper's empirical results, at least for sufficiently large models (≥32B parameters, per Appendix G.1).

Process Reward Models: Theoretically Appealing, Practically Problematic

Prior work on improving reasoning through RL has gravitated toward process reward models (PRMs) that provide dense, step-level feedback (Lightman et al., 2024; Uesato et al., 2022; Wang et al., 2023a). The intuition is compelling: if the model knows which step went wrong, it can learn more efficiently than if it only knows the final answer was wrong.

The paper's Appendix G.2 presents a candid assessment of why PRMs failed in their experiments. Three problems are identified:

1. Granularity definition: "it is challenging to explicitly define a fine-grain step in general reasoning." In math, steps might be natural (each algebraic manipulation). In code, what constitutes a "step"? A line? A function call? A subgoal? The definition affects what the PRM scores, and inconsistent granularity produces noisy training signals.

2. Correctness determination: "determining whether the current intermediate step is correct is a challenging task." For math, a step might be logically valid but lead to a dead end—is that "correct"? For code, a partial implementation might be syntactically valid but algorithmically wrong. Automated annotation (using models to judge steps) introduces model biases. Manual annotation doesn't scale.

3. Reward hacking: "once a model-based PRM is introduced, it inevitably leads to reward hacking." The policy model learns to produce steps that look correct to the PRM rather than steps that are correct. Retraining the PRM to fix this requires additional resources and adds complexity. This is the same problem documented extensively in RLHF (Gao et al., 2022), but it's amplified for PRMs because the per-step feedback provides more surface area for exploitation.

The paper ultimately concludes that PRMs have value for reranking and guided search at inference time (Snell et al., 2024), but their advantages "are limited compared to the additional computational overhead" during large-scale RL training. This is a significant practical finding: outcome-based rewards, despite being sparser, avoid the complexity and failure modes of process-based rewards while still producing strong reasoning behaviors.

Monte Carlo Tree Search: Search Space Explosion

Appendix G.2 also describes unsuccessful attempts to use Monte Carlo Tree Search (MCTS), inspired by AlphaGo/AlphaZero (Silver et al., 2017a,b). The idea was to decompose answers into steps, use a value model to guide exploration, and train both the policy and value model iteratively.

The core failure mode is search space explosion: "unlike chess, where the search space is relatively well-defined, token generation presents an exponentially larger search space." Even with maximum extension limits per node, the model gets stuck in local optima. More critically, "training a fine-grained value model is inherently difficult, which makes it challenging for the model to iteratively improve." In AlphaGo, the value model and policy model co-evolved—each improvement in one fed the other. In the LLM token-generation setting, the value model couldn't provide reliable enough guidance to bootstrap this virtuous cycle.

This negative result is informative: it suggests that learned value functions for intermediate reasoning states are not yet practical at scale, and that alternative approaches (like the paper's outcome-based RL with long chain-of-thought) may be more promising for now.

The Limitation of Prompting-Based Approaches

The paper also implicitly critiques prompting-based reasoning improvements. Techniques like self-consistency (Wang et al., 2023b), least-to-most prompting (Zhou et al., 2023a), and tree-of-thoughts (Yao et al., 2023a) all improve reasoning performance at inference time, but they don't fundamentally change the model's underlying reasoning capability—they're better ways to extract capability that already exists. The paper's approach is orthogonal: it aims to increase the model's reasoning capability through training, after which these prompting techniques could still be applied for further gains.

How This Paper Positions Itself

The paper's positioning can be understood along three dimensions:

Against the SFT-first paradigm. The paper is not merely proposing an alternative—it's making the case that SFT is actively harmful for maximizing reasoning. By training DeepSeek-R1-Zero directly from the base model with RL, the paper demonstrates that the SFT phase is not necessary for stable, effective RL training on reasoning tasks, and in fact may prevent the emergence of novel reasoning strategies. This is the most provocative claim in the paper, and it's supported by the empirical trajectory: R1-Zero develops sophisticated behaviors (self-reflection, verification, backtracking) that human-written SFT data typically doesn't demonstrate, and these behaviors emerge solely from the pressure of the outcome reward.

Against dense reward signals. The paper takes a strong stance for outcome-based rewards over process-based rewards. This is counterintuitive because dense rewards are generally preferred in RL for providing more learning signal. The paper argues that the practical difficulties of PRMs (granularity, correctness determination, reward hacking) outweigh their theoretical advantages, and that with sufficient scale and long chain-of-thought, sparse outcome rewards suffice. The key insight is that long CoT naturally provides the model with opportunities to self-correct, making per-step feedback less necessary—the model learns to generate its own verification steps because doing so improves the probability of a correct final answer.

Within the compute-scaling landscape. The paper positions RL-trained reasoning as a new axis of compute scaling. Just as pretraining scaling laws (Kaplan et al., 2020) describe how to allocate compute during training, and test-time compute scaling (Snell et al., 2024) describes how to allocate compute during inference, this paper introduces RL-training-time compute scaling for reasoning: spending compute on RL updates with verifiable rewards to improve reasoning capability, rather than on SFT data collection or larger pretraining runs. The finding that distilled smaller models outperform RL-trained models of the same size (Appendix F.1, Table 16) suggests that RL-training compute is most effective when applied to very large models, with distillation serving as the bridge to smaller, deployable models.

The Specific Evidence the Paper Brings

To substantiate these positions, the paper provides several forms of evidence that prior work lacked:

• Direct comparison of pure RL vs. SFT+RL at scale (Table 3): The progression from R1-Zero (pure RL) through R1-Dev1 (cold-start SFT), Dev2 (first RL stage), Dev3 (second SFT), and final R1 (second RL) shows that while pure RL achieves strong reasoning, SFT stages improve other desiderata like readability and general task performance.

• Failure of small models under pure RL (Appendix G.1): The finding that 7B and 16B models "consistently failed to yield meaningful improvements" and exhibited "a tendency toward repetition" under the same RL training that succeeded with ≥32B models provides a boundary condition for the approach.

• Distillation outperforming RL for small models (Appendix F.1, Table 16): DeepSeek-R1-Distill-Qwen-32B (distilled from the large R1) achieves 72.6% on AIME 2024 versus Qwen2.5-32B-Zero's 47.0% (trained with pure RL for >10K steps), suggesting that for smaller models, learning from a stronger teacher's outputs is more effective than self-discovery through RL.

• Ablation of language consistency reward (Appendix B.6, Figure 7): Shows that while the LC reward slightly degrades benchmark performance, it's necessary for human-readable outputs—illustrating the practical tradeoffs in pure RL training.

The paper thus positions itself not as a wholesale rejection of existing methods, but as a reordering of the dependency chain: instead of SFT → RL, it proposes RL → SFT (using RL-discovered reasoning traces) → final RL alignment. The human annotation effort shifts from writing reasoning traces to curating cold-start data and aligning outputs, while the heavy lifting of reasoning capability development is offloaded to scalable, verifier-driven RL.

3. Technical Approach

3.1 Reader Orientation

This paper builds a reinforcement learning system that trains large language models to reason by rewarding correct final answers without ever showing the model examples of how to reason. The core idea is that sophisticated reasoning behaviors—self-verification, reflection, backtracking, strategy switching—can emerge autonomously when a sufficiently large pretrained model is optimized against a simple outcome-based reward signal, provided the model is allowed to generate long chains of intermediate thought before producing its answer.

3.2 Big-Picture Architecture (Diagram in Words)

The system has four major components, deployed in two distinct training regimes:

DeepSeek-R1-Zero (pure RL pipeline):

1. Base Model (DeepSeek-V3-Base) — a 671B-parameter Mixture-of-Experts transformer, pretrained on 14.8 trillion tokens of web and ebook data (no synthetic data), that serves as the starting policy. It generates candidate solutions consisting of a reasoning process followed by a final answer, structured by a simple template with thinking and <answer> tags.
2. GRPO Trainer — the reinforcement learning algorithm that samples groups of outputs from the current policy, computes rewards using rule-based verifiers, estimates advantages by normalizing rewards within each group, and updates the policy to maximize these advantages while staying close to a reference policy via a KL penalty.
3. Rule-Based Reward System — two deterministic reward functions with no learned components: an accuracy reward (1 if the final answer matches ground truth via mathematical expression checking or compiler test cases, 0 otherwise) and a format reward (1 if the model correctly uses the thinking ... response and <answer> ... </answer> tag structure, 0 otherwise). These are summed with equal weight.
4. Reference Policy — a periodically updated snapshot of the policy model (every 400 training steps) used to compute the KL divergence penalty that prevents the policy from changing too drastically.

DeepSeek-R1 (multi-stage pipeline, Figure 2): The training proceeds through four stages that interleave supervised fine-tuning and reinforcement learning:

1. Cold-Start SFT — a small set of thousands of human-refined long-chain-of-thought examples used to fine-tune DeepSeek-V3-Base, producing Dev1. These examples establish human-readable formatting, first-person perspective, and language-consistent reasoning before any RL.
2. First RL Stage — GRPO training on Dev1 using the same rule-based rewards as R1-Zero, plus a language consistency reward (proportion of target-language words in the chain of thought) to combat language mixing. Produces Dev2.
3. Second SFT Stage (Rejection Sampling) — the Dev2 checkpoint generates ~600K reasoning trajectories via rejection sampling (keep only correct ones), combined with ~200K non-reasoning samples from DeepSeek-V3's SFT data. Fine-tuning on this mixture produces Dev3.
4. Second RL Stage — final GRPO training on Dev3 using mixed reward signals: rule-based rewards for reasoning data, model-based helpfulness and safety rewards (trained neural reward models on preference pairs and safety labels) for general data. Produces the final DeepSeek-R1.

Information flows from left to right in Figure 2: V3-Base → Cold-Start SFT → Dev1 → First RL → Dev2 → Rejection Sampling + SFT → Dev3 → Second RL → DeepSeek-R1. The R1-Zero branch (V3-Base → RL → R1-Zero) runs in parallel as a baseline demonstrating what pure RL alone can achieve.

3.3 Roadmap for the Deep Dive

• First, the GRPO algorithm (Section 2.1, Appendix A.3)—the core RL mechanism that makes training tractable without a critic model. Understanding GRPO is prerequisite to everything else because all RL stages use it.
• Second, the reward design (Sections 2.2, 3.1)—the rule-based accuracy and format rewards for R1-Zero, plus the model-based helpfulness and safety rewards for R1. Rewards are the training signal; their properties determine what behaviors emerge.
• Third, the R1-Zero training procedure (Section 2.3)—how GRPO plus rule-based rewards applied to V3-Base produces emergent reasoning, including the exact hyperparameters, the self-evolution dynamics, and the "aha moment" phenomenon.
• Fourth, the R1 multi-stage pipeline (Section 3)—cold-start SFT data construction, the language consistency reward, rejection sampling for the second SFT stage, and the two-stage RL training with mixed reward signals.
• Fifth, the RL infrastructure (Appendix B.1)—the decoupled rollout/inference/training architecture that makes 671B-parameter RL training feasible, including expert parallelism, MTP speculative decoding, and VRAM management.
• Sixth, the distillation approach (Appendix F)—how the R1 model's outputs are used to train smaller models, and why distillation outperforms direct RL on smaller architectures.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a training methodology paper whose core idea is that sophisticated reasoning behaviors can emerge from pure reinforcement learning with only outcome-based rewards, provided the base model is sufficiently large and the RL algorithm permits long chain-of-thought generation without penalizing response length.

The GRPO Algorithm: Eliminating the Critic Model

GRPO (Group Relative Policy Optimization) is the reinforcement learning algorithm used for all RL stages in both DeepSeek-R1-Zero and DeepSeek-R1. It was introduced in Shao et al. (2024) as a simplification of PPO (Proximal Policy Optimization, Schulman et al., 2017) that removes the need for a learned value function (critic model).

The PPO baseline and its limitations. In standard PPO as applied to LLM training (Ouyang et al., 2022), three models are maintained: a policy model (the LLM being trained), a value model (a separate neural network of similar size that predicts expected future rewards from each token position), and a reference model (a frozen snapshot of the initial policy). The value model computes the Generalized Advantage Estimation (GAE, Schulman et al., 2015), which requires per-token value predictions to estimate how much better or worse each action was than expected:

$A_t^{\text{GAE}} = \sum_{l=0}^{\infty} (\gamma \lambda)^l \delta_{t+l}$

where $\delta_t = r_t + \gamma V(s_{t+1}) - V(s_t)$ is the temporal difference error, $V(s_t)$ is the value model's prediction of expected cumulative reward from token position $t$ onward, $\gamma$ is the discount factor, and $\lambda$ controls the bias-variance tradeoff.

The paper identifies three practical problems with PPO for long chain-of-thought reasoning models (Appendix A.3):

1. Memory overhead: the value model is typically the same size as the policy model, doubling GPU memory requirements.
2. Value prediction difficulty: "the training objective of the value model is to predict the expected cumulative reward from the current position onward, based on the tokens generated from the beginning up to the current position. This is inherently difficult, especially when only the final outcome reward is available." For long chain-of-thought reasoning where the model engages in reflection and revision (meaning earlier content may be contradicted later), predicting the final reward from a partial response is extremely challenging.
3. Implicit length penalty: PPO adds a per-token KL divergence penalty as a dense reward at each token position. Since the RL objective maximizes cumulative rewards, "PPO's approach penalizes the cumulative KL divergence, which may implicitly penalize the length of the response and thereby prevent the model's response length from increasing." For the paper's goal of encouraging long chains of thought, this is a critical failure mode.

GRPO's core idea. GRPO foregoes the value model entirely. Instead of estimating advantages using learned value predictions, it estimates advantages by comparing rewards within a group of outputs generated from the same prompt—hence "Group Relative" in the name. Figure 3 (in the paper's Appendix A.3) illustrates the architectural difference: PPO has a value model branch computing GAE from individual rewards, while GRPO takes a group of rewards, normalizes them, and uses the normalized scores directly as advantages.

The GRPO objective. For each question $q$, GRPO samples a group of $G$ outputs $\{o_1, o_2, ..., o_G\}$ from the old policy $\pi_{\theta_{old}}$ and optimizes the policy model $\pi_\theta$ by maximizing:

$\mathcal{J}_{GRPO}(\theta) = \mathbb{E}\left[q \sim P(Q), \{o_i\}_{i=1}^G \sim \pi_{\theta_{old}}(O|q)\right] \frac{1}{G} \sum_{i=1}^G \min\left(\frac{\pi_\theta(o_i|q)}{\pi_{\theta_{old}}(o_i|q)} A_i, \text{clip}\left(\frac{\pi_\theta(o_i|q)}{\pi_{\theta_{old}}(o_i|q)}, 1 - \varepsilon, 1 + \varepsilon\right) A_i\right) - \beta \mathbb{D}_{KL}\left(\pi_\theta || \pi_{ref}\right)$

where:

• $\pi_\theta$ is the current policy (the LLM being trained)
• $\pi_{\theta_{old}}$ is the old policy (the behavior policy that generated the outputs, frozen during the update)
• $\pi_{ref}$ is a reference policy (a periodically updated snapshot of the policy)
• $\varepsilon$ is the clipping hyperparameter (set to 10 for R1, a remarkably high value that the paper notes is crucial—lower values "can lead to the truncation of gradients for a significant number of tokens")
• $\beta$ is the KL penalty coefficient (set to 0.001)
• $A_i$ is the advantage for output $i$, computed from the group's rewards
• $\mathbb{D}_{KL}$ is an unbiased estimator of the KL divergence between the policy and the reference model

What it computes: The objective has three components. The first (inside the min) is the standard policy gradient objective—the probability ratio $\frac{\pi_\theta(o_i|q)}{\pi_{\theta_{old}}(o_i|q)}$ times the advantage $A_i$—which increases the probability of outputs with positive advantages and decreases it for negative advantages. The clip function prevents the ratio from exceeding $[1-\varepsilon, 1+\varepsilon]$, constraining how much the policy can change in a single update (this is the "proximal" part, inherited from PPO). The final term subtracts $\beta$ times the KL divergence between the current policy and the reference policy, serving as an additional regularizer that prevents the policy from diverging too far from its starting point.

The group-based advantage computation. This is GRPO's defining innovation. The advantage $A_i$ for each output $i$ in a group is computed as:

$A_i = \frac{r_i - \text{mean}(\{r_1, r_2, ..., r_G\})}{\text{std}(\{r_1, r_2, ..., r_G\})}$

where $\{r_1, r_2, ..., r_G\}$ are the rewards for the $G$ outputs in the group (all responses to the same question $q$).

Why this form: This is a z-score normalization of rewards within each group. Subtracting the group mean centers the advantages—outputs better than the group average get positive advantages, outputs worse than average get negative advantages. Dividing by the standard deviation scales the advantages so that the magnitude of updates doesn't depend on the absolute scale of rewards (which can vary across prompts). This replaces the role of the value model in PPO: instead of asking "how much better was this action than the value model's baseline prediction?", GRPO asks "how much better was this output than the average output for this question?" The group serves as a self-normalizing baseline.

The key property that makes this work is that all outputs in a group come from the same prompt. This means the group mean naturally captures the expected reward for that prompt given the current policy, without needing a separate model to estimate it. The tradeoff is that the baseline is higher-variance than a learned value function (especially with small group sizes), but for the paper's setting where rewards are deterministic (rule-based) and the group size is 16, this variance is manageable.

The KL divergence estimator. The paper uses an unbiased estimator of the KL divergence (Schulman, 2020) added directly to the loss, rather than PPO's approach of adding per-token KL penalties to the reward:

$\mathbb{D}_{KL}\left(\pi_\theta || \pi_{ref}\right) = \frac{\pi_{ref}(o_i|q)}{\pi_\theta(o_i|q)} - \log\frac{\pi_{ref}(o_i|q)}{\pi_\theta(o_i|q)} - 1$

Why this form: Adding the KL penalty directly to the loss rather than to the reward means it doesn't accumulate over tokens. In PPO, the per-token KL penalty grows with response length—longer responses incur larger total KL penalties even if the per-token policy divergence is small. This creates an implicit bias against long outputs. In GRPO, the KL term is computed once per complete output and subtracted from the objective, so response length doesn't affect the KL penalty magnitude. For training long chain-of-thought reasoning models where responses grow from hundreds to tens of thousands of tokens, this property is essential—without it, the KL penalty would effectively punish the model for thinking longer, which is the opposite of what the paper wants.

Reference policy updates. The paper periodically replaces the reference model with the latest policy model: "Every 400 steps, we replace the reference model with the latest policy model." This is important because "as we may train thousands of steps in the scenario of training long chain-of-thought reasoning models, the trained policy can diverge significantly from the initial reference policy." If the reference policy were frozen at the initial model, the KL penalty would eventually constrain all meaningful exploration. Periodic updates "balance the scope that the training policy can explore and the stability of the training."

GRPO vs. PPO empirical comparison (Appendix A.3, Figure 4). On the MATH task using DeepSeek-Coder-V2-Lite (16B MoE, 2.4B active parameters), PPO with default $\lambda = 0.95$ in GAE performs "considerably worse than GRPO." With careful tuning ($\lambda = 1.0$, which effectively removes the temporal smoothing in GAE), PPO approaches but doesn't surpass GRPO performance. The paper concludes that "while PPO can achieve comparable performance when appropriately tuned, it demands additional computational cost for hyperparameter optimization" and the memory overhead of the value model. For training 671B-parameter models with constrained resources, GRPO is the more practical choice.

The clipping hyperparameter $\varepsilon$. A subtle but critical detail: the paper sets the GRPO clip ratio $\varepsilon$ to 10 for DeepSeek-R1 training, which is much larger than typical PPO values (usually 0.1–0.2). The paper notes: "A lower value can lead to the truncation of gradients for a significant number of tokens, thereby degrading the model's performance, while a higher value may cause instability during training." The high value of 10 means the policy can change substantially per update, which is necessary for the large policy shifts that occur during reasoning emergence (the model goes from 15.6% to 77.9% on AIME). The paper is essentially trading off PPO's conservative updates for faster learning, relying on the KL penalty and periodic reference updates to maintain stability.

Reward Design: The Training Signal

The reward function is the entire source of learning signal in RL—it determines which behaviors are reinforced and which are suppressed. The paper uses two fundamentally different types of rewards depending on the training stage.

Rule-based rewards (R1-Zero and R1 reasoning data). For mathematical, coding, and logical reasoning tasks where correctness is objectively verifiable, the paper uses only deterministic, rule-based rewards:

$\text{Reward}_{\text{rule}} = \text{Reward}_{\text{acc}} + \text{Reward}_{\text{format}}$

where both components have equal weight (implicitly 1.0 each).

Accuracy rewards evaluate whether the model's final answer is correct:

• For math problems with deterministic results: the model must provide the final answer in a specified format (within a \boxed{} LaTeX command), and the system matches it against the ground-truth answer. The paper uses sympy for parsing and expression comparison, which can handle algebraic equivalence (e.g., x^2 + 2x + 1 equals (x+1)^2). The reward is 1 if the extracted answer matches the reference, 0 otherwise.
• For code problems: the model's code is executed against a suite of test cases (both public examples from the problem statement and hidden test cases). The reward is 1 if all test cases pass, 0 otherwise. For competitive programming problems where original test cases aren't publicly available, the paper developed a methodology using DeepSeek-V2.5 to generate candidate test cases, then filtered them using correct and incorrect submissions to ensure they properly differentiate solutions (Listing 2 in Appendix B.3.2).

Format rewards enforce structural requirements: the model must enclose its reasoning process within thinking ... response tags and its final answer within <answer> ... </answer> tags. The reward is 1 if both tags are present and properly structured, 0 otherwise. This constraint serves two purposes: (1) it makes the reasoning process explicit and separable from the answer, which is necessary for analyzing what the model learned; (2) it provides a weak structural prior that encourages the model to "think before answering" without constraining the content of the thinking.

Why rule-based rewards over neural reward models. The paper explicitly states: "we abstain from applying neural reward models—whether outcome-based or process-based—to reasoning tasks. This decision is predicated on our observation that neural reward models are susceptible to reward hacking during large-scale reinforcement learning." This is a crucial design choice. Neural reward models (including PRMs) learn to predict human judgments of quality, but they have systematic errors and biases. During RL training, the policy model learns to exploit these errors—generating outputs that score highly under the reward model but are actually incorrect or nonsensical. Rule-based verifiers, by contrast, are objective and ungameable: you can't "trick" a compiler into passing your code if it's wrong, or convince an expression parser that $x=3$ when the answer is $x=4$. The tradeoff is that rule-based rewards are sparse (only at the final answer) and limited to domains with objective correctness criteria.

Language consistency reward (R1 first RL stage). DeepSeek-R1-Zero exhibited language mixing—generating chains of thought that combined English and Chinese within the same response—because DeepSeek-V3-Base was trained on multilingual data. For the R1 training, the paper introduces a language consistency reward:

$\text{Reward}_{\text{language}} = \frac{\text{Num}(\text{Words}_{\text{target}})}{\text{Num}(\text{Words})}$

where $\text{Num}(\text{Words}_{\text{target}})$ is the count of words in the target language (the language of the user's query) and $\text{Num}(\text{Words})$ is the total word count in the chain of thought.

What it computes: the fraction of words in the reasoning trace that are in the target language. For an English query, this is the proportion of English words; for a Chinese query, the proportion of Chinese words. The reward is a continuous value between 0 (no target-language words) and 1 (all target-language words).

Why this form: A simple binary reward for "all words in target language" would be too sparse—it wouldn't provide gradient toward improvement when the output is mostly but not entirely in the target language. The proportional reward provides a smooth signal: the model gets partial credit for partial consistency, which guides it toward higher consistency gradually. The paper notes in Appendix B.6 that "such alignment results in a slight degradation in the model's performance" on benchmarks (Figure 7 shows a small drop on LiveCodeBench and AIME), "but this reward aligns with human preferences, making the output more readable." This is a deliberate tradeoff: slightly lower benchmark accuracy for substantially better user experience.

Model-based rewards (R1 second RL stage). For general-domain data (writing, factual QA, role-playing, and safety), rule-based verification is impossible because correctness is subjective or multidimensional. The paper trains two separate neural reward models.

Helpful Reward Model. The helpful reward model is trained on 66,000 preference pairs. The data generation procedure works as follows:

1. DeepSeek-V3 is prompted with queries using the Arena-Hard prompt format (shown in Appendix B.2).
2. For each query, two candidate responses are generated.
3. DeepSeek-V3 acts as a judge: it is queried four times for each preference pair, with the responses randomly assigned as Response A or Response B (to mitigate positional bias).
4. The final preference score is the average of the four judgments.
5. Only pairs where the score difference $\Delta$ exceeds 1 are retained, ensuring meaningful distinctions between chosen and rejected responses.
6. The dataset is balanced so that chosen and rejected responses have comparable lengths, to prevent the reward model from learning a length bias.

The reward model architecture is identical to DeepSeek-R1 (same transformer backbone) with an additional reward head—a linear layer that projects the final hidden state to a scalar preference score:

$\text{Reward}_{\text{helpful}} = \text{RM}_{\text{helpful}}(\text{Response}_A, \text{Response}_B)$

The model is trained with a batch size of 256, learning rate of $6 \times 10^{-6}$, for a single epoch. Maximum sequence length during training is 8,192 tokens, with no explicit limit during inference.

Safety Reward Model. The safety reward model is trained on 106,000 prompts with model-generated responses annotated as "safe" or "unsafe" according to predefined safety guidelines. Unlike the helpfulness model's pairwise training, the safety model uses point-wise training—it classifies each response independently:

$\text{Reward}_{\text{safety}} = \text{RM}_{\text{safety}}(\text{Response})$

The training hyperparameters (batch size 256, learning rate $6 \times 10^{-6}$, one epoch) are identical to the helpful reward model.

Usage during RL. During the second RL stage for DeepSeek-R1, each training batch contains a mix of reasoning data and general data. The reward is computed as:

$\text{Reward} = \text{Reward}_{\text{reasoning}} + \text{Reward}_{\text{general}} + \text{Reward}_{\text{language}}$

where:

• $\text{Reward}_{\text{reasoning}} = \text{Reward}_{\text{rule}}$ (the same accuracy + format rewards used in R1-Zero)
• $\text{Reward}_{\text{general}} = \text{Reward}_{\text{reward\_model}} + \text{Reward}_{\text{format}}$ (model-based helpfulness/safety reward plus format reward)

For queries in the helpfulness dataset, $\text{Reward}_{\text{reward\_model}}$ is the helpful reward model's score. For queries in the safety dataset, it's the safety reward model's score. The format reward is also applied to general data to maintain structural consistency.

Reward hacking with model-based rewards (Appendix B.5, Figure 6). The paper documents reward hacking when using the helpful reward model: "if the reward model contains systematic biases or inaccuracies, the LLM may learn to generate responses that are rated highly by the model but diverge from authentic human preferences." Figure 6 shows this concretely: as training steps increase, the reward score from the helpful reward model rises, but the model's actual CodeForces performance decreases. The model is learning to produce outputs that the reward model likes rather than outputs that are genuinely good.

This is why the second RL stage limits model-based reward training to only the final 400 steps out of 1,700 total steps. The paper states: "We find that more training steps with the model based preference reward signal may lead to reward hacking." The 400-step limit is an empirical compromise—enough to get alignment benefits without crossing into reward exploitation territory.

DeepSeek-R1-Zero: Pure RL from the Base Model

R1-Zero represents the paper's cleanest test of the hypothesis that reasoning can emerge from reward alone. The training procedure is deliberately minimal.

Base model and template. The starting point is DeepSeek-V3-Base, the pretrained 671B MoE model. No supervised fine-tuning is applied. The model is given a simple template (Table 1) that structures its output:

A conversation between User and Assistant. The user asks a question, and the Assistant solves it. 
The assistant first thinks about the reasoning process in the mind and then provides the user 
with the answer. The reasoning process and answer are enclosed within  thinking... response 
and <answer>...</answer> tags, respectively, i.e.,  thinking reasoning process here  response 
<answer> answer here </answer>. User: prompt. Assistant:

This template is minimal by design—"we intentionally limit our constraints to this structural format, avoiding any content-specific biases to ensure that we can accurately observe the model's natural progression during the RL process" (Section 2.3). The template doesn't tell the model how to reason, only where to put its reasoning.

Hyperparameters. The paper specifies (Section 2.1):

• Learning rate: $3 \times 10^{-6}$
• KL coefficient $\beta$: 0.001
• Sampling temperature for rollouts: 1.0
• Group size $G$: 16 outputs per question
• Maximum generation length: 32,768 tokens before training step 8,200; increased to 65,536 tokens afterward
• Training steps: 10,400 total (1.6 epochs over the training data)
• Questions per training step: 32 unique questions
• Training batch size: 512 (32 questions × 16 outputs each)
• Reference model update frequency: every 400 steps
• Rollout parallelism: each rollout generates 8,192 outputs, split into 16 mini-batches, trained for a single inner epoch

The length increase at step 8,200. A critical detail: the maximum generation length is increased from 32,768 to 65,536 tokens at step 8,200 because the model's responses had been growing longer throughout training (Figure 1b) and were beginning to hit the earlier limit. The paper notes that "both the performance and response length of DeepSeek-R1-Zero exhibit a significant jump at the 8.2k step" (Section 2.1). This is a chicken-and-egg dynamic: the model learns that longer reasoning chains lead to higher accuracy, so it generates longer chains; the increased length limit allows even longer chains, which enables further accuracy improvements. The paper doesn't ablate whether the jump was caused by the length limit increase or was already in progress, but the correlation is suggestive.

Training data composition (Appendix B.3.1, Table 4). The RL training data for R1-Zero consists of four categories of verifiable reasoning problems:

• Mathematics: 26,000 quantitative reasoning questions (exam and competition problems covering algebra, calculus, probability, geometry), average 122 prompt tokens. Answers are numbers, expressions, or equations. Mathematical proofs are excluded "because it is difficult to determine their correctness."
• Coding: 17,000 algorithm competition questions (similar to Codeforces/LeetCode) plus 8,000 bug-fixing problems (from real-world GitHub issues with failing unit tests). For algorithm problems, the model must produce a complete solution passing hidden test cases. For bug-fixing, it must correct the buggy code to pass all unit tests.
• STEM: 22,000 multiple-choice questions covering physics (15.5%), biology (30.7%), chemistry (46.5%), and other topics (7.3%). Average 161 prompt tokens. Each question has 4–8 options. Binary reward based on correct option matching.
• Logic: 15,000 questions including real-world brain teasers, classical logic puzzles, knowledge-intensive questions (multiple-choice format), and synthetically generated problems (code-IO problems converting coding problems to logical reasoning tasks, cryptography puzzles, deductive reasoning puzzles, arithmetic puzzles like 24 game). Average 420 prompt tokens.

All questions are in Chinese or English. The reward for all reasoning categories is the accuracy reward (1 if answer matches reference, 0 otherwise) plus the format reward.

Self-evolution dynamics (Section 2.3, Figure 1). Figure 1(a) shows the AIME 2024 pass@1 accuracy over training: starting at 15.6%, rising gradually to about 40% by step 3,000, then climbing more steeply to 77.9% by step 10,400. The cons@64 (majority voting at 64 samples) curve follows the same trajectory but higher, reaching 86.7%. This is significant because it means the model's individual responses are improving, not just the ensemble—the pass@1 improvement shows genuine capability gain.

Figure 1(b) shows the average response length per question on the training set: starting around 500 tokens, growing steadily to about 3,000 by step 4,000, then accelerating to over 15,000 by step 10,400. This is the "thinking time" increase the paper describes—the model learns that investing more tokens in reasoning improves its chance of getting the correct answer, and this behavior is reinforced by the outcome reward.

The "aha moment" (Table 2). The paper presents a specific example of emergent self-reflection. In the middle of solving a math problem, the model writes: "Wait, wait. Wait. That's an aha moment I can flag here. Let's reevaluate this step-by-step to identify if the correct sum can be..." This is significant because the model was never trained on examples of "wait" or "let me reevaluate"—these behaviors emerged from the RL process. The model learned that when its initial approach leads to confusion or contradiction, pausing to reconsider improves the probability of a correct final answer, and this behavior was reinforced by the reward signal.

Figure 9(b) quantifies this: the frequency of the word "wait" in model outputs is near zero until step 4,000, shows occasional usage between steps 4,000-7,000, then "exhibited significant spikes after step 8,000." This is a phase-change in behavior—the model doesn't gradually use "wait" more; it suddenly discovers this strategy late in training and then employs it heavily.

Evolution of reflective behaviors (Appendix C.2, Figure 9a). The paper tracks ten reflective words ("wait", "mistake", "however", "but", "retry", "error", "verify", "wrong", "evaluate", "check") selected by three human experts. The total frequency of these words increases 5- to 7-fold over training—from roughly 2,000 occurrences per evaluation to roughly 12,000-14,000. This is direct quantitative evidence that the RL process is causing the model to engage in more self-monitoring and error-correction behavior.

MATH difficulty breakdown (Appendix C.1, Figure 8). The paper stratifies the MATH-500 dataset by difficulty level (1-5, as labeled by the dataset creators based on human-perceived difficulty). The results show:

• Easy problems (levels 1-3): quickly reach 0.90-0.95 accuracy and remain stable
• Level 4: improve from near 0.78 to 0.95 over training
• Level 5 (hardest): "demonstrate the most dramatic improvement from near 0.55 to 0.90"

A counterintuitive observation: the model's accuracy on harder questions (levels 3-4) occasionally surpasses its performance on level-1. The paper explains this as a dataset artifact—level-1 has only 43 of 500 examples, so 95-97% accuracy represents just 1-2 unsolved problems (primarily in geometry), while higher levels have ~100 examples each. More importantly, the difficulty labels were "annotated based on human perception of problem complexity rather than machine learning considerations"—what's easy for humans isn't necessarily easy for the model, and vice versa.

Why this works: the base model hypothesis. The paper's explanation for why pure RL succeeds is grounded in the properties of the base model. Appendix A.1 notes that DeepSeek-V3-Base's pretraining data "contains a substantial amount of mathematical and code-related content, indicating that DeepSeek-V3-Base has been exposed to a significant volume of reasoning trace data." The model already possesses the knowledge and sub-skills needed for reasoning (arithmetic, algebraic manipulation, code syntax, logical operators). What RL adds is the metacognitive layer—the strategic decisions about when to verify, when to backtrack, when to try an alternative approach. The base model can generate plausible solution candidates; RL selects for the patterns of generation that lead to correct answers.

This explains why the approach fails for models below ~32B parameters (Appendix G.1): smaller models have weaker base capabilities—they can't reliably perform the constituent operations of reasoning, so no amount of metacognitive optimization can produce correct answers. The paper's experiments with 7B dense and 16B MoE models showed that "as response lengths increased, these smaller models exhibited a tendency toward repetition and were unable to effectively leverage long chains of thought (CoT) to improve reasoning accuracy." They simply didn't have enough capacity to maintain coherent multi-step reasoning.

DeepSeek-R1: The Multi-Stage Pipeline

While R1-Zero demonstrates that pure RL can produce strong reasoning, it has practical issues: poor readability, language mixing, and limited performance on non-reasoning tasks like writing and open-domain QA. DeepSeek-R1 addresses these through a four-stage pipeline (Figure 2) that strategically interleaves SFT and RL.

Stage 1: Cold-Start SFT Data Collection (Appendix B.3.2). The paper collects "thousands of cold-start data that exhibits a conversational, human-aligned thinking process." The motivation is "primarily product-driven"—users prefer responses where the reasoning uses first-person perspective ("I" rather than "we" or no pronoun), maintains language consistency with the query, and presents a clear, readable structure.

The data construction procedure:

1. Gather diverse reasoning prompts. The paper collects high-quality reasoning questions across domains.

2. Generate trajectories with R1-Zero. DeepSeek-R1-Zero generates multiple reasoning trajectories per prompt at temperature 1.0 (high diversity).

3. Filter for correctness and readability. Only trajectories with correct final answers and readable format are retained. For math, sympy parses and compares expressions. For formatting, rules detect repetition and language mixing.

4. Human post-processing. Human annotators "convert the reasoning trace into a more natural, human conversational style" using a first-person perspective. The paper acknowledges that "the observed vivid reasoning patterns primarily reflect DeepSeek-engineered heuristics, rather than indicating that the model has inherently acquired human-like intelligence."

5. LLM-based expansion. The human-refined examples are used as few-shot prompts for an LLM (presumably DeepSeek-V3) to rewrite additional data in similar style. All LLM-generated outputs undergo a second round of human verification.

6. Summary generation (Listing 1). Since R1-Zero's summaries only provided the final answer, DeepSeek-V3 is prompted to produce a human-readable solution that "show[s] key steps leading to final answer(s) in clear, well-formatted LaTeX" while staying faithful to the original thought process.

7. Simple math examples (Listing 3). For very simple problems like "1 + 1 = ?", few-shot prompts ensure the model produces appropriately concise responses rather than overthinking.

For code data, the paper compiles 5,151 Codeforces problems and 2,504 AtCoder problems. Since original test cases are not public, they develop a test case generation pipeline: DeepSeek-V2.5 writes Python programs that generate test cases (Listing 2), correct submissions filter out invalid test cases, and strategic subsets that differentiate correct from incorrect solutions are selected. This ensures the code training data has reliable ground-truth verification.

SFT on cold-start data. The cold-start SFT fine-tunes DeepSeek-V3-Base for 2-3 epochs with cosine decay learning rate from $5 \times 10^{-5}$ to $5 \times 10^{-6}$, maximum context length 32,768 tokens, batch size 128. The resulting model is DeepSeek-R1-Dev1.

Stage 2: First RL Stage (Section 3.2.1). Dev1 undergoes GRPO training with the same hyperparameters as R1-Zero (learning rate $3 \times 10^{-6}$, KL coefficient 0.001, GRPO clip $\varepsilon = 10$, temperature 1.0, 16 outputs per question, maximum length 32,768, 32 questions per step, batch size 512, reference update every 400 steps), but with the addition of the language consistency reward described above.

The language consistency reward is applied "to both reasoning and non-reasoning data by directly adding it to the final reward." This means the total reward for this stage is:

$\text{Reward} = \text{Reward}_{\text{acc}} + \text{Reward}_{\text{format}} + \text{Reward}_{\text{language}}$

The resulting model is DeepSeek-R1-Dev2.

Stage 2 results (Table 3). Comparing Dev1 to Dev2: AIME 2024 improves from 59.0% to 74.0% (the RL stage substantially improves reasoning), LiveCodeBench from 57.5% to 63.5%, Codeforces from 84.5 to 90.5 percentile. However, general benchmarks like IF-Eval and ArenaHard show minimal improvement (Dev1→Dev2: 71.7%→72.0% and 77.0→73.2—the latter actually slightly declines), because "reasoning-oriented RL considerably enhances reasoning capabilities while exerting limited influence on user preference-oriented benchmarks."

Stage 3: Second SFT Stage (Rejection Sampling, Appendix B.3.3). This is the most data-intensive stage, producing ~800,000 supervised training samples.

Reasoning data (~600K samples). The Dev2 checkpoint generates reasoning trajectories for a large set of prompts. For each prompt, multiple responses are sampled, and only correct ones are retained (rejection sampling). The paper expands the data scope beyond rule-verifiable problems by using DeepSeek-V3 as a judge for tasks where ground-truth answers exist but can't be automatically parsed:

"we expand the dataset by incorporating additional data, some of which uses a generative reward model by feeding the ground-truth and model predictions into DeepSeek-V3 for judgment"

Listing 4 shows the judge prompt: it asks DeepSeek-V3 to classify answers as "correct" (fully aligns with reference) or "incorrect" (contains errors or deviates from the core question). This is a hybrid approach—the judge is neural (and therefore potentially biased), but it's comparing against a known ground-truth rather than evaluating quality in the abstract, which reduces the risk of systematic errors.

The paper also filters out "chain-of-thought with mixed languages, long paragraphs, and code blocks" to maintain quality. The final reasoning dataset spans math (~395K samples), code (~211K), STEM (~10K), logic (~10K), with an additional ~10K from other reasoning domains.

Non-reasoning data (~200K samples). For tasks like writing, factual QA, self-cognition, and translation, the paper reuses portions of DeepSeek-V3's SFT dataset. Software engineering data (program repair, front-end web development) is added. For certain non-reasoning tasks, DeepSeek-V3 generates a potential chain of thought before answering. For simple queries like "hello", no CoT is provided.

Thinking process design principles. The paper specifies three principles for the thinking process style: (1) keep paragraphs concise and digestible, (2) adopt a conversational, natural tone without markdown formatting, (3) begin by understanding the complete user context including unstated needs. Human annotators verify the accuracy of outputs. The paper notes that "these artificial reasoning traces enhance the model's precision in interpreting user queries" by highlighting format constraints, clarifying user intentions, and elucidating output structure requirements.

SFT data statistics (Table 5). The final SFT dataset contains 804,745 samples with an average of 1.0 rounds (mostly single-turn) and 5,355.3 average tokens per sample. The math data averages 6,094.2 tokens; code averages 7,435.7; general domain averages only 1,419.8. The paper acknowledges that "the majority of the data consists of single-turn interactions, which may limit the multi-turn conversational capabilities of DeepSeek-R1."

SFT training. DeepSeek-V3-Base is fine-tuned on this dataset for 2-3 epochs with cosine decay learning rate $5 \times 10^{-5} \rightarrow 5 \times 10^{-6}$, maximum context length 32,768, batch size 128. The resulting model is DeepSeek-R1-Dev3.

Stage 3 results (Table 3). Comparing Dev2 to Dev3: AlpacaEval2.0 jumps from 55.8 to 62.1 (+6.3%), Aider-Polyglot from 25.6% to 44.8% (+19.2%), attributable to "the inclusion of large-scale non-reasoning corpora and code engineering datasets" in the SFT data. Reasoning benchmarks show modest gains (AIME 74.0%→78.1%, LiveCodeBench 63.5%→64.6%).

Stage 4: Second RL Stage (Section 3.2.2). Dev3 undergoes final GRPO training with mixed reward signals and diverse prompt distributions.

Hyperparameter changes from the first RL stage:

• Temperature reduced from 1.0 to 0.7 because "higher temperatures in this stage lead to incoherent generation"
• Total training steps: 1,700
• General instruction data and preference-based rewards are incorporated "exclusively in the final 400 steps"

Reward formulation. For a batch containing both reasoning and general data:

$\text{Reward} = \text{Reward}_{\text{reasoning}} + \text{Reward}_{\text{general}} + \text{Reward}_{\text{language}}$

where $\text{Reward}_{\text{reasoning}} = \text{Reward}_{\text{rule}}$ (accuracy + format, same as R1-Zero) and $\text{Reward}_{\text{general}} = \text{Reward}_{\text{reward\_model}} + \text{Reward}_{\text{format}}$ (model-based score + format compliance).

The 400-step limit on model-based rewards. The paper discovered that prolonged exposure to model-based rewards causes reward hacking (Appendix B.5, Figure 6). By limiting preference-based reward training to only the final 400 steps out of 1,700, the model gets alignment benefits (helpfulness, harmlessness) without sufficient time to exploit reward model biases. The remaining 1,300 steps use only rule-based rewards, maintaining reasoning capability.

Stage 4 results (Table 3). Comparing Dev3 to final R1:

• Reasoning benchmarks show marginal improvement: AIME 78.1%→79.8%, LiveCodeBench 64.6%→65.9%, Codeforces 92.1→96.3 percentile. The paper notes this is because "substantial reasoning-specific RL was done in prior stages."
• General benchmarks show dramatic improvement: AlpacaEval2.0 from 62.1 to 87.6 (+25.5%), ArenaHard from 75.6 to 92.3 (+16.7%). "The primary advancements in the final DeepSeek-R1 were in general instruction-following and user-preference benchmarks."

Interpretation of the pipeline. The multi-stage design reflects a fundamental insight: different capabilities require different training signals, and applying them in the right order matters. Pure RL (R1-Zero) excels at reasoning but produces unreadable output. Cold-start SFT (Dev1) fixes readability but degrades reasoning. RL on the SFT'd model (Dev2) recovers reasoning while maintaining format. Rejection sampling SFT (Dev3) adds general capabilities but may slightly degrade specialized reasoning. Final RL with mixed signals (R1) polishes general alignment without harming reasoning (because rule-based rewards still dominate the training signal).

RL Infrastructure: Training at 671B Scale

Conducting RL training on a 671B-parameter MoE model requires specialized infrastructure. Appendix B.1 describes the system architecture, diagrammed in Figure 5.

Decoupled module architecture. The framework partitions the RL pipeline into four independent modules, each executing as a separate phase:

1. Rollout Module: Prompts are loaded from the training dataset and dispatched across multiple vLLM workers (Kwon et al., 2023), each equipped with the actor (policy) model, to sample responses. For the DeepSeek-V3 MoE architecture, the paper implements expert parallelism across nodes to reduce memory access overhead and deploys redundant copies of hotspot experts to balance computational loads. The Multi-Token Prediction (MTP) component from DeepSeek-V3 is leveraged for self-speculative decoding, "significantly accelerating the decoding speed and effectively minimizing the completion time for the longest samples."

2. Inference Module: Loads the reward model and reference model to perform forward passes on the samples from the rollout phase, obtaining model-based rewards (helpfulness/safety scores) and reference log-probabilities (needed for the KL penalty).

3. Rule-based Reward Module: Computes rule-based rewards (accuracy via code executors and answer matchers, format via format checkers). This module does not require GPU memory, but "its execution tends to be time-consuming." An asynchronous scheduling approach overlaps its execution with the Rollout and Inference modules, "effectively hiding the associated latency."

4. Training Module: Loads the actor model (and critic model, if using PPO) to compute the loss and update parameters. Supports multiple RL algorithms (PPO, GRPO, DPO).

Data packing strategy for efficient training. To minimize computational waste from sequence padding (where shorter sequences in a batch waste compute on padding tokens), the paper implements: (1) all data in a global batch is sorted by length and distributed across processes in the data parallel group; (2) within each process, a Best-Fit strategy packs data into fixed-length chunks with minimal padding; (3) the number of chunks is adjusted to be equal across all processes. This is combined with the DualPipe algorithm from DeepSeek-V3 training for efficient pipeline parallelism.

VRAM management. A critical engineering detail for training 671B models: "upon completion of each module... the model instances utilized in that phase are automatically offloaded from VRAM to either system memory or disk storage, thereby freeing up VRAM for the subsequent phase." Without this, the combined memory of the actor model, reference model, reward model, and optimizer states would exceed GPU capacity.

Training cost (Table 7). The paper reports costs in H800 GPU hours:

• DeepSeek-R1-Zero: 101,000 GPU hours (~198 hours on 64×8 H800s)
• SFT data creation: 5,000 GPU hours
• DeepSeek-R1 (all stages): 41,000 GPU hours (~80 hours on 64×8 H800s)
• Total: 147,000 H800 GPU hours (~$294K at$2/GPU-hour)

Distillation: Transferring Reasoning to Smaller Models

The paper demonstrates that the reasoning capabilities of DeepSeek-R1 can be transferred to smaller, more deployable models through supervised fine-tuning on R1-generated outputs (Appendix F).

Distillation procedure (Appendix B.4.3). For each student model, the corresponding base model is fine-tuned for 2-3 epochs on the 800K SFT data described in Appendix B.3.3 (the same data used to train Dev3). The training uses cosine decay learning rate schedulers that decrease to one-tenth of the initial value, with maximum context length 32,768 and batch size 64. Table 6 lists the base models and initial learning rates:

The paper notes: "For distilled models, we apply only SFT and do not include an RL stage, even though incorporating RL could substantially boost model performance. Our primary goal here is to demonstrate the effectiveness of the distillation technique, leaving the exploration of the RL stage to the broader research community."

Distillation vs. reinforcement learning for small models (Appendix F.1, Table 16). This is a critical comparison that reveals a boundary condition for the pure RL approach:

Qwen2.5-32B-Zero is trained with the same pure RL methodology as R1-Zero (GRPO with rule-based rewards on math, code, and STEM data for >10K steps). It achieves performance on par with QwQ-32B-Preview—respectable, but dramatically below the distilled model (72.6% vs. 47.0% on AIME). The paper draws two conclusions:

1. "Distilling more powerful models into smaller ones yields excellent results, whereas smaller models relying on the large-scale RL mentioned in this paper require enormous computational power and may not even achieve the performance of distillation."

2. "While distillation strategies are both economical and effective, advancing beyond the boundaries of human intelligence may still require more powerful base models and larger-scale reinforcement learning."

This suggests a two-tier strategy: use large-scale RL on very large models (>100B parameters) to discover superior reasoning strategies beyond human examples, then distill these strategies into smaller, deployable models through supervised fine-tuning. The large model does the exploration; the small models inherit the results.

Additional evidence from Qwen2-Math-7B (Table 17). The paper conducted experiments on Qwen2-Math-7B (released August 2024, before any reasoning model launches) to ensure the base model wasn't exposed to reasoning trajectory data. After ~10,000 GRPO steps, Qwen2-Math-7B-Zero achieved 22.3% on AIME 2024 and 18.1% on AIME 2025, significantly outperforming Qwen2-Math-7B-Instruct (7.9% and 4.6%) and GPT-4o (9.3% on AIME 2024). This demonstrates that even a 7B model can develop reasoning through pure RL, though the absolute performance is far below what larger models or distillation can achieve. The paper uses this as evidence that "the model can autonomously develop advanced reasoning strategies through large-scale reinforcement learning" across model scales, but the practical lesson is that more parameters make a qualitative difference in what RL can unlock.

Distilled model performance (Table 15). The distilled models show remarkably strong performance, particularly in light of their size:

• DeepSeek-R1-Distill-Qwen-1.5B: 28.9% on AIME 2024—a 1.5B model outperforming GPT-4o (9.3%) and Claude-3.5-Sonnet (16.0%)
• DeepSeek-R1-Distill-Qwen-7B: 55.5% on AIME, 92.8% on MATH-500, 49.1% on GPQA Diamond
• DeepSeek-R1-Distill-Qwen-32B: 72.6% on AIME, approaching the full 671B R1-Zero's 77.9% (which required pure RL)

The paper's distillation results are a significant practical contribution: they show that the reasoning patterns discovered by the large-scale RL process can be effectively packaged and distributed to models that run on consumer hardware.

4. Key Insights and Innovations

Innovation 1: Reasoning Can Emerge from Outcome-Based Reward Alone Without Any Reasoning Demonstrations

The paper's most profound intellectual contribution is not a new algorithm or architecture, but a refutation of a widely-held assumption: that sophisticated reasoning behaviors—self-verification, error detection, backtracking, strategy switching—must be taught through examples of those very behaviors. The dominant post-training paradigm in the field (Ouyang et al., 2022; Chung et al., 2024) treats supervised fine-tuning on human-written reasoning traces as a necessary prerequisite for stable RL training. The paper demonstrates that this assumption is false for sufficiently large models.

What makes this a conceptual innovation rather than merely an empirical finding is the diagnostic clarity of the experimental design. By training DeepSeek-R1-Zero directly from DeepSeek-V3-Base with no SFT whatsoever—only GRPO with accuracy and format rewards—the paper isolates the causal variable: reward signal alone. The template (Table 1) provides structural constraints (think in thinking tags, answer in <answer> tags) but zero content about how to reason. The model must discover that generating intermediate verification steps, catching its own errors ("Wait, wait. Wait." in Table 2), and exploring alternative approaches improves its probability of receiving the outcome reward. No human ever demonstrated these behaviors; they emerged because they worked.

The significance of this finding extends far beyond the specific model or benchmark. It suggests that pretrained LLMs contain latent reasoning capabilities that are gated not by missing knowledge but by missing incentive structures. The base model already "knows" how to perform algebraic manipulation, check its work, and backtrack from dead ends—these patterns exist in its training distribution from the web. What it lacks is the metacognitive policy of when to deploy these skills. RL with outcome rewards provides exactly this: the model discovers that deploying verification and reflection contingent on uncertainty increases expected reward, without anyone having to label which steps are correct or which decision points demand reflection.

This reframes the problem of building reasoning-capable AI. Prior work implicitly assumed that human reasoning patterns represent the ceiling—that we must first collect expert demonstrations, then train models to imitate them, then possibly improve beyond them. The paper's framing inverts this: human reasoning patterns may be a floor, not a ceiling. The model discovered strategies (extended backtracking, multi-step verification, dynamic allocation of thinking tokens proportional to problem difficulty) that human-written CoT examples rarely demonstrate because humans produce cleaned, linear solution presentations. By removing the human prior, RL allowed the model to find strategies that are more effective for machine cognition, even if they don't resemble how humans present their reasoning.

The phase-change dynamics provide additional evidence for this interpretation. Figure 9(b) shows that the word "wait" was nearly absent until step 4,000, appeared occasionally between steps 4,000-7,000, and then spiked dramatically after step 8,000. This is not gradual acquisition—it is discovery. The model stumbled upon a strategy (pausing to reconsider) that improved reward, and then rapidly incorporated it into its policy. The fact that this happened without any demonstration of "wait"-style reflection means the model autonomously recognized a failure mode (confusion or contradiction) and invented a remediation strategy.

The boundary condition is equally informative: the approach fails for models below approximately 32B parameters (Appendix G.1). The 7B and 16B models "exhibited a tendency toward repetition and were unable to effectively leverage long chains of thought." This negative result transforms the finding from an existence proof ("reasoning can emerge") into a scaling law observation: the capacity threshold for emergent metacognition through RL is substantially higher than for basic task competence. This has direct implications for the field's understanding of what different model scales can and cannot achieve through post-training.

Innovation 2: Eliminating the Critic Model Through Group-Normalized Advantages

The adoption of GRPO represents a methodological innovation with architectural implications: the paper demonstrates that a learned value function—long considered essential for stable policy gradient methods in language model training—can be replaced by group-level reward normalization without performance degradation, while simultaneously removing a barrier to long chain-of-thought generation.

Prior work on RL for language models (Ouyang et al., 2022; Bai et al., 2022) used PPO with a separate critic model of comparable size to the policy, trained to predict expected future rewards at each token position. This architecture was inherited from continuous control domains where value functions are essential for credit assignment over long horizons. The paper's insight is that for language generation with outcome-only rewards, within-group comparisons provide an adequate baseline without learning. The advantage estimator in GRPO—z-score normalization of rewards within a group of outputs from the same prompt—replaces the critic's role of estimating "how good was this action relative to expectations" with "how good was this complete output relative to its peers for the same input."

This is not merely a computational convenience (though saving the memory and training cost of a critic model at 671B scale is practically significant). It is a conceptual reframing of what constitutes a baseline in language model RL. In PPO, the baseline is learned: the value model approximates $V(s_t) = \mathbb{E}[\sum_{k=0}^\infty \gamma^k r_{t+k} | s_t]$, which requires predicting long-horizon outcomes from partial sequences. The paper argues this is "inherently difficult, especially when only the final outcome reward is available" and "even more pronounced when training long chain-of-thought reasoning models" where early content may be revised or contradicted later (Appendix A.3). GRPO sidesteps this problem entirely: the baseline is the empirical mean reward of sampling the policy multiple times on the same prompt. This baseline is always correct by construction (it is the Monte Carlo estimate of expected reward under the current policy), requires no training, and adapts instantly to policy changes.

The second architectural insight concerns the KL penalty. PPO adds a per-token KL divergence penalty as a dense reward, meaning longer responses accumulate larger total KL penalties even if the per-token policy divergence is identical. The paper recognizes that this implicitly penalizes response length—a critical failure mode when the training objective is to incentivize the model to think longer. GRPO adds the KL penalty directly to the loss (Equation 11), computed once per complete output, so response length does not affect the magnitude of the regularization term. The paper notes that "in the scenario of training long chain-of-thought reasoning models, the trained policy can diverge significantly from the initial reference policy," and GRPO's loss-level KL penalty, combined with periodic reference model updates (every 400 steps), allows exploration of longer reasoning chains without being penalized for the length per se.

The empirical validation that this matters comes from Figure 4 (Appendix A.3): PPO with default $\lambda = 0.95$ in GAE performs "considerably worse than GRPO" on the MATH task, and only approaches GRPO performance when $\lambda$ is tuned to 1.0 (effectively removing temporal smoothing). The paper's interpretation is that "while PPO can achieve comparable performance when appropriately tuned, it demands additional computational cost for hyperparameter optimization," and that the value model's memory overhead makes GRPO "a more practical alternative, especially when training large-scale models." This is a significant methodological contribution: it establishes that the critic model, long considered central to PPO, is not just unnecessary but actively harmful for long-form reasoning RL due to its length-penalty effect, and it provides a drop-in replacement that is both simpler and more effective.

Innovation 3: The SFT-First Paradigm Is Actively Harmful for Reasoning Capability Discovery

While Innovation 1 established that pure RL can produce reasoning, this innovation makes a stronger claim: the standard practice of applying SFT before RL is detrimental to discovering optimal reasoning strategies. The paper's multi-stage pipeline comparison provides direct evidence for this claim.

The conventional post-training paradigm (Appendix A.2) treats SFT as a necessary initialization step: fine-tune on human demonstrations to establish basic task competence, then apply RL to refine behavior. The paper's R1-Zero results demonstrate that this initialization is unnecessary for reasoning—the base model already possesses sufficient competence to begin exploration. More provocatively, the comparison between R1-Zero (pure RL) and R1-Dev1 (cold-start SFT) in Table 3 shows that introducing SFT before RL degrades reasoning performance: R1-Zero achieves 77.9% on AIME 2024 pass@1 while Dev1, which received thousands of high-quality human-refined reasoning examples, achieves only 59.0%.

The paper's explanation is that human-provided reasoning examples "often omit critical reasoning components such as explicit reflection and verification steps" (Appendix A.2). Human experts, when writing solutions, present cleaned, linear derivations that skip the messy exploratory process. When the model is fine-tuned on these examples, it learns to imitate the presentation style of reasoning rather than the cognitive process of reasoning. The SFT effectively trains the model to skip the verification and backtracking that make reasoning robust. When RL is subsequently applied, the model starts from a policy that has been explicitly trained not to generate self-correction behaviors, and must unlearn this bias before it can discover more effective strategies.

This finding has significant implications for how the field should approach post-training. The paper is not arguing that SFT is useless—the multi-stage pipeline deliberately re-introduces SFT at specific points for specific purposes (readability in cold-start, general capabilities in second SFT). Rather, it is arguing that the sequencing of SFT and RL matters critically, and the conventional sequence (SFT first) is suboptimal for reasoning. The optimal sequence, per the paper's pipeline, is: RL first (to discover effective reasoning strategies without human constraints) → SFT using RL-discovered outputs (to make reasoning readable and add general capabilities) → RL again (to polish alignment while preserving reasoning).

The distillation results (Appendix F.1) reinforce this interpretation. DeepSeek-R1-Distill-Qwen-32B achieves 72.6% on AIME by learning from the outputs of the RL-trained model, while Qwen2.5-32B-Zero achieves only 47.0% by undergoing the same RL process directly. The paper concludes: "distilling more powerful models into smaller ones yields excellent results, whereas smaller models relying on the large-scale RL mentioned in this paper require enormous computational power and may not even achieve the performance of distillation." The RL process is the engine of discovery, but it requires sufficient model capacity to work. SFT on RL-discovered patterns can transfer those discoveries to smaller models, but SFT on human-demonstrated patterns cannot substitute for the RL discovery process itself.

This represents a fundamental shift in the field's understanding of the relationship between supervised learning and reinforcement learning for reasoning. The prior assumption—that SFT provides a necessary foundation that RL refines—is replaced by a more nuanced view: RL provides the discovery mechanism that can find reasoning strategies beyond human priors, while SFT provides the distillation and alignment mechanism that makes those strategies deployable. The two techniques have asymmetric roles, and applying them in the wrong order is actively counterproductive.

Innovation 4: Test-Time Compute Scaling Is a Learned Behavior, Not Just an Inference Strategy

The paper provides evidence that adaptive allocation of computation at inference time can emerge as a learned behavior through reinforcement learning, rather than being externally imposed through search algorithms. This distinguishes DeepSeek-R1's approach from prior work on test-time compute scaling and has implications for both capability and efficiency.

Prior work on scaling inference compute (Snell et al., 2024; Brown et al., 2024; Lightman et al., 2024) treats the allocation of additional computation as an external mechanism applied to a frozen model: generate multiple samples (best-of-N, majority voting), use search algorithms (beam search, MCTS), or employ verifier-guided selection (PRM reranking). The model itself does not decide how much to think; the inference procedure does. DeepSeek-R1 operates differently: the model learns during RL training to dynamically adjust its own reasoning length based on problem difficulty, generating more thinking tokens for harder problems and fewer for simpler ones. This is an internal compute-scaling mechanism, learned through reward optimization, rather than an external one imposed at inference time.

The evidence comes from Section E.4 and Figure 18. On a collection of 366 competition math problems from 2024, DeepSeek-R1 achieves 61.8% pass@1 while generating an average of 8,793 thinking tokens per problem. Crucially, the model's token usage scales with difficulty: fewer than 7,000 thinking tokens for easy problems and more than 18,000 for the hardest ones. The paper notes that "the model adaptively adjusts its computational effort based on problem difficulty" and that "the complexity of a problem directly correlates with the number of thinking tokens required." This behavior was not programmed—it emerged during RL training because allocating more computation to harder problems improves expected reward.

This has significant implications for understanding what RL training actually optimizes. The model is not merely learning better reasoning content; it is learning a reasoning budget allocation policy. For each problem, the model makes an implicit decision about how much computation to invest before committing to an answer. The RL process shapes this decision by rewarding correct answers: spending too few tokens risks errors, while spending excessive tokens incurs no direct penalty (the KL penalty is output-level, not token-level) but represents wasted computation. The emergent equilibrium is a difficulty-conditioned compute allocation policy that the paper's analysis makes visible for the first time.

The contrast with external scaling methods is instructive. The paper notes that majority voting across 64 samples improves DeepSeek-R1's AIME 2024 accuracy from 79.8% to 86.7%—so external scaling still helps. But the source of the primary capability gain (from the base model's near-zero performance to 79.8%) is the learned internal scaling behavior, not the external voting. For non-reasoning models like GPT-4o, majority voting across 64 samples only improves AIME accuracy from 9.3% to 13.4% (Section E.4). The paper argues that "since non-reasoning models lack the ability to backtrack or self-correct, scaling the sample size merely results in repeatedly sampling potentially incorrect final solutions without increasing the probability of finding correct solutions in any single attempt." The learned internal scaling behavior—generating a long, coherent chain of thought that includes verification and error correction—is qualitatively more effective than parallel independent sampling because the reasoning steps can build on each other.

This innovation reframes the test-time compute scaling conversation. Prior work asks: "given a fixed model, how should we allocate additional inference compute?" This paper demonstrates that the model itself can be trained to internalize this allocation, making external scaling mechanisms complementary rather than primary. The practical implication is that the most effective test-time compute scaling may come from training models that inherently scale their computation, rather than from designing better search algorithms over frozen models. This does not make external scaling obsolete—the paper shows both can be combined. But it shifts the emphasis from inference-time algorithms to training-time objectives that incentivize adaptive computation.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The primary benchmark suite spans mathematical reasoning, coding, general knowledge, and safety. Mathematical benchmarks include AIME 2024 (American Invitational Mathematics Examination, 30 problems, integer answers from 0–999), MATH-500 (500 competition-level math problems from Hendrycks et al., 2021), and CNMO 2024 (Chinese National High School Mathematics Olympiad). Coding benchmarks include LiveCodeBench (problems collected from LeetCode, AtCoder, and Codeforces between August 2024 and January 2025), Codeforces (problems from 10 Div.2 contests with expert-crafted test cases, evaluated as percentile and Elo rating), SWE-Bench Verified (real-world software engineering bug fixes, evaluated via the agentless framework from Xia et al., 2024), and Aider-Polyglot (multi-language code editing accuracy). Knowledge benchmarks include MMLU (57 tasks), MMLU-Redux (5,700 manually re-annotated MMLU questions), MMLU-Pro (higher-difficulty MMLU subset), GPQA Diamond (graduate-level STEM multiple-choice), SimpleQA and C-SimpleQA (factuality evaluation), FRAMES (long-context retrieval-augmented QA), DROP (reading comprehension requiring discrete reasoning), C-Eval (Chinese multi-discipline evaluation), and CLUEWSC (Chinese Winograd schema). Instruction-following is measured by IFEval (verifiable constraint compliance). Open-ended generation quality is assessed by AlpacaEval 2.0 and ArenaHard (both using GPT-4-Turbo-1106 as judge). Safety benchmarks include SimpleSafetyTests, BBQ (bias), Anthropic Red Team, XSTest, Do-Not-Answer, HarmBench, and a custom in-house taxonomy spanning 4 major categories and 28 subcategories across 50 languages and jailbreak attacks. All datasets use their original evaluation protocols with default prompts, except MMLU-Pro, C-Eval, and CLUEWSC which are modified from few-shot to zero-shot. Decontamination is performed via 10-gram matching against evaluation text in both pre-training and post-training data, with approximately six million texts removed from the mathematics domain alone. Models are evaluated at a maximum output length of 32,768 tokens.

• Base model(s). All experiments build on DeepSeek-V3-Base, a 671B-parameter Mixture-of-Experts transformer (37B activated per token) pretrained on 14.8 trillion tokens of web and ebook data with no synthetic data incorporation. The pretraining data contains substantial mathematical and code content but no intentionally added synthetic reasoning traces. The base model's pass@1 on AIME 2024 before any post-training is 15.6%. For distillation experiments, base models include Qwen2.5-Math-1.5B, Qwen2.5-Math-7B, Qwen2.5-14B, Qwen2.5-32B, Llama-3.1-8B, and Llama-3.3-70B-Instruct. The model family choice is driven by the hypothesis that the base checkpoint must possess sufficient capacity for RL to induce reasoning—experiments with 7B dense and 16B MoE models failed (Appendix G.1).

• Metrics. For mathematical benchmarks (AIME, MATH-500, CNMO), the metric is pass@1—the probability that a single sample from the model produces the correct answer. For AIME, the paper also reports cons@64 (majority voting across 64 samples). Pass@k is estimated following Chen et al. (2021): for each question, k responses are generated (typically k = 64 for AIME and GPQA, 16 for MATH and Codeforces, 8 for LiveCodeBench) at temperature 0.6 and top-p 0.95, correctness is determined per-response, and pass@1 is the average correctness across the k responses. For Codeforces, both percentile (relative to human competitors) and Elo rating are reported. For MMLU, MMLU-Redux, MMLU-Pro, C-Eval, GPQA Diamond, and CLUEWSC, the metric is exact match (EM) with the ground-truth answer choice. For DROP, the metric is 3-shot F1. For IFEval, Prompt Strict accuracy. For SimpleQA and C-SimpleQA, Correct (binary factuality judgment via LLM-as-judge). For FRAMES, accuracy. For AlpacaEval 2.0, LC-winrate (length-controlled win rate against reference). For ArenaHard, GPT-4-1106 judged win rate. For SWE-Bench Verified, Resolved (fraction of issues correctly fixed). For Aider-Polyglot, accuracy. For safety benchmarks, safety score (percentage of responses judged safe). Statistical significance is assessed via t-test with p < 0.01 on benchmark scores. Decontamination prevents data leakage using 10-gram overlap filtering.

• Baselines. The paper compares against DeepSeek-V3 (the instructed non-reasoning variant of the same base architecture), DeepSeek-V3-Base (the pretrained model without any post-training), Claude-3.5-Sonnet-1022, GPT-4o-0513, OpenAI-o1-mini, and OpenAI-o1-1217. For distilled models, baselines include GPT-4o-0513, Claude-3.5-Sonnet-1022, and QwQ-32B-Preview (Qwen, 2024a). The OpenAI-o1-1217 results are taken from official reports when API access was unavailable. All baselines except V3-Base and V3 represent the state of frontier closed-source and open-source reasoning and non-reasoning models at the time of evaluation. For the pure RL vs. distillation comparison, the paper also reports Qwen2.5-32B-Zero (trained with the same GRPO methodology as R1-Zero for >10K steps) and Qwen2-Math-7B-Zero and Qwen2-Math-7B-Instruct (Appendix F.1, Tables 16–17).

• Generation budget / compute accounting. All models are evaluated with a maximum generation length of 32,768 tokens. The paper uses a sampling temperature of 0.6 and top-p of 0.95 for pass@k estimation. Some evaluations use greedy decoding, but the paper notes that "using greedy decoding to evaluate long-output reasoning models results in higher repetition rates and significant variability across different checkpoints," motivating the temperature-based approach. For DeepSeek-R1-Zero training, compute is measured in training steps (10,400 total), with each step processing 32 unique questions × 16 outputs = batch size 512. Total training cost is 101,000 H800 GPU hours. For DeepSeek-R1, total training cost across all stages is 41,000 H800 GPU hours. SFT data creation consumed 5,000 GPU hours. The FLOPs-matched comparison from prior work on test-time compute scaling is not replicated in this paper, as the focus is on training-time RL compute rather than inference-time compute allocation.

• Cross-validation / statistical protocol. The paper does not employ cross-validation for model selection, as the training pipeline is deterministic given fixed hyperparameters. Statistical significance between models is assessed via t-test with p < 0.01, applied to benchmark scores. For AIME 2024 evaluation, pass@1 is computed by averaging across k = 64 samples per question, providing a reliable point estimate. For human evaluation on ChatbotArena, Elo scores are computed using a bootstrap-like technique with pairwise comparisons from anonymous user votes. For the in-house safety benchmark, LLM-as-judge consistency with human assessments was verified at above 95% on sampled results. The paper acknowledges that n-gram based decontamination cannot prevent paraphrase-based contamination, so "it is possible that benchmarks released before 2024 may suffer from contamination issues."

Main Quantitative Results

Pure RL from the Base Model (DeepSeek-R1-Zero)

Headline result: DeepSeek-R1-Zero, trained exclusively with GRPO and rule-based rewards from DeepSeek-V3-Base with no supervised fine-tuning, achieves 77.9% pass@1 and 86.7% cons@64 on AIME 2024. This surpasses the average human participant score and represents a 62.3 percentage point improvement over the base model's 15.6% (Section 2.3, Figure 1a).

The training trajectory (Figure 1a) shows the pass@1 score rising from 15.6% at initialization to approximately 40% by step 3,000, then climbing to 77.9% by step 10,400. The cons@64 curve tracks consistently above pass@1, reaching 86.7% at the final step. The AIME baseline for human participants is shown as a horizontal reference line that the model crosses during training.

Concurrently, the average response length on the training set (Figure 1b) grows from approximately 500 tokens at initialization to over 15,000 tokens by step 10,400. The growth accelerates particularly after step 4,000, coinciding with the period when reflective behaviors emerge. At step 8,200, the maximum generation length was increased from 32,768 to 65,536 tokens, and the paper notes that "both the performance and response length of DeepSeek-R1-Zero exhibit a significant jump at the 8.2k step."

On broader benchmarks (Table 8, Figure 10): R1-Zero achieves 88.8% on MMLU (vs. DeepSeek-V3's 88.5%), 75.8% on GPQA Diamond (vs. V3's 59.1%), 50.0% on LiveCodeBench (vs. V3's 36.2%), 80.4 percentile on Codeforces (vs. V3's 58.7), and 95.9% on MATH-500 (vs. V3's 90.2%). On open-ended generation, R1-Zero scores only 24.7 on AlpacaEval2.0 and 53.6 on ArenaHard—substantially below V3's 70.0 and 85.5—because pure RL without general-domain training produces unreadable, poorly formatted outputs.

The evolution of reasoning behaviors is quantified in Appendix C.2, Figure 9a: the frequency of ten reflective words ("wait", "mistake", "however", "but", "retry", "error", "verify", "wrong", "evaluate", "check") increases 5- to 7-fold from roughly 2,000 to 12,000–14,000 total occurrences over training. Figure 9b shows that the word "wait" specifically was nearly absent before step 4,000, appeared occasionally between steps 4,000–7,000, and then spiked dramatically after step 8,000, indicating a phase-change in reasoning strategy.

Multi-Stage Pipeline Results (DeepSeek-R1)

Headline result: DeepSeek-R1, trained through the four-stage pipeline, achieves 79.8% pass@1 on AIME 2024 (on par with OpenAI-o1-1217's 79.2%), 97.3% on MATH-500 (vs. o1's 96.4%), 96.3 percentile on Codeforces (vs. o1's 96.6), and 71.5% on GPQA Diamond (vs. o1's 75.7%), while dramatically outperforming all non-reasoning baselines and matching or exceeding o1 on general benchmarks (AlpacaEval2.0: 87.6 vs. o1's unreported; ArenaHard: 92.3 vs. o1's unreported) (Table 8, Figure 10).

Stage-by-stage progression (Table 3): The four intermediate checkpoints reveal the contribution of each pipeline component:

• R1-Zero → Dev1 (cold-start SFT): AIME 2024 drops from 77.9% to 59.0% (-18.9 pp), confirming that introducing human-style SFT before RL degrades the model's discovered reasoning strategies. However, Dev1 gains substantially on instruction-following (IF-Eval: 46.6% → 71.7%) and general quality (AlpacaEval2.0: 24.7 → 50.1; ArenaHard: 53.6 → 77.0). This shows the cold-start SFT trades reasoning capability for readability and formatting.

• Dev1 → Dev2 (first RL stage): AIME 2024 recovers to 74.0% (+15.0 pp), and coding benchmarks jump (LiveCodeBench: 57.5% → 63.5%; Codeforces percentile: 84.5 → 90.5). General benchmarks show marginal or slightly negative change (IF-Eval: 71.7% → 72.0%; ArenaHard: 77.0 → 73.2), confirming the paper's claim that "reasoning-oriented RL considerably enhances reasoning capabilities while exerting limited influence on user preference-oriented benchmarks."

• Dev2 → Dev3 (second SFT with rejection sampling): AIME improves further to 78.1% (+4.1 pp), and coding engineering benchmarks show large gains (Aider-Polyglot: 25.6% → 44.8%, +19.2 pp; SWE Verified: 44.6% → 45.6%, +1.0 pp). General benchmarks improve moderately (AlpacaEval2.0: 55.8 → 62.1; ArenaHard: 73.2 → 75.6), attributable to "the inclusion of large-scale non-reasoning corpora and code engineering datasets."

• Dev3 → R1 (second RL stage): Reasoning benchmarks show marginal gains (AIME: 78.1% → 79.8%; LiveCodeBench: 64.6% → 65.9%; Codeforces: 92.1 → 96.3 percentile), but general benchmarks improve dramatically: AlpacaEval2.0 jumps from 62.1 to 87.6 (+25.5 pp) and ArenaHard from 75.6 to 92.3 (+16.7 pp). The paper notes that "the primary advancements in the final DeepSeek-R1 were in general instruction-following and user-preference benchmarks."

Comparison to frontier models (Table 8): DeepSeek-R1 matches or exceeds all baselines on reasoning benchmarks: on AIME 2024, R1's 79.8% edges out o1-1217's 79.2% and dramatically exceeds GPT-4o's 9.3% and Claude-3.5-Sonnet's 16.0%. On MATH-500, R1's 97.3% surpasses o1's 96.4%. On Codeforces, R1's 96.3 percentile essentially ties o1's 96.6, with ratings of 2029 vs. 2061. On LiveCodeBench, R1's 65.9% exceeds o1's 63.4%. On GPQA Diamond, R1's 71.5% trails o1's 75.7%. On SWE-Bench Verified, R1's 49.2% slightly exceeds o1's 48.9%, but on Aider-Polyglot, R1's 53.3% trails o1's 61.7%. On general benchmarks, R1 achieves 87.6 on AlpacaEval2.0 and 92.3 on ArenaHard, both exceeding GPT-4o and Claude-3.5-Sonnet. On IF-Eval, R1's 83.3% is comparable to other frontier models. On MMLU, R1's 90.8% trails o1's 91.8% but exceeds V3's 88.5%.

Human evaluation (ChatbotArena, Figures 11–12): As of January 24, 2025, DeepSeek-R1 shares the #1 overall position alongside OpenAI-o1 and Gemini-Exp-1206 on the style-controlled leaderboard. The model ranks particularly high on mathematics and coding categories. The paper notes this as "a huge milestone that an open-source model under the MIT License could achieve comparable performance with closed-source models."

Difficulty-Stratified Performance

LiveCodeBench difficulty breakdown (Table 14): The paper stratifies LiveCodeBench problems into easy, medium, and hard difficulty levels and tracks each stage's performance:

Easy problems are nearly solved by R1-Zero and reach perfection by Dev2. The main improvements across stages come from medium and hard problems, with hard problems improving from 17.09% to 34.44% over the full pipeline.

MMLU and MMLU-Pro category breakdown (Figures 15–16): On MMLU (Figure 15), DeepSeek-R1 improves over DeepSeek-V3 across all four major categories, with the largest gains in STEM (95.3% vs. 90.5%) and Humanities (86.5% vs. 80.0%). On MMLU-Pro (Figure 16), which has substantially harder questions, R1 outperforms V3 across all 14 subject categories, with Math showing the largest absolute gap (93.5% vs. 84.2%). The paper hypothesizes that "MMLU represents a relatively easier challenge compared to MMLU-Pro," and that "in STEM tasks of MMLU, post-training on DeepSeek-V3 may have already achieved near-saturation performance, leaving minimal room for further improvement." Notably, non-STEM categories like Social Sciences and Humanities also improve with long CoT, which the paper attributes to "the better understanding of the question."

MATH difficulty breakdown during R1-Zero training (Figure 8, Appendix C.1): Tracking R1-Zero's performance on MATH-500 by difficulty level (1–5, human-annotated) across training steps reveals: easy problems (levels 1–3) quickly reach 0.90–0.95 accuracy and remain stable; level 4 improves from ~0.78 to 0.95; level 5 (hardest) shows the most dramatic improvement from ~0.55 to 0.90. A counterintuitive observation: the model's accuracy on harder questions occasionally surpasses level-1 performance, which the paper explains as a dataset artifact—level-1 contains only 43 of 500 examples, so 95–97% accuracy represents just 1–2 unsolved geometry problems. The difficulty labels were "annotated based on human perception of problem complexity rather than machine learning considerations."

2024 competition math breakdown (Figure 17, Appendix E.3): On a collection of 366 problems from 93 mathematics competitions held in 2024, DeepSeek-R1 significantly outperforms GPT-4o across all eight problem categories. The strongest performance is in Functional Equation (73.4% vs. 32.3%), Number Theory (72.6% vs. 26.5%), and Algebra (70.9% vs. 19.0%). The weakest is Combinatorial Geometry (14.5% vs. 4.5%). The overall pass@1 is 61.8% vs. 24.7% for GPT-4o.

Test-Time Compute Scaling Behavior

Adaptive token allocation (Figure 18, Section E.4): DeepSeek-R1 shows learned adaptive compute scaling: on the 2024 competition math set, it generates an average of 8,793 thinking tokens to reach correct answers, with usage varying from fewer than 7,000 tokens for easy problems to more than 18,000 tokens for the hardest. The paper notes that the model "adaptively adjusts its computational effort based on problem difficulty" and that "the complexity of a problem directly correlates with the number of thinking tokens required." The relationship is shown with a smoothed curve (UnivariateSpline with smoothing factor 5), with the standard deviation band widening at higher difficulties, indicating variability in token usage for hard problems.

Comparison to non-reasoning models: GPT-4o achieves only 24.7% on the same problem set while generating an average of 711 tokens—"an order of magnitude less than DeepSeek-R1." Majority voting across 16 samples per problem yields "minimal improvement" in GPT-4o's solve rate despite consuming more total tokens than DeepSeek-R1. On AIME 2024, majority voting across 64 samples increases GPT-4o's accuracy from 9.3% to 13.4%, still dramatically below DeepSeek-R1's 79.8%. The paper attributes this gap to a fundamental difference: "in majority voting, samples are generated independently rather than building upon each other. Since non-reasoning models lack the ability to backtrack or self-correct, scaling the sample size merely results in repeatedly sampling potentially incorrect final solutions without increasing the probability of finding correct solutions in any single attempt."

Complementary external scaling: Despite the learned internal scaling, external majority voting still helps DeepSeek-R1: cons@64 on AIME 2024 improves from pass@1 of 79.8% to 86.7%. The pass@64 score (the best of 64 samples) is 90.0%, indicating that while individual responses are strong, independent sampling can still find correct solutions that some reasoning chains miss. The paper notes this as evidence that "traditional test-time scaling methods like majority voting or Monte Carlo Tree Search (MCTS) can complement DeepSeek-R1's long reasoning."

Generalization to Unseen Competitions

AIME 2025 results (Table 13): To address concerns about benchmark contamination, the paper evaluates on AIME 2025, released after the model's training cutoff. DeepSeek-R1 achieves a 75% solve rate (implied pass@1 ~11.3/15), approaching OpenAI-o1-1217's 80% (12.0/15). On AMC 12 2024, R1 scores 143.7/150, and when combined with its AIME performance, yields a USAMO Index of 256.7, exceeding the 251.5 qualification threshold for the United States of America Mathematical Olympiad. The paper presents this as evidence that the model's capabilities generalize beyond its training distribution.

Distilled Model Performance

Headline result (Table 15): Models distilled from DeepSeek-R1's outputs achieve performance that dramatically exceeds their baseline counterparts and often surpasses much larger non-reasoning models. The smallest distilled model, DeepSeek-R1-Distill-Qwen-1.5B, achieves 28.9% pass@1 on AIME 2024—exceeding GPT-4o's 9.3% and Claude-3.5-Sonnet's 16.0%. The distilled models scale predictably with size: the 7B achieves 55.5%, 14B achieves 69.7%, 32B achieves 72.6%, and the Llama-70B distilled model achieves 70.0%. On MATH-500, even the 1.5B model reaches 83.9%, the 7B reaches 92.8%, and the 32B reaches 94.3%. On LiveCodeBench, the 32B distilled model reaches 57.2% pass@1, exceeding GPT-4o's 32.9% and Claude-3.5-Sonnet's 38.9%.

Distillation vs. RL for small models (Table 16, Appendix F.1): The critical comparison between distillation and direct RL training on smaller models shows:

• Qwen2.5-32B-Zero (pure RL for >10K steps on the same base model): 47.0% AIME, 91.6% MATH-500, 55.0% GPQA Diamond, 40.2% LiveCodeBench
• DeepSeek-R1-Distill-Qwen-32B (SFT on R1 outputs): 72.6% AIME (+25.6 pp), 94.3% MATH-500, 62.1% GPQA Diamond, 57.2% LiveCodeBench

The distilled model dramatically outperforms the RL-trained model of the same size, despite the RL model training for over 10,000 steps "with enormous computational power." The paper concludes that "distilling more powerful models into smaller ones yields excellent results, whereas smaller models relying on the large-scale RL mentioned in this paper require enormous computational power and may not even achieve the performance of distillation."

Additional evidence from 7B scale (Table 17): Qwen2-Math-7B-Zero, trained with ~10,000 GRPO steps before any reasoning model releases (to ensure the base model wasn't exposed to reasoning trajectories), achieves 22.3% on AIME 2024 and 18.1% on AIME 2025, significantly outperforming Qwen2-Math-7B-Instruct (7.9% and 4.6%) and GPT-4o (9.3% on AIME 2024). This demonstrates that even 7B models can develop some reasoning capability through pure RL, but the absolute performance is far below distillation from larger models.

Safety Evaluations

Standard benchmarks (Table 9): DeepSeek-R1 achieves safety scores comparable to other frontier models. On the six-benchmark average: Claude-3.7-Sonnet 94.6%, o1 93.6%, GPT-4o 92.2%, DeepSeek-V3 96.3% (with risk control), DeepSeek-R1 95.0% (with risk control, without hiding CoT). Without risk control: DeepSeek-V3 91.5%, DeepSeek-R1 85.9% (hiding CoT) / 89.7% (not hiding CoT). The paper notes that "R1 exhibits a significant performance gap compared to other models on the HarmBench benchmark" due to poor performance on intellectual property questions (e.g., failing to refuse when asked to generate copyrighted lyrics).

Fine-grained in-house taxonomy (Table 10): Across four major safety categories (Discrimination, Illegal, Harmful, Ethical), DeepSeek-R1 with risk control achieves unsafe rates of 8.5% overall (vs. 10.7% for Claude-3.7-Sonnet and 22.0% for GPT-4o). Without risk control, the unsafe rate rises to 25.2%. The model performs best on Illegal and Ethical categories, and worst on Discrimination and Harmful Behavior. The rejection rate with risk control is 27.3% (vs. Claude's 3.6% and GPT-4o's 17.1%).

Multilingual safety (Figure 14): Across 50 languages, DeepSeek-R1 with risk control achieves an 85.9% safety score, approaching Claude-3.7-Sonnet's 88.3%. Without risk control, the score is 74.2%, comparable to GPT-4o's 75.2%. The paper finds zero high-risk languages (below 60 points) for DeepSeek-R1 without risk control, suggesting no obvious language-specific safety vulnerabilities.

Jailbreak robustness (Table 11): Under jailbreak attacks (2,232 templates concatenated with safety questions), DeepSeek-R1 without risk control shows an unsafe rate of 85.9% (vs. 25.2% on original questions)—a 60.7 percentage point increase, the largest degradation among tested models. With risk control, the unsafe rate drops to 4.3%, and the rejection rate rises to 87.3%. This demonstrates that "compared to non-reasoning models, the two reasoning models—DeepSeek-R1 and o1—rely more heavily on the risk control system for security checks."

Ablation Studies and Robustness Checks

GRPO vs. PPO (Appendix A.3, Figure 4): On the MATH task with DeepSeek-Coder-V2-Lite (16B MoE, 2.4B active), GRPO consistently outperforms PPO with default GAE parameters (λ = 0.95). When PPO's λ is tuned to 1.0 (removing temporal smoothing in advantage estimation), PPO approaches but does not surpass GRPO. The paper notes that PPO requires "additional computational cost for hyperparameter optimization" and the value model's memory overhead, making GRPO preferable for large-scale training. This ablation validates the core architectural choice of eliminating the critic model.

Language consistency reward (Appendix B.6, Figure 7): On DeepSeek-R1-Distill-Qwen-7B, training without the LC reward leads to gradually deteriorating language consistency (dropping from ~0.98 to ~0.88 over 5,000 steps). With the LC reward, consistency remains stable at ~0.97. On benchmarks: AIME accuracy is comparable (slightly higher without LC reward at some steps, ending at ~0.57 with vs. ~0.59 without at step 5,000), LiveCodeBench shows a small degradation (~0.48 with vs. ~0.50 without), confirming the paper's statement that "such alignment results in a slight degradation in the model's performance, but this reward aligns with human preferences, making the output more readable."

Reward hacking with model-based rewards (Appendix B.5, Figure 6): When training with the helpful reward model for extended steps, the reward score increases (from ~3.0 to ~5.0 over 700 steps) while Codeforces test pass@1 decreases (from ~0.35 to ~0.29). This is direct evidence of reward hacking: "if the reward model contains systematic biases or inaccuracies, the LLM may learn to generate responses that are rated highly by the model but diverge from authentic human preferences." This motivates the paper's decision to limit model-based reward training to only the final 400 of 1,700 steps in the second RL stage.

Clipping hyperparameter ε (Section 3.2.1): The paper sets ε = 10 for DeepSeek-R1 training, which is substantially larger than standard PPO values (typically 0.1–0.2). The paper notes: "A lower value can lead to the truncation of gradients for a significant number of tokens, thereby degrading the model's performance, while a higher value may cause instability during training." This ablation finding—that large clipping ranges are necessary for the substantial policy shifts during reasoning emergence—is presented as a practical insight rather than a formal ablation with comparison values.

Maximum generation length increase (Section 2.1): The paper notes that increasing the maximum length from 32,768 to 65,536 tokens at step 8,200 coincided with a significant performance jump. While not framed as a controlled ablation, this is a robustness-relevant observation: the model's reasoning capability growth was constrained by the earlier length limit, and providing more headroom unlocked further improvement. This suggests that output length limits are an important hyperparameter for pure RL reasoning training.

Cold-start SFT vs. no SFT (Table 3, R1-Zero vs. Dev1): The 18.9 percentage point drop in AIME performance from R1-Zero (77.9%) to Dev1 (59.0%) when introducing cold-start SFT before RL is presented as evidence that SFT constrains reasoning exploration. This is not a formal ablation study (the cold-start SFT includes other changes like human refinement and first-person perspective), but the magnitude of the drop strongly supports the paper's claim that human priors can interfere with RL-based reasoning discovery.

Distillation-only vs. RL-only for small models (Table 16): The 25.6 percentage point gap between distilled and RL-trained 32B models on AIME is a striking finding. The paper frames it as evidence that "distilling more powerful models into smaller ones yields excellent results, whereas smaller models relying on the large-scale RL mentioned in this paper require enormous computational power and may not even achieve the performance of distillation." This boundary condition—that pure RL's effectiveness depends critically on model scale—is one of the paper's most important qualifications.

Impact of including non-reasoning data in SFT (Table 3, Dev2 vs. Dev3): The addition of ~200K non-reasoning SFT samples in stage 3 yields: Aider-Polyglot improves 25.6% → 44.8% (+19.2 pp), AlpacaEval2.0 improves 55.8 → 62.1 (+6.3 pp), IF-Eval improves 72.0% → 78.1% (+6.1 pp). This validates the paper's claim that the second SFT stage "enabl[es] the model to not only excel in reasoning tasks but also demonstrate advanced writing capabilities." The contrast with the cold-start SFT (which harmed reasoning) highlights the importance of SFT timing and content: SFT using RL-discovered reasoning traces plus general data helps, while SFT using only human-curated reasoning traces (without prior RL) hurts.

Simple math CoT examples (Appendix B.3.2, Listing 3): The paper includes few-shot prompts for simple math problems to ensure the model doesn't overthink trivial queries. An example shows "1 + 2 = ?" receiving a concise CoT of roughly 50 words. This is a data curation robustness measure rather than a formal ablation, addressing the risk that the model might learn to always generate long chains of thought regardless of problem difficulty. Figure 18 provides evidence this works: the model uses fewer thinking tokens for easier problems.

DeepSeek-V3 as judge for non-rule-verifiable data (Appendix B.3.3, Listing 4): The paper uses DeepSeek-V3 to judge correctness for some SFT data where rule-based verification is impossible. This introduces potential model bias into the training data. The paper doesn't ablate the impact of this choice versus human-only verification, but notes that "in this stage, we expand the dataset by incorporating additional data, some of which uses a generative reward model." The reliance on a potentially biased judge for data quality is an uncontrolled variable in the pipeline.

Critical Assessment

Does the paper demonstrate that reasoning can emerge from pure RL without human demonstrations?

The evidence is strong but narrower than the broad claim suggests. The paper convincingly shows that DeepSeek-R1-Zero, trained from DeepSeek-V3-Base with only rule-based outcome rewards, achieves 77.9% on AIME 2024 and exhibits emergent behaviors like self-reflection and verification (Figure 1, Table 2, Figure 9). The "aha moment" (Table 2) and the quantified increase in reflective word frequency (Figure 9a) provide compelling process evidence that these behaviors were not present in the base model and emerged during training.

However, several qualifications are necessary:

The claim is demonstrated on a single base model family. All results use DeepSeek-V3-Base or Qwen/Llama variants. The paper explicitly tested only 7B dense, 16B MoE, and 32B+ dense models within this architectural lineage. The boundary condition—that models below ~32B parameters fail (Appendix G.1)—is well-documented, but whether the emergent reasoning generalizes to other model families (Gemini, Claude, etc.) with different pretraining data, tokenizers, and architectural choices is untested. The paper states that V3-Base's pretraining data "contains a substantial amount of mathematical and code-related content," which may be a critical enabling condition. A base model pretrained primarily on non-technical text might not exhibit the same emergent reasoning under pure RL, but this is not tested.

"No human demonstrations" is true for reasoning, but not for formatting. The template (Table 1) provides a structural prior: "The assistant first thinks about the reasoning process in the mind and then provides the user with the answer." This implicitly teaches the model to separate reasoning from answering and to reason before answering. The format reward reinforces this. While no content-level demonstrations are provided, the structural scaffolding is non-trivial—it tells the model that it should reason, just not how. An interesting ablation that is not run: what happens if the model is trained with only the accuracy reward and no format constraints? Would it still discover chain-of-thought reasoning, or would it learn to output answers directly? The paper doesn't explore this.

The base model may have been exposed to reasoning traces during pretraining. Appendix A.1 acknowledges: "we have observed that some web pages contain a significant number of OpenAI-model-generated answers, which may lead the base model to acquire knowledge from other powerful models indirectly." Additionally, the pretraining data "contains a substantial amount of mathematical and code-related content." This means the base model may have already seen examples of step-by-step reasoning during pretraining. The claim "no human demonstrations" refers to the post-training phase specifically, but the base model's knowledge of reasoning patterns from pretraining is uncontrolled. The paper's experiments with Qwen2-Math-7B (released before any reasoning model launches) partially address this (Table 17), showing that pure RL still produces improvement over the instruct baseline, but the absolute AIME performance (22.3%) is modest and doesn't approach R1-Zero levels.

The distinction between "reasoning" and "specific task knowledge" is blurred. The MATH difficulty breakdown (Figure 8) shows that hard problems (level 5) improve from ~0.55 to 0.90 during RL training. Some of this improvement could be due to the model learning better math-specific strategies (algebraic manipulation, equation solving) rather than general reasoning strategies. The paper doesn't disentangle domain-specific skill improvement from domain-general metacognitive improvement. The reflective behaviors (Figure 9) suggest metacognitive gains, but the relative contribution of each is unknown.

Does the paper demonstrate that SFT is actively harmful for reasoning discovery?

Supported with conditions. The 18.9 percentage point drop from R1-Zero (77.9%) to Dev1 (59.0%) on AIME is striking and supports the claim that introducing human-style SFT before RL degrades reasoning. However:

The cold-start SFT introduced multiple changes simultaneously. The cold-start data was "converted by human annotators into a more natural, human conversational style" using first-person perspective, with language consistency, and with explicit solution formatting (Listing 1). The drop in AIME performance could be due to any combination of: (a) the model learning to use first-person perspective ("I") rather than the problem-solving perspective R1-Zero discovered, (b) the model spending tokens on conversational framing rather than reasoning, (c) the human-refined reasoning being suboptimal for the model's own reasoning process, or (d) simple catastrophic forgetting from the domain shift. The paper attributes the drop to human priors constraining exploration, but cannot isolate which specific aspect of the cold-start SFT is responsible.

The pipeline ultimately re-introduces SFT (stages 3 and 4) and achieves the best results. If SFT were universally harmful for reasoning, Dev3 and R1 should underperform Dev2. Instead, Dev3 improves over Dev2 on AIME (74.0% → 78.1%), and R1 marginally improves further (79.8%). The harmful effect is specific to SFT before the model has discovered its own reasoning strategies through RL, not SFT in general. The paper's position—that the sequencing matters—is well-supported, but the strong claim that "the SFT stage may impede a model's ability to explore and develop effective reasoning strategies" (Appendix A.2) should be qualified as "SFT before RL impedes exploration, while SFT after RL using RL-discovered outputs can improve performance."

The Dev1 → Dev2 comparison also confounds RL training with SFT presence. Dev1 received cold-start SFT; Dev2 received cold-start SFT + RL. The RL stage improved AIME from 59.0% to 74.0%, but we don't know what RL directly on V3-Base with language consistency reward would achieve—R1-Zero didn't use the LC reward, so the degradation from SFT vs. the benefit from RL are partially confounded.

Does the paper demonstrate that GRPO is superior to PPO for reasoning RL?

Partially demonstrated with a scale limitation. Figure 4 (Appendix A.3) shows GRPO outperforming PPO with default parameters on a 16B MoE model. The paper argues that with tuned λ = 1.0, PPO "approaches but does not surpass GRPO." However:

The comparison is only at 16B scale, not 671B. The computational cost of training a separate value model at 671B scale (doubling memory requirements) may be the more important practical consideration than the algorithmic performance difference. The paper's strongest argument for GRPO is the memory savings, not a clear performance advantage at scale. Would PPO with a value model training on 671B-parameter long CoT trajectories even be feasible? The paper doesn't test this, and the feasibility argument is empirically unsupported.

The λ tuning finding is post-hoc. The paper found that λ = 0.95 (default) performs poorly and λ = 1.0 performs better. But this tuning was discovered after the decision to use GRPO was made. We don't know whether further PPO hyperparameter tuning (learning rate, batch size, KL coefficient) could close the performance gap entirely. The paper's claim that GRPO is "a more practical alternative" is reasonable given the evidence, but the claim that PPO cannot match GRPO's performance with sufficient tuning is not rigorously proven.

The KL penalty difference between GRPO and PPO is a hypothesis, not an empirically verified mechanism. The paper argues that PPO's per-token KL penalty "implicitly penalizes the length of the response" (Appendix A.3). This is a plausible mechanism, but no ablation directly tests it—for example, by modifying PPO to use output-level KL penalty and measuring whether the length penalty disappears. The GRPO vs. PPO performance difference could be due to the group-based advantage estimation, the KL placement, or other factors.

Does the paper demonstrate that smaller models cannot develop reasoning through pure RL?

Supported with important nuance. The paper's Appendix G.1 states that 7B and 16B models "consistently failed to yield meaningful improvements" and exhibited "a tendency toward repetition." However:

The 7B experiment (Qwen2-Math-7B-Zero, Table 17) actually shows meaningful improvement over baselines. AIME 2024 performance of 22.3% vs. 7.9% for the instruct version and 9.3% for GPT-4o represents a 2.8× improvement. The 18.1% on AIME 2025 (vs. 4.6% for instruct) shows generalization. This is not "failure to yield meaningful improvements"—it's a substantial capability gain, just far below what 671B models achieve. The paper's framing of 7B models as "failing" is relative to the dramatic gains at larger scales, not an absolute failure.

The "repetition" failure mode is described but not quantified. The paper doesn't show examples of the 7B/16B repetition behavior, measure its frequency, or analyze whether it's a fundamental capacity limitation or a training dynamics issue (e.g., insufficient exploration, poor KL penalty tuning). The failure could be fixable with different hyperparameters or reward shaping for smaller models, but this is not explored.

The distillation vs. RL comparison at 32B (Table 16) has a confound. Qwen2.5-32B-Zero was trained with the R1-Zero methodology, but potentially with different data, different hyperparameter tuning, or for insufficient steps compared to the full R1-Zero training. The paper reports "over 10K steps" but doesn't specify whether this was sufficient for convergence or whether additional training would close the gap. The distilled model benefited from the full R1 training pipeline's outputs; the RL-trained model had to discover strategies from scratch. A fairer comparison would give the RL-trained model comparable total compute to the R1 training pipeline, but this isn't attempted.

Does the paper demonstrate that R1 matches or exceeds o1's performance?

On aggregate: yes for the benchmarks tested, with important caveats. Table 8 shows R1 essentially tied with o1-1217 on AIME (79.8% vs. 79.2%), MATH-500 (97.3% vs. 96.4%), and Codeforces (96.3 vs. 96.6 percentile), slightly behind on GPQA Diamond (71.5% vs. 75.7%), and behind on Aider-Polyglot (53.3% vs. 61.7%). However:

o1-1217 results are from official reports, not the paper's own evaluation. The paper notes: "Since accessing the OpenAI-o1-1217 API is challenging in mainland China, we report its performance based on official reports." This introduces potential evaluation protocol differences—prompt format, temperature settings, sample sizes may differ between OpenAI's reports and DeepSeek's evaluations. The paper's own evaluation protocol uses temperature 0.6 and top-p 0.95 with 64 samples for AIME, while OpenAI's evaluation protocol for o1 may differ. Direct head-to-head comparison under identical conditions is not performed.

The benchmarks where R1 trails o1 (GPQA, Aider) are not thoroughly analyzed. GPQA Diamond tests PhD-level STEM knowledge; R1's 71.5% vs. o1's 75.7% is a meaningful gap. Aider-Polyglot tests multi-language code editing; R1's 53.3% vs. o1's 61.7% is substantial. The paper doesn't investigate whether these gaps are due to base model knowledge differences, RL training data coverage, or fundamental architectural limitations. The paper acknowledges that software engineering tasks received limited RL data: "Due to the long evaluation times, which impact the efficiency of the RL process, large-scale RL has not been applied extensively in software engineering tasks."

The ChatbotArena evaluation (Figures 11–12) is crowdsourced and uncontrolled. While the Arena provides real-world human preference data, the user population, prompt distribution, and voting behavior are not controlled. The paper notes the ranking is "dynamically updated in real time as the number of votes increases" and the screenshot is from January 24, 2025—one week after release. The ranking may have changed subsequently. More importantly, the style-controlled leaderboard specifically attempts to separate content quality from presentation style, but the mechanism for doing so is not detailed.

Missing experiments that would strengthen the paper

No direct comparison between different RL algorithms at 671B scale. The GRPO vs. PPO comparison is only at 16B. Whether PPO could match or exceed GRPO at the full 671B scale with sufficient hyperparameter tuning is unknown.

No ablation of the format reward. What happens if the model is trained with only accuracy rewards and no format constraint? Would CoT still emerge, or would the model learn to output answers directly? This would test whether the structural prior (reasoning-then-answering) is necessary for emergent metacognition.

No ablation of group size in GRPO. The paper uses G = 16 outputs per group. How does performance scale with group size? Larger groups provide better advantage estimates but cost more compute. Smaller groups are cheaper but noisier. This tradeoff could be practically important for resource-constrained training.

No analysis of the effect of the length limit increase at step 8,200. The paper observes a performance jump coinciding with the length limit increase but doesn't isolate whether the jump was caused by the increased limit or would have occurred anyway. A controlled experiment holding the limit constant would clarify this.

No controlled comparison of cold-start SFT content. The paper shows that cold-start SFT degrades reasoning but doesn't test whether this is due to the first-person perspective, the language style, the specific reasoning content, or simply the act of fine-tuning on any reasoning data before RL.

No exploration of whether RL-discovered reasoning strategies transfer across domains. R1-Zero was trained on math, code, STEM, and logic. Do the reasoning strategies discovered in these domains transfer to entirely new domains not seen during RL (e.g., legal reasoning, medical diagnosis)? The paper doesn't test zero-shot transfer to held-out reasoning domains.

Limited analysis of the source of R1's improvements over V3 on non-STEM benchmarks. Figure 15 shows R1 improving over V3 on Social Sciences and Humanities categories of MMLU. The paper attributes this to "better understanding of the question" but provides no evidence for this mechanism. An analysis of whether R1's CoT for non-STEM questions includes different reasoning patterns than V3's direct answers would strengthen this claim.

No comparison to models trained with SFT-first-then-RL at comparable scale. The paper argues that SFT-first is harmful, but this claim is based on comparing R1-Zero (no SFT) to Dev1 (cold-start SFT with specific content). A direct comparison between the full R1 pipeline and a pipeline that starts with large-scale SFT on human reasoning traces before RL would more directly test the paper's central methodological claim.

Assessment of specific weaknesses

Test set size for in-house safety evaluation: The 28 subcategories each contain only 20 Chinese questions (translated to English for bilingual evaluation), totaling 1,120 questions. The multilingual extension uses 9,330 questions across 50 languages, but the per-language sample sizes for low-frequency languages are small (the paper notes "sampling translation" for low-frequency languages). The jailbreak evaluation concatenates 2,232 templates with the original safety questions, but the per-category jailbreak sample sizes aren't reported.

Potential data contamination despite decontamination efforts: The paper uses 10-gram overlap filtering, which cannot detect paraphrased test questions. The paper acknowledges this: "it is possible that benchmarks released before 2024 may suffer from contamination issues." For the 2024 competition math set and AIME 2025, contamination is less likely because these were released after training.

Single evaluation protocol for reasoning models: The paper uses pass@k with temperature 0.6 and top-p 0.95. While this is justified by the observation that greedy decoding produces higher repetition rates, it means the reported pass@1 is actually an average over k samples, not a single deterministic output. For practical deployment where users expect a single answer, performance may differ.

No latency or throughput measurements: The paper reports training costs (GPU hours) but doesn't report inference latency or throughput. For a model that generates 8,793 thinking tokens on average (Section E.4), the time-to-first-token and total generation time are practically important metrics that go unreported.

Safety evaluation judges are LLMs: The safety evaluation uses GPT-4o (2024-11-20) as judge. While the paper reports >95% consistency with human judgments on sampled results, the potential for systematic judge bias (particularly on nuanced ethical questions) is not deeply analyzed.

The relationship between CoT length and accuracy is correlational, not causal. Figure 18 shows that harder problems receive more thinking tokens on average. This could mean that longer thinking causes better accuracy, or that problems where the model happens to think longer (for whatever reason) are more likely to be solved correctly. The paper's language implies causation ("the model adaptively adjusts its computational effort"), but the experimental design doesn't establish this direction. An interventional experiment—forcing the model to generate more or fewer thinking tokens and measuring the effect on accuracy—would be needed to establish causality.

6. Limitations and Trade-offs

The Pure RL Approach Requires Base Models Above a Critical Capacity Threshold

The assumption or constraint. The paper's core finding—that sophisticated reasoning can emerge from pure reinforcement learning with outcome-based rewards—rests on a sharp capacity requirement. The base model must be sufficiently large for RL to induce meaningful reasoning improvements. Appendix G.1 documents this explicitly:

"During the initial phase of our development, we experimented with smaller-scale models, specifically a 7B dense model and a 16B Mixture-of-Experts (MoE) model, as the foundational architectures for RL training. However, these configurations consistently failed to yield meaningful improvements when evaluated on the AIME benchmark... We observed that as response lengths increased, these smaller models exhibited a tendency toward repetition and were unable to effectively leverage long chains of thought (CoT) to improve reasoning accuracy."

The paper reports that transitioning to larger models (32B dense, 230B MoE, 671B MoE) was necessary before "substantial performance gains attributable to pure RL training" were observed. The critical threshold appears to lie somewhere between 16B and 32B parameters—likely 32B dense or equivalent effective capacity.

The consequence. This capacity threshold has profound practical and scientific implications. For practitioners, it means the pure RL methodology is not a general-purpose training recipe—it is specifically a technique for very large models. Organizations without the resources to train or even fine-tune models at the 100B+ scale cannot meaningfully replicate the R1-Zero result on their own base models. The paper's recommendation—"future research in this area prioritize the use of sufficiently large and expressive models when aiming to validate the efficacy of RL from scratch"—effectively gates the approach behind substantial computational infrastructure.

Scientifically, the capacity threshold raises unanswered questions about what the larger models possess that smaller ones lack. Is it raw memorized knowledge (more math facts, more exposure to reasoning traces during pretraining)? Is it a qualitative shift in in-context learning or compositional generalization ability that emerges with scale? Is it simply greater representational capacity allowing more diverse exploration before mode collapse? The paper does not investigate the mechanism. This matters because without understanding why scale matters, it is impossible to predict whether architectural innovations, better pretraining data, or different RL algorithms could lower the threshold. The approach may be not just currently expensive but fundamentally tied to model scale in a way that cannot be engineered around.

What evidence exists in the paper. The evidence is qualitative and somewhat anecdotal rather than systematic. Appendix G.1 describes the 7B and 16B failures but provides no quantitative results—no accuracy curves, no examples of the repetition behavior, no measurements of how response quality degrades with length for small models. The Qwen2-Math-7B-Zero experiment (Table 17) complicates the narrative: this 7B model did show meaningful improvement from pure RL, achieving 22.3% on AIME 2024 (vs. 7.9% for the instruct version and 9.3% for GPT-4o). This is a 2.8× improvement over baselines—hardly "failure to yield meaningful improvements." The paper's framing of 7B models as failing appears to mean "failed to reach frontier-level performance," not "failed to improve at all." The distinction matters: smaller models can develop some reasoning through pure RL, just far less dramatically than 671B models.

The distillation comparison in Table 16 provides indirect evidence: Qwen2.5-32B-Zero (pure RL for >10K steps) achieves 47.0% on AIME, while DeepSeek-R1-Distill-Qwen-32B (distilled from the 671B R1) achieves 72.6%. The 25.6 percentage point gap suggests that even at 32B scale, pure RL substantially underperforms distillation from a larger model. However, this comparison confounds model scale with training methodology, data quality, and potentially insufficient RL training (the paper doesn't establish that >10K steps was enough for convergence at 32B).

Mitigation status. The paper does not attempt to lower the capacity threshold or to characterize it precisely. The distillation experiments (Appendix F) provide a practical workaround—train a large model with pure RL, then distill into smaller models—but this does not address the underlying limitation: the RL discovery process itself still requires a very large model. The paper suggests that "advancing beyond the boundaries of human intelligence may still require more powerful base models and larger-scale reinforcement learning," implying the capacity threshold is a fundamental property rather than an engineering limitation. No experiments are proposed to test whether different RL algorithms, reward structures, or curriculum learning approaches could enable pure RL reasoning emergence at smaller scales.

Pure RL Produces Models with Fundamentally Limited Practical Usability

The assumption or constraint. DeepSeek-R1-Zero demonstrates that pure RL with rule-based rewards produces strong reasoning—but also produces outputs that are poorly readable, mix languages unpredictably, and perform weakly on non-reasoning tasks. The paper states in Section 3:

"DeepSeek-R1-Zero struggles with challenges like poor readability, and language mixing, as DeepSeek-V3-Base is trained on multiple languages, especially English and Chinese. Furthermore, the rule-based RL training stage of DeepSeek-R1-Zero is narrowly focused on reasoning tasks, resulting in limited performance in broader areas such as writing and open-domain question answering."

Table 3 quantifies this: R1-Zero achieves only 24.7 on AlpacaEval2.0 (vs. DeepSeek-V3's 70.0) and 53.6 on ArenaHard (vs. V3's 85.5). The model scores 46.6 on IF-Eval (instruction-following), far below V3's 86.1. These are not marginal degradations—they represent a model that is essentially unusable for general-purpose deployment despite its reasoning strength.

The consequence. This limitation means that pure RL alone does not produce a deployable product. The paper implicitly acknowledges this by developing the full DeepSeek-R1 pipeline (four stages, including cold-start SFT, rejection sampling, and model-based reward RL) specifically to address these deficiencies. The R1-Zero result is a scientific demonstration, not a practical model. Any organization seeking to deploy a reasoning model would need to implement substantial additional training stages—SFT for readability, RL with language consistency rewards, general-domain data incorporation—that the paper did not ablate for their independent contribution. The pure RL methodology is therefore a component of a training pipeline, not a standalone solution for building usable models.

The distinction between capability and usability matters for interpreting the paper's central claim. The paper argues that "sophisticated reasoning behaviors... can emerge organically during the reinforcement learning process" without human demonstrations. This is true for the cognitive behaviors (self-reflection, verification, backtracking) but not for the communicative behaviors (clear formatting, language consistency, appropriate conciseness). A model that thinks brilliantly but communicates in an unreadable language mixture is not a functioning reasoning assistant. The paper's framing occasionally elides this distinction between reasoning capability and usable reasoning.

What evidence exists in the paper. The gap between R1-Zero and R1 on general benchmarks is stark and well-documented in Table 3:

The language mixing issue is documented qualitatively in Section 3 and the cold-start SFT description (Appendix B.3.2), which notes that DeepSeek-V3 is prompted to "Translate the thinking process to the same language as the question" during data refinement. The language consistency reward ablation (Appendix B.6, Figure 7) quantifies the issue on a 7B distilled model: without the LC reward, language consistency drops from ~0.98 to ~0.88 over 5,000 RL steps.

The paper does not provide examples of R1-Zero's unreadable outputs or quantify the frequency of language mixing. The reader must infer the severity from the large gaps in Table 3 and the fact that the authors felt compelled to build a four-stage pipeline to address these issues.

Mitigation status. The entire DeepSeek-R1 pipeline is the mitigation. Cold-start SFT (stage 1) addresses readability and formatting. The language consistency reward (stage 2) addresses language mixing. The second SFT stage with general data (stage 3) addresses broad task performance. The second RL stage with model-based rewards (stage 4) addresses helpfulness and harmlessness. Table 3 shows this multi-stage approach largely closes the gap with DeepSeek-V3 on general benchmarks while preserving reasoning gains. However, each mitigation introduces its own costs and tradeoffs. The cold-start SFT degrades reasoning (R1-Zero 77.9% → Dev1 59.0% on AIME). The language consistency reward slightly reduces benchmark performance (Appendix B.6). The model-based rewards risk reward hacking (Appendix B.5). The paper does not present ablations showing which specific components of the R1 pipeline are necessary and which could be simplified, leaving practitioners uncertain about the minimal set of additional training stages needed to convert a pure-RL reasoning model into a usable product.

The Pipeline Relies on a Multi-Stage, Empirically-Tuned Recipe with No Theoretical Guarantees

The assumption or constraint. The DeepSeek-R1 training pipeline consists of four stages with numerous hand-tuned hyperparameters and design decisions, each justified by empirical performance rather than theoretical principle. The cold-start SFT uses "thousands" of examples with specific human refinement procedures (first-person perspective, language consistency, summary generation). The first RL stage adds a language consistency reward weighted equally with accuracy and format rewards. The second SFT stage uses rejection sampling with a DeepSeek-V3 judge for non-rule-verifiable data. The second RL stage uses model-based rewards for exactly the final 400 of 1,700 steps because "more training steps with the model based preference reward signal may lead to reward hacking" (Appendix B.5). The GRPO clip ratio ε is set to 10 because "a lower value can lead to the truncation of gradients" while "a higher value may cause instability" (Section 3.2.1). The maximum generation length is increased from 32,768 to 65,536 at step 8,200 of R1-Zero training, coinciding with a performance jump.

Each of these choices represents a degree of freedom that a practitioner replicating the approach would need to tune for their own base model, data distribution, and compute budget. The paper provides the final hyperparameter values but little insight into how they were arrived at (grid search? manual tuning? single attempts that happened to work?) or how sensitive performance is to deviations.

The consequence. The pipeline's complexity and reliance on empirical tuning means that the paper demonstrates existence of a successful recipe, not a robust methodology with predictable transfer. A team attempting to apply pure RL reasoning training to a different base model (not DeepSeek-V3-Base), a different reward structure (not the paper's specific math/code/STEM/logic mix), or a different compute budget would need to re-discover the correct staging, mixing ratios, and hyperparameters through expensive trial and error. The paper provides no guidance on how to adapt the pipeline to new settings.

This matters particularly for the cold-start SFT content. The paper describes a specific data creation process involving R1-Zero generation, human refinement, and LLM-based expansion using DeepSeek-V3. The quality and style of this cold-start data—first-person perspective, conversational tone, language consistency—are described as "primarily product-driven" and "DeepSeek-engineered heuristics" (Appendix B.3.2). A different team targeting different user preferences (third-person, more formal, different languages) would need to re-develop the cold-start data creation pipeline, with unknown effects on downstream reasoning performance. The paper does not ablate cold-start data properties (e.g., first-person vs. third-person, with vs. without explicit reflection markers) to determine which aspects are essential for the pipeline's success.

The empirical nature of the pipeline also means that the paper's central scientific claim—that reasoning emerges from pure RL—is demonstrated through a specific implementation with many auxiliary design choices, making it difficult to attribute causality. The R1-Zero experiment (no SFT, no language reward, no model-based rewards) cleanly isolates the effect of pure RL. But the path from R1-Zero to R1 involves multiple simultaneous changes whose individual contributions are not ablated. When R1 outperforms R1-Zero on general benchmarks, we cannot determine whether the improvement comes from cold-start SFT, the language consistency reward, the second SFT stage, the second RL stage, or specific interactions between stages.

What evidence exists in the paper. The stage-by-stage results in Table 3 provide the only decomposition of the pipeline's effects. However, each stage transition changes multiple variables simultaneously:

• R1-Zero → Dev1: adds cold-start SFT (thousands of human-refined examples with specific style)
• Dev1 → Dev2: adds RL with language consistency reward (but also continues RL training)
• Dev2 → Dev3: adds second SFT with 800K samples (600K reasoning + 200K general), but also uses a different base (DeepSeek-V3-Base) rather than continued training from Dev2
• Dev3 → R1: adds second RL with mixed rewards (but also changes temperature from 1.0 to 0.7)

The paper does not report ablations that would isolate specific design choices. We don't know:

• Whether the language consistency reward alone (without cold-start SFT) would fix readability
• Whether the second SFT stage's general data is necessary or whether pure reasoning SFT data would suffice
• Whether the model-based rewards in stage 4 improve anything beyond what rule-based rewards would achieve with continued training
• Whether the 400-step limit on model-based rewards is optimal or just a safe choice
• How sensitive the final R1 performance is to the cold-start data size, the rejection sampling threshold, or the SFT data mixture ratios

The paper's discussion of unsuccessful attempts (Appendix G.2) provides some negative results (PRMs, MCTS) that informed the final design, but these are retrospective justifications, not controlled comparisons.

Mitigation status. The paper does not present the pipeline as a theoretically principled framework. Section 3 describes the stages sequentially with justifications for each design choice, but these justifications are empirical ("we find that," "we observe that") rather than derived from underlying principles. The paper does not claim to have optimized the pipeline or to have established that all stages are necessary. The open-source release of model weights and data partially mitigates the replication challenge—other teams can study the effects rather than re-discovering them—but does not provide guidance for adapting the pipeline to new settings. Future work on "the exploration of the RL stage" for distilled models (Appendix F) is suggested, but no systematic study of pipeline sensitivity is proposed.

The Approach Provides No Path Forward for Problems That Lack Reliable Verifiers

The assumption or constraint. The paper's methodology depends fundamentally on the existence of reliable, automated reward signals for the target tasks. For R1-Zero, these are rule-based: mathematical answer matching via sympy, code correctness via compiler test cases, multiple-choice accuracy via ground-truth comparison. The paper acknowledges this dependency in Section 6:

"The success of pure RL depends on reliable reward signals. In this study, we ensure reward reliability through a reasoning-domain rule-based reward model (RM). However, such dependable RMs are difficult to construct for certain tasks, such as writing."

The paper further notes in Appendix G.2 that even for reasoning tasks, certain problem types are excluded: "Mathematical proofs are excluded because it is difficult to determine their correctness" (Appendix B.3.1). For the R1 training pipeline, tasks without rule-based verifiability are handled through supervised fine-tuning on human-labeled or LLM-judged data rather than through RL, and RL with model-based rewards is restricted to only the final 400 steps due to reward hacking concerns.

The consequence. This limitation means that the pure RL methodology cannot be directly applied to the majority of tasks that language models are used for in practice. Open-ended generation (creative writing, dialogue, summarization), subjective evaluation (helpfulness, engagement, nuance), and tasks requiring complex multi-step verification that resists automation (mathematical proofs, legal reasoning, medical diagnosis) all lack the kind of reliable, automated reward signal that the paper's approach requires. For these tasks, the paper falls back on conventional SFT and limited RL with model-based rewards—essentially the standard post-training paradigm it argues against for reasoning.

This has two implications. First, it bounds the scope of the paper's central claim: reasoning can emerge from pure RL specifically for tasks where correctness is machine-verifiable. This is an important class of tasks (mathematics, programming, formal logic, certain scientific QA) but far from all reasoning. The paper does not establish whether pure RL would produce emergent reasoning for tasks where the reward signal is noisy, delayed, or subjective.

Second, it creates a methodological cliff between verifiable and non-verifiable domains. For math and code, the paper advocates pure RL from the base model, bypassing SFT entirely. For writing and general QA, the paper uses conventional SFT and limits RL to prevent reward hacking. A practitioner building a general-purpose model must maintain two entirely different training regimes and somehow combine them—as the R1 pipeline does—without clear principles for how to balance the two or whether the reasoning strategies discovered through pure RL transfer usefully to the SFT-trained domains.

What evidence exists in the paper. The contrast between reasoning and general benchmarks in Table 3 provides indirect evidence. R1-Zero achieves 77.9% on AIME (pure RL, verifiable rewards) but only 24.7 on AlpacaEval2.0 (no verifiable reward available). The R1 pipeline partially closes this gap (87.6 on AlpacaEval2.0), but the improvement comes from SFT on human-curated data and limited model-based RL—not from extending the pure RL methodology to non-verifiable domains.

The paper's discussion of reward hacking (Appendix B.5, Figure 6) provides direct evidence of what happens when model-based rewards are used for extended RL training: the reward score increases while actual Codeforces performance decreases. This is the failure mode that prevents extending the pure RL approach to domains without rule-based verifiers. The paper's mitigation—limiting model-based reward training to 400 steps—is an acknowledgment that the problem is not solved, only managed.

The paper does not report experiments on whether reasoning strategies discovered through pure RL on verifiable tasks transfer to improved performance on non-verifiable tasks without additional training. The MMLU results (Figure 15) show R1 improving over V3 on Humanities and Social Sciences, which the paper attributes to "better understanding of the question," but this improvement comes from the full R1 pipeline (which includes SFT on general data), not from pure RL transfer.

Mitigation status. The paper treats the verifier requirement as a fundamental constraint, not a limitation to be overcome. Section 6 states: "for complex tasks that cannot be effectively evaluated by a reliable reward model, scaling up pure RL methods remains an open challenge." The suggested path forward is developing "a robust reward model" for such tasks, but the paper's own experience with reward hacking (Figure 6) and process reward models (Appendix G.2) suggests this is a difficult research problem, not an incremental engineering challenge. No concrete approach for building robust reward models for subjective or open-ended tasks is proposed.

The paper does not explore alternative reward structures that might extend the pure RL methodology without requiring perfect verifiers—for example, using constrained generation to keep outputs within verifiable bounds, training discriminator models adversarially, or using human feedback more efficiently. These are left entirely to future work.

The Pipeline's Most Expensive Stage (Difficulty Estimation and Data Generation) Is Not Ablated for Necessity

The assumption or constraint. The DeepSeek-R1 training pipeline involves two extremely computationally expensive data generation steps whose individual contributions are not measured. The rejection sampling stage (Section 3, Appendix B.3.3) generates ~600K reasoning trajectories by sampling multiple responses per prompt from the Dev2 checkpoint and retaining only correct ones. The paper does not report the total number of samples generated to produce these 600K correct trajectories—but it must be substantially larger, likely millions of samples, given that Dev2 achieves 74.0% on AIME (pass@1) and likely lower accuracy on the broader reasoning prompt distribution. Each sample involves generating up to 32,768 tokens from a 671B-parameter model. The computational cost of this data generation—separate from the SFT training that uses the data—is not reported.

Similarly, the paper notes in Section 2.1 that R1-Zero training involves generating 8,192 outputs per rollout (32 questions × 256 rollouts, split into mini-batches). Over 10,400 training steps, this amounts to approximately 85 million complete outputs generated from the 671B model during RL training alone. The total inference compute spent on generating training data (rollouts for RL, rejection sampling for SFT) likely dwarfs the compute spent on updating model parameters. The paper reports training costs in GPU hours (Table 7: 147K total) but this figure appears to include only the parameter update computation, not the inference compute for rollout generation, which the decoupled infrastructure (Appendix B.1) runs on separate vLLM workers.

The consequence. If the inference compute for data generation substantially exceeds the training compute, then the headline result—that reasoning emerges from RL with rule-based rewards—is misleading in an important way. The mechanism of improvement may be not just RL's credit assignment but also the massive scale of exploration: the model sees millions of its own outputs with outcome feedback, effectively performing a brute-force search over reasoning strategies where successful patterns are reinforced. This is qualitatively different from "learning to reason" in a sample-efficient way. A practitioner hoping to replicate the results with a smaller exploration budget (fewer rollouts, less rejection sampling) may find that the approach fails not because RL cannot induce reasoning but because insufficient exploration prevents discovery of effective strategies.

This matters for the paper's implicit claim about democratization. The paper argues that pure RL "obviat[es] the need for human-labeled reasoning trajectories" (Abstract), reducing the scaling constraint from human effort to compute. But if the required compute for data generation is enormous—requiring millions of samples from a 671B-parameter model—then the approach merely substitutes one expensive resource (human expertise) for another (large-scale inference compute), which may be equally inaccessible to under-resourced groups. The distillation results (Appendix F) partially address this by showing that smaller models can inherit the reasoning capabilities, but the discovery process itself remains gated behind massive compute.

What evidence exists in the paper. The paper reports total training costs (Table 7): R1-Zero 101K GPU hours, SFT data creation 5K GPU hours, R1 41K GPU hours, total 147K H800 GPU hours (~$294K at$2/GPU-hour). However, the paper does not disaggregate these costs into inference (data generation) vs. training (parameter updates) components. The SFT data creation cost (5K GPU hours) is surprisingly low given the description of "sampling multiple responses" per prompt and retaining only correct ones, suggesting either that the sampling was done with smaller models or that the cost accounting is incomplete.

The paper does not report:

• The total number of samples generated during R1-Zero RL training (rollouts for advantage estimation and policy updates)
• The total number of samples generated during R1 RL stages
• The total number of samples generated during rejection sampling, or the acceptance rate (fraction of generated samples that were correct and retained)
• The inference compute cost as a fraction of total training cost
• The wall-clock time required for data generation vs. training

Without these numbers, it is impossible to assess whether the approach is practically replicable at smaller scale or with different compute budgets. The paper's description of the RL infrastructure (Appendix B.1) indicates that rollout generation runs on "multiple vLLM workers" with expert parallelism and MTP speculative decoding—engineering optimizations that may be necessary for the approach to be feasible but are not quantified in terms of their contribution to throughput or cost reduction.

Mitigation status. The paper does not address this limitation. The cost reporting (Table 7) is presented as a single aggregate number without decomposition. The paper does not ablate the number of samples per prompt, the group size G in GRPO, or the rejection sampling acceptance threshold to characterize the tradeoff between exploration budget and final performance. The paper does not report whether RL training would succeed with fewer rollouts (reducing the exploration budget) or with stronger priors (reducing the need for exploration). This leaves a significant gap in understanding the resource requirements for replicating the approach.

The distillation experiments (Appendix F) partially address the cost concern for deployment—showing that smaller models can achieve strong performance through SFT on R1 outputs—but do not address the cost of the discovery process. The paper does not explore whether a smaller-scale version of the R1-Zero training (fewer parameters, fewer samples, shorter training) could discover reasoning strategies that, while weaker than R1-Zero's, could then be scaled up through distillation or continued training. The minimum viable exploration budget for emergent reasoning remains unknown.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper initiates a paradigm shift in the relationship between human data and machine reasoning—not by incrementally improving supervised fine-tuning, but by demonstrating that sophisticated reasoning behaviors can emerge from reward optimization alone when the base model possesses sufficient capacity. The shift is not that RL can improve reasoning (this was known) but that human demonstrations of reasoning are not merely unnecessary—they can be actively harmful as a starting point, because they train models to imitate the presentation of reasoning (clean, linear, reflection-free) rather than the process of reasoning (messy, iterative, self-correcting).

The magnitude of this shift should not be overstated. The paper does not render SFT obsolete. The R1 pipeline deliberately re-introduces SFT at multiple stages for readability, general capability, and alignment—the final R1 model depends on it. What the paper changes is the default assumption about where reasoning capability comes from. Prior to this work, the dominant post-training paradigm (Ouyang et al., 2022; Chung et al., 2024) treated SFT as the foundation that RL refines. After this work, at least for reasoning tasks with verifiable outcomes, the default should be inverted: RL provides the discovery mechanism that finds strategies beyond human priors; SFT provides the distillation and alignment mechanism that makes those strategies deployable. The two techniques have asymmetric roles, and the sequencing is critical.

The paper resolves a latent tension in the field about whether scaling inference compute and scaling RL training compute are fundamentally different. Prior work on test-time compute scaling (Snell et al., 2025; Brown et al., 2024) treats computation as an external mechanism—generate more samples, search more broadly, aggregate better. DeepSeek-R1 demonstrates that the model can learn to internalize compute allocation during training, generating more thinking tokens for harder problems and fewer for easier ones (Figure 18) without being explicitly programmed to do so. This blurs the boundary between training-time capability acquisition and inference-time compute scaling: the model learns during RL to dynamically scale its own computation, making external scaling mechanisms (majority voting, beam search) complementary rather than primary. The implication is that future work on inference-time compute should not treat the model as fixed—training the model to scale its own computation may yield larger gains than designing better search procedures over frozen models.

The paper also reconciles conflicting intuitions about process rewards. Several prior works (Lightman et al., 2024; Uesato et al., 2022; Wang et al., 2023a) argued that dense, step-level feedback is necessary for effective reasoning RL. The paper's Appendix G.2 presents a systematic case that PRMs introduce three practical failures—ambiguous step granularity, unreliable correctness determination, and inevitable reward hacking—that outweigh their theoretical advantages. With sufficient model scale and long chain-of-thought, sparse outcome rewards suffice. This does not mean PRMs are worthless—the paper acknowledges their value for inference-time reranking—but it redirects research investment away from building ever-better process reward models for RL training and toward scaling outcome-based RL with larger models and longer reasoning horizons.

A less obvious but important shift concerns what counts as a "reasoning improvement." The paper's difficulty-stratified analysis (Figures 8, 17; Table 14) demonstrates that RL improves performance primarily on medium and hard problems, not on easy ones where the base model already succeeds. This differentiates outcome-based RL from prompt engineering techniques (self-consistency, least-to-most) that improve performance across the difficulty spectrum by better extracting existing capability. RL is increasing the ceiling, not raising the floor. For researchers evaluating reasoning improvements, this implies that benchmark-level aggregate metrics can mask the mechanism of improvement—a method that improves easy-problem accuracy through better prompt formatting looks identical to one that improves hard-problem accuracy through deeper reasoning, but the source of gain is fundamentally different.

Finally, the paper establishes a capacity threshold for emergent metacognition through RL that changes how the field should think about model scale. The finding that 7B and 16B models "consistently failed to yield meaningful improvements" while 32B+ models succeeded (Appendix G.1) suggests that the ability to discover novel reasoning strategies through RL is not a smooth function of parameter count—it may be a phase transition. This has direct implications for model architecture research: future work on scaling laws for post-training must account for qualitative capability thresholds, not just continuous performance improvements. It also implies that RL-based reasoning research at small scale may produce misleadingly negative results, because the phenomenon under study simply doesn't occur below the critical capacity. The field's history of negative results for self-correction and reflection (Huang et al., 2023) may partly reflect experiments conducted below this threshold.

Follow-Up Research This Work Enables

Systematic characterization of the capacity threshold for emergent reasoning through RL. The paper documents that 7B and 16B models fail while 32B+ models succeed, but provides no quantitative scaling curve. A systematic study would train base models at 1B, 3B, 7B, 13B, 30B, 70B, and 150B+ parameters using identical pure-RL protocols (same data, same GRPO hyperparameters, same reward structure) and measure not just final benchmark accuracy but the emergence trajectory of specific reasoning behaviors—reflection frequency, backtracking rate, strategy-switching entropy—as a function of model scale. The key question is whether these behaviors exhibit a smooth power-law relationship with scale, a sharp phase transition at some critical threshold, or different thresholds for different behaviors. This would determine whether the capacity requirement is a fundamental limitation or an artifact of current training recipes that could be engineered around (e.g., through better exploration bonuses, auxiliary losses, or architectural modifications).

Can process-level behaviors be incentivized without process-level rewards? The paper's outcome-based RL produced process-level behaviors (reflection, verification) as an emergent byproduct of optimizing final-answer correctness. This raises a specific mechanistic question: what properties of the base model, the reward structure, and the RL algorithm determine which process-level strategies emerge? A controlled experiment would manipulate three factors independently: (a) whether the reward is outcome-only or includes process-level shaping, (b) whether the base model's pretraining data includes reasoning traces (clean vs. filtered), and (c) whether the generation length is constrained or unconstrained. The hypothesis—implied but not tested in the paper—is that long, unconstrained generation horizons are the critical enabling factor, because they allow the model to experiment with intermediate behaviors (pausing, checking, backtracking) before committing to a final answer. If true, this would mean that process-level capabilities emerge not from process-level supervision but from process-level opportunity—the model needs room to explore before answering, not labels on its exploration steps.

Does pure RL discover reasoning strategies that are genuinely inaccessible to SFT-trained models, or does it merely find them more efficiently? The paper shows that pure RL (R1-Zero) reaches 77.9% on AIME while cold-start SFT followed by RL (Dev1 → Dev2) reaches 74.0%—a small gap. But this comparison is confounded: the cold-start SFT was a specific, small dataset with human refinement. A stronger test would be to compare pure RL against a model trained on the maximum possible SFT data—hundreds of thousands of high-quality human reasoning traces across diverse domains, representing the best that human expertise can provide—then apply identical RL training to both. If pure RL still produces different or superior reasoning strategies (qualitatively, not just quantitatively), this would establish that the approach discovers strategies humans don't or can't demonstrate, rather than just being more sample-efficient at finding the same strategies. The diagnostic would be behavioral: do pure-RL models exhibit reasoning patterns (specific backtracking sequences, verification heuristics, strategy-switching criteria) that are absent from or rare in the SFT training distribution?

Distillation as a probe for the nature of RL-discovered reasoning. The paper shows that distilling R1's outputs into smaller models (Appendix F) produces dramatically better reasoning than training those same small models with pure RL directly (Table 16: 72.6% vs. 47.0% on AIME for 32B). This gap could be investigated as a scientific probe: if R1's outputs are used as SFT data, does the distilled model learn to replicate R1's specific reasoning strategies, or does it learn something more general that improves its own reasoning? By comparing the behavioral patterns (reflection frequency, backtracking depth, error types) of (a) the full R1 model, (b) the distilled model trained on R1 outputs, and (c) the same base model trained on an equal amount of high-quality human data, researchers could determine whether distillation transfers specific strategic knowledge or simply provides better-configured base policies that the small model can then extend. This has practical implications for whether distillation datasets should be curated to maximize output correctness or to maximize strategic diversity.

What is the minimum verifier quality needed for outcome-based RL to induce reasoning? The paper uses deterministic, rule-based verifiers (math expression matching, compiler test cases). But many practically important reasoning domains have noisy, incomplete, or proxy-based verifiability—automated essay scoring, code review quality, medical diagnosis correctness. A systematic degradation study would take a task with perfect verifiability (like MATH) and deliberately introduce noise into the reward signal—random flipping at rates of 1%, 5%, 10%, 20%—and measure whether emergent reasoning behaviors degrade smoothly with reward noise or exhibit a threshold effect below which they collapse entirely. The hypothesis from the paper's discussion of reward hacking (Appendix B.5) is that noisy rewards not only slow learning but actively teach the model to exploit verifier biases rather than develop genuine reasoning. Characterizing this tradeoff would establish the practical scope of the pure-RL approach and identify which real-world tasks are within reach.

Can the multi-stage pipeline be simplified or automated? The DeepSeek-R1 pipeline involves four stages with multiple hand-tuned design decisions. A practical research direction is to test whether specific stages can be merged or eliminated: (a) can cold-start SFT be replaced by a simple formatting prompt without human-curated examples? (b) can the language consistency reward alone (without cold-start SFT) produce readable outputs, or does the combination matter? (c) is the second SFT stage necessary, or would continued RL with mixed rule-based and model-based rewards achieve similar results? (d) can the final RL stage's 400-step limit on model-based rewards be replaced by an automated early-stopping criterion based on divergence between reward model scores and held-out ground-truth performance? Answering these would transform the paper's existence proof (a specific recipe that works) into a robust methodology (principles for constructing pipelines that work across base models and domains).

Practical Applications and Downstream Use Cases

Cost-efficient specialization of large models for verifiable domains. The paper's core result—that a base model can develop strong reasoning through RL with only rule-based verifiers—enables a deployment pattern where a single pretrained base model is specialized for different clients or tasks without collecting domain-specific human demonstration data. For a financial services company needing models that reason about regulatory compliance (where correctness is verifiable against rule books), or a chip design firm needing models that optimize circuit layouts (where correctness is verifiable through simulation), the recipe is: take the base model, construct rule-based verifiers for the target domain, and run GRPO training. No human experts need to write reasoning traces for regulatory analysis or circuit optimization—the verifiers provide the training signal. The distillation results (Table 15) mean the final specialized model can be 7B–32B parameters and deployable on modest hardware. The cost of specialization is RL training compute (~$200K for R1-Zero scale, potentially less for narrower domains) rather than ongoing human expert annotation. This is particularly valuable for domains where the verification logic is well-understood (and can be encoded in rules) but human expertise is scarce or expensive.

Bootstrapping reasoning for new programming languages and frameworks. The paper's code RL data includes competitive programming problems and bug-fixing tasks where correctness is verified by compiler output and test case execution. This approach can be extended to new programming languages, frameworks, or APIs as they emerge: construct a set of verifiable problems in the new language (unit tests, compilation checks, runtime behavior assertions), train with GRPO from a base model, and the model develops reasoning capability for that language without human-written tutorials or solution examples. The rapid evolution of programming ecosystems (new languages like Mojo, new frameworks, new API versions) means that waiting for human experts to produce high-quality reasoning demonstrations creates a permanent lag. The paper's methodology closes this gap: as soon as an execution environment exists, verifiable training data can be generated programmatically, and RL can induce reasoning capability. The 1.5B distilled model's 28.9% on AIME (Table 15) demonstrates that even very small models can acquire non-trivial reasoning through this pipeline, making the approach viable for resource-constrained deployment environments like IDE plugins or edge devices.

Automated curriculum generation for self-improving reasoning systems. The paper's finding that RL-discovered reasoning strategies outperform human-written ones (the 18.9-point AIME gap between R1-Zero and Dev1) suggests a specific architecture for continuous self-improvement: a large model (the "explorer") continually undergoes pure RL on new verifiable problems, discovering progressively better reasoning strategies; its outputs are periodically distilled into a smaller model (the "deployer") that serves production traffic. The large model never needs human data—only new verifiable problems, which can be generated programmatically (new math problems from templates, new coding problems from specification-to-test-case generators). The small model benefits from the explorer's discoveries without incurring the RL training cost. This decouples capability discovery (expensive, large-scale RL) from capability deployment (cheap, distillation-based SFT), creating a flywheel where the explorer improves the deployer, the deployer enables new applications that generate new verifiable problems, and those problems feed back into the explorer's training. The paper provides the key enabling result: distillation works dramatically better than direct RL for small models (Table 16), meaning the explorer-deployer gap is not just practical but optimal.

When to Prefer This Method

The paper explicitly positions pure RL against two alternatives: SFT-first-then-RL (the conventional post-training paradigm) and distillation from larger models. The following decision rules are grounded in the paper's empirical results:

Prefer pure RL (R1-Zero methodology) over SFT-first when:

• The base model has sufficient capacity (≥32B parameters based on Appendix G.1; the exact threshold may vary by architecture and pretraining data).
• The target tasks have reliable, rule-based verifiability (deterministic correctness criteria that can be automated—math with expression matching, code with test cases, logic with ground-truth labels).
• The goal is maximum reasoning capability on the target domain, and output readability or general-domain performance are not constraints (or will be addressed through subsequent pipeline stages).
• Evidence: R1-Zero's 77.9% on AIME vs. Dev1's 59.0% (Table 3); the 18.9-point gap demonstrates that SFT before RL constrains reasoning discovery.

Prefer distillation from a large RL-trained model when:

• The deployment model must be small (≤32B parameters) and cannot support the computational cost of large-scale RL.
• The large "explorer" model already exists or can be trained once and reused.
• Evidence: DeepSeek-R1-Distill-Qwen-32B at 72.6% vs. Qwen2.5-32B-Zero at 47.0% on AIME (Table 16); the 25.6-point gap shows distillation is dramatically more effective than direct RL for smaller models.

Prefer the full multi-stage pipeline (R1 methodology) when:

• The model must serve as a general-purpose assistant with strong reasoning, readable output, and broad task coverage.
• Both verifiable reasoning tasks and subjective/open-ended tasks are in scope.
• Human preference alignment (helpfulness, harmlessness) matters.
• Evidence: R1 achieves 79.8% on AIME (preserving reasoning) while reaching 87.6 on AlpacaEval2.0 (vs. R1-Zero's 24.7) and passing safety benchmarks at frontier-model levels (Table 9, Table 10). The pipeline trades some reasoning ceiling (R1-Zero's 77.9% vs. R1's 79.8%—note R1 actually exceeds R1-Zero here, but the cold-start SFT initially degraded performance before later stages recovered it) for massive gains in usability.

Prefer conventional SFT-first when:

• The base model is below the capacity threshold for emergent metacognition (<32B parameters based on the paper's findings) and no larger explorer model exists for distillation.
• The target domain lacks reliable verifiers (open-ended generation, subjective evaluation, creative tasks).
• Readability and format compliance are critical from the start and cannot be deferred to later pipeline stages.
• Evidence: The paper's 7B and 16B pure-RL experiments "consistently failed to yield meaningful improvements" (Appendix G.1), and the paper's response to non-verifiable tasks is to fall back to SFT (the second SFT stage with general data, Table 3). For these conditions, the paper provides no evidence that pure RL succeeds.