ArXiv: 2505.00551

🎯 Pitch

A fully open-source 1.5B parameter model trained with simple rule-based reinforcement learning can beat OpenAI's o1-preview on the AIME math benchmark—proving that emergent reasoning is not exclusive to massive proprietary systems. This survey maps the 100-day scramble to replicate DeepSeek-R1, showing that curated SFT data and stabilized RL algorithms, not scale, are the real unlock.

1. Executive Summary

This survey systematically catalogs and analyzes the open-source replication efforts that emerged in the 100 days following DeepSeek-R1's release, focusing on how the community has reconstructed the model's reasoning capabilities through two complementary training paradigms—supervised fine-tuning (distilling long chain-of-thought traces from DeepSeek-R1 into smaller models) and reinforcement learning from verifiable rewards (optimizing base models with rule-based outcome rewards and GRPO-style policy gradient algorithms)—across models ranging from 1.5B to 32B parameters on benchmarks such as AIME24, AIME25, and MATH500. The survey documents that carefully curated SFT datasets with as few as 817 examples can reproduce strong reasoning performance (LIMO achieving 57.1% on AIME24 with a 32B model), while RLVR applied to a 1.5B model can surpass o1-preview on AIME24 (DeepScaleR reaching 43.1%), together establishing that the primary bottleneck in replicating DeepSeek-R1 lies not in model scale but in data quality, verification signal design, and algorithm stabilization—provided the base model already possesses non-trivial prior knowledge that RL can exploit.

2. Context and Motivation

The Core Problem: DeepSeek-R1's Reasoning Pipeline Is Partially Undisclosed

The fundamental problem this survey addresses is deceptively simple: DeepSeek-R1 released a powerful reasoning model alongside a high-level training recipe, but withheld the implementation details needed to actually reproduce it. The DeepSeek team publicly described their approach as combining supervised fine-tuning (SFT) with Group Relative Policy Optimization (GRPO) and rule-based reward systems (Guo et al., 2025). However, as the survey authors state in Section 1:

"the implementation details of the released models have not been fully open-sourced by DeepSeek, including DeepSeek-R1-Zero, DeepSeek-R1, and the distilled small models... the training data and configurations of the SFT and RLVR stages are not released, leaving the impact on model performance to be further examined."

This is not a minor omission. The missing details span every critical component of the training pipeline: the exact composition and preprocessing of the 800k sample SFT dataset (which includes 600k reasoning examples and 200k non-reasoning examples), the precise reward function design beyond the high-level description of "accuracy and format rewards," the hyperparameter settings for GRPO, the data sampling strategies used during RL training, and the specific checkpoints and training schedules employed at each stage. Without these details, the community could observe DeepSeek-R1's impressive benchmark results—79.8% on AIME24, 97.3% on MATH500—but could not deterministically reconstruct how those results were achieved.

This gap matters because it creates a replication crisis in the making. When a model demonstrates emergent behaviors like self-verification, reflection, and spontaneous long chain-of-thought reasoning (the so-called "aha moment" that DeepSeek-R1-Zero exhibited during RL training), the scientific value of those observations depends on independent verification. If the community cannot reproduce the training process, it cannot distinguish between genuine algorithmic advances and artifacts of undisclosed data curation, hyperparameter tuning, or compute scale. The survey accordingly positions itself as a necessary consolidation:

"many replication works have attempted to explore the optimal design for RLMs from various perspectives... yet a comprehensive list and comparison of these works are still lacking."

Why This Problem Is Important

The significance of this replication effort extends beyond academic reproducibility into three practical domains.

First, economic accessibility. DeepSeek-R1 demonstrated that explicit reasoning capabilities—previously associated with proprietary models like OpenAI's o1 series—could be achieved through a training recipe that, while computationally intensive, is conceptually straightforward: generate long chain-of-thought traces, fine-tune on them, then optimize with outcome-based RL. If this recipe can be reliably reproduced and understood, organizations beyond the largest AI labs can develop reasoning models tailored to their specific domains. The survey documents that reproduction efforts have succeeded across a wide range of model sizes (1.5B to 32B parameters), with DeepScaleR-1.5B-Preview achieving 43.1% on AIME24—surpassing o1-preview—using only 40k curated math problems. This suggests that the barrier to entry for reasoning capabilities is lower than the original DeepSeek-R1 paper might imply, provided the community converges on effective data curation and training stabilization practices.

Second, scientific understanding of emergent reasoning. DeepSeek-R1-Zero's training process revealed that pure RL on a base model could spontaneously induce behaviors like self-verification and backtracking—the model learned to "think" without being explicitly taught how. Understanding why and under what conditions these behaviors emerge requires systematic replication across different model architectures, scales, and data distributions. The survey catalogs multiple studies that observed (or failed to observe) the "aha moment" under varying conditions, providing the raw material for a more rigorous theory of emergent reasoning. For instance, TinyZero (Pan et al., 2025) and Mini-R1 (Schmid, 2025) attempt to reproduce this phenomenon on the Countdown game, while Logic-RL (Xie et al., 2025a) examines it on deductive reasoning puzzles—each experiment constraining the conditions under which the behavior manifests.

Third, safety and alignment implications. Reasoning models that can self-verify and reflect introduce new safety considerations. The survey notes (Section 4.3) that DeepSeek-R1 produces more unsafe responses than OpenAI models on certain adversarial tests, and that overthinking attacks can inflate operational costs. Replication studies that expose the training pipeline enable safety researchers to study which components of the recipe most affect alignment, rather than treating DeepSeek-R1 as an opaque artifact.

Where Prior Approaches Fall Short

The survey identifies several distinct limitations in the pre-DeepSeek-R1 landscape and in the initial DeepSeek release itself.

Limitation 1: The DeepSeek-R1 technical report describes a recipe, not a reproducible procedure.

The DeepSeek-R1 paper (Guo et al., 2025) provides a high-level narrative: collect cold-start SFT data, fine-tune a base model, apply GRPO with accuracy and format rewards, and then further refine through additional SFT and RL stages. But the report does not specify:

  • The exact filtering criteria used to select the 800k SFT samples from a presumably larger pool.
  • The hyperparameters for GRPO (group size, learning rate, KL penalty coefficient, maximum response length scheduling).
  • Whether the reward function includes components beyond accuracy and format (e.g., length penalties, which several replication studies later found necessary).
  • The data sources used for RL training, their preprocessing, and their deduplication procedures.

As the survey authors note, "the optimal design of the reinforcement learning algorithm and reward system remains underexplored" in the original work. The community was left with a successful outcome but an underspecified method.

Limitation 2: Existing open-source reasoning models relied on imitation, not emergent behavior.

Prior to DeepSeek-R1, the dominant approach to building reasoning models was to fine-tune on reasoning traces generated by larger proprietary models—a form of distillation or imitation learning. This produces models that can mimic reasoning patterns but do not develop those patterns through exploration. The survey contrasts this with DeepSeek-R1-Zero, which "achieved strong performance on reasoning and knowledge tasks through a standalone RLVR process" (Section 3.1) starting from a base model with no explicit reasoning demonstrations. The distinction matters because imitation-based models are fundamentally bounded by the quality of their teacher, while RL-based models can, in principle, discover reasoning strategies that exceed those demonstrated in the training data.

Limitation 3: No systematic comparison of SFT data curation strategies existed.

Before the replication wave documented in this survey, there was no organized understanding of how dataset size, difficulty distribution, domain coverage, and verification rigor affected the downstream reasoning performance of fine-tuned models. Different projects used their own ad hoc datasets, making it impossible to attribute performance differences to specific data design choices. The survey addresses this by presenting, for the first time, a side-by-side comparison of datasets ranging from LIMO's 817 carefully curated examples to AM's 1.4M broad-coverage samples, with detailed analysis of their token length distributions (Figure 2), cross-referencing patterns (Figure 3), and benchmark results (Table 2).

Limitation 4: RLVR algorithm variants proliferated without theoretical unification.

The survey documents a rapidly expanding alphabet soup of algorithm variants: GRPO, Dr. GRPO, DAPO, VAPO, VC-PPO, CPPO, REINFORCE++, REINFORCE-Kimi, GPG. Each paper proposed modifications—different clipping strategies, different normalization schemes, different KL penalty implementations—with empirical justifications but no framework for understanding how these modifications relate to each other. Practitioners choosing an algorithm faced a combinatorial explosion of design choices with no principled guidance. The survey addresses this by establishing a unified theoretical framework (Section 3.2.1) that derives all these variants from a common policy gradient foundation, explicitly showing where each method departs from the shared formulation and why.

Limitation 5: The interaction between SFT and RLVR was poorly characterized.

DeepSeek-R1 used both SFT and RLVR in sequence, but the original report did not isolate their contributions. Several replication studies revealed that SFT on high-quality long-CoT data alone could achieve strong reasoning performance (Table 2), while RLVR from a base model without SFT could induce emergent reasoning behaviors but with lower final accuracy. The survey's side-by-side presentation of SFT-only and RLVR-only results enables readers to understand the complementary roles of these stages—a question the original DeepSeek-R1 paper left largely to speculation.

How This Paper Positions Itself

The survey explicitly frames itself as a consolidation and comparative analysis, not a proposal of new methods. In Section 1, the authors state their goal:

"This survey aims to provide a clear review of the open-source replication works on DeepSeek-R1... We primarily focus on SFT and RLVR as two main directions, introducing the details for data construction, method design and training procedure of current replication studies. Moreover, we conclude key findings from the implementation details and experimental results reported by these studies, anticipating to inspire future research."

This positions the survey at the meta-analytic level—one step removed from the individual replication studies, extracting cross-cutting insights that no single study could provide. The taxonomy presented in Figure 1 reflects this synthesis orientation: rather than listing works chronologically or by institution, it organizes them by methodological component (SFT datasets, SFT models, RL data curation, RL algorithm design, RL reward curation, RL sampling strategy, and orthogonal techniques), enabling readers to compare approaches within each component.

Several design choices in the survey's structure reinforce this positioning:

  • Comparative tables as primary evidence. Tables 1, 2, 3, 4, and 5 provide dense, side-by-side comparisons of dataset compositions, training configurations, algorithm choices, and benchmark results across dozens of studies. These tables implicitly argue that the replication landscape can be understood through a small set of key dimensions—data size, difficulty calibration, RL algorithm variant, reward design—rather than through study-specific idiosyncrasies.

  • Unified mathematical framework. Rather than describing each RL algorithm variant in isolation, the survey derives all of them from a common policy gradient formulation (Equations 2–4), then systematically shows how each variant modifies the base objective. This transformation—from a list of methods to a single parameterized family—is a theoretical contribution in itself, making the space of algorithm design choices navigable.

  • Explicit identification of open questions. The survey does not merely report what has been done; it identifies what remains unknown. For instance, it notes that format rewards "rapidly saturate within approximately 60 steps" but "there are no rigorous ablation studies to prove the effectiveness of the format reward" (Section 3.3.5). It observes that KL regularization "may not be essential" but that some studies retain it with good results (Section 3.3.6). These observations function as a research agenda, directing future work toward the most impactful unresolved questions.

  • Connections to applications beyond math and code. Section 3.4 extends the survey's scope beyond the MATH/AIME benchmarks that dominate replication studies, documenting RLVR applications to logical reasoning (Countdown games, Sudoku, deductive puzzles), software engineering (GitHub issue fixing), chemistry, machine translation, creative writing, and even algorithm discovery. This broadening signals that the survey is not merely a retrospective of DeepSeek-R1 replication but a forward-looking assessment of the RLVR paradigm's generality.

The survey's relationship to the original DeepSeek-R1 work is thus one of decomposition and systematization. Where DeepSeek-R1 presented a single, integrated training pipeline with many black-box components, the survey decomposes that pipeline into its constituent parts, examines each through the lens of independent replication attempts, and synthesizes cross-cutting principles that characterize successful approaches. The intended contribution is not a new state-of-the-art number on AIME24 but a shared understanding of the design space—what matters, what doesn't, and what remains unknown—that can accelerate future research across the entire reasoning language model community.

3. Technical Approach

3.1 Reader Orientation

This paper is a survey and meta-analysis—it does not propose a new model or algorithm. Instead, it systematically catalogues, categorizes, and compares—across dozens of independent research projects—the concrete engineering decisions and experimental outcomes involved in reproducing the reasoning capabilities of DeepSeek-R1 through supervised fine-tuning and outcome-reward-based reinforcement learning. The problem it solves is that of design-space navigation: a practitioner wanting to build a reasoning language model faces a bewildering array of choices (dataset size from 817 to 1.4M examples, RL algorithm from vanilla PPO to DAPO, reward design from accuracy-only to multi-component rule-based systems, KL penalty on or off, etc.), and the survey's "shape" of solution is to extract the common principles and critical trade-offs that characterize successful replication efforts, organizing them into a unified conceptual framework that makes the space of options intelligible.

3.2 Big-Picture Architecture (Diagram in Words)

The survey organizes the reproduction landscape into a four-component pipeline corresponding to the stages of building a reasoning language model:

  1. Data Collection and Curation — The process of gathering raw problems (math, code, logic puzzles, etc.) and transforming them into training-ready datasets through filtering, deduplication, difficulty calibration, verification, and (for SFT) chain-of-thought distillation from teacher models like DeepSeek-R1. This component answers: what do we train on, and how is it prepared?

  2. Supervised Fine-Tuning (SFT) — Training a base or instruct model on curated long chain-of-thought demonstrations by minimizing the standard next-token prediction loss. This component answers: how do we teach the model to produce reasoning traces that look like the teacher's?

  3. Reinforcement Learning from Verifiable Rewards (RLVR) — Optimizing a model (potentially the SFT checkpoint, a distilled model, or a raw base model) using policy gradient methods with rule-based outcome rewards (accuracy, format, length) rather than learned reward models. This component answers: how do we make the model discover better reasoning strategies through trial and error?

  4. Alternative and Complementary Techniques — Methods that augment or replace components of the main pipeline: process reward models (step-level feedback), preference optimization (DPO as an alternative to online RL), and extensions beyond math/code (logical reasoning, creative writing, multilingual transfer).

Information flows sequentially but conditionally: a project may start with an instruct-model SFT path, a base-model RLVR path, or a combined SFT→RLVR path; the survey documents the performance implications of each choice.

3.3 Roadmap for the Deep Dive

  • First, the SFT data pipeline (3.4.1): how projects collect problems, generate chain-of-thought traces, verify correctness, and select subsets—because data quality is the single most consequential variable in SFT-based reproduction.
  • Second, the SFT training procedure (3.4.2): the formal objective, training configurations, and a comparative analysis of how dataset choices interact with base model selection to determine benchmark performance.
  • Third, the RLVR data pipeline (3.4.3): how RL training datasets differ from SFT datasets in their emphasis on verifiability, difficulty calibration, and deduplication, and why these differences matter.
  • Fourth, the unified RL algorithm framework (3.4.4): deriving PPO, GRPO, and all their variants from a shared policy gradient foundation, showing that apparent diversity masks a small set of fundamental design dimensions.
  • Fifth, reward design and sampling strategies (3.4.5–3.4.6): the surprisingly nuanced role of format and length rewards, and the mechanisms by which curriculum learning and dynamic sampling stabilize training.
  • Sixth, alternative training paradigms (3.4.7): process reward models and preference optimization as complementary or substitute approaches to outcome-reward-based RLVR.

3.4 Detailed, Sentence-Based Technical Breakdown

This is a survey paper whose core intellectual contribution is a systematic taxonomy of design choices, a unified mathematical derivation of RL algorithm variants, and a synthesis of empirical patterns across dozens of independently conducted replication studies. It does not introduce novel algorithms or datasets but instead provides the conceptual scaffolding needed to understand and navigate the rapidly expanding landscape of DeepSeek-R1 reproduction.

3.4.1 SFT Data Collection and Curation

The SFT component of DeepSeek-R1 reproduction centers on one deceptively simple idea: if you can collect high-quality reasoning traces from a strong teacher model and fine-tune a smaller model on them, the smaller model will inherit much of the teacher's reasoning capability. The survey documents that the primary variation across replication projects lies not in whether they do this, but in how they construct the training dataset—the sources, the filtering pipeline, the verification strategy, and the selection criteria.

Raw data collection. Most projects begin by aggregating problems from existing math and coding benchmarks and competition archives. Common sources include NuminaMath 1.5, AIME (1984-2023), AMC, MATH, Omni-MATH, AoPS, TACO, CodeForces, and LeetCode. The AM dataset (Zhao et al., 2025a) additionally incorporates web-crawled problems and applies embedding-based deduplication using cosine similarity on text embeddings. The survey notes a significant practical issue documented in Figure 3: many of these source datasets are not independent. For instance, multiple reasoning datasets (Open-R1, Light-R1, Bespoke-Stratos, CuratedThoughts) derive from NuminaMath 1.5, while others draw from overlapping competition archives. This cross-referencing pattern means that two projects reporting results on a benchmark may have used training data with unknown overlap, complicating fair comparison.

Chain-of-thought distillation. Once problems are collected, projects generate long chain-of-thought reasoning traces using DeepSeek-R1 as the teacher. The AM dataset generates new CoTs for samples whose existing reasoning traces fail verification. OpenR1-Math-220k generates 2–4 CoTs per problem. The S1k-1.1 dataset forms "question-trace-solution" triplets where the trace is the full chain-of-thought and the solution is the final answer extract. A key practical consideration is that DeepSeek-R1's outputs for the same problem can vary in length and style across multiple samples—this is why some datasets retain multiple traces per problem while others select a single best trace (Light-R1 keeps one per question after filtering).

Verification methods. Different domains require different verification strategies, and projects vary in how they ensure that a CoT trace leads to a factually correct answer:

  • Math problems are typically verified using Math-Verify (Kydlíček, 2024), a symbolic formula parser that evaluates whether the extracted final answer matches the ground truth. This is notably more reliable than string matching because mathematical expressions can be written in equivalent forms (e.g., "1/2" vs. "0.5" vs. "\frac{1}{2}").
  • Coding problems are verified through execution—the generated code is run against provided unit tests, and correctness is determined by whether all tests pass. DeepCoder (Luo et al., 2025a) enforces that every problem has at least 5 test cases, while KodCode-V1 (Xu et al., 2025b) generates test cases synthetically along with the solutions.
  • General reasoning problems (puzzles, science, STEM) rely on LLM judges. Bespoke-Stratos (Bespoke-Labs, 2025) reports that using GPT-4o-mini as a judge increased the proportion of retained examples from 25% to 73% compared to rule-based filtering—a striking difference that highlights the brittleness of purely rule-based verification for non-formal domains.
  • LIMO's quality framework goes beyond correctness verification to assess reasoning trace quality along three dimensions: Optimal Structural Organization (clear step-by-step decomposition), Effective Cognitive Scaffolding (explicit intermediate goals and sub-goals), and Rigorous Verification (self-checking within the trace).

Difficulty filtering. This is one of the most consequential design choices in the SFT pipeline. The underlying principle is that easy problems produce short, uninformative CoT traces that teach the model little, while impossible problems produce incorrect traces that teach the model wrong reasoning patterns. Several projects operationalize this through pass-rate-based filtering:

  • Light-R1 (Wen et al., 2025a) collects 1 million math problems, generates initial responses using DeepScaleR-1.5B-Preview, and retains only questions where the model's pass rate falls below a threshold $\alpha$. This selects problems that are challenging but not impossible. The resulting set of ~76k examples is further filtered using DeepSeek-R1, retaining only questions where DeepSeek-R1's responses were correct—ensuring the final dataset contains only valid reasoning traces on appropriately difficult problems.
  • LIMO (Ye et al., 2025) takes a more aggressive stance, using Qwen2.5-Math-7B-Instruct to remove problems that can be solved within a few attempts (too easy), then collecting solutions from human experts and state-of-the-art models, ultimately producing only 817 examples from what was initially "tens of millions" of candidate problems. The phrase "Less is More" in LIMO's title refers precisely to this extreme filtering: a small number of exceptionally high-quality, challenging examples can outperform much larger but less-curated datasets.
  • The S1k-1.1 dataset (Muennighoff et al., 2025) applies a three-stage filtering process (decontamination, deduplication, then quality/diversity/difficulty selection) to reduce 59k initial questions to 1,000.

Data decontamination. The survey authors identify a critical methodological concern: only the technical reports of Light-R1 and LIMO explicitly mention proper decontamination against popular benchmarks (AIME24/25, MATH500, GPQA Diamond) during dataset curation. Wen et al. (2025a) specifically point out that MATH500 is "partially compromised across several open-source datasets, including OpenThoughts, Open-R1, Bespoke Stratos, and others." This means that SFT models trained on these datasets may be memorizing solutions that appear in the test set, artificially inflating benchmark performance. The survey's Figure 3, showing cross-referencing patterns among math reasoning datasets, is partly motivated by this concern: if two projects both evaluate on a benchmark that overlaps with their shared training source, their results are not independently informative.

Token length distributions. Figure 2 presents the token length distributions of major SFT datasets. These exhibit "observable differences" despite all traces originating from the same teacher model (DeepSeek-R1). AM and Synthetic-1 are "skewed toward shorter sequences," while Light-R1 and Open-R1 "display broader distributions with longer tails, suggesting a higher proportion of complex problems." This variation matters because the length of training sequences directly affects the model's learned behavior: models trained on longer-CoТ datasets tend to produce longer reasoning traces at inference time, which can improve accuracy but increases computational cost.

3.4.2 SFT Training Procedure

Formal objective. The SFT objective is the standard causal language modeling loss applied to chain-of-thought traces. Given a dataset $\mathcal{D}_{\text{SFT}} \triangleq \{(q_i, c_i)\}_{i=1}^{|\mathcal{D}|}$, where $q_i$ is a question and $c_i$ is a long chain-of-thought (decomposable into intermediate rationale followed by final answer), the loss is:

LSFT(θ)≜−E(q,c)∼DSFT[log⁡πθ(c∣q)]\mathcal{L}_{\text{SFT}}(\theta) \triangleq -\mathbb{E}_{(q,c) \sim \mathcal{D}_{\text{SFT}}} \left[ \log \pi_\theta(c \mid q) \right]

where $\pi_\theta(c \mid q)$ is the probability assigned by the policy model (parameterized by $\theta$) to the CoT response $c$ conditioned on the question $q$.

What it computes: For each question, the model generates a probability distribution over possible next tokens at every position, and the loss sums the negative log-probability of the actual token that appears in the training trace. This is averaged over all tokens in all traces. The loss is minimized when the model assigns high probability to the exact CoT trace from the training data.

Why this form: This is the maximum-likelihood objective for autoregressive sequence generation. No special reasoning-aware loss is used—the model learns to imitate the step-by-step reasoning patterns purely from the token-level supervision. The survey notes that this is "remarkably effective" despite not explicitly encoding any notion of logical validity, suggesting that the structure of reasoning is adequately captured in the token sequence itself when the traces are sufficiently detailed.

Training configurations. The survey aggregates training details from replication studies: common learning rates are $1.0 \times 10^{-5}$ and $5.0 \times 10^{-5}$, typical batch sizes are 96 or 128, and packing (concatenating multiple examples into a single sequence up to the maximum context length) is "usually employed to improve training efficiency" (Wang et al., 2024a). For long-context reasoning tasks, the RoPE scaling factor $\theta$ and maximum context length are often adjusted—Open-R1 sets $\theta = 300,000$ and context length to 32,768 tokens (Chen et al., 2023).

Initial checkpoint choice. Table 2 reveals a clear pattern: projects fine-tuning from Qwen2.5-Instruct models generally achieve stronger results than those fine-tuning from Qwen2.5-Base models, even with smaller datasets. The survey cites a related finding by Li et al. (2025d): "instruct models exhibit higher learning efficiency than their base counterparts." However, DeepSeek-R1's own distilled models and the AM project fine-tune from base models—the difference is that these projects incorporate non-reasoning data (writing, role-playing, general tasks) in their SFT mixture, which other projects omit. The survey flags this as underexplored: "the interplay between reasoning and non-reasoning data" remains an open question.

The LIMO paradox. The survey highlights LIMO's 817-example dataset achieving 57.1% on AIME24 with Qwen2.5-32B-Instruct as a critical data point. This is competitive with or exceeds models trained on much larger datasets (Synthetic-1 with 894k examples achieves 30.0% on the smaller 7B model; OpenThoughts with 114k examples achieves 68.0% on the 32B model but on an instruct base). The implication is that dataset quality and difficulty calibration dominate dataset size—a finding that the survey presents as one of the key actionable insights from the SFT replication studies.

3.4.3 RLVR Data Curation

RL training datasets differ from SFT datasets in a fundamental way: rather than needing complete reasoning traces, they require only queries with verifiable answers. The model generates its own reasoning traces during training (the "rollout" phase), which are then scored by the reward function. This shifts the data curation challenge from trace quality to answer verifiability and difficulty calibration.

Verifiability as the central constraint. The survey documents a consistent practice across RLVR projects: removing samples whose answers cannot be automatically verified. This includes:

  • Multiple-choice problems (the model could guess correctly without reasoning, providing noisy reward signals).
  • Proof-oriented problems (no single verifiable final answer).
  • Multi-part questions (unclear what the "final answer" should be).
  • Questions referring to figures or charts (the model cannot "see" the visual content).
  • Questions with yes/no or true/false answers (binary outcomes make reward signals coarse and verification prone to error when the answer can be expressed in multiple ways).
  • Non-English questions (training stability and final model performance improve with English-only data, per Open-Reasoner-Zero's observations).

DAPO (Yu et al., 2025) takes verifiability to an extreme: they modify the original AoPS questions so that every expected answer is always an integer. The advantages are twofold: integer answers are trivial to parse (no formula-parser errors), and the resulting reward signal is fully deterministic—a response is either exactly right or exactly wrong, with no ambiguous intermediate states.

Difficulty calibration. The same principle that applies to SFT data applies to RL data, but with different operationalization. For RL, the goal is to construct datasets where the model's current policy has a moderate success rate—neither 0% (the model never encounters positive reward, so learning cannot begin) nor 100% (all rollouts receive maximum reward, providing zero gradient signal). DAPO formalizes this through its dynamic sampling constraint:

0<∣{oi∣is_equivalent(a,oi)}∣<G0 < |\{ o_i \mid \text{is\_equivalent}(a, o_i) \}| < G

where $G$ is the number of rollouts per prompt, $a$ is the ground-truth answer, and $o_i$ are the model's generated responses. This constraint ensures that each training batch contains both correct and incorrect responses for every prompt, producing non-zero advantages that drive learning.

Several projects implement difficulty calibration through pass-rate-based filtering:

  • DeepScaleR (Luo et al., 2025b) explicitly states: "samples with an overly high pass rate are too easy for model training, while samples with a zero pass rate are often unverifiable or contain errors, therefore, both should be filtered out."
  • Skywork-OR1 (He et al., 2025a) marks the difficulty level of each sample based on its pass rate when evaluated by DeepSeek-R1-Distilled models, enabling curriculum learning strategies.
  • LIMR (Li et al., 2025b) goes beyond pass-rate filtering to propose "learning impact measurement"—selecting samples whose learning patterns complement the model's overall performance trajectory, identifying examples that are particularly informative for the current stage of training.

Dataset sizes and composition. Table 3 documents RLVR datasets ranging from LIMR's 1k examples to Math-RLVR's 773k examples. The survey observes "a consistent trend: datasets used for RL training are carefully designed to include data where models are likely to make mistakes... Such uncertainty creates opportunities for learning. This calibrated challenge level encourages models to engage in deeper reasoning and reflection, often resulting in longer and more informative responses."

Deduplication and decontamination. The survey notes that Light-R1 emphasizes the necessity of decontamination for fair evaluation, while DeepScaleR and Skywork-OR1 perform "elaborated de-duplication for efficient training." This is particularly important for RL because the model generates its own responses and receives rewards—if the training set contains problems that closely resemble test-set problems, the model can learn to produce correct answers through memorization rather than reasoning, and the RL process will reinforce this behavior.

3.4.4 Unified RL Algorithm Framework

The survey makes its most significant theoretical contribution in Section 3.2.1, where it derives all major RLVR algorithm variants from a common policy gradient foundation. This derivation reveals that the apparent diversity—GRPO, Dr. GRPO, DAPO, VAPO, VC-PPO, REINFORCE++, etc.—arises from a small set of choices along a few dimensions, not from fundamentally different approaches.

Foundational policy optimization objective. The starting point is the standard RL objective for LLM fine-tuning:

max⁡θJ(πθ)≜E(q,a)∼DEo∼πθ(⋅∣q)[r(o,a)]\max_\theta \mathbf{J}(\pi_\theta) \triangleq \mathbb{E}_{(q,a) \sim \mathcal{D}} \mathbb{E}_{o \sim \pi_\theta(\cdot|q)} \left[ r(o, a) \right]

where $\pi_\theta$ is the LLM policy (parameterized by $\theta$), $(q, a)$ is a query-answer pair from the data distribution $\mathcal{D}$, $o$ is a response generated by the policy, and $r(o, a)$ is a reward function quantifying response quality.

Policy gradient estimation. The fundamental challenge is that the gradient $\nabla_\theta \mathbb{E}_{o \sim \pi_\theta(\cdot|q)} r(o, a)$ cannot be computed via standard Monte Carlo sampling because the expectation is over a distribution that depends on $\theta$. The survey presents two classical solutions:

REINFORCE estimator:

∇θEo∼πθ(⋅∣q)r(o,a)=Eo∼πθ(⋅∣q)[∇θlog⁡πθ(o∣q)⋅r(o,a)]\nabla_\theta \mathbb{E}_{o \sim \pi_\theta(\cdot|q)} r(o, a) = \mathbb{E}_{o \sim \pi_\theta(\cdot|q)} \left[ \nabla_\theta \log \pi_\theta(o|q) \cdot r(o, a) \right]

where the gradient is taken through the log-probability of the generated response, weighted by the reward.

Importance sampling estimator:

∇θEo∼πθ(⋅∣q)r(o,a)=Eo∼πθ′(⋅∣q)[∇θ(πθ(o∣q)πθ′(o∣q))⋅r(o,a)]\nabla_\theta \mathbb{E}_{o \sim \pi_\theta(\cdot|q)} r(o, a) = \mathbb{E}_{o \sim \pi_{\theta'}(\cdot|q)} \left[ \nabla_\theta \left( \frac{\pi_\theta(o|q)}{\pi_{\theta'}(o|q)} \right) \cdot r(o, a) \right]

where $\pi_{\theta'}$ is any parameter-frozen LLM policy, typically a past snapshot of $\pi_\theta$.

Practical approximation. In practice, these estimators are approximated by generating $G$ independent responses for each query (the "rollout phase"):

∇θJ(πθ)=E(q,a)∼D,{oi}i=1G∼πθ(⋅∣q)[1G∑i=1G∇θlog⁡πθ(oi∣q)⋅r(oi,a)]\nabla_\theta \mathbf{J}(\pi_\theta) = \mathbb{E}_{(q,a) \sim \mathcal{D}, \{o_i\}_{i=1}^G \sim \pi_\theta(\cdot|q)} \left[ \frac{1}{G} \sum_{i=1}^G \nabla_\theta \log \pi_\theta(o_i|q) \cdot r(o_i, a) \right]

or, for the importance sampling variant with a frozen policy $\pi_{\theta_{\text{old}}}$:

∇θJ(πθ)=E(q,a)∼D,{oi}i=1G∼πθold(⋅∣q)[1G∑i=1G∇θ(πθ(oi∣q)πθold(oi∣q))⋅r(oi,a)]\nabla_\theta \mathbf{J}(\pi_\theta) = \mathbb{E}_{(q,a) \sim \mathcal{D}, \{o_i\}_{i=1}^G \sim \pi_{\theta_{\text{old}}}(\cdot|q)} \left[ \frac{1}{G} \sum_{i=1}^G \nabla_\theta \left( \frac{\pi_\theta(o_i|q)}{\pi_{\theta_{\text{old}}}(o_i|q)} \right) \cdot r(o_i, a) \right]

What these compute: The policy gradient is an average over generated responses, where the gradient for each response is the product of (a) how much the policy parameters affect the log-probability of that response, and (b) the reward that response received. This has an intuitive operational meaning: if a response received a high reward, the gradient pushes the policy to make that response more likely in the future; if low, the policy is pushed away from generating similar responses.

What makes this challenging: The reward signals $\{r(o_i, a)\}_{i=1}^G$ are "highly sparse"—a single scalar per complete response, with no per-token feedback. This creates high variance in the gradient estimates across different rollouts. The major algorithm variants all address this variance through different stabilization strategies, which the survey categorizes into three families:

  1. Reward normalization: reducing variance by centering and scaling rewards within each group of rollouts.
  2. Importance sampling clipping: preventing destructive parameter updates by bounding how much the new policy can differ from the old one.
  3. KL-divergence penalties: constraining the policy from drifting too far from a reference (usually the initial SFT model) to prevent catastrophic forgetting.

PPO (Proximal Policy Optimization). The PPO objective, as used in Open-Reasoner-Zero (Hu et al., 2025), is:

JPPO(πθ)≜E(q,a)∼D,{oi}i=1G∼πθold(⋅∣q)[1G∑i=1G1∣oi∣∑t=1∣oi∣(min⁡(ri,t(θ)A^i,t(ϕ),clip(ri,t(θ),1−ε,1+ε)A^i,t(ϕ)))]\mathbf{J}_{\text{PPO}}(\pi_\theta) \triangleq \mathbb{E}_{(q,a) \sim \mathcal{D}, \{o_i\}_{i=1}^G \sim \pi_{\theta_{\text{old}}}(\cdot|q)} \left[ \frac{1}{G} \sum_{i=1}^G \frac{1}{|o_i|} \sum_{t=1}^{|o_i|} \left( \min\left( r_{i,t}(\theta) \hat{A}_{i,t}(\phi), \text{clip}(r_{i,t}(\theta), 1-\varepsilon, 1+\varepsilon) \hat{A}_{i,t}(\phi) \right) \right) \right]

where:

  • $r_{i,t}(\theta) \triangleq \pi_\theta(o_{i,t} \mid q, o_{i,<t}) / \pi_{\theta_{\text{old}}}(o_{i,t} \mid q, o_{i,<t})$ is the per-token likelihood ratio—how much more (or less) likely the current policy makes token $t$ of response $i$ compared to the old policy.
  • $\hat{A}_{i,t}(\phi)$ is the Generalized Advantage Estimator (GAE, Schulman et al., 2018), computed from a learned critic model $V_\phi$ that estimates the value of each generation state. GAE uses a trace decay parameter $\lambda$ (typically 0.95 or 1.0) to balance bias and variance in advantage estimation.
  • $\varepsilon$ is the clipping threshold (typically 0.2).
  • The $\min$ operation implements conservative policy updates: it uses the unclipped objective when it would reduce the advantage of negative-advantage tokens or increase the advantage of positive-advantage tokens, but falls back to the clipped version when the policy change would be too aggressive.

What it computes: For every token in every generated response, PPO computes an advantage (how much better or worse this token's outcome was compared to expected), then adjusts the policy to favor tokens with positive advantages while preventing any single update from being too large (via clipping). The critic model $V_\phi$ is trained concurrently to predict the expected future reward from each state, using a mean-squared-error loss against observed returns.

Why this form and its variants: PPO is the most widely studied and influential policy optimization algorithm for LLMs. Its key innovation is the clipped surrogate objective—by clipping the importance sampling ratio to $[1-\varepsilon, 1+\varepsilon]$, PPO prevents the policy from changing so much in a single update that it "forgets" what it previously learned, while still allowing gradual improvement. The critic model provides per-token advantage estimates rather than using a single per-response reward, which significantly reduces gradient variance for long sequences. However, the need to train a separate critic model of comparable size to the actor doubles memory requirements.

GRPO (Group Relative Policy Optimization). DeepSeek-R1's primary algorithm discards the critic model entirely and instead normalizes rewards within each rollout group:

JGRPO(πθ)≜E(q,a)∼D,{oi}i=1G∼πθold(⋅∣q)[1G∑i=1G1∣oi∣∑t=1∣oi∣(min⁡(ri,t(θ)A^i,t,clip(ri,t(θ),1−ε,1+ε)A^i,t)−βKL(πθ∣πref)i,t)]\mathbf{J}_{\text{GRPO}}(\pi_\theta) \triangleq \mathbb{E}_{(q,a) \sim \mathcal{D}, \{o_i\}_{i=1}^G \sim \pi_{\theta_{\text{old}}}(\cdot|q)} \left[ \frac{1}{G} \sum_{i=1}^G \frac{1}{|o_i|} \sum_{t=1}^{|o_i|} \left( \min\left( r_{i,t}(\theta) \hat{A}_{i,t}, \text{clip}(r_{i,t}(\theta), 1-\varepsilon, 1+\varepsilon) \hat{A}_{i,t} \right) - \beta \text{KL}(\pi_\theta | \pi_{\text{ref}})_{i,t} \right) \right]

where the critical difference from PPO is the advantage definition:

A^i,t≜r(oi,a)−mean({r(oi,a)}i=1G)std({r(oi,a)}i=1G)\hat{A}_{i,t} \triangleq \frac{r(o_i, a) - \text{mean}(\{r(o_i, a)\}_{i=1}^G)}{\text{std}(\{r(o_i, a)\}_{i=1}^G)}

and the KL penalty uses the K3 estimator (Schulman, 2020):

KL(πθ∣πref)i,t≜πref(oi,t∣q,oi,<t)πθ(oi,t∣q,oi,<t)−log⁡πref(oi,t∣q,oi,<t)πθ(oi,t∣q,oi,<t)−1\text{KL}(\pi_\theta | \pi_{\text{ref}})_{i,t} \triangleq \frac{\pi_{\text{ref}}(o_{i,t} | q, o_{i,<t})}{\pi_\theta(o_{i,t} | q, o_{i,<t})} - \log \frac{\pi_{\text{ref}}(o_{i,t} | q, o_{i,<t})}{\pi_\theta(o_{i,t} | q, o_{i,<t})} - 1

What GRPO computes: Instead of using a learned critic to estimate advantages, GRPO simply computes each response's reward, subtracts the group mean, and divides by the group standard deviation. This "group relative" normalization means that advantages are defined relative to other responses to the same question, not relative to a learned value function. A response that scores 1.0 but is one of many correct responses in a group will have a small advantage; a response that scores 0.5 but is the only correct response when others scored 0.0 will have a large advantage.

Why GRPO over PPO: The primary motivation is efficiency—removing the critic model approximately halves memory consumption during training, enabling larger models or larger batch sizes on the same hardware. The KL penalty to a frozen reference model (the initial SFT checkpoint) serves as regularization, replacing some of the stabilization that the critic provides in PPO. DeepSeek-R1's empirical results demonstrate that GRPO can achieve stable large-scale training while naturally incentivizing the emergence of long chain-of-thought patterns.

Dr. GRPO (GRPO Done Right). Liu et al. (2025c) identify two biases in the original GRPO formulation and propose a simplified objective:

JDr.GRPO(πθ)≜E(q,a)∼D,{oi}i=1G∼πθold(⋅∣q)[1G∑i=1G1∣oi∣∑t=1∣oi∣(min⁡(ri,t(θ)A^i,t,clip(ri,t(θ),1−ε,1+ε)A^i,t)−βKL(πθ∣πref))]\mathbf{J}_{\text{Dr.GRPO}}(\pi_\theta) \triangleq \mathbb{E}_{(q,a) \sim \mathcal{D}, \{o_i\}_{i=1}^G \sim \pi_{\theta_{\text{old}}}(\cdot|q)} \left[ \frac{1}{G} \sum_{i=1}^G \frac{1}{|o_i|} \sum_{t=1}^{|o_i|} \left( \min\left( r_{i,t}(\theta) \hat{A}_{i,t}, \text{clip}(r_{i,t}(\theta), 1-\varepsilon, 1+\varepsilon) \hat{A}_{i,t} \right) - \beta \text{KL}(\pi_\theta | \pi_{\text{ref}}) \right) \right]

The same advantage $\hat{A}_{i,t}$ as GRPO is used, but two biases are identified:

  • Response-level length bias: The original GRPO divides by $|o_i|$ (response length) within the objective, which means longer correct responses contribute less per token to the gradient than shorter correct responses. Liu et al. argue this encourages the policy to prefer shorter correct responses. Dr. GRPO "eliminates the normalization terms $1/|o_i|$"—though the formula appears identical, the key difference is in how the expectation is structured to avoid length-dependent weighting.

  • Question-level difficulty bias: Normalizing by $\text{std}(\{r(o_i, a)\}_{i=1}^G)$ causes questions with lower reward variance (typically either very easy or very hard questions, where most responses get the same reward) to receive disproportionately larger gradient updates. Dr. GRPO removes the standard deviation normalization from the advantage calculation (though the exact formulation of this removal requires careful reading of the source paper).

DAPO (Decoupled Clip and Dynamic sAmpling Policy Optimization). Yu et al. (2025) identify three shortcomings: entropy collapse (sampled responses becoming nearly identical), training instability from prompts with all-correct or all-incorrect rollouts, and length bias from sample-level loss computation. Their solution:

JDAPO(πθ)≜E(q,a)∼D,{oi}i=1G∼πθold(⋅∣q)[1∑i=1G∣oi∣∑i=1G∑t=1∣oi∣min⁡(ri,t(θ)A^i,t,clip(ri,t(θ),1−εlow,1+εhigh)A^i,t)]\mathbf{J}_{\text{DAPO}}(\pi_\theta) \triangleq \mathbb{E}_{(q,a) \sim \mathcal{D}, \{o_i\}_{i=1}^G \sim \pi_{\theta_{\text{old}}}(\cdot|q)} \left[ \frac{1}{\sum_{i=1}^G |o_i|} \sum_{i=1}^G \sum_{t=1}^{|o_i|} \min\left( r_{i,t}(\theta) \hat{A}_{i,t}, \text{clip}(r_{i,t}(\theta), 1-\varepsilon_{\text{low}}, 1+\varepsilon_{\text{high}}) \hat{A}_{i,t} \right) \right]

subject to:

0<∣{oi∣is_equivalent(a,oi)}∣<G0 < |\{ o_i \mid \text{is\_equivalent}(a, o_i) \}| < G

Key modifications:

  • Decoupled clip: The upper clip $\varepsilon_{\text{high}}$ is set higher than the lower clip $\varepsilon_{\text{low}}$. This "allows more room for increasing low-probability tokens," which addresses entropy collapse by making it easier for the policy to explore new token distributions.
  • Token-level loss: The average is computed over all tokens across all responses ($\sum_{i=1}^G |o_i|$ in the denominator) rather than averaging per-sample first and then across samples. This "encourages longer sequences to make greater contributions to the overall gradient update, which is critical in long CoT RL scenarios."
  • Dynamic sampling constraint: Prompts where all $G$ rollouts are correct or all are incorrect are filtered out of the training batch. These provide zero advantage and thus zero gradient signal, but "may contribute to the KL loss or entropy loss, leading to a more unstable training process."

VC-PPO (Value-Calibrated PPO). Yuan et al. (2025) identify two PPO failure modes specific to long chain-of-thought training:

  • Value initialization bias: Initializing the critic model with a reward model trained only on <EOS> tokens creates a position-dependent bias that artificially favors shorter completions—the critic "thinks" later tokens are more important because it was only trained to predict outcomes at the final token.
  • Reward signal decay: The GAE trace decay rate $\lambda < 1$ causes the advantage estimate for early tokens in a long sequence to be severely attenuated, effectively removing useful gradient signal from early reasoning steps.

VC-PPO addresses these through:

  1. Value pretraining: The value model is pretrained under a fixed SFT policy using Monte Carlo returns ($\lambda = 1$)—the critic learns to predict the true outcome without temporal decay, eliminating the initialization bias.
  2. Decoupled-GAE: The policy (actor) uses $\lambda = 0.95$ for variance reduction in advantage estimation, while the value (critic) update uses $\lambda = 1$ for unbiased value targets. This means the critic learns the true expected return while the actor benefits from lower-variance gradients.

VAPO (Value-based Augmented PPO). Yue et al. (2025b) builds directly on VC-PPO and DAPO, combining their innovations:

JVAPO(πθ)≜JPPO-CH(πθ)+μJNLL(πθ)\mathbf{J}_{\text{VAPO}}(\pi_\theta) \triangleq \mathbf{J}_{\text{PPO-CH}}(\pi_\theta) + \mu \mathbf{J}_{\text{NLL}}(\pi_\theta)

where $\mathbf{J}_{\text{PPO-CH}}$ is DAPO's clip-higher objective with VC-PPO's critic-based advantages (using Length-Adaptive GAE: $\lambda_{\text{policy}} \triangleq 1 - 1/(\alpha l)$ where $l$ is the rollout length and $\alpha$ is a scaling hyperparameter), and $\mathbf{J}_{\text{NLL}}$ is a negative log-likelihood penalty restricted to correct rollouts:

JNLL(πθ)≜E(q,a)∼D,{oi}i=1G∼πθold(⋅∣q)[−1∑i∈T∣oi∣∑i∈T∑t=1∣oi∣log⁡πθ(oi,t∣q,oi,<t)]\mathbf{J}_{\text{NLL}}(\pi_\theta) \triangleq \mathbb{E}_{(q,a) \sim \mathcal{D}, \{o_i\}_{i=1}^G \sim \pi_{\theta_{\text{old}}}(\cdot|q)} \left[ -\frac{1}{\sum_{i \in \mathcal{T}} |o_i|} \sum_{i \in \mathcal{T}} \sum_{t=1}^{|o_i|} \log \pi_\theta(o_{i,t} | q, o_{i,<t}) \right]

where $\mathcal{T}$ is the set of rollout indices that achieved correct answers, and $\mu > 0$ is the penalty rate.

What the NLL term does: This is interpreted as performing SFT over only the correct rollout samples during RL training. It provides an additional gradient signal that directly increases the probability of responses that are known to be correct, beyond the advantage-weighted policy gradient. This addresses reward sparsity—when many rollouts are incorrect, the advantage signal alone may be too weak to effectively reinforce the structure of correct responses.

REINFORCE variants. The survey also documents simpler approaches based on the REINFORCE estimator:

  • REINFORCE++ (Hu, 2025): Normalizes penalized rewards within each group, where the penalty is a token-wise KL divergence incorporated into the reward function before normalization:

    R^i,t≜r(oi,a)−β∑j=t∣oi∣log⁡(πθold(oi,j∣q,oi,<j)πθref(oi,j∣q,oi,<j))\hat{R}_{i,t} \triangleq r(o_i, a) - \beta \sum_{j=t}^{|o_i|} \log\left( \frac{\pi_{\theta_{\text{old}}}(o_{i,j} | q, o_{i,<j})}{\pi_{\theta_{\text{ref}}}(o_{i,j} | q, o_{i,<j})} \right)

    This shifts the KL penalty from the objective (as in GRPO) to the reward (as in earlier RLHF work), producing a simpler update rule.

  • REINFORCE-Kimi (Kimi-Team et al., 2025): A remarkably minimal approach that uses the basic REINFORCE gradient with centralized rewards (subtracting the group mean) and a K2 KL penalty:

    ∇θJKIMI(πθ)≜E(q,a)∼D,{oi}i=1G∼πθold(⋅∣q)[1G∑i=1G1∣oi∣∑t=1∣oi∣(∇θlog⁡πθ(oi∣q)(r(oi,a)−rˉ)−β2∇θ(log⁡πθ(oi∣q)πθold(oi∣q))2)]\nabla_\theta \mathbf{J}_{\text{KIMI}}(\pi_\theta) \triangleq \mathbb{E}_{(q,a) \sim \mathcal{D}, \{o_i\}_{i=1}^G \sim \pi_{\theta_{\text{old}}}(\cdot|q)} \left[ \frac{1}{G} \sum_{i=1}^G \frac{1}{|o_i|} \sum_{t=1}^{|o_i|} \left( \nabla_\theta \log \pi_\theta(o_i|q) (r(o_i, a) - \bar{r}) - \frac{\beta}{2} \nabla_\theta \left( \log \frac{\pi_\theta(o_i|q)}{\pi_{\theta_{\text{old}}}(o_i|q)} \right)^2 \right) \right]

    where $\bar{r} \triangleq \text{mean}(\{r(o_i, a)\}_{i=1}^G)$. This demonstrates that even REINFORCE—the simplest policy gradient method—can achieve stable training on 72B models with appropriate reward normalization and KL regularization.

  • GPG (Group Policy Gradient, Chu et al., 2025b): An even more radical simplification that directly optimizes:

    JGPG(πθ)≜E(q,a)∼D,{oi}i=1G∼πθ(⋅∣q)[1∑i=1G∣oi∣∑i=1G∑t=1∣oi∣(−log⁡πθ(oi,t∣q,oi,<t)A^i,t)]\mathbf{J}_{\text{GPG}}(\pi_\theta) \triangleq \mathbb{E}_{(q,a) \sim \mathcal{D}, \{o_i\}_{i=1}^G \sim \pi_\theta(\cdot|q)} \left[ \frac{1}{\sum_{i=1}^G |o_i|} \sum_{i=1}^G \sum_{t=1}^{|o_i|} \left( -\log \pi_\theta(o_{i,t} | q, o_{i,<t}) \hat{A}_{i,t} \right) \right]

    where $\hat{A}_{i,t}$ is the group-normalized reward advantage. GPG eliminates the importance sampling ratio, the clipping, the critic, and the reference model—using only the REINFORCE gradient with group normalization and token-level loss. The survey notes this "offers significant advantages for scalability in distributional training."

  • CPPO (Completions Pruning Policy Optimization, Lin et al., 2025b): A computational efficiency intervention that filters rollouts:

    I≜{i∈{1,…,G}∣∣A^i,t∣≥γ}\mathcal{I} \triangleq \{ i \in \{1, \dots, G\} \mid |\hat{A}_{i,t}| \geq \gamma \}

    where $\gamma$ is a predefined threshold. Only rollouts with sufficiently high absolute advantage contribute to the policy update. The remaining rollouts are pruned—their forward pass and gradient computation are skipped—"accelerating the training process."

Survey's synthesis. The survey's key meta-level observation is: "these algorithms are not significantly different from the theoretical perspective." All are based on the same policy gradient foundation (REINFORCE or importance sampling), and their differences reduce to choices along a small set of dimensions: whether to use a critic (PPO vs. GRPO), how to normalize advantages (group vs. GAE), whether to clip importance ratios and how (symmetric vs. decoupled), and how to apply KL regularization (in-objective vs. in-reward vs. none). The empirical finding that multiple variants achieve strong results suggests that the exact algorithm choice is less important than the stabilization it provides—"the community has made engineering efforts to adapt the algorithms for more stable training."

3.4.5 Reward Design

The survey identifies rule-based outcome rewards as "the cornerstone of RL training" and documents three categories:

Accuracy rewards. The most fundamental component: assign a score of 1 for correct answers and 0 or -1 for incorrect ones. Several projects "sometimes use [accuracy rewards] exclusively, reflecting a minimalist approach to reward design" (Hu et al., 2025; Liu et al., 2025c). The survey's position is that this minimalism is a strength: "the minimal reward function design reduces the risk of reward hacking by leaving little room for unintended optimization."

However, the survey also notes that rule-based verification can fail for answers that are expressed in unexpected but mathematically equivalent forms. Seed-Thinking-v1.5 (ByteDance-Seed, 2025) addresses this with an LLM-based verifier for complex cases, while DAPO (Yu et al., 2025) sidesteps the issue entirely by ensuring all expected answers are integers.

Format rewards. These assign a score of 1 when the response follows a predefined structure (typically a "think" section followed by an "answer" section, with appropriate XML-style tags) and 0 or -1 otherwise. The survey's findings on format rewards are nuanced and somewhat contradictory:

  • Open-Reasoner-Zero (Hu et al., 2025) reports that "the format reward rapidly saturates, typically reaching its maximum within approximately 60 steps"—the model quickly learns the desired format and maintains it thereafter.
  • However, the same study also found that "models trained solely with accuracy rewards, when guided by well-designed prompts, can still quickly learn and reinforce the desired formatting," suggesting format rewards "may be unnecessary in some cases."
  • Logic-RL (Xie et al., 2025a) warns that format rewards "may inadvertently incentivize reward hacking behaviors"—the model can learn to produce the correct format without the correct reasoning, exploiting the reward structure.
  • The survey explicitly notes: "there are no rigorous ablation studies to prove the effectiveness of the format reward"—this is presented as a significant gap in the literature.

Length rewards. These are the most diverse category, varying from simple truncation penalties to sophisticated shape-based incentives:

  • Aggarwal and Welleck (2025) propose "rewarding generating responses of a desired length"—explicitly controlling verbosity.
  • Arora and Zanette (2025) propose "incentivizing brevity without sacrificing accuracy"—the model should produce concise correct answers.
  • DAPO (Yu et al., 2025) implements a "linear length penalty when responses exceed a predefined maximum length"—a hard constraint that prevents runaway generation.
  • Yeo et al. (2025) propose a "cosine-based reward that encourages longer reasoning processes for incorrect answers and more concise ones for correct responses." This is adopted by Light-R1 (Wen et al., 2025a). The operational logic is: when the model gets the wrong answer, it should think more (longer chains); when it gets the right answer, it should be efficient. This shape encourages exploration without rewarding verbosity for its own sake.

3.4.6 Sampling Strategies During RL Training

The survey documents several mechanisms by which training samples are selected, filtered, or scheduled during the RL process:

Curriculum learning on difficulty. Open-Reasoner-Zero (Hu et al., 2025) implements a two-stage curriculum. In the first stage (1100 steps), the model trains on data uniformly sampled from the full 129k dataset. Then, a challenging 13k subset is selected based on the model's current success rate—problems the model finds moderately difficult—and used to fine-tune the model "for improved performance on the most difficult reasoning problems." This mirrors the SFT data curation philosophy: once the model has mastered the basics, training should focus on the frontier of its capability.

Dynamic sampling. Both DAPO (Yu et al., 2025) and Skywork-OR1 (He et al., 2025a) implement dynamic sampling that "filters out zero advantage sample groups"—prompts for which all generated rollouts receive the same reward. The argument is that these groups "do not contribute to the policy loss, but may contribute to the KL loss or entropy loss, leading to a more unstable training process." By removing them, training focuses exclusively on prompts where the model is learning something.

Epoch-level history resampling. Skywork-OR1 (He et al., 2025a) and SRPO (Zhang et al., 2025b) introduce a strategy of dropping samples that were answered correctly by all rollouts in the previous epoch. The intuition is that if the model has fully mastered a problem, continuing to train on it wastes compute and may lead to overfitting.

Easy data resampling. MiMo (Xiaomi LLM-Core Team, 2025) takes a countervailing approach: maintaining an "easy data pool" of problems with perfect pass rates, and sampling from this pool with 10% probability during rollouts. The MiMo team argues that completely removing easy problems "introduces instability in policy updates" and that occasional exposure to mastered problems helps maintain performance while the model tackles harder cases.

Curriculum learning on sequence length. DeepScaleR (Luo et al., 2025b) progressively increases the maximum response length from 8k to 16k to 24k, with "performance improving consistently at each step." Skywork-OR1 similarly extends context length up to 32k during multi-stage training. This gradual approach prevents the model from being overwhelmed by the exploration space at the start while allowing it to develop increasingly sophisticated reasoning chains.

Handling truncated rollouts. Skywork-OR1 (He et al., 2025a) conducted an ablation study on whether truncated rollouts (responses that hit the maximum length limit) should be masked from the advantage calculation—i.e., should they be treated as having zero advantage to avoid penalizing potentially valid reasoning that was cut off? The result: "applying this masking strategy does not yield improved scaling behavior in later training stages, typically when the context length reaches 32k. Consequently, Skywork-OR1 opts not to apply masking for truncated rollouts during training." This suggests that at sufficiently long context lengths, truncation is rare enough that special handling is unnecessary.

3.4.7 Alternative Approaches: Process Reward Models and Preference Optimization

The survey identifies two major alternative paradigms to outcome-reward-based RLVR.

Process Reward Models (PRMs). Unlike outcome rewards that provide a single score per complete response, PRMs provide feedback at each intermediate step within the reasoning process. The survey notes that PRMs "transcend simplistic outcome-based score annotations" by enabling "models to navigate complex, multi-step tasks more effectively." Two approaches are highlighted:

  • rStar-Math (Guan et al., 2025): Trains a Process Preference Model (PPM) that learns to rank intermediate reasoning steps—distinguishing between more and less promising paths—during Monte Carlo Tree Search. The PPM and the policy model are jointly refined through iterative self-evolution.
  • PRIME (Cui et al., 2025): Introduces an "implicit PRM" trained solely on outcome labels, eliminating the need for expensive step-level annotations. The PRM is updated online using policy rollouts and outcome labels, providing a scalable way to integrate process-level feedback without the annotation bottleneck.

Preference Optimization. These methods replace online RL with direct optimization on preference pairs (chosen vs. rejected responses):

  • DPO (Direct Preference Optimization, Rafailov et al., 2023): Optimizes the policy directly from preference data without training a separate reward model or performing online rollouts. The survey reports that DPO "demands much less computational resources" than PPO or GRPO.

  • Iterative DPO (Tu et al., 2025): Extends DPO by constructing preference pairs through multiple rounds—the model generates responses, the best are selected as "chosen" and the worst as "rejected," and the model is fine-tuned, then the process repeats. The survey reports this "can rival the performance of online RL approaches."

  • DPO-R1 (Zhang et al., 2025a): Explicitly compares DPO with PPO for reasoning, finding that "DPO substantially enhances model performance while maintaining high training efficiency, and incorporating a SFT warm-up phase before DPO further boosts performance. Nonetheless, DPO still lags slightly behind PPO in overall effectiveness."

  • Light-R1 (Wen et al., 2025a): Implements a curriculum of two SFT stages followed by a DPO stage, where chosen samples are DeepSeek-R1 rollouts with verified correct answers and rejected samples are incorrect rollouts from the SFT checkpoint.

The survey's overall assessment is that preference optimization offers a computational efficiency advantage over online RL but has not yet matched its peak performance—making it an attractive option for resource-constrained settings or as a final refinement stage after SFT.

4. Key Insights and Innovations

Innovation 1: A Unified Mathematical Derivation Reveals RL Algorithm Diversity Is Surface-Deep

The single most intellectually distinctive contribution of this survey emerges in Section 3.2.1, where the authors derive every major RLVR algorithm variant—PPO, GRPO, Dr. GRPO, DAPO, VC-PPO, VAPO, REINFORCE++, REINFORCE-Kimi, GPG, CPPO—from a shared policy gradient foundation. This is not a taxonomic exercise in naming and categorizing; it is a conceptual decompression that transforms a bewildering landscape into a small, navigable decision space.

Before this survey, the literature presented these algorithms as distinct innovations. Each paper proposed its variant with empirical justifications—GRPO removes the critic for efficiency, DAPO decouples the clip to fix entropy collapse, VC-PPO pretrains the value model to fix initialization bias—but no work had shown how these choices relate to one another within a common parameterization. A practitioner selecting an algorithm faced what appeared to be a combinatorial explosion: GRPO with or without KL? With or without dynamic sampling? With symmetric or decoupled clipping? The space seemed to contain 2^n independent decisions.

The survey's derivation collapses this apparent complexity. It shows that every variant is built from the same policy gradient estimator (Equation 3–4), and that all differences reduce to choices along four dimensions:

  1. Advantage estimation: critic-based (GAE in PPO/VC-PPO/VAPO) versus group normalization (GRPO and its descendants) versus simple centralized rewards (REINFORCE-Kimi).
  2. Importance sampling handling: clipped ratio (PPO, GRPO) versus no ratio (GPG, REINFORCE-Kimi) versus filtered ratio (CPPO prunes low-advantage rollouts).
  3. KL regularization placement: in-objective penalty (GRPO, Dr. GRPO) versus in-reward penalty (REINFORCE++) versus post-gradient penalty (REINFORCE-Kimi's K2 formulation) versus none (DAPO, Dr. GRPO, SRPO, MiMo).
  4. Loss aggregation granularity: per-sample averaging (original GRPO) versus per-token averaging (DAPO, VAPO, GPG).

The theoretical punchline—"these algorithms are not significantly different from the theoretical perspective"—is quietly radical. It implies that the primary contribution of each paper was not a novel algorithm in the traditional sense (a new optimization procedure with different convergence guarantees) but rather an engineering intervention that stabilized training under specific conditions (model scale, data distribution, compute budget). The value of PPO's critic is not that it theoretically enables something GRPO cannot do, but that it provides more stable advantage estimates for certain training regimes. The value of DAPO's decoupled clip is not a new conceptual framework but a practical fix for entropy collapse observed in long chain-of-thought training.

This reframing matters because it redirects future work. Rather than asking "which algorithm is best?"—a question the survey's Table 5 shows has no universal answer—researchers should ask "which stabilization mechanisms are necessary for my specific setting?" The survey implicitly argues for a configurable framework approach to RLVR algorithm design: start from the shared gradient estimator, then activate stabilization components (critic, KL penalty, dynamic sampling, token-level loss) based on empirical need rather than methodological allegiance.

The evidence for this unifying view is distributed across the survey's comparative tables. Table 4 shows VAPO (value-based, critic, NLL penalty), DAPO (value-free, no KL, token-level loss), and REINFORCE-Kimi (minimal REINFORCE with group rewards) all achieving competitive results on 32B base models (60.4%, 50.0%, and 47.0% on AIME24 respectively). GPG-1.5B—the most radically simplified variant—achieves 33.3% on AIME24, competitive with GRPO-based DeepScaleR-1.5B at 43.1%. If fundamentally different algorithms were required for different scales, these results would not cluster. The fact that they do supports the survey's implicit claim: stabilization is the commodity; algorithm architecture is the packaging.

This is a fundamental intellectual contribution, not an incremental one. It does for RLVR what the Chinchilla scaling laws (Hoffmann et al., 2022) did for pretraining: it collapses a high-dimensional design space into a manageable set of trade-offs, enabling systematic rather than ad hoc exploration going forward.

Innovation 2: Data Difficulty as the Central Mediating Variable Across Both SFT and RLVR

The survey's second major conceptual contribution is the extraction of difficulty calibration as the unifying design principle spanning both supervised fine-tuning and reinforcement learning from verifiable rewards—two paradigms that the field had previously treated as methodologically distinct.

Prior to this survey, the SFT literature emphasized dataset size and domain coverage as primary drivers of reasoning performance. The RLVR literature emphasized algorithm design and reward specification. The survey's comparative analysis reveals that both paradigms are actually governed by the same underlying constraint: the model learns most efficiently from examples at the boundary of its current capability—not too easy (provides no gradient signal) and not too hard (provides no correct examples to learn from).

The evidence for this cross-paradigm principle is striking in its consistency:

  • In SFT: LIMO achieves 57.1% on AIME24 with 817 examples, competitive with models trained on 114k (OpenThoughts: 68.0% on 32B) or 894k (Synthetic-1: 30.0% on 7B) examples—but only because those 817 examples were selected through an elaborate difficulty-filtering pipeline that removed problems Qwen2.5-Math-7B-Instruct could solve easily and retained only challenging examples with verified high-quality reasoning traces. Light-R1's two-stage SFT pipeline explicitly operationalizes difficulty through pass-rate thresholds: the second stage uses only problems where the SFT checkpoint's pass rate is below 𝛼, creating a curriculum that concentrates training on the model's weaknesses.

  • In RLVR: DeepScaleR explicitly filters out problems with "overly high pass rates" (too easy) and "zero pass rates" (unverifiable or containing errors). Open-Reasoner-Zero selects a challenging 13k subset from 129k examples based on the model's intermediate success rate for curriculum fine-tuning. DAPO's dynamic sampling constraint 0 < |{o_i | is_equivalent(a, o_i)}| < G is a formalization of the difficulty principle: prompts where all rollouts are correct or all are incorrect are removed because they provide zero advantage. LIMR's "learning impact measurement" goes a step further, selecting samples whose learning patterns "complement the model's overall performance trajectory"—a dynamic, per-model difficulty calibration rather than a static, dataset-level one.

  • In both paradigms: The survey's synthesis (Section 3.3.1) explicitly states: "We observe a consistent trend: datasets used for RL training are carefully designed to include data where models are likely to make mistakes (i.e., models neither consistently succeed nor completely fail). Such uncertainty creates opportunities for learning."

This cross-paradigm unification is a conceptual reframing of significant practical importance. It implies that the primary question for both SFT and RLVR data curation is not "how much data do we need?" but "how do we identify the data at our model's learning frontier?" This shifts the research agenda from dataset scaling to difficulty estimation—a problem the survey flags as critical but largely unsolved. Current approaches require either generating thousands of rollouts per problem to estimate pass rates (expensive), using off-the-shelf LLMs to rate difficulty (unreliable across model families), or relying on static difficulty labels from competition archives (not model-specific). The survey implicitly calls for research on cheap, reliable, model-specific difficulty prediction.

The practical value of this reframing is evident in Table 2 and Table 5. The projects that most carefully calibrate difficulty—LIMO (817 examples), Light-R1 (3k second-stage examples), DeepScaleR (40k RL examples filtered by pass rate)—achieve performance competitive with or exceeding much larger datasets. The projects that emphasize scale over calibration (Synthetic-1 at 894k, AM at 1.4M) do not consistently outperform the calibrated smaller datasets on the benchmarks reported. This is not to say scale is irrelevant—DeepSeek-R1 itself used 800k SFT examples—but that scale without calibration is inefficient, and calibration can reduce the required scale by orders of magnitude.

This finding is fundamental, not incremental. It reorganizes the field's understanding of what matters in reasoning model training, demoting dataset size from a primary to a secondary concern and elevating difficulty estimation to a first-class design problem.

Innovation 3: A Documented Negative Result—KL Regularization Is Optional for Large-Scale RLVR

The survey's third distinctive contribution is a documented negative result with significant practical implications: KL divergence regularization—long considered essential for stable RL fine-tuning of language models, dating back to the originalRLHF formulation (Ouyang et al., 2022; Schulman et al., 2017)—appears to be unnecessary and possibly detrimental for outcome-reward-based reasoning training at scale.

The evidence for this claim is unusual because it comes not from a single controlled experiment but from a convergent pattern across independent replication studies, which the survey aggregates and makes legible for the first time:

  • Open-Reasoner-Zero (Hu et al., 2025): "ablation studies suggest that KL regularization may not be essential for large-scale RL. In fact, it can significantly restrict the increase in response length."
  • DAPO (Yu et al., 2025): omits KL loss entirely, achieves 50.0% on AIME24 from a 32B base model.
  • Dr. GRPO (Liu et al., 2025c): removes KL (along with other normalizations), demonstrates effectiveness on 7B.
  • SRPO (Zhang et al., 2025b): omits KL, achieves strong cross-domain performance.
  • MiMo (Xiaomi LLM-Core Team, 2025): omits KL, achieves 56.4% on AIME24 from a 7B base model—the strongest 7B result in the survey.
  • Skywork-OR1 (He et al., 2025a): conducts an explicit ablation, finding that "incorporating KL loss causes the actor to stay too closely aligned with the reference model. The KL divergence quickly drops toward zero, limiting policy exploration. As a result, performance on AIME24 plateaus, with limited improvement over training."

This convergent evidence contradicts the default assumption inherited from RLHF that KL regularization is necessary to prevent catastrophic forgetting and reward hacking. The survey's explanation—implicit in its presentation—is that outcome-reward-based reasoning training differs fundamentally from RLHF in the nature of the reward signal. In RLHF, the reward comes from a learned model that can be exploited (producing high-reward but nonsensical outputs), so KL to a safe reference policy is essential. In RLVR, the reward comes from a deterministic rule (exact answer match, format compliance) that cannot be "hacked" in the traditional sense—it can be exploited by producing correct answers with degenerate reasoning, but the survey documents that this is addressed through other mechanisms (length penalties, dynamic sampling) rather than KL constraints.

This is a fundamental practical finding, not incremental. It removes a major implementation burden from RLVR training pipelines: the need to maintain a frozen reference model in memory, compute per-token KL divergences during training, and tune the KL penalty coefficient β. For large-scale training where memory is the binding constraint, eliminating the reference model (which is the same size as the policy) effectively doubles available capacity for larger batch sizes or longer sequences. This partly explains why DAPO, MiMo, and Skywork-OR1 can train 32B models with long chain-of-thought rollouts—they are not paying the memory cost of a reference model.

However, the survey is appropriately cautious in its presentation. It does not claim KL is always harmful—Light-R1 and Logic-RL retain KL and report strong results—but rather that KL is not necessary, and that its presence can actively hurt exploration in long-CoT training. The distinction matters: practitioners should not blindly include KL because prior work did; they should evaluate whether it helps or hurts in their specific setting. This shifts KL from a default-on safety mechanism to a tunable hyperparameter whose value is setting-dependent.

The negative-result framing is important because the field has a publication bias toward positive findings. The survey's aggregation of independent observations that KL can be removed without performance degradation—and may even improve exploration—provides a counterweight to the default assumption, enabling practitioners to simplify their training pipelines with empirical justification.

Innovation 4: The SFT-RL Complementarity Thesis—SFT Stabilizes Format, RL Drives Generalization

The survey's fourth distinctive contribution is the articulation and evidence-collation for a specific thesis about the complementary roles of supervised fine-tuning and reinforcement learning in reasoning model training: SFT establishes format and basic reasoning patterns; RL enables generalization beyond the training distribution. This thesis emerges not from a single paper but from the survey's synthesis of apparently contradictory findings across multiple studies.

The evidence structure is:

SFT's role as format stabilizer: Chu et al. (2025a) directly demonstrates that "RL significantly enhances a model's ability to generalize across both textual and visual domains. In contrast, SFT often encourages memorization of the training data, which can impair performance on out-of-distribution tasks. Interestingly, while RL drives generalization, SFT remains crucial for stabilizing the model's output format—an essential property that facilitates effective downstream RL optimization." The survey's own SFT analysis (Section 2) implicitly supports this: models fine-tuned on DeepSeek-R1 traces learn to produce well-structured think/answer responses, but their generalization to unseen domains is limited by the coverage of the training data.

RL's role as generalization driver: The survey's Section 4.2 documents multiple cases where RL training on a narrow task produces broad generalization:

  • Llama3-SWE-RL (Huang et al., 2025a), trained solely on code repair, "demonstrates improved results on five out-of-domain tasks, including function coding, library use, code reasoning, mathematics, and general language understanding"—while an SFT baseline "led to an average performance degradation" on the same out-of-domain tasks.
  • RL-Poet (Doria, 2025), trained almost exclusively on English poetry with format rewards, "demonstrates the ability to generate literary poems in multiple languages with correct poetic rules."
  • The DeepSeek-R1-Zero training process itself—pure RL from a base model with no SFT—produced emergent reasoning behaviors (self-verification, backtracking) that were not present in any supervised data.

The tension and resolution: The complementarity thesis resolves a tension in the literature. If SFT alone can achieve strong benchmark performance (Table 2: Light-R1-32B at 73.0% AIME24, AM-72B at 76.5%), why use RL at all? Conversely, if RL alone can produce emergent reasoning from base models (DeepSeek-R1-Zero at 71.0% AIME24), why bother with SFT? The survey's answer—visible in the comparative structure of Tables 2 and 5—is that SFT provides a higher performance floor with less compute, while RL provides a higher ceiling with better generalization, and the combination of both outperforms either alone. DeepSeek-R1's pipeline (SFT → RL → SFT → RL) is the canonical example, but the survey shows that even simpler combinations (Light-R1's SFT → DPO, Skywork-OR1's distillation → GRPO) benefit from both stages.

This is a diagnostic reframing with practical implications. It suggests that projects should not choose between SFT and RL but should sequence them deliberately: SFT first to teach the model what reasoning traces look like and how to format them; RL second to push the model beyond imitation into exploration and generalization. It also implies that pure-RL approaches (DeepSeek-R1-Zero style) are viable but inefficient—they require the model to discover format and basic reasoning patterns through trial and error, which is computationally expensive compared to providing a few hundred high-quality SFT examples.

The significance of this thesis extends beyond reasoning. It suggests a general principle for post-training: imitation (SFT) provides structure; exploration (RL) provides capability. This reframes the long-standing debate about whether LLMs "understand" or merely "memorize"—the combination of SFT and RL may produce models that are structurally grounded (via SFT's format stabilization) while being behaviorally flexible (via RL's generalization), achieving a form of robust competence that neither paradigm achieves alone.

Innovation 5: The Overthinking and Safety Tax as First-Class Diagnostic Concepts for Reasoning Models

The survey's final distinctive conceptual contribution—developed primarily in Section 4.3—is the articulation of overthinking and the safety tax as diagnostic concepts that characterize failure modes specific to reasoning language models, distinct from the failure modes of standard LLMs.

Overthinking is defined through a cluster of empirical observations: reasoning models "allow for extended reasoning chains during inference, but this capability can sometimes cause issues like overthinking" (Sui et al., 2025; Chen et al., 2024a). The concept has two operational manifestations: (1) cost overthinking, where models generate unnecessarily long reasoning traces that increase inference cost without improving accuracy, exploited by attacks like OverThink (Kumar et al., 2025) that "trigger excessive reasoning, raising operational and environmental costs"; and (2) agentic overthinking, where models "exhibit reduced performance in agentic scenarios when environmental feedback is neglected" (Cuadron et al., 2025; Feng et al., 2025a)—the model thinks so much about what to do that it fails to incorporate feedback from what it actually did.

This is a new diagnostic category not present in pre-reasoning-model literature. Standard LLM failure modes include hallucination, factual error, and instruction-following failure, but "thinking too much" was not a coherent concept because standard models did not have explicit reasoning budgets. The emergence of reasoning language models with controllable chain-of-thought lengths creates a new failure axis: the model's internal deliberation, rather than being a pure good, can become pathological.

Safety tax refers to the observation that "incorporating safety alignment can inadvertently compromise the reasoning capabilities of these models" (Huang et al., 2025b). Zhao et al. (2025b) and Jiang et al. (2025) "observe substantial decreases in both helpfulness and harmlessness in reasoning language models compared to baseline models." This creates a genuine trade-off that was less acute for non-reasoning LLMs: safety training that constrains the model's output distribution may directly conflict with the exploration that RLVR requires to discover novel reasoning strategies.

The survey's framing of these concepts as first-class diagnostic tools—not merely as bugs to be fixed but as inherent tensions in reasoning model design—is a conceptual contribution. It transforms the safety discussion from "how do we prevent bad outputs?" to "how do we balance competing objectives (accuracy, safety, efficiency) when the very mechanisms that produce accurate reasoning (long chains, exploration, self-modification) can also produce unsafe or wasteful behavior?"

The evidence for these concepts is distributed across multiple sections but converges on a clear implication: reasoning models require new evaluation frameworks. Standard accuracy benchmarks (AIME24, MATH500) capture only one dimension of model quality. The survey's inclusion of overthinking attacks, the safety tax, and the complementary findings on generalization (RL generalizes but can narrow pass@k at large k; Yue et al., 2025a) collectively argue that the field needs multi-dimensional evaluation that accounts for reasoning quality, reasoning efficiency, safety under extended chains-of-thought, and out-of-distribution robustness. This is not a call for better metrics on existing benchmarks—it is a call for new benchmark categories that measure the unique failure modes of reasoning models.

The significance of this contribution is forward-looking. The survey is mapping the contours of a problem space that the field is only beginning to recognize. By naming and characterizing overthinking and the safety tax, it provides conceptual vocabulary that subsequent research can use to formulate hypotheses, design experiments, and report results. This is the survey operating at its highest level of synthesis—not merely cataloguing what has been done, but identifying what needs to be done and why it matters.

5. Experimental Analysis

Evaluation Methodology

  • Dataset. The primary evaluation benchmarks are AIME24, AIME25, and MATH500—standard competition-level and advanced mathematics problem sets drawn from the American Invitational Mathematics Examination (years 1984–2023 for training, 2024 and 2025 for evaluation) and the MATH benchmark (Hendrycks et al., 2021). These are the same benchmarks used by DeepSeek-R1 to report its own performance. Several SFT projects also report on GPQA Diamond (Rein et al., 2024) for out-of-domain generalization testing, though this is less consistently used across replication studies. The coding-focused RLVR projects (DeepCoder, SYNTHETIC-1, KodCode) use LiveCodeBench, CodeForces, or LeetCode test sets. The survey does not introduce a new evaluation dataset; it reports the benchmark results claimed by each individual replication study.

  • Base model(s). The replication landscape is dominated by the Qwen2.5 model family at three scales: 1.5B, 7B, and 32B parameters—with most projects using either the Base or Instruct variant, and several using the Math-specific variant (Qwen2.5-Math-7B). Certain projects (DeepScaleR, Skywork-OR1, Light-R1, GPG) start from DeepSeek-R1-Distilled Qwen checkpoints rather than raw Qwen2.5 models. The survey also documents projects using MiMo-7B-Base (Xiaomi LLM-Core Team, 2025), Pleias-350M (Doria, 2025), and Qwen2.5-3B-Base/Instruct (TinyZero, Mini-R1). The dominance of Qwen2.5 reflects its availability as a strong open-weight model family with published training details, making it a natural substrate for replication attempts. No single base model is used across all studies; cross-study comparison is inherently confounded by base model capability differences.

  • Metrics. The universal metric across all replication studies is accuracy—the fraction of problems for which the model's extracted final answer exactly matches the ground-truth answer. For math benchmarks, answer extraction is performed by parsing the model's output (typically looking for a boxed or otherwise delimited final answer), and equivalence is determined by symbolic evaluation (e.g., Math-Verify for LaTeX math expressions, execution-based verification for code). Several studies report pass@1 (single-sample accuracy) and some report pass@k (accuracy when selecting the best of k samples, either by majority voting or verifier scoring). The survey's Tables 2 and 5 report single-model accuracy numbers; they do not provide confidence intervals, standard deviations, or statistical significance tests. This is a notable limitation: for MATH500 (500 test problems), a difference of ~2 percentage points corresponds to approximately 10 problems, and without variance estimates, it is unclear whether reported differences between models are statistically reliable.

  • Baselines. The primary baselines against which replication studies compare their results are the official DeepSeek-R1 and DeepSeek-R1-Distilled model checkpoints at corresponding parameter scales. For SFT studies, the baseline is typically the DeepSeek-R1-Distilled model of the same size (e.g., DeepSeek-R1-Distill-Qwen-7B achieving 55.5% on AIME24, 92.8% on MATH500). For RLVR studies, the baselines are: (a) the initial checkpoint before RL training (often 0% on AIME24 for base models that have never seen competition math), (b) the DeepSeek-R1-Zero result when starting from a comparable base model (47.0% on AIME24 for DeepSeek-R1-Zero-Qwen-32B, per Guo et al., 2025), and (c) proprietary model performances (OpenAI o1, o4-mini, Qwen3-235B, Seed-Thinking-v1.5) as upper-reference points. Several RLVR projects also compare against the corresponding DeepSeek-R1-Distilled model to assess whether RL can surpass distillation—a key question the survey flags as unresolved for models ≤32B.

  • Generation budget / compute accounting. The survey does not standardize compute measurement across studies; each replication project uses its own accounting. RL training is measured in training steps and the number of rollouts per step (group size G, typically 8–64). Inference-time evaluation typically uses pass@1 with a single sample, though some studies additionally report pass@k for k > 1. The survey does not attempt to estimate total FLOPs consumed by each replication study—a significant gap, since training costs vary dramatically: DeepScaleR's progressive context length increase from 8k to 24k implies different per-step compute than DAPO's 16k fixed-length training, but the survey does not provide a common-cost basis for comparison. Several studies report "number of training steps" or "number of epochs," but these are not directly comparable across different dataset sizes and model scales.

  • Cross-validation / statistical protocol. The survey itself does not perform experiments and therefore does not have a cross-validation protocol. Among the replication studies it surveys, few report explicit cross-validation procedures. The exception is in data decontamination: several projects (Light-R1, LIMO) report explicit decontamination of their training sets against evaluation benchmarks, and LIMR reports that it uses the model's "overall performance trajectory" to select training samples—an implicit form of per-model validation. The survey authors themselves note (Section 2.1.3) that "only the technical reports of Light-R1 and LIMO explicitly mention conducting proper data decontamination against popular reasoning benchmarks." This means results from projects that did NOT perform decontamination (which includes most of the entries in Table 2 and Table 5) may be inflated by training-test overlap. This is a serious methodological concern that the survey identifies but cannot retrospectively correct.

Main Quantitative Results

Supervised Fine-Tuning Performance (Table 2)

The headline SFT finding is that high-quality, difficulty-filtered SFT datasets enable small-to-medium models to approach or match the performance of DeepSeek-R1's officially distilled models, with data curation quality dominating dataset size as a predictor of success. Table 2 organizes results on AIME24, AIME25, and MATH500 across projects using Qwen2.5 checkpoints.

On AIME24, the strongest SFT-only results at the 32B scale come from AM (Zhao et al., 2025a) at 72.7% starting from Qwen2.5-32B-Base, and Light-R1 (Wen et al., 2025a) at 73.0% starting from Qwen2.5-32B-Instruct. Both approach the DeepSeek-R1-Distill-Qwen-32B baseline of 72.6%. At the 7B scale, Light-R1 reports 59.1% when starting from the R1-distilled checkpoint (applying further RL, not pure SFT), while Open-R1 achieves 36.7% from Qwen2.5-Math-7B-Instruct—a substantial gap that the survey attributes primarily to data quality differences (Open-R1 uses a larger but less-difficulty-filtered dataset of 220k examples vs. Light-R1's aggressively curated 76k + 3k examples).

On AIME25, which was released after DeepSeek-R1's training data cutoff and thus serves as a cleaner out-of-distribution test, Light-R1-32B achieves 64.3%—a substantial drop from its AIME24 score, but still competitive with Skywork-OR1-32B-Preview at 69.0% (though Skywork-OR1 includes RL training, not pure SFT). S1k-1.1 achieves 50.0% on AIME25, competitive with LIMO's 44.5%—both using ~1k or fewer examples, reinforcing the "data quality over quantity" thesis.

On MATH500, the pattern is compressed: most 32B models achieve 90–97%, suggesting MATH500 is approaching saturation for strong models and provides less discriminative power than AIME24/25. The notable outlier is AM-72B at 97.0% from a base model—matching DeepSeek-R1's own 97.3%—but this should be interpreted cautiously given the lack of decontamination reporting.

The survey's critical cross-cutting observation from Table 2 is the base model vs. instruct model split: all projects achieving >70% on AIME24 at 32B scale start from either Qwen2.5-Base (AM, DeepSeek-R1 distilled series) or Qwen2.5-Instruct (Light-R1, LIMO, S1k-1.1). Projects starting from smaller or weaker base models achieve substantially lower scores—Open-R1-7B at 36.7%, Bespoke-Stratos-7B at 20.0%. This strongly suggests that the base model's inherent capability is a critical (and perhaps limiting) factor that data curation alone cannot fully compensate for. The survey cites Li et al. (2025d): "small models struggle to learn from strong reasoners."

RLVR Performance (Table 5)

Table 5 organizes RLVR results across model scales from 1.5B to 32B, with a crucial organizational split: models trained from base models are grouped separately from models trained from already-distilled or instruct models, because the starting checkpoint's capability dramatically affects the RLVR outcome.

At the 32B scale from base models: VAPO (Yue et al., 2025b) achieves 60.4% on AIME24, the strongest open-source RLVR result from a base checkpoint, surpassing DeepSeek-R1-Zero-Qwen-32B's 47.0% and the original DeepSeek-R1-Zero's 71.0% (though the latter starts from DeepSeek-V3-Base, a much stronger foundation). DAPO, SRPO, VC-PPO, and Open-Reasoner-Zero cluster in the 48.1–50.0% range—a remarkably tight band given their algorithmic diversity (DAPO uses GRPO with decoupled clip; VC-PPO uses value-calibrated PPO; Open-Reasoner-Zero uses vanilla PPO). This clustering is the survey's strongest empirical evidence for its central claim that algorithm architecture matters less than training stabilization—four different algorithms with different components (critic, KL, dynamic sampling) converge to similar performance from the same base model on the same benchmark.

At the 32B scale from distilled models: Skywork-OR1-32B-Preview (He et al., 2025a) achieves 79.7% on AIME24, substantially exceeding the DeepSeek-R1-Distill-Qwen-32B baseline of 72.6%. This demonstrates that applying RL to an already-distilled model can produce further gains—a finding that challenges the claim in Guo et al. (2025) that "smaller models may fail to match the performance of distillation models through RL training." The discrepancy likely reflects Skywork-OR1's extensive data cleaning and difficulty calibration pipeline, which may provide better RL training data than DeepSeek's internal pipeline.

At the 14B scale: Light-R1-14B-DS (Wen et al., 2025a) achieves 74.0% on AIME24 from an SFT checkpoint, exceeding DeepSeek-R1-Distill-Qwen-14B's 69.7%. This is a direct demonstration that SFT + GRPO can outperform pure distillation, at least at this scale and on this benchmark.

At the 7B scale: The strongest result is MiMo-7B-RL-Zero (Xiaomi LLM-Core Team, 2025) at 56.4% on AIME24 from MiMo-7B-Base—a 7B model trained purely with RLVR (GRPO with clip-higher and dynamic sampling) achieving performance exceeding DeepSeek-R1-Distill-Qwen-7B's 55.5%. This is the survey's most striking data point for the claim that RLVR can surpass distillation even at small model scales. Skywork-OR1-Math-7B achieves 69.8% on AIME24 starting from DeepSeek-R1-Distill-Qwen-7B—reinforcing that further RL on distilled checkpoints yields gains. However, Oat-Zero-7B at 43.3% and Qwen2.5-7B-SimpleRL-Zero at 36.7% demonstrate that the specific RLVR recipe strongly affects outcomes—starting from the same Qwen2.5-Math-7B base, the gap between MiMo-7B-RL-Zero's 56.4% and SimpleRL-Zero's 36.7% is nearly 20 percentage points, despite both using GRPO variants.

At the 1.5B scale: DeepScaleR-1.5B-Preview (Luo et al., 2025b) achieves 43.1% on AIME24, which the survey describes as "surpassing o1-preview"—but this requires careful contextualization. OpenAI's o1-preview is a proprietary model of much larger scale; DeepScaleR surpasses it specifically on a single benchmark (AIME24) using a specific training recipe. This is impressive but not a general capability claim. GPG-1.5B (Chu et al., 2025b) achieves 33.3% on AIME24 using the simplest algorithm variant—a REINFORCE-based method with no critic, no clipping, and no reference model—while DeepSeek-R1-Distill-Qwen-1.5B achieves only 28.9%. The gap between DeepScaleR (43.1%) and the distilled baseline (28.9%) illustrates the magnitude of improvement that carefully tuned RLVR can provide over distillation at very small scales.

Cross-benchmark consistency: On MATH500, the performance range is compressed: most 7B+ models achieve 80–94%, with the strongest being AM-72B at 97.0% and DeepSeek-R1 at 97.3%. This compression reinforces the survey's observation that MATH500 is approaching saturation and provides limited discriminative power for comparing state-of-the-art models. AIME24, with its harder problems and lower baseline performance, provides better separation.

The SFT Data Quantity-Quality Tradeoff

Figure 2 and the accompanying analysis in Section 2.1.3 provide the first systematic visualization of token length distributions across major SFT datasets. The key quantitative observation is that datasets "skewed toward shorter sequences" (AM, Synthetic-1) produce models with different reasoning characteristics than datasets with "broader distributions with longer tails" (Light-R1, Open-R1), even when all traces originate from the same teacher model. The survey does not provide a controlled experiment isolating this effect—different datasets also differ in domain composition and filtering criteria—but the observation is methodologically important because it documents that the teacher model's raw output distribution is not what determines SFT effectiveness; rather, the selection process over that distribution (which length traces to retain, which difficulty level to target) is the active ingredient.

Cross-Study Patterns in RL Training

Section 3.3 aggregates implementation details across RLVR replication studies to extract common practices and their empirical justifications. The key cross-study findings are:

  • Context length matters and scales non-trivially. From 1.5B models (DeepScaleR's progressive 8k→16k→24k increase) to 32B models (Skywork-OR1 up to 32k), every project that reports context length scaling observes performance improvements with longer maximum response lengths—but also diminishing returns and increased training instability. Skywork-OR1's ablation on masking truncated rollouts (finding it "does not yield improved scaling behavior in later training stages") is the only controlled experiment on this question that the survey documents, and it suggests that at sufficiently long context lengths, truncation becomes rare enough that special handling is unnecessary.

  • KL regularization is frequently omitted with no performance degradation. Open-Reasoner-Zero removes KL, DAPO removes KL, Dr. GRPO removes KL, SRPO removes KL, MiMo removes KL—all achieve strong results. Skywork-OR1 provides the only direct ablation, showing "incorporating KL loss causes the actor to stay too closely aligned with the reference model... performance on AIME24 plateaus." The survey does not claim KL is universally harmful (Light-R1 retains it and reports strong results at 14B scale), but the weight of independent evidence converges on "KL is optional, possibly detrimental for long-CoT exploration."

  • Format rewards saturate rapidly and their necessity is unproven. Open-Reasoner-Zero reports format reward reaching its maximum "within approximately 60 steps." Multiple projects note that well-designed prompts can achieve the same formatting compliance without explicit rewards. The survey explicitly flags the absence of rigorous ablation studies on format rewards as a gap in the literature.

Ablation Studies and Robustness Checks

The survey itself does not run experiments; it reports and synthesizes ablations from the individual replication studies it covers. These are distributed across Section 3.3 and organized below by the aspect being ablated.

KL regularization (Skywork-OR1, Open-Reasoner-Zero, DAPO, Dr. GRPO): Multiple independent projects converge on the finding that removing KL divergence regularization does not degrade and may improve RLVR performance. Skywork-OR1's explicit ablation on DeepSeek-R1-Distill-Qwen-7B shows KL causing the actor to stay too closely aligned with the reference model, with performance on AIME24 plateauing. Open-Reasoner-Zero reports that KL "can significantly restrict the increase in response length." DAPO, Dr. GRPO, SRPO, and MiMo all omit KL and achieve strong results at various model scales. Countervailing evidence: Light-R1 and Logic-RL retain KL and also achieve strong results, suggesting setting-dependent effects. The survey does not synthesize these into a clear recommendation, instead flagging the lack of a systematic cross-setting KL ablation as an open question.

Masking truncated rollouts (Skywork-OR1): Applying a masking strategy to exclude truncated rollouts from advantage calculation (to avoid penalizing valid reasoning cut off by length limits) "does not yield improved scaling behavior in later training stages, typically when the context length reaches 32k." This is a negative result with practical implications: the added complexity of masking is unnecessary at sufficiently long context lengths.

Format reward ablation (Logic-RL, Open-Reasoner-Zero): Logic-RL warns that format rewards "may inadvertently incentivize reward hacking behaviors." However, the survey explicitly notes: "there are no rigorous ablation studies to prove the effectiveness of the format reward." This is a gap in the literature—no single replication study has systematically compared accuracy-only vs. accuracy+format rewards while controlling for all other variables.

GRPO vs. PPO vs. REINFORCE++ (Logic-RL): In a controlled comparison on the K&K text logic dataset, Logic-RL (Xie et al., 2025a) finds that PPO (with λ=1.0, γ=1.0) "significantly outperforms GRPO and REINFORCE++ in both accuracy and reward," with REINFORCE++ consistently surpassing GRPO and GRPO showing "the weakest overall performance among the three." This is one of the few direct algorithm comparisons in the literature. However, the survey cautions that these results are on a specific logic dataset and may not generalize to math and coding domains where other studies find GRPO variants highly effective.

Value pretraining and decoupled GAE (VC-PPO): Yuan et al. (2025) provide targeted ablations for their two interventions: value pretraining (pretraining the critic under a fixed SFT policy with Monte Carlo returns, λ=1) eliminates position-dependent value initialization bias that favors shorter completions; decoupled-GAE (actor λ=0.95, critic λ=1) addresses reward signal decay for early tokens in long sequences. Both interventions are shown to improve PPO stability on long-CoT benchmarks like AIME24, but are specific to PPO-based methods and do not apply to GRPO variants that lack a critic.

Clip-higher and dynamic sampling (DAPO): Yu et al. (2025) demonstrate that decoupling the clip threshold (ε_high > ε_low) addresses entropy collapse by allowing more room for increasing low-probability tokens, while dynamic sampling (filtering prompts with all-correct or all-incorrect rollouts) stabilizes training by removing zero-advantage examples. These are presented as complementary fixes to distinct GRPO failure modes. The survey does not report a formal ablation isolating each component's contribution, but DAPO's 50.0% AIME24 from Qwen2.5-32B-Base (matching SRPO's 50.0%) suggests the combination is effective without being uniquely so.

Curriculum learning on difficulty (Open-Reasoner-Zero): The two-stage curriculum (1100 steps on full dataset → fine-tune on challenging 13k subset) is ablated implicitly by comparison to non-curriculum baselines. However, the survey does not report a direct ablation (training on the full dataset for all steps vs. the curriculum approach), making it difficult to isolate the curriculum's contribution from other Open-Reasoner-Zero design choices.

Curriculum learning on context length (DeepScaleR, Skywork-OR1): The progressive increase from 8k to 16k to 24k (DeepScaleR) or up to 32k (Skywork-OR1) is described as yielding consistent performance improvements at each step. This functions as an implicit ablation: training directly at 24k without intermediate stages would likely be unstable or computationally wasteful, though neither project reports a direct comparison.

PRM aggregation strategy (borrowed from process-reward-model literature, reported in Section 4.1): The survey notes that process-level reward models (PRMs) provide step-level feedback that improves over outcome-only rewards, but this finding comes from separate work (rStar-Math, PRIME) rather than from the DeepSeek-R1 replication studies per se. The comparison of PRM vs. outcome reward is not a controlled ablation within the survey's scope.

Critical Assessment

Does the evidence support the survey's central claims?

Claim: Dataset curation quality dominates dataset size for SFT-based reasoning reproduction. The evidence for this is distributed across Table 2 and Figure 2 but is fundamentally observational, not experimental. LIMO (817 examples, 57.1% AIME24) substantially outperforms Synthetic-1 (894k examples, 30.0% AIME24 at 7B) and OpenThoughts (114k examples, 31.3% AIME24 at 7B)—but these comparisons confound dataset quality with base model capability, domain composition, verification rigor, and training configuration. LIMO starts from Qwen2.5-32B-Instruct; Synthetic-1 from Qwen-2.5-7B-Instruct; OpenThoughts-7B from Qwen2.5-7B-Instruct. The survey acknowledges the base model confound but does not have the data to disentangle how much of LIMO's advantage comes from data curation vs. from the 32B instruct model's higher learning efficiency. A proper test of the data-quality claim would require: same base model, same training configuration, varying only the dataset—and no such experiment is reported by any of the surveyed studies. The survey's claim is best interpreted as an empirically-grounded hypothesis rather than a verified finding.

Claim: RLVR can surpass distillation even at small model scales (≤7B). MiMo-7B-RL-Zero at 56.4% AIME24 vs. DeepSeek-R1-Distill-Qwen-7B at 55.5% provides a single positive data point. DeepScaleR-1.5B at 43.1% vs. DeepSeek-R1-Distill-Qwen-1.5B at 28.9% provides another. However, these are point comparisons without statistical quantification—the survey does not report whether the 0.9 percentage point gap at 7B (56.4% vs. 55.5%) is within sampling error. AIME24 has 30 problems; a difference of 0.9 points corresponds to roughly 0.27 problems, which is well within the expected variance of a 30-question test. DeepScaleR's 14.2-point gap at 1.5B is more convincing, but DeepScaleR was trained with an elaborate multi-stage curriculum (progressive context length increase, pass-rate-based data filtering) that may not reflect "standard" RLVR. The claim that RLVR generally surpasses distillation at small scales is suggestive but not robustly established by the evidence surveyed.

Claim: Algorithm variants are not significantly different from a theoretical perspective, and stabilization engineering dominates performance. This is the survey's strongest empirical claim, and the evidence is the tight clustering of diverse algorithms at the 32B base-model scale: VAPO (value-based PPO) at 60.4%, DAPO (value-free GRPO variant) at 50.0%, VC-PPO (value-calibrated PPO) at 48.8%, Open-Reasoner-Zero (vanilla PPO) at 48.1%, DeepSeek-R1-Zero-Qwen-32B (GRPO) at 47.0%. The 13.4-point spread between VAPO and the cluster at ~48% complicates the "algorithms don't matter" narrative—VAPO incorporates value pretraining, decoupled GAE, clip-higher, token-level loss, and an NLL penalty, suggesting that the combination of stabilizations may matter even if no single component is decisive. More critically, the comparison is across different projects with different datasets, different training schedules, and different hyperparameter tuning budgets, making algorithm effects inseparable from implementation quality effects. A proper test would require: same base model, same data, same compute budget, systematic variation of algorithm components—and no such experiment exists in the surveyed literature.

Claim: SFT stabilizes format while RL drives generalization. This claim comes from Chu et al. (2025a), which the survey cites but does not independently verify. The evidence within the survey's scope is circumstantial: SFT models excel at in-distribution benchmarks (Table 2) while RLVR models show generalization to out-of-domain tasks (Section 4.2, citing RL-Poet, Llama3-SWE-RL). But the survey does not present a controlled comparison of the same base model trained with SFT-only vs. RL-only vs. SFT+RL on both in-distribution and out-of-distribution tasks. The claim is better characterized as a framing hypothesis that organizes observations rather than a causal claim verified by experiment.

Genuine weaknesses in the surveyed evidence

The survey's most important critical function is to identify methodological weaknesses across the replication literature, and it does so with commendable specificity:

  • Decontamination is the elephant in the room. The survey explicitly reports that only two projects (Light-R1, LIMO) confirm decontamination of their training data against evaluation benchmarks. Wen et al. (2025a) found that "MATH500 is partially compromised across several open-source datasets, including OpenThoughts, Open-R1, Bespoke Stratos." This means some fraction of the results in Table 2 and Table 5 may be inflated by training-test overlap, and the survey cannot determine which results or by how much. This is a fundamental validity concern that applies to the majority of the replication literature. The survey's Figure 3 (cross-referencing dataset sources) is partly motivated by this concern, but it documents dependencies, not contamination—a dataset can derive from NuminaMath without containing MATH500 problems, or it can contain them depending on the subset used.

  • No confidence intervals anywhere. Every number in Tables 2 and 5 is a point estimate without variance information. AIME24 has 30 questions; MATH500 has 500. The sampling error on 30 questions is substantial—a 3-point difference corresponds to roughly one problem. The survey does not discuss this, and neither do most of the surveyed papers. This makes fine-grained ranking of models (e.g., VAPO at 60.4% vs. Light-R1 at 59.1% vs. MiMo-7B at 56.4%) statistically meaningless without additional information.

  • The base model confound is pervasive. Projects compare their models against DeepSeek-R1-Distilled baselines, but the training data, training hyperparameters, and base model differ simultaneously. When Skywork-OR1-32B surpasses DeepSeek-R1-Distill-Qwen-32B (79.7% vs. 72.6%), is this because of better RL training, better data curation, better hyperparameters, or some interaction? The survey cannot answer this because no study performs a controlled comparison varying only the training procedure while holding data and base model constant.

  • The proprietary model comparisons are asymmetric. Several studies claim to "surpass o1-preview" or "match o1" on specific benchmarks, but o1-preview's performance is evaluated with unknown prompting, unknown decoding parameters, and unknown evaluation protocol. The survey reports these claims but does not critically assess their validity, which is a missed opportunity given that these comparisons are often used to headline replication papers.

  • Single-benchmark focus limits generality claims. The vast majority of results in Tables 2 and 5 are on AIME24 and MATH500—both competition-level math benchmarks. The survey's Section 3.4 documents RLVR applications to code, logic, chemistry, and creative writing, but performance on these tasks is reported qualitatively or anecdotally without standardized benchmarks. Claims about generalization (Section 4.2) thus rest on a thin evidentiary base.

Missing experiments that would strengthen the surveyed literature

  • A controlled data-quality ablation: Same base model, same training configuration, vary only the dataset (e.g., LIMO's 817 examples vs. a random subset of Synthetic-1 at 817 examples vs. Synthetic-1 at full scale). None of the surveyed studies performs this experiment, making the "data quality > data quantity" claim fundamentally correlational.

  • A systematic algorithm component ablation: Same base model, same data, vary one component at a time (KL on/off, clip symmetric/decoupled, loss token-level/sample-level). The DAPO and Dr. GRPO papers perform some ablations of their modifications against the original GRPO, but no study crosses the project boundary to compare, say, DAPO's token-level loss against Dr. GRPO's bias corrections against VAPO's NLL penalty on a common substrate.

  • Scale sweeps with matched compute: Compare RLVR at 1.5B, 7B, 32B with matched total FLOPs (not matched steps, which favor larger models due to higher per-step cost). The absence of FLOPs-matched comparisons across model scales is a notable gap in a literature that otherwise emphasizes efficiency.

  • Multiple-evaluation-run reporting: Given small test sets (especially AIME24/25 with 30 problems), reporting mean and standard deviation across multiple evaluation runs (with different random seeds for sampling) would provide basic statistical grounding that is currently absent.

Conditional nature of the claims

The survey's findings are conditioned on several factors that it explicitly acknowledges:

  • Base model capability boundary: RLVR cannot create reasoning capability from nothing—it requires a base model with non-trivial prior knowledge. This is demonstrated by the 1.5B results (28.9% distilled baseline, 43.1% with DeepScaleR's aggressive RL) vs. the inability of RLVR to achieve meaningful performance on base models with near-zero starting capability (the survey does not document specific failures, but notes this principle in Section 3.3.1).

  • Benchmark difficulty range: The MATH500 compression effect (most strong models achieving 90%+) means that findings about relative model performance on MATH500 are less reliable than on AIME24. The survey's emphasis on AIME24 as the primary comparison benchmark is justified but limits generalizability to problems of similar difficulty.

  • Language and domain scope: Almost all replication studies use English math problems. The survey's Section 3.4 documents applications beyond math (code, logic, chemistry, poetry) but the quantitative evidence for these domains is sparse. Claims about RLVR's generality are aspirational rather than empirically grounded by the data presented.

  • Training data overlap: As noted above, the unknown degree of training-test contamination across projects means that all quantitative comparisons in Tables 2 and 5 should be interpreted as upper bounds on true generalization performance—any model whose training data was not properly decontaminated may have inflated scores. The survey cannot quantify this inflation without access to each project's training data and decontamination procedure, which is typically not provided.

6. Limitations and Trade-offs

Limitation 1: Difficulty Estimation Cost Is Unaccounted for in Reported Efficiency Gains

The assumption or constraint. The entire compute-optimal framework described in the original DeepSeek-R1 paper and partially replicated by the surveyed studies rests on knowing which problems are easy, medium, or hard before allocating training or inference compute. The original work estimates difficulty by generating 2048 solutions per question and computing pass@1 rates, then binning questions into quintiles. The survey does not challenge this assumption directly—it reproduces difficulty-based filtering as a consensus best practice—but it also does not account for the cost of difficulty estimation in any of the reported comparisons.

The survey states this implicitly when discussing SFT data curation (Section 2.1.1): Light-R1 "uses a trained checkpoint to sample and verify responses... retaining only samples with a moderate pass rate." DeepScaleR similarly filters RL training data by pass rate. These procedures require generating large numbers of rollouts per candidate problem before training begins, but the cost of these rollouts is not included in any training budget calculation. The survey acknowledges this as a procedural detail, not a cost to be amortized.

The consequence. In realistic deployment, the total cost of building a reasoning model includes (1) estimating which data to train on, (2) training the model, and (3) inference. The survey's reported efficiency comparisons—such as LIMO achieving competitive performance with only 817 examples, or DeepScaleR exceeding o1-preview with a 1.5B model—count only steps (2) and (3). Step (1) can dominate: generating 2048 rollouts per candidate problem to estimate difficulty across a pool of millions of potential training examples is extraordinarily expensive. If a project screens 1 million math problems (as Light-R1 does) by generating even a small number of rollouts per problem, the difficulty-estimation compute can exceed the SFT or RL training compute itself.

The practical implication is that the survey's implicit endorsement of difficulty-filtered data curation ("We observe a consistent trend: datasets... are carefully designed to include data where models are likely to make mistakes," Section 3.3.1) comes with an undisclosed price tag. A practitioner following the survey's guidance would need to budget for difficulty estimation as a substantial line item, potentially making the "efficient" small-dataset approach (LIMO, Light-R1 stage 2) less efficient than it appears when total end-to-end compute is accounted for.

What evidence exists in the paper. The survey provides no quantitative analysis of difficulty estimation cost. It does not report how many rollouts were generated per candidate problem in any of the surveyed studies. It does not compare total FLOPs (difficulty estimation + training + evaluation) across projects. The "data curation" descriptions in Section 2.1 and Section 3.1 describe filtering procedures qualitatively—"samples with a moderate pass rate are kept," "samples with zero pass rate are filtered out"—without reporting the computational budget required to estimate those pass rates.

Mitigation status. Not addressed. The survey does not flag difficulty estimation cost as a limitation, does not propose cheaper alternatives (e.g., training a lightweight difficulty predictor from question text alone), and does not suggest that future work should amortize this cost in comparative analyses. The original DeepSeek-R1 paper is similarly silent on this point. This is a significant omission given that difficulty estimation is the linchpin of the data curation strategies the survey identifies as best practice.

Limitation 2: Single Benchmark and Single Model Family Dominance

The assumption or constraint. The survey's quantitative evidence—Tables 2 and 5, which contain every head-to-head performance comparison—draws almost exclusively from math reasoning benchmarks (AIME24, AIME25, MATH500) evaluated on the Qwen2.5 model family (at 1.5B, 7B, 14B, 32B, and 72B scales). Section 3.4 ("RLVR on Other Tasks") and Section 4.2 ("Generalizability") discuss applications to code, logical reasoning, chemistry, machine translation, and creative writing, but these discussions are qualitative and lack standardized comparative metrics. The survey's strongest claims—that data quality dominates data quantity, that RLVR can surpass distillation, that algorithm variants are theoretically equivalent—are supported entirely by math reasoning results on Qwen2.5-derived models.

The survey does not claim universality; it is transparent that the replication landscape is concentrated. But this concentration is not analyzed as a limitation—it is presented as a description of the state of the field, without discussing whether findings from MATH/AIME on Qwen2.5 would transfer to, say, code generation on StarCoder or scientific reasoning on Llama-based models.

The consequence. Two distinct failure modes arise from this concentration:

  1. Domain specificity of difficulty calibration: MATH and AIME problems have unique, verifiable, numerical or symbolic answers. This enables the entire RLVR paradigm—rule-based accuracy rewards, pass-rate-based difficulty filtering, integer-answer simplification (DAPO's approach). Code problems share some of these properties (unit tests provide deterministic verification), but the survey's Section 3.4 documents that code-specific RLVR projects (DeepCoder, KodCode, Code-r1) use fundamentally different reward designs—execution-based verification, test case coverage requirements, multi-language translation of problems. Logical reasoning, creative writing, and open-ended QA lack deterministic verification entirely, requiring either LLM judges (which introduce their own biases) or multi-component heuristic rewards (which create new reward hacking surfaces). The survey's core findings about what makes training effective—difficulty calibration via pass rate, accuracy-only rewards, minimal KL regularization—may not transfer to domains without clean verifiability.

  2. Model family specificity of RL dynamics: The Qwen2.5 family shares architectural choices (attention mechanism, normalization, tokenizer) and pretraining data mixtures. The survey's central claim that "KL regularization is optional" is supported by five independent projects using Qwen2.5-derived models (Open-Reasoner-Zero, DAPO, Dr. GRPO, SRPO, MiMo) plus one ablation on DeepSeek-R1-Distill-Qwen-7B (Skywork-OR1). It is entirely possible that models with different pretraining distributions, different tokenizer vocabularies, or different architectural inductive biases require KL regularization for stable RL training—the surveyed evidence does not speak to this question because no project tests a non-Qwen base model with and without KL in a controlled comparison.

A practitioner using a non-Qwen base model and targeting a non-math domain must therefore extrapolate the survey's findings across both dimensions simultaneously, with no empirical grounding for doing so.

What evidence exists in the paper. The qualitative Sections 3.4 and 4.2 document breadth but not depth. Table 4 shows that most RLVR studies use Qwen2.5-Base, Qwen2.5-Math, or DeepSeek-R1-Distill-Qwen checkpoints. The only non-Qwen entries are MiMo-7B-Base (Xiaomi's proprietary model), Pleias-350M (for poetry, not benchmarked), and unspecified checkpoints used in some logic/code projects. Table 5's performance comparisons are entirely within the Qwen2.5 and DeepSeek-R1-Distill-Qwen ecosystem. The survey does not report a single cross-model-family controlled experiment.

Mitigation status. Not addressed. The survey treats the Qwen2.5 concentration as a contingent fact about the replication landscape rather than a methodological limitation. It does not call for replication on alternative model families, does not discuss how architectural or pretraining differences might interact with RLVR training dynamics, and does not caution readers that findings may not generalize. The discussion of "Generalizability" in Section 4.2 addresses out-of-distribution task generalization (does a model trained on math generalize to code?) but not out-of-model-family generalization (do training recipes developed on Qwen2.5 work on Llama or Gemma?).

Limitation 3: No Standardized Compute Accounting Across Studies

The assumption or constraint. The survey reports performance outcomes (accuracy on AIME24, AIME25, MATH500) across dozens of projects without normalizing for the computational resources consumed. Table 2 compares AM-72B (trained on 1.4M examples, 72B parameters) against LIMO (817 examples, 32B parameters) against Open-R1 (220k examples, 7B parameters)—all on the same benchmark, but with training costs that differ by orders of magnitude. Similarly, Table 5 places VAPO-32B (value-based PPO with a critic model of comparable size to the policy, effectively doubling memory requirements) alongside GPG-1.5B (REINFORCE-based, no critic, no reference model, no clipping) without discussing the per-step or total FLOPs differential.

The survey acknowledges this implicitly in Section 3.3.3, noting that model sizes range from 1.5B to 32B, and in Section 3.3.4, documenting context length ranges from 8k to 32k. But it does not attempt to construct a common-cost basis—e.g., total FLOPs for training, or FLOPs per unit of benchmark improvement—that would enable fair efficiency comparisons.

The consequence. The central trade-off that practitioners care about—how much compute buys how much reasoning capability—cannot be extracted from the survey's tables as presented. A team deciding between training a 7B model with DeepScaleR's multi-stage RL pipeline (progressive context lengthening, pass-rate filtering, GRPO) versus fine-tuning a 32B model with LIMO's 817 carefully curated examples has no basis for cost-benefit comparison. The survey reports that LIMO-32B achieves 57.1% on AIME24 and DeepScaleR-1.5B achieves 43.1%, but the compute to train LIMO-32B (SFT on 817 examples at 32B scale, plus the cost of curating those 817 examples from "tens of millions" of candidates) is not comparable to the compute to train DeepScaleR-1.5B (RL on 40k examples with progressive context length increase and multiple training stages) in any straightforward way.

This also creates a potential efficiency illusion. The survey's most striking results—MiMo-7B-RL-Zero at 56.4% outperforming DeepSeek-R1-Distill-Qwen-7B at 55.5%, DeepScaleR-1.5B at 43.1% surpassing the distilled 1.5B baseline at 28.9%—are presented as evidence that RLVR can exceed distillation. But RLVR training is expensive: each training step requires generating G rollouts (typically 8–64) of length up to 24k–32k tokens, performing forward and backward passes on all rollouts, and (for PPO-based methods) training a critic model concurrently. Distillation, by contrast, is a single SFT pass over pre-generated traces. The survey does not report training wall-clock time, GPU-hours, or total FLOPs for any project, making it impossible to assess whether the performance gains from RLVR justify the additional compute.

What evidence exists in the paper. The survey reports some training configuration details—context lengths (Section 3.3.4), group sizes G (implicit in algorithm descriptions), dataset sizes—but never translates these into compute units. The RL algorithm derivations in Section 3.2.1 discuss the theoretical cost of different approaches (e.g., PPO requires a critic model, GRPO does not) but do not quantify this cost for the specific training runs reported. The discussion of CPPO (Section 3.2.1) as a method to "accelerate the training process by skipping the forward pass and gradient backpropagation on rollout samples with low advantages" is the only explicit compute-efficiency discussion, and it is specific to one algorithm variant.

Mitigation status. Not addressed. The survey does not call for standardized compute reporting, does not propose FLOPs-matched comparisons across studies, and does not note the absence of cost normalization as a limitation. This is a missed opportunity: a survey that positions itself as a guide for practitioners should at minimum flag that the performance numbers it presents are not cost-adjusted, and that "best" depends on budget.

Limitation 4: Training-Test Contamination Is Pervasive and Undetected

The assumption or constraint. Every benchmark number in Tables 2 and 5 assumes that the model was not trained on problems that appear in (or closely resemble) the evaluation set. The survey explicitly documents that this assumption is violated for an unknown fraction of the replication literature. In Section 2.1.3, the survey states:

"only the technical reports of Light-R1 and LIMO explicitly mention conducting proper data decontamination against popular reasoning benchmarks, e.g., AIME24/25, MATH500, and GPQA Diamond (Rein et al., 2024), during dataset curation. Notably, Wen et al. (2025a) point out that MATH500 is partially compromised across several open-source datasets, including OpenThoughts, Open-R1, Bespoke Stratos, and others."

This is not a hypothetical concern—it is a documented fact that major SFT datasets contain evaluation-set problems. The survey's Figure 3 (cross-referencing dataset sources) partially illuminates the problem by showing which datasets derive from common sources, but overlap with a source dataset (e.g., NuminaMath) does not guarantee overlap with evaluation benchmarks—and the absence of documented decontamination does not guarantee contamination. The fundamental problem is uncertainty: for most projects in Tables 2 and 5, we simply do not know whether their reported performance reflects genuine reasoning capability or memorization of benchmark solutions encountered during training.

The consequence. This casts a shadow over every comparative claim in the survey. When Table 2 shows AM-72B at 76.5% AIME24 versus Light-R1-32B at 73.0%, the difference could reflect genuine capability differences, or it could reflect the fact that Light-R1 decontaminated its training data (removing AIME24 problems) while AM did not report doing so. When Table 5 shows Skywork-OR1-32B at 79.7% AIME24—substantially exceeding the DeepSeek-R1-Distill-Qwen-32B baseline of 72.6%—the improvement might be real, or it might be inflated by training on problems that overlap with AIME24. The survey cannot resolve these ambiguities because it can only report what each project claimed, not independently verify decontamination.

More subtly, the RLVR training paradigm introduces a dynamic contamination risk that SFT alone does not. In SFT, contamination is a property of the training data—if a benchmark problem appears in the training set with its solution, the model can memorize it. In RLVR, the model generates its own solutions during training and receives accuracy rewards. If a benchmark problem appears in the RL training set, the model gets to practice solving it repeatedly with feedback, potentially learning problem-specific shortcuts that inflate benchmark performance without developing general reasoning capability. The survey does not discuss this RL-specific contamination dynamic, which may be more pernicious than SFT memorization because the RL process actively reinforces whatever patterns produce correct answers on the contaminated problems.

What evidence exists in the paper. Direct evidence of contamination is reported for MATH500 across "several open-source datasets" (Section 2.1.3). The survey does not report contamination rates for AIME24 or AIME25. It does not audit any specific dataset for benchmark overlap. It does not discuss whether RL training datasets (Table 3) were decontaminated—the decontamination discussion is specific to SFT datasets (Section 2.1.3), with no parallel analysis for RL datasets in Section 3.1.

Mitigation status. Partially addressed through documentation. The survey performs a valuable service by flagging the decontamination gap—this is one of its most important critical contributions. However, it does not attempt to quantify the magnitude of the problem, does not adjust its comparative analyses to account for it (e.g., by giving more weight to results from decontamination-verified projects), and does not recommend specific decontamination protocols for future work. The survey's tables present all results at face value with a footnote noting the decontamination concern; a reader skimming the tables without reading Section 2.1.3 would miss the caveat entirely.

Limitation 5: The "Hard Problem" Regime Remains Inaccessible

The assumption or constraint. Both the SFT and RLVR paradigms documented in the survey operate under a fundamental capability bound: the model must already possess enough prior knowledge to produce correct reasoning traces at some non-trivial rate. SFT requires high-quality reasoning traces from a teacher model; if the teacher cannot solve a problem class, no amount of SFT data curation will teach the student to solve it. RLVR requires the base model to occasionally produce correct answers through exploration; if the model's pass@1 on a problem class is exactly zero across all reasonable sampling budgets, RL training provides no positive reward signal and learning cannot begin.

The survey documents this boundary in multiple places. Section 3.3.1 notes that DeepScaleR "revealed that samples with... a zero pass rate are often unverifiable or contain errors, therefore, both should be filtered out"—problems the model cannot solve are removed from training, not tackled. Section 2.1.1 describes Light-R1 filtering for problems where DeepScaleR-1.5B-Preview's pass rate is below a threshold but not zero—problems that are completely unsolvable are excluded. The difficulty calibration principle the survey identifies as best practice is, in effect, avoiding hard problems entirely.

The consequence. This creates a sharp capability ceiling that no amount of algorithmic refinement—better data curation, better RL stabilization, better reward design—can breach. The survey's entire framework (SFT + RLVR + difficulty calibration) optimizes performance on problems within the base model's latent capability range, but provides no mechanism for expanding that range. A model that cannot solve competition-level geometry problems after pretraining will not learn to solve them through RLVR, because it will never generate a correct geometry solution to receive positive reward. A student model fine-tuned on DeepSeek-R1 traces will not learn to solve problems that DeepSeek-R1 itself cannot solve.

This limitation is structurally analogous to the "hard problems remain essentially unsolved" finding in the compute-optimal test-time scaling literature, but it operates at the training level rather than the inference level. The survey's replication studies all demonstrate that training compute amplifies existing capability but does not create it from nothing. The dramatic improvements reported—DeepScaleR-1.5B improving from 28.9% to 43.1% on AIME24, MiMo-7B from near-zero to 56.4%—represent the extraction of latent capability that was already present in the base model's pretraining, not the acquisition of fundamentally new mathematical knowledge.

For practitioners, this means the survey's guidance applies only to problems within the base model's reach. If the target application involves genuinely novel reasoning (e.g., proving unproven theorems, solving competition problems at the IMO level, reasoning in domains where no strong teacher model exists), the SFT+RLVR paradigm documented here is insufficient. Continued investment in pretraining—larger models, more diverse pretraining data, better data mixtures—remains necessary for expanding the capability frontier, even if it is less efficient than post-training for exploiting existing capability.

What evidence exists in the paper. The survey does not present quantitative data on this limitation—no study reports "RLVR improvement as a function of base model pass@1" or "SFT improvement on problems the teacher model cannot solve." The evidence is indirect: the consistent practice of filtering out zero-pass-rate problems (documented across Sections 2.1, 3.1, and 3.3.1) implies that such problems are recognized as useless for training, but no project explicitly tests whether they could become useful with different methods. The survey's discussion of "hard" problems is entirely in terms of difficulty relative to the current model (maintaining a moderate pass rate), not absolute difficulty (problems beyond any current model's capability).

Mitigation status. Not addressed as a limitation. The survey treats the capability boundary as a practical constraint to be worked around (via difficulty filtering) rather than as a fundamental limitation of the paradigm. It does not discuss whether alternative training approaches—process reward models with human feedback, formal verification-guided search, interleaved pretraining and RL—might expand the capability frontier, and it does not flag the reliance on teacher-model capability as a ceiling for SFT-based approaches. The discussion of "exploration beyond supervision" (Section 3.4) and "generalizability" (Section 4.2) gestures toward the possibility of models exceeding their teachers, but the documented cases (sorting algorithm optimization, multilingual poetry generation) are narrow and do not address the core question of whether RLVR can teach a model to solve problems that no current model can solve.

Limitation 6: No Systematic Latency, Memory, or Hardware Accounting

The assumption or constraint. The survey reports performance in terms of accuracy on benchmarks. It does not report, compare, or analyze the systems-level costs of different approaches: training wall-clock time, GPU memory requirements, inference latency, or hardware accessibility. These factors are critical for practitioners deciding between alternatives because they determine whether an approach is feasible under real-world constraints—not just whether it is better in the abstract.

The algorithmic diversity documented in Section 3.2.1 implies substantial systems-level diversity that goes unremarked. PPO-based methods (VC-PPO, VAPO, Open-Reasoner-Zero) require maintaining a critic model of comparable size to the policy, effectively doubling GPU memory requirements during training. GRPO-based methods eliminate the critic but require maintaining a frozen reference model if KL regularization is used (another full model copy). REINFORCE-based methods (GPG, REINFORCE-Kimi) eliminate both critic and reference model, requiring only the policy model in memory. CPPO further reduces computation by pruning low-advantage rollouts. Yet the survey treats these as algorithm design choices—discussing their theoretical properties and stabilization effects—without translating them into concrete resource requirements.

Similarly, SFT approaches at different scales and dataset sizes have vastly different hardware footprints. Training a 72B model on 1.4M examples (AM) requires different hardware than training a 7B model on 817 examples (LIMO), but the survey only reports the performance outcomes, not the accessibility implications.

The consequence. The survey's implicit guidance—that careful data curation, difficulty calibration, and algorithm stabilization are the keys to reproducing DeepSeek-R1—is incomplete without addressing the resource question. A small team with limited GPU access may not be able to run PPO with a critic on a 32B model, regardless of how well it works in published experiments. They need to know: which approaches work at my scale? The survey provides some signal on this (DeepScaleR-1.5B shows that small models can benefit from RLVR; LIMO shows that small datasets can work), but it does not organize findings by resource tier.

More specifically, the survey's discussion of KL regularization as "optional" (Section 3.3.6) has a latency dimension that goes unmentioned. Removing KL during training eliminates the need for a reference model in memory, which for large models can mean the difference between fitting on a single node and requiring model parallelism. This is a practical consequence of the KL finding that the survey could highlight but does not.

The inference-time cost of different approaches also goes unanalyzed. Models trained with SFT on long CoT traces (Open-R1, Light-R1) tend to produce long reasoning chains at inference time. RLVR-trained models may produce even longer chains as they learn to explore. The survey does not report average inference token counts, generation times, or the accuracy-per-token efficiency of different models—metrics that matter enormously for deployment cost.

What evidence exists in the paper. The survey reports context lengths used during training (Section 3.3.4: 8k to 32k) and occasionally mentions memory considerations in algorithm descriptions (GRPO "discards the critic model... to improve efficiency and memory consumption," Section 3.2.1). But these are qualitative mentions, not quantitative comparisons. No table reports GPU-hours, peak memory usage, training throughput, or inference latency for any project.

Mitigation status. Not addressed. The survey does not flag the absence of systems-level analysis as a limitation, does not call for standardized resource reporting in future replication studies, and does not organize its comparative analysis along resource-tiers. This is a significant gap for a survey that positions itself as a practical guide for "researchers and developers" (Abstract) aiming to build their own reasoning models.

7. Implications and Future Directions

How This Work Changes the Landscape

This survey fundamentally reorients the conversation around reasoning language model training from an algorithmic innovation race toward a systems-level design-space mapping. Before this survey, the literature presented DeepSeek-R1 reproduction as a series of independent discovery efforts—each project proposing its own algorithm variant, dataset recipe, or reward design, with no framework for understanding how these choices relate. The survey's central intellectual move is to demonstrate that the apparent diversity of approaches collapses into a small set of design dimensions, and that success along any one of those dimensions is governed by a shared underlying principle: difficulty calibration relative to the model's current capability.

This is a reframing, not a paradigm shift. The survey does not introduce new training methods, new benchmarks, or new theoretical results. What it introduces is a coordinate system for reasoning about reasoning model training. A practitioner who reads the survey emerges not with a single recommended recipe but with the ability to navigate the design space: to understand that data curation quality matters more than dataset size because both SFT and RLVR are governed by the same difficulty-calibration constraint; that KL regularization is optional because outcome-reward-based training does not present the same reward-hacking surface as learned-reward RLHF; that algorithm choice matters less than training stabilization because all variants are built from the same policy gradient foundation; and that decontamination is the most pervasive uncontrolled variable in the literature because only two projects have properly addressed it.

The survey resolves several specific tensions that pervaded earlier work:

  • "GRPO vs. PPO vs. REINFORCE": The unified derivation in Section 3.2.1 shows that these are not competing paradigms but points in a continuous space of advantage estimation, clipping, and KL placement choices. The clustering of diverse algorithms at similar performance levels on the same base model (DAPO at 50.0%, VC-PPO at 48.8%, Open-Reasoner-Zero at 48.1% on AIME24 from Qwen2.5-32B-Base) empirically validates the theoretical claim that the specific algorithm matters less than the stabilization it provides.

  • "Large datasets vs. small, high-quality datasets": The survey's side-by-side presentation of LIMO (817 examples, 57.1% AIME24) against Synthetic-1 (894k examples, weaker performance at comparable scale) resolves the apparent contradiction by revealing that quality and difficulty calibration—not size—are the active ingredients. The tension was an artifact of comparing datasets that differed on unobserved dimensions.

  • "KL regularization is essential vs. optional": The convergent evidence from five independent projects that omit KL without degradation, plus Skywork-OR1's explicit ablation showing KL limits exploration, resolves this debate for outcome-reward-based reasoning training. The earlier consensus (inherited from RLHF) that KL is necessary was based on a different reward structure (learned reward models) that does not apply here.

  • "RL cannot surpass distillation for small models": DeepSeek-R1's original claim that "smaller models may fail to match the performance of distillation models through RL training" is contradicted by MiMo-7B-RL-Zero (56.4% vs. distilled baseline 55.5% on AIME24) and DeepScaleR-1.5B (43.1% vs. distilled baseline 28.9%). The survey's documentation of these counterexamples—and the careful training pipelines that enabled them—shows that the original claim was contingent on specific (undisclosed) training choices, not a fundamental limitation.

The research directions that become more attractive after this survey are: (1) cheap, model-specific difficulty estimation, since difficulty calibration is identified as the unifying principle but current methods are prohibitively expensive; (2) systematic component-level ablation studies, since the survey reveals that no existing study has crossed project boundaries to compare, say, DAPO's token-level loss against Dr. GRPO's bias corrections on a common substrate; (3) contamination auditing of existing benchmark results, since the survey documents that most projects did not decontaminate their training data; and (4) expansion beyond math to domains where deterministic verification is unavailable, since the survey's strongest findings are all from MATH/AIME on Qwen2.5 models.

The directions that become relatively less attractive are: (1) proposing yet another GRPO variant without demonstrating that it addresses a replication failure mode not already covered by DAPO, Dr. GRPO, or VAPO; (2) scaling dataset size without difficulty calibration, since the survey shows this is inefficient; and (3) reporting benchmark results without decontamination protocols, since the survey has established that this is a baseline methodological requirement.

Follow-Up Research This Work Enables

A systematic, multi-project contamination audit of AIME and MATH500 training data. The survey documents that only two projects (Light-R1, LIMO) performed decontamination, and that MATH500 is "partially compromised across several open-source datasets." This creates a significant validity concern for every number in Tables 2 and 5. A strong follow-up would: (a) collect the training datasets from all major replication projects; (b) compute exact n-gram and embedding-similarity overlap with AIME24, AIME25, MATH500, and GPQA Diamond test sets; and (c) report contamination-adjusted performance estimates. For RLVR projects, this audit would need to be dynamic—assessing not just whether benchmark problems appear in the training query pool, but whether the model's RL training process (which generates and reinforces its own solutions) creates an additional contamination pathway that static decontamination cannot address. This work is newly tractable because the survey has catalogued exactly which datasets were used by which projects (Tables 1, 3) and identified the cross-referencing patterns (Figure 3) that make overlap likely.

A controlled ablation of difficulty estimation cost vs. training benefit across the full pipeline. The survey identifies difficulty calibration as the unifying design principle but provides no quantitative analysis of what it costs to estimate difficulty—generating thousands of rollouts per candidate problem, computing pass rates, and filtering accordingly. A rigorous follow-up would: use a single base model (e.g., Qwen2.5-32B-Base), a single training algorithm (e.g., GRPO with group size 8), and a large candidate pool of math problems, then systematically vary the difficulty estimation budget (number of rollouts per problem: 4, 16, 64, 256, 1024, and "no filtering" as baseline). For each budget, measure: (a) wall-clock time and GPU-hours for difficulty estimation; (b) training time and final AIME24 accuracy; and (c) total compute (estimation + training) per unit of benchmark improvement. This would answer the survey's open question about whether difficulty estimation costs are amortizable in practice or dominate the training budget. It would also test whether cheap difficulty proxies—training a lightweight classifier on question text alone to predict pass rates—can substitute for expensive rollout-based estimation. The survey makes this tractable by providing the conceptual framework: we know difficulty matters; we just don't know what it costs.

A model-family transfer study of the "KL is optional" finding. The survey's evidence for KL regularization being unnecessary comes exclusively from Qwen2.5-derived models. Whether this generalizes is unknown. A targeted experiment would: select base models from three different pretraining distributions (e.g., Qwen2.5-7B, Llama-3-8B, Gemma-2-9B), train each with GRPO on the same RL dataset (e.g., DeepScaleR's 40k math problems) with and without KL regularization, and track training dynamics—response length growth, entropy collapse, reward hacking behaviors, and final AIME24 accuracy. The hypothesis (from the survey's synthesis) is that models with stronger pretraining alignment to the instruction-following format may need KL to prevent catastrophic deviation, while models with weaker prior alignment may benefit from removing KL to enable exploration. A result showing that KL necessity depends on base model properties would refine the survey's finding from "KL is optional" to "KL is optional for Qwen2.5-class models, necessary for models with property X." The survey enables this by providing a clear description of what "KL removal" means operationally (Section 3.3.6: no reference model in memory, no KL penalty in the objective) and a list of projects that have successfully done it.

A head-to-head DAPO vs. Dr. GRPO vs. VAPO comparison on identical data and compute. The survey claims that algorithm variants are "not significantly different from the theoretical perspective" and that stabilization engineering dominates. But this claim rests on cross-project comparisons confounded by different datasets, different base models, and different hyperparameter tuning budgets. A definitive test would: fix the base model (e.g., Qwen2.5-32B-Base), the training data (e.g., DAPO-Math-17k, since DAPO uses it), the total training compute budget (matched FLOPs), and the evaluation protocol (AIME24 pass@1 with multiple seeds), then run DAPO, Dr. GRPO, VAPO, vanilla GRPO, and vanilla PPO side-by-side with systematic hyperparameter sweeps for each. Report mean, standard deviation, and training curves. This experiment would either confirm the survey's "algorithms don't matter beyond stabilization" thesis—if all methods converge to similar performance at matched compute—or identify specific failure modes (entropy collapse, reward hacking, training instability) that certain variants handle better than others, refining the thesis. The survey makes this tractable by providing the unified mathematical framework that shows exactly which components differ between variants, enabling a factorial experimental design (KL on/off × clip symmetric/decoupled × loss token/sample × critic present/absent).

Extension of difficulty-calibrated RLVR to non-verifiable domains using LLM judges. The survey's strongest quantitative findings are from math, where ground-truth answers enable deterministic accuracy rewards. A critical follow-up would test whether the same principles—difficulty calibration, minimal reward design, group-relative advantage normalization—transfer to domains where correctness must be evaluated by an LLM judge rather than a rule-based verifier. A concrete experiment: collect a dataset of multi-step reasoning questions in a domain without clean verification (e.g., legal reasoning, medical diagnosis, or multi-hop question answering with free-form answers). Use a strong LLM (e.g., GPT-4o) as a judge to provide binary correctness feedback. Train a model with GRPO using judge-based rewards, with and without difficulty calibration (filtering out prompts where the model's initial pass rate is 0% or 100%). Measure whether difficulty calibration improves training efficiency and final test-set accuracy despite the judge's imperfections, and whether reward hacking against the judge emerges as a failure mode (analogous to PRM over-optimization in the test-time compute literature). This would stress-test the survey's central claim that difficulty calibration is a cross-paradigm principle, not just a math-specific heuristic. The survey enables this by documenting the exact operationalization of difficulty calibration across multiple projects (Light-R1's pass-rate thresholds, DAPO's dynamic sampling constraint, DeepScaleR's zero-pass-rate filtering).

A negative-result study: when does RLVR fail to produce emergent reasoning? The survey documents multiple success cases but also gestures at boundary conditions—models with insufficient prior knowledge, domains without clean verification, training with overly easy or overly hard data. A systematic negative-result study would deliberately stress-test RLVR under conditions hypothesized to prevent emergent reasoning: (a) train on a math benchmark where the base model's pass@1 is exactly zero even with high-temperature sampling (e.g., IMO-level problems for a 1.5B model), measuring whether any amount of RL training can bootstrap learning from zero; (b) train with accuracy rewards alone (no format rewards, no length penalties) on a model that initially produces unstructured outputs, measuring whether the model spontaneously develops formatting without explicit incentive; (c) train on a dataset where all problems have the same difficulty (no calibration), measuring whether the lack of a learning gradient impairs convergence; and (d) train with intentionally noisy rewards (e.g., 10% of accuracy labels flipped), measuring the robustness of the RLVR paradigm to verifier errors. These experiments would establish the failure boundaries that the survey's success cases only imply—crucial information for practitioners deciding whether RLVR is appropriate for their setting. The survey enables this by providing a clear taxonomy of what "success" looks like (Tables 2, 5) and what design choices are believed to matter (difficulty calibration, reward minimalism, algorithm stabilization).

Practical Applications and Downstream Use Cases

Cost-constrained reasoning model development for math education technology. A company building an AI-powered math tutoring system needs a model that can solve competition-level high school math problems (AIME, AMC difficulty) with step-by-step reasoning, but cannot afford the inference costs of a 70B+ proprietary model or the training costs of a full DeepSeek-R1-scale pipeline. The survey's synthesis directly informs their build decision. They would: (1) select Qwen2.5-7B-Instruct as a base model (sufficient prior math capability, per the survey's analysis of base-model effects); (2) curate 1,000–5,000 SFT examples by filtering a large candidate pool through the difficulty-calibration pipeline documented in Section 2.1 (generate rollouts with a cheap proxy model, retain only problems with moderate pass rates, verify traces with Math-Verify); (3) fine-tune with standard SFT (learning rate 1e-5, batch size 128, packing enabled) to produce a model that generates well-structured reasoning traces; and (4) optionally apply one round of DPO (Section 4.1, following Light-R1's curriculum: chosen = DeepSeek-R1 traces, rejected = SFT checkpoint rollouts) for a further performance boost at low additional compute cost. The expected outcome—based on LIMO's 57.1% AIME24 with 817 examples at 32B, scaled down to 7B but with a stronger instruct base—would be a model achieving 25–35% on AIME-level problems, sufficient for generating practice-problem solutions with visible reasoning steps, at a total training cost of a few hundred GPU-hours on commodity hardware. The survey makes this feasible by providing exact training configurations, dataset sources, and verification tools across a range of scales.

On-device reasoning for coding assistants via small-model RLVR. A developer tools company wants to integrate local code reasoning into an IDE plugin—generating explanations for why code fails, suggesting fixes, and reasoning about algorithmic choices—without sending code to cloud APIs (privacy constraint). The model must run on a developer's laptop (requiring ≤7B parameters quantized to 4-bit) but handle LeetCode-medium to LeetCode-hard difficulty problems. The survey's documentation of DeepScaleR-1.5B achieving 43.1% on AIME24 (exceeding o1-preview on that benchmark) demonstrates that a small model with aggressive RLVR training can punch far above its weight class. The build plan: (1) start from DeepSeek-R1-Distill-Qwen-1.5B or Qwen2.5-Coder-1.5B as base; (2) curate a code-specific RL dataset following the verifiability standards documented in Table 3 (problems with ≥5 unit tests, execution-based verification, no multi-file dependencies); (3) train with GPG (the simplest REINFORCE variant, requiring no critic and no reference model—the survey shows GPG-1.5B achieves 33.3% AIME24 on math with this approach) adapted for code rewards (accuracy = all tests pass, with additional reward for solution efficiency if desired); (4) use DeepScaleR's progressive context length increase (8k→16k→24k) to develop longer reasoning chains. The survey's documentation of Skywork-OR1's finding that masking truncated rollouts is unnecessary at 32k context lengths removes one implementation burden. The expected result is a model that can explain and fix common algorithmic bugs locally, with accuracy on LeetCode-medium problems approaching 40–50%—usable as an instant feedback tool even if not fully reliable as an autonomous solver.

Self-improving data generation pipeline for a domain-specific reasoning corpus. A research lab studying scientific reasoning wants to create a large corpus of high-quality reasoning traces for molecular biology problems (e.g., predicting experimental outcomes, designing protocols, interpreting assay results), but no strong teacher model exists for this specific domain—DeepSeek-R1 was trained on general math and code, not biology. The survey's documentation of the SFT→RL→SFT→RL cycle from DeepSeek-R1, combined with the finding that RL drives generalization (Section 4.2), suggests a bootstrapping procedure: (1) use GPT-4o or Claude to generate a small seed set of ~500 biology reasoning traces with expert-verified answers; (2) SFT a Qwen2.5-7B-Instruct model on these traces to establish basic reasoning format and domain terminology; (3) apply RLVR on a larger set of unlabeled biology problems where ground-truth answers are known from experimental databases (enabling accuracy rewards) but reasoning traces are not available—the model generates its own traces during RL and receives binary correctness feedback; (4) use the RL-trained model to generate reasoning traces on a much larger corpus of biology problems, filtering for those where the model's answer matches the known ground truth; (5) SFT a stronger model (e.g., 32B) on this expanded corpus; (6) repeat RLVR on an even larger problem set. The survey enables this by: (a) documenting that RL can improve performance beyond the teacher (MiMo-7B-RL-Zero exceeding distilled baselines), so the seed model does not need to be perfect; (b) providing the recipe for difficulty calibration during RL (filter out problems with 0% or 100% pass rates) to maintain a productive learning signal; and (c) showing that format learning through SFT before RL (the complementarity thesis, Section 4.3) is a reliable pattern. The expected outcome is an iterative capability expansion in a domain where no off-the-shelf reasoning model exists, with each cycle producing better training data for the next.

Safety evaluation framework for reasoning models using overthinking diagnostics. A red-teaming organization needs to evaluate the safety properties of a deployed reasoning model (e.g., an internal fine-tune of DeepSeek-R1-Distill-Qwen-32B used for automated code review). The survey provides two concrete diagnostic concepts—overthinking and the safety tax—that can be operationalized as evaluation protocols. For overthinking: construct a test suite of 100 coding problems where the optimal solution is straightforward (O(n) time, <20 lines of code) but the model might engage in unnecessary reasoning (e.g., exploring multiple algorithms, re-verifying already-correct intermediate results). Measure the ratio of reasoning tokens to solution tokens; a model with a ratio >10:1 on simple problems is exhibiting cost overthinking, which inflates API costs and increases latency without accuracy benefit—exploitable by an adversary sending intentionally trivial-looking problems (the OverThink attack documented in Section 4.3). For the safety tax: evaluate the model on a standard harmfulness benchmark (e.g., HarmBench) under two conditions—(a) standard zero-shot prompting and (b) with an explicit instruction to "think step by step about the safety implications before responding." A substantial increase in unsafe responses under condition (b) indicates that the model's reasoning process, when extended, discovers justifications for violating safety constraints—the safety tax in action. The survey enables these evaluations by providing the conceptual vocabulary and citing specific papers (Chen et al., 2024a for overthinking; Huang et al., 2025b for safety tax) that contain operationalizable metrics.

When to Prefer This Method

The survey does not propose a single method—it is a meta-analysis of many methods. However, it does articulate clear trade-offs among the approaches it surveys, based on cross-study patterns in Tables 1–5 and the synthesis in Section 3.3. These trade-offs can be summarized as a decision framework for practitioners choosing how to build a reasoning model:

  • Prefer pure SFT (data distillation from a strong teacher) when: (a) you have access to a strong teacher model (e.g., DeepSeek-R1, o1, or a larger in-house model) that can generate high-quality reasoning traces for your target domain; (b) your target domain has clean verification (math, code with unit tests) to ensure trace correctness; (c) you are compute-constrained for training—SFT requires a single pass over pre-generated traces, substantially cheaper than RLVR's repeated online rollouts; (d) the base model is an instruct variant, which per the survey's evidence (Table 2, Li et al., 2025d) exhibits higher learning efficiency from SFT than base models; and (e) your primary requirement is strong in-distribution performance on problem types well-represented in the teacher's output distribution. The evidence: LIMO achieves 57.1% AIME24 with 817 examples; Light-R1-32B achieves 73.0% with 76k examples; both approach DeepSeek-R1-Distill-Qwen-32B's 72.6%.

  • Prefer pure RLVR (training from a base model with no SFT) when: (a) no strong teacher model exists for your target domain (the survey documents RLVR generalizing to novel tasks in Section 3.4); (b) you need the model to develop reasoning strategies that go beyond imitating a teacher; (c) you have sufficient compute budget for repeated online rollouts (G=8–64 per prompt per training step) and can implement the stabilization mechanisms the survey identifies as necessary (dynamic sampling, difficulty-calibrated data, appropriate context length scheduling); and (d) you are willing to accept that the model may take longer to converge and may require more hyperparameter tuning than SFT. The evidence: MiMo-7B-RL-Zero achieves 56.4% AIME24 from a base model, exceeding the distilled 7B baseline of 55.5%; DeepScaleR-1.5B achieves 43.1% from a 1.5B model, exceeding o1-preview on that benchmark.

  • Prefer SFT followed by RLVR (the combined pipeline) when: (a) you have access to a teacher model for initial SFT but want to push beyond imitation; (b) the base model needs format training before RL can be effective (the complementarity thesis: "SFT stabilizes format, RL drives generalization," Section 4.2); (c) you observe that pure RLVR from a base model is unstable or plateaus early—the SFT checkpoint provides a stronger starting policy, reducing exploration variance; and (d) you can afford both stages. The evidence: Light-R1-14B-DS (SFT + GRPO) achieves 74.0% AIME24 vs. the distilled baseline of 69.7%; Skywork-OR1-32B (distillation + RL) achieves 79.7% vs. the distilled baseline of 72.6%.

  • Prefer SFT followed by DPO (off-policy preference optimization) when: (a) you are compute-constrained and cannot afford online RLVR with repeated rollouts; (b) you already have an SFT checkpoint that performs reasonably well but needs refinement; (c) you can construct clean preference pairs (chosen = teacher traces with verified correct answers, rejected = SFT checkpoint traces with verified incorrect answers). The evidence: DPO "demands much less computational resources" than PPO/GRPO; "DPO substantially enhances model performance while maintaining high training efficiency" (Zhang et al., 2025a), though "DPO still lags slightly behind PPO in overall effectiveness."

  • Across all approaches, prefer difficulty-calibrated data over large-scale data. The survey's most consistent finding is that filtering training data to maintain a moderate model pass rate (problems where the model makes mistakes but occasionally succeeds) dominates dataset size as a predictor of training efficiency. This applies whether the dataset is for SFT (LIMO's 817 carefully filtered examples vs. Synthetic-1's 894k) or RLVR (DeepScaleR filtering out zero-pass-rate and high-pass-rate problems). The operational rule: before training, generate a modest number of rollouts from your base model or current checkpoint on each candidate training problem, estimate pass rates, and retain only problems in the 10–90% pass-rate range—discard both the trivially easy and the currently impossible.

  • When choosing an RL algorithm, start simple. The survey's theoretical framework and the empirical clustering of algorithm variants suggest: begin with the simplest REINFORCE variant (GPG or REINFORCE-Kimi: group-normalized rewards, no critic, no clipping, token-level loss), and add complexity (critic, KL penalty, importance sampling clipping, dynamic sampling) only if training instability or entropy collapse is observed. The evidence for this minimalism: GPG-1.5B achieves 33.3% AIME24 with no critic, no clipping, and no reference model; REINFORCE-Kimi achieves stable training on 72B models with centralized rewards and K2 KL penalty. Advanced components (critic via VAPO reaching 60.4%, value pretraining via VC-PPO reaching 48.8%) provide further gains but at substantial implementation and memory cost that may not be justified in all settings.