Supervised Fine-Tuning versus Reinforcement Learning: A Study of Post-Training Methods for Large Language Models

🎯 Pitch

LLM post-training has undergone a decisive paradigm shift—hybrid SFT+RL pipelines exploded from just 20% of studies in 2023 to 73.8% in 2024. This survey reveals that SFT and RL are not separate stages but deeply connected under a unified gradient framework, where SFT is simply RL with an implicit trajectory-reproduction reward.

1. Executive Summary

This survey systematically analyzes and unifies two post-training paradigms for LLMs—Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL)—by examining their objectives, algorithmic structures, data requirements, and practical applications across reasoning, mathematical, agentic, and code domains from 2023 to 2025. The paper establishes a unified framework showing that SFT can be viewed as a special case of RL (where an indicator function checking whether the current policy reproduces an offline trajectory acts as a surrogate reward), and documents the rapid shift toward hybrid post-training pipelines that combine SFT and RL—for example, methods like SRFT and UFT that integrate supervised losses on offline demonstrations with RL objectives on online rollouts. Drawing from a benchmark-oriented search across roughly 195,000 arXiv preprints, the survey reports that hybrid "SFT+RL" training grew from 20% of studies in 2023 to 73.8% in 2024 (+269%), that open-weight model usage more than doubled over the same period (12.2% → 25%), and that research volume across all surveyed domains more than doubled between 2023 and 2024, establishing that the dominant paradigm for LLM post-training has shifted decisively away from single-stage approaches toward integrated frameworks that leverage the complementary strengths of both SFT and RL.

2. Context and Motivation

The central gap this survey addresses is the fragmented understanding of LLM post-training methodologies. The field has converged on two dominant paradigms—Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL)—but the vast majority of research, surveys, and practical guidance treats them as separate, sequential stages rather than as interconnected components of a unified optimization landscape. As the paper states in Section 1:

"While recent studies offer valuable insights into LLM post-training, the majority of them typically examine SFT or RL separately... leaving the relationships between these approaches comparatively underexplored."

This fragmentation matters because SFT and RL are not merely two independent tools that happen to be used in sequence. They share a common mathematical structure—both optimize the expected log-probability of desirable outputs, differing primarily in how "desirable" is defined (ground-truth tokens versus reward signals). The paper demonstrates this explicitly in Section 4.1, showing that the SFT gradient can be rewritten as an RL gradient with a surrogate reward function $-\mathbb{I}\{y = y_i\} / \pi_\theta(y|x)$ , meaning that every insight developed for one paradigm has potential relevance to the other. Without a systematic framework for understanding this relationship, researchers are effectively rediscovering solutions in parallel, missing opportunities for cross-pollination, and making suboptimal design choices about when to use which method.

Why This Problem Is Important: Practical and Theoretical Stakes

The question of "SFT vs. RL vs. hybrid" is not merely academic taxonomy—it has direct, high-stakes consequences for how LLMs are developed and deployed.

Practical impact on model development pipelines. Most state-of-the-art LLMs today follow a multi-stage post-training recipe: pre-training → SFT → RLHF or some variant thereof (Ouyang et al., 2022; Bai et al., 2022). Each stage consumes substantial computational resources, human annotation effort, and engineering time. Understanding precisely what each stage contributes and whether they can be combined more efficiently directly affects the cost and timeline of developing capable models. The paper documents that the field is already moving toward hybrid approaches—with hybrid "SFT+RL" training growing from 20% to 73.8% of surveyed studies between 2023 and 2024 (Figure 2)—but this shift is happening largely through empirical trial-and-error rather than principled understanding. A unified framework can accelerate this convergence by identifying which hybrid strategies are likely to succeed and why.

Theoretical significance: resolving seemingly contradictory findings. The field contains apparent contradictions that a unified perspective can resolve. For instance, SFT is known to suffer from distribution shift and compounding errors (Ross and Bagnell, 2010; De Haan et al., 2019)—a problem inherited from behavior cloning in classical RL—yet it remains the standard first stage of post-training because of its stability. RL can generalize better (as demonstrated in recent work like Chu et al., 2025, which the paper cites in Section 7) but suffers from entropy collapse (Cui et al., 2025a), reward hacking (Pan et al., 2024a,b), and training instability. By showing that SFT is a special case of RL under a particular reward parameterization, the paper provides a theoretical basis for understanding why SFT is stable (the implicit reward is sparse and exactly aligned with the demonstration) and why RL can generalize better (the reward signal covers a broader distribution of outputs). This unification transforms what appeared to be conflicting empirical observations into a coherent tradeoff space.

Economic and resource allocation decisions. The survey's trend analysis (Section 5 and Appendix B) reveals that research volume across all surveyed domains more than doubled between 2023 and 2024, with math-related research alone growing from 492 to 1,098 papers (+123%) and projected to reach 2,399 in 2025. This explosive growth means that the field is collectively investing enormous resources in post-training research. A unified framework helps direct these resources more efficiently by clarifying which problems are best addressed through SFT, which through RL, and which through their combination—potentially saving millions of dollars in misallocated compute and human effort.

Where Existing Surveys and Prior Work Fall Short

The paper identifies several specific limitations in how the existing literature addresses SFT and RL:

1. Most surveys treat SFT and RL as separate topics. Prior surveys either focus exclusively on SFT (Parthasarathy et al., 2024; Mao et al., 2025 on LoRA methods), exclusively on RL (Zhang et al., 2025c on RL for reasoning models), or mention both only as sequential stages in a pipeline without analyzing their relationship. The paper explicitly notes this gap:

"Other works focus on specialized dimensions of post-training, such as vision-centric adaptation (Chu et al., 2025), advances in reasoning (Kumar et al., 2025), agentic behaviors (Du et al., 2025a), or scaling strategies (Lai et al., 2025)."

While these specialized surveys are valuable for their respective subdomains, none of them addresses the fundamental question: how do SFT and RL relate to each other, and when should each be used (or combined)?

2. The sequential application of SFT-then-RL is treated as a given, not a choice to be optimized. The standard RLHF pipeline (Ouyang et al., 2022) applies SFT first to inject general knowledge, then RL to optimize a specific reward. But as Section 4 points out, there is growing evidence that this sequential approach may be suboptimal. Several recent works (Wu et al., 2025; Fu et al., 2025b; Liu et al., 2025b) have proposed single-stage or interleaved approaches that outperform the sequential baseline. The field lacks a systematic framework for understanding why these approaches work and when they are appropriate—this survey fills that gap by organizing hybrid methods according to how they integrate the SFT and RL objectives (Table 1).

3. The theoretical connection between SFT and RL is underexploited. While a few pioneering works (Wu et al., 2025; Qin and Springenberg, 2025) have begun formalizing SFT as a special case of RL, this insight has not been systematically developed or connected to practical design choices. The survey consolidates these theoretical threads by deriving the gradient equivalence explicitly (Section 4.1) and showing how SFT's indicator-function reward leads to both its strengths (stability, efficiency with high-quality data) and its weaknesses (distribution shift, poor generalization). It then organizes a wide range of hybrid methods (DFT, iw-SFT, UFT, SRFT, Prefix Sampling, CHORD, etc.) according to whether they modify the SFT objective from an RL perspective ("RL → SFT," e.g., DFT which rescales token losses by policy probabilities) or modify the RL objective from an SFT perspective ("SFT → RL," e.g., NFT which learns from both correct and incorrect outputs under supervision).

4. No systematic empirical mapping of how the field is actually using these methods. Beyond theoretical synthesis, the paper provides something unique among existing surveys: a quantitative, benchmark-oriented analysis of publication trends across 2023–2025 (Section 5 and Appendix B). By searching roughly 195,000 arXiv preprints for mentions of 26 standardized benchmarks across four domains (QA, math, agentic, code), the survey maps not just what methods exist but what methods are actually being used and how this usage is changing. This empirical grounding distinguishes the paper from purely methodological surveys—it doesn't just describe methods, it documents the trajectory of the field and identifies where practice is converging.

How This Paper Positions Itself

The survey positions itself as the first systematic, integrated perspective on SFT and RL as complementary post-training tools, with three explicit contributions stated in Section 1:

Taxonomic organization: systematically summarizing what SFT and RL are, and how they can be extended from both algorithm-centric and data-centric perspectives
Unified theoretical framework: establishing that SFT is a special case of RL and organizing hybrid methods according to their objective-level integration
Empirical trend analysis: documenting the rapid shift toward hybrid training, the expansion of application domains, and the transition from API-based labeling to open-weight model-generated datasets

What makes this positioning novel is its bidirectional perspective. Rather than treating SFT as "the thing you do before RL," the paper examines how RL insights can improve SFT (e.g., importance sampling to mitigate distribution shift, as in iw-SFT; Qin and Springenberg, 2025) and how SFT insights can improve RL (e.g., seeding exploration with demonstrations to stabilize early training, as in Huang et al., 2025's prefix sampling). This symmetrical treatment is absent from prior surveys and reflects a genuine conceptual shift: SFT and RL are not sequential stages but complementary optimization strategies operating on a shared objective landscape.

The survey also distinguishes itself by its practical orientation. Section 6 distills clear takeaways from the theoretical and empirical analysis: "When high-quality expert data are available, SFT is generally preferred as an initial training stage over RL... due to its simpler implementation and greater stability" and "If a strong policy model or prompt is available to generate expert data, incorporating importance sampling on failed queries can mitigate distribution shift by treating them as informative positive samples for RFT/RL." These actionable guidelines are directly useful to practitioners making day-to-day decisions about their post-training pipelines, and they derive from the unified framework rather than from ad hoc empirical findings.

Finally, the paper explicitly acknowledges its scope limitations (Section 8 and the Limitations section): it focuses on text-only tasks for clearer side-by-side comparison, it may miss some recent advances due to the rapid pace of the field, and its benchmark-oriented search method may introduce approximation bias. This transparency about boundaries strengthens the paper's credibility by clarifying exactly what claims it does and does not make.

3. Technical Approach

This is a survey paper that synthesizes and organizes existing research rather than proposing a new method. The core idea is that Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) for LLMs share a deep mathematical connection—SFT can be reformulated as a special case of RL with a specific surrogate reward—and that understanding this relationship enables more principled design of hybrid post-training pipelines that combine the strengths of both approaches.

3.1 Reader Orientation

The "system" this paper builds is a conceptual framework and taxonomy for understanding how SFT and RL relate to each other in LLM post-training, supported by a quantitative empirical analysis of how the research community is actually deploying these methods across application domains. The problem it solves is the fragmented, isolated treatment of SFT and RL in existing literature: by establishing that SFT is mathematically a special case of RL (with an indicator-function reward that fires only when the model exactly reproduces an offline trajectory), the framework explains why SFT is stable but vulnerable to distribution shift, why RL generalizes better but risks entropy collapse and reward hacking, and—most importantly—how insights from each paradigm can transfer to improve the other. The "shape" of the solution is a two-axis organization: one axis organizes methods by whether they are algorithm-centric (modifying training objectives or loss functions) or data-centric (curating or synthesizing training data); the other axis organizes hybrid methods by the direction of influence (RL insights applied to SFT, SFT insights applied to RL, or joint optimization of both objectives simultaneously).

3.2 Big-Picture Architecture (Diagram in Words)

The paper's analytical framework has four major components:

SFT and RL as baseline post-training paradigms — the two foundational methods, each with their canonical objective functions, gradient structures, and failure modes. SFT maximizes the log-likelihood of expert-provided responses given prompts; RL maximizes expected reward from model-generated responses, typically with a KL penalty that prevents the policy from deviating too far from a reference model.
A unified objective formulation — the mathematical derivation showing that the SFT gradient can be rewritten as an RL gradient with a surrogate reward function, establishing that both paradigms optimize the same underlying quantity (expected log-probability of outputs) with different reward parameterizations. This formulation reveals that the standard post-training objective can be expressed as maximizing expected reward minus a KL divergence penalty from a reference policy.
A taxonomy of hybrid methods — organized in Table 1 according to four categories: (a) methods that modify the SFT objective using RL principles like importance sampling or online rollouts; (b) methods that enhance RL with SFT-style supervision through offline demonstrations or expert prefixes; (c) methods that jointly optimize weighted combinations of SFT and RL losses; and (d) methods that interleave SFT and RL steps based on model performance metrics.
An empirical trend analysis framework — a benchmark-oriented search methodology (Appendix B) that maps approximately 195,000 arXiv preprints across four application domains (QA, math, agentic, code) using 26 standardized dataset keywords, classifying papers by their training methodology (SFT-only, RL-only, or hybrid) and data sources (proprietary API, open-weight models, benchmarks, human/web data) to quantify the field's trajectory from 2023 to 2025.

Information flows as follows: the theoretical framework (Component 2) motivates the taxonomy of methods (Component 3) by explaining why certain hybrid approaches work; the taxonomy organizes the vast landscape of methods described in the paper's literature review (Sections 3–4); and the trend analysis (Component 4) provides empirical grounding by showing which methodological categories are gaining versus losing adoption, confirming that the theoretical convergence toward hybrid approaches is reflected in actual research practice.

3.3 Roadmap for the Deep Dive

First, the formal definitions of SFT and RL objectives (Section 2 of the paper), including their gradient structures and the key practical distinction—SFT trains on expert-provided pairs, while RL trains on model-generated outputs scored by a reward function.
Second, the taxonomy of SFT methods from algorithm-centric and data-centric perspectives (Section 3.1), since understanding how researchers extend SFT provides the baseline for later comparison.
Third, the taxonomy of RL methods from algorithm-centric and data-centric perspectives (Section 3.2), with particular attention to the shift from PPO toward critic-free methods like GRPO and the growing emphasis on data efficiency through rollout selection and curriculum learning.
Fourth, the unified objective derivation (Section 4.1)—the mathematical core of the paper showing that SFT is a special case of RL—along with the organization of hybrid methods in Table 1, since this is the paper's primary conceptual contribution.
Fifth, the empirical trend analysis methodology (Section 5 and Appendix B), explaining how the benchmark-oriented search and classification pipeline works and what it reveals about the field's trajectory.

Each stage builds on the previous: the baseline definitions establish vocabulary and notation; the individual taxonomies show what each paradigm achieves in isolation; the unified framework reveals why these isolated achievements are complementary rather than competitive; and the empirical analysis validates that the field is converging toward the hybrid approaches the framework predicts should be effective.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a conceptual synthesis paper whose core idea is that SFT and RL for LLMs share a common optimization structure—both maximize expected log-probability of desirable outputs, differing only in how "desirable" is defined—and that exploiting this connection through hybrid methods yields better post-training outcomes than either approach alone.

Formal Definitions of SFT and RL Objectives

The paper begins (Section 2) by establishing the canonical formulations of both post-training paradigms, which serve as the foundation for all subsequent analysis and hybridization.

Supervised Fine-Tuning (SFT). The SFT objective minimizes the negative log-likelihood of expert-provided responses given prompts:

$\min_\theta \mathbb{E}_{(x,y) \sim \mathcal{D}}\left[-\log \pi_\theta(y \mid x)\right]$

where $\pi_\theta$ is the policy (language) model parameterized by $\theta$ , $\mathcal{D} = \{(x_i, y_i)\}$ is a dataset of prompt-response pairs with $x$ representing the input prompt and $y$ representing the target response, and $-\log \pi_\theta(y \mid x)$ is the negative log-likelihood of the target response under the current model.

What it computes: for each training example, the model outputs a probability distribution over all possible response tokens conditioned on the prompt $x$ . The SFT loss extracts the probability assigned to the ground-truth token $y_t$ at each position $t$ , takes the negative log, and averages over all tokens and all examples. The optimizer adjusts $\theta$ to increase the probability of the expert-provided tokens.

Why this form: maximum-likelihood estimation on expert demonstrations is equivalent to behavior cloning (BC) in classical RL (Pomerleau, 1991). It is the simplest way to transfer expert knowledge to a model because it requires no learned reward function and no online interaction—just a static dataset of $(x, y)$ pairs. The negative log-likelihood is the proper scoring rule for categorical distributions, meaning it is minimized precisely when the model's predicted distribution matches the empirical distribution of the training data. However, the paper explicitly notes that this approach inherits BC's fundamental weakness: "it suffers from distribution shift, which can lead to compounding errors (Ross and Bagnell, 2010; De Haan et al., 2019), and depends heavily on the quality of the demonstrations."

Reinforcement Learning (RL). The RL objective maximizes expected reward from model-generated responses:

$\max_\theta \mathbb{E}_{x \sim \mathcal{D}_x, y \sim \pi_\theta(\cdot \mid x)}\left[r(x, y)\right]$

where $\mathcal{D}_x$ is a distribution over prompts (not full prompt-response pairs—the responses are generated by the model itself), $y \sim \pi_\theta(\cdot \mid x)$ means the response is sampled from the current policy rather than drawn from a fixed dataset, and $r(x, y)$ is a reward function that scores the quality of the generated response. The reward may be manually specified (e.g., a rule-based correctness checker for math problems), learned from human preference data (as in RLHF), or derived from automated feedback (e.g., compiler output for code tasks).

What it computes: the model generates a response $y$ by sampling from its own output distribution given prompt $x$ . The reward function $r$ evaluates this response, producing a scalar score. The expectation is taken over the distribution of prompts and the model's own stochastic generation process. The optimizer adjusts $\theta$ to increase the probability of responses that receive high rewards, which implicitly decreases the probability of low-reward responses.

Why this form: unlike SFT, which only learns from the specific responses in the training dataset, RL explores the space of possible responses through the model's own generations and receives feedback on their quality. This enables two key advantages: (1) the model can discover novel high-quality responses that were never present in the training data, improving generalization; and (2) the reward signal provides more nuanced feedback than binary "correct/incorrect" imitation—for example, a partially correct math solution might receive a partial reward, guiding the model toward the right answer even when it doesn't match any single training trajectory.

The practical distinction between training data sources. The paper draws a sharp line between the data regimes of SFT and RL (Section 2):

"SFT is a supervised learning paradigm that trains on expert-annotated prompt–response pairs, whereas RL is a reward-driven optimization paradigm that learns by updating the model from its own generations."

This distinction is crucial because it determines the cost structure and failure modes of each approach. SFT requires high-quality annotated data upfront—often expensive to collect—but training is stable and predictable. RL requires a reward function (which may itself need to be learned) and involves online sampling from the model, which introduces computational overhead and training instability, but can improve performance beyond what the expert demonstrations alone could teach.

The gradient structures. The paper derives the gradients for both objectives (Section 4.1):

$\nabla_\theta \mathcal{L}_{\text{SFT}} = \mathbb{E}_{(x,y) \sim \mathcal{D}}\left[-\nabla_\theta \log \pi_\theta(y \mid x)\right]$

$\nabla_\theta \mathcal{L}_{\text{RL}} = \mathbb{E}_{x \sim \mathcal{D}_x, y \sim \pi_\theta(\cdot \mid x)}\left[\nabla_\theta \log \pi_\theta(y \mid x) \cdot r(x, y)\right]$

where $\nabla_\theta \log \pi_\theta(y \mid x)$ is the score function—the gradient of the log-probability of response $y$ with respect to model parameters. In the SFT gradient, this term is averaged over the fixed dataset $\mathcal{D}$ . In the RL gradient, this term is weighted by the reward $r(x, y)$ and averaged over the model's own sampling distribution.

What these gradient structures reveal. Both gradients share the same core component: $\nabla_\theta \log \pi_\theta(y \mid x)$ , which tells the optimizer how to change parameters to increase the probability of response $y$ . The difference is in which responses get their probabilities increased and by how much. SFT increases probability uniformly for all expert-provided responses (each $(x,y)$ pair gets equal weight). RL increases probability more for high-reward responses and less (or decreases probability) for low-reward ones. This structural similarity is the foundation for the unified objective that follows.

SFT Methods: Algorithm-Centric and Data-Centric Taxonomies

The paper organizes SFT research (Section 3.1) along two axes: algorithm-centric approaches that modify the training objective or loss function, and data-centric approaches that focus on curating or synthesizing high-quality training data. This taxonomy is not merely descriptive—it identifies the key levers researchers use to address SFT's fundamental weaknesses (distribution shift and dependence on demonstration quality).

Algorithm-centric SFT methods modify how the SFT loss is computed or what it optimizes for, without necessarily changing the training data:

Entropic Distribution Matching (GEM) (Li et al., 2024c): Reformulates SFT as a distribution-matching problem with entropy regularization. Rather than simply maximizing the likelihood of the training responses, GEM encourages the model's output distribution to match the empirical distribution of the training data while preserving output diversity through an entropy bonus. This addresses a known SFT failure mode: standard maximum-likelihood training can collapse the model's output distribution to a narrow peak around the training responses, reducing the model's ability to generate diverse but still-correct alternatives.
Token Cleaning (Pang et al., 2025): Estimates the contribution of each token to model updates and removes uninformative tokens from the loss computation. The insight is that not all tokens in a training response are equally useful—some tokens (e.g., function words, punctuation) contribute little to learning the task, while others (e.g., key reasoning steps, numerical values) are critical. By filtering out low-contribution tokens, Token Cleaning reduces noise in the supervision signal and allows the model to focus its capacity on the informative parts of each response.
One-Token Rollout (Ming et al., 2025): Treats each token prediction as a one-step trajectory and uses the ground-truth token as a reward signal, creating a policy-gradient-inspired variant of SFT. Specifically, instead of minimizing the negative log-likelihood directly, it computes $\nabla_\theta \log \pi_\theta(y_t \mid x, y_{<t}) \cdot \mathbb{I}\{y_t = y_t^*\}$ , where $\mathbb{I}\{y_t = y_t^*\}$ is 1 if the model's sampled token matches the ground truth and 0 otherwise. This introduces on-policy learning signals—the model learns from the tokens it actually generates, not just from the tokens in the training data—without the complexity of full RL (no value function, no advantage estimation, no KL penalty).

Data-centric SFT methods focus on what data to train on and how to construct it, without necessarily changing the loss function:

LIMA (Zhou et al., 2023): Demonstrates that fine-tuning on only 1,000 high-quality and diverse instruction-response pairs can achieve alignment performance comparable to much larger models. The key insight is that most of a model's knowledge comes from pretraining, and SFT primarily serves to surface this knowledge in the desired format—so data quality matters far more than data quantity. This finding motivates the entire data-centric research direction: if 1,000 carefully chosen examples suffice, then significant resources should be invested in data curation rather than data collection.
FisherSFT (Deb et al., 2025b): Selects training examples that maximize information gain, measured through the Fisher information matrix. For each candidate example, FisherSFT estimates how much the model's parameters would change if trained on that example, and selects the subset that collectively provides the most information. This achieves efficient learning with limited data by ensuring that every training example teaches the model something new, rather than reinforcing patterns it already knows.
Data Mixing Optimization (Li et al., 2025b): Learns domain-specific weights for mixing training data from different sources to minimize validation loss. If the training data comes from multiple domains (e.g., math problems, code generation, general QA), naively mixing them uniformly may be suboptimal—some domains might benefit from more training examples than others. This method automatically determines the optimal mixing ratio, improving generalization with minimal tuning cost.
Context-Driven Data Generation (Quan, 2025): Automatically generates instruction-response pairs from context documents, enriching SFT data without heavy human annotation. Rather than requiring humans to write prompts and responses, this method extracts instructional material from existing text corpora (e.g., documentation, tutorials) and converts it into $(x, y)$ pairs, enabling scalable data synthesis.
Condor (Cao et al., 2025a): Integrates knowledge-guided synthesis and iterative refinement to produce high-fidelity alignment data. The method first uses a knowledge base to guide the generation of initial instruction-response pairs, then iteratively refines these pairs using quality filtering and correction, producing data that is both diverse and accurate.

Design choice: why organize by algorithm-centric vs. data-centric? This taxonomy reflects a fundamental tension in SFT: the loss function itself (maximum likelihood) is theoretically well-motivated, but its practical performance depends heavily on what data it is applied to. The algorithm-centric methods address the observation that maximum-likelihood can overfit, collapse diversity, or waste capacity on uninformative tokens—problems that arise even with perfect data. The data-centric methods address the observation that high-quality demonstrations are expensive and scarce, and that the distribution of training data strongly influences model behavior—problems that arise even with a perfect loss function. By separating these concerns, the taxonomy helps practitioners diagnose which problem they are facing and select the appropriate intervention.

RL Methods: Algorithm-Centric and Data-Centric Taxonomies

The paper organizes RL research (Section 3.2) along the same two axes, but the content differs substantially because RL introduces additional challenges around training stability, reward quality, and exploration efficiency.

Algorithm-centric RL methods focus on improving the optimization procedure itself:

Policy optimization algorithms beyond PPO. The paper notes that while PPO (Schulman et al., 2017) with a learned value critic has been the dominant approach, recent work increasingly favors critic-free methods. GRPO (Shao et al., 2024) replaces the value network with group-relative normalized advantages: for each prompt, the model generates a group of responses, computes their rewards, and normalizes the advantages within that group rather than relying on a separately trained value function. This simplifies training by eliminating the value network (which can be unstable and requires careful tuning) while preserving the variance reduction benefits of advantage normalization. REINFORCE-style methods (Li et al., 2023; Ahmadian et al., 2024; Hu, 2025) go even further, using direct policy gradient updates without clipping, value functions, or group normalization—essentially returning to the simplest possible RL algorithm and showing that, with appropriate regularization, it can match or exceed PPO's performance on LLM alignment tasks.
Entropy regularization for preventing mode collapse. The paper highlights that entropy collapse is a central challenge in RL for LLMs: as the policy becomes more confident, it generates less diverse outputs, which reduces exploration and can cause training to stall at a suboptimal point. Several methods address this: Cheng et al. (2025) and Chen et al. (2025c) incorporate entropy information into advantage estimation, effectively rewarding the model for maintaining output diversity. He et al. (2025a) and Shrivastava et al. (2025) introduce weighted token entropy, which applies stronger regularization to tokens where entropy is dropping fastest. Cui et al. (2025a) takes a more targeted approach, identifying the covariance between an action's probability and its advantage as the key entropy "driver" and proposing covariance-based KL clipping to selectively constrain tokens with exceptionally high covariance—precisely those tokens that would otherwise dominate the policy update and collapse diversity.
Stability improvements through regularized objectives. The standard RL objective $\mathbb{E}[r(x,y)]$ provides no explicit constraint on how far the policy can move from its starting point. Without regularization, RL can produce policies that achieve high reward but have diverged so far from the base model that they lose general capabilities or produce nonsensical outputs. The paper notes that most modern RL methods incorporate some form of KL regularization—penalizing the divergence between the current policy $\pi_\theta$ and a reference policy (usually the SFT checkpoint)—but differ in how they implement it. Some apply it as a separate loss term, others as a constraint on policy updates (as in PPO's clipping), and others through reward shaping (subtracting a KL penalty from the raw reward).

Data-centric RL methods focus on what data the RL process uses for training:

Rollout selection (Zhang et al., 2025a; Xu et al., 2025c): Rather than training on all model-generated responses, these methods selectively train on a subset of informative rollouts—for example, those with high variance (indicating the model is uncertain and could learn from the outcome) or those that are diverse (covering different solution strategies). This is data-efficient because low-variance rollouts (where the model is consistently correct or consistently wrong) provide little learning signal: if the model always gets it right, there is nothing to improve; if it always gets it wrong, the failure mode is likely systematic and individual examples are redundant.
Prompt selection (Zheng et al., 2025; Qu et al., 2025): Selects which prompts to generate rollouts for before generation, reducing computational cost. The insight is that not all prompts benefit equally from RL—some prompts are too easy (the model already produces correct answers consistently), some are too hard (the model cannot produce correct answers even with optimization), and only a middle range provides useful learning signal. By identifying this middle range in advance and only running RL on those prompts, computational cost is reduced while maintaining (or improving) final performance.
Curriculum learning (Zhang et al., 2025d; Yao et al., 2025): Dynamically selects prompts of intermediate difficulty to maximize the learning signal. Rather than sampling prompts uniformly, curriculum methods estimate each prompt's difficulty (e.g., by the model's current success rate on it) and select prompts that are challenging but not impossible—the "zone of proximal development" where learning is most efficient. As the model improves, the set of prompts at this optimal difficulty level shifts, so the curriculum adapts.
Distribution-level adaptation (Chen et al., 2025d; Wang et al., 2025a): Prioritizes tasks where the model exhibits the greatest advantage (the gap between current performance and potential performance) or lowest visitation (tasks the model rarely encounters). This ensures that RL allocates training compute to the tasks that will benefit most, rather than spending resources on tasks where the model is already optimal or where improvement is impossible.

Design choice: why does RL need both algorithm-centric and data-centric innovations? RL for LLMs faces a unique challenge: the action space (all possible token sequences) is astronomically large, the reward signal is often sparse or noisy (only the final answer is scored, not intermediate steps), and the model must maintain coherence and general capabilities while optimizing for a specific reward. Algorithm-centric methods address the optimization challenge—how to navigate this vast space without collapsing or diverging. Data-centric methods address the efficiency challenge—how to focus the expensive process of generating and scoring rollouts on the examples that provide the most learning signal. Both are necessary because a perfect optimizer applied to noisy or redundant data will still produce poor results, and perfect data fed to an unstable optimizer will fail to converge.

The Unified Objective: SFT as a Special Case of RL

This is the mathematical core of the paper (Section 4.1). The derivation proceeds in three steps: first, writing the SFT gradient in a form that reveals its structural similarity to an RL gradient; second, identifying the implicit reward function that SFT optimizes; third, showing that both SFT and RL can be expressed as instances of a common regularized objective.

Step 1: Rewriting the SFT gradient as a policy gradient. The paper begins with the standard SFT gradient:

$\nabla_\theta \mathcal{L}_{\text{SFT}} = \mathbb{E}_{(x_i, y_i) \sim \mathcal{D}}\left[-\nabla_\theta \log \pi_\theta(y_i \mid x_i)\right]$

and rewrites it by expanding the expectation over $x$ and introducing a sum over the model's own output distribution:

$\nabla_\theta \mathcal{L}_{\text{SFT}} = \mathbb{E}_{x_i \sim \mathcal{D}_x}\left[\mathbb{E}_{y \sim \pi_\theta(\cdot \mid x_i)}\left[-\frac{\mathbb{I}\{y = y_i\}}{\pi_\theta(y \mid x_i)} \nabla_\theta \log \pi_\theta(y \mid x_i)\right]\right]$

where $\mathbb{I}\{y = y_i\}$ is the indicator function that equals 1 when the sampled output $y$ exactly matches the ground-truth response $y_i$ from the SFT dataset, and 0 otherwise. The denominator $\pi_\theta(y \mid x_i)$ is the probability the current policy assigns to that exact response.

What this rewriting achieves. The original SFT gradient only references $\nabla_\theta \log \pi_\theta(y_i \mid x_i)$ —the gradient of the log-probability of the ground-truth response. The rewritten version references $\nabla_\theta \log \pi_\theta(y \mid x_i)$ for any response $y$ that the model could generate, weighted by the importance sampling ratio $-\mathbb{I}\{y = y_i\} / \pi_\theta(y \mid x_i)$ . When $y \neq y_i$ , the indicator is zero and the term contributes nothing. When $y = y_i$ , the weight is $-1 / \pi_\theta(y_i \mid x_i)$ , which increases the effective learning rate for responses the model currently assigns low probability to—exactly the behavior of importance sampling.

Step 2: Identifying the implicit reward function. Comparing the rewritten SFT gradient to the RL gradient $\mathbb{E}[\nabla_\theta \log \pi_\theta(y \mid x) \cdot r(x, y)]$ , the paper identifies that the quantity $-\mathbb{I}\{y = y_i\} / \pi_\theta(y \mid x_i)$ acts as a surrogate reward. This surrogate reward has a specific structure: it is zero for all outputs except the ground-truth response, for which it is negative (since the SFT objective minimizes negative log-likelihood, the "reward" in an RL framing is negative to encourage probability increase). The magnitude of this reward is inversely proportional to the model's current probability of generating the ground-truth response: responses the model currently considers unlikely receive a larger magnitude reward, driving faster probability increase.

Why this is a useful perspective. By recognizing that SFT implicitly uses this particular reward function, the paper opens up a design space: what if we used a different reward function while keeping the same optimization structure? This is exactly what hybrid methods do—they modify the reward to incorporate information beyond binary "matches the ground truth or not," such as partial correctness, output diversity, or consistency with a learned reward model. Conversely, what if we applied RL-style techniques (importance sampling, online rollouts, KL regularization) to SFT? This is the "RL → SFT" direction in Table 1.

Step 3: The unified regularized objective. The paper then states the general form that encompasses both SFT and RL:

$\max_\pi \mathbb{E}_{x \sim \mathcal{D}_x, y \sim \pi_\theta(\cdot \mid x)}\left[r(x, y)\right] - \beta \cdot \text{KL}\left(\pi_\theta(\cdot \mid x) \parallel \pi_0(\cdot \mid x)\right)$

where $\pi_0$ is the base (reference) policy model—typically the pretrained model or an SFT checkpoint— $r(x, y)$ is a proxy for the reward (which could be the SFT surrogate reward $-\mathbb{I}\{y = y_i\} / \pi_\theta(y \mid x)$ for SFT, or a learned reward model or rule-based scorer for RL), and $\beta$ is a hyperparameter controlling the strength of the KL penalty.

What the KL term computes. The Kullback-Leibler divergence $\text{KL}(\pi_\theta \parallel \pi_0) = \mathbb{E}_{y \sim \pi_\theta}[\log \pi_\theta(y \mid x) - \log \pi_0(y \mid x)]$ measures how much the current policy's output distribution diverges from the reference policy's distribution. When $\pi_\theta = \pi_0$ , the KL divergence is zero. When $\pi_\theta$ concentrates probability on outputs that $\pi_0$ considers unlikely, the KL divergence is large.

Why the KL penalty is essential. Without the KL penalty ( $\beta = 0$ ), the optimizer would push the policy toward whatever outputs maximize $r(x, y)$ , regardless of how unnatural or incoherent those outputs are. This is the mechanism behind reward hacking: the policy discovers an output that achieves high reward according to the reward function (which may be imperfect or misspecified) but is not actually a good response. The KL penalty anchors the policy to the reference model, which (presumably) produces coherent and safe outputs, preventing catastrophic divergence. The paper explicitly notes:

"The KL regularization between the policy model and the reference model restricts $\pi_\theta$ from deviating too much from a pre-trained checkpoint, primarily for stability reasons."

The hyperparameter $\beta$ as a temperature parameter. Larger $\beta$ means stronger regularization—the policy stays closer to the reference model, which increases stability but limits how much it can improve over the reference. Smaller $\beta$ allows more aggressive optimization toward the reward, which can achieve higher performance but risks instability and reward hacking. In practice, $\beta$ is treated as a tunable hyperparameter that must be set appropriately for each task and reward function.

The paper's core argument about mutual reinforcement. From this unified perspective, the paper draws a central conclusion about the relationship between SFT and RL:

"Therefore, optimizing LLMs with both SFT and RL objectives ultimately collapses to the RL objective, and the tricks that work for one thus have potential to be applied to the other. To mitigate the drop in generalization ability from SFT, one could consider applying importance sampling and online rollout, as in RL. Meanwhile, to help LLMs memorize additional knowledge, one could integrate the SFT loss into the RL objective. The roles of these two stages should be regarded as a mutually reinforcing and interdependent relationship instead of merely being applied alternatively."

This conclusion is the intellectual foundation for the taxonomy of hybrid methods that follows.

The Taxonomy of Hybrid Methods: Table 1

The paper organizes hybrid SFT-RL methods (Section 4.2–4.4) in Table 1, which lists methods along with their objective expressions, data sources, and the direction of integration (whether SFT modifies RL, RL modifies SFT, or both are jointly optimized). The taxonomy reveals four distinct strategies for combining SFT and RL, each with different theoretical motivations and practical tradeoffs.

Strategy 1: Using RL insights to improve SFT ("RL → SFT"). These methods modify the SFT objective using techniques borrowed from RL, without introducing a separate RL training phase:

DFT (Decoupled Fine-Tuning) (Wu et al., 2025): Rescales each token's SFT loss by the model's current predicted probability of that token: $\mathbb{E}_{(x_i, y_i) \sim \mathcal{D}}[-\text{sg}(\pi_\theta(y_i \mid x_i)) \nabla_\theta \log \pi_\theta(y_i \mid x_i)]$ , where $\text{sg}(\cdot)$ is the stop-gradient operator. The term $\pi_\theta(y_i \mid x_i)$ is the probability the current policy assigns to the ground-truth token—used as a weight but not differentiated through (hence "stop-gradient"). Tokens the model already assigns high probability to get smaller loss weights; tokens the model considers unlikely get larger weights. This is functionally equivalent to importance sampling in RL, where rare but important events get upweighted. Why this helps: standard SFT treats all tokens equally, but some tokens represent critical reasoning steps while others are predictable function words. By downweighting already-learned tokens, DFT focuses the model's capacity on the tokens where improvement is needed, reducing overfitting and improving generalization.
iw-SFT (Importance-Weighted SFT) (Qin and Springenberg, 2025): Interprets curated SFT data as optimizing a lower bound on a sparse-reward RL objective, and tightens that bound via importance weights. The method trains on a filtered dataset $\mathcal{D}^+$ (containing only high-quality demonstrations) with an importance-weighted loss: $\mathbb{E}_{\tau \in \mathcal{D}^+}[-\frac{q(\tau)}{\pi_{\text{ref}}(\tau)} \log p(\tau; \theta)]$ , where $\tau$ is a trajectory (full prompt-response pair), $q(\tau)$ is a target distribution over trajectories (derived from the RL objective the user actually wants to optimize), and $\pi_{\text{ref}}(\tau)$ is the probability of that trajectory under a reference policy. Why this form: standard SFT implicitly assumes the training data is drawn from the optimal policy, which is typically false—the demonstrations come from humans or other models, not from the truly optimal behavior. The importance weight $q(\tau) / \pi_{\text{ref}}(\tau)$ corrects for this distribution mismatch, effectively telling the model "these demonstrations are valuable, but you should put more probability on the ones that are underrepresented in the dataset."
Proximal SFT (Zhu et al., 2025a): Adopts PPO-style clipping for SFT updates: $\mathbb{E}_{(x_t, y_t) \sim \mathcal{D}}[\min(\frac{\pi_\theta(y_t \mid x_t)}{\pi_{\theta_{\text{old}}}(y_t \mid x_t)}, \text{clip}(\frac{\pi_\theta(y_t \mid x_t)}{\pi_{\theta_{\text{old}}}(y_t \mid x_t)}, 1-\epsilon, 1+\epsilon))]$ , where $\pi_{\theta_{\text{old}}}$ is the policy from the previous optimization step and $\epsilon$ is the clipping threshold. If the ratio of new-to-old probabilities exceeds $1+\epsilon$ or falls below $1-\epsilon$ , the gradient is clipped to prevent the update from being too large. Why this helps: standard SFT can take excessively large steps on individual examples, causing the model to overfit to a single datapoint at the expense of others. The clipping constraint—borrowed directly from PPO in RL—ensures that no single example can change the policy too much in one update, improving training stability and final generalization.
VAR (Variational Reward-Weighted SFT) (Du et al., 2025b): Proposes a reward-weighted SFT objective derived through variational inference: $\mathbb{E}[\frac{\pi_{\text{ref}}(y \mid x) \exp(\frac{1}{\lambda} r(x, y))}{Z(x)} \log \pi_\theta(y \mid x)]$ , where $Z(x) = \sum_y \pi_{\text{ref}}(y \mid x) \exp(r(x,y)/\lambda)$ is a normalization constant (partition function) and $\lambda$ is a temperature parameter. The weight for each demonstration is proportional to $\exp(r(x,y)/\lambda)$ , meaning higher-reward demonstrations get exponentially more weight. Why this form: it bridges SFT and RL by showing that an RLHF-style objective (maximizing reward subject to KL penalty) has an equivalent formulation as a reward-weighted SFT problem, where the weights are determined by the reward function and reference policy. This enables using SFT infrastructure (which is simpler and more stable) to approximately implement RLHF.

Strategy 2: Using SFT data to enhance RL ("SFT → RL"). These methods incorporate offline demonstrations into the RL process to improve exploration, stability, or sample efficiency:

Off-Policy Guided RL (Yan et al., 2025a): Augments on-policy RL updates with reasoning traces from offline demonstrations. The method maintains a buffer of both online rollouts (generated by the current policy) and offline demonstrations (from the SFT dataset), and computes policy gradients using a mixture of both. Why this helps: pure RL can waste many training steps exploring low-quality regions of the output space before discovering good strategies. By seeding the training process with demonstrations of successful behavior, the model starts from a better initialization and explores more efficiently.
SRFT (Single-stage Reinforcement Fine-Tuning) (Fu et al., 2025b): Integrates supervised and reinforcement objectives in a single training stage, avoiding the inefficiency of sequential SFT-then-RL pipelines. The loss has three components: $\mathcal{L}_{\text{SRFT}} = \mathcal{L}_{\text{SFT}}^{\text{demo}} + \mathcal{L}_{\text{RL}}^{\text{demo}} + \mathcal{L}_{\text{RL}}^{\text{self-rollout}}$ , where $\mathcal{L}_{\text{SFT}}^{\text{demo}}$ is standard SFT loss on offline demonstrations, $\mathcal{L}_{\text{RL}}^{\text{demo}}$ is GRPO-style advantage-weighted loss also applied to demonstrations (with advantage estimated from a group that includes both demonstrations and online rollouts), and $\mathcal{L}_{\text{RL}}^{\text{self-rollout}}$ is the same GRPO loss applied to the model's own online generations. Why this helps: sequential SFT-then-RL can be inefficient because the RL phase "forgets" some SFT knowledge and has to rediscover it, or because the SFT phase wastes capacity learning patterns that RL would have discovered more effectively. By training on both objectives simultaneously, SRFT avoids these inefficiencies and achieves better final performance with less total compute.
Prefix Sampling (Huang et al., 2025): Uses the prefix of a ground-truth response (e.g., the first few tokens of a correct solution) to guide the model's generation before applying RL. The method first samples a prefix from the ground-truth response, then lets the model complete the response, and applies RL loss only to the model-generated continuation. The prefix partition receives SFT loss (train to reproduce the ground-truth prefix), while the continuation receives RL loss (train to maximize reward given the prefix). Why this helps: pure RL can struggle with exploration when the correct response requires a specific initial step that the model never samples. By providing the prefix from a demonstration, the method guarantees that the model starts on the right track, and RL only needs to learn how to continue correctly—a much easier problem.
NFT (Negative Feedback Training) (Chen et al., 2025a): Enables models to learn from both correct and incorrect outputs under a unified objective. For each prompt, the method generates multiple rollouts and identifies which are correct and incorrect. The loss function has two terms: for correct outputs, it maximizes $\log R_t^\theta(x, y)$ , where $R_t^\theta(x, y) = \pi_\theta^+(y_t \mid x, y_{<t}) / \pi(y_t \mid x, y_{<t})$ is a ratio measuring how much the policy has increased the probability of the correct token; for incorrect outputs, it minimizes $\log \max_v(\frac{1 - \hat{r}_x R_t^\theta(x,y)}{1 - \hat{r}_x}, \epsilon)$ , where $\hat{r}_x$ is the expected reward for prompt $x$ . Why this helps: most RL methods only learn from the reward signal on the generated outputs—they increase probability for high-reward outputs and decrease for low-reward ones. But they don't explicitly penalize specific incorrect behaviors. By directly penalizing incorrect outputs (not just low-reward outputs, but outputs known to be wrong), NFT provides a stronger and more targeted learning signal.
BREAD (Branched Rollouts from Expert Anchors) (Zhang et al., 2025f): Uses expert prefixes as anchors and generates branched rollouts—multiple completions from each anchor point—then applies RL to optimize the branches. This reduces reliance on large demonstration sets because a single expert prefix can generate many diverse rollouts through branching, each of which provides a learning signal. Why this helps: collecting full expert demonstrations is expensive. BREAD shows that partial demonstrations (prefixes) are sufficient to seed effective RL, because the model only needs to learn how to complete the trajectory from a known-good starting point.
SRL (Supervised Reinforcement Learning) (Deng et al., 2025): Decomposes the reasoning process into intermediate steps and compares online rollouts with offline expert trajectories at each step. The reward is proportional to the matching between the model's generated steps and the expert's steps: $R = 2 \sum_{(i,j,n) \in \text{MatchingBlocks}} n / (|S_1| + |S_2|)$ , where $S_1$ and $S_2$ are the sequences being compared and $n$ is the length of each matching subsequence. Why this helps: this provides dense, step-level feedback rather than sparse, outcome-level feedback. If the model makes a small error early in a long reasoning chain, outcome-based RL might still give zero reward, providing no signal about where the error occurred. Step-level matching identifies exactly which steps diverge from the expert, enabling more targeted learning.

Strategy 3: Hybrid training with combined objectives. These methods jointly optimize weighted combinations of SFT and RL losses in a single training process:

HPT (Hybrid Post-Training) (Lv et al., 2025): Uses a unified objective $\mathcal{L} = \alpha \mathcal{L}_{\text{RL}} + \beta \mathcal{L}_{\text{SFT}}$ where $\alpha$ and $\beta$ control the relative weight of each loss. The method interleaves SFT and RL based on the policy model's performance during online rollouts: when performance is above a preset threshold (the model is generating good outputs on its own), online RL is preferred for exploration; when performance is poor (the model struggles), correct guidance from SFT is preferred. Why this helps: this adaptive weighting addresses the observation that RL is most beneficial when the model already has some competence—it can explore effectively—while SFT is most beneficial when the model is struggling and needs explicit guidance. By dynamically adjusting the weights based on current performance, HPT uses each method when it is most effective.
CHORD (Zhang et al., 2025e): Combines a weighted SFT loss with GRPO loss: $\mathcal{L}_{\text{CHORD}} = (1 - \mu) \mathcal{L}_{\text{GRPO}} + \mu \mathcal{L}_{\text{SFT}-\phi}$ , where $\mu$ is a mixing coefficient. The SFT loss component is modified with token-level weights $\phi(y_t^*; \pi_\theta) = \pi_\theta(y_t^* \mid x, y_{<t}^*)(1 - \pi_\theta(y_t^* \mid x, y_{<t}^*))$ , which upweights tokens the model is uncertain about (probability near 0.5) and downweights tokens it is confident about (probability near 0 or 1). Why this uncertainty-weighting form: $\pi(1-\pi)$ is the variance of a Bernoulli distribution—it is maximized at $\pi = 0.5$ and minimized at $\pi = 0$ or $\pi = 1$ . By weighting SFT tokens by this variance, CHORD focuses SFT supervision on exactly the tokens where the model is most uncertain, complementing the RL signal which already provides strong guidance on high-confidence tokens through the reward.
UFT (Unified Fine-Tuning) (Liu et al., 2025b): Formulates the objective as a combination of value-based RL on generated steps and maximum-likelihood on expert steps: $\mathcal{J}^{\text{UFT}} = \mathbb{E}[\mathcal{J}^{\text{value}} - \beta \sum \text{KL}(\pi \parallel \pi_{\text{ref}}) + \beta \sum_{h=0}^{l-1} \log \pi(y_h^* \mid x_h^*)]$ , where the first $l$ steps use expert data and the remaining steps use the model's own generations evaluated through a value function. Why this helps: early steps in a reasoning chain are critical—if the model starts wrong, it will almost certainly end wrong. By using expert data for the first few steps and RL for the rest, UFT ensures the model learns correct initial reasoning while still benefiting from RL's ability to explore alternative completions. The paper reports this achieves "exponentially improved sample efficiency."
RPO (Regularized Preference Optimization) (Liu et al., 2024): Extends the DPO objective by directly injecting an SFT loss on samples from the base model: $\mathcal{L}_{\text{RPO}} = \eta\beta \cdot \mathbb{E}_{x \sim d_0, y^0 \sim \pi_{\text{base}}}[-\log \pi_\theta(y^0 \mid x)] + \mathcal{L}_D(\beta \cdot \log \frac{\pi_\theta}{\pi_{\text{ref}}})$ , where $\mathcal{L}_D$ is the standard DPO loss on preference pairs. The first term is an SFT loss on base model samples (encouraging the policy to maintain the base model's generation patterns), and the second term is the DPO preference loss. Why this helps: DPO can suffer from overoptimization—the policy learns to exploit the preference model, producing outputs that score highly on the learned reward but are actually poor. The SFT term acts as an implicit adversarial regularizer, anchoring the policy to the base model's distribution and preventing the kind of distribution shift that enables reward hacking.
Q-SFT (Hong et al., 2024): Combines SFT with Q-learning: an SFT loss $\mathcal{L}_{\text{CE}} = \mathbb{E}_{(x,y) \sim \mathcal{D}}[\log \pi_\phi(y \mid x)]$ is used to train a separate policy, while a Q-function $Q_\theta(x, y)$ is trained via temporal difference learning $\mathcal{L}_{\text{TD}} = \mathbb{E}[(r + \gamma \max_{y'} \bar{Q}_\theta(x', y') - Q_\theta(x, y))^2]$ on an offline dataset of $(x, y, r, x')$ tuples. Why this helps: Q-learning provides a value estimate for each $(x, y)$ pair, which can be used downstream for planning or for selecting the best response among multiple candidates at inference time. The SFT component ensures the policy generates coherent responses, while the Q-function provides a way to rank them by expected future reward.

Strategy 4: Interleaved or sequential SFT-RL pipelines. These methods alternate between SFT and RL phases based on model performance or training progress:

SASR (Step-wise Adaptive SFT-RL) (Chen et al., 2025b): For each training step $t$ , the method chooses between SFT and RL loss based on an indicator function $\mathbb{I}(t)$ : $\mathcal{L}(\theta) = \frac{1}{S} \sum_{s=1}^S [(1 - \mathbb{I}(t)) \cdot \mathcal{L}_{\text{SFT}}(\theta) + \mathbb{I}(t) \cdot \mathcal{L}_{\text{GRPO}}(\theta)]$ . The decision rule $\mathbb{I}(t)$ can be based on training progress, validation performance, or other heuristics. Why this helps: rather than committing to a fixed schedule (e.g., "1000 steps of SFT then 500 steps of RL"), SASR adapts the schedule to the model's current needs, potentially avoiding both undertraining and overtraining in each phase.
IFT (Intuitive Fine-Tuning) (Hua et al., 2024): Transforms the RL problem into a sequence of SFT problems by constructing trajectories where the expert action at each step is known. Specifically, it minimizes $-\sum_{i=n}^N \log \mathcal{T}_\theta(a_i^*, \delta_\theta(s_i^*))$ , where $\mathcal{T}_\theta$ is the model's action distribution, $a_i^*$ is the expert action at step $i$ , and $\delta_\theta(s_i^*)$ is the model's internal state. Why this helps: by reducing RL to SFT on carefully constructed trajectories, IFT avoids the instability of policy gradient methods while still learning from sequential decision-making data.

Design choice: why organize hybrid methods by direction of integration? The four strategies above represent fundamentally different answers to the question "how should SFT and RL be combined?" Strategy 1 ("RL → SFT") says: SFT is basically right, but it can be improved by borrowing specific techniques from RL. Strategy 2 ("SFT → RL") says: RL provides the core optimization, but it needs to be seeded with expert knowledge to explore efficiently. Strategy 3 (combined objectives) says: neither paradigm is sufficient alone; they address complementary aspects of learning and should be jointly optimized. Strategy 4 (interleaved) says: the optimal combination depends on the training stage, and the best approach is to dynamically switch between them. By organizing methods according to this taxonomy, the paper helps practitioners understand not just what each method does but why it might be appropriate for their specific situation—whether they have abundant expert data (favoring Strategy 2 or 3), limited data but a good reward model (favoring Strategy 1), or a complex task requiring both imitation and exploration (favoring Strategy 3 or 4).

The Empirical Trend Analysis Framework (Section 5 and Appendix B)

The paper's methodological contribution is complemented by a systematic empirical analysis of how the research community is actually deploying SFT and RL. The framework described in Appendix B is not a machine learning method but a research methodology for surveying the literature quantitatively.

Benchmark-oriented keyword search. The paper uses 26 standardized datasets as query keys, organized across four domains (Table 4 in Appendix B):

QA: HotpotQA, StrategyQA, TriviaQA, PubMedQA, MuSiQue, 2WikiMultihopQA, QASPER
Math: GSM8K, ASDiv, SVAMP, AIME
Agentic: WebShop, WebArena, Mind2Web, MiniWoB++, ScienceWorld, ALFWorld, TDW-MAT, C-WAH, ALFRED, RLCard
Code: SWE-bench, HumanEval, LiveCodeBench, BIRD, InterCodeSQL

Classification procedure. A paper is assigned to a domain if it contains at least five mentions of any dataset associated with that domain. The threshold of five is justified by the observation that "a paper that genuinely employs a dataset typically introduces it, reports results on it, and provides comparative or analytical discussion, making five mentions a conservative filter against purely methodological contributions." Papers can be assigned to multiple domains (mutual exclusivity is not enforced) because some papers evaluate across domains.

Training type classification. The paper uses an LLM-based classifier (gpt-oss-120b, with FP4 precision, 128k context length, temperature 0.8, top-k 40, top-p 0.95) to extract the training type from each paper's text. The classifier assigns each paper to exactly one of seven categories: SFT, RL, SFT+RL, Prompt Optimization, none-text Modality, Other Methods, or Survey. The prompt used for classification is provided verbatim in Appendix E. Human readers subsequently verify all candidate papers to confirm publication status and training paradigm relevance—this human verification step is important because LLM-based classification can introduce errors, and the paper is transparent about this limitation.

Data source classification. Papers are also categorized by their primary data source: benchmark datasets, open-weight model outputs, proprietary API model outputs (e.g., GPT-4), or human-curated/web-scraped data. This enables tracking the shift from API-based labeling to open-weight model-generated datasets that the paper identifies as a key trend.

Projection methodology for 2025. Since the data collection covers January 2023 through June 2025, the paper projects full-year 2025 counts by noting that the first six months of 2023 and 2024 account for 47.38% and 49.81% of their respective annual totals, and scaling the observed 2025 first-half counts accordingly. This projection assumes similar seasonal patterns, which the paper acknowledges as an approximation.

Results produced by this framework. The framework generates the quantitative evidence for the paper's main empirical claims (Figure 2 and Table 5):

Research volume across all four domains more than doubled between 2023 and 2024 (e.g., QA: 292 → 652; Math: 492 → 1,098; Agentic: 100 → 174; Code: 115 → 428)
Hybrid "SFT+RL" training grew from 20.0% of studies in 2023 to 73.8% in 2024 (+269%)
SFT-only declined from 73.3% to 19.1% over the same period
Proprietary API usage declined from 32.2% to 11.1% (projected 2025), while open-weight model usage more than doubled from 12.2% to 25.0%
The five-mention threshold provides a conservative filter: Table 5 shows that paper counts decrease consistently as the threshold increases, with the >5 threshold representing a balance between specificity and recall

Why this empirical framework matters. Most survey papers are purely qualitative—they describe methods and draw conclusions based on the authors' expertise and reading of the literature. This paper's quantitative trend analysis adds an empirical dimension that is rare in survey papers: it doesn't just claim that the field is moving toward hybrid methods; it measures this movement and quantifies its magnitude. This makes the paper's conclusions more falsifiable and more actionable—if a reader disagrees with the trend analysis, they can replicate the methodology with different keywords or thresholds and see whether the conclusions hold.

4. Key Insights and Innovations

Innovation 1: SFT Is a Special Case of RL — A Unifying Mathematical Framework That Recontextualizes Both Paradigms

The paper's most fundamental conceptual contribution is not a new algorithm or an empirical finding, but a reframing of the relationship between SFT and RL that transforms how the field should think about post-training. By demonstrating that the SFT gradient can be rewritten as a policy gradient with the surrogate reward function $-\mathbb{I}\{y = y_i\} / \pi_\theta(y \mid x)$ , the paper establishes that SFT is not an alternative to RL but rather a specific instance of RL with a particular (sparse, demonstration-anchored) reward parameterization.

Why this reframing is intellectually distinctive. Before this paper, the standard mental model in the LLM community treated SFT and RL as fundamentally different beasts: SFT was "supervised learning on static data" and RL was "reward-driven optimization on model-generated outputs." They were applied sequentially (SFT then RLHF) with little theoretical justification for why this order worked, and techniques developed for one paradigm rarely transferred to the other. The paper's derivation (Section 4.1) collapses this distinction: both are policy gradient methods differing only in their reward function. The SFT reward is an indicator that fires only when the model exactly reproduces the ground-truth response, weighted inversely by the model's current probability of generating that response. The RL reward is a (typically learned) function that can assign partial credit, penalize undesirable outputs, and generalize beyond the training demonstrations.

This shift in perspective has generative consequences — it unlocks a design space that was previously invisible. If SFT is just RL with a particular reward, then every technique developed for RL (importance sampling, KL regularization, online rollouts, advantage estimation) should, in principle, be applicable to SFT. Conversely, if RL can be understood through the lens of SFT, then the stability and simplicity of supervised learning can be brought to bear on RL's instability challenges. The paper's taxonomy of hybrid methods (Table 1) is a direct product of this reframing: DFT, iw-SFT, Proximal SFT, and VAR all emerge from the question "what if we applied RL techniques to the SFT objective?" — a question that only makes sense once you recognize they share the same optimization structure.

Comparison to prior work. The dominant narrative before this survey, exemplified by the original RLHF pipeline (Ouyang et al., 2022; Bai et al., 2022), treated SFT as a necessary but theoretically uninteresting first step: you do SFT to "warm up" the model, then you do RL to actually optimize for desired behavior. The idea that these stages might be combined, interleaved, or unified was present in isolated works (e.g., Liu et al., 2024's RPO, which added an SFT loss to DPO), but these efforts were ad hoc — motivated by empirical improvements rather than a principled understanding of why combining objectives should help. The paper provides that principled understanding by showing that the combination is not an arbitrary trick but a natural consequence of the shared mathematical structure. This is a fundamental conceptual shift rather than an incremental refinement: it changes the question from "which method is better?" to "what reward function best captures our objectives, and what optimization strategy best suits that reward function?"

Significance beyond performance. The practical payoff of this reframing is the paper's ability to explain and predict empirical phenomena that were previously puzzling. Why does SFT suffer from distribution shift (Section 2)? Because its reward is zero for all outputs except the ground truth — the model never receives feedback on what happens when it deviates from the demonstration, so it cannot learn to recover from errors. Why can RL generalize better (as documented in Chu et al., 2025, which the paper cites)? Because its reward function covers a broader distribution of outputs, providing gradient information even for imperfect responses. Why does combining SFT and RL objectives often outperform either alone (as shown across the methods in Table 1)? Because the SFT reward provides dense, correct-but-narrow supervision while the RL reward provides broad-but-noisy exploration — they are complementary reward signals, not competing optimization strategies. These explanations are not post-hoc rationalizations; they follow directly from the unified objective and would be difficult to articulate without it.

The paper anchors this innovation in the formal derivation connecting the SFT gradient to the RL gradient (Section 4.1) and in the organization of hybrid methods in Table 1, which systematically categorizes approaches according to which direction of the SFT-RL relationship they exploit.

Innovation 2: The First Systematic Empirical Mapping of How the Post-Training Landscape Is Actually Shifting

The paper's second distinctive contribution is methodological rather than theoretical: it provides the first quantitative, benchmark-oriented analysis of how the research community's use of SFT and RL has evolved across application domains, training paradigms, and data sources from 2023 to 2025. This is not merely a literature review — it is an empirical study of the literature itself, treating the corpus of arXiv preprints as a dataset to be analyzed for trends, shifts, and convergence patterns.

Why this methodology is intellectually distinctive. Most survey papers are qualitative: authors read papers, identify themes, and synthesize findings based on their expertise. This is valuable but subjective — different authors might organize the same literature differently, and claims about "growing adoption" or "shifting paradigms" are typically supported by citation counts or the authors' impressions rather than systematic measurement. The paper's benchmark-oriented search methodology (Appendix B) changes this by making trend claims falsifiable and reproducible. A reader who disagrees with the finding that hybrid SFT+RL training grew from 20% to 73.8% of studies between 2023 and 2024 can replicate the methodology — same keywords, same thresholds, same time window — and check whether the conclusions hold. This transforms the survey from an opinion piece into a scientific instrument.

What the methodology reveals that qualitative surveys cannot. The paper's key empirical findings (Figure 2) include several trends that are non-obvious and, in some cases, counterintuitive:

Hybrid training dominance is accelerating faster than expected. The growth from 20.0% to 73.8% in a single year (+269%) is not a gradual shift — it is a phase transition. By 2024, hybrid methods had become the default approach, with pure SFT dropping to just 19.1%. This suggests that the field has passed a tipping point where the "SFT-then-RL" sequential pipeline is being abandoned in favor of integrated approaches, and the paper documents this empirically rather than asserting it impressionistically.
Open-weight model usage is replacing proprietary APIs as the primary data source, doubling from 12.2% to 25.0% while API usage halved from 32.2% to 11.1%. This trend has practical implications for reproducibility (open-weight models enable exact replication), cost (API labeling is expensive at scale), and the democratization of post-training research (researchers without API access can still participate). The paper quantifies what many researchers have anecdotally observed: the field is becoming less dependent on GPT-4 as a data-generation oracle.
Research volume is growing explosively across all domains, but the growth pattern varies. Code-related research grew 272% (115 → 428 papers) between 2023 and 2024, while agentic research grew 79% (100 → 174). These differential growth rates reveal where research energy is concentrating — code generation is maturing rapidly, while agentic tasks remain a smaller but steadily growing niche.

Comparison to prior work. Prior surveys in this space (Parthasarathy et al., 2024; Tao et al., 2024; Kumar et al., 2025; Lai et al., 2025) provide valuable qualitative synthesis but do not attempt systematic quantitative trend analysis. The paper explicitly distinguishes itself from these works in Section 1: "In contrast, our survey provides a systematic and integrated perspective on SFT and RL as complementary post-training tools, with particular emphasis on their interplay and practical applications." The "systematic" claim is backed by the benchmark-oriented methodology; the "practical applications" claim is backed by the domain-specific trend data.

Significance beyond the specific numbers. The empirical framework's broader significance is that it establishes a methodology for tracking the evolution of a fast-moving field that other surveys can adopt or adapt. The approach — using standardized benchmark mentions as domain proxies, applying consistent keyword thresholds, and projecting partial-year data — is generalizable to other subfields of ML. It also provides a baseline against which future shifts can be measured: if, in 2026, hybrid training drops back to 50% or open-weight usage stalls, that would be a meaningful signal that something has changed in the research ecosystem.

The paper anchors these claims in Figure 2 (which visualizes the trends across domain, training methodology, and data source), Table 5 (which shows paper counts under different keyword-mention thresholds, establishing robustness to the threshold choice), and Appendix B (which provides the full methodology, enabling replication).

Innovation 3: A Taxonomy That Reveals Four Fundamentally Different Strategies for Combining SFT and RL — Not Just "Add Both Losses"

The paper's third distinctive contribution is the taxonomic organization of hybrid SFT-RL methods into four categories based on the direction of influence between the two paradigms: (1) RL insights applied to improve SFT ("RL → SFT"), (2) SFT data used to enhance RL ("SFT → RL"), (3) joint optimization of combined objectives, and (4) interleaved or adaptive switching between the two. This taxonomy does more than organize existing work — it reveals that "combining SFT and RL" is not a single idea but a family of strategies with different theoretical motivations and practical tradeoffs.

Why this taxonomy is intellectually distinctive. Before this paper, the literature on combining SFT and RL was fragmented. Individual papers proposed specific combinations (e.g., "add an SFT loss to DPO" or "use expert prefixes to seed RL") without a shared vocabulary for describing how their approach related to others. The paper's taxonomy provides that vocabulary by asking a simple organizing question: which paradigm is being modified, and in what direction? This reveals that superficially different methods often share deep structure:

DFT (which rescales SFT token losses by policy probabilities) and iw-SFT (which applies importance weights to curated SFT data) are both "RL → SFT" methods because they take techniques originally developed for RL (importance sampling, probability weighting) and apply them to improve the SFT objective. They differ in mechanism but share the same conceptual move: SFT is treated as a policy optimization problem, and RL-derived techniques make that optimization more efficient.
SRFT (which integrates demonstration loss and rollout loss in a single stage) and Prefix Sampling (which uses expert prefixes to guide RL exploration) are both "SFT → RL" methods because they take data or supervision originally associated with SFT (offline demonstrations, expert prefixes) and use it to improve the RL process. They differ in what SFT-derived element they incorporate but share the conceptual move: RL benefits from being grounded in expert knowledge.
CHORD (which combines weighted SFT and GRPO losses) and UFT (which uses expert data for early steps and RL for later steps) are both "joint optimization" methods because they simultaneously apply both losses in a single training process. They differ in how they weight the two components but share the conceptual move: the two objectives are complementary and should be optimized together rather than sequentially.

What the taxonomy explains that individual methods cannot. The taxonomy reveals that the effectiveness of any particular hybrid method depends on which problem you are trying to solve. If your primary challenge is that SFT overfits and loses diversity, "RL → SFT" methods like DFT (which downweights already-learned tokens) are natural fits because they directly address SFT's failure mode. If your primary challenge is that RL explores inefficiently and wastes compute on obviously wrong outputs, "SFT → RL" methods like Prefix Sampling (which seeds exploration with correct prefixes) are natural fits because they directly address RL's exploration problem. If your primary challenge is that sequential SFT-then-RL causes forgetting or inefficiency, joint optimization methods like SRFT are natural fits because they avoid the sequential bottleneck. The taxonomy transforms the question from "should I use a hybrid method?" to "which type of hybrid method addresses my specific bottleneck?"

Comparison to prior work. Prior to this survey, the literature on SFT-RL combinations was organized chronologically (who published what when) or by application domain (hybrid methods for math, hybrid methods for code, etc.). Neither organization reveals the structural relationships between methods. The paper's taxonomy is a fundamental conceptual advance over these ad hoc organizations because it groups methods by their theoretical mechanism rather than their surface characteristics. This makes the taxonomy useful for both understanding existing work and guiding future research: a researcher developing a new hybrid method can ask "which category does my method fall into, and what does that category tell me about likely failure modes and appropriate baselines?"

Significance beyond taxonomy for its own sake. The taxonomy directly generates the practical takeaways in Section 6. When the paper advises "When high-quality expert data are available, SFT is generally preferred as an initial training stage" and "If a strong policy model or prompt is available to generate expert data, incorporating importance sampling on failed queries can mitigate distribution shift by treating them as informative positive samples for RFT/RL," these recommendations follow from the taxonomy's analysis of when each strategy is appropriate. The taxonomy is not just a filing system — it is a decision framework.

The paper anchors this innovation in Table 1 (which lists methods with their objective expressions, data sources, and integration direction) and in the structured discussion of Sections 4.2–4.4, which walk through each category with representative methods and their theoretical justifications.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper does not train or evaluate models on a single dataset in the traditional sense. Instead, it conducts a meta-analysis of the research literature by searching approximately 195,000 arXiv preprints in the Computer Science categories from January 2023 through June 2025. Papers are classified into four application domains—QA, math, agentic, and code—using a benchmark-oriented keyword search methodology detailed in Appendix B. Specifically, 26 standardized datasets serve as domain-specific query keys (listed in Table 4 of the paper): HotpotQA, StrategyQA, TriviaQA, PubMedQA, MuSiQue, 2WikiMultihopQA, and QASPER for QA; GSM8K, ASDiv, SVAMP, and AIME for math; WebShop, WebArena, Mind2Web, MiniWoB++, ScienceWorld, ALFWorld, TDW-MAT, C-WAH, ALFRED, and RLCard for agentic tasks; and SWE-bench, HumanEval, LiveCodeBench, BIRD, and InterCodeSQL for code. A paper is assigned to a domain if it contains at least five mentions of any dataset associated with that domain; mutual exclusivity is not enforced, so multi-domain papers are counted in every applicable category. The initial dataset comprises approximately 64,000 papers from 2023, 78,000 from 2024, and 52,000 from the first half of 2025. Full-year 2025 projections are derived by observing that the first six months of 2023 and 2024 account for 47.38% and 49.81% of their respective annual totals, then scaling the observed 2025 first-half counts accordingly.

Base model(s). The empirical trend analysis does not use a base model for benchmarking—it analyzes which models the surveyed papers use as their data sources or training backbones. The paper classifies data sources into four categories: proprietary API models (e.g., GPT-4), open-weight models, benchmark datasets, and human-curated or web-scraped data. The trend results (Figure 2, right panel) track the proportion of surveyed papers using each data source type. No specific model family or scale is the object of evaluation; rather, the analysis measures community-wide shifts in model preferences over time.

Metrics. The primary metrics are publication counts and proportions across several categorical dimensions: (1) research volume per application domain (QA, math, agentic, code) per year, measured as the number of papers meeting the five-mention threshold for domain-specific benchmark keywords; (2) training methodology distribution (SFT-only, RL-only, or SFT+RL hybrid), expressed as a proportion of papers within each year; and (3) data source distribution (proprietary API, open-weight models, benchmarks, or human/web data), also expressed as proportions. The paper reports percentage changes year-over-year (e.g., "+123%" for QA paper count growth from 2023 to 2024) as well as raw counts. For projection purposes, counts are filtered at varying keyword-mention thresholds (from >0 to >20) and reported in Table 5, with the >5 threshold used as the primary reporting standard.

Baselines. The paper's trend analysis does not compare against specific algorithmic baselines. The "baseline" is implicitly the 2023 distribution of methodologies and data sources, against which 2024 and projected 2025 distributions are compared. For example, the finding that SFT-only methods constituted 73.3% of surveyed papers in 2023 but only 19.1% in 2024 (a 74% relative decline) uses the 2023 proportion as the reference point. The training type classification into SFT, RL, SFT+RL, Prompt Optimization, none-text Modality, Other Methods, and Survey is performed by an LLM-based classifier (gpt-oss-120b, FP4 precision, 128k context, temperature 0.8, top-k 40, top-p 0.95), with the prompt provided verbatim in Appendix E. Human readers subsequently verify all candidate papers to confirm publication status and training paradigm relevance.

Generation budget / compute accounting. The paper does not measure or compare FLOPs, GPU-hours, or inference budgets because it is not benchmarking model performance. Compute is accounted for only indirectly: the paper notes in Appendix C that SFT typically requires approximately 16GB of VRAM per billion parameters in FP16, while LoRA-based RL training consumes roughly 15–25GB VRAM per billion parameters (1.5–3× higher memory demand per parameter than SFT). These figures are presented as community-reported heuristics, not as experimentally measured quantities. No compute-matched comparison between SFT and RL is attempted in the trend analysis.

Cross-validation / statistical protocol. The paper does not apply cross-validation or statistical significance testing to its trend data because it is a descriptive census, not a sample-based experiment. The robustness of domain assignments is assessed through sensitivity to the keyword-mention threshold: Table 5 reports paper counts at thresholds from >0 to >20 across all four domains, showing that counts decrease consistently as the threshold increases (e.g., for code papers in 2024: 1,091 at >0, 767 at >1, 621 at >2, 481 at >4, 428 at >5, and 254 at >10), but the relative trends across domains and years remain stable. The authors acknowledge that their methodology "may introduce approximation bias, as commonly adopted benchmarks are not fully inclusive of all real-world scenarios or methodologies" (Limitations section). The 2025 projections assume that seasonal publication patterns in 2025 match those of 2023 and 2024, which is explicitly noted as an assumption rather than a verified fact.

Main Quantitative Results

The paper's experimental output consists of a single composite figure (Figure 2) and a supporting table (Table 5 in Appendix B). All quantitative claims in the paper derive from these two exhibits. The results are organized around three axes of analysis: domain-specific research volume trends, training methodology shifts, and data source transitions.

Domain-Specific Research Volume Trends

Figure 2 (left panel) and the accompanying text in Sections 5 and Appendix B.4 report paper counts across four domains for 2023, 2024, and projected 2025.

Headline numbers. Research volume grows sharply across all domains between 2023 and 2024, with differential growth rates by domain:

QA: 292 papers (2023) → 652 papers (2024), a +123% increase. Projected 2025: 983 papers (+118% over 2024).
Math: 492 papers (2023) → 1,098 papers (2024), a +123% increase. Projected 2025: 2,399 papers (+118% over 2024), representing a near 5× increase from 2023.
Agentic: 100 papers (2023) → 174 papers (2024), a +79% increase. Projected 2025: 261 papers (+46% over 2024).
Code: 115 papers (2023) → 428 papers (2024), a +272% increase. Projected 2025: 786 papers (+84% over 2024).

These figures are based on the >5 keyword-mention threshold. The paper states: "Results show rapid growth across all surveyed domains, with substantial increases in research volume and diversification of application areas" (Figure 2 caption).

Relative growth patterns. Code-related research exhibits the fastest relative growth (272%) but from a smaller absolute base (115 papers in 2023). Math-related research has the largest absolute growth (606 additional papers from 2023 to 2024) and is projected to become the largest domain by volume in 2025 at 2,399 papers. Agentic research, while showing the slowest relative growth (79% from 2023 to 2024), still more than doubles in absolute volume over the three-year window (100 → 261). The paper notes that "this trajectory reflects the rapid maturation of code-centric benchmarks, tools, and evaluation pipelines, and highlights coding as one of the most rapidly diversifying application domains for LLMs" (Appendix B.4).

Robustness to threshold choice. Table 5 demonstrates that while raw paper counts are sensitive to the keyword-mention threshold, the relative trends are stable. At the >1 threshold (more inclusive, potentially capturing papers that mention benchmarks only in passing), math paper counts are 3,870 (2023), 7,143 (2024), and 4,889 (first half of 2025). At the >10 threshold (more conservative), the corresponding counts are 221, 538, and 794. The growth direction (2024 > 2023) is preserved across all thresholds for all domains.

Training Methodology Shifts

Figure 2 (center panel) and Section 5 report the distribution of training methodologies (SFT-only, RL-only, or hybrid SFT+RL) across the surveyed papers for 2023, 2024, and projected 2025.

Headline numbers for methodology distribution. The paper reports:

2023: SFT-only dominates at 73.3% of studies. Hybrid SFT+RL represents 20.0%. RL-only represents 6.7%.
2024: Hybrid SFT+RL becomes the dominant approach at 73.8% (+269% relative growth from 2023). SFT-only drops to 19.1% (a 74% relative decline). RL-only rises slightly to 7.1%.
Projected 2025: Hybrid SFT+RL continues to dominate at 70.6%. SFT-only declines further to 17.6%. RL-only rises to 11.8%.

The paper characterizes this as "a marked consolidation toward hybrid approaches that combine SFT with RL or other post-training strategies" and notes that "by 2024, hybrid training expands by 269%, becoming the most common approach at 73.8%, superseding pure SFT (19.1%)" (Appendix B.4).

Side-by-side comparison. In 2023, SFT-only methods outnumbered hybrid methods by a factor of roughly 3.7× (73.3% vs. 20.0%). In 2024, hybrid methods outnumbered SFT-only by a factor of roughly 3.9× (73.8% vs. 19.1%)—an almost exact reversal. The magnitude and speed of this reversal (occurring within a single year) is the paper's strongest empirical finding. The paper does not report confidence intervals or error bars on these proportions, as they represent a full census of the filtered papers rather than a sample.

RL-only trajectory. RL-only methods remain a consistent minority throughout the period (6.7% → 7.1% → 11.8%). The paper attributes the modest growth to "ongoing improvements in open-source RLHF frameworks and increased accessibility of preference data" (Appendix B.4). The paper does not break down what fraction of RL-only methods use online vs. offline RL, PPO vs. GRPO vs. REINFORCE, or other algorithmic distinctions—the RL category is treated as monolithic in the quantitative analysis.

Data Source Transitions

Figure 2 (right panel) and Section 5 report trends in the primary data sources used by surveyed papers across 2023, 2024, and projected 2025.

Headline numbers for data source distribution. The paper reports:

Proprietary API models: 32.2% (2023) → 19.9% (2024) → 11.1% (projected 2025), a cumulative decline of 66% from 2023 to 2025.
Open-weight models: 12.2% (2023) → 17.5% (2024) → 25.0% (projected 2025), more than doubling from 2023 to 2025.
Benchmarks: 48.9% (2023) → 62.1% (2024) → 61.1% (projected 2025), representing the most commonly used resource category throughout.
Human/Web data: 6.7% (2023) → 0.5% (2024) → 2.8% (projected 2025), declining sharply.

The paper describes this as "a pronounced realignment in model choice" and notes that "reliance on proprietary API-based models declines sharply" while "open-weight usage more than doubles" (Appendix B.4).

Side-by-side comparison. In 2023, proprietary API usage exceeded open-weight model usage by a factor of roughly 2.6× (32.2% vs. 12.2%). By projected 2025, this relationship inverts: open-weight usage exceeds proprietary API usage by a factor of roughly 2.3× (25.0% vs. 11.1%). Simultaneously, benchmark usage consolidates as the dominant data source category, growing from approximately half to approximately three-fifths of all surveyed papers. The paper does not break down open-weight model usage by specific model families (e.g., Llama vs. Qwen vs. DeepSeek), so the trends aggregate over all non-proprietary models.

Human/web data decline. The sharp decline in human-curated and web-scraped data (6.7% → 0.5% → 2.8%) is noted but not deeply analyzed in the paper. The authors attribute it to "improved benchmark coverage and growing concerns around legality, license compliance, and data provenance" (Appendix B.4). The anomalous jump from 0.5% in 2024 to a projected 2.8% in 2025 is not explained in the text.

Ablation Studies and Robustness Checks

Because this is a meta-analysis of publication trends rather than a model evaluation, traditional ablation studies (varying hyperparameters, removing components) are not applicable. The paper instead provides robustness checks on its trend analysis methodology through sensitivity analysis on the keyword-mention threshold and through a discussion of methodological limitations.

Keyword-mention threshold sensitivity (Table 5). The paper reports paper counts at thresholds ranging from >0 (most inclusive, likely capturing any paper that mentions a benchmark even in passing) to >20 (most conservative, capturing only papers with extensive benchmark engagement). The results show that all four domains exhibit consistent growth from 2023 to 2024 across all thresholds. For example, QA paper counts at the >1 threshold (580 → 1,321 → 1,088 for first-half 2025) and at the >20 threshold (56 → 114 → 90) both show the same 2023-to-2024 growth direction, though the absolute numbers and growth rates differ. This consistency across thresholds supports the robustness of the trend direction claims, though the paper does not compute formal sensitivity metrics (e.g., how much the percentage growth rate changes as the threshold varies).

Methodological limitations acknowledged. The paper identifies several sources of potential bias in its trend analysis (Limitations section and Appendix B.2):

Approximation bias from benchmark-oriented search: "the use of a paper-filtering strategy may introduce approximation bias, as commonly adopted benchmarks are not fully inclusive of all real-world scenarios or methodologies." Papers that use custom or less-common benchmarks may be excluded, potentially undercounting certain subdomains or novel evaluation approaches.
Multi-domain papers are double-counted: mutual exclusivity is not enforced, meaning a paper evaluating on both MATH and HumanEval would be counted in both the math and code categories. The paper acknowledges this but does not report what fraction of papers are assigned to multiple domains or how this might affect trend interpretations.
LLM-based classification of training types: the gpt-oss-120b classifier that assigns papers to SFT, RL, SFT+RL, or other categories is subject to its own errors. While human readers verify all candidate papers, the paper does not report inter-annotator agreement between the LLM classifier and human verifiers, nor does it report the fraction of papers that required reclassification upon human review.
2025 projections assume stable seasonality: the paper scales first-half 2025 counts by assuming that the first six months of 2025 will constitute the same fraction of annual output as they did in 2023 (47.38%) and 2024 (49.81%). Changes in submission patterns, conference deadlines, or the pace of research could make this assumption invalid. The paper does not report uncertainty bounds or alternative projection models.

Domain characteristic analysis (Table 3). The paper presents a qualitative characterization of each domain's input complexity, reasoning requirements, and output complexity (Appendix B.1, Table 3). While not a formal ablation, this serves as a sanity check on the domain categorization: the domains differ meaningfully in their requirements, supporting the claim that the trends reflect real shifts across diverse task types rather than artifacts of a single narrow benchmark. However, these characterizations are the authors' qualitative assessments and are not empirically validated (e.g., through measurements of average prompt length, number of reasoning steps, or output token counts in the surveyed papers).

Hardware requirements discussion (Appendix C). The paper provides community-reported heuristics for hardware requirements (e.g., 16GB VRAM per billion parameters for full SFT fine-tuning, 15–25GB VRAM per billion parameters for LoRA-based RL training), but these are presented as approximate guidelines rather than experimentally measured quantities. They serve to contextualize the practical differences between SFT and RL but are not part of the core empirical analysis and are not validated against the actual hardware configurations used in the surveyed papers.

Critical Assessment

The paper makes three central empirical claims, each supported by the trend analysis in Figure 2 and Table 5. Here I examine whether the experiments actually demonstrate what is claimed, with attention to what is and is not tested.

Claim 1: "Hybrid SFT+RL training became the dominant paradigm, growing from 20% of studies in 2023 to 73.8% in 2024." This claim is directly supported by the data reported in Figure 2 (center panel) and described in Section 5 and Appendix B.4. The numbers are clear: 20.0% → 73.8% is a measured shift, not an extrapolation. However, there are important caveats about what this number represents:

The classification granularity is coarse. The "SFT+RL" category in Figure 2 aggregates all methods that use both SFT and RL, regardless of how they are combined. This includes sequential SFT-then-RL pipelines (the standard RLHF recipe), joint optimization methods (like SRFT or CHORD), and methods that merely use SFT to warm-start RL. The paper's qualitative analysis (Sections 4.2–4.4 and Table 1) carefully distinguishes these different integration strategies, but the quantitative trend analysis collapses them into a single category. As a result, we cannot tell from Figure 2 alone whether the growth in "SFT+RL" represents a shift toward novel hybrid methods (which the paper emphasizes) or simply reflects the increasing adoption of the standard sequential pipeline (which has been the dominant RLHF recipe since Ouyang et al., 2022). The paper's claim that the field is converging toward "integrated frameworks that leverage the complementary strengths of both SFT and RL" (Section 1) would be stronger if it could break down the SFT+RL category by integration type.
The LLM classifier's accuracy is not reported. The paper states that human readers verify candidate papers, but does not report what fraction of the LLM's classifications were corrected upon human review, nor whether the classifier exhibits systematic biases (e.g., overcategorizing papers that mention both SFT loss and RL reward as "SFT+RL" even when the RL component is a minor ablation). This matters because a 20.0% → 73.8% shift could be partly an artifact of classifier behavior if, for example, papers in 2024 are more likely to include SFT+RL keywords in their abstracts without actually using a hybrid method.
Selection bias is not addressed. The 2023 paper count (at the >5 threshold) is 292+492+100+115 = 999 papers across all four domains. The 2024 count is 652+1,098+174+428 = 2,352 papers. The methodology captures more than twice as many papers in 2024 as in 2023. This could reflect genuine growth in the field, but it could also reflect changes in arXiv submission practices (more CS papers being posted), changes in benchmark popularity (more papers mentioning the specific 26 datasets used as keywords), or improvements in the search methodology's coverage of 2024 papers versus 2023 papers. If the 2023 paper set is biased toward well-established benchmarks (which might correlate with more conservative methodology choices), the 2023 SFT-only proportion could be overstated. The paper does not discuss selection bias as a potential confound.

Claim 2: "Open-weight model usage more than doubled over the same period (12.2% → 25%)." This claim is supported by Figure 2 (right panel) and shows a clear trend. However, the interpretation of this trend as representing "a continued shift from API-based labeling to open-weight–generated datasets" (Section 1) is subject to several caveats:

The data source categories are not mutually exclusive. A paper could use both open-weight models and API models (e.g., using GPT-4 to generate training data and Llama as the base model for fine-tuning). The percentages reported (e.g., 25.0% open-weight in projected 2025) represent the fraction of papers that use open-weight models as a data source, not necessarily as the primary or exclusive data source. The paper does not report what fraction of papers use multiple data source types.
"Open-weight model" is a heterogeneous category. The category includes everything from small models like Llama-7B to large models like DeepSeek-V3. A shift from "proprietary API" to "open-weight model" could represent a genuine democratization of research, or it could represent a shift toward using the largest open-weight models (which may themselves have been trained on proprietary data or with massive compute budgets inaccessible to most researchers) as API replacements. The paper does not break down open-weight model usage by model scale, which would be needed to distinguish these interpretations.
The trend line for "Benchmarks" dominates throughout (48.9% → 62.1% → 61.1%). The primary story in the data source analysis is not the shift from API to open-weight models (though that is real), but the consolidation around benchmark datasets as the central resource for post-training research. The paper emphasizes the API-to-open-weight shift in its narrative but the benchmark consolidation is arguably the more robust and practically significant finding, since it reflects the maturation of standardized evaluation infrastructure across domains.

Claim 3: "Research volume across all surveyed domains more than doubled between 2023 and 2024." This claim is supported for QA (+123%), math (+123%), and code (+272%), but only partially supported for agentic tasks (+79% — substantial growth but less than doubling). The paper's text in Section 1 states "more than doubled between 2023 and 2024," which is accurate for three of four domains but overstates the agentic trend. The projected 2025 growth rates (+118% QA, +118% math, +46% agentic, +84% code) suggest that the "more than doubling" characterization is a 2023–2024 phenomenon and may not persist uniformly.

What experiments would have strengthened the paper:

Breakdown of hybrid SFT+RL by integration type. The paper's most important empirical finding—the shift to hybrid training—would be far more informative if it distinguished sequential SFT-then-RL from genuinely integrated approaches. This would directly test the paper's central thesis that the field is moving toward novel hybrid paradigms rather than simply adopting the standard RLHF pipeline more widely.
Inter-annotator agreement for the LLM-based classifier. The paper would benefit from reporting the agreement rate between the gpt-oss-120b classifier and human verifiers, along with an analysis of common failure modes. This would allow readers to assess whether the measured trends could be artifacts of classifier behavior.
Confidence intervals or trend stability tests. The paper reports point estimates (e.g., 73.8% hybrid in 2024) without uncertainty quantification. Bootstrap resampling of the paper corpus or sensitivity analysis to keyword selection (e.g., leave-one-benchmark-out) would provide insight into the stability of these estimates.
Breakdown by model scale. The claim that open-weight models are replacing API-based labeling would be more nuanced if it showed whether this shift is toward large open-weight models (which may replicate some of the same centralization and access issues as proprietary APIs) or toward truly accessible smaller models.
Longitudinal consistency analysis. The 2025 projections assume stable seasonality, but the paper could test this assumption by checking whether the 2023 first-half vs. full-year ratio (47.38%) itself varied across domains, which would indicate whether the projection is reliable or domain-dependent.

Summary of evidential strength. The paper's trend analysis is a valuable and novel contribution to the survey literature—it provides quantitative grounding for claims that most surveys make qualitatively. The measured shift from SFT-dominated to hybrid-dominated post-training is large (73.3% → 19.1% for SFT-only, 20.0% → 73.8% for hybrid), and the robustness checks in Table 5 show that this shift persists across keyword-mention thresholds. However, the coarse granularity of the training type classification (lumping all SFT+RL combinations into one category) limits the strength of the paper's central narrative about novel integrated frameworks: the data show that SFT and RL are being used together more often, but cannot distinguish whether this represents a qualitative shift toward new hybrid paradigms or simply the quantitative growth of the standard sequential RLHF pipeline. The paper's theoretical framework (unified objective, Table 1 taxonomy) provides strong conceptual support for the value of hybrid methods, but the empirical analysis cannot validate that the field is adopting these specific hybrid strategies—only that it is adopting some form of combined SFT and RL. This gap between the theoretical synthesis (which is the paper's strongest contribution) and the empirical trend analysis (which is novel but coarsely instrumented) is the paper's primary limitation as an experimental contribution.

6. Limitations and Trade-offs

The Unified SFT-as-RL Framework Does Not Quantify Practical Deployment Trade-offs

The paper's central theoretical contribution—that SFT is a special case of RL with a surrogate reward $-\mathbb{I}\{y = y_i\} / \pi_\theta(y \mid x)$ —provides a clean conceptual unification but operates entirely at the level of objectives and gradients, without addressing the practical deployment trade-offs that distinguish SFT from RL in production systems.

The assumption or constraint. The unified framework (Section 4.1) abstracts away from implementation details: both SFT and RL are characterized as maximizing expected reward minus a KL penalty. This abstraction treats the differences between SFT and RL as reward parameterization choices rather than as fundamentally different computational and operational regimes. The paper acknowledges this implicitly in Section 6, noting that "SFT is generally preferred as an initial training stage over RL... possibly due to its simpler implementation and greater stability," but never quantifies what "simpler" or "greater stability" means in concrete terms.

The consequence. Practitioners choosing between SFT and RL face decisions that the unified framework does not illuminate: SFT training is typically offline (you pre-collect a dataset and train on it, with no model-generated rollouts during training), while RL training is typically online (the model generates responses, receives rewards, and updates based on those rollouts—a fundamentally more complex distributed systems problem). The paper notes in Appendix C that RL "generally incurs 1.5–3× higher memory demands per parameter" than SFT, but this is a community-reported heuristic, not a measured quantity. More importantly, the paper does not analyze the wall-clock time, engineering complexity, or failure recovery differences between SFT and RL pipelines. A practitioner who adopts a hybrid method based on the paper's taxonomy may discover that the theoretical elegance of combining SFT and RL objectives is overshadowed by the operational complexity of maintaining two different training regimes, managing online rollout infrastructure, and debugging convergence issues that only appear at scale. The paper's framework explains why SFT and RL are complementary at the objective level, but cannot tell a practitioner whether the complementarity is worth the implementation cost for their specific use case.

What evidence exists in the paper. None. The paper does not benchmark training time, GPU-hours, or engineering effort for any of the methods it surveys. Appendix C provides rough VRAM heuristics but frames them as approximate guidelines and explicitly notes that "actual requirements depend on factors including model architecture, task complexity, batch size, rollout length, and ongoing improvements in software and hardware technologies." The trend analysis (Figure 2) shows that hybrid methods are popular—73.8% of surveyed papers used them in 2024—but popularity does not equal practicality. The paper does not report what fraction of these hybrid methods were successfully deployed in production systems versus used only in academic experiments.

Mitigation status. Not addressed. The paper does not attempt to characterize the deployment cost vs. benefit trade-off of hybrid methods, nor does it propose frameworks for estimating whether a given combination of SFT and RL is worth the engineering overhead. This is a legitimate scope limitation for a survey paper—the authors cannot experimentally characterize every method they cite—but it means that the paper's practical guidance (Section 6: "When high-quality expert data are available, SFT is generally preferred" and "If a strong policy model... is available... incorporating importance sampling on failed queries can mitigate distribution shift") is based on conceptual reasoning rather than empirical cost-benefit analysis.

The Benchmark-Oriented Search Methodology Cannot Capture Hybrid Methods' Integration Type

The paper's most empirically distinctive contribution—the quantitative trend analysis showing that hybrid SFT+RL training grew from 20% to 73.8% of studies between 2023 and 2024—relies on an LLM-based classifier that assigns each paper to one of seven coarse categories, with all forms of SFT-RL combination collapsed into a single "SFT+RL" bin.

The assumption or constraint. The classifier categorizes papers into mutually exclusive training types: SFT, RL, SFT+RL, Prompt Optimization, none-text Modality, Other Methods, or Survey. The paper's narrative emphasizes the shift toward novel integrated frameworks—methods that jointly optimize SFT and RL objectives, interleave them adaptively, or use one to enhance the other in principled ways. However, the "SFT+RL" category is defined broadly enough to include any paper that uses both SFT and RL, regardless of integration depth. This includes: the standard sequential SFT-then-RLHF pipeline (which has been dominant since Ouyang et al., 2022—what the paper might call "merely being applied alternatively"), genuinely integrated single-stage methods like SRFT or UFT, and marginal cases where RL is used only as a minor post-processing step after primarily SFT-based training. The paper explicitly critiques the sequential approach in Section 4, stating that SFT and RL "should be regarded as a mutually reinforcing and interdependent relationship instead of merely being applied alternatively," but its quantitative analysis cannot distinguish between "mutually reinforcing" and "merely applied alternatively."

The consequence. The paper's headline empirical claim—that the field is shifting toward "integrated frameworks that leverage the complementary strengths of both SFT and RL" (Section 1)—overstates what the data actually show. The data show that papers using both SFT and RL in some capacity grew from 20% to 73.8% of surveyed publications. This could reflect a genuine shift toward novel hybrid architectures (the paper's interpretation), or it could reflect the maturation of the standard RLHF pipeline: as more researchers gain access to RLHF infrastructure, more papers include both an SFT warm-up phase and an RL alignment phase, even though the two phases remain strictly sequential and methodologically independent. The paper provides no evidence to distinguish these interpretations, because the classifier does not break down "SFT+RL" by integration strategy.

This limitation is particularly important because the paper's theoretical contribution—the unified objective and the taxonomy of hybrid methods—is specifically designed to argue that certain types of combination are more principled and effective than others. The trend analysis, which should provide empirical validation of this argument, is instead instrumented at a granularity that cannot test it. A reader who accepts the claim at face value might conclude that the community has validated the paper's specific taxonomy through widespread adoption, when in fact the community may simply be adopting the standard sequential pipeline at larger scale.

What evidence exists in the paper. The paper provides some indirect evidence through its qualitative survey: Table 1 lists 18 hybrid methods with their integration types, and the paper's discussion in Sections 4.2–4.4 argues that genuinely integrated approaches are emerging as a distinct research direction. But the proportion of surveyed papers using each integration type is never quantified. The prompt used for the LLM-based classifier (reproduced in Appendix E) asks for "the training type of the proposed method, exactly one of: SFT, RL, SFT+RL, Prompt Optimization, none-text Modality, Other Methods, Survey"—it makes no attempt to sub-classify SFT+RL by integration depth. The paper acknowledges that "human readers subsequently verified all candidate papers" but does not report what fraction of SFT+RL classifications represented sequential versus integrated approaches, nor the inter-annotator agreement rate between the LLM classifier and human verifiers.

Mitigation status. Not addressed. The paper does not acknowledge this granularity mismatch as a limitation—in fact, the narrative treatment of the trend analysis consistently frames "SFT+RL" as synonymous with "hybrid" or "integrated" approaches, which is the interpretation the data cannot support. The paper could have addressed this by: (1) manually sub-classifying a random sample of SFT+RL papers to estimate the proportion using genuinely integrated vs. sequential approaches, or (2) modifying the classifier prompt to distinguish sequential SFT-then-RL pipelines from methods that jointly optimize both objectives. Neither was done.

Difficulty Estimation and Compute Efficiency Are Not Addressed in the Unified Framework

The paper's unified framework characterizes SFT and RL as alternative reward parameterizations within a common optimization structure, but it does not address a fundamental practical question that determines when each paradigm is appropriate: how much compute must be invested in each to achieve equivalent performance, and how does this depend on task difficulty?

The assumption or constraint. The paper treats SFT and RL as methodologically distinct but functionally interchangeable within the unified objective: both maximize expected reward minus KL divergence, with the only difference being the reward function. This implies that the choice between SFT and RL is a choice about what reward signal to use, not about fundamentally different scaling properties. The paper's practical guidance (Section 6) recommends SFT when "high-quality expert data are available" and RL when "a reliable reward model can be trained," but these recommendations are based on data availability rather than compute efficiency.

The consequence. A practitioner with a fixed compute budget faces a resource allocation problem that the paper's framework does not help solve: given $N$ GPU-hours, should they invest in collecting high-quality demonstrations for SFT, training a reward model for RL, or some combination? The paper provides no guidance on this question because it never compares the compute efficiency of SFT and RL at equivalent performance levels. Appendix C notes that RL training consumes 1.5–3× more memory per parameter than SFT, but this is a per-step cost, not a total-cost-to-convergence metric. If RL requires 3× more memory per step but converges in 1/10th the number of steps—or if RL achieves a higher performance ceiling that SFT cannot reach regardless of compute—then the per-step cost comparison is misleading. The paper's trend analysis (Figure 2) shows that hybrid methods are increasingly popular, but popularity could indicate that researchers have access to abundant compute (making the higher cost of RL acceptable) rather than that hybrid methods are compute-efficient. This is particularly critical for practitioners with constrained resources, who must decide whether the potential benefits of RL justify its higher implementation complexity and computational cost.

What evidence exists in the paper. The paper provides no compute-matched comparison between SFT and RL. It does not report training FLOPs, wall-clock time, or total cost for any of the methods it surveys. The trend analysis measures paper counts and proportions, not resource consumption. The hardware requirements in Appendix C are approximate heuristics explicitly presented as "rough starting points" rather than "strict requirements." The paper does not report what hardware configurations were used in the surveyed studies, making it impossible to estimate the total compute invested in SFT-only vs. RL-only vs. hybrid research.

Mitigation status. Not addressed. The paper's future directions (Section 7) identify "sample- and compute-efficient methodologies" as an open problem—citing early work on data-efficient SFT (FisherSFT; Deb et al., 2025b) and quantization-aware methods (Wei et al., 2025b)—but frames this as an efficiency problem within each paradigm rather than a comparative efficiency question between paradigms. The paper does not suggest that future work should establish compute-matched baselines for SFT vs. RL comparisons, which would be necessary to determine whether the shift toward hybrid methods (documented in Figure 2) reflects genuine improvements in compute efficiency or simply reflects the field's willingness to spend more compute on post-training.

The Survey's Scope Is Limited to Text-Only Tasks with Well-Defined Correctness Signals

The paper explicitly restricts its application analysis (Section 5) to text-only tasks, and within those, to domains where correctness can be evaluated through standardized benchmarks with clear ground-truth answers or verifiable outputs (math problems with exact answers, code generation with unit tests, QA with factual correctness, agentic tasks with environment-defined success).

The assumption or constraint. The paper states in Appendix B.1: "Section 5 focuses on papers addressing text-only tasks. We make this choice because text-only tasks enable clearer side-by-side comparison, whereas multimodal settings introduce additional sources of uncertainty in model selection and fusion techniques, which lie outside the scope of this work." The domains analyzed—QA, math, agentic, and code—all share the property that task success can be objectively measured: math problems have exact answers, code can be executed against test cases, QA questions have factual answers, and agentic tasks have environment-defined reward signals. This is a reasonable scoping choice for a survey, but it means the paper's findings about the effectiveness of SFT, RL, and hybrid methods are restricted to tasks with clean, objective correctness signals.

The consequence. Many important LLM applications—open-ended dialogue, creative writing, summarization, translation, tutoring, therapy, negotiation—do not have objective correctness criteria. In these domains, the distinction between SFT and RL becomes more complex because the "reward function" in RL must itself be learned from subjective human preferences (as in RLHF), and the quality of this learned reward fundamentally constrains the effectiveness of RL. The paper's unified framework (Section 4.1) represents the reward function as $r(x, y)$ without interrogating how this reward is obtained. In math and code domains, $r$ can be a rule-based verifier—nearly perfect, zero-noise, and generalizing to all possible outputs. In open-ended domains, $r$ is a learned preference model trained on noisy, sparse, and potentially inconsistent human judgments. The paper's recommendations—"When a reliable reward model can be trained, the common practice is to first perform SFT and then RL, which typically achieves the highest reported performance" (Section 6)—assume reward model reliability, which is precisely what is most difficult to achieve in the domains the paper excludes.

Additionally, the paper's trend analysis (Figure 2) shows that math and code research are growing fastest (math: 492 → 1,098 papers in one year; code: 115 → 428). These domains benefit most from verifiable rewards, which may be driving both their rapid growth and the apparent success of hybrid methods. If the field's convergence toward hybrid SFT+RL (73.8% of studies in 2024) is partly an artifact of the domains most amenable to RL-based optimization growing fastest, then the trend may not generalize to other domains. The paper cannot distinguish between "hybrid methods work better" and "hybrid methods are especially effective in the domains that also happen to be growing fastest"—these are confounded in the trend analysis.

What evidence exists in the paper. The paper does not empirically test whether its findings generalize beyond text-only, objective-correctness domains. It explicitly acknowledges this scope limitation in Appendix B.1 but frames it as a choice for "clearer side-by-side comparison" rather than as a potential threat to external validity. Section D provides additional references to multimodal and speech-related work (Cui et al., 2025c; Yang et al., 2025c; Abootorabi et al., 2025) but does not analyze whether the SFT-RL relationship identified in the paper holds in these settings.

Mitigation status. Partially addressed through acknowledgment. The paper is transparent about its scope (Limitations section: "this study may omit some recent advances in SFT/RL research due to the rapid pace of progress"), but this acknowledgment focuses on pace rather than domain coverage. The survey does not claim generalizability to multimodal or open-ended tasks, but it also does not warn readers that its central framework—SFT as a special case of RL with an indicator reward—assumes the existence of a well-defined reward signal, which may not hold in many practical applications. A clearer statement about the boundary conditions of the framework would strengthen the paper's practical utility.

The Trend Analysis Provides Population-Level Statistics but Cannot Explain Causal Drivers of Observed Shifts

The paper's empirical contribution—the quantitative trend analysis in Figure 2 and Table 5—documents what is changing in the post-training research landscape but provides no evidence about why these changes are occurring.

The assumption or constraint. The methodology described in Appendix B is purely descriptive: it counts papers, classifies them by domain, training type, and data source, and reports proportions and growth rates. This is valuable as a measurement exercise, but the paper's narrative (Sections 1, 5, and 6) consistently interprets these trends as evidence for the effectiveness of hybrid methods and the superiority of open-weight models, rather than as outcomes of complex sociotechnical dynamics in the research community. For example, the paper states that the shift from proprietary to open-weight models "underscores the field's movement toward reproducible, transparent experimentation" (Appendix B.4)—a causal claim about researchers' motivations that the data cannot support. Alternative explanations for the same trends are equally plausible: the shift could be driven by cost (API labeling became more expensive), by API access restrictions (rate limits, content policies), by the release of capable open-weight models (making API dependence unnecessary rather than undesirable), or by journal/conference norms that increasingly favor reproducible methods.

The consequence. A reader who accepts the paper's causal framing might conclude that hybrid SFT+RL methods are inherently better and that the research community has converged on them because of their demonstrated superiority. But the trend data are equally consistent with a scenario where hybrid methods are simply the fashionable approach—adopted because they are exciting to work on, because they generate more publishable results, or because the infrastructure for hybrid training has become more accessible (through libraries like TRL, TRLX, and verl)—without being provably more effective or efficient than well-tuned SFT-only or RL-only baselines. The paper does not control for baseline quality: it documents that papers using hybrid methods are a growing fraction of the literature, but cannot assess whether these papers actually achieve better results than papers using SFT-only or RL-only approaches, because it does not compare the performance numbers reported in each category.

This limitation is serious because it undermines the paper's practical recommendations. Section 6 advises that "the common practice is to first perform SFT and then RL, which typically achieves the highest reported performance." This recommendation is based on the observation that hybrid methods are common, not on a meta-analysis showing that they empirically outperform alternatives on specific benchmarks. A practitioner who follows this advice invests in building an RL infrastructure that the paper's own analysis cannot confirm is worth the cost.

What evidence exists in the paper. The paper provides no performance comparison between the methodology categories. The trend analysis measures popularity, not effectiveness. Table 5 and Figure 2 report paper counts and proportions; they do not report average performance gains, ablation results, or head-to-head comparisons between SFT-only and hybrid methods on standardized benchmarks. The paper's qualitative survey (Sections 3–4) cites individual papers that demonstrate improvements from specific hybrid methods, but these are cherry-picked success cases, not systematic comparisons.

Mitigation status. Not addressed. The paper does not acknowledge the gap between measuring methodological adoption and measuring methodological effectiveness, nor does it caution readers that popularity trends should not be interpreted as performance endorsements. A future version of this work could strengthen its practical recommendations by conducting a meta-analysis of reported benchmark scores across methodology categories, controlling for task domain and base model, to determine whether hybrid methods actually outperform alternatives and by what margin. Such an analysis would transform the paper's contribution from "the field is shifting" (descriptive) to "the shift is justified" (prescriptive).

The Universal KL-Regularized Objective Does Not Capture Differences in Exploration-Exploitation Behavior

The paper's unified objective (Section 4.1)—maximizing expected reward minus a KL penalty from a reference policy—treats the KL term as a generic stabilization mechanism. However, this formulation conflates two fundamentally different regularizers: KL divergence from a fixed reference policy (SFT's implicit regularizer) and KL divergence from a moving reference (RL's typical practice of regularizing toward the previous policy checkpoint or the initial SFT model).

The assumption or constraint. The unified objective $\max_\pi \mathbb{E}[r(x,y)] - \beta \cdot \text{KL}(\pi_\theta \parallel \pi_0)$ specifies a single reference policy $\pi_0$ . The paper states that $\pi_0$ is "the base (reference) policy model"—typically the pretrained model or an SFT checkpoint—and that the KL term "restricts $\pi_\theta$ from deviating too much from a pre-trained checkpoint, primarily for stability reasons." This formulation assumes that the stabilization function of the KL penalty is the same regardless of whether $\pi_0$ is a static pretrained model (as in SFT, where the reference is the starting checkpoint and never changes) or an evolving checkpoint (as in RL, where some methods update the reference policy periodically, effectively tracking the policy as it improves).

The consequence. The exploration-exploitation trade-off—a central concern in RL that directly affects whether the model discovers novel high-reward outputs or stagnates at local optima—is collapsed into the single hyperparameter $\beta$ . In practice, SFT with a static reference (the pretrained model) and RL with an evolving reference (the previous policy checkpoint, updated every $K$ steps) have qualitatively different exploration dynamics. With a static reference, the KL penalty acts as an absolute anchor: the policy can only move so far from the pretrained distribution, regardless of how many training steps it takes. This limits exploration but provides strong stability guarantees. With a moving reference, the KL penalty acts as a relative anchor: the policy can drift arbitrarily far from the pretrained distribution over many small steps, each of which is individually constrained. This enables more exploration—potentially discovering entirely new behavioral regimes—but at the cost of weaker stability guarantees (the policy can gradually "forget" pretraining knowledge through cumulative drift). The paper's recommendations (Section 6) do not distinguish these regimes: it advises SFT for "simpler implementation and greater stability" and RL for better "exploratory capacity," but does not connect this to the KL reference policy choice.

This limitation is particularly relevant for practitioners because the choice of reference policy update schedule is a hyperparameter that can dramatically affect RL outcomes—too frequent updates cause instability, too infrequent updates cause stagnation—and the paper's framework provides no guidance on how to set it.

What evidence exists in the paper. None. The paper's discussion of RL algorithms (Section 3.2) mentions entropy regularization and KL clipping as stabilization techniques but does not analyze how reference policy update frequency affects exploration or final performance. The taxonomy of hybrid methods (Table 1) lists whether each method uses an offline dataset, online rollouts, or both, but does not specify reference policy update schedules. The paper does not cite empirical work comparing different KL anchoring strategies.

Mitigation status. Not addressed. The paper does not identify the static vs. dynamic reference policy distinction as a dimension along which SFT and RL differ, nor does it suggest that future work should characterize the exploration-exploitation implications of different KL regularization schedules. This is a missed opportunity, because the unified objective formulation could naturally accommodate an analysis of how the choice of $\pi_0$ update schedule interacts with $\beta$ to control the exploration-exploitation trade-off—but the paper treats $\pi_0$ as a fixed given rather than a design choice.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not propose a new algorithm, dataset, or training recipe. It is a synthesis and reframing paper, and its primary impact on the field is conceptual rather than empirical: it provides the first unified mathematical framework that positions Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) for LLMs not as sequential stages in a pipeline, but as two instances of the same optimization problem differing only in their reward parameterization. This is a reframing, not a paradigm shift — the underlying algorithms (maximum likelihood, policy gradients) remain the same — but the reframing has substantial consequences for how researchers and practitioners reason about post-training design choices.

The magnitude of the shift is medium-to-large for how the community thinks about post-training, and small for what the community does day-to-day (for now). The paper's empirical trend analysis (Figure 2) reveals that the community was already moving toward hybrid SFT+RL approaches — from 20% of studies in 2023 to 73.8% in 2024 — before this survey was written. The paper does not cause this shift; it documents and explains it. The contribution is to provide the theoretical vocabulary that transforms "we do SFT then RL because it works" into "we do SFT then RL because SFT provides a dense-but-narrow reward signal that bootstraps exploration for RL's broad-but-noisy reward signal, and these signals are complementary within a shared policy gradient structure." This explanation is not post-hoc rationalization — it follows directly from the unified objective derivation in Section 4.1 and would have been difficult to articulate without it.

The paper resolves a specific set of apparent contradictions in the literature. Why does SFT suffer from distribution shift and compounding errors while still being the standard first stage of post-training? Because its implicit reward (the indicator function $-\mathbb{I}\{y = y_i\} / \pi_\theta(y \mid x)$ ) is zero for all outputs except the exact ground truth — providing no gradient signal for error recovery, hence compounding errors — but extremely stable because the reward is perfectly aligned with the training data. Why can RL generalize better but also collapse or reward-hack? Because its reward function covers a broader output distribution (providing gradient information even for imperfect responses, enabling generalization) but is typically learned or heuristic (introducing noise and misspecification that can be exploited). These are not separate phenomena requiring separate explanations — they are two regimes of the same KL-regularized policy optimization objective with different reward functions. The paper does not prove this mathematically (it states the equivalence and argues from it, but does not, for example, formally characterize the conditions under which the SFT reward leads to compounding errors), but it provides the conceptual machinery to make the explanation coherent.

Research directions that become more attractive. The unified framework makes method transfer across paradigms a natural research strategy rather than an unusual cross-pollination. If SFT is RL with a specific reward, then every technique from the RL literature (importance sampling, advantage estimation, entropy regularization, curriculum learning, reference policy annealing) is a candidate for improving SFT. Conversely, every insight from SFT about the value of high-quality demonstrations, the importance of data diversity, and the dangers of overfitting is applicable to RL's exploration and reward design. The paper's taxonomy (Table 1) organizes existing work along these transfer directions, and the framework predicts that more such transfers will be productive.

Research directions that become less attractive. The paper's framework implicitly argues against the "SFT-only vs. RL-only" framing that characterized much earlier work. If SFT and RL are points on a continuum rather than competing alternatives, then papers that ask "is SFT or RL better for task X?" are asking an underspecified question — the answer depends on the reward function quality, the availability of demonstrations, and the exploration budget, all of which are design choices rather than fixed properties of the task. The trend analysis confirms that the field has already largely abandoned this framing (SFT-only dropped from 73.3% to 19.1% of studies in one year), and the unified framework explains why: the question was never well-posed to begin with.

A secondary landscape change: the survey as a measurement instrument. The paper's benchmark-oriented trend analysis methodology (Appendix B) — using 26 standardized dataset keywords to classify ~195,000 arXiv preprints by domain, training type, and data source — establishes a reproducible, quantitative approach to tracking the evolution of a fast-moving field. This is rare in survey papers, which are typically qualitative. The methodology is imperfect (coarse classification granularity, LLM-based classifier with unknown error rates, projection assumptions), but it is transparent and falsifiable: a reader who disagrees with the claim that hybrid training grew from 20% to 73.8% can replicate the analysis with different keywords, thresholds, or classifiers. This is more than a contribution to understanding SFT and RL — it is a contribution to how the community can conduct surveys, particularly in fields where the literature is growing too fast for traditional manual review.

What the paper does not change. The unified framework does not provide guidance on the economic or engineering trade-offs between SFT and RL. It does not tell a practitioner with a fixed compute budget whether to invest in better demonstrations (for SFT) or a better reward model (for RL). It does not characterize the wall-clock time, infrastructure complexity, or failure recovery costs of hybrid pipelines versus pure SFT or pure RL. These are first-order concerns for deployment, and the paper's framework is silent on them. The practical takeaways in Section 6 ("When high-quality expert data are available, SFT is generally preferred... due to its simpler implementation and greater stability") are based on conceptual reasoning, not on cost-benefit analysis. A practitioner who follows these recommendations may find that the engineering overhead of maintaining a hybrid pipeline outweighs the performance gains the paper's framework predicts — a risk the paper does not quantify or even flag.

Follow-Up Research This Work Enables

A compute-matched head-to-head comparison of SFT, RL, and hybrid methods at equivalent total FLOPs. The paper's unified framework predicts that hybrid methods should outperform pure SFT or pure RL because they combine complementary reward signals. The trend analysis (Figure 2) shows that researchers increasingly believe this — 73.8% of studies used hybrid approaches in 2024 — but provides no evidence that hybrid methods are more efficient: it documents popularity, not performance-per-FLOP. A follow-up study should replicate the Chinchilla-style compute-matched comparison paradigm from Hoffmann et al. (2022) but applied to post-training: fix a total FLOPs budget, and compare (a) pure SFT on a large demonstration dataset, (b) pure RL with a fixed reward model, (c) sequential SFT-then-RL (the standard RLHF pipeline), (d) joint optimization (e.g., SRFT or CHORD-style combined loss), and (e) interleaved switching (e.g., HPT-style adaptive weighting) — all evaluated on the same MATH, GSM8K, HumanEval, or agentic benchmarks using the same base model. The key question is not "which method achieves the highest accuracy at convergence" but "which method achieves the highest accuracy per FLOP invested in post-training." This would directly test the paper's central thesis that hybrid methods are not just popular but efficient, and would provide practitioners with actionable guidance on how to allocate their post-training compute budget. The paper's current data cannot answer this question because it measures adoption, not efficiency.

A granular re-analysis of the hybrid SFT+RL category in the trend data. The paper's headline empirical finding — that hybrid SFT+RL training grew from 20% to 73.8% of studies between 2023 and 2024 — is instrumented at a granularity that cannot distinguish between genuinely integrated methods (joint loss optimization, interleaved switching) and the standard sequential SFT-then-RL pipeline that has been dominant since Ouyang et al. (2022). This is a critical gap because the paper's theoretical contribution specifically argues that certain types of combination are more principled than others, yet the trend analysis cannot validate whether the community is adopting these specific types. A follow-up study should manually sub-classify a random sample of ~200 papers from the SFT+RL category in 2023 and 2024 into (a) sequential SFT-then-RL, (b) joint optimization (single loss combining SFT and RL terms), (c) interleaved/adaptive switching, (d) SFT-enhanced RL (RL with demonstration seeding), and (e) RL-enhanced SFT (SFT with importance weights or KL regularization). The result would reveal whether the explosive growth in "SFT+RL" represents a genuine shift toward novel hybrid architectures (the paper's narrative) or simply the maturation of the standard RLHF pipeline (an alternative explanation the paper cannot rule out). This is a tractable annotation study — three annotators, a codebook derived from Table 1's taxonomy, 200 papers — that would substantially strengthen (or qualify) the paper's central empirical claim.

A systematic study of how reward function quality shapes the SFT-RL boundary. The unified framework (Section 4.1) implies that the optimal choice between SFT, RL, and hybrid methods depends on the quality of the reward function relative to the quality of the demonstrations. When the reward function is nearly perfect (rule-based verifiers for math, compilers for code), RL should dominate because it can explore beyond the demonstrations and discover novel high-reward outputs. When the reward function is noisy or misspecified (learned preference models for open-ended dialogue), SFT should dominate because its implicit reward is perfectly aligned with the (high-quality) training data, even if it cannot generalize beyond it. Hybrid methods should be most valuable in the intermediate regime where the reward function is informative but imperfect — the demonstrations provide a stable anchor, and the RL reward provides exploration. A follow-up study should test this prediction by systematically degrading reward function quality (e.g., training reward models on progressively smaller preference datasets, or adding controlled noise to rule-based verifiers) and measuring the performance gap between SFT, RL, and hybrid methods at each quality level on MATH and GSM8K. This would transform the paper's conceptual framework into a predictive model: given a reward function quality estimate, a practitioner could decide whether SFT, RL, or a hybrid approach is likely to be most effective. The paper currently provides no such guidance — its recommendations are binary ("when high-quality expert data are available, SFT is generally preferred") rather than gradient-based.

An empirical characterization of KL reference policy dynamics in SFT vs. RL. The paper's unified objective $\max_\pi \mathbb{E}[r(x,y)] - \beta \cdot \text{KL}(\pi_\theta \parallel \pi_0)$ treats $\pi_0$ as a fixed reference policy, but in practice, RL methods differ in how frequently they update $\pi_0$ : some use a static pretrained checkpoint (strong anchor, limited exploration), others update $\pi_0$ to the current policy every $K$ steps (weak anchor, more exploration but risk of drift). The paper does not discuss this distinction, yet it may explain why some RL methods generalize better than others and why some hybrid methods (which typically use a static reference from the SFT phase) are more stable than pure RL. A follow-up study should train the same base model with the same RL algorithm (e.g., GRPO) but vary the reference policy update frequency from every step (effectively no KL penalty) to never (static anchor) on MATH reasoning tasks, measuring both final accuracy and the rate of reward hacking / entropy collapse. This would characterize the trade-off between exploration and stability as a function of $\pi_0$ update schedule — a dimension the paper's framework can accommodate but does not explore. The hypothesis (which the paper's logic implies but does not state) is that easy problems benefit from static anchors (limited exploration prevents over-optimization) while hard problems benefit from dynamic anchors (exploration is necessary to discover correct solutions), mirroring the difficulty-dependent findings in the test-time compute scaling literature (Snell et al., 2024, though this is not cited in the current paper).

A survey extension to multimodal and open-ended task domains. The paper's application analysis (Section 5) is explicitly restricted to text-only tasks with objective correctness signals (math with exact answers, code with unit tests, QA with factual answers, agentic tasks with environment rewards). The unified framework assumes a well-defined reward function $r(x, y)$ , which is straightforward in these domains but becomes the central challenge in multimodal, open-ended, or subjective tasks (image captioning, dialogue, creative writing, medical diagnosis). A follow-up survey should extend the benchmark-oriented search methodology (Appendix B) to multimodal benchmarks (e.g., MMBench, MME, SEED-Bench, Video-MME) and open-ended evaluation settings (e.g., Chatbot Arena, AlpacaEval, MT-Bench) and ask: does the shift toward hybrid SFT+RL hold in domains where the reward function is necessarily learned from sparse, noisy human preferences rather than derived from ground-truth verifiers? If hybrid methods are less dominant in these domains — or if the specific types of hybrid methods differ (e.g., more emphasis on RL-enhanced SFT rather than joint optimization) — that would establish a boundary condition on the paper's framework and provide practical guidance for researchers in multimodal and subjective domains. The paper currently cannot claim generalizability beyond text-only, verifiable-reward settings, and this follow-up would determine whether that is a minor scope limitation or a fundamental constraint on the unified framework's applicability.

A negative-result study: when does combining SFT and RL hurt? The paper's narrative — and the trend analysis — emphasizes the success of hybrid methods, but there are documented cases where adding RL to SFT degrades performance: the paper itself notes in Appendix K that ReST^EM-based revision model training "substantially hurt performance" with sequential revisions, and the broader literature contains examples of reward hacking, entropy collapse, and catastrophic forgetting during RL phases. A systematic negative-result study would deliberately search for failure modes of hybrid approaches: test whether adding an RL phase to a strong SFT model on a narrow task (where the SFT demonstrations already cover the output distribution well) causes overfitting to the reward and loss of generalization; test whether joint optimization with a misspecified reward model causes the SFT component to learn incorrect patterns from the RL signal; and characterize the conditions under which "SFT then RL" is worse than "SFT alone." This would establish the boundary of the unified framework's practical applicability — the paper currently implies that hybrid methods are generally beneficial, but the framework itself suggests they should be harmful when the reward function introduces more noise than signal relative to the demonstrations. Documenting these failure conditions would make the paper's recommendations more actionable by telling practitioners when not to follow them.

Practical Applications and Downstream Use Cases

Efficient post-training pipeline design for domain-specific LLM deployment. Organizations deploying LLMs for specialized tasks — customer support, legal document analysis, medical QA, internal code generation — face a recurring decision: how to allocate their post-training budget between collecting expert demonstrations (for SFT) and building reward models (for RL). The paper's unified framework provides a conceptual decision tree: if the task has a verifiable correctness signal (code execution, mathematical equivalence, database query accuracy), prioritize RL with a rule-based reward, seeded by a small number of high-quality demonstrations (e.g., the Prefix Sampling strategy from Huang et al., 2025, which uses expert prefixes to guide exploration while applying RL to the continuation). If the task requires subjective quality judgments (tone, helpfulness, style) and high-quality demonstrations are available, prioritize SFT with data-efficient curation (e.g., FisherSFT-style information-theoretic selection) and use RL only as a light final refinement step with a learned preference model — the paper's trend analysis shows that SFT+RL hybrid approaches are now the dominant paradigm (73.8% of studies in 2024), but the specific integration strategy should depend on reward quality. If neither verifiable signals nor high-quality demonstrations are available, the framework suggests that neither SFT nor RL alone is likely to succeed — the task may require fundamental capability improvements through pretraining rather than post-training. This decision tree is not explicitly laid out in the paper, but it follows directly from the unified objective and the taxonomy of hybrid methods in Table 1.

Cost reduction through data source substitution. The paper documents a sharp shift in data source usage: proprietary API reliance dropped from 32.2% (2023) to 11.1% (projected 2025), while open-weight model usage more than doubled from 12.2% to 25.0% over the same period (Figure 2, right panel). For practitioners, this trend has a direct cost implication: generating SFT training data or RL reward signals using GPT-4 API calls at scale (e.g., 100,000 demonstrations at $0.01–0.03 per response) costs$ 1,000–3,000, while using an open-weight model like Llama-3-70B or Qwen-2.5-72B on self-hosted hardware amortizes to near-zero incremental cost per generation after the initial infrastructure investment. The paper's trend data suggest that the research community has already validated this substitution — open-weight model outputs are increasingly accepted as training data — which lowers the barrier for practitioners to follow suit. The practical workflow is: (1) fine-tune an open-weight model on a small set of human-verified demonstrations using data-efficient SFT (e.g., LIMA-style with ~1,000 high-quality examples, as described in Section 3.1); (2) use this fine-tuned model to generate additional training data for iterative self-improvement or to train a reward model; (3) apply hybrid SFT+RL training using the generated data and reward model. The paper does not prescribe this exact workflow, but each component is supported by methods surveyed in Sections 3–4.

Resource-constrained deployment through method selection based on data availability. For teams with limited compute budgets (academic labs, startups, non-ML organizations adopting LLMs), the paper's Section 6 recommendations translate to a concrete resource allocation strategy. If the team has access to domain experts who can produce ~1,000 high-quality demonstrations: use SFT with data-efficient curation (FisherSFT or LIMA-style selection) and do not invest in RL infrastructure — the paper argues that SFT alone can achieve strong performance when demonstrations are high-quality, and the added complexity of RL (1.5–3× higher memory demand per parameter per Appendix C, plus online rollout infrastructure, plus reward model training) may not be justified. If the team lacks domain experts but has access to a strong open-weight model (e.g., Llama-3-70B) and a clear correctness metric (e.g., unit tests for code, answer matching for math): use RL with a rule-based reward, seeded by a small number of demonstrations generated by the open-weight model itself (the "SFT → RL" strategy from the taxonomy). If the team has neither: invest in pretraining data or model access rather than post-training — the paper's analysis of hard problems (Section 7's discussion of Chu et al., 2025, which showed that RL generalizes while SFT memorizes) suggests that post-training alone cannot compensate for fundamental capability gaps. These recommendations are grounded in the paper's theoretical framework rather than in empirical cost-benefit analysis, so they should be treated as hypotheses to validate rather than as proven prescriptions.

Benchmark selection for post-training research based on reward function characteristics. The paper's trend analysis reveals differential growth rates across domains: math research grew 123% (492 → 1,098 papers) from 2023 to 2024, while code research grew 272% (115 → 428). These are the two domains with the cleanest verifiable reward signals (exact answer matching for math, unit test execution for code). The paper's unified framework explains why: RL and hybrid methods are most effective when the reward function is reliable, and these domains provide ground-truth rewards for free. For practitioners choosing which domain to focus their post-training research on, this implies a strategic consideration: if you are developing a new RL algorithm or a new hybrid integration strategy, test it first on math or code where the reward signal is clean — this isolates the algorithmic contribution from reward noise. If you are developing a new reward modeling technique (e.g., learning from sparse preferences, active reward learning), test it on QA or agentic tasks where the reward signal is less reliable — this is where improved reward modeling has the most impact. The paper's domain characteristic analysis (Table 3) provides a qualitative mapping of input complexity, reasoning requirements, and output complexity that can guide this selection, though the mapping would benefit from quantitative instantiation (e.g., average reward sparsity per domain, inter-annotator agreement on preference labels).

When to Prefer This Method

The paper does not propose a specific new method — it is a survey and synthesis that organizes existing methods into a taxonomy. As such, a "Prefer A when B" decision matrix would not be faithful to the paper's contribution. The paper provides conceptual guidance for navigating the SFT-RL design space (summarized in Section 6), organized around three decision points:

Data-driven decision: what resources are available?

If you have abundant high-quality expert demonstrations → start with SFT (possibly data-efficient variants like FisherSFT or LIMA-style curation). RL may still add value, but the marginal benefit is smaller when demonstrations already cover the desired output distribution well.
If you have a reliable, automated reward signal (rule-based verifier, compiler, unit tests) but limited demonstrations → start with RL, seeded by a small number of demonstrations (the "SFT → RL" strategy from Table 1, e.g., Prefix Sampling or SRFT).
If you have both high-quality demonstrations and a reliable reward signal → use a joint optimization hybrid method (e.g., CHORD, SRFT, or HPT) that trains on both objectives simultaneously. The trend analysis (Figure 2) confirms this is the most common approach in current research (73.8% of studies in 2024).

Capability-driven decision: how far is the model from task competence?

If the model has near-zero pass@1 on the task (hard problems, difficulty bin 5 in the test-time compute literature) → post-training alone is unlikely to help, regardless of method. Invest in pretraining or model access instead. The paper cites Chu et al. (2025)'s finding that "RL generalizes, SFT memorizes," but notes that neither can create capabilities absent from the base model.
If the model has moderate pass@1 (e.g., 10–50%) → RL and hybrid methods provide the largest gains, because the model can generate partially correct outputs that provide learning signal through the reward function. The paper documents this regime as the "sweet spot" for RL.
If the model already has high pass@1 (e.g., >80%) → SFT on diverse, high-quality data may be sufficient, with RL used only for fine-grained preference alignment (e.g., reducing minor errors, improving style). Over-aggressive RL in this regime risks reward hacking.

Stability-driven decision: how complex is the RL infrastructure?

If training stability and implementation simplicity are primary concerns → prefer SFT or "RL → SFT" methods (e.g., DFT, iw-SFT, Proximal SFT) that apply RL insights (importance sampling, KL regularization, clipping) within the simpler SFT optimization framework, without requiring online rollouts, value networks, or reward model training.
If exploration and generalization beyond the training data are primary concerns → invest in full RL infrastructure (online rollouts, reward model, KL anchoring), potentially combined with SFT demonstrations to stabilize early training (the "SFT → RL" strategy).
If you are unsure → use interleaved or adaptive methods (e.g., HPT, SASR) that dynamically switch between SFT and RL based on model performance during training. These methods are more complex to implement but provide insurance against choosing the wrong fixed schedule.

These decision rules are derived from the paper's synthesis of existing work, not from the paper's own experiments. They represent the paper's conceptual mapping of the design space rather than empirically validated prescriptions. The paper's primary contribution is making this mapping explicit and systematizing it — before this survey, practitioners were making these decisions based on intuition and folklore. After this survey, they can make them based on a coherent framework that connects SFT's implicit reward function to its failure modes, RL's exploration dynamics to its instability risks, and hybrid methods to specific integration strategies tailored to different resource profiles. The next step — validating these decision rules through compute-matched comparisons and systematic ablation studies — is the follow-up research agenda the paper enables.