Good SFT Optimizes for SFT, Better SFT Prepares for Reinforcement Learning

🎯 Pitch

Stronger SFT checkpoints can underperform weaker ones after identical RL training—a rank reversal caused by the distribution mismatch between offline data and online rollouts. PEAR corrects this with importance-sampling loss reweighting and delivers up to a 14.6% pass@8 gain on AIME-2025, proving that SFT should be optimized for RL readiness, not offline accuracy.

1. Executive Summary

This paper studies a fundamental mismatch in the standard SFT→RL post-training pipeline for reasoning LLMs, demonstrating that optimizing SFT objectives for offline performance alone is an unreliable proxy for post-RL accuracy—stronger SFT checkpoints can underperform weaker ones after identical GRPO training on logic games and math benchmarks using Qwen2.5/3 and DeepSeek-distilled models. The authors propose PEAR (Policy Evaluation–inspired Algorithm for Offline Learning Loss Reweighting), an importance-sampling-based reweighting scheme that corrects the distribution mismatch between the offline behavior policy and the online target policy by down-weighting logged continuations that are implausible under the current model and up-weighting those that remain plausible, with three variants operating at the token level (suffix likelihood ratios), block level, and sequence level. PEAR consistently improves post-RL performance over canonical SFT across Qwen3 models from 0.6B to 8B parameters, delivering up to a 14.6% pass@8 gain on AIME-2025 and 40% absolute accuracy improvement on logic games for the 1.7B model, establishing that the offline stage should prioritize reproducible successes under the target policy that will be optimized online rather than maximizing SFT accuracy in isolation.

2. Context and Motivation

The Core Problem: Offline SFT Metrics Are a False Oracle for RL Readiness

The standard pipeline for post-training reasoning language models is now well-established: an offline Supervised Fine-Tuning (SFT) phase produces a checkpoint, and an online Reinforcement Learning (RL) phase—typically using algorithms like GRPO (Shao et al., 2024) or RLVR—further refines the model by training on the model's own sampled rollouts against verifiable reward signals (Guo et al., 2025). This two-stage approach has produced state-of-the-art results in mathematical and logical reasoning, most notably in DeepSeek-R1's training. The implicit assumption in essentially all existing SFT research is straightforward: if an SFT method produces better offline accuracy, then it should also produce a better initialization for subsequent RL. Most papers proposing new SFT objectives—probability-based losses (Li et al., 2025a), adaptive token reweighting (Qin & Springenberg, 2025), KL regularization, masked token losses—evaluate their methods primarily or exclusively on offline metrics (pass@1 after SFT, SFT validation loss, benchmark accuracy before RL). The downstream RL stage is either entirely omitted from evaluation or treated as a separate concern.

This paper demonstrates that this assumption is empirically false and potentially dangerous. The central finding, illustrated in Figure 1 and detailed in Section 2, is a phenomenon the authors call rank reversal: when multiple SFT objectives are evaluated on the same data and each resulting checkpoint undergoes identical GRPO training, the ordering of methods by offline performance does not predict their ordering by online (post-RL) performance. In some cases, a method that substantially outperforms another after SFT ends up substantially worse after both receive the same RL budget. The most striking example occurs with Qwen3-1.7B-Base on SynLogic games: the TopLogP objective (a hard-thresholded negative log-likelihood that trains only on high-confidence tokens) achieves strong offline pass@1—appearing to beat standard SFT—yet after identical GRPO, it produces the worst post-RL performance among all tested methods, underperforming even standard SFT initialization.

"One may be tempted to pick TopLogP for Qwen3-1.7B-Base because of the best offline scores, yet this choice leads to worst-among-all post-RL performance, even under-performing SFT initialized model." (Section 2.1)

This is not a one-off anomaly. The paper tests a wide spectrum of SFT objectives (Table 1), including KL-regularized SFT with various coefficients, probability-family objectives from Li et al. (2025a), token-adaptive loss reweighting (TALR), and hard-threshold masking variants that selectively train on high- or low-probability tokens. Across all these variations, the Pearson correlation between offline pass@1 and online pass@1 on Qwen3-1.7B-Base is a meager 0.09 (Figure 11). Even when using offline pass@128 or majority voting accuracy—two alternative offline metrics proposed as more correlated with RL performance by recent work (Yue et al., 2025; Kang et al., 2025)—the correlations remain weak (r = −0.67 and r = −0.42 respectively), and in some cases are actually negatively correlated with online performance.

This is a genuinely surprising result. It means that the entire existing methodology for evaluating and comparing SFT techniques—optimize them in isolation, report offline metrics, pick the best one—may be systematically selecting for properties that harm subsequent RL. The community has been optimizing the wrong objective.

Why This Problem Is Important: Practical and Theoretical Consequences

The practical stakes are substantial. The RL phase in modern reasoning LM training is computationally expensive—it requires simultaneous rollout generation on a training set and parameter updates, and when the RL environment involves tool calling or code execution (Wei et al., 2025; Qian et al., 2025), the cost can become extreme. Understanding which SFT checkpoints will actually benefit from this expensive RL investment—versus which will waste it—is a high-value signal. As the authors note:

"understanding the signals that can suggest a model's potential after RLVR can save a huge amount of compute, and has received great attention in the LLM community" (Appendix A)

If organizations can reliably predict which SFT initialization will yield the best post-RL model, they can avoid running expensive RL on suboptimal checkpoints and allocate compute budgets more efficiently. The current practice of using offline performance as a proxy is essentially a heuristic with no empirical validation—and this paper shows it is a poor one.

More fundamentally, the finding exposes a conceptual gap in how the field thinks about the SFT→RL pipeline. The two stages have been treated as modular: SFT teaches the model to imitate correct behavior; RL then polishes the model by exploring and reinforcing successful trajectories. This modularity assumption is implicit in essentially all work that proposes new SFT methods without evaluating post-RL performance. The paper argues that this modularity is illusory: properties that make an SFT checkpoint "good" in isolation (high offline accuracy, low loss, good calibration on the SFT data distribution) may actually make it harder for RL to improve—a phenomenon that Kang et al. (2025) also observed for repetition and data homogeneity, but which this paper shows applies much more broadly across loss function designs.

The theoretical significance extends to how we understand the optimization dynamics of the two-stage pipeline. If offline and online optimization pull the model in different directions, then the SFT stage is not just providing a "warm start" for RL—it is imposing a set of inductive biases that can either facilitate or obstruct subsequent RL updates. The challenge is to design SFT objectives that produce not the most accurate model on the offline data, but the model most amenable to RL refinement.

Where Prior Approaches Fall Short

The dominant approach: optimizing SFT in isolation. Current practice in the post-training literature is to compare SFT methods purely on offline benchmarks and treat the RL stage as a black box that will magnify any offline advantage. The paper demonstrates that this implicit assumption fails systematically. The authors tested a broad range of objectives from recent work, all evaluated at their recommended hyperparameters (Appendix E), and found that their relative performance after RL bears essentially no relationship to their relative performance after SFT.

Alternative offline metrics don't help. Some recent work has suggested that rather than looking at simple pass@1 after SFT, one should look at other offline descriptors that might better predict RL performance: pass@K with very large K (Yue et al., 2025) or majority voting accuracy (Kang et al., 2025). The paper explicitly tests these alternatives in Figure 1 and Figure 11 and finds that they also fail to reliably preserve the ranking: methods beneficial for offline majority voting can be harmful for online RL, and vice versa. The diagnostic that selects the best SFT method remains elusive using any straightforward offline metric.

Stabilization heuristics don't target the right thing. Several recent works propose modifications to SFT to improve stability—probability-based objectives for smoothness (Li et al., 2025a), entropy-regularized distribution matching to reduce forgetting (Diao et al., 2026), token/sample reweighting to suppress destructive gradients (Sanyal et al., 2025; Lin et al., 2025), and various importance-weighted or proximal SFT methods that constrain KL/entropy drift (Wu et al., 2025; Zhu et al., 2025d;a). These methods are typically designed with a "stabilization" or "trust-region" motivation: they aim to prevent SFT from overfitting, forgetting general capabilities, or diverging too far from the base model distribution. While these are reasonable goals, they do not directly address the alignment between the offline training objective and the online RL objective. The paper argues in Appendix G that common stabilization signals like smaller KL divergence or sparser updates "primarily measure conservatism, not actually an accurate 'mismatch correction'." In fact, PEAR often produces larger KL divergence and denser updates than standard SFT (Figure 8-a, Figure 10), yet performs substantially better after RL. The goal is not to move less—it is to move in the right direction.

The behavior–target mismatch is recognized in offline RL but underexplored in LM post-training. In classical reinforcement learning, the problem of learning from logged data generated by one policy (the behavior policy, $\pi_\beta$ ) while optimizing a different policy (the target policy, $\pi_\theta$ ) is the central challenge of offline RL (Levine et al., 2020; Precup et al., 2000). The mismatch between state/action visitation distributions under $\pi_\beta$ versus $\pi_\theta$ can cause catastrophic overestimation, policy collapse, and poor generalization when the policy is deployed. Standard remedies include importance sampling (reweighting logged transitions by the likelihood ratio $\pi_\theta(a_t|s_t)/\pi_\beta(a_t|s_t)$ ), conservative value estimates, and behavior-regularized policy updates. In the language domain, several works have imported these ideas: treating LM responses as logged decision trajectories and applying offline policy optimization (Lanchantin et al., 2025; Wang et al., 2024; Snell et al., 2023), or using offline data to improve online RL efficiency (Li et al., 2025b; Zhang et al., 2025b).

However, these works largely focus on improving either the offline stage in isolation or the online stage in isolation. The specific problem of designing the offline stage to serve as an optimal initialization for a subsequent online phase—rather than as a standalone policy—is underexplored. The paper's key insight is that the SFT checkpoint should not imitate the behavior policy's actions uniformly, but should preferentially learn from trajectories and continuations that the target policy will actually revisit during subsequent rollout-based RL. This requires correcting for the behavior–target distribution mismatch at the loss level during SFT, not just during RL policy optimization.

Prior observations about SFT–RL tension. Independent lines of work have observed that SFT and RL exhibit different learning dynamics. Chu et al. (2025) find that SFT memorizes while RL generalizes; Jin et al. (2025b) characterize when RL outperforms SFT and vice versa; Shenfeld et al. (2025) connect RL's reduced forgetting to on-policy sampling bias. These analyses explain why SFT and RL behave differently, motivating the need for better integration, but they do not provide a concrete mechanism for addressing the mismatch at the SFT stage.

How This Paper Positions Itself

The paper frames the problem through the lens of off-policy evaluation (OPE)—a subfield of RL that deals precisely with the question: given trajectories sampled from a behavior policy $\pi_\beta$ , how can we estimate the expected performance of a different target policy $\pi_\theta$ ? The fundamental technique in OPE is importance sampling: reweighting logged trajectories by the likelihood ratio $\pi_\theta(\tau) / \pi_\beta(\tau)$ to unbias expectations. The paper adapts this idea to the SFT loss rather than to return estimation.

Instead of asking "how can we estimate $\pi_\theta$ 's return from $\pi_\beta$ 's data?" (the standard OPE question), the paper asks: "how can we train $\pi_\theta$ on $\pi_\beta$ 's data so that the gradient updates reflect what $\pi_\theta$ will actually encounter during on-policy RL?" The answer is to reweight each token's SFT loss by a suffix importance ratio: how likely the continuation after that token is under $\pi_\theta$ relative to $\pi_\beta$ . Intuitively, if the logged continuation after token $y_t$ is a dead end that the target policy would rarely explore, the loss on $y_t$ should be down-weighted. If the continuation is a promising path that $\pi_\theta$ will revisit, the token should receive full learning signal. This corrects the distribution mismatch at the granularity most relevant for the autoregressive generation setting, where prefixes determine which continuations are explored.

Crucially, PEAR is not proposed as a better SFT objective for standalone performance—in fact, it often underperforms standard SFT on offline metrics. It is proposed specifically as an initialization strategy for downstream RL. The paper thus reframes the goal of the offline stage:

"We contend that the goal of an offline stage is not merely strong offline accuracy, but an initialization that facilitates improvement under the online RL." (Section 1)

This is a subtle but significant shift in perspective. Rather than treating SFT and RL as two independent modules connected by a checkpoint file, the paper advocates for designing SFT objectives with the downstream RL dynamics in mind. PEAR is the first concrete instantiation of this principle for language model post-training, and the empirical results across four model sizes (0.6B to 8B), two model families (Qwen2.5/Qwen3, DeepSeek-distilled), and two task domains (logic games and math reasoning) provide compelling evidence that this perspective yields practical gains over the status quo of optimizing SFT in isolation.

3. Technical Approach

3.1 Reader Orientation

The paper is primarily an empirical analysis and method paper whose core idea is that the SFT loss should be reweighted by how likely each training example's continuation is under the current model—not to improve offline accuracy, but to better prepare the model for subsequent online RL by correcting the mismatch between the data-generating distribution and the distribution the RL policy will actually encounter during its own rollouts. PEAR itself is a lightweight reweighting wrapper applied to any standard per-token SFT loss (negative log-likelihood or knowledge distillation loss) that uses importance sampling ratios computed from the behavior policy and the current target policy, requiring no changes to the underlying training objective, optimizer, or architecture—only a modification to how each token's contribution to the total loss is weighted.

3.2 Big-Picture Architecture (Diagram in Words)

The system has four major components that operate during the offline SFT stage to produce a single checkpoint that will initialize the online GRPO stage:

Offline Dataset (D) — a fixed set of prompt–response pairs $(x, y)$ generated by a known behavior policy $\pi_\beta$ (typically a larger teacher model like Qwen3-8B). Each response is a token sequence $y = (y_1, \ldots, y_T)$ produced under prefixes induced by $\pi_\beta$ 's sampling.
Behavior Policy Probabilities — for every token position $t$ in every response, the log-probability $\log\pi_\beta(y_t | x, y_{<t})$ that the data-generating model assigned to the token under the given prefix. These are computed once, offline, and stored alongside the data. They do not change during training.
Target Policy ( $\pi_\theta$ ) — the current model being trained. At each training step, for every token being supervised, the model computes its own log-probability $\log\pi_\theta(y_t | x, y_{<t})$ . The ratio between $\pi_\theta$ and $\pi_\beta$ at each token—and across continuations—determines the reweighting.
PEAR Weighting Module — given the per-token log-ratios $\log(\pi_\theta/\pi_\beta)$ , this module computes a scalar weight $\hat{G}_t$ for each token using one of three aggregation strategies (sequence-level, block-level, or token-level suffix weighting), applies numerical stabilization (clipping, log-space computation), stops gradients through the weights, and multiplies the per-token SFT loss by the resulting coefficient.

Information flow: For each training batch, the model forward-passes each $(x, y)$ pair → per-token probabilities $\pi_\theta(y_t | x, y_{<t})$ are computed → per-token log-ratios $\delta_t = \log\pi_\theta(y_t|x, y_{<t}) - \log\pi_\beta(y_t|x, y_{<t})$ are formed → these are aggregated into weights $\hat{G}_t$ via one of the three strategies (suffix product, block product, or sequence product) → the per-token SFT loss $\ell_\theta(x, y_{<t}, y_t)$ is multiplied by $\text{sg}[\hat{G}_t]$ → the weighted sum is backpropagated. The behavior policy probabilities are never updated; the target policy probabilities update normally through the loss.

3.3 Roadmap for the Deep Dive

First, the formal definition of the PEAR-weighted objective (Section 3.2), which specifies the mathematical form of the reweighting and clarifies that gradients are stopped through the weights—this establishes the core computation.
Second, the concept of behavior–target mismatch and why uniform SFT loss is misaligned, including the connection to off-policy evaluation and importance sampling—this provides the theoretical motivation for why reweighting is necessary.
Third, the three PEAR variants (sequence-level, token-level suffix, block-level), their exact formulas, and what each one captures about the behavior–target mismatch—this is the core technical contribution.
Fourth, the numerical stabilization techniques (log-space computation, per-decision clipping, weight clipping, discount factor) that make the reweighting practical for long sequences—without these, importance weights over long horizons would have prohibitively high variance.
Fifth, the optional extension to negative examples and how repulsive gradients are weighted with sequence-level importance ratios.
Sixth, the concrete training configuration (hyperparameters, learning rates, dataset construction) that connects the method to the experimental results.

3.4 Detailed, Sentence-Based Technical Breakdown

The paper proposes a loss reweighting scheme for offline SFT that does not change the per-token objective (NLL or KL divergence) but instead multiplies each token's contribution by a scalar weight derived from the importance ratio between the current target policy $\pi_\theta$ and the fixed behavior policy $\pi_\beta$ that generated the training data. The weights are designed to down-weight tokens whose logged continuations are unlikely under the target policy—since those tokens lead into regions the model will rarely visit during subsequent on-policy RL—and up-weight tokens whose continuations remain plausible. The result is an SFT checkpoint that has concentrated its learning on prefixes and continuations that the online RL stage will actually encounter, making the initialization more amenable to further improvement.

The PEAR-Weighted Objective: Core Formulation

The method begins with a standard per-token training loss $\ell_\theta(x, y_{<t}, y_t)$ —this could be negative log-likelihood $\ell = -\log\pi_\theta(y_t|x, y_{<t})$ , knowledge distillation loss $\ell = \text{KL}(\pi_\beta(\cdot|x, y_{<t}) \|\pi_\theta(\cdot|x, y_{<t}))$ , or any other per-token objective. PEAR wraps this loss in a weighted expectation:

$L_{\text{PEAR}}(\theta) \triangleq \mathbb{E}_{(x,y)\sim\mathcal{D}} \left[ \sum_{t=1}^T \text{sg}[\hat{G}_t] \; \ell_\theta(x, y_{<t}, y_t) \right]$

where $\mathcal{D}$ is the offline dataset of prompt–response pairs, $T$ is the sequence length, $\hat{G}_t$ is the stabilized scalar weight for position $t$ , and $\text{sg}[\cdot]$ is the stop-gradient operator.

What it computes: For each training example, the model computes its per-token loss as usual (NLL or KD), multiplies each token's loss by a pre-computed fixed coefficient $\hat{G}_t$ , sums the weighted losses across the sequence, and takes the expectation over the dataset. The stop-gradient means that during backpropagation, $\hat{G}_t$ is treated as a constant—there is no gradient flowing through the weight computation itself, only through the base loss $\ell_\theta$ scaled by the weight.

Why this form: Stopping gradients through the weights is crucial because the weights depend on $\pi_\theta$ probabilities (through the log-ratio $\log\pi_\theta/\pi_\beta$ ), and allowing gradients through both the weight and the loss term would create a different (and incorrect) optimization objective—it would optimize $\pi_\theta$ to maximize the weight rather than to minimize the reweighted loss on the correct tokens. The stop-gradient isolates the reweighting as a pure coefficient that says "this token matters this much for the SFT objective" without the model trying to game the weight itself.

Why Uniform SFT Loss Is Misaligned: The Behavior–Target Mismatch

Standard SFT applies the same per-token loss to every token in every sequence, regardless of whether the model $\pi_\theta$ would actually produce that continuation during its own sampling. The formal problem is:

Behavior policy occupancy vs. target policy occupancy. The offline dataset is generated by some $\pi_\beta$ (the data-generating policy, typically a larger or differently-trained model). During online RL under GRPO, the model generates rollouts from its own current policy $\pi_\theta$ , which—after SFT—differs from $\pi_\beta$ . The prefixes (partial sequences) that $\pi_\theta$ actually visits during sampling are the prefixes it assigns high probability to, which may not be the same prefixes that appeared in the offline data. This is the prefix distribution shift or covariate shift problem well-documented in imitation learning (Ross et al., 2011): small early deviations compound over long horizons.

Concrete failure mode (Figure 3 intuition). Suppose the behavior policy frequently generates a token A followed by continuation B→C, because $\pi_\beta$ assigns high probability to this path (e.g., $\pi_\beta(B|A) = 0.9$ , $\pi_\beta(C|A,B) = 0.8$ ). However, the target policy $\pi_\theta$ —the model we are training and will use for RL—assigns much lower probability to this continuation (e.g., $\pi_\theta(B|A) = 0.1$ , $\pi_\theta(C|A,B) = 0.6$ ). Under uniform SFT, every occurrence of A, B, and C in the dataset receives full loss weight. This means the model learns to associate A with continuation B→C at full strength, even though during RL rollouts the model will almost never follow A with B→C. The learning on token A is wasted—or worse, it actively pushes the model toward a region it won't explore during RL, creating a conflict between what SFT reinforced and what RL will need to optimize. In long-form reasoning tasks where trajectories involve trial, backtracking, and self-correction, $\pi_\beta$ may over-represent dead-end continuations that $\pi_\theta$ will rarely revisit. PEAR down-weights these tokens so the model focuses its SFT capacity on the prefixes and continuations that align with its own policy.

Connection to Off-Policy Evaluation (OPE)

The theoretical foundation draws directly from off-policy evaluation in RL. Given trajectories $\tau$ from behavior policy $\pi_\beta$ , and a target policy $\pi_\theta$ , the expected reward (or any function) under $\pi_\theta$ can be estimated using the importance sampling identity:

$\mathbb{E}_{\tau \sim \pi_\theta}[f(\tau)] = \mathbb{E}_{\tau \sim \pi_\beta}\left[ \frac{\pi_\theta(\tau)}{\pi_\beta(\tau)} f(\tau) \right]$

where $\tau = (s_1, a_1, \ldots, s_T, a_T)$ is a full trajectory (states and actions), and the importance ratio $\pi_\theta(\tau)/\pi_\beta(\tau)$ corrects for the distribution shift.

What it computes: The identity says that if you have trajectories sampled from one distribution ( $\pi_\beta$ ) but want to estimate an expectation under a different distribution ( $\pi_\theta$ ), you can reweight each trajectory by the ratio of its probability under the two distributions. Trajectories that are more likely under $\pi_\theta$ than $\pi_\beta$ get weight > 1 (they are under-represented in the logged data), and those less likely get weight < 1 (they are over-represented). The reweighted empirical average then provides an unbiased estimate of the target expectation.

Why this form: The importance ratio acts as a Radon-Nikodym derivative or change of measure between the two distributions. For autoregressive policies, the trajectory probability factorizes as $\pi(\tau) = \prod_{t=1}^T \pi(a_t | s_t)$ , so the importance ratio also factorizes as a product of per-step probability ratios. This factorization is what PEAR exploits to compute token-level and suffix-level weights.

Extending from return estimation to loss weighting. PEAR adapts this idea from estimating expected returns to weighting the SFT loss. Instead of asking "what is the expected return of $\pi_\theta$ given $\pi_\beta$ 's trajectories?", PEAR asks "for each token in $\pi_\beta$ 's trajectories, how relevant is the continuation for $\pi_\theta$ 's future rollouts?" Specifically, for a token at position $t$ , the suffix continuation $y_{t:T}$ has probability $\pi_\theta(y_{t:T} | x, y_{<t})$ under the target policy and $\pi_\beta(y_{t:T} | x, y_{<t})$ under the behavior policy. The suffix importance ratio $\rho_{t:T} = \pi_\theta(y_{t:T} | x, y_{<t}) / \pi_\beta(y_{t:T} | x, y_{<t})$ tells us: given the prefix up to token $t$ , how much more or less likely is the remaining sequence under $\pi_\theta$ than $\pi_\beta$ ? If this ratio is small, the logged continuation is not something $\pi_\theta$ would generate, and the token's loss should be down-weighted.

Per-Token Log-Ratio Computation

For every token position $t$ in every sequence, PEAR first computes the per-token log-probability ratio:

$\delta_t = \text{clip}\left( \log\pi_\theta(y_t | x, y_{<t}) - \log\pi_\beta(y_t | x, y_{<t}), \; \ell_\Delta, \; u_\Delta \right)$

where $\pi_\theta(y_t | x, y_{<t})$ is the target model's probability for the correct token under the given prefix, $\pi_\beta(y_t | x, y_{<t})$ is the behavior policy's probability for the same token under the same prefix, and $[\ell_\Delta, u_\Delta]$ are clipping bounds applied in log-space.

What it computes: For each supervised token, we compute how much more or less likely the target model finds this token compared to the behavior policy, measured in log-probability. The clipping bounds (set to $[-0.08, 0.3]$ in experiments) prevent individual tokens from contributing extreme values that would destabilize downstream products. A positive $\delta_t$ means $\pi_\theta$ assigns higher probability to the correct token than $\pi_\beta$ did (the target finds this token more natural); a negative $\delta_t$ means the opposite.

Why this form: Working in log-space and clipping early prevents numerical issues that would arise from multiplying many raw probability ratios across long sequences. The per-token log-ratio $\delta_t$ is the building block for all three PEAR variants—the variants differ only in how they aggregate these $\delta_t$ values over the sequence to form final token weights $\hat{G}_t$ .

Variant 1: Sequence-Level Weighting

This is the simplest variant. For every token in a given sequence $(x, y)$ , all tokens receive the same weight equal to the importance ratio of the entire sequence:

$G_t^{\text{seq}} = \prod_{j=1}^T \Delta_j = \frac{\pi_\theta(y|x)}{\pi_\beta(y|x)} \quad \forall t \in \{1, \ldots, T\}$

where $\Delta_j = \exp(\delta_j) = \pi_\theta(y_j | x, y_{<j}) / \pi_\beta(y_j | x, y_{<j})$ is the per-token probability ratio (in linear space, before clipping/stabilization).

What it computes: The product of all per-token probability ratios across the entire sequence, giving a single scalar weight that is applied uniformly to every token in that sequence. If the sequence as a whole is 3× more likely under $\pi_\theta$ than $\pi_\beta$ , every token in the sequence gets weight 3. If it is 10× less likely under $\pi_\theta$ , every token gets weight 0.1.

Operational meaning: The uniform weight $G_t^{\text{seq}}$ estimates the global compatibility between the entire logged sequence and the target policy. It is equivalent to using the standard OPE importance weight $\pi_\theta(\tau)/\pi_\beta(\tau)$ from Rowland et al. (2020) and applying it uniformly to all tokens—a sequence-level change of measure. Sequences that are plausible under the current model receive full learning signal; sequences that the model finds implausible receive attenuated signal.

Why this form: It is the simplest possible importance correction and has lower variance than per-token weights because it aggregates over the entire sequence before forming a ratio. However, it cannot distinguish between a sequence that is mostly plausible but has one highly implausible suffix (where most tokens should still be learned) versus a sequence that is uniformly implausible. This granularity tradeoff is addressed by the token-level variant.

Variant 2: Token-Level Weighting Based on Suffix Ratios

This variant assigns each token a weight that depends only on the importance ratio of the continuation after that token—the suffix. For token at position $t$ , the weight is:

$G_t^{\text{suffix}} = \gamma^{T-t} \prod_{j=t+1}^T \Delta_j = \gamma^{T-t} \cdot \frac{\pi_\theta(y_{t+1:T} | x, y_{\leq t})}{\pi_\beta(y_{t+1:T} | x, y_{\leq t})}$

where $\gamma \in (0, 1]$ is a discount factor and the product $\prod_{j=t+1}^T \Delta_j$ is the likelihood ratio of the remaining suffix $y_{t+1:T}$ after position $t$ , conditioned on the prefix up to and including $y_t$ .

What it computes: For each token $y_t$ , we compute how likely the continuation $y_{t+1:T}$ is under the target policy relative to the behavior policy, given that we are at prefix $(x, y_{\leq t})$ . The factor $\gamma^{T-t}$ discounts the contribution of tokens far from the end of the sequence to control variance (long-horizon products have higher variance). The weight $G_t$ then tells us: "after generating $y_t$ , would $\pi_\theta$ continue along the same path as the logged data, or would it go somewhere different?"

Operational meaning and intuition: Consider token A from the earlier example. Even if the model assigns high probability to A itself (so $\delta_A$ is positive and large), the suffix weight evaluates whether the model would continue from A along B→C (the logged continuation). If $\pi_\theta(B|A)$ and $\pi_\theta(C|A,B)$ are low, the suffix product $\prod_{j=A+1}^T \Delta_j$ will be small, and $G_A$ will be small. This down-weights the loss on A because learning to assign high probability to A—when it leads into a region the model won't explore during RL—provides little useful signal for RL readiness. Conversely, if the model would naturally follow A with the logged continuation, the weight is high and the token receives full learning signal.

Connection to off-policy value estimation (Appendix B and C). The paper provides an alternative interpretation: the suffix weight $G_t$ can be viewed as a single-sample estimator of the action-value function $Q^{\pi_\theta}(s_t, a_t)$ —the expected future return (discounted terminal reward) starting from state $s_t$ after taking action $a_t$ under the target policy. In the terminal-reward setting where the reward $R(\tau)$ is observed only at the end of the sequence (1 for correct, 0 for incorrect), the importance-weighted estimator is:

$G_t(\tau) = \gamma^{T-t} R(\tau) \prod_{j=t+1}^T \frac{\pi_\theta(a_j|s_j)}{\pi_\beta(a_j|s_j)}$

which is exactly the suffix weight multiplied by the terminal reward. Uniform SFT corresponds to replacing $Q^{\pi_\theta}$ with a constant (every token gets equal credit), while PEAR replaces it with a return estimate that propagates terminal outcome back to early tokens (via $\gamma^{T-t}R$ ) and discounts suffixes that $\pi_\theta$ would not realize (via the importance ratio).

Why this form over single-step weighting: A natural alternative would be to weight each token only by the probability ratio of that single token— $w_t = \pi_\theta(y_t|x, y_{<t}) / \pi_\beta(y_t|x, y_{<t})$ —which the paper calls "Single Step IS" and tests as a baseline. This myopic objective only considers whether the current action is plausible under $\pi_\theta$ , ignoring whether the continuation after that action aligns with $\pi_\theta$ 's likely behavior. The paper shows empirically that single-step weighting is substantially less effective than suffix weighting (Figure 4a, Table 3), with up to 6% absolute degradation on math benchmarks, because what matters for RL readiness is not whether individual actions are plausible in isolation but whether successful continuations from the offline data are compatible with the current policy over the remaining horizon.

Variant 3: Block-Level Weighting for Stability

Multiplying probability ratios over very long sequences (common in reasoning tasks where solutions can span thousands of tokens) introduces high variance in the importance weights—a well-known issue in off-policy evaluation called the "curse of horizon" (Liu et al., 2018; 2020). A single token with a very small ratio (e.g., $\Delta_j = 0.01$ ) can drive the entire product to near-zero, making the weight for all preceding tokens vanish. To trade granularity for stability, PEAR introduces a block-level variant.

Block partitioning. The sequence positions $\{1, \ldots, T\}$ are partitioned into $K = \lceil T/B \rceil$ contiguous blocks $\{I_k\}_{k=1}^K$ , each of length at most $B$ (in experiments, $B=4$ is used). Let $e_k = \max I_k$ be the last index of block $k$ . Instead of computing suffix ratios at every token, the method aggregates within each block first and then applies suffix weighting at the block level.

Block-level computation. For each block $k$ , define the block-level product of token ratios:

$\Delta_k^{\text{blk}} = \prod_{t \in I_k} \Delta_t$

The suffix weight for block $k$ is the discounted product of all subsequent block ratios:

$S_k = \prod_{m=k+1}^K \Delta_m^{\text{blk}} = \prod_{j=e_k+1}^T \Delta_j$

which is exactly the importance ratio of the suffix after the last token of block $k$ . Then every token in block $k$ receives the same weight:

$G_t^{\text{blk}} = \gamma^{T-e_k} S_k \quad \forall t \in I_k$

What it computes: For a block size $B$ , the effective horizon for importance weight computation is reduced from $T$ to $\lceil T/B \rceil$ block-level products, reducing variance. All tokens within the same block share the same suffix weight, meaning the method assumes that tokens within a short span (up to $B$ positions) have similar suffix compatibility. When $B=1$ , this recovers the token-level variant exactly. When $B$ is large (e.g., $B=T$ ), it approaches the sequence-level variant (since there is only one block and $S_1 = 1$ , giving weight $\gamma^{T-e_1} \approx 1$ to all tokens).

Why this form: The block-level variant interpolates between the fine granularity of token-level suffix weighting (which can be high-variance on long sequences) and the stability of sequence-level weighting (which cannot distinguish between compatible and incompatible suffixes within a sequence). In practice, $B=4$ provides a useful compromise, as shown in Figure 6 where PEAR_{B=4} outperforms SFT and is competitive with token-level PEAR.

Discount Factor and Its Role

All suffix-based weight computations (token-level and block-level) include a discount factor $\gamma \in (0, 1]$ that multiplies the suffix importance ratio. The full weight for position $t$ in suffix mode is:

$G_t = \gamma^{T-t} \prod_{j=t+1}^T \Delta_j$

where $\gamma$ is set to $0.999$ in experiments.

What it computes: The discount exponentially attenuates the contribution of the suffix importance ratio based on the distance to the end of the sequence. For tokens near the end ( $t$ close to $T$ ), $\gamma^{T-t} \approx 1$ and the weight is dominated by the suffix ratio. For tokens very early in a long sequence, $\gamma^{T-t}$ can be small even if the suffix ratio is moderate, reducing the weight.

Why this form: From the off-policy evaluation perspective, $\gamma$ is the standard discount factor in RL that controls the effective horizon for credit assignment and reduces variance (Sutton & Barto, 2018; Jiang & Li, 2016). Without discounting ( $\gamma=1$ ), variance grows exponentially with sequence length. With $\gamma < 1$ , the effective horizon is roughly $1/(1-\gamma) \approx 1000$ tokens for $\gamma=0.999$ , which is sufficient to capture long-range dependencies in reasoning traces while preventing variance explosion from partial products at the very beginning of sequences.

Numerical Stabilization: Algorithm 1

The paper presents Algorithm 1 as the complete, numerically-stabilized implementation of PEAR. The stabilization strategy has five components:

1. Log-space computation. All importance ratios are computed in log-space to avoid multiplying potentially thousands of numbers near zero or infinity. The per-token log-ratio $\delta_t = \log\pi_\theta - \log\pi_\beta$ is computed and clipped, then block-level log-ratios $\rho_k = \sum_{t \in I_k} \delta_t$ are formed by summing (equivalent to log of the product). The suffix weight is computed by accumulating $\rho_k$ in a backward pass (lines 15–20 of Algorithm 1). The final weight is exponentiated only at the end: $\hat{G}_{e_k} = \text{clip}(\exp((T-e_k)\log\gamma + u), G_{\min}, G_{\max})$ where $u$ accumulates the future log-ratios.

2. Per-decision clipping. Each $\delta_t$ is clipped to $[\ell_\Delta, u_\Delta] = [-0.08, 0.3]$ before any aggregation. This means a single token can contribute at most an exponent of $0.3$ (ratio ≈ 1.35) or at minimum $-0.08$ (ratio ≈ 0.92) to the product. Extreme per-token ratios—which often occur from rare tokens or distributional artifacts—are truncated.

Why these bounds: The asymmetry ( $|u_\Delta| > |\ell_\Delta|$ ) allows tokens where the target finds the correct continuation much more likely than the behavior policy (positive $\delta$ ) to contribute more to the weight than tokens where the target finds it less likely (negative $\delta$ ). This is because down-weighting (negative ratios) reduces learning signal, and being too aggressive about down-weighting can starve the model of data. Conversely, being permissive about up-weighting amplifies the signal on tokens most compatible with the target policy.

3. Weight clipping. After exponentiation, the final weight $\hat{G}$ is clipped to $[G_{\min}, G_{\max}] = [\exp(-10), \exp(5)]$ , which bounds the linear weight to approximately $[4.5 \times 10^{-5}, 148.4]$ . This prevents any single token from dominating the batch loss (if weight is too high) or from being completely zeroed out (if weight is too low).

4. Backward scan for suffix accumulation. In suffix mode (lines 15–20 of Algorithm 1), the method scans backward from the end of the sequence, maintaining a running sum $u$ of future block log-ratios. At block $k$ , $u$ contains $\sum_{m=k+1}^K \rho_m$ , the log of the suffix importance product. The weight for the last token of block $k$ is $\hat{G}_{e_k} = \text{clip}(\exp((T-e_k)\log\gamma + u), G_{\min}, G_{\max})$ , and all tokens in that block share this weight. The running sum is then updated $u \leftarrow u + \rho_k$ to include the current block for earlier positions.

5. Stop-gradient through weights. As specified in the objective, all weights are wrapped in $\text{sg}[\cdot]$ , meaning the computational graph treats $\hat{G}_t$ as a constant during backpropagation.

Optional Extension: Incorporating Negative Examples

When the offline dataset contains verified incorrect trajectories $\mathcal{D}^- = \{(x, y^-)\}$ (responses that were checked against ground truth and found wrong), PEAR can optionally add a repulsive gradient that pushes the model away from these trajectories. The negative loss is:

$L_{\text{neg}}(\theta) \triangleq \mathbb{E}_{(x,y^-)\sim\mathcal{D}^-} \left[ -\lambda \; \text{sg}[\hat{G}^-_t] \sum_{t=1}^T \ell_\theta(x, y^-_{<t}, y^-_t) \right]$

where $\lambda$ is a positive coefficient controlling the strength of repulsion, and $\hat{G}^-_t$ is a sequence-level importance weight on the negative trajectory (the same uniform weight applied to all tokens in the sequence).

What it computes: For each negative trajectory, the standard SFT loss (NLL) is computed—this loss measures how well the model predicts the incorrect tokens—and then the model performs gradient ascent (note the negative sign) on this loss, scaled by $\lambda$ and the sequence-level weight. If the negative trajectory is plausible under the current model (high $\hat{G}^-$ ), the repulsive force is strong; if the model already finds it implausible (low $\hat{G}^-$ ), the repulsive force is weak.

Why this form: Using token-level or suffix-level signed ratios on negative trajectories would be even more unstable than on positive trajectories—the importance ratios could have either sign and fluctuate violently. The sequence-level weight provides a stable global compatibility measure. The repulsive term is conceptually similar to unlikelihood training or DPO's negative gradient, but with policy-consistent weighting: only negative trajectories that the model might actually generate (high $\hat{G}^-$ ) receive strong repulsion, preventing wasted gradient effort on trajectories already far from the model's distribution.

Experimental configuration: Experiments in Section 4.5 subsample 50K positive examples and include 50K negative examples from the same behavior policy on the same instructions. The combination of positive PEAR reweighting (suffix-based) and negative sequence-level repulsion produces additional gains over positive-only PEAR (Figure 4a).

Training Configuration and Hyperparameters

The method operates within a standard SFT training loop with the following settings (Section 4.2):

Offline training. Models are trained for 1 epoch with learning rate $3 \times 10^{-5}$ on logic games and $1 \times 10^{-5}$ on math reasoning tasks. The base model for offline data generation is Qwen3-8B (the behavior policy $\pi_\beta$ ), which generates responses for the training prompts. Both the generated responses and the behavior policy's per-token log-probabilities are stored in the offline dataset.

PEAR hyperparameters. The discount factor is $\gamma = 0.999$ . The per-decision log-ratio clipping bounds are $[\ell_\Delta, u_\Delta] = [-0.08, 0.3]$ . The final weight clipping bounds are $[G_{\min}, G_{\max}]$ corresponding to log-range $[-10, 5]$ (so $\hat{G}_t \in [e^{-10}, e^5] \approx [4.5\times10^{-5}, 148]$ ). For block-level variants, $B=4$ is the standard block size.

Online RL (post-SFT). All SFT checkpoints, regardless of their training objective, undergo identical GRPO training (Shao et al., 2024) with learning rate $10^{-6}$ , batch size 128, and KL coefficient 0.01. The GRPO configuration is held fixed across all methods to isolate the effect of the SFT initialization.

Design choice: No additional training overhead. PEAR's only additional computation during SFT is the per-token log-probability subtraction ( $\log\pi_\theta - \log\pi_\beta$ ), the backward accumulation of log-ratios (a single $O(T)$ scan), and the weight multiplication. Since $\log\pi_\theta(y_t|x, y_{<t})$ must already be computed for the SFT loss forward pass, and $\log\pi_\beta(y_t|x, y_{<t})$ is pre-computed and stored offline, the overhead is essentially a few element-wise operations per token per step—negligible compared to the forward/backward pass through the model. When combined with knowledge distillation, the log-ratios can reuse the log-probabilities already computed for the KL divergence, adding "minimal overhead to the KL-based KD baseline" (Section 4.3).

Summary: What PEAR Does and Does Not Do

PEAR does not change the SFT objective, architecture, optimizer, or data. It adds a scalar multiplicative coefficient to each token's loss contribution, computed from a stop-gradiented importance ratio between the training model and the data-generating model. This coefficient down-weights tokens whose logged continuations are unlikely under the current model, focusing the SFT gradient signal on prefixes and continuations that align with the model's own distribution—the same prefixes it will actually visit during on-policy RL rollouts. The method does not aim to improve offline SFT accuracy (and often does not), but rather to produce a checkpoint whose learned behaviors are concentrated in regions the RL stage will naturally explore, making subsequent GRPO more efficient. The three variants (sequence, block, token-level suffix) trade off between granularity of mismatch correction and variance of the importance weights, with token-level suffix weighting being the primary variant used in experiments.

4. Key Insights and Innovations

Innovation 1: Reframing SFT as an RL Initialization Problem Rather Than an Offline Optimization Problem

The paper's most fundamental conceptual move is to redefine what the SFT stage is for. Prior to this work, the dominant framing—implicit in virtually all SFT research from probability-based objectives (Li et al., 2025a) to adaptive reweighting (Qin & Springenberg, 2025) to KL-regularized distillation—was that SFT should produce the most accurate model possible on the offline data, and that stronger offline performance would naturally translate to stronger post-RL performance. This was an assumption about modularity: SFT and RL were treated as independent stages connected only by a checkpoint file, and optimizing each stage separately was assumed to optimize the pipeline.

This paper demonstrates that this assumption is not just slightly wrong—it is systematically inverted in important cases. The finding in Figure 1 that TopLogP produces the best offline pass@1 but the worst post-RL performance on Qwen3-1.7B-Base is not a minor ranking perturbation. It is a rank reversal, where optimizing for the wrong objective (offline accuracy in isolation) actively harms the downstream objective (post-RL accuracy). The Pearson correlation of 0.09 between offline and online pass@1 (Figure 11) is not just low—it is effectively zero, meaning offline metrics carry essentially no information about which SFT method will produce the best post-RL model under the experimental conditions tested.

This reframing matters because it changes the optimization criterion at the SFT stage. Instead of asking "which objective minimizes validation loss on the offline data?", the paper argues we should ask "which objective produces a checkpoint that GRPO can most effectively improve?" These are fundamentally different optimization targets, and the paper provides compelling evidence that they pull in incompatible directions. The practical implication is that the entire methodology for developing and evaluating SFT techniques—comparing methods on offline benchmarks and selecting the best one—may be systematically selecting for properties that harm RL, making the community's evaluation protocol actively counterproductive.

The significance extends beyond a methodological critique. The paper is proposing a new principle for two-stage training pipelines: the offline stage should prioritize reproducible successes under the target policy that will be optimized online. This is not a specific algorithm but a design philosophy, and it connects to a broader insight from reinforcement learning: that off-policy data should be weighted by its relevance to the learning policy, not treated uniformly. Prior work on SFT modifications (Zhu et al., 2025d;a; Wu et al., 2025) focused on stabilizing SFT against overfitting or capability loss—prudent goals, but ones that implicitly accept the modular framing. PEAR abandons modularity entirely: the SFT loss is explicitly shaped by the model's own current policy, making the offline stage aware of its downstream purpose in a way that no prior SFT method attempted.

Innovation 2: Diagnosis That Offline–Online Policy Mismatch Explains the Performance Reversal

Having established that offline performance doesn't predict online performance, the paper goes further by providing a mechanistic explanation for why—and this diagnosis is itself a contribution. The explanation centers on prefix distribution shift: the behavior policy that generates offline SFT data induces a different distribution over prefixes (partial sequences) than the target policy that will be optimized during online RL. Under uniform SFT loss, every token in every logged sequence receives equal learning signal, regardless of whether the model would actually visit those prefixes during its own rollouts.

This diagnosis is not entirely novel in the abstract—covariate shift in imitation learning is well-known from Ross et al. (2011), and the behavior–target mismatch is a central concept in offline RL (Levine et al., 2020). What is novel is the concrete mapping of this concept onto the SFT→RL pipeline for language model reasoning. The paper identifies a specific, measurable mechanism: tokens whose logged continuations are implausible under the current model (because the model, even after partial SFT training, assigns low probability to the continuation path taken by the behavior policy) receive full training signal during SFT but lead into regions the model will never revisit during subsequent GRPO rollouts. This wasted learning capacity not only fails to help RL—it can actively harm it by reinforcing associations that conflict with the model's own sampling distribution.

The evidence for this diagnosis is multi-layered. The core empirical finding in Figure 1—that methods producing stronger offline checkpoints can produce weaker post-RL checkpoints—is consistent with the mismatch hypothesis: methods that more aggressively fit the offline data (like the hard-thresholded objectives that train only on high-confidence tokens) may actually amplify the mismatch by forcing the model to imitate continuations that diverge from its own policy. The gradient analysis in Figure 7 provides mechanistic support: PEAR-initialized models exhibit a smaller principal angle between offline and online GRPO gradients (~51°) compared to SFT (~53°) and KL-regularized variants (up to ~56°), suggesting that PEAR's reweighting makes the offline learning direction more consistent with the online RL update direction. And the parameter drift analysis in Figure 8 shows that PEAR-initialized models undergo relatively more parameter change during offline training but less during online training (smaller NSS between online and offline checkpoints), consistent with PEAR doing the "heavy lifting" of distribution correction during SFT so that GRPO spends fewer updates correcting misalignments.

This diagnosis is significant because it provides a falsifiable causal model rather than just empirical correlation. The model predicts that any SFT method which more accurately weights tokens by their suffix compatibility with the current policy should reduce the offline–online gradient mismatch and show smaller parameter drift during RL—predictions that are borne out in the analysis section. It also explains why prior stabilization heuristics (KL constraints, sparsity-inducing objectives) don't reliably help: they measure conservatism, not alignment between offline and online learning directions.

Innovation 3: Importance Sampling on Continuations as an Offline-to-Online Bridge for Language Models

While importance sampling is a standard technique in off-policy RL for correcting distribution shift in return estimation, its application to loss reweighting during SFT for language model post-training is a genuinely novel synthesis. The paper's key technical insight is that the relevant quantity for RL readiness is not how likely each individual token is under the target policy (the single-step ratio tested as a baseline) but rather how likely the entire continuation after that token is—the suffix importance ratio.

This distinction between single-step weighting and suffix-based weighting is subtle but critically important, and the empirical evidence for its superiority is clear. Single-step importance weighting reweights each token by $\pi_\theta(y_t | x, y_{<t}) / \pi_\beta(y_t | x, y_{<t})$ , asking: "does the target model find this particular action plausible given the prefix?" The suffix weighting instead reweights by $\pi_\theta(y_{t:T} | x, y_{\leq t}) / \pi_\beta(y_{t:T} | x, y_{\leq t})$ , asking: "given that the model takes this action, would it then continue along the same path as the logged data, or would it diverge?" In the autoregressive generation setting, this captures the compounding effect of prefix shift: a token that is plausible in isolation may lead into a region that the model's own policy would never explore, making the learning on that token largely irrelevant for downstream RL.

The Single Step IS baseline in Figure 4a and Table 3 provides the direct comparison. On Qwen3-1.7B, single-step weighting achieves 18% average accuracy on math (vs. 23% for SFT)—a 5-point degradation. In contrast, PEAR's suffix weighting achieves 26%—a 3-point gain over SFT. On Qwen3-8B, single-step weighting drops to 30% (vs. 36% for SFT, a 6-point degradation), while PEAR reaches 41%. These are not marginal differences; single-step weighting is actively harmful compared to uniform SFT, while suffix weighting is beneficial. This demonstrates that the horizon-aware nature of the correction—looking forward at the continuation, not just backward at the prefix—is the operative mechanism, not just the presence of any importance ratio.

The intellectual contribution here is not the mathematics of importance sampling itself (which is standard), but the identification of the suffix as the correct unit of analysis for offline-to-online transfer in autoregressive models. Prior work applying offline RL to language models (Snell et al., 2023; Richemond et al., 2024; Baheti et al., 2024) focused on optimizing policies from offline data using conservative value estimates or behavior regularization—techniques designed for the offline phase treated as a complete training process. PEAR instead provides a lightweight, loss-level correction specifically designed for the offline-to-online transition, where the offline stage is not the final product but a preparatory step. This is a different use case and a different technical solution, filling a gap between pure offline optimization (which produces a final policy) and pure online optimization (which starts from scratch). The innovation is in targeting the initialization quality for subsequent RL rather than the standalone policy quality from SFT.

Innovation 4: The Curse of Horizon in Importance-Weighted SFT and the Block-Level Solution

A well-known challenge in off-policy evaluation is the curse of horizon: importance weights formed as products of per-step ratios over long trajectories have exponentially growing variance (Liu et al., 2018; 2020). In the standard OPE setting, this manifests as estimators with high mean-squared error on long-horizon tasks, requiring techniques like per-decision importance sampling, stationary distribution corrections, or doubly-robust estimators. The paper identifies that this same challenge arises in the SFT→RL setting—language model reasoning traces can span hundreds or thousands of tokens—and that naive suffix importance weighting can become unstable due to the multiplicative accumulation of per-token ratios.

The block-level PEAR variant (Section 3.5) directly addresses this by partitioning the sequence into contiguous blocks and computing suffix weights at the block level rather than the token level. This reduces the effective horizon for importance weight computation from $T$ (the full sequence length) to $\lceil T/B \rceil$ (the number of blocks), trading granularity for stability. The design is reminiscent of state-based importance sampling in RL (Bossens & Thomas, 2024), which aggregates ratios over groups of actions to reduce variance, but adapted to the autoregressive token-generation setting where the "state" is the prefix and "actions" are tokens.

The empirical comparison in Figure 6 shows that block-level PEAR ( $B=4$ ) is competitive with token-level PEAR ( $B=1$ ) on SynLogic games, with both substantially outperforming standard SFT. On Qwen3-1.7B, token-level PEAR achieves 63.7% pass@1 / 83.8% pass@8, while block-level PEAR achieves 63.7% pass@1 / 82.5% pass@8, compared to SFT's 42.5% / 70.0%. This near-equivalence is encouraging because it suggests that the method is robust to the choice of granularity—the benefits of suffix-aware reweighting do not critically depend on per-token precision, and stability-oriented aggregation does not sacrifice the core signal. This robustness property is practically important because it means PEAR can be applied to very long reasoning traces without hyperparameter-sensitive tuning of the block size.

The significance of this insight extends beyond PEAR itself. As reasoning models generate increasingly long chains of thought (the DeepSeek-R1 and o1 models regularly produce multi-thousand-token reasoning traces), variance in any importance-weighted method will become a first-order concern. The block-level strategy provides a template for how to handle this: identify the natural granularity at which the importance ratio remains informative (a block of 4-8 tokens may capture a meaningful reasoning step) and compute weights at that level rather than at the raw token level. This is not a solved problem—the optimal block size likely depends on the task and model—but the paper provides the first concrete demonstration that block-level aggregation is viable in this setting.

Innovation 5: The Negative Result That Common Stabilization Heuristics Are Orthogonal to RL Readiness

Not all innovations are positive results. One of the paper's most intellectually important contributions is a diagnostic negative finding: the properties that existing SFT methods optimize for—small KL divergence from the base model, sparse parameter updates, conservative movement in weight space—are not merely insufficient for RL readiness but are orthogonal to it. Methods that produce "stable" SFT checkpoints by these traditional metrics can be worse RL initializations than methods that produce more aggressive, denser updates.

The evidence for this is stark and multilayered. Figure 8-a shows that PEAR leads to substantially larger offline parameter drift (measured by NSS) than standard SFT and most variants—it moves further from the base model during SFT. Simultaneously, Figure 8-b shows that PEAR leads to smaller online parameter drift during GRPO—it moves less during RL relative to its offline checkpoint. This inversion of the usual "stability" pattern suggests that PEAR's aggressive offline movement is not random drift but targeted correction: it moves the model in directions that align with what GRPO would otherwise have to fix, so GRPO spends fewer updates undoing SFT-induced misalignments. The standard advice to "keep SFT conservative to leave headroom for RL" (implicit in KL-regularized SFT and proximal methods) may be actively counterproductive if it prevents the model from making the necessary distribution corrections during the cheaper offline stage.

The gradient analysis in Figure 7 reinforces this: methods with stronger KL constraints (higher β) show larger principal angles between offline and online gradients—they are moving in less aligned directions, not more. The intuition is that KL penalties pull the model toward the behavior policy's distribution, which is precisely the distribution that RL will need to move away from during on-policy optimization. A strong KL penalty during SFT may make the checkpoint look "stable" (close to the base model) but leaves more misalignment for RL to correct, increasing the effective workload during the expensive online phase.

This finding challenges the prevailing "stabilization" narrative in the SFT literature. Works like Proximal SFT (Zhu et al., 2025d), Anchored SFT (Zhu et al., 2025a), and KL-regularized distillation are motivated by the concern that unconstrained SFT causes catastrophic forgetting, capability loss, or destructive gradient interference. While these concerns may be valid for the SFT stage in isolation, the paper's results suggest that the SFT stage should be evaluated not by its stability but by its alignment with subsequent RL optimization. A checkpoint that "looks good" by convergence metrics (low loss, close to base, sparse updates) may be a poor RL initialization, while a checkpoint that "looks aggressive" (high drift, dense updates, large KL from base) may be an excellent one—if the drift is in the right direction.

The practical implication is that the community needs new diagnostics for SFT checkpoints that measure RL readiness rather than SFT quality. The paper explores several candidates in Appendix A—offline KL, update sparsity, spectrum shift, gradient rotation—and finds that none of them reliably predict post-RL performance as a single scalar (Figure 9). The strong correlation between offline and online gradient alignment (Figure 7) is suggestive but requires a pilot RL run to compute, defeating the purpose of an offline diagnostic. Developing cheap, reliable "RL readiness" metrics remains an open problem that this paper's negative results make urgently visible.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper uses two task domains: (1) SynLogic Games — synthetic, verifiable logic puzzles from SynLogic (Liu et al., 2025) and Enigmata (Chen et al., 2025b), with procedurally generated puzzle instances, deduplicated prompts across train/test splits, and a held-out evaluation set with original verifiers. The offline buffer contains roughly 100,000 correct trajectories generated by Qwen3-8B. (2) Math Reasoning — the SYNTHETIC-2 dataset (Prime Intellect, 2025), using a subset of 33,400 unique math instructions, with 100,000 question-response pairs sampled from Qwen3-8B and verified by final answer. For online RL, the DAPO-17k dataset (Yu et al., 2025) is used. Evaluation benchmarks include MATH-500 (Hendrycks et al., 2021), MINERVA (Lewkowycz et al., 2022), AIME-2024, AIME-2025, and AMC-2023 (Balunović et al., 2026).
Base model(s). The paper tests across six models spanning multiple families and scales: Qwen3-Base at 0.6B, 1.7B, 4B, and 8B parameters (Yang et al., 2025); Qwen2.5-1.5B-Math (Yang et al., 2024); and DeepSeek-Distill-Qwen-1.5B (Guo et al., 2025). The range from 0.6B to 8B allows testing whether the benefits of PEAR scale with model size. The diversity of base models (Qwen2.5, Qwen3, DeepSeek-distilled) tests robustness across model families. The off-the-shelf models (Qwen2.5-Math, DeepSeek-distilled) represent different pretraining and distillation paradigms, strengthening the generality claim.
Metrics. On SynLogic games, the paper reports Pass@1 and Pass@8 on a held-out set of puzzles evaluated with the original verifiers. On math benchmarks, the paper reports average accuracy across 64 samples (Avgs@64) to reduce variance from sampling, along with Pass@K (Pass@1, Pass@8, and sometimes Pass@64). For the offline vs. online analysis in Section 2, additional offline metrics include Pass@128 and majority voting accuracy. All results are reported as percentages. Parameter drift during training is quantified via Normalized Spectrum Shift (NSS) (Zhu et al., 2025c), which measures the normalized distance between singular value spaces of weight matrices before and after training. Gradient alignment between offline and online phases is measured via average principal angle between gradient subspaces computed over 10 pilot training steps.
Baselines. The paper compares against a comprehensive set of SFT objectives (Table 1): Standard SFT (NLL) with negative log-likelihood; SFT+KL with KL regularization coefficients β ∈ {0.03, 0.1, 0.3, 1} against a reference model; GeneralFamily-α from Li et al. (2025a), with α = 1 corresponding to the 1−p objective; TopP-q and BottomP-q, which threshold by token probability p and train only on tokens above or below quantile q; TopLogP-q and BottomLogP-q, the same thresholding applied to negative log-likelihood; and TALR (Token-Adaptive Loss Reweighting) from Qin & Springenberg (2025), which reweights tokens by an exponential function of their NLL. For PEAR specifically, the paper also tests Single Step IS, a baseline that uses only the per-token probability ratio πθ(yt) / πβ(yt) without considering the suffix continuation, and tests PEAR applied to KL-based knowledge distillation as well as vanilla NLL. All methods use their recommended hyperparameters (Appendices E.1, E.2). For online RL, identical GRPO training is applied to all checkpoints, making the comparison purely about SFT initialization quality.
Generation budget / compute accounting. The paper does not use a formal FLOPs accounting as in pretraining–inference tradeoff papers. Instead, the comparison is structured around the fixed RL budget: all SFT checkpoints undergo identical GRPO training with the same number of steps, batch size (128), and rollout budget. The offline data budget is fixed at 100,000 trajectories for both games and math, with 1 epoch of training. The key comparative unit is the online RL performance at the end of identical training, isolating the effect of the SFT initialization. PEAR's additional overhead is the per-token log-probability subtraction and backward accumulation, described as "negligible" relative to the forward/backward pass. In the negative example experiment (Section 4.5), the data budget is balanced: 50K positive + 50K negative trajectories from the same instructions.
Cross-validation / statistical protocol. The paper does not employ formal cross-validation for strategy selection (unlike the reference example paper which used two-fold CV within difficulty bins). Instead, the logic games evaluation uses a deduplicated held-out set with removal of any train instances overlapping with evaluation prompts, providing a contamination-free evaluation. For math benchmarks, the paper uses standard public evaluation sets (MATH-500, AIME, AMC) with no explicit mention of cross-validation or statistical significance testing. Parameter drift metrics (NSS, gradient rotation) are computed over 10 pilot training steps and averaged. The offline–online correlation analysis in Figure 11 and Appendix H reports Pearson correlation coefficients for 8–11 model variants per metric. The paper does not report confidence intervals or error bars for the main performance numbers.

Main Quantitative Results

Offline Performance Does Not Predict Online RL Performance (Section 2) — The Foundational Negative Result

The paper first establishes the motivating finding: stronger offline SFT checkpoints do not reliably produce stronger post-RL models, and in some cases, the ranking reverses. On Qwen3-1.7B-Base trained on SynLogic games (Table 6, Figure 1, Figure 11):

PEAR achieves 32.5% offline pass@1 (tied for highest) and 45% online pass@1 (tied for highest) — the best overall RL initialization.
TopLogP achieves 20% offline pass@1 (competitive) but only 10% online pass@1 — the worst among all tested methods, underperforming standard SFT's 30% online pass@1 by 20 percentage points, despite having stronger offline pass@1.
Standard SFT achieves 22.5% offline pass@1 and 30% online pass@1 — a substantial 7.5-point online improvement over TopLogP, even though offline it was only 2.5 points better.
SFT-KL(β=0.03) achieves only 10% offline pass@1 (low) but 42.75% online pass@1 — the second-highest online performance, dramatically outperforming its offline ranking.
TALR achieves 8.75% offline pass@1 (among the lowest) but 13.75% online pass@1 — still poor, but the offline-to-online ranking shift is substantial.

On Qwen3-4B-Base (Table 7): PEAR achieves 32.5% offline pass@1 and 63.75% online pass@1 (best); BottomP achieves 32.5% offline pass@1 (tied) but only 38.75% online pass@1 (a 25-point degradation relative to PEAR online). The Pearson correlation between offline pass@1 and online pass@1 is 0.09 on Qwen3-1.7B (Figure 11, left panel) — effectively zero. Alternative offline metrics fare no better: offline pass@128 has Pearson r = −0.67 with online pass@1 (negatively correlated!), and offline majority vote accuracy has r = −0.42 (Figure 11, bottom panels). These negative correlations mean that higher offline majority voting accuracy is associated with lower online performance — the opposite of what the modularity assumption predicts.

This finding is replicated qualitatively across model sizes in Figure 1, showing rank reversals and line intersections between offline and online pass@1 for 1.7B and 4B models. The paper explicitly states:

"stronger offline performance is an unreliable proxy for post-RL performance: objectives that dominate after SFT can be overtaken after identical RL, producing substantial rank reversals" (Section 6)

The implications for evaluation methodology are severe: the standard practice of comparing SFT methods on offline metrics alone is selecting for properties that may actively harm downstream RL performance.

PEAR Improves Post-RL Performance on Logic Games Across Model Sizes (Section 4.3, Figure 4a)

The primary positive result on SynLogic games compares PEAR (token-level suffix weighting) against standard SFT and the Single Step IS baseline, all followed by identical GRPO training. Results in Figure 4a:

Qwen3-1.7B-Base: SFT+GRPO achieves 42.5% pass@1; PEAR (B=1)+GRPO achieves 63.7% pass@1 (a 21.2-point absolute gain). Single Step IS+GRPO achieves 45.0% (only 2.5 points above SFT). For pass@8, SFT+GRPO is 70.0%; PEAR+GRPO is 83.8% (13.8-point gain); Single Step IS+GRPO is 70.0% (no gain over SFT).
Qwen3-4B-Base: SFT+GRPO achieves 45.0% pass@1; PEAR+GRPO achieves 63.7% pass@1 (18.7-point gain). Single Step IS+GRPO achieves 60.0% (15-point gain over SFT, competitive with PEAR at this scale). For pass@8, SFT+GRPO is 67.5%; PEAR+GRPO is 82.5% (15-point gain); Single Step IS+GRPO is 67.5% (no pass@8 gain over SFT despite the pass@1 improvement).
Qwen3-8B-Base: SFT+GRPO achieves 28.7% pass@1; PEAR+GRPO achieves 73.8% pass@1 (a striking 45.1-point absolute gain). For pass@8, SFT+GRPO is 41.2%; PEAR+GRPO is 78.8% (37.6-point gain).
Qwen3-0.6B-Base (smallest model): SFT+GRPO achieves 6.2% pass@1; PEAR+GRPO achieves 8.8% pass@1 (a modest 2.6-point gain). For pass@8, SFT+GRPO is 15.0%; PEAR+GRPO is 23.8% (8.8-point gain). The gains exist but are smaller for the smallest model, suggesting PEAR's benefit may be conditioned on the base model having sufficient capacity to learn the distribution correction.

Single Step IS compared to PEAR. The myopic baseline (weighting by per-token ratio only) shows dramatically different behavior: on 1.7B, it provides minimal improvement over SFT at pass@1 (45% vs. 42.5%) and zero improvement at pass@8 (both 70%); on 8B, it actually degrades performance to 40% pass@1 vs. SFT's 28.7% (still an improvement, but far smaller than PEAR's 73.8%). This demonstrates that the horizon-aware suffix weighting is the critical mechanism, not just the presence of any importance ratio.

PEAR with negative examples. Adding 50K negative trajectories with sequence-level repulsive weights (Section 4.5, Figure 4a) further improves performance: on Qwen3-1.7B, PEAR+negative+GRPO achieves 70.0% pass@1 (6.3 points above PEAR positive-only at 63.7%) and 86.7% pass@8 (2.9 points above PEAR positive-only at 83.8%). On Qwen3-8B, PEAR+negative+GRPO achieves 77.5% pass@1 (3.7 points above PEAR positive-only at 73.8%) and 81.2% pass@8 (2.4 points above positive-only). The gains are consistent but modest relative to the positive-only PEAR improvement over SFT, suggesting that the suffix reweighting captures most of the benefit and negative examples provide a small additional signal.

PEAR applied to knowledge distillation. Figure 4b shows that PEAR's suffix weighting works when the underlying per-token loss is KL divergence (knowledge distillation from the behavior policy) rather than NLL. On Qwen3-1.7B: KD+GRPO achieves 52.5% pass@1 / 62.5% pass@8; PEAR+KD+GRPO achieves 70.0% pass@1 / 78.8% pass@8 (gains of 17.5 and 16.3 points respectively). On Qwen3-4B: KD+GRPO achieves 55.0% pass@1 / 77.5% pass@8; PEAR+KD+GRPO achieves 73.8% pass@1 / 86.7% pass@8 (gains of 18.8 and 9.2 points). This underscores PEAR's generality: the reweighting mechanism works across both NLL and KD objectives.

PEAR Improves Post-RL Performance on Math Benchmarks Across Model Families (Section 4.3, Table 2, Table 3)

The paper evaluates PEAR-initialized GRPO on standard math reasoning benchmarks across four model configurations. Key headline results from Table 2:

Qwen2.5-1.5B-Math: On AIME-2025, SFT+GRPO achieves 8% Pass@8; PEAR+GRPO achieves 24% Pass@8 (16-point gain). On AIME-2024, 23% vs. SFT's 19% Pass@8 (4-point gain). On AMC-2023, 78% vs. SFT's 80% Pass@8 (a 2-point loss — notably the only Pass@8 metric where PEAR underperforms SFT across all model-benchmark pairs). Average Pass@1 across all benchmarks: 28% for SFT+GRPO vs. 33% for PEAR+GRPO (5-point gain). Average Pass@8: 51% vs. 55% (4-point gain). On MATH-500, the Avgs@64 metric shows 63% for SFT+GRPO vs. 70% for PEAR+GRPO (7-point gain in average accuracy across 64 samples).

DeepSeek-Distill-Qwen-1.5B: This distilled model shows the most dramatic gains. On AIME-2025, SFT+GRPO achieves 5% Pass@8 vs. PEAR+GRPO's 35% Pass@8 (a 30-point gain). On AIME-2024, SFT's 2% Pass@8 becomes 38% (36-point gain). On AMC-2023, 50% becomes 91% (41-point gain). On MATH-500, 68% becomes 94% Pass@8 (26-point gain). Average Pass@1: 14% vs. 39% (25-point gain). Average Pass@8: 32% vs. 65% (33-point gain). These gains are substantially larger than on the Qwen2.5-Math model, suggesting that PEAR's distribution correction may be particularly important when the base model is heavily distilled (and thus may have a larger mismatch between its own policy and the teacher's data-generating policy).

Qwen3-4B-Base: Average Pass@1 improves from 39% to 43% (4-point gain); average Pass@8 from 56% to 65% (9-point gain). Individual benchmark results show gains across the board: AIME-2024 Pass@8 from 25% to 40% (15-point gain); AIME-2025 Pass@8 from 21% to 35% (14-point gain); Olympiad Avgs@64 from 41% to 46% (5-point gain).

Qwen3-8B-Base: Average Pass@1 improves from 35% to 41% (6-point gain); average Pass@8 from 59% to 64% (5-point gain). On AIME-2024 Pass@64, SFT+GRPO achieves 35% vs. PEAR+GRPO's 41% (6-point gain); AIME-2025 Pass@8 is flat at 35% for both. The 8B model shows smaller relative gains than the 4B, and Pass@8 on AIME-2025 shows no benefit, suggesting possible ceiling effects at larger scales or on particularly difficult benchmarks.

Aggregated across the four model sizes (0.6B, 1.7B, 4B, 8B) on math benchmarks (Table 3): PEAR+GRPO consistently outperforms SFT+GRPO with gains of 3, 3, 5, and 5 percentage points in average accuracy respectively. In contrast, Single Step IS+GRPO shows negative gains: -1, -5, -1, and -6 points relative to SFT+GRPO. This systematic degradation of single-step weighting demonstrates that the suffix-aware reweighting is not merely a beneficial heuristic but is essential—myopic importance correction actively harms RL readiness.

PEAR Transfers to Different RL Task Distributions (Section 4.4, Table 4)

The paper tests whether PEAR's benefit is specific to the domain the offline data was drawn from, or whether the RL readiness transfers when the online RL is run on a different task distribution. Concretely, SFT is performed on SynLogic games, but GRPO is then run on the Enigmata training set (12.8K problems), with evaluation on held-out Enigmata tasks after deduplication. Results in Table 4:

Qwen3-0.6B-Base: SFT+GRPO: 8.4%; PEAR+GRPO: 13.1% (4.7-point gain).
Qwen3-1.7B-Base: SFT+GRPO: 13.1%; PEAR+GRPO: 38.3% (25.2-point gain — by far the largest relative improvement).
Qwen3-4B-Base: SFT+GRPO: 49.5%; PEAR+GRPO: 59.8% (10.3-point gain).
Qwen3-8B-Base: SFT+GRPO: 53.3%; PEAR+GRPO: 61.7% (8.4-point gain).

PEAR consistently provides a stronger initialization even when the RL distribution shifts, confirming that "the benefit of PEAR is not overfit to the offline domain; it transfers better to a shifted online RL distribution under identical RL compute" (Section 4.4). This is a strong robustness result: the distribution correction learned during SFT generalizes because it is not about memorizing the specific offline data continuations but about learning to weight tokens by suffix compatibility—a property that transfers across task distributions.

PEAR Variant Comparison: Token, Block, and Sequence Level (Section 4.3, Figure 6)

Figure 6 compares the three PEAR variants on SynLogic games:

Qwen3-1.7B-Base: PEAR(B=1, token-level) achieves 63.7% pass@1 / 83.8% pass@8; PEAR(B=4, block-level) achieves 63.7% pass@1 / 82.5% pass@8 (nearly identical); PEAR(seq, sequence-level) achieves 57.5% pass@1 / 68.3% pass@8 (somewhat lower); SFT achieves 42.5% pass@1 / 70.0% pass@8. All three PEAR variants substantially outperform SFT, with token and block level nearly tied.
Qwen3-4B-Base: PEAR(B=1) achieves 63.7% pass@1 / 82.5% pass@8; PEAR(B=4) achieves 60.0% pass@1 / 76.2% pass@8 (slightly lower); PEAR(seq) achieves 45.0% pass@1 / 80.0% pass@8 (lower pass@1 but competitive pass@8); SFT achieves 45.0% pass@1 / 67.5% pass@8.

The key finding is that sequence-level weighting, despite its simplicity (a single global weight per sequence), is surprisingly effective—it outperforms SFT by significant margins (e.g., 57.5% vs. 42.5% on 1.7B pass@1), though it trails the more granular token-level and block-level variants. Block-level weighting with B=4 is nearly as effective as token-level on the 1.7B model but shows a slight degradation on the 4B model, suggesting that at larger scales, the per-token granularity may matter more. The paper notes that "sequence-level weighing turns out highly effective despite its simplicity" (Section 4.3).

Ablation Studies and Robustness Checks

PEAR with knowledge distillation (Figure 4b): PEAR's suffix weighting is not tied to the NLL loss—it works equally well when the underlying objective is KL divergence. On Qwen3-1.7B, PEAR+KD improves pass@1 from 52.5% to 70.0% over KD-only (17.5-point gain); on Qwen3-4B, from 55.0% to 73.8% (18.8-point gain). This ablation establishes PEAR as a loss-agnostic reweighting wrapper.
Single-step vs. suffix weighting (Figure 4a, Table 3): The single-step IS baseline (weighting by πθ(yt) / πβ(yt) only) is systematically inferior to suffix-based PEAR, and in some cases worse than uniform SFT. On Qwen3-8B math, single-step IS drops to 30% average accuracy vs. SFT's 36% — a 6-point degradation. This ablation isolates the suffix horizon as the critical mechanism, not the presence of any importance ratio.
Block size sensitivity (Figure 6, comparing PEAR(B=1) vs. PEAR(B=4) vs. PEAR(seq)): The block-level variant with B=4 achieves performance very close to token-level (B=1) on Qwen3-1.7B (63.7% vs. 63.7% pass@1) and slightly lower on Qwen3-4B (60.0% vs. 63.7% pass@1). This suggests moderate robustness to granularity choice, with token-level being safest for larger models.
Negative examples (Figure 4a, Section 4.5): Adding 50K negative trajectories with sequence-level repulsive weights provides incremental gains: +6.3 pass@1 points on 1.7B, +3.7 points on 8B. The effect is additive to positive-only PEAR, suggesting that avoiding wrong continuations provides a small but real signal beyond reweighting correct ones.
Model size scaling (Figure 4a, Table 3): PEAR's benefit is not monotonic with model size. On the 0.6B model, gains are modest (+2.6 pass@1 on games, +3 average on math). On 1.7B and 4B, gains are largest (+21.2 and +18.7 pass@1 on games). On 8B, gains are very large on games (+45.1 pass@1) but more moderate on math (+5 average accuracy). This pattern suggests PEAR helps most at intermediate scales where there is sufficient capacity to learn the distribution correction and sufficient headroom for RL to benefit.
Cross-task transfer (Table 4): When RL is run on Enigmata tasks after SFT on SynLogic, PEAR still provides gains (+4.7 to +25.2 percentage points depending on model size), showing that the benefit is not an artifact of task-specific overfitting during SFT.
Offline performance of PEAR itself (Figure 1, Table 6): Notably, PEAR does not always dominate SFT on offline metrics. On Qwen3-1.7B, PEAR achieves 32.5% offline pass@1 vs. SFT's 22.5% (a gain), but on Qwen3-4B, PEAR achieves 32.5% vs. SFT's 10.0% (a larger gain). What matters is not PEAR's offline superiority but its online RL benefit, which is consistent across models even when offline differences vary. This supports the paper's central thesis that offline metrics are a poor proxy.
Parameter drift analysis (Figure 8): PEAR-initialized models show larger offline parameter drift (higher NSS between offline and base model) but smaller online parameter drift (lower NSS between online and offline model) compared to SFT and other variants. This inversion pattern is consistent across learning rates and KL coefficients. The paper interprets this as PEAR doing the distribution correction during the cheaper offline stage, reducing the workload for GRPO.
Gradient alignment analysis (Figure 7): The average principal angle between offline and online GRPO gradients is smallest for PEAR (~51.3°) compared to SFT (~52.5°), SFT+KL(β=0.03) (~55.2°), and stronger KL variants (up to ~56.3°). Higher KL penalties produce larger gradient misalignment with subsequent RL, suggesting that KL regularization may be actively harmful for RL readiness despite its stabilizing effect during SFT.
Correlation of offline diagnostics with online performance (Figure 9, Appendix A): The paper explores whether offline KL divergence from the base model, update sparsity, spectrum shift, or gradient rotation can predict post-RL performance. Figure 9 shows these diagnostics plotted against pass@8 for various methods, but no strong predictor emerges: PEAR does not cluster cleanly on any of these axes, and methods with similar diagnostic values can have very different online performance. The paper concludes that "a checkpoint that 'looks stable' by these proxies may still yield weak post-RL gains" (Appendix G), underscoring the difficulty of offline RL readiness prediction.

Critical Assessment

Claim 1: "Stronger SFT checkpoints can significantly underperform weaker ones after identical RL." Strongly supported. The evidence is comprehensive: Figure 1 and Table 6 demonstrate clear rank reversals across 11 SFT variants on two model sizes. The specific examples are stark—TopLogP with best offline pass@1 but worst online pass@1 on 1.7B, SFT-KL(β=0.03) with poor offline but excellent online. The near-zero Pearson correlation (r = 0.09) between offline and online pass@1 on the full set of 19 models in Figure 1 is a damning statistic. Even the alternative offline metrics tested (pass@128, majority vote) show negative correlations with online performance. However, the claim is demonstrated on a single task domain (SynLogic games) for the correlation analysis, and while the math results (Tables 2, 3) show PEAR outperforming SFT post-RL despite comparable offline performance, the full rank-reversal analysis with many variants is not replicated on math. The generality of the "offline doesn't predict online" phenomenon across reasoning domains beyond logic puzzles is partially but not fully established.

Claim 2: "PEAR consistently improves post-RL performance over canonical SFT across model sizes and families." Supported, with important qualifications. The gains on SynLogic games are large and consistent across 1.7B, 4B, and 8B models (Figure 4a), and the math results in Tables 2 and 3 show consistent improvement across all four model sizes (0.6B–8B) and across three model families (Qwen2.5-Math, DeepSeek-distilled, Qwen3-Base). The weakest result is Qwen3-8B on AIME-2025 where Pass@8 is flat at 35% for both SFT and PEAR — no gain. The 0.6B model on games shows only modest improvement (+2.6 pass@1). So the consistency claim holds across models and benchmarks but with variable magnitude. A genuine weakness: the paper does not test PEAR on any model larger than 8B or on any frontier-scale reasoning models (e.g., 70B+), which is where the community's interest in post-training optimization is most intense. The claim that PEAR works across model families is supported by Qwen vs. DeepSeek results, but these are all relatively similar transformer architectures — no test on non-Qwen/DeepSeek families.

Claim 3: "PEAR works by correcting the offline-to-online distribution mismatch, and suffix-based weighting is the critical mechanism." Supported through multiple converging analyses, though the causal chain has gaps. The superiority of suffix weighting over single-step IS (Figure 4a, Table 3) directly demonstrates that the horizon-aware correction is the operative mechanism. The gradient alignment analysis (Figure 7) provides mechanistic evidence: PEAR produces offline gradients more aligned with online GRPO gradients. The parameter drift analysis (Figure 8) is consistent with the mismatch-correction narrative: PEAR moves more during offline, less during online. However, the paper does not perform a direct manipulation that would prove the causal claim — for instance, an experiment showing that the specific tokens PEAR down-weights are exactly the tokens whose continuations are rarely visited during GRPO rollouts. The connection between suffix ratios and actual RL rollout statistics is asserted but not directly verified with trajectory-level analysis. The analysis in Figure 5 (PEAR's weight distribution vs. KL to base) shows that high-weight tokens are indeed updated more aggressively, but this shows that PEAR concentrates updates, not why those concentrated updates help RL.

Claim 4: "PEAR transfers to different RL task distributions." Supported but limited in scope. Table 4 shows transfer from SynLogic offline training to Enigmata online RL — both are synthetic logic puzzle domains with verifiable answers. This is a distribution shift but within the same broad task category (structured reasoning with ground-truth answers). A stronger test would be SFT on math → RL on logic, or vice versa. The transfer claim is therefore more precisely stated as "PEAR transfers to a different logic puzzle distribution" rather than "PEAR transfers across task families."

Genuine weaknesses in the experimental design:

No direct measurement of the behavior–target mismatch. The paper argues that PEAR's mechanism is to correct distribution mismatch, but it never actually measures the mismatch itself. A convincing experiment would show: (1) the overlap between prefixes visited in offline data vs. prefixes visited during GRPO rollouts from a standard SFT checkpoint (low), (2) the overlap for a PEAR checkpoint (higher), and (3) correlation between per-token PEAR weights and per-token visitation frequency during GRPO. Without this, the mechanism is inferred from downstream performance and gradient analysis, not directly validated.
The oracle behavior policy requirement is a practical limitation. PEAR requires storing the data-generating model's per-token log-probabilities. In many real-world scenarios (e.g., SFT on human-written or curated data without a known generator model), πβ is undefined or unknown. The paper acknowledges this implicitly by using Qwen3-8B as the known behavior policy, but does not discuss how PEAR would be applied when the behavior policy is human, a closed-source API, or an ensemble. This is not a flaw in the experimental results but a significant deployment constraint that is not addressed.
GRPO hyperparameters are fixed and not tuned. All comparisons use a single GRPO configuration (learning rate 10^-6, batch size 128, KL coefficient 0.01). It's possible that the rank reversal phenomenon is sensitive to RL hyperparameters — some SFT checkpoints might require different RL settings to realize their potential, and the fixed GRPO training could disadvantage certain initializations unfairly. The paper does not ablate RL hyperparameters or show that the PEAR advantage is robust to RL configuration choices.
No comparison to RL-only (no SFT) or to online-only baselines. The paper compares SFT→RL against PEAR→RL, but does not include a baseline where RL starts from the base model (no SFT at all). This would establish the absolute value of the SFT stage and contextualize the gains from PEAR relative to skipping SFT entirely. If RL from the base model performs comparably to SFT→RL (which is unlikely for complex reasoning but possible), the entire goal of improving SFT as an initialization would be called into question.
Single epoch of SFT training. All experiments use 1 epoch of SFT. It is unclear whether the offline–online mismatch and PEAR's benefit are specific to the early-training regime or would persist (or change character) with multiple epochs. Multiple SFT epochs could exacerbate the mismatch (more overfitting to the behavior policy's distribution) or could saturate the gains from PEAR's reweighting.
Small or medium-scale models only. The largest model tested is 8B parameters. The paper's findings are relevant to the contemporary frontier of reasoning model post-training (where models are often 70B+), but the experimental scale is substantially smaller. Whether PEAR's benefits scale to larger models is unknown, and the 8B results showing flat Pass@8 on AIME-2025 and diminishing relative gains are a cautionary signal.
Missing statistical rigor. The paper does not report confidence intervals, standard errors, or any measure of statistical significance for the performance numbers. With test sets like MATH-500 (500 questions), the 5–7 point average accuracy gains in Table 2 could have overlapping confidence intervals. The SynLogic evaluation sets are not described with explicit sizes, making it impossible to assess whether the pass@1 differences are statistically significant or could arise from sampling noise.

What would strengthen the paper:

A direct analysis of prefix/continuation overlap between SFT data and GRPO rollouts, correlating PEAR weights with actual rollout visitation frequencies.
An RL budget scaling experiment: does PEAR's advantage persist as the RL budget increases, or does it primarily accelerate early RL learning?
A formal statistical significance test or bootstrap confidence intervals on the main performance results.
A comparison against starting RL from the base model without SFT, to bound the absolute contribution of the SFT stage.
An experiment with multiple SFT epochs, testing whether the offline–online mismatch compounds or saturates.
Testing on models larger than 8B to assess scalability of the approach.
A discussion of how to apply PEAR when the behavior policy is unknown (e.g., for human-curated SFT data), which is the most common real-world deployment scenario.

6. Limitations and Trade-offs

The Behavior Policy Must Be Known and Its Token-Level Probabilities Must Be Stored

The assumption or constraint. PEAR fundamentally requires access to $\pi_\beta$ , the data-generating policy's per-token log-probabilities for every training example: the weight $\hat{G}_t$ is computed from $\delta_t = \log\pi_\theta(y_t | x, y_{<t}) - \log\pi_\beta(y_t | x, y_{<t})$ . The paper's entire experimental setup uses Qwen3-8B as a known, accessible behavior policy whose log-probabilities are collected once and stored alongside the offline dataset (Section 4.1.1). This is feasible when the SFT data is generated by an open-weight model that the practitioner controls. But in many real-world post-training scenarios—SFT on human-written responses, on data scraped from the web where the generating model is unknown, on outputs from closed-source API models (GPT-4, Claude) that do not expose token-level log-probabilities, or on mixtures of data from multiple unknown sources— $\pi_\beta$ is either entirely undefined or its per-token probabilities are inaccessible. The paper does not discuss this constraint anywhere in the main text, nor does it propose a method for approximating $\pi_\beta$ when it is unavailable.

The consequence. Without access to $\pi_\beta$ , PEAR cannot be applied at all—the core reweighting computation has a missing term. This is not a minor implementation detail but a hard requirement that excludes PEAR from the most common SFT data regimes. The vast majority of instruction-tuning datasets (OpenHermes, ShareGPT, UltraChat, etc.) consist of responses from models whose per-token log-probabilities were not recorded or are inaccessible. Even when the source model is known, practitioners may not have stored log-probabilities at generation time. A natural workaround would be to estimate $\pi_\beta$ using the same model that generated the data (if available) or a surrogate, but this introduces estimation error in the importance weights, and the paper provides no analysis of how sensitive PEAR is to noise in $\pi_\beta$ 's probabilities. If $\pi_\beta$ is approximated by a different model than the true data-generating policy, the importance sampling correction could become biased in unknown ways—potentially making PEAR worse than uniform SFT.

What evidence exists in the paper. The paper provides no ablation or analysis of this constraint. Every experiment uses Qwen3-8B as the explicitly known behavior policy whose log-probabilities are pre-computed. The paper does not test a setting where $\pi_\beta$ is approximated, estimated from a different model, or corrupted by noise. The limitation is not acknowledged in Section 6 (Conclusion) or in the limitations discussion (there is no designated limitations section outside this one). This is an undocumented hard dependency that a practitioner reading the paper for deployment guidance would need to discover independently.

Mitigation status. Not addressed. The paper does not propose any method for applying PEAR when $\pi_\beta$ is unknown, does not discuss the sensitivity of results to $\pi_\beta$ approximation quality, and does not flag this as a direction for future work. This is a significant omission because it means PEAR, as described, is only directly applicable to a narrow sliver of SFT pipelines where the practitioner both controls the data generation model and has the foresight to store per-token log-probabilities—a combination that may be uncommon outside of research labs running fully controlled post-training pipelines.

No Evidence That the Behavior–Target Mismatch Is Directly Corrected

The assumption or constraint. The paper's central mechanistic claim is that PEAR works by correcting the distribution mismatch between the behavior policy $\pi_\beta$ (which generated the offline data) and the target policy $\pi_\theta$ (which online RL will optimize). The mathematical framing in Section 3 derives importance weights from the suffix likelihood ratio $\pi_\theta(y_{t:T} | x, y_{\leq t}) / \pi_\beta(y_{t:T} | x, y_{\leq t})$ , and the intuition in Figure 2 and throughout Section 2.2 is that uniform SFT over-weights continuations that $\pi_\theta$ will not revisit during on-policy rollouts. This is a causal claim: PEAR improves RL readiness because it down-weights tokens leading into implausible continuations and up-weights tokens leading into plausible ones, thereby aligning SFT learning with the prefixes GRPO will actually encounter. However, the paper validates this mechanism entirely through indirect proxies—gradient alignment (Figure 7), parameter drift (Figure 8), and weight-vs-KL distribution analysis (Figure 5)—rather than through any direct measurement of the claimed mismatch correction.

The consequence. Without direct evidence that PEAR's weights correlate with actual GRPO rollout visitation frequencies, the causal chain remains unverified. Several alternative explanations for PEAR's effectiveness are possible and not excluded by the experiments: (1) PEAR might simply be a form of adaptive difficulty weighting that happens to benefit RL, independent of any distribution correction; (2) the suffix-based reweighting might act as a regularizer that prevents overfitting to noisy or atypical examples, similar to methods like TALR or confidence-based filtering; (3) the benefits might come from the numerical properties of the weighting scheme (e.g., reducing loss magnitude on long sequences) rather than from importance sampling logic. The gradient alignment evidence (Figure 7) is suggestive but limited—it shows PEAR's offline gradients are more aligned with online GRPO gradients, but a 1.2° smaller principal angle (51.3° vs. 52.5°) is a small difference, and the measurement is taken over only 10 pilot steps. The parameter drift evidence (Figure 8) shows PEAR moves more offline and less online, but this could occur for many reasons unrelated to distribution matching (e.g., PEAR might simply produce a checkpoint that saturates early in RL, reducing further drift independent of alignment quality).

What evidence exists in the paper. The analysis in Figure 5 shows that high-weight tokens (as determined by PEAR's suffix ratios) undergo larger parameter updates (higher KL from base model) than low-weight tokens. This demonstrates that PEAR concentrates learning on the tokens it deems important—but it does not verify that those tokens are the ones corresponding to prefixes GRPO actually revisits. No experiment tracks which prefixes $\pi_\theta$ generates during GRPO rollouts and compares them to the prefixes in the offline data, nor does any experiment show that the per-token PEAR weights are correlated with future GRPO visitation frequency. The paper describes the off-policy evaluation connection in detail (Section 2.3, Appendix B and C) but stops short of the empirical verification that would close the loop: measuring whether the importance-sampling correction actually reduces the bias of SFT gradients relative to on-policy RL gradients.

Mitigation status. Partially addressed through indirect proxies. The gradient alignment analysis (Figure 7) and parameter drift analysis (Figure 8) provide converging but circumstantial evidence. The single-step IS ablation (Figure 4a, Table 3) shows that horizon-aware weighting is necessary—myopic weighting fails—which is consistent with the suffix-based distribution correction story but does not uniquely support it (other horizon-aware mechanisms might also work). The paper does not acknowledge the absence of direct mismatch measurement as a limitation or propose future experiments to close the gap. A convincing direct test—correlating PEAR token weights with GRPO rollout prefix visitation statistics—would require engineering effort (instrumenting the RL training loop to log prefix distributions) but is feasible and would substantially strengthen the mechanistic claim.

Tested Only on Relatively Small Models (0.6B–8B) and a Specific Reasoning Task Family

The assumption or constraint. The paper tests PEAR on models ranging from Qwen3-0.6B to Qwen3-8B, with additional experiments on Qwen2.5-1.5B-Math and DeepSeek-Distill-Qwen-1.5B. The largest model is 8B parameters. The task domains are synthetic logic puzzles (SynLogic, Enigmata) and competition math (MATH-500, AIME, AMC, MINERVA)—both belonging to the family of verifiable reasoning tasks with ground-truth answers. This is a coherent but narrow slice of the post-training landscape. Modern reasoning LM post-training pipelines routinely involve models at the 70B+ scale (e.g., DeepSeek-R1's base models, Qwen-72B, Llama-70B) and apply to more diverse domains including code generation, scientific reasoning, tool use, and open-ended instruction following. The paper's experimental scope, while covering multiple model sizes within the 0.6B–8B range, does not establish whether PEAR's benefits persist, diminish, or reverse at the scales where the community's practical interest is highest.

The consequence. The scalability of PEAR's benefit is genuinely uncertain. The paper's own results show mixed scaling behavior: on SynLogic games, the absolute gain from PEAR over SFT varies non-monotonically with model size—+2.6 pass@1 on 0.6B, +21.2 on 1.7B, +18.7 on 4B, +45.1 on 8B (Figure 4a). On math benchmarks (Table 3), the gains are 3, 3, 5, and 5 percentage points for 0.6B, 1.7B, 4B, and 8B respectively—essentially flat in absolute terms despite a 13× parameter increase. On AIME-2025 Pass@8 specifically, the 8B model shows zero gain from PEAR (35% for both SFT and PEAR, Table 2). This pattern does not inspire confidence that PEAR's benefit will increase or even persist at 70B+ scale. If the mechanism depends on the base model having sufficient capacity to learn the distribution correction while also maintaining task performance, then at very large scales the model may already implicitly handle the behavior–target mismatch (better-calibrated probabilities, more robust in-context learning), reducing PEAR's marginal benefit. Alternatively, large models might exhibit different mismatch patterns that require different correction strategies. The paper provides no theoretical or empirical basis for extrapolation.

The task domain limitation is equally significant. Verifiable reasoning tasks have a clean terminal reward signal (correct/incorrect) that the paper exploits conceptually—the suffix importance weight is interpreted as estimating $Q^{\pi_\theta}$ under a terminal reward (Appendix C). For tasks without such clean outcomes—open-ended dialogue, summarization, creative writing, multi-step planning with partial credit—the conceptual foundation weakens because there is no natural terminal reward to propagate backward. PEAR's weights are computed purely from model probabilities without any reward signal, so the method does not strictly require a terminal reward, but the intuition and the OPE connection are built on the reward propagation analogy. Whether PEAR's reweighting would be beneficial for non-reward-based RL (e.g., RLHF with learned reward models, constitutional AI) is untested.

What evidence exists in the paper. The scaling evidence is mixed—large gains at 8B on games coexist with flat pass@8 on AIME-2025 at the same model size. The transfer experiment (Table 4) shows that PEAR works when the RL task distribution shifts (SynLogic → Enigmata), but this shift is within the same task family (synthetic verifiable logic). The paper does not test PEAR on any non-math/non-logic task, on any model larger than 8B, or with any online RL algorithm other than GRPO. These gaps are not discussed as limitations.

Mitigation status. Not addressed. The paper does not acknowledge the scale limitation or the task domain limitation as constraints on the generalizability of the findings. The conclusion (Section 6) presents PEAR as a general method for "reasoning LLM post-training" without qualification about the model scales, task domains, or RL algorithms for which effectiveness has been demonstrated. A transparent statement of the tested regime and a call for larger-scale validation would help practitioners assess the relevance of the method to their own pipelines.

The Offline Stage Is Evaluated Only After a Single Epoch of SFT

The assumption or constraint. All experiments in the paper train the SFT stage for exactly 1 epoch on the offline dataset—100,000 trajectories for both SynLogic games (Section 4.1.1) and math reasoning (Section 4.1.2). This is stated as the training configuration (Section 4.2): "Unless stated otherwise, we train for 1 epoch." The paper does not provide any rationale for this choice, nor does it ablate the number of SFT epochs to test whether the offline–online mismatch and PEAR's benefit are sensitive to how long the SFT stage runs. This is a significant assumption because the dynamics of distribution mismatch likely depend on the degree of SFT optimization: a model trained for many epochs may more closely approximate the behavior policy's distribution (increasing the mismatch with its own subsequent rollouts), or may overfit to noise in the offline data (changing which continuations appear plausible), or may saturate in a way that leaves no headroom for RL regardless of initialization quality. A single-epoch regime is not necessarily representative of how SFT is typically deployed—many post-training pipelines use multiple epochs, especially on high-quality data.

The consequence. The paper's central finding—that optimizing SFT for offline performance is a poor proxy for RL readiness—may be partially an artifact of the 1-epoch regime. In early SFT training, different objectives can produce checkpoints at very different points on their optimization trajectories: some methods might converge faster (producing higher offline metrics after 1 epoch) but reach a lower-quality plateau, while others might converge more slowly and look worse at 1 epoch but surpass the faster methods with additional epochs. The rank reversal observed in Figure 1 could reflect differences in learning speed rather than fundamental misalignment between offline and online optimization. If extended SFT training causes the rankings to stabilize (or to reverse again), the paper's main argument—that offline performance is inherently an unreliable proxy for RL readiness—would need qualification: it might only be unreliable in the early-training regime, and with sufficient SFT training, offline metrics might become predictive. This is a critical unknown because practitioners making decisions about which SFT method to use need to know whether the rank instability is a transient early-training phenomenon or a persistent feature of the two-stage pipeline.

Relatedly, if multiple SFT epochs exacerbate the behavior–target mismatch (by driving the model closer to $\pi_\beta$ 's distribution), PEAR's benefit might actually increase with more epochs—the correction becomes more necessary as the model overfits to the behavior policy. Alternatively, if multiple epochs saturate SFT gains uniformly across methods, PEAR's relative advantage might shrink. The paper cannot distinguish between these scenarios.

What evidence exists in the paper. None. The paper provides no ablation over the number of SFT epochs, no learning curves showing offline or online performance as a function of SFT training duration, and no discussion of why 1 epoch was chosen. The experiment in Appendix K testing ReST $^{EM}$ , while not a direct epoch ablation, shows that additional sequential revision training can degrade performance—hinting that SFT training dynamics are non-trivial—but this is a different intervention (on-policy revision data, not multiple NLL epochs) and cannot substitute for a direct epoch sweep.

Mitigation status. Not addressed. The paper does not acknowledge the single-epoch regime as a limitation or discuss its implications for the generality of the findings. A straightforward experiment—varying SFT epochs from 1 to, say, 4, and measuring both offline performance and post-RL performance across multiple initializations—would substantially clarify whether the offline–online rank reversal is a robust phenomenon or a transient effect of early stopping. The paper's recommended practice (use PEAR during SFT) is implicitly conditioned on the 1-epoch setting, and practitioners who train SFT for longer cannot know from the paper's evidence whether PEAR remains beneficial or whether the offline–online correlation changes.

The GRPO Hyperparameters Are Fixed and Not Tuned Per Initialization

The assumption or constraint. The paper's core experimental design holds the online RL stage completely fixed: all SFT checkpoints undergo identical GRPO training with learning rate $10^{-6}$ , batch size 128, and KL coefficient 0.01 (Section 4.2). The goal is to isolate the effect of the SFT initialization by ensuring that "the comparison [is] purely about SFT initialization quality." This design is clean and well-motivated for the paper's primary question (does offline performance predict online performance given the same RL recipe?). However, it also means that the paper implicitly assumes the optimal RL hyperparameters are invariant to the SFT initialization. If different SFT checkpoints benefit from different RL configurations—for example, a checkpoint with higher KL divergence from the base model might need a different KL penalty coefficient during GRPO, or a checkpoint that learned more aggressively during SFT might benefit from a lower learning rate during RL—then the fixed-GRPO comparison could unfairly disadvantage certain initializations that would perform better under their own optimal RL settings.

The consequence. The rank reversal observed in Figure 1 could partially reflect a mismatch between SFT initialization and fixed RL hyperparameters rather than a fundamental misalignment between offline and online optimization objectives. A method like SFT-KL(β=0.03) that achieves strong online performance (42.75% pass@1 on SynLogic, Table 6) might be doing so because the KL regularization during SFT happens to produce a checkpoint well-suited to the particular GRPO configuration used—not because KL regularization is generally good for RL readiness. Conversely, TopLogP's catastrophic post-RL performance (10% vs. 30% for SFT) might reflect extreme sensitivity to the fixed RL hyperparameters—a different learning rate or KL coefficient might rescue its performance. The paper cannot distinguish between these possibilities because it never varies the RL configuration.

This matters practically because a practitioner who adopts PEAR based on the paper's results needs to know whether the reported gains are robust to their specific RL setup. If PEAR's advantage is large only under a narrow range of RL hyperparameters, the method is less practically useful than if the advantage is robust across RL configurations. The paper's transfer experiment (Table 4) shows that PEAR's benefit persists when the RL task distribution shifts, but this does not test robustness to RL algorithm hyperparameters.

What evidence exists in the paper. The paper provides no ablation over RL hyperparameters. The GRPO learning rate ( $10^{-6}$ ), batch size (128), and KL coefficient (0.01) are stated but never varied. There is no experiment showing that the ranking of SFT methods by post-RL performance is stable under different GRPO configurations, nor any test of whether PEAR's benefit over SFT is larger or smaller at different RL learning rates or KL penalties.

Mitigation status. Not addressed. The paper does not acknowledge the fixed RL hyperparameter assumption as a limitation. The conclusion that PEAR "consistently improves post-RL performance" (Section 6) is technically accurate given the paper's fixed-RL design, but a more complete claim would require evidence that the improvement is robust to the RL configuration. A minimal robustness check—running one or two alternative GRPO configurations (e.g., a higher and lower learning rate, or a higher and lower KL coefficient) on the top few SFT methods—would substantially strengthen confidence in the main result. Without it, the possibility that some SFT methods are merely poorly matched to the specific GRPO settings used cannot be ruled out.

PEAR Does Not Address the Hardest Problems Where Test-Time Compute Is Most Needed

The assumption or constraint. The paper evaluates PEAR on competition-level benchmarks (AIME-2024, AIME-2025, AMC-2023, MATH-500) where the base models already achieve non-trivial performance. For Qwen3-8B-Base, SFT+GRPO reaches 35% Pass@8 on AIME-2024 and 35% on AIME-2025 (Table 2). For DeepSeek-Distill-Qwen-1.5B, the pre-PEAR baseline is much lower (2% Pass@8 on AIME-2024)—but this model achieves substantial gains from PEAR (+36 percentage points), bringing it to 38%. The critical observation is that PEAR's benefit is largest on models and benchmarks where the baseline post-RL performance is moderate but improvable, and it provides essentially zero benefit on the hardest benchmark for the largest model: Qwen3-8B-Base on AIME-2025 shows 35% Pass@8 with both SFT+GRPO and PEAR+GRPO (Table 2). The method does not break through performance ceilings on the most difficult problems for capable models.

The consequence. PEAR improves the efficiency of the SFT→RL pipeline—it gets more out of a fixed RL budget—but does not appear to expand the capability frontier of what the model can ultimately solve. In the reference paper's terminology (Section 7), test-time strategies have sharply bounded effectiveness on "hard" problems where the base model's pass@1 is near zero. An analogous dynamic appears here: PEAR's suffix reweighting helps the model concentrate SFT learning on continuations it will revisit during RL, which improves the signal-to-noise ratio of offline training for the RL stage. But if the model fundamentally lacks the capacity to solve certain problems—even after RL—no amount of distribution correction during SFT can create that capacity. The flat Pass@8 on AIME-2025 for Qwen3-8B is consistent with this interpretation: at 8B parameters, AIME-2025 problems may be near the model's absolute capability ceiling, and PEAR cannot push beyond it.

For practitioners, this means PEAR is best understood as an efficiency tool—it reduces the amount of RL compute needed to reach a given performance level, or equivalently, achieves higher performance for the same RL compute budget—but not as a mechanism for solving problems that are out of reach. This is an important distinction because the paper's narrative around "better SFT for RL" could be misinterpreted as claiming that SFT redesign can substitute for model scale or fundamentally better training. The 8B AIME-2025 result suggests it cannot.

What evidence exists in the paper. The flat AIME-2025 Pass@8 for Qwen3-8B (Table 2) is the clearest signal: 35% for SFT+GRPO, 35% for PEAR+GRPO. On AIME-2024, the 8B model shows a smaller gain (35% to 41% Pass@64) and flat Pass@8 (35% for both). For the smaller models (1.5B–4B), the gains are larger, consistent with these models having more headroom below their capability ceiling. The 0.6B model shows modest absolute gains even on easier benchmarks (+3 average accuracy, Table 3), suggesting that at very small scales, base capability is so limited that distribution correction during SFT provides minimal benefit—the model simply cannot learn the task well enough for RL to matter. The paper does not explicitly discuss this capability-ceiling dynamic or characterize the conditions under which PEAR provides large vs. small vs. zero benefit.

Mitigation status. Not addressed. The paper does not discuss the hardness-dependence of its results, does not break down gains by problem difficulty (unlike the reference paper, which partitioned results into five difficulty quintiles), and does not characterize the regime where PEAR is most vs. least beneficial. The conclusion presents PEAR as a general improvement without qualification about the difficulty level or base model capability where it helps. A difficulty-stratified analysis—analogous to the reference paper's Figure 3 (right) and Figure 7 (right)—would show whether PEAR's gains are concentrated on easy/medium problems and vanish on the hardest ones, which would help practitioners decide whether PEAR is relevant to their setting (e.g., frontier math vs. general instruction following). The absence of such an analysis is a missed opportunity to clarify the method's practical scope.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper's primary impact is conceptual, not algorithmic: it forces a re-evaluation of what the SFT stage is for in two-stage post-training pipelines and exposes a widespread evaluation methodology as systematically misleading. Prior to this work, the field operated under an implicit modularity assumption—SFT and RL are independent stages connected by a checkpoint file, and stronger SFT performance naturally leads to stronger post-RL models. The paper's central finding—that the Pearson correlation between offline pass@1 and online pass@1 is 0.09 on Qwen3-1.7B-Base (Figure 11), and that methods optimizing directly for offline accuracy can be the worst RL initializations (TopLogP achieves best offline but worst online across 11 variants in Table 6)—renders this assumption untenable. The community has been comparing SFT techniques on a proxy metric with essentially zero predictive validity for the outcome that actually matters.

This constitutes a diagnostic reframing rather than a paradigm shift in algorithm design. PEAR itself is a relatively lightweight loss-reweighting scheme—it does not introduce a new model architecture, optimizer, or training paradigm. The shift is in the evaluation criterion: the paper argues that SFT research should be judged by post-RL performance rather than offline metrics, and that SFT objectives should be designed with the downstream RL dynamics in mind rather than optimized in isolation. This echoes the reference paper's reframing of test-time compute from "more is better" to "difficulty-conditioned allocation," but applied to the training pipeline rather than inference strategy. In both cases, the key contribution is not a new method but a new way of thinking about an existing stage of the pipeline—plus a concrete technique that operationalizes the new perspective.

Several prior contradictions are reconciled by this reframing. The observation that SFT memorizes while RL generalizes (Chu et al., 2025) seemed to suggest that SFT and RL are fundamentally at odds, yet both stages are essential in practice. The paper's diagnosis explains why: they are not at odds per se, but standard SFT optimizes a distribution (the behavior policy's) that RL must subsequently move away from, creating gradient conflict (Figure 7) and wasted parameter updates (Figure 8). Similarly, prior work found that repetition and data homogeneity boost SFT but reduce RL headroom (Kang et al., 2025)—the paper generalizes this finding beyond data construction to loss function design, showing that the SFT objective itself can either facilitate or obstruct RL improvement. The resolution is that SFT should not merely imitate the behavior policy but should prepare the model for its own on-policy exploration.

The paper also clarifies which research directions are more and less promising. The finding that common stabilization heuristics—KL regularization, sparse updates, conservative movement in weight space—are orthogonal or even counterproductive for RL readiness (Figure 8, Appendix G) suggests that the substantial investment in "stable SFT" methods (Proximal SFT, Anchored SFT, KL-distillation variants) may be optimizing the wrong objective. A method that looks stable by convergence metrics can be a poor RL initialization, while PEAR—which often produces larger KL divergence and denser updates (Figure 8-a, Figure 10)—substantially outperforms these methods post-RL. This does not mean stabilization is worthless, but it means stabilization alone is insufficient and may be actively harmful if it prevents the model from making distribution corrections during the cheaper offline stage. Research attention should shift from "making SFT well-behaved in isolation" to "making SFT aligned with downstream RL optimization."

Conversely, the paper opens the door for off-policy evaluation techniques as a design toolkit for LM post-training. Importance sampling, stationary distribution corrections, doubly-robust estimators, and other OPE machinery have been developed over decades in the RL community primarily for policy evaluation. The paper demonstrates that these same mathematical tools can be repurposed for loss reweighting during training to bridge the offline-to-online gap. This connection had been underexplored because language model SFT is typically viewed through a supervised learning lens (maximize likelihood of the data) rather than a policy optimization lens (learn behaviors that will generalize under distribution shift). PEAR's success suggests that importing more sophisticated OPE techniques—beyond simple importance sampling—into SFT loss design could yield further gains.

The paper also validates a broader principle that may apply beyond the SFT→RL pipeline: whenever a model is trained on data from one distribution and deployed under another, the training loss should be weighted by compatibility with the deployment distribution. This principle is obvious in classical machine learning (domain adaptation, importance-weighted empirical risk minimization) but has been largely ignored in the post-training literature, where the assumption was that offline data is representative of the online policy's behavior. PEAR provides a concrete template for how to apply this principle in the autoregressive generation setting, and the transfer experiment (Table 4, SynLogic→Enigmata) suggests the correction generalizes across task distributions. This opens the door for applying similar reweighting strategies in other multi-stage training paradigms: instruction tuning followed by RLHF, continued pretraining followed by domain-specific SFT, or multi-task training where data mixtures change between stages.

Follow-Up Research This Work Enables

Directly measuring and verifying the behavior–target mismatch during the SFT→RL transition. The paper's central mechanistic claim—that PEAR works by down-weighting tokens whose logged continuations are unlikely under the target policy, aligning SFT learning with prefixes GRPO will actually visit—is supported by indirect evidence (gradient alignment in Figure 7, parameter drift in Figure 8) but never directly verified. A strong follow-up would instrument the GRPO training loop to log the prefixes (partial sequences) generated during on-policy rollouts and compare them to the prefixes in the offline SFT data. The prediction is twofold: (1) for a standard SFT checkpoint, there should be substantial divergence between the offline prefix distribution and the GRPO rollout prefix distribution, particularly on tokens where the model's policy disagrees with the behavior policy; and (2) the per-token PEAR weights should be positively correlated with the frequency with which GRPO rollouts actually visit tokens with similar continuations—high-weight tokens correspond to prefixes that appear frequently in GRPO rollouts, low-weight tokens correspond to prefixes that are rare. Performing this analysis across difficulty levels (easy vs. hard SynLogic puzzles, easy vs. hard MATH problems) would reveal whether PEAR's correction is more important when the base model's policy diverges more from the behavior policy, which would explain the difficulty-dependent scaling of benefits observed in the AIME results (Table 2). If the direct mismatch measurement confirms the mechanism, it would transform PEAR from an empirically-motivated heuristic into a principled correction with predictable behavior.

Scaling PEAR to frontier-size models (70B+) with realistic SFT→RL pipelines. The paper tests models up to 8B parameters with a single epoch of SFT and a single GRPO configuration. The most impactful unknown is whether PEAR's benefits persist, diminish, or reverse at the scales where the community's practical interest is highest—the 70B-to-hundreds-of-billions regime. A natural follow-up would replicate the core comparison (SFT+GRPO vs. PEAR+GRPO) on a model like Qwen2.5-72B or DeepSeek-V2-Lite, with a standard multi-epoch SFT stage (e.g., 3 epochs) and variable GRPO configurations. The key questions: (1) Does PEAR's benefit scale with model size, or does the offline–online mismatch become less severe as models become better calibrated? The mixed scaling evidence in the paper—large gains at 8B on SynLogic games but flat Pass@8 on AIME-2025 at the same model size—provides no clear trend. (2) Does PEAR remain beneficial with multi-epoch SFT, or does extended training saturate the mismatch correction? (3) Does the ranking of SFT methods by post-RL performance stabilize with more SFT training, reducing the practical importance of the rank reversal phenomenon? A negative result—PEAR provides no benefit at 70B+—would not invalidate the paper's contributions but would sharply bound their practical scope. A positive result—PEAR provides 5-10% absolute gains on AIME or GPQA at the 70B scale—would make it a standard component of frontier post-training pipelines.

Training a difficulty predictor or mismatch predictor to make PEAR's benefit estimable offline. One of the paper's most significant unaddressed practical limitations is that post-RL performance can only be measured by actually running RL. The paper's analyses (Figures 7, 8, 9) attempted to find offline diagnostics that predict online performance but found that standard metrics (KL to base, update sparsity, spectrum shift) are weak predictors. A targeted follow-up would train a lightweight probe—a small linear classifier or MLP—that takes the PEAR weights (or summary statistics thereof: mean, variance, distribution across weight bins) computed on a small held-out validation set as input and predicts the post-GRPO performance. The training data would come from the paper's existing matrix of 19 SFT variants on SynLogic (Figure 1, Table 6), where both the PEAR weight distributions (computable offline after SFT) and the post-GRPO performance (requiring RL) are known. If such a probe achieves strong predictive accuracy (e.g., R² > 0.7 in predicting post-RL pass@1 from offline weight statistics), it would provide a practical tool for SFT checkpoint selection without running expensive RL. Even a negative result—the probe fails to predict post-RL performance better than offline metrics—would be informative, showing that the offline–online gap is not simply a function of the weight distribution and requires deeper mechanistic understanding. This research direction is directly enabled by the paper's release of systematic comparisons across many SFT variants.

Applying PEAR-style reweighting to the RL stage itself. PEAR corrects the mismatch between offline data and the target policy during SFT. But a related mismatch exists during online RL: GRPO uses the model's own rollouts, but these rollouts are sampled from the current policy while the updates are applied to the next policy. This is the standard on-policy vs. off-policy tension in policy gradient methods, typically addressed by importance sampling with the ratio $\pi_{\theta_{\text{new}}}/\pi_{\theta_{\text{old}}}$ or by trust-region constraints like PPO's clipping. A natural extension would be to apply PEAR's suffix-based reweighting to the GRPO advantage estimation: when computing advantages over a rollout, weight the contribution of each token by the suffix importance ratio between the current policy and the policy that generated the rollout. This would provide a finer-grained off-policy correction than standard GRPO's clipping, potentially reducing the variance of advantage estimates and improving sample efficiency. The experiment would compare standard GRPO against GRPO with per-token suffix importance weights on the same SynLogic benchmark, measuring both final performance and learning curves. A positive result would show that the suffix-based correction is a general principle for policy optimization in autoregressive models, not just for SFT initialization.

Testing whether PEAR can substitute for larger offline data budgets. The paper uses a fixed 100,000 trajectories for offline training. Since PEAR effectively filters the offline data by relevance to the target policy—down-weighting tokens with implausible continuations—it might allow equivalent post-RL performance with substantially less offline data. A data-scaling experiment would train PEAR and standard SFT on increasingly large subsets of the offline data (e.g., 10K, 25K, 50K, 100K trajectories) and compare post-GRPO performance. The prediction is that PEAR's performance saturates at smaller data sizes because it extracts more signal per relevant example, while standard SFT requires larger datasets to overcome the noise from misaligned continuations. If confirmed, this would be practically significant because offline data generation (sampling from teacher models) can be expensive at scale. A negative result—PEAR's benefit is independent of dataset size—would suggest that the reweighting serves a purpose orthogonal to data efficiency (e.g., improving gradient alignment regardless of data volume), which would also be informative for understanding the mechanism.

Extending PEAR to domains without clean terminal rewards, using learned or heuristic continuation scores. The paper's conceptual grounding in off-policy evaluation and the value-function interpretation in Appendix C relies on the existence of a terminal reward—correct/incorrect for verifiable reasoning. However, PEAR's actual computation uses only model probabilities (the suffix ratio $\pi_\theta/\pi_\beta$ ) and does not incorporate the reward signal into the weight computation; the reward interpretation is motivational, not computational. This means PEAR can be applied, in principle, to any SFT dataset with known behavior policy probabilities, regardless of whether a terminal reward exists. The open question is whether the suffix importance ratio is a good proxy for "RL readiness" in domains where success is not binary or verifiable. A follow-up experiment would test PEAR on instruction-following tasks (e.g., AlpacaEval, MT-Bench) where the RL stage uses a learned reward model (RLHF) rather than ground-truth verifiers. The prediction is uncertain: if PEAR's benefit comes from aligning SFT with the model's own distribution, it should help regardless of the reward type; but if the suffix ratio's effectiveness depends on the implicit value-propagation structure (where transitions near the end of a successful trajectory are more informative), it might be less beneficial when rewards are dense, learned, or subjective. This experiment would substantially broaden (or bound) PEAR's applicability beyond the verifiable reasoning domain.

Practical Applications and Downstream Use Cases

Cost-efficient SFT checkpoint selection for organizations running large-scale GRPO pipelines. The paper's most actionable practical implication is not PEAR itself but the demonstration that offline SFT performance is an unreliable guide to post-RL quality. Organizations with the resources to run multiple SFT variants and expensive GRPO training on each should adopt a pilot RL protocol rather than selecting SFT checkpoints based on offline metrics. Concretely: train several candidate SFT checkpoints with different objectives, run a small-scale GRPO pilot (e.g., 10-20% of the full training steps, reduced rollout budget) on each, evaluate on a validation set, and select the best performer for full-scale RL. The paper's results suggest this could prevent the catastrophic scenario where a checkpoint selected for strong offline performance (like TopLogP in Table 6) produces the worst post-RL model—a waste of compute that the pilot protocol would catch early. The cost of running multiple small GRPO pilots is substantially lower than running full GRPO on a suboptimal checkpoint, especially given that "RLVR for LLMs has become increasingly expensive" (Appendix A). The paper's transfer experiment (Table 4) suggests the pilot need not even be on the exact same task distribution—a small-scale pilot on a related task can identify the best initialization for the full-scale task.

Integrating PEAR into existing SFT pipelines when the behavior policy is known. For any organization that generates SFT data from a known, accessible model—a common pattern in self-improvement loops (generate data from a teacher, train a student, RL on the student) and controlled post-training pipelines—PEAR can be integrated with minimal engineering overhead. The implementation requires: (1) storing per-token log-probabilities from the data-generating model alongside the training data (a one-time cost at data generation time, adding moderate storage but no training-time overhead), and (2) modifying the SFT loss to multiply each token's contribution by the stop-gradiented suffix weight computed from the stored behavior probabilities and the current model's forward-pass probabilities. The paper reports that this adds "negligible" training overhead because the log-probabilities are already computed for the base loss (Section 4.3). The practical benefit, using the paper's numbers: on Qwen3-1.7B-Base with SynLogic games, PEAR improves post-GRPO pass@1 from 42.5% to 63.7% (a 50% relative improvement, Figure 4a); on DeepSeek-Distill-Qwen-1.5B across math benchmarks, average pass@1 improves from 14% to 39% (a 2.8× relative improvement, Table 2). For an organization running weekly SFT→RL cycles on a 1.5B–8B model, this translates to substantially more capable models for the same RL compute budget—or equivalently, reaching a target performance level with fewer expensive RL steps. The integration is most straightforward when the behavior policy is the same as the model used for knowledge distillation (Figure 4b), where the log-probabilities are already available as part of the KD objective, making PEAR essentially free to add.

Improving data efficiency in offline-to-online self-improvement loops. Several recent works (STaR, ReST $^{EM}$ , self-rewarding LMs) implement iterative self-improvement where a model generates training data, is fine-tuned on that data, and then undergoes RL using its own rollouts. In these loops, the SFT stage is trained on data from a previous version of the model—which is precisely a known behavior policy with accessible log-probabilities. PEAR's suffix reweighting is directly applicable: during each iteration's SFT phase, weight tokens by the suffix ratio between the current (new) model and the previous (data-generating) model. This should focus each iteration's SFT learning on the continuations that the updated model will actually revisit during the subsequent RL phase, potentially accelerating the self-improvement cycle or reducing the amount of RL compute needed per iteration. The paper's finding that PEAR transfers across task distributions (SynLogic→Enigmata, Table 4) is encouraging because in self-improvement loops, the data distribution shifts each iteration as the model improves. A practical implementation would store the generating model's probabilities alongside the self-generated data, apply PEAR during SFT, and compare cycle-over-cycle improvement rates against standard SFT→RL. The paper's results on the DeepSeek-distilled model—which saw the largest relative gains from PEAR (average pass@8 from 32% to 65%, a 2× improvement, Table 2)—are particularly relevant because distillation is a common starting point for self-improvement loops.

When to Prefer This Method

The paper explicitly positions PEAR against canonical SFT and a broad set of alternative SFT objectives (KL-regularized, probability-family, adaptive reweighting). The decision rule is not about choosing between PEAR and other reweighting schemes for standalone SFT performance, but about how to structure the SFT stage given that RL will follow:

Prefer PEAR over standard SFT when the offline data is generated by a known, accessible behavior policy (so that $\pi_\beta$ log-probabilities are available), and the planned downstream training includes an online RL stage using the model's own rollouts (GRPO, PPO, or RLVR). Under these conditions, the paper's evidence shows PEAR consistently improves post-RL performance across model sizes 0.6B–8B, model families (Qwen2.5, Qwen3, DeepSeek-distilled), and task domains (logic games, competition math). The gains are largest when there is substantial headroom between the base model's capability and the task difficulty—on models that are not already near their performance ceiling. The implementation cost is minimal (per-token weight multiplication, one $O(T)$ backward scan per sequence) and the overhead is negligible relative to the forward/backward pass.
Prefer standard SFT over PEAR when the behavior policy is unknown or its per-token log-probabilities are inaccessible (e.g., SFT on human-written data, web-scraped data, API model outputs without probability access). PEAR cannot be applied without $\pi_\beta$ . In these settings, the paper's diagnostic finding is still actionable: do not assume the SFT checkpoint with the best offline accuracy will be the best RL initialization, regardless of which SFT objective you use. Test multiple checkpoints with small-scale pilot RL before committing to a full-scale RL run.
Do not expect PEAR to help when the base model is near its absolute capability ceiling on the target tasks (e.g., Qwen3-8B on AIME-2025 in Table 2, where PEAR provides zero Pass@8 gain). The method improves RL efficiency within the model's existing capability envelope but does not create new capabilities. If your baseline post-RL performance is already at or near what the model scale can achieve, PEAR's distribution correction will not push beyond that ceiling.
PEAR with block-level weighting (e.g., $B=4$ ) is a practical default that nearly matches token-level performance (Figure 6) while being more stable for very long sequences. For reasoning tasks where solutions span thousands of tokens, the block-level variant reduces variance from long-horizon importance products without sacrificing the core mismatch correction.