Learning to Reason under Off-Policy Guidance

🎯 Pitch

On-policy RL for reasoning hits a hard ceiling—models like LLaMA-3.1-8B completely fail on difficult math when training rewards collapse to zero. LUFFY breaks this barrier by dynamically blending off-policy traces from a stronger model into the reinforcement learning loop, achieving a +6.4 point average gain across math benchmarks and, critically, making weak models learn where standard RLVR cannot.

1. Executive Summary

This paper introduces LUFFY (Learning to Reason under Off-Policy Guidance), a framework that augments reinforcement learning with verifiable rewards (RLVR) by incorporating off-policy reasoning traces from a stronger teacher model into the GRPO algorithm. The approach combines Mixed-Policy GRPO (adding off-policy rollouts directly into the advantage computation group alongside on-policy samples) with policy shaping via regularized importance sampling (re-weighting gradients to emphasize low-probability tokens from off-policy traces using a nonlinear shaping function $f(x) = x/(x + \gamma)$ ), evaluated on Qwen2.5-Math-7B across six competition-level math benchmarks including AIME 2024/25, AMC, MATH-500, Minerva, and OlympiadBench. LUFFY achieves an average gain of +6.4 points over prior RLVR methods and +6.2 points on out-of-distribution tasks, establishing a new state-of-the-art on RLVR with Qwen2.5-Math-7B. Crucially, LUFFY successfully trains weak foundation models like LLaMA-3.1-8B on difficult data where on-policy RLVR completely fails—training rewards collapse to zero—demonstrating that off-policy guidance enables learning beyond a model’s initial cognitive boundaries only when imitation is dynamically balanced with exploration rather than applied rigidly through supervised fine-tuning.

2. Context and Motivation

The Core Problem: On-Policy RLVR Is Bounded by Initial Model Capabilities

The fundamental limitation this paper tackles is deceptively simple: reinforcement learning with verifiable rewards (RLVR), the paradigm behind DeepSeek-R1 and similar breakthroughs, is inherently on-policy, meaning the model can only learn from its own generated outputs. This creates a hard ceiling on what the model can acquire — it can amplify and refine existing behaviors but cannot discover genuinely new cognitive strategies that lie outside its initial sampling distribution.

The paper articulates this constraint directly (Section 1):

"on-policy RL is bounded by the base LLM itself... reinforcement learning under this setting amplifies existing behaviors rather than introducing genuinely novel cognitive capacities."

This is not a hypothetical concern. The paper cites concrete evidence from recent work (Gandhi et al., 2025) showing that models like Llama 3.2 "quickly reach performance plateaus under RL training precisely because they lack certain foundational cognitive behaviors necessary for further advancement." In other words, if a model doesn't already have the seeds of self-reflection, backtracking, or verification in its output distribution, no amount of on-policy RL will create them — there are no positive reward signals to reinforce because the model never stumbles upon these behaviors by chance.

The implications are profound for the RLVR research agenda. DeepSeek-R1 demonstrated that reasoning behaviors like self-reflection and self-correction can "emerge" through RL with simple outcome rewards, famously termed the "aha moment." But this emergence is contingent: it only occurs when the base model already has a non-trivial probability of producing those behaviors. For weaker base models, on-policy RLVR produces no "aha moment" at all — the training signal collapses because the exploration space is too vast and the model never finds correct solutions to learn from (Section 5.2, Figure 4).

Why This Problem Matters

The paper identifies three interlocking reasons why the on-policy bottleneck is critical:

1. Democratization of reasoning capabilities. DeepSeek-R1's success relied on starting from strong base models (DeepSeek-V3) that already possessed sophisticated reasoning behaviors. But what about the vast ecosystem of smaller, open-weight models — Llama-3.1-8B, Qwen2.5-Math-1.5B, and others — that lack these foundational capabilities? If on-policy RLVR only works on already-capable models, the reasoning revolution remains restricted to organizations that can train frontier-scale models. Breaking the on-policy constraint would enable smaller, more accessible models to acquire reasoning behaviors they could never discover independently.

2. Sample efficiency and plateau avoidance. Even for models where on-policy RLVR initially succeeds, the paper highlights that performance plateaus are a practical concern. The exploration space for reasoning is combinatorially vast — for a math problem requiring 1000+ tokens of coherent reasoning, the number of plausible token sequences that "look like reasoning" but are incorrect dwarfs the number of correct solutions. On-policy methods, which must find correct solutions through random exploration to receive positive reinforcement, become increasingly sample-inefficient as task complexity grows. An approach that provides guided exploration — showing the model what correct reasoning looks like — could break through plateaus that on-policy methods cannot.

3. The gap between imitation and reinforcement learning for reasoning. There is a known dichotomy in the field: supervised fine-tuning (SFT) on high-quality reasoning traces can teach models to produce solutions that look correct in format and length, but the learned behavior is often brittle — the paper shows SFT-trained models fail catastrophically at higher sampling temperatures (Figure 12, Appendix F.2) and produce excessively long, repetitive outputs that don't actually solve problems (Appendix D, Figure 7). On the other hand, pure RL teaches models to explore and find correct solutions, but only within their existing capability envelope. The paper's motivation is to find a synthesis that captures the guided learning benefits of SFT (showing the model what success looks like) with the exploration benefits of RL (letting the model discover its own strategies within informed boundaries). This gap is particularly visible in the poor out-of-distribution generalization of SFT-based distillation methods (Table 1: SFT scores 47.5 vs. LUFFY's 57.8 on OOD tasks).

Where Prior Approaches Fall Short

On-policy RLVR methods (GRPO and its variants). The state-of-the-art in open-source RLVR — represented by SimpleRL-Zero (Zeng et al., 2025), Oat-Zero (Liu et al., 2025), PRIME-Zero (Cui et al., 2025), and OpenReasonerZero (Hu et al., 2025) — all operate exclusively on-policy. The model samples $N$ responses per prompt from its current policy, receives binary (correct/incorrect) rewards, computes advantages based on relative performance within that group, and updates toward higher-reward behaviors. This has proven effective (Table 1: On-Policy RL achieves 45.5 avg on Qwen2.5-Math-7B), but the paper identifies a fundamental ceiling: the model can only learn from patterns that appear in its own rollouts. If a certain reasoning strategy — say, explicitly checking intermediate calculations for arithmetic errors — never appears in the model's sampled outputs, it never receives positive reinforcement, and the model never learns it, regardless of how many iterations of RL are run.

The paper provides empirical evidence for this limitation (Section 5.2). On the Hard training subset derived from OpenR1-Math-220k, LLaMA-3.1-8B's on-policy RL training rewards "collapse to zero" — the model never generates correct solutions, so all advantages are negative or zero, and the policy gradient provides no meaningful learning signal. This is not a slow convergence problem; it's a complete failure mode where learning stops entirely because the model's initial policy doesn't intersect with the space of correct solutions.

The implicit exploration assumption. Prior RLVR work largely assumes that with sufficient sampling (typically 8–64 rollouts per prompt at temperature 1.0), the model will occasionally produce correct solutions through stochastic exploration, providing the positive signals needed to bootstrap learning. The paper shows this assumption breaks for weaker models on difficult problems — the exploration space is simply too vast for random sampling to land on correct solutions within a feasible number of attempts. This aligns with theoretical results from Zhao et al. (2025) and Yue et al. (2025), who demonstrate that RL post-training primarily amplifies behaviors already present in the pretrained model's distribution.

SFT as an alternative — and its failure modes. The most direct way to provide off-policy guidance is supervised fine-tuning on high-quality reasoning traces (e.g., from DeepSeek-R1). This is exactly the approach used in DeepSeek's own distillation pipeline (Guo et al., 2025) and in open-source reproductions like OpenR1-Qwen-7B. The paper identifies two critical weaknesses of this approach:

Rigid imitation of surface patterns. SFT models learn to reproduce the form of reasoning traces — the length, the structure, the "thinking" tags — without necessarily internalizing the underlying reasoning skills. The paper's analysis (Appendix F.1) shows SFT models generate outputs that are 64% longer on average (4,646 tokens vs. 2,832 for LUFFY) and frequently produce "overly long and ultimately unproductive reasoning traces" on incorrect problems. The case study (Appendix D, Figure 7) vividly illustrates this: the SFT model produces over 8,129 tokens of circular "Let me break this down... Wait, but in the problem statement... Hmm, maybe..." without ever reaching a coherent solution.
Catastrophic degradation under test-time scaling. When sampling at higher temperatures during inference — a standard technique for test-time compute scaling — SFT models suffer severe performance degradation (Appendix F.2, Figure 12). At temperature 1.0, SFT's pass@8 on AIME/AMC drops to roughly 25%, compared to LUFFY's approximately 40%. This suggests SFT memorizes a narrow distribution of "correct-looking" outputs rather than learning generalizable reasoning strategies that hold across diverse sampling conditions.
Poor out-of-distribution generalization. On out-of-distribution tasks (ARC-c, GPQA-diamond, MMLU-Pro), SFT achieves 47.5 average vs. On-Policy RL's 57.3 (Table 1) — a nearly 10-point gap. The rigid imitation learned through SFT does not transfer to problem formats and domains that differ from the training distribution.

SFT + RL as a hybrid — and why it underperforms. A natural idea is to combine SFT's guided initialization with RL's exploration benefits: first fine-tune on off-policy traces, then apply RL. The paper tests this and finds it underperforms LUFFY (48.2 vs. 50.1 on in-distribution, 44.8 vs. 57.8 on OOD) despite requiring significantly more resources (Table 2: 130 × 8 vs. 77 × 8 GPU hours). The problem is that SFT creates "excessively long generations induced by rigid imitation... which substantially increase the computational overhead during the RL roll-out stage" (Section 5.1). In other words, SFT's initial imitation corrupts the RL process — the model starts RL from a policy that already overproduces lengthy, unproductive reasoning, and RL cannot easily unlearn these patterns because they're never wrong (they just don't produce correct answers in a sample-efficient way, but the sparse binary reward provides no signal about output length or efficiency).

RL with SFT loss as a multi-task objective. Another straightforward hybrid — computing both the SFT loss on off-policy traces and the RL loss on on-policy samples simultaneously — also performs poorly (40.1 avg, Table 1). The paper hypothesizes (and provides evidence in Appendix F.1, Figure 11) that this approach causes the model to rapidly converge toward imitating the surface features of off-policy traces (generation length increases sharply early in training) without the balancing mechanism that LUFFY provides through its dynamic advantage computation.

How This Paper Positions Itself

The paper frames LUFFY as a principled solution to the imitation-exploration tradeoff in off-policy RLVR — a problem that, to the authors' knowledge, has not been previously addressed in the LLM reasoning literature. The central insight is that off-policy traces should be used as scaffolding that selectively guides when on-policy exploration fails, not as targets to be uniformly imitated.

The theoretical positioning draws on connections to importance sampling in policy gradient methods (Sutton et al., 1999) and off-policy PPO variants (Meng et al., 2023). By integrating off-policy trajectories directly into the GRPO group computation — rather than treating them as separate SFT targets — the advantage estimation naturally becomes dynamic: when the model's on-policy rollouts are mostly incorrect, the off-policy traces (with their high rewards) dominate the advantage normalization, providing strong learning signals. When the model begins producing its own correct solutions, on-policy rollouts take over in the advantage computation, encouraging self-driven exploration. This is formalized in Equation 4:

$\hat{A}_i = \frac{R(\tau_i) - \text{mean}(\mathcal{G}_{\text{on}} \cup \mathcal{G}_{\text{off}})}{\text{std}(\mathcal{G}_{\text{on}} \cup \mathcal{G}_{\text{off}})}$

where $\mathcal{G}_{\text{on}}$ and $\mathcal{G}_{\text{off}}$ are the reward sets from on-policy and off-policy rollouts, respectively. This is a unified advantage computation rather than separate SFT and RL objectives, which the paper argues is what enables the dynamic balance.

The paper positions its policy shaping contribution (Section 3.2) as addressing a previously unremarked failure mode of mixed-policy training: entropy collapse. Even though mixed-policy GRPO successfully incorporates off-policy signals, the model's entropy collapses much faster than in on-policy RL (Figure 2, left) because the model quickly learns to assign high probability to off-policy tokens that are already likely under its current distribution, while ignoring off-policy tokens with low probability — precisely the tokens that represent genuinely new reasoning capabilities. Policy shaping via the nonlinear transformation $f(x) = x/(x + \gamma)$ re-weights the gradient to amplify learning from low-probability tokens (Figure 2, right), maintaining exploration throughout training.

The paper also positions itself within the broader "distillation vs. RL" debate. While distillation from stronger models (e.g., DeepSeek-R1) has become a standard pipeline, LUFFY argues that how you use off-policy traces matters enormously. The paper's ablation (Table 4) shows that each component — Mixed-Policy GRPO, policy shaping, and clip removal — contributes additively to performance, suggesting they address distinct aspects of the off-policy learning challenge.

A critical contextual note: the paper explicitly acknowledges it focuses on math reasoning with verifiable rewards, and that tasks without ground-truth answers (where reward hacking is a concern) are outside scope. This is an important scoping choice — the verifiable reward setting is precisely where the on-policy constraint is most stark, because there is a clear correct/incorrect signal that the model may simply never stumble upon through random exploration.

3. Technical Approach

3.1 Reader Orientation

LUFFY is a training algorithm that teaches a language model to solve math problems through reinforcement learning, but instead of forcing the model to discover correct solutions entirely on its own, it provides high-quality reasoning demonstrations from a stronger model as scaffolding — the model learns which reasoning patterns to imitate when its own attempts fail, while retaining the freedom to explore its own strategies when they succeed. The problem it solves is that standard on-policy RLVR (where the model learns only from its own generated outputs) fails completely on weak models or hard problems because the model never stumbles upon correct solutions to receive positive feedback; LUFFY addresses this by dynamically mixing off-policy guidance with on-policy exploration within a single unified advantage computation, creating a natural imitation-exploration balance that adapts automatically as the model improves.

3.2 Big-Picture Architecture (Diagram in Words)

The LUFFY training pipeline has five interconnected components:

Training Dataset — a filtered subset of OpenR1-Math-220k containing 45k prompts paired with reasoning traces generated by DeepSeek-R1 (the off-policy guidance). Each prompt has one off-policy trace that is verified correct by Math-Verify.
Base Policy Model ( $\pi_\theta$ ) — the LLM being trained (default: Qwen2.5-Math-7B). It generates both on-policy rollouts (7 per prompt, at temperature 1.0) and computes probabilities for off-policy tokens.
Off-Policy Reference ( $\pi_\phi$ ) — the provider of high-quality reasoning traces (DeepSeek-R1). In practice, $\pi_\phi$ is set to 1 (a uniform distribution) for computational efficiency, avoiding cross-model tokenization issues.
Mixed-Policy GRPO Objective — the core learning mechanism that combines on-policy and off-policy rollouts into a single group for advantage computation (Equation 4) and applies importance-weighted policy gradients to update the model (Equation 5).
Policy Shaping Mechanism — a regularized importance sampling transformation $f(x) = x/(x + \gamma)$ that replaces the raw importance ratio in the off-policy gradient, amplifying learning from low-probability tokens that represent genuinely new reasoning capabilities (Equation 6).

Information flow: For each training step, the base model generates 7 on-policy responses per prompt (sampled at temperature 1.0) → these are combined with 1 pre-computed off-policy response per prompt from DeepSeek-R1 → reward is computed for all 8 responses using Math-Verify (binary: 1 if final answer matches gold, 0 otherwise) → advantages are computed jointly over the mixed group using Equation 4 → the policy gradient is computed using Equation 6 with shaping function $f$ applied to off-policy importance ratios → the model parameters are updated → repeat for 500 steps.

3.3 Roadmap for the Deep Dive

First, the verifiable reward function and on-policy GRPO baseline, since LUFFY extends GRPO and understanding the base algorithm is prerequisite.
Second, Mixed-Policy GRPO — how off-policy rollouts are injected into the advantage computation and how the objective is extended with importance sampling, including the convergence analysis (Theorem 1).
Third, the practical choice of $\pi_\phi = 1$ and why this simplification works despite theoretical concerns, since it's a non-obvious design choice.
Fourth, the entropy collapse problem in Mixed-Policy GRPO — why it happens and how it motivates policy shaping.
Fifth, policy shaping via regularized importance sampling — the shaping function, its gradient properties, and the variance reduction analysis.
Sixth, the training configuration and hyperparameters needed to make all of this work in practice.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a methodological paper that introduces a new training algorithm combining off-policy guidance with on-policy RL for LLM reasoning. The core idea is that by incorporating high-quality reasoning traces into the GRPO advantage computation — rather than using them as separate SFT targets — the model can dynamically balance imitation of expert behavior with exploration of its own strategies, overcoming the fundamental limitation that on-policy RLVR can only amplify existing capabilities.

Verifiable Reward Function

The foundation of RLVR is a reward function that returns a binary signal indicating whether the model's output contains the correct final answer, without evaluating the intermediate reasoning steps. This design choice is deliberate and important: by only checking the final answer, the reward avoids the risk of reward hacking that comes with intermediate evaluation (e.g., the model learning to produce steps that "look good" to a learned verifier without actually being correct).

Formally, given a question $q$ and the model's output $\tau$ , the reward is:

$R(\tau) = \begin{cases} 1 & \text{if } \tau \text{ outputs the correct final answer to } q \\ 0 & \text{otherwise} \end{cases}$

where $\tau$ is the complete generation (the entire chain of thought plus final answer), and correctness is determined by extracting the answer from a formatted box (e.g., \boxed{}) using a regex function and comparing it against the ground-truth answer.

What it computes: a binary scalar per response — 1 if the extracted final answer matches the gold answer exactly, 0 otherwise. The extraction uses Math-Verify, a rule-based parser that identifies the answer within \boxed{} tags and normalizes formatting (e.g., handling equivalent mathematical expressions) before comparison.

Why this form: binary outcome rewards are extremely difficult to hack because the reward is sparse and directly tied to the ground truth. Unlike learned verifiers (process reward models or outcome reward models), there are no proxy metrics for the model to exploit — the only way to get positive reward is to produce the correct final answer. This sparsity is exactly what makes on-policy RLVR challenging on difficult problems: the model may never encounter a positive reward signal through random exploration, which is the failure mode LUFFY addresses.

The paper explicitly does not include format rewards or length penalties — rewards are purely based on answer correctness. This avoids the complication of multi-objective reward tuning and keeps the training signal focused entirely on problem-solving capability.

On-Policy GRPO Baseline

Group Relative Policy Optimization (GRPO), introduced by Shao et al. (2024) for DeepSeekMath, is the base algorithm that LUFFY extends. GRPO is designed to remove the need for a separate value model (a critic network that estimates expected future rewards) by using the relative performance of multiple samples from the same prompt to estimate advantages.

The algorithm works as follows for a single training step:

Step 1: Group sampling. For a given question $q$ , the model's current policy $\pi_{\theta_{\text{old}}}$ generates $N$ complete solutions $\tau_1, \tau_2, \dots, \tau_N$ independently. Each solution is scored with the binary reward function $R(\tau_i)$ .

Step 2: Advantage computation. The advantage of each solution is computed by normalizing its reward within the group:

$A_i = \frac{R(\tau_i) - \text{mean}(\{R(\tau_i) \mid \tau_i \sim \pi_{\theta_{\text{old}}}(\tau), i = 1, 2, \dots, N\})}{\text{std}(\{R(\tau_i) \mid \tau_i \sim \pi_{\theta_{\text{old}}}(\tau), i = 1, 2, \dots, N\})}$

where $R(\tau_i)$ is the binary reward for the $i$ -th sampled solution, the mean is computed over all $N$ rewards in the group, and the standard deviation normalizes the advantages to have unit variance within each question group.

What it computes: for each response, its advantage is the number of standard deviations its reward is above or below the group mean. A correct solution (reward = 1) in a group where most solutions are incorrect (mean ≈ 0.125 if 1 of 8 is correct) receives a large positive advantage. An incorrect solution (reward = 0) in a group with a high mean receives a negative advantage. The normalization by standard deviation ensures that advantage magnitudes are comparable across questions with different difficulty levels.

Why this form: the group-relative normalization serves two critical purposes. First, it eliminates the need for a learned value function — the group mean serves as a baseline, and the advantage is positive for solutions that are better than average and negative for solutions worse than average. This is a form of self-normalization that automatically adapts to question difficulty (hard questions where most samples are incorrect will have a lower mean, making correct samples receive very high advantages). Second, the standard deviation normalization (which the paper's baseline actually removes, following Dr.GRPO) prevents advantages from having unbounded magnitudes when the model is either very good or very bad at a particular question. The paper follows Dr.GRPO (Liu et al., 2025) in removing both the length normalization and standard deviation normalization from the original GRPO.

Step 3: Policy gradient update. The GRPO objective is inherited from PPO's clipped surrogate objective:

$\mathcal{J}_{\text{GRPO}}(\theta) = \frac{1}{\sum_{i=1}^N |\tau_i|} \sum_{i=1}^N \sum_{t=1}^{|\tau_i|} \text{CLIP}(r_{i,t}(\theta), A_i, \epsilon) - \beta \cdot \mathbb{D}_{\text{KL}}[\pi_{\theta} \parallel \pi_{\text{ref}}]$

where:

$|\tau_i|$ is the length (number of tokens) of the $i$ -th solution,
$r_{i,t}(\theta) = \pi_{\theta}(\tau_{i,t} | q, \tau_{i,<t}) / \pi_{\theta_{\text{old}}}(\tau_{i,t} | q, \tau_{i,<t})$ is the importance sampling ratio — the ratio of the probability of token $\tau_{i,t}$ under the current (updating) policy to its probability under the old (rollout) policy. This corrects for the fact that the samples were generated by $\pi_{\theta_{\text{old}}}$ but the gradient is taken with respect to $\pi_{\theta}$ ,
$\text{CLIP}(r, A, \epsilon) = \min[r \cdot A, \text{clip}(r; 1 - \epsilon, 1 + \epsilon) \cdot A]$ is the PPO clipping function that prevents the policy from changing too much in a single update — when $r$ is outside $[1-\epsilon, 1+\epsilon]$ , the gradient is clipped to zero if it would push the policy further away from the old policy,
$\mathbb{D}_{\text{KL}}[\pi_{\theta} \parallel \pi_{\text{ref}}]$ is the Kullback-Leibler divergence between the current policy and a reference policy (typically the initial model), and $\beta$ controls its weight.

What it computes: the total objective is a sum over all tokens in all sampled solutions, where each token contributes either $r \cdot A$ (if the ratio is within the clipping range) or a clipped value (if the ratio would cause too large a policy change), minus a KL penalty that keeps the model from diverging too far from its initial behavior. The result is a scalar objective that is maximized to increase the probability of tokens from high-advantage solutions and decrease the probability of tokens from low-advantage solutions.

Why this form: the importance sampling correction $r_{i,t}(\theta)$ is necessary because the standard policy gradient theorem (Sutton et al., 1999) assumes samples are drawn from the current policy $\pi_\theta$ . Since GRPO uses samples from $\pi_{\theta_{\text{old}}}$ (the policy before the update), the ratio re-weights each sample's contribution to account for the distribution shift. Without this correction, the gradient would be biased if the policy changes significantly during the update. The clipping mechanism is a trust-region technique: it prevents the policy from making large jumps that could destabilize training, particularly important in the sparse-reward setting where a single successful solution could otherwise cause an enormous policy update.

KL divergence and its removal. The paper notes that "recent practices have increasingly omitted the KL divergence term, making these methods somewhat less 'On-Policy'" and follows this trend by setting $\beta = 0$ . Without the KL penalty, the only constraint on policy changes is the PPO clipping, which allows the policy to deviate more freely from its initial distribution — a design choice that becomes important for LUFFY because the model needs to learn behaviors very different from its initial policy by following off-policy guidance.

The paper also removes the length normalization term ( $1/\sum|\tau_i|$ ) and the standard deviation normalization from the advantage computation, following Dr.GRPO. This means each token contributes equally to the gradient regardless of solution length, and advantages are computed using only mean subtraction (not division by standard deviation). The entropy loss coefficient is set to 0.01 to maintain some exploration pressure on the policy.

Mixed-Policy GRPO: Injecting Off-Policy Guidance into Advantage Computation

The fundamental innovation of LUFFY is replacing the purely on-policy group in GRPO with a mixed group containing both on-policy and off-policy rollouts. The off-policy rollouts are pre-generated by DeepSeek-R1 and are always correct (reward = 1), which fundamentally changes the dynamics of advantage computation.

Mixed-group advantage computation. For each question $q$ , instead of sampling $N$ on-policy solutions, the system constructs a group with $N_{\text{on}}$ on-policy rollouts (generated by the current model $\pi_{\theta_{\text{old}}}$ ) and $N_{\text{off}}$ off-policy rollouts (pre-generated by the teacher model $\pi_\phi$ ):

$\hat{A}_i = \frac{R(\tau_i) - \text{mean}(\mathcal{G}_{\text{on}} \cup \mathcal{G}_{\text{off}})}{\text{std}(\mathcal{G}_{\text{on}} \cup \mathcal{G}_{\text{off}})}$

where:

$\mathcal{G}_{\text{on}} = \{R(\tau_i) \mid \tau_i \sim \pi_{\theta_{\text{old}}}(\tau), i = 1, 2, \dots, N_{\text{on}}\}$ is the set of rewards from the model's own generated solutions,
$\mathcal{G}_{\text{off}} = \{R(\tau_j) \mid \tau_j \sim \pi_\phi(\tau), j = 1, 2, \dots, N_{\text{off}}\}$ is the set of rewards from the teacher's solutions,
The mean and standard deviation are computed over the union of both sets.

What it computes: each response's advantage is its reward's z-score within the combined on-policy and off-policy group. The critical consequence: because off-policy traces always have reward 1, they raise the group mean, making it harder for on-policy correct solutions to receive positive advantages and easier for on-policy incorrect solutions to receive strongly negative advantages.

Why this form creates dynamic balance. This is the paper's key insight. Consider three scenarios as training progresses:

Early training (model mostly incorrect): Most on-policy rollouts get reward 0, while the off-policy rollouts get reward 1. The group mean is pulled up by the off-policy successes (e.g., with 1 off-policy correct and 7 on-policy incorrect, mean = 1/8 = 0.125). On-policy incorrect solutions (reward = 0) have slightly negative advantages. More importantly, the off-policy solutions (reward = 1) have large positive advantages, providing strong learning signals for the model to imitate the teacher's reasoning patterns. Imitation dominates because the model can't produce correct solutions yet.
Mid training (model occasionally correct): When the model starts producing some correct solutions (say, 2 on-policy correct out of 7), the group mean rises further (3/8 = 0.375). Now on-policy correct solutions receive moderate positive advantages, and off-policy solutions still receive positive (but smaller) advantages. The model learns from both its own successes and the teacher's guidance.
Late training (model mostly correct): When the model produces many correct solutions (say, 6 on-policy correct out of 7), the group mean becomes very high (7/8 = 0.875). Off-policy solutions (reward = 1) have only a small positive advantage (just above the mean), while on-policy correct solutions also have small positive advantages. Incorrect on-policy solutions receive strongly negative advantages. At this stage, on-policy successes dominate the learning signal because the model's own correct solutions are more informative about its current policy's distribution, and off-policy guidance naturally fades into the background.

This adaptive behavior is emergent from the group computation — no explicit schedule or curriculum switches from imitation to exploration. The paper describes this as "selectively imitat[ing] these high-quality reasoning traces when [the model's] own roll-outs fail to achieve correctness, while preserving the capacity for self-driven exploration whenever its generated reasoning steps are successful."

Practical configuration. The paper uses $N_{\text{on}} = 7$ and $N_{\text{off}} = 1$ , for a total of 8 rollouts per prompt. This 7:1 ratio was chosen "to ensure fairness" with the on-policy baseline which uses 8 on-policy rollouts. The paper notes that only one off-policy trajectory is used and finds it "already strong," suggesting that even a single high-quality demonstration per prompt provides substantial guidance.

The Mixed-Policy Objective: Importance-Weighted Policy Gradient with Off-Policy Data

Incorporating off-policy rollouts into the GRPO objective requires an extension because the standard importance sampling ratio $r_{i,t}(\theta) = \pi_\theta / \pi_{\theta_{\text{old}}}$ assumes samples were generated by the previous iteration of the same model. Off-policy samples were generated by a different model ( $\pi_\phi$ ), so they need a different importance sampling correction.

The Mixed-Policy GRPO objective extends Equation 3:

$\mathcal{J}_{\text{Mixed}}(\theta) = \underbrace{\frac{1}{Z} \left( \sum_{j=1}^{N_{\text{off}}} \sum_{t=1}^{|\tau_j|} \text{CLIP}(\hat{r}_{j,t}(\theta, \phi), \hat{A}_j, \epsilon) \right)}_{\text{off-policy objective}} + \underbrace{\sum_{i=1}^{N_{\text{on}}} \sum_{t=1}^{|\tau_i|} \text{CLIP}(r_{i,t}(\theta), \hat{A}_i, \epsilon)}_{\text{on-policy objective}}$

where:

$\hat{r}_{j,t}(\theta, \phi) = \pi_\theta(\tau_{j,t}|q, \tau_{j,<t}) / \pi_\phi(\tau_{j,t}|q, \tau_{j,<t})$ is the off-policy importance sampling ratio: the ratio of the token's probability under the current policy to its probability under the teacher's policy. This corrects for the distribution shift between the teacher (which generated the sample) and the current policy (which we're updating),
$r_{i,t}(\theta) = \pi_\theta(\tau_{i,t}|q, \tau_{i,<t}) / \pi_{\theta_{\text{old}}}(\tau_{i,t}|q, \tau_{i,<t})$ is the standard on-policy importance ratio from GRPO,
$\hat{A}_j$ and $\hat{A}_i$ are the mixed-group advantages from Equation 4 (applied to off-policy and on-policy samples respectively, though computed from the same group statistics),
$Z = \sum_{j=1}^{N_{\text{off}}} |\tau_j| + \sum_{i=1}^{N_{\text{on}}} |\tau_i|$ is the total number of tokens across all rollouts, normalizing the total objective to be per-token.

What it computes: the objective is a sum over all tokens from all rollouts (both on-policy and off-policy). Each token contributes a clipped importance-weighted advantage term. For on-policy tokens, this is identical to standard GRPO. For off-policy tokens, the importance ratio uses the teacher's probability in the denominator, which is typically very different from $\pi_{\theta_{\text{old}}}$ — the divergence between student policy $\pi_\theta$ and teacher policy $\pi_\phi$ is much larger than between successive student policies.

Why this form: the off-policy importance ratio $\hat{r}_{j,t}$ plays a different role than the on-policy $r_{i,t}$ . In on-policy RL, $r_{i,t}$ is usually close to 1 because the policy changes slowly. In off-policy learning, $\hat{r}_{j,t}$ can be extremely small (when the student assigns very low probability to tokens the teacher considered highly probable) or occasionally large (when the student assigns high probability to teacher-unlikely tokens). The clipping mechanism helps control this variance, but the paper notes that the clip operation is actually removed for off-policy rollouts in the final implementation (a design choice discussed below).

Convergence guarantee (Theorem 1). The paper provides a theoretical analysis showing that the importance-weighted policy gradient estimator in the Mixed-Policy objective converges to a stationary point at rate $O(1/\sqrt{K})$ , where $K$ is the total number of training iterations. This is the standard convergence rate for stochastic gradient descent in nonconvex optimization (equivalent to requiring $O(1/\epsilon^2)$ iterations to achieve $\epsilon$ -accuracy).

The theorem assumes:

The objective $J$ is in the class of finite-sum Lipschitz smooth functions ( $L$ -smooth, meaning the gradient doesn't change too rapidly),
Gradients are $\sigma$ -bounded: $\|\nabla J_i(\theta)\| \leq \sigma$ for all samples $i$ and parameters $\theta$ ,
The importance weight $w = \pi_\theta/\pi_\phi$ is clipped to be bounded between $\underline{w}$ and $\bar{w}$ (preventing extreme values),
The learning rate is set to $\alpha = c/\sqrt{K}$ with the specific constant $c = \sqrt{\frac{2(J(\theta^*) - J(\theta^0))}{L\sigma^2 \underline{w}\bar{w}}}$ .

What this guarantee means: despite using samples from a different distribution (off-policy data), the gradient estimator is unbiased in expectation and the algorithm will converge to a point where the gradient magnitude is small (a stationary point of the objective), with the convergence time scaling as $O(1/\sqrt{K})$ . The boundedness assumption on importance weights is critical — without clipping, extreme importance ratios could cause arbitrarily large variance and prevent convergence.

Why this analysis matters: it addresses a valid concern about mixing on-policy and off-policy gradients — importance-weighted estimators can have unbounded variance if the behavior policy ( $\pi_\phi$ ) assigns very low probability to actions that the target policy ( $\pi_\theta$ ) considers important. By showing convergence with bounded importance weights, the theorem provides theoretical justification that the approach is sound (doesn't diverge or oscillate) even though the training data distribution differs from the model's sampling distribution.

The proof (Appendix B.1) follows a standard SGD convergence argument: bound the improvement per step using Lipschitz smoothness, sum over $K$ iterations, and optimize the step size to get the $O(1/\sqrt{K})$ rate. The key difference from standard SGD is that the importance weight $w$ appears explicitly in the step size, and the bound scales with $\sqrt{\bar{w}/\underline{w}}$ — meaning that if the weight range is large (very different policies), convergence is proportionally slower.

Practical Choice: Setting $\pi_\phi = 1$ (Uniform Off-Policy Distribution)

The theoretical framework assumes a well-defined teacher policy $\pi_\phi$ that assigns probabilities to tokens — these probabilities would appear in the denominator of the off-policy importance ratio $\hat{r}_{j,t} = \pi_\theta / \pi_\phi$ . However, the paper makes a deliberately practical choice:

"In practice, to facilitate direct integration of high-quality demonstrations from large, powerful models (e.g., DeepSeek-R1), we adopt $\pi_\phi = 1$ for computational efficiency."

This means that instead of computing $1/\pi_\phi(\tau_{j,t}|q, \tau_{j,<t})$ — which would require running DeepSeek-R1 inference on every off-policy token to get its probability — they simply set the denominator to 1, making the off-policy importance ratio $\hat{r}_{j,t} = \pi_\theta$ (just the current model's probability of that token).

What this choice means computationally: the off-policy gradient for token $\tau_{j,t}$ becomes proportional to $\pi_\theta \cdot \hat{A}_j \cdot \nabla_\theta \log \pi_\theta$ (after removing the clip, as discussed below). This is equivalent to treating the teacher's policy as uniform — every token is equally likely under the teacher. The learning signal tells the model: "increase the probability of this token, where the strength of the update is proportional to (a) how unlikely the model currently considers this token (if $\pi_\theta$ is small, the gradient is small, but see policy shaping for how this is addressed) and (b) the advantage of the overall trajectory."

Why this is acceptable: three justifications are provided:

Avoids tokenization mismatch. DeepSeek-R1 likely uses a different tokenizer than Qwen2.5-Math-7B. Computing $\pi_\phi$ on Qwen2.5-Math's tokens would require either converting tokens between tokenizers (lossy) or evaluating DeepSeek-R1 on Qwen2.5-Math's tokenized sequences (nonsensical for the model). Setting $\pi_\phi = 1$ avoids this entirely.
Enables use of off-the-shelf datasets. The training data comes from OpenR1-Math-220k, which provides the text of DeepSeek-R1's reasoning traces but not the per-token probabilities. Computing those probabilities would require re-running inference on DeepSeek-R1 for 45k traces × thousands of tokens each — computationally prohibitive. Setting $\pi_\phi = 1$ allows training from any available reasoning dataset without access to the generating model.
Preserves theoretical guarantees. The authors claim that "our derivations and guarantees hold for any well-defined $\pi_\phi$ distribution," and that the boundedness assumption on importance weights in Theorem 1 is satisfied even with $\pi_\phi = 1$ because $\pi_\theta$ is itself bounded between near-0 and near-1 values for any given token.

Consequence: removing the off-policy clip. With $\pi_\phi = 1$ , the clipping operation $\text{CLIP}(\hat{r}, \hat{A}, \epsilon)$ becomes problematic. The original PPO clip is designed to keep $\hat{r}$ near 1, but with $\pi_\phi = 1$ and $\pi_\theta$ typically $< 1$ , the ratio is always $\leq 1$ , so only the lower clip boundary $1 - \epsilon$ is relevant. The paper states:

"We omit the clip operation for the off-policy rollouts, as the clip operation will be imbalanced when $\pi_\phi = 1$ ."

This means the off-policy objective simplifies to an unclipped importance-weighted advantage: the model simply increases probability for tokens from high-advantage trajectories and (through the advantage sign) reduces probability for tokens from low-advantage trajectories, without any clipping constraint. The paper's ablation (Appendix E.1, Figure 8) shows that even the on-policy clip is removed in the final LUFFY configuration because it suppresses learning from off-policy guidance early in training when the policy is far from the teacher.

The Entropy Collapse Problem in Mixed-Policy Training

While Mixed-Policy GRPO successfully incorporates off-policy signals, the paper identifies a critical failure mode: entropy collapses much faster than in on-policy RL. Figure 2 (left) shows that generation entropy (a measure of how diverse the model's outputs are, computed as the average negative log-probability of sampled tokens) drops precipitously in Mixed-Policy training compared to On-Policy RL.

Why entropy collapses. The paper provides a mechanistic explanation rooted in how the model responds to off-policy gradients:

"When combining both learning off-policy and on-policy signals, the model tends to quickly converge toward reinforcing off-policy tokens that are also likely in the on-policy $\pi_\theta$ distribution, and ignoring off-policy tokens that are deviated from the model's original policy, i.e., low-probability tokens that may represent essential reasoning capabilities the model has yet to acquire."

In other words, the model is presented with off-policy reasoning traces containing tokens with a wide range of probabilities under its current policy. Some tokens — common words, standard mathematical notation, frequent reasoning phrases — already have high probability under $\pi_\theta$ . Others — specific reasoning strategies, verification statements, backtracking patterns — have very low probability because they represent behaviors the model hasn't yet learned. The mixed-policy objective naturally encourages the model to increase probability for all off-policy tokens (they have positive advantages), but the gradient magnitude for high-probability tokens is much larger (because $\pi_\theta$ is larger, making the product $\pi_\theta \cdot \nabla_\theta \log \pi_\theta$ larger), so the model preferentially learns the "easy" tokens it already knows while gaining very little from the novel tokens. The result is rapid convergence to a policy that produces reasoning that superficially resembles the teacher (high-probability patterns) but lacks the substantive reasoning capabilities (low-probability patterns).

Consequences of entropy collapse. A model with collapsed entropy produces nearly deterministic outputs — it always follows the same reasoning patterns regardless of problem variation. This means:

It cannot explore alternative solution strategies when its default approach fails,
It cannot adapt to out-of-distribution problems that require different reasoning structures,
It is vulnerable to compounding errors — a single mistake derails the entire solution because the model can't backtrack or reconsider.

The paper's evidence for this phenomenon is both visual (the entropy curve in Figure 5) and performance-based: Mixed-Policy without policy shaping plateaus early (Figure 6), converging to roughly the same final performance as On-Policy RL despite the initial advantage from off-policy guidance.

Policy Shaping via Regularized Importance Sampling

To address entropy collapse, LUFFY introduces a shaping function $f$ that replaces the raw importance sampling ratio $\hat{r}_{j,t}(\theta, \phi)$ in the off-policy objective. The shaped objective is:

$\mathcal{J}_{\text{SHAPING}}(\theta) = \frac{1}{Z} \left( \sum_{j=1}^{N_{\text{off}}} \sum_{t=1}^{|\tau_j|} f(\hat{r}_{j,t}(\theta, \phi)) \cdot \hat{A}_j + \sum_{i=1}^{N_{\text{on}}} \sum_{t=1}^{|\tau_i|} \text{CLIP}(r_{i,t}(\theta), \hat{A}_i, \epsilon) \right)$

where $f(\cdot)$ is a nonlinear transformation applied to the off-policy importance ratio, and the on-policy term is unchanged (still using the original PPO clip, though the final LUFFY removes this clip as well).

The shaping function. The paper selects:

$f(x) = \frac{x}{x + \gamma}, \quad \gamma \in [0, 1]$

with $\gamma = 0.1$ as the default (selected through hyperparameter sweep over $\{0.05, 0.1, 0.2, 0.3, 0.5\}$ , with 0.1 performing best — Appendix E.4, Figure 9).

Why this specific function. The rational function $x/(x + \gamma)$ has precisely the properties needed:

When $x$ is large ( $\pi_\theta$ is high for a token), $f(x) \approx x/(x + 0.1) \approx 1$ (since $x \gg 0.1$ ), saturating at approximately 1. This prevents the model from over-reinforcing already-high-probability tokens.
When $x$ is small ( $\pi_\theta$ is low for a token), $f(x) \approx x/0.1 = 10x$ , amplifying the gradient for low-probability tokens. This is the key effect: tokens that the model considers unlikely get their learning signal boosted by approximately a factor of $1/\gamma = 10$ .
The function is smooth (differentiable), monotonically increasing (preserving the ordering of which tokens are more likely), and bounded above (preventing any single token from dominating the gradient).

Figure 2 (middle) shows $f(x)$ as a function of $\pi_\theta$ : it rises steeply for low probabilities and flattens for high probabilities, creating a "progressive" weighting that emphasizes learning from unfamiliar patterns.

Gradient analysis. The paper derives the gradient of the shaped off-policy objective to show exactly how the shaping function affects learning:

$\nabla_{\theta} \mathcal{J}_{\text{SHAPING-OFF}}(\theta) = \mathbb{E}_{\tau \sim \pi_{\phi}} \left[ f'(\pi_{\theta}) \frac{\pi_{\theta}}{\pi_{\phi}} \nabla_{\theta} \log \pi_{\theta} \cdot \hat{A}_j \right]$

where $f'(\pi_{\theta})$ is the derivative of $f$ with respect to $\pi_\theta$ . For the chosen shaping function, $f'(\pi_\theta) = \gamma/(\pi_\theta + \gamma)^2$ .

What this gradient does: for each off-policy token, the gradient pushes $\pi_\theta$ toward 1 for tokens from positive-advantage trajectories (and toward 0 for negative advantage, though all off-policy traces have positive advantage). The weighting factor is $f'(\pi_{\theta}) \pi_{\theta} / \pi_{\phi}$ . With $\pi_\phi = 1$ , this simplifies to $f'(\pi_{\theta}) \pi_{\theta} \cdot \nabla_\theta \log \pi_\theta \cdot \hat{A}_j$ .

To make the per-token effect concrete, the paper decomposes the gradient for a single action (token) $\tau'_{j,t}$ :

$\left| \frac{\partial \mathcal{J}_{\text{SHAPING-OFF}}(\theta)}{\partial M_{\theta}(\tau'_{j,t})} \right| \leq \mathbb{E}_{\tau \sim \pi_{\phi}} \left[ |f'(\pi_{\theta})| \pi_{\theta} (1 - \pi_{\theta}) \cdot |\hat{A}_j| \right]$

where $M_{\theta}(\tau'_{j,t})$ is the logit for that token.

What this inequality tells us: the magnitude of the gradient update for any token is bounded by $f'(\pi_{\theta}) \pi_{\theta} (1 - \pi_{\theta})$ times the advantage magnitude. The term $\pi_{\theta} (1 - \pi_{\theta})$ is the standard policy gradient weighting — it peaks at $\pi_{\theta} = 0.5$ and goes to zero for both very low and very high probabilities, meaning the model receives weak gradients for tokens it's either very confident or very unconfident about. This is the "vanishing gradient" problem in standard policy gradients.

How shaping fixes this. Figure 2 (right) compares the gradient weighting with and without shaping:

Without shaping ( $f(\pi_\theta) = \pi_\theta$ , so $f'(\pi_\theta) = 1$ ): the gradient weight is $\pi_\theta (1 - \pi_\theta)$ , which is near-zero for both $\pi_\theta \approx 0$ (the model finds the token very unlikely) and $\pi_\theta \approx 1$ (the model already finds it very likely).
With shaping ( $f(\pi_\theta) = \pi_\theta/(\pi_\theta + 0.1)$ ): the gradient weight is shifted upward, particularly for low $\pi_\theta$ . At $\pi_\theta = 0.01$ , the shaping function provides approximately 10× more gradient signal than the unshaped version. This ensures that even tokens the model initially considers extremely unlikely receive meaningful learning signals, provided they come from high-advantage trajectories.

The operational effect: the model is pushed harder to learn reasoning patterns it doesn't already know (low $\pi_\theta$ tokens from off-policy traces), while patterns it already knows well (high $\pi_\theta$ tokens) receive reduced gradient signals, preventing them from dominating the update. This maintains higher entropy throughout training (Figure 5, right) because the model never fully converges to a deterministic policy — it continues exploring low-probability actions that may represent better reasoning strategies.

Entropy behavior under shaping. Figure 5 (right) shows that LUFFY consistently sustains higher entropy compared to On-Policy RL throughout training. On-Policy RL entropy "rapidly converges to nearly zero after approximately 200 steps." LUFFY not only maintains higher entropy overall but exhibits "entropy fluctuations and even occasional increases, such as between steps 200 and 250." The paper interprets these fluctuations as evidence of the model "escaping local optima" by exploring previously low-probability actions that the shaping function has amplified, potentially discovering "pivotal tokens" (Zhong et al., 2025) that represent critical reasoning decisions.

Variance reduction analysis (informal). Appendix B.2 provides an informal analysis showing that the shaped importance weights have lower variance than the raw importance weights. The key argument: using a first-order Taylor expansion of $f(x) = x/(x + \gamma)$ around $x = 1$ (the expected value of the importance ratio when $\pi_\phi$ dominates $\pi_\theta$ ):

$\text{Var}[f(x)] \approx \left( \frac{\gamma}{(1+\gamma)^2} \right)^2 \text{Var}[x]$

With $\gamma = 0.1$ , the coefficient is $(0.1/1.21)^2 = 0.0068$ , meaning the variance of the shaped weights is reduced by approximately 147× compared to raw importance weights. This is because the shaping function compresses the range of weights — extremely large importance ratios (which occur when $\pi_\theta$ places high probability on tokens the teacher considered unlikely) are squashed toward 1, while extremely small ratios are amplified, narrowing the overall distribution.

The paper further verifies this with a special case where the raw weights follow an exponential distribution $p(x) = e^{-x}$ (motivated by the fact that importance weights tend to be small in practice since $\pi_\theta$ for teacher tokens is often small). Under this distribution, $\text{Var}[x] = 1$ (for the raw weights), while $\text{Var}[f(x)] < 1$ (proven analytically), confirming variance reduction.

Why lower variance matters: importance-weighted policy gradient estimators are notoriously high-variance because a few samples with extreme importance ratios can dominate the gradient estimate, causing unstable updates. By reducing variance, policy shaping makes the training more stable — the model receives a more consistent learning signal from off-policy data, preventing the destructive large parameter updates that could cause catastrophic forgetting or policy collapse.

Training Configuration and Hyperparameters

The paper provides detailed training configuration (Section 4, Appendix C):

Data construction. The training set starts from OpenR1-Math-220k (94k prompts with DeepSeek-R1-generated reasoning traces). Two filters are applied:

Remove generations longer than 8192 tokens,
Remove generations that Math-Verify marks as incorrect (the final extracted answer doesn't match the gold answer).

This filtering retains approximately 45k prompts with verified-correct off-policy reasoning traces. The prompts originally come from NuminaMath 1.5, a competition-math-focused dataset.

Rollout configuration. For each training step, the batch contains 64 prompts. For each prompt:

7 on-policy rollouts are generated by the current policy at temperature 1.0,
1 off-policy rollout is provided from the pre-computed DeepSeek-R1 trace.

Total batch size: 64 × 8 = 512 rollouts per step, with 7/8 from on-policy and 1/8 from off-policy.

Optimization hyperparameters:

Learning rate: constant $1 \times 10^{-6}$
Number of training steps: 500 for standard LUFFY, 860 for LUFFY† (extended training to match SFT+RL GPU budget)
KL divergence coefficient: $\beta = 0$ (removed)
Entropy loss coefficient: 0.01
PPO clip ratio: $\epsilon$ (standard value, though the on-policy clip is removed in the final version)
Shaping function parameter: $\gamma = 0.1$ , selected from $\{0.05, 0.1, 0.2, 0.3, 0.5\}$ via hyperparameter sweep (Appendix E.4, Figure 9)

Model configuration for Qwen2.5-Math-7B. Since Qwen2.5-Math has a native context length of 4096 tokens and DeepSeek-R1's reasoning traces can be significantly longer, the paper adjusts the model's position encoding:

Rope theta: increased from 10000 to 40000 (extending the effective context window by increasing the wavelength of rotary position embeddings),
Context window: extended to 16384 tokens.

These modifications allow the model to handle longer reasoning sequences during training without truncation.

Computational resources. Standard LUFFY training requires $77 \times 8 = 616$ GPU hours on A100s (8 GPUs for 77 hours). LUFFY† requires $130 \times 8 = 1040$ GPU hours. By comparison:

SFT (3 epochs on 64K traces): $24 \times 8 = 192$ GPU hours,
SFT+RL (SFT + 500 more RL steps): $130 \times 8 = 1040$ GPU hours (same as LUFFY†),
RL w/ SFT Loss: $133 \times 8 = 1064$ GPU hours.

The paper notes that SFT+RL and RL w/ SFT loss have higher GPU costs because "excessively long generations induced by rigid imitation... substantially increase the computational overhead during the RL roll-out stage" — SFT causes the model to generate much longer outputs during RL, increasing generation time per rollout.

Evaluation configuration. For in-distribution benchmarks:

AIME 2024, AIME 2025, AMC: avg@32 (average accuracy over 32 sampled solutions),
MATH-500, Minerva, OlympiadBench: pass@1 (accuracy of a single greedy sample).
Temperature: 0.6 for all evaluations.

For out-of-distribution benchmarks (ARC-c, GPQA-diamond, MMLU-Pro): multiple-choice options are shuffled to prevent position bias, and pass@1 is reported.

Why these specific choices matter. The 7:1 on-policy-to-off-policy ratio reflects a deliberate balance — the design space includes extremes like pure SFT (0:8) and pure RL (8:0), as well as various mixed ratios not explored in the paper. The choice of only one off-policy trace is partly computational (less data to load and process) and partly based on the empirical finding that "one trajectory is already strong." The paper acknowledges that "extending off-policy guidance to multiple trajectories and multiple teachers could help the performance even further" as a limitation.

The removal of the KL penalty ( $\beta = 0$ ) is a significant departure from the original PPO/GRPO formulation. The paper implicitly justifies this by noting that "recent practices have increasingly omitted the KL divergence term" and that without it, methods become "somewhat less 'On-Policy'" — the policy is allowed to deviate more freely from its starting point, which is necessary when learning from a very different distribution (the teacher's reasoning traces). The entropy loss coefficient of 0.01 provides some regularization to prevent complete determinism, but without the KL penalty, the model can potentially forget its initial capabilities — a risk that the paper's experiments suggest is manageable but not explicitly analyzed.

Summary of Design Choices and Their Justifications

Mixed-group advantage computation over separate SFT + RL objectives: enables dynamic, automatic balance between imitation and exploration without requiring an explicit curriculum or schedule. The balance emerges from the group statistics — when on-policy rollouts are poor, off-policy advantages dominate; when on-policy improves, off-policy naturally fades.
$\pi_\phi = 1$ over computing true teacher probabilities: avoids tokenizer mismatch between teacher and student models, enables use of pre-computed reasoning datasets without storing/accessing teacher model probabilities, and preserves the theoretical convergence guarantee under bounded importance weights.
One off-policy rollout per prompt over multiple: computationally efficient, and empirically sufficient — the paper finds "one trajectory is already strong." The mixing of on-policy and off-policy in the group computation means the off-policy signal gets amplified when the model is struggling, making even a single trajectory impactful.
$f(x) = x/(x + \gamma)$ with $\gamma = 0.1$ over alternative functions or constant scaling: the rational function saturates for high probabilities (preventing over-reinforcement of already-learned patterns) while amplifying low-probability tokens (ensuring novel reasoning behaviors get sufficient gradient signal). The specific $\gamma$ value was chosen through hyperparameter sweep and represents a tradeoff — too small $\gamma$ amplifies too aggressively (causing instability), too large $\gamma$ provides insufficient amplification (allowing entropy collapse).
Removing the on-policy clip over keeping it: off-policy tokens from the teacher come from a very different distribution than the student's current policy, meaning the importance ratio is often very small (far outside the typical clip range $[1-\epsilon, 1+\epsilon]$ ). Clipping these small ratios would suppress most of the off-policy learning signal. The removal allows the model to make larger policy updates toward the teacher, particularly early in training when the policy distributions are most different.
7:1 sampling ratio over other splits: the paper's baseline uses 8 on-policy rollouts, so 7 + 1 keeps the total batch size and on-policy sample count comparable for fair comparison. The paper acknowledges that exploring different ratios and multiple teachers is future work.
Temperature 1.0 for rollout generation over lower temperatures: higher temperature increases exploration diversity in on-policy rollouts, which is critical for discovering correct solutions that the model might miss with more deterministic sampling. Since the base model's pass@1 on many problems is low, temperature 1.0 improves the chances of occasionally generating correct solutions and receiving positive rewards.

4. Key Insights and Innovations

Innovation 1: Reframing Off-Policy Data as Dynamic Scaffolding Rather Than Static Targets

The paper's most fundamental conceptual move is not the mechanics of mixing on-policy and off-policy data — mixing is straightforward — but the reframing of what off-policy traces are in the learning process. Prior work treats off-policy reasoning demonstrations from stronger models in one of two ways: as supervised targets for SFT-based distillation (DeepSeek-R1's own distillation pipeline, OpenR1-Qwen-7B), or as initialization for subsequent RL (SFT+RL). Both framings are static: the off-policy data is consumed once, either to clone behavior or to warm-start a policy, after which the model is on its own.

LUFFY reframes off-policy traces as dynamic scaffolding — structural support that selectively bears weight only when the model's own policy cannot. This is not a training trick but a different way of thinking about what the teacher-student relationship should be. Instead of "learn to behave like the teacher" (SFT) or "learn broad patterns from the teacher, then refine them yourself" (SFT+RL), LUFFY implements "the teacher is always in the room, but you only look at their work when you're stuck." The mechanism that enables this — mixed-group advantage computation — is described in Section 3, but the reason it's intellectually distinctive is that it inverts the usual teacher-student hierarchy: the student's own successes take precedence over the teacher's demonstrations, and the teacher's role naturally diminishes as the student improves, without any explicit curriculum design.

This reframing matters because it resolves a tension that the paper's experiments make visible but the field has not systematically articulated. SFT-based distillation produces models that score well on in-distribution evaluations (SFT achieves 44.1 on in-distribution math, Table 1) but generalize poorly: SFT's OOD average is 47.5 versus On-Policy RL's 57.3. The paper's analysis (Appendix F.2, Figure 12) shows why — SFT models "memorize reasoning patterns rather than learning generalizable reasoning capability," and their performance degrades under test-time temperature scaling while RL-based models improve. Pure on-policy RL (On-Policy RL: 45.5 in-distribution, 57.3 OOD) generalizes well but cannot learn from data it cannot generate — it fails entirely on weak models with hard problems (Section 5.2, Figure 4). The static framing of off-policy data forces a choice between these failure modes. The dynamic scaffolding framing escapes the choice.

One way to see the significance of this reframing is to ask: what would a researcher who internalized this idea do differently from someone following the distillation-then-RL paradigm? They would stop thinking about off-policy data as "training data" and start thinking about it as "contingent guidance" — the design question shifts from "what's the optimal ratio of teacher data to student data" to "under what conditions should the model attend to the teacher versus trust itself." The paper's answer — conditions determined by group-relative advantage — is one implementation, but the reframing opens a broader design space: other conditioning variables (uncertainty estimates, difficulty predictions, per-step verifier scores) could determine when off-policy guidance is most valuable.

Tie to evidence: The dynamic effect is visible in Figure 5 (left): LUFFY initially shows lower training rewards than On-Policy RL (the model is adjusting to unfamiliar reasoning patterns in the off-policy data), then surpasses it as on-policy exploration takes over. The automatic balance is visible in the ablation (Table 4): removing the mixed-policy objective (i.e., pure On-Policy + Shaping) yields no benefit (42.0 avg), demonstrating that the dynamic mixing — not just the regularization — is what drives the gains. The contrast with RL w/ SFT Loss (40.1 avg, Table 1) is particularly telling: that method provides exactly the same data (64K on-policy × 7 + 64K off-policy per step) but fails because it treats off-policy traces as static targets (SFT loss) rather than dynamic scaffolding.

Innovation 2: Diagnosing Entropy Collapse as the Central Failure Mode of Mixed-Policy Training — and Providing a Principled Remedy

The second conceptual contribution is the identification and characterization of entropy collapse as a distinct, previously unexamined failure mode in mixed-policy RLVR, and the introduction of policy shaping via regularized importance sampling as a theoretically motivated solution. This is not an incremental hyperparameter tuning discovery; it's a diagnostic contribution that names a problem the field didn't know it had and explains why naive approaches to incorporating off-policy data systematically underperform.

What was the prior assumption? The default approach to combining off-policy data with RL — whether through multi-task SFT+RL loss or SFT pretraining followed by RL — implicitly assumes that off-policy learning and on-policy exploration are compatible objectives: the model can simultaneously imitate expert behavior and explore its own policy. The paper's entropy analysis (Figure 2 left, Figure 5 right) shows this assumption is false: mixed-policy training causes dramatically faster entropy collapse than pure on-policy RL because of an asymmetric gradient phenomenon that the paper explicitly characterizes. High-probability tokens (those the model already finds natural) receive disproportionately large gradient signals from off-policy traces, causing rapid convergence to a narrow, surface-level imitation of the teacher's output distribution. Low-probability tokens (those representing genuinely new reasoning strategies) receive vanishingly small gradients and are effectively ignored.

This is a mechanistic explanation for why mixed training can be worse than either pure approach: it combines SFT's tendency toward rigid imitation with RL's tendency toward entropy collapse, producing a model that is more deterministic than either SFT or RL alone, and whose determinism is concentrated on the most superficial features of the off-policy data. The paper shows this in Figure 6: Mixed-Policy without shaping initially outperforms On-Policy RL (because it benefits from off-policy guidance) but then plateaus and converges to the same final performance — the early advantage is squandered because the model stops exploring before it can internalize the deeper reasoning patterns.

Why is this a fundamental diagnostic rather than a minor observation? Entropy collapse in RL is well-known (PPO, GRPO, and related algorithms all battle with policy determinism), but the paper identifies a specific, off-policy-induced mechanism for it that is distinct from the standard on-policy entropy collapse. Standard collapse occurs because the policy gradient naturally drives the model toward deterministic optimal actions — a fundamental property of policy optimization. The LUFFY paper's observed collapse is faster than standard because of the interaction between on-policy and off-policy gradients: the off-policy signal provides a "shortcut" to high probability for tokens the model already knows, accelerating the collapse toward surface-level patterns before the model has time to explore.

The policy shaping remedy is conceptually elegant because it targets the gradient asymmetry directly rather than patching it with entropy bonuses. The shaping function $f(x) = x/(x + \gamma)$ is not an arbitrary nonlinearity — it's a variance-reducing reweighting (proven informally in Appendix B.2) that specifically amplifies the gradient contribution of low-probability tokens from off-policy traces while saturating high-probability tokens. This is fundamentally different from adding an entropy bonus to the loss (which the paper does anyway, with coefficient 0.01): an entropy bonus encourages any exploration, uniformly, while policy shaping encourages directed exploration toward reasoning patterns the teacher uses but the student hasn't yet learned. The difference matters because uniform exploration in token space is combinatorially unlikely to discover complex reasoning strategies — the model needs to be pushed specifically toward the low-probability tokens that represent valuable new behaviors.

Tie to evidence: Figure 5 (right) shows LUFFY maintaining higher entropy than On-Policy RL throughout training, with "fluctuations and even occasional increases" that the paper interprets as exploration-driven escapes from local optima. Figure 6 shows that Mixed-Policy without shaping plateaus early while LUFFY continues improving — the shaping function converts what would be a transient early advantage into sustained improvement. The hyperparameter study (Appendix E.4, Figure 9) shows that performance is sensitive to γ: γ = 0.1 is optimal, with both smaller (0.05, too aggressive amplification causing instability) and larger values (0.3–0.5, insufficient amplification allowing entropy collapse) degrading performance. This non-monotonic relationship confirms that shaping is not just "adding regularization" but actively managing the imitation-exploration tradeoff at a specific operating point. The ablation (Table 4) quantifies the shaping contribution: Mixed-Policy alone achieves 44.4 avg, adding shaping lifts to 47.8 (+3.4 points), and adding clip removal lifts to 50.1 (+2.3 points) — each component addresses a distinct aspect of the collapse problem.

Innovation 3: Demonstrating That Off-Policy Guidance Enables RLVR Where On-Policy RL Completely Fails — Reframing the "Weak Model" Problem

The third contribution is primarily empirical but with significant conceptual implications: the demonstration in Section 5.2 that LUFFY can successfully train models in regimes where on-policy RLVR training rewards collapse to zero. This is not about achieving higher scores on an established benchmark; it's about expanding the applicability frontier of RLVR from strong base models to models that lack the foundational reasoning behaviors necessary to bootstrap through exploration alone.

What was the prior understanding? The RLVR paradigm — including DeepSeek-R1, SimpleRL-Zero, Oat-Zero, and other recent methods — is empirically validated on base models that already possess non-trivial reasoning capabilities. DeepSeek-R1 started from DeepSeek-V3, a frontier-scale model. SimpleRL-Zero and Qwen2.5-Math-7B already had 19.0 average on the six math benchmarks before RL training (Table 1). The implicit assumption, reinforced by theoretical work from Zhao et al. (2025) and Yue et al. (2025), is that RL amplifies existing behaviors — if a behavior doesn't exist in the base model's output distribution, RL cannot create it.

LUFFY's results on LLaMA-3.1-8B (Section 5.2, Figure 4) directly challenge the scope of this assumption. On the Hard training subset, on-policy RLVR fails completely — training rewards collapse to zero because the model never generates correct solutions to learn from. This is the "cold start" problem in RLVR: no positive signal means no learning, and no learning means no future positive signals. It's a deadlock. LUFFY breaks this deadlock not by making the model smarter, but by providing an alternative source of positive advantage — the off-policy traces serve as a bootstrap signal that keeps the gradient flowing even when on-policy rollouts are entirely incorrect. The model initially imitates the teacher to receive positive advantages, which raises its policy into a region of the output space where it occasionally produces correct solutions, at which point on-policy successes begin contributing to the learning signal.

Why is this a conceptual advance rather than just a performance gain? It reframes the relationship between model capability and training methodology. The dominant narrative — RL amplifies existing capabilities — implies that model capability is a prerequisite for RLVR. LUFFY shows that with appropriate off-policy guidance, model capability is partially an output of training, not just an input. A model that initially cannot solve a single problem in a dataset can, through guided exploration, develop problem-solving strategies it never independently discovered. This doesn't contradict Zhao et al. (2025) — the model still isn't creating behaviors ex nihilo — but it shows that "existing capabilities" can include abilities the model has never demonstrated because they're too deeply buried in the tail of its output distribution for random exploration to uncover them. The off-policy guidance functions like a search heuristic: it points the model toward regions of the output space where useful behaviors exist, enabling it to discover capabilities that pure exploration would never find.

The failure boundary is informative. On the Easy subset, both On-Policy RL and LUFFY succeed, though LUFFY achieves higher final performance (13.2 avg vs. 9.6 avg, Table 3). On the Hard subset, On-Policy RL collapses while LUFFY continues learning. This establishes a capability gradient: there exists a threshold of problem difficulty beyond which pure on-policy exploration fails, and off-policy guidance extends the trainable difficulty range. The paper doesn't fully map this threshold (it's characterized for one model on one dataset), but the existence of the threshold is itself an important finding — it means the "on-policy RL works" narrative is implicitly conditioned on a model-difficulty match that is not guaranteed in practice, particularly when trying to train smaller or weaker models on challenging tasks.

Tie to evidence: Figure 4 shows the training reward curves: On-Policy RL on Easy rises steadily, On-Policy RL on Hard collapses to zero, and LUFFY on both subsets shows stable improvement. This is a qualitative difference (failure vs. success) not a quantitative one. Table 3 quantifies the downstream impact on LLaMA-3.1-8B: while On-Policy RL's average across six benchmarks is 9.6 (barely above the Instruct baseline of 17.1? Actually below — the base model's Instruct version scores 17.1 on the benchmarks, and On-Policy RL reduces this to 9.6, suggesting the RL training on a simplified dataset actually degraded capabilities on harder benchmarks), LUFFY achieves 13.2. The absolute numbers are low because LLaMA-3.1-8B is fundamentally weak at math, but the relative pattern — LUFFY succeeding where on-policy RL fails — is what carries the conceptual weight. The extension to multiple base models (Figure 3) shows this is not model-specific: LUFFY improves over On-Policy RL on Qwen2.5-Math-1.5B (+8.0), Qwen2.5-Instruct-7B (+5.5), and LLaMA-3.1-8B (+3.6), with the largest absolute gains on the smaller, weaker models where on-policy RL provides the least benefit.

Innovation 4: The Unified Advantage Computation as a Mechanism for Emergent Curriculum Learning

The fourth contribution is more subtle but methodologically significant: the recognition that computing advantages over a mixed on-policy/off-policy group creates an emergent curriculum without any explicit difficulty estimation, scheduling, or hyperparameter annealing. This is a design principle rather than a method, and it explains why LUFFY outperforms alternatives like SFT+RL that use the same data but structure the learning differently.

What is the standard approach to curriculum in RLVR? Most RLVR methods either apply uniform training across all prompts (the GRPO default), or use explicit curriculum strategies like training on easier problems first (curriculum learning), using SFT to warm-start on easier problems before applying RL to harder ones (SFT+RL), or filtering training data by difficulty and gradually increasing it. These approaches require defining "difficulty" — typically through problem source labels, answer correctness rates, or solution lengths — and designing a schedule for introducing harder data.

LUFFY achieves a similar effect implicitly through the mixed-group advantage computation (Equation 4). Consider what happens to the advantage signal for off-policy traces as the model's on-policy performance changes:

On problems where the model is weak (most on-policy rollouts incorrect, reward = 0): the off-policy trace (reward = 1) receives a large positive advantage relative to the group mean, producing a strong imitation gradient. The model is pushed hard to learn from the teacher on these problems.
On problems where the model is strong (most on-policy rollouts correct, reward = 1): the off-policy trace still has reward = 1 but receives a much smaller positive advantage (it's only slightly above the group mean), producing a weak imitation gradient. The model's own correct solutions dominate the update. The model effectively "graduates" from needing teacher guidance on these problems.
On problems where the model is inconsistent (some on-policy correct, some incorrect): both on-policy successes and the off-policy trace contribute meaningful gradients, creating a blended learning signal.

This is an emergent curriculum because the model implicitly spends more learning capacity on problems it's currently bad at (where the off-policy advantage is large) and less on problems it's already good at (where on-policy successes provide sufficient signal). No difficulty labels, no scheduling logic, no hyperparameter for when to "turn off" the teacher — the group statistics handle it automatically through gradient magnitudes, not through explicit problem selection.

Why is this conceptually important? It reframes curriculum design from a data-selection problem to an objective-design problem. The question shifts from "in what order should we present training examples" to "how should we structure the learning objective so that the model naturally attends more to examples where guidance is most needed." This is a more scalable approach because it doesn't require per-problem difficulty estimation (which the paper's earlier work on compute-optimal test-time scaling, described in the prior sections, showed is expensive and imperfect). The curriculum emerges from the interaction between the model's current capabilities and the fixed structure of the mixed-group advantage — as the model improves, the curriculum automatically adapts.

The contrast with SFT+RL makes this concrete. SFT+RL first trains on all off-policy data uniformly (SFT), then applies RL on a separate dataset. The SFT phase treats all problems identically regardless of whether the model needs heavy guidance (hard problems) or light guidance (easy problems where it could learn independently). The RL phase then inherits a policy that may have overfit to surface patterns on easy problems while "running out of capacity" before learning deep reasoning on hard ones. The paper's evidence (Table 2: SFT+RL uses 135K off-policy traces to LUFFY's 64K, yet scores 48.2 vs. 50.1) suggests that the emergent curriculum in LUFFY is more data-efficient because it allocates learning attention where it's most useful.

Tie to evidence: The training dynamics in Figure 5 provide suggestive evidence for the emergent curriculum effect. Generation length — which correlates with adoption of off-policy reasoning patterns — increases gradually in LUFFY (consistent with gradual learning from off-policy guidance) rather than spiking immediately (as would happen with uniform imitation). The entropy curve shows the model exploring throughout training rather than converging rapidly, consistent with a curriculum where hard problems continue providing challenging learning signals rather than being exhausted early. The training reward curve (Figure 5, left) shows LUFFY initially below On-Policy RL and then crossing above — this is exactly what one would expect from an emergent curriculum: early training emphasizes imitation on hard problems (dragging down average reward because the model hasn't yet integrated the new patterns), while later training benefits from improved capabilities across the difficulty spectrum.

Assessment: How Fundamental Are These Contributions?

Innovation 1 (dynamic scaffolding) is the most fundamental — it changes how researchers should think about off-policy data in RLVR from static targets to adaptive support. This is a framing contribution that, if adopted, would influence how future methods are designed, not just how they're evaluated. It builds on known concepts from RL (importance sampling for off-policy learning) but applies them in a way that creates behavior qualitatively different from prior approaches.

Innovation 2 (entropy collapse diagnosis) is a diagnostic contribution that fills an explanatory gap: it explains why naive mixed-policy approaches underperform and provides a principled remedy. The shaping function itself is incrementally derived from standard importance sampling, but the diagnosis — that off-policy guidance specifically accelerates entropy collapse through an asymmetric gradient mechanism — is novel and may generalize to other domains where mixed-distribution training is attempted.

Innovation 3 (weak model training) is an empirical contribution with theoretical implications: it shows that the "RL amplifies existing behaviors" boundary is more porous than previously understood, and that off-policy guidance can bootstrap capabilities from behaviors too improbable for random exploration to discover. This is a counterexample to a narrative that was solidifying (Zhao et al., 2025; Yue et al., 2025) and should prompt more nuanced theoretical work on exactly what conditions enable RLVR to create versus merely amplify capabilities.

Innovation 4 (emergent curriculum) is a design-principle contribution: it identifies an implicit property of the mixed-group advantage computation that explains data efficiency and distinguishes LUFFY from alternatives that use more data less effectively. It is the most speculative of the four — the paper provides suggestive evidence but does not rigorously ablate the curriculum effect against explicit curriculum baselines — but it points toward a general principle for structuring objectives when combining heterogeneous data sources.

5. Experimental Analysis

Evaluation Methodology

Dataset. The training set is a filtered subset of OpenR1-Math-220k (94k prompts with DeepSeek-R1-generated reasoning traces). Two filters are applied: generations longer than 8192 tokens are removed, and generations Math-Verify marks as incorrect are discarded, yielding approximately 45k prompts with verified-correct off-policy reasoning traces. The prompts originate from NuminaMath 1.5, a competition-math-focused dataset. For the LLaMA-3.1-8B weak-model experiments (Section 5.2), two stratified subsets are constructed based on the length of DeepSeek-R1's correct solution: the Easy set (problems R1 solves within 2k tokens, 7.3k prompts) and the Hard set (problems R1 solves within 4k tokens, 25.4k prompts), with length serving as a proxy for difficulty — problems requiring longer reasoning traces are presumed harder.
Base model(s). The primary model is Qwen2.5-Math-7B (Yang et al., 2024), selected because it is the standard backbone in prior RLVR work (SimpleRL-Zero, Oat-Zero, PRIME-Zero, OpenReasonerZero). Extensions are tested on Qwen2.5-Math-1.5B (a smaller math-specialized model), Qwen2.5-Instruct-7B (a general instruction-tuned model), and LLaMA-3.1-8B (a weaker model for studying the capability boundary). For Qwen2.5-Math models, the context window is extended from the native 4096 to 16384 tokens by increasing the rope theta from 10000 to 40000, since DeepSeek-R1's reasoning traces can be longer than the default window. For the off-policy teacher, DeepSeek-R1 serves as the source of high-quality reasoning traces, though the teacher policy itself is never evaluated or run during LUFFY training ( $\pi_\phi$ is set to 1).
Metrics. Six competition-level math reasoning benchmarks are used for in-distribution evaluation: AIME 2024, AIME 2025, AMC (Li et al., 2024), MATH-500 (Hendrycks et al., 2021), Minerva (Lewkowycz et al., 2022), and OlympiadBench (He et al., 2024). For AIME 2024, AIME 2025, and AMC, avg@32 is reported (average accuracy over 32 sampled solutions per problem) because the test sets are relatively small and averaging over multiple samples reduces variance. For MATH-500, Minerva, and OlympiadBench, pass@1 is reported (accuracy of a single sample per problem). Three out-of-distribution benchmarks test generalization: ARC-c (Clark et al., 2018, open-domain reasoning), GPQA-diamond (Rein et al., 2024, graduate-level science knowledge, denoted GPQA*), and MMLU-Pro (Wang et al., 2024, reasoning-focused academic questions). Multiple-choice options are shuffled to prevent position bias. All evaluations use temperature 0.6. The reward signal during training uses Math-Verify, a rule-based answer extraction and comparison tool that checks whether the final answer in \boxed{} matches the ground truth (binary 1/0). No format or length rewards are included. The primary aggregated metric is the arithmetic mean across the six in-distribution benchmarks and across the three out-of-distribution benchmarks.
Baselines. Baseline methods fall into four categories. (1) Prior RLVR methods (results taken from published papers, not re-run): SimpleRL-Zero (Zeng et al., 2025) trains Qwen2.5-Math-7B with rule-based reward; Oat-Zero (Liu et al., 2025) removes standard deviation normalization in GRPO advantage and token-level normalization in the policy loss; PRIME-Zero (Cui et al., 2025) uses implicit process rewards from outcome labels; OpenReasonerZero (Hu et al., 2025) is a recent open-source RLVR implementation. The paper notes these methods differ in dataset composition (source and difficulty) and optimization strategies, so comparisons are approximate. (2) On-Policy RL replication: the authors train a pure GRPO baseline using Dr.GRPO with the same reward function, same data (only the prompts, without off-policy traces), and identical hyperparameters (8 on-policy rollouts per prompt, β = 0, no length normalization, no standard error normalization, entropy coefficient 0.01). This is the primary controlled comparison. (3) Alternative off-policy learning methods: SFT trains the model for 3 epochs on the same prompts and DeepSeek-R1 reasoning traces as LUFFY; RL w/ SFT Loss computes both SFT loss on 1 off-policy trace and RL loss on 7 on-policy traces per prompt; SFT+RL performs SFT first (3 epochs on 64K traces) then RL (500 steps on a held-out 49K prompt subset). (4) Qwen2.5-Math-7B base and instruct models provide reference points for the model before any RL training.
Generation budget / compute accounting. Fair comparison is enforced by maintaining a constant 8 rollouts per prompt across all RL-trained models. On-Policy RL uses 8 on-policy rollouts; LUFFY uses 7 on-policy + 1 off-policy rollout, keeping the total group size fixed. All models are trained for 500 steps (except LUFFY†, trained for 860 steps to match the GPU-hour budget of SFT+RL). GPU hours are reported in Table 2: LUFFY uses 77 × 8 = 616 GPU hours on A100s, SFT uses 24 × 8 = 192, SFT+RL uses 130 × 8 = 1040, and RL w/ SFT Loss uses 133 × 8 = 1064. The paper notes that SFT+RL and RL w/ SFT Loss incur higher costs because SFT-induced rigid imitation causes excessively long generations during the RL rollout stage, increasing generation time per step. Training batch size is 64 prompts, with rollouts generated at temperature 1.0. Constant learning rate is 1 × 10⁻⁶. All training uses 8 A100 GPUs.
Cross-validation / statistical protocol. Statistical significance for the main results (Table 1) is tested using paired bootstrapping resampling with 1000 iterations (Koehn, 2004), comparing LUFFY and LUFFY† against all baseline methods. The null hypothesis is that observed differences are due to random sampling variation. Results marked with * indicate p < 0.05. No cross-validation is used for strategy selection (unlike the compute-optimal paper), since LUFFY is a single training recipe rather than an adaptive policy that selects among strategies — the only hyperparameter sweep reported is for γ in the shaping function (Appendix E.4), which is done on validation performance.

Main Quantitative Results

Comparison Against Prior RLVR Methods and On-Policy RL

Headline result. LUFFY achieves an average score of 50.1 across six in-distribution math benchmarks on Qwen2.5-Math-7B, outperforming the best prior RLVR method (Oat-Zero, 43.7) by +6.4 points and the authors' own On-Policy RL baseline (45.5) by +4.6 points (Table 1). LUFFY† (extended training to match SFT+RL GPU budget) achieves 50.4 average.

Per-benchmark breakdown (Table 1):

AIME 2024: LUFFY scores 29.4, compared to Oat-Zero's 33.4 (best prior method, -4.0) and On-Policy RL's 25.1 (+4.3). LUFFY† reaches 30.7. This is the one benchmark where a prior method outperforms LUFFY, though the paper does not discuss this exception.
AIME 2025: LUFFY scores 23.1, compared to OpenReasonerZero's 15.0 (+8.1) and On-Policy RL's 15.3 (+7.8). The paper highlights this as evidence of "generalization to internalize nuanced reasoning behaviors from off-policy traces," since AIME 2025 was released after DeepSeek-R1's training data cutoff (implicitly — the paper doesn't state this but emphasizes the newness of the test set).
AMC: LUFFY scores 65.6 vs. On-Policy RL's 62.0 (+3.6). LUFFY† reaches 66.2, the highest across all methods.
MATH-500: LUFFY scores 87.6 vs. On-Policy RL's 84.4 (+3.2). SFT+RL achieves 87.2, nearly matching LUFFY on this benchmark.
Minerva: LUFFY scores 37.5 vs. On-Policy RL's 39.3 (-1.8). This is the only benchmark where LUFFY underperforms On-Policy RL. The paper does not analyze this reversal. SFT achieves 40.8, the best among all methods on this benchmark.
OlympiadBench: LUFFY scores 57.2 vs. On-Policy RL's 46.8 (+10.4). This is LUFFY's largest single-benchmark gain, accounting for approximately 38% of the total improvement over On-Policy RL.

Key observation within the benchmark results: LUFFY's gains are unevenly distributed. The largest gains are on OlympiadBench (+10.4) and AIME 2025 (+7.8), both challenging competition-level benchmarks where off-policy guidance plausibly provides the most value. Gains on MATH-500 (+3.2) and AMC (+3.6) are more modest. Minerva shows a slight regression (-1.8). The paper does not analyze why performance improves unevenly or why Minerva regresses — these patterns are simply reported without interpretation.

Out-of-distribution results. On three OOD benchmarks, LUFFY achieves 57.8 average, outperforming the best prior RLVR method (OpenReasonerZero, 51.6) by +6.2 points and On-Policy RL (57.3) by only +0.5 points. The near-parity with On-Policy RL on OOD tasks — despite a 4.6-point gap on in-distribution tasks — is notable and not discussed. Specifically:

ARC-c: LUFFY 80.5 vs. On-Policy RL 82.3 (-1.8). On-Policy RL actually performs best here.
GPQA:* LUFFY 39.9 vs. On-Policy RL 40.4 (-0.5). Essentially tied.
MMLU-Pro: LUFFY 53.0 vs. On-Policy RL 49.3 (+3.7). This benchmark accounts for essentially all of LUFFY's OOD advantage over On-Policy RL.

The LUFFY† extended training improves OOD performance substantially to 61.8, with GPQA* jumping from 39.9 to 49.0 (+9.1). This suggests that the benefits of off-policy guidance for generalization may require longer training to manifest fully — a point the paper does not develop.

Statistical significance. LUFFY and LUFFY† are noted as significantly better than all baselines (p < 0.05) on the in-distribution average. The per-benchmark significance is not reported, so we cannot tell whether the -1.8 on Minerva or -4.0 on AIME 2024 vs. Oat-Zero are significant differences.

Comparison Against Alternative Off-Policy Learning Methods

Headline result. LUFFY outperforms SFT (44.1, +6.0), RL w/ SFT Loss (40.1, +10.0), and SFT+RL (48.2, +1.9) on in-distribution math benchmarks, while requiring substantially fewer GPU hours and less off-policy data than SFT+RL (Table 2: 77×8 vs. 130×8 GPU hours, 64K vs. 135K off-policy traces). On OOD benchmarks, the gap is much larger: LUFFY 57.8 vs. SFT+RL 44.8 (+13.0), demonstrating that LUFFY's dynamic approach to off-policy learning preserves generalization capability that SFT+RL sacrifices.

Per-method comparison (Table 1):

SFT alone: Achieves 44.1 in-distribution average — comparable to On-Policy RL (45.5) and Oat-Zero (43.7), suggesting SFT distillation from DeepSeek-R1 produces competent but not outstanding performance. The OOD average is 47.5, substantially below On-Policy RL's 57.3 (-9.8). This gap is the paper's primary evidence that SFT "imitates rigidly" and fails to generalize. The paper's analysis (Appendix F.1, Figure 10) shows SFT generates much longer outputs on average (4,646 tokens vs. 2,832 for LUFFY) and that this is especially pronounced on incorrect answers, suggesting SFT produces "overly long and ultimately unproductive reasoning traces" when it cannot solve the problem.
RL w/ SFT Loss: Achieves only 40.1 in-distribution — worse than pure On-Policy RL (45.5) and pure SFT (44.1). This is a negative result: naively adding SFT loss to RL training is worse than either approach alone. The paper explains this through generation length analysis (Appendix F.1, Figure 11): RL w/ SFT Loss causes generation length to spike early in training and exceed even the off-policy trace length, suggesting the model converges to a pathological imitation pattern that inflates output length without improving correctness.
SFT+RL: Achieves 48.2 in-distribution — the closest baseline to LUFFY (50.1). Inspection of per-benchmark results reveals an interesting pattern: SFT+RL outscores LUFFY on AIME 2025 (23.1 tie), essentially ties on MATH-500 (87.2 vs. 87.6), and is close on AMC (62.7 vs. 65.6). The main gaps are on OlympiadBench (50.4 vs. 57.2) and AIME 2024 (25.8 vs. 29.4). On OOD benchmarks, SFT+RL scores 44.8 vs. LUFFY's 57.8, a catastrophic gap of -13.0 points, with the largest difference on MMLU-Pro (37.7 vs. 53.0). This pattern suggests SFT+RL overfits to in-distribution reasoning patterns at the expense of general reasoning capability — exactly the failure mode LUFFY's policy shaping is designed to prevent.

Resource efficiency (Table 2). LUFFY achieves its 50.1 average using 77×8 GPU hours and 64K off-policy traces (one per prompt, used once as the mixed-group reference). SFT+RL achieves 48.2 using 130×8 GPU hours (69% more) and 135K off-policy traces (111% more). The paper attributes SFT+RL's higher cost to "excessively long generations induced by rigid imitation... which substantially increase the computational overhead during the RL roll-out stage." LUFFY†, trained for 860 steps to match SFT+RL's GPU budget, achieves 50.4, widening the gap to +2.2 points.

Test-time scaling behavior (Appendix F.2, Figure 12). The paper evaluates pass@8 accuracy on the combined AIME 2024 + AMC set under varying generation temperatures (0.1, 0.3, 0.5, 0.7, 1.0). Both RL-based methods (On-Policy RL and LUFFY) show pass@8 improving with temperature — a hallmark of exploration capability and potential for test-time compute scaling. In contrast, SFT's pass@8 degrades at higher temperatures (from near-LUFFY performance at temperature 0.1 to substantially worse at 1.0), indicating that SFT memorizes a narrow distribution of "correct-looking" outputs that doesn't generalize to diverse sampling conditions. At temperature 1.0, LUFFY's pass@8 is approximately 40% vs. SFT's approximately 25%. This is the paper's most direct evidence that LUFFY learns a genuinely exploratory policy while SFT learns a fragile imitation policy — a finding consistent with concurrent work (Chu et al., 2025; Chen et al., 2025) showing SFT memorizes while RL generalizes.

Extension to Multiple Base Models

Headline result (Figure 3, Table 3). LUFFY generalizes across model families and scales, consistently outperforming both SFT and On-Policy RL on every model tested.

Qwen2.5-Math-1.5B (small math model): LUFFY achieves 38.0 average, beating SFT (31.9) by +6.1 and On-Policy RL (30.0) by +8.0. The On-Policy RL result is particularly weak — at 30.0, it barely improves over the base model (16.0) and significantly underperforms the instruct version (35.7), suggesting that on-policy RLVR alone is insufficient for a 1.5B-parameter math model on this task distribution. LUFFY's gains are distributed across benchmarks: AIME 2024 (16.0 vs. SFT's 11.7), AMC (47.1 vs. SFT's 37.8), MATH-500 (80.2 vs. SFT's 70.6), Olympiad (41.0 vs. SFT's 31.3). This demonstrates that off-policy guidance is particularly valuable for smaller models where on-policy exploration is least effective.

Qwen2.5-Instruct-7B (general instruction model): LUFFY achieves 40.7 average, beating SFT (29.0) by +11.7 and On-Policy RL (35.2) by +5.5. Notably, SFT actually degrades performance relative to the base instruct model (34.4 → 29.0), suggesting that naive imitation of DeepSeek-R1's math-specific reasoning patterns interferes with the model's general instruction-following capabilities. On-Policy RL recovers partially (35.2), and LUFFY pushes substantially further (40.7). The largest gains are on Olympiad (47.4 vs. 37.6 for On-Policy RL) and AIME 2025 (14.8 vs. 8.3).

LLaMA-3.1-8B (weak model): LUFFY achieves 13.2 average vs. On-Policy RL's 9.6 (+3.6) and SFT's 5.9 (+7.3). All models perform poorly in absolute terms on this model — the LLaMA-3.1-8B instruct baseline is 17.1, meaning all training methods degrade in-distribution math performance relative to the instruct model. The paper explains this by noting LLaMA-3.1-8B was trained on a simplified subset of the data (the Easy set, 0-2k token solutions) because "on-policy RL fails on other subsets." This is a crucial qualification: LUFFY improves over alternatives on LLaMA-3.1-8B, but all methods perform worse than the instruct baseline on these benchmarks, suggesting that the RL training setup (simplified data, R1-style reasoning format) may be fundamentally misaligned with LLaMA-3.1-8B's pretraining distribution. The positive spin — LUFFY succeeds where On-Policy RL fails — should be understood relative to this low absolute baseline.

LUFFY Succeeds Where On-Policy RL Fails

Headline result (Section 5.2, Figure 4). On LLaMA-3.1-8B trained on the Hard subset of the data, on-policy RL training rewards "collapse to zero" — the model never generates correct solutions, receives no positive reinforcement, and learning stops entirely. In contrast, LUFFY achieves stable reward improvements on both the Easy and Hard subsets.

What Figure 4 shows: Training reward curves (y-axis: group-average reward, x-axis: training steps) for four configurations: On-Policy RL on Easy (rises steadily from ~0.15 to ~0.35), On-Policy RL on Hard (collapses from ~0.05 to ~0.00), LUFFY on Easy (rises from ~0.20 to ~0.40), and LUFFY on Hard (rises from ~0.15 to ~0.25). The Hard subset curves are the key: On-Policy RL Hard shows a flat line near zero, indicating the model never discovers correct solutions through exploration. LUFFY Hard shows steady improvement, reaching approximately 0.25 average reward by the end of training — the off-policy traces provide the positive signal that on-policy rollouts cannot.

Why this matters for the paper's central claims. This result is the strongest evidence for the claim that LUFFY "transcends the limitations of model capacity" (Section 1). It shows a qualitative difference — success vs. complete failure — rather than a quantitative improvement, establishing a capability boundary that on-policy methods cannot cross and LUFFY can. However, the result is limited to a single weak model (LLaMA-3.1-8B) on a single difficulty stratification. The paper does not test whether this failure boundary exists for other model families at different scales, nor does it characterize the difficulty threshold precisely (beyond "2k tokens" vs. "4k tokens" in R1's solution length).

Connection to downstream performance. Table 3 shows the consequences in benchmark scores: On-Policy RL on LLaMA-3.1-8B achieves only 9.6 average (vs. 13.2 for LUFFY), with particularly stark gaps on AMC (9.4 vs. 13.5) and MATH-500 (23.4 vs. 39.0). On AIME, both methods score near zero (0.3 vs. 1.9 on AIME 2024, 0.5 vs. 0.1 on AIME 2025). The practical significance of "succeeding where on-policy fails" is therefore concentrated on medium-difficulty benchmarks — on the hardest problems (AIME), even LUFFY cannot make LLaMA-3.1-8B competent.

Ablation Studies and Robustness Checks

Policy shaping contribution (Table 4, left block). Starting from Mixed-Policy RL (44.4 avg), adding policy shaping improves to 47.8 (+3.4), and further adding on-policy clip removal (NoClip) improves to 50.1 (+2.3, the full LUFFY configuration). The cumulative gain from shaping + NoClip over base Mixed-Policy is +5.7 points. Each component contributes additively: shaping addresses entropy collapse, NoClip prevents the PPO clipping mechanism from suppressing learning from off-policy traces when the policy is far from the teacher distribution. On a per-benchmark level, shaping provides the largest boost on Olympiad (49.9 → 53.0, +3.1) and AIME 2024 (19.4 → 27.4, +8.0), while NoClip adds to AIME 2024 (27.4 → 29.4, +2.0), AMC (61.2 → 65.6, +4.4), and Olympiad (53.0 → 57.2, +4.2).

On-Policy RL with shaping or NoClip (Table 4, right block). Applying shaping or NoClip to pure On-Policy RL (without off-policy data) does not improve performance and in some cases degrades it. On-Policy + Shaping achieves 42.0 (-3.5 from On-Policy RL's 45.5), and On-Policy + NoClip achieves 44.9 (-0.6). This confirms that the benefits of shaping and clip removal are specific to the off-policy setting — they amplify learning from external guidance but provide no benefit (or harm) when applied to pure on-policy data. The degradation under shaping likely occurs because the shaping function's amplification of low-probability tokens in the absence of off-policy guidance simply adds noise, disrupting the standard GRPO gradient.

Clipping frequency analysis (Appendix E.1, Figure 8). The paper measures what fraction of tokens have their importance sampling ratio clipped during training. LUFFY experiences "more frequent clipping compared to On-Policy RL," which the paper interprets as evidence that the off-policy traces come from a distribution substantially different from the model's current policy — the importance ratio $r_{i,t}(\theta)$ more often falls outside the $[1-\epsilon, 1+\epsilon]$ range, triggering the clipping mechanism. This motivates the removal of the on-policy clip in the final LUFFY configuration, since clipping would "suppress learning from high-quality off-policy traces" by limiting the gradient magnitude when the policy needs to change most.

Shaping function hyperparameter γ (Appendix E.4, Figure 9). Values tested: {0.05, 0.1, 0.2, 0.3, 0.5}. Performance peaks at γ = 0.1 (50.1 avg) and degrades for both smaller and larger values. γ = 0.05 achieves approximately 48.5 avg, γ = 0.2 achieves ~48, γ = 0.3 achieves ~46, γ = 0.5 achieves ~44. The non-monotonic relationship confirms that γ controls a genuine tradeoff: too small (0.05), the amplification of low-probability tokens is too aggressive, potentially causing training instability or overfitting to rare off-policy patterns; too large (≥0.3), the shaping function approaches linear behavior ( $f(x) \approx x/\gamma$ for small x), providing insufficient amplification to prevent entropy collapse. The paper uses γ = 0.1 consistently across all experiments without further analysis of why this value is optimal.

SFT training details ablation (Appendix C.1). The SFT baseline follows the OpenR1-Qwen-7B recipe: 3 epochs, batch size 64, learning rate 5 × 10⁻⁵, warmup ratio 0.1, max length 16k. The SFT+RL baseline uses the SFT model trained for 3 epochs, then RL for 500 steps on a held-out 49K prompt subset of OpenR1-Math-220k. The paper does not ablate SFT training duration or data quantity to test whether SFT+RL's underperformance is due to overfitting during the SFT phase or inherent to the two-stage approach.

Temperature sensitivity analysis (Appendix F.2, Figure 12). Pass@8 is evaluated on AIME 2024 + AMC at temperatures {0.1, 0.3, 0.5, 0.7, 1.0} for SFT, On-Policy RL, and LUFFY. Both RL-based methods improve with temperature (On-Policy RL: ~26% at 0.1 to ~38% at 1.0; LUFFY: ~28% at 0.1 to ~40% at 1.0), while SFT degrades (~27% at 0.1 to ~25% at 1.0). At temperature 0.1, SFT is competitive with LUFFY; at temperature 1.0, the gap is approximately 15 points. This confirms that LUFFY's learned policy supports exploration under temperature scaling, a property relevant to test-time compute scaling and distinct from SFT's brittle memorization. The paper does not evaluate pass@32 or majority voting at these temperatures, which would provide a more complete picture of test-time scaling potential.

Generation length analysis (Appendix F.1, Figures 10, 11). On the combined six-benchmark set, LUFFY's average generation length is 2,832 tokens vs. SFT's 4,646 tokens (64% longer for SFT). The length distribution (Figure 10) shows SFT produces a long tail of very lengthy outputs on incorrect answers, while LUFFY's incorrect-answer lengths are more controlled. During training (Figure 11), RL w/ SFT Loss shows "a steep increase in generation length early in training" that eventually exceeds the off-policy trace length, while LUFFY's length "grows more gradually and steadily." This behavioral evidence supports the claim that LUFFY learns selective, strategic imitation rather than rigid surface-level copying.

BLEU similarity to off-policy traces (Appendix F.1). On 1,000 held-out samples, BLEU scores against DeepSeek-R1 generations are: SFT 57.5, On-Policy RL 8.8, LUFFY 44.8. SFT's high BLEU confirms near-verbatim imitation; On-Policy RL's low BLEU confirms independent exploration; LUFFY's intermediate BLEU suggests a blend — the model adopts some reasoning patterns from the teacher without copying wholesale. The paper does not provide BLEU broken down by correctness, which would clarify whether LUFFY's imitation is concentrated on problems it cannot solve independently (as predicted by the dynamic scaffolding mechanism).

Format and length reward absence ablation (Section 4). The paper explicitly states "no format or length reward" is used — rewards are purely based on Math-Verify correctness. This is a deliberate contrast with DeepSeek-R1 and other methods that include format rewards to encourage structured thinking. The paper does not ablate whether adding format rewards would improve or harm LUFFY's performance. Given that LUFFY learns to produce appropriately structured reasoning (Figure 7 case study shows clear thinking/response formatting), format rewards may be unnecessary, but the ablation is missing.

Multiple off-policy trajectories limitation. The paper uses exactly one off-policy trace per prompt and notes this as a limitation (Appendix A): "extending off-policy guidance to multiple trajectories and multiple teachers could help the performance even further." No experiments test varying the number of off-policy traces (e.g., 2 off-policy + 6 on-policy) or mixing traces from multiple teacher models. This is a significant unexplored dimension — the dynamic scaffolding mechanism might work differently with multiple off-policy references, potentially providing diversity in reasoning strategies or creating conflicting signals if teachers disagree.

Critical Assessment

Does LUFFY genuinely "transcend the limitations of on-policy RLVR" as claimed in the abstract?

The strongest evidence is Section 5.2 (Figure 4): on-policy RL on LLaMA-3.1-8B with the Hard dataset collapses to zero reward while LUFFY continues learning. This is genuinely a qualitative difference — not just better performance, but the difference between learning and not learning. The paper's abstract claims that this "provides compelling evidence that LUFFY transcends the fundamental limitations of on-policy RLVR." The evidence supports this for the specific configuration tested (LLaMA-3.1-8B, Hard difficulty defined by 2-4k token R1 solutions), but the claim of "transcending fundamental limitations" implies generality that the experiments do not fully establish. Only one weak model is tested for the collapse phenomenon; the failure boundary is not systematically mapped across model scales, architectures, or difficulty definitions; and we don't know whether LUFFY would similarly rescue an even weaker model (e.g., LLaMA-3.2-3B) or whether there exists a difficulty threshold beyond which even LUFFY's off-policy guidance is insufficient. The claim would be stronger with a systematic sweep of model capability × data difficulty showing the shift in the trainability frontier.

Does LUFFY achieve state-of-the-art on RLVR with Qwen2.5-Math-7B?

The paper claims LUFFY "establishes a new state-of-the-art on RLVR with Qwen2.5-Math-7B." Table 1 supports this: LUFFY's 50.1 average exceeds all prior methods listed, including Oat-Zero (43.7) and OpenReasonerZero (41.0). There are two important caveats. First, the prior methods differ in dataset composition, which is not controlled — the paper's On-Policy RL baseline (45.5) uses the same data as LUFFY and provides a fairer comparison. The +4.6 gain over this controlled baseline is substantial but less dramatic than the +6.4 over all prior methods. Second, the claim of SOTA applies specifically to Qwen2.5-Math-7B at the time of writing. Whether LUFFY would exceed methods on other base models or newer RLVR variants is unknown.

Does policy shaping actually prevent entropy collapse, and does this matter for performance?

Figure 5 (right) shows LUFFY maintaining higher entropy than On-Policy RL throughout training, with fluctuations interpreted as exploration events. Table 4 shows that adding shaping to Mixed-Policy RL improves performance from 44.4 to 47.8. This establishes that shaping helps and that entropy is higher under shaping — but the causal chain "shaping → higher entropy → better performance" is not directly tested. An alternative interpretation is that shaping provides a better bias for off-policy gradient estimation (as the variance reduction analysis suggests) and the entropy effect is a correlated but not causal outcome. The paper doesn't run an experiment that directly manipulates entropy (e.g., varying the entropy bonus coefficient) to see whether higher entropy alone reproduces the shaping benefit. The shaping function's variance reduction property (Appendix B.2) may be the mechanism, with entropy preservation as a side effect.

Does LUFFY genuinely learn to "selectively imitate" rather than just doing a better job of imitation than SFT?

The paper's central narrative is that LUFFY achieves "dynamic balance" between imitation and exploration, learning to imitate only when necessary. The evidence for selectivity comes from: (1) BLEU score 44.8 vs. SFT's 57.5 (less imitation by surface-form similarity), (2) generation length 2,832 vs. SFT's 4,646 (less verbose imitation), (3) improved OOD generalization (57.8 vs. SFT's 47.5, suggesting less overfitting to the teacher's distribution), and (4) the emergent curriculum argument that off-policy advantages are larger when the model is incorrect. However, the paper does not directly demonstrate selectivity — it does not show, for instance, that LUFFY's token-level imitation probability correlates with problem difficulty or with whether its own answer is correct. Such an analysis (e.g., showing that on incorrect answers, LUFFY's outputs are more similar to the teacher than on correct answers) would directly support the dynamic scaffolding claim. The BLEU score is aggregated across all problems; a conditional breakdown would be more informative.

Is the comparison to SFT+RL fair given the resource differences?

Table 2 shows LUFFY uses 77×8 GPU hours and 64K off-policy traces, while SFT+RL uses 130×8 GPU hours and 135K off-policy traces. LUFFY achieves better results with less compute and less data — this is a favorable comparison. However, the SFT+RL configuration (3 epochs SFT, 500 RL steps on held-out data) is only one possible instantiation of the two-stage approach. The paper does not explore whether SFT+RL with fewer SFT epochs (less overfitting), a different data split, or different RL hyperparameters would close the gap. The extended LUFFY† training (860 steps, matching SFT+RL GPU budget) shows further improvement to 50.4, suggesting that even at equal cost, LUFFY is preferable. But the SFT+RL GPU cost is inflated by long generations during RL — a better-optimized SFT+RL (with generation length penalties during RL, which the paper doesn't try) might be more competitive.

What about the OOD results — does LUFFY genuinely improve generalization, or is the OOD advantage a side effect of improved in-distribution training?

LUFFY's OOD average (57.8) exceeds On-Policy RL (57.3) by only 0.5 points in the standard configuration. This is negligible and suggests that the OOD benefit is not from off-policy guidance per se but from improved training overall. LUFFY† shows a larger gap (61.8 vs. 57.3 for On-Policy RL, +4.5), but this comes from extended training, not from the off-policy mechanism specifically. The large OOD gap between LUFFY and SFT+RL (57.8 vs. 44.8) is more about SFT+RL's catastrophic OOD degradation than LUFFY's OOD strength — On-Policy RL already achieves 57.3 OOD without any off-policy data. The paper's claim that LUFFY "demonstrates superior generalization capability" is better supported by the temperature scaling experiment (Figure 12), where LUFFY handles diverse sampling conditions better than SFT, than by the absolute OOD benchmark numbers.

Missing experiments that would strengthen the paper:

Systematic sweep of on-policy/off-policy ratio. The paper uses 7:1 throughout. How does performance vary with 6:2, 4:4, 2:6? If the dynamic scaffolding hypothesis is correct, there should be diminishing returns as more off-policy traces are added (the group mean saturates, off-policy advantages shrink). Finding the optimal ratio would inform practical deployment.
Varying teacher quality. All off-policy traces come from DeepSeek-R1. What if the teacher is weaker (e.g., Qwen2.5-Math-7B-Instruct's own outputs) or stronger (e.g., a hypothetical better model)? Does LUFFY's performance degrade gracefully with weaker teachers, and does it saturate with stronger ones? This would clarify whether LUFFY is a distillation method or genuinely a new learning paradigm.
Ablation of the shaping function form. Only $f(x) = x/(x + \gamma)$ is tested. Would a simple temperature-scaled softmax or a piecewise linear function with similar properties produce similar results? The theoretical motivation (variance reduction, gradient amplification for low-probability tokens) is not unique to this functional form, and testing alternatives would distinguish whether the specific shape matters or just the general property of deemphasizing high-probability tokens.
Conditional BLEU/ similarity analysis. For LUFFY, measure output similarity to the teacher separately for problems the model gets correct vs. incorrect. If dynamic scaffolding works, similarity should be higher when the model is incorrect (it relies on the teacher) and lower when correct (it uses its own strategies). This would directly validate the core mechanism.
Difficulty-stratified results for the main benchmarks. The paper reports difficulty-stratified analysis only for the LLaMA-3.1-8B training reward experiment (Easy vs. Hard sets). For Qwen2.5-Math-7B, we don't know whether LUFFY's gains are concentrated on easy, medium, or hard problems — this would reveal whether off-policy guidance provides the most value where on-policy exploration is least effective (hard problems) or where imitation is easiest (medium problems).

Bottom-line assessment. The experimental evidence strongly supports the claim that LUFFY outperforms both on-policy RLVR and alternative off-policy learning methods on in-distribution math reasoning benchmarks for Qwen2.5-Math-7B. The evidence for "transcending fundamental limitations" is real but narrowly demonstrated — one weak model, one difficulty boundary, one teacher. The evidence for the dynamic scaffolding mechanism (selective imitation) is suggestive but indirect — the behavioral signatures (entropy, generation length, BLEU) are consistent with the mechanism but could have alternative explanations. The evidence for improved generalization is mixed — strong against SFT-based methods, weak against On-Policy RL in the standard configuration, and stronger with extended training. The paper's most robust finding is that integrating off-policy data into the advantage computation (rather than treating it as a separate SFT objective) yields better results, and that the policy shaping regularizer is important for sustaining these gains through training. The more ambitious claims about transcending on-policy limitations and enabling weak-model training are directionally supported but require broader validation across model scales, teacher qualities, and task difficulties.

6. Limitations and Trade-offs

The Black-Box Difficulty Assumption: Why Off-Policy Guidance Works on Some Problems but Not Others Is Uncharacterized

The assumption or constraint. LUFFY provides one off-policy reasoning trace per prompt to every prompt in the training set, regardless of the prompt's difficulty or the model's current capability on that prompt. The dynamic scaffolding mechanism responds to whether the model's on-policy rollouts are correct (via the mixed-group advantage computation), but the paper does not analyze which types of problems benefit most from off-policy guidance. The difficulty-stratified analysis (Section 5.2, Figure 4) exists only for LLaMA-3.1-8B on a single difficulty split (Easy vs. Hard defined by R1 solution length), and the benchmark-level results (Table 1) are reported in aggregate without difficulty breakdowns for the main Qwen2.5-Math-7B experiments.

The consequence. A practitioner cannot predict whether LUFFY will help on a given problem distribution. The benchmark results show large variance in improvement: on Qwen2.5-Math-7B, LUFFY gains +10.4 points on OlympiadBench but loses -1.8 points on Minerva relative to On-Policy RL (Table 1). This is not random noise — it suggests that off-policy guidance is differentially effective depending on problem characteristics (competition math vs. quantitative reasoning, multi-step proof vs. calculation, etc.). Without understanding which problem features correlate with LUFFY's benefits, a practitioner cannot forecast whether applying LUFFY to a new domain (e.g., code generation, scientific reasoning, theorem proving) would yield similar gains or would waste compute on a method that provides no benefit or regresses relative to on-policy RL. The Minerva regression is particularly concerning because it is not analyzed — the paper simply reports the number without investigating why off-policy guidance from DeepSeek-R1 (a math reasoning model) might interfere with performance on Minerva (a quantitative reasoning benchmark with different problem styles).

What evidence exists in the paper. Table 1 provides the raw per-benchmark numbers showing uneven gains: OlympiadBench +10.4, AIME 2025 +7.8, AIME 2024 +4.3, AMC +3.6, MATH-500 +3.2, Minerva -1.8 (all vs. On-Policy RL). Section 5.2 and Figure 4 show the training reward collapse on Hard vs. success on Easy for LLaMA-3.1-8B, but this is a single model on a binarized difficulty metric (solution length) and is not connected to the benchmark-level results on Qwen2.5-Math-7B. The paper provides no analysis of how LUFFY's gains correlate with problem difficulty, problem type, or required reasoning skills. The dynamic scaffolding hypothesis (Section 3.4) predicts that off-policy guidance helps most when on-policy rollouts are incorrect — but this prediction is never tested against difficulty-stratified benchmark performance.

Mitigation status. Not addressed. The paper acknowledges in Appendix A that it "mainly focus[es] on math reasoning RL training" and that extending to "broader domains" is future work, but the uneven gains within the math domain itself are not discussed as a limitation. A brief analysis of per-benchmark gains by difficulty or problem type would have substantially strengthened the practical guidance for deployment.

The Hidden Cost of Teacher Dependence: Teacher Model Quality and Availability Are Unstated Prerequisites

The assumption or constraint. LUFFY assumes access to a stronger teacher model (DeepSeek-R1) that can produce verified-correct reasoning traces for the training prompts. The paper states that "one trajectory is already strong" (Section 5.1) and uses exactly one off-policy trace per prompt. However, the method is entirely dependent on the existence and quality of this teacher — LUFFY does not work with incorrect traces (the data is filtered to remove Math-Verify failures, Section 4), and the paper provides no evidence on how teacher quality affects LUFFY's performance. The teacher model (DeepSeek-R1) is a frontier-scale reasoning model that required substantial resources to train, and its reasoning traces are not guaranteed to exist for arbitrary domains or problem types.

The consequence. LUFFY's applicability is bounded by teacher availability — it cannot be used to train models on tasks where no strong teacher exists, which is precisely the regime where "transcending the limitations of model capacity" would be most valuable. If a teacher model can already solve all the training problems, LUFFY is a distillation method (albeit a more efficient one than SFT) — but the paper frames it as enabling weaker models to learn capabilities they couldn't otherwise acquire. This framing only holds if the teacher provides reasoning strategies that the student can partially internalize, but the paper provides no evidence on what happens with weaker teachers or with teachers that are only slightly better than the student. Would LUFFY fail gracefully with a teacher that is correct on only 60% of problems (providing partially noisy guidance) or with multiple teachers that disagree? Would the off-policy advantage normalization break if the teacher's rewards were not uniformly 1? The paper's theoretical framework assumes "the quality of off-policy rollouts is high (yielding high rewards)" (Section 3.1), but does not specify what "high" means or test degradation with lower teacher accuracy.

What evidence exists in the paper. Table 1 compares LUFFY against SFT, On-Policy RL, and SFT+RL — all using the same DeepSeek-R1 traces (where applicable). There is no experiment varying teacher model quality (e.g., using Qwen2.5-Math-7B-Instruct as teacher, using a weaker model, using an ensemble, using traces with artificially introduced errors). The filtering step (Section 4: "filter out... those that are verified wrong") ensures 100% teacher correctness on the training data, but this filtering is possible only because Math-Verify provides ground-truth labels — in domains without verifiable rewards, teacher filtering would be much harder. The paper acknowledges in Appendix A that "extending off-policy guidance to multiple trajectories and multiple teachers could help the performance even further" but frames this as an opportunity rather than a limitation.

Mitigation status. None. The paper treats teacher quality as fixed and does not analyze sensitivity. This is a significant gap because, in practice, the strongest available teacher for a given domain may vary substantially in quality, and practitioners need to know whether LUFFY provides benefits even with imperfect teachers (e.g., human-written solutions, model-generated traces with 80% accuracy) or whether teacher quality exhibits a threshold effect below which LUFFY underperforms on-policy RL. The theoretical convergence guarantee (Theorem 1) assumes bounded importance weights but does not depend on teacher accuracy — however, the practical mechanism depends on off-policy traces having positive advantages, which requires the teacher to be correct.

The Unexplained OOD Generalization Gap: LUFFY Barely Improves Over On-Policy RL on Out-of-Distribution Tasks

The assumption or constraint. The paper claims that LUFFY "demonstrates superior generalization capability" (Section 5.1) and highlights an "advantage of over +6.2 points on out-of-distribution tasks" in the abstract. However, this +6.2 figure is relative to prior RLVR methods (specifically OpenReasonerZero at 51.6), not relative to the paper's own controlled On-Policy RL baseline. Against On-Policy RL (57.3), LUFFY's OOD average (57.8) represents a gain of only +0.5 points — negligible and almost certainly not statistically significant. The gap to SFT-based methods is large (57.8 vs. 44.8 for SFT+RL), but this reflects SFT+RL's catastrophic OOD failure rather than LUFFY's OOD strength.

The consequence. The abstract's framing of LUFFY as providing superior generalization could mislead practitioners into believing that LUFFY improves OOD robustness over on-policy RL — it does not, at least in the standard training configuration. The OOD benefit is almost entirely from avoiding the generalization collapse caused by SFT-based approaches. This matters because a practitioner choosing between On-Policy RL and LUFFY might pay the additional complexity cost of LUFFY (implementing mixed-policy advantages, policy shaping, off-policy data management) expecting better OOD performance, when in fact the two methods are essentially tied on OOD tasks. The extended training (LUFFY†) does show larger OOD gains (61.8 vs. 57.3, +4.5), but this comes from training longer (860 vs. 500 steps) and is a confounded comparison — we don't know whether On-Policy RL trained for 860 steps would also improve, or whether the on-policy RL gains would saturate sooner than LUFFY's.

What evidence exists in the paper. Table 1 reports OOD results: On-Policy RL achieves 57.3 (82.3 ARC-c, 40.4 GPQA*, 49.3 MMLU-Pro), LUFFY achieves 57.8 (80.5, 39.9, 53.0), and LUFFY† achieves 61.8 (81.8, 49.0, 54.7). The per-benchmark breakdown reveals that the MMLU-Pro gap (53.0 vs. 49.3, +3.7) accounts for essentially all of LUFFY's OOD advantage in the standard configuration, while ARC-c and GPQA* show slight regressions. The paper does not analyze why MMLU-Pro specifically benefits or why ARC-c and GPQA* regress. The temperature scaling experiment (Figure 12) provides a different type of generalization evidence — robustness to sampling temperature — but this is a property of the learned policy distribution, not a measure of performance on held-out task distributions.

Mitigation status. Not acknowledged. The abstract and Section 5.1 frame the OOD results as a strength without noting the near-parity with On-Policy RL. A more transparent presentation would have stated that LUFFY matches On-Policy RL's OOD performance while substantially exceeding it on in-distribution tasks, and that the +6.2 figure in the abstract compares against weaker prior methods rather than the controlled baseline. The LUFFY† result hints that longer training may widen the OOD gap, but this is presented as an additional data point, not as a systematic investigation.

The Single-Teacher, Single-Trace Design: Unexplored Sensitivity to the Amount and Diversity of Off-Policy Data

The assumption or constraint. LUFFY uses exactly one off-policy reasoning trace per training prompt (7 on-policy + 1 off-policy), and that trace comes from a single teacher model (DeepSeek-R1). The 7:1 ratio is chosen "to ensure fairness" with the 8-rollout on-policy baseline (Section 4), not because it was found to be optimal through experimentation. The paper explicitly acknowledges in Appendix A that "extending off-policy guidance to multiple trajectories and multiple teachers could help the performance even further," but treats this as an opportunity for future work rather than a limitation of the current method that practitioners need to know about.

The consequence. A practitioner implementing LUFFY faces an under-explored design space with no guidance on critical choices: (1) How many off-policy traces per prompt are optimal? The dynamic scaffolding mechanism predicts that off-policy advantages shrink as on-policy performance improves — with multiple off-policy traces per prompt (all with reward 1), the group mean would be higher, potentially reducing the advantage signal for on-policy correct solutions and distorting the imitation-exploration balance. Could 2 off-policy + 6 on-policy be worse than 1 + 7? Could 4 + 4 cause entropy collapse because off-policy signals dominate? The paper provides no evidence. (2) Does LUFFY benefit from diverse off-policy traces (different teachers, different reasoning strategies) or is a single high-quality trace per prompt sufficient? If a practitioner has access to multiple teacher models or multiple correct solutions per problem, should they use ensemble-like mixing or pick the single best trace? (3) The total amount of off-policy data (45K traces after filtering) is not ablated — could LUFFY achieve similar performance with 22.5K or 10K traces? If off-policy data is expensive to generate (requiring a large teacher model), understanding data efficiency is crucial for cost estimation.

What evidence exists in the paper. None. The ratio experiment (varying $N_{\text{on}}$ vs. $N_{\text{off}}$ ) is not conducted. The number of off-policy traces is not ablated. Teacher diversity is not tested. The paper's single empirical data point on this dimension is that 1 off-policy trace with the 7:1 ratio works — but we don't know whether 2:6, 4:4, or 0:8 (pure on-policy) is on the same performance curve with different tradeoffs or whether 1:7 is a lucky choice.

Mitigation status. Noted as future work in Appendix A ("extending off-policy guidance to multiple trajectories and multiple teachers could help the performance even further"), but not presented as a limitation that affects the interpretability or reproducibility of the current results. A practitioner reading the paper would reasonably assume 1 off-policy trace is the recommended configuration without realizing that no alternative was tested.

The Scope Restriction to Verifiable Rewards: LUFFY Cannot Be Applied to Open-Ended or Subjective Tasks

The assumption or constraint. LUFFY, like all RLVR methods, requires a verifiable reward function that can definitively determine whether a model output is correct. The paper uses Math-Verify for answer extraction and exact-match comparison against ground-truth answers (Section 2). This restricts LUFFY's applicability to tasks with well-defined correct answers — math competition problems, certain coding tasks with unit tests, formal theorem proving — and excludes open-ended generation tasks (summarization, dialogue, creative writing, instruction following) and tasks where correctness is subjective or multi-dimensional (essay grading, scientific explanation quality, legal argument strength).

The consequence. A significant fraction of real-world LLM deployment tasks cannot use LUFFY. The paper does not discuss this scope restriction as a limitation — it is stated as a framing choice ("we mainly focus on math reasoning RL training," Appendix A) — but a practitioner evaluating whether to adopt LUFFY needs to know that the method provides no path forward for tasks where reward signals are learned (reward models) rather than rule-based. This is particularly important because learned reward models are susceptible to reward hacking in ways that verifiable rewards are not, and it is unclear whether LUFFY's off-policy guidance would mitigate or exacerbate reward hacking when using a learned verifier. The dynamic scaffolding mechanism relies on off-policy traces having consistently high reward — if the off-policy traces occasionally receive low rewards from a noisy learned verifier, the advantage computation would produce misleading signals, potentially causing the model to unlearn from high-quality teacher traces.

What evidence exists in the paper. None — learned reward models are not tested. The paper's entire empirical evaluation is on math benchmarks with exact-match grading. The related work section mentions that "tasks lacking verifiable rewards are not addressed in this manuscript" (Appendix A) and cites Fu et al. (2025) on how scaling reasoning can impair instruction-following, but does not investigate whether LUFFY could be extended to learned-reward settings or what would break if attempted. The theoretical analysis (Theorem 1) does not depend on the reward structure and would apply to any scalar reward signal, but the practical mechanism (off-policy traces provide consistent positive signal, enabling bootstrapping when on-policy rewards are zero) would fail if off-policy rewards were unreliable.

Mitigation status. Acknowledged briefly in Appendix A: "Tasks lacking verifiable rewards are not addressed in this manuscript." No mitigation is attempted or suggested for future work. A natural extension — using a learned reward model in place of Math-Verify, with off-policy traces filtered by reward model score rather than ground-truth correctness — is not discussed.

The Missing Latency and Inference Overhead Analysis: Training Efficiency Gains May Not Translate to Deployment Efficiency

The assumption or constraint. The paper evaluates LUFFY based on training efficiency (GPU hours, Table 2) and benchmark accuracy (Table 1). All evaluation uses temperature 0.6 and either pass@1 or avg@32 (Section 4). The paper does not measure or discuss inference-time latency, throughput, or deployment cost. LUFFY's generated outputs average 2,832 tokens on the combined benchmark set (Appendix F.1), compared to On-Policy RL's presumably shorter outputs (not directly reported, but the paper notes On-Policy RL's generation length is lower than LUFFY's, since LUFFY learns to produce longer, more thorough reasoning by imitating DeepSeek-R1's style).

The consequence. A practitioner deploying LUFFY faces a tradeoff that the paper does not quantify: LUFFY's accuracy gains may come with increased inference cost due to longer outputs. If LUFFY's average output length is 2,832 tokens vs. (say) 1,800 tokens for On-Policy RL, then the +4.6 accuracy gain on in-distribution benchmarks must be weighed against approximately 57% more tokens generated per query — which translates to higher latency, higher per-query cost, and potentially lower throughput in batch inference settings. This tradeoff is particularly relevant for the applications where the paper claims LUFFY is most valuable: training "weak models" (Section 5.2) for deployment. If a LLaMA-3.1-8B trained with LUFFY achieves 13.2 avg but generates 2× the tokens of a simpler model, a practitioner on a latency budget might prefer a faster but less accurate model. The temperature scaling results (Figure 12) show LUFFY maintains accuracy at higher temperatures, which could mitigate the throughput concern (higher temperature → shorter outputs in expectation), but the interaction between temperature, output length, and accuracy is not analyzed.

What evidence exists in the paper. Figure 10 (Appendix F.1) provides average generation lengths: LUFFY 2,832 tokens, SFT 4,646 tokens. On-Policy RL's length is not reported on this figure, but the training dynamics plot (Figure 5, middle) shows On-Policy RL's generation length is substantially lower than LUFFY's throughout training (On-Policy RL plateauing around ~1,500-2,000 tokens, LUFFY stabilizing around ~2,500-3,000). The case study (Figure 7) shows LUFFY producing 2,623 tokens vs. On-Policy RL's 1,002 tokens for the same problem — a 2.6× difference. This single example is not necessarily representative, but combined with the training dynamics, it suggests a systematic length increase that the paper does not frame as a cost. The GPU hour comparison (Table 2) accounts for training cost but not inference cost.

Mitigation status. Not addressed. The paper presents LUFFY primarily as a training method and evaluates it on accuracy, not on deployment efficiency. A brief analysis of accuracy-per-output-token or accuracy-vs-latency would significantly improve the practical guidance for practitioners deciding whether to deploy a LUFFY-trained model versus a shorter-output on-policy RL model or a distilled SFT model.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the conversation around off-policy data in RLVR from a data-selection problem to an objective-design problem. The dominant paradigm for incorporating stronger-model reasoning traces into LLM training is distillation: supervised fine-tuning on teacher outputs, optionally followed by RL (SFT+RL). LUFFY demonstrates that how you incorporate off-policy data matters as much as whether you do — and that the standard approaches systematically underperform because they create pathologies (rigid imitation, entropy collapse, catastrophic OOD degradation) that LUFFY's mixed-group advantage computation and policy shaping avoid. This is not an incremental improvement to distillation; it's a different way of thinking about what off-policy data is in the learning process: dynamic scaffolding rather than static targets.

The magnitude of this shift is moderate. LUFFY does not invent a fundamentally new optimization algorithm — it extends GRPO with importance-weighted off-policy gradients and a regularized importance ratio, both techniques with established roots in RL literature (Sutton et al., 1999; Meng et al., 2023). The contribution is in recognizing that simply injecting off-policy rollouts into the GRPO group computation creates emergent behaviors (automatic imitation-exploration balancing, difficulty-conditioned attention to teacher guidance) that previous methods achieve only through explicit multi-stage pipelines or loss mixing. The +6.4 average gain over prior RLVR methods on Qwen2.5-Math-7B (Table 1) and the qualitative success on LLaMA-3.1-8B where on-policy RL collapses to zero reward (Section 5.2, Figure 4) are substantial results that make the case for this reframing. But LUFFY remains within the RLVR paradigm — it does not propose a new reward structure, a new policy gradient theorem, or a new model architecture. Its impact will be felt primarily through adoption: if subsequent work converges on mixed-group advantage computation as the default way to incorporate off-policy data (replacing SFT+RL pipelines), LUFFY will have meaningfully changed how the field builds reasoning models.

The paper reconciles a tension in the literature between two empirical observations: (1) distillation works — DeepSeek-R1's own pipeline and open-source reproductions like OpenR1-Qwen-7B show that SFT on teacher traces produces competent models — and (2) distillation produces brittle models — the paper's evidence (Figure 12: SFT degrades under temperature scaling; Table 1: SFT OOD is 47.5 vs. On-Policy RL's 57.3) aligns with concurrent findings from Chu et al. (2025) and Chen et al. (2025) that SFT memorizes while RL generalizes. LUFFY's resolution: the problem isn't distillation per se — it's the static treatment of off-policy data. When teacher traces are used as fixed targets (SFT), the model overfits to surface patterns. When they're used as contingent guidance within an RL objective (LUFFY), the model selectively internalizes the reasoning strategies they contain while retaining exploratory capacity. This explains why SFT+RL (which inherits SFT's rigid imitation as its starting point) underperforms LUFFY (48.2 vs. 50.1 in-distribution, 44.8 vs. 57.8 OOD, Table 1) despite using more data and more compute — the two-stage approach cannot recover from the brittleness baked in during the SFT phase.

The paper identifies entropy collapse as the central failure mode of naive mixed-policy training (Section 3.2, Figure 2 left, Figure 6), and this diagnosis has implications beyond LUFFY. Any method that combines heterogeneous data sources — on-policy and off-policy, multiple tasks, multiple modalities, human and synthetic data — must contend with the asymmetric gradient phenomenon the paper identifies: high-probability patterns in the model's current distribution receive disproportionately large gradient signals from off-policy data, causing rapid convergence to surface-level imitation while ignoring genuinely novel capabilities buried in low-probability teacher tokens. The policy shaping remedy ( $f(x) = x/(x + \gamma)$ ) is one solution, but the diagnosis itself is a conceptual contribution: it tells researchers working on multi-source training to look for entropy collapse as a diagnostic signal and to consider gradient-reweighting mechanisms that counteract the asymmetry.

The paper also reframes the capability boundary for RLVR. Prior theoretical and empirical work (Zhao et al., 2025; Yue et al., 2025) established that RL primarily amplifies behaviors already present in the pretrained distribution. LUFFY's results on LLaMA-3.1-8B (Section 5.2) do not contradict this — the model still isn't creating reasoning strategies ex nihilo — but they show that the boundary is more porous than the "amplification only" narrative suggests. Behaviors that are too improbable for random exploration to discover (the "cold start" problem where training rewards collapse to zero) can be bootstrapped through off-policy guidance into the model's sampling distribution, after which RL can amplify them. This implies that the effective capability threshold for RLVR is not "does the model occasionally produce this behavior" but "can the behavior be introduced into the model's distribution through guided exploration," which is a lower bar.

The research directions that become more attractive after this work:

Objective-design approaches to multi-source training. LUFFY shows that how you combine data matters more than how much data you have — SFT+RL uses 111% more off-policy traces yet underperforms (Table 2). This encourages research on loss functions, advantage computations, and gradient reweighting mechanisms for heterogeneous data, rather than simply scaling up data quantity.
Entropy-maintaining regularizers for RL. The paper provides both a diagnosis and a remedy for off-policy-induced entropy collapse. This opens a line of work on regularizers that specifically counteract the asymmetric gradient phenomenon — not just uniform entropy bonuses, but adaptive reweighting that amplifies learning from distribution-shifted data.
Teacher-student co-training and iterative self-improvement. If off-policy guidance can bootstrap capabilities from teacher traces, a natural extension is iterative: train a model with LUFFY, use it to generate better traces, use those as off-policy guidance for the next iteration. The paper's finding that one off-policy trace is "already strong" (Section 5.1) suggests diminishing returns, but teacher quality improvements through iteration might compound.

Directions that become less urgent:

Pure on-policy RLVR scaling without guidance. LUFFY demonstrates that even a single teacher trace per prompt provides substantial benefits over pure on-policy exploration (+4.6 avg on Qwen2.5-Math-7B, Table 1), and that pure on-policy fails entirely in some regimes (Figure 4). This suggests that scaling on-policy RLVR alone — through larger models, more rollouts, or longer training — may be fundamentally less efficient than incorporating even minimal off-policy guidance. The field's attention should shift from "how do we make on-policy exploration more efficient" to "how do we optimally combine guidance and exploration."
Complex multi-stage distillation pipelines. The SFT+RL baseline (48.2 avg) uses two training stages, separate datasets, and 69% more GPU hours than standard LUFFY (50.1 avg). LUFFY† (50.4 avg) matches SFT+RL's GPU budget and widens the gap to +2.2 points. If these results hold across models and domains, the expensive two-stage paradigm of "distill then RL" should be replaced by integrated approaches where teacher guidance and RL exploration coexist within a single objective.

Follow-Up Research This Work Enables

Systematically mapping the trainability frontier across model scales, teacher qualities, and task difficulties. The paper demonstrates a single point on this frontier: LLaMA-3.1-8B fails with on-policy RL on the Hard subset (2-4k token R1 solutions) but succeeds with LUFFY (Figure 4). A comprehensive follow-up would sweep model scales (0.5B → 70B parameters), teacher qualities (ranging from same-model self-generated traces to frontier-model traces to human-written solutions), and difficulty metrics (solution length, benchmark score, human difficulty ratings) to produce a phase diagram showing where on-policy RL succeeds, where LUFFY succeeds, and where even LUFFY fails. The key output would be a predictive model: given a student model's base MATH accuracy, a teacher model's accuracy on the training set, and a problem difficulty measure, can we predict whether LUFFY will improve over on-policy RL? This would directly test whether LUFFY "transcends fundamental limitations" (abstract) across conditions or only in specific regimes.

Testing whether LUFFY's dynamic scaffolding mechanism actually produces the claimed selective imitation. The paper provides behavioral signatures consistent with selective imitation (BLEU 44.8 vs. SFT's 57.5, intermediate generation lengths, maintained entropy), but does not directly show that LUFFY imitates the teacher more when it's incorrect and less when it's correct — the core prediction of the dynamic scaffolding hypothesis. A targeted follow-up would compute token-level or trajectory-level similarity between LUFFY's outputs and the teacher's outputs during training, conditioned on whether the model's own answer is correct. The prediction: on problems where the model answers correctly, similarity to the teacher decreases over training (the model develops its own strategies); on problems where it answers incorrectly, similarity remains high or increases (the model relies on the teacher). If this conditional pattern does not emerge, the dynamic scaffolding interpretation fails and the performance gains must be attributed to some other mechanism (perhaps the regularization effect of policy shaping alone, or the off-policy data serving as a better exploration curriculum).

Ablating the shaping function form to determine whether the specific functional choice matters or just the general property of amplifying low-probability tokens. The paper uses $f(x) = x/(x + \gamma)$ with $\gamma = 0.1$ , motivated by its gradient amplification properties (Equation 8) and variance reduction (Appendix B.2). But any function that is concave for $x \in (0, 1)$ and saturates at high $x$ would produce qualitatively similar behavior: softmax temperature scaling, $f(x) = x^\alpha$ for $\alpha < 1$ , piecewise linear with a hinge, or learned gating functions. A strong follow-up would compare these alternatives at equal $\gamma$ -equivalent hyperparameter values, measuring not just final accuracy but training dynamics (entropy, generation length, BLEU vs. teacher) to test whether performance differences emerge from the specific derivative shape or from the general property of deemphasizing high-probability tokens. If many functional forms produce equivalent results, the specific choice of $x/(x + \gamma)$ is incidental and the contribution is the general principle of gradient reweighting. If performance is sensitive to functional form, the specific mathematical properties of the shaping function (smoothness, derivative behavior, asymptotic limits) matter in ways the paper's informal analysis does not fully capture.

Extending LUFFY to non-verifiable reward settings using learned reward models, with careful attention to reward hacking dynamics. The paper explicitly scopes itself to math reasoning with verifiable rewards (Appendix A). The next frontier is adapting LUFFY to domains where correctness signals come from learned reward models: instruction following, summarization, dialogue, code quality. This is challenging because LUFFY's mechanism depends on off-policy traces having consistent positive advantage — with a noisy learned reward model, teacher traces might occasionally receive low rewards, producing negative advantages that push the model away from high-quality behavior. A strong follow-up would benchmark LUFFY against SFT and on-policy RL on, say, the IFEval instruction-following benchmark or the AlpacaEval dialogue benchmark, using an off-the-shelf reward model (e.g., ArmoRM, PairRM) in place of Math-Verify, and measure whether (a) LUFFY still outperforms baselines, (b) reward hacking (Goodhart's law effects) is more or less severe under LUFFY than under on-policy RL, and (c) the entropy-maintaining property of policy shaping helps resist reward over-optimization. Negative results here would clarify the boundary beyond which LUFFY's approach requires verifiable ground truth; positive results would dramatically expand LUFFY's applicability.

Investigating whether multiple diverse teachers or multiple traces per prompt provide diminishing or synergistic returns. The paper uses exactly one off-policy trace per prompt from a single teacher (DeepSeek-R1) and acknowledges this limitation (Appendix A). A systematic sweep of the on-policy/off-policy ratio $N_{\text{on}}:N_{\text{off}}$ (from 8:0 to 0:8) would reveal whether the 7:1 choice is near-optimal or whether more off-policy data provides further gains. More interestingly, experiments with teacher ensembles — e.g., traces from DeepSeek-R1, Qwen2.5-Math-7B-Instruct, and GPT-4 on each prompt, mixed into the group computation — would test whether LUFFY benefits from diverse reasoning strategies or whether conflicting teacher signals degrade performance. The prediction from the dynamic scaffolding mechanism: diverse teachers provide more opportunities for the model to internalize reasoning patterns it doesn't already know (more low-probability tokens get amplified), but conflicting signals (different teachers produce different correct solutions) could destabilize the advantage computation if their relative quality differs. This experiment would bridge LUFFY to the literature on ensemble distillation and mixture-of-experts for reasoning.

Stress-testing LUFFY on the absence of teacher data — does the model internalize teacher-derived capabilities or remain dependent on teacher-style prompting at test time? The paper evaluates LUFFY under standard inference conditions (temperature 0.6, no special prompting beyond the system prompt, Section 4). But if LUFFY's reasoning capabilities were bootstrapped from DeepSeek-R1 traces during training, it's possible that at test time the model benefits from prompting that resembles the teacher's distribution — e.g., using DeepSeek-R1-style system prompts, few-shot examples, or specific reasoning cues. A strong follow-up would evaluate LUFFY under two test conditions: (a) the standard system prompt used during training, and (b) a "neutral" prompt that avoids any resemblance to the teacher (e.g., "Solve this problem step by step"). If performance degrades significantly under the neutral prompt, LUFFY has learned a context-dependent reasoning capability rather than a robust, internalized skill — a form of prompt overfitting that would limit its practical deployment. If performance transfers, the bootstrapping claim is stronger: the teacher guidance produced lasting capability improvements, not just context-dependent behavior cloning.

Practical Applications and Downstream Use Cases

Cost-efficient distillation from large reasoning models to smaller deployable models. The paper's most direct practical implication: if you have access to DeepSeek-R1 (or a similar frontier reasoning model) and want to train a smaller model (7B, 1.5B) that preserves as much reasoning capability as possible, LUFFY is substantially more data-efficient and compute-efficient than the standard SFT-based distillation pipeline. Table 2 quantifies this: LUFFY achieves 50.1 average on six math benchmarks using 64K teacher traces and 616 GPU hours (77 × 8 A100), while SFT+RL achieves only 48.2 using 135K traces and 1,040 GPU hours. For an organization with limited compute, LUFFY provides a 2.2-point accuracy gain at 59% of the GPU cost and 47% of the teacher data requirement. On the smallest tested model (Qwen2.5-Math-1.5B, Table 3), LUFFY achieves 38.0 vs. SFT's 31.9 (+6.1) — a proportionally larger gain for the weakest student, suggesting LUFFY is particularly valuable when the teacher-student capability gap is large.

Bootstrapping reasoning in weak base models for specialized domains. Section 5.2 demonstrates that LLaMA-3.1-8B (a general-purpose model with minimal math capability) can be trained via LUFFY to achieve 13.2 avg on competitive math benchmarks, while on-policy RL fails completely (9.6 avg, with training rewards collapsing to zero on harder data). For practitioners deploying models in domains where no strong domain-specific base model exists — medical reasoning, legal analysis, scientific research — LUFFY offers a path to bootstrap reasoning capabilities from a general teacher model into a smaller or weaker domain-specialized model, without the cost of pretraining a domain-specific model from scratch. The key requirement is access to a teacher that can produce verified-correct reasoning traces for the target domain. With DeepSeek-R1 or similar models available via API, this requirement is increasingly feasible for domains with clear correctness criteria.

Replacing SFT-based distillation in self-improvement loops. Many self-improvement pipelines (ReST $^{\text{EM}}$ , STaR, self-play) follow an iterate-then-distill pattern: train a model via RL, use it to generate high-quality traces, then distill into a smaller model via SFT for the next iteration. LUFFY offers an alternative: at each iteration, use the previous iteration's best model as the off-policy teacher for LUFFY training of the next model. The advantage over SFT-based distillation is that LUFFY's dynamic scaffolding would naturally weight teacher traces more heavily when the student is weak (early in training or on hard problems) and less when the student catches up — creating an automatic curriculum within each iteration without explicit difficulty scheduling. The paper's result that LUFFY† (extended training) increases OOD performance from 57.8 to 61.8 (Table 1) suggests that LUFFY's benefits compound with additional training, which is exactly the property needed for iterative self-improvement. The risk — to be tested empirically — is that using a model as its own teacher across iterations might amplify biases or cause capability collapse if the teacher doesn't improve between iterations.

When to Prefer This Method

The paper positions LUFFY against three alternatives — pure on-policy RLVR, SFT distillation, and SFT+RL — and the experimental results (Tables 1, 2; Figures 4, 12) articulate clear conditions for preferring LUFFY over each:

Prefer LUFFY over pure on-policy RLVR when the base model is weak enough that on-policy exploration fails to discover correct solutions (training rewards collapse to zero, as with LLaMA-3.1-8B on the Hard set, Figure 4), or when simply improving in-distribution performance is the primary goal and OOD degradation is acceptable (LUFFY matches On-Policy RL on OOD while providing +4.6 points in-distribution, Table 1). Prefer on-policy RLVR when OOD generalization is the only objective and training compute is limited — LUFFY's standard configuration provides negligible OOD advantage (+0.5 points over On-Policy RL, Table 1), and the OOD gains require extended training (LUFFY†).
Prefer LUFFY over SFT distillation when test-time exploration matters — LUFFY maintains accuracy under high-temperature sampling while SFT degrades sharply (Figure 12: at temperature 1.0, LUFFY pass@8 ≈ 40% vs. SFT ≈ 25%), or when out-of-distribution generalization is important (LUFFY 57.8 OOD vs. SFT 47.5, Table 1), or when inference cost from excessively long outputs is a concern (LUFFY average output 2,832 tokens vs. SFT 4,646 tokens, Appendix F.1). Prefer SFT when maximizing in-distribution accuracy on a narrow benchmark with minimal training budget — SFT achieves 44.1 average using only 192 GPU hours (Table 2) and is simpler to implement.
Prefer LUFFY over SFT+RL when training compute, teacher data, or human effort is constrained — LUFFY achieves 50.1 avg using 59% of the GPU hours and 47% of the teacher data that SFT+RL requires for 48.2 avg (Table 2) — or when OOD generalization cannot be sacrificed (SFT+RL OOD is 44.8 vs. LUFFY's 57.8, Table 1), or when avoiding the implementation complexity of a two-stage pipeline is desirable. Prefer SFT+RL only if existing infrastructure is already built around a two-stage distill-then-RL workflow and the switching cost to LUFFY's single-stage mixed-policy approach is prohibitive — but the performance and efficiency gaps suggest that migrating to LUFFY would pay back the switching cost quickly in most settings.

🎯 Pitch

1. Executive Summary

2. Context and Motivation

The Core Problem: On-Policy RLVR Is Bounded by Initial Model Capabilities

Why This Problem Matters

Where Prior Approaches Fall Short

How This Paper Positions Itself

3. Technical Approach

3.1 Reader Orientation

3.2 Big-Picture Architecture (Diagram in Words)

3.3 Roadmap for the Deep Dive

3.4 Detailed, Sentence-Based Technical Breakdown

Verifiable Reward Function

On-Policy GRPO Baseline

Mixed-Policy GRPO: Injecting Off-Policy Guidance into Advantage Computation

The Mixed-Policy Objective: Importance-Weighted Policy Gradient with Off-Policy Data

Practical Choice: Setting πϕ=1\pi_\phi = 1πϕ​=1 (Uniform Off-Policy Distribution)

The Entropy Collapse Problem in Mixed-Policy Training

Policy Shaping via Regularized Importance Sampling

Training Configuration and Hyperparameters

Summary of Design Choices and Their Justifications

4. Key Insights and Innovations

Innovation 1: Reframing Off-Policy Data as Dynamic Scaffolding Rather Than Static Targets

Innovation 2: Diagnosing Entropy Collapse as the Central Failure Mode of Mixed-Policy Training — and Providing a Principled Remedy

Innovation 3: Demonstrating That Off-Policy Guidance Enables RLVR Where On-Policy RL Completely Fails — Reframing the "Weak Model" Problem

Innovation 4: The Unified Advantage Computation as a Mechanism for Emergent Curriculum Learning

Assessment: How Fundamental Are These Contributions?

5. Experimental Analysis

Evaluation Methodology

Main Quantitative Results

Comparison Against Prior RLVR Methods and On-Policy RL

Comparison Against Alternative Off-Policy Learning Methods

Extension to Multiple Base Models

LUFFY Succeeds Where On-Policy RL Fails

Ablation Studies and Robustness Checks

Critical Assessment

6. Limitations and Trade-offs

The Black-Box Difficulty Assumption: Why Off-Policy Guidance Works on Some Problems but Not Others Is Uncharacterized

The Hidden Cost of Teacher Dependence: Teacher Model Quality and Availability Are Unstated Prerequisites

The Unexplained OOD Generalization Gap: LUFFY Barely Improves Over On-Policy RL on Out-of-Distribution Tasks

The Single-Teacher, Single-Trace Design: Unexplored Sensitivity to the Amount and Diversity of Off-Policy Data

The Scope Restriction to Verifiable Rewards: LUFFY Cannot Be Applied to Open-Ended or Subjective Tasks

The Missing Latency and Inference Overhead Analysis: Training Efficiency Gains May Not Translate to Deployment Efficiency

7. Implications and Future Directions

How This Work Changes the Landscape

Follow-Up Research This Work Enables

Practical Applications and Downstream Use Cases

When to Prefer This Method

Adjacent in domain

Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?

TTRL: Test-Time Reinforcement Learning

Practical Choice: Setting $\pi_\phi = 1$ (Uniform Off-Policy Distribution)