ArXiv: 2603.01223

🎯 Pitch

On hard math problems, RL for LLMs often collapses because the model never samples a correct answer to learn from—a catastrophic reward sparsity problem. ReGFT solves this by having the model generate its own reasoning traces while guided by partial human reference solutions, producing synthetic correct trajectories that live in the model’s native reasoning distribution rather than imitating alien human proofs. This simple pre-RL fine-tuning step consistently boosts final RL performance across benchmarks, demonstrating that reference solutions unlock stronger reasoning only when they serve as scaffolding, not as imitation targets.

1. Executive Summary

This paper introduces Reference-Guided Fine-Tuning (ReGFT), a method that uses human-written reference solutions to synthesize correct model-generated trajectories for hard math problems before reinforcement learning begins — operationalized by prompting a Qwen3-4B model with partial reference solutions (e.g., the first 80% of sentences) and requiring it to produce its own reasoning trace, then fine-tuning on this mixture of guided and self-generated correct trajectories. Across three benchmarks (AIME 2024, AIME 2025, and Beyond-AIME) with DAPO-based RL on the OmniMath training set, ReGFT consistently improves supervised pass rates, accelerates RL convergence, raises the final performance plateau over both raw and ReFT baselines, and scales pass@k more reliably under increased test-time compute. The gains are driven specifically by model-derived reasoning that remains aligned with the model's own generation distribution, establishing that human reference solutions are effective for overcoming reward sparsity only when they serve as guidance for the model to construct its own trajectories rather than as direct imitation targets.

2. Context and Motivation

The Core Problem: Reinforcement Learning Fails When the Model Cannot Generate Correct Trajectories

The central problem this paper addresses is deceptively simple: reinforcement learning for mathematical reasoning can completely stall on hard problems because the model never produces a single correct answer, which means it never receives a positive reward signal to learn from. This is the reward sparsity problem, and it is not a minor inconvenience — it is a fundamental failure mode of the dominant post-training paradigm.

To understand why this is so consequential, consider how modern reasoning LLMs are trained. The standard recipe, which the paper explicitly positions itself within, is Reinforcement Learning with Verifiable Rewards (RLVR). In RLVR, the training loop works as follows: for each math problem, the model samples multiple reasoning trajectories (e.g., 16 or 64 complete solutions), a rule-based verifier checks whether each final answer is correct, and the model's parameters are updated to increase the probability of correct trajectories while suppressing incorrect ones. The elegance of RLVR is that it requires no human labels — correctness is determined automatically by comparing the model's output to a ground-truth answer using a deterministic grading function.

But this elegance masks a brittle dependency: RLVR can only provide learning signals when at least one sampled trajectory happens to be correct. If the model's base capability is too low on a particular problem — for instance, a challenging Olympiad-level math problem requiring multi-step algebraic manipulation and creative insight — then all 16 (or 64, or even 1024) generated solutions will be wrong. The verifier returns zero reward for every single trajectory. The gradient update becomes meaningless: the model receives no information about which directions in parameter space would improve performance, because there is no positive example to reinforce and no relative signal distinguishing less-wrong from more-wrong trajectories. Training stalls.

This is not a hypothetical concern. The paper's experimental setup uses OmniMath, a dataset of 4,428 Olympiad-level mathematics problems, with the Qwen3-4B-Instruct model. On these problems, the base model fails to generate a single correct trajectory on a substantial fraction of problems — the paper reports that under standard sampling with 64 attempts, 68.58% of problems are solvable, meaning 31.42% yield zero correct solutions. For those roughly 1,400 problems, RLVR would contribute nothing to learning. Every FLOP spent sampling and evaluating trajectories on those problems is wasted computation.

Why This Problem Matters: The Practical and Theoretical Stakes

The reward sparsity problem matters for several distinct reasons that the paper makes clear, both explicitly and through the framing of its experimental design:

It caps the return on investment from scaling RL compute. A common response to reward sparsity is to simply sample more trajectories per problem — if 16 samples yield zero rewards, try 64; if 64 fail, try 1024. This is the approach advocated by work on scaling reinforcement learning (e.g., BRoRL from Hu et al., 2025). But the paper's ablation in Table 2 shows that scaling samples alone is insufficient: even with 64 samples per problem, models initialized without reference-guided fine-tuning lag behind ReGFT-initialized models. The paper explicitly demonstrates this as a complementarity argument: more samples help, but they are not a substitute for improving the model's baseline competence. The wasted computation from sampling on unsolvable problems grows linearly with the sampling budget, creating a harsh economic trade-off between exploration and efficiency.

It prevents RL from expanding the model's capability frontier. Perhaps most critically, a growing body of evidence — which the paper engages with directly in Section 2 — suggests that naïve RL may not actually teach models to solve problems they could not solve before. Recent studies (Yue et al., 2025; Wu et al., 2026) observe that RL training can exhibit pass@k saturation: as you increase the number of samples at inference time (k), the accuracy plateaus, and on problems that were initially unsolvable, RL may even underperform the base model's raw inference. The implication is sobering: standard RL might primarily amplify and stabilize existing reasoning patterns rather than teaching genuinely new ones — it makes the model more reliably correct on what it already kind of knows, but does not teach it to solve fundamentally harder problems. If true, this represents a ceiling on what RLVR alone can achieve, independent of how much compute and algorithmic sophistication is thrown at it.

Hard problems are where the value is. In mathematical reasoning benchmarks like AIME and Beyond-AIME, the difficulty distribution is heavy-tailed: a small number of extremely challenging problems account for most of the performance differentiation between models. A method that improves accuracy from 30% to 60% on easy problems is less impactful than one that moves from 0% to 20% on the hardest tier, because the latter represents capability expansion rather than capability consolidation. The paper's focus on "hard problems" — operationally defined as those with less than 25% accuracy under the original model when sampled 16 times — is not arbitrary; it targets precisely the problems where RLVR currently contributes nothing.

Automatic verifiability makes reasoning the ideal testbed, but also imposes constraints. Math reasoning is the canonical RLVR domain because correctness is unambiguous and automatically checkable, which eliminates the need for expensive and potentially miscalibrated reward models. But this same verifiability property means that the reward sparsity problem is especially stark: the verifier is binary and exacting. On open-ended generation tasks, a learned reward model might assign partial credit to a "close" answer, providing some gradient signal even when the answer is technically wrong. In math RLVR, there is no partial credit — the answer either matches the ground truth or it does not. This makes math reasoning both the best case for RLVR (clean signal) and the worst case for reward sparsity (no signal at all on hard problems).

Where Prior Approaches Fall Short

The paper identifies three broad families of prior work addressing the capability-expansion problem, and positions ReGFT relative to each. Understanding these prior approaches — and their specific limitations — is essential for appreciating why ReGFT's contribution is meaningfully different.

Scaling RL and Adaptive Sampling

A natural first response to reward sparsity is to increase the number of samples per problem, hoping that at least one trajectory stumbles onto the correct answer. This is the logic behind BRoRL (Hu et al., 2025), ProRL (Liu et al., 2025), and KnapsackRL (Li et al., 2025). These methods scale exploration breadth (more rollouts per example), extend training horizons (more RL steps to allow rare successful strategies to emerge and propagate), and design difficulty-aware sampling schemes (allocating more rollouts to harder problems with stronger learning signals).

The paper acknowledges the value of these approaches — indeed, its own RL experiments use 64 samples per prompt, and the ablation in Table 2 confirms that increasing from 16 to 64 samples improves performance across all settings. However, the paper's key critique is implicit in its results: scaling sampling is a blunt instrument. If a problem is fundamentally outside the model's reasoning distribution — if no sequence of tokens the model is likely to generate could possibly constitute a correct solution — then no amount of sampling will help. The probability mass on correct trajectories is effectively zero. Increasing from 16 to 64 to 1024 samples multiplies the computational cost but multiplies zero by a larger constant. The paper's ReGFT approach, by contrast, aims to shift that probability mass — to make correct trajectories actually achievable — which is a qualitatively different intervention.

Moreover, the pass@k saturation findings from Yue et al. (2025) suggest that even when RL does improve performance, it may do so in a way that does not scale well with test-time compute. The paper's own pass@k analysis (Figure 5) shows that ReGFT + DAPO maintains an advantage over raw DAPO even at k = 1024, indicating it provides more than just a pass@1 boost — it genuinely expands the set of solvable problems. This is evidence that ReGFT addresses a dimension of capability that scaling sampling alone does not.

Question Augmentation During RL

A second family of approaches modifies the problem distribution during RL to make hard problems easier. Questa (Li et al., 2025) augments difficult questions with partial solutions or scaffolding, increasing the probability that the model's rollouts will land on a correct answer. Chen et al. (2025) similarly introduce hints — self-generated or externally provided — for initially unsolvable questions. These methods transform zero-reward cases into learnable instances by giving the model a "leg up" during the RL process itself.

The paper's key departure from this family is temporal: ReGFT operates entirely before RL, in a supervised fine-tuning stage that precedes reinforcement learning. This is not merely an implementation detail — it reflects a different theory of how capability expansion should work. Question augmentation methods inject guidance during RL, which means the model is simultaneously learning from both guided trajectories (where the hint did the work) and unguided trajectories (where the model succeeded on its own). There is a risk that the model learns to depend on the augmentation rather than internalizing the underlying reasoning patterns. By contrast, ReGFT uses reference guidance to synthesize training data for a pre-RL SFT stage, then strips away the guidance entirely during RL. The model never sees partial solutions during RL — it must generate complete correct solutions autonomously. The guidance serves to bootstrap competence, and then RL reinforces and generalizes that competence without the crutch of hints.

The paper also distinguishes itself from BREAD (Zhang et al., 2025), which anchors RL rollouts to partial expert prefixes when self-generated trajectories fail. BREAD modifies the RL process itself — it's an exploration strategy that ensures at least some rollouts succeed by starting them from a known-good prefix. ReGFT, by contrast, modifies the model's weights before RL begins. The model trained with ReGFT can generate correct trajectories from scratch, without prefix-level guidance at inference or RL time. This is a stronger form of capability acquisition: the model internalizes the reasoning patterns rather than relying on external scaffolding to complete them.

Interleaving SFT and RL

A third family alternates between supervised fine-tuning and reinforcement learning, using SFT to inject correct solutions on hard problems and RL to reinforce those behaviors. Ma et al. (2025) explicitly alternates between online SFT and RL phases. SRFT (Fu et al., 2025) proposes a single-stage objective that jointly optimizes supervised and reinforcement losses. Expo (Zhou et al., 2026) augments RL with self-generated explanations as auxiliary supervision.

The paper's approach differs from these in that the SFT stage is entirely pre-RL and uses a fundamentally different data source: reference-guided trajectories rather than self-generated correct trajectories or pure reference solutions. This is important because the interleaved approach still faces the cold-start problem on the very hardest questions: if the model cannot generate any correct trajectories at all, there is nothing for the SFT phase to train on. Interleaved methods can address this by using reference solutions directly (as some do), but the paper's ablation in Figure 4 shows that directly fine-tuning on raw reference solutions performs poorly — the model fails to internalize reasoning patterns that are not expressed in its own generative style. ReGFT solves this by transforming references into model-aligned trajectories before any training occurs.

The closest prior works are ReFT (Luong et al., 2024) and RL Teachers (Cetin et al., 2025). ReFT applies SFT to model-generated correct trajectories to increase their probability before RL — this is essentially the "self-generated" component of ReGFT without the reference-guided component. ReFT's fundamental limitation is that it assumes the model can already generate correct trajectories: if a problem yields zero correct samples in the initial sampling phase, ReFT has no training data for it. The paper directly compares ReFT and ReGFT throughout its experiments (Figures 3, 5; Table 1) and shows that ReFT accelerates early RL but fails to raise the final performance ceiling, and sometimes underperforms raw DAPO (as on Beyond-AIME). This is a crucial finding: self-generated trajectories alone are not sufficient for capability expansion.

RL Teachers distills trajectories from an already-RL-trained strong teacher model into a student. This requires access to a stronger model, which may not always be available (the "teacher" itself had to be trained somehow, which begs the question on the hardest problems). ReGFT uses static human-written references that come bundled with the training dataset (OmniMath includes verified reference solutions), making no assumption about access to stronger models.

How This Paper Positions Itself

The paper situates its contribution at the intersection of timing and methodology, with both dimensions being essential to its novelty.

On timing: The paper argues that improving the model before RL is fundamentally different from improving the RL process itself, and that this pre-RL stage is under-explored. The logic is straightforward but powerful: if RLVR's failure mode is the absence of any correct trajectories in the model's output distribution, then the most direct solution is to expand that output distribution to include correct trajectories before RL starts. This is analogous to the observation in pretraining that better mid-training leads to more effective downstream RL (Zhang et al., 2025, cited in Section 2). The paper is not claiming that pre-RL SFT is a new idea — ReFT does exactly that — but rather that how you construct the pre-RL training data is the critical variable that prior work got wrong, and that reference guidance is the missing ingredient.

On methodology: The paper's key methodological insight is what it calls reference-guided sampling: providing the model with a partial reference solution (specifically, the first 80% of sentences from the human-written solution) and requiring it to generate the complete reasoning trace from the beginning, in its own words and reasoning style. This is neither (a) directly fine-tuning on the human reference (which Figure 4 shows fails) nor (b) pure self-generation (which is ReFT and fails on hard problems) nor (c) injecting reference prefixes during RL rollouts (which is BREAD). It is a deliberate hybrid designed to solve a specific tension: human references contain correct reasoning patterns that the model needs to learn, but those patterns are expressed in a style and structure that the model cannot directly imitate because it falls outside its generation distribution. By using the reference as a hint rather than a target, the model translates the human reasoning into its own reasoning language, producing trajectories that are both correct (inherited from the reference) and distributionally aligned with the model (because the model generated them).

A subtle but important design choice that the paper does not belabor but that carries significant weight: the reference provides only structural guidance (the high-level solution approach, the key mathematical insight) while the model fills in all the reasoning steps. The paper notes that the model "almost always derives its own reasoning independently even when exposed to the full reference solution" (Section 4.3.1), indicating that the model is not simply copying or completing the reference — it genuinely reasons from scratch, informed but not determined by the hint. This is what makes the resulting trajectories viable as supervised training data: they represent reasoning patterns the model can actually reproduce, rather than patterns it can only parrot.

The paper also positions ReGFT as orthogonal to advances in RL algorithms. By using DAPO — a state-of-the-art RL method with dynamic sampling and decoupled clipping specifically designed to handle sparse rewards — the paper makes a deliberate point: even with a strong RL algorithm that already mitigates some forms of reward sparsity, initializing from a ReGFT checkpoint provides additional gains. This suggests that ReGFT addresses a fundamental data problem (the absence of correct trajectories in the model's output distribution) that algorithmic improvements alone cannot fully solve. Better optimization cannot create gradients where gradients do not exist.

Finally, the paper connects to a broader narrative in the field about the limits of RL for capability expansion. The findings from Yue et al. (2025) and Wu et al. (2026) — that RL may primarily amplify existing behaviors rather than inducing new capabilities — cast a shadow over pure RLVR approaches. ReGFT can be understood as a response to this challenge: if RL alone cannot teach new capabilities, then the solution is to teach those capabilities through a different mechanism (guided SFT) and then use RL to reinforce and generalize them. This reframes SFT not as a weaker alternative to RL, but as a complementary tool for capability injection that RL can subsequently optimize.

3. Technical Approach

3.1 Reader Orientation

This paper presents a pre-reinforcement-learning data augmentation pipeline that converts human-written math reference solutions into model-generated correct reasoning trajectories, then uses those trajectories to fine-tune the base model before RL begins. The core problem it solves is reward sparsity during RLVR: when a base model cannot generate any correct answers on hard problems, RL receives no positive gradients. The solution's shape is a two-stage process — first, use partial reference solutions as hints to elicit model-generated correct reasoning traces (reference-guided sampling), then supervised-fine-tune on a mixture of these reference-guided trajectories and self-generated correct trajectories, producing a checkpoint that can successfully generate correct solutions on previously-unsolvable problems. This checkpoint then serves as the initialization for standard RL, where it naturally receives positive rewards on problems that were previously zero-reward, enabling learning to proceed.

3.2 Big-Picture Architecture (Diagram in Words)

The ReGFT system has five major components, deployed in sequence before RL training:

  1. Base LLM (Qwen3-4B-Instruct) — the pretrained language model that generates all reasoning trajectories. It serves as both the target of fine-tuning and the generator in the reference-guided sampling process.
  2. Reference Solution Corpus (OmniMath) — a dataset of 4,428 Olympiad-level math problems, each paired with a human-written reference solution. These references provide high-level structural guidance but are written in a style outside the model's generation distribution.
  3. Trajectory Sampler — produces two kinds of trajectories per problem: (a) standard independent samples from the base model, and (b) reference-guided samples where the model conditions on the first ~80% of the human reference solution (with final answer withheld) and generates its own complete reasoning trace. Trajectories are verified by a rule-based answer checker.
  4. Hard Problem Filter — selects problems where the original base model has less than 25% accuracy when sampled 16 times. Only these hard problems enter the fine-tuning dataset. Easier problems are excluded to avoid overfitting.
  5. Supervised Fine-Tuning (SFT) — trains the base model on a mixture of self-generated correct trajectories (from standard sampling on hard problems) and reference-guided correct trajectories (from reference-guided sampling on hard problems). The output is a ReGFT checkpoint.

Information flows sequentially: base model → standard sampling → filter hard problems → reference-guided sampling on hard problems → collect correct trajectories from both sources → SFT on the mixture → ReGFT checkpoint → standard DAPO RL training from this checkpoint.

3.3 Roadmap for the Deep Dive

  • First, the base RLVR problem formulation (Section 3, opening) — what RLVR is, why reward sparsity occurs, and the formal definition of the failure mode (zero reward on all sampled trajectories). This establishes the technical problem that ReGFT must solve.
  • Second, the ReFT baseline (Section 3, "On-policy finetuning") — the existing approach of supervised fine-tuning on self-generated correct trajectories, its mechanism, and its fundamental limitation (dependency on existing correct trajectories). Understanding ReFT is essential because ReGFT directly extends it.
  • Third, the reference-guided sampling procedure (Section 3, "Reference-Guided Finetuning for Unsolved Problems") — how partial reference solutions are used as hints, what percentage is provided versus withheld, and why this produces model-aligned correct trajectories. This is the core technical contribution.
  • Fourth, the hard problem definition and training data construction (Section 3, "Reference-Guided Finetuning") — the 25% accuracy threshold, why easy problems are excluded, and how the mixture of self-generated and reference-guided trajectories is assembled.
  • Fifth, the transition to RL (Section 3, "Reinforcement learning from improved initialization") — how the ReGFT checkpoint feeds into DAPO, and why DAPO's dynamic sampling and decoupled clipping complement ReGFT's data improvements.
  • Sixth, the DAPO algorithm specifics (Section 3 and Appendix 6) — the choice of DAPO over PPO/GRPO, its key mechanisms (decoupled clipping, dynamic sampling), and the hyperparameter configurations used in all experiments.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a methodology paper whose core idea is that human-written reference solutions can be transformed into model-aligned correct trajectories through partial-hint-guided generation, and that training on these trajectories before RL addresses reward sparsity by expanding the set of problems on which the model can produce correct answers. The method itself is simple — a specific prompting strategy followed by standard SFT — but its effectiveness depends on careful design choices about what information to provide, what to withhold, and which problems to target.

The Reinforcement Learning with Verifiable Rewards (RLVR) Framework

ReGFT is designed to operate within — and improve — the RLVR post-training paradigm, so understanding RLVR's mechanics is prerequisite to understanding what ReGFT changes.

Standard RLVR training loop. The paper assumes the following procedure: for each problem $q$ in the training set, the current policy model $\pi_\theta$ (parameterized by weights $\theta$) samples $N$ reasoning trajectories $\{\tau_1, \tau_2, ..., \tau_N\}$, where each trajectory $\tau_i = (t_{i,1}, t_{i,2}, ..., t_{i,L_i})$ is a sequence of tokens representing a complete chain-of-thought solution ending in a final answer $a_i$. A rule-based verifier $V$ maps each final answer to a binary reward:

ri=V(ai,a∗)∈{0,1}r_i = V(a_i, a^*) \in \{0, 1\}

where $a^*$ is the ground-truth answer string, and $V$ returns 1 if and only if the extracted answer $a_i$ matches $a^*$ under a deterministic grading function (e.g., exact string match after normalization, or mathematical equivalence checking).

What it computes: a binary correctness label for each sampled trajectory — 1 if the model's final answer matches the ground truth, 0 otherwise. This label is the sole learning signal: there is no partial credit for partially correct reasoning or near-miss answers.

Why this form: using a rule-based verifier eliminates the need for human labeling or trained reward models, which are expensive, potentially miscalibrated, and can introduce reward hacking. The binary verifier is a gold-standard oracle on math problems where answers are exact. The cost is that the signal is extremely sparse: a trajectory that contains 15 correct reasoning steps but one arithmetic error at the end receives exactly the same zero reward as a completely nonsensical trajectory.

The gradient update. In a policy gradient framework like PPO or GRPO (of which DAPO is a variant), the model is updated to increase the log-probability of tokens in high-reward trajectories and decrease the log-probability of tokens in low-reward trajectories. The general form (simplified) is:

∇θJ≈Eq,τ∼πθ[A(τ)⋅∇θlog⁡πθ(τ∣q)]\nabla_\theta \mathcal{J} \approx \mathbb{E}_{q, \tau \sim \pi_\theta} \left[ A(\tau) \cdot \nabla_\theta \log \pi_\theta(\tau \mid q) \right]

where $A(\tau)$ is the advantage — a normalized version of $r_i$ that centers rewards relative to a baseline (often the mean reward across the sampled group for that problem).

What it computes: a gradient direction that moves the model's parameters to make trajectories with positive advantage more likely and trajectories with negative advantage less likely. The magnitude of the update on each trajectory is proportional to the magnitude of $A(\tau)$.

Why this form (and its failure mode on hard problems): this objective can only produce meaningful updates when the rewards within a sampled group have variance. Specifically, there must be at least one trajectory with $r_i = 1$ and at least one with $r_i = 0$. If all $N$ trajectories are incorrect ($r_i = 0$ for all $i$), then after normalization, all advantages are equal (all zero or all negative depending on the baseline), the gradient is uniform — it encourages or discourages all trajectories equally — and no differential learning occurs. The model receives no information about which types of reasoning are better or worse because there is no contrast. The paper's core problem is precisely this: on hard problems, $r_i = 0$ for all sampled trajectories, so RL stalls.

The reward sparsity-causing gap. The paper quantifies the severity of this problem in Section 4.3.1: on the OmniMath training set of 4,428 problems, standard sampling with 64 attempts produces correct trajectories on only 68.58% of problems. The remaining 31.42% — approximately 1,390 problems — yield zero correct trajectories and therefore provide no gradient signal during RL. These are the "hard problems" that ReGFT targets.

The ReFT Baseline: On-Policy Supervised Fine-Tuning

Before describing ReGFT, the paper establishes ReFT (REinforced Fine-Tuning, from Luong et al., 2024) as the baseline that ReGFT extends. Understanding ReFT is essential because ReGFT's innovation is precisely what it adds to ReFT's recipe.

ReFT's procedure. Given a base model $\pi_{\text{base}}$ and a training set of problems $\mathcal{Q}$:

  1. Sampling phase: For each problem $q \in \mathcal{Q}$, sample $K$ reasoning trajectories from $\pi_{\text{base}}$ (the paper uses $K = 64$ in its experiments).
  2. Verification phase: For each trajectory, extract the final answer and check it against the ground truth using $V$. Collect all trajectories where $V = 1$ — these are "verified correct" self-generated trajectories.
  3. Fine-tuning phase: Use standard next-token prediction (supervised learning) to fine-tune $\pi_{\text{base}}$ on the collected correct trajectories. The loss is the usual language modeling cross-entropy:

LSFT=−1∣τ∣∑t=1∣τ∣log⁡πθ(xt∣x<t,q)\mathcal{L}_{\text{SFT}} = -\frac{1}{|\tau|} \sum_{t=1}^{|\tau|} \log \pi_\theta(x_t \mid x_{<t}, q)

where $\tau$ is a correct trajectory, $|\tau|$ is its length in tokens, $x_t$ is the $t$-th token, and $x_{<t}$ are all preceding tokens. The loss is computed only over the trajectory tokens (the model's answer), not over the question prompt.

What it computes: the standard autoregressive language modeling loss — it maximizes the probability the model assigns to the actual tokens in correct trajectories. This is a maximum-likelihood objective that directly encourages the model to reproduce successful reasoning patterns.

Why ReFT works (for solvable problems): since the trajectories are generated by the model itself, they are on-policy — they lie within the model's generation distribution. This means the model does not need to learn a foreign reasoning style; it only needs to upweight reasoning patterns it already occasionally produces. This avoids the distributional mismatch that occurs when fine-tuning on human-written reference solutions (which Figure 4 shows performs poorly). ReFT effectively increases $\text{pass}@1$ by increasing the probability mass on trajectories that were previously low-probability but occasionally sampled.

ReFT's fundamental limitation. ReFT's sampling phase requires that the model produce at least one correct trajectory per problem for that problem to contribute to training. If $K$ samples all yield $V = 0$, ReFT has nothing to train on for that problem. Formally, if we define $p_{\text{base}}(q)$ as the probability that the base model produces a correct answer on problem $q$ in one attempt, then the probability that at least one of $K$ attempts succeeds is:

P(at least one correct)=1−(1−pbase(q))KP(\text{at least one correct}) = 1 - (1 - p_{\text{base}}(q))^K

When $p_{\text{base}}(q)$ is very small (hard problems), even with large $K$, this probability may be near zero. For instance, if $p_{\text{base}}(q) = 0.001$ and $K = 64$, the probability of at least one correct trajectory is only about 6.2%. For the hardest problems, $p_{\text{base}}(q)$ is effectively zero, and no amount of feasible sampling will find a correct trajectory. ReFT therefore cannot help on the hardest problems — it can only reinforce capabilities the model already possesses, not teach new ones.

The paper's experimental results confirm this: ReFT accelerates early-stage RL (Figures 2, 3) — because it upweights existing correct trajectories, making them more accessible to the RL process — but it does not raise the final performance plateau, and on Beyond-AIME it even underperforms raw DAPO. The interpretation is that ReFT helps the model more reliably access what it already knows, but does not teach it to solve problems it could not solve before.

The Hard Problem Definition

ReGFT does not apply to all problems — it specifically targets "hard problems" as defined by a threshold on the base model's accuracy. This triage is essential: easy problems already generate positive rewards and benefit from RL, so spending SFT training capacity on them risks overfitting without providing new capability.

Operational definition. A problem is classified as "hard" if:

accuracybase(q)<0.25\text{accuracy}_{\text{base}}(q) < 0.25

where $\text{accuracy}_{\text{base}}(q)$ is the fraction of correct answers when the base model samples 16 independent trajectories on problem $q$.

What it computes: this is an empirical estimate of $\mathbb{E}_{\tau \sim \pi_{\text{base}}} [V(\tau, q)]$ — the base model's expected correctness rate on problem $q$. With 16 samples, a problem with true $p_{\text{base}}(q) = 0.25$ has an expected 4 correct trajectories. The 25% threshold is chosen so that problems below it are ones where even with aggressive sampling during RL, the probability of a batch containing zero correct trajectories is non-negligible. For example, with $p_{\text{base}} = 0.20$ and $N = 64$ RL samples, the probability of all-wrong is $(1 - 0.20)^{64} \approx 6.3 \times 10^{-7}$ — essentially zero. But for $p_{\text{base}} = 0.01$, the probability of all-wrong with $N = 64$ is $0.99^{64} \approx 0.53$ — more than half the batches provide no positive signal. The threshold captures problems where RL reward sparsity is a genuine practical concern.

Why 16 samples and 25%: the paper does not ablate this threshold, so the specific choice is empirical rather than derived. The logic is that 16 samples is computationally cheap enough to run across the full training set, and 25% is a natural first-quartile-like cutoff that separates problems into "the model has some handle on" versus "the model is essentially guessing." The threshold is used as a hard boundary: problems above it are excluded from ReGFT training, and problems below it are included. The paper explicitly states that easier problems are excluded "to avoid overfitting and to focus learning on cases where the model lacks sufficient competence" (Section 3).

Consequence of the hard problem definition. Only problems that genuinely lack model competence enter ReGFT. This means ReGFT's training data is drawn from precisely the problems where RLVR would otherwise fail. The ReGFT checkpoint therefore has an expanded competence profile exactly where it matters — on problems that were previously zero-reward.

Reference-Guided Sampling: The Core Mechanism

This is the paper's central technical contribution. The goal is: given a hard problem $q$ for which the base model generates zero correct trajectories under standard sampling, produce at least one correct trajectory $\tau$ that (a) has the correct final answer, and (b) is expressed in the model's own reasoning style so it can serve as effective SFT training data. The mechanism is partial reference hinting: provide the model with a truncated human-written reference solution as context, then require it to generate a complete solution from scratch.

The hint construction. For each hard problem, the human-written reference solution — a chain-of-thought proof or derivation included with the OmniMath dataset — is split into sentences. The paper provides the model with the first approximately 80% of sentences as context, while withholding the remaining approximately 20% (which typically contains the final answer). Specifically, from Section 4.3.1:

"the model is provided with the first 80% sentences of the human-written reference solution as contextual guidance, while the remaining 20%—which typically contains the final answer—is withheld"

What this provides to the model: the reference hint encodes high-level structural information about the solution approach — the key mathematical insight, the sequence of transformations, the intermediate lemmas needed — without revealing the final answer. This is intentionally "structural guidance" rather than step-by-step instruction. The model sees what strategy to use but must execute that strategy using its own reasoning process.

Why 80/20 split: the withholding of the final 20% prevents trivial answer copying. If the model could see the final answer, it might simply reproduce the reference verbatim or generate a superficial solution that reaches that answer without genuine reasoning. By withholding the conclusion, the model must derive the answer itself, even though it knows the intended solution path. Interestingly, the paper notes (Section 4.3.1) that:

"in practice we found that the model almost always derives its own reasoning independently even when exposed to the full reference solution"

This suggests the model's generation process is robust against simple copying — perhaps because the reference solutions are stylistically foreign (concise, proof-like) and the model naturally translates them into its own more verbose reasoning style. However, the 80/20 split is retained as a principled safeguard.

The prompting procedure. Although the paper does not provide the exact prompt template in the main text, Appendix 7 ("Prompts") presumably contains the system/user prompt used for reference-guided sampling. The model receives:

  • The problem statement (same as in standard sampling).
  • The truncated reference solution as part of the context (likely in a designated "Reference" or "Hint" section of the prompt).
  • An instruction to generate its own solution (implicit in the standard math-solving prompt format).

The model then autoregressively generates a complete reasoning trace $\tau_{\text{guided}}$ including a final answer $a_{\text{guided}}$. The verifier $V$ checks whether $a_{\text{guided}} = a^*$.

What the guided trajectory contains: a complete chain-of-thought solution generated entirely by the model. Even though the model was "prompted" with the reference hint, the tokens in the trajectory come from the model's own distribution $\pi_{\text{base}}(\cdot \mid q, \text{hint})$ — the hint shifts the conditioning, but the generation is still autoregressive model output. This preserves the distributional alignment that direct reference fine-tuning loses: the trajectory tokens are plausible under the model's generation distribution (they were actually generated by it), so SFT on these tokens is a form of on-policy learning rather than off-policy imitation.

Empirical effectiveness of reference-guided sampling. Section 4.3.1 presents the key quantitative evidence that reference-guided sampling works:

  • Standard sampling solves 68.58% of the 4,428 OmniMath problems (i.e., produces at least one correct trajectory in 64 attempts).
  • Reference-guided sampling solves 70.82% — a modest overall increase.
  • Critically, reference-guided sampling solves an additional 5.85% of problems that are never solved under standard sampling. These are the problems ReGFT targets: problems for which the base model's unguided pass@64 = 0, but guided pass@64 > 0.
  • Conversely, 3.61% of problems are solved only by standard sampling — some problems are actually harder for the model with reference guidance, perhaps because the reference style confuses the model or the truncation removes critical context.

Why some problems remain unsolvable even with guidance. The paper identifies two limitations (Section 4.3.1):

  1. Model capability ceiling: the model's ability to interpret and internalize complex human reasoning is limited, particularly when reference solutions use "advanced mathematical tools or domain-specific insights that fall outside the model's training distribution." If the reference solution invokes, say, a theorem from algebraic topology that the model has never seen, providing the first 80% of that proof as a hint does not magically teach the model algebraic topology.
  2. Verifier false negatives: the rule-based verifier in OmniMath "cannot reliably assess open-ended or proof-style solutions," leading to false negatives where the model's reasoning is substantively correct but the extracted final answer does not match the ground truth format. This is a known limitation of answer-matching verifiers on problems where the answer format is complex or ambiguous.

Training Data Construction for ReGFT

Once reference-guided sampling produces verified correct trajectories on previously-unsolvable hard problems, these trajectories must be combined with self-generated correct trajectories (from ReFT) to form the ReGFT training dataset. The paper is explicit about the composition:

Data sources. For each hard problem (as defined by the 25% threshold), the ReGFT training set includes:

  1. Self-generated correct trajectories: from standard (unguided) sampling from the base model, verified by $V$.
  2. Reference-guided correct trajectories: from reference-guided sampling, verified by $V$.

Trajectories from both sources are correct (verified by the same verifier) and on-policy (generated by the same base model), differing only in whether the model had access to the partial reference hint during generation.

Why both sources? The mixture serves two purposes:

  • Self-generated trajectories provide training data for the subset of hard problems where the model can occasionally produce correct answers on its own — these trajectories represent the model's "natural" reasoning patterns on these problems. Including them prevents the SFT from shifting the model entirely toward reference-influenced solutions, which might hurt performance on problems where the model already had some competence.
  • Reference-guided trajectories provide training data for problems where the model never succeeds on its own — these are the incremental gains that ReGFT provides over ReFT. Without them, the hard problems with zero self-generated correct trajectories would contribute no training data.

The mixture ensures that the fine-tuning process both (a) reinforces existing correct reasoning patterns (like ReFT) and (b) teaches new reasoning patterns on previously unsolvable problems (the ReGFT addition). The paper does not specify the exact ratio of self-generated to reference-guided trajectories in the mixture; this is presumably determined by the empirical success rates of the two sampling strategies.

Exclusion of easy problems. Problems with base model accuracy ≥ 25% (when sampled 16 times) are entirely excluded from ReGFT training. The justification is two-fold:

  1. Focus: RL already works well on these problems because the model generates correct trajectories with sufficient frequency. SFT on easy problems would be redundant — RL will handle them naturally.
  2. Overfitting prevention: SFT on a large number of correct trajectories for easy problems could cause the model to memorize problem-specific solution patterns rather than learning generalizable reasoning. Since easy problems are numerous and the model already succeeds on them, excluding them reduces the risk of overfitting without sacrificing training signal where it is needed (hard problems).

The SFT procedure. The paper does not specify the SFT hyperparameters in the main text (e.g., learning rate, batch size, number of epochs), but the objective is standard next-token prediction cross-entropy loss on the concatenated set of correct trajectories:

LReGFT=−1∣DReGFT∣∑τ∈DReGFT1∣τ∣∑t=1∣τ∣log⁡πθ(xt∣x<t,q)\mathcal{L}_{\text{ReGFT}} = -\frac{1}{|\mathcal{D}_{\text{ReGFT}}|} \sum_{\tau \in \mathcal{D}_{\text{ReGFT}}} \frac{1}{|\tau|} \sum_{t=1}^{|\tau|} \log \pi_\theta(x_t \mid x_{<t}, q)

where $\mathcal{D}_{\text{ReGFT}}$ is the combined set of self-generated and reference-guided correct trajectories on hard problems, and the loss is averaged over all tokens in all trajectories.

What it computes: the log-likelihood of the correct trajectory tokens under the model's current parameters. Minimizing this loss (or equivalently maximizing log-likelihood) pushes the model to assign higher probability to correct reasoning patterns on hard problems.

Why this form: it is standard supervised fine-tuning, compatible with any autoregressive language model training pipeline. No special loss function or RL-style advantage weighting is needed. The innovation is entirely in what data goes into $\mathcal{D}_{\text{ReGFT}}$, not in how the model is trained on it. This simplicity is a strength: ReGFT can be plugged into any existing SFT pipeline with no infrastructure changes.

The Transition from ReGFT to Reinforcement Learning

The ReGFT checkpoint — the base model fine-tuned on the mixture of self-generated and reference-guided trajectories — is then used as the initialization for standard RLVR training using DAPO. This transition is where ReGFT's benefits manifest operationally.

What changes between the ReGFT checkpoint and the base model. On hard problems where the base model previously generated zero correct trajectories (even with 64 attempts), the ReGFT checkpoint now generates correct trajectories with non-trivial probability. Specifically:

  • Before ReGFT: $p_{\text{base}}(q) \approx 0$ → probability of at least one correct trajectory in a batch of 64 ≈ 0 → zero reward → no gradient signal.
  • After ReGFT: $p_{\text{ReGFT}}(q) > 0$ (the exact value is not reported, but empirical pass rates improve — Table 1 shows ReGFT pass@64 on OmniMath training set is higher than both raw and ReFT) → probability of at least one correct trajectory in a batch of 64 is non-negligible → positive rewards appear → RL can proceed.

Why the RL phase is still necessary. One might ask: if ReGFT already produces correct trajectories, why not just stop there? The paper's argument — supported by the experimental results — is that ReGFT provides an initialization that makes RL feasible, but RL is still needed to reinforce and generalize the learned reasoning patterns. SFT increases the probability of correct trajectories but does not actively suppress incorrect ones (beyond the normalization effect of a well-calibrated language model). RL, by contrast, explicitly downweights incorrect trajectories through the negative advantage component of the policy gradient, which sharpens the model's behavior and reduces the probability of near-miss or plausible-but-wrong answers. Additionally, RL's exploration during training can discover new correct reasoning patterns beyond those present in the ReGFT training data, further improving performance.

From the experimental results: ReGFT + DAPO consistently outperforms ReGFT alone (Figure 2 shows the ReGFT checkpoint's supervised accuracy — the leftmost point — is lower than the eventual RL accuracy after hundreds of steps). The RL phase adds substantial value, but it can only do so because ReGFT created the initial correct trajectories that provide gradient signal.

Why ReGFT is orthogonal to the choice of RL algorithm. The paper deliberately uses DAPO — a strong, state-of-the-art RL algorithm — to demonstrate that ReGFT provides benefits beyond what algorithmic sophistication alone can achieve. Section 3 states:

"We deliberately choose DAPO to demonstrate that the benefits of ReGFT are orthogonal to advances in reinforcement learning algorithms. Despite DAPO already incorporating dynamic sampling to mitigate uninformative gradients, initializing training from ReGFT checkpoints consistently leads to further improvements."

This is an important methodological point: DAPO's dynamic sampling already tries to handle reward sparsity by ensuring each update sees diverse reward signals, but it cannot create positive rewards where none exist. ReGFT addresses the root cause (no correct trajectories) rather than optimizing around it. The two operate at different levels of the training stack and are complementary.

DAPO: The Reinforcement Learning Algorithm

The paper uses DAPO (Decoupled Clip and Dynamic sAmpling Policy Optimization, from Yu et al., 2025) as the RL algorithm for all experiments. DAPO is a variant of GRPO (Group Relative Policy Optimization, from Shao et al., 2024), which itself is a variant of PPO designed for reasoning tasks. Understanding DAPO's key mechanisms explains why it was chosen and how it interacts with ReGFT.

DAPO's two key innovations over GRPO/PPO:

  1. Decoupled clipping. In standard PPO, the clipped surrogate objective uses a single clipping parameter $\varepsilon$ that applies symmetrically to both positive and negative advantages. DAPO decouples this into two separate thresholds:

    • $\varepsilon_{\text{low}}$ (set to $0.2$ in the paper's experiments): the clipping threshold for trajectories with negative advantage — prevents the model from reducing the probability of low-reward trajectories too aggressively, which would collapse policy diversity.
    • $\varepsilon_{\text{high}}$ (set to $0.28$ in the paper's experiments): the clipping threshold for trajectories with positive advantage — allows the model to reinforce high-reward trajectories more aggressively than standard PPO would permit.

    This asymmetry is designed for sparse-reward settings: when most trajectories have zero reward, aggressively downweighting them (low $\varepsilon_{\text{low}}$) could eliminate exploration diversity, but allowing stronger upweighting of rare positive trajectories (high $\varepsilon_{\text{high}}$) helps propagate their signal.

  2. Dynamic sampling. Instead of using all sampled trajectories for each update, DAPO filters trajectory groups to ensure reward diversity within each group. Specifically, it only uses trajectory groups where at least one trajectory has positive reward and at least one has zero reward — precisely the condition for meaningful gradient signal that we discussed earlier. Groups where all trajectories are correct or all are incorrect are discarded for that update step. This prevents "wasted" updates on batches with no contrast.

Why DAPO is still insufficient without ReGFT. Dynamic sampling ensures that updates only occur when there is reward contrast, but it does not increase the probability that such contrast exists. On problems where the base model generates zero correct trajectories (the hard cases ReGFT targets), dynamic sampling simply discards all groups and no updates occur for those problems. DAPO's dynamic sampling is a filter — it prevents bad updates — but it cannot create positive examples where none exist. ReGFT creates those positive examples by expanding the model's competence, after which DAPO's dynamic sampling can effectively use them.

Hyperparameters (from Appendix 6). The paper reports the following DAPO configuration, matching the recommendations from the original DAPO paper:

  • Optimizer: AdamW
  • Learning rate: constant $1 \times 10^{-6}$ with linear warmup over the first 20 rollout steps
  • Batch size: depends on samples per prompt:
    • 16 samples/prompt → prompt batch size 512 (total 8,192 trajectories per rollout step)
    • 64 samples/prompt → prompt batch size 128 (total 8,192 trajectories per rollout step)
  • Mini-batch size: 2,048 (4 gradient updates per rollout step, since 8,192 / 2,048 = 4)
  • Clipping thresholds: $\varepsilon_{\text{low}} = 0.2$, $\varepsilon_{\text{high}} = 0.28$
  • Generation parameters: maximum 16,384 tokens per trajectory, temperature $0.7$, top-p sampling with $p = 0.9$

What the batch size configuration implies. At 64 samples per prompt with 128 prompts per rollout step, the model generates 8,192 complete reasoning trajectories in parallel before computing a single gradient update. Each trajectory can be up to 16,384 tokens — about 12,000 words — so a single rollout step generates up to approximately 100 million tokens, then processes them in 4 mini-batch gradient steps. This is computationally intensive but provides the broad exploration coverage needed for sparse-reward math reasoning.

The interaction between DAPO's dynamic sampling and ReGFT. Dynamic sampling discards trajectory groups that lack reward contrast. On hard problems before ReGFT, most groups are all-zero-reward and get discarded — RL makes essentially no progress. After ReGFT, some groups now contain correct trajectories, so they pass the dynamic sampling filter and contribute to parameter updates. This is the mechanism by which ReGFT "enables" RL: it converts problems from "always filtered out" to "sometimes contributes," increasing the effective number of training signals the model receives.

Summary of Design Choices and Their Justifications

  • Partial reference hint (80/20 split) over full reference: prevents trivial answer copying while providing structural guidance. Even though the model rarely copies verbatim, withholding the final answer ensures the model must derive it independently.
  • Reference-guided sampling over direct reference fine-tuning: Figure 4 conclusively shows that directly fine-tuning on human references fails because the model cannot internalize reasoning patterns outside its generation distribution. Reference-guided sampling produces trajectories that are both correct (informed by the reference) and distributionally aligned (generated by the model).
  • ReGFT's pre-RL timing over mid-RL augmentation: by operating before RL, ReGFT ensures the model learns to generate correct trajectories autonomously, without dependency on hints at inference time. Mid-RL augmentation methods (e.g., BREAD, Questa) risk the model learning to exploit augmentation rather than internalizing reasoning.
  • Hard problem definition (25% accuracy, 16 samples) over training on all problems: focuses SFT capacity on problems where RL would otherwise fail, avoiding redundant training on easy problems and preventing overfitting.
  • Mixture of self-generated and reference-guided trajectories over either alone: self-generated trajectories reinforce existing competence; reference-guided trajectories teach new competence. Both are needed because hard problems span a spectrum — some have occasional correct solutions (self-generated data exists) while others never do (reference-guided data is essential).
  • DAPO over standard PPO/GRPO: decoupled clipping and dynamic sampling are specifically designed for sparse-reward reasoning tasks, making DAPO a strong testbed for evaluating whether ReGFT provides benefits beyond algorithmic sophistication.

4. Key Insights and Innovations

Innovation 1: Reward Sparsity Is a Data Problem, Not an Algorithm Problem — And the Solution Lies Before RL Begins

The paper's most fundamental conceptual move is reframing the reward sparsity bottleneck from an algorithmic challenge to a data preconditioning challenge. This is not a semantic distinction — it has concrete implications for where research effort should be directed and what kinds of solutions are viable.

The dominant framing before ReGFT. The field has largely treated reward sparsity as an optimization problem to be solved within the RL process. BRoRL scales up sampling budgets so that rare correct trajectories eventually appear. KnapsackRL allocates more rollouts to harder problems. DAPO introduces dynamic sampling to filter out uninformative updates. Questa augments prompts during RL. BREAD prefixes rollouts with expert hints. Each of these methods — and the paper cites them explicitly — accepts the base model's competence as given and tries to extract learning from an impoverished signal through better exploration, better filtering, or external augmentation during training.

What the paper argues instead. ReGFT is built on a quieter but more fundamental claim: if the model cannot generate any correct trajectories, the problem is not that RL is insufficiently clever — the problem is that the model's output distribution contains zero probability mass on correct solutions. No amount of algorithmic sophistication can learn from gradients that do not exist. The correct intervention is to shift the probability mass before RL begins, transforming the problem from "zero signal" to "sparse but present signal" at which point standard RL — even without algorithmic enhancements — can make progress.

This reframing matters because it changes where you invest effort. If reward sparsity is an algorithmic problem, you invest in better exploration strategies, adaptive sampling, or RL objective design. If it is a data problem, you invest in better pre-RL training data construction — which is precisely what ReGFT does. The paper's experimental architecture is deliberately set up to test this distinction: it uses DAPO, a state-of-the-art RL algorithm specifically designed for sparse rewards, and shows that ReGFT provides gains on top of DAPO. The gains are not from making DAPO better — they are from giving DAPO a problem it can actually solve.

Why this is a fundamental shift, not an incremental refinement. Prior work in the "interleaving SFT and RL" family (Ma et al., 2025; Fu et al., 2025) also combines supervised and reinforcement learning, which might make ReGFT seem like a variant. But these methods use SFT during RL to inject corrective signals, mixing objectives and complicating optimization. ReGFT's sharp separation — SFT first, RL second, with no overlap — reflects a different mental model: SFT expands the set of solvable problems, and then RL polishes and generalizes. This temporal decoupling is cleaner theoretically (each phase has a single objective), simpler to implement (no alternating schedules or joint losses), and empirically effective. It also aligns with emerging evidence (Zhang et al., 2025) that stronger mid-training leads to better downstream RL, suggesting a broader principle that the paper's specific instantiation validates.

Evidence anchor. The claim is substantiated by Figure 2, which shows that ReGFT + DAPO consistently outperforms raw DAPO across all three benchmarks throughout training. The gap is not just in convergence speed — it persists at the final plateau — indicating that ReGFT changes what the model can ultimately learn, not just how fast it learns. Table 2 reinforces this: even with 64 samples per problem (a large exploration budget), models without ReGFT initialization lag behind, confirming that algorithmic scaling alone cannot substitute for improved baseline competence.

Innovation 2: Human Reference Solutions Can Teach the Model — But Only When the Model Translates Them Into Its Own Reasoning Language

The paper's second conceptual contribution is a specific diagnosis of why directly fine-tuning on human-written reference solutions fails, paired with a mechanism that succeeds. This is not merely a "we tried SFT on references and it didn't work, so we did something else" empirical observation — it is a principled argument about distributional alignment as a requirement for effective knowledge transfer from human demonstrations to language models.

The dominant assumption before ReGFT. The natural instinct when you have human-written reference solutions is to fine-tune on them directly. After all, they are correct, they are high-quality, and they demonstrate the reasoning patterns you want the model to learn. This is standard practice in many SFT pipelines. The paper's ablation (Figure 4) shows that this instinct is wrong — but more importantly, it provides a coherent explanation for why it is wrong: human reference solutions "lie outside [the model's] own reasoning distribution" (Section 1), meaning the model's generation process is not structured to produce tokens in the style, granularity, or structure of human-written proofs. When forced to predict those tokens during SFT, the model memorizes surface patterns rather than internalizing the underlying reasoning — leading to poor generalization.

What ReGFT does differently at the conceptual level. ReGFT introduces a translation interface between human knowledge and model capabilities. The partial reference hint serves as a semantic bridge: it conveys the high-level mathematical strategy (the "what to do") without dictating the implementation (the "how to express it"). The model then translates that strategy into its own generation distribution — producing tokens it can actually reproduce, in a style it naturally generates, with reasoning steps it can articulate. The resulting trajectory is correct (verified by the same ground-truth checker) but is expressed in the model's "native" reasoning language rather than the human's "foreign" one.

This translation concept is the paper's most original theoretical contribution. It reframes the challenge of incorporating human knowledge as a cross-distributional transfer problem: human solutions come from distribution P_human, the model generates from distribution P_model, and simply minimizing cross-entropy between P_model and P_human fails because the two distributions have fundamentally different support structures. The reference-guided approach instead maps P_human knowledge into P_model tokens by having the model perform the translation itself, producing training data that lies within P_model's support.

Why this is broader than math reasoning. While the paper's instantiation is in mathematical problem-solving, the distributional alignment argument generalizes. Any domain where human expert demonstrations differ stylistically from model outputs — code with different naming conventions, scientific explanations at different levels of abstraction, creative writing with different voice — faces the same direct-SFT failure mode. ReGFT's "have the model translate" approach is a template for knowledge transfer that preserves correctness while respecting the model's generation characteristics. This is a conceptual contribution to the broader challenge of learning from human demonstrations in language models, not just a math-specific trick.

Evidence anchor. Figure 4 is the decisive comparison: directly fine-tuning on raw human reference solutions produces substantially weaker performance both during SFT and after RL compared to ReGFT. The gap is large and consistent across AIME 2024, AIME 2025, and Beyond-AIME. The paper's qualitative note that "the model almost always derives its own reasoning independently even when exposed to the full reference solution" (Section 4.3.1) further supports the claim that the model naturally translates rather than copies — it cannot help but produce its own reasoning style.

Innovation 3: ReFT Accelerates, ReGFT Expands — A Diagnostic Distinction Between Two Types of Pre-RL Fine-Tuning

The paper's most nuanced contribution is not showing that ReGFT outperforms ReFT (which is expected, since ReGFT has more training data) but rather characterizing the qualitative difference in what each method accomplishes. This distinction — between capability consolidation and capability expansion — provides a diagnostic framework for evaluating pre-RL interventions that extends beyond this paper's specific methods.

The diagnostic pattern in the results. Looking across Figures 2, 3, and 5, a clear pattern emerges:

  • ReFT improves early-stage RL convergence speed (visible in the steeper initial slope in Figures 2 and 3) and improves pass@1 (higher accuracy with a single sample, visible in Table 1). However, it does not raise the final RL plateau — on Beyond-AIME, it actually underperforms raw DAPO at convergence (Figure 3c) — and its pass@k gains diminish as k increases (Figure 5, where ReFT + DAPO and raw DAPO converge at high k).
  • ReGFT also accelerates early RL (similar initial slope advantage), but crucially raises the final convergence plateau (Figures 2, 3) and maintains pass@k advantages even at k = 1024 (Figure 5).

What this pattern reveals about the underlying mechanisms. ReFT takes correct trajectories the model already occasionally produces and increases their probability. This makes the model more efficient at accessing its existing capabilities — hence the faster convergence (RL wastes fewer updates on problems where correct trajectories were low-probability but present) and better pass@1 (higher probability mass on the most likely correct solution). But it does not enable the model to solve problems it could never solve before — the set of solvable problems is unchanged, only the probability of landing on a correct solution within that set increases.

ReGFT, by contrast, takes problems where the model previously had zero correct trajectories and creates correct trajectories through reference guidance. This expands the set of solvable problems — the model can now generate correct solutions where it literally could not before. RL then reinforces these newly-acquired patterns, raising the ceiling. The pass@k advantage at high k (Figure 5) is the signature of capability expansion: when the model explores broadly at inference time, it finds correct solutions in regions of the solution space that were previously empty.

Why this distinction matters beyond this paper. The consolidation-vs-expansion diagnostic provides a framework for evaluating any pre-RL or mid-RL intervention. If a method only improves convergence speed or pass@1 but does not raise asymptotic performance or high-k pass@k, it is likely consolidating existing capabilities rather than teaching new ones. This has direct implications for research prioritization: if the goal is to push the frontier on genuinely hard problems (as the paper's focus on AIME and Beyond-AIME suggests), then capability-expanding methods are essential, and consolidation-only methods are insufficient regardless of their compute efficiency benefits.

The distinction also explains the apparent contradiction in prior literature between papers showing RL improves pass@k (supporting capability expansion) and papers showing pass@k saturation (suggesting only consolidation). The outcome depends on whether the base model had any correct trajectories to begin with — a variable that is often uncontrolled in experiments. ReGFT's explicit hard-problem threshold makes this variable visible and intervenable.

Evidence anchor. The consolidation-vs-expansion pattern is visible across multiple metrics: Figure 3 (ReFT underperforms raw DAPO on Beyond-AIME at convergence despite early advantage), Figure 5 (ReFT pass@k gains shrink at high k, ReGFT gains persist), and Section 4.3.1 (reference-guided sampling solves 5.85% of problems never solved by standard sampling — direct evidence of capability expansion).

Innovation 4: Inference-Time Scaling Benefits from Pre-RL Capability Expansion, Not Just Post-RL Search

The paper's pass@k analysis (Section 4.2.4, Figure 5) surfaces a finding that is easy to overlook but carries significant implications for how the field thinks about test-time compute: improving the model's baseline competence before RL produces more reliable inference-time scaling than RL alone, and the quality of that scaling depends on how the baseline competence was improved.

The standard expectation about RL and pass@k. A natural assumption is that RL improves pass@k because it teaches the model to explore more effectively — to generate a broader, more diverse set of potentially-correct solutions at inference time. If this were true, then any RL-trained model (raw DAPO, ReFT + DAPO, ReGFT + DAPO) should show similar pass@k scaling behavior relative to their respective pass@1. The gap between models should be roughly constant across k, or perhaps widen if some RL methods teach better exploration.

What the paper actually finds. The results in Figure 5 show something more nuanced. ReFT + DAPO improves pass@1 over raw DAPO but its advantage shrinks at higher k — at k = 1024, ReFT + DAPO and raw DAPO are nearly identical on some benchmarks. This suggests that ReFT's benefit is primarily in making the most-likely correct solution more probable (a pass@1 effect) rather than increasing the diversity of correct solutions in the model's output distribution.

ReGFT + DAPO, by contrast, maintains a clear advantage over raw DAPO even at k = 1024. On AIME 2025 and Beyond-AIME, the ReGFT pass@k curve sits consistently above both ReFT and raw curves across all k. This suggests that ReGFT genuinely expands the support of the model's correct-solution distribution — there are more distinct correct solutions that the model can generate, not just higher probability on the ones it already could.

Why this matters for the test-time compute debate. The field is currently debating whether scaling test-time compute (via pass@k or tree search) can substitute for scaling model capability (via pretraining or RL). The paper's finding adds a crucial nuance: the effectiveness of test-time compute depends on how the model's capability was acquired. A model that learned primarily through consolidation (ReFT-style) may show diminishing returns from additional samples because it has few distinct correct solution strategies in its distribution — sampling more just explores the same narrow peak more thoroughly. A model that learned through genuine capability expansion (ReGFT-style) has a broader set of correct strategies, so additional samples continue to discover new correct solutions rather than re-sampling the same one.

This is not a claim the paper makes explicitly, but it follows directly from the data. It suggests that future work on inference-time compute scaling should consider not just the quantity of test-time compute but the quality of the model's solution diversity — and that pre-RL training data construction is a lever for improving that diversity.

Evidence anchor. Figure 5 across all three benchmarks: ReGFT + DAPO maintains separation from raw DAPO at k = 1024, while ReFT + DAPO and raw DAPO converge. The gap is most pronounced on Beyond-AIME (Figure 5c), which is the hardest benchmark — supporting the interpretation that capability expansion is most valuable where the solution space is sparsest.

Innovation 5: The Hard Problem Threshold Operationalizes "Capability Boundary" as an Empirical, Model-Specific Quantity

While the hard problem definition (25% accuracy with 16 samples) appears to be a minor implementation detail, it represents a conceptual contribution in its own right: replacing dataset-level difficulty labels (e.g., "AIME is harder than MATH") with model-specific, empirically measured difficulty. This operationalization enables targeted intervention and reveals structure that aggregate metrics obscure.

The common approach to difficulty in RL for reasoning. Most RLVR papers evaluate on standard benchmarks (MATH, AIME, etc.) and report aggregate accuracy. Some distinguish between "easy," "medium," and "hard" subsets based on the benchmark's own difficulty taxonomy (e.g., MATH level 1-5). But these taxonomies are designed by dataset creators based on human judgments or problem characteristics, not calibrated to any specific model's capabilities. A problem that is "hard" for humans (because it requires advanced number theory) might be easy for an LLM (because it has seen many similar examples in training), and vice versa.

What ReGFT's threshold does differently. The 25% threshold on the base model's own accuracy is a model-relative difficulty measure. It answers the question: "for this specific model, with this specific sampling budget, how reliably can it solve this problem?" rather than "how hard is this problem in some absolute sense?" This is the right question for RLVR because reward sparsity depends on the model's capabilities, not on the problem's intrinsic mathematical difficulty. Two problems that are equally "hard" by human standards can have drastically different reward sparsity profiles for different models.

The threshold also serves a practical triage function that the paper does not emphasize but that is significant for scaling: it identifies exactly which problems will remain zero-reward during RL, enabling targeted intervention. Without such a threshold, ReGFT would either waste SFT capacity on problems RL already solves (overfitting risk) or miss problems where RL would fail (leaving reward sparsity unaddressed). The threshold makes the intervention efficient.

Why this is conceptually novel despite its simplicity. The idea of conditioning on estimated difficulty is not new — the DAPO paper itself uses dynamic sampling based on reward signals. But using the base model's accuracy to define a pre-RL training curriculum — deciding which problems get SFT attention based on their expected RL-learnability — is a different application of the same insight. It treats the base model as an oracle for its own learnability, creating a feedback loop between model capability assessment and training data construction. This is a design pattern that could generalize: future work could use model-specific difficulty estimates to decide not just which problems to include in SFT, but how much reference guidance to provide, how many trajectories to generate, or which RL hyperparameters to use.

Evidence anchor. Table 1 shows that ReGFT improves pass@64 on the OmniMath training set more than ReFT does — the threshold successfully identifies problems where ReGFT's reference guidance adds value beyond self-generated trajectories. The fact that this improvement transfers to out-of-distribution benchmarks (AIME, Beyond-AIME) without overfitting suggests the threshold-based filtering is well-calibrated.

A Note on What the Paper Does Not Claim as Innovation

The paper is commendably precise about what is and is not novel. It does not claim to invent pre-RL SFT (ReFT predates it), reference utilization (BREAD, Questa, and RL Teachers all use reference solutions), DAPO (which is adopted from Yu et al., 2025), or the idea that reward sparsity is a problem (widely recognized). Its contributions are specifically in:

  • When the reference guidance is applied (pre-RL, not during RL).
  • How the reference is used (as a translation hint, not a direct target or a rollout prefix).
  • Why this works (distributional alignment via model-generated reasoning, verified through ablation).
  • What this accomplishes (capability expansion, not just consolidation, as diagnosed through convergent-vs-plateau RL dynamics and pass@k scaling behavior).

These distinctions are subtle but consequential — they are the difference between "we applied SFT before RL, which others have done" and "we identified a specific failure mode (distributional mismatch in knowledge transfer), designed a mechanism to address it (reference-guided translation), and provided diagnostic evidence distinguishing our solution's effects from those of superficially similar approaches."

5. Experimental Analysis

Evaluation Methodology

  • Dataset. The training data is OmniMath (Gao et al., 2024), consisting of 4,428 Olympiad-level mathematics problems, each accompanied by a verified human-written reference solution. The paper specifically selects OmniMath because it "induces severe reward sparsity, making it well suited for studying verifier-based reinforcement learning" (Section 4.1). For evaluation, three held-out benchmarks are used: AIME 2024, AIME 2025, and Beyond-AIME (ByteDance Seed, 2025), a benchmark of 100 problems described as "more challenging and unsaturated" than AIME (Section 4.1). The paper does not report the exact number of problems in AIME 2024 or AIME 2025.

  • Base model. All experiments use Qwen3-4B-2507-Instruct (Yang et al., 2025) as the base model. The paper justifies this choice on two grounds: first, it "provides strong reasoning performance and reliable instruction following ability" (Section 4.1); second, prior work showed that Qwen2.5 models "can be sensitive to spurious reward signals" (Shao et al., 2025, cited in Section 4.1), making Qwen3's improved stability a more reliable testbed for isolating the effects of ReGFT from confounding RL instability. The 4B scale is deliberately chosen as a regime where the model is capable but not saturated — the base model's pass@64 on OmniMath is 68.58% (Section 4.3.1), leaving substantial headroom for improvement from both SFT and RL.

  • Metrics. The primary evaluation metric throughout is accuracy: the fraction of problems for which the model's final answer matches the ground truth under a rule-based verifier. This is reported as pass@1 (accuracy with a single generated solution) in the RL training curves (Figures 2-4) and in Table 1, and as pass@k in the inference-time scaling analysis (Section 4.2.4, Figure 5). Pass@k is estimated using the standard unbiased estimator from N = 1024 independent samples:

    pass@k=1−(N−ck)(Nk)\text{pass@k} = 1 - \frac{\binom{N - c}{k}}{\binom{N}{k}}

    where c is the number of correct samples among the N generations. The rule-based verifier that determines correctness operates on final answer matching, not on solution quality — the paper explicitly notes that this verifier "cannot reliably assess open-ended or proof-style solutions, leading to false negatives even when the model's reasoning is substantively correct" (Section 4.3.1). This means reported accuracies on Beyond-AIME (which includes proof-style problems) may underestimate true reasoning capability. For the OmniMath training set, pass@64 is also reported (Table 1) as a measure of the fraction of problems for which at least one correct trajectory is generated in 64 attempts.

  • Baselines. The paper compares against three main baselines:

    1. Raw DAPO: The base model initialized directly into DAPO-based RL with no pre-RL fine-tuning. This represents the standard RLVR pipeline against which all pre-RL interventions are measured (Figures 2-4; Table 1).
    2. ReFT + DAPO (Luong et al., 2024): The base model fine-tuned via REinforced Fine-Tuning — SFT on self-generated correct trajectories from standard sampling — before DAPO RL. This isolates the contribution of on-policy self-generated training data (Figures 3, 5; Table 1).
    3. Direct SFT on references + DAPO: The base model fine-tuned directly on raw human-written reference solutions (no model-generated reasoning required) before DAPO RL. This ablation tests whether exposure to correct human reasoning without the model's own generative translation is sufficient (Figure 4; the paper does not give this configuration a separate name).

    Additionally, Table 1 reports the supervised-only (pre-RL) performance of raw model, ReFT, and ReGFT to quantify the improvement from fine-tuning alone before any RL occurs.

  • Generation budget / compute accounting. The paper measures compute in two complementary ways. During fine-tuning (both ReFT and ReGFT), the key quantity is the number of sampled trajectories per problem: 16 trajectories per problem are used to define the hard problem threshold (the 25% accuracy criterion in Section 3), though the actual sampling budget for trajectory collection during ReFT/ReGFT data generation is 64 trajectories per problem (as implied by the 68.58% solvable rate at 64 attempts in Section 4.3.1). During RL, the generation budget is reported as samples per prompt per rollout step: the default is 64 responses per prompt (Section 4.2.1), with an ablation at 16 responses per prompt (Section 4.3.3, Table 2). Each response can be up to 16,384 tokens; generation uses temperature 0.7 and top-p sampling with p = 0.9 (Section 4.1). The batch size is adjusted to maintain constant total trajectory count: 512 prompts at 16 samples/prompt and 128 prompts at 64 samples/prompt, yielding 8,192 trajectories per rollout step in both configurations (Appendix 6). For inference-time scaling (Figure 5), N = 1,024 independent samples are generated per problem to estimate pass@k up to k = 1,024.

  • Cross-validation / statistical protocol. The paper does not report any cross-validation, multiple random seeds, or confidence intervals. All results in Figures 2-5 appear to be from single training runs. The RL training curves (Figures 2-4) plot accuracy against training steps, with the trajectory showing per-step evaluation rather than averages over multiple seeds. This is a notable absence: without multiple seeds, it is impossible to quantify the variance in RL training dynamics (which are known to be noisy) or to assess whether the observed performance differences between ReGFT and baselines are statistically reliable. The training set size (4,428 problems) and test set sizes (AIME 2024/2025 typically have 30 problems each; Beyond-AIME has 100) are small enough that sampling variance could be substantial. Table 1 and Table 2 report point estimates without error bars. The paper's claims about convergence plateaus and relative performance rankings should be interpreted with appropriate caution given the single-run nature of the experiments.

Main Quantitative Results

Reference-Guided Fine-Tuning Improves RL Training (Figure 2)

The central claim of the paper is tested by comparing DAPO-based RL training starting from the ReGFT checkpoint versus starting from the raw (un-fine-tuned) checkpoint. The results are presented in Figure 2, which plots evaluation accuracy on AIME 2024, AIME 2025, and Beyond-AIME over the course of RL training steps.

Headline finding: Models initialized with ReGFT consistently outperform those initialized from the raw checkpoint across all three benchmarks throughout the entire RL training process (Figure 2). The two curves (ReGFT + DAPO in blue, raw DAPO in orange) are separated at every visible training step, with ReGFT maintaining a clear margin from the earliest steps through convergence.

Magnitude of the advantage at convergence. The paper does not report exact numerical values for the final accuracy in the text — the results are presented only as line plots in Figure 2. From visual inspection of the figures:

  • On AIME 2024 (Figure 2a): ReGFT + DAPO converges to what appears to be approximately 55-60% accuracy, while raw DAPO reaches approximately 45-50%. The gap is roughly 10 percentage points.
  • On AIME 2025 (Figure 2b): Both curves appear lower, with ReGFT + DAPO reaching approximately 40-45% and raw DAPO around 30-35%, maintaining a similar absolute gap.
  • On Beyond-AIME (Figure 2c): The advantage is most pronounced — ReGFT + DAPO converges to roughly 25-30% while raw DAPO reaches only about 10-15%, representing a near-doubling of final accuracy.

Convergence speed. The paper states that ReGFT-initialized models "exhibit substantially faster improvement during the early and middle stages of training" (Section 4.2.1). This is visible in the steeper initial slope of the ReGFT curves in Figure 2 — the ReGFT model reaches a given accuracy threshold (say, 30% on AIME 2024) in substantially fewer training steps than raw DAPO. The paper attributes this to "more informative learning signals" being available from the start of RL, since ReGFT increases the fraction of problems for which correct trajectories are sampled.

Interpretation of the plateau behavior. Both ReGFT + DAPO and raw DAPO appear to reach approximate plateaus in Figure 2, but the plateaus are at different heights. This is the key evidence for the paper's claim that ReGFT raises the final performance ceiling rather than merely accelerating convergence. If ReGFT only helped with early training speed, the raw DAPO curve would eventually catch up. It does not — the separation persists.

Why this matters mechanistically. The persistent gap supports the interpretation that ReGFT expands the set of solvable problems rather than just making existing solutions more accessible. If the raw model cannot generate correct trajectories on certain hard problems, RL will never receive gradient signals on those problems regardless of how long training continues. ReGFT creates those initial trajectories through reference guidance, enabling RL to learn on problems it would otherwise ignore entirely. The plateau separation is the empirical signature of this capability-expansion effect.

ReFT vs. ReGFT: The Contribution of Reference Guidance Beyond Self-Generated Trajectories (Figure 3, Table 1)

To isolate the specific contribution of reference-guided demonstrations, the paper compares DAPO RL initialized from ReFT (self-generated correct trajectories only) versus ReGFT (self-generated + reference-guided trajectories). Figure 3 presents the RL training curves for this comparison across all three benchmarks.

Headline finding: ReFT + DAPO accelerates early-stage RL compared to raw DAPO, but fails to raise the final convergence plateau, and in one case underperforms raw DAPO. ReGFT + DAPO provides both early acceleration and a higher final plateau (Section 4.2.2).

Detailed pattern across benchmarks:

  • AIME 2024 (Figure 3a): ReFT + DAPO (green) starts with a clear early advantage over raw DAPO (orange) but the curves converge by the end of training, with ReFT showing at best a marginal improvement at the final step. ReGFT + DAPO (blue) tracks ReFT early but continues improving, ending clearly above both. This suggests ReFT provides primarily a pass@1 boost that raw DAPO eventually matches through exploration, while ReGFT provides new capabilities that raw DAPO cannot match.
  • AIME 2025 (Figure 3b): The pattern is similar but more compressed — all curves are lower, and the ReFT advantage over raw DAPO is narrower. ReGFT + DAPO maintains a clearer separation.
  • Beyond-AIME (Figure 3c): This is the most diagnostically revealing comparison. ReFT + DAPO actually underperforms raw DAPO at convergence — the green curve sits below the orange curve at the final training steps, despite an early advantage. ReGFT + DAPO, by contrast, substantially outperforms both. The paper interprets this as ReFT "primarily reinforces the model's existing successful trajectories, which can bias exploration and limit the diversity of solutions" (Section 4.2.4). On the hardest benchmark (Beyond-AIME), this reduced exploration diversity hurts more than the pass@1 boost helps — the model becomes over-confident in a narrow set of strategies that work on easier problems but fail on harder ones. ReGFT, by introducing reference-guided trajectories that the model could not discover on its own, avoids this exploration collapse.

Supervised-only performance (Table 1). Before RL, the ReGFT checkpoint achieves higher pass@64 on the OmniMath training set than the ReFT checkpoint, confirming that reference-guided sampling successfully generates correct trajectories for problems where self-generated sampling fails. However, on the evaluation benchmarks (AIME 2024, AIME 2025, Beyond-AIME), the supervised-only ReGFT and ReFT checkpoints show "largely comparable" performance (Section 4.3.2). This is an important finding: the benefit of reference guidance is not primarily from the SFT phase alone — it is from the interaction with subsequent RL. ReGFT provides correct trajectories on hard training problems that ReFT misses; these trajectories become RL training signals that teach the model to solve similar problems, and this learning transfers to the evaluation benchmarks even though the evaluation problems are out-of-distribution.

Interpretation of the ReFT divergence on Beyond-AIME. The fact that ReFT underperforms raw DAPO on the hardest benchmark is a negative result with significant implications. It suggests that fine-tuning on self-generated correct trajectories can be harmful if those trajectories represent a biased subset of possible solution strategies. The model overfits to the strategies it happens to sample during the ReFT data collection phase, and RL — which normally explores broadly — starts from a distribution that has already collapsed toward those strategies. This is a cautionary finding for the broader practice of SFT-before-RL: the quality and diversity of SFT data matter, and data from the model's own outputs can be counterproductive if it is not sufficiently diverse.

Necessity of Model-Derived Reasoning: Direct SFT on References Fails (Figure 4)

The paper tests whether the model-generated component of ReGFT trajectories is genuinely necessary, or whether direct exposure to human reference solutions would suffice. The ablation compares ReGFT + DAPO against a baseline where the model is directly fine-tuned on raw human-written reference solutions from OmniMath, then trained with DAPO (Figure 4).

Headline finding: Directly fine-tuning on human reference solutions "leads to substantially weaker performance, both during supervised training and after reinforcement learning" (Section 4.2.3). The direct-SFT curve in Figure 4 sits consistently and substantially below both ReGFT + DAPO and raw DAPO across all three benchmarks.

Magnitude of the gap. From visual inspection of Figure 4, the direct-SFT + DAPO model performs roughly 15-20 percentage points worse than ReGFT + DAPO at convergence on AIME 2024 (Figure 4a), and the gap is comparably large on AIME 2025 and Beyond-AIME (Figures 4b-c). On Beyond-AIME, direct-SFT + DAPO appears to perform roughly at or below the raw DAPO level — meaning the pre-RL SFT on references provided essentially zero benefit, or was slightly harmful.

Why the direct-SFT baseline arguably starts lower than raw DAPO. The paper does not extensively discuss why direct SFT on references underperforms even the raw model, but the mechanism from Section 3 provides the explanation: the model is fine-tuned to produce tokens that lie outside its generation distribution. This degrades the model's ability to generate coherent, distributionally-aligned reasoning, which hurts the quality of trajectories sampled during RL — even on easy problems where the raw model would have succeeded. The SFT on references does not just fail to help on hard problems; it actively damages the model's existing capabilities by pulling its generation distribution toward a foreign style. This is consistent with the theoretical framing of the paper: knowledge transfer from human demonstrations is only effective when the model translates that knowledge into its own reasoning language.

Diagnostic significance. This ablation is the paper's strongest evidence for its core methodological claim: that the specific mechanism of reference-guided sampling — having the model generate reasoning under partial guidance rather than imitating reference solutions — is essential. Simply having access to correct reference solutions is not enough; the model must produce its own trajectories for SFT to be effective. This rules out a simple alternative hypothesis (that any exposure to correct solutions helps) and validates the distributional-alignment argument.

Inference-Time Scaling: Pass@k Analysis (Figure 5)

Section 4.2.4 evaluates how different training methods affect the model's ability to benefit from increased test-time compute. The paper samples N = 1,024 independent trajectories per problem and estimates pass@k for k ranging from 1 to 1,024.

Headline finding: "ReGFT + DAPO exhibits the strongest and most consistent scaling performance" across all benchmarks, with particular advantages on Beyond-AIME (Section 4.2.4). The key diagnostic insight is that ReGFT maintains a clear performance margin at high k, while ReFT's advantage over raw DAPO diminishes as k increases.

Detailed pass@k behavior across methods:

  • ReGFT + DAPO (blue solid lines): The curve sits highest across essentially all k on all three benchmarks. On AIME 2024 (Figure 5a), ReGFT + DAPO reaches roughly 70-75% at k = 1,024. On AIME 2025 (Figure 5b), approximately 55-60%. On Beyond-AIME (Figure 5c), approximately 40%. Critically, the separation between ReGFT and the other methods widens or remains stable at high k, rather than narrowing.
  • ReFT + DAPO (green solid lines): Improves over raw DAPO at low k (pass@1 benefit), but the gap shrinks at higher k. On AIME 2024 (Figure 5a), ReFT + DAPO and raw DAPO nearly converge at k = 1,024. This supports the interpretation that ReFT increases the probability of the model's existing correct solutions (boosting pass@1) but does not substantially increase the diversity of correct solutions in the distribution — additional sampling at inference time discovers the same correct solutions that raw DAPO would eventually discover with enough attempts.
  • Raw DAPO (orange solid lines): Middle performance, with the slowest pass@k growth on the hardest benchmarks.
  • Raw model (dashed lines): Substantially below all RL-trained models, confirming that RL provides significant gains across all sampling budgets. The dashed line's shape indicates the base model's pass@k scaling behavior, which improves with k but saturates earlier.

The contrast with prior pass@k saturation findings. The paper explicitly notes that prior work (Yue et al., 2025) reported pass@k saturation — RL-trained models failing to improve with increased test-time compute. The paper attributes its different finding to "the difference of the base model, as well as we have a larger sampling budget during RL" (Section 4.2.4). Specifically, using 64 samples per prompt during RL training (rather than fewer) may teach the model to explore more broadly, leading to better coverage of the solution space at inference time. However, the paper does not isolate whether it is the model family (Qwen3 vs. the models used in Yue et al.) or the sampling budget that explains the different outcome.

Interpretation of ReGFT's pass@k advantage. The sustained separation at high k is the paper's strongest evidence for the capability expansion versus capability consolidation distinction. If ReGFT had only increased pass@1 (making the most-likely correct solution more probable), the curves would converge at high k — with enough samples, raw DAPO would find those solutions too. The fact that they do not converge means ReGFT has enabled the model to generate correct solutions that are genuinely inaccessible to the raw DAPO model, even in 1,024 attempts. These are problems the raw model simply cannot solve, regardless of sampling budget — the probability mass on correct trajectories is effectively zero. ReGFT shifted that mass to be non-zero.

Effect of Sampling Scale During RL (Table 2)

Table 2 presents an ablation comparing RL training with 16 versus 64 sampled responses per prompt, across all three initialization methods (raw, ReFT, ReGFT). The goal is to test whether increased exploration scale during RL can substitute for the improved initialization that ReGFT provides.

Headline finding: Increasing sampling from 16 to 64 improves performance across all methods — "scaling alone is not sufficient" (Section 4.3.3). Even with 64 samples per prompt, models initialized with ReGFT outperform those initialized with ReFT or raw.

Specific comparisons from Table 2:

  • Under DAPO (raw initialization), moving from 16 to 64 samples provides improvements across all benchmarks, consistent with the expectation that broader exploration helps overcome reward sparsity.
  • Under ReFT + DAPO, the same scaling provides improvements, and ReFT + DAPO at 16 samples often outperforms raw DAPO at 64 samples — indicating that improving initialization can be more sample-efficient than increasing RL exploration.
  • Under ReGFT + DAPO, the model achieves the highest accuracy in every configuration (both 16 and 64 samples, across all three benchmarks). At 64 samples, ReGFT + DAPO achieves the strongest results overall.

What this tells us about complementarity. The paper's stated conclusion is that "improving the model's initial competence and increasing exploration scale are complementary and jointly necessary for robust gains" (Section 4.3.3). This is supported by the data: the best results come from combining ReGFT initialization with high sampling budgets. Neither alone is optimal. ReGFT without sufficient RL exploration (16 samples) leaves some capability on the table that broader exploration (64 samples) can access. Raw DAPO with broad exploration (64 samples) still falls short of ReGFT with moderate exploration (16 samples), indicating that the initialization quality is the more impactful of the two factors for these benchmarks.

Reference-Guided Sampling Expands the Solvable Problem Set (Section 4.3.1)

While not a comparative result against baselines, the paper reports a critical measurement of the reference-guided sampling mechanism itself: how many problems that were previously unsolvable become solvable when the model is given partial reference guidance.

Key numbers from Section 4.3.1:

  • Standard sampling with 64 attempts: 68.58% of OmniMath's 4,428 problems are solvable (at least one correct trajectory).
  • Reference-guided sampling with 64 attempts: 70.82% are solvable.
  • 5.85% of problems are solved only under reference guidance and never under standard sampling. These are the problems ReGFT targets — problems where the base model's unguided pass@64 = 0, but guided pass@64 > 0.
  • Conversely, 3.61% of problems are solved only under standard sampling — reference guidance actually hurts on some problems, perhaps because the truncated reference confuses the model or provides an unhelpful solution direction.

Interpretation. The 5.85% figure (approximately 259 problems out of 4,428) represents the maximum potential scope of ReGFT's benefit over ReFT: these are problems where ReFT would have zero training data (no self-generated correct trajectories), but ReGFT can construct training data via reference guidance. This provides a rough upper bound on how many "new" problems ReGFT can address. The fact that ReGFT + DAPO substantially outperforms ReFT + DAPO on the evaluation benchmarks (which are different problems than those in OmniMath) suggests that learning on these ~259 problems transfers to improved reasoning on unseen problems — the model is learning generalizable reasoning patterns, not just memorizing reference-guided solutions.

Ablation Studies and Robustness Checks

Direct SFT on human reference solutions vs. ReGFT (Figure 4): Fine-tuning directly on raw human-written reference solutions, without requiring the model to generate its own reasoning, leads to substantially worse performance both before and after RL compared to ReGFT. This is the paper's primary ablation isolating the importance of model-derived reasoning. The gap is large (roughly 15-20 percentage points at convergence on AIME 2024) and consistent across all three benchmarks, establishing that exposure to correct human reasoning alone is not sufficient — the model must translate that reasoning into its own generation style for SFT to be effective.

ReFT vs. ReGFT (Figures 3, 5; Table 1): Removing the reference-guided component and training only on self-generated correct trajectories (ReFT) accelerates early RL but fails to raise the final convergence plateau, and underperforms raw DAPO on Beyond-AIME. This isolates the specific contribution of reference-guided trajectories beyond what the model can discover on its own. The critical negative result is on Beyond-AIME (Figure 3c), where ReFT + DAPO falls below raw DAPO — suggesting that SFT on narrow self-generated data can bias the model toward limited strategies and harm performance on the hardest problems. ReGFT's inclusion of reference-guided trajectories mitigates this bias by introducing solution patterns the model would not discover independently.

Sampling scale during RL: 16 vs. 64 samples per prompt (Table 2): Increasing the number of sampled responses per prompt from 16 to 64 improves performance across all initialization methods (raw, ReFT, ReGFT), confirming that broader exploration helps mitigate reward sparsity. However, the improvement from scaling samples does not close the gap between initialization methods: ReGFT + DAPO at 16 samples outperforms raw DAPO at 64 samples on some benchmarks, and ReGFT + DAPO at 64 samples achieves the strongest results overall. This establishes that initialization quality and exploration scale are complementary rather than substitutable — improving one does not remove the need for the other.

Hard problem definition threshold (Section 4.3): The paper defines hard problems as those with less than 25% accuracy under the base model with 16 samples, and trains ReGFT only on these problems. The paper does not ablate this threshold (e.g., comparing 10%, 25%, 50% cutoffs, or training on all problems). This is a notable missing ablation: the threshold value directly determines how many problems enter ReGFT training and which problems they are, and it is unclear whether the results are sensitive to this choice. The paper's justification (avoiding overfitting on easy problems, focusing SFT on reward-sparse cases) is reasonable but empirically unverified.

Reference hint length: 80/20 sentence split (Section 4.3.1): The paper uses approximately the first 80% of sentences from the human reference solution as the hint, withholding the final 20% (which typically contains the answer). The paper does not ablate this ratio (e.g., 50/50, 90/10, full reference). The qualitative observation that "the model almost always derives its own reasoning independently even when exposed to the full reference solution" (Section 4.3.1) suggests that the exact split may not be critical — the model naturally generates its own reasoning style regardless — but this is not tested systematically. The paper also does not investigate whether providing less guidance (e.g., only the first 50% or only the key mathematical insight) would be equally effective, which would be informative about how much structural information is needed.

Choice of RL algorithm (Section 3, Appendix 6): The paper uses DAPO specifically and does not compare against other RL algorithms (e.g., standard PPO, GRPO without decoupled clipping, or simpler REINFORCE variants). The claim that "ReGFT is orthogonal to advances in RL algorithms" is supported indirectly — ReGFT improves performance on top of a strong RL algorithm — but is not directly tested by showing similar gains with a different RL method. This leaves open the possibility that ReGFT's benefits are specific to DAPO's optimization dynamics (e.g., the decoupled clipping might interact with ReGFT's trajectory distribution in ways that standard PPO would not).

Training data composition ratio (Section 3): The paper trains ReGFT on a mixture of self-generated correct trajectories and reference-guided correct trajectories, but does not specify the mixing ratio or ablate different ratios. If reference-guided trajectories are substantially outnumbered by self-generated trajectories (because reference-guided sampling only adds 5.85% additional solvable problems, per Section 4.3.1), the effective training signal from the reference-guided data might be diluted. The paper does not report whether weighting reference-guided trajectories more heavily (e.g., via oversampling) would improve results.

Supervised-only evaluation (Table 1): The paper reports pre-RL performance for raw, ReFT, and ReGFT checkpoints, providing a baseline for how much each SFT stage contributes before RL. ReGFT achieves higher pass@64 on the OmniMath training set than ReFT, confirming that reference guidance successfully generates additional correct trajectories. However, on the evaluation benchmarks, ReGFT and ReFT show comparable performance, indicating that the SFT-only benefit does not directly transfer to out-of-distribution problems — the transfer occurs only after RL generalizes the training-set improvements. This is an important nuance: ReGFT's value is realized primarily through its interaction with RL, not as a standalone SFT method.

Critical Assessment

Claim 1: ReGFT Consistently Improves RL Performance Over Raw Initialization

The experiments do demonstrate this, but with important caveats about the strength of the evidence. Figure 2 shows ReGFT + DAPO outperforming raw DAPO across all three benchmarks at every visible training step. The separation is clear and appears substantial. The supporting evidence from Table 2 (sampling scale ablation) shows the advantage persists at both 16 and 64 samples per prompt, and the pass@k analysis (Figure 5) shows the advantage extends across all sampling budgets.

However, the evidence would be substantially stronger with:

  • Multiple random seeds. RL training is known to be high-variance, and Figures 2-4 show single runs. Without error bars or seed averages, we cannot tell whether the observed separation between ReGFT and raw DAPO is larger than run-to-run variance. It is possible that raw DAPO with a different random seed would occasionally match or exceed the ReGFT curve shown. The paper's conclusions about "consistent" outperformance and convergence plateaus are qualitative judgments from single trajectories.
  • Significance testing. The test sets are small (AIME has ~30 problems per year; Beyond-AIME has 100). A 10-percentage-point gap on AIME 2024 represents approximately 3 more correct answers out of 30 — substantively meaningful but potentially within sampling variance. The paper does not report whether differences are statistically significant.
  • More training steps. The curves in Figure 2 appear to plateau, but it is unclear whether training was stopped at the plateau or whether longer training might eventually close the gap between ReGFT and raw DAPO. The paper does not specify a stopping criterion or report whether training was run until accuracy stabilized.

Claim 2: ReGFT Raises the Final Performance Plateau (Capability Expansion), Not Just Convergence Speed (Capability Consolidation)

This claim is supported by the evidence presented, but the distinction between plateau-raising and convergence-acceleration is harder to establish definitively than the paper suggests. The key evidence is in Figure 3:

  • On AIME 2024 and AIME 2025, the ReFT and raw DAPO curves do appear to converge by the end of training, while ReGFT maintains separation. This supports the plateau-raising interpretation.
  • On Beyond-AIME, ReFT actually drops below raw DAPO, which is strong evidence that ReFT's effect is qualitatively different from ReGFT's.
  • The pass@k analysis (Figure 5) provides converging evidence: ReFT's advantage over raw DAPO shrinks at high k (consolidation — more samples eventually find the same solutions), while ReGFT's advantage persists (expansion — new solutions exist).

However, the following weaken the claim:

  • Single training runs make it impossible to distinguish a genuine plateau gap from run-to-run variance in final performance. If raw DAPO is noisier than ReGFT + DAPO, a single unlucky run could produce an apparent plateau gap that would disappear with averaging.
  • The paper does not define "convergence" operationally — no formal stopping criterion is reported. It is possible that training was simply terminated at a step where the visual separation looked favorable, and that continuing training would show the curves converging.
  • The capability-expansion interpretation relies on the assumption that ReGFT teaches the model to solve problems it literally could not solve before, but the paper does not directly measure this on the evaluation benchmarks. It would be informative to see, for example, which specific AIME problems are solved by ReGFT + DAPO but never solved by raw DAPO (even with 1,024 attempts in the pass@k evaluation). This would provide direct evidence of capability expansion on the evaluation set, rather than inferring it from pass@k curves.

Claim 3: Reference-Guided Demonstrations Are Necessary; Direct SFT on References Fails

This is the paper's most robustly supported claim. Figure 4 provides a clean, decisive comparison: direct-SFT + DAPO underperforms ReGFT + DAPO by a large margin (15-20 percentage points at convergence by visual estimate) and even underperforms raw DAPO on some benchmarks. The ablation is well-designed — the only difference between the conditions is whether the model generates its own reasoning (ReGFT) or directly imitates human solutions (direct SFT). The result is unambiguous and consistent across all three benchmarks.

The one thing that would strengthen this claim further: the paper does not report what fraction of direct-SFT training trajectories are verifiably correct. If the model fine-tuned on references produces outputs that fail the verifier (because the fine-tuning degraded its answer formatting or reasoning coherence), then the direct-SFT checkpoint might simply be a worse starting point for any kind of training, not specifically for RL. A control condition showing direct-SFT performance without subsequent RL (comparable to the ReGFT row in Table 1) would help isolate whether the degradation occurs during SFT alone or during the interaction of SFT with RL.

Claim 4: ReGFT Improves Inference-Time Scaling (Pass@k)

Supported with important nuance. Figure 5 clearly shows ReGFT + DAPO maintaining pass@k advantages at k = 1,024, which is strong evidence that ReGFT expands the solution coverage rather than just boosting pass@1. However, the paper's claim that this represents "improved coverage of the solution space" (Section 4.2.4) would be stronger with a direct diversity measure (e.g., number of distinct correct solutions per problem, or semantic diversity of correct reasoning traces) rather than being inferred from pass@k alone. It is theoretically possible for pass@k advantages at high k to arise from a single additional correct solution with moderate probability, rather than from broad solution-space coverage. Without a diversity metric, the pass@k curves are suggestive but not conclusive about the mechanism.

The contrast with prior pass@k saturation findings (Yue et al., 2025) is interesting but not fully explained. The paper attributes the difference to model family and sampling budget, but does not run the Yue et al. experimental configuration (same model, lower sampling budget) to verify this attribution. Controlled experiments varying only the sampling budget during RL, while holding the model and other hyperparameters constant, would isolate whether sampling budget is the key variable.

Potential Weaknesses in the Experimental Design

Single model family, single scale. All experiments use Qwen3-4B-Instruct. The paper justifies this choice (stability, strong reasoning), but the generalizability of the findings to other model families (Llama, DeepSeek, Gemma), other scales (1B, 8B, 70B), and other training paradigms (base models vs. instruct-tuned) is entirely untested. The paper's claims implicitly assume that the distributional-alignment argument applies universally, but the strength of that effect may vary with model scale — larger models may be better at imitating human reference solutions directly, potentially reducing ReGFT's relative advantage.

Single training dataset. All fine-tuning uses OmniMath. While the evaluation benchmarks (AIME, Beyond-AIME) are held-out, they are drawn from the same domain (competition mathematics). The paper does not test whether ReGFT's benefits transfer to non-math reasoning domains, or even to math problems from a substantially different distribution (e.g., theorem proving, applied math word problems).

No comparison to mid-RL augmentation methods. The paper positions ReGFT as an alternative to methods that inject reference guidance during RL (BREAD, Questa). This is a key conceptual distinction, but the paper does not empirically compare ReGFT + DAPO against, for example, DAPO with BREAD-style prefix anchoring. Without such a comparison, the claim that pre-RL guidance is superior to mid-RL guidance remains a theoretical argument rather than an empirical finding.

Difficulty estimation cost is unaccounted for. Defining the hard problem set requires sampling 16 trajectories per problem (4,428 × 16 = 70,848 generations) and evaluating them. This cost is incurred before ReGFT training begins and is not included in any efficiency calculation. While this cost is small relative to RL training (which generates 8,192 trajectories per rollout step), it is a prerequisite for ReGFT that raw DAPO does not require. If the goal is to argue that ReGFT is more efficient than scaling RL samples, this fixed cost should be amortized into the comparison.

No exploration of reference quality. The paper assumes reference solutions are high-quality and correct, which is reasonable for OmniMath (verified Olympiad solutions). However, in domains where reference quality varies, low-quality references used as hints could actively mislead the model. The paper does not include an ablation with degraded or noisy references to test robustness to reference quality.

Missing direct measurement of reward sparsity reduction. The paper argues that ReGFT works by reducing reward sparsity — increasing the fraction of RL batches that contain at least one correct trajectory. This is a directly measurable quantity (e.g., plot the fraction of trajectory groups that pass DAPO's dynamic sampling filter over the course of RL training) but is not reported. The evidence for reward sparsity reduction is indirect (faster convergence, higher plateau) rather than directly observed. Including such a measurement would transform the mechanistic claim from inferential to demonstrated.

6. Limitations and Trade-offs

6.1 Single Model Family, Single Scale, Single Domain

The assumption or constraint. All experiments in the paper use exactly one base model — Qwen3-4B-2507-Instruct — at a single parameter scale (4B) and on a single training dataset (OmniMath, consisting of Olympiad-level mathematics problems). The evaluation benchmarks (AIME 2024, AIME 2025, Beyond-AIME) are all drawn from the same domain of competition mathematics. The paper acknowledges none of these as limitations explicitly; the model choice is justified positively ("provides strong reasoning performance and reliable instruction following ability," Section 4.1) and the benchmark choice is justified by the severity of reward sparsity in Olympiad-level math, but the scope boundaries are not discussed.

The consequence. The central claim of the paper — that reference-guided fine-tuning before RL expands a model's capability frontier by translating human knowledge into model-aligned reasoning — rests on a mechanistic argument about distributional alignment between human-written and model-generated reasoning. This argument is domain- and model-agnostic in its formulation, but is validated only on a single model family at a single scale on a single reasoning domain. Several forms of generalization failure are plausible and unexamined:

  • Across model families: Qwen3 may have particular properties — its pretraining data mixture, its instruction-tuning recipe, its tokenizer, its architectural inductive biases — that make the gap between human reference reasoning and model-generated reasoning especially wide or narrow. Other model families (Llama, DeepSeek, Gemma) might be better at directly imitating human reference solutions, potentially narrowing or eliminating ReGFT's advantage over direct SFT. Conversely, models that are worse at instruction following might fail to benefit from partial reference hints at all (the model needs to understand and incorporate the hint, which is a meta-reasoning capability not all models possess equally).
  • Across scales: The paper's 4B model sits in a regime of nontrivial but far-from-saturated capability on Olympiad math (68.58% of training problems solvable at pass@64). At smaller scales (1B, 0.5B), the base model may be so weak that even reference-guided hints fail to elicit correct trajectories — the model would lack the fundamental mathematical knowledge to execute the hinted strategy — rendering ReGFT ineffective. At larger scales (70B, 400B), the base model may already have high pass rates on most problems, making the reward sparsity problem less severe and ReGFT's incremental benefit proportionally smaller. The paper provides no evidence about where on this scale spectrum ReGFT remains beneficial.
  • Across domains: Mathematical reasoning with verifiable answers is an unusually favorable setting for ReGFT. Reference solutions are typically structured, step-by-step, and encode clear high-level strategies that can be effectively summarized in an 80% sentence prefix. It is unclear whether reference-guided sampling would work for domains where reference solutions are less structured (e.g., open-ended essay writing), where correctness is non-binary (e.g., dialogue quality), or where the "strategy" is harder to convey without revealing the answer (e.g., code generation, where the first 80% of a reference implementation often contains the core algorithmic logic and the remaining 20% is boilerplate).

What evidence exists in the paper. None. The paper contains no cross-model, cross-scale, or cross-domain experiments. All results are from Qwen3-4B on OmniMath → AIME/Beyond-AIME. The paper makes no claim about generalizability beyond this setting, but the Introduction and Approach sections are written in general language that invites readers to assume broader applicability.

Mitigation status. The paper does not attempt to address this limitation. No experiments with alternative models, scales, or domains are reported, and no discussion of scope boundaries is included. Future work on replicating ReGFT with different model families (particularly models with known sensitivity to distributional shift in SFT, as noted for Qwen2.5 in Section 4.1) would test whether the distributional-alignment mechanism is genuinely general or Qwen3-specific.

6.2 The Hard Problem Threshold Is Unexplored and Potentially Brittle

The assumption or constraint. ReGFT trains only on problems where the base model achieves less than 25% accuracy with 16 samples (Section 3). This threshold serves as both a filter (excluding easy problems to prevent overfitting) and a targeting mechanism (focusing SFT capacity on problems where RL would otherwise receive zero reward). The paper presents this as an implementation detail — the threshold value is stated, not derived, and no ablation over different threshold values is performed.

The consequence. The choice of threshold directly determines three critical properties of the ReGFT training procedure:

  1. The size and composition of the training set. Varying the threshold from, say, 10% to 50% changes how many problems enter ReGFT — and which problems. A higher threshold (e.g., 50%) includes more problems where the model already has some competence, potentially diluting the reference-guided signal with problems RL would handle on its own and increasing overfitting risk. A lower threshold (e.g., 10%) restricts ReGFT to only the very hardest problems, which may be so difficult that even reference-guided sampling fails to produce correct trajectories, leaving ReGFT with little training data. The optimal threshold depends on the base model's accuracy distribution across problems, which varies by model and dataset.
  2. The balance between self-generated and reference-guided trajectories in the training mixture. On problems just below the threshold (e.g., 20-25% accuracy), the base model occasionally generates correct trajectories under standard sampling, so both self-generated and reference-guided data exist. On problems far below the threshold (e.g., 0-5% accuracy), only reference-guided trajectories (if any) are available. The threshold controls the ratio of problems with both data sources versus reference-guided-only, which in turn controls how much the model's training distribution is shifted toward reference-influenced solutions.
  3. The upper bound on ReGFT's potential benefit. Section 4.3.1 reports that reference-guided sampling adds 5.85% of OmniMath problems to the solvable set (relative to standard sampling). But this is measured across all 4,428 problems. If many of those 5.85% are already above the 25% threshold (i.e., the model solves them under standard sampling, just not in the particular 64-sample run), they would be excluded from ReGFT training, reducing the effective training data gain. The paper does not report how many of the 5.85% fall below the 25% threshold.

Without understanding the sensitivity of ReGFT's performance to the threshold, a practitioner cannot reliably set this hyperparameter for a new model or dataset — and guessing wrong could mean either wasting SFT compute on problems RL would handle or failing to address the reward sparsity on problems that need it most.

What evidence exists in the paper. Very little. Section 4.3.1 reports the 5.85% increment from reference-guided sampling, and Table 1 reports ReGFT's improved pass@64 on the training set relative to ReFT. But neither of these is an ablation over different thresholds. The paper does not report performance comparisons for ReGFT with thresholds of 10%, 25%, 50%, or "all problems." The sensitivity of downstream RL performance to this choice is entirely unexamined.

Mitigation status. The paper does not acknowledge this as a limitation, and no threshold ablation is performed or suggested for future work. A practitioner adopting ReGFT would need to either replicate the paper's exact threshold (25% with 16 samples) and hope it transfers to their setting, or conduct their own threshold sweep — adding to the already substantial computational cost of the method (see Limitation 6.3).

6.3 The Computational Overhead of Difficulty Estimation and Reference-Guided Sampling Is Substantial and Not Amortized

The assumption or constraint. Before ReGFT training can begin, the method requires: (a) sampling 16 trajectories per problem across the full training set to estimate per-problem accuracy and identify hard problems (the threshold step), and (b) for hard problems, performing reference-guided sampling (which requires additional generations beyond the standard sampling already done for ReFT-style data collection) to produce correct reference-guided trajectories. The paper does not account for either of these costs in any efficiency comparison against raw DAPO or ReFT + DAPO. Section 3 mentions the 16-sample threshold and the reference-guided sampling step but does not quantify the FLOPs or wall-clock time they consume.

The consequence. The headline results in Figures 2-4 and the claims about improved RL efficiency mask a significant pre-RL computational overhead that varies by problem:

  • Difficulty estimation cost: For OmniMath's 4,428 problems, generating 16 trajectories per problem requires 70,848 complete solution generations. At a maximum of 16,384 tokens per trajectory, this is up to ~1.16 billion tokens generated — comparable to a non-trivial number of RL rollout steps (each RL step generates 8,192 trajectories, so the difficulty estimation cost is about 8.6 RL steps' worth of generation). This cost is incurred before any training benefit is realized and is entirely wasted if ReGFT is subsequently abandoned.
  • Reference-guided sampling cost: For the hard problems (those below the 25% threshold), the paper generates additional reference-guided trajectories. The exact number is not reported, but Section 4.3.1's measurement that reference-guided sampling raises the solvable fraction from 68.58% to 70.82% (with 64 samples per problem) implies that reference-guided sampling was run at scale — potentially 64 samples × 4,428 problems = 283,392 additional generations, though this may overlap with the difficulty estimation samples if those were reused. Even conservatively estimating 64 guided samples on the ~31.42% of problems that are hard (~1,390 problems) gives roughly 89,000 additional generations.
  • The amortization problem: These costs are "fixed costs" — they must be paid once before RL begins, regardless of how many RL steps are subsequently run. If RL is run for many steps, the fixed cost amortizes to a small fraction of total compute. If RL is run for few steps (or if the practitioner wants to try ReGFT, evaluate results, and decide whether to continue), the overhead can dominate the total budget. The paper's RL runs continue for what appears to be hundreds of steps (exact number not stated), which may make the overhead proportionally small, but shorter RL schedules would see a proportionally larger efficiency penalty.

A fair efficiency comparison against raw DAPO or ReFT + DAPO would amortize this overhead into the RL step count — e.g., "at the point where ReGFT + DAPO reaches accuracy X, the total compute spent (difficulty estimation + ReGFT SFT + RL steps) is Y, while raw DAPO reached accuracy X at compute Z." The paper provides no such analysis, making the headline claims about efficiency and convergence speed incomplete.

What evidence exists in the paper. The paper partially acknowledges the difficulty estimation cost in Section 4.3.1 by reporting the 68.58% baseline and 5.85% reference-guided increment, but does not integrate these numbers into any total-cost comparison. The sampling budgets for data collection (64 per problem, per Section 4.3.1) are mentioned but not summed or compared to RL step costs. The Appendix 6 reports RL rollout costs (8,192 trajectories per step) in enough detail that a reader could reconstruct approximate total costs, but the paper itself draws no conclusions from such a reconstruction.

Mitigation status. Not addressed. The paper does not discuss this overhead as a trade-off, does not amortize it into convergence comparisons, and does not propose methods for reducing it (e.g., using fewer samples for difficulty estimation, sharing samples between difficulty estimation and training data collection, or using a lightweight difficulty predictor as an alternative). A practitioner implementing ReGFT would need to budget for this overhead independently and decide whether it is worth paying, without guidance from the paper.

6.4 The Reference Hint Construction (80/20 Split) Is Not Ablated; Optimal Hinting Strategy Is Unknown

The assumption or constraint. ReGFT's reference-guided sampling provides the model with the first approximately 80% of sentences from the human-written reference solution as a hint, withholding the remaining 20% (which typically contains the final answer) to prevent trivial answer copying (Section 4.3.1). The 80/20 split is presented as a design choice, not a tuned hyperparameter, and is not ablated. The paper notes qualitatively that "the model almost always derives its own reasoning independently even when exposed to the full reference solution," suggesting that the exact split may not be critical, but this is an observation based on model behavior, not a systematic evaluation.

The consequence. The hinting strategy is the core mechanism by which ReGFT translates human knowledge into model-aligned trajectories. Several aspects of this strategy could substantially affect ReGFT's effectiveness, and none are explored:

  • How much reference to provide (the split ratio). Providing more of the reference (e.g., 95%) gives the model more structural guidance and might increase the probability of generating a correct trajectory, especially on extremely hard problems where even a strategy hint is insufficient without detailed step-level guidance. However, more guidance risks the model simply continuing the reference rather than generating its own reasoning, producing trajectories that are closer to direct reference imitation — which Figure 4 shows is harmful. Providing less (e.g., 50%, or only the first key insight sentence) forces the model to do more independent reasoning, which may produce more distributionally aligned trajectories but at the cost of lower correctness rates (since the model may fail to reconstruct the intended solution strategy). The optimal split likely varies by problem difficulty and by the style of the reference solution.
  • What information to provide. The paper splits by sentence count (first 80%), which is a coarse proxy for semantic content. A 3-sentence reference where the key insight is in sentence 2 would provide that insight at both 66% and 100% splits, but a 10-sentence reference where the key insight spans sentences 4-6 would provide it at 60% but not 30%. The paper's split strategy ignores the semantic structure of the reference. Alternative strategies — providing the first N tokens, providing all sentences except the one containing the final answer, or having a language model extract and provide only the "strategy" sentences — could be more effective but are not explored.
  • The interaction with problem difficulty. The paper's hard problem set (those below 25% accuracy) spans a range of difficulties. For problems at 20-24% accuracy, the model can occasionally solve them unguided, and a light hint might be optimal. For problems at 0-5% accuracy, the model's mathematical toolkit may be insufficient to execute the hinted strategy even with guidance, and the hinting strategy is irrelevant because the model fails regardless (a capability ceiling issue). The single 80/20 split treats all hard problems identically, potentially over-guiding some and under-guiding others.
  • Robustness to reference quality. The paper uses OmniMath, which provides "verified reference solutions" (Section 4.1). In practice, reference solutions may contain errors, be suboptimal, or use notation or theorems unfamiliar to the model. Providing a flawed or confusing reference as a hint could actively mislead the model, causing it to generate incorrect trajectories even when it might have succeeded unguided (consistent with the 3.61% of problems that are solved only under standard sampling in Section 4.3.1). The paper does not test ReGFT's sensitivity to reference quality.

What evidence exists in the paper. Only the qualitative observation quoted above (Section 4.3.1) that the model generates its own reasoning even with full references. No ablation of the split ratio is performed. No analysis of how the split interacts with problem difficulty or reference characteristics is provided. The 3.61% regression rate (problems solved under standard sampling but not under reference-guided sampling) provides indirect evidence that reference guidance can sometimes hurt, but the paper does not analyze these cases to understand whether a different hinting strategy would avoid the regression.

Mitigation status. Not addressed. The paper treats the 80/20 split as a fixed design choice and does not discuss it as a hyperparameter requiring tuning. The observation about full-reference behavior is presented as reassuring but is not tested systematically. A practitioner seeking to apply ReGFT to a new domain would need to guess at an appropriate hinting strategy without guidance from ablation results.

6.5 No Direct Measurement of the Claimed Mechanism: Reward Sparsity Reduction

The assumption or constraint. The paper's central mechanistic claim is that ReGFT works by reducing reward sparsity during RL: "By increasing the likelihood that sampled trajectories contain correct ones on hard problems, ReGFT supplies RL with more informative learning signals" (Section 4.2.1), and "ReGFT checkpoints tend to be stronger and are more capable of generating trajectories that receive positive rewards on previously unsolvable questions" (Section 3). This is a directly measurable claim — one can track, during RL training, what fraction of trajectory groups contain at least one correct trajectory (or equivalently, what fraction of problems contribute useful gradients at each step). DAPO's dynamic sampling mechanism makes this particularly easy to measure: it explicitly filters groups to those with reward contrast, so the fraction of groups that pass the filter is a direct measure of reward sparsity.

The consequence. Without this measurement, the paper's causal narrative — ReGFT enables more positive trajectories → more gradient signals → faster convergence + higher plateau — remains inferential rather than demonstrated. Several alternative explanations for ReGFT's benefits are consistent with the reported results but would not involve reward sparsity reduction:

  • ReGFT might work primarily through better initialization of the policy, placing it in a more favorable region of parameter space where DAPO's optimization dynamics are more effective regardless of reward sparsity. The improved supervised accuracy on the training set (Table 1) is consistent with this.
  • ReGFT might increase the quality of correct trajectories (more thorough reasoning, better step-by-step structure) rather than their quantity, providing stronger gradient signals when correct trajectories do appear rather than making them appear more often.
  • ReGFT might improve the diversity of correct trajectories, preventing the exploration collapse that the paper hypothesizes for ReFT (Section 4.2.4), without necessarily changing the fraction of batches containing any correct trajectory.

Without a direct measurement, the paper cannot distinguish among these mechanisms or validate its core narrative. This matters practically because it changes what a practitioner would optimize: if reward sparsity reduction is the mechanism, the key lever is maximizing the number of previously-unsolvable problems that become solvable (driving the 5.85% from Section 4.3.1 as high as possible). If policy initialization quality is the mechanism, the key lever might be the quality of SFT training rather than the coverage of hard problems.

What evidence exists in the paper. All evidence is indirect:

  • Faster convergence (steeper initial slope in Figures 2-3) is consistent with reduced reward sparsity but also with better initialization.
  • Higher final plateau is consistent with capability expansion from newly-solvable problems but also with better optimization dynamics from a better starting point.
  • Table 1 shows higher pass@64 on the training set for ReGFT, confirming the pre-RL model solves more problems — but this is pre-RL, not during RL, and does not directly measure RL batch-level reward sparsity.
  • The 5.85% increment from reference-guided sampling (Section 4.3.1) shows that guided sampling creates correct trajectories on previously-unsolvable problems, but does not measure how many of those trajectories persist (i.e., how many of those problems become solvable by the ReGFT model without the reference hint, which is what matters for RL since RL does not provide hints).

Mitigation status. Not addressed. The paper does not report any reward-sparsity-related metric during RL training. A simple plot of "fraction of trajectory groups with at least one correct sample" over RL steps for ReGFT + DAPO vs. raw DAPO would directly test the mechanism and is feasible given the experimental setup (DAPO already computes this for dynamic sampling). The paper does not suggest this analysis for future work.

6.6 The "Hard Problem" Focus Leaves Easy and Medium Problems Unimproved; ReGFT-Derived Gains May Not Generalize Within-Task

The assumption or constraint. ReGFT trains only on hard problems, defined as those with less than 25% accuracy under the base model with 16 samples. Easy problems (those above the threshold) are entirely excluded from ReGFT's SFT stage. This design choice is justified as avoiding overfitting and focusing SFT capacity on reward-sparse cases (Section 3). The implicit assumption is that easy problems do not need pre-RL improvement — RL will handle them naturally — and that excluding them prevents the model from overfitting to problem-specific patterns on problems it already knows.

The consequence. This creates an asymmetry in the model's training: hard problems receive SFT attention (both self-generated and reference-guided trajectories), while easy problems receive none before RL. Several consequences follow:

  • The benefit of ReGFT on hard problems may come partly at the expense of performance on easy problems, if the SFT stage shifts the model's generation distribution in ways that degrade easy-problem reasoning. This would appear in aggregate accuracy as a trade-off (ReGFT gains on hard problems offset by losses on easy problems) that could mask the true magnitude of the hard-problem improvement or make ReGFT's benefit look smaller than it actually is on the subpopulation it targets. The paper's aggregate evaluation metrics (overall AIME/Beyond-AIME accuracy) cannot distinguish between these effects.
  • The model may not learn to transfer reasoning patterns from hard problems to structurally similar easy/medium problems. This is particularly relevant because mathematical reasoning often requires the same underlying skills applied at different levels of complexity — if ReGFT teaches the model a new algebraic manipulation technique on a hard problem, that technique might also improve performance on easy problems that use it, but only if the SFT stage includes easy problems where the technique can be practiced in simpler contexts. By excluding easy problems, ReGFT may miss this synergistic transfer.
  • The assumption that easy problems are "handled by RL" may not hold uniformly. Among problems above the 25% threshold, some may have accuracy of 30-40% — these are "easy" in the sense that they generate some positive rewards, but they are still sparse enough that RL would benefit from denser signals. Excluding them from ReGFT leaves potential improvement on the table.

What evidence exists in the paper. None that separates easy-problem and hard-problem performance. The paper reports only aggregate accuracy on the evaluation benchmarks (AIME 2024/2025, Beyond-AIME), without difficulty-stratified breakdowns. The ReGFT vs. ReFT comparison in Figure 3 shows ReGFT winning in aggregate, but if ReGFT improved hard-problem accuracy by +15 points while losing 3 points on easy problems relative to ReFT, the aggregate would show +12 points — still a win, but the easy-problem regression would be an important limitation for practitioners targeting balanced performance. No such stratified analysis is provided.

The paper's pass@k analysis (Figure 5) provides partial evidence: if ReGFT's advantage persists at high k, the gains are likely concentrated on problems that are hard enough that raw DAPO cannot solve them even with 1,024 attempts. But this is an inference about hard problems, not a direct measurement of per-difficulty-bin performance.

Mitigation status. Not addressed. The paper does not report difficulty-stratified evaluation results, does not analyze whether easy-problem performance degrades under ReGFT, and does not ablate the inclusion of easy/medium problems in the SFT stage. A practitioner concerned with maintaining strong performance across all difficulty levels would need to conduct their own stratified evaluation to determine whether ReGFT's hard-problem gains come with easy-problem trade-offs.

7. Implications and Future Directions

How This Work Changes the Landscape

ReGFT introduces a temporal displacement strategy for handling the hardest instances in RL-based reasoning. Instead of asking "How can we make RL work better on these zero-reward problems?" it asks "How can we make these problems stop being zero-reward before RL sees them?" This flips the locus of improvement from the RL algorithm itself to the data preconditioning stage — a shift that is conceptually modest but operationally significant.

The magnitude of the shift is best characterized as a reframing with diagnostic consequences, not a paradigm shift. The paper does not introduce a new RL algorithm, a new loss function, or a new model architecture. It introduces a specific prompting strategy — provide the first 80% of a human-written reference solution as context, require the model to generate its own complete reasoning trace, then fine-tune on the verified correct outputs — and demonstrates that this strategy, applied in a pre-RL SFT stage, produces a checkpoint that enables downstream RL to learn on problems it otherwise could not. The simplicity of the method is a strength: it can be dropped into any existing SFT → RL pipeline with no infrastructure changes, using data (reference solutions) that already ships with many math reasoning datasets.

Where the work does advance the field's conceptual apparatus is in its diagnostic distinction between capability consolidation and capability expansion, made empirically visible through the contrast between ReFT and ReGFT. ReFT (SFT on self-generated correct trajectories) accelerates early RL but does not raise the final plateau, and on the hardest benchmark (Beyond-AIME) it actually underperforms raw DAPO (Figure 3c). ReGFT raises the plateau. The pass@k analysis (Figure 5) reinforces this: ReFT's advantage over raw DAPO shrinks at high k — consistent with consolidation, where the model more reliably accesses what it already knows but does not know new things — while ReGFT maintains separation at k = 1,024 — consistent with expansion, where genuinely new correct solutions exist in the model's output distribution. This consolidation-vs-expansion axis provides a concrete diagnostic that future pre-RL and mid-RL interventions can be evaluated against: if your method only improves convergence speed and pass@1 but not the asymptotic plateau and high-k pass@k, it is likely consolidating rather than expanding.

The work also reconciles contradictory impulses in the literature about whether RL can teach genuinely new capabilities. On one side, Yue et al. (2025) and Wu et al. (2026) show that naïve RL exhibits pass@k saturation and may even underperform the base model on initially unsolvable problems — evidence that RL primarily amplifies existing behaviors. On the other side, the scaling-RL literature (BRoRL, ProRL, KnapsackRL) shows that broader exploration and longer training can push performance beyond what the base model achieves. ReGFT's results suggest a synthesis: RL can teach new capabilities, but only when the model's output distribution already contains those capabilities at non-zero (even if vanishingly small) probability. The base model on hard problems has effectively zero probability mass on correct trajectories — no feasible amount of RL sampling can surface what isn't there. ReGFT injects that probability mass through reference-guided SFT, converting unsolvable problems into solvable ones before RL begins. RL then reinforces and generalizes. In this framing, the "RL can't teach new things" finding is correct for a specific failure mode (zero-mass problems), and the "RL can teach new things" finding is correct when that failure mode is addressed (by ReGFT or similar pre-RL competence injection). The field's disagreement was partly a disagreement about the base model's initial competence distribution on the problems being studied — a variable ReGFT makes controllable.

Three research directions become more attractive in light of this work:

  • Pre-RL data generation as a first-class design space. The paper shows that how you construct pre-RL training data matters enormously — direct SFT on references fails (Figure 4), SFT on self-generated trajectories only consolidates (Figure 3), and SFT on reference-guided trajectories expands (Figure 2, 5). This opens a design space around prompting strategies, hint construction, and data mixture ratios that has received far less attention than RL algorithm design. The paper's distributional-alignment argument (the model must translate human knowledge into its own reasoning language) provides a theoretical principle to guide exploration of this space.

  • Mid-training quality as a determinant of RL outcomes. The paper connects to emerging evidence (Zhang et al., 2025, cited in Section 2) that stronger mid-training leads to more effective downstream RL. ReGFT operationalizes "stronger mid-training" as targeted competence injection on identified weak spots, rather than uniform SFT on all problems. This suggests a broader principle: mid-training should be diagnostic and targeted, not uniform, with resources concentrated on problems where the base model's competence is insufficient for RL to succeed.

  • Reference solutions as a translation resource, not an imitation target. The paper's core mechanism — using references as hints for model-generated reasoning rather than as direct training targets — reframes how the field should think about the vast repositories of human-written solutions, tutorials, and proofs that accompany many reasoning datasets. These are not SFT training data in raw form, but they are valuable as guidance for on-policy data generation. This is a more nuanced and potentially more productive relationship between human demonstrations and model training than either pure imitation (SFT on references) or pure self-play (RL without any external knowledge).

Conversely, some research directions become less attractive:

  • Solely scaling RL sampling as a solution to reward sparsity. Table 2 shows that increasing from 16 to 64 samples per prompt helps, but models without ReGFT initialization still lag behind. If the probability mass on correct trajectories is truly zero, scaling samples multiplies zero. The paper's evidence suggests that sampling-scale improvements have diminishing returns when the base model lacks competence, and that competence injection is the higher-leverage intervention.

  • Mid-RL reference injection without pre-RL competence building. Methods like BREAD and Questa inject reference guidance during RL to ensure some rollouts succeed. ReGFT's results suggest that building competence before RL — so the model can generate correct trajectories autonomously — may be more effective than providing crutches during RL that the model never internalizes. The paper does not empirically compare against these methods (see Limitation discussion in Section 5), so this suggestion is inferential, but the theoretical argument is coherent: trajectories generated with external scaffolding during RL may teach the model to rely on scaffolding rather than to reason independently.

Follow-Up Research This Work Enables

1. Ablating the 80/20 reference split and characterizing the optimal hinting surface. The paper's use of the first 80% of reference sentences as the hint is a fixed, unablated design choice (Section 4.3.1). A systematic follow-up would sweep the hint fraction from, say, 0% (no hint — pure self-generation, i.e., ReFT) to 100% (full reference provided, tested both with and without answer masking) in increments, measuring (a) the fraction of hard problems that become solvable under guided sampling, (b) the pass@1 of the resulting SFT checkpoint, and (c) the final RL plateau when initialized from each checkpoint. This would map the "hinting surface" and identify whether there is a sweet spot, whether the optimal fraction varies by problem difficulty (e.g., very hard problems may benefit from more guidance), and whether the model's natural tendency to generate its own reasoning (the paper's qualitative observation in Section 4.3.1) makes the exact split robust or brittle. A strong version of this experiment would also vary which sentences are provided — e.g., provide only the first sentence (the problem restatement and high-level strategy), provide all sentences except the last (the answer), or provide only sentences containing mathematical notation — to understand whether structural or content-based hinting is more effective.

2. Direct measurement of reward sparsity reduction during RL, with and without ReGFT. The paper's central mechanistic claim — that ReGFT reduces reward sparsity by increasing the fraction of RL batches containing at least one correct trajectory — is never directly measured (see Limitation 6.5). A straightforward follow-up would instrument the DAPO training loop to log, at each rollout step, the fraction of trajectory groups that pass the dynamic sampling filter (i.e., groups with at least one correct and at least one incorrect trajectory), stratified by whether the problems in the group were classified as hard or easy at the pre-RL stage. The prediction is that ReGFT + DAPO would show (a) a consistently higher pass-through fraction on hard problems compared to raw DAPO, (b) this fraction starting higher and remaining higher throughout training, and (c) the gap in pass-through fraction between hard and easy problems narrowing over training for ReGFT but remaining wide for raw DAPO. This would convert the paper's inferential causal narrative into a directly observed mechanism. If the prediction fails — if ReGFT does not actually increase batch-level reward sparsity — that would be a highly informative negative result forcing a reconsideration of why ReGFT helps.

3. ReGFT with different model families at matched scale to test the distributional-alignment hypothesis. The paper argues that ReGFT works because human reference solutions lie outside the model's generation distribution and must be translated into model-native reasoning before they can serve as effective SFT data (Section 3). This hypothesis predicts that the benefit of ReGFT over direct SFT on references will vary with the model's stylistic distance from the reference solutions. A cross-model-family experiment would test this: apply the identical ReGFT and direct-SFT pipelines to Qwen3-4B, Llama-3-8B, DeepSeek-7B, and Gemma-7B (or their current equivalents), all using the same OmniMath training set and AIME evaluation. Measure the gap between ReGFT + DAPO and direct-SFT + DAPO for each model family. If the distributional-alignment hypothesis is correct, models whose pretraining data distribution is "closer" to formal mathematical proof style (perhaps DeepSeek, which is strong at math) would show a smaller ReGFT-vs-direct-SFT gap, because their generation distribution overlaps more with the reference solutions. Models with more conversational or code-heavy pretraining would show a larger gap. This would provide the first cross-model evidence for the paper's theoretical mechanism and guide practitioners on which models benefit most from ReGFT.

4. ReGFT as a targeted intervention: per-problem difficulty estimation without the 25% threshold, plus dynamic hinting. The paper's hard problem threshold (25% accuracy with 16 samples) is a binary, static filter. A more sophisticated follow-up would treat difficulty as a continuous variable and vary the hinting strategy accordingly. Specifically: (a) estimate per-problem accuracy continuously (e.g., run 64 or 128 samples and compute exact pass@1), (b) define a continuous "guidance dosage" function that maps estimated accuracy to hint length — problems at 0-5% accuracy receive 95% of the reference, problems at 5-15% receive 80%, problems at 15-25% receive 50%, problems above 25% receive no guidance — and (c) evaluate whether this adaptive strategy outperforms the fixed 80/20 split on the aggregate evaluation benchmarks. If adaptive hinting works, it validates the intuition that harder problems need more structural guidance and easier (but still hard) problems need less, potentially expanding ReGFT's benefit to a wider range of the difficulty spectrum. A negative result (fixed 80/20 is as good or better) would simplify the method for practitioners by removing the need for fine-grained difficulty estimation.

5. Combining ReGFT with mid-RL reference injection to test for complementarity or redundancy. The paper positions ReGFT as a pre-RL alternative to methods like BREAD and Questa that inject reference guidance during RL, but never empirically compares them. A direct comparison would train four conditions: (a) raw DAPO, (b) ReGFT + DAPO, (c) raw DAPO + BREAD-style prefix anchoring (where RL rollouts on hard problems are prefixed with expert reference prefixes when self-generated rollouts fail), and (d) ReGFT + DAPO + BREAD. If ReGFT and BREAD are complementary, condition (d) should outperform both (b) and (c), suggesting that pre-RL competence building and mid-RL guidance address different aspects of the reward sparsity problem. If they are redundant or one dominates, the best condition would not benefit from the other. This experiment would directly inform whether practitioners should invest in pre-RL SFT, mid-RL augmentation, or both.

6. ReGFT on non-math reasoning with imperfect reference solutions. The paper operates in the cleanest possible setting: math problems with verifiable ground-truth answers and verified correct reference solutions. A stress-test would apply ReGFT to a domain where references are available but of variable quality — for example, code generation (where references are human-written solutions from programming competitions, but optimality and correctness may vary), or scientific question answering (where references are textbook explanations that may use different notation or conceptual frameworks than the model expects). The key measurement would be: does ReGFT's performance degrade gracefully with reference quality, or is there a sharp threshold below which reference-guided sampling becomes actively harmful (consistent with the 3.61% regression finding in Section 4.3.1)? This would establish the robustness boundary of the method and determine whether it requires curated, verified references (limiting its applicability) or can tolerate noisy, real-world reference data.

Practical Applications and Downstream Use Cases

Self-improving math reasoning systems with finite human reference budgets. A team building a math-solving LLM for competition-level problems (Olympiad, AIME, Putnam) has access to a finite corpus of human-written reference solutions — perhaps a few thousand problems with solutions from contest archives. Applying ReGFT, they can use those references to generate model-aligned training trajectories for the subset of problems their current model cannot solve, fine-tune on those trajectories, and then apply RL. The paper's numbers provide a concrete estimate of the leverage: reference-guided sampling makes an additional 5.85% of OmniMath's 4,428 problems solvable (Section 4.3.1), and the resulting ReGFT checkpoint raises the final RL plateau by approximately 10-15 percentage points on AIME 2024 (Figure 2a) and nearly doubles accuracy on Beyond-AIME (Figure 2c). For a fixed reference corpus, ReGFT extracts more training value per reference than direct SFT (which Figure 4 shows fails) or ignoring references entirely (raw DAPO). The method also requires no additional human annotation — the references already exist, and the rest is automated sampling and verification.

Bootstrapping RL on a new, hard dataset with existing reference solutions. Suppose a research group acquires a new dataset of expert-annotated reasoning problems (e.g., in physics, chemistry, or formal logic) where each problem includes a reference solution. The base model's accuracy on this dataset is low — perhaps 30% pass@64 — meaning RLVR would stall on 70% of problems. ReGFT provides a recipe for converting those reference solutions into effective pre-RL training data without manual prompt engineering or RL algorithm modification. The procedure is mechanically simple: for each hard problem (pass rate < 25% with 16 samples), truncate the reference to the first 80% of sentences, sample guided trajectories from the base model, verify correctness, and fine-tune on the mixture of guided and self-generated correct trajectories. The resulting checkpoint can then be handed to any RL algorithm. The only computational overhead is the initial sampling for difficulty estimation and guided trajectory generation — roughly 70,000 + 89,000 = ~159,000 complete solution generations for a 4,428-problem dataset (see Limitation 6.3), which is comparable to ~20 DAPO rollout steps (at 8,192 trajectories per step). For any RL run longer than ~20 steps, this overhead amortizes favorably. The practical value is that a team can take an off-the-shelf model, an off-the-shelf dataset with references, and an off-the-shelf RL framework (verl, as used in the paper), and apply ReGFT with no custom components beyond the reference truncation and prompting step.

Improving inference-time scaling for production math services. A deployed math-solving API that serves AIME-style problems to users can use ReGFT + DAPO to improve its pass@k scaling behavior, which directly translates to higher accuracy under increased test-time compute budgets. Figure 5 shows that ReGFT + DAPO maintains a clear performance advantage over raw DAPO even at k = 1,024 — on AIME 2024, ReGFT reaches approximately 70-75% pass@1024 versus roughly 60-65% for raw DAPO. For a service that can afford to sample 1,024 solutions per query (e.g., an asynchronous batch evaluation service), this 10-15 percentage point improvement represents solving roughly 3-4 additional problems out of a 30-problem AIME set — a substantial accuracy gain at fixed inference cost. The alternative of achieving that gain through model scaling (moving from a 4B to a larger model) would involve higher per-token latency and cost, whereas ReGFT's overhead is paid once during training and imposes zero additional inference cost. This makes ReGFT particularly attractive for deployment scenarios where training compute can be amortized over many inference queries — exactly the regime where the paper's R≪1R \ll 1 analysis (if adapted to this setting) would favor test-time compute over model scaling.

When to Prefer This Method

The paper does not position ReGFT against a specific named alternative with explicit conditions for preferring one over the other. It compares against raw DAPO, ReFT, and direct SFT on references, and ReGFT wins across all three benchmarks throughout RL training. There is no regime in the paper's experiments where ReGFT is not preferred over these baselines — it dominates. The paper also does not empirically compare against mid-RL augmentation methods (BREAD, Questa) or against scaling RL sampling as a standalone strategy (beyond the 16-vs-64 ablation in Table 2). As a result, a conditional "prefer A when X, prefer B when Y" decision matrix would be a fabrication unsupported by the paper's data. The paper's contribution is demonstrating that ReGFT works and characterizing why it works (reference-guided translation into model-aligned trajectories, pre-RL competence injection), not delineating a tradeoff space against peer methods. A practitioner reading this paper can conclude that if they have reference solutions available and their base model struggles on a subset of problems, ReGFT is likely worth trying before or alongside RL — but cannot make finer-grained method-selection decisions without additional experiments that the paper does not provide.