ArXiv: 2604.25907
🎯 Pitch
RLVR alone can stall at cold start not due to variance but because escaping requires Ω(1/p₀) time under a bounded loss that resists label noise, whereas SFT escapes in Θ(log(1/p₀)) by amplifying gradients on unfamiliar examples via log-marginal-likelihood. The paper shows that a single-parameter commitment q in the JQ loss family interpolates between these extremes, and that intermediate q values allow direct cold-start escape without SFT.
1. Executive Summary
This paper introduces the JQ loss family — a single-parameter continuum indexed by a commitment scalar q ∈ [0,1] that interpolates between RLVR (q=0, the exploitation pole) and log-marginal-likelihood maximization over latent trajectories (q=1, the density-estimation pole) — and uses it to provide a unifying theoretical account of why SFT-then-RLVR succeeds: SFT (q=1) amplifies gradients on unfamiliar examples by a factor P⁻q to escape cold start in Θ(log(1/p₀)) time, while RLVR (q=0) resists label-noise memorization but stalls with Ω(1/p₀) escape. The paper derives two Monte Carlo estimators that directly optimize fixed-q points on this continuum — Gradient-Amplified RL (GARL) (prior sampling, amplified by P⁻q) and Posterior-Attenuated Fine-Tuning (PAFT) (posterior resampling, attenuated by P¹⁻q) — and validates them on FinQA, HotPotQA, and MuSiQue with Qwen 3 0.6B and 8B models under exact-match training rewards. In cold start, GARL at high q escapes where GRPO fails entirely; in warm start, GARL at low q beats GRPO by +10.9 maj@16 on FinQA (38.7 vs. 27.8), while PAFT at q=0.75 beats GRPO by +13.9 maj@16 on HotPotQA (47.9 vs. 34.0), establishing that stable warm-start gains require switching estimators when GARL destabilizes.
2. Context and Motivation
The Core Problem: Why Does SFT-Then-RLVR Work?
The standard recipe for post-training reasoning models is well-established: first perform supervised fine-tuning (SFT) on annotated reasoning traces, then apply reinforcement learning from verifiable rewards (RLVR) using outcome-level correctness signals [Ouyang et al., 2022, DeepSeek-AI, 2025, Shao et al., 2024, Chu et al., 2025]. This two-stage pipeline has driven remarkable empirical progress — models like DeepSeek-R1 demonstrate sophisticated reasoning capabilities that emerge from exactly this training protocol. Yet two fundamental questions about it have lacked a satisfying theoretical account:
Why this specific ordering? Why must SFT come first and RLVR second? What would happen if you reversed the stages or used only one of them? The empirical answer is that RLVR alone stalls at cold start — when the model's initial probability of generating a correct answer () is near zero, reinforcement learning cannot make meaningful progress because it receives no reward signal to learn from. This is not a subtle failure of tuning; it's a hard barrier. The model generates incorrect outputs, receives zero reward, and the gradient signal collapses, preventing any learning whatsoever. SFT circumvents this by providing dense supervision on annotated rationales, moving the model into a regime where is non-trivial. RLVR can then take over from there.
But why does this ordering succeed in particular? Many alternatives could in principle break the cold-start deadlock — you could use a weaker RL signal, a shaped reward, a different optimization algorithm, or a different data ordering. The fact that SFT → RLVR works so reliably suggests there's something structurally right about the progression from dense, example-level supervision to sparse, outcome-level supervision. The paper identifies this structural property as a shift in commitment to supervision: SFT commits fully to every example's supervision signal, amplifying gradients on hard (low-) examples to force learning; RLVR commits selectively, robustly filtering noise but requiring sufficient initial probability to make progress. The JQ continuum shows this is a continuous parameter , not a binary choice.
Why does RLVR alone stall? The naive explanation is "no reward signals, therefore no gradient." But this is incomplete. Recent advances — specifically Rao-Blackwellized REINFORCE variants like RLOO [Zhou et al., 2026] — ensure non-zero gradients even when all sampled outputs are incorrect by analytically marginalizing over the output token rather than relying on sampled binary rewards. These variants do reduce variance, but as the paper demonstrates (Section 5, Table 1), they still fail at cold start: GARL at (which recovers RB-RLOO) gets zero accuracy on all benchmarks in cold start. The bottleneck is not variance — it's the rate at which the gradient can move the model when is extremely small. Rao-Blackwellization ensures there is a gradient signal, but that signal is too weak to escape the cold-start regime within any practical training budget.
This distinction — between having some gradient and having enough gradient — is the paper's central theoretical insight. It's not about whether the gradient exists; it's about how fast it can increase , and that speed is governed by an amplification factor that differs by orders of magnitude depending on .
The Deeper Gap: No Unifying Language for Reasoning Model Training
The SFT-then-RLVR pipeline, while effective, has developed largely through empirical trial-and-error. The field lacks:
A common mathematical framework. SFT on annotated rationales, REINFORCE with outcome rewards, Rao-Blackwellized variants, importance-weighted objectives, and variational EM methods are typically treated as distinct algorithm families with separate derivations, separate hyperparameters, and separate intuitions about when they work. Yet they all share a common goal: maximizing some measure of the model's probability of producing correct answers given latent reasoning trajectories. The isolation of these methods means practitioners cannot reason clearly about tradeoffs — when should I use more SFT? When should I switch to RL? What happens if I blend them? These questions currently have only empirical answers.
An explanation for the cold-start/warm-start boundary. It's empirically known that RL works when the model already has some non-trivial probability of success, but the precise nature of this boundary is unclear. Is it a smooth transition where RL starts working gradually as increases? Is there a critical threshold? Does the threshold depend on model scale, problem difficulty, or training hyperparameters? Without a theory of escape rates, these questions cannot be answered systematically.
A principled way to train without annotated rationales. SFT requires human-annotated reasoning traces, which are expensive, domain-specific, and constrain the model to imitate human reasoning patterns rather than discovering its own. RLVR removes the annotation requirement but only works from warm start. The field needs methods that can cold-start without annotated rationales — directly from verifiable outcome supervision — and the JQ continuum provides the theoretical foundation for such methods by showing that intermediate values can supply the gradient amplification of SFT while retaining the verifiability of RLVR.
Where Prior Approaches Fall Short
Rao-Blackwellized estimators reduce the wrong bottleneck. Zhou et al. [2026] introduced Rao-Blackwellized REINFORCE, which analytically computes the expected reward given a sampled rationale rather than relying on a single sampled output. This reduces variance substantially — there's no Monte Carlo noise from the output sampling step. But as the cold-start experiments in Section 5 demonstrate, RB-RLOO () still gets zero accuracy across all benchmarks in cold start (Table 1). The paper's analysis explains why: variance reduction addresses stochasticity in the gradient estimate, but the cold-start bottleneck is deterministic — the gradient direction is correct on average, but its magnitude is proportional to (since at ). When starts at, say, , even a perfectly estimated gradient moves the model at a rate of per unit time. No amount of variance reduction can fix a gradient magnitude problem.
RLVR methods (GRPO, REINFORCE) lack cold-start capability. DeepSeek-AI [2025] and Shao et al. [2024] apply GRPO and similar RLVR algorithms to reasoning tasks, but only after SFT warm-start. The cold-start experiments in Section 5 (Table 1) confirm that GRPO fails entirely at cold start — zero accuracy across all three benchmarks for both 0.6B and 8B models. This is not a tuning issue; it's a fundamental property of the pole where gives escape time (Theorem 3.1). The RLVR loss at is bounded () and noise-robust, but this robustness comes at the cost of glacial learning speed when is small.
SFT alone has the opposite problem: it memorizes noise. At the pole, the loss is , which is unbounded as and provides massive amplification () on hard examples. This enables the fast escape from cold start (Theorem 3.2), but it also means the model commits indiscriminately to all supervision signals — including label errors. The noise-fitting analysis (Proposition D.2) shows that at , the model fits corrupted labels at a rate , while at , noise-fitting time diverges to infinity (the model simply ignores label noise). This is why the standard pipeline switches from SFT to RLVR: SFT provides the fast cold-start escape, then RLVR's noise robustness takes over once is non-trivial.
Existing RL-MLE bridges treat these as separate frameworks, not a continuum. Several prior works connect reinforcement learning and maximum likelihood estimation [Levine, 2018, Norouzi et al., 2016, Guu et al., 2017], but they present them as distinct frameworks with different objectives, different optimization procedures, and different theoretical properties. The JQ continuum shows they are endpoints of a single loss family with a continuous parameter controlling commitment. This is more than a philosophical reframing — it means you can optimize at any intermediate , inheriting partial cold-start speed and partial noise robustness proportional to .
Importance-weighted and variational methods exist at one pole. The IWAE gradient estimator [Burda et al., 2015] corresponds to GARL at (Proposition E.2), and the EM gradient update with posterior samples [Dempster et al., 1977, Phan et al., 2023] corresponds to PAFT at . Both operate at the density-estimation pole, inheriting its cold-start advantages and noise-memorization disadvantages. The JQ continuum generalizes these to arbitrary , providing estimators (GARL at , PAFT at ) that attenuate the amplification to balance escape speed against noise robustness and estimator quality.
Concurrent work on ML-RL continua treats the problem differently. Tajwar et al. [2026] propose MaxRL, an RL-to-ML continuum via Maclaurin truncation of . Their estimator is unbiased for the truncated objective but is exactly zero when no sample succeeds — precisely the cold-start scenario where help is needed most. In contrast, GARL targets the true -loss and always has non-zero gradient because (the likelihood weight under the prior is never zero, even when sampled outputs are incorrect). This difference is critical for cold-start viability. Concurrently, Wang et al. [2026] apply a deformed-log family at the token level for SFT, but their exact softmax probability is directly computable, whereas is the intractable marginal over latent trajectories — a fundamentally harder estimation problem that requires the Monte Carlo machinery developed in Section 4.
How This Paper Positions Itself
The paper positions JQ not as a single new training algorithm, but as a unifying theoretical framework that explains why the standard pipeline works and enables principled training at any point on the commitment spectrum. Rather than proposing yet another post-training recipe, it asks: what is the underlying structure that makes SFT-then-RLVR the right answer, and can we exploit that structure to do better?
The framework explains the pipeline, it doesn't replace it. The SFT-then-RLVR paradigm emerges from the JQ continuum as a stepwise schedule: SFT operates at , using amplification to escape cold start in time; then RLVR operates at , where noise-fitting time diverges to infinity (Proposition D.2), providing immunity to label errors. The paper articulates this as a hard switch between two poles of a continuous family. The reverse order () would fail: cannot escape cold start, and by the time takes over, there's nothing to refine. The paper makes this dynamical tradeoff explicit: accelerates commitment to both clean and corrupted supervision by the same multiplicative factor, so the optimal schedule must exploit the -dependence of what the model commits to, not just how fast.
It provides the first cold-start escape rate analysis for latent-variable reasoning. The theorems in Section 3 are novel not because they analyze gradient flow — that's a standard technique — but because they isolate as the sole determinant of escape speed through the factor in the dynamics . This is a qualitative separation: gives logarithmic escape, gives reciprocal () escape, and intermediate give intermediate rates . The separation ratio diverges as , meaning these are not small constant-factor differences — they are exponentially large gaps in the initial probability. A model with would take roughly units of time to escape at but only units at .
It derives the first gradient estimators for direct -optimization. Prior to this work, there was no way to optimize JQ at arbitrary on latent-variable models where is intractable. The dual factorization in Proposition 2.2 () immediately suggests two Monte Carlo strategies, but both require estimating or from samples — a ratio estimation problem with finite-sample bias. The bias expansion in Theorem 4.1 shows that this bias is : higher and lower both degrade estimator quality, creating a cold-start escape vs. estimator quality tradeoff that matches the empirical pattern (Table 1: escapes faster but achieves lower final accuracy than ).
It demonstrates that the right estimator depends on the training regime. The empirical results in Section 5 show a regime-dependent pattern: GARL works well when training is stable (FinQA, all ), but collapses to zero on benchmarks where latent-rationale variance is high (HotPotQA, MuSiQue). PAFT, by filtering to approximately-posterior samples, remains stable on those benchmarks but is inapplicable at cold start (because attenuation suppresses the gradient entirely). The paper thus establishes a three-phase recipe: GARL at high for cold start, GARL at low where stable, PAFT at otherwise. This is a specific, actionable prescription derived from the theory that the experiments only partially validate — annealing schedules and automatic switching remain future work.
The paper explicitly scopes itself as foundational, not comprehensive. The theoretical analysis is stylized (single-example, gradient flow, bounded score, exact-match reward) to isolate the role of the amplification factor without stochastic optimization confounds. The experiments use fixed values rather than schedules, fixed benchmarks, and a single model family. The paper acknowledges these limitations in Section 6 and positions its contributions as: (1) the JQ family itself as a conceptual foundation, (2) the cold-start escape rate analysis, (3) the GARL/PAFT estimators as first demonstrations, and (4) initial empirical validation that confirms the theoretical predictions. The SFT-then-RLVR pipeline is thus reframed not as a recipe to be followed blindly, but as an instance of a broader design space that can be navigated with the commitment parameter .
3. Technical Approach
3.1 Reader Orientation
This is primarily a theoretical analysis paper that proposes a unified loss family for training reasoning models with latent (unobserved) reasoning trajectories, rather than a single new algorithm. The system being developed is a mathematical framework and corresponding gradient estimators that allow practitioners to train language models for reasoning tasks by choosing a single scalar parameter $q \in [0, 1]$ that controls how aggressively the model commits to its supervision signal — trading off cold-start escape speed against noise robustness and estimator quality in a principled, continuous way.
3.2 Big-Picture Architecture (Diagram in Words)
The JQ framework has five conceptual layers:
-
The JQ Loss Family (
$J_Q(\theta, q)$) — a single-parameter continuum of per-example losses defined via the Tsallis$q$-logarithm, ranging from a bounded, noise-robust exploitation loss at$q=0$to an unbounded, mode-covering density-estimation loss at$q=1$. Every$q$shares the same per-example gradient direction but amplifies it by a factor$P_\theta^{-q}$that grows with$q$and shrinks with$P_\theta$. -
The Latent-Variable Generative Model — an autoregressive language model
$p_\theta$that, given an input$x$, samples an unannotated latent rationale$z \sim p_\theta(\cdot \mid x)$(e.g., chain-of-thought, proof trace, program), then samples an output$\hat{y} \sim p_\theta(\cdot \mid x, z)$. The marginal probability of the correct output is$P_\theta = p_\theta(y^* \mid x^*) = \sum_z p_\theta(z, y^* \mid x^*)$, which is intractable because the latent space is enormous. -
The Gradient Geometry (Proposition 2.2) — the dual factorization
$\nabla_\theta \ell_q = P_\theta^{-q} \nabla_\theta \ell_0 = P_\theta^{1-q} \nabla_\theta \ell_1$that connects all$q$to the two endpoint losses. This factorization is what makes estimation possible: the RL factorization ($P_\theta^{-q} \nabla_\theta \ell_0$) motivates GARL, and the FT factorization ($P_\theta^{1-q} \nabla_\theta \ell_1$) motivates PAFT. -
Monte Carlo Gradient Estimators (GARL and PAFT) — two sampling-based procedures for approximating
$\nabla_\theta J_Q$at fixed$q$without requiring annotated rationales. Both reuse the same pool of$M$latent trajectories sampled from the prior. GARL amplifies the RL gradient by a plug-in estimate of$P_\theta^{-q}$; PAFT resamples trajectories proportional to their likelihood weights (approximate posterior sampling) and applies attenuated teacher-forcing. -
Gradient Flow Analysis — a continuous-time, single-example analysis that isolates how the amplification factor
$P_\theta^{-q}$governs the rate$\dot{p} = p^{2-q} \|s(\theta)\|^2$at which the success probability$p = P_\theta$increases, yielding exponential separation in escape time between$q=0$($\Omega(1/p_0)$) and$q=1$($\Theta(\log(1/p_0))$), with intermediate$q$giving intermediate rates$\Theta(p_0^{-(1-q)}/(1-q))$.
Information flows as follows: an input-output pair $(x^*, y^*)$ enters the system → the generative model $p_\theta$ produces $M$ latent trajectories $z^{(1)}, \ldots, z^{(M)}$ → likelihood weights $w_m = p_\theta(y^* \mid x^*, z^{(m)})$ are computed → the chosen estimator (GARL or PAFT, at the chosen $q$) forms a gradient estimate → parameters $\theta$ are updated → $P_\theta$ increases → the amplification $P_\theta^{-q}$ shifts, changing the effective learning dynamics for the next step.
3.3 Roadmap for the Deep Dive
- First, the JQ loss family itself — how the Tsallis
$q$-logarithm defines the per-example loss, how$q$controls commitment, and what the escort minimizer tells us about mode-seeking vs. mode-covering — since the whole framework rests on this continuous interpolation. - Second, the gradient geometry (Proposition 2.2) — why all
$q$share one gradient direction, how the dual factorization works, and why this makes Monte Carlo estimation tractable — because both GARL and PAFT are direct consequences of this factorization. - Third, the gradient flow analysis of cold-start escape (Section 3) — how
$\dot{p} = p^{2-q} \|s(\theta)\|^2$governs the dynamics, why the exponent$2-q$creates exponential separation, and the precise$\Omega$/$\Theta$rates — because this explains why SFT-then-RLVR works and what intermediate$q$buys you. - Fourth, the GARL estimator (Section 4.1) — how it constructs a plug-in ratio estimator
$\bar{g}_M / (\bar{w}_M)^q$, where its bias$O(q/MP_\theta^q)$comes from, and how the RLOO control variate is applied — since GARL is the primary cold-start and warm-start estimator. - Fifth, the PAFT estimator (Section 4.2) — how importance resampling approximates the posterior, why the attenuation factor
$(\bar{w}_M)^{1-q}$modifies the gradient, and why PAFT is more stable than GARL on unstable benchmarks despite higher variance — since PAFT fills the gap where GARL collapses.
3.4 Detailed, Sentence-Based Technical Breakdown
The core idea of the paper is that the SFT-then-RLVR pipeline corresponds to a stepwise $q=1 \to 0$ schedule along a continuous loss family, and that this insight enables both a theoretical explanation of the pipeline's success and the design of gradient estimators that can optimize at any intermediate $q$ without annotated rationales.
The Tsallis $q$-Logarithm and the JQ Loss Family
The mathematical foundation of the entire framework is the Tsallis $q$-logarithm, a deformation of the natural logarithm that introduces a parameter $q$ controlling the shape of the function. The standard definition from Tsallis [1988] is:
with the limit $\log_1(u) = \log u$ recovered by L'Hôpital's rule as $q \to 1$.
where $u$ is a real number in $(0, 1]$ (a probability), $q \in [0, 1]$ is the commitment parameter, and $\log_q(u)$ is the deformed logarithm evaluated at $u$.
What it computes: The $q$-logarithm transforms a probability $u$ into a loss-like value that grows as $u$ shrinks. At $q=0$, the formula simplifies to $\log_0(u) = u - 1$ (a linear function, bounded below by $-1$). At $q=1$, it recovers the standard natural logarithm. For $q \in (0,1)$, it is a continuous, monotonically increasing interpolation between these two extremes. When negated (which is what JQ does), it becomes a loss that penalizes low probabilities.
Why this form: The $q$-logarithm is not an arbitrary interpolation — it is the unique function (up to constants) that maintains the key algebraic property $\frac{d}{du} \log_q(u) = u^{-q}$, meaning its derivative is a simple power law in $u$. This derivative property is what makes the per-example gradient $\nabla_\theta \ell_q = -P_\theta^{-q} \nabla_\theta P_\theta$ factor cleanly into the amplification factor $P_\theta^{-q}$ times the endpoint gradient $\nabla_\theta \ell_0$, which is the entire mechanism underlying both the convergence analysis and the estimator design. Alternative interpolations (e.g., a convex combination of $\log$ and linear) would not produce this multiplicative gradient factorization and would not yield the clean exponential separation in escape rates.
From the $q$-logarithm, the paper defines the per-example JQ loss by negating it and applying it to the model's success probability $P_\theta$:
and the dataset-level objective is the expectation over the training distribution:
where $(x^*, y^*)$ is a supervised input-output pair from the training set $\mathcal{D}$, $P_\theta = p_\theta(y^* \mid x^*) = \sum_z p_\theta(z, y^* \mid x^*)$ is the marginal probability that the model assigns to the correct output given the input, and $q \in [0, 1]$ is the commitment parameter.
What it computes: For each training example, JQ measures how much the model's success probability $P_\theta$ deviates from 1. At $q=0$, the per-example loss is $\ell_0 = 1 - P_\theta$ — bounded between 0 and 1, and equal to $1 - \mathbb{E}[\text{reward}]$ under exact-match supervision (Proposition B.1), so minimizing $J_Q(\theta, 0)$ is literally maximizing expected reward. At $q=1$, the loss is $\ell_1 = -\log P_\theta$ — unbounded above, growing to infinity as $P_\theta \to 0$, and equal to the negative log-marginal-likelihood of the correct output under the latent-variable model. At intermediate $q$, the loss smoothly interpolates between these two behaviors: the $P_\theta^{1-q}$ term in the numerator means the loss grows more steeply than linear but less steeply than logarithmic as $P_\theta$ decreases.
Why this form: The $1 - P_\theta^{1-q}$ numerator makes the loss bounded at $q<1$ (since $P_\theta \leq 1$ implies $P_\theta^{1-q} \leq 1$, so $\ell_q \leq 1/(1-q)$), which gives noise robustness — a single mislabeled example cannot drive the loss to infinity and thus cannot dominate the gradient. The denominator $1-q$ normalizes the slope so that $\ell_q$ approaches $-\log P_\theta$ smoothly as $q \to 1$. The boundedness at $q<1$ is the key structural property that separates it from the unbounded $q=1$ pole: models trained with $q<1$ will not memorize label errors, because no single example can incur infinite loss regardless of how small $P_\theta$ becomes.
The paper further shows that for $q > 0$, the function $h_q(u) = -\log_q(u)$ is strictly convex, which yields a dispersion penalty via Jensen's inequality:
where $\bar{P} = \mathbb{E}_{(x^*, y^*) \sim \mathcal{D}}[P_\theta]$ is the mean success probability across all training examples.
What it computes: The left-hand side is the actual JQ loss averaged over examples. The right-hand side is the $q$-logarithm of the average success probability. Jensen's inequality says the left side is always at least as large as the right side, with equality if and only if $P_\theta$ is constant across all examples. This means that for $q > 0$, JQ penalizes uneven success across examples — a model that gets 100% on half the data and 0% on the other half incurs higher JQ loss than a model that gets 50% on everything, even though both have the same average $\bar{P}$. This dispersion penalty grows with $q$: at $q=0$, the convexity vanishes and the model only cares about the average (which is why $q=0$ can be mode-seeking — it can concentrate all its capacity on the easiest examples); at $q=1$, the penalty is strongest (which is why $q=1$ is mode-covering — it forces the model to spread its probability mass across all examples).
Why this form: The dispersion penalty provides a mechanistic explanation for why $q=1$ (SFT) forces the model to learn from all examples, including hard ones, while $q=0$ (RLVR) allows the model to ignore hard examples in favor of perfecting easy ones. This is not just an observation about behavior — it's a mathematical consequence of convexity, meaning the gradient on each example is anti-proportional to the current $P_\theta$ for that example, forcing the hardest examples to receive the strongest updates.
The Escort Minimizer and $q$ as Training-Time Temperature
To build intuition for what JQ optimization does to a model, the paper analyzes a simplified setting: a categorical model with $K$ categories and empirical label frequencies $\alpha_j > 0$ (the fraction of training examples with label $j$). This is not the full latent-variable setting, but it exposes the loss's mode-seeking vs. mode-covering behavior in closed form. The objective becomes:
where $\theta_j$ is the model's predicted probability for category $j$ (constrained to the simplex $\Delta_K$: all $\theta_j \geq 0$ and $\sum_j \theta_j = 1$), and $\alpha_j$ is the fraction of training data with label $j$.
What it computes: For a fixed data distribution $\alpha$, JQ asks: what probability vector $\theta$ minimizes the average $q$-logarithmic loss? The answer, derived via Lagrange multipliers enforcing the simplex constraint, is the escort distribution [Beck and Schlögl, 1993]:
where $\alpha_j$ is the empirical frequency of class $j$ and $q \in (0, 1]$ is the commitment parameter.
What it computes: The optimal model probability for each class is proportional to the class frequency raised to the power $1/q$. At $q=1$, the exponent is 1, so $\theta^*_j = \alpha_j$ — the model exactly recovers the data distribution (mode-covering). At $q \to 0^+$, the exponent $1/q \to \infty$, so the largest $\alpha_j$ dominates: the model places all mass on the single most frequent class (pure mode-seeking). At intermediate $q$, the distribution is "sharpened" relative to the data distribution — more frequent classes get amplified, less frequent classes get suppressed, but not entirely eliminated. This sharpening is monotonic: for any two classes with $\alpha_j > \alpha_k$, the ratio $\theta^*_j / \theta^*_k = (\alpha_j / \alpha_k)^{1/q}$ grows as $q$ decreases, meaning lower $q$ increasingly favors the dominant class.
Why this form: The escort distribution explains the mode-seeking-to-mode-covering interpolation in closed form and connects it to a single parameter $q$. It also shows why the $q$-logarithmic scoring rule is only strictly proper at $q=1$ (Corollary C.3): at $q<1$, the minimizer is not the data distribution itself but a sharpened version of it, so the model has no incentive to perfectly calibrate its probabilities. This is the formal reason why $q<1$ provides noise robustness: the model does not try to perfectly fit every observed label proportion, so label errors (which bias $\alpha$ away from the true data distribution) have less influence on the minimizer. The escort is also useful for the noise-fitting analysis (Section D.5): it gives the asymptotic equilibrium $\tilde{p}^*(q) = (\epsilon/(1-\epsilon))^{1/q}$ that the model converges to under label noise, showing explicitly how $q \to 0$ drives noise contamination to zero.
The Gradient Geometry and Dual Factorization
The centerpiece connecting the loss family to practical optimization is Proposition 2.2, which shows that all members of JQ share the same per-example gradient direction, differing only by a scalar multiplicative factor:
where $P_\theta = p_\theta(y^* \mid x^*)$ is the success probability, $\nabla_\theta \ell_0$ is the gradient of the exploitation loss at $q=0$, $\nabla_\theta \ell_1$ is the gradient of the density-estimation loss at $q=1$, and $q \in [0, 1]$ is the commitment parameter.
What it computes: Starting from the definition $\ell_q = -\log_q(P_\theta)$, the chain rule gives $\nabla_\theta \ell_q = -P_\theta^{-q} \nabla_\theta P_\theta$ because $\frac{d}{du} \log_q(u) = u^{-q}$. Meanwhile, $\ell_0 = 1 - P_\theta$ gives $\nabla_\theta \ell_0 = -\nabla_\theta P_\theta$, and $\ell_1 = -\log P_\theta$ gives $\nabla_\theta \ell_1 = -(1/P_\theta) \nabla_\theta P_\theta = P_\theta^{-1} \nabla_\theta \ell_0$. Substituting, we get $\nabla_\theta \ell_q = P_\theta^{-q} \nabla_\theta \ell_0$, and since $\nabla_\theta \ell_0 = P_\theta \nabla_\theta \ell_1$, the second equality $\nabla_\theta \ell_q = P_\theta^{1-q} \nabla_\theta \ell_1$ follows. The first factorization (multiplying $\nabla_\theta \ell_0$ by $P_\theta^{-q}$) is the RL factorization, because $\ell_0$ is the expected reward objective. The second (multiplying $\nabla_\theta \ell_1$ by $P_\theta^{1-q}$) is the FT factorization, because $\ell_1$ is the log-marginal-likelihood and its gradient can be expressed as an expectation over the posterior.
Why this form: This dual factorization is the operational engine of the entire paper because it tells us exactly what we need to estimate to optimize JQ. Instead of constructing a new gradient estimator from scratch for each $q$, we can take a known estimator for either endpoint ($\nabla_\theta \ell_0$ or $\nabla_\theta \ell_1$) and simply multiply it by the appropriate $P_\theta$-power. The RL factorization suggests sampling from the prior $p_\theta(z \mid x^*)$ to estimate $\nabla_\theta \ell_0$ and then amplifying by a plug-in estimate of $P_\theta^{-q}$ — this becomes GARL. The FT factorization suggests sampling from the posterior $p_\theta(z \mid x^*, y^*)$ to estimate $\nabla_\theta \ell_1$ and then attenuating by $P_\theta^{1-q}$ — this becomes PAFT. The key challenge is that $P_\theta$ is intractable (it is the marginal over all latent trajectories), so both approaches must estimate this amplification/attenuation factor from samples — making them ratio estimators with finite-sample bias.
The amplification factor $P_\theta^{-q}$ ranges from 1 (at $q=0$, no amplification) to $1/P_\theta$ (at $q=1$, inverse-proportional amplification). When $P_\theta$ is very small (cold start), this factor is enormous — e.g., if $P_\theta = 10^{-5}$ and $q=1$, the gradient is amplified by $10^5$ relative to the $q=0$ gradient. This is why $q=1$ escapes cold start quickly: the gradient magnitude is hundreds of thousands of times larger on the hardest examples. The price is that the same amplification applies to gradient noise, to estimator bias, and to corrupted examples — the factor is completely agnostic to whether the supervision is correct or incorrect.
Gradient Flow Analysis of Cold-Start Escape (Section 3)
The theoretical centerpiece of the paper is the analysis of how $q$ controls the speed at which a model escapes cold start. The analysis is stylized — it tracks a single example's success probability under continuous-time gradient flow, isolating the effect of the amplification factor $P_\theta^{-q}$ without the confounding effects of stochastic optimization, mini-batching, adaptive optimizers, or multi-example interactions.
The key dynamical equation comes from applying gradient flow $\dot{\theta} = -\nabla_\theta \ell_q(\theta)$ and using the chain rule to track the evolution of the success probability $p(t) = P_{\theta(t)}$:
where $p = P_\theta$ is the current success probability, $\dot{p}$ is its time derivative (rate of improvement), $s(\theta) = \nabla_\theta \log P_\theta$ is the score function (the gradient of the log-probability with respect to parameters), $\|s(\theta)\|^2$ is the squared norm of the score (a measure of how much parameter movement changes the log-probability), and $q \in [0, 1]$ is the commitment parameter.
What it computes: This is an ordinary differential equation (ODE) for $p(t)$ in terms of itself. The term $\|\nabla_\theta P_\theta\|^2$ captures how sensitive the success probability is to parameter changes — if small parameter changes produce large changes in $P_\theta$, the score norm is large and learning is fast. The factor $P_\theta^{-q}$ comes from the $q$-logarithm's derivative: at higher $q$, the loss is more sensitive to small $P_\theta$, so the gradient is larger and $p$ increases faster. Rewriting as $p^{2-q} \|s\|^2$ isolates the $q$-dependence entirely in the exponent $2-q$ of $p$. The score norm $\|s\|^2$ does not depend on $q$ — it is a property of the model parameterization alone — so the effect of $q$ on escape speed is completely captured by how $p^{2-q}$ behaves as $p \to 0$.
Why this form: The exponent $2-q$ is the crux. When $p$ is very small, $p^{2-q}$ is much smaller than $p$ itself (since $2-q \geq 1$), and the smaller the exponent, the more aggressively small $p$ is downweighted. At $q=0$, the exponent is 2, so $\dot{p} \propto p^2$ — when $p_0 = 10^{-6}$, the initial rate is proportional to $10^{-12}$. At $q=1$, the exponent is 1, so $\dot{p} \propto p$ — the initial rate is proportional to $10^{-6}$, which is one million times larger than at $q=0$. This is an enormous difference, and it scales exponentially in the initial probability: the lower $p_0$, the larger the gap between $q=0$ and $q=1$ escape speeds. Intermediate $q$ give intermediate exponents $2-q$ between 1 and 2, yielding proportional differences in escape rate.
The paper formalizes this into two theorems. Theorem 3.1 provides a lower bound on escape time assuming only that the score norm is bounded above ($\|s(\theta)\| \leq C$), which is a standard regularity condition (the model cannot have arbitrarily large score even when $p$ is tiny). Theorem 3.2 provides a matching upper bound (and thus a tight $\Theta$ characterization) under the additional assumption that the score norm is bounded below ($\|s(\theta)\| \geq c > 0$) — meaning the score doesn't vanish, so the model can always make progress.
Theorem 3.1 (Lower Bound):
where $T_q(p_0, \delta)$ is the time to increase the success probability from initial value $p_0 \in (0, 1/2)$ to target value $\delta \in (p_0, 1/2]$, and the big-Omega notation means "grows at least as fast as."
What it computes: The proof uses the ODE comparison principle: since $\dot{p} = p^{2-q} \|s\|^2 \leq C^2 p^{2-q}$ (using the upper bound on $\|s\|$), the actual $p(t)$ grows no faster than the solution to the simpler ODE $\dot{p}^* = C^2 (p^*)^{2-q}$. Separating variables and integrating gives the time bound. For $q \in [0,1)$, the integral of $u^{-(2-q)}$ from $p_0$ to $\delta$ evaluates to $(p_0^{-(1-q)} - \delta^{-(1-q)})/(1-q)$, which is dominated by the $p_0^{-(1-q)}$ term as $p_0 \to 0$. For $q=1$, the integral is $\log(\delta/p_0)$, which grows logarithmically in $1/p_0$.
Theorem 3.2 (Tight Rates):
What it computes: With the additional lower bound $\|s\| \geq c$, we also have $\dot{p} \geq c^2 p^{2-q}$, giving an upper bound on escape time via the same integration. Since the upper bound matches the lower bound up to constant factors, we have a tight $\Theta$ characterization. The ratio $T_q / T_{q'} = \Theta(p_0^{-(q'-q)})$ diverges to infinity as $p_0 \to 0$ for any $q < q'$ — meaning the escape time difference between two $q$ values grows exponentially in $1/p_0$. For $p_0 = 10^{-6}$, $T_0/T_{0.5} = \Theta(10^3) = 1000\times$ and $T_0/T_1 = \Theta(10^6 / \log(10^6)) \approx 70,000\times$. These are not small constant-factor differences; they are qualitative differences in whether escape is possible within any practical training budget.
Why these particular bounds: The lower bound in Theorem 3.1 requires only the mild assumption that $\|s\|$ is bounded above, which is almost always true (a model cannot have unbounded score unless it has pathological parameterization). This makes the $\Omega$ bound very general — at $q=0$, cold-start escape is provably slow under almost any realistic model. The matching upper bound requires the additional assumption that $\|s\| \geq c$ throughout the trajectory, which may fail near optima where the score vanishes. The paper restricts the analysis to $\delta \leq 1/2$ to stay in the regime where this assumption is reasonable — far from the optimum, the score is non-zero because there is room for improvement. The sigmoid example in Section D.1 validates this: $\|s\|^2 = (1-p)^2$, which is bounded between $(1-\delta)^2$ and $1$ on $[p_0, \delta]$, satisfying both the upper and lower bound assumptions and yielding the exact asymptotic $T_q = p_0^{-(1-q)}/(1-q) \cdot (1 + o(1))$.
A complementary result (Proposition D.1) shows that near optimality (when $p \approx 1$), the escape time is $q$-independent up to a vanishing correction. This is because the amplification factor $P_\theta^{-q}$ converges to 1 as $P_\theta \to 1$ — the $q$-logarithm and natural logarithm have the same slope near $u=1$. So the $q$-dependence matters only in the low-$P_\theta$ regime, which is precisely cold start.
Noise-Fitting Dynamics: The Symmetric Dual
The same gradient flow machinery also characterizes how fast the model commits to incorrect supervision under label noise. The paper analyzes a binary sigmoid model with symmetric label noise: the clean label $c$ is flipped to noise label $k$ with probability $\epsilon < 1/2$. The key quantity is the noise contamination $\tilde{p}(t) = p_\theta(k \mid x^*)$ — the model's probability of predicting the corrupted label.
The dynamics are governed by:
where $\tilde{p} = p_\theta(k \mid x^*)$ is the noise probability, $\epsilon$ is the label-noise rate, and the two terms represent the gradient from noise examples (pushing $\tilde{p}$ up) and clean examples (pushing $\tilde{p}$ down).
What it computes: When $\tilde{p}$ is very small (the model initially gives near-zero probability to the noise label), the noise term $\epsilon \tilde{p}^{2-q}$ dominates the clean term $(1-\epsilon)(1-\tilde{p})^{2-q} \approx (1-\epsilon)$ because $\tilde{p}^{2-q}$ can be arbitrarily small for small $\tilde{p}$, but for $q>0$ the noise term has a larger exponent on $\tilde{p}$ ($2-q$ vs. 0 for the clean term's near-constant behavior), meaning the dynamics are initially driven upward by the noise gradient.
Proposition D.2 (Noise-Fitting Rate):
What it computes: The time to grow noise contamination from initial $\tilde{p}_0$ to a fixed target $\eta$ follows the exact same exponent $-(1-q)$ as cold-start escape from the clean mode — the $P_\theta^{-q}$ amplification accelerates commitment to both clean and corrupted supervision by the same multiplicative factor. The only difference is the additional factor $1/\epsilon$ in the denominator: lower noise rates slow down noise commitment, but the $q$-dependent exponent is identical. At $q=0$, the noise term is $\epsilon \tilde{p}^2$ while the clean term is $(1-\epsilon)(1-\tilde{p})^2$, so the clean term dominates even at small $\tilde{p}$ (since $(1-\epsilon)$ is a constant while $\epsilon \tilde{p}^2 \to 0$), meaning noise commitment time diverges to infinity — the $q=0$ model simply ignores label noise entirely. This is the formal justification for the SFT → RLVR switch: SFT at $q=1$ escapes cold start in log time but memorizes noise at the same log rate; RLVR at $q=0$ ignores noise entirely but cannot escape cold start. The pipeline exploits the $q$-dependence of what the model commits to, not just how fast.
Why this symmetric form: The symmetry between clean escape and noise fitting is not a coincidence — it follows directly from the fact that $\dot{p} = p^{2-q} \|s\|^2$ applies for any target label, clean or corrupted, because the $q$-logarithm's derivative $u^{-q}$ multiplies the gradient of the probability assigned to whatever label the supervision says is correct. The amplification factor $P_\theta^{-q}$ is completely agnostic to whether the supervision is correct or not. The only asymmetry comes from the gradient of the probability itself, which points toward the observed label regardless of its truth value.
Gradient Estimation: The Intractability Problem and Ratio Estimators
For the full latent-variable model, the marginal $P_\theta = \sum_z p_\theta(z, y^* \mid x^*)$ is intractable because the latent space $\mathcal{Z}$ (all possible reasoning trajectories) is astronomically large. Both $P_\theta^{-q}$ (the amplification factor) and $\nabla_\theta \ell_1 = -\nabla_\theta \log P_\theta$ (the gradient of the log-marginal) involve sums over this space. The paper uses Monte Carlo sampling to estimate these quantities, drawing $M$ i.i.d. latent trajectories $z^{(1)}, \ldots, z^{(M)} \sim p_\theta(\cdot \mid x^*)$ from the model's prior distribution over rationales.
For each sampled trajectory, two quantities are computed:
where $w_m$ is the likelihood weight — the probability that the correct output $y^*$ is generated given this particular rationale $z^{(m)}$ — and $g_m$ is the per-sample gradient contribution — the gradient of the negative joint log-probability $-\log p_\theta(z^{(m)}, y^* \mid x^*)$, weighted by $w_m$.
What $w_m$ computes: For autoregressive models, $w_m = \prod_{t=1}^T p_\theta(y^*_t \mid x^*, z^{(m)}, y^*_{<t})$ — the product of token-level probabilities of generating the correct output tokens $y^*$, conditioning on the sampled rationale $z^{(m)}$. This is the likelihood that the model would have produced the correct output if we had forced it to use rationale $z^{(m)}$. If the rationale is good (leads to the correct output), $w_m$ is near 1; if bad (leads to a wrong output), $w_m$ is near 0.
What $g_m$ computes: Via the log-derivative trick, $g_m = -\nabla_\theta p_\theta(z^{(m)}, y^* \mid x^*)$ — it is the gradient of the negative joint probability of $(z^{(m)}, y^*)$ under the model. Its expectation over $z \sim p_\theta(\cdot \mid x^*)$ is $-\nabla_\theta \sum_z p_\theta(z, y^* \mid x^*) = -\nabla_\theta P_\theta = \nabla_\theta \ell_0$ — it is an unbiased estimator of the $q=0$ exploitation loss gradient.
The batch means of these quantities are denoted:
where $\bar{w}_M$ is an unbiased estimator of $P_\theta$ (the expected likelihood weight) and $\bar{g}_M$ is an unbiased estimator of $\nabla_\theta \ell_0$ (the $q=0$ gradient).
Why these estimators: The batch means are natural Monte Carlo estimates because $\mathbb{E}[w_m] = P_\theta$ and $\mathbb{E}[g_m] = \nabla_\theta \ell_0$ (both follow from the log-derivative trick: $\nabla_\theta p_\theta(z, y^*) = p_\theta(z, y^*) \nabla_\theta \log p_\theta(z, y^*)$, and summing over $z$ gives $\nabla_\theta P_\theta = \mathbb{E}_{z \sim p_\theta(\cdot \mid x^*)}[p_\theta(y^* \mid x^*, z) \nabla_\theta \log p_\theta(z, y^* \mid x^*)]$, so $\mathbb{E}[g_m] = -\mathbb{E}[w_m \nabla_\theta \log p_\theta(z^{(m)}, y^* \mid x^*)] = -\nabla_\theta P_\theta = \nabla_\theta \ell_0$). The key property is that $\bar{w}_M$ and $\bar{g}_M$ use the same $M$ trajectories — they share the same randomness.
GARL: Gradient-Amplified RL (Section 4.1)
GARL constructs a plug-in estimator for $\nabla_\theta \ell_q$ by combining the unbiased estimators for the two factors in the RL factorization $\nabla_\theta \ell_q = P_\theta^{-q} \cdot \nabla_\theta \ell_0$:
where $\bar{g}_M = \frac{1}{M} \sum_m g_m$ estimates $\nabla_\theta \ell_0$, $\bar{w}_M = \frac{1}{M} \sum_m w_m$ estimates $P_\theta$, and $(\bar{w}_M)^{-q}$ is the plug-in estimate of the amplification factor $P_\theta^{-q}$.
What it computes: GARL draws $M$ trajectories from the prior, computes each trajectory's likelihood weight $w_m$ and gradient contribution $g_m$, averages them to get $\bar{w}_M$ and $\bar{g}_M$, and then amplifies the average gradient by $(\bar{w}_M)^{-q}$. Since $\bar{w}_M$ is between 0 and 1 (it's an average of probabilities), $(\bar{w}_M)^{-q} \geq 1$, and the amplification grows as $\bar{w}_M$ gets smaller. When the model is bad (low $P_\theta$), $\bar{w}_M$ is small, the amplification is large, and the effective gradient magnitude increases — exactly the mechanism that Theorem 3.2 identifies as accelerating cold-start escape.
Why this is a ratio estimator: Because $\bar{g}_M$ and $\bar{w}_M$ are computed from the same set of samples, they are correlated. The expectation of a ratio is not the ratio of expectations: $\mathbb{E}[\bar{g}_M / (\bar{w}_M)^q] \neq \mathbb{E}[\bar{g}_M] / (\mathbb{E}[\bar{w}_M])^q = \nabla_\theta \ell_0 / P_\theta^q = \nabla_\theta \ell_q$. This means GARL is biased at finite $M$ — its expected value is not exactly the true gradient. The bias arises because the same randomness that makes $\bar{g}_M$ an imperfect estimate of $\nabla_\theta \ell_0$ also makes $\bar{w}_M$ an imperfect estimate of $P_\theta$, and the nonlinear division amplifies the noise.
The bias is characterized in Theorem 4.1, which uses a delta-method (Taylor expansion) approach. The leading-order bias for large $M$ is:
where $\operatorname{Var}(w_m)$ is the variance of the likelihood weight across trajectories, $\operatorname{Cov}(g_m, w_m)$ is the covariance between the per-sample gradient and its weight, and $\nabla_\theta \ell_1 = -\nabla_\theta \log P_\theta$ is the density-estimation pole gradient.
What it computes: The bias is proportional to $q$ (higher $q$ gives more bias), inversely proportional to $M$ (more samples reduce bias), and inversely proportional to $P_\theta^{q}$ (after simplifying the bracketed term, which the paper shows is $O(P_\theta)$ under bounded-score assumptions, giving the final simplified form $O(q / M P_\theta^q)$). The bracketed term represents the relative variability of the estimator components: $\operatorname{Var}(w_m)$ measures how much the likelihood weights vary across trajectories (high variance means some are good and some are bad), and $\operatorname{Cov}(g_m, w_m)$ measures whether high-weight trajectories tend to have larger or smaller gradients (a systematic correlation between quality and gradient magnitude).
Why this bias form matters: The bias shares the same $P_\theta^{-q}$ scaling as the cold-start amplification. This means that in cold start, where $P_\theta$ is tiny and the amplification is most needed, the estimator quality is also worst — the bias grows as $O(1/P_\theta^q)$. This creates a cold-start escape vs. estimator quality tradeoff: $q=1$ gives the fastest escape ($\Theta(\log(1/p_0))$) but the largest bias ($O(1/MP_\theta)$), while $q=0$ gives zero bias (the estimator is exactly unbiased because $\bar{g}_M / (\bar{w}_M)^0 = \bar{g}_M$ is a simple sample mean) but the slowest escape ($\Omega(1/p_0)$). At $q=0$, the bias term vanishes entirely because $q=0$ in the prefactor $q/(M P_\theta^{q+1})$ — the plug-in estimator reduces to the unbiased sample mean. Intermediate $q$ provides partial amplification with proportionally reduced bias. This tradeoff is directly visible in the cold-start experiments: $q=0.75$ achieves higher final accuracy than $q=1$ despite escaping slightly later (Table 1; Figure 2a in Appendix G).
Control variate via RLOO. The paper applies the standard leave-one-out control variate [Kool et al., 2019] to reduce variance without affecting the bias. The raw plug-in estimator $\bar{g}_M / (\bar{w}_M)^q$ can be expanded via the chain rule:
where the first term is the score-function component (gradient of the log-probability of the rationale) and the second term is the pathwise component (gradient of the likelihood weight itself, which can be computed via reparameterization or the log-derivative trick applied to the output tokens).
What this decomposition achieves: Only the score-function component has expectation zero under the prior — $\mathbb{E}_{z \sim p_\theta}[\nabla_\theta \log p_\theta(z \mid x^*)] = 0$ by the standard score-function identity. This means we can subtract any baseline (any scalar that doesn't depend on $z^{(m)}$) from the score-function weight $-w_m / (\bar{w}_M)^q$ without biasing the estimator, provided the baseline also has zero expectation. The RLOO baseline uses $-(\bar{w}_{\neg m})^{1-q}$ where $\bar{w}_{\neg m} = \frac{1}{M-1} \sum_{j \neq m} w_j$ is the leave-one-out mean. The centered weight becomes:
and the RLOO estimator is:
What this computes: For each trajectory, the score-function term is weighted by $c_m$ (which can be positive or negative, giving a valid advantage function), and the pathwise term is weighted by the batch-level amplification $(\bar{w}_M)^{-q}$. The subtraction of $(\bar{w}_{\neg m})^{1-q}$ ensures that trajectories with below-average weights get negative centered weights (pushing the model away from those rationales), while above-average weights get positive centered weights (pulling the model toward those rationales). Proposition E.1 proves that this control variate preserves the bias expansion of Theorem 4.1 exactly.
Endpoint recovery. At $q=0$, the centered weight simplifies to $w_m - \bar{w}_{\neg m}$, and GARL recovers exactly the Rao-Blackwellized RLOO estimator of Zhou et al. [2026]. At $q=1$, the centered weight becomes $w_m / \bar{w}_M - 1$, which is a self-normalizing importance weight centered at zero, and GARL recovers the IWAE gradient estimator [Burda et al., 2015]. This endpoint recovery is formalized in Proposition E.2.
Practical normalization. The per-sample weight $w_m / (\bar{w}_M)^q$ can reach magnitude up to $M^q$ (when $w_m \approx 1$ and $\bar{w}_M \approx 1/M$). To keep the advantage signal uniformly bounded as $q$ varies, Algorithms 1 and 2 divide the final gradient by $M^q$, yielding $c_m / M^q \in [-1, 1]$. This is equivalent to applying a $q$-independent learning rate to the bounded-advantage form, rather than a $q$-dependent learning rate to the unscaled form. The mathematical estimators target $\nabla_\theta \ell_q$ directly; the $1/M^q$ rescaling in the algorithm is a practical choice to maintain consistent optimization dynamics across different $q$ values.
PAFT: Posterior-Attenuated Fine-Tuning (Section 4.2)
PAFT exploits the FT factorization $\nabla_\theta \ell_q = P_\theta^{1-q} \nabla_\theta \ell_1$. The endpoint gradient $\nabla_\theta \ell_1 = -\nabla_\theta \log P_\theta$ can be expressed as a posterior expectation:
where the expectation is over the posterior distribution over rationales — trajectories that are consistent with the observed correct output $y^*$ — rather than the prior distribution over arbitrary trajectories.
What this means: The gradient of $-\log P_\theta$ is the average negative gradient of the log-joint probability, where the average is taken over trajectories weighted by how well they explain the observed output. This is fundamentally different from GARL's prior expectation: GARL averages over all trajectories and uses the likelihood weight $w_m$ to upweight good ones, while PAFT (in principle) only considers trajectories that are already consistent with $y^*$ — no reweighting needed because the sampling distribution already favors good trajectories.
Substituting into the FT factorization gives:
where the amplification factor is now $P_\theta^{1-q} \in [0, 1]$ — an attenuation rather than an amplification. When $P_\theta$ is small, $P_\theta^{1-q}$ is very small (for $q<1$), suppressing the gradient. When $P_\theta$ is near 1, $P_\theta^{1-q} \approx 1$ and PAFT approximates standard SFT on posterior samples.
Why this attenuation matters: The attenuation factor $P_\theta^{1-q}$ acts as an automatic curriculum: hard instances (low $P_\theta$) get heavily downweighted, so the model only receives strong gradients from instances it already has some chance of getting right. This makes PAFT unusable at cold start (where $P_\theta \approx 0$ gives vanishing gradients) but potentially more stable in warm start, because it avoids the large, noisy updates that GARL's amplification $P_\theta^{-q}$ produces on hard examples.
Approximate posterior sampling via importance resampling. The true posterior $p_\theta(z \mid x^*, y^*)$ is intractable because it requires normalizing by $P_\theta = \sum_z p_\theta(z, y^* \mid x^*)$. PAFT approximates it using importance resampling [Rubin, 1988] from the same prior sample pool used by GARL. Given the $M$ prior trajectories and their likelihood weights $w_m$:
- Compute the normalized importance weights
$w_m / \sum_j w_j$. - Draw
$K$indices$r_1, \ldots, r_K \in \{1, \ldots, M\}$with replacement, where index$m$is selected with probability proportional to$w_m$.
The resampled trajectories are an approximate sample from the posterior: high-weight trajectories (those likely to produce the correct output) get resampled many times; low-weight trajectories get resampled few times or not at all. The PAFT estimator is then:
where $(\bar{w}_M)^{1-q}$ is the plug-in estimate of the attenuation factor $P_\theta^{1-q}$, and the sum is standard SFT (teacher forcing) on the $K$ resampled trajectories.
What it computes: PAFT reuses the same $M$ prior samples as GARL, but instead of computing per-sample gradient contributions $g_m$ and averaging them with weights, it first resamples trajectories from the importance-weighted distribution (favoring good rationales), then applies standard teacher-forcing gradient computation on those resampled trajectories, and finally multiplies by the attenuation factor $(\bar{w}_M)^{1-q}$. The teacher-forcing gradient $\nabla_\theta \log p_\theta(z, y^* \mid x^*)$ is computed by running the model on the (input, sampled rationale) context and computing the cross-entropy loss of the output tokens $y^*$ — this is exactly what SFT does, except the rationales are sampled rather than human-annotated.
Why resampling rather than reweighting: The key difference from GARL is that PAFT excludes bad trajectories before the gradient is formed. GARL computes $g_m$ for every trajectory (including incoherent ones) and weights them by $w_m / (\bar{w}_M)^q$ — but the per-sample gradient $g_m = -w_m \nabla_\theta \log p_\theta(z^{(m)}, y^* \mid x^*)$ for a bad trajectory still involves the log-gradient of that trajectory, which may be noisy or point in pathological directions. PAFT only computes gradients on trajectories that are likely to be coherent (because they survived the importance resampling filter). The paper hypothesizes that this filtering is why PAFT remains stable on HotPotQA and MuSiQue where GARL collapses: GARL's pathwise gradient term $-\nabla_\theta w_m / (\bar{w}_M)^q$ updates $p_\theta(y^* \mid x^*, z^{(m)})$ on every sampled trajectory, including incoherent ones, potentially corrupting the output-likelihood model; PAFT only updates on resampled coherent trajectories.
Bias and variance. Proposition E.3 proves that PAFT has the same expectation as GARL — conditioning on the sample pool and taking the expectation over the resampling step yields exactly the GARL estimate, so the outer expectation (over the pool) gives the same bias expansion. Proposition E.4 proves that PAFT has strictly higher variance than GARL by the law of total variance: $\operatorname{Var}(\hat{\nabla}^{\text{PAFT}}) = \operatorname{Var}(\hat{\nabla}^{\text{GARL}}) + \mathbb{E}[\operatorname{Var}(\hat{\nabla}^{\text{PAFT}} \mid \text{pool})]$, where the second term is the additional variance from resampling noise. So PAFT is theoretically worse than GARL in both bias and variance — yet empirically more stable on two of three benchmarks. This paradox is a key finding: the "stability" that matters in practice is not captured by these asymptotic metrics, and likely relates to the pathwise-term corruption hypothesis or benchmark-specific overfitting patterns.
Endpoint recovery. At $q=1$, the attenuation vanishes ($(\bar{w}_M)^0 = 1$) and PAFT recovers the EM gradient update with E-step posterior samples, which is also the TRICE estimator [Phan et al., 2023]. At $q=0$, PAFT reduces to posterior-resampled SFT scaled by $\bar{w}_M \approx P_\theta$, which so aggressively downweights hard instances that it is "overly conservative in practice" per the paper and does not correspond to a standard method.
Drop-in compute cost. Both GARL and PAFT are designed as drop-in replacements for standard RLVR estimators at the same rollout budget $M$. GARL adds only an $O(M)$ scalar reweighting on top of RB-RLOO (computing the batch mean $\bar{w}_M$ and applying the power $q$). PAFT adds one categorical resample (sampling $K$ indices from a multinomial) followed by teacher-forcing on already-generated tokens — no extra forward passes through the model beyond those already used to generate the $M$ trajectories. The resampling step is trivially parallelizable and, for $K = M$ (the setting used in experiments), uses the same number of gradient computations as GARL.
Summary of Design Choices and Their Justifications
-
Tsallis
$q$-logarithm over alternative deformations: Chosen because its derivative is exactly$u^{-q}$, giving the clean multiplicative gradient factorization$\nabla_\theta \ell_q = P_\theta^{-q} \nabla_\theta \ell_0$that enables both the convergence analysis and the dual estimator design. Alternative deformations (e.g., convex combinations) would not produce this multiplicative structure. -
Gradient flow (continuous-time, single-example) over discrete stochastic optimization: Chosen to isolate the role of the amplification factor
$P_\theta^{-q}$without confounding effects from step-size schedules, mini-batch noise, adaptive optimizers, and multi-example interactions. The analysis explicitly trades realism for clarity — the$\dot{p} = p^{2-q} \|s\|^2$dynamics capture the essential$q$-dependence that persists in the full stochastic setting. -
Ratio estimator (
$\bar{g}_M / (\bar{w}_M)^q$) over separate estimation: Reusing the same samples for$\bar{g}_M$and$\bar{w}_M$introduces bias ($O(q/MP_\theta^q)$) but avoids the variance blowup that would come from independent sample sets. At$q=0$, the bias vanishes entirely, and at$q=1$, the estimator reduces to the well-studied self-normalizing importance sampling estimator. -
Importance resampling (PAFT) over rejection sampling: The STaR approach [Zelikman et al., 2022] rejects all incorrect trajectories — a hard acceptance rule. Importance resampling instead samples with replacement proportional to weights, which preserves more diversity and provides a known bias characterization (Proposition E.3). PAFT's
$K = M$resampling keeps the same computational budget while providing the stability benefits of posterior filtering. -
RLOO control variate over simpler baselines: The leave-one-out baseline
$(\bar{w}_{\neg m})^{1-q}$is designed to exactly match the$q$-dependence of the weight coefficient, ensuring that at$q=0$it recovers the standard RLOO advantage$w_m - \bar{w}_{\neg m}$and at$q=1$it recovers the self-normalizing importance weight$w_m / \bar{w}_M - 1$. Simpler baselines (e.g., constant zero) would not provide this seamless interpolation. -
Algorithmic
$1/M^q$rescaling over unnormalized estimation: Keeps per-sample advantage$c_m / M^q \in [-1, 1]$bounded uniformly in$q$, enabling a single learning rate schedule across all$q$values. Without this rescaling, the effective learning rate would vary by factors of$M^q$(up to$M$at$q=1$, which would be 32× different for the$M=32$experiments), confounding comparisons across$q$. -
Fixed
$q$in experiments over annealed schedules: The paper deliberately tests fixed$q$values (0, 0.25, 0.5, 0.75, 1) as a first-step validation of the continuum concept. Annealed schedules that smoothly decrease$q$from 1 to 0 over training — directly instantiating the SFT-then-RLVR intuition within a single loss — are flagged as future work. The fixed-$q$experiments establish the baseline behavior that annealing would need to interpolate between.
4. Key Insights and Innovations
Innovation 1: Unifying SFT and RLVR as Endpoints of a Single Commitment Continuum
The paper's most fundamental contribution is not a new training algorithm but a conceptual reframing that explains why the established SFT-then-RLVR pipeline works — by showing that it corresponds to traversing a single continuous parameter $q$ from 1 to 0 along the JQ loss family. This is a genuine intellectual shift, not an incremental refinement.
What the field assumed before: SFT and RLVR were understood as categorically different training paradigms with distinct objectives, optimization procedures, and theoretical properties. SFT maximizes log-likelihood of annotated rationales under a teacher-forcing objective; RLVR maximizes expected reward via policy-gradient methods like REINFORCE or GRPO. The connection between them — that SFT on rationales and RL on outcomes could be combined sequentially — was discovered empirically, but the theoretical basis for the ordering was absent. The field operated with a tacit dichotomy: you either do likelihood-based training (SFT, MLE) or reward-based training (RL, policy gradients), and you sequence them because that's what works.
What this paper shows instead: The JQ continuum reveals that both SFT and RLVR are extreme points of a single loss family parameterized by a commitment scalar $q$. At $q=1$, the loss is $-\log P_\theta$ — exactly the negative log-marginal-likelihood that SFT optimizes when the latent rationale is fixed to the annotated trace. At $q=0$, the loss is $1 - P_\theta$ — exactly $1 - \mathbb{E}[\text{reward}]$ under exact-match supervision (Proposition B.1), which is what RLVR minimizes. Every intermediate $q$ defines a valid loss that blends properties of both extremes. The standard pipeline is therefore revealed as a stepwise $q=1 \to 0$ schedule: start at the density-estimation pole to leverage maximum gradient amplification ($P_\theta^{-1}$), then switch to the exploitation pole once the model escapes cold start, gaining noise robustness and unbiased gradient estimation.
Why this reframing matters beyond the pipeline explanation: It transforms the design space for post-training from a discrete choice between qualitatively different method classes into a continuous optimization over commitment. Rather than asking "should I use SFT or RLVR?", a practitioner can now ask "what $q$ schedule should I use?" — a question that admits principled answers based on per-instance $P_\theta$, estimator quality, and noise conditions. The SFT-then-RLVR recipe that required careful engineering (when to switch? how much SFT? what RL algorithm?) becomes an instance of a more general design pattern.
Relationship to prior RL-MLE bridges: Prior work connecting RL and MLE [Levine, 2018; Norouzi et al., 2016; Guu et al., 2017] presented them as distinct frameworks that could be related through probabilistic inference — e.g., RL maximizes a reward-weighted likelihood, MLE maximizes an unweighted likelihood. The JQ continuum goes further: it embeds them as endpoints of a single objective with identical per-example gradient direction (Proposition 2.2). This is a stronger claim than "they are related" — it asserts they are literally the same optimization problem under different loss transformations, differing only in how aggressively each example's gradient is amplified.
The gradient factorization as unifying mechanism: The key insight that enables this unification is Proposition 2.2: $\nabla_\theta \ell_q = P_\theta^{-q} \nabla_\theta \ell_0 = P_\theta^{1-q} \nabla_\theta \ell_1$. This factorization states that all members of the JQ family share the same direction of gradient per example — the model is always being pushed to increase $P_\theta$ on every example, regardless of $q$. The only difference is the magnitude of that push, controlled by the amplification factor $P_\theta^{-q}$. This reveals that the apparent categorical difference between SFT and RLVR — likelihood vs. reward — is an artifact of the endpoint losses, not a fundamental distinction in what the model learns. At every $q$, the model is learning to assign higher probability to correct outputs; the $q$ parameter only controls how hard it pushes on examples it currently gets wrong.
Evidence anchoring: The cold-start experiments in Table 1 provide the most direct validation of this conceptual move: GARL at intermediate $q=0.75$ achieves non-trivial cold-start performance (e.g., 30.5% p@1 on FinQA, 53.4% on HotPotQA) where both endpoints fail — $q=0$ (RB-RLOO) and GRPO get zero, and $q=1$ escapes but with degraded final accuracy. This demonstrates that the continuum is not just theoretically elegant; intermediate points on it have practical value that neither endpoint alone provides. The warm-start results in Table 2 further validate that different benchmarks and training regimes benefit from different points on the continuum — GARL at $q=0.25$ on FinQA, PAFT at $q=0.75$ on HotPotQA — confirming that the $q$ parameter captures meaningful variation in training dynamics beyond what endpoint methods offer.
Scope of the contribution: This is a foundational theoretical advance disguised as an empirical methods paper. It doesn't claim to outperform the standard pipeline — indeed, the best cold-start GARL results ($q=0.75$) match warm-start GRPO rather than dramatically exceeding it. Its contribution is providing the intellectual infrastructure to reason clearly about why the standard pipeline is structured as it is, and what alternative structures (intermediate $q$, annealed schedules, $q$-dependent estimator selection) become accessible once you recognize the continuum.
Innovation 2: Cold-Start Escape as a Deterministic Gradient Magnitude Problem, Not a Variance Problem
The paper delivers a sharp diagnostic correction to the prevailing understanding of cold-start failure in RL-based reasoning model training. Prior to this work, the dominant intuition was that RLVR stalls at cold start because of reward sparsity — the model never samples a correct output, so all rewards are zero, and the gradient vanishes. This framing motivated variance-reduction techniques like Rao-Blackwellization [Zhou et al., 2026], which analytically marginalize over the output to produce non-zero gradient estimates even when all sampled outputs are incorrect. But the paper's cold-start experiments deliver a decisive empirical result: RB-RLOO (GARL at $q=0$) still gets zero accuracy in cold start (Table 1: 0% on all three benchmarks at both model scales). Variance reduction is insufficient because the bottleneck is not variance — it's gradient magnitude.
The diagnostic move: The paper isolates the failure to a specific mathematical quantity — the amplification factor $P_\theta^{-q}$ in the gradient $\nabla_\theta \ell_q = P_\theta^{-q} \nabla_\theta \ell_0$. At $q=0$, this factor is identically 1, so the gradient magnitude is $\|\nabla_\theta \ell_0\| = \|-\nabla_\theta P_\theta\|$. When $P_\theta$ is, say, $10^{-6}$, the gradient is proportional to $P_\theta^2 \approx 10^{-12}$ in the squared norm (from $\dot{p} = p^2 \|s\|^2$ at $q=0$). This is not a noisy estimate of a large gradient — it's a clean, low-variance estimate of a tiny gradient. No amount of variance reduction can fix a signal that is structurally small.
This shift in diagnosis — from "the gradient is noisy" to "the gradient is too small" — is a fundamental reconceptualization of the cold-start problem. It explains why Rao-Blackwellization, despite being mathematically elegant and demonstrably variance-reducing, provides no practical benefit for cold-start escape: it addresses the wrong bottleneck.
Contrast with concurrent approaches: Tajwar et al. [2026] propose MaxRL, an RL-to-ML continuum via Maclaurin truncation of $\log p$. Their estimator is unbiased for the truncated objective but is exactly zero when no sample succeeds — the precise scenario where gradient help is needed most. The JQ approach is fundamentally different: the likelihood weight $w_m$ is never zero because $p_\theta(y^* \mid x^*, z^{(m)})$ is the model's assessed probability of the correct output given the sampled rationale, not a binary success/failure signal. Even when the model wouldn't have generated the correct output, it can still assess that the correct output is compatible with a given rationale (e.g., a partially correct reasoning chain), yielding $w_m > 0$. This means GARL always has a non-zero gradient signal, and the question is only the signal's magnitude — which $q$ controls.
The theoretical separation via Theorems 3.1 and 3.2: The paper formalizes the magnitude problem through gradient flow analysis of $\dot{p} = p^{2-q} \|s(\theta)\|^2$. The separation between $q=0$ and $q=1$ is not a small constant factor — it is exponential in $1/p_0$. At $p_0 = 10^{-6}$, the factor is approximately $10^6 / \log(10^6) \approx 70,000\times$ in escape time. This means that for sufficiently small initial $P_\theta$ (which is typical at cold start, especially for challenging reasoning tasks), the $q=0$ pole is not just slow — it is provably impossible to escape within any realistic training budget. The $\Omega(1/p_0)$ lower bound in Theorem 3.1 requires only the mild assumption of bounded score norm, making it a very general result. This provides the first theoretical explanation for the empirically observed cold-start failure of RLVR that is grounded in the loss function's mathematical structure rather than in implementation details like reward sparsity or exploration strategies.
The dual diagnosis for noise fitting (Proposition D.2): The same mathematical machinery yields the symmetric insight: the amplification factor $P_\theta^{-q}$ accelerates commitment to both clean and corrupted supervision. At $q=0$, noise-fitting time diverges to infinity (the model ignores label errors); at $q=1$, noise is memorized at the same $\Theta(\log(1/\tilde{p}_0))$ rate as clean cold-start escape. This formalizes the intuition that SFT "memorizes" and RL "generalizes" [Chu et al., 2025] into precise dynamical rates. It also reveals that the SFT-then-RLVR pipeline exploits not just the speed difference between poles, but the divergence in what the model commits to — the hard $q=1 \to 0$ switch first uses amplification to escape, then eliminates amplification to halt noise commitment.
Evidence anchoring: The cold-start experiments in Table 1 provide the cleanest test of this diagnostic claim. The sharp threshold behavior — $q \leq 0.5$ fails completely, $q \geq 0.75$ escapes at 0.6B; $q \geq 0.85$ required at 8B — is consistent with Theorem 3.1's lower bound: as $q$ decreases, the required escape time grows as $p_0^{-(1-q)}/(1-q)$, so at fixed training budget there exists a critical $q$ below which escape fails. The scale-dependent threshold (8B requiring higher $q$ than 0.6B) is consistent with a lower effective initial $P_\theta$ at larger scale (plausible if larger models have more diffuse initial distributions over the larger output space), though the paper does not directly measure this. Figure 2a visualizes the dynamics: the maximum amplified advantage $c_m/M^q$ remains flat for $q \leq 0.5$ and spikes for $q=0.75$ around step 35, exactly as the theory predicts — gradient amplification, not reduced variance, drives the escape.
Scope and limitations: The theoretical analysis is stylized (single-example, gradient flow, bounded score), so it captures the mechanism but not the full complexity of multi-example stochastic optimization with adaptive optimizers. The claim is not that the $\Theta$ rates are exact for real training — it's that the ordering between poles and the exponential separation are robust structural properties of the loss family that manifest even when the analysis assumptions are violated. This is a significant intellectual contribution: it replaces the vague intuition "RL doesn't work from cold start because there's no reward signal" with a precise, quantitative account of when it fails, why it fails, and what parameter controls the failure boundary.
Innovation 3: The Escape-Speed vs. Estimator-Quality vs. Noise-Robustness Trilemma
The paper identifies and formalizes a three-way tradeoff that governs all fixed-$q$ optimization on the JQ continuum. This is not a new algorithm or a new theorem, but a diagnostic framework that explains the empirical pattern of results — why no single $q$ dominates, why the optimal $q$ depends on the training regime, and why the standard pipeline's hard $1 \to 0$ switch is a heuristic for navigating this trilemma rather than a fundamental optimum.
The three competing forces:
-
Cold-start escape speed (Section 3): Governed by the amplification factor
$P_\theta^{-q}$. Escape time scales as$\Theta(p_0^{-(1-q)}/(1-q))$for$q<1$and$\Theta(\log(1/p_0))$at$q=1$. Higher$q$is strictly better for escape speed — the$q=0$pole is provably glacial,$q=1$is fast. -
Estimator quality (Theorem 4.1): Governed by the ratio-estimator bias
$O(q/MP_\theta^q)$. Higher$q$and lower$P_\theta$both degrade the estimator — the bias scales with the same$P_\theta^{-q}$factor that enables fast escape. At$q=0$, the estimator is unbiased (no amplification to estimate); at$q=1$, the bias is$O(1/MP_\theta)$, which can be enormous in cold start. -
Noise robustness (Proposition D.2): Governed by whether the model commits to corrupted supervision. Noise-fitting time scales as
$\Theta(\tilde{p}_0^{-(1-q)}/((1-q)\epsilon))$for$q \in (0,1)$and diverges to infinity at$q=0$(the model ignores label noise entirely). Lower$q$is strictly better for noise robustness.
What makes this a genuine trilemma rather than a simple tradeoff: The same mechanism — the amplification factor $P_\theta^{-q}$ — drives all three effects, and it cannot be disentangled. You cannot have fast cold-start escape without also amplifying estimator bias and noise commitment, because all three are proportional to the same $P_\theta^{-q}$ factor. The standard pipeline's hard switch is one solution: use $q=1$ only long enough to escape cold start, then switch to $q=0$ to eliminate bias and halt noise fitting. But this leaves efficiency on the table — intermediate $q$ might provide enough escape speed while better managing bias in the post-escape phase.
The warm-start pattern as evidence for the trilemma: On FinQA, where training is stable (no collapse), GARL at $q=0.25$ achieves the best m@16 (38.7, vs. 38.3 at $q=0$ and 37.6 at $q=0.75$). This is consistent with the trilemma: in warm start, escape speed is less critical (Pθ is already non-trivial), so the optimal $q$ shifts toward lower values to reduce estimator bias, but not all the way to $q=0$ because some amplification still helps on the hardest examples. On HotPotQA and MuSiQue, where GARL collapses at all tested $q$, PAFT at $q=0.75$ delivers the best stable results (47.9 and 22.4 m@16). This is consistent with the trilemma shifted by estimator stability: GARL's lower variance turns into destructive updates when latent-rationale variance is high; PAFT's higher per-step variance is offset by its posterior filtering, which keeps gradients pointed at coherent rationales.
The significance of formalizing this trilemma: Prior work addressed the cold-start problem through ad hoc solutions — warm-start with SFT, use shaped rewards, add exploration bonuses, reduce variance. Each solution addressed one facet of the problem in isolation. The trilemma framework reveals that these facets are intrinsically coupled through a single parameter, and that any fixed-point solution must compromise. This reframes the problem from "how do we fix cold start?" to "how do we navigate the escape-bias-noise tradeoff dynamically?" — a fundamentally different question that motivates scheduled $q$ approaches rather than point solutions.
Evidence anchoring: The cold-start threshold behavior (Table 1: $q=0.75$ escapes but $q=0.5$ does not) validates the escape-speed axis. The $q=0.75$ vs. $q=1$ comparison in cold start (e.g., FinQA p@1: 30.5 vs. 21.9) validates the estimator-quality axis — $q=1$ escapes faster (Figure 2a: immediate spike) but achieves lower final accuracy because its higher bias degrades the quality of updates post-escape. The noise-fitting analysis (Proposition D.2) is not directly validated in the experiments (which use clean supervision), but it provides the theoretical foundation for the SFT → RLVR switch in the standard pipeline and predicts that training with $q<1$ would be more robust to label noise, a testable hypothesis left to future work.
Scope: This is a conceptual framework contribution rather than an algorithmic one. It doesn't solve the trilemma — it names it and provides the analytical tools to reason about it. The three-phase recipe proposed in Section 6 (GARL high $q$ for cold start, GARL low $q$ where stable, PAFT $q \geq 0.75$ otherwise) is a first heuristic derived from the framework, not a validated solution. The value is in providing a language and a set of concepts — escape speed, estimator bias, noise commitment — that are all expressible in terms of the same parameter $q$ and the same amplification factor $P_\theta^{-q}$, enabling systematic reasoning about training dynamics that previously relied on empirical intuition.
Innovation 4: Dual Gradient Factorization as a Design Principle for Estimators
The paper demonstrates that the mathematical structure of the JQ loss family — specifically, the dual gradient factorization $\nabla_\theta \ell_q = P_\theta^{-q} \nabla_\theta \ell_0 = P_\theta^{1-q} \nabla_\theta \ell_1$ — is not merely a theoretical curiosity but a design principle that mechanically generates complementary gradient estimators with distinct stability properties. This is an insight about how theory can guide algorithm design: the factorization doesn't just explain existing methods; it tells you exactly what quantities to estimate and suggests two natural estimation strategies based on which endpoint gradient you choose to target and how you handle the intractable $P_\theta$ power.
What's novel about this as a design principle: Prior work on gradient estimation for latent-variable models developed specific estimators for specific objectives: REINFORCE for expected reward, IWAE for the marginal log-likelihood, EM for maximum likelihood with posterior samples. Each estimator was derived separately for its objective, and the relationships between them — e.g., that IWAE at $M \to \infty$ targets the same gradient as EM — were discovered post-hoc. The JQ factorization inverts this: it starts from the shared gradient direction and shows that the only difference between objectives is a scalar power of $P_\theta$, which immediately suggests two estimation strategies — amplify the prior-based gradient (GARL) or attenuate the posterior-based gradient (PAFT). The estimators are not independently invented and then unified; they are mechanically derived from the factorization.
The dual factorization as a completeness result: The factorization $\nabla_\theta \ell_q = P_\theta^{-q} \nabla_\theta \ell_0 = P_\theta^{1-q} \nabla_\theta \ell_1$ means that any valid gradient estimator for JQ must effectively estimate either $P_\theta^{-q}$ times the $q=0$ gradient (the RL path) or $P_\theta^{1-q}$ times the $q=1$ gradient (the FT path) — there are no other options because these two factorizations span the space. GARL and PAFT are therefore not arbitrary choices among many possible estimators; they are the natural realizations of the only two estimation strategies compatible with the loss structure. Any future estimator for JQ will necessarily fall into one of these two families (prior-based amplification or posterior-based attenuation) or will combine them.
The stability-variance paradox as a cautionary insight: Proposition E.4 proves that PAFT has strictly higher variance than GARL by the law of total variance. Theorem 4.1 gives both estimators the same leading-order bias. By standard statistical criteria, GARL dominates PAFT — it is uniformly better in both bias and variance. Yet the experiments reveal the opposite pattern: PAFT remains stable on HotPotQA and MuSiQue where GARL collapses to zero validation accuracy (Figure 2b). This is a significant finding not because PAFT is "better" — it isn't, on the metrics theory says matter — but because it reveals that the relevant notion of "stability" for reasoning model training is not captured by asymptotic bias and variance of the per-step gradient estimate. Something about GARL's gradient composition — the paper hypothesizes pathwise-term corruption when updating $p_\theta(y^* \mid x^*, z)$ on every sampled trajectory, including incoherent ones — causes catastrophic training dynamics that bias and variance alone cannot predict.
This is an important negative-capability result: the paper identifies a failure mode of its own theoretically-superior estimator and uses that failure to motivate the complementary estimator. Rather than claiming GARL is universally better and dismissing the collapse as a tuning issue, the paper treats the collapse as informative — it tells us something about what matters for stable training that our current theoretical tools don't capture. This intellectual honesty strengthens the contribution: the dual factorization is not presented as a solved problem but as a design space with known structure and unknown dynamics.
Endpoint recovery as validation of the design principle: The fact that GARL recovers RB-RLOO at $q=0$ and IWAE at $q=1$, while PAFT recovers the EM gradient update at $q=1$ and a novel conservative estimator at $q=0$ (Proposition E.2), is not a coincidence — it validates that the factorization correctly identifies the common structure underlying these seemingly disparate estimators. RB-RLOO, IWAE, and EM were developed for different objectives (expected reward, marginal log-likelihood, marginal log-likelihood via posterior samples). The JQ factorization shows they are all instances of the same estimation strategy (prior-based or posterior-based) applied at different $q$, and that intermediate $q$ yields previously unknown estimators that interpolate between them.
Evidence anchoring: The warm-start results in Table 2 provide the key evidence for the dual-estimator design principle. On FinQA, where GARL is stable, GARL at $q=0.25$ achieves the best m@16 (38.7). On HotPotQA, where GARL collapses at every tested $q$, PAFT at $q=0.75$ achieves the best m@16 (47.9, +13.9 over GRPO). On MuSiQue, PAFT at $q=0.75$ achieves the best stable result (22.4), with GARL peaking higher but collapsing (24.3 peak-before-collapse). This pattern validates the core design claim: no single estimator dominates, and the dual factorization provides complementary tools for different stability regimes. The three-phase recipe proposed in Section 6 — GARL at high $q$ for cold start, GARL at low $q$ where stable, PAFT at $q \geq 0.75$ where GARL destabilizes — is a direct operationalization of the dual-estimator design space.
Relationship to prior estimator development: Prior work on gradient estimators for latent-variable models developed one estimator for one objective at a time: REINFORCE with baselines for RL, IWAE for tighter marginal likelihood bounds, TRICE for variational EM. Each required separate derivation, separate variance analysis, and separate empirical validation. The JQ factorization provides a unified derivation framework: start from the same per-example gradient direction, choose RL or FT endpoint, plug in the $P_\theta$ power, and you get a family of estimators parameterized by $q$. The paper doesn't claim GARL and PAFT are individually novel — they recover known methods at the endpoints — but rather that the systematic derivation from a shared structure is novel, and that this derivation immediately yields previously unknown intermediate-$q$ estimators with practical value (GARL at $q=0.75$ for cold start, PAFT at $q=0.75$ for stable warm start).
Scope: This is a methodological contribution in estimator design — not in the sense of proposing a single new estimator, but in demonstrating that a simple mathematical structure (the dual factorization) can generate a family of estimators whose properties vary continuously with a single parameter, and that different points in this family are optimal for different training regimes. The stability-variance paradox reveals a gap in our theoretical understanding that is itself a contribution: it identifies where existing theory (asymptotic bias, variance) fails to predict practical outcomes, pointing toward the need for new analytical tools that capture training instability.
5. Experimental Analysis
Evaluation Methodology
-
Dataset. The paper uses three reasoning benchmarks: FinQA [Chen et al., 2021] (numerical reasoning over financial data), HotPotQA [Yang et al., 2018] (multi-hop question answering), and MuSiQue [Trivedi et al., 2022] (multi-hop questions via single-hop composition). Subsets are sampled from Huggingface, filtered to keep instances whose pre-tokenization input length falls below caps of 8000, 4000, and 10000 characters respectively, yielding train/val/test sizes of 6145/872/1132 (FinQA), 9067/342/343 (HotPotQA), and 9985/579/445 (MuSiQue).
-
Base model(s). All experiments use post-trained Qwen 3 models [Yang et al., 2025] at two scales: 0.6B and 8B parameters. The paper argues these are contemporary, representative models with non-trivial baseline reasoning capability (sufficient for warm-start comparisons) but far from saturation on the benchmarks used. The 0.6B model is the primary testbed; 8B experiments validate scale-dependent phenomena in cold start.
-
Metrics. Three evaluation metrics are reported at test time using 16 samples per prompt: p@1 (single-sample pass accuracy — fraction of first samples that are correct), p@16 (best-of-16 — fraction of prompts where at least one of 16 samples is correct, measuring coverage), and m@16 (majority vote over 16 samples — majority-consensus accuracy [Wang et al., 2023], measuring the ability to identify the correct answer from a pool of candidates). Training uses exact-match rewards (
R(ŷ, y*) = I(ŷ = y*)); evaluation uses a relaxed substring match (correct if the ground-truth answery*appears as a substring of the generated outputŷ). Test numbers are taken from the checkpoint with the highest validation m@16. -
Baselines. Three primary baselines are used: (1) GRPO [DeepSeek-AI, 2025, Shao et al., 2024] — the standard RLVR algorithm used in reasoning model post-training, evaluated in both cold-start and warm-start configurations; (2) RB-RLOO (Rao-Blackwellized REINFORCE Leave-One-Out) [Zhou et al., 2026] — recovered by GARL at
q=0, representing the verifier-free variant of REINFORCE with analytic reward marginalization; (3) Base (no training, prompted) — the un-adapted Qwen 3 0.6B model evaluated on prompted inputs without any fine-tuning, providing a zero-shot reference point for warm-start tables. GARL and PAFT at variousqvalues are compared against GRPO as the primary practical baseline throughout. -
Generation budget / compute accounting. All methods use the same number of latent trajectories
Mper prompt during training:M = 32for Qwen 3 0.6B experiments,M = 16for 8B experiments. At evaluation, all methods use 16 samples per prompt. The compute cost of GARL and PAFT is characterized as drop-in equivalent to standard RLVR estimators at the same rollout budget: GARL addsO(M)scalar reweighting on top of RB-RLOO (computingẇMand applying the powerq); PAFT adds one categorical resample followed by teacher forcing on already-generated tokens. Neither requires extra forward passes through the model beyond those used to generate theMtrajectories. Per-rationale token budgets cap generation length (FinQA: 4096, HotPotQA: 3072, MuSiQue: 2048, plus 128 tokens for the answer), following Muennighoff et al. [2025] to force the thinking-end token once the budget is exhausted. -
Cross-validation / statistical protocol. Model selection uses the checkpoint with highest validation m@16 evaluated every 50 steps and at the end of training. For Qwen 3 0.6B cold-start GARL
q=0.75and warm-start GRPO results, the paper reports mean and standard deviation over 3 different random seeds; other numbers are single-seed unless marked with±. The warm-start GRPO baseline is explicitly run with 3 seeds (Table 1 caption), establishing variance estimates for the primary comparison point. Training uses the AdamW optimizer [Loshchilov and Hutter, 2019] with batch size 64, learning rate5 × 10⁻⁷for 0.6B (higher rates were unstable in preliminary experiments) and1 × 10⁻⁶for 8B, constant learning rate (no warmup or decay), 2 epochs for all datasets, and rollout temperature 1.0 (top-k/top-p sampling disabled). No KL regularization to a reference policy is applied, following the VeriFree setup [Zhou et al., 2026] and the finding by Liu et al. [2025] that KL does not improve performance in this regime.
Main Quantitative Results
The paper organizes empirical findings around three research questions that directly test the theoretical predictions of Sections 3 and 4: RQ1 — can fixed-q JQ optimization escape cold start without annotated rationales? RQ2 — does JQ optimization provide gains over GRPO in warm start? RQ3 — is PAFT empirically more stable than GARL in warm start?
RQ1: Cold-Start Escape (Table 1)
Headline result: GARL at sufficiently high q escapes cold start where both GRPO and RB-RLOO (q=0) fail entirely, with the critical q threshold rising with model scale.
Table 1 reports cold-start results on all three benchmarks at both 0.6B and 8B scales. Cold start is operationally defined as using linearized (x*, y*) pairs with no task description and no formatting instructions — the model must discover both how to solve the problem and how to format the answer, and initial Pθ is very low.
Qwen 3 0.6B cold start: GRPO scores zero on every metric across all three benchmarks — complete failure, consistent with the escape rate at q=0 (Theorem 3.1). Similarly, GARL at q=0 (RB-RLOO), q=0.25, and q=0.5 all score zero on every metric. The first escape occurs at q=0.75, which achieves:
- FinQA: 30.5 ±0.3 p@1, 61.1 ±0.5 p@16, 38.6 ±0.6 m@16
- HotPotQA: 53.4 ±0.6 p@1, 74.1 ±1.0 p@16, 57.4 ±0.9 m@16
- MuSiQue: 27.5 ±0.9 p@1, 58.2 ±0.7 p@16, 35.6 ±1.5 m@16
GARL at q=1 also escapes (no sharp threshold between non-escape and escape at 0.6B — both q=0.75 and q=1 escape), but with key differences from q=0.75:
- FinQA p@1: 21.9 vs. 30.5 at
q=0.75(lower final accuracy despite immediate escape in Figure 2a) - HotPotQA p@16: 75.5 vs. 74.1 at
q=0.75(higher coverage, consistent withq=1's mode-covering behavior) - HotPotQA m@16: 56.6 vs. 57.4 at
q=0.75(coverage advantage does not survive majority voting)
This pattern — q=0.75 beats q=1 on p@1 and m@16 on two of three benchmarks — validates the escape-speed vs. estimator-quality tradeoff of Theorem 4.1: q=1's stronger amplification enables faster escape (Figure 2a: immediate spike at step 0) but yields higher-bias gradient estimates that degrade final accuracy post-escape.
Qwen 3 8B cold start: The critical q threshold shifts upward to q ≥ 0.85. At q=0.75, GARL now fails entirely (0 on all metrics), unlike at 0.6B where the same q=0.75 was sufficient. At q=0.85:
- FinQA: 45.0 p@1, 75.2 p@16, 52.9 m@16
- HotPotQA: 64.8 p@1, 81.5 p@16, 68.6 m@16
- MuSiQue: 58.7 p@1, 78.8 p@16, 62.9 m@16
At q=1 on 8B: FinQA p@1 drops to 38.4 (vs. 45.0 at q=0.85), while MuSiQue m@16 edges slightly higher at 64.5 (vs. 62.9) — the same pattern of q=1 trading final accuracy for broader coverage, now shifted to a higher absolute performance level.
Side-result — cold-start GARL vs. warm-start GRPO (cross-regime comparison): Table 1 includes warm-start prompted GRPO baselines for reference. Cold-start GARL at q=0.75 (0.6B) substantially exceeds warm-start GRPO on every metric: p@1 margins of +9.9 (FinQA), +23.8 (HotPotQA), +14.6 (MuSiQue). At 8B, cold-start GARL at q=0.85 exceeds warm-start GRPO by +26.3 (FinQA p@1), +29.9 (HotPotQA), +32.0 (MuSiQue). The paper explicitly flags this as hypothesis-generating, not conclusive: cold- and warm-start runs differ in more than prompts (input formatting, output constraints, target distribution), and isolating the prompt factor requires a controlled ablation left to future work. The FinQA 8B GRPO warm-start m@16 of 19.6 is notably lower than the 0.6B GRPO warm-start m@16 of 27.8 — an inverted scale ordering that the paper notes without explanation.
Theoretical consistency: The sharp threshold at q=0.75 (0.6B) and q=0.85 (8B) is qualitatively consistent with Theorem 3.1: at fixed training budget (2 epochs, constant learning rate), there exists a critical q below which the required escape time exceeds the available training time. The upward shift in threshold at larger scale is consistent with lower effective initial Pθ (plausible for larger models with more diffuse output distributions), though the paper does not directly measure Pθ. Figure 2a in Appendix G visualizes this threshold behavior on FinQA: the maximum amplified advantage c_m/M^q remains flat near zero for q ≤ 0.5 and spikes sharply around step 35 for q=0.75, with q=1 showing immediate escape from step 0.
RQ2 and RQ3: Warm-Start Utility and PAFT vs. GARL Stability (Table 2)
Warm start is operationally defined using natural-language prompts with task descriptions and answer-formatting instructions (the prompting approach of Wei et al., 2022, DeepSeek-AI, 2025), without SFT on annotated rationales. The un-adapted model can occasionally produce correct answers, so Pθ is non-trivial and standard RL already makes progress. All warm-start comparisons use exact-match training rewards.
RQ2 headline: JQ optimization at low q provides substantial gains over GRPO when training is stable; at higher q, it achieves higher peaks but destabilizes on certain benchmarks.
Table 2 reports warm-start m@16 across all three benchmarks for Qwen 3 0.6B. The baseline (no training, prompted) achieves 12.6 (FinQA), 22.2 (HotPotQA), 8.9 (MuSiQue) m@16. GRPO improves these to 27.8, 34.0, and 15.4 respectively.
On FinQA (stable training): GARL is stable at all tested q. GARL at q=0 (RB-RLOO) achieves 38.3, already +10.5 over GRPO — Rao-Blackwellization alone provides a substantial gain where the base model has non-trivial Pθ. GARL at q=0.25 achieves the best result of 38.7 (+10.9 over GRPO), with performance roughly flat across q ∈ [0, 0.75]: q=0.25 at 38.7, q=0.75 at 37.6. This flatness is consistent with the model already being in a regime where cold-start escape speed is non-critical, so the optimal q shifts toward lower values to reduce the O(q/MP_θ^q) bias. PAFT at q=0.25 and q=0.75 achieves 26.6 and 28.6, underperforming GARL on this benchmark — FinQA's latent-rationale variance is apparently low enough that GARL's lower-variance gradient estimation dominates PAFT's stability advantage.
On HotPotQA (unstable training): GARL collapses at every tested q: validation accuracy peaks early then drops to literal zero before training ends, and the best-validation checkpoint gives test m@16 of 22.9 (q=0.25) and 46.8 (q=0.75) — the latter representing a peak-before-collapse rather than steady-state performance. At q=0 (RB-RLOO), GARL achieves only 21.6, well below GRPO's 34.0 — showing that Rao-Blackwellized REINFORCE alone actually hurts on this benchmark. GRPO also degrades on HotPotQA (the paper notes it peaks ~37.4 around step 100 and declines steadily to ~5.0), but GARL's collapse is qualitatively different: a sharp drop to zero rather than a gradual decline.
PAFT is stable on HotPotQA: PAFT at q=0.75 achieves 47.9 m@16, the best warm-start result across any method (+13.9 over GRPO, and +1.1 over GARL's peak-before-collapse of 46.8). PAFT at q=0.25 achieves 47.0 — also stable and also exceeding GRPO. Figure 2b in Appendix G compares validation curves at matched q=0.25: GARL peaks at step 50 (30.6 m@16) and collapses to zero by step 100; PAFT remains stable throughout, peaking at 53.6.
On MuSiQue (mixed stability): GARL shows a similar peak-and-collapse pattern, with the best-validation checkpoint giving test m@16 of 24.3 (q=0.25) — exceeding PAFT's best stable result of 22.4 (q=0.75) but not surviving training. At q=0 (RB-RLOO), GARL achieves only 9.1, below GRPO's 15.4 and barely above the untrained baseline of 8.9. PAFT at q=0.25 underperforms GRPO (9.0 vs. 15.4), but the paper notes its validation curve is still rising at end of training — the P_θ^{0.75} attenuation heavily down-weights hard instances, slowing learning without destabilizing it. PAFT at q=0.75 recovers speed while remaining stable, delivering the recommended MuSiQue result of 22.4 m@16 (+7.0 over GRPO).
RQ3 headline: PAFT is empirically more stable than GARL on HotPotQA and MuSiQue, despite having provably higher variance (Proposition E.4) and the same leading-order bias (Proposition E.3).
The paper identifies this as a stability-variance paradox: by standard asymptotic criteria, GARL dominates PAFT (lower variance, same bias), yet on two of three benchmarks, GARL catastrophically collapses while PAFT remains stable. The paper hypothesizes — but does not verify — that the mechanism is pathwise-term corruption: GARL computes the pathwise gradient ∇θ w_m on every sampled trajectory (including incoherent ones), updating pθ(y* | x*, z^(m)) for all M trajectories regardless of quality. PAFT only computes gradients on resampled trajectories that passed the importance-resampling filter, so it never forms a gradient on an incoherent rationale. The collapse timing appears to correlate with latent-rationale variance Var_z[w(z)] under the prior, ranking: FinQA (no collapse — lowest variance) < MuSiQue (late collapse — moderate) < HotPotQA (early collapse — highest). Direct measurement of this variance and a pathwise-zeroed ablation are left to future work.
Speed vs. stability tradeoff across q: PAFT at q=0.25 underperforms GRPO on MuSiQue because the P_θ^{0.75} attenuation heavily downweights hard instances, slowing learning. Raising q to 0.75 recovers speed without compromising stability — the best PAFT results on HotPotQA (47.9) and the stable MuSiQue recommendation (22.4) both use q=0.75. This creates a practical tension: lower q reduces the O(q/MP_θ^q) bias but also weakens the gradient signal on hard instances via PAFT's P_θ^{1-q} attenuation; higher q strengthens the signal but amplifies noise and bias. The paper notes that PAFT additionally acts as an automatic curriculum: only the easiest rationales pass the resampling filter early in training, broadening as Pθ grows.
Endpoint behavior: At q=0 (RB-RLOO), GARL beats GRPO on FinQA (+10.5 m@16) but underperforms on HotPotQA (−12.4) and MuSiQue (−6.3). This demonstrates that conditional reward alone does not generalize — Rao-Blackwellization's variance reduction helps when training is inherently stable but cannot overcome the collapse dynamics on unstable benchmarks. GARL at q=1 and PAFT at q=1 are not separately reported in Table 2 (the warm-start experiments focus on q ∈ [0, 0.75]), but the cold-start q=1 results suggest that q=1 would escape quickly but degrade post-escape accuracy relative to intermediate q.
Ablation Studies and Robustness Checks
Rao-Blackwellization alone (GARL q=0 vs. GRPO in warm start): On FinQA, RB-RLOO (GARL q=0) achieves 38.3 m@16 vs. GRPO's 27.8 — a +10.5 gain showing that analytic reward marginalization substantially helps when training is stable. On HotPotQA, RB-RLOO drops to 21.6 vs. GRPO's 34.0 (−12.4), and on MuSiQue to 9.1 vs. 15.4 (−6.3). This confirms that variance reduction addresses the wrong bottleneck for unstable benchmarks (Table 2).
q sweep in cold start (Table 1, 0.6B): GARL at q ∈ {0, 0.25, 0.5, 0.75, 1} reveals a sharp transition between q=0.5 (zero everywhere) and q=0.75 (non-trivial performance). This validates that gradient amplification, not variance reduction, is the cold-start bottleneck — adding more q at fixed M=32 qualitatively changes the escape behavior, consistent with Theorem 3.1's scaling (Table 1).
q=0.75 vs. q=1 in cold start (Table 1, 0.6B): At q=0.75, FinQA p@1 is 30.5 vs. 21.9 at q=1; HotPotQA m@16 is 57.4 vs. 56.6; MuSiQue m@16 is 35.6 vs. 32.5. The q=0.75 advantage on p@1 and m@16 (where q=1's broader coverage does not survive voting) validates Theorem 4.1's O(q/MP_θ^q) bias: q=1's stronger amplification causes higher-bias gradient estimates that degrade final accuracy despite enabling faster escape (Table 1, Figure 2a).
Scale-dependent threshold (Table 1, 8B): The cold-start critical q shifts from ≥0.75 at 0.6B to ≥0.85 at 8B — q=0.75 fails entirely at 8B. This is consistent with lower effective initial Pθ at larger scale (harder optimization landscape or more diffuse initial distributions), but the paper acknowledges the mechanism is "not directly measured" (Table 1).
GARL vs. PAFT stability at matched q (Table 2, Figure 2b): On HotPotQA at q=0.25, GARL's validation peak of 30.6 at step 50 collapses to zero by step 100, while PAFT remains stable and peaks at 53.6. On MuSiQue, GARL peaks higher (24.3) but collapses, while PAFT at q=0.75 delivers the stable recommendation (22.4). This pattern — GARL peaks higher but collapses, PAFT stays stable — is the core empirical evidence for the dual-estimator design principle (Table 2, Figure 2b).
PAFT q=0.25 vs. q=0.75 in warm start (Table 2): On HotPotQA, PAFT at q=0.25 (47.0) and q=0.75 (47.9) are comparable — the stability benefit of PAFT is robust to q. On MuSiQue, PAFT at q=0.25 collapses to 9.0 (below GRPO's 15.4), while q=0.75 achieves 22.4 — the attenuation P_θ^{1-q} at low q downweights hard instances too aggressively, and higher q is needed for sufficient gradient signal. This validates the speed-vs-stability tradeoff within PAFT itself (Table 2).
GRPO warm-start baseline (Table 2): GRPO achieves 27.8 (FinQA), 34.0 (HotPotQA), 15.4 (MuSiQue) m@16, establishing that standard RLVR provides a non-trivial warm-start baseline. However, GRPO itself degrades on HotPotQA (peaking ~37.4 around step 100 and declining to ~5.0), demonstrating that instability is not unique to GARL — but GARL's collapse is qualitatively sharper (to literal zero rather than gradual decline). The three-seed reporting on GRPO (±2.0 on FinQA p@1, ±0.7 on m@16) provides variance estimates for the primary comparison point (Table 1 caption).
FinQA 8B GRPO warm-start scale inversion (Table 1): At 8B, GRPO warm-start m@16 on FinQA is 19.6, lower than the 0.6B result of 27.8. HotPotQA and MuSiQue scale as expected (8B > 0.6B). The paper notes this inversion without explanation — it may reflect benchmark-specific overfitting or optimization difficulties at larger scale that happen to affect FinQA more severely.
Critical Assessment
Claim 1: "GARL at sufficiently high q escapes cold start where GRPO fails entirely."
What was tested: Cold-start GARL at five q values on three benchmarks at 0.6B, and four q values at 8B, compared against GRPO and RB-RLOO (q=0) cold-start baselines. GRPO scores zero everywhere (Table 1).
Assessment: This claim is strongly supported for the tested conditions — Qwen 3 models, FinQA/HotPotQA/MuSiQue, exact-match training rewards, M=32 (0.6B) or M=16 (8B) rollouts, 2 epochs of training. The sharp threshold at q=0.75 (0.6B) and q=0.85 (8B) is consistent with the theoretical prediction of a critical q below which escape fails at fixed budget. However, the cold-start definition (linearized pairs, no task description, no formatting instructions) is somewhat synthetic — it represents a more extreme cold start than typical RLVR applications where models are at least instruction-tuned. The claim would be strengthened by testing intermediate initial conditions (e.g., prompted but without SFT, or with partial SFT on a fraction of the data) to map out how the critical q varies with initial Pθ.
Relevant gaps: No cold-start PAFT experiments (the paper explains that PAFT's P_θ^{1-q} attenuation suppresses the gradient at cold start, and importance resampling suffers particle degeneracy when all w_m are near zero). The claim is therefore specific to GARL among JQ estimators. The scale-dependent threshold is demonstrated at two scales but the mechanism (lower Pθ at larger scale) is hypothesized, not measured.
Claim 2: "In warm start, GARL at low q gives sizable gains over GRPO when training is stable; PAFT at q ≥ 0.75 gives gains where GARL destabilizes."
What was tested: Warm-start GARL and PAFT at selected q values (0, 0.25, 0.75) on all three benchmarks at 0.6B, compared against GRPO. FinQA results show stable GARL at all q; HotPotQA and MuSiQue show GARL collapse and PAFT stability.
Assessment: This claim is supported with important qualifications. The FinQA result (GARL stable, q=0.25 best at 38.7 vs. GRPO 27.8) is clean and reproducible (single-seed for GARL, mean±std for GRPO on 3 seeds). The claim of "gains over GRPO" holds on all three benchmarks for at least one JQ method — PAFT on HotPotQA (47.9 vs. 34.0), PAFT on MuSiQue (22.4 vs. 15.4). However, the recommended method differs by benchmark — GARL on FinQA, PAFT on HotPotQA and MuSiQue — meaning the claim is about the existence of a good JQ variant per benchmark, not about a single method dominating GRPO universally. This is consistent with the paper's three-phase recipe framework but means practitioners would need benchmark-specific method selection, which weakens the practical utility claim unless the selection criteria can be predicted from measurable properties (e.g., latent-rationale variance).
The collapse mechanism is not experimentally isolated. The paper hypothesizes pathwise-term corruption (GARL updates pθ(y* | x*, z) on every trajectory; PAFT only on resampled coherent ones) but provides no ablation to test this hypothesis — no GARL variant with the pathwise term removed, no measurement of Var[w(z)] that correlates with collapse timing, no controlled experiment demonstrating that higher-variance rationales cause collapse. The collapse timing correlation (FinQA < MuSiQue < HotPotQA) is suggestive but observational.
The 8B warm-start experiments are ongoing. All warm-start results are at 0.6B only; the paper notes "8B ongoing" in Section 6. This is a significant gap — the warm-start stability pattern (GARL collapse / PAFT stability) is validated only at one scale, and the scale-dependent cold-start threshold suggests warm-start dynamics might also shift. The claim is therefore scale-limited until 8B results are available.
Claim 3: "The JQ continuum explains why SFT-then-RLVR works — SFT escapes cold start via P_θ^{-1} amplification, RLVR halts noise commitment at q=0."
What was tested: The theoretical escape rates (Theorems 3.1, 3.2) and noise-fitting rates (Proposition D.2) are derived analytically but not experimentally validated on the SFT-then-RLVR pipeline directly. The cold-start experiments validate that q=0 fails and q≥0.75 escapes, consistent with the escape-rate prediction. The noise-fitting prediction is not experimentally tested — all experiments use clean supervision.
Assessment: This claim is theoretically supported (the math is rigorous within its assumptions) but empirically incomplete. The paper does not run SFT-then-RLVR experiments to compare against fixed-q or annealed JQ; it does not test JQ under label noise to validate Proposition D.2; it does not demonstrate that a q=1 → 0 schedule (the JQ instantiation of SFT-then-RLVR) achieves the same or better performance as the standard pipeline. The explanation for why SFT-then-RLVR works is therefore a theoretical contribution — a plausible mechanistic account grounded in the mathematics of the loss family — rather than an experimentally validated causal explanation. This is a legitimate type of contribution (theoretical frameworks don't require exhaustive empirical validation), but the paper's abstract and introduction frame it as "providing a unifying theoretical account," and the experiments validate the components of that account (escape rates, estimator bias) rather than the account as a whole (that this is why the pipeline succeeds rather than one possible reason it could succeed).
Claim 4: "The cold-start bottleneck is gradient magnitude (P_θ^{-q} amplification), not variance (Rao-Blackwellization)."
What was tested: GARL at q=0 (RB-RLOO) vs. GARL at q ≥ 0.75 in cold start. RB-RLOO has non-zero gradient but gets zero accuracy everywhere (Table 1). GARL at higher q escapes successfully.
Assessment: This is the most cleanly validated claim in the paper. The experiment is a direct test of the bottleneck hypothesis: if variance were the bottleneck, RB-RLOO (which reduces variance) would escape cold start; it doesn't. If gradient magnitude (controlled by q) were the bottleneck, higher q would enable escape; it does. The sharp threshold between q=0.5 and q=0.75 provides within-method evidence: the only difference between these runs is the amplification factor P_θ^{-q}, and it makes the difference between zero and non-trivial performance. The scale-dependent upward shift (8B requiring q ≥ 0.85) further supports the magnitude account — larger models plausibly have smaller initial Pθ, requiring stronger amplification to escape within fixed training budget. The limitation is that initial Pθ is not directly measured, so the "plausibly smaller" mechanism is a reasonable but unverified inference.
Missing experiments that would strengthen the paper:
-
Annealed
qschedules. The paper's central conceptual contribution is that SFT-then-RLVR is a stepwiseq=1 → 0schedule. Testing an annealed schedule (e.g., linear decay from 1 to 0, or step decay matching the pipeline's switch point) against fixedqand against the standard pipeline would directly validate this claim. The fixed-qexperiments establish baseline behavior but don't test scheduling. -
Direct Pθ measurement. The theoretical escape rates and bias expansion both depend on Pθ, but Pθ is never directly measured or reported — even as an average over the training set. Measuring Pθ evolution during cold-start training for different
qvalues would validate theṗ = p^{2-q} ‖s‖²dynamics and the threshold behavior. -
Label-noise experiments. Proposition D.2 predicts that higher
qmemorizes label noise faster. Testing GARL at differentqwith synthetic label noise (flipping a fraction of training answers) would validate this directly and provide empirical grounding for the noise-robustness axis of the trilemma. -
PAFT at cold start with learned proposals. The paper states PAFT fails at cold start due to vanishing attenuation and particle degeneracy, but doesn't test alternatives — learned proposal distributions, auxiliary models, or hybrid GARL-then-PAFT schedules — that could extend PAFT to lower-Pθ regimes.
-
Pathwise-term ablation. The GARL collapse hypothesis (pathwise-term corruption) is testable: run GARL without the pathwise term
∇θ w_m(using only the score-function component) on HotPotQA and observe whether collapse is eliminated or attenuated. This would isolate the mechanism. -
Latent-rationale variance measurement. Compute
Var_z[w(z)]during training for all three benchmarks and correlate with collapse timing. If FinQA consistently shows lower variance than HotPotQA, the variance-collapse hypothesis gains empirical support. -
Multiple model families. All experiments use Qwen 3. Validating the cold-start threshold pattern (critical
qrises with model scale) on a different family (e.g., Llama, Gemma) would confirm it's a general phenomenon rather than Qwen-specific. Similarly, the GARL-collapse / PAFT-stability pattern on HotPotQA could be model-specific. -
Warm-start 8B experiments. The warm-start claims are validated only at 0.6B. The scale-dependent cold-start threshold suggests warm-start dynamics might also shift — GARL could become more stable or more unstable at larger scale, and the optimal
qfor PAFT might shift.
Overall experimental assessment: The experiments effectively validate the paper's two core empirical claims: (1) high q GARL escapes cold start where GRPO cannot, and (2) JQ optimization provides warm-start gains over GRPO with the estimator choice depending on training stability. The cold-start results are particularly clean — the sharp threshold between q=0.5 and q=0.75 (and between q=0.75 and q=0.85 at 8B) is a strong, interpretable pattern that directly supports the gradient-magnitude bottleneck hypothesis. The warm-start results are more nuanced — the GARL/PAFT stability dichotomy is empirically clear but mechanistically unexplained — and are limited to a single model scale. The theoretical claim about why SFT-then-RLVR works is supported by the component validations (escape rates, noise-fitting rates) but not tested as an integrated hypothesis. The paper's scope as "foundational" is appropriate: it establishes baseline fixed-q behavior that future work on schedules, annealing, and automatic switching will build upon.
6. Limitations and Trade-offs
Difficulty Estimation Cost Dominates the Headline Efficiency Gains
The assumption or constraint. The paper's theoretical claims implicitly assume that Pθ — the marginal probability of the correct output — is known or can be estimated at negligible cost. This is true for the continuous-time gradient flow analysis (Section 3), which treats Pθ as directly observable, but it is emphatically NOT true for the practical estimators GARL and PAFT. Both estimators must estimate Pθ⁻q or Pθ¹⁻q from the same M-samples used to estimate the gradient, creating the ratio-estimator bias that Theorem 4.1 characterizes. However, there is a deeper, unaccounted cost that the paper does not address.
The theoretical escape-rate analysis (Theorems 3.1 and 3.2) and its practical consequence — GARL at high q escapes cold start — depends on the amplification factor Pθ⁻q being large when Pθ is small. But in practice, Pθ is never known. It is estimated from exactly M samples via ẇM = (1/M) Σ wm. When Pθ is genuinely tiny (say, 10⁻⁶ in cold start), ẇM with M = 32 will be near zero for most minibatches. The plug-in amplification (ẇM)⁻q is then estimated from a nearly-zero denominator with enormous relative variance — it is unreliable precisely in the regime where accurate amplification matters most.
This is not the O(q/MPθ^q) bias from Theorem 4.1, which is a systematic error in the estimator's expectation. This is a finite-sample reliability problem: at cold start with M = 32, the denominator ẇM may be zero or near-zero for many examples, rendering the amplification factor large by accident rather than by design. The paper does not account for this effect in its theoretical analysis (which assumes continuous-time flow with exact Pθ) or in its computational cost accounting (which treats M = 32 as sufficient for both numerator and denominator estimation).
The consequence. The headline cold-start results in Table 1 — GARL at q = 0.75 achieving 30.5% p@1 on FinQA, 53.4% on HotPotQA from zero — were obtained with M = 32 rollouts per example. However, these results do not tell us how many of those 32 samples were wasted on examples where ẇM ≈ 0 and the amplification factor was effectively random noise. The practical reliability of GARL at cold start depends on whether M = 32 is sufficient to get a non-degenerate estimate of Pθ⁻⁰·⁷⁵ for a typical cold-start example. If Pθ = 10⁻⁶, then E[ẇM] = 10⁻⁶, and the probability that ALL M = 32 samples yield wm ≈ 0 (and thus ẇM ≈ 0) is substantial. The amplification factor (ẇM)⁻⁰·⁷⁵ would be enormous when ẇM happens to be tiny, but this amplification would be driven by sampling noise rather than by genuine gradient signal.
In other words, the practical cold-start escape that GARL demonstrates relies on a hidden hyperparameter — the number of samples M — that must be large enough to reliably estimate the amplification factor. The paper sweeps q extensively but does not sweep M to determine the minimum sample size needed for reliable amplification estimation at cold start. The M = 32 setting was chosen for comparison with standard RLVR (which also uses M = 32), but RLVR does not need to estimate Pθ⁻q — its amplification factor is identically 1. The sample-size requirements of GARL are fundamentally different, and the paper does not characterize them.
What evidence exists in the paper. The cold-start results in Table 1 show that M = 32 IS sufficient for escape at q = 0.75 (0.6B) and q = 0.85 (8B). However, the results at q = 0.5 (zero everywhere) do not distinguish between two possible causes: (a) the amplification is genuinely insufficient at q = 0.5 (as Theorem 3.1 predicts — Pθ⁻⁰·⁵ amplification is too weak), or (b) the amplification IS sufficient in principle but M = 32 is too small to estimate it reliably at q = 0.5, and more samples would enable escape. Figure 2a shows that at q = 0.5, the maximum amplified advantage cm/M^q remains flat — consistent with (a) — but without an M sweep, we cannot confirm that the flatness reflects genuine gradient starvation rather than estimation collapse.
Mitigation status. The paper does not address this limitation. There is no ablation over M, no discussion of the minimum M required for reliable cold-start amplification, and no reporting of ẇM distributions during training. The paper's cost accounting (Section 4: "drop-in compute cost") treats GARL as equivalent to RB-RLOO at the same M, ignoring that GARL's ratio-estimator structure may demand larger M than RB-RLOO to achieve comparable reliability. This is a significant practical gap for anyone attempting to reproduce the cold-start results: the paper provides no guidance on choosing M as a function of q and the expected initial Pθ.
The Three-Phase Recipe Is Postulated, Not Validated
The assumption or constraint. Section 6 proposes a "three-phase post-training recipe":
- Phase 1 (cold start, Pθ ≈ 0): GARL at large q (≥ 0.75)
- Phase 2 (warm start, stable): GARL at low q where stable
- Phase 3 (warm start, unstable): PAFT at q ≥ 0.75; as Pθ → 1, anneal q → 0
This recipe is derived from the theory — the escape-speed vs. estimator-quality vs. noise-robustness trilemma, plus the observed PAFT stability on two benchmarks — but it has never been tested as an integrated procedure. The paper's experiments test each phase IN ISOLATION: cold-start GARL at fixed q, warm-start GARL at fixed q, warm-start PAFT at fixed q. No experiment runs a schedule that transitions between them — no cold-start GARL followed by warm-start PAFT, no warm-start GARL switching to PAFT at collapse onset, no annealing of q from 0.75 down to 0 over the course of training.
The recipe also requires knowing, or detecting, which phase the model is in and when to switch. The paper provides no mechanism for this detection — no diagnostic for "Pθ has risen enough to reduce q" or "GARL is about to collapse, switch to PAFT." The theoretical analysis (Theorems 3.1, 3.2) gives escape times in terms of Pθ, which is unobserved in practice. The paper acknowledges this gap explicitly in Section 6:
"Validating these switches empirically is future work."
and in Section H:
"Principled schedule design adaptive to the current Pθ, and automatic switching between GARL and PAFT, remain open."
The consequence. The three-phase recipe is a hypothesis, not a ready-to-deploy training protocol. A practitioner attempting to use JQ for a new task faces a series of unresolved decisions: at what q to start cold start? When to reduce q? When (if at all) to switch from GARL to PAFT? At what q to run PAFT? Should q be annealed continuously or dropped in discrete steps? The paper's fixed-q results provide candidate q values for each phase, but do not demonstrate that transitioning between them works without introducing new instabilities (e.g., what happens to the optimizer state when q changes mid-training? Does the effective learning rate shift require learning-rate warmup or reset?).
Moreover, the recipe's claimed advantage over the standard SFT-then-RLVR pipeline — that JQ enables training without annotated rationales — depends on cold-start GARL successfully bootstrapping the model to a regime where warm-start methods can take over. Table 1 shows that cold-start GARL at q = 0.75 (0.6B) achieves impressive results (30.5% FinQA p@1, 53.4% HotPotQA p@1), but these are compared against warm-start GRPO, not against the full SFT-then-RLVR pipeline that the recipe aims to replace. We do not know whether cold-start GARL followed by warm-start PAFT matches, exceeds, or underperforms standard SFT-then-RLVR, because no such comparison is run.
What evidence exists in the paper. The individual phases have empirical support. Cold-start GARL works at high q (Table 1). Warm-start GARL works on FinQA where stable (Table 2). Warm-start PAFT works on HotPotQA and MuSiQue where GARL destabilizes (Table 2). But these are disconnected observations. The paper does not even report a simple sequential experiment: cold-start GARL at q = 0.75 for 1 epoch, then switch to GARL at q = 0.25 for 1 epoch, then evaluate. The "recipe" is purely a conceptual synthesis of separate findings, not an empirically validated procedure.
Mitigation status. The paper is transparent about this being future work (Section 6, Section H). The three-phase recipe is presented as a prescription that "the continuum prescribes" — a logical consequence of the theory, not a tested algorithm. This transparency is appropriate for a foundational theory paper, but it means the paper's most actionable-sounding recommendation (Section 6: "A three-phase post-training recipe") is, at present, untested. The paper's actual validated contributions are the fixed-q results in Tables 1 and 2; the recipe is a roadmap, not a deliverable.
The GARL Collapse Mechanism Is Unidentified and Unresolved
The assumption or constraint. On HotPotQA and MuSiQue warm-start, GARL exhibits catastrophic collapse — validation accuracy peaks early, then drops to literal zero before training ends — at every tested q value (Table 2, Figure 2b). This is not a gradual performance degradation like GRPO's slow decline on HotPotQA (~37.4 at step 100 to ~5.0 by training end). It is a sharp, complete collapse to zero accuracy, suggesting a qualitative failure mode rather than simple overfitting.
The paper does not identify the mechanism. Section 5.3 offers a hypothesis:
"Candidate explanations include pathwise-term corruption (GARL updates pθ(y* | x*, z) on every sampled z, including incoherent ones; PAFT only on resampled coherent rationales) and HotPotQA-specific overfitting (also visible in GRPO)."
This hypothesis is plausible — the pathwise term ∇θ wm in GARL's decomposition (Equation 15) updates the model's output-likelihood parameters for EVERY sampled trajectory, including rationales that are completely incoherent or lead to wrong answers. PAFT only computes gradients on resampled trajectories that passed the importance-weight filter, so it never updates the model based on poor rationales. If updating on incoherent rationales corrupts the model's ability to generate good rationales or assess output likelihoods, this could cause the sharp collapse.
However, the paper provides no experimental evidence for this hypothesis. There is no ablation removing the pathwise term from GARL to see whether collapse is eliminated. There is no measurement of Varz[w(z)] (latent-rationale quality variance) during training to correlate with collapse onset. The paper notes a suggestive pattern — "Collapse timing appears to correlate with latent-rationale variance ranking FinQA (none) < MuSiQue (late) < HotPotQA (early)" — but calls this a correlation needing "direct measurement."
The consequence. The GARL collapse is the single most important practical failure mode in the paper. It means that the theoretically-superior estimator (GARL has provably lower variance than PAFT, Proposition E.4, and the same leading-order bias, Proposition E.3) is unusable on two of the three tested benchmarks in warm-start, and its failure is unpredictable from the theory. A practitioner cannot look at a new benchmark, compute any summary statistic of the data or model, and determine whether GARL will be stable or collapse. They must run the experiment and observe whether collapse occurs — which defeats the purpose of having a principled estimator selection criterion.
The collapse also means that the clean story "use GARL at low q in warm start for best results" (the natural prescription from the bias-variance tradeoff in Theorem 4.1) is false on two-thirds of the tested benchmarks. The actual advice is "use GARL on FinQA-like tasks, use PAFT on HotPotQA/MuSiQue-like tasks, and we don't yet know how to tell the difference a priori." This significantly weakens the practical deployment case for JQ — it requires benchmark-specific trial-and-error rather than principled selection.
What evidence exists in the paper. Figure 2b shows the GARL vs. PAFT validation curve at matched q = 0.25 on HotPotQA: GARL peaks at step 50 (m@16 30.6) and hits zero by step 100; PAFT remains stable and peaks at 53.6. Table 2 shows GARL's best-validation-checkpoint results for HotPotQA (22.9 at q = 0.25, 46.8 at q = 0.75) and MuSiQue (24.3 at q = 0.25, 19.7 at q = 0.75), all identified as "peak-before-collapse" rather than steady-state. The paper explicitly notes that GRPO also degrades on HotPotQA (peaking ~37.4 and declining to ~5.0), establishing that some instability is benchmark-specific — but GARL's collapse to literal zero is qualitatively different.
Mitigation status. The paper identifies PAFT as the workaround — PAFT is stable where GARL collapses — but this is a replacement, not a fix. PAFT has higher variance (Proposition E.4), is inapplicable at cold start (vanishing attenuation), and underperforms GARL on FinQA (26.6-28.6 vs. 37.6-38.7 m@16; Table 2). The pathwise-term corruption hypothesis is flagged for future investigation (Section 5.3: "direct measurement and a pathwise-zeroed ablation are left to future work"). The paper does not resolve the GARL collapse; it routes around it with PAFT. The fundamental question — why does a lower-variance, same-bias estimator catastrophically fail, and can we detect or prevent this failure? — is left open.
All Results Are on a Single Model Family and Three Benchmarks Under Exact-Match Reward
The assumption or constraint. Every experiment in the paper uses post-trained Qwen 3 models (0.6B and 8B) [Yang et al., 2025] and three question-answering benchmarks (FinQA, HotPotQA, MuSiQue) with exact-match training rewards — the reward is 1 if and only if the generated output exactly matches the ground-truth answer. The theoretical framework in Sections 2-4 is formulated under the same exact-match assumption: Proposition B.1 shows that 1 - Pθ = ED[1 - reward] under R(ŷ, y*) = I(ŷ = y*), so minimizing J0 exactly maximizes expected reward. The JQ loss family, the gradient factorization, and the convergence analysis all depend on this identity: Pθ = ED[R(ŷ, y*)] under exact-match reward. This section explicitly acknowledges: "General rewards are open" (Section 6, Limitations).
The paper acknowledges the model-family limitation partially: "The cold-start theorems are scale-agnostic and the cold-start ordering replicates at Qwen 3 8B across all three benchmarks (Section 5); the warm-start GARL collapse / PAFT stability finding is verified only at 0.6B (8B ongoing)."
The consequence. There are two independent generalization gaps.
First, the model-family gap: the cold-start threshold pattern (critical q rises from 0.75 to 0.85 with scale) and the warm-start GARL-collapse / PAFT-stability pattern may be specific to Qwen 3's architecture, pretraining data, or tokenization. The paper's claim to theoretical generality (the JQ family should apply to any latent-variable generative model) is not empirically supported beyond one model family. Different model families (Llama, Gemma, Mistral) may have different score-function norms ‖s(θ)‖, different initial Pθ distributions, and different latent-rationale variance properties — all of which affect the critical q threshold, the optimal warm-start q, and the GARL/PAFT stability boundary.
Second, the exact-match reward gap: many important reasoning tasks do not admit binary correctness signals. Code generation (pass@k over multiple test cases), mathematical reasoning with partial credit, open-ended generation evaluated by LLM-as-judge, and reinforcement learning with learned reward models all involve non-binary, non-deterministic rewards. Under these reward structures, Pθ ≠ ED[R] in general, and Proposition B.1's identity fails. The entire JQ loss family is defined in terms of Pθ = pθ(y* | x*), which treats y* as the single correct output. For tasks where multiple outputs are acceptable (e.g., multiple valid code solutions, multiple valid explanations), the mapping from reward to Pθ is undefined — there is no single y* to compute pθ(y* | x*) against.
This is not merely a scope limitation ("we only tested on these benchmarks"). It is a structural constraint of the framework: JQ requires a well-defined marginal probability of the correct output. This probability is meaningful for exact-match tasks (question answering, factual verification) but ambiguous for tasks where correctness is multi-dimensional or graded.
What evidence exists in the paper. The paper makes no attempt to test JQ on non-exact-match rewards. All experiments (Tables 1 and 2) use exact-match training rewards (Section 5.1: "Training uses exact-match rewards"). The evaluation uses relaxed substring match, but this is a post-hoc evaluation metric, not the training signal — the model was optimized against exact-match reward during training. The warm-start GRPO baseline also uses exact-match training rewards, so the comparisons are fair within this constraint, but no comparison against methods designed for non-binary rewards (e.g., PPO with a learned reward model) is possible from the reported results.
The model-family limitation is partially addressed by the 8B cold-start results (Table 1), which replicate the critical-q threshold phenomenon. However, the warm-start results at 8B are "ongoing" — the paper's central warm-start claims (GARL collapse on HotPotQA/MuSiQue, PAFT stability, +13.9 maj@16 over GRPO) are validated at only one scale and one model family. The paper cannot claim empirical generality for these findings until they replicate in at least one additional family or scale.
Mitigation status. The paper acknowledges both limitations. For exact-match reward: "General rewards are open" (Section 6). For model family: "the warm-start GARL collapse / PAFT stability finding is verified only at 0.6B (8B ongoing)." Both are listed as future work in Section H. There is no conceptual obstacle to extending JQ to non-binary rewards — one could redefine Pθ = ED[R] for expected reward, or use the escort distribution to weight trajectory likelihoods — but this extension is not developed or tested. The paper's theoretical framework is complete for exact-match supervision; its empirical validation is limited to one model family at one scale for warm-start dynamics.
PAFT Is Inapplicable at Cold Start, Creating a Regime Where No JQ Estimator Works Well
The assumption or constraint. PAFT relies on two mechanisms that fail when Pθ is very small:
-
Attenuation suppression: The PAFT gradient is multiplied by
(ẇM)¹⁻q ≈ Pθ¹⁻q. When Pθ ≈ 0 (cold start), this attenuation drives the gradient to zero — the exact opposite of the amplification that cold-start escape requires. This is by design: PAFT's attenuation acts as an automatic curriculum, down-weighting hard instances. But in cold start, ALL instances are hard, so PAFT produces no meaningful gradient on any example. -
Particle degeneracy in importance resampling: PAFT resamples trajectories from the importance-weighted distribution
w_m / Σ_j w_j. When allw_mare near zero (which happens when Pθ is small — the expected weight is small, so individual weights are small), the importance weights become nearly uniform (all near1/M). The resampling step then selects trajectories essentially at random from the prior, providing no posterior concentration. Worse, if the weights vary but are all extremely small, numerical underflow can make the categorical distribution degenerate (division by sum of near-zero numbers).
The paper explicitly acknowledges this in a footnote in Section 5.3:
"PAFT is not evaluated at cold start:
Pθ¹⁻q ≈ 0suppresses the gradient, and importance resampling suffers particle degeneracy (effective sample size ≈ 1) when allwmare near zero."
The consequence. There is a cold-start gap in the JQ estimator landscape: GARL works at cold start but requires high q (≥ 0.75 at 0.6B, ≥ 0.85 at 8B), which brings the O(q/MPθ^q) bias and the GARL collapse risk documented in warm-start. PAFT would in principle offer the stable, low-bias alternative, but it cannot be used until Pθ has already risen — exactly the regime where GARL is already working and the escape-speed advantage of high q is no longer needed. This creates an awkward situation: the estimator you want for warm-start stability (PAFT) cannot bootstrap itself to warm-start; the estimator that can bootstrap (GARL) may destabilize once you reach the regime where you'd prefer PAFT.
The three-phase recipe's proposed solution — "Phase 1: GARL at large q, Phase 2: GARL at low q where stable, Phase 3: PAFT at q ≥ 0.75 otherwise" — acknowledges this gap but papers over it with a handoff that has never been tested. Phase 2 ("GARL at low q where stable") assumes GARL will be stable on the target benchmark post-cold-start — but the paper has no way to predict GARL stability a priori. If GARL turns out to be unstable (as it is on HotPotQA and MuSiQue), you would want to switch to PAFT after Phase 1 — but the paper provides no tested mechanism for making this switch, no diagnostic for when to switch, and no demonstration that switching from GARL to PAFT mid-training works without introducing new optimization artifacts.
What evidence exists in the paper. The paper provides no PAFT cold-start results, as explicitly acknowledged. The cold-start results (Table 1) are exclusively GARL. The warm-start results (Table 2) show that where GARL is stable (FinQA), it outperforms PAFT; where GARL is unstable (HotPotQA, MuSiQue), PAFT is the only viable JQ option. But these warm-start runs start from the prompted base model — Pθ is already non-trivial because the model can occasionally answer correctly with prompts. We do not know whether a model bootstrapped by cold-start GARL would, after cold-start escape, behave like the prompted warm-start model (where PAFT works) or whether the cold-start training trajectory introduces distribution shift that makes PAFT less effective.
The gap is most acute for practitioners who want to use JQ in a fully annotation-free setting — no SFT on rationales, no prompting. The paper's cold-start GARL results show this is possible in principle (Table 1), but the post-cold-start phase is unaddressed. Would cold-start GARL at q = 0.75 followed by PAFT at q = 0.75 achieve the 47.9 m@16 that prompted warm-start PAFT achieves on HotPotQA? Would it destabilize during the transition? The paper provides no evidence either way.
Mitigation status. The paper flags this as a key direction for future work. Section H (Future Directions) states: "PAFT's importance resampling from the prior pool fails at cold start (vanishing attenuation and particle degeneracy); learned proposals, MCMC, or infilling models conditioned on both x* and y* could extend PAFT to lower-Pθ regimes." This is a recognition of the gap, not a solution. The three-phase recipe (Section 6) is a proposed workaround (GARL for cold start, PAFT for warm start where GARL is unstable), but the handoff is unspecified and untested. A practitioner attempting fully annotation-free JQ training currently has no validated path from cold start to final model — only the demonstration that cold-start escape is possible, and the separate demonstration that warm-start PAFT works, with a theoretical handoff between them that has never been executed end-to-end.
No Accounting for Wall-Clock Time or the Serial Nature of Sequential Methods
The assumption or constraint. The paper measures computational cost exclusively in terms of "number of generations" (M rollouts per prompt during training, 16 per prompt at evaluation) and "drop-in compute cost" (Section 4: GARL adds O(M) scalar reweighting; PAFT adds one categorical resample + teacher forcing on already-generated tokens). The claim is that GARL and PAFT are cost-equivalent to standard RLVR at the same M — "Neither requires extra forward passes."
This cost accounting is correct for total FLOPs but ignores latency — wall-clock time per training step. The forward passes to generate M rollouts dominate the FLOPs budget for any reasonable M (the reweighting and resampling are negligible by comparison). However, the methods' computational graphs have different dependency structures that affect parallelism and hardware utilization.
More importantly, the paper's recommended three-phase recipe involves a sequential handoff between methods — cold-start GARL, then warm-start GARL, then warm-start PAFT — and the performance of this handoff may depend on training duration (number of steps) in each phase. The paper's experiments use a fixed training budget (2 epochs, constant learning rate), but the theoretical escape rates (Theorems 3.1, 3.2) are in continuous time, mapping loosely to iteration count. If cold-start escape takes most of the training budget at high q, there may be insufficient remaining budget for warm-start fine-tuning at low q or for PAFT stabilization.
The consequence. The practical cost of using JQ in a fully annotation-free pipeline is potentially higher than the "drop-in equivalent" framing suggests, for three reasons:
-
Cold-start GARL at high q may require more steps than warm-start RLVR. The escape process governed by
ṗ = p²⁻q ‖s‖²is slow early in training, even with amplification. The theoretical escape times are inversely proportional to the score norm‖s‖², which may be small at initialization. The paper provides no comparison of training time (steps or wall-clock) between cold-start GARL and warm-start GRPO to achieve comparable accuracy. -
The M requirement for GARL may exceed that for RLVR. As discussed in the first limitation, GARL's ratio-estimator structure may demand larger M than standard RLVR to achieve reliable amplification estimation, especially at cold start. The paper's
M = 32setting is the same as GRPO's, but this is a choice, not a validated minimum. A fair cost comparison would determine the minimum M at which each method achieves its reported performance, then compare FLOPs or wall-clock at those minima. -
The three-phase recipe may extend total training time. If Phase 1 (cold-start GARL) requires 1 epoch to escape, Phase 2 (warm-start GARL or PAFT) requires another epoch to stabilize, and Phase 3 (q-annealing) requires additional steps, the total training budget for the JQ pipeline may exceed the 2-epoch budget used in the fixed-q experiments, or may exceed the budget needed for standard SFT+RLVR with annotated rationales. The paper provides no head-to-head training-time comparison between JQ and the standard pipeline.
What evidence exists in the paper. Figure 2a shows the cold-start escape dynamics on FinQA: q = 1 escapes immediately (amplified advantage spikes at step 0), q = 0.75 escapes around step 35, q ≤ 0.5 never escapes within the training budget. This indicates that q = 0.75 spends approximately 35 steps (out of roughly 192 steps per epoch for FinQA at batch size 64 and 6145 training examples, i.e., ~96 steps per epoch — the exact number depends on the epoch length) in the near-zero-Pθ regime before escape. This is a non-trivial fraction of training time with near-zero learning progress. The paper does not report how training time partitions across phases in any experiment, nor does it measure wall-clock time.
Mitigation status. The paper does not address latency, wall-clock time, or training-time comparisons. The "drop-in compute cost" claim (Section 4) is about FLOPs per training step, not about total training cost to reach a target accuracy. The fixed training budget (2 epochs) is the same for all methods, so the comparisons in Tables 1 and 2 are fair in the sense that each method got the same number of gradient updates. But this fairness obscures the possibility that some methods require more updates than others to reach the same performance — a question the paper does not investigate. The paper's scope as a foundational methods paper makes this a reasonable omission, but practitioners deploying JQ at scale need to know not just that GARL can escape cold start, but how many GPU-hours it takes compared to annotating rationales and running standard SFT+RLVR. This comparison is absent.
7. Implications and Future Directions
How This Work Changes the Landscape
This paper introduces a unifying theoretical language for reasoning model post-training that replaces a categorical distinction (SFT vs. RLVR) with a continuous parameter q controlling per-instance commitment to supervision. This is not a paradigm shift in the sense of rendering prior methods obsolete — SFT-then-RLVR remains an effective and widely deployed pipeline — but it is a conceptual reframing that changes how researchers should think about the design space.
From "which method?" to "what q schedule?" Before this work, the central question in reasoning model post-training was: should I use SFT, RLVR, or both? The answer (both, in that order) was empirically derived and theoretically unexplained. The JQ continuum transforms this from a discrete choice among qualitatively different method classes into a continuous optimization over commitment. The gradient factorization ∇ℓq = Pθ⁻q ∇ℓ0 = Pθ¹⁻q ∇ℓ1 (Proposition 2.2) shows that all points on the continuum share the same per-example gradient direction — the model is always being pushed to increase Pθ — differing only in the magnitude of that push. This reframing makes previously mysterious phenomena (why SFT first, why RLVR stalls at cold start, why intermediate approaches like reward-weighted SFT sometimes work and sometimes don't) legible as consequences of a single parameter.
A new diagnostic for cold-start failure. The paper's sharpest contribution is the re-diagnosis of cold-start failure in RLVR as a gradient magnitude problem rather than a variance problem. The prevailing intuition — RLVR stalls because no sampled output is correct and the gradient vanishes — motivated variance-reduction techniques like Rao-Blackwellization. The cold-start experiments decisively falsify this: RB-RLOO (GARL at q=0) gets zero accuracy everywhere, despite having non-zero gradient (Table 1). The bottleneck is not variance; it is that ṗ = p²‖s‖² at q=0 gives Ω(1/p₀) escape time, which is structurally glacial. The adjustment this requires in the field's thinking is significant: improving cold-start RL is not about better exploration or tighter variance bounds — it's about gradient amplification, controlled by a scalar parameter q in the loss function.
Reconciliation of conflicting prior findings. The paper resolves several apparent contradictions in the literature:
-
"LLMs cannot self-correct reasoning" [Huang et al., 2023] vs. self-refinement works [Madaan et al., 2023]. The JQ perspective reveals that correctness depends on Pθ: when Pθ is non-trivial, SFT-like updates (high
q) refine reasoning; when Pθ is near zero, no amount of self-correction helps because the gradient magnitude is proportional to Pθ²⁻q. These studies were implicitly testing on different difficulty distributions, differing in initial Pθ. -
Rao-Blackwellization reduces variance but doesn't help cold start. The paper shows why: variance reduction addresses the wrong bottleneck. The cold-start failure of
q=0(Table 1) and the sharp escape atq≥0.75with identicalM=32samples demonstrates that amplification, not variance, determines escape. -
SFT memorizes, RL generalizes [Chu et al., 2025]. The JQ analysis provides the mechanism: at
q=1, the model fits both clean and corrupted supervision at rateΘ(log(1/p̃₀)/ϵ)(Proposition D.2); atq=0, noise-fitting time diverges. The SFT-then-RLVR ordering exploits this: SFT atq=1escapes cold start, then RL atq=0halts noise commitment. -
IWAE [Burda et al., 2015], EM [Dempster et al., 1977], and RB-RLOO [Zhou et al., 2026] as endpoints of one family. The endpoint recovery in Proposition E.2 shows these are not independently invented algorithms but instances of the same prior-based or posterior-based estimation strategy applied at different
q. This unification suggests that future estimator development should focus on theqcontinuum rather than inventing new objective-specific estimators.
Which research directions become more attractive, and which less so:
-
More attractive: Developing cheap, online Pθ estimation methods to enable adaptive
qscheduling; training verifiers/PRMs that can be plugged into JQ for tasks beyond exact-match; extending PAFT to cold start via learned proposals or infilling models; characterizing the GARL collapse mechanism (pathwise-term corruption) with controlled ablations; designingq-annealing schedules validated end-to-end against the standard SFT-then-RLVR pipeline. -
Less attractive: Developing ever-more-sophisticated RLVR variance reduction techniques for cold start — the paper shows variance is not the bottleneck. Designing new, from-scratch loss functions for reasoning model training without understanding their position on the commitment spectrum — the JQ family demonstrates that many desiderata (escape speed, noise robustness, estimator quality) are governed by a single parameter, and new proposals should articulate where they sit on this continuum. Treating SFT and RL as categorically distinct paradigms requiring separate theoretical tools — the JQ factorization provides a unified analytical framework.
The scale of the contribution. This is a foundational theory paper whose primary impact will be on how researchers think and talk about reasoning model post-training, not on immediate deployment recipes. The three-phase recipe (Section 6) is a hypothesis; the fixed-q experiments demonstrate the continuum's existence but not an end-to-end training protocol. The paper's value is in providing a generative framework — a set of concepts (commitment q, amplification Pθ⁻q, the escape-bias-noise trilemma, the dual estimator factorization) that enable systematic reasoning about phenomena that were previously understood only through empirical intuition. The most important downstream consequence is likely to be a shift from "SFT-then-RLVR" as a fixed recipe to "commitment scheduling" as a design dimension, with q being tuned, annealed, or adaptively adjusted per-instance based on estimated Pθ.
Follow-Up Research This Work Enables
Direct measurement of Pθ and ‖s‖² during JQ training to validate the gradient flow escape dynamics. The theoretical analysis in Section 3 makes specific, quantitative predictions: ṗ = p²⁻q ‖s‖², escape time Tq = Θ(p₀^{-(1-q)}/(1-q)), and the near-optimality convergence being q-independent (Proposition D.1). These predictions are untested beyond the qualitative observation that higher q enables cold-start escape. A strong follow-up would instrument the training loop to measure Pθ (by averaging ẇM over a held-out set throughout training) and ‖∇θ log Pθ‖² (via finite differences or the Fisher information), then fit the observed p(t) curves to the predicted p²⁻q dynamics for multiple q values. This would either confirm the gradient flow analysis as quantitatively predictive (not just qualitatively correct) or reveal where the stylized assumptions (single-example, bounded score, continuous time) break down. The 0.6B FinQA cold-start setting is ideal for this because GARL is stable and the sharp threshold between q=0.5 (failure) and q=0.75 (escape) at M=32 samples provides a clear region where the dynamics should produce measurable differences.
Pathwise-term ablation to isolate the GARL collapse mechanism. The paper hypothesizes that GARL's collapse on HotPotQA and MuSiQue is caused by the pathwise gradient term ∇θwm updating pθ(y* | x*, z) on every sampled trajectory, including incoherent rationales, while PAFT only computes gradients on resampled coherent rationales (Section 5.3). This hypothesis is directly testable: run GARL on HotPotQA with the pathwise term removed (using only the score-function component −cm ∇θ log pθ(z(m) | x*) from Equation 17), with all other settings identical to the q=0.25 run that collapsed in Figure 2b. If collapse is eliminated or substantially delayed, the pathwise-term corruption mechanism is confirmed, and the practical fix is to drop the pathwise term on unstable benchmarks (at the cost of some gradient information). If collapse persists, the mechanism lies elsewhere — perhaps in the score-function amplification (w_m / ẇM^q) itself amplifying noise on high-variance trajectories — and the investigation shifts to the score-function term. A second ablation would measure Varz[w(z)] (the variance of likelihood weights across sampled rationales) during training for all three benchmarks and correlate with collapse timing; the paper's observation that "collapse timing appears to correlate with latent-rationale variance ranking FinQA (none) < MuSiQue (late) < HotPotQA (early)" would be quantitatively verified or refuted within the first 100 training steps.
Cold-start GARL followed by warm-start PAFT as an end-to-end annotation-free pipeline. The paper demonstrates that cold-start GARL at high q escapes (Table 1) and that prompted warm-start PAFT at q=0.75 achieves strong stable results on HotPotQA and MuSiQue (Table 2). But these are separate experiments with different initial conditions — cold-start GARL starts from linearized (x*, y*) pairs with no prompts, while warm-start PAFT starts from a prompted base model with non-trivial Pθ. The critical follow-up is an integrated experiment: train GARL at q=0.75 from cold start on HotPotQA for some number of steps (until escape, as diagnosed by rising validation accuracy), then switch to PAFT at q=0.75 for the remaining training budget, and compare against (a) cold-start GARL at q=0.75 for the full budget, (b) prompted warm-start PAFT at q=0.75, and (c) the standard prompted GRPO baseline. This experiment tests whether the model bootstrapped by cold-start GARL is in a state where PAFT can take over effectively — i.e., whether Pθ has risen enough that PAFT's Pθ¹⁻q attenuation no longer suppresses the gradient, and whether the cold-start training trajectory introduces distribution shift that degrades PAFT's effectiveness relative to the prompted warm-start setting. The HotPotQA benchmark is the right testbed because it exhibits both the cold-start escape (Table 1, 0.6B) and the GARL-collapse / PAFT-stability pattern in warm-start (Table 2, Figure 2b). A positive result would validate the three-phase recipe as more than a theoretical proposal; a negative result would indicate that the cold-start and warm-start Pθ distributions are sufficiently different that the GARL→PAFT handoff requires additional machinery (e.g., gradual q annealing during the transition, or a mixed estimator that blends GARL and PAFT).
Annealed-q schedules against the standard SFT-then-RLVR pipeline with annotated rationales. The paper's central conceptual claim is that SFT-then-RLVR corresponds to a stepwise q=1 → 0 schedule. This claim is supported theoretically (Sections 2-3) but has never been tested empirically against a q-annealed JQ schedule. A strong follow-up would implement continuous q annealing — e.g., linear decay from q=1 to q=0 over the course of training — using GARL throughout, and compare against (a) the standard pipeline (SFT on human-annotated rationales for 1 epoch, then GRPO for 1 epoch), (b) fixed-q GARL at the best intermediate q from Section 5, and (c) a stepwise q schedule that drops q at manually chosen steps. The comparison must control for total training FLOPs: the standard pipeline has the advantage of dense supervision during the SFT phase (every token of the annotated rationale provides a training signal), while annealed GARL only has outcome supervision throughout. The annealed JQ schedule might require more training steps to match SFT-then-RLVR's final accuracy, and measuring this efficiency gap is the key practical question. The FinQA benchmark is the right testbed because GARL is stable at all q (Table 2), removing the GARL-collapse confound from the annealing comparison. If annealed JQ matches or exceeds SFT-then-RLVR without requiring annotated rationales, the paper's conceptual claim becomes a practical deployment option. If it underperforms, the contribution remains theoretical, and the standard pipeline's advantage likely stems from the dense token-level supervision during SFT (which JQ at any q does not provide when trained only on outcome rewards) rather than from the q-scheduling per se.
JQ under label noise to validate Proposition D.2's noise-fitting predictions. The noise-fitting analysis (Proposition D.2) predicts that higher q memorizes label errors faster: at q=1, noise commitment time is Θ(log(1/p̃₀)/ϵ); at q=0.5, it's Θ(p̃₀⁻⁰·⁵/ϵ); at q=0, it diverges (the model ignores label noise entirely). This prediction has not been experimentally tested — all experiments in Section 5 use clean supervision. A strong follow-up would systematically corrupt a fraction of training labels (e.g., flip 5%, 10%, 20% of answers to incorrect alternatives) on a benchmark where GARL is stable (FinQA), then train GARL at q ∈ {0, 0.25, 0.5, 0.75, 1} and measure both clean-test accuracy and noise-memorization rate (the fraction of corrupted training examples where the model learns the incorrect label). The prediction is that q=1 will show the highest noise memorization and q=0 the lowest, with a monotonic relationship, and that the memorization dynamics will follow the predicted p̃⁻(¹⁻q) time-course. This experiment would directly validate the noise-robustness axis of the trilemma and provide practical guidance for choosing q in settings where label noise is expected (crowd-sourced datasets, automatically constructed supervision, distillation from imperfect teacher models). A negative result — e.g., q=0.25 memorizing noise faster than q=0.75 — would indicate that the theoretical analysis misses important effects of stochastic optimization or multi-example interactions that alter the noise-commitment dynamics.
Scale and model-family replication of the warm-start GARL-collapse / PAFT-stability pattern. The warm-start stability findings (Table 2) are validated only on Qwen 3 0.6B — the 8B warm-start experiments are reported as "ongoing" (Section 6). A minimal but essential follow-up is to complete the 8B warm-start experiments and report whether the same pattern holds: does GARL collapse on HotPotQA and MuSiQue at 8B? Does PAFT remain stable? Does the optimal q for stable PAFT shift with scale, as the cold-start critical q shifts from 0.75 (0.6B) to 0.85 (8B)? Beyond Qwen, replicating the HotPotQA warm-start experiment on at least one other model family (e.g., Llama 3.2 3B, Gemma 2 2B) would determine whether the GARL-collapse / PAFT-stability pattern is model-specific or a general property of the estimators. If the pattern replicates across families, it suggests a fundamental estimator property (consistent with the pathwise-term corruption hypothesis). If it fails to replicate — e.g., GARL is stable on HotPotQA with Llama — then the instability is confounded with Qwen-specific training dynamics, and the three-phase recipe's GARL-to-PAFT switch cannot be prescribed without model-specific validation.
Practical Applications and Downstream Use Cases
Annotation-free post-training for specialized reasoning domains. The clearest practical scenario enabled by this work is training a reasoning model on a domain-specific dataset where annotating rationales is expensive or infeasible — financial analysis (FinQA), legal reasoning, medical diagnosis, multi-hop fact verification (HotPotQA, MuSiQue). The cold-start GARL results in Table 1 demonstrate that a 0.6B model can be trained from zero (no prompts, no formatting instructions, no annotated rationales) to 30.5% p@1 on FinQA and 53.4% p@1 on HotPotQA using only outcome-level exact-match supervision, with fixed q=0.75 and M=32 rollouts per prompt. This translates to: give the model input-output pairs (x*, y*), specify that successful outputs must contain y* as a substring, and GARL handles the rest — no human annotation of intermediate reasoning steps required. The 8B results (45.0% FinQA p@1, 64.8% HotPotQA p@1 at q=0.85) suggest the approach scales. For a domain expert with a dataset of 5,000–10,000 examples, running cold-start GARL at q ≥ 0.75 for 2 epochs on a single 8-GPU A100 machine (the paper's compute setup, taking ~3 minutes per step per Section F) could produce a specialized reasoning model without writing a single rationale. The practical caveat is that cold-start GARL's escape depends on the critical q threshold, which may need tuning for a new domain — the paper provides q=0.75 (0.6B) and q=0.85 (8B) as reference points, but the safe procedure is to sweep q ∈ {0.5, 0.75, 0.85, 1} and select the checkpoint that escapes.
Stable warm-start fine-tuning on noisy or crowd-sourced label datasets. When a model is already instruction-tuned and can occasionally produce correct answers (prompted warm start), the PAFT results in Table 2 provide a drop-in replacement for GRPO that delivers substantially higher accuracy on benchmarks where standard RLVR is unstable. On HotPotQA, prompted warm-start PAFT at q=0.75 achieves 47.9 m@16, +13.9 over GRPO's 34.0, and remains stable throughout training while GRPO degrades (the paper notes GRPO peaks around 37.4 and declines to ~5.0). On MuSiQue, PAFT at q=0.75 achieves 22.4 m@16, +7.0 over GRPO's 15.4, also without collapse. These are large absolute gains — HotPotQA goes from roughly one-third to nearly half of questions answered correctly by majority vote. For a practitioner deploying reasoning models on multi-hop QA or similar compositional reasoning tasks, swapping GRPO for PAFT at q=0.75 (with the same M=32 rollout budget) provides a significant accuracy improvement without changing any other aspect of the training pipeline. The GARL-stable case (FinQA, where GARL at q=0.25 achieves 38.7 m@16, +10.9 over GRPO) offers a different route: if the practitioner can verify stability on their domain (by monitoring validation accuracy for early collapse), GARL at low q provides the largest gains. The paper does not provide an a priori stability diagnostic, so the practical workflow is: try GARL at q=0.25 first; if validation accuracy collapses, fall back to PAFT at q=0.75.
Compute-efficient adaptation of small models for on-device reasoning. The cold-start results at 0.6B scale (Table 1: 30.5% FinQA, 53.4% HotPotQA, 27.5% MuSiQue p@1) demonstrate that a sub-billion-parameter model can acquire non-trivial reasoning capabilities from outcome supervision alone, without the distillation or SFT-on-rationales steps that typically require a larger teacher model. This matters for on-device deployment: a 0.6B model runs on consumer hardware (phones, laptops), while the 8B+ models that typically power reasoning applications require server-grade GPUs. The paper's M=32 rollout budget during training translates to 32 forward passes per example per step, which is substantial but amortized over the entire training corpus (6,000–10,000 examples, 2 epochs, ~200–400 steps per epoch). At inference, the model is standard autoregressive generation with no special test-time compute requirements. For a mobile application needing offline financial QA or multi-hop reasoning, cold-start GARL at q=0.75 on a 0.6B model could produce a specialized on-device reasoner trained entirely on the target domain's input-output pairs, matching or exceeding the prompted warm-start GRPO baseline (Table 1: cold-start GARL q=0.75 m@16 of 38.6 on FinQA vs. warm-start GRPO m@16 of 27.8). The practical limitation is that the cold-start escape requires q ≥ 0.75 at 0.6B — lower q values fail entirely — so the practitioner must commit to the higher-bias q=0.75 estimator and cannot benefit from the lower-bias q=0.25 that provides the best FinQA warm-start results (Table 2: 38.7 m@16).
When to Prefer This Method
The paper articulates a clear tradeoff between fixed-q JQ optimization (GARL or PAFT) and standard RLVR (GRPO/REINFORCE), conditioned on the training regime and benchmark-specific stability. The decision rule is:
-
Prefer cold-start GARL at
q ≥ 0.75(0.6B) orq ≥ 0.85(8B) when: you have no annotated rationales, no task prompts that give non-trivial initial Pθ, and only outcome-level supervision (exact-match rewards). Use GARL withM ≥ 32rollouts; sweepqin{0.5, 0.75, 0.85, 1}to find the critical threshold for your model scale. Cold-start GARL at the criticalqmatches or exceeds prompted warm-start GRPO on the three tested benchmarks (Table 1: +9.9 to +23.8 p@1 at 0.6B). Do not use PAFT in this regime —Pθ¹⁻qattenuation suppresses the gradient and importance resampling suffers particle degeneracy (Section 5.3 footnote). -
Prefer warm-start GARL at low
q(0–0.25) when: the model already has non-trivial Pθ (via prompting or prior SFT), training is verified to be stable on your benchmark (monitor validation accuracy for early collapse to zero), and you want the largest gains over GRPO. On FinQA, this achieves +10.9 m@16 over GRPO (38.7 vs. 27.8, Table 2). GARL atq=0recovers RB-RLOO, which already beats GRPO on stable benchmarks (+10.5 on FinQA). The risk is that GARL may collapse to zero accuracy on your benchmark, as it does on HotPotQA and MuSiQue (Figure 2b) — if this occurs, fall back to PAFT. -
Prefer warm-start PAFT at
q = 0.75when: GARL collapses on your benchmark (you observe validation accuracy peaking early then dropping to zero), or when you want a stable, collapse-free JQ estimator without tuningq. PAFT atq=0.75achieves +13.9 m@16 over GRPO on HotPotQA (47.9 vs. 34.0) and +7.0 on MuSiQue (22.4 vs. 15.4) without collapse (Table 2). PAFT atq=0.25may be too conservative — on MuSiQue it underperforms GRPO (9.0 vs. 15.4) becausePθ⁰·⁷⁵attenuation heavily down-weights hard instances. PAFT additionally acts as an automatic curriculum (only coherent rationales pass the resampling filter), which may be beneficial for benchmarks with high latent-rationale variance. -
Prefer standard GRPO over JQ methods when: you have access to annotated rationales for SFT warm-start (the standard SFT-then-GRPO pipeline is not directly compared to JQ in this paper), or when your benchmark exhibits stable GRPO training and the implementation complexity of GARL/PAFT is not justified by the expected gains (but the paper shows GRPO underperforms the best JQ method on all three benchmarks in warm start, by margins of +7.0 to +13.9 m@16 — Table 2 — so the threshold for "not justified" is narrow).
The paper does NOT provide guidance for scenarios with: non-exact-match rewards (Section 6: "General rewards are open"), annealed q schedules (Section 6: "validating these switches empirically is future work"), or cold-start PAFT (explicitly stated as inapplicable).
