ArXiv: 2511.20347
🎯 Pitch
GRPO and GSPO often collapse early during RL fine-tuning because hard clipping discards too many useful gradients or admits noisy ones. SAPO replaces that brittle clipping with a temperature-tuned sigmoid gate that smoothly down-weights off-policy tokens while preserving near-policy updates, enabling stable training without sacrificing sample efficiency.
1. Executive Summary
The paper introduces Soft Adaptive Policy Optimization (SAPO), a smooth and temperature-controlled policy-gradient method that replaces the hard clipping used in group-based RL algorithms like GRPO and GSPO with a sigmoid-shaped soft gate that continuously attenuates off-policy token updates (for example, selectively down-weighting outlier tokens while preserving gradients from near-on-policy tokens within the same sequence). Evaluated on mathematical reasoning benchmarks (AIME25, HMMT25, BeyondAIME) using a cold-start Qwen3-30B-A3B MoE model, SAPO achieves both improved training stability and higher final Pass@1 performance compared to GSPO and GRPO-R2 under comparable compute budgets, with the asymmetric temperature design—setting τ_neg > τ_pos to more aggressively dampen high-variance negative-token gradients—proving critical for avoiding early training collapse. Large-scale experiments on Qwen3-VL models across MoE and dense architectures further demonstrate consistent gains on diverse text and multimodal tasks, establishing that smooth, token-adaptive gating can extend stable RL training duration and improve final performance without relying on auxiliary stabilization techniques like routing replay.
2. Context and Motivation
The Core Problem: Hard Clipping in Group-Based RL Creates a Stability-Efficiency Trade-off
The paper addresses a specific, practical failure mode in the current paradigm of using reinforcement learning to improve LLM reasoning: hard clipping mechanisms in group-based policy optimization methods create a brittle trade-off between training stability and sample efficiency. While RL has become central to advancing LLM reasoning capabilities—powering breakthroughs in mathematical problem-solving, code generation, and multimodal understanding (OpenAI, 2024; DeepSeek-AI, 2025; Qwen, 2025)—the optimization algorithms used to train these models suffer from a fundamental tension that SAPO aims to resolve.
The problem manifests concretely as early training collapse. In the paper's own experiments (Figure 4), both GSPO and GRPO-R2 exhibit "early-stage training collapse" when fine-tuning a Qwen3-30B-A3B model on mathematical reasoning, whereas SAPO sustains stable learning. This is not a niche issue—it affects the practical deployment of RL for LLMs at scale, where training runs are expensive, and unexpected divergence wastes substantial compute resources. The paper identifies the root cause as high variance in token-level importance ratios , which measure how much more or less likely the current policy is to generate a given token compared to the behavior policy that originally sampled the response.
This variance is particularly severe in Mixture-of-Experts (MoE) architectures, where different tokens may be routed to different expert sub-networks. The routing heterogeneity creates a wider distribution of token-level likelihood ratios because a single gradient update can shift the expert assignment for some tokens while leaving others unchanged, producing uneven changes in generation probabilities across the sequence. The paper documents this empirically in Figures 2 and 3: the MoE model (Qwen3-30B-A3B) exhibits a wider spread of values and higher per-sequence log-ratio variance compared to a dense model (Qwen3-4B), confirming that MoE architectures amplify the variance problem that hard clipping must contend with.
The stakes of this problem are both empirical and conceptual. Empirically, when training runs collapse early, the model never reaches its potential performance ceiling—compute is wasted, and the resulting model underperforms on downstream tasks. Conceptually, the fact that how you constrain policy updates matters so much for stability suggests that our theoretical understanding of policy optimization for autoregressive sequence models is incomplete. The paper's framing implies that the field has converged on group-based methods as a practical recipe—Shao et al. (2024) with GRPO, Zheng et al. (2025) with GSPO—but that these methods are held together by a crude engineering fix (hard clipping) whose failure modes are poorly understood and whose alternatives are unexplored.
Why This Problem Matters: The Centrality of RL in Modern LLM Training
The significance of stable policy optimization extends well beyond mathematical reasoning benchmarks. The paper makes clear that RL is now "a key driver of recent advances in large language models" (Section 1), powering the reasoning breakthroughs in models like OpenAI's o1 (OpenAI, 2024), DeepSeek-R1 (DeepSeek-AI, 2025), and Qwen3 (Qwen, 2025). These models use RL to learn how to reason—producing longer chains of thought, verifying intermediate steps, backtracking from dead ends—capabilities that are difficult to instill through supervised fine-tuning alone because the space of correct reasoning trajectories is combinatorially large and hard to specify with static demonstrations.
The group-based RL paradigm has emerged as the dominant approach for this task. The recipe, as outlined in Section 2, is: sample multiple responses per query from the current (or behavior) policy, compute sequence-level rewards (e.g., whether the final answer is correct), normalize those rewards within the group to compute advantages, and update the policy to increase the likelihood of high-advantage responses while decreasing the likelihood of low-advantage ones. This recipe appears in GRPO (Shao et al., 2024), GSPO (Zheng et al., 2025), and variants. It is appealing because it requires no learned value function or critic network—the group acts as a self-normalizing baseline—and because sequence-level rewards (correctness, passing unit tests) are often easier to define than token-level rewards.
However, the very structure that makes group-based methods practical—operating on pre-sampled rollouts from a behavior policy that differs from the current policy—creates the off-policy problem that hard clipping is designed to address. The importance ratios can deviate substantially from 1 as the policy is updated, meaning the gradient estimates become biased and high-variance. If these off-policy tokens are weighted naively (as in an unclipped policy gradient), the updates can be destructively large, causing the policy to collapse to degenerate behavior—generating repetitive tokens, empty responses, or gibberish. The problem is thus existential for the entire paradigm: without effective policy constraint, group-based RL for LLMs is not just suboptimal, but actively harmful.
This makes the search for better constraint mechanisms directly consequential. If SAPO can extend the duration of stable training—the paper claims it "sustains coherent learning for a longer duration and reaches higher Pass@1 accuracy before divergence" (Section 1)—then it directly enables better final models from the same training compute. For organizations training frontier reasoning models, where a single training run can cost millions of dollars, reliability improvements that prevent early collapse are high-value. Moreover, if SAPO eliminates the need for auxiliary stabilization techniques like the "routing replay" used in GRPO-R2 (Section 5.1), it reduces engineering complexity and removes a hyperparameter that must be tuned.
Beyond the immediate practical benefits, the problem has theoretical significance because it exposes a gap in how we understand policy constraints for sequence models. In traditional RL (e.g., PPO for continuous control), clipping the importance ratio works reasonably well because the action space is low-dimensional and the ratio distribution is relatively concentrated. In LLMs, the action space is the entire vocabulary (~100K+ tokens), and the policy is a product distribution over many time steps, making the ratio distribution both high-dimensional and potentially heavy-tailed. The failure of hard clipping in this setting suggests that the standard PPO-style constraint may be fundamentally mismatched to the geometry of autoregressive language generation, and that alternative constraint forms—like SAPO's continuous trust region—may be more natural.
Prior Approaches: GRPO, GSPO, and the Hard Clipping Paradigm
The paper identifies two primary prior methods that represent the state of the art in group-based policy optimization for LLMs, both of which rely on hard clipping but apply it at different granularities.
GRPO (Group Relative Policy Optimization). Introduced by Shao et al. (2024) and used in DeepSeekMath, GRPO operates at the token level. For each token in each sampled response, it computes an importance ratio and clips the surrogate objective using the standard PPO-style clip function:
where is the clipping range (typically 0.2) and is the group-normalized advantage. The gradient weight is a step function: it is 1 for tokens whose ratio falls within the clipping band and 0 for tokens outside it (Equation 24 in Section 4.2).
This hard binary gating creates two problems. First, it is all-or-nothing: a token with (just barely outside a band) receives zero gradient, while a token with receives the full unclipped gradient. There is no graduated penalty for increasing off-policyness, which makes the optimization landscape discontinuous and potentially brittle. Second, it is uniform within the band: all tokens inside are treated identically regardless of how close they are to the on-policy point . This means the clipping mechanism provides no incentive for the policy to stay close to the behavior policy—only a cliff edge beyond which updates cease entirely.
GSPO (Group Sequence Policy Optimization). Introduced by Zheng et al. (2025), GSPO proposes to apply clipping at the sequence level rather than the token level. It defines a length-normalized sequence importance ratio:
and clamps this single scalar per response. The motivation is that length normalization reduces variance: by taking the geometric mean of token ratios, places all sequences on a comparable numerical scale regardless of length. This is a genuine insight—without normalization, longer sequences would have more extreme sequence-level ratios simply because they multiply more token-level ratios together, making the clipping band ineffective for long responses or overly restrictive for short ones.
However, GSPO's sequence-level clipping introduces a subtler but equally severe problem: the sequence-level gate is token-invariant. If a single token in a 500-token response has a wildly off-policy ratio (say ), pushing outside the clipping band , GSPO zeroes out gradients for all 500 tokens in that sequence, including 499 near-on-policy tokens that could provide useful learning signals. The paper describes this as "suppressing gradients for the many near-on-policy tokens in that sequence, hurting sample efficiency" (Section 3). This is particularly damaging in MoE models where routing heterogeneity makes outlier tokens more common—Figures 2 and 3 confirm that the MoE model has a heavier tail in its distribution and higher per-sequence variance , meaning more sequences will contain at least one outlier token that triggers the hard clip.
Both GRPO and GSPO thus suffer from the same underlying pathology: hard clipping creates discontinuities in the optimization landscape that discard potentially useful information. GRPO does this at the token level (individual tokens are either wholly accepted or wholly rejected), while GSPO does it at the sequence level (entire sequences are accepted or rejected based on their average behavior). Neither provides a mechanism for partially weighting tokens based on their degree of off-policyness.
Why hard clipping has persisted. It is worth understanding why these hard clipping methods became standard despite their apparent crudeness. In PPO (Schulman et al., 2017), from which GRPO inherits the clip function, hard clipping worked well for continuous control tasks where the action space is low-dimensional (e.g., 6-DOF joint torques) and importance ratios are typically concentrated near 1. The clip function was a simple engineering solution that was easy to implement, had a clear interpretation (don't trust samples too far from the current policy), and empirically stabilized training. When GRPO adapted PPO's objective to the LLM setting, it preserved the clip function without reconsidering whether the geometry of the problem—massive action space, product distribution over time, heavy-tailed ratio distributions in MoE models—might call for a different constraint mechanism. GSPO recognized that sequence-level clipping was more coherent with the sequence-level reward structure, but retained the hard clip, merely shifting it to a different granularity.
The paper's contribution can be understood as questioning this inherited design choice and asking: what would a constraint mechanism designed specifically for the structure of autoregressive sequence generation look like?
How SAPO Positions Itself Relative to Existing Work
SAPO is positioned as a synthesis and refinement of GRPO and GSPO that inherits the strengths of each while addressing their weaknesses through a single mechanism: the smooth sigmoidal gate.
Relative to GRPO: SAPO shares GRPO's token-level granularity but replaces the hard clip with a continuous, temperature-controlled weighting function. Instead of a binary accept/reject decision at the boundary , SAPO's weight where follows a bell-shaped curve that peaks at when (fully on-policy) and decays smoothly to near-zero as deviates. This means:
- A token with does not receive zero gradient (as in GRPO with ), but rather a reduced weight of approximately 0.1–0.3 (depending on ). The learning signal is attenuated, not discarded.
- A token with receives a weight close to 0.8–0.9, still contributing substantially to the update, rather than the full unattenuated gradient that GRPO would apply.
- The transition is smooth: there is no cliff edge where a tiny change in flips the gradient from fully present to fully absent, reducing optimization noise from discretization.
The paper argues this "avoids gradient vanishing and enables more stable update dynamics" (Section 4.2). When the policy change is small, gradients remain responsive; as deviation grows, gradients shrink continuously, producing more conservative adjustments. This is the continuous trust region concept: rather than a binary inside/outside distinction, SAPO defines a graduated region where the degree of trust decays with distance from the on-policy point.
Relative to GSPO: SAPO shares GSPO's concern for sequence-level coherence—the idea that optimization should respect the fact that rewards are assigned to complete sequences, not individual tokens. However, rather than enforcing sequence-level coherence through a hard sequence-level clip (which discards entire sequences when the average is off), SAPO achieves sequence-level coherence as an emergent property under mild conditions while retaining the ability to handle outliers at the token level.
The paper formalizes this through the analysis in Section 4.1. Under two assumptions—(A1) small on-policy steps where , and (A2) low intra-sequence dispersion where the variance of token log-ratios within a sequence is small—the average of SAPO's token-level gates concentrates to a smooth sequence-level gate:
This means that when the sequence is reasonably homogeneous (most tokens have similar importance ratios), SAPO automatically behaves like GSPO with a smooth gate—coherent, sequence-level weighting aligned with the sequence-level reward. However, when the assumptions are violated—when a sequence contains outlier tokens that push it far from the on-policy regime—SAPO falls back to its token-level gating, selectively down-weighting the offending tokens while preserving gradients from near-on-policy tokens in the same sequence.
The paper provides empirical evidence that these assumptions hold in practice. Figures 2 and 3 show that is "sharply concentrated around 1" and "typically remains below 0.02" for both MoE and dense models, with the dense model showing tighter concentration. The rightmost panels of these figures show that the bound is small in practice—typically well below 0.001—meaning the average token gate is numerically close to the sequence-level gate for most sequences. This validates the reduction.
The key advantage over GSPO is what happens when a sequence contains a few highly off-policy tokens. In GSPO, one outlier can cause the entire sequence to exceed the clipping band, zeroing out gradients for all tokens. In SAPO, only the outlier tokens are strongly down-weighted; near-on-policy tokens continue to contribute meaningful gradients. The paper describes this as SAPO "selectively down-weighting only the offending tokens while preserving informative gradients from near on-policy tokens within the same sequence, improving sample efficiency" (Section 1).
The asymmetric temperature innovation. A novel component that SAPO introduces beyond simply smoothing the clipping function is the use of different temperatures for positive and negative advantage tokens: . This design choice is motivated by an analysis of how token-level gradients propagate through the logits (Equation 9 in Section 3).
The analysis shows that for a positive advantage (), the gradient increases the logit of the sampled token and decreases the logits of all unsampled tokens. For a negative advantage (), the gradient does the opposite: it decreases the sampled token's logit and increases the logits of all unsampled tokens in the vocabulary. Since the vocabulary size is typically or larger, a negative gradient diffuses to an enormous number of inappropriate tokens, while only a tiny fraction of those unsampled tokens are genuinely desirable alternatives. This makes negative updates inherently noisier and more prone to causing instability.
By setting , SAPO applies a steeper attenuation to negative-token gradients: they decay more rapidly as the token becomes off-policy. This means SAPO is more conservative with negative updates than with positive ones. The ablation in Figure 5 confirms the importance of this choice: training is "most stable when negative tokens are assigned a higher temperature () and most unstable when they are assigned a lower temperature ()." The asymmetry is not merely a fine-tuning knob—it is critical for avoiding early collapse.
This design choice has no counterpart in GRPO or GSPO, which apply the same clipping band symmetrically to positive and negative advantages. The paper does not claim that GRPO or GSPO couldn't be made asymmetric—one could imagine a GRPO variant with a tighter clip for negative advantages—but the smooth gate makes the asymmetry natural to implement and interpret, since the temperature parameter directly controls the rate of weight decay.
Positioning summary. SAPO is thus not a radical departure from the group-based RL paradigm, but rather a principled refinement of the constraint mechanism at its core. It preserves the overall structure: group sampling, advantage normalization, and policy-gradient updates weighted by importance ratios. What it changes is how the importance ratios are converted into update weights—replacing the discontinuous clamp with a smooth, temperature-controlled, asymmetric gate that is token-adaptive while retaining sequence-level coherence. The paper positions this as a more natural fit for the geometry of autoregressive sequence generation in MoE models, where token-level heterogeneity and high importance-ratio variance make hard clipping's cliff-edge behavior particularly damaging.
3. Technical Approach
3.1 Reader Orientation
SAPO is a reinforcement learning algorithm that determines how much to update each token in a language model's generated response based on how "off-policy" that token has become—replacing the brittle yes/no decision of hard clipping with a smooth, temperature-controlled weighting function that gradually reduces the influence of tokens as they drift further from what the original sampling policy would have produced. The core problem it solves is training instability in group-based RL for LLMs: when the policy changes during training, some tokens in previously-sampled responses become wildly unlikely under the new policy, and naively including them in gradient updates can cause the model to collapse to degenerate behavior. SAPO's solution is a continuous trust region—rather than a binary in/out decision at a fixed threshold, it assigns each token a weight between 0 and 1 that peaks at full on-policy (weight = 1) and decays smoothly toward zero as the token becomes more off-policy, with negative-advantage tokens (those the model should avoid) decaying faster than positive-advantage tokens (those the model should reinforce).
3.2 Big-Picture Architecture (Diagram in Words)
The SAPO training loop has five major components:
-
Behavior Policy (π_θold) — the current frozen copy of the language model that generates responses. It samples
Gresponses per query, producing the rollouts that will be used for off-policy learning. -
Reward Function — an external scorer (e.g., a math verifier) that assigns a scalar reward
R_ito each complete response based on correctness or quality. This operates at the sequence level: it sees the full generated text and the correct answer, returning a score. -
Group-Normalized Advantage Computation — a sequence-level operation that takes the
Grewards within each query's group, computes their mean and standard deviation, and normalizes each to produceÂ_i = (R_i - mean({R_j})) / std({R_j}). Positive advantages mean the response was better than average within its group; negative advantages mean worse. -
Token-Level Importance Ratios — for each token in each response, a ratio
r_i,t(θ) = π_θ(token | context) / π_θold(token | context)is computed by forward-passing through both the current policy (being trained) and the behavior policy (frozen). This ratio measures how much the policy's probability for that specific token has changed since the response was sampled. -
SAPO Soft Gate — a sigmoid-based function that converts each token's importance ratio and its sequence's advantage into a gradient weight
w_i,t(θ) ∈ [0, 1]. The weight peaks at 1 whenr_i,t = 1(perfectly on-policy) and decays smoothly asr_i,tdeviates from 1, with the rate of decay controlled by a temperature parameter that is larger for negative-advantage tokens (faster decay, more conservative) than for positive-advantage tokens (slower decay, more permissive).
Information flows as follows: a batch of queries enters → the behavior policy samples G responses per query → rewards are computed externally → advantages are normalized within each group → the current policy (updated since the rollouts were generated) computes token probabilities for every token in every response → importance ratios are computed by comparing current and behavior policy probabilities → SAPO's soft gate converts each ratio to a weight → the weight modulates the policy gradient, which is accumulated and used to update the model parameters → after the update, the current policy becomes the new behavior policy for the next round of sampling.
3.3 Roadmap for the Deep Dive
-
First, the SAPO objective function (Equation 5): the full mathematical definition, what each term means, and how it differs structurally from GRPO and GSPO. This defines what SAPO optimizes.
-
Second, the gradient weight function (Equations 6–8): the sigmoid-based soft gate that replaces hard clipping, including why the
4/τscaling factor is present and what the weight curve looks like as a function of the importance ratio. -
Third, the asymmetric temperature design: why SAPO uses different temperatures (
τ_pos,τ_neg) for positive and negative advantage tokens, grounded in an analysis of how gradients propagate through the softmax over the vocabulary. -
Fourth, the reduction to sequence-level behavior (Section 4.1): how, under mild conditions (small steps, low within-sequence variance), SAPO's average token gate concentrates to a smooth sequence-level gate—showing that SAPO achieves GSPO-like coherence without GSPO's brittleness.
-
Fifth, the contrast with GRPO's token-level hard clipping (Section 4.2): a precise comparison of the gradient weight functions, showing that GRPO's is a step function (0 or 1) while SAPO's is a smooth bell-shaped curve, and why this matters for optimization dynamics.
-
Sixth, the practical training setup: hyperparameters, the data pipeline (how rollouts are generated and consumed), and implementation details that make the algorithm work at scale.
3.4 Detailed, Sentence-Based Technical Breakdown
This is primarily an algorithm design paper whose core idea is that replacing hard clipping in group-based policy optimization with a smooth, temperature-controlled, asymmetric soft gate improves both training stability and final performance without requiring auxiliary stabilization techniques.
The SAPO Objective Function
The SAPO objective defines what the algorithm maximizes at each training step. It is a token-level surrogate objective that weights each token's contribution by a smooth function of its importance ratio.
The complete objective:
where:
$\mathcal{D}$is the distribution of queries (training prompts) from which batches are drawn;$G$is the number of responses sampled per query (the group size);$q$is a single query;$y_i$is the$i$-th response to query$q$, consisting of$|y_i|$tokens;$\pi_{\theta_{\text{old}}}$is the behavior policy—the frozen version of the model that generated the rollouts;$r_{i,t}(\theta) = \frac{\pi_\theta(y_{i,t} | q, y_{i,<t})}{\pi_{\theta_{\text{old}}}(y_{i,t} | q, y_{i,<t})}$is the token-level importance ratio: the probability the current policy assigns to token$t$of response$i$divided by the probability the behavior policy assigned to that same token at sampling time;$\hat{A}_{i,t} = \hat{A}_i = \frac{R_i - \text{mean}(\{R_j\}_{j=1}^G)}{\text{std}(\{R_j\}_{j=1}^G)}$is the group-normalized advantage, shared across all tokens within a response (it does not vary with$t$);$f_{i,t}(\cdot)$is SAPO's core innovation: the soft gating function that replaces hard clipping.
What it computes: The objective takes an expectation over queries and sampled responses. For each response, it computes a per-token weighted contribution: the gating function $f_{i,t}(r_{i,t}(\theta))$ maps the importance ratio to a scalar, which is then multiplied by the advantage $\hat{A}_{i,t}$. These are summed over tokens (normalized by response length to prevent long responses from dominating) and averaged over the group. The result is a scalar that measures how well the current policy aligns with the advantage-weighted behavior, with off-policy tokens attenuated by the soft gate. Maximizing this objective pushes the policy to increase probabilities for tokens in high-advantage sequences (near $r_{i,t} \approx 1$) while decreasing them for tokens in low-advantage sequences.
Why this form: The structure mirrors GRPO and GSPO—expectation over groups, length-normalized sum over tokens, advantage-weighted—but replaces the min/clip/max operations with $f_{i,t}$. This preserves the group-based, advantage-normalized framework (which is the established recipe for LLM RL) while changing only the constraint mechanism. The length normalization $1/|y_i|$ prevents longer responses from contributing disproportionately to the objective simply because they have more tokens. The group normalization of advantages creates a self-baselining effect: within each group, responses are compared to each other, so the absolute scale of rewards doesn't matter—only relative quality within the group.
The Soft Gating Function
The gating function $f_{i,t}$ is the algorithmic core of SAPO. It maps a token's importance ratio $r_{i,t}(\theta)$ to a scalar, using a sigmoid with a temperature-controlled slope.
The gating function definition:
where:
$\sigma(x) = 1 / (1 + e^{-x})$is the logistic sigmoid function, which smoothly maps real numbers to the interval$[0, 1]$;$\tau_i$is the temperature parameter, which takes one of two values:$\tau_i = \tau_{\text{pos}}$when$\hat{A}_i > 0$(the response was better than the group average) and$\tau_i = \tau_{\text{neg}}$when$\hat{A}_i \leq 0$(the response was worse than or equal to the group average);$r_{i,t}(\theta) - 1$centers the sigmoid at the on-policy point: when$r_{i,t}(\theta) = 1$(the current and behavior policies agree), the argument is$0$, and$\sigma(0) = 0.5$.
What it computes: For a given token, the gating function produces a value between $0$ and $4 / \tau_i$. At $r_{i,t}(\theta) = 1$, the output is $0.5 \cdot (4 / \tau_i) = 2 / \tau_i$. As $r_{i,t}(\theta)$ increases above $1$ (the policy has raised this token's probability), $\sigma$ increases toward $1$, so the output increases toward $4 / \tau_i$. As $r_{i,t}(\theta)$ decreases below $1$ (the policy has lowered this token's probability), $\sigma$ decreases toward $0$, so the output decreases toward $0$. The function is monotonic increasing in $r_{i,t}$.
Why this form—centering: Centering the sigmoid at $r = 1$ means the gate is most sensitive precisely where we want it: near the on-policy regime. When $r_{i,t} \approx 1$, small changes in policy probability produce meaningful changes in the gate output, allowing the optimization to respond to moderate off-policy drift. When $r_{i,t}$ is far from $1$ (e.g., $r \to 0$ or $r \to \infty$), the sigmoid saturates, and further changes produce negligible gate variation—the token is already fully attenuated or fully passed through.
Why this form—the $4 / \tau_i$ scaling factor: This factor ensures that at the on-policy point $r_{i,t}(\theta) = 1$, the gradient of the gated objective equals the gradient of the unclipped objective. To see this, note that the policy gradient (Equation 7) involves $f_{i,t}'(r_{i,t}) \cdot r_{i,t}$. At $r_{i,t} = 1$:
So $f_{i,t}'(1) \cdot 1 = 1$, matching the unclipped gradient weight of $r_{i,t} \cdot 1 = 1$ at the on-policy point. Without the $4 / \tau_i$ factor, the gradient at $r = 1$ would be $\tau_i / 4$, meaning even perfectly on-policy tokens would have their gradients scaled by a temperature-dependent fraction, changing the effective learning rate. The scaling factor preserves on-policy behavior—when the policy hasn't changed, SAPO reduces to standard policy gradient.
The Gradient Weight Function
Differentiating the objective reveals the effective weight applied to each token's log-policy gradient.
The gradient of the SAPO objective:
where $w_{i,t}(\theta)$ is the token-level gradient weight:
What it computes: The gradient is a weighted average of per-token policy gradients $\nabla_\theta \log \pi_\theta(y_{i,t} | \dots)$, each multiplied by three factors: (1) the importance ratio $r_{i,t}(\theta)$, which corrects for the distribution shift between the current and behavior policies (standard importance sampling); (2) the advantage $\hat{A}_i$, which determines the direction and magnitude of the update (positive = increase probability, negative = decrease); and (3) the soft gate $w_{i,t}(\theta)$, which attenuates the update based on how off-policy the token is. The weight $w_{i,t}(\theta)$ is a bell-shaped curve centered at $r_{i,t} = 1$:
- At
$r_{i,t} = 1$:$p = \sigma(0) = 0.5$, so$w = 4 \cdot 0.5 \cdot 0.5 = 1$. The token receives the full gradient (matching unclipped policy gradient). - As
$r_{i,t}$deviates from$1$in either direction:$p$approaches$0$or$1$, making$p(1-p)$approach$0$, so$w$approaches$0$. The gradient is smoothly down-weighted. - The rate of decay is controlled by
$\tau_i$: larger$\tau$makes the sigmoid steeper, causing$p$to saturate faster as$r$deviates, so$w$decays more rapidly to zero.
Why this form—the bell-shaped weight: The product $p(1-p)$ is the derivative of the sigmoid (up to scaling). It peaks at $p = 0.5$ (the on-policy point) and goes to zero as $p$ approaches the extremes. This creates a continuous trust region: the policy is most influenced by tokens that are near-on-policy, with influence decaying smoothly as tokens become more off-policy. Unlike hard clipping, there is no discontinuity where a token suddenly switches from fully weighted (1) to zero. A token with $r_{i,t} = 1.5$ and $\tau = 1$ receives $p = \sigma(0.5) \approx 0.622$, so $w = 4 \cdot 0.622 \cdot 0.378 \approx 0.94$—still contributing 94% of the full gradient, rather than being zeroed out as in GRPO with $\varepsilon = 0.2$. A token with $r_{i,t} = 5.0$ and $\tau = 1$ receives $p = \sigma(4) \approx 0.982$, so $w \approx 0.07$—still contributing a small gradient rather than being completely discarded.
Why this form—no gradient vanishing: In the unclipped regime, when $r_{i,t}$ becomes extremely large or small, the gradient $r_{i,t} \nabla_\theta \log \pi_\theta$ can explode or vanish respectively, causing unstable updates. Hard clipping prevents this by zeroing out gradients past a threshold—which also discards potentially useful information. SAPO's smooth weight $w_{i,t}(\theta)$ serves as a soft bound: it never exactly reaches zero (the sigmoid never exactly reaches 0 or 1), so gradients never truly vanish. Instead, extremely off-policy tokens contribute minimally, while moderately off-policy tokens contribute proportionally to their reliability.
Asymmetric Temperature Design
SAPO introduces a novel component not present in GRPO or GSPO: different temperatures for positive and negative advantage tokens, with $\tau_{\text{neg}} > \tau_{\text{pos}}$. The paper justifies this through an analysis of how token gradients propagate through the softmax over the vocabulary.
The gradient analysis (Equation 9): Consider a single token $y_{i,t}$ with advantage $\hat{A}_i$. Let $z = [z_1, z_2, \dots, z_{|V|}]$ be the logits output by the model before the softmax, where $|V|$ is the vocabulary size (typically ~100K-200K tokens). The token probability is $\pi_\theta(v | q, y_{i,<t}) = \exp(z_v) / \sum_{v'} \exp(z_{v'})$. The gradient of the log-probability with respect to each logit is:
Multiplying by the advantage $\hat{A}_i$:
What this means for positive advantages ($\hat{A}_i > 0$): The gradient increases the logit of the sampled token (by $(1 - \pi) \cdot \hat{A}_i$, a positive quantity) and decreases the logits of all $|V| - 1$ unsampled tokens (by $-\pi(v) \cdot \hat{A}_i$, a negative quantity for each). The total magnitude across unsampled tokens is $\hat{A}_i \sum_{v \neq y_{i,t}} \pi(v) = \hat{A}_i (1 - \pi(y_{i,t}))$, which exactly matches the magnitude of the increase to the sampled token. The update is balanced: it redistributes probability mass from unsampled tokens to the sampled token, keeping total mass approximately constant.
What this means for negative advantages ($\hat{A}_i \leq 0$): The gradient decreases the logit of the sampled token (by $(1 - \pi) \cdot \hat{A}_i$, a negative quantity since $\hat{A}_i \leq 0$) and increases the logits of all $|V| - 1$ unsampled tokens (by $-\pi(v) \cdot \hat{A}_i$, a positive quantity). The increase diffuses across the entire vocabulary—potentially hundreds of thousands of tokens—most of which are irrelevant or inappropriate alternatives to the sampled token. This makes negative updates inherently noisier: the gradient encourages the model to raise the probability of many unrelated tokens, which can destabilize the learned probability distribution, especially when the importance ratio is off-policy (meaning the behavior policy's token probabilities may not reflect what the current policy should learn).
The asymmetric temperature solution: SAPO sets $\tau_{\text{neg}} > \tau_{\text{pos}}$, making the weight decay faster for negative-advantage tokens. Concretely, with $\tau_{\text{pos}} = 1.0$ and $\tau_{\text{neg}} = 1.05$ (the main experimental setting):
- A positive-advantage token with
$r_{i,t} = 1.5$receives weight$w \approx 0.94$(near full gradient). - A negative-advantage token with the same ratio receives weight using
$\tau = 1.05$:$p = \sigma(1.05 \cdot 0.5) \approx 0.628$, so$w \approx 0.935$—slightly lower, but the difference grows for larger deviations. - At
$r_{i,t} = 3.0$: positive weight$\approx 0.42$, negative weight$\approx 0.38$. At$r_{i,t} = 10.0$: positive weight$\approx 0.002$, negative weight$\approx 0.0007$.
The asymmetry is modest in absolute terms but critical for stability. The paper's ablation (Figure 5) shows that reversing the asymmetry ($\tau_{\text{neg}} = 0.95 < \tau_{\text{pos}} = 1.0$) causes "significant instability," with training reward collapsing early. Using symmetric temperatures ($\tau_{\text{neg}} = \tau_{\text{pos}} = 1.0$) is more stable than the reversed setting but still less stable than the asymmetric configuration. This confirms that negative-token gradients are the primary source of instability and that damping them more aggressively is crucial.
Why this form rather than different clipping ranges: One could imagine a GRPO variant with a tighter clip for negative advantages (e.g., $\varepsilon_{\text{neg}} = 0.1$ vs. $\varepsilon_{\text{pos}} = 0.2$). The paper's temperature-based approach has two advantages. First, it is continuous: there is no hard boundary, so the transition from "trusted" to "distrusted" is smooth for both positive and negative tokens. Second, the temperature directly controls the rate of decay rather than the range of acceptance, which is a more natural parameterization for the soft gate. In GRPO's hard clip, changing $\varepsilon$ changes which tokens are included at all, not how much they are weighted. In SAPO, changing $\tau$ changes how quickly weight declines with off-policyness, preserving the property that all tokens contribute some gradient, just with varying confidence.
SAPO as a Sequence-Level Method (Reduction to GSPO)
Section 4.1 establishes a theoretical connection: under mild conditions, SAPO's token-level soft gate averages out to a sequence-level soft gate, meaning SAPO implicitly achieves GSPO-like sequence coherence without GSPO's hard sequence-level clipping.
The key insight: The gradient weight $w_{i,t}(\theta)$ can be written in terms of the token log-ratio $z_{i,t}(\theta) = \log r_{i,t}(\theta)$:
Under Assumption A1 (small-step/on-policy: $r_{i,t}(\theta) \approx 1$), we can approximate $r_{i,t}(\theta) - 1 \approx \log r_{i,t}(\theta)$, giving:
where $g_\tau(z) = \text{sech}^2(\frac{\tau}{2} z)$.
The sequence-level quantity: Define the log sequence ratio $\mu_i(\theta) = \log s_i(\theta) = \frac{1}{|y_i|} \sum_t \log r_{i,t}(\theta) = \frac{1}{|y_i|} \sum_t z_{i,t}(\theta)$.
Under Assumption A2 (low intra-sequence dispersion: the variance $\text{Var}_i(\theta) = \frac{1}{|y_i|} \sum_t (z_{i,t}(\theta) - \mu_i(\theta))^2$ is small), a second-order Taylor expansion shows:
The linear term vanishes because $\sum_t (z_{i,t} - \mu_i) = 0$. The second-order remainder is bounded by:
What this means operationally: When the within-sequence variance $\text{Var}_i(\theta)$ is small, the average of per-token soft gates across a sequence is well-approximated by a single sequence-level soft gate evaluated at the log sequence ratio $\log s_i(\theta)$. The gradient update then becomes:
This is structurally identical to GSPO's gradient, but with the hard clip replaced by the smooth gate $g_{\tau_i}(\log s_i(\theta))$. SAPO recovers GSPO's behavior as a special case when tokens within sequences are homogeneous.
When the assumptions hold: Figures 2 and 3 provide empirical validation. For the MoE model (Qwen3-30B-A3B):
- Token ratios
$r_{i,t}$are concentrated around 1, with the bulk of the distribution between 0.6 and 1.4 (Figure 2, left). - Per-sequence variance
$\text{Var}_i(\theta)$typically stays below 0.02, with a mode near 0.002 (Figure 2, middle). - The discrepancy
$D_i(\theta)$between average token gate and sequence gate is tiny—below 0.006 for most sequences, and well below the theoretical bound$(\tau^2/4) \cdot \text{Var}$(Figure 2, right).
For the dense model (Qwen3-4B), the concentration is even stronger: $r_{i,t}$ is between 0.85 and 1.15, $\text{Var}$ is below 0.006, and $D_i$ is below 0.0014 (Figure 3). The assumptions hold in the majority of cases, supporting the reduction.
When the assumptions are violated—SAPO's advantage: The crucial case is when a sequence contains a few outlier tokens with large $|z_{i,t} - \mu_i|$, making $\text{Var}_i(\theta)$ large. In this case:
- GSPO sees
$s_i(\theta)$pushed outside$[1 - \varepsilon, 1 + \varepsilon]$by the outliers and zeroes out gradients for all tokens in the sequence. - SAPO's token-level gate automatically handles the heterogeneity: the outlier tokens receive weight near zero (since
$|r_{i,t} - 1|$is large), while near-on-policy tokens continue to receive weights near 1. SAPO "defaults to its token-level gating" (Section 4.1), selectively down-weighting only the problem tokens.
This is the sequence-coherent, token-adaptive property the paper claims: SAPO behaves like GSPO when sequences are homogeneous (most of the time), but unlike GSPO, it gracefully handles heterogeneous sequences by falling back to per-token weighting. This prevents the signal loss that occurs when GSPO discards entire sequences due to a few outliers.
SAPO vs. GRPO: Smooth Gate vs. Hard Clip
Section 4.2 provides a direct comparison of the gradient weight functions.
GRPO's hard token gate (Equation 24): The derivative of GRPO's clipped surrogate gives the effective gradient weight:
This is a step function: weight = 1 inside the band $[1 - \varepsilon, 1 + \varepsilon]$, weight = 0 outside.
SAPO's smooth token gate: $w_{i,t}(\theta) = 4 \cdot \sigma(\tau_i(r_{i,t} - 1)) \cdot (1 - \sigma(\tau_i(r_{i,t} - 1)))$, a bell-shaped curve peaking at 1 when $r_{i,t} = 1$ and decaying continuously to near-zero at the extremes.
Key difference 1—continuity at the boundary: In GRPO, a token with $r_{i,t} = 1 + \varepsilon + 10^{-6}$ (just barely outside) receives zero gradient; a token with $r_{i,t} = 1 + \varepsilon - 10^{-6}$ receives full gradient. There is a cliff edge in the optimization landscape, creating potential for oscillatory behavior: a small parameter update can push a large batch of tokens across the boundary, causing the effective gradient to discontinuously jump. SAPO eliminates this cliff: the gradient weight changes smoothly with $r_{i,t}$, so small changes in policy produce small changes in effective gradient.
Key difference 2—non-uniform weighting inside the band: In GRPO, every token inside $[1 - \varepsilon, 1 + \varepsilon]$ receives exactly the same gradient weight (1), regardless of whether it is perfectly on-policy ($r = 1$) or near the boundary ($r = 1 + \varepsilon - \delta$). There is no incentive for the policy to stay close to the behavior policy, only to stay within the band. In SAPO, the weight peaks at $r = 1$ and decays even within the "trusted" region, providing a continuous incentive to minimize off-policyness: the closer to $r = 1$, the stronger the update. This encourages the policy to change gradually, which is a form of implicit trust-region regularization.
Key difference 3—non-zero weight outside the band: In GRPO, tokens with $|r_{i,t} - 1| > \varepsilon$ are discarded entirely—their gradient is zero. In SAPO, these tokens still contribute a small gradient, proportional to $w_{i,t}(\theta) \approx 4 \cdot \sigma(\tau_i(r_{i,t} - 1)) \cdot (1 - \sigma(\tau_i(r_{i,t} - 1)))$, which decays roughly exponentially for large $|r_{i,t}|$. This means SAPO preserves some learning signal from moderately off-policy tokens rather than discarding them, improving sample efficiency. The paper argues this is especially important in MoE models where routing heterogeneity produces more outlier tokens (Figures 2–3): GRPO would zero out these tokens' gradients, losing information, while SAPO down-weights them smoothly.
Key difference 4—shape symmetry: GRPO's hard gate is asymmetric in its treatment of upwards vs. downwards deviations: for positive advantages, it accepts $r \leq 1+\varepsilon$ and rejects $r > 1+\varepsilon$; for negative advantages, it accepts $r \geq 1-\varepsilon$ and rejects $r < 1-\varepsilon$. SAPO's gate is symmetric around $r = 1$ for a given $\tau$: the weight at $r = 1 + \delta$ equals the weight at $r = 1 - \delta$. The asymmetry in SAPO comes from $\tau$ differing by advantage sign, not from the gate shape. This decomposes the design into two orthogonal choices: (a) how quickly to decay with off-policyness (controlled by $\tau$), and (b) whether to decay faster for negative advantages (controlled by $\tau_{\text{neg}} > \tau_{\text{pos}}$). In GRPO, these are conflated in the clip parameter $\varepsilon$ and the sign-dependent direction of the clip.
Why smoothness matters for optimization dynamics: The paper argues that the discontinuities in GRPO's gradient lead to "brittle and unstable optimization behavior" (Section 4.2). When many tokens in a batch are near the clip boundary, small stochastic variations in $r_{i,t}$ (due to minibatch sampling, model stochasticity, or optimizer momentum) can cause tokens to flicker in and out of the trusted set, making the effective gradient noisy and potentially causing the optimizer to oscillate rather than converge smoothly. SAPO's smooth gate eliminates this flicker: as $r_{i,t}$ crosses any particular value, the weight changes continuously, so the effective gradient changes continuously. This is a form of gradient smoothing that reduces optimization noise.
Practical Implementation and Hyperparameters
The paper specifies several practical details for SAPO training:
Temperature values. The primary experimental configuration uses $\tau_{\text{pos}} = 1.0$ and $\tau_{\text{neg}} = 1.05$ (Section 5.1). The difference is small—only 5%—but the ablation in Figure 5 shows it is critical: $\tau_{\text{neg}} = 1.0$ (symmetric) is less stable, and $\tau_{\text{neg}} = 0.95$ (reversed) causes significant instability.
Group sampling and advantage normalization. Following the standard group-based recipe, SAPO uses the same advantage computation as GRPO and GSPO: $\hat{A}_i = (R_i - \text{mean}({R_j}_{j=1}^G)) / \text{std}({R_j}_{j=1}^G)$. The advantage is token-invariant within a response—every token in response $i$ receives the same $\hat{A}_i$. This is a deliberate design choice: since the reward $R_i$ is sequence-level (e.g., correctness of the final answer), there is no per-token reward signal to exploit. The advantage reflects the overall quality of the response, not any specific token's contribution.
Importance ratio computation. For each token $y_{i,t}$, two forward passes are required: one through the current policy $\pi_\theta$ and one through the frozen behavior policy $\pi_{\theta_{\text{old}}}$ (which can be cached from the rollout generation step). The ratio $r_{i,t}(\theta) = \pi_\theta(y_{i,t} | q, y_{i,<t}) / \pi_{\theta_{\text{old}}}(y_{i,t} | q, y_{i,<t})$ is computed directly from the token probabilities output by the softmax layer. No approximation or clipping is applied to the ratio itself before feeding it into the soft gate.
Mini-batching. During RL training, "each batch of rollout data is divided into four mini-batches for gradient updates" (Section 5.1). This means the model takes four gradient steps per batch of collected rollouts, reusing the same data with an updated policy. This improves data efficiency but increases off-policyness—since the policy changes between mini-batches, the importance ratios in later mini-batches reflect larger deviations from $1$. SAPO's soft gate is particularly well-suited to this setting because it can handle moderate off-policy drift without discarding data, whereas GRPO's hard clip might zero out most tokens after a few mini-batches.
No routing replay. The paper notes that SAPO "does not rely on routing replay for stabilization or strong performance" (Section 5.1). GRPO-R2 uses routing replay—a technique for MoE models that resamples expert routing decisions from the behavior policy during gradient computation to reduce variance. SAPO's stability does not require this auxiliary technique, which "improves exploration and reduces engineering overhead for RL systems." This is a practical advantage: routing replay requires storing and replaying expert assignments, adding complexity and potentially constraining the policy update (since routing decisions are frozen).
Large-scale training (Qwen3-VL). For the Qwen3-VL experiments (Section 5.2), the training setup includes: "a broad collection of text and multimodal tasks, including mathematics, coding, and logical reasoning," with "a fixed sampling ratio for each task within each batch" to support multi-task learning. Each batch of rollout data is split into "two mini-batches for gradient updates, ensuring that each mini-batch provides sufficient learning signal for all tasks." The larger batch size and fewer mini-batches (2 vs. 4) reflect the multi-task setting, where each mini-batch must contain examples from all tasks to prevent catastrophic forgetting.
Model architecture. The controlled experiments use a cold-start checkpoint fine-tuned from Qwen3-30B-A3B-Base—a Mixture-of-Experts model with 30B total parameters and 3B active parameters. The dense model ablation uses Qwen3-4B. The VL experiments extend to additional model sizes and architectures within the Qwen3-VL family, covering both MoE and dense variants.
Design Choice Summary
The paper makes four key design choices that distinguish SAPO from prior work, each with a specific justification:
-
Sigmoidal soft gate instead of hard clip (Sections 3, 4.2): Hard clipping creates discontinuities in the optimization landscape (tokens flicker between fully weighted and zero-weighted) and discards moderately off-policy tokens that still carry useful signal. The sigmoid provides a continuous trust region where gradient contribution decays smoothly with off-policyness, reducing optimization noise and preserving sample efficiency.
-
Temperature-controlled decay rate via
$\tau$(Section 3): The temperature parameter directly controls how quickly weight decays with off-policyness, providing a tunable knob for the trust-region width. Smaller$\tau$allows larger policy deviations before attenuation; larger$\tau$enforces stricter on-policy behavior. This is more interpretable than GRPO's$\varepsilon$clip range because it controls the rate of decay rather than a binary boundary. -
Asymmetric temperatures (
$\tau_{\text{neg}} > \tau_{\text{pos}}$) (Section 3): Negative-advantage gradients diffuse to the entire vocabulary of unsampled tokens, making them inherently noisier than positive-advantage gradients which concentrate on a single token. Setting$\tau_{\text{neg}} > \tau_{\text{pos}}$applies stronger attenuation to negative updates, damping their destabilizing effect. The ablation (Figure 5) confirms this is critical for avoiding early collapse. -
Token-adaptive with sequence-level coherence (Sections 4.1, 4.3): SAPO operates at the token level (like GRPO) but its gate averages to a sequence-level gate under mild conditions (like GSPO). This hybrid design automatically handles both homogeneous sequences (where it behaves like GSPO with a smooth gate) and heterogeneous sequences (where it selectively down-weights outlier tokens while preserving near-on-policy tokens within the same sequence). The paper demonstrates empirically that the conditions for sequence-level coherence hold in practice (Figures 2–3).
4. Key Insights and Innovations
Innovation 1: Reframing Policy Constraints as a Continuous Trust Region Problem Rather Than a Binary Accept/Reject Decision
The dominant paradigm for constraining policy updates in group-based RL for LLMs—inherited from PPO and preserved in both GRPO and GSPO—treats the trust region as a hard boundary: tokens inside the clipping band [1 - ε, 1 + ε] contribute fully to the gradient, while tokens outside contribute nothing. SAPO's most fundamental conceptual move is to reject this binary framing entirely and replace it with a continuous trust region, where every token contributes some learning signal, but the degree of contribution decays smoothly with off-policyness.
This reframing matters because it changes what the optimization algorithm is communicating to the policy. In the hard-clipping paradigm, the message is "stay within this band or your updates will be discarded"—a cliff-edge incentive that punishes exploration while also being brittle to stochasticity. In SAPO's continuous framing, the message is "the more you deviate from the behavior policy, the less we trust your gradient"—a graduated incentive that naturally encourages conservative updates without ever fully discarding information. The optimization landscape becomes smooth rather than piecewise, eliminating the discontinuities that cause the flickering behavior where tokens near the boundary oscillate between fully-weighted and zero-weighted across minibatches.
This is not merely a smoother version of GRPO's clip—it is a fundamentally different philosophy about what a trust region means. In GRPO, the trust region is a constraint (enforced by zeroing gradients outside the band). In SAPO, the trust region is a regularizer (enforced by continuously down-weighting contributions proportional to their unreliability). The paper does not frame it in these terms explicitly, but the mathematical structure makes the distinction clear: SAPO's gradient weight w_{i,t}(θ) is never zero, only asymptotically small, meaning the algorithm always trusts the direction of the gradient to some degree, just with modulated confidence. This is a more principled approach to off-policy learning because importance sampling correction is itself a continuous quantity—r_{i,t} being 1.01 versus 1.99 reflects genuinely different signal reliability—and a binary threshold throws away that gradation.
The significance of this reframing extends beyond SAPO's specific sigmoidal gate. It suggests that future work on policy constraints for LLMs should move away from PPO-style clipping entirely and toward continuous attenuation functions, of which SAPO's sigmoid is one instance. The paper does not explore alternative decay functions (exponential, Gaussian), but the conceptual framework—constraints as continuous regularizers rather than hard boundaries—opens that design space.
Evidence for this innovation's practical importance comes from Figure 4: SAPO sustains stable training for 1750+ gradient steps while GSPO and GRPO-R2 both exhibit early collapse. The key mechanism is not empirically ablated in isolation (we cannot turn off smoothness while keeping other SAPO properties), but the theoretical argument in Section 4.2—that hard clipping creates all-or-nothing gates leading to "brittle and unstable optimization behavior"—is well-supported by the juxtaposition of GRPO's step-function gate (Equation 24) against SAPO's bell-shaped weight (Figure 1, right panel). The visual contrast in Figure 1 makes the conceptual point economically: the unclipped objective grows unboundedly, the hard-clipped objective has a sharp kink, and the soft-gated objective smoothly saturates—three qualitatively different optimization landscapes.
Innovation 2: Diagnosing and Addressing the Asymmetric Instability of Positive vs. Negative Token Updates
Prior work on group-based RL for LLMs applies the same clipping mechanism symmetrically to positive-advantage and negative-advantage tokens. GRPO uses the identical clip range ε regardless of whether the advantage is positive or negative; GSPO does the same at the sequence level. SAPO introduces a diagnostic insight that the paper formally derives but whose implications extend beyond the specific solution: negative-advantage token gradients are fundamentally noisier and more destabilizing than positive-advantage gradients because of how the softmax over a massive vocabulary diffuses gradient signal.
This is a genuine diagnostic contribution—it identifies why training instability occurs rather than merely observing that it does. The derivation in Equation 9 is simple but revealing: when the advantage is positive, the gradient concentrates probability mass onto the single sampled token (increasing its logit) while symmetrically decreasing mass from all unsampled tokens. The total magnitude of the increase to the sampled token exactly balances the total decrease across all unsampled tokens—a zero-sum redistribution that preserves the overall shape of the distribution. When the advantage is negative, the gradient does the opposite: it deflates the sampled token and inflates every one of the potentially hundreds of thousands of unsampled tokens in the vocabulary. Since only a tiny fraction of those unsampled tokens are genuinely appropriate alternatives, the negative gradient effectively injects noise into the model's probability distribution across a vast action space.
This asymmetry is not a bug in GRPO or GSPO—it is inherent to the structure of autoregressive language generation with a softmax output layer. But prior work did not recognize or address it. The field's default assumption was that policy updates should be symmetric: if you clip positive updates at 1 + ε, you should clip negative updates at 1 - ε. SAPO's insight is that this symmetry is a category error: the gradient geometry is different for positive vs. negative advantages, so the constraint mechanism should be different too.
The solution—τ_neg > τ_pos—is an engineering response to a theoretical diagnosis, but the diagnosis itself is the more valuable contribution. It reveals that the search for stable RL algorithms for LLMs should not treat all tokens as interchangeable units of gradient signal. Some tokens (those with negative advantages) are inherently riskier to update on, and the optimization algorithm should reflect that asymmetry. The paper's ablation in Figure 5 provides empirical validation: reversing the asymmetry (τ_neg < τ_pos) causes "significant instability," while the symmetric case (τ_neg = τ_pos) is intermediate. This three-way comparison confirms that the asymmetry is not merely a minor hyperparameter tuning choice but a structural requirement for stability.
This insight is fundamental rather than incremental because it changes what practitioners should optimize when designing policy constraint mechanisms for LLMs. Rather than searching for a single optimal clip range, the design space now includes per-advantage-sign constraint parameters—or more generally, constraint mechanisms that are explicitly sensitive to the gradient's diffusion properties. SAPO implements this through asymmetric temperatures; future work might implement it through asymmetric clip ranges, different decay functions for positive and negative tokens, or even advantage-dependent learning rates. The paper opens this design axis without exhaustively exploring it.
Innovation 3: Achieving Sequence-Level Coherence as an Emergent Property of Token-Level Design
GSPO made a compelling argument: because rewards in LLM RL are assigned to complete sequences, policy optimization should operate at the sequence level to maintain coherence between the optimization granularity and the reward structure. It implemented this through explicit sequence-level clipping—a direct structural choice. SAPO takes a more subtle approach: it designs a token-level mechanism that reduces to sequence-level behavior under the conditions that empirically hold most of the time, while retaining token-level adaptivity as a fallback when those conditions are violated.
This is a distinctive conceptual move because it inverts the relationship between token-level and sequence-level optimization. In GSPO, sequence-level behavior is enforced—the algorithm explicitly computes a single ratio s_i(θ) and applies a single gate to all tokens. In SAPO, sequence-level behavior is emergent—it arises from the concentration of token-level gates under mild assumptions about the data distribution. The paper formalizes this through the Taylor expansion argument in Section 4.1, which shows that the average token gate concentrates to the sequence gate g_τ(log s_i(θ)) with error bounded by (τ² / 4) · Var_i(θ).
The practical consequence is a "best of both worlds" property that neither GRPO nor GSPO achieves. When sequences are homogeneous (most tokens have similar importance ratios, which Figures 2–3 show is common), SAPO automatically behaves like GSPO—coherent sequence-level weighting aligned with the reward. When sequences contain outlier tokens (a few tokens with wildly different importance ratios, caused perhaps by expert routing changes in MoE models or by rare vocabulary items), SAPO gracefully degrades to token-level weighting, down-weighting only the outliers while preserving signal from near-on-policy tokens. GSPO, by contrast, would discard the entire sequence when outliers push s_i(θ) outside the clipping band—a catastrophic loss of information.
This is fundamentally an argument about robustness through design rather than through tuning. GSPO's sequence-level clipping is theoretically clean but practically fragile: its behavior depends on a single threshold ε applied to a scalar summary statistic s_i(θ). A small number of outlier tokens can corrupt that summary statistic and trigger the clip. SAPO's design is robust to outliers not because it has a better-tuned threshold, but because its mechanism operates at the granularity where outliers manifest (the token level) while still aggregating to sequence-level behavior when outliers are absent.
The empirical validation in Figures 2–3 is crucial: it shows that the conditions for sequence-level coherence (small r_{i,t} deviations, low Var_i(θ)) hold in practice, meaning SAPO achieves GSPO-like behavior without GSPO's brittleness for the majority of training examples. The rightmost panels show that the bound D_i(θ) ≤ (τ² / 4) · Var_i(θ) is tight—the actual discrepancy between average token gate and sequence gate is small, typically below 0.001. This provides a rare example in deep learning of a theoretical reduction that is both mathematically clean and empirically validated on real training data at scale (10⁹ tokens, 10⁵ sequences).
This innovation is fundamental rather than incremental because it proposes a new design principle for policy optimization in sequence models: design at the token level for robustness, but ensure that the mechanism reduces to sequence-level behavior under the data distribution's typical conditions. This principle could guide the design of future algorithms beyond SAPO's specific sigmoidal gate, and it suggests that the token-vs-sequence debate in LLM RL is a false dichotomy—the right answer is to have both, with the algorithm transitioning smoothly between regimes based on data characteristics.
Innovation 4: Demonstrating That Smoothness Alone—Without Auxiliary Stabilization—Suffices for Stable MoE Training
The paper includes a negative finding that carries substantial practical significance: SAPO achieves stable training on MoE architectures (Qwen3-30B-A3B) without routing replay, an auxiliary technique used in GRPO-R2 specifically to stabilize MoE training. The paper states this plainly in Section 5.1: "SAPO does not rely on routing replay for stabilization or strong performance, which improves exploration and reduces engineering overhead for RL systems."
This is important because Mixture-of-Experts models present unique challenges for RL fine-tuning. Expert routing decisions change when the model parameters are updated, creating additional variance in importance ratios beyond what occurs in dense models. Figures 2–3 confirm this empirically: the MoE model shows wider r_{i,t} distributions and higher Var_i(θ) than the dense model. Prior work responded to this challenge by introducing routing replay—a technique that stores and replays the expert assignments from the behavior policy, effectively freezing routing decisions during gradient computation to reduce variance. This is an engineering fix that adds complexity, constrains the policy update (since routing cannot adapt during the replay), and requires additional memory.
SAPO's finding suggests that routing replay may be unnecessary when the policy constraint mechanism is sufficiently robust to handle the variance that MoE routing introduces. The smooth gate's continuous down-weighting naturally handles the wider r_{i,t} distribution: outlier tokens caused by routing changes are attenuated rather than triggering hard clips that discard entire sequences. This is a cleaner solution than routing replay because it addresses the root cause (high variance) rather than patching one of its sources (routing changes).
The significance of this finding extends beyond SAPO. It suggests that the perceived instability of MoE models under RL may be partly an artifact of hard clipping's brittleness—when the constraint mechanism creates cliff edges, the additional variance from routing pushes more tokens across the boundary, causing training collapse. A smoother constraint mechanism that can handle moderate variance may eliminate the need for MoE-specific stabilization. If this finding generalizes to other MoE architectures and RL algorithms, it would simplify the RL training pipeline for the increasingly common class of MoE LLMs, removing a specialized component that requires additional engineering and hyperparameter tuning.
This innovation is incremental relative to SAPO's other contributions—it is a consequence of the smooth gate design rather than a separate conceptual advance—but it has outsized practical importance for the growing number of practitioners training MoE models with RL. The paper does not ablate whether GRPO with a wider clip range could also eliminate the need for routing replay, so the precise attribution (smoothness vs. simply being more permissive) is unclear. But the result stands as evidence that SAPO's design choices compound to meaningful practical simplifications.
5. Experimental Analysis
Evaluation Methodology
-
Dataset. The controlled experiments (Section 5.1) use mathematical reasoning queries—the paper does not specify the exact training dataset name, size, or source, describing it only as training "on mathematical reasoning queries." Validation is conducted on three benchmarks: AIME25 (AIME, 2025), HMMT25 (HMMT, 2025), and BeyondAIME (Seed et al., 2025), all of which are competition-level mathematical reasoning datasets. The Qwen3-VL experiments (Section 5.2) use "a broad collection of text and multimodal tasks, including mathematics, coding, and logical reasoning," evaluated on AIME25, LiveCodeBench v6 (Jain et al., 2024), ZebraLogic (Lin et al., 2025), and MathVision (Wang et al., 2024).
-
Base model(s). The controlled experiments use a cold-start checkpoint fine-tuned from Qwen3-30B-A3B-Base—a Mixture-of-Experts model with 30B total parameters and 3B active parameters. The paper also uses Qwen3-4B (a dense model) for the empirical validation of assumptions in Figures 2–3. The Qwen3-VL experiments extend to multiple model sizes and architectures within the Qwen3-VL family, covering both MoE and dense variants (Section 5.2). The choice of Qwen3 models is motivated by their representativeness of contemporary LLM capabilities and the practical goal of demonstrating SAPO's effectiveness in real training pipelines.
-
Metrics. The primary metric is Pass@1 accuracy on validation benchmarks, computed as the average over multiple samples per query—16 samples for AIME25, HMMT25, and BeyondAIME in the controlled experiments; 32 samples for AIME25 and 8 samples for LiveCodeBench v6 in the VL experiments. For the VL experiments, an aggregated validation score (Figure 6, right panel) combines performance across all four benchmarks into a single metric. Training progress is also monitored via training reward (the average reward of sampled responses during RL training), plotted alongside validation performance.
-
Baselines. Three baselines are compared against SAPO:
- GRPO-R2—GRPO (Shao et al., 2024) equipped with routing replay (Zheng et al., 2025), a stabilization technique for MoE models that replays expert routing decisions from the behavior policy during gradient computation.
- GSPO—Group Sequence Policy Optimization (Zheng et al., 2025), which applies hard clipping at the sequence level using the length-normalized importance ratio
s_i(θ). - For the temperature ablation in Figure 5, SAPO is compared against itself with two alternative temperature configurations:
τ_neg = τ_pos = 1.0(symmetric) andτ_neg = 0.95 < τ_pos = 1.0(reversed asymmetry).
The controlled experiments use "the same hyperparameter configurations as in Zheng et al. (2025)" for GRPO-R2 and GSPO (Section 5.1), ensuring a fair comparison under identical experimental settings.
-
Generation budget / compute accounting. The paper compares methods under equal compute budgets, measured implicitly by the number of gradient steps and the training wall-clock time. In the controlled experiments (Figure 4), all methods are trained for the same number of gradient steps (~1750), with identical rollout generation costs per step (since all methods sample
Gresponses per query from the behavior policy, and the group size is held constant across methods). In the VL experiments (Section 5.2), the comparison is explicitly "under the same compute budget" (Figure 6 caption). The paper does not provide a FLOPs accounting or generation budget in tokens, relying instead on the equal-gradient-steps metric. Mini-batching is consistent across methods: each batch of rollout data is divided into four mini-batches for gradient updates in the controlled experiments, and two mini-batches in the VL experiments. -
Cross-validation / statistical protocol. The paper does not report any cross-validation procedure. Validation performance is reported as Pass@1 accuracy at specific gradient steps during training, computed on the full validation sets of each benchmark. The paper does not report error bars, confidence intervals, or statistical significance tests. Results are presented as single curves over gradient steps (Figures 4–6), leaving the statistical reliability of the observed performance gaps unquantified.
Main Quantitative Results
Controlled Experiments: SAPO vs. GSPO and GRPO-R2 on Mathematical Reasoning
Headline result. Figure 4 shows SAPO maintaining stable training dynamics throughout the full 1750 gradient steps and achieving the highest final Pass@1 performance across all three benchmarks, while both GSPO and GRPO-R2 exhibit early training collapse. Specifically:
-
Training reward (Figure 4, top-left): SAPO's training reward rises steadily from approximately 0.50 to approximately 0.82–0.84 by step 1750, with no visible degradation. GSPO's reward rises to roughly 0.80 by step ~500, then enters a volatile phase with large oscillations and eventually collapses downward to approximately 0.45 by step 1250. GRPO-R2 follows a similar trajectory: reward peaks around 0.80 by step ~500, then collapses to roughly 0.50 by step 1250.
-
HMMT25 (Figure 4, top-right): SAPO reaches approximately 0.640–0.650 Pass@1 by step 1750, continuing to improve throughout training. GSPO peaks around 0.625–0.630 at step ~500–750, then declines to roughly 0.55 by step 1500. GRPO-R2 peaks around 0.600 at step ~500 and then deteriorates—the curve ends early, consistent with training collapse.
-
BeyondAIME (Figure 4, bottom-left): SAPO achieves approximately 0.56–0.58 Pass@1 at the final steps, with steady improvement. GSPO reaches roughly 0.55 at its peak (step ~500) and then declines. GRPO-R2 peaks around 0.54 and collapses.
-
AIME25 (Figure 4, bottom-right): SAPO reaches approximately 0.80–0.82 Pass@1 by step 1750. GSPO peaks around 0.80 at step ~500–750 before declining. GRPO-R2 peaks around 0.78 before collapsing.
Key timing pattern. Across all benchmarks, the performance gap between SAPO and the baselines is not primarily about peak performance—GSPO and GRPO-R2 achieve peak Pass@1 values that are within 2–5 percentage points of SAPO's early values (e.g., on AIME25, GSPO reaches ~0.80 while SAPO at that step is ~0.76). The gap emerges from SAPO's ability to sustain and extend learning beyond the point where baselines collapse. By step 1750, SAPO's Pass@1 exceeds the baseline peaks by 2–5 points on most benchmarks, and the baselines have fallen substantially below their own peaks.
Side-by-side comparison at matched compute. At step 1750 (the final reported point), SAPO outperforms:
- GSPO on HMMT25 by approximately 0.09–0.10 Pass@1 (0.64–0.65 vs. ~0.55)
- GSPO on BeyondAIME by approximately 0.02–0.03 Pass@1 (0.57 vs. ~0.54)
- GSPO on AIME25 by approximately 0.02 Pass@1 (0.81 vs. ~0.79 post-peak decline)
- GRPO-R2 shows even larger gaps, with final performance substantially below SAPO across all benchmarks (GRPO-R2 curves terminate or degrade visibly in each panel)
The paper does not provide a table of exact final numbers, so these estimates are read from Figure 4. The absence of tabulated results with precise values is a notable omission that makes quantitative comparison approximate.
The "early-stage training collapse" claim. The paper asserts that both GSPO and GRPO-R2 "exhibit early-stage training collapse" (Section 5.1). Figure 4 supports this: both baselines show training reward increasing to a peak around step 500 and then declining, accompanied by a drop in validation Pass@1. However, the collapse is not catastrophic—GSPO on AIME25 maintains Pass@1 above 0.75 even as reward declines, and on BeyondAIME the decline is modest (~0.55 to ~0.53). The collapse is most dramatic in training reward, where both baselines eventually fall well below their peaks, though validation performance degrades more gradually. The paper's claim of "collapse" is primarily supported by the training reward trajectory; the validation performance curves show a softer degradation that might be characterized as "stagnation followed by decline" rather than catastrophic collapse.
Asymmetric Temperature Ablation
Headline result. Figure 5 demonstrates that SAPO's asymmetric temperature design (τ_neg > τ_pos) is critical for training stability. Three configurations are compared:
-
SAPO (
τ_neg = 1.05 > τ_pos = 1.0): This is the primary configuration. Training is stable throughout 1750 steps, with reward reaching ~0.82 and validation Pass@1 climbing steadily on all benchmarks. -
SAPO (
τ_neg = τ_pos = 1.0): Symmetric temperatures. Training reward initially rises similarly but begins to exhibit instability around step 1000–1250, with reward dropping from ~0.80 to ~0.72 before partially recovering. On HMMT25, Pass@1 reaches ~0.62—lower than the asymmetric configuration's ~0.65. On BeyondAIME, Pass@1 plateaus around 0.53–0.54, 3–4 points below asymmetric SAPO. On AIME25, Pass@1 reaches ~0.79, about 1–2 points below asymmetric SAPO. The degradation is visible but not catastrophic—training does not fully collapse, but performance is consistently lower and reward is more volatile. -
SAPO (
τ_neg = 0.95 < τ_pos = 1.0): Reversed asymmetry (negative tokens given smaller temperature = less attenuation). This configuration exhibits "significant instability" (Section 5.1): training reward collapses sharply around step 750, falling from ~0.80 to below 0.60. On HMMT25, Pass@1 plateaus and then declines, ending below 0.60—worse than both other configurations. On BeyondAIME, performance collapses to ~0.46 by step 1250. On AIME25, performance degrades to ~0.74. The paper describes this as "significant instability," and Figure 5 supports this characterization: the reversed configuration is clearly the worst across all benchmarks.
Quantitative impact of asymmetry. The difference between asymmetric (τ_neg = 1.05) and symmetric (τ_neg = 1.0) is modest but consistent: approximately 2–3 Pass@1 points on most benchmarks at the final step. The difference between asymmetric and reversed (τ_neg = 0.95) is substantial: 5+ points on HMMT25, 10+ points on BeyondAIME, and 6+ points on AIME25 by the end of training. This confirms that while the asymmetry is small in magnitude (5% difference in τ), its effect on training dynamics is significant—particularly that reversing the asymmetry is actively harmful rather than merely suboptimal.
Important nuance. The symmetric configuration (τ_neg = τ_pos) does not collapse—it underperforms the asymmetric configuration but maintains reasonable stability. This suggests that the smooth gating mechanism itself provides substantial stabilization, and the asymmetric temperature is a further refinement that adds robustness against the specific instability caused by negative-token gradients. The paper's narrative emphasizes the asymmetry as critical, but the data in Figure 5 suggests the smooth gate alone handles the majority of the stability problem, with asymmetry providing the final margin. No ablation of "smooth gate without asymmetry vs. GRPO" is reported, so the independent contribution of smoothness cannot be isolated from these experiments.
Large-Scale Multi-Task Training: Qwen3-VL-30B-A3B
Headline result. Figure 6 demonstrates that SAPO scales effectively to large-scale, multi-task, multimodal RL training. Starting from a preliminary cold-start checkpoint of Qwen3-VL-30B-A3B and training on a mixture of text and multimodal tasks, SAPO achieves consistent performance gains throughout 500 gradient steps and outperforms both GSPO and GRPO-R2 under the same compute budget.
-
Training reward (Figure 6, left): SAPO's reward rises from approximately 0.525 at step 0 to approximately 0.675 at step 500, with a steady upward trajectory and no visible instability. GSPO's reward rises to roughly 0.625 then plateaus. GRPO-R2's reward rises to roughly 0.600 then begins to decline after step 400, ending near 0.575.
-
Aggregated validation score (Figure 6, right): SAPO improves from approximately 0.720 to approximately 0.760 by step 500. GSPO reaches approximately 0.745 at its peak (step ~250) before stagnating. GRPO-R2 reaches approximately 0.740 at its peak and then declines. At step 500, SAPO leads GSPO by roughly 0.010–0.015 and GRPO-R2 by roughly 0.020 in the aggregated metric.
Key observations for the VL setting. The performance gaps in this multi-task setting are smaller than in the controlled mathematical reasoning experiments (roughly 1–2% relative improvement over GSPO vs. 3–15% on individual math benchmarks). Several factors may contribute: (1) the training run is shorter (500 steps vs. 1750)—baselines may not have had time to fully collapse; (2) the multi-task mixture may inherently dampen instability because gradient conflicts between tasks prevent any single task from dominating and causing collapse; (3) the aggregated metric obscures per-task differences—SAPO might show larger gains on specific tasks that are masked by aggregation. The paper does not provide per-task breakdowns for the VL experiments, making it impossible to assess whether SAPO's benefits are uniform across tasks or concentrated in specific domains.
Consistency with controlled experiments. Despite smaller gaps, the qualitative pattern from Figure 4 is replicated in Figure 6: SAPO sustains monotonic improvement throughout training, GSPO plateaus, and GRPO-R2 exhibits late-stage decline. The stability advantage of SAPO is robust across model scales, architectures (MoE), and task mixtures (text + multimodal).
Ablation Studies and Robustness Checks
Temperature symmetry ablation (Figure 5): As discussed in the Main Quantitative Results section above, testing τ_neg = 1.05, τ_neg = 1.0, and τ_neg = 0.95 with fixed τ_pos = 1.0 demonstrates that the asymmetric configuration (τ_neg > τ_pos) yields the best stability and performance, the symmetric configuration is intermediate, and the reversed configuration causes significant instability. This is the only systematic ablation reported in the paper.
Empirical validation of theoretical assumptions (Figures 2–3): While not an ablation in the traditional sense, the paper provides histograms of r_{i,t}(θ) and per-sequence log-ratio variance Var_i(θ) for both MoE (Qwen3-30B-A3B) and dense (Qwen3-4B) models, computed over "more than 10^5 sequences and 10^9 tokens drawn from off-policy mini-batches" (Section 4.1). These validate Assumptions A1 (small on-policy steps, r_{i,t} ≈ 1) and A2 (low intra-sequence dispersion, Var_i(θ) small). The rightmost panels show the discrepancy D_i(θ) between average token gate and sequence gate is well below the theoretical bound (τ²/4) · Var_i(θ), confirming the reduction to sequence-level behavior. For the MoE model: r_{i,t} is concentrated between 0.6 and 1.4, Var_i(θ) mode ~0.002, D_i(θ) below 0.006. For the dense model: r_{i,t} between 0.85 and 1.15, Var_i(θ) below 0.006, D_i(θ) below 0.0014. The assumptions hold more tightly for dense models than for MoE models, consistent with the paper's claim that MoE routing heterogeneity increases importance ratio variance.
MoE vs. dense architecture (Figures 2–3): The side-by-side histograms for MoE and dense models serve as a qualitative ablation of model architecture's effect on SAPO's assumptions. The MoE model shows a wider r_{i,t} distribution and higher Var_i(θ) than the dense model, confirming that MoE architectures amplify the variance problem that SAPO is designed to handle. The paper does not, however, provide a direct SAPO-vs-baseline comparison on dense models—all RL training experiments use the MoE Qwen3-30B-A3B. This leaves open the question of whether SAPO's advantages over GSPO and GRPO-R2 are larger, smaller, or similar on dense architectures.
Absence of routing replay (Section 5.1): SAPO does not use routing replay, unlike GRPO-R2. The paper frames this as evidence that "SAPO does not rely on routing replay for stabilization or strong performance" (Section 5.1). While not a formal ablation (there is no SAPO-with-routing-replay condition to compare against), the fact that SAPO without routing replay outperforms GRPO-R2 with routing replay suggests that the smooth gate provides sufficient stabilization that the auxiliary technique is unnecessary. However, the comparison confounds two differences: (1) SAPO's smooth gate vs. GRPO's hard clip, and (2) SAPO's token-level operation with sequence-level coherence vs. GRPO's pure token-level operation. The absence of routing replay cannot be attributed solely to smoothness.
Missing ablations. Several experiments that would strengthen the paper are absent:
- SAPO with hard clip would isolate the contribution of smoothness from other design choices. If SAPO-hard-clip performed similarly to SAPO-soft-gate, the smoothness claim would be weakened.
- GRPO with asymmetric clipping (different
εfor positive and negative advantages) would test whether the asymmetry insight is SAPO-specific or generalizable to hard-clipping methods. - Varying
τ_posandτ_negindependently beyond the singleτ_pos = 1.0setting would characterize the sensitivity of performance to these hyperparameters and potentially identify better configurations. - SAPO on dense models would test whether the stability advantage is specific to MoE architectures or general.
- Longer training runs beyond 1750 steps would reveal whether SAPO itself eventually collapses or whether it can sustain stable learning indefinitely. The paper states SAPO "sustains coherent learning for a longer duration" but does not establish whether it sustains it indefinitely.
- Per-task breakdown for VL experiments would reveal whether SAPO's benefits are concentrated in specific domains (e.g., mathematical reasoning) or uniform across tasks.
Critical Assessment
Claim from executive summary: "SAPO achieves both improved training stability and higher final Pass@1 performance compared to GSPO and GRPO-R2 under comparable compute budgets."
What the experiments demonstrate: Figure 4 clearly shows SAPO maintaining stable training through 1750 steps while GSPO and GRPO-R2 exhibit training reward degradation and validation performance decline after ~500 steps. SAPO's final Pass@1 exceeds the baselines on all three mathematical reasoning benchmarks. Figure 6 replicates this pattern in a large-scale multi-task VL setting over 500 steps.
What the experiments do not demonstrate: The paper does not establish that GSPO and GRPO-R2 are optimally tuned. The hyperparameters are taken from Zheng et al. (2025)—"the same hyperparameter configurations as in Zheng et al. (2025)" (Section 5.1)—which were optimized for GSPO and GRPO-R2 in that prior work. However, it is possible that different clip ranges ε, learning rates, or group sizes would improve baseline stability without requiring SAPO's smooth gate. The paper does not report any hyperparameter sweep for any method. The collapse of GSPO and GRPO-R2 might reflect suboptimal hyperparameters rather than an inherent limitation of hard clipping. A fairer comparison would tune each method's key hyperparameters independently and report the best configuration for each. The current comparison demonstrates that SAPO with its chosen hyperparameters outperforms GSPO and GRPO-R2 with their chosen hyperparameters—a weaker claim than "SAPO outperforms optimally-tuned GSPO and GRPO-R2."
Conditions and boundaries: The claim holds for the specific setting tested: Qwen3-30B-A3B (MoE) fine-tuned from a cold-start checkpoint on mathematical reasoning, with τ_pos = 1.0, τ_neg = 1.05, and four mini-batches per rollout. The paper does not test sensitivity to model size, architecture family (non-Qwen models), task domain (non-math reasoning), or RL training stage (warm-start vs. cold-start beyond the single cold-start setting). The VL experiments partially address the task domain and architecture generalization, but with the caveat that performance gaps are small and per-task breakdowns are not provided.
Claim from executive summary: "The asymmetric temperature design—setting τ_neg > τ_pos to more aggressively dampen high-variance negative-token gradients—[proves] critical for avoiding early training collapse."
What the experiments demonstrate: Figure 5 provides a clean three-way comparison that supports this claim. The asymmetric configuration (τ_neg = 1.05) is clearly best; the symmetric configuration (τ_neg = 1.0) is intermediate; the reversed configuration (τ_neg = 0.95) causes the worst instability. The qualitative ordering is consistent with the paper's theoretical argument that negative-token gradients are inherently noisier.
What the experiments do not demonstrate: The ablation tests only three temperature values (0.95, 1.0, 1.05), all with τ_pos = 1.0. This is a narrow sweep around the chosen configuration. The paper does not establish that 1.05 is optimal—larger τ_neg values (e.g., 1.1, 1.2, 1.5) might yield even better stability, or might over-dampen useful negative signals and reduce performance. The paper also does not test whether the optimal τ_neg value depends on τ_pos—perhaps the ratio τ_neg / τ_pos matters more than the absolute values. Furthermore, whether this asymmetry is specifically needed for MoE models or generalizes to dense models is untested, since all RL experiments use MoE architectures. The theoretical argument about vocabulary size and gradient diffusion (Equation 9) is architecture-agnostic and should apply to dense models as well, but the magnitude of the effect may differ.
Conditions and boundaries: The "critical" label is supported for the specific τ_pos = 1.0 configuration on Qwen3-30B-A3B. Whether asymmetric temperatures are always critical, or critical only in certain regimes (cold-start, MoE, specific τ_pos values), is not established. The symmetric configuration, while worse, does not catastrophically collapse—it sustains reasonable performance—so "critical" may overstate the empirical necessity for this specific training setup.
Claim from executive summary: "SAPO sustains coherent learning for a longer duration and reaches higher Pass@1 accuracy before divergence."
What the experiments demonstrate: Figure 4 shows SAPO still improving at step 1750 while baselines have plateaued or declined. The claim about "longer duration" is supported. The phrase "before divergence" implies SAPO will eventually diverge too—this is consistent with the paper's statement that "all methods may ultimately exhibit signs of instability" (Section 1)—but the experiments do not run long enough to observe SAPO's own divergence point. A reader might reasonably ask: does SAPO diverge at step 3000? 5000? Never? The claim about "higher Pass@1 accuracy" at the point of measurement (step 1750) is supported.
What the experiments do not demonstrate: The claim is framed as a general property of SAPO, but it is tested only on one model, one task domain (mathematical reasoning, plus a shorter multi-task VL run), and one training duration. Whether SAPO's stability advantage persists at much longer training horizons (tens of thousands of steps), at different model scales, or in different RL paradigms (e.g., online RL with frequent policy updates) is unknown. The paper's own caveat—"all methods may ultimately exhibit signs of instability"—is acknowledged but unexplored; the experiments do not characterize SAPO's own failure modes or stability limits.
Claim from executive summary: "SAPO achieves consistent gains on diverse text and multimodal tasks and across different model sizes and architectures."
What the experiments demonstrate: Section 5.2 reports that SAPO "consistently improves performance across models of varying sizes and across both MoE and dense architectures" when applied to Qwen3-VL training. Figure 6 shows the aggregated validation score for Qwen3-VL-30B-A3B, with SAPO outperforming GSPO and GRPO-R2.
What the experiments do not demonstrate: The paper does not actually show results for "different model sizes and architectures" in the experimental section. Section 5.2 states that "our experiments show that SAPO consistently improves performance across models of varying sizes and across both MoE and dense architectures," but Figure 6 reports only the Qwen3-VL-30B-A3B results. Results for other model sizes and dense architectures are not presented in any figure or table, nor are they described with specific numbers. The paper does not provide per-task breakdowns for the VL experiments, so "consistent gains on diverse text and multimodal tasks" is asserted rather than demonstrated—we cannot see whether gains are concentrated on mathematical reasoning tasks (consistent with the controlled experiments) or distributed across coding, logical reasoning, and multimodal understanding tasks. The aggregated metric obscures task-level heterogeneity.
Methodological concerns:
-
No error bars or statistical testing. All figures present single curves with no indication of variance across random seeds, data orderings, or initialization. RL training is known to be sensitive to random seeds; without multiple runs, it is impossible to assess whether the observed performance gap (e.g., 2–3 Pass@1 points on AIME25) is statistically reliable or within the range of run-to-run variation. The paper's strongest results (SAPO vs. GSPO on HMMT25: ~0.09 Pass@1 gap) are large enough to likely be significant, but the smaller gaps (SAPO vs. GSPO on AIME25: ~0.02 gap) might not survive replication with different seeds.
-
No tabulated results. All quantitative comparisons require reading values off figures, introducing approximation error. Precise final Pass@1 values, peak values, and degradation amounts are not provided in tabular form. This is a notable omission for a paper making quantitative performance claims.
-
GSPO and GRPO-R2 collapse may reflect hyperparameter sensitivity. The baseline methods use hyperparameters from Zheng et al. (2025), which were tuned for that paper's experimental setup. It is unknown whether adjusting the clip range
ε, learning rate, group size, or number of mini-batches would prevent or delay the observed collapse. The paper does not report any effort to optimize baseline hyperparameters for the specific cold-start Qwen3-30B-A3B setting used here. A reader cannot distinguish between "hard clipping is inherently unstable" and "the specific hard-clip configuration used here is suboptimal." -
Controlled experiments use a single seed/model initialization. The paper does not mention multiple training runs, random seeds, or statistical replicates. All curves in Figures 4–6 appear to be single runs. This is a significant limitation for an empirical paper whose primary claims are about training stability—a phenomenon known to exhibit high run-to-run variance in deep RL.
-
The "cold-start" checkpoint is not characterized. The paper uses "a cold-start checkpoint fine-tuned from Qwen3-30B-A3B-Base" (Section 5.1) without specifying what this fine-tuning entailed, how many steps it involved, what data it used, or what performance it achieved before RL training began. This makes the experimental setup difficult to reproduce and raises questions about whether the "cold-start" state might influence which methods are stable (e.g., if the cold-start model already has certain pathologies that interact with hard clipping).
Experiments that would have strengthened the paper:
- Multi-seed runs with error bars for at least the main comparison (Figure 4) to establish statistical reliability.
- Hyperparameter sweeps for baselines to ensure the comparison is against well-tuned GSPO and GRPO-R2, not just the configurations from prior work.
- Per-task breakdowns for VL experiments to assess whether SAPO's gains are uniform or concentrated.
- Results on dense architectures in the RL experiments, not just in the assumption-validation histograms, to test whether SAPO's advantages are MoE-specific.
- SAPO with varying
τ_pos(e.g., 0.5, 1.0, 2.0) to characterize hyperparameter sensitivity. - Longer training runs to determine whether SAPO's stability is indefinite or merely extended.
- Ablation of the soft gate shape—testing alternative smooth decay functions (Gaussian, exponential) against the sigmoidal gate to determine whether the specific functional form matters or any smooth decay works.
- Comparison against a GRPO variant with asymmetric clipping (
ε_pos ≠ ε_neg) to disentangle the smoothness contribution from the asymmetry contribution.
6. Limitations and Trade-offs
6.1 The Paper Tests SAPO on a Single Base Model Family with No Evidence of Cross-Architecture Generalization
The assumption or constraint. All RL training experiments—both the controlled mathematical reasoning runs (Section 5.1) and the large-scale VL experiments (Section 5.2)—use models from the Qwen3 family (Qwen3-30B-A3B-Base and Qwen3-VL variants). The paper does not test SAPO on models from other families (e.g., LLaMA, DeepSeek, Mistral) or on non-Qwen MoE architectures. The only cross-architecture comparison is the histogram analysis in Figures 2–3, which validates that the theoretical assumptions (A1 and A2) hold for both a Qwen3 MoE model and a Qwen3 dense model, but this is a descriptive statistic check rather than a training comparison. The paper states in Section 1 that it "believe[s]" the approach generalizes but provides no empirical evidence beyond Qwen3.
The consequence. A practitioner using non-Qwen models—particularly those with different tokenizers, different MoE routing algorithms, or different pretraining distributions—cannot determine from this paper whether SAPO's stability advantage will transfer. The importance ratio distribution r_{i,t}(θ) (which determines how much the soft gate attenuates gradients) depends on the model's specific probability calibration, which varies across model families. A model family with systematically different calibration properties (e.g., overconfident or underconfident token probabilities) would produce a different r_{i,t} distribution, potentially changing the effective behavior of SAPO's temperature-controlled gate. Specifically, if a different model's r_{i,t} distribution were substantially wider than Qwen3's (Figures 2–3 show r_{i,t} concentrated between roughly 0.6 and 1.4 for the MoE model), the default τ_pos = 1.0 might either over-attenuate (if the distribution is naturally wider) or under-attenuate (if narrower), requiring re-tuning that the paper provides no guidance for.
What evidence exists in the paper. Figures 2–3 provide histograms of r_{i,t} and per-sequence variance Var_i(θ) for Qwen3-30B-A3B (MoE) and Qwen3-4B (dense). These show that the assumptions underlying SAPO's sequence-level reduction hold for these specific models. However, the training results (Figures 4–6) are exclusively on Qwen3 architectures. The VL experiments (Section 5.2) mention "different model sizes and architectures within the Qwen3-VL family," but this is still within a single model family. No results are reported for, e.g., a LLaMA-based MoE model or a DeepSeek dense model, despite these being widely used alternatives. The paper does not discuss whether Qwen3's specific architecture—its attention mechanism, its expert routing algorithm, its pretraining data mixture—might interact with SAPO's soft gate in ways that would not replicate on other architectures.
Mitigation status. The paper does not acknowledge this as a limitation. It asserts that SAPO "provides a more reliable, scalable, and effective optimization strategy for RL training of LLMs" (Section 1) in a general sense, without qualifying the scope of this claim to Qwen3 architectures. The VL experiments' multi-task, multi-size framing partially addresses generalization concerns (showing SAPO works on multiple Qwen3-VL sizes), but does not cross the model-family boundary. A practitioner deploying SAPO on a non-Qwen model would need to run their own stability comparisons from scratch, with no guidance from this paper on whether the τ hyperparameters transfer or need re-tuning.
6.2 The Difficulty Estimation Cost in the Prior Paper Has No Counterpart Here, But SAPO Introduces Its Own Unaccounted Computational Overhead
The assumption or constraint. SAPO requires computing per-token importance ratios r_{i,t}(θ) = π_θ(y_{i,t} | q, y_{i,<t}) / π_{θ_old}(y_{i,t} | q, y_{i,<t}) for every token in every sampled response during gradient computation. This involves a forward pass through the current policy π_θ to obtain current token probabilities, plus access to the behavior policy probabilities π_θ_old from the rollout generation step. The paper does not report the computational cost of this additional forward pass relative to the total training FLOPs, nor does it compare SAPO's per-step wall-clock time or memory footprint against GSPO or GRPO-R2.
The consequence. While all group-based RL methods require importance ratio computation (GRPO and GSPO also need r_{i,t}(θ) or s_i(θ)), SAPO's token-level soft gate requires computing r_{i,t} for every token individually—there is no way to reduce this to a sequence-level summary as GSPO does. This means SAPO's per-step cost is at least as high as GRPO's (both are token-level) and potentially higher than GSPO's (which could, in principle, compute only s_i(θ) without per-token ratios if the sequence-level probabilities are available directly). Furthermore, because SAPO's default configuration uses "four mini-batches for gradient updates" (Section 5.1) per batch of rollout data, the current policy is updated four times and thus r_{i,t}(θ) must be recomputed for each mini-batch (since π_θ changes), multiplying the forward-pass cost. The paper reports no wall-clock timing or FLOPs comparison between SAPO, GSPO, and GRPO-R2, making the "under comparable compute budgets" claim (Section 5.2) ambiguous—does it mean equal gradient steps, equal wall-clock time, or equal FLOPs? If SAPO's per-step computation is higher, then equal gradient steps means SAPO received more total compute, biasing the comparison in its favor.
What evidence exists in the paper. The paper states that comparisons are "under comparable training budgets" (Section 1) and "under the same compute budget" (Figure 6 caption), but defines this only implicitly as equal gradient steps and identical rollout generation costs. Section 5.1 notes that "each batch of rollout data is divided into four mini-batches for gradient updates" and that SAPO, GSPO, and GRPO-R2 all use this same mini-batching scheme, which equalizes the number of gradient steps. However, no analysis of per-step FLOPs, forward-pass overhead, memory usage, or wall-clock time is provided. The paper does not report whether the soft gate computation (σ(τ_i (r_{i,t} - 1)) · 4/τ_i) adds meaningful overhead relative to GRPO's min(r, 1+ε) or GSPO's sequence-level clip, though both are negligible compared to the forward passes through the model. The real question—whether SAPO requires additional forward passes that GSPO avoids—remains unaddressed.
Mitigation status. The paper does not acknowledge this as a limitation or discuss computational overhead. The "comparable compute budgets" language is vague and could refer to any of several metrics (gradient steps, total FLOPs, wall-clock time). A practitioner trying to reproduce or budget for SAPO training would need to measure the per-step cost independently. The absence of any wall-clock or FLOPs comparison is a significant gap given that the paper's primary practical claim is about training efficiency and stability at scale.
6.3 The Paper Does Not Demonstrate That the Baselines Are Well-Tuned, Confounding the Algorithm Comparison with Hyperparameter Sensitivity
The assumption or constraint. The GSPO and GRPO-R2 baselines use "the same hyperparameter configurations as in Zheng et al. (2025)" (Section 5.1). These hyperparameters—including the clip range ε, learning rate, group size G, and mini-batch count—were originally tuned for the experimental setup in Zheng et al. (2025), which used a different base model, possibly a different training data distribution, and a different training stage. The paper does not report any attempt to re-tune these baselines for the specific Qwen3-30B-A3B cold-start setting used in the controlled experiments. SAPO's own hyperparameters (τ_pos = 1.0, τ_neg = 1.05) were presumably chosen by the authors for this setting, but no sweep or sensitivity analysis is reported for any method.
The consequence. The observed "early-stage training collapse" of GSPO and GRPO-R2 (Section 5.1, Figure 4) might reflect suboptimal hyperparameter choices rather than an inherent limitation of hard clipping. For instance, a smaller clip range ε for GRPO-R2 or a larger group size G for GSPO might stabilize training by reducing the variance of the advantage estimates or by allowing fewer off-policy tokens to pass through. The paper cannot distinguish between "hard clipping with the specific hyperparameters from Zheng et al. (2025) is unstable on Qwen3-30B-A3B" and "hard clipping is inherently brittle compared to SAPO's smooth gate." This is a fundamental confound: the comparison is between a method tuned (even if minimally) for the current setting (SAPO) and methods whose hyperparameters were inherited from prior work on a different setting (GSPO, GRPO-R2). If a practitioner were to invest effort in tuning ε for GSPO on their specific model and data, they might achieve stability comparable to or better than SAPO—but this experiment is not performed.
What evidence exists in the paper. Figure 4 shows GSPO and GRPO-R2 both exhibiting training reward decline after ~500 steps, while SAPO remains stable. The paper attributes this to SAPO's smooth gate, stating that "hard clipping makes it difficult to strike a favorable trade-off" (Section 1). However, the only hyperparameter ablation in the paper is SAPO's temperature symmetry study (Figure 5), which tests three τ_neg values at fixed τ_pos = 1.0. No corresponding ablation of ε for GSPO or GRPO-R2 is reported. The paper does not describe how the baselines' hyperparameters were selected; the phrase "same hyperparameter configurations as in Zheng et al. (2025)" is the sole specification. A reader cannot assess whether Zheng et al.'s ε value (likely 0.2, following standard PPO practice, though this is not stated in the SAPO paper) is appropriate for Qwen3-30B-A3B cold-start training or whether a different ε would prevent the observed collapse.
Mitigation status. The paper does not acknowledge this as a limitation. The narrative presents SAPO's stability advantage as evidence of smooth gating's superiority over hard clipping without controlling for the possibility that the hard-clipping methods were suboptimally configured. This is a significant methodological concern because the central empirical claim—that SAPO is more stable than GSPO and GRPO-R2—rests on a comparison where baseline hyperparameters are fixed to values from prior work. A fair comparison would either (a) tune each baseline's key hyperparameter (ε for GSPO/GRPO-R2) via a sweep and report the best result for each, or (b) acknowledge that the comparison is between SAPO's chosen configuration and GSPO/GRPO-R2's inherited configuration, making it a weaker claim than "SAPO outperforms well-tuned baselines."
6.4 The Paper Provides No Evidence That SAPO's Stability Advantage Extends Beyond the 1750-Step Horizon Tested
The assumption or constraint. The controlled experiments in Figure 4 run for approximately 1750 gradient steps, and the VL experiments in Figure 6 run for approximately 500 steps. At the end of these runs, SAPO is still improving (its Pass@1 curves have not plateaued) and has not collapsed. However, the paper itself acknowledges in Section 1 that "all methods may ultimately exhibit signs of instability"—an acknowledgment that SAPO is not immune to the eventual degradation that affects GSPO and GRPO-R2, only that it "sustains coherent learning for a longer duration." The experiments do not test this claim by running SAPO to its own point of divergence.
The consequence. A practitioner planning a long training run (tens of thousands of steps, as is common in large-scale RL fine-tuning of LLMs) cannot determine from this paper whether SAPO will remain stable throughout their training or whether it merely delays collapse by some factor (e.g., 2× or 3× the stable duration of GSPO). The paper claims SAPO "extends the duration of stable training" (Section 6), but provides no characterization of how much longer this duration is—does it extend it indefinitely, or by 50% more steps, or by 500%? Without running SAPO to collapse, the "longer duration" claim is directional but unquantified. Furthermore, if SAPO does eventually collapse, the practitioner does not know whether the collapse is gradual (allowing early stopping before performance degrades) or catastrophic (sudden divergence that ruins the checkpoint), since the paper's experiments stop before any SAPO instability appears.
What evidence exists in the paper. Figure 4 shows SAPO's Pass@1 curves with positive slopes through step 1750 on all three benchmarks (HMMT25, BeyondAIME, AIME25), while GSPO and GRPO-R2 have plateaued or declined by step 500–750. Figure 6 shows SAPO still improving through step 500 in the VL setting. The paper's claim about "longer duration" is supported in a relative sense—SAPO outlasts the baselines within the observed window—but the absolute duration of stability is unmeasured. The training reward curves (Figure 4, top-left; Figure 5, top-left) show SAPO's reward rising steadily with no downward inflection, suggesting stability at least through the tested horizon. However, GSPO and GRPO-R2 also showed steady reward increases before their collapse points (~step 500), so the absence of collapse by step 1750 does not guarantee absence at step 3000 or 5000.
Mitigation status. The paper partially acknowledges this through the "all methods may ultimately exhibit signs of instability" caveat in Section 1, which is a rare example of the authors qualifying a broad claim. However, the experiments do not probe this boundary. A longer training run—even doubling the steps to 3500—would substantially strengthen the "extended duration" claim by either demonstrating continued stability or characterizing SAPO's own failure mode. The paper suggests no mechanism by which SAPO would eventually collapse (e.g., would the soft gate eventually saturate as r_{i,t} distributions widen with continued training? Would the asymmetric temperature become insufficient as the policy drifts further from the behavior policy?), leaving the stability ceiling unspecified. Practitioners must treat SAPO's stability as proven only within the tested horizon and remain vigilant for collapse at longer durations, with no guidance on warning signs or mitigation strategies.
6.5 The Paper Does Not Report Statistical Variation, Making the Reliability of Observed Performance Gaps Unquantifiable
The assumption or constraint. All experimental results in Figures 4–6 are presented as single curves with no error bars, confidence intervals, or multi-seed statistics. The paper does not report how many independent training runs were conducted, whether results are averaged over multiple seeds, or the variance across runs. RL training is known to exhibit substantial run-to-run variation due to stochasticity in rollout sampling, minibatch ordering, and optimizer dynamics. The paper's headline comparisons—SAPO outperforming GSPO by ~0.09 Pass@1 on HMMT25 and ~0.02 on AIME25 at step 1750—are presented as point estimates from what appear to be single runs.
The consequence. The reliability of the paper's quantitative performance claims cannot be assessed. If the observed gaps (particularly the smaller ones, like ~0.02 on AIME25 and ~0.01–0.015 on aggregated VL score) are within the range of run-to-run variation, then the claim that SAPO "achieves higher final Pass@1 performance" (Section 5.1) may not hold under replication. The stability advantage (SAPO not collapsing while baselines do) is more visually robust—the qualitative pattern of GSPO and GRPO-R2 degrading after step 500 while SAPO continues improving is unlikely to be purely stochastic—but the magnitude of the performance gap at the final step could vary substantially across seeds. Without error bars, a practitioner cannot determine whether the expected improvement from switching to SAPO is 0.02 Pass@1 (potentially within noise) or 0.09 Pass@1 (clearly meaningful), nor can they assess the worst-case performance (SAPO might underperform baselines in some seeds even if its mean is higher).
What evidence exists in the paper. Figures 4–6 contain single-line plots with no indication of variance. The paper reports no seed counts, no standard deviations, and no statistical tests. The validation metrics (Pass@1 averaged over 16 or 32 samples per query) include Monte Carlo estimation error from the finite sample size, but this error is not propagated or reported. The paper's sample sizes for validation—500 test questions for the mathematical reasoning benchmarks (typical for AIME/HMMT)—are large enough that sampling error in Pass@1 estimation is likely small relative to the observed gaps, but this does not address run-to-run training variation, which is typically the dominant source of variance in RL experiments.
Mitigation status. The paper does not acknowledge the absence of statistical variation metrics as a limitation. This is a significant methodological gap, especially for a paper whose primary claims are about training stability—a property that is inherently stochastic and whose measurement is incomplete without variance characterization. At minimum, reporting the standard deviation of final Pass@1 across 3–5 independent training runs for each method would allow readers to assess whether the observed gaps are statistically reliable. The field's norms for RL training experiments increasingly expect multi-seed results, and the absence here weakens the strength of the empirical claims.
6.6 The Asymmetric Temperature Design Is Validated Only at a Single τ_pos Value with a Narrow τ_neg Sweep, Leaving the Hyperparameter Sensitivity Largely Uncharted
The assumption or constraint. The paper's temperature ablation (Figure 5) fixes τ_pos = 1.0 and tests only three values of τ_neg: 0.95, 1.0, and 1.05. This sweep demonstrates that τ_neg > τ_pos is better than τ_neg = τ_pos, which is better than τ_neg < τ_pos—a qualitative ordering that supports the theoretical argument about negative-token gradient instability. However, the sweep does not establish that τ_neg = 1.05 is optimal or near-optimal, nor does it explore the sensitivity of SAPO's performance to the absolute scale of τ. The paper provides no ablation varying τ_pos itself (e.g., 0.5, 1.0, 2.0, 5.0) to determine whether performance is robust to this choice or whether τ_pos = 1.0 was cherry-picked post-hoc.
The consequence. A practitioner seeking to deploy SAPO on a new model, dataset, or training stage has no guidance on how to select τ_pos and τ_neg. If SAPO's stability and performance are sensitive to these hyperparameters—and the paper's own ablation (Figure 5) shows that a 5% change in τ_neg (from 1.05 to 0.95) causes "significant instability"—then transferring SAPO to a new setting may require substantial hyperparameter tuning. The paper's theoretical framework (the continuous trust region with temperature-controlled decay rate) provides qualitative guidance (larger τ = faster decay, more conservative; smaller τ = slower decay, more permissive), but no quantitative principle for choosing τ_pos given a model's observed r_{i,t} distribution. For instance, the MoE model's r_{i,t} distribution in Figure 2 shows tokens ranging from ~0.6 to ~1.4; the paper does not explain why τ_pos = 1.0 is the right scale for this distribution width, nor whether a different r_{i,t} distribution (e.g., from a dense model or a later training stage where policies drift more) would require a different τ_pos. The absence of any τ_pos sweep means the paper's main results are demonstrated at a single, potentially cherry-picked temperature value.
What evidence exists in the paper. Figure 5 provides the three-way τ_neg comparison at τ_pos = 1.0. No τ_pos sweep is reported. The paper's derivation of the gradient weight function (Equation 8) shows that w_{i,t}(θ) peaks at 1 at r = 1 regardless of τ, and that τ controls the rate of decay—this provides a theoretical interpretation of τ but not a prescription for its value. The assumption-validation histograms (Figures 2–3) show r_{i,t} distributions, which could in principle inform the choice of τ (e.g., matching the soft gate's width to the distribution's spread), but the paper does not make this connection or provide a heuristic.
Mitigation status. The paper does not acknowledge the narrowness of the temperature sweep as a limitation or discuss hyperparameter sensitivity. Section 3 provides the theoretical motivation for asymmetric temperatures, and Section 5.1 presents the three-value comparison as confirmation of this theory. However, the absence of any τ_pos sweep means the paper's primary result—SAPO's superior stability and performance—is demonstrated at a single temperature configuration without evidence that it is robust or near-optimal. A practitioner who cannot replicate the paper's results at τ_pos = 1.0, τ_neg = 1.05 on a different model has no guidance on how to adjust these values, and the sensitivity demonstrated in Figure 5 (τ_neg = 0.95 causes collapse) suggests that small changes can have large effects, making this a practical deployment risk. The paper would be strengthened by either (a) a τ_pos sweep demonstrating robustness across a range (e.g., 0.5–2.0) or (b) a heuristic for selecting τ_pos based on observable properties of the model's r_{i,t} distribution.
7. Implications and Future Directions
How This Work Changes the Landscape
SAPO represents an algorithmic refinement with outsized practical consequences rather than a paradigm shift. It does not overturn the group-based RL framework—it preserves the core recipe of group sampling, advantage normalization, and importance-weighted policy gradients—but it fundamentally rethinks the constraint mechanism at that framework's center. The magnitude of the contribution lies in demonstrating that a conceptually simple change (replacing a discontinuous clip with a smooth sigmoidal gate) can eliminate the training collapse that has been a persistent failure mode in this line of work, without introducing new hyperparameters that are harder to tune or requiring auxiliary stabilization techniques like routing replay.
Reframing of policy constraints. The paper's most significant conceptual move is to argue, through both theory and empirics, that hard clipping's binary accept/reject logic is fundamentally mismatched to the geometry of autoregressive sequence generation in large-vocabulary models. The continuous trust region framing—where every token contributes some gradient, with weight decaying smoothly as a function of off-policyness—is not itself novel (it connects to soft clipping ideas in Chen et al., 2023, and to the broader literature on robust off-policy evaluation). But SAPO is the first to apply this framing to group-based LLM RL at scale, and more importantly, to demonstrate that the smoothness alone—without any of the previously-necessary stabilization tricks—suffices for stable MoE training through 1750 gradient steps. This shifts the field's default assumption about what a "policy constraint" should look like for LLMs: not a hard boundary enforced by a clip, but a continuous regularizer expressed through a decay function.
What becomes less attractive. The paper's results weaken the case for investing in increasingly sophisticated hard-clipping variants. GSPO's innovation—moving clipping from the token level to the sequence level—was a genuine insight that improved coherence with sequence-level rewards, but SAPO achieves comparable sequence-level coherence as an emergent property while avoiding GSPO's fragility to outlier tokens. Routing replay, the MoE stabilization technique used in GRPO-R2, is shown to be unnecessary when the constraint mechanism is sufficiently robust—SAPO without routing replay outperforms GRPO-R2 with it. This suggests that future engineering effort should shift from patching hard clipping's failure modes (through architectural tricks like routing replay or through careful ε tuning) toward designing better constraint functions. The paper does not render hard clipping obsolete—GSPO and GRPO remain widely deployed and understood—but it establishes that superior alternatives exist and provides a concrete template for what they look like.
Resolution of an implicit tension. Prior work on group-based RL for LLMs faced an unacknowledged tension: the community knew that importance ratios could be high-variance (especially in MoE models), that off-policy updates could destabilize training, and that some form of constraint was necessary—but the inherited solution (PPO-style clipping) was adopted without systematic interrogation of whether it was the right constraint for this specific problem geometry. The result was a series of papers that made incremental improvements (token-level vs. sequence-level, with vs. without routing replay) while leaving the underlying clip function untouched. SAPO resolves this tension by identifying the clip function itself as the bottleneck and proposing an alternative that is better-matched to the structure of the problem. The paper's juxtaposition of GRPO's step-function gradient weight (Equation 24) against SAPO's bell-shaped weight (Figure 1, right panel) makes this diagnosis visually immediate: the optimization landscape of hard clipping has a cliff edge, and that cliff edge is what causes the brittleness.
Where the field should focus next. SAPO's results redirect attention toward two interconnected research questions. First, what is the optimal shape of the decay function? SAPO uses a specific sigmoid-derived kernel sech², but the paper does not claim this is uniquely optimal—it is the first instance of a broader class of smooth constraint functions. Understanding how different decay shapes (exponential, Gaussian, Student's t) interact with different model architectures, training stages, and task distributions is a natural next step. Second, can we automate the choice of constraint hyperparameters? SAPO introduces τ_pos and τ_neg as tunable knobs; the paper shows that a 5% asymmetry in τ is critical for stability (Figure 5), but provides no principle for setting these values beyond the specific Qwen3-30B configuration tested. A method that sets τ adaptively based on online estimates of the importance ratio distribution—widening the trust region when the policy is changing slowly and narrowing it when variance increases—would eliminate the need for manual temperature tuning and make SAPO truly "adaptive" in the sense its name implies.
Follow-Up Research This Work Enables
Characterizing SAPO's stability ceiling: does the soft gate prevent collapse indefinitely, or merely delay it? The paper acknowledges that "all methods may ultimately exhibit signs of instability" (Section 1) but stops training at 1750 steps, well before SAPO shows any degradation. A direct follow-up would run SAPO, GSPO, and GRPO-R2 on the same Qwen3-30B-A3B cold-start setup for 5000–10000 steps—or until collapse, whichever comes first—with multiple seeds per method, and measure the stable training duration (defined as the number of steps before training reward drops below 90% of its peak and stays there). This would quantify the "longer duration" claim: does SAPO extend stability by 2x, 5x, or indefinitely? It would also characterize SAPO's own failure mode when it does collapse—does performance degrade gradually (allowing early stopping) or catastrophically (ruining the checkpoint)? The soft gate's boundedness (w ∈ [0,1]) suggests collapse should be gradual if it occurs, since the gate cannot produce arbitrarily large updates even for extremely off-policy tokens, but this hypothesis needs testing. If SAPO never collapses within practical training horizons while baselines consistently do, the paper's contribution shifts from "more stable" to "effectively solves the stability problem," which is a substantially stronger claim.
Does the asymmetric temperature design generalize across model families, or is it Qwen3-specific? The paper's theoretical argument for τ_neg > τ_pos—that negative-advantage gradients diffuse to the full vocabulary and are therefore noisier—is architecture-agnostic and should apply to any autoregressive LLM with a softmax output layer. But the magnitude of the effect may differ across model families due to differences in tokenizer vocabulary size, probability calibration, pretraining data distribution, and MoE routing algorithms. A cross-model-family study would train SAPO on, e.g., a LLaMA-3-based MoE model, a DeepSeek-based MoE model, and a dense model (all at comparable parameter counts), sweeping τ_neg from 0.9 to 1.2 at fixed τ_pos = 1.0 for each. The key measurements would be: (a) whether τ_neg > τ_pos is universally optimal or whether some model families prefer symmetric or even reversed asymmetry; (b) whether the optimal τ_neg / τ_pos ratio correlates with observable properties like the model's average token entropy, the width of the r_{i,t} distribution, or vocabulary size; and (c) whether dense models, which show tighter r_{i,t} concentration in Figures 2–3, benefit from asymmetry at all, or whether symmetry suffices when variance is low. A negative result—finding that asymmetry matters only for Qwen3 MoE models—would scope the paper's contribution more narrowly and suggest that the gradient diffusion argument (Equation 9) is incomplete without accounting for model-specific calibration.
Can SAPO's soft gate be combined with GRPO's clipped objective as a hybrid constraint? The paper presents SAPO and GRPO as alternatives—hard clip vs. soft gate—but these are not mutually exclusive. A natural hybrid would apply SAPO's soft gate w_{i,t}(θ) as a continuous weight, but additionally apply a hard clip at an extreme threshold (e.g., r_{i,t} > 3.0) to completely zero out catastrophically off-policy tokens. The question is whether the soft gate alone attenuates extreme tokens sufficiently, or whether there remain outlier tokens (e.g., r_{i,t} > 10) where even the exponentially small soft-gate weight provides a destabilizing gradient. This could be tested by comparing SAPO against SAPO-with-hard-clip on the same Qwen3 setup, measuring both final performance and training stability. If the hard clip provides no additional benefit, it confirms that the soft gate's attenuation is sufficient even in the tails; if it improves stability, it suggests that extremely off-policy tokens—rare but impactful—require zeroing out, and that a hybrid approach combining SAPO's graduated weighting with a safety floor is optimal. This experiment would also test the paper's implicit claim that the soft gate "avoids gradient vanishing" (Section 4.2): is there a regime where gradient vanishing (via hard clip) is actually beneficial because the alternative (tiny but non-zero gradients from extreme outliers) accumulates noise over many steps?
Adaptive temperature scheduling: can τ be set automatically based on online estimates of policy drift? The paper's temperature parameters are fixed throughout training (τ_pos = 1.0, τ_neg = 1.05), but the appropriate trust region width likely changes as training progresses: early in training, when the policy is changing rapidly, a narrower trust region (larger τ) may be needed; later, when the policy converges, a wider trust region (smaller τ) could improve sample efficiency by trusting more off-policy tokens. An adaptive scheme would monitor the empirical distribution of r_{i,t} (or the per-sequence variance Var_i(θ)) within each batch and adjust τ online: for example, set τ = τ_base / (1 + c · σ_batch) where σ_batch is the standard deviation of log-ratios in the current batch and c is a scaling constant. This would eliminate τ as a hyperparameter (replacing it with the easier-to-set τ_base and c) and make SAPO more robust to changing conditions during training. The experiment would compare fixed-τ SAPO against adaptive-τ SAPO on the Qwen3-30B-A3B setup, measuring whether adaptive τ (a) matches or exceeds fixed-τ performance without manual tuning, and (b) prevents instability in the "reversed asymmetry" condition (τ_neg < τ_pos) that caused collapse in Figure 5—if adaptive τ can automatically widen the trust region for negative tokens when they are near-on-policy and narrow it when they drift, it might rescue configurations that are unstable under fixed τ. A negative result—adaptive τ failing to beat well-tuned fixed τ—would suggest that the optimal trust region width is surprisingly stable across training and that the manual tuning burden is low.
Does SAPO enable longer or more ambitious RL training runs that were previously infeasible due to stability concerns? The paper's experiments demonstrate stability up to 1750 steps, but many production RL training runs for reasoning models extend to tens of thousands of steps. A large-scale stress test would take a frontier reasoning model (e.g., a Qwen3-235B or comparable) and run SAPO against GSPO (the most competitive baseline in the paper's own results) for 20,000+ steps on a challenging reasoning benchmark, measuring not just final Pass@1 but also the area under the Pass@1 curve—which captures whether SAPO's stability translates to higher average performance throughout training, not just at the final checkpoint. The hypothesis is that SAPO's stability advantage compounds over long training horizons: GSPO may need to be stopped early and rolled back when collapse is detected, wasting compute and leaving performance on the table, while SAPO can be trained uninterrupted to convergence. This experiment would also reveal whether SAPO's continuous improvement in Figures 4 and 6 eventually plateaus and at what level—is SAPO still gaining 0.5 Pass@1 per 100 steps at step 1750, or is the curve flattening? Quantifying the asymptotic performance gap would inform practitioners about the total expected benefit of switching to SAPO for long training runs, not just the benefit within the 1750-step window the paper tests.
Can the soft gate's functional form be learned rather than hand-designed? SAPO's gate is the sigmoid-derived sech² kernel, chosen for its analytical convenience (it produces the clean reduction to GSPO in Section 4.1) and its empirical effectiveness. But the space of possible smooth decay functions is large, and the optimal shape likely depends on the specific model, task, and training stage. A meta-learning approach would parameterize the gating function f(r) as a small neural network (e.g., a 2-layer MLP) that is trained jointly with the policy to maximize a meta-objective—perhaps the policy's validation performance after K gradient steps. This would learn the optimal decay shape for the specific training dynamics of the model at hand, potentially discovering asymmetric, non-monotonic, or context-dependent gating patterns that a fixed sigmoid cannot express. The experiment would compare learned-gate SAPO against fixed-gate SAPO on the Qwen3 setup, with the caveat that the learned gate introduces additional parameters and compute overhead. A strong positive result—the learned gate substantially outperforming the fixed sigmoid—would suggest that the hand-designed sech² kernel is a reasonable default but far from optimal, opening a new subfield of meta-learned policy constraints. A negative result—the learned gate recovering something close to the sigmoid—would strengthen the paper's claim that the specific functional form is well-motivated and near-optimal.
Practical Applications and Downstream Use Cases
Stabilizing RL fine-tuning for on-device or edge-deployed MoE models. MoE architectures are attractive for on-device deployment because they offer large model capacity with sparse activation (e.g., 30B total parameters, 3B active), fitting within tight memory and compute budgets. However, when these models are fine-tuned with RL for domain-specific reasoning tasks (e.g., a math tutor app that adapts to student questions), the training instability documented in this paper—particularly the amplification of importance ratio variance due to routing heterogeneity (Figures 2–3)—can cause training runs to collapse before reaching acceptable performance. SAPO provides a drop-in replacement for GRPO or GSPO in these fine-tuning pipelines that, based on Figure 4, extends stable training by at least 3–4× (GSPO collapses by step ~500; SAPO is still improving at step 1750). The elimination of routing replay as a requirement further simplifies the pipeline: practitioners do not need to implement or maintain the additional infrastructure for storing and replaying expert assignments. For a team fine-tuning an on-device MoE model on a task-specific dataset of 10,000–100,000 queries, where each training run costs hundreds of GPU-hours and early collapse means restarting from scratch, SAPO's reliability improvement directly translates to reduced compute waste and faster time-to-deployment. The asymmetric temperature design (τ_neg > τ_pos) is critical here: the on-device setting likely involves a narrower task distribution than the paper's multi-task VL training, meaning negative-advantage tokens (responses that failed the task) may be more frequent and their destabilizing effect stronger, making the asymmetric damping particularly valuable.
Production RL training of large-scale reasoning models where training runs cost millions of dollars. The paper's Qwen3-VL experiments (Section 5.2, Figure 6) demonstrate SAPO's effectiveness in the exact setting where stability matters most: multi-task, multi-modal, large-scale RL training on models that will be deployed to millions of users. In this regime, a single training run can cost millions of dollars in compute, and early collapse is catastrophic—not only is the spent compute wasted, but the release schedule is delayed while a new run is initiated. SAPO's consistent performance gains throughout 500 gradient steps (Figure 6, aggregated validation score improving from ~0.720 to ~0.760) and its outperformance of GSPO and GRPO-R2 under equal compute budgets mean that, for the same training cost, SAPO delivers a better final model. Even if the per-step compute overhead of SAPO's token-level gate is slightly higher than GSPO's sequence-level computation (a limitation the paper does not quantify), the elimination of collapse risk is likely worth the marginal cost increase for production teams. The practical recommendation is to adopt SAPO as the default policy optimizer for large-scale RL runs, with the caveat that τ_pos and τ_neg should be validated on a small-scale pilot run before committing to the full training budget—Figure 5 shows that a 5% difference in τ_neg can mean the difference between stable training and collapse, so blindly copying τ_neg = 1.05 without validation on the specific model and data is risky.
Multi-task RL pipelines where task balancing and stability interact. The VL experiments (Section 5.2) train on "a broad collection of text and multimodal tasks, including mathematics, coding, and logical reasoning," with "a fixed sampling ratio for each task within each batch." In such multi-task settings, different tasks may produce responses with different length distributions, different reward densities, and different importance ratio characteristics. A task that typically produces short, high-certainty responses (e.g., factual QA) will have a narrower r_{i,t} distribution than a task that produces long, creative responses (e.g., code generation with diverse solutions). SAPO's token-adaptive gating naturally handles this heterogeneity: on tasks with narrow r_{i,t} distributions, most tokens receive weights near 1, preserving learning signal; on tasks with wide distributions, off-policy tokens are smoothly attenuated. GRPO's hard clip, by contrast, applies the same ε band to all tasks, potentially zeroing out useful gradients from the wide-distribution tasks while providing insufficient constraint for the narrow-distribution tasks. The paper does not provide per-task breakdowns (a notable omission), but a practitioner building a multi-task RL pipeline could use SAPO to avoid the need for per-task clip ranges—a single set of τ values may work across diverse tasks, reducing the hyperparameter tuning burden in an already-complex multi-task setup. The practical benefit is a simplified training configuration with fewer task-specific knobs, combined with the stability guarantees SAPO provides.
When to Prefer This Method
The paper explicitly positions SAPO against GRPO (token-level hard clipping) and GSPO (sequence-level hard clipping), and provides empirical comparisons against both (Figures 4, 6). The conditions under which SAPO is preferable are clear from the paper's claims and evidence, though the paper does not provide a formal decision rule. Based on the experimental results and theoretical analysis, the tradeoffs are:
-
Prefer SAPO over GSPO when training MoE models, or whenever sequences may contain outlier tokens that corrupt sequence-level averages. GSPO zeroes out gradients for all tokens in a sequence when a few outlier tokens push the sequence-level ratio
s_i(θ)outside the clipping band. In MoE models, Figures 2–3 show widerr_{i,t}distributions and higher per-sequence varianceVar_i(θ), making such outliers more common. SAPO's token-adaptive gating preserves gradients from near-on-policy tokens within outlier-containing sequences (Section 4.1). The paper does not test this on dense models, but the tighterr_{i,t}concentration in Figure 3 suggests GSPO may be more competitive there—a practitioner training a dense model should verify whether GSPO's sequence-level clipping discards significant fractions of sequences before switching. -
Prefer SAPO over GRPO when training stability to long horizons is more important than per-step computational simplicity, or when negative-advantage token variance is high. GRPO's hard token-level clip can cause flickering instability as tokens oscillate across the clip boundary (Section 4.2), and it treats positive and negative advantages symmetrically despite their different gradient propagation properties (Equation 9). SAPO's smooth gate eliminates boundary flicker, and its asymmetric temperature specifically dampens the noisier negative-token gradients. Figure 5 shows that even a small asymmetry (
τ_neg = 1.05 > τ_pos = 1.0) prevents the early collapse that occurs under symmetric temperatures. If the training data has a high proportion of negative-advantage responses (e.g., tasks with low success rates where most sampled responses are incorrect), SAPO's asymmetric damping becomes increasingly valuable. -
Prefer SAPO when engineering simplicity matters: it eliminates the need for auxiliary stabilization. GRPO-R2 requires routing replay for stable MoE training. SAPO achieves superior stability without routing replay (Section 5.1). For teams that do not already have routing replay infrastructure, adopting SAPO removes an implementation and maintenance burden. For teams that do have routing replay, switching to SAPO may still be worthwhile if it improves exploration (SAPO does not freeze routing decisions during gradient computation, allowing the policy to learn better routing simultaneously with better token generation)—but the paper does not ablate this exploration benefit, so this remains speculative.
-
Prefer the baseline methods when τ tuning is infeasible, or when the model architecture produces importance ratio distributions far from those in Figures 2–3. The paper's hyperparameters (
τ_pos = 1.0,τ_neg = 1.05) are validated only on Qwen3 models. A practitioner working with a substantially different model family, tokenizer, or training data distribution may find that these τ values are inappropriate—and Figure 5 shows that a 5% change inτ_negcan cause collapse. If resources for hyperparameter tuning are extremely limited, GSPO or GRPO with their single clip parameterεmay be easier to tune than SAPO's two-temperature design, even if the resulting stability is worse. This caveat would be weakened if follow-up work establishes τ-setting heuristics or demonstrates thatτ_pos = 1.0, τ_neg = 1.05transfers across model families without re-tuning.
