ArXiv: 2511.08567

🎯 Pitch

RLVR’s dramatic reasoning gains surprisingly arise from sparse-looking updates, but this sparsity is an illusion created by a model-conditioned bias that steers optimization into low-curvature, off-principal subspaces—the opposite of SFT, which targets principal weights and distorts the spectrum.

1. Executive Summary

This paper studies how Reinforcement Learning with Verifiable Rewards (RLVR) modifies language model parameters, analyzing a suite of publicly released checkpoints spanning multiple model families (Qwen, Llama, Mistral) and RL algorithms (GRPO, DAPO, Reinforcement++). The authors uncover a persistent, model-conditioned optimization bias that concentrates visible updates into narrow, stable regions of the pretrained model — a pattern they mechanistically explain through a Three-Gate Theory: Gate I (KL Anchor) imposes a KL-constrained per-step update, Gate II (Model Geometry) steers updates off principal directions into low-curvature, spectrum-preserving subspaces (e.g., RL avoids the top singular-subspace weights that SFT targets; instead it favors non-principal, low-magnitude parameters), and Gate III (Precision) hides micro-updates in non-preferred regions, making the bias appear as sparsity when stored in bfloat16. In parameter-level validation, RLVR preserves spectral structure with minimal subspace rotation and singular-value drift, whereas SFT distorts the spectrum and targets principal weights — establishing that RL and SFT operate in fundamentally distinct optimization regimes, and that SFT-era parameter-efficient fine-tuning methods aligned with principal directions (e.g., PiSSA, principal-targeted sparse masks) are misaligned with RLVR’s off-principal dynamics, often degrading performance or destabilizing training.

2. Context and Motivation

The Core Problem: We Don't Understand How RL Modifies Language Model Parameters

The paper addresses a fundamental gap in our understanding of post-training: when Reinforcement Learning with Verifiable Rewards (RLVR) improves a model's reasoning capabilities, what actually happens to the model's parameters? This is not merely an academic curiosity. The recent success of Large Reasoning Models like OpenAI-o3 and DeepSeek-R1 has established RLVR as a critical stage in the LLM development pipeline, yet the mechanisms by which RL shapes model representations remain opaque. As the authors note, given the substantial computational resources devoted to RL—"especially relative to supervised fine-tuning (SFT)"—and the emergence of striking new behaviors like chain-of-thought reasoning and self-verification, "one might naturally assume that such progress arises from significant parameter changes."

This assumption turns out to be wrong, and understanding why it is wrong reveals something deep about how RL interacts with pretrained model geometry.

A Paradox That Demands Explanation

The paper is motivated by a specific, counterintuitive observation that crystallizes the gap in our understanding. Mukherjee et al. (2025) reported that RL induces sparse parameter updates, whereas SFT yields dense ones. This is a paradox: a high-cost, high-gain process (RL) that relies on surprisingly minimal weight modification, while a simpler and cheaper process (SFT) changes many more parameters.

The authors frame this as a paradox worth resolving:

"This counterintuitive finding reveals a paradox — a high-cost, high-gain process that relies on surprisingly minimal weight modification."

The paradox invites deeper questions. If RL updates only a fraction of parameters, which parameters are they? Is the sparsity pattern random, or does it reflect something systematic about how RL navigates the optimization landscape? Does the pattern depend on the specific RL algorithm, the training data, or the model architecture? Prior work had documented the existence of sparsity but had not explained its origin or characterized where updates land.

Where Prior Work Falls Short

The paper identifies several specific limitations in the existing literature that motivate its investigation:

1. Prior analyses of RLVR focus on policy-level effects, not parameter-level dynamics. Concurrent work by Wu et al. (2025) and Shenfeld et al. (2025) had shown that RL-trained policies remain KL-proximal to the base policy—an important finding about distributional behavior. But these analyses operate at the level of token probabilities and sequence distributions. They do not explain how such KL-proximal policies are realized in weight space, nor do they characterize which parameters change and why. The authors explicitly position their work as complementary:

"Together, these findings provide the first parameter-space account linking RL optimization dynamics to weight evolution, complementing concurrent work that focuses primarily on policy-level or distributional effects."

2. The sparsity finding by Mukherjee et al. (2025) was suggestive but incomplete. That work observed that RL "fine-tunes small subnetworks in large language models" but left the underlying mechanism unexplored. It did not characterize whether the updated subnetwork was structured (e.g., specific attention heads, specific rows/columns of weight matrices), whether it was consistent across runs, or whether it reflected something deeper about the pretrained model's geometry. The authors argue that "magnitude alone does not reveal where changes occur, impeding deep analysis of how sparse changes arise." Without understanding where updates land, the sparsity observation remains a curiosity rather than an actionable insight.

3. No mechanistic theory existed for RL's parameter-level optimization dynamics. While the pretraining scaling laws literature (Hoffmann et al., 2022) and the SFT literature (Liu et al., 2025c; Meng et al., 2024a) had developed relatively mature accounts of how training modifies model parameters—SFT is known to target principal weights, high-energy directions captured by top singular subspaces—no analogous framework existed for RL. The paper identifies this as a critical gap because RL is qualitatively different from SFT: it uses on-policy sampling with ratio clipping or KL penalties, optimizes a reward signal rather than mimicking a teacher distribution, and operates in a regime where the reference policy is the model's own previous checkpoint rather than a distant target.

4. SFT-era PEFT methods were being applied to RL without understanding their suitability. The paper notes that parameter-efficient fine-tuning methods developed for SFT—sparse fine-tuning masks targeting principal weights (Liu et al., 2025c), low-rank adapters initialized along principal singular vectors (PiSSA; Meng et al., 2024a)—were being recommended for RL use cases, including in a recent report by Schulman & Lab (2025) that suggested principal-targeted LoRA variants "should yield further gains." The field lacked a principled basis for evaluating whether these methods, designed for SFT's optimization geometry, would transfer to RL's. The paper identifies this as a practical motivation: if RL and SFT operate in fundamentally different parameter-space regimes, then importing SFT-era heuristics into RL could be actively harmful.

The Practical and Theoretical Significance

The paper's motivation extends beyond pure academic understanding to several concrete implications:

Resource allocation in post-training pipelines. Modern LLM development pipelines devote enormous computational resources to RL. According to xAI (2025), RL consumes substantial GPU-hours relative to SFT. Understanding how RL modifies parameters could inform more efficient training recipes—for example, if RL naturally concentrates updates in specific parameter regions, perhaps one could freeze other regions without loss of performance, reducing memory and compute costs.

Design of RL-native PEFT methods. If the paper's hypothesis is correct—that RL and SFT operate in distinct optimization regimes—then the design space for parameter-efficient RL is fundamentally different from that for parameter-efficient SFT. Methods that succeed for SFT (e.g., principal-targeted adaptation) might not only fail for RL but actively destabilize training. The paper aims to provide the mechanistic understanding necessary to design "geometry-aware, RLVR-native learning algorithms, rather than repurposed SFT-era heuristics."

Toward a white-box understanding of RL training. The paper frames its contribution as "marking a step toward a white-box understanding of RLVR training." Currently, RL post-training is largely a black box: practitioners tune hyperparameters, observe reward curves, and evaluate downstream performance, but lack visibility into the internal parameter changes driving these outcomes. The paper's parameter-space diagnostics—spectral drift, principal-subspace rotation, update-principal overlap—provide concrete, quantitative tools for opening this black box.

Resolution of conflicting intuitions. The paradox of sparse-yet-effective RL updates challenges a natural intuition that larger behavioral changes require larger parameter changes. Resolving this paradox—showing that RL's effectiveness comes not from the magnitude of parameter changes but from their direction and their alignment with the pretrained model's geometry—provides a conceptual reframing of what "learning" means in the RL post-training context.

How This Paper Positions Itself

The paper positions itself at the intersection of several lines of work but distinct from each:

Against prior sparsity work (Mukherjee et al., 2025). The paper acknowledges the prior observation of RL sparsity but argues it is a "superficial readout" of a deeper mechanism. The authors show that sparsity "largely disappears under float32 storage" and that the underlying phenomenon is not sparsity per se but a model-conditioned optimization bias that determines where updates land, with bfloat16 precision merely amplifying visibility by hiding sub-ULP micro-updates. This reframes the conversation from "RL is sparse" to "RL is biased—here is why and where."

Against concurrent policy-level analyses (Wu et al., 2025; Shenfeld et al., 2025). These works established that RL policies remain KL-proximal to the base policy. The paper's Gate I (KL Anchor) builds on this insight but pushes further: it connects the policy-level KL leash to weight-level constraints via the Fisher information matrix, and then shows (Gates II and III) that the KL leash alone does not determine where within the constraint region updates land—geometry does. The paper thus provides the missing parameter-space link between policy-level KL proximity and the observed weight-space patterns.

Against SFT parameter analysis (Liu et al., 2025c; Meng et al., 2024a). These works established that SFT targets principal weights—the high-energy directions in the pretrained model's weight matrices. The paper adopts the same diagnostic toolkit (principal masks based on rank-k SVD reconstruction) but discovers the opposite pattern for RL: RL updates systematically avoid principal weights. This empirical contrast is central to the paper's claim that RL and SFT operate in disjoint optimization regimes, and it directly motivates the case studies in Section 5 showing that principal-targeted PEFT fails for RL.

Against PEFT-for-RL recommendations (Schulman & Lab, 2025). The paper engages directly with a recent practitioner report suggesting that low-rank LoRA can match full-parameter RL and that principal-targeted variants like PiSSA should help. The paper's theory predicts the opposite: PiSSA enforces updates along principal directions, which are precisely the high-curvature, spectrum-distorting directions that RL's geometry bias steers away from. The paper tests this prediction empirically and finds that PiSSA provides no gain over standard LoRA and often destabilizes training at the higher learning rates needed for low-rank adapters to match full-parameter performance.

The Paper's Central Thesis

The paper's motivating hypothesis can be stated succinctly: RLVR's parameter updates are not randomly sparse but are deterministically routed by the pretrained model's geometry into off-principal, low-curvature, spectrum-preserving directions, and this routing is enforced by the interaction of three mechanisms: a KL anchor that constrains step size, a geometric landscape that determines feasible directions, and finite precision that filters sub-threshold updates in non-preferred regions.

This thesis makes several falsifiable predictions that structure the paper's experimental program: (1) RL should preserve the pretrained spectral structure, while SFT should distort it; (2) RL should avoid principal weights, while SFT should target them; (3) disrupting the pretrained geometry should abolish the update bias; (4) PEFT methods aligned with principal directions should perform poorly for RL. Each of these predictions is tested in Sections 4 and 5.

The Scope and Limitations the Paper Acknowledges Up Front

The paper is transparent about its boundaries. The Three-Gate Theory is a mechanistic account of post-training RLVR applied to pretrained models, not a theory of RL from random initialization. The model-conditioned optimization bias is explicitly tied to the structured geometry of a well-pretrained model:

"This geometry gate explains the model-conditioned nature of the bias: it arises from the pretrained landscape rather than particular datasets or RL recipes."

The paper also acknowledges that directly quantifying curvature in Large Reasoning Models with long chain-of-thought outputs is computationally prohibitive. Instead, it adopts "principal weights" as a proxy for high-curvature directions, following Liu et al. (2025c). This is a practical compromise, not a theoretical claim that principal weights are high-curvature directions; it is a tractable diagnostic that has been empirically validated as a proxy.

Finally, while the paper's analysis spans multiple model families (Qwen, Llama, Mistral), RL algorithms (GRPO, DAPO, Reinforcement++), and task types (math, code, agents, RLHF), it does not claim to exhaust the space of possible post-training configurations. The consistent patterns across this diverse suite suggest a generic phenomenon, but the paper cautiously frames this as evidence for a "persistent, model-conditioned optimization bias" rather than a universal law.

3. Technical Approach

3.1 Reader Orientation

This paper is primarily a mechanistic analysis and diagnostic study, not a method proposal. The authors do not build a new RL algorithm or a new architecture. Instead, they develop a theoretical framework—the Three-Gate Theory—and a suite of parameter-space diagnostic tools to explain why RLVR produces sparse, localized parameter updates and where those updates land. The problem they solve is conceptual: given the paradox that expensive RL training modifies surprisingly few parameters while producing dramatic behavioral improvements, what underlying mechanisms determine which parameters change and which do not? The solution takes the form of a mechanistic account linking policy-level KL constraints to weight-level geometric steering, validated through careful measurement of spectral properties, update localization, and causal interventions on pretrained model geometry.

3.2 Big-Picture Architecture (Diagram in Words)

The system under analysis has five major components, though the paper studies them diagnostically rather than building them:

  1. Pretrained Language Model ($\pi_\theta$) — the base model whose geometry shapes all subsequent RL updates. This is the source of the "model-conditioned" bias: its weight matrices have structured singular value spectra that determine high-curvature vs. low-curvature directions.

  2. RLVR Training Loop — an on-policy policy gradient algorithm (typically DAPO, GRPO, or PPO) that samples completions, computes verifiable rewards, and applies gradient updates with ratio clipping or KL penalties. This loop imposes Gate I (the KL Anchor) through its constrained update mechanics.

  3. Parameter-Space Diagnostics — a set of measurement tools the authors apply to checkpoints before and after RL training:

    • Spectral analysis: singular value decomposition of weight matrices to measure subspace rotation and spectral drift.
    • Principal masks: binary masks identifying the top-$\alpha$ fraction of weights with largest magnitude in the rank-$k$ SVD reconstruction, following Liu et al. (2025c).
    • Update masks: bfloat16-aware binary masks identifying which stored parameters actually changed.
    • Consensus ratios: per-coordinate fraction of independent RL runs that modified a given weight.

  4. Three-Gate Theoretical Framework — the mechanistic explanation of how KL constraint (Gate I), pretrained geometry (Gate II), and bfloat16 precision (Gate III) interact to produce the observed update patterns.

  5. Causal Intervention Tools — function-preserving orthogonal rotations and head permutations that "scramble" pretrained geometry in specific layers, allowing the authors to test whether geometry is genuinely causal for update localization.

Information flows through this analysis pipeline as follows: a pretrained model checkpoint enters → RLVR training produces a fine-tuned checkpoint → parameter-space diagnostics compare base and fine-tuned weights → the Three-Gate Theory predicts and explains the observed patterns → causal interventions verify that geometry is the steering mechanism.

3.3 Roadmap for the Deep Dive

  • First, the Three-Gate Theory (Section 3.4.1–3.4.3): I will explain each gate in isolation—the KL Anchor, the Model Geometry, and Precision—because the theory is the paper's central contribution and everything else flows from it. Understanding each gate's mechanism is prerequisite to understanding the experimental validation.
  • Second, the bfloat16-aware measurement apparatus (Section 3.4.4): Before explaining any results, I need to establish how the paper measures "unchanged" vs. "changed" weights, since this measurement choice directly shapes what counts as an "update" and what gets hidden.
  • Third, the principal-weights diagnostic framework (Section 3.4.5): This is the operationalization of "high-curvature directions" that enables all the update-localization experiments.
  • Fourth, the experimental design for validating the theory (Section 3.4.6): How the paper designs experiments to test each gate's predictions, including the causal intervention protocol and the sparse fine-tuning probe.
  • Fifth, the LoRA/PiSSA comparison methodology (Section 3.4.7): The practical case study testing whether geometry-aligned PEFT design matters for RL.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a theoretical and diagnostic paper whose core idea is that RLVR's optimization dynamics are shaped by the interaction of three "gates"—a KL constraint, pretrained model geometry, and numerical precision—which together route updates into off-principal, spectrum-preserving directions, producing the appearance of sparsity as a side effect.

3.4.1 Gate I: The KL Anchor (How Policy Constraints Bound Weight Updates)

The first gate explains why RL updates cannot move parameters arbitrarily far, even when the reward signal strongly favors a different policy. This gate operates at the policy level but has direct consequences for weight-space dynamics.

The RLVR objective. The paper starts from the standard KL-regularized policy optimization objective used across RLVR algorithms (PPO, GRPO, DAPO, Reinforcement++):

max⁡θEy∼πθ(⋅∣x),x∼X[R(x,y)−β KL(πθ(⋅∣x)∥πref(⋅∣x))]\max_\theta \mathbb{E}_{y \sim \pi_\theta(\cdot \mid x), x \sim \mathcal{X}} \left[ R(x, y) - \beta \, \text{KL}(\pi_\theta(\cdot \mid x) \parallel \pi_{\text{ref}}(\cdot \mid x)) \right]

where $R(x, y)$ is the verifiable reward (normalized and bounded, typically pass/fail or execution scores), $\pi_\theta$ is the current policy, $\pi_{\text{ref}}$ is a fixed reference policy (the pretrained model or a previous checkpoint), and $\beta \ge 0$ controls KL regularization strength. When $\beta = 0$ (as in DAPO), the explicit KL term vanishes but ratio clipping still imposes an implicit KL constraint.

What this objective computes: it maximizes expected reward while penalizing divergence from a reference policy. The KL term acts as a leash—the policy can improve rewards but cannot drift arbitrarily far from where it started. The trade-off parameter $\beta$ controls how tightly the leash is held.

Why this form: without the KL penalty or ratio clipping, policy gradient methods could move the policy to a degenerate distribution that assigns probability 1 to the highest-reward sequence, collapsing exploration and overfitting to the reward signal. The KL regularization keeps the policy in a trust region where the advantage estimates remain valid. For DAPO with $\beta = 0$, ratio clipping to $[1 - \varepsilon, 1 + \varepsilon]$ achieves the same conservative effect in the small-step regime, bounding KL divergence by $O(\varepsilon^2)$ (Lemma F.9 in the Appendix).

The one-step surrogate. In practice, RLVR uses a frozen-policy surrogate where advantages are computed under the old policy $\pi_{\text{old}}$:

LPG(θ)=−Ex∼X,y∼πθ(⋅∣x)[A⊥(x,y)log⁡πθ(y∣x)]\mathcal{L}_{\text{PG}}(\theta) = -\mathbb{E}_{x \sim \mathcal{X}, y \sim \pi_\theta(\cdot \mid x)} \left[ A^\perp(x, y) \log \pi_\theta(y \mid x) \right]

where $A^\perp$ is a normalized advantage estimate optionally shaped by a reference-KL log-ratio term. This is not fully on-policy (updates happen over mini-batches of collected data), but Lemma F.1 shows the error after a small step $\Delta\theta$ is $O(\|\Delta\theta\|^2)$—second-order tight, meaning the surrogate is accurate for small updates.

Proposition 3.1: The one-step policy-KL leash. This is the paper's formal statement that on-policy RL imposes a KL bound between successive policies. Let $\tilde{q}_\beta(\cdot \mid x) \propto q(\cdot \mid x) \exp(R/\beta)$ be the soft-regularized improvement oracle—the policy that would be optimal if we could directly tilt the reference distribution toward higher rewards. Let $\theta^+$ be the parametric fit obtained by M-projection of $\tilde{q}_\beta$ onto the policy class (minimizing $D_{\text{KL}}(\tilde{q}_\beta \parallel \pi_\theta)$). Then for a sufficiently small one-step update:

DKL(πθ+∥πθ)≤(1+o(1)) DKL(q~β∥πθ)D_{\text{KL}}(\pi_{\theta^+} \parallel \pi_\theta) \le (1 + o(1)) \, D_{\text{KL}}(\tilde{q}_\beta \parallel \pi_\theta)

Defining the symbols: $\pi_{\theta^+}$ is the policy after one RL update step, $\pi_\theta$ is the policy before the step, $\tilde{q}_\beta$ is the KL-regularized improvement target (the "oracle" policy we would like to reach), and $D_{\text{KL}}(p \parallel q)$ is the Kullback-Leibler divergence from $q$ to $p$, measuring how many extra bits are needed to encode samples from $p$ using a code optimized for $q$.

What it computes: the KL divergence from the old policy to the updated policy is bounded by (approximately) the KL divergence from the old policy to the improvement oracle. Since the oracle is itself KL-constrained toward the reference, this means the policy cannot jump far in a single RL step—it is anchored near its starting point by a KL leash whose tightness is determined by $\beta$.

Why this form: the inequality captures the conservative nature of policy-gradient methods. The M-projection step (fitting parameters to the oracle) introduces error, but that error is second-order and vanishes as the oracle gets closer to the current policy. The $(1 + o(1))$ factor means the bound becomes exact in the limit of small updates—the leash is tight for small steps.

Proposition 3.2: From policy-KL leash to weight bound. This is the critical bridge from policy space to weight space. Under the same conditions, with $F(\theta) \succeq \mu I$ on the update subspace (the Fisher information matrix is positive-definite with minimum eigenvalue $\mu > 0$):

∥Δ∥F(θ)≜Δ⊤F(θ)Δ≤2K (1+o(1)),∥Δ∥2≤2Kμ (1+o(1))\|\Delta\|_{F(\theta)} \triangleq \sqrt{\Delta^\top F(\theta) \Delta} \le \sqrt{2K} \, (1 + o(1)), \qquad \|\Delta\|_2 \le \frac{\sqrt{2K}}{\sqrt{\mu}} \, (1 + o(1))

Defining the symbols: $\Delta = \theta^+ - \theta$ is the parameter update vector, $F(\theta) = -\mathbb{E}_{\pi_\theta}[\nabla^2 \log \pi_\theta]$ is the Fisher information matrix (the expected negative Hessian of the log-likelihood, measuring local curvature of the KL divergence), $\mu$ is the minimum eigenvalue of $F(\theta)$ on the update subspace, and $K$ is the KL budget (the bound from Proposition 3.1).

What it computes: the Fisher-norm of the parameter update (which directly measures the induced KL change via a quadratic approximation) is bounded by $\sqrt{2K}$, and the ordinary Euclidean norm is bounded by $\sqrt{2K/\mu}$. For any weight matrix block $W \subset \theta$, this implies $\|\Delta W\|_F \le \sqrt{2K/\mu}$—the Frobenius norm (root-sum-of-squares of all entries) of changes to a specific weight matrix is bounded by the per-layer KL budget divided by the minimum Fisher curvature.

Why this form: the Fisher norm is the natural metric for policy space—by the standard quadratic expansion $D_{\text{KL}}(\pi_{\theta+\Delta} \parallel \pi_\theta) \approx \frac{1}{2} \Delta^\top F(\theta) \Delta$. The conversion to Euclidean norm requires dividing by $\mu$, the minimum eigenvalue of the Fisher matrix. When $\mu$ is large (high curvature), even a small Euclidean step causes a large KL change, so the KL leash restricts Euclidean movement tightly. When $\mu$ is small (low curvature), the same KL budget permits larger Euclidean movement. This sets up Gate II: the KL leash constrains total movement, but the direction of movement is determined by where the Fisher matrix has low curvature.

Corollary F.8: Per-layer KL budget. The paper partitions the global KL leash into per-layer budgets. Writing $\theta = (\text{vec}(W), \theta_{\neg W})$, the Schur complement of the Fisher matrix gives a layer-conditioned curvature proxy $S_W = F_{W,W} - F_{W,\neg W} F_{\neg W, \neg W}^{-1} F_{\neg W, W} \succeq 0$. The per-layer budget is $\delta_W = \frac{1}{2} \text{vec}(\Delta W)^\top S_W \, \text{vec}(\Delta W) \le K$. With $\mu_W = \lambda_{\min}(S_W) > 0$, this gives the layer-wise norm bounds used throughout Gate II:

∥ΔW∥F≤2δWμW,∥ΔW∥2≤2δWμW\|\Delta W\|_F \le \sqrt{\frac{2\delta_W}{\mu_W}}, \qquad \|\Delta W\|_2 \le \sqrt{\frac{2\delta_W}{\mu_W}}

Take-away from Gate I: RLVR's on-policy updates are anchored by an implicit KL leash. The policy cannot diverge far from its predecessor in a single step, which translates to bounded weight changes. But the leash constrains the magnitude of the step, not its direction—that is where Gate II enters.

3.4.2 Gate II: Model Geometry (How Pretrained Structure Steers Update Direction)

Gate II is the paper's central mechanism explaining where the KL-constrained update lands. The key claim: a pretrained model has a highly structured Hessian geometry, and the KL leash naturally routes updates toward directions that minimize KL change per unit Euclidean distance—which are precisely the low-curvature, off-principal directions.

Layer-wise norm bounds from the KL leash. From Gate I, we already have $\|\Delta W\|_F \le \sqrt{2\delta_W / \mu_W}$ and $\|\Delta W\|_2 \le \sqrt{2\delta_W / \mu_W}$. These are conservative bounds—the actual update is much smaller than the worst case—but they establish that the update operator norm is tightly controlled. The paper then uses classical matrix perturbation theory to derive three consequences of small-norm updates on pretrained weight matrices.

Theorem 3.3: Constrained subspace rotation (Wedin's sin-Θ theorem). For a pretrained weight matrix $W_0$ and its updated version $W_+ = W_0 + \Delta W$, let $\gamma_k = \sigma_k(W_0) - \sigma_{k+1}(W_0)$ be the singular value gap between the $k$-th and $(k+1)$-th singular values. Then for any $k$ with $\gamma_k > 0$:

max⁡(∥sin⁡Θ(Uk(W0),Uk(W+))∥2,∥sin⁡Θ(Vk(W0),Vk(W+))∥2)≤∥ΔW∥2γk≤2δW/μWγk\max\left( \|\sin \Theta(U_k(W_0), U_k(W_+))\|_2, \|\sin \Theta(V_k(W_0), V_k(W_+))\|_2 \right) \le \frac{\|\Delta W\|_2}{\gamma_k} \le \frac{\sqrt{2\delta_W / \mu_W}}{\gamma_k}

Defining the symbols: $U_k(W)$ is the matrix whose columns are the top-$k$ left singular vectors of $W$ (the principal input directions), $V_k(W)$ is the corresponding matrix of top-$k$ right singular vectors (principal output directions), $\Theta(U_k(W_0), U_k(W_+))$ is the vector of principal angles between the old and new subspaces, and $\sin \Theta$ applies sine element-wise. The matrix $\sin \Theta$ has diagonal entries $\sin \theta_i$ where $\theta_i$ are the principal angles—when the subspaces are aligned, $\sin \theta_i \approx 0$; when they are orthogonal, $\sin \theta_i = 1$.

What it computes: the sine of the largest principal angle between the original top-$k$ singular subspace and the updated top-$k$ singular subspace is bounded by the ratio of the update operator norm to the singular value gap $\gamma_k$. When the gap is large (well-separated singular values), even a moderate-norm update produces minimal subspace rotation. When the gap is small (nearly degenerate singular values), the same update could cause larger rotation—but the KL leash keeps $\|\Delta W\|_2$ itself small.

Why this form: Wedin's theorem is the standard tool for bounding subspace perturbation under additive noise. The singular value gap $\gamma_k$ appears in the denominator because degenerate subspaces are inherently unstable—a tiny perturbation can mix them. But the key insight is that the KL leash from Gate I makes $\|\Delta W\|_2$ small enough that even for modest gaps, the rotation remains constrained. This predicts that RLVR should preserve the top-$k$ singular subspace orientation, which the paper tests in Section 4.1.

Corollary 3.4: Singular value stability. Using Weyl's inequality and the Hoffman-Wielandt theorem:

∣σk(W+)−σk(W0)∣≤∥ΔW∥2≤2δWμW,∑i(σi(W+)−σi(W0))2≤∥ΔW∥F2≤2δWμW|\sigma_k(W_+) - \sigma_k(W_0)| \le \|\Delta W\|_2 \le \sqrt{\frac{2\delta_W}{\mu_W}}, \qquad \sum_i (\sigma_i(W_+) - \sigma_i(W_0))^2 \le \|\Delta W\|_F^2 \le \frac{2\delta_W}{\mu_W}

What it computes: each individual singular value changes by at most the operator norm of the update, and the total squared change across all singular values is bounded by the squared Frobenius norm of the update. Since the KL leash makes both norms small, the pretrained singular value spectrum is preserved.

Why this form: singular values measure the energy (variance) captured by each principal direction. Stability of singular values means the pretrained model's allocation of representational capacity across directions is largely maintained—RL does not destroy the pretrained spectral structure, it works within it.

Corollary 3.5: Top-k energy preservation (Ky Fan norms). The cumulative energy of the top-$k$ components:

∣∥W+∥(k)−∥W0∥(k)∣≤∑i=1k∣σi(W+)−σi(W0)∣≤k∥ΔW∥2≤k2δWμW|\|W_+\|_{(k)} - \|W_0\|_{(k)}| \le \sum_{i=1}^k |\sigma_i(W_+) - \sigma_i(W_0)| \le k \|\Delta W\|_2 \le k \sqrt{\frac{2\delta_W}{\mu_W}}

where $\|\cdot\|_{(k)} = \sum_{i=1}^k \sigma_i(\cdot)$ is the Ky Fan $k$-norm (sum of top-$k$ singular values).

What it computes: the total "energy" captured by the top-$k$ principal components changes by at most $k$ times the operator norm bound. This means the pretrained model's principal subspace not only preserves its orientation (Theorem 3.3) but also its magnitude—the model does not significantly amplify or suppress its primary computational pathways.

Why the principal-weight proxy is needed. Directly measuring curvature (the Hessian of the RL loss) for Large Reasoning Models with long chain-of-thought outputs is computationally prohibitive—the Hessian would be enormous and the loss involves sampling from the policy. The paper sidesteps this by adopting principal weights as a proxy for high-curvature directions, following Liu et al. (2025c). A principal weight is defined operationally: given a weight matrix $W_0$, compute its rank-$k$ SVD reconstruction $W_0^{(k)}$, and identify the top-$\alpha$ fraction of entries with largest absolute magnitude in $W_0^{(k)}$. Liu et al. (2025c) validated this proxy through perturbation experiments showing that modifying these specific weights causes sharp degradation in reasoning performance, consistent with their location in high-curvature regions (where small weight changes cause large loss changes, by Taylor expansion).

The paper's geometric prediction. From Theorem 3.3 and Corollaries 3.4–3.5, the paper derives a testable prediction: RLVR updates should avoid principal weights and instead concentrate in off-principal, low-magnitude parameter regions, because:

  • Principal weights inhabit high-curvature directions where even small Euclidean steps cause large KL changes (large $\mu$), and the KL leash restricts movement there.
  • Off-principal, low-magnitude weights inhabit low-curvature directions where the same KL budget permits larger Euclidean movement (small $\mu$), allowing the optimizer to make meaningful parameter changes without violating the policy-level constraint.
  • The spectrum-preserving nature of small-norm updates means RL can achieve its behavioral goals without restructuring the pretrained model's principal components—it routes information differently rather than rebuilding the foundational representations.

This contrasts with SFT, which (as established by Liu et al., 2025c and Meng et al., 2024a) targets principal weights because supervised learning pulls the model toward a potentially distant target distribution, requiring changes to the high-energy, high-curvature components that dominate the model's behavior.

3.4.3 Gate III: Precision (How bfloat16 Amplifies the Visibility of the Bias)

Gate III explains why the optimization bias from Gates I–II appears as sparsity rather than as a more subtle pattern of small-but-nonzero updates everywhere. The mechanism is numerical: bfloat16 has limited mantissa precision, which creates a magnitude-dependent threshold for whether a weight change is representable in storage.

bfloat16 format properties. bfloat16 (brain floating point 16) allocates 1 bit for sign, 8 bits for exponent, and 7 bits for mantissa. For a normalized value $x$ in the binade $[2^e, 2^{e+1})$, the unit-in-the-last-place (ULP) is $\text{ULP}_{\text{bf16}}(x) = 2^{e-7}$. This means:

  • Around $x \approx 1$ ($e = 0$), the smallest representable difference is $2^{-7} \approx 0.0078$—roughly 0.78%.
  • Around $x \approx 1024$ ($e = 10$), the smallest representable difference is $2^3 = 8$—a change smaller than 8 is invisible.
  • Around $x \approx 10^{-6}$ ($e = -20$), the smallest representable difference is $2^{-27} \approx 7.45 \times 10^{-9}$—very small changes are visible.

Corollary 3.6: Magnitude-dependent realization threshold. A stored weight $W_{ij}$ changes at a step if and only if:

∣ΔWij∣≳12ULPbf16(Wij)|\Delta W_{ij}| \gtrsim \frac{1}{2} \text{ULP}_{\text{bf16}}(W_{ij})

What this means operationally: if the RL optimizer produces a weight change smaller than half a ULP, the stored bfloat16 value does not change—the update is "invisible" in the saved checkpoint, even though it exists in the float32 optimizer states and gradient accumulations. Conversely, if the change exceeds half a ULP, the stored value updates and the change becomes "visible."

Why the absolute-tolerance approach fails. The paper explicitly contrasts its bfloat16-aware probe with the fixed absolute-tolerance check used by Mukherjee et al. (2025). Appendix E.2 provides a rigorous justification. Lemma E.2 shows that $\text{ULP}_{\text{bf16}}(x) / |x| \in (2^{-8}, 2^{-7}]$, meaning the relative threshold for representability is between 0.39% and 0.78%. A fixed absolute tolerance like $10^{-5}$ fails in both directions:

  • False positives at large scale: weights around 1024 differing by 0.001 (below the bf16 ULP of 8) are flagged as "changed" by a $10^{-5}$ tolerance but are actually bitwise identical in storage.
  • False negatives at small scale: weights around $10^{-6}$ differing by $10^{-6}$ (below $10^{-5}$ threshold) are flagged as "equal" but actually represent ~134 ULPs apart and produce different stored values.

Definition 2.1: bfloat16-aware unchanged weight. The paper defines a weight $w_i$ as unchanged with respect to $\hat{w}_i$ if:

∣w^i−wi∣≤ηmax⁡(∣wi∣,∣w^i∣),η=10−3|\hat{w}_i - w_i| \le \eta \max(|w_i|, |\hat{w}_i|), \quad \eta = 10^{-3}

Defining the symbols: $w_i$ is the pretrained weight, $\hat{w}_i$ is the fine-tuned weight, both stored in bfloat16. The parameter $\eta = 10^{-3} < 2^{-9}$ is chosen to be strictly smaller than half the minimum relative ULP, making the test equivalent to bitwise equality by Proposition E.3.

What it computes: instead of checking absolute difference, it checks relative difference scaled by the larger of the two magnitudes. If the relative change is below $10^{-3}$, the stored values are guaranteed bitwise identical (Proposition E.3).

Why this form: the $\max(|w_i|, |\hat{w}_i|)$ scaling accounts for the fact that bfloat16 resolution depends on magnitude—a change of 0.001 is invisible at $x = 1000$ but highly visible at $x = 10^{-4}$. Using the maximum handles edge cases where the update crosses a binade boundary. Setting $\eta = 10^{-3}$ is safe because $10^{-3} < 2^{-9} \approx 1.95 \times 10^{-3}$, ensuring the test flags equality only when storage is truly unchanged.

Definition 2.2: bfloat16-aware update sparsity. The change count and sparsity are:

∥θ1−θ0∥0,ηbf16:=∣{i:θ1i̸≈ηbf16θ0i}∣\|\theta_1 - \theta_0\|^{\text{bf16}}_{0, \eta} := |\{i : \theta_{1i} \not\approx^{\text{bf16}}_\eta \theta_{0i}\}|

sparsitybf16(θ0,θ1;η):=1−∥θ1−θ0∥0,ηbf16n\text{sparsity}_{\text{bf16}}(\theta_0, \theta_1; \eta) := 1 - \frac{\|\theta_1 - \theta_0\|^{\text{bf16}}_{0, \eta}}{n}

where $n$ is the total number of parameters.

What it computes: the fraction of parameters whose stored bfloat16 values are identical before and after training. Values near 1 indicate few stored changes (high apparent sparsity); values near 0 indicate dense apparent change.

The precision critique that completes the argument. Gate III is not claiming that precision causes the optimization bias—it is claiming that precision amplifies its visibility as sparsity. The paper provides two pieces of evidence for this:

  1. Learning rate scaling experiment: increasing the learning rate pushes otherwise sub-ULP updates above the representable threshold, and "the apparent update sparsity largely disappears."
  2. Concurrent finding from Shenfeld et al. (2025): "sparsity mostly vanishes under float32 storage... even though task performance does not improve." The same optimization bias exists in float32, but without the precision filter, the updates are visible everywhere—tiny in non-preferred regions, larger in preferred regions—rather than appearing as a binary mask.

The paper also clarifies a potential misunderstanding: "It may be tempting to blame precision limit for sparsity. In fact, verl keeps optimizer states and gradient reductions/accumulation in float32. Thus, sparsity cannot be explained by precision alone." The optimization bias must exist first (Gates I–II routing updates toward specific regions), and precision then hides the small updates that occur elsewhere.

3.4.4 The bfloat16-Aware Measurement Apparatus

This subsection documents the specific measurement protocols the paper uses throughout all experiments, not as theory but as operational definitions.

Update mask construction. For a given layer $\ell$ and run $r$, the bf16-aware changed mask is:

Mℓ(r):=1[Wℓ(r)̸≈ηbf16Wℓ0]M^{(r)}_\ell := \mathbb{1}\left[W^{(r)}_\ell \not\approx^{\text{bf16}}_\eta W^0_\ell\right]

where $W^{(r)}_\ell$ is the fine-tuned weight matrix, $W^0_\ell$ is the base weight matrix, and the indicator function returns 1 for coordinates where the bf16 values differ (by Definition 2.1) and 0 otherwise. This produces a binary matrix the same shape as the weight matrix.

Cross-run stability (Jaccard overlap). For two independent runs $r$ and $s$, let $A = \{(i, j) : M^{(r)}_{\ell, ij} = 1\}$ and $B = \{(i, j) : M^{(s)}_{\ell, ij} = 1\}$ be the sets of updated coordinates. The Jaccard overlap is:

J(A,B)=∣A∩B∣∣A∪B∣J(A, B) = \frac{|A \cap B|}{|A \cup B|}

The paper reports the mean off-diagonal of the pairwise Jaccard matrix across all pairs of runs and compares it to the independent Bernoulli baseline:

E[J]=pqp+q−pq\mathbb{E}[J] = \frac{p q}{p + q - p q}

where $p$ and $q$ are the update densities (fractions of weights changed) in runs $r$ and $s$ respectively. The independent Bernoulli baseline is what you would expect if updates were randomly scattered with the same per-run density—a value substantially above this baseline indicates shared structure.

Consensus ratio. For each coordinate $(i, j)$ in a weight matrix, across $R$ independent runs:

Cℓ,ij=1R∑r=1RMℓ,ij(r)C_{\ell, ij} = \frac{1}{R} \sum_{r=1}^R M^{(r)}_{\ell, ij}

Values near 1 indicate that all runs consistently modify that weight; values near 0 indicate that no run modifies it. Visualization of $C_{\ell, ij}$ as a heatmap (Figure 2) reveals stripe-like patterns—contiguous rows or columns where updates concentrate—rather than random scatter.

Temporal analysis (row/column-wise ratios). For a checkpoint at training step $t$, the row-wise update ratio for row $i$ of layer $\ell$ is:

ρℓ,i(t)=1nℓ∑jMℓ,ij(t)\rho_{\ell, i}(t) = \frac{1}{n_\ell} \sum_j M_{\ell, ij}(t)

where $n_\ell$ is the number of columns. The column-wise ratio $\kappa_{\ell, j}(t)$ is defined analogously. The paper tracks these ratios across checkpoints to assess whether the update pattern stabilizes early (temporal persistence) or shifts during training. The profiles $\rho_{\ell, \cdot}(t)$ and $\kappa_{\ell, \cdot}(t)$ are tracked with their relative shapes (peaks and troughs) staying aligned while overall density grows (Figure 3), indicating the routing bias emerges early and is reinforced, not a transient artifact.

3.4.5 The Principal-Weights Diagnostic Framework

This section documents the operational definitions used to test Gate II's predictions about where updates land.

Principal mask construction (following Liu et al., 2025c). For a weight matrix $W_0 \in \mathbb{R}^{m \times n}$:

  1. Compute the rank-$k$ SVD reconstruction: $W_0^{(k)} = U_k \Sigma_k V_k^\top$, where $U_k$ and $V_k$ are the top-$k$ left and right singular vectors and $\Sigma_k$ is the diagonal matrix of top-$k$ singular values.
  2. For each entry $(i, j)$, compute the score: $s_{ij}^{(k)} = |W_0^{(k)}(i, j)|$—the absolute value of the rank-$k$ reconstruction at that position.
  3. The principal mask $M^{(k)}_{\text{princ}}$ selects the top-$\alpha$ fraction of entries with the highest $s_{ij}^{(k)}$ values:

Mprinc(k)=Topα(sij(k))M^{(k)}_{\text{princ}} = \text{Top}_\alpha\left(s_{ij}^{(k)}\right)

What this mask identifies: weights that contribute most to the top-$k$ singular components—the high-energy computational pathways that dominate the model's forward pass. These weights are empirically validated by Liu et al. (2025c) as residing in high-curvature regions where perturbations cause sharp performance degradation.

Low-magnitude mask. The paper also defines a complementary mask based on the original (non-reconstructed) weight magnitudes:

Mlow=Bottomα(∣W0(i,j)∣)M_{\text{low}} = \text{Bottom}_\alpha\left(|W_0(i, j)|\right)

This selects the smallest-magnitude weights in the original matrix. The paper treats these as "low-resistance pathways" because small magnitudes mean the bfloat16 relative threshold is more forgiving (small absolute changes can still be representable), making these weights easier to update without violating the precision gate.

Overlap metrics. For an RL-derived update mask $M$ (the bf16-aware changed mask from Section 3.4.4) and a reference mask $M_\bullet$ (e.g., $M_{\text{princ}}$ or $M_{\text{low}}$):

Overlap(M∙,M)=∣M∙∩M∣∣M∣\text{Overlap}(M_\bullet, M) = \frac{|M_\bullet \cap M|}{|M|}

This measures what fraction of the RL-updated weights fall within the reference mask. The random baseline for comparison is the density of $M_\bullet$ itself (i.e., $\alpha$)—if RL updates were scattered randomly, the overlap would equal $\alpha$. Values above $\alpha$ indicate super-random concentration; values below $\alpha$ indicate active avoidance.

Prediction from Gate II: RL updates should show sub-random overlap with $M_{\text{princ}}$ (avoid principal weights) and super-random overlap with $M_{\text{low}}$ (favor low-magnitude weights), even after controlling for the overlap between the two masks.

3.4.6 Experimental Design for Theory Validation (Section 4)

The paper designs four classes of experiments to test specific predictions from the Three-Gate Theory.

Experiment class 1: Spectral geometry comparison (Section 4.1). The prediction from Gate II is that RLVR preserves pretrained spectral structure while SFT distorts it. The protocol:

  • Models: Qwen3-8B-Base (SFT→RLVR pipeline), DS-Qwen-1.5B (long-horizon RL), Qwen3-14B-Base (separate SFT and RL runs matched on in-domain math performance).
  • Metrics:
    • Subspace rotation: for the top-$k$ singular subspaces, compute principal angles via $\cos \theta_i(U) = \sigma_i(U_{0,k}^\top U_{+,k})$ (singular values of the cross-covariance between old and new subspace bases). $\theta_i = 0$ means perfect alignment; $\theta_i = \pi/2$ means orthogonal.
    • Spectral drift: normalized $\ell_2$ shift: $\text{NSS}(W) = \|\sigma(W_+) - \sigma(W_0)\|_2 / \|\sigma(W_0)\|_2$.
  • Expected outcome: RLVR shows consistently small principal angles (near-zero rotation) and low NSS values, while SFT shows larger rotations and drifts. Figure 4 (main text) and Figures 16–17 (appendix) present these results.

Experiment class 2: Update-principal overlap (Section 4.2). The prediction from Gate II is that RLVR updates avoid principal weights, while SFT targets them. The protocol:

  • Compute the bf16-aware update mask $M$ from an RLVR run.
  • Compute the principal mask $M_{\text{princ}}$ and low-magnitude mask $M_{\text{low}}$ from the pretrained weights.
  • Report layer-wise $\text{Overlap}(M_\bullet, M)$ against the random baseline of $\alpha$.
  • Also report overlap with $M_{\text{princ}} \cap M_{\text{low}}^c$ (principal weights that are NOT low-magnitude), to disentangle the principal effect from the magnitude effect.
  • Expected outcome: RL updates show sub-random overlap with $M_{\text{princ}}$, super-random overlap with $M_{\text{low}}$, and very low overlap with $M_{\text{princ}} \cap M_{\text{low}}^c$.

Experiment class 3: Causal intervention on geometry (Section 4.3). Gate II claims pretrained geometry is causal—disrupting it should abolish the update bias. The protocol:

  • Intervention 1 (Rotate): For specific V and O projection layers, apply function-preserving orthogonal rotations. Let $D$ be the head dimension, $H_{\text{kv}}$ the number of key/value heads, $H_q$ the number of query heads. Draw an orthogonal $R \in \mathbb{R}^{D \times D}$ and form block rotations $R_{\text{kv}} = \text{diag}(R, \ldots, R)$ (repeated $H_{\text{kv}}$ times) and $R_q = \text{diag}(\underbrace{R, \ldots, R}_{n_{\text{rep}}}, \underbrace{R, \ldots, R}_{n_{\text{rep}}}, \ldots)$ (expanded for grouped query attention). Edit weights: $W_v' = W_v R_{\text{kv}}$, $W_o' = W_o R_q$. Proposition D.1 proves exact functional invariance: $\text{Attn}(Q, K, V R_{\text{kv}})(W_o R_q)^\top = \text{Attn}(Q, K, V) W_o^\top$.
  • Intervention 2 (Permute): For Q/K/V/O layers, apply head permutations. Let $P_{\text{kv}}$ permute KV heads, $P_q$ its grouped expansion to Q heads, and $P_q^{-1}$ the inverse applied to O columns. This relabels which head carries which subspace without changing the block function.
  • Measurement: Apply RLVR fine-tuning to the intervened model. Compute the update overlap $\text{Overlap}(M_{\text{run1}}, M_{\text{run2}})$ between two independent runs—one with and one without intervention—on the same base model. Compare intervened layers to untouched layers.
  • Expected outcome: Update overlap collapses to random levels in intervened layers but remains high in untouched layers.

Experiment class 4: Sparse fine-tuning probe (Section 5.1). This tests whether the theory correctly identifies which weights are "touchable" (can be updated without impeding learning) vs. "untouchable" (must be frozen to preserve the training trajectory). The protocol:

  • Mask design (all at 50% density or equivalent):
    • $M_{\text{princ}}$: top-50% principal weights only (the SFT-aligned mask—predicted to perform worst).
    • $M_{\text{princ}}^c$: bottom-50% non-principal weights only (predicted to perform well).
    • $M_{\text{low}}$: bottom-50% low-magnitude weights only.
    • $M_{\text{low}} \cup M_{\text{princ}}^c$: the "safe mask"—non-principal AND low-magnitude weights (predicted to best track dense performance).
    • $\text{rand-}M_{\text{low}} \cup M_{\text{princ}}^c$: a random mask with the same layer-wise sparsity as the safe mask (control for density effects).
  • Training: Run RLVR fine-tuning with each mask (freezing masked-out weights). Track token-wise forward KL divergence $\text{KL}(\pi \parallel \pi_{\text{ref}})$ throughout training (following Shenfeld et al., 2025) and final evaluation accuracy on MATH500, AMC23, AIME24, AIME25.
  • KL metric justification: If freezing certain weights impedes learning, the policy's KL divergence from the base model will grow more slowly (or not at all), indicating blocked optimization progress. A mask that closely tracks the dense KL curve is correctly identifying the parameters RL needs to update.
  • Expected outcome: The safe mask $M_{\text{low}} \cup M_{\text{princ}}^c$ most closely tracks the dense KL curve and reaches comparable accuracy using ~74% of parameters in linear layers (Table 4, 320 steps; Table 5, 500 steps). The principal-only mask $M_{\text{princ}}$ shows the slowest KL growth and worst accuracy, directly confirming that principal-targeted directions (SFT-favored) are ineffective for RL.

3.4.7 LoRA/PiSSA Comparison Methodology (Section 5.2)

This case study tests the practical prediction that principal-aligned LoRA variants (PiSSA) should underperform standard LoRA for RLVR because they enforce updates along precisely the directions RL's geometry bias avoids.

Adapter configuration. Both LoRA and PiSSA apply low-rank adapters to attention projection matrices (Q, K, V, O) and optionally MLP layers:

  • LoRA initializes $A$ with random Gaussian and $B$ with zeros, so the effective update $\Delta W = BA$ starts at zero and learns freely.
  • PiSSA initializes $A$ and $B$ to approximate the top-$r$ singular components of the pretrained weight matrix $W_0$ via SVD: $W_0 \approx U_r \Sigma_r V_r^\top$, then sets $A = U_r \sqrt{\Sigma_r}$ and $B = \sqrt{\Sigma_r} V_r^\top$. This biases the adapter toward updating principal directions from the start.

Hardware and training configuration. The paper uses the same DAPO recipe but with reduced compute for learning-rate sweeps:

  • Global batch size 128 (mini-batch 32), 4 gradient updates per step, 16 rollouts per prompt.
  • Both DS-Qwen-1.5B (on DeepMath-103K) and LLaMA-3.2-3B-Instruct (on a Math corpus) trained for 200 steps.
  • Rank sweep: $\{8, 32, 64\}$.
  • Learning rate sweep: $\{1 \times 10^{-4}, 5 \times 10^{-5}, 1 \times 10^{-5}\}$.
  • Actor optimizer: AdamW (constant learning rate $1 \times 10^{-6}$ for full fine-tuning baseline, $\beta_1 = 0.9$, $\beta_2 = 0.999$).
  • Rewards: +1.0 if the extracted final answer is correct, −1.0 otherwise (no format score).
  • Over-length penalty: additional 1024-token budget, penalty factor 1.0.

Evaluation protocol:

  • DS-Qwen-1.5B: pass@1 averaged over 16 samples (avg@16) on AIME24 and AMC23.
  • LLaMA-3.2-3B: pass@1 averaged over 4 samples (mean@4) on MATH500.

Why the learning rate sweep matters. The paper's theoretical argument is that PiSSA's principal-direction initialization forces updates along high-curvature directions. At low learning rates, this effect is mild because both LoRA and PiSSA make small steps. But for low-rank adapters to match full-parameter RL performance, higher learning rates are typically needed (to compensate for the reduced capacity). At these higher learning rates, PiSSA's principal-direction bias becomes pronounced—it pushes updates in directions the KL leash and geometry gate would naturally avoid, leading to training instability and collapse.

Expected outcome: Across ranks and learning rates, PiSSA provides no clear gain over standard LoRA. At high learning rates, PiSSA often destabilizes and collapses earlier than LoRA. Standard LoRA, with its random initialization and frozen base weights, naturally approximates off-principal updates—the frozen base discourages principal-direction moves, and the small adapter capacity is sufficient because effective RL updates are concentrated in a narrow subspace anyway. Figures 10 and 11 present these results.

Model suite for the main analysis (Table 1). The paper's primary observations draw from publicly released checkpoints spanning:

  • Models: Qwen-1.5B, Qwen3-8B-Base, Qwen3-14B-Base, Qwen3-4B, Qwen3-30B-A3B (Mixture-of-Experts), DS-R1-Distill-Qwen-7B.
  • RL algorithms: GRPO, DAPO, Reinforcement++.
  • Data domains: Math (DeepScaleR, DeepCoder, Polaris, Klear-Reasoner, GT-Qwen3, UniReason, SimpleRL-Zoo), Code (Archer-Code, DeepCoder), Mixed (NV-ProRL includes math, coding, STEM, logic puzzles, instruction-following).
  • Extended suite for agents and RLHF (Table 3): AgentFlow (planning), GiGPO (WebShop, ALFWorld), SkyRL (WebResearch), VERL-Agent (DeepSearch, SWE), plus DPO and SimPO checkpoints for RLHF.

The paper places particular emphasis on DS-Qwen-1.5B (DeepSeek-R1-Distill-Qwen-1.5B further fine-tuned with RLVR for over 3,000 steps on a diverse data mixture) as a "robust case study given its extensive training," with multiple independently trained checkpoints available for cross-run stability analysis.

4. Key Insights and Innovations

Innovation 1: Reframing RLVR Sparsity from a Mystery to a Surface Readout of a Deeper Geometric Bias

The paper's most fundamental conceptual move is reinterpreting the reported sparsity of RLVR updates (Mukherjee et al., 2025) not as the phenomenon of interest but as a visual artifact of an underlying optimization bias interacting with finite precision. Prior work had observed that RL modifies only a fraction of parameters and treated this sparsity as the primary puzzle: why does expensive RL training change so few weights? This paper inverts the question: sparsity is what you see when you have a consistent bias about where updates go, and bfloat16 precision filters out small changes elsewhere.

This reframing matters because it shifts the research agenda. If sparsity were the core phenomenon, the natural response would be to exploit it—design sparse training algorithms that only update the "important" subset, prune everything else, and celebrate the efficiency gains. The paper shows this would be misguided: the sparsity is an epiphenomenon of bfloat16 storage, not a property of the gradient signal itself. Under float32 (as concurrent work by Shenfeld et al., 2025 confirms), "sparsity mostly vanishes... even though task performance does not improve." The same optimization bias exists—the optimizer concentrates meaningful updates in specific parameter regions—but without the precision filter, those small updates in non-preferred regions remain visible. The paper provides direct evidence by increasing the learning rate (Section 2.3): sub-ULP updates become representable, and "the apparent update sparsity largely disappears."

What makes this reframing intellectually distinctive is that it separates the signal (optimization bias) from the sensor (numerical precision). The prior literature conflated them. By distinguishing Gates II (geometry, which determines where updates accumulate) from Gate III (precision, which determines where updates are visible), the paper provides a framework that explains why (a) sparsity patterns are consistent across runs and datasets (shared geometry), (b) sparsity increases with lower precision (precision threshold), and (c) sparsity can be "tuned" by changing the learning rate or storage format without affecting the underlying learning dynamics. This is a conceptual clarification, not merely a measurement improvement—it tells us what to optimize (geometry-aligned update routing) versus what to ignore (the bf16 change mask as a literal description of learning).

The practical implication is significant: practitioners interpreting bfloat16 checkpoint diffs should not mistake storage-level sparsity for algorithmic sparsity. The optimizer states and gradient accumulations (in float32 in frameworks like verl) contain richer update information than the stored weights reveal. Sparse fine-tuning methods based purely on bf16 update masks would capture a precision-distorted view of the actual learning dynamics.

Innovation 2: A Parameter-Level, Mechanistic Account of Where RL Updates Land, Not Just How Much the Policy Changes

The paper provides what it claims is "the first parameter-space account linking RL optimization dynamics to weight evolution." Prior work on RLVR's training dynamics—even sophisticated concurrent analyses (Wu et al., 2025; Shenfeld et al., 2025)—operated at the policy level: they showed that RL policies remain KL-proximal to the base policy, that RL causes less forgetting than SFT, and that the effective update is constrained by a KL budget. These are distributional statements about what the model outputs, not about what happens to the model's weights.

The gap this paper fills is non-trivial. Knowing that the policy stays KL-close to the base tells you nothing about which weights change to achieve that. The same KL-constrained policy shift could, in principle, be realized by modifying principal weights, off-principal weights, a random subset, or all weights uniformly. The authors show it is realized through a highly structured, predictable pattern: updates concentrate in off-principal, low-magnitude parameter regions, avoid principal weights (defined via rank-k SVD reconstruction), and preserve the pretrained spectral structure (minimal subspace rotation, minimal singular-value drift). This is not a trivial consequence of KL proximity—it requires Gate II (model geometry) to explain.

What's distinctive here is the diagnostic toolkit itself. The paper operationalizes geometric concepts that had been discussed abstractly—"high-curvature directions," "low-curvature subspaces," "principal components"—into concrete, computable measurements on real LLM checkpoints. The principal mask (top-α weights by magnitude in the rank-k SVD reconstruction, following Liu et al., 2025c), the spectral drift metric (normalized ℓ₂ shift of singular values), the principal angle between subspaces (via Wedin's sin-Θ theorem), and the consensus ratio (fraction of independent runs that modify a given weight coordinate) form a reusable diagnostic suite. Future work can apply these tools to new models, new RL algorithms, or new task domains without reinventing the measurement apparatus.

The comparison with SFT is particularly instructive. The paper doesn't just characterize RL's dynamics in isolation—it shows they are qualitatively opposite to SFT's on the same diagnostic metrics. SFT targets principal weights (consistent with Liu et al., 2025c and Meng et al., 2024a), distorts the singular value spectrum, and rotates principal subspaces substantially. RL avoids principal weights, preserves the spectrum nearly identically to the base model, and induces minimal subspace rotation. This is captured concisely in Figure 1(c): the RL update mask shows sub-random overlap with principal positions, while SFT's mask would show super-random overlap. The principal-angle curves show RL's top-k subspaces barely rotate, while SFT's rotate significantly. This empirical contrast is not predicted by the policy-level KL proximity account alone—both RL and SFT could be KL-constrained relative to their starting points, but SFT's starting point is the pretrained model while RL's starting point is the previous checkpoint. The difference in weight-space dynamics reveals that the KL leash (Gate I) operates differently when the reference is distant (SFT) versus proximal (RL's on-policy updates).

This contribution is fundamental rather than incremental: it establishes a new level of analysis (parameter-space geometry) for understanding post-training, complementing the existing policy-level and behavioral-level perspectives.

Innovation 3: Causal Demonstration That Pretrained Geometry—Not Data or Algorithm—Steers RL Updates

Many papers report correlations between model properties and training behavior. This paper goes further by manipulating the hypothesized causal mechanism and observing the predicted effect. The geometric intervention experiment (Section 4.3) is methodologically distinctive because it cleanly isolates model geometry from other confounds.

The intervention: take specific layers of a pretrained Qwen3-4B model, apply function-preserving orthogonal rotations to V and O projections (Proposition D.1 proves exact output invariance), or permute attention heads (also functionally invariant). Then run RLVR fine-tuning on the intervened model and measure whether the update overlap between independent runs at the intervened layers matches the overlap at untouched layers.

The result: update overlap collapses to random levels in the intervened layers while remaining high in all untouched layers (Figure 6). This is strong causal evidence because everything else is held constant—the training data, the RL algorithm, the reward signal, the optimizer, the learning rate—and only the pretrained geometry is scrambled. The functional behavior of the model is unchanged by the intervention (same forward pass, same outputs), so any difference in training dynamics must be attributed to the geometry of the parameter space, not to differences in initial policy behavior.

Why this matters beyond the paper's specific claims: it establishes a template for causal-mechanistic analysis of deep learning training dynamics. The field often relies on observational correlations (e.g., "RL updates are sparse," "SFT targets principal weights") without testing whether the hypothesized mechanism is genuinely causal. The orthogonal-rotation intervention is particularly elegant because it preserves all functional properties while reshuffling the coordinate basis, directly testing whether the specific arrangement of weights—their geometry—matters, or whether any functionally equivalent parameterization would produce the same training dynamics. The answer is clear: geometry matters.

This finding also explains the "model-conditioned" nature of the optimization bias that the paper emphasizes throughout. The bias is invariant to datasets and RL algorithms but highly consistent for a fixed pretrained model precisely because it is encoded in the pretrained weight geometry. Change the model (e.g., Llama vs. Qwen), and the specific parameters where updates concentrate will differ, but the principle—updates avoid principal directions and preserve spectral structure—generalizes. The authors show this by observing "similar stripe-structured footprints on Llama and Mistral" (Appendix, Figure 13), confirming the phenomenon is generic to RLVR applied to pretrained models, not specific to one architecture.

Innovation 4: Demonstrating That SFT-Era PEFT Methods Are Actively Misaligned with RLVR's Optimization Geometry

This is a negative result with substantial practical consequences. The paper's case studies on sparse fine-tuning (Section 5.1) and LoRA variants (Section 5.2) provide principled evidence that parameter-efficient methods designed for SFT can be not merely suboptimal but counterproductive when applied to RLVR. This is more than "method X doesn't work for RL"—it is a diagnosis of why it fails, grounded in the geometric theory.

The sparse fine-tuning experiment is particularly clean. The paper constructs parameter masks directly from the pretrained model (no training required) and shows that:

  • Freezing principal weights (M_princ, 50% density) yields the worst KL trajectory and degraded accuracy (Table 4: 53.82% average vs. 57.44% dense at 320 steps; Table 5: 55.44% vs. 58.59% at 500 steps).
  • Freezing non-principal weights (M_princ^c) performs substantially better (55.40% at 320 steps, 56.90% at 500 steps).
  • A "safe mask" combining non-principal and low-magnitude weights (M_low ∪ M_princ^c) closely tracks the dense training trajectory and reaches comparable accuracy (56.24% at 320 steps, 58.37% at 500 steps) using only ~74% of linear-layer parameters.
  • A random mask with identical layer-wise sparsity (same density, random locations) underperforms the safe mask (55.34% at 320 steps, 56.89% at 500 steps), confirming that the specific choice of which weights to freeze matters, not just the density.

This is actionable: practitioners can identify a subset of parameters that RL needs to update without running any RL training, purely from the pretrained weights. The principal mask and low-magnitude mask require only SVD and thresholding.

The LoRA/PiSSA comparison (Section 5.2) addresses a timely practitioner debate. Schulman & Lab (2025) had reported that low-rank LoRA (even rank-1) can match full-parameter RL performance, and suggested that principal-targeted variants like PiSSA "should yield further gains." The paper's geometric theory predicts the opposite: PiSSA initializes adapters along the top-r singular directions, precisely the high-curvature, principal-weight directions that RL's geometry bias steers away from. The empirical results bear this out: across ranks {8, 32, 64} and learning rates {1×10⁻⁴, 5×10⁻⁵, 1×10⁻⁵} on two model families (DS-Qwen-1.5B and LLaMA-3.2-3B), PiSSA provides "no clear gain over LoRA" and "at higher learning rates... often collapses early" (Figures 10, 11). The learning-rate effect is key: low-rank adapters typically need higher learning rates to compensate for reduced capacity and match full-parameter performance, but these higher rates amplify PiSSA's principal-direction bias, causing the destabilization the geometric theory predicts.

What makes this innovation significant is that it provides a decision rule, not just a benchmark result. Rather than empirically trying every PEFT method for RL and reporting which one wins (which would be fragile to hyperparameter choices and model-specific effects), the paper offers a geometric criterion: if a PEFT method biases updates toward principal directions (high-curvature, spectrum-distorting), it is misaligned with RLVR's dynamics and should be avoided. If it allows off-principal updates while regularizing principal directions (as standard LoRA does by freezing the base weights, or as the safe sparse mask does by freezing principal and large-magnitude weights), it is better aligned. This transforms PEFT-for-RL from an empirical crapshoot into a geometry-aware design problem. The paper does not claim to have solved this design problem—it explicitly calls for "a new generation of RL-native, geometry-aware parameter-efficient algorithms"—but it has provided the diagnostic framework and the cautionary evidence needed to motivate that development.

5. Experimental Analysis

Evaluation Methodology

  • Dataset. The primary training and evaluation domain is mathematical reasoning, using the DAPO-MATH-17k set combined with the MATH dataset (Hendrycks et al., 2021) for the "DM" corpus, and the DeepScaler collection (Luo et al., 2025b) combined with high-difficulty (levels 3–5) problems from SimpleRL (Zeng et al., 2025a) for the "DS+SR" corpus. For evaluation, the paper uses four benchmarks: AIME24 (MAA, 2024), AIME25 (MAA, 2025), AMC23 (MAA, 2023), and MATH-500 (Lightman et al., 2023). The LoRA/PiSSA comparison additionally uses DeepMath-103K (He et al., 2025) for DS-Qwen-1.5B training and a Math corpus for LLaMA-3.2-3B training. The agent experiments use checkpoints from WebResearch, DeepSearch, SWE, Planning, WebShop, and ALFWorld tasks (Table 3). The RLHF experiments use DPO and SimPO checkpoints trained on instruction-following data.

  • Base model(s). The primary analysis uses publicly released checkpoints spanning multiple model families and scales: DeepSeek-R1-Distill-Qwen-1.5B (DS-Qwen-1.5B) as the most extensively studied case (over 3,000 RL steps on a diverse data mixture), Qwen3-8B-Base and Qwen3-14B-Base for SFT→RLVR comparisons, Qwen3-4B for the geometric intervention experiments, and additional models including Qwen3-30B-A3B (Mixture-of-Experts), DS-R1-Distill-Qwen-7B, Llama-3.1-8B, and Mistral-Small-24B for cross-family generalization. The paper justifies DS-Qwen-1.5B as a "robust case study given its extensive training" on a mixture encompassing mathematics, coding, STEM, logic puzzles, and instruction-following tasks. For the agent analysis, models include Qwen3-8B and Qwen2.5-7B-Instruct fine-tuned with various RL algorithms on agent tasks (Table 3). For the RLHF analysis, Meta-Llama-3-8B-Instruct is used with DPO and SimPO.

  • Metrics. The paper employs two distinct metric families. Behavioral metrics: pass@1 averaged over k samples (avg@k), with k = 64 for AIME24/AIME25, k = 32 for AMC23, and k = 2 for MATH-500, evaluated at temperature 0.7 and top-p 0.8 using Evalchemy (Raoof et al., 2025). Parameter-space metrics: (i) bfloat16-aware update sparsity (sparsity_bf16), the fraction of parameters with bitwise-identical stored values before and after training (Definition 2.2); (ii) Jaccard overlap between update masks across runs, measuring spatial agreement of changed weight positions; (iii) consensus ratio, the per-coordinate fraction of independent runs that modify a given weight; (iv) principal angles between top-k singular subspaces, computed as cos θ_i(U) = σ_i(U_{0,k}^⊤ U_{+,k}) where larger cosine values indicate less rotation; (v) normalized spectral shift, NSS(W) = ‖σ(W_+) − σ(W_0)‖_2 / ‖σ(W_0)‖_2; (vi) overlap ratio between RL update masks and reference masks (principal, low-magnitude), Overlap(M_●, M) = |M_● ∩ M| / |M|. For the sparse fine-tuning probe, the paper additionally tracks token-wise forward KL divergence KL(π ‖ π_ref) between the fine-tuned policy and the base model throughout training, following Shenfeld et al. (2025), used to quantify whether freezing certain weights impedes optimization progress.

  • Baselines. (i) Dense full-parameter RLVR: using DAPO without explicit KL penalty (ratio clipping only), the primary baseline against which sparse fine-tuning masks are compared via KL trajectory tracking and final accuracy (Table 4, Figure 9). (ii) SFT checkpoints: for spectral comparisons, the SFT versions of DS-Qwen-1.5B (Liu et al., 2025a), Qwen3-8B (Su et al., 2025), and Qwen3-14B (Huan et al., 2025) serve as the contrasting optimization regime, with SFT matched to in-domain math performance where applicable. (iii) Random Bernoulli baseline: for Jaccard overlap, the expected value E[J] = pq/(p+q−pq) where p and q are per-run update densities, serving as the null hypothesis of independent random update placement. (iv) Random density baseline: for overlap ratios, the density α of the reference mask M_● itself—if RL updates were scattered randomly, the overlap would equal α. (v) Random mask control: for sparse fine-tuning, a random mask with the same layer-wise sparsity as the safe mask (rand-M_low ∪ M_princ^c) controls for the effect of density alone versus the specific choice of which weights to freeze. (vi) Standard LoRA: serving as the off-principal adapter baseline for the PiSSA comparison, with random Gaussian initialization for A and zero initialization for B. (vii) Full fine-tuning for the LoRA/PiSSA experiments at learning rate 1×10^{−6}, providing the dense performance ceiling (Figure 11).

  • Generation budget / compute accounting. The paper does not use "generations" as a budget metric in the traditional sense—this is not a study of test-time compute scaling. Instead, compute accounting is done at two levels. For the main RL training experiments, the budget is measured in gradient update steps (e.g., 320 steps for DS-Qwen-1.5B on DS+SR, 300 steps for Qwen2.5-Math-7B on DM, 150 steps for Qwen3-4B-Base on DS+SR, 200 steps for LoRA/PiSSA sweeps), with per-step details: global batch size 256 (mini-batch 64), 4 gradient updates per step, and 12–16 rollouts per prompt depending on the model. For the sparse fine-tuning probe, the fraction of trainable parameters is reported as #params relative to the dense baseline (100% for dense, 50% for M_princ and M_princ^c, 58.59% for M_low, 74.02% for M_low ∪ M_princ^c). For the LoRA/PiSSA comparison, the rank r ∈ {8, 32, 64} controls adapter capacity. The paper does not report total FLOPs or GPU-hours.

  • Cross-validation / statistical protocol. The paper does not employ cross-validation splits for strategy selection, since it is not proposing a learned allocation policy. Instead, statistical reliability is established through multi-run replication: for the cross-run stability analysis (Section 2.2), 5 independent RLVR checkpoints from the same DS-Qwen-1.5B base model trained on disjoint datasets and RL algorithms are compared via Jaccard overlap and consensus ratio. For the geometric intervention experiment (Section 4.3), update overlap is compared between independent runs with and without intervention, using untouched layers as within-experiment controls. For evaluation metrics (pass@1), the paper reports mean accuracy over k samples with fixed random seeds (temperature 0.7, top-p 0.8). No confidence intervals or error bars are reported for any metric—the paper relies on consistency across models, algorithms, and datasets rather than formal statistical testing.

Main Quantitative Results

Update Sparsity Across Model Families and RL Algorithms (Section 2.1)

Table 1 reports bfloat16-aware update sparsity for 12 publicly released checkpoints spanning 6 model families (Qwen-1.5B, Qwen3-8B, Qwen3-14B, Qwen3-4B, Qwen3-30B-A3B, DS-R1-Distill-Qwen-7B), 3 RL algorithms (GRPO, DAPO, Reinforcement++), and multiple data domains (math, code, mixed). The headline finding: RLVR consistently yields substantially higher update sparsity than SFT across all models and configurations.

SFT sparsity values are uniformly low: 2.8% for DS-R1-Distill-Qwen-1.5B, 0.6% for Klear-Reasoner-8B-SFT, 18.8% for UniReason-Qwen3-14B-think-SFT. In contrast, RLVR sparsity ranges from 36.3% (NV-ProRL-v2, Reinforcement++) to 91.7% (UloRL-A3B, GRPO), an order of magnitude higher in most cases. The paper notes that "absolute levels on recent checkpoints are lower than earlier reports (Mukherjee et al., 2025)" (the earlier work reported ~99% sparsity), attributing this to the bfloat16-aware probe and re-evaluation on more recent models. For instance, DS-Qwen-1.5B fine-tuned with GRPO on math (DeepScaleR-1.5B-Preview) shows 53.8% sparsity, while the same base model fine-tuned on code (DeepCoder-1.5B-Preview) shows 45.5%, and on the diverse mixture (NV-ProRL) shows 38.4%—all substantially lower than the ~99% figure from prior work, yet still far sparser than SFT's ~3%.

The variation across data domains and algorithms for the same base model (DS-Qwen-1.5B spans 36.3%–53.8%) indicates some sensitivity to training configuration, but all values remain in a regime qualitatively distinct from SFT. The Qwen3-8B-Base results are particularly striking: GRPO on math yields 79.9% sparsity, and DAPO on math yields 79.7% sparsity—nearly identical despite different algorithms, consistent with the paper's claim that the optimization bias is "largely invariant to datasets and RL recipes" for a fixed pretrained model. The Qwen3-30B-A3B model (Mixture-of-Experts) shows the highest sparsity at 91.7%, though the paper does not discuss whether MoE architecture amplifies the effect.

Cross-Run Stability and Consensus of Update Locations (Section 2.2)

For the 5 independent RLVR runs from DS-Qwen-1.5B, Table 2 reports Jaccard overlap for the 13th attention block's projection layers. Across Q, K, V, O projections and MLP down/up/gate layers, the Jaccard overlap ranges from 0.552 (O) to 0.597 (V), consistently and substantially above the random Bernoulli baseline (0.373–0.467). For example, the Q projection overlap of 0.580 is 35% higher than its random baseline of 0.430. This confirms that RL updates share a common spatial footprint across runs trained on different datasets and algorithms, consistent with a model-conditioned (rather than data-conditioned or algorithm-conditioned) bias.

The consensus ratio visualization (Figure 2) for the same 13th block reveals stripe-like patterns rather than scattered noise. In the Q and V projections, contiguous row-wise bands show high consensus (many runs updating the same rows), while the O projection shows column-wise stripes. This structured routing pattern—entire rows or columns of weight matrices being consistently updated or consistently untouched—is visually striking and argued to be inconsistent with random or data-driven update placement. The paper emphasizes that this pattern is "not random scatter" and invites readers to "zoom in to see fine structure."

Temporal analysis (Figure 3) tracks row-wise and column-wise update ratios across training steps (t ∈ {240, 720, 1200}) for the 13th attention block. The row-wise ratios for Q projections and column-wise ratios for O projections show persistent peaks and troughs whose relative profiles remain aligned even as overall update density grows with training. The Q projection's row-dominant pattern and the O projection's column-dominant pattern are "consistent with the bias structures in Fig. 2." The column-wise ratios for Q and row-wise ratios for O show "a much weaker bias" (Figure 14, Appendix), confirming the pattern is structured and directional, not simply all rows or all columns updating equally. The paper notes the bias appears "not only across heads but also within heads," with head boundaries marked by grey dashed lines in Figure 3.

For cross-family generalization, the paper reports (Appendix, Figure 13) "similar stripe-structured footprints on Llama (Llama-3.1-8B) and Mistral (Mistral-Small-24B) models" using zero-RL checkpoints from Zeng et al. (2025b). No quantitative Jaccard or consensus metrics are reported for these models, but the visual evidence is presented as qualitative confirmation that the routing bias is "generic to RLVR."

Spectral Geometry Preservation: RLVR vs. SFT (Section 4.1)

Figure 4 (main text) presents the spectral comparison for Qwen3-8B-Base (SFT→RLVR pipeline). The top-left panel shows principal angle curves for an exemplar layer: RLVR's top-k singular subspace angles remain near zero (cosine near 1), while SFT's angles increase substantially for k ≥ 20, indicating rotation of the principal subspace. The top-right panel shows singular value curves: RLVR's curve nearly overlays the base model's curve, while SFT's curve shows visible distortion. The bottom-left panel aggregates normalized spectral drift across all layers, showing consistently lower values for RLVR than SFT. The bottom-right panel shows maximum principal angle across all layers, again with RLVR substantially lower.

Appendix Figure 16 replicates this analysis for DS-Qwen-1.5B (long-horizon RL vs. its SFT predecessor), finding the same pattern: "RLVR shows surprisingly stable top-k spectrum with minimal subspace rotation and top-k eigenvalue changes." Appendix Figure 17 extends to Qwen3-14B-Base, where SFT and RL are applied separately and matched on in-domain math performance (Huan et al., 2025), again confirming that RLVR preserves spectral structure while SFT distorts it. The consistency across three model families and training configurations (pipelined SFT→RL, long-horizon RL, separate SFT and RL) strengthens the claim that this is a general property of RLVR, not an artifact of a specific training setup.

No quantitative values are extracted from these figures in the text—the results are presented visually, with the paper stating qualitative conclusions: "RLVR preserves the pretrained spectral structure," "SFT induces substantially larger rotations and pronounced drifts on the same metrics," and "the singular-value profiles are even nearly identical to the base model." This is a limitation of the presentation—without tabulated spectral drift values or principal angle thresholds, the magnitude of the RL-SFT difference is difficult to assess precisely.

Update–Principal Misalignment (Section 4.2)

Figure 5 presents the layer-wise overlap between RL update masks and three reference masks for DS-Qwen-1.5B: M_princ (principal weights), M_low (low-magnitude weights), and M_princ ∩ M_low^c (principal weights that are not low-magnitude). All overlap ratios are plotted against a random baseline equal to the mask density α.

The results show a clear dichotomy. RL updates exhibit sub-random overlap with principal weights across all layers—bars fall consistently below the random baseline, indicating active avoidance. Conversely, RL updates show super-random overlap with low-magnitude weights—bars fall above the random baseline, indicating concentration. The residual overlap between RL updates and principal weights is substantially accounted for by the intersection M_princ ∩ M_low: when this intersection is excluded (M_princ ∩ M_low^c), the overlap "drops significantly." This means the small fraction of RL updates that do land on principal-weight positions are predominantly weights that are both principal (high rank-k reconstruction magnitude) and low-magnitude in the original matrix—a narrow subset.

The paper does not report exact overlap percentages, but the visual presentation in Figure 5 makes the qualitative pattern unambiguous. The finding directly supports Gate II's prediction that RL avoids high-curvature (principal) directions and favors low-resistance (low-magnitude) pathways, with the important nuance that the distinction between "principal" and "low-magnitude" masks is itself informative—many weights classified as principal by the rank-k SVD criterion turn out to have small raw magnitudes, and it is these weights (rather than large-magnitude principal weights) that account for most of the apparent principal-weight overlap.

Causal Intervention on Pretrained Geometry (Section 4.3)

Figure 6 presents the overlap ratio results from the geometric intervention experiment on Qwen3-4B-Base. Layer 20 receives both Rotate (function-preserving orthogonal rotation of V and O projections) and Permute (attention head permutation for all Q/K/V/O layers). Layer 25 receives only Rotate. The metric is Overlap(M_run1, M_run2), comparing the update mask from a baseline run with the update mask from an intervened run.

In the baseline comparison (no intervention, two independent runs), the overlap ratio is high—as expected from the cross-run stability results in Section 2.2. In the intervened layers (Layer 20, Layer 25), the overlap ratio collapses to the random level (the density baseline). In all untouched layers, the overlap remains high. The paper states this "provides strong causal evidence that the pretrained model's geometry is the source of the optimization bias." Because the interventions are function-preserving (Proposition D.1), the model's forward pass behavior is unchanged—any difference in update localization must stem from the parameter-space geometry, not from initial policy behavior or reward signals.

The experimental design is elegant: each model serves as its own control, with intervened and untouched layers within the same training run, eliminating confounds from different random seeds, data ordering, or optimization trajectories. However, the paper tests only 2 layers out of the full network, leaving open whether the effect generalizes across all layer types or is specific to the chosen attention projections.

Sparse Fine-Tuning Probe: Which Weights Can Be Frozen? (Section 5.1)

Figure 9 presents the token-wise forward KL divergence curves during RL training for DS-Qwen-1.5B under different sparse masks. The dense baseline curve shows steady KL growth from near zero to approximately 0.0125 at 300 steps. The mask performances stratify clearly:

  • M_princ (principal-only, 50% density): the KL curve rises slowest, reaching only ~0.0075 at 300 steps—about 60% of the dense KL growth—indicating severely impeded optimization when only principal weights can be updated.
  • M_princ^c (non-principal-only, 50% density): the KL curve rises faster, reaching ~0.0100–0.0110, substantially closer to dense.
  • M_low (low-magnitude-only, 58.59% density): the KL curve tracks close to dense, reaching ~0.0115.
  • M_low ∪ M_princ^c (safe mask, 74.02% density): the KL curve most closely tracks the dense baseline, nearly overlapping throughout training.
  • rand-M_low ∪ M_princ^c (random mask, same density as safe mask): the KL curve is noticeably lower than the safe mask, confirming that the specific choice of weights matters, not just the overall density.

Table 4 reports final evaluation accuracy at 320 steps. The dense baseline achieves an average of 57.44% across MATH500, AMC23, AIME24, and AIME25. The safe mask (M_low ∪ M_princ^c) achieves 56.24%—a gap of only 1.2 percentage points while using 74.02% of linear-layer parameters. The principal-only mask (M_princ) achieves 53.82%—a gap of 3.62 percentage points, the worst among all masks. The non-principal mask (M_princ^c) achieves 55.40%, and the low-magnitude mask (M_low) achieves 56.69%. The random density-matched mask achieves 55.34%, underperforming the safe mask by 0.9 points despite having the same number of trainable parameters.

Table 5 extends the experiment to 500 steps, with the dense baseline reaching 58.59% average accuracy. The pattern persists: the safe mask achieves 58.37% (0.22 point gap), while M_princ achieves 55.44% (3.15 point gap). The extended training does not close the gap between principal-only and other masks—if anything, the principal-only mask's relative disadvantage remains stable, suggesting the limitation is not merely slower learning but a fundamental bottleneck in optimization capacity when restricted to principal directions.

The paper highlights the practical implication: "Freezing principal and large-magnitude weights while updating non-principal, low-magnitude ones closely reproduces dense RLVR behavior (KL trajectory and final accuracy) using roughly 70% the parameters." The masks are "one-shot and fixed," constructed entirely from the pretrained model with no additional training.

LoRA vs. PiSSA: Principal-Aligned Adapters Destabilize RL (Section 5.2)

Figure 10 presents the learning rate sweep results for DS-Qwen-1.5B on DeepMath-103K, evaluating pass@1 (avg@16) on AIME24 (top row) and AMC23 (bottom row) across 200 training steps. Three learning rates are swept: 1×10^{−4}, 5×10^{−5}, and 1×10^{−5}, and three ranks: 8, 32, 64.

At lr=1×10^{−5} (the lowest learning rate): Across all ranks and both benchmarks, LoRA and PiSSA perform similarly, with curves largely overlapping. Both show gradual improvement over 200 steps, reaching roughly comparable plateau levels. The paper interprets this as expected—at low learning rates, the principal-direction bias of PiSSA's initialization is mild, and both adapters can learn effectively.

At lr=5×10^{−5}: PiSSA begins to show instability. On AIME24 with rank 64, PiSSA's curve initially tracks LoRA but then plateaus earlier and at a lower accuracy than LoRA's continued improvement. On AMC23 with rank 32, PiSSA shows a visible degradation after approximately 100 steps, while LoRA continues improving.

At lr=1×10^{−4} (the highest learning rate): PiSSA frequently collapses early in training. On AIME24, PiSSA with rank 8 and rank 32 shows an initial rise followed by a sharp decline after approximately 50 steps, falling well below LoRA and often below the starting accuracy. On AMC23, PiSSA with rank 64 shows a steep early drop. LoRA, while also sometimes unstable at this high learning rate (particularly rank 8 on AIME24), consistently maintains higher accuracy and recovers more reliably than PiSSA. The paper interprets this as confirming the geometric prediction: "Scaling the learning rate in PiSSA enforces updates along principal directions, higher-curvature and spectrum-distorting, precisely the directions RLVR tends to avoid. The result is brittle optimization and early collapse."

Figure 11 replicates the comparison on LLaMA-3.2-3B-Instruct with a fixed rank of 64, sweeping the same three learning rates (plus a full fine-tuning baseline at lr=1×10^{−6}) and evaluating pass@1 (mean@4) on MATH500. The pattern is consistent with Figure 10: at lr=1×10^{−5}, LoRA and PiSSA converge to similar accuracies (~0.42–0.44). At lr=5×10^{−5}, PiSSA underperforms LoRA by a visible margin throughout training. At lr=1×10^{−4}, PiSSA collapses early (accuracy drops sharply after ~50 steps), while LoRA reaches approximately 0.50 accuracy, approaching the full fine-tuning baseline (~0.52). The cross-model replication (Qwen vs. LLaMA) strengthens the claim that this is not an architecture-specific artifact.

The paper summarizes the finding: "Across settings, the principal-targeted PiSSA provides no clear gain over LoRA. At the higher learning rates used for low-rank adapters to match full-parameter performance, PiSSA often becomes unstable and collapses earlier than LoRA."

Generalization to Agent Tasks and RLHF (Section 4.4)

Figure 8 presents spectral geometry diagnostics for RL-trained agent checkpoints. The left panel shows singular value curves for representative layers (second block O-projection) across agent models including agentflow-planner-7b, SkyRL-Agent-WebResearch-8B, VT-deepsearch-8B, VT-SWE-8B, and GiGPO checkpoints for WebShop and ALFWorld. In all cases, the RL-trained curves nearly overlay the base model curves, showing minimal spectral distortion. The right panel shows maximum principal angles: all agent models exhibit small rotations (< 0.2 radians for most layers), consistent with the spectrum-preserving RLVR regime.

Figure 18 (Appendix) extends this to RLHF (DPO and SimPO on Llama-3-Instruct-8B), showing the same pattern: "RL training preserves layer spectra and induces only minor rotation of the top-k subspaces, consistent with the RLVR regime." No quantitative values are reported, but the visual evidence across 6 agent checkpoints and 2 RLHF checkpoints is presented as confirmation that the geometric signatures persist beyond verifiable math/code tasks.

Figure 7 provides a qualitative visualization of update–principal misalignment for two representative agent layers: agentflow-planner-7b (Layer 16, o_proj) and SkyRL-Agent-WebResearch-8B (Layer 11, k_proj). The left panels show the bf16-aware update mask (white pixels indicate weights that changed under RL), and the right panels show the principal mask (white pixels indicate principal-weight positions). Dashed red boxes highlight "stripe regions where RL updates concentrate outside principal-weight bands," visually confirming that the off-principal routing observed in math models generalizes to multi-turn agent and tool-use settings. The paper states this indicates "robust off-principal routing in agent and tool-use settings," though no quantitative overlap metrics are reported for these models.

Ablation Studies and Robustness Checks

  • bfloat16 probe validity (Appendix E.2): Lemma E.1 establishes that for distinct bfloat16 numbers in the same binade, |x−y| / max(|x|,|y|) > 2^{−8}. Lemma E.2 shows that ULP_bf16(x) / |x| ∈ (2^{−8}, 2^{−7}], meaning the relative threshold for representability is between 0.39% and 0.78%. Proposition E.3 proves that choosing η = 10^{−3} < 2^{−9} makes the test equivalent to bitwise equality. Corollary E.4 confirms η = 10^{−3} is a safe choice. This formal justification is critical because the entire sparsity measurement framework (Table 1, all update masks) depends on this probe's correctness.

  • Fixed absolute-tolerance probe failures (Appendix E.1): The paper demonstrates two failure modes of the 10^{−5} absolute-tolerance approach used by Mukherjee et al. (2025). Around magnitude 1024 (bf16 binade [2^{10}, 2^{11})), the bf16 ULP is 8, so numbers differing by 10^{−3} are flagged as "changed" by a 10^{−5} rule but round to the same bf16 value (false positive). Around magnitude 10^{−6} (bf16 binade [2^{−20}, 2^{−19})), the bf16 ULP is 7.45×10^{−9}, so weights differing by 10^{−6} would be marked "equal" by a 10^{−5} rule yet are separated by ~134 ULPs (false negative). This demonstrates why the bfloat16-aware relative probe is necessary for accurate sparsity measurement.

  • Learning rate scaling dissolves apparent sparsity (Section 2.3): The paper reports an experiment where increasing the learning rate pushes sub-ULP updates above the representable threshold. The result: "the apparent update sparsity largely disappears." No figure or table is provided for this specific experiment, but the finding is cited as evidence that sparsity is a precision artifact, not a reflection of zero gradients. The paper also references concurrent work by Shenfeld et al. (2025) showing that "sparsity mostly vanishes under float32 storage... even though task performance does not improve," reinforcing the point without replicating the float32 experiment.

  • Temporal stability of update patterns (Section 2.2, Figure 3): The row-wise update ratios ρ_{ℓ,i}(t) and column-wise ratios κ_{ℓ,j}(t) are tracked at steps 240, 720, and 1200 for the 13th attention block, smoothed with a 3-step window. The relative profiles (positions of peaks and troughs) remain aligned while overall density grows, confirming the bias "emerges early and is reinforced over training, indicating a temporally stable phenomenon rather than a transient artifact." The column-wise ratios for Q and row-wise ratios for O (Figure 14, Appendix) show "a much weaker bias," confirming the directionality is structured (e.g., Q updates concentrate in specific rows, O updates in specific columns), not uniform across both dimensions.

  • Mask density control for sparse fine-tuning (Section 5.1, Figure 9 and Table 4): The random mask (rand-M_low ∪ M_princ^c) with the same layer-wise sparsity as the safe mask (74.02%) serves as a critical control. It achieves 55.34% average accuracy at 320 steps (Table 4) and 56.89% at 500 steps (Table 5), compared to the safe mask's 56.24% and 58.37% respectively. The gap of 0.9–1.48 points confirms that the safe mask's superior performance is not merely a density effect—the specific choice of which weights to freeze matters. The paper does not ablate the density itself (e.g., 50% vs. 75% vs. 90% for the safe mask), leaving open the question of whether denser safe masks would close the remaining gap to dense performance.

  • Cross-model replication of PiSSA instability (Section 5.2, Figures 10 and 11): The PiSSA vs. LoRA comparison is replicated on two model families (DS-Qwen-1.5B and LLaMA-3.2-3B-Instruct) with different training corpora (DeepMath-103K vs. a Math corpus) and evaluation benchmarks (AIME24/AMC23 vs. MATH500). The consistent pattern—PiSSA provides no gain and often collapses at high learning rates—across both settings strengthens the claim that this is a geometric phenomenon, not a model-specific or data-specific artifact. However, the paper tests only rank 64 for LLaMA-3.2-3B, so the rank dependence is not replicated across models.

  • Agent and RLHF generalization (Section 4.4, Figures 7, 8, 18): The spectral diagnostics (principal angles, singular value preservation) are extended to 6 agent checkpoints and 2 RLHF checkpoints, showing the same qualitative pattern as math/code models. However, the paper does not report consensus ratios, Jaccard overlaps, or update-principal overlap metrics for these checkpoints—only visualizations (Figure 7) for two representative layers. This limits the quantitative strength of the generalization claim, as the possibility remains that the off-principal routing is less pronounced or more variable in non-math domains.

  • Explicit vs. implicit KL penalty (Appendix F.2.4, Figure 15): The paper uses DAPO (β = 0, ratio clipping only) for its primary experiments to "eliminate the confounding effect of the KL penalty." Appendix Figure 15 shows the token-wise KL loss during a DAPO run without explicit KL penalty, demonstrating that the KL divergence "shows a steadily increasing KL loss instead of being unconstrained." This confirms that ratio clipping alone imposes an implicit KL leash (Lemma F.9, bounding KL by O(ε^2) in the small-step regime), validating that Gate I applies even without an explicit KL term in the objective.

  • Extended training for sparse fine-tuning (Table 5 vs. Table 4): Extending training from 320 to 500 steps does not qualitatively change the mask performance ordering. The principal-only mask's gap to dense remains approximately 3 percentage points (53.82% vs. 57.44% at 320 steps; 55.44% vs. 58.59% at 500 steps). The safe mask's gap remains small (56.24% vs. 57.44% at 320 steps; 58.37% vs. 58.59% at 500 steps). This suggests the principal-only limitation is not merely slower learning but a fundamental capacity constraint, consistent with the geometric theory's prediction that principal directions are poor optimization pathways for RL.

Critical Assessment

Does the Evidence Support the Claim that RLVR Exhibits a Model-Conditioned Optimization Bias?

The evidence for a model-conditioned bias (as opposed to a data-conditioned or algorithm-conditioned one) is strongest in the cross-run stability analysis (Section 2.2). Five independent RLVR runs from the same base model but with different datasets and RL algorithms show high Jaccard overlap (Table 2) and stripe-like consensus patterns (Figure 2). This is genuinely compelling: if the bias were driven by data, different datasets should route updates differently; if driven by algorithm, GRPO and Reinforcement++ should produce different patterns. The temporal stability result (Figure 3) further supports this by showing the pattern emerges early and is reinforced rather than shifting as training progresses.

However, the paper's claim that the bias is "largely invariant to datasets and RL recipes" is demonstrated only for DS-Qwen-1.5B. Table 1 shows that sparsity varies by ~17 percentage points across different RL runs from the same base model (36.3% for NV-ProRL-v2 vs. 53.8% for DeepScaleR-1.5B-Preview), which could indicate some data/algorithm sensitivity in the magnitude of the bias, even if the spatial pattern remains consistent. The paper does not report cross-run Jaccard overlaps for models other than DS-Qwen-1.5B, so the generality of the consensus finding is untested. The visual evidence of "similar stripe-structured footprints on Llama and Mistral" (Appendix, Figure 13) is suggestive but qualitative—without quantitative overlap metrics for these models, the strength of the model-conditioned claim across architectures cannot be rigorously assessed.

A more fundamental limitation: the paper demonstrates that update patterns are consistent within a pretrained model but does not systematically characterize how patterns differ between models. If the bias is truly model-conditioned, then Qwen, Llama, and Mistral should show different routing patterns (reflecting their different pretrained geometries), and these differences should be predictable from geometric properties. The paper's cross-model analysis is limited to showing that all models exhibit some form of structured routing, not that the routing is predictably model-specific.

Does the Spectral Geometry Evidence Genuinely Distinguish RL from SFT?

The spectral comparisons in Section 4.1 (Figures 4, 16, 17) are visually clear and consistent across three model families. However, the paper reports these results entirely qualitatively—no tabulated values for normalized spectral shift or maximum principal angle, and no statistical quantification of the RL-SFT difference. This makes it difficult to assess effect sizes: is SFT's spectral drift 2× or 10× larger than RL's? Are there layers where the difference is negligible? The paper's claim that RLVR "preserves the pretrained spectral structure" would be stronger with a quantitative threshold (e.g., "RLVR's maximum principal angle across all layers is below X degrees for all models tested, compared to SFT's Y degrees").

A subtler issue: the SFT and RL checkpoints being compared differ not only in training algorithm but also in the amount and nature of training data. For the Qwen3-8B pipeline, the SFT checkpoint presumably receives substantial supervised training before RL, so the RL checkpoint has seen more total updates. The spectral comparison should ideally control for total optimization steps or for final task performance, since a larger-magnitude training process might naturally distort the spectrum more regardless of algorithm. The Qwen3-14B comparison (Figure 17), where SFT and RL are matched on in-domain math performance, partially addresses this concern, but even here the number of gradient steps likely differs between the two methods.

Is the Causal Intervention Experiment Sufficient to Establish Geometry as the Steering Mechanism?

The geometric intervention experiment (Section 4.3, Figure 6) is methodologically the paper's strongest claim to causal evidence. The function-preserving rotations and permutations are elegant because they maintain exact output equivalence. The result—overlap collapses in intervened layers but not in untouched layers—cleanly isolates geometry.

However, the experiment has several limitations. First, only 2 layers out of the full network are intervened upon, and both are attention projection layers. It remains possible that geometry is causal for attention projections but not for MLP layers, embedding layers, or other components. Second, the intervention changes the geometry in a specific way (orthogonal rotation, head permutation) that may not represent all aspects of "geometry" that could steer updates. For example, the singular value spectrum is preserved under orthogonal rotation (singular values are invariant to orthogonal transformations), so the intervention tests whether the specific orientation of principal directions matters, not whether the spectrum matters. Third, the experiment uses only Qwen3-4B, leaving cross-model replication open.

An additional experiment that would strengthen the causal claim: applying RLVR to a randomly initialized model (no pretrained geometry) and showing that the bias does not emerge. This would directly test the paper's assertion that the bias "arises from the pretrained landscape rather than particular datasets or RL recipes." The paper does not run this experiment.

Do the Sparse Fine-Tuning Results Prove That RL and SFT Operate in Disjoint Regimes?

The sparse fine-tuning probe (Section 5.1) convincingly shows that principal-targeted sparse masks perform poorly for RL, while off-principal masks perform well. However, the paper frames this as evidence that "RLVR and SFT operate in distinct optimization regions of parameter space," which requires comparing RL sparse fine-tuning with SFT sparse fine-tuning under the same masks. The paper does not run SFT with these masks; it cites Liu et al. (2025c) for the SFT finding that principal weights are important. This is a reasonable appeal to prior work but means the direct RL-vs-SFT comparison under identical sparse masks is not demonstrated within this paper. A within-paper replication—showing that M_princ performs best for SFT while performing worst for RL on the same model and data—would substantially strengthen the claim of disjoint regimes.

The safe mask result (74.02% of parameters achieving near-dense performance) is practically significant but leaves open the question of whether even sparser masks could work. The paper does not sweep mask densities—all masks are at fixed densities (50% for M_princ/M_princ^c, 58.59% for M_low, 74.02% for the safe mask). The "safe mask" might be considerably denser than necessary, and a principled method for determining the minimum required density per layer would be a natural extension that the paper does not pursue.

Do the LoRA/PiSSA Results Generalize Beyond the Swept Hyperparameters?

The LoRA/PiSSA comparison (Figures 10, 11) tests a reasonable range of ranks ({8, 32, 64}) and learning rates ({1×10^{-5}, 5×10^{-5}, 1×10^{-4}}) on two model families. However, several confounds are not controlled:

  • Adapter placement: The paper does not specify which layers receive LoRA/PiSSA adapters. If adapters are applied only to attention projections (Q, K, V, O), the results may not generalize to adapter configurations that include MLP layers.
  • Training duration: All experiments run for 200 steps. RLVR training in practice runs for thousands of steps (DS-Qwen-1.5B in Table 1 was trained for over 3,000 steps). The collapse behavior of PiSSA might be a transient phenomenon that recovers with longer training, or conversely, the instability might compound and worsen. The 200-step window may not capture long-run behavior.
  • The rank-64, lr=1×10^{-4}, LLaMA-3.2-3B result (Figure 11): PiSSA collapses early, but LoRA reaches ~0.50 accuracy, approaching the full fine-tuning baseline (~0.52). This is presented as evidence for LoRA's superiority, but it also shows that LoRA itself is somewhat unstable at this learning rate relative to full fine-tuning (which uses lr=1×10^{-6}). The paper does not discuss the trade-off between adapter capacity, learning rate, and stability for LoRA itself.

The paper's claim that "standard LoRA, with its random initialization and frozen base weights, naturally approximates off-principal updates" is an interpretation consistent with the results but not directly tested. An experiment that initialized LoRA with random directions orthogonal to principal directions and compared it to standard LoRA would provide more direct evidence for the geometric mechanism.

Are the Agent and RLHF Generalization Results Sufficiently Quantified?

Section 4.4 extends the parameter-space diagnostics to agents and RLHF, which is important for establishing generality beyond math reasoning. However, the analysis is substantially thinner than the math-model analysis:

  • No sparsity values: Table 3 lists the checkpoints but does not report sparsity_bf16 for them. The reader cannot assess whether agent and RLHF models exhibit the same magnitude of apparent sparsity as math models.
  • No overlap metrics: Figure 7 shows qualitative update masks for two layers, but no overlap ratios, Jaccard similarities, or consensus ratios are reported. We cannot quantify how strongly the off-principal routing holds in these domains.
  • Spectral analysis is visual only: Figures 8 and 18 show singular value curves and principal angles, but without numerical values, the strength of the effect cannot be compared quantitatively to the math models in Figure 4.

This asymmetry in analysis depth between the primary (math) and generalization (agent, RLHF) results weakens the universality claim. The paper interprets the visual evidence as showing the same "optimization signature," but a skeptical reader could argue that the agent/RLHF results are cherry-picked layers that show the desired pattern, while other layers might behave differently.

Missing Experiments That Would Strengthen the Paper

  1. Float32 comparison throughout: The paper acknowledges (Section 2.3, citing Shenfeld et al., 2025) that sparsity largely vanishes under float32. An experiment running the full diagnostic suite (spectral analysis, update-principal overlap, cross-run stability) under float32 would show whether the underlying bias (stripes, off-principal routing) persists even when precision doesn't hide micro-updates, or whether the bias itself is partly a precision artifact.

  2. SFT with the same sparse masks on the same model: To directly demonstrate the RL-SFT regime distinction within the paper, running SFT with the same mask suite (M_princ, M_princ^c, M_low, safe mask) on DS-Qwen-1.5B and showing the opposite performance ordering (M_princ best for SFT) would be a clean, self-contained demonstration.

  3. Random-initialization baseline for the geometric bias: Training a randomly initialized model with RLVR and checking whether update patterns are structured (they shouldn't be, according to Gate II) would test the "pretrained geometry" claim more directly than the intervention experiment.

  4. Density sweeps for sparse fine-tuning: Testing whether the safe mask can be made sparser (e.g., 50%, 60%, 70%) without performance degradation would characterize the pareto frontier of parameter efficiency for RL-aware sparse training, and would be practically valuable.

  5. Learning rate scaling for sparse fine-tuning: The LoRA/PiSSA experiment shows learning rate sensitivity. The sparse fine-tuning experiment uses a single learning rate (1×10^{-6}). Testing whether the performance gap between M_princ and the safe mask narrows or widens at higher learning rates would test the paper's claim that principal directions are high-curvature (where larger steps cause larger KL changes, potentially destabilizing training).

  6. Layer-type breakdown: The paper focuses on attention projection layers (Q, K, V, O) and MLP layers. It does not systematically characterize whether the optimization bias differs across layer types (early vs. late layers, attention vs. MLP), which would provide a more complete parameter-space account.

  7. Larger-scale models: The largest model analyzed is Qwen3-30B-A3B (Table 1), but the detailed diagnostics (Jaccard, consensus, spectral analysis) are reported only for 1.5B–8B models. Whether the geometric bias persists at the 70B+ scale (where RLVR is most commonly applied in practice) is untested.

Overall Assessment

The paper's central empirical claims are well-supported within their tested scope: RLVR reliably produces localized, structured update patterns that avoid principal weights and preserve spectral geometry, and this pattern is consistent with a model-conditioned geometric bias rather than being driven by data or algorithm choice. The causal intervention experiment provides genuine evidence for geometry as a steering mechanism. The practical demonstration that SFT-aligned PEFT underperforms for RL is timely and actionable.

The primary weaknesses are: (1) heavy reliance on qualitative/visual presentation where tabulated quantitative metrics would strengthen claims, particularly for spectral analysis and generalization experiments; (2) limited analysis depth for agent and RLHF domains compared to math models; (3) the causal intervention is tested on only 2 layers of 1 model; (4) the sparse fine-tuning mask densities are not swept, leaving the efficiency limits unexplored; and (5) the absence of a randomly initialized baseline makes the "pretrained geometry" claim less directly tested than it could be. The paper's contributions are nonetheless substantial—the diagnostic toolkit, the geometric theory, and the cautionary PEFT findings together advance understanding of RLVR's training dynamics beyond the prior state of the art, even if some supporting evidence is more suggestive than conclusive.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Unaccounted for Yet Completely Dominates the Inference Budget

The paper’s compute-optimal framework requires estimating the difficulty of every prompt before allocating test-time compute. The method for doing so — generating 2,048 samples per question and either scoring them against ground truth (oracle bins) or averaging the PRM’s final-answer score (predicted bins) — is extraordinarily expensive. The authors acknowledge this directly: “estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity” (Section 3.2). What makes this consequential is the absolute scale: 2,048 generations is 4–8× larger than the largest test-time budgets studied (256–512 generations). Amortizing this cost over a single question means the total compute is dominated not by the strategy execution the paper optimizes, but by the preliminary difficulty assessment it treats as free.

The consequence is that the headline 4× efficiency gains (e.g., matching best-of-N performance with 16 generations instead of 64 in Figure 4; matching parallel best-of-256 with 64 generations of compute-optimal revisions in Figure 8) are computed after difficulty is already known. If a practitioner must spend 2,048 generations to determine that a problem is “easy” and then spend 16 generations on the optimal strategy, the actual cost is 2,064 generations — far worse than simply running best-of-256 (which costs 256 and requires no difficulty estimation). In a deployment context where each question is encountered once, this makes the compute-optimal framework strictly more expensive than the baseline it claims to outperform.

The paper partially mitigates this by showing that predicted difficulty bins (using the PRM’s average score without ground truth) track oracle bins closely (Figures 4 and 8), but the prediction still requires the same 2,048 samples. The authors flag this as “a key avenue for future work” (Section 3.2), suggesting pretraining or fine-tuning models to predict difficulty directly from question text. No such model is developed or evaluated. Until cheap difficulty estimation exists — either through a lightweight classifier, adaptive estimation that starts with few samples and adjusts, or amortization across many similar problems — the compute-optimal framework is primarily an analytical tool, not a practical deployment strategy. The paper’s efficiency claims should be understood as upper bounds conditional on free difficulty oracles.

Hard Problems Remain Unsolved — Test-Time Compute Cannot Substitute for Capability Gaps

The paper is honest that the hardest questions (difficulty bin 5) show essentially zero improvement from any test-time compute strategy, regardless of budget or method. In Figure 3 (right), bin 5 accuracy for both beam search and best-of-N weighted hovers at 1–3% across all budgets from 4 to 256 generations. In Figure 7 (right), bin 5 accuracy under revisions at a fixed 128-generation budget is roughly 2–3% for all sequential-to-parallel ratios. In the FLOPs-matched comparison (Figure 9), the bin 5 scaling line is essentially flat near 0–5%, substantially below the ~14× larger model’s greedy performance (which itself is modest but non-zero). The authors state this clearly: “the base model simply lacks the capability to produce correct solutions regardless of how the budget is allocated” (Section 5.3) and “test-time compute amplifies existing capability but does not create it” (Section 7).

The consequence is a hard boundary on when compute-optimal test-time scaling can substitute for pretraining. The FLOPs-matched analysis in Section 7 shows that for easy-to-medium problems at low inference-to-pretraining ratios (R ≪ 1), the smaller model with test-time compute can substantially outperform a ~14× larger model. But for hard problems (bins 4–5), the smaller model with test-time compute shows a disadvantage of up to −52.9% (PRM search, R ≫ 1) compared to simply training larger. The paper’s framing — “test-time and pretraining compute are not 1-to-1 exchangeable” — acknowledges that test-time compute address a fundamentally different axis of capability than pretraining scale. For problems where the base model’s pass@1 is near zero, no amount of search or revision will help because there are no correct solutions in the proposal distribution to find or refine.

The paper does not attempt to mitigate this — it is a fundamental limitation of the approach, not a weakness in execution. The practical implication for deployment decisions is clear: if the problem distribution includes a non-trivial fraction of genuinely hard problems (those substantially outside the base model’s training distribution or requiring capabilities the model has not acquired), compute-optimal test-time scaling provides no path forward. Pretraining remains the only viable strategy for those cases. This scope limitation is well-documented but should temper enthusiasm for test-time compute as a universal substitute for model scale.

The Results Come from a Single Benchmark (MATH) and a Single Model Family (PaLM 2-S*), Limiting Generality Claims

All experiments in the paper — the search algorithm comparisons, revision model analysis, FLOPs-matched comparisons, and compute-optimal strategy selection — are conducted on the MATH benchmark (Hendrycks et al., 2021) using PaLM 2-S* (Anil et al., 2023). The MATH dataset consists exclusively of high-school competition-level mathematics problems requiring symbolic reasoning and producing closed-form answers. The authors argue this choice is deliberate — “test-time compute is expected to help most when the model already possesses the necessary knowledge and the challenge is drawing complex inferences” (Section 4) — but this means the paper provides no evidence about how the difficulty-dependent scaling patterns generalize to other reasoning domains (code generation, logical reasoning, scientific QA, multi-step planning), to tasks requiring factual recall rather than inference, or to open-ended generation where correctness is ambiguous.

The consequence is that a practitioner cannot assume the 4× efficiency gains, the difficulty-dependent strategy ordering (beam search good for medium problems, bad for easy ones; sequential revisions good for easy problems, balanced ratio for hard ones), or the compute-optimal policy lookups (which strategy to use at which difficulty-budget combination) will transfer. Code generation, for example, has different structural properties — solutions can fail partially (a unit test passes 3/5 cases), intermediate states are more interpretable, and verification via execution provides a different signal than answer matching. The PRM training procedure (Monte Carlo rollout supervision) depends on having clean per-step correctness labels, which are natural for math (each step is a logical deduction) but less natural for other domains. The revision model training procedure (edit-distance-based pairing of incorrect and correct solutions) may not produce coherent revision trajectories in tasks where “closeness” in edit distance does not correspond to “targeted correction.”

The paper acknowledges this limitation implicitly — all experimental design choices are justified for MATH, but no cross-domain experiments are proposed or run. Neither the search nor revision results are replicated on a second benchmark. The FLOPs-matched comparison (Section 7) is particularly sensitive to this: the finding that test-time compute can outperform a ~14× larger model on easy-to-medium problems at low R may be specific to math reasoning, where the base model’s knowledge is largely present and the challenge is procedural. In a knowledge-intensive domain like closed-book QA, the base model may lack the relevant facts entirely, and no amount of chain-of-thought search will recover them — a different failure mode than the “hard problem” limitation discussed above, but one the MATH benchmark cannot reveal. The paper’s contributions are foundational for understanding test-time compute scaling in math reasoning, but the generality of the framework to other domains remains an open question requiring separate investigation.

The Revision Model’s Correct-to-Incorrect Reversion Rate (~38%) Is a Significant Unresolved Reliability Problem

The paper reports that “approximately 38% of correct answers get converted back to incorrect ones” during sequential revision chains (Section 6.1). This is a direct consequence of the training data construction: the model is only trained on sequences where all in-context answers are incorrect and the final target is correct. The model never sees examples where the current answer is already correct and should be preserved, so at test time, when a revision chain produces a correct answer, the model has no learned behavior for “recognize correctness and stop” — it applies its trained transformation (modify toward correctness) and inadvertently corrupts a good solution.

The consequence is that sequential revision chains are inherently unreliable at the per-step level. The paper mitigates this with within-chain selection — using majority voting or verifier-based selection to pick the best answer from any point in the chain rather than always taking the last revision (Section 6.1). This is a patch, not a solution: it requires generating the entire chain and then discarding most of it, wasting the compute spent on post-correctness revisions. It also means that the optimal revision chain length cannot be known in advance — generating more revisions than needed risks corrupting good answers, while generating too few risks missing the point where the model would have converged. The sequential-to-parallel ratio analysis (Figure 7) operates at the level of budget allocation across chains, but the within-chain length (how many revisions to generate before stopping) is not optimized — the paper simply generates chains of fixed length and selects the best point post hoc.

The ReST^EM experiment (Appendix K, Figure 16) further reveals fragility: an attempt to further optimize the revision model using RL-style training caused “additional sequential revisions to substantially hurt performance,” with the authors hypothesizing that “on-policy data collection in ReST^EM exacerbates spurious correlations in revision data.” This suggests the revision training procedure is brittle — the positive results depend on specific offline data construction choices (independent sampling, edit-distance pairing) that may not transfer to other settings or scale well with additional optimization.

The paper does not propose a solution to the reversion problem beyond within-chain selection, nor does it analyze how the reversion rate scales with chain length or problem difficulty. A principled fix — training the model with “no revision needed” examples, adding a stopping criterion, or using the verifier to dynamically decide when to stop revising — is left to future work. For practitioners, this means revision chains are a useful but unreliable tool: they improve pass@1 on average (Figure 6, left) but introduce a failure mode (correct→incorrect reversion) that parallel sampling does not have. The compute-optimal allocation policy in Figure 8 accounts for this implicitly by selecting the best ratio per difficulty bin, but it does not characterize or bound the reliability cost.

The Computational Cost of Difficulty Estimation Is Not Amortized, and Dynamic Adaptation Is Not Explored

This limitation extends the first one (unaccounted difficulty estimation cost) by focusing on the structural problem with the paper’s allocation framework rather than just the numerical magnitude of the cost. The paper treats difficulty estimation as a preprocessing step: estimate the question’s difficulty once, then look up the optimal strategy, then execute it. This is a static, two-phase approach that wastes information. The 2,048 samples generated during difficulty estimation produce candidate solutions and verifier scores that are then discarded — they are not used as part of the solution process. Similarly, the strategy is committed to before any actual problem-solving begins; there is no mechanism for adjusting the strategy mid-computation based on how the initial attempts are going.

The consequence is that the compute-optimal framework misses an entire class of potentially more efficient allocation strategies: dynamic policies that interleave estimation and optimization. For example, a system could start by generating 4 parallel samples, check the verifier’s score distribution and the degree of consensus among those samples, and use that signal to decide whether to continue with best-of-N (if scores are high and consistent, suggesting an easy problem), switch to beam search (if scores are middling, suggesting a medium problem where exploration would help), or stop and escalate to a larger model (if scores are near zero, suggesting the problem is outside the base model’s capability). This adaptive approach amortizes the difficulty estimation cost into the solution process itself — the samples used for estimation also contribute to the final answer pool. It also avoids the worst case where 2,048 samples are spent to determine that a problem is bin 5 and essentially unsolvable — an adaptive policy could detect this after a much smaller number of samples and redirect compute elsewhere.

The paper acknowledges the general idea in Section 3.2, framing the difficulty estimation cost as “an exploration-exploitation tradeoff — compute spent assessing difficulty versus compute spent solving the problem,” and flagging it as “a key avenue for future work.” But no dynamic policy is developed or evaluated. The static two-phase approach is used throughout all experiments, and all efficiency comparisons assume difficulty is known in advance. This is a substantial gap between the analytical framework and practical deployment, and it means the paper’s results are better understood as analyzing the structure of the test-time compute allocation problem rather than providing a directly deployable solution.

The FLOPs-Matched Comparison Is Biased in Favor of Test-Time Compute by Using a Suboptimally Trained Baseline and Denying the Larger Model Test-Time Compute

Section 7’s central empirical claim — that a smaller model with test-time compute can outperform a ~14× larger model — relies on a specific comparison that advantages the test-time compute approach in several ways the paper acknowledges only partially. First, the ~14× larger model is scaled in parameters only while holding training data fixed, following the LLaMA paradigm (Touvron et al., 2023) rather than compute-optimal pretraining where both parameters and data are scaled equally (Hoffmann et al., 2022). The paper states: “We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work.” This matters because a Chinchilla-optimal model trained with 14× more total FLOPs would likely outperform a parameter-only-scaled model, making the pretraining baseline weaker than it could be. The magnitude of this effect is unknown — the paper provides no comparison against a compute-optimally trained larger model.

Second, and perhaps more consequentially, the ~14× larger model uses greedy decoding only — no majority voting, no best-of-N, no verifier-guided search, no revision chains. The test-time compute approach gets the full power of the compute-optimal framework (difficulty-conditioned strategy selection, PRM-guided beam search, sequential revisions, best-of-N weighted selection), while the larger model gets none of these. This is an asymmetric comparison: the paper is evaluating “small model + sophisticated inference” versus “large model + naive inference.” A fairer comparison would give the larger model some test-time compute budget — even a modest best-of-4 or best-of-8 — and measure whether the small model’s advantage persists. The paper’s FLOP accounting (Equations in Section 7) accounts for the per-token inference cost of the larger model, but the type of inference (greedy vs. search-augmented) is fundamentally different between the two conditions.

The consequence is that the reported advantages of test-time compute over pretraining — e.g., +27.8% on easy-to-medium questions for revisions at R ≪ 1 — are best understood as upper bounds on the substitution potential. Against a compute-optimally trained larger model with even a modest test-time compute budget, these advantages would shrink or potentially reverse. The paper’s own difficulty-bin analysis shows that the larger model’s greedy performance on easy questions is already quite high (Figure 9, purple line for revisions), and adding best-of-N or majority voting to the larger model would likely push its performance further into the regime where the smaller model cannot catch up, even with compute-optimal test-time allocation.

The paper is transparent about the parameter-only scaling choice but does not discuss the asymmetry of giving test-time compute only to the smaller model. For a practitioner deciding between training larger or investing in inference infrastructure, this comparison does not answer the relevant question: “Given a fixed total budget that I can split between training a larger model and running smarter inference on whichever model I choose, what is the optimal allocation?” Answering that would require giving both models some inference budget and optimizing jointly over model scale and inference strategy — a substantially harder experiment that the paper does not attempt.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper effects a reframing of RLVR sparsity from a curiosity to a diagnostic, and in doing so provides the first parameter-level mechanistic account of how RL post-training modifies language model weights. Before this work, the field had a paradox—expensive RL training produces sparse updates while cheap SFT produces dense ones—but no framework for understanding why or where updates land. The paper's central move is to separate the signal (a model-conditioned optimization bias routing updates toward off-principal, low-curvature directions) from the sensor (bfloat16 precision filtering sub-ULP micro-updates in non-preferred regions). This is not merely a measurement improvement; it reorients the research question from "why is RL sparse?" to "what geometric properties of the pretrained model determine where RL updates concentrate?"

The magnitude of this contribution is substantial but foundational rather than paradigm-shifting. It does not propose a new training algorithm, a new architecture, or a new scaling law. Instead, it builds a diagnostic toolkit—principal masks, spectral drift metrics, update-principal overlap ratios, consensus maps, causal geometric interventions—that converts RLVR's weight-space dynamics from a black box into a measurable, predictable phenomenon. This toolkit is reusable: future work on new models, new RL algorithms, or new task domains can apply the same measurements without reinventing the apparatus. The paper's analogy to the Chinchilla scaling laws (Hoffmann et al., 2022) is apt not because it provides an inference-time scaling law, but because it provides the conceptual foundation upon which such laws could be built: just as Chinchilla established the right axes (model size, data quantity) and measurement methodology for pretraining scaling, this paper establishes the right axes (principal subspace alignment, spectral preservation, update localization) and measurement methodology for post-training dynamics.

The paper resolves a specific contradiction in the literature. Mukherjee et al. (2025) reported RL sparsity as a primary phenomenon, while concurrent work by Shenfeld et al. (2025) showed that sparsity largely vanishes under float32 storage. These observations appeared contradictory: is RL sparse or not? The paper's Three-Gate Theory reconciles them. RL is biased toward updating specific parameter regions (Gate II: Model Geometry), but the visibility of this bias as sparsity depends on numerical precision (Gate III). Under bfloat16, micro-updates in non-preferred regions fall below the representable threshold and the bias appears as binary sparsity. Under float32, those same micro-updates become visible and the bias appears as a gradient of update magnitudes—large in preferred regions, small elsewhere—rather than a binary mask. The underlying geometric steering is the same; the storage format merely changes how it is read out. This resolution converts a confusing set of contradictory empirical claims into a coherent picture with clear boundary conditions.

The paper also redirects research priorities in concrete ways. First, it demonstrates that verifier over-optimization is not the only geometric bottleneck in RLVR—the pretrained model's own curvature structure actively constrains which parameter updates are feasible under the KL leash, independent of reward signal quality. This means that understanding and potentially modifying pretrained geometry (through architecture design, initialization schemes, or pre-RL interventions) becomes a lever for improving RLVR efficiency, a direction that prior work focused on reward design and algorithm choice had not considered. Second, it shows that SFT-era PEFT methods should not be naively ported to RLVR—the geometric misalignment between principal-targeted adapters (designed for SFT) and RLVR's off-principal dynamics means that an entire class of methods must be rethought, not merely re-tuned. This is a negative result with positive practical consequences: it prevents wasted effort on approaches that are geometrically doomed and channels research toward RL-native PEFT design.

Perhaps most significantly, the paper establishes a causal-mechanistic template for analyzing deep learning training dynamics. The geometric intervention experiment—function-preserving orthogonal rotations that scramble pretrained geometry while maintaining exact output equivalence, followed by observation that update localization collapses to random—is methodologically clean in a way that correlational analyses are not. It directly tests whether the arrangement of weights (their geometry) matters, rather than merely observing that certain weight patterns correlate with certain training behaviors. This template—manipulate a hypothesized causal mechanism while holding all other factors constant, observe the predicted effect—could be applied to other open questions in training dynamics: does the spectral structure of attention weights matter for in-context learning? Does the singular value gap between principal components predict catastrophic forgetting? The paper provides both the conceptual framework and the specific intervention techniques (orthogonal rotation, head permutation) for such investigations.

Follow-Up Research This Work Enables

Cheap, adaptive difficulty estimation for compute-optimal RLVR allocation. The paper's most immediate practical bottleneck—difficulty estimation requiring thousands of samples per prompt—is flagged but not addressed (Section 3.2). The Three-Gate Theory suggests a more efficient approach: instead of 2,048 samples, one could estimate difficulty from the KL leash behavior itself. Since Gate I constrains how far the policy can move per step, the rate at which the policy's KL divergence grows during early RL training should correlate with problem difficulty—hard problems (where the base model has near-zero pass@1) would show slower KL growth because the policy has little room to improve within the constraint, while easy problems would show faster KL growth as the policy rapidly tilts toward high-reward completions. This would repurpose the KL trajectory tracking already used in the sparse fine-tuning probe (Figure 9) as a difficulty estimator, eliminating the separate estimation phase and amortizing difficulty assessment into the training process itself. A strong follow-up would measure the correlation between early-training KL divergence and the oracle difficulty bins (pass@1 rate from 2,048 samples) on the MATH benchmark, and test whether a policy that adaptively switches between search algorithms based on this KL signal achieves comparable efficiency to the oracle-bin policy.

Dynamic, mid-training strategy switching informed by geometric diagnostics. The paper's compute-optimal allocation is static: estimate difficulty once, look up the best strategy, execute. But the parameter-space diagnostics developed here—spectral drift, principal-subspace rotation, update-principal overlap—could be computed during training from intermediate checkpoints, enabling dynamic strategy adjustment. For example, if the spectral drift starts accelerating (suggesting the policy is being pulled toward high-curvature directions, risking over-optimization), the system could reduce the learning rate or switch from beam search to best-of-N. If the update-principal overlap increases beyond the sub-random baseline (suggesting the optimizer is being forced into principal directions, potentially destabilizing), it could activate a regularization term that penalizes principal-weight movement. This would require efficient online SVD (already feasible for the matrix sizes in models up to ~8B parameters) and a controller policy that maps diagnostic states to strategy adjustments. A concrete experiment: train DS-Qwen-1.5B with GRPO, compute per-layer spectral drift every 50 steps, and test whether early-terminating runs where drift exceeds a threshold (calibrated from the stable RL runs in Figure 4) prevents the performance degradation that the paper observes when beam search over-optimizes the verifier at high budgets.

Geometry-aware PEFT methods designed specifically for RLVR's off-principal dynamics. The paper's case studies (Sections 5.1, 5.2) demonstrate that SFT-aligned PEFT fails for RL, but they only test binary masks and low-rank adapters. The finding that freezing principal weights while updating off-principal, low-magnitude weights closely tracks dense performance (Table 4: 56.24% vs. 57.44% at 320 steps using 74% of parameters) suggests a more sophisticated approach: learned sparsity masks that adapt per-layer based on the pretrained geometry. Instead of a one-shot mask from SVD, one could parameterize a mask as a continuous function of the pretrained weight matrix (e.g., sigmoid(g(|W_0|)) where g is a small neural network or a per-layer learnable threshold) and optimize it jointly with the RL objective, regularized to encourage off-principal, low-magnitude updates. This would be a "geometry-aware" sparse fine-tuning method in the literal sense: the mask is a function of geometry, not a fixed binary pattern. A concrete experiment: on DS-Qwen-1.5B, train a per-layer mask network that takes |W_0| as input and outputs a continuous sparsity mask, optimize it for 50 warmup steps on the RL objective with a penalty term λ · Overlap(M, M_princ), then compare the final accuracy and KL trajectory against the one-shot safe mask from Table 4. The hypothesis: a learned mask can achieve comparable performance with higher sparsity because it can fine-tune the boundary between touchable and untouchable weights rather than using a hard threshold.

Extending the Three-Gate Theory to multi-turn RL and iterative self-improvement loops. The paper analyzes single-stage RLVR from a fixed pretrained checkpoint. But modern post-training pipelines often involve multiple rounds: SFT → RL → distillation → RL, with the model serving as its own reference policy across iterations. In such self-improvement loops (a la STaR, ReST^EM), Gate I's KL anchor is relative to the previous iteration's policy, not the base pretrained model. Over multiple iterations, the accumulated KL drift from the original pretrained geometry could compound, potentially causing the policy to gradually escape the off-principal, spectrum-preserving regime that Gate II describes. At some point, the model's geometry may have shifted enough that the original principal-subspace structure no longer constrains updates—a phase transition where RL behavior qualitatively changes. The paper's diagnostic toolkit makes this tractable to study: run 5–10 iterations of RLVR on DS-Qwen-1.5B, each using the previous iteration's final checkpoint as the reference policy, and track (a) the cumulative spectral drift from the base pretrained model, (b) the update-principal overlap at each iteration, and (c) the consensus ratio of updated positions across iterations (do later iterations still route updates to the same stripe patterns, or does the bias decay?). The paper's finding that the ReST^EM-trained revision model degraded (Appendix K) hints that such phase transitions may occur; a systematic characterization would establish the limits of the "RL stays near the base model" regime and inform how many RL iterations are feasible before geometry-aware re-anchoring is needed.

Causal decomposition of the Three Gates: which one dominates the bias? The paper presents Gate I (KL Anchor), Gate II (Model Geometry), and Gate III (Precision) as a unified mechanism, but does not quantify their relative contributions. Are the update patterns primarily driven by the KL leash (Gate I), with geometry merely determining which low-curvature directions are available? Or is geometry (Gate II) the dominant force, with the KL leash setting a scale but not a direction? One could isolate these effects through targeted interventions. To isolate Gate I: run RLVR with varying KL penalty strengths β (including β = 0 with loose clipping, and β large enough to keep the policy nearly frozen) and measure how the update-principal overlap and spectral drift change. The hypothesis: stronger KL penalty (tighter leash) should push updates further off-principal, because high-curvature directions become even more expensive; weaker penalty should allow some principal-direction movement. To isolate Gate II: apply the orthogonal rotation intervention from Section 4.3, which scrambles geometry while preserving function, and measure whether the KL divergence trajectory changes (it shouldn't, since forward-pass behavior is identical) but update localization does (it should randomize, as the paper already shows). To isolate Gate III: run the same RL training stored in float32 vs. bfloat16 and compare the bf16-aware update mask overlap between the two—the underlying geometric bias should produce the same relative update magnitudes in both, but the bf16 storage will threshold them into a binary mask. A full decomposition would quantify, for each layer, what fraction of the variance in update localization is explained by KL constraint strength, pretrained geometry (as measured by singular value gap γ_k), and precision effects.

Adversarial testing: can we force RLVR into principal directions, and what breaks? The paper's theory predicts that forcing RLVR to update principal weights should degrade performance or destabilize training—the sparse fine-tuning (Table 4) and PiSSA (Figures 10, 11) results support this. But these are correlational: they show that principal-targeted methods perform worse, not that principal-direction updates are causally harmful. A stronger test: design a "principal-biasing" regularizer that explicitly penalizes off-principal updates and rewards principal-direction updates (R_principal(ΔW) = −‖ΔW − proj_{U_k}(ΔW)‖_F^2, where proj_{U_k} projects onto the top-k singular subspace), add this to the RL objective, and observe whether (a) the KL trajectory slows (consistent with Gate II's prediction that principal directions are high-curvature and KL-expensive), (b) training destabilizes (as PiSSA does at high learning rates), and (c) the spectral drift accelerates (as SFT does in Figure 4). The converse experiment—a regularizer that penalizes principal-direction updates—should improve stability and preserve the spectrum even more than standard RLVR. This would provide causal evidence that the off-principal bias is not merely a side effect but an adaptive property of RLVR: the optimizer discovers off-principal directions because they are genuinely better pathways for improving reward under the KL constraint, not because they happen to be available.

Practical Applications and Downstream Use Cases

Geometry-informed sparse RLVR training for cost reduction at scale. The safe mask result (Table 4: M_low ∪ M_princ^c achieves 56.24% average accuracy vs. 57.44% dense at 320 steps using 74.02% of linear-layer parameters) directly enables memory and compute savings in large-scale RL post-training. For a practitioner training a 70B-parameter model with RLVR, the linear layers (attention projections and MLP weights) constitute the vast majority of parameters. Freezing roughly 26% of these—specifically the principal, large-magnitude weights identified by SVD and thresholding—would reduce optimizer state memory proportionally (AdamW stores two momentum buffers per parameter) and reduce the backward-pass FLOPs for frozen weights. The key practical advantage: the mask requires no training to compute (only SVD of the pretrained weights, which is a one-time cost), and the paper shows it works across model families (Qwen, Llama, Mistral in Table 1, Figure 13) and RL algorithms (GRPO, DAPO, Reinforcement++). The trade-off: the 1.2 percentage point accuracy gap to dense training at 320 steps (Table 4) must be weighed against the ~26% parameter reduction, and the optimal mask density likely varies per layer and per model scale—the paper's fixed 74% is not necessarily the pareto-optimal point.

Checkpoint diff debugging and training monitoring via parameter-space diagnostics. The paper's diagnostic toolkit—spectral drift, principal angle tracking, update-principal overlap—can be deployed as a monitoring dashboard during RL post-training to detect anomalous behavior before it manifests in evaluation metrics. For example, if the maximum principal angle for a layer suddenly increases (suggesting the principal subspace is rotating, as SFT does in Figure 4), this could indicate that the KL leash has become too loose (learning rate too high, clipping epsilon too large) and the policy is being pulled toward high-curvature directions, risking over-optimization. If the update-principal overlap increases above the sub-random baseline (Figure 5), it could indicate that the optimizer is being forced into principal directions by a reward signal that is too strong relative to the KL constraint. These diagnostics are cheap to compute from periodic checkpoints (SVD of weight matrices is fast for the layer sizes in typical transformer models) and provide early warning of training instability that might take hundreds of steps to manifest in downstream accuracy. This is immediately actionable for teams running large-scale RLVR training, where silent degradation can waste substantial GPU-hours before being detected.

Adapter selection for RL post-training: choose LoRA, not PiSSA or principal-targeted variants. The paper's LoRA vs. PiSSA comparison (Figures 10, 11) provides a clear, actionable recommendation for practitioners: when using low-rank adapters for RLVR, do not use principal-targeted initialization variants like PiSSA. Standard LoRA with random initialization and frozen base weights naturally approximates off-principal updates (the frozen base discourages principal-direction movement, and the small adapter capacity is sufficient because effective RL updates are concentrated in a narrow subspace), while PiSSA's principal-direction bias causes training instability, particularly at the higher learning rates needed for low-rank adapters to match full-parameter performance. The recommendation is robust across two model families (DS-Qwen-1.5B and LLaMA-3.2-3B) and multiple ranks and learning rates. The practical benefit is avoiding wasted experimentation: a team that read Schulman & Lab (2025) and decided to try PiSSA for RL would likely encounter training collapses that they might attribute to hyperparameter sensitivity or data issues, when the root cause is geometric misalignment. The paper's geometric account provides the diagnosis that enables this recommendation without requiring each team to independently replicate the full rank/learning-rate sweep.

When to Prefer This Framework Over Alternatives

The paper itself does not propose a specific method that competes with named alternatives—it is a diagnostic and mechanistic study, not a method paper. However, it does articulate a clear conceptual trade-off between two approaches to understanding and optimizing RL post-training:

Use the Three-Gate / parameter-space diagnostics framework when:

  • The goal is to understand why RL training behaves as it does (stability, sample efficiency, final performance) rather than merely achieving state-of-the-art accuracy. The diagnostics reveal mechanisms (spectral preservation, off-principal routing) that behavioral metrics alone cannot.
  • Designing new PEFT methods for RL post-training, and need to avoid geometrically misaligned approaches. The framework provides a decision rule: methods that bias updates toward principal directions (high-curvature, spectrum-distorting) are predicted to underperform; methods that allow off-principal updates while regularizing principal directions are predicted to work better. This is validated for sparse masks (Section 5.1) and LoRA variants (Section 5.2).
  • Training runs are expensive and early detection of instability is valuable. The diagnostics (spectral drift, principal angle) can serve as early-warning indicators before downstream accuracy degrades.

The parameter-space diagnostics are less informative when:

  • The primary goal is pure accuracy maximization on a specific benchmark, and the cost of running the diagnostics (SVD of all weight matrices at multiple checkpoints, computing update masks) outweighs the value of mechanistic understanding.
  • Working with models where SVD is computationally prohibitive (100B+ parameters) and efficient approximations (randomized SVD, power iteration) have not been validated against the paper's exact-SVD results.
  • The RL training pipeline differs fundamentally from the on-policy, KL-constrained regime studied here—for example, offline RL from fixed datasets without a reference policy, or RL from human feedback where the reward signal is a learned preference model rather than verifiable. The Three-Gate Theory's reliance on the KL leash (Gate I) and pretrained geometry (Gate II) may not hold in these regimes without modification.