RAGEN-2: Reasoning Collapse in Agentic RL

🎯 Pitch

Even when entropy stays high, RL-trained agents can collapse into input-agnostic reasoning templates that look diverse per-input but fail to distinguish across problems—a failure mode invisible to all standard metrics. The authors introduce a mutual information proxy that detects this "template collapse" and show it predicts final performance far better than entropy. They trace the cause to low reward variance weakening task signal versus regularization noise, and fix it with SNR-Aware Filtering that boosts success rates by over 15 percentage points.

1. Executive Summary

This paper diagnoses and mitigates a previously invisible failure mode in multi-turn LLM agent reinforcement learning called template collapse, where reasoning appears diverse within any single input (high conditional entropy) but becomes input-agnostic across different inputs (low mutual information), making the degradation undetectable by standard entropy-based monitoring. The authors introduce a family of mutual information (MI) proxies that cross-score reasoning traces against batch inputs to measure input dependence without external models, and explain template collapse through a signal-to-noise ratio (SNR) mechanism: low within-prompt reward variance weakens task-discriminative gradients while input-agnostic regularization terms—KL divergence and entropy bonuses—remain constant and dominate updates, systematically erasing cross-input reasoning differences. To counteract this, they propose SNR-Aware Filtering, which uses reward variance as a lightweight SNR proxy to retain only high-signal prompts per iteration before computing parameter updates. Across planning (Sokoban), mathematical reasoning (MetaMathQA, Countdown), web navigation (WebShop), and code execution (DeepCoder) under PPO, DAPO, GRPO, and Dr. GRPO, SNR-Aware Filtering consistently improves both input dependence and task performance—for instance, boosting PPO on Sokoban from 12.9% to 28.9% (+16.0) with Qwen2.5-3B—while the MI proxy achieves a Spearman correlation of +0.39 with final performance versus −0.11 for entropy, establishing that input dependence, not within-input diversity, is the reliable diagnostic for reasoning quality in agent RL, and that template collapse is a gradient-level consequence of low reward variance that cannot be prevented by entropy or KL tuning alone.

2. Context and Motivation

The Core Problem: Reasoning Degradation That Entropy Cannot See

The paper addresses a specific and previously invisible failure mode in training multi-turn LLM agents with reinforcement learning: template collapse, where an agent's reasoning becomes fluent and superficially diverse, yet completely input-agnostic — generating structurally varied but semantically interchangeable boilerplate regardless of the problem it faces. The critical danger is that this degradation occurs while standard monitoring metrics remain stable, particularly the conditional entropy of the reasoning process, which practitioners routinely track as a health indicator during RL training.

This matters because multi-turn agent RL represents one of the most promising paths toward autonomous LLM systems that can plan, reason, self-correct, and interact with environments over extended horizons. The training process is inherently unstable: models are updated on their own self-generated trajectories, creating a closed feedback loop where small deviations in reasoning quality can compound across iterations. When reasoning collapses into templates, the agent loses the ability to adapt its thinking to specific inputs — it becomes unreliable in ways that are hard to detect because its outputs still look reasonable and its reward may remain stable. The paper argues this is not a hypothetical risk but a systematic failure that "can persist unnoticed during training, making agents unreliable and silently hurting their reasoning abilities" (Section 1).

The urgency of this problem is amplified by recent trends in the field. Large-scale RL training of reasoning models (e.g., DeepSeek-R1, GRPO-based training) has shown that reinforcement learning can dramatically improve LLM reasoning capabilities. However, the community's understanding of what can go wrong during such training — beyond obvious failure modes like reward hacking or complete mode collapse — remains limited. Template collapse represents a class of failures that are particularly insidious because they corrupt the very capability that RL is meant to enhance (input-driven reasoning) while leaving surface metrics healthy. The paper's framing makes clear that stable entropy and stable reward are necessary but not sufficient conditions for healthy training.

The paper identifies a fundamental limitation in how the field monitors reasoning quality during RL training. The standard approach, inherited from the broader RL and language model alignment literature, tracks two signals:

Reward as an outcome stability metric — is the agent achieving consistent task success?
Entropy (specifically, conditional entropy $H(Z|X)$ of reasoning tokens given the input) as a process stability metric — is the reasoning distribution remaining diverse rather than collapsing to a single mode?

This two-signal monitoring regime has an important lineage. In RLHF and related alignment paradigms (Ouyang et al., 2022; Schulman et al., 2017), entropy is explicitly used as a regularizer to prevent policy collapse — the phenomenon where the model's output distribution concentrates on a small set of high-reward responses, losing diversity. Practitioners have therefore come to treat stable or high entropy as evidence that reasoning remains healthy. The paper cites works that "proxy process stability with entropy and outcome stability with reward, treating both as evidence of healthy training" (Section 2.2).

The critical gap the paper identifies is that entropy measures within-input diversity but cannot detect whether reasoning actually responds to different inputs. The standard identity from information theory makes this explicit:

$H(Z) = I(X; Z) + H(Z|X)$

where $H(Z)$ is marginal entropy (overall reasoning diversity), $I(X; Z)$ is mutual information between inputs and reasoning (input dependence), and $H(Z|X)$ is conditional entropy (within-input diversity). Entropy-based monitoring effectively tracks $H(Z|X)$ — how varied the model's outputs are for a given input — but says nothing about $I(X; Z)$ , which measures whether those outputs are systematically different across inputs.

The paper's key insight is that these two quantities can move in opposite directions: $I(X; Z)$ can decline sharply while $H(Z|X)$ remains high or even increases. When this happens, the model produces reasoning that looks diverse within any single prompt but is effectively the same across prompts — it has learned reusable templates that satisfy the surface requirement of "generating varied tokens" without engaging with the specific content of each input. This is precisely what entropy-based monitoring misses.

Conflicting Prior Evidence on Reasoning Quality Under RL

The paper is motivated by a landscape of partially understood degradation phenomena that have been reported across the LLM agent RL literature but never systematically diagnosed. The authors cite a family of observations that have appeared under different names:

Reasoning collapse: "rationales, plans, or explanations becoming increasingly templated and less diverse, while their correspondence to the input goal weakens" (Appendix A, citing Wei et al., 2025; Yun et al., 2025; Yao et al., 2025)
Policy-level degeneracy: "behavioral choices concentrating on a small set of easy-to-reproduce action patterns that yield stable scores, with less exploration and less error correction" (Appendix A, citing Feng et al., 2025; Wang and Ammanabrolu, 2025)
Model collapse in self-training: the well-documented phenomenon where models trained on their own outputs gradually lose distributional coverage, even when average metrics appear stable (Gerstgrasser et al., 2024; Shumailov et al., 2024)

The paper positions template collapse as a specific subtype within this broader family — one that is distinguishable from other forms of degradation because it leaves conditional entropy intact. This distinction matters because prior work has sometimes conflated different collapse modes. For instance, entropy collapse (where $H(Z|X)$ drops) is a known and visible failure that regularization can address. Template collapse (where $I(X; Z)$ drops while $H(Z|X)$ stays high) is invisible to the same interventions and requires different diagnostics and remedies.

The authors also note that reasoning-level degradation can decouple from policy-level degradation: "even when the agent's external behavior remains effective or yields stable rewards, language-level reasoning expressions can become concentrated earlier" (Appendix A, citing Wei et al., 2025; Xu et al., 2025). This means an agent might continue to solve tasks successfully while its reasoning has already collapsed into templates — a particularly dangerous scenario because the agent appears to be training normally but is actually losing the generalization capability that reasoning is meant to provide.

Where Existing Stabilization Methods Fall Short

The paper identifies a broad set of stabilization techniques from the RL and LLM alignment literature, then argues that none of them address the root cause of template collapse:

KL constraints and trust-region methods (Schulman et al., 2017; Stiennon et al., 2022) keep the policy close to a reference model, preventing it from deviating too far into degenerate regions. However, KL constraints act uniformly across all inputs — they apply the same contraction force to every reasoning chain regardless of whether the chain is input-driven or templated. If the reference model already has reasonable $I(X; Z)$ , KL anchoring can help preserve it (formalized in the paper's Appendix L), but it cannot increase $I(X; Z)$ that is being eroded by other forces.

Entropy regularization (Ouyang et al., 2022; Xu et al., 2025) explicitly encourages output diversity by adding a bonus proportional to $H(Z|X)$ . This directly increases within-input variability, which can prevent the most obvious form of mode collapse. However, the paper demonstrates in Section 5.1 (Figure 13) that sweeping the entropy coefficient moves training dynamics primarily along the $H(Z|X)$ axis — increasing within-input diversity — without significantly shifting $I(X; Z)$ . In fact, the entropy decomposition in Appendix M proves that if an entropy bonus increases $H(Z|X)$ more than it increases the marginal diversity $H(Z)$ , mutual information necessarily decreases. An entropy bonus can therefore actively contribute to template collapse while making the surface metric (entropy itself) look healthier.

Imitation-to-RL pipelines and stepwise rewards (Cobbe et al., 2021; Uesato et al., 2022; Madaan et al., 2023) provide denser supervision signals that could, in principle, discriminate between input-driven and templated reasoning. However, the paper notes that these methods still operate within a framework where "if most rollouts for the same prompt receive similar rewards regardless of reasoning quality, then the gradient update carries little information about which reasoning path matters" (Appendix A, citing Moskovitz et al., 2023; O'Mahony et al., 2024; Shumailov et al., 2024; Yun et al., 2025). The fundamental issue is not the sparsity of rewards but the within-prompt reward variance — if all trajectories for a given prompt receive identical rewards, no amount of stepwise decomposition can recover a gradient that distinguishes good reasoning from bad.

DAPO and Dr. GRPO (Yu et al., 2025; Liu et al., 2025) are recent algorithms that specifically target stable training and collapse mitigation. The paper explicitly evaluates SNR-Aware Filtering on top of these baselines (Table 4) and finds it provides additional gains, suggesting that even state-of-the-art stabilization methods do not prevent the SNR imbalance that drives template collapse. DAPO's built-in filtering/acceptance step "can be interpreted as a special case of our framework where the selection is fixed" (Section 4.3), while SNR-Aware Filtering provides an explicit, tunable knob for controlling the signal-to-noise ratio of each update — a dimension that existing methods do not directly address.

The Gap in Reasoning Quality Evaluation

Beyond training interventions, the paper identifies a limitation in how the field evaluates reasoning quality itself. Most diversity metrics — n-gram statistics, embedding dispersion, distributional distances, token-level uncertainty — "primarily capture overall randomness or within-input variability" and are "often less sensitive to whether the reasoning distribution changes coherently across inputs" (Appendix A, citing Tevet and Berant, 2021; Yun et al., 2025).

Recent work has started probing input dependence through behavioral tests (Gardner et al., 2020; Ribeiro et al., 2020), retrieval-style output-input matching (Morris et al., 2023; Gao et al., 2024; Zhang et al., 2024), and prompt reconstruction signals (Li and Klabjan, 2025). However, the paper notes that "a unified and scalable treatment tailored to closed-loop agent RL remains limited" — existing methods typically require external models, human evaluation, or carefully constructed test suites that are not practical for online monitoring during RL training. The MI proxy family introduced in this paper fills this gap by reusing the training model's own scoring to measure input dependence from standard rollout data, requiring no additional models or inference passes.

A parallel line of work on reasoning faithfulness (Lanham et al., 2023; Turpin et al., 2023; Siegel et al., 2024; Zaman and Srivastava, 2025) asks whether a model's stated reasoning reflects its true decision basis. While related, the paper draws a clear distinction: faithfulness concerns whether reasoning causally supports a particular decision, while template collapse concerns whether reasoning remains sensitive to inputs at all during optimization. An agent could have faithful but templated reasoning (the same causal structure applied to every input) or unfaithful but input-dependent reasoning (different post-hoc rationalizations for different inputs).

How This Paper Positions Itself

The paper positions itself at the intersection of three research threads:

Diagnosing collapse phenomena in LLM training — providing the first systematic characterization of template collapse as a distinct failure mode separable from entropy collapse, and introducing practical proxies to detect it online.
Understanding the mechanisms of RL-driven degradation — proposing the SNR framework as a causal explanation for why template collapse occurs, grounded in a gradient decomposition of the policy update into task-signal, task-noise, and regularization-noise components.
Developing lightweight interventions — introducing SNR-Aware Filtering as a method that controls the signal quality of each update without requiring additional models, human feedback, or changes to the underlying RL algorithm.

The paper's central claim is not that prior stabilization methods are ineffective, but that they target the wrong axis. KL constraints and entropy regularization primarily control $H(Z|X)$ — within-input diversity — while template collapse is fundamentally a problem of declining $I(X; Z)$ — cross-input dependence. The SNR mechanism explains why $I(X; Z)$ degrades (low reward variance weakens the task gradient's ability to discriminate between inputs, letting input-agnostic regularization dominate) and SNR-Aware Filtering provides a direct intervention on the causal pathway (by removing low-variance prompts whose updates would be dominated by regularization noise).

This framing has a specific practical implication: practitioners monitoring LLM agent RL training should track input dependence ( $I(X; Z)$ ) alongside traditional metrics (reward, entropy), and should be skeptical of runs where entropy remains stable but the MI proxy declines — such runs may be silently losing the reasoning capability that RL is intended to enhance.

3. Technical Approach

This is primarily a diagnostic and analytical paper that introduces a framework for detecting, explaining, and mitigating template collapse in multi-turn agent RL — a failure mode where reasoning degrades from input-driven to input-agnostic without triggering traditional stability monitors. The core idea is to decompose reasoning quality into two orthogonal axes: within-input diversity (tracked by entropy) and cross-input distinguishability (tracked by mutual information), then explain why the latter degrades through a signal-to-noise ratio mechanism in policy gradients, and finally intervene by filtering training prompts based on reward variance to restore input-conditioned reasoning.

3.1 Reader Orientation

The paper builds two interconnected systems: (1) a diagnostic framework that monitors whether an agent's reasoning remains input-dependent during RL training using mutual information proxies computed from the model's own log-probabilities, and (2) a training intervention (SNR-Aware Filtering) that selects which prompts to update on at each iteration based on within-prompt reward variance. The problem being solved is that standard RL training of multi-turn LLM agents causes reasoning to drift toward fluent but input-agnostic templates — a degradation invisible to entropy-based monitoring — and the solution takes the form of measuring input dependence directly (via MI proxies) while filtering out training updates that would be dominated by input-agnostic regularization noise (via SNR-Aware Filtering).

3.2 Big-Picture Architecture

The system has five major components:

Policy and Environment Loop — the base LLM agent (e.g., Qwen2.5-3B) generates reasoning-then-action trajectories in a multi-turn environment (Sokoban, FrozenLake, MetaMathQA, Countdown, etc.), receives rewards, and collects rollout data for RL updates under PPO/GRPO/DAPO/Dr. GRPO.
Mutual Information (MI) Proxy Calculator — reuses the training model's teacher-forced log-probabilities to compute in-batch cross-scoring matrices between prompts and reasoning traces, producing continuous (MI-Est, MI-ZScore-EMA) and discrete (Retrieval-Acc) proxies that estimate input dependence $I(X; Z)$ without external models or additional inference passes.
Reward Variance Estimator — computes per-prompt within-input reward variance $\widehat{\text{Var}}(R|X)$ from the $G$ trajectories sampled for each prompt in a training batch, serving as a lightweight proxy for the signal-to-noise ratio of the gradient update that prompt would produce.
SNR-Aware Filtering Operator — ranks prompts by reward variance, retains the top fraction (via top- $p$ nucleus-style selection with keep rate $\rho$ ), and masks the RL objective to update only on the selected high-signal prompts, effectively concentrating gradient budget on updates where task-discriminative signal dominates regularization noise.
Training Monitor — tracks the MI proxy alongside traditional metrics (task success rate, conditional entropy) during training, using the proxy as an early warning signal: declining MI while entropy remains stable or increases indicates template collapse is occurring.

Information flows as follows: the agent interacts with the environment → trajectories are collected → the MI proxy calculator scores all $(Z, X)$ pairs in the batch to estimate $I(X; Z)$ → the reward variance estimator computes $\widehat{\text{Var}}(R|X)$ per prompt → the filtering operator selects high-variance prompts → the RL update is computed only on the filtered subset → the MI proxy and task metrics are logged for monitoring.

3.3 Roadmap for the Deep Dive

First, the formal decomposition of reasoning quality into $I(X; Z)$ and $H(Z|X)$ using information theory, establishing why entropy alone is insufficient and defining template collapse precisely in these terms.
Second, the MI proxy family — how it estimates input dependence from in-batch cross-scoring of reasoning traces against prompts using the model's own log-probabilities, including the discrete Retrieval-Acc proxy (interpretable, with a chance-level baseline) and the continuous MI-ZScore-EMA proxy (robust, normalized for training monitoring).
Third, the empirical gradient analysis that motivates the SNR mechanism — how sorting prompts by reward variance reveals that task gradient norms scale with variance while regularization gradients remain flat, establishing the empirical basis for the theoretical framework.
Fourth, the formal SNR mechanism — the gradient decomposition into signal, task-noise, and regularization-noise components, the proof that task gradient magnitude is bounded by $\sqrt{\text{RV}(x)}$ , and the explanation of why low reward variance causes regularization to dominate updates and erase cross-input reasoning differences.
Fifth, SNR-Aware Filtering — the top- $p$ selection procedure, the reward variance computation, the filtering operator definition, and the design choices behind the adaptive keep rate.
Sixth, the supporting theoretical results (from the appendix) that formalize the connection between reward variance, gradient quality, and input dependence, including the template mixing lemma and the entropy decomposition theorem.

3.4 Detailed, Sentence-Based Technical Breakdown

The Information-Theoretic Decomposition of Reasoning Quality

The paper's foundational move is to define reasoning quality not as a single quantity but as a pair of orthogonal quantities drawn from information theory. Given a prompt $X$ and a reasoning trace $Z$ generated by the policy $\pi_\theta(z|x)$ , the marginal entropy of reasoning $H(Z)$ decomposes via the standard Shannon identity:

$H(Z) = I(X; Z) + H(Z|X)$

where $H(Z)$ is the marginal entropy of the reasoning distribution (how diverse reasoning is overall, averaging across all inputs), $I(X; Z)$ is the mutual information between inputs and reasoning (how much observing the reasoning tells you about which input produced it — a measure of input dependence), and $H(Z|X)$ is the conditional entropy of reasoning given the input (how diverse reasoning is for a fixed input — a measure of within-input variability).

What this decomposition computes: given the joint distribution $P(X, Z) = P(X)\pi_\theta(Z|X)$ induced by the prompt distribution and the current policy, it partitions total reasoning diversity into a component that is attributable to input variation $(I)$ and a component that remains after accounting for the input $(H(Z|X))$ . If reasoning is perfectly input-agnostic (the same distribution regardless of $X$ ), then $I(X; Z) = 0$ and $H(Z) = H(Z|X)$ — all diversity is within-input noise with no cross-input structure. If reasoning perfectly encodes its input (a one-to-one mapping), then $H(Z|X) = 0$ and $H(Z) = I(X; Z)$ — all diversity comes from input variation with zero within-input randomness.

Why this decomposition: the paper argues that standard entropy monitoring tracks only $H(Z|X)$ (or a proxy thereof), which measures whether the model is producing varied outputs for a single input. This tells you nothing about whether outputs for different inputs are systematically different. A model could achieve high $H(Z|X)$ by randomizing over a set of fluent but input-agnostic phrases — the conditional entropy would look healthy, but $I(X; Z)$ would be near zero. The decomposition makes explicit that these are two independent axes of reasoning quality, and that a decline in one can be masked by stability in the other. Template collapse is precisely the regime where $H(Z|X)$ remains high or even increases while $I(X; Z)$ drops to zero.

The paper operationalizes this decomposition via scorer-based proxies. Since true Shannon quantities have no closed form for high-dimensional token sequences, the paper defines empirical proxies computed from a fixed batch of prompts and rollouts using the model's own log-probabilities as a scoring function. For a scorer $q$ (which in practice is the current policy $\pi_\theta$ itself), the key quantities are:

$\text{matched}_{i,k} = \frac{\log q(Z_{i,k} | X_i)}{|Z_{i,k}|}, \quad \text{marginal}_{i,k} = \frac{1}{|Z_{i,k}|} \log\left( \frac{1}{P} \sum_{j=1}^P \exp(\log q(Z_{i,k} | X_j)) \right)$

where $P$ is the number of prompts in the batch, $X_i$ is the $i$ -th prompt, $Z_{i,k}$ is the $k$ -th reasoning trace generated from prompt $X_i$ , $\log q(Z_{i,k} | X_i)$ is the teacher-forced log-probability of the reasoning trace under its true source prompt (the "matched" score), and $\log q(Z_{i,k} | X_j)$ is the teacher-forced log-probability of the same trace under a different prompt $X_j$ . The division by $|Z_{i,k}|$ (the number of tokens in the trace) provides length normalization, preventing longer traces from dominating the scores.

What these quantities compute: matched $_{i,k}$ is the per-token log-likelihood of reasoning trace $Z_{i,k}$ conditioned on its true source prompt $X_i$ — it measures how well the model's own scoring function recognizes that this reasoning trace "belongs" to this prompt. marginal $_{i,k}$ is the per-token log-likelihood of the same trace under a uniform mixture over all prompts in the batch — it measures how well the trace fits under an average prompt, approximating the marginal log-likelihood $\log p_\theta(Z_{i,k})$ via the empirical mixture $\frac{1}{P} \sum_j q(Z_{i,k} | X_j)$ . The difference matched $-$ marginal estimates how much more likely the trace is under its true prompt than under a random prompt — when this difference is large, the reasoning is strongly input-dependent; when it is near zero, the reasoning is equally compatible with any prompt, indicating template collapse.

Why this form: the paper uses teacher-forced scoring (evaluating $q(Z|X)$ for a pre-generated $Z$ ) rather than generation-based scoring (evaluating how likely $X$ is to produce $Z$ under autoregressive sampling) because teacher-forced log-probabilities are deterministic and cheap to compute — they require only a single forward pass per $(Z, X)$ pair through the model, not repeated sampling. Using the model's own scoring function $(q = \pi_\theta)$ means no external model is needed, making the proxy lightweight enough for online monitoring during training. The batch-based mixture approximation avoids the need to maintain a running estimate of the marginal distribution $p_\theta(Z)$ across the entire training set.

The MI Proxy Family: Design, Variants, and Computation

The paper defines a family of proxies for $I(X; Z)$ by varying three dimensions: (1) the aggregation method (discrete retrieval vs. continuous MI estimate), (2) the turn scope (first-turn only vs. trajectory-uniform sampling across all agent turns), and (3) the normalization scheme (per-token vs. per-sequence, raw vs. z-score vs. EMA-stabilized). All proxies are computed from the in-batch cross-scoring matrix.

Step 1: Construct the scoring matrix. Given a training batch with $P$ prompts and $G$ reasoning samples per prompt (total $P \times G$ traces), the system computes a $P \times G \times P$ tensor $\mathbf{L}$ where:

$L_{i,k,j} = \log \pi_\theta(Z_{i,k} | X_j)$

where $L_{i,k,j}$ is the teacher-forced log-probability of the $k$ -th reasoning trace from prompt $i$ when conditioned on prompt $j$ . This requires evaluating the forward pass of the current policy on every $(Z, X)$ pair in the batch — $P \times G \times P$ evaluations. Since $P$ is typically small (8–64) and $G$ is similarly modest (2–16), this cost is dominated by the rollout generation cost and adds negligible overhead.

Step 2: Extract matched and marginal scores. For each reasoning trace $Z_{i,k}$ , extract:

matched $_{i,k}$ : the log-probability under the true prompt $X_i$ , length-normalized — this is the diagonal element $L_{i,k,i}$ divided by $|Z_{i,k}|$ .
marginal $_{i,k}$ : the log-probability under the batch mixture $\frac{1}{P} \sum_j \exp(L_{i,k,j})$ , length-normalized — this is the log-sum-exp over the prompt dimension (including the true prompt) divided by $|Z_{i,k}|$ .

Step 3: Compute the MI proxy value. Different proxy variants aggregate the matched and marginal scores differently:

(a) Retrieval-Acc (discrete, interpretable). For each reasoning trace $Z_{i,k}$ , identify which prompt $j$ maximizes the raw (non-length-normalized) log-probability $L_{i,k,j}$ . If the argmax matches the true prompt index $i$ , this trace is "correctly retrieved." The proxy value is the fraction of traces correctly retrieved:

$\text{Retrieval-Acc} = \frac{1}{PG} \sum_{i=1}^P \sum_{k=1}^G \mathbb{1}\left[ i = \arg\max_j L_{i,k,j} \right]$

where $\mathbb{1}[\cdot]$ is the indicator function that equals 1 when the condition is true and 0 otherwise, and $\arg\max_j L_{i,k,j}$ returns the prompt index under which the trace achieves the highest raw log-probability.

What it computes: the fraction of reasoning traces that are "more compatible" with their true source prompt than with any other prompt in the batch, where compatibility is measured by the model's own log-probability scoring. Under perfect input dependence, each trace is uniquely compatible with its source prompt and Retrieval-Acc approaches 1.0. Under template collapse, traces are equally compatible with all prompts and Retrieval-Acc approaches the chance level $1/P$ (e.g., $1/64 \approx 1.56\%$ at $P = 64$ ).

Why this form: Retrieval-Acc has an absolute, interpretable baseline: chance level is known exactly from the batch size, providing a ground-truth reference point that requires no calibration. It is a discrete metric (each trace either "retrieves" or doesn't), which makes it intuitive but potentially noisier at small batch sizes. The paper uses raw (non-length-normalized) log-probabilities for the argmax because length differences between traces can carry signal — a trace that is genuinely more compatible with its source prompt may naturally be longer or shorter than it would be under a mismatched prompt.

(b) MI-Est (continuous, raw). The continuous MI estimate is the batch-averaged difference between matched and marginal scores:

$\widehat{I}(X; Z) = \frac{1}{PG} \sum_{i=1}^P \sum_{k=1}^G \left( \text{matched}_{i,k} - \text{marginal}_{i,k} \right)$

where each term is length-normalized per Equation 1.

What it computes: the average excess log-probability (per token) that a reasoning trace receives under its true prompt compared to the batch mixture baseline. When $\widehat{I}(X; Z)$ is large and positive, the model assigns substantially higher probability to its own reasoning traces when given the correct prompt — evidence of strong input dependence. When $\widehat{I}(X; Z)$ is near zero or negative, the model's reasoning traces are no more likely under their true prompts than under random prompts — evidence of template collapse. The paper notes that under template collapse, matched $_{i,k} \approx$ marginal $_{i,k}$ for many samples, driving $\widehat{I}$ toward zero.

Why this form: the continuous MI estimate uses more information than the discrete retrieval proxy — it captures degree of compatibility rather than just the binary correct/incorrect retrieval outcome, making it potentially more sensitive to early-stage degradation. However, raw continuous estimates can be noisy across training iterations due to batch composition effects and changes in the marginal score distribution as the policy evolves.

(c) MI-ZScore-EMA (continuous, stabilized). To stabilize the continuous estimate for training monitoring, the paper applies z-score normalization followed by exponential moving average smoothing:

$\text{MI-ZScore} = \frac{1}{PG} \sum_{i=1}^P \sum_{k=1}^G \frac{\text{matched}_{i,k} - \text{marginal}_{i,k}}{\sigma_{\text{batch}} + \epsilon}$

where $\sigma_{\text{batch}}$ is the standard deviation of the marginal scores across all traces in the current batch (capturing batch-level variation in reasoning compatibility), and $\epsilon = 10^{-3}$ prevents division by zero. The EMA-smoothed version updates a running estimate:

$\sigma_{\text{EMA}}^{(t)} = \alpha \sigma_{\text{EMA}}^{(t-1)} + (1 - \alpha) \sigma_{\text{batch}}^{(t)}$

with $\alpha = 0.9$ , then substitutes $\sigma_{\text{EMA}}$ for $\sigma_{\text{batch}}$ in the z-score denominator. This produces MI-ZScore-EMA, the primary continuous proxy used in the paper's analyses.

What it computes: the same excess log-probability as MI-Est, but divided by an estimate of the batch-level score dispersion, which removes the effect of overall shifts in the score distribution (e.g., if the policy's absolute log-probabilities drift over training). The EMA further smooths the normalizer across iterations, preventing single-batch outliers from dominating the proxy trajectory.

Why this form: z-score normalization is a standard technique for removing scale effects from metrics that are tracked over time; here, it ensures that the proxy measures relative input dependence (how much more compatible is the trace with its source prompt compared to the cross-prompt variation within the same batch) rather than absolute log-probability magnitudes, which can drift for reasons unrelated to input dependence (e.g., the policy becoming more or less confident overall). The EMA with $\alpha = 0.9$ provides a slowly adapting baseline that prevents the z-score from being dominated by a single anomalous batch.

(d) Variants by turn scope and aggregation. Table 1 lists additional variants:

Recall@k (discrete): instead of strict argmax retrieval, checks whether the true prompt is in the top- $k$ highest-scoring prompts ( $k \in \{2, 4, 8\}$ ). This relaxes the retrieval criterion, capturing cases where reasoning is somewhat input-dependent but not perfectly discriminable.
MI-Seq-Est (per-sequence): computes the score per sequence without length normalization — $L_{i,k,i} - \log(\frac{1}{P} \sum_j e^{L_{i,k,j}})$ — which preserves sequence-level information but can be dominated by sequence length effects.
First-turn vs. trajectory-uniform: first-turn variants use only the first agent reasoning turn; trajectory variants sample uniformly across all turns in the trajectory. Trajectory-level proxies capture whether input dependence persists throughout multi-turn interactions, not just at the initial reasoning step.

Computational cost. All MI proxies reuse $(X_i, Z_{i,k})$ pairs already collected for RL training rollouts and require no additional model generation or external models. The only additional computation is $P \times G \times P$ forward passes of the reasoning traces through the model under different prompt conditionings. For typical configurations ( $P = 8, G = 16$ , so $8 \times 16 \times 8 = 1024$ additional forward passes per iteration), this is a small fraction of the total training cost, which is dominated by the $P \times G = 128$ trajectory rollouts (each potentially hundreds of tokens).

Empirical validation. Section 5.1 and Figure 8 demonstrate that Trajectory MI-ZScore achieves a Spearman correlation of +0.39 with final task performance across runs, while Reasoning Entropy (conditional entropy) shows a negative correlation of −0.11. All MI-family proxies achieve positive correlations; all entropy-family proxies achieve near-zero or negative correlations. This validates the paper's core claim: MI is a superior training monitor because it captures the reasoning quality dimension (input dependence) that entropy misses.

Empirical Motivation: Observing the Signal-Noise Imbalance

Before formalizing the SNR mechanism, the paper presents an empirical gradient analysis that motivates the theoretical framework. The procedure is:

For a training run (PPO or GRPO, Qwen2.5-3B), collect all prompt groups in a batch and compute each prompt's within-prompt reward variance: $\widehat{\text{Var}}(R|X) = \frac{1}{G-1} \sum_{g=1}^G (R_g(X) - \bar{R}(X))^2$ , where $R_g(X)$ is the return of the $g$ -th trajectory from prompt $X$ , $\bar{R}(X)$ is the mean return across the $G$ trajectories, and $G = 16$ in the main experiments.
Sort prompts into six equal-sized quantile buckets (Q1 = highest variance, Q6 = lowest variance) based on this reward variance.
For each bucket, measure the gradient norm contributed by (a) the task objective (the advantage-weighted policy gradient $E[A \cdot \nabla_\theta \log \pi_\theta]$ ), and (b) the regularization terms (KL divergence penalty and entropy bonus).

Results (Figure 3):

Task gradient norm increases monotonically with reward variance. Buckets with higher within-prompt reward variance produce substantially larger task gradient norms. This is the "signal" pattern — prompts where trajectories receive meaningfully different rewards produce stronger, more task-discriminative updates.
Regularization gradient norm is flat across buckets. The gradient norms from KL and entropy regularization terms remain approximately constant regardless of which reward-variance bucket the prompt falls into. This is the "noise floor" — regularization applies uniform contraction to every reasoning chain, independent of whether the prompt carries useful task signal.
Low-variance prompts produce updates dominated by regularization. In the lowest-variance buckets (Q5, Q6), task gradient norms approach zero while regularization gradient norms persist at their flat level, meaning the total update for these prompts is driven almost entirely by input-agnostic regularization forces.
Non-zero task gradients persist even at near-zero reward variance. The paper notes that "when RV near 0, substantial task gradients persist despite carrying almost no useful signal" (Figure 3 caption) — these are task-noise gradients arising from sampling stochasticity and environment randomness, which contribute to the gradient norm without carrying discriminative information about which reasoning paths are better.

This empirical pattern — task gradient scaling with reward variance, regularization gradient flat, leading to regularization dominance at low variance — is the direct motivation for the SNR mechanism formalized next. It is reproduced across both PPO and GRPO (Figure 3, right panels), establishing that the phenomenon is not algorithm-specific.

The SNR Mechanism: Formal Gradient Decomposition and Why Template Collapse Occurs

The paper formalizes the empirical observation into a signal-to-noise ratio (SNR) theory of policy gradient updates. The core claim is that template collapse is a gradient-level consequence of low within-prompt reward variance, not a side effect of aggressive regularization or insufficient model capacity.

Setup. For a fixed prompt $x$ , let $z \sim \pi_\theta(\cdot | x)$ be a sampled trajectory, $R(z; x)$ the scalar return, $b(x) = \mathbb{E}[R(Z; x) | X = x]$ the conditional-mean baseline, and $A(z; x) = R(z; x) - b(x)$ the advantage, which is zero-mean within each prompt. The task gradient for prompt $x$ is:

$g_{\text{task}}(x) = \mathbb{E}_{z \sim \pi_\theta(\cdot | x)} \left[ A(z; x) \cdot \nabla_\theta \log \pi_\theta(z | x) \right]$

Theorem (Task gradient magnitude is RV-controlled, Appendix H). The paper proves that the norm of the task gradient is bounded by the square root of the within-prompt reward variance:

$\|g_{\text{task}}(x)\| \leq \sqrt{\text{RV}(x)} \cdot \sqrt{\mathbb{E}\left[ \|\nabla_\theta \log \pi_\theta(z | x)\|^2 \mid X = x \right]}$

where $\text{RV}(x) = \text{Var}(R(Z; x) | X = x) = \mathbb{E}[A^2 | X = x]$ is the within-prompt reward variance, and the second term is the expected squared norm of the score function (the gradient of the log-probability), which captures how sensitive the model's output distribution is to parameter changes at prompt $x$ .

What this inequality means: the task gradient — the component of the update that carries information about which trajectories are better or worse for this specific prompt — cannot be arbitrarily large when the reward variance is small. If all $G$ trajectories for a prompt receive identical rewards (RV $\approx 0$ ), the advantage estimates collapse to zero and the task gradient is provably near zero, regardless of how complex or interesting the individual trajectories are. The bound is tight in the sense that it uses only the Cauchy-Schwarz inequality without further assumptions; the actual task gradient can be even smaller than the bound if the score function and advantage are poorly aligned.

Why this matters for template collapse: when the task gradient is weak, the total parameter update is dominated by the regularization gradient, which the paper shows is flat across reward-variance buckets (Figure 3). The regularization gradient — from KL divergence penalty $-\lambda_{\text{KL}} D_{\text{KL}}(\pi_\theta \| \pi_{\text{ref}})$ and entropy bonus $+\lambda_H H(\pi_\theta)$ — applies uniform contraction to every reasoning chain regardless of which prompt it came from or what reward it received. This regularization is inherently input-agnostic: it pushes the policy toward the reference distribution and encourages diversity, but it does not discriminate between prompts. When the regularization gradient outweighs the task gradient, each update step moves the policy parameters in directions that are common across all prompts, systematically erasing the prompt-specific differences in the reasoning distribution — which is exactly what drives $I(X; Z)$ to zero.

The three-noise decomposition (Table 2). The paper further decomposes the total gradient into three components:

$g_{\text{total}}(x) = g_{\text{signal}}(x) + g_{\text{task-noise}}(x) + g_{\text{reg}}(x)$

where:

$g_{\text{signal}}(x)$ is the gradient from meaningful reward differences across same-prompt trajectories — this is the "signal" that pushes reasoning toward input-dependent patterns. It exists at the prompt level and is not directly controllable (it depends on the environment and the policy's current behavior).
$g_{\text{task-noise}}(x)$ is the gradient from sampling and environment stochasticity — irreducible variance in the advantage estimates that does not carry task-discriminative information. Also at the prompt level.
$g_{\text{reg}}(x)$ is the gradient from KL and entropy regularization — uniform per-chain contraction independent of input. At the chain level and directly controllable through the coefficients $\lambda_{\text{KL}}$ and $\lambda_{\text{ent}}$ .

The SNR definition. For prompt $x$ , the signal-to-noise ratio is:

$\text{SNR}(x) = \frac{\|g_{\text{signal}}(x)\|}{\|g_{\text{task-noise}}(x)\| + \|g_{\text{reg}}(x)\|}$

When SNR is high, the update is dominated by task-discriminative signal — the model learns to produce different reasoning for different prompts. When SNR is low (because reward variance is small, weakening $g_{\text{signal}}$ , while $g_{\text{reg}}$ remains constant), the update is dominated by input-agnostic noise — the model learns to produce reasoning that satisfies regularization constraints but ignores prompt-specific requirements.

The critical practical consequence: in a training batch, not all prompts have the same SNR. Some prompts produce trajectories with clearly distinguishable rewards (high RV → high SNR), generating useful updates that strengthen input-dependent reasoning. Other prompts produce trajectories with nearly identical rewards (low RV → low SNR), generating updates that are dominated by regularization and actually weaken input dependence. The paper's central intervention — SNR-Aware Filtering — is to identify and retain only the high-SNR prompts for each parameter update, removing the low-SNR prompts whose updates would be counterproductive.

Formalization of the SNR upper bound (Appendix H.4). For the $G$ -sample Monte Carlo gradient estimator $\hat{g}_{\text{task}}(x) = \frac{1}{G} \sum_{k=1}^G A_k s_k$ , the paper proves that the estimator's signal-to-noise ratio is upper-bounded by:

$\text{SNR}(x) \leq \sqrt{G} \cdot \frac{\sqrt{\text{RV}(x)}}{\sigma(x)}$

where $\sigma^2(x)$ is the variance of the reward noise (the irreducible stochasticity in the reward function that is not explained by the trajectory). When $\sigma(x)$ is large relative to $\sqrt{\text{RV}(x)}$ , even $G$ samples cannot produce a reliable gradient estimate — a formal statement of why low-RV prompts are fundamentally limited in their ability to provide useful training signal, not just empirically noisy.

Connection to input dependence. Appendix K formalizes the link between low-SNR updates and declining $I(X; Z)$ . The dominance ratio $\rho(x) = \frac{\|g_{\text{reg}}(x)\|}{\|g_{\text{task}}(x)\| + \|g_{\text{reg}}(x)\|}$ — the fraction of the total update attributable to regularization — admits the lower bound:

$\rho(x) \geq \frac{\|g_{\text{reg}}(x)\|}{\|g_{\text{reg}}(x)\| + \sqrt{\text{RV}(x)} \cdot C}$

where $C = \sqrt{\mathbb{E}[\|\nabla_\theta \log \pi_\theta\|^2 \mid x]}$ . Smaller RV(x) implies larger $\rho(x)$ , meaning the total update is more strongly shaped by reward-agnostic regularizers on low-RV prompts. Since these regularizers are input-agnostic, sustained training with $\rho(x)$ close to 1 drives $I(X; Z)$ toward zero — the mechanism of template collapse.

SNR-Aware Filtering: The Intervention

Given the SNR mechanism, the natural intervention is to remove low-SNR prompts from each training update, concentrating the parameter gradient on prompts where task-discriminative signal dominates regularization noise. The paper implements this as SNR-Aware Filtering.

Step 1: Estimate per-prompt reward variance. For a training batch with $P$ prompts, each having $G$ trajectories, compute:

$\widehat{\text{Var}}(R | X = x_i) = \frac{1}{G-1} \sum_{g=1}^G \left( R_g(x_i) - \bar{R}(x_i) \right)^2, \quad \bar{R}(x_i) = \frac{1}{G} \sum_{g=1}^G R_g(x_i)$

where $R_g(x_i)$ is the episode return of the $g$ -th trajectory from prompt $x_i$ . This requires $G \geq 2$ trajectories per prompt — a group sampling setup. The paper's default configuration uses $P = 8, G = 16$ (128 total trajectories per iteration), but Table 5 sweeps alternative ratios.

Why reward variance specifically: within-prompt reward variance is a lightweight proxy for SNR because (i) it is directly computable from rollout data with no additional models or inference passes, (ii) it correlates with task gradient magnitude (Theorem H.2) and upper-bounds the gradient estimator's SNR (Theorem H.3), and (iii) it captures the specific condition — indistinguishable trajectories — that causes task gradient collapse. Alternative filtering metrics (entropy, entropy variance, reward sum, response length) are evaluated in Appendix C (Table 9) and found to be less reliable: entropy-based and length-based filters "either suppress MI or fail to prevent collapse," while reward variance "consistently achieves strong task performance while better preserving MI."

Step 2: Rank prompts by reward variance and apply top- $p$ selection. Given keep rate $\rho \in (0, 1]$ , rank the $P$ prompts in descending order of $\widehat{\text{Var}}(R|X = x_i)$ . Let $\sigma$ be the permutation such that:

$\widehat{\text{Var}}(R|X = x_{\sigma(1)}) \geq \widehat{\text{Var}}(R|X = x_{\sigma(2)}) \geq \cdots \geq \widehat{\text{Var}}(R|X = x_{\sigma(P)})$

Define the selection threshold as the cumulative variance mass:

$\tau = \rho \sum_{i=1}^P \widehat{\text{Var}}(R|X = x_i)$

Then retain the smallest prefix of the sorted prompts whose cumulative variance reaches $\tau$ :

$k^* = \min\left\{ k : \sum_{j=1}^k \widehat{\text{Var}}(R|X = x_{\sigma(j)}) \geq \tau \right\}, \quad S = \{\sigma(1), \ldots, \sigma(k^*)\}$

where $S$ is the set of kept prompt indices and $k^*$ is the number of kept prompts (which varies per iteration depending on the variance distribution).

What this computes: the top- $p$ procedure adaptively selects prompts to retain based on the shape of the variance distribution. If most prompts have high variance, $k^*$ will be close to $P$ — little filtering occurs because most prompts carry useful signal. If many prompts have near-zero variance, $k^*$ may be much smaller than $P$ or even zero (if the cumulative variance threshold cannot be reached with the available prompts) — heavy filtering occurs, potentially rejecting the entire batch when signal quality is uniformly poor. This adaptivity is the key difference from top- $k$ filtering, which "retains a fixed fraction regardless of signal quality, potentially including low-quality updates that dilute the training signal" (Section 4.3).

Step 3: Compute the filtered objective. The standard RL objective (e.g., PPO) applies to all prompts in the batch. The filtered objective restricts the loss computation to the kept prompts:

$\mathcal{L}_\rho(\theta) = \frac{1}{k^*} \sum_{i \in S} \sum_{j \in \mathcal{B}_i} L_\theta(\xi_j)$

where $\mathcal{B}_i$ is the set of trajectory samples from prompt $i$ , $L_\theta(\xi_j)$ is the per-sample RL loss (policy gradient + value loss + regularization), and the outer sum is only over kept prompts. The normalization by $k^*$ (rather than the original $P$ ) ensures the effective learning rate per kept sample remains comparable to the unfiltered setting.

Why top- $p$ over alternatives:

Top- $k$ filtering Keeps exactly $\lfloor \rho P \rfloor$ prompts regardless of variance distribution. As Section 4.3 notes, this "can be less adaptive: when most prompts have near-zero variance, top-k still keeps the highest-variance subset even if all retained prompts carry weak signal." Top- $p$ can reject the entire batch if the cumulative variance threshold cannot be reached.
Min- $p$ filtering Keeps all prompts whose variance exceeds a fraction $p$ of the maximum variance in the batch. This is highly adaptive but "potentially unstable when the maximum variance fluctuates" (Appendix G.1). The paper includes it as a variant in sampling strategy ablations (Table 9) but does not evaluate it as a primary filtering strategy in the main experiments.
Trajectory-level filtering (Section 5.2, Table 7) retains all prompts but sub-selects trajectories within each prompt (top-8 and bottom-8 by reward). This improves over no filtering but underperforms prompt-level SNR-Aware Filtering because "within a naturally low-RV prompt, forcing within-prompt variance by sub-selecting trajectories amplifies noise." The key advantage of prompt-level filtering is that it selects prompts that naturally produce discriminative signals, rather than artificially creating variance within low-signal prompts.

The default keep rate. The paper uses $\rho = 0.9$ as the default in most experiments (Figure 5, Table 5). This means the filter retains prompts whose cumulative variance forms the top 90% of total batch variance — a relatively mild filter that primarily removes prompts with extremely low or zero reward variance. The paper acknowledges that "the kept mass requires per-task tuning" (Section 7, Limitations) but demonstrates that $\rho = 0.9$ is effective across the diverse set of environments tested.

Computational overhead. SNR-Aware Filtering requires $G \geq 2$ trajectories per prompt to estimate variance, which means the total rollout budget ( $K = P \times G = 128$ trajectories) must be partitioned differently than a standard $128 \times 1$ configuration (one trajectory each for 128 unique prompts). However, the rollout cost is identical — generating 128 trajectories costs the same whether they are organized as $128 \times 1$ or $8 \times 16$ . Table 5 shows that the variance computation itself "adds <0.1% of iteration time," and filtering actually reduces per-step time by 26–41% because fewer prompt groups enter gradient computation. The VRAM overhead is negligible (<0.5 GB difference from unfiltered).

Interaction with existing stabilization. SNR-Aware Filtering is explicitly designed as a complementary intervention to KL and entropy regularization, not a replacement. Section 5.2 states: "SNR-Aware Filtering is complementary: it selects high-signal prompts at each iteration, directly boosting the fraction of task-discriminative gradient in each update. This acts as a signal-enhancement mechanism rather than a noise-control mechanism." While KL tuning and entropy tuning primarily affect $\|g_{\text{reg}}\|$ (the noise floor), SNR-Aware Filtering affects which prompts contribute to the update at all, directly manipulating the signal-to-noise ratio of the aggregate batch gradient.

Integration with DAPO and Dr. GRPO. The paper notes that DAPO "itself also includes a filtering/acceptance step" and "can be interpreted as a special case of our framework where the selection is fixed (equivalently, a top- $P$ filter with $P \to 1.0$ ), while our SNR-Aware Filtering provides an explicit, tunable SNR knob via the keep rate $\rho$ " (Section 4.3). The experiments in Table 4 apply SNR-Aware Filtering on top of DAPO's built-in mechanisms, finding additional gains (+5.1% on Sokoban, +2.1% on FrozenLake, +2.8% on MetaMathQA, +1.6% on Countdown), demonstrating that even state-of-the-art stabilization methods leave SNR-imbalance headroom that filtering can address.

When SNR-Aware Filtering helps most (Section 5.2, Table 8). The paper identifies a practical diagnostic for whether filtering will be effective: the ratio $\text{Std}(\text{RV}) / \text{Mean}(\text{RV})$ , computable from a single rollout batch. When this ratio is high (>1.0), the per-prompt variance distribution is bimodal — some prompts clearly carry signal while others are purely noise — and filtering cleanly separates them. When the ratio is low (<0.5), all prompts carry similar variance and filtering discards data uniformly with no benefit. For example, Sokoban with 3B model shows a ratio of 1.16 and a filtering gain of +3.2%, while FrozenLake GRPO shows a ratio of 0.33 and a filtering loss of −5.0%, because "filtering discards data uniformly" without separating signal from noise in that setting.

Supporting Theoretical Results: Template Mixing and Entropy Decomposition

The paper provides several additional theoretical results in its appendices that support the SNR mechanism and the choice of MI as a diagnostic:

Template mixing lemma (Appendix I). If the policy's conditional distribution is contaminated by a prompt-independent "template" distribution $q(z)$ with mixing weight $\alpha$ :

$p_\alpha(z|x) = (1 - \alpha) p(z|x) + \alpha q(z)$

then the mutual information contracts by at least a factor of $(1 - \alpha)$ :

$I_\alpha(X; Z) \leq (1 - \alpha) I(X; Z)$

What this means: even partial drift toward a shared template — the policy mixing in a prompt-agnostic distribution alongside its input-conditioned distribution — directly reduces mutual information. This formalizes the intuition that template collapse corresponds to the mixing weight $\alpha$ increasing over training, with $I(X; Z)$ declining proportionally.

Entropy decomposition theorem (Appendix M). The change in mutual information between a reference policy $\pi_0$ and the current policy $\pi_\theta$ decomposes as:

$\Delta I = I_\theta(X; Z) - I_0(X; Z) = (H_\theta(Z) - H_0(Z)) - (H_\theta(Z|X) - H_0(Z|X)) = \Delta_{\text{marg}} - \Delta_{\text{in}}$

where $\Delta_{\text{marg}}$ is the change in marginal entropy (overall reasoning diversity) and $\Delta_{\text{in}}$ is the change in conditional entropy (within-input diversity). If an intervention increases within-input diversity more than it increases marginal diversity ( $\Delta_{\text{in}} > \Delta_{\text{marg}}$ ), mutual information necessarily decreases( $\Delta I < 0$ ).

What this means for entropy regularization: an entropy bonus directly increases $H(Z|X)$ (within-input diversity) because it rewards the policy for producing diverse outputs on each input. However, it does not explicitly encourage cross-prompt separation that would increase $H(Z)$ (marginal diversity). Therefore, it is "plausible that $\Delta_{\text{in}}$ exceeds $\Delta_{\text{marg}}$ , in which case... $I_\theta(X; Z)$ decreases" (Appendix M). An entropy bonus can thus actively contribute to template collapse while making the surface metric (entropy itself) look healthier — a direct formalization of why entropy-based monitoring is misleading.

KL-closeness and MI preservation (Appendix L). If the current policy stays within $\varepsilon$ KL divergence of a reference policy (uniformly across prompts), then the mutual information also stays within $f(\varepsilon)$ of the reference policy's mutual information, where $f(\varepsilon) \to 0$ as $\varepsilon \to 0$ .

What this means for KL constraints: KL regularization can preserve input dependence by anchoring the policy near a reference that has non-trivial $I(X; Z)$ , but it cannot increase $I(X; Z)$ that is being actively eroded by other forces. This explains why KL tuning alone is insufficient to prevent template collapse (as shown in Section 5.1, Figure 13) — it is a constraint on the rate of change, not a counterforce to the SNR imbalance that drives the change.

GRPO normalization amplifies noise at low RV (Appendix N). GRPO divides advantages by $\sqrt{\text{RV}(x)}$ , which causes the gradient estimator's variance to scale as $\text{RV}(x)^{-1}$ . For prompts with small reward variance, this normalization inflates gradient noise rather than reducing it — a formal explanation for why GRPO's normalization does not resolve the SNR problem and can actually make it worse in low-RV regimes.

Design Choices and Their Justifications

Decomposing reasoning quality into $I(X; Z)$ and $H(Z|X)$ rather than using a single metric: this is the paper's foundational insight. A single diversity metric (e.g., BLEU, embedding variance, entropy) conflates within-input variability with cross-input structure. The decomposition makes explicit that these can move independently, which is both the diagnosis of template collapse and the explanation for why prior diagnostics failed.
Using in-batch cross-scoring with the model's own log-probabilities for MI estimation rather than external models or generation-based metrics: this makes the proxy computable online with negligible overhead (no additional inference passes beyond standard training forward passes). External models would introduce distribution shift (the scoring model might not recognize the training model's reasoning patterns); generation-based metrics would require multiple sampling passes per trace, which is expensive.
The Retrieval-Acc proxy with chance level $1/P$ : provides an absolute, calibration-free baseline. Unlike continuous MI estimates where "zero" is not clearly defined in practice, Retrieval-Acc has a known lower bound under complete template collapse (chance-level retrieval) and a known upper bound under perfect input dependence (1.0). This makes it interpretable across different models, tasks, and training stages.
Z-score normalization with EMA in the continuous proxy: removes the effect of overall distribution shifts in log-probability magnitudes while the EMA (with $\alpha = 0.9$ ) prevents single-batch outliers from dominating. Without normalization, the proxy would conflate changes in input dependence with changes in the model's overall confidence or calibration.
Top- $p$ (nucleus-style) filtering over top- $k$ : the adaptive keep count automatically adjusts to the variance distribution. When many prompts carry near-zero variance, top- $p$ can reject the entire batch; top- $k$ would always keep exactly $k$ prompts regardless of signal quality. The paper empirically demonstrates this advantage in Figure 6.
Prompt-level filtering over trajectory-level filtering: selecting whole prompts that naturally produce discriminative signals is more effective than artificially creating variance within low-signal prompts by sub-selecting trajectories. The paper provides empirical evidence for this in Table 7: trajectory-level filtering improves over no filtering (16.8% vs. 12.9% on Sokoban) but prompt-level filtering provides larger gains (23.6%).
Reward variance as the filtering metric over entropy, entropy variance, reward sum, or response length: Appendix C (Table 9) shows that entropy-based filters "either suppress MI or fail to prevent collapse," length-based filters produce runs that collapse despite high Retrieval-Acc, and reward-sum filtering shows unstable training. Only reward variance consistently produces both high task performance and high MI proxy values.
Keeping two-fold cross-validation and early stopping for robustness: the paper applies standard RL training stability measures but adds reward-variance collapse detection — stopping training if reward variance drops below 10% of the baseline variance for 5 consecutive iterations — as an additional safeguard against template collapse in extremely low-signal regimes.

4. Key Insights and Innovations

Innovation 1: Decomposing Reasoning Quality into Orthogonal Axes — Input Dependence vs. Within-Input Diversity

The paper's deepest conceptual contribution is the recognition that reasoning quality during RL training is not a single dimension that can be monitored with a single metric, but rather decomposes into two independent axes that can move in opposite directions. This reframing — treating $I(X; Z)$ (mutual information between inputs and reasoning) and $H(Z|X)$ (conditional entropy of reasoning given input) as separable, independently trackable quantities — is what makes template collapse diagnosable at all.

What the field did before. Prior work on collapse in LLM training treated reasoning degradation as fundamentally a problem of diversity loss. The dominant monitoring paradigm — tracking reward for outcome stability and entropy for process stability (Ouyang et al., 2022; Xu et al., 2025; Schulman et al., 2017) — implicitly assumes that maintaining high conditional entropy is sufficient to prevent reasoning degradation. This assumption is baked into the standard RLHF and agent RL toolchain: entropy regularization is added specifically to prevent policy collapse, and stable entropy is interpreted as evidence that reasoning remains healthy. The information-theoretic identity $H(Z) = I(X; Z) + H(Z|X)$ is a textbook fact, but prior work had not operationalized it as a training diagnostic, nor recognized that $I(X; Z)$ and $H(Z|X)$ can move independently under standard RL training dynamics.

What makes this distinctive. The paper's move is not to introduce new mathematics — the Shannon decomposition is standard — but to recognize that the independence of these two terms has direct, previously overlooked consequences for RL training. A model can sustain or even increase $H(Z|X)$ (producing diverse outputs within each prompt) while $I(X; Z)$ collapses to near zero (the outputs become equally compatible with any prompt). This is not a hypothetical edge case: the paper demonstrates it as a systematic outcome of low-SNR gradient updates (Section 3, the SNR mechanism). The critical insight is that entropy-based monitoring answers the question "is the model producing varied outputs?" but the question that actually matters for reasoning quality is "are the outputs systematically different across inputs?" — and these two questions can have opposite answers.

Significance beyond performance. This decomposition is a diagnostic reframing, not a method improvement. It does not directly improve any metric, but it changes what practitioners should look at and what they should worry about. The paper's finding that MI-family metrics achieve positive Spearman correlations with final performance (+0.39 for Trajectory MI-ZScore) while entropy metrics show near-zero or negative correlations (−0.11 to −0.14, Figure 8) is not a claim that MI is a "better metric" in a narrow sense — it is evidence that the field has been systematically monitoring the wrong quantity. Entropy points in the wrong direction as a predictor of task success because it tracks within-input diversity, which can be sustained by input-agnostic templates, rather than cross-input distinguishability, which is what actually enables reasoning to adapt to new problems.

A fundamental shift, not an incremental refinement. This is a fundamental reorientation of what "reasoning quality" means in the context of RL training. Prior work conflated diversity with quality; this paper establishes that diversity is necessary but not sufficient, and that the missing dimension — input dependence — is the one that collapses first and most reliably predicts downstream performance. The operationalization via lightweight in-batch cross-scoring (the MI proxy family) makes this reframing practically deployable without external models or additional inference, converting an abstract information-theoretic insight into a concrete training monitor.

Innovation 2: The SNR Mechanism as a Causal Explanation for Template Collapse

The paper's second major contribution is a mechanistic theory of why template collapse occurs — not just that it does, but what specific property of the RL update causes it. The SNR framework identifies the interaction between within-prompt reward variance and input-agnostic regularization as the gradient-level driver of declining $I(X; Z)$ .

What the field did before. Prior explanations for collapse in LLM training invoked broad, non-specific causes. The model collapse literature (Gerstgrasser et al., 2024; Shumailov et al., 2024) attributes degradation to distributional feedback loops — models training on their own outputs progressively narrow the effective data distribution. The RLHF literature identifies reward over-optimization (Moskovitz et al., 2023) and mode collapse (O'Mahony et al., 2024) as risks, but generally attributes them to aggressive optimization against imperfect reward models. In the agent RL setting, works like Wei et al. (2025) and Yun et al. (2025) documented reasoning collapse empirically but did not isolate the causal mechanism. The default assumption, implicit in the widespread use of entropy and KL regularization, was that collapse is primarily a problem of the policy becoming too deterministic or drifting too far from its reference distribution — and that stronger regularization is the solution.

What makes this distinctive. The SNR mechanism makes a specific, falsifiable causal claim: template collapse occurs because low within-prompt reward variance weakens the task-discriminative gradient component while input-agnostic regularization components (KL, entropy) remain constant, causing their relative influence on the parameter update to grow. This is not a claim about reward quality (reward hacking, reward misspecification) or about the policy becoming too narrow (mode collapse), but about the balance of forces within each gradient update. The key insight is that regularization — usually viewed as a benign or protective force — can become actively harmful when it dominates the update, because it applies uniform contraction across all inputs and systematically erases the cross-input differences that constitute input-dependent reasoning.

The paper supports this causal claim with converging evidence: the gradient norm decomposition (Figure 3) showing task gradient scaling with reward variance while regularization gradient remains flat; the quartile ablation (Table 6) showing that training exclusively on low-RV prompts degrades both MI and performance; the controlled noise injection experiment (Figure 9) showing that increasing environment stochasticity predictably weakens MI; and the formal bounds (Appendix H) proving that task gradient magnitude is upper-bounded by $\sqrt{\text{RV}(x)}$ and that the gradient estimator's SNR is bounded by $\sqrt{G} \cdot \sqrt{\text{RV}(x)} / \sigma(x)$ . This is a rare example in the LLM training literature of a mechanistic theory that makes directional predictions verified through intervention, not just correlation.

Significance beyond performance. The SNR mechanism has an important negative implication: it explains why standard stabilization techniques (KL tuning, entropy tuning) cannot prevent template collapse, even when optimally tuned. These techniques primarily control $\|g_{\text{reg}}\|$ — the noise floor — but do not affect $\|g_{\text{signal}}\|$ , which is determined by reward variance. Reducing $\|g_{\text{reg}}\|$ (e.g., by lowering the KL coefficient) can slow the rate of collapse but cannot reverse it if the underlying SNR imbalance persists. This is not a limitation of specific hyperparameter choices but a structural property of the gradient decomposition. Section 5.1 (Figure 13) demonstrates this empirically: sweeping entropy and KL coefficients moves the training trajectory primarily along the $H(Z|X)$ axis without significantly shifting $I(X; Z)$ , while SNR-Aware Filtering — which directly affects which prompts contribute signal — moves the trajectory into the high- $I(X; Z)$ , high-performance regime.

A theoretical advance with immediate practical implications. The SNR mechanism is both explanatory and actionable. It explains why the field's contradictory findings about reasoning collapse (some works finding it, others not) might be attributable to differences in reward variance across environments and tasks. It predicts when filtering will help: when the ratio $\text{Std}(\text{RV}) / \text{Mean}(\text{RV})$ is high, indicating bimodal variance distributions where filtering can separate signal from noise (Table 8). And it suggests what not to do: increasing entropy regularization to "fix" template collapse is counterproductive, because it increases $H(Z|X)$ without restoring $I(X; Z)$ , potentially accelerating the very degradation it aims to prevent (Appendix M).

Innovation 3: Mutual Information as a Training Diagnostic That Outperforms and Supersedes Entropy

The paper's third contribution is empirical and methodological: the demonstration that MI-based proxies consistently and substantially outperform entropy-based metrics as predictors of task performance across training runs, and that entropy can be actively misleading.

What the field did before. Entropy (or proxies thereof, such as token-level diversity measures, n-gram statistics, or embedding-based dispersion) has been the standard process-level metric for monitoring LLM training health. In the RLHF and agent RL literature, entropy is both a training objective (the entropy bonus) and a monitoring signal (declining entropy is interpreted as collapse, stable entropy as healthy training). The assumption is that entropy captures what matters about reasoning quality — that if the model is generating diverse outputs, its reasoning is functioning correctly. This assumption is so ingrained that works proposing alternatives to entropy (e.g., diversity-aware policy optimization, Yao et al., 2025; guided thought reinforcement, Wei et al., 2025) still operate within the diversity-as-quality framework, attempting to measure or preserve different kinds of diversity rather than questioning whether diversity is the right target.

What makes this distinctive. The paper does not propose MI as a slightly better metric — it demonstrates that entropy is misleading in direction. Across a sweep of interventions (entropy coefficient, KL coefficient, SNR-Aware Filtering keep rate), entropy-family metrics show Spearman correlations with final performance that are near zero or slightly negative (−0.11 to −0.14, Figure 8), while MI-family metrics show consistent positive correlations (up to +0.39). This is not a matter of MI being "more sensitive" or "more robust" — it means that a practitioner monitoring entropy would draw the opposite conclusion from a practitioner monitoring MI. A training run with high entropy might look healthy but actually have low input dependence and poor eventual performance; a run with moderate entropy but high MI would look worse by the traditional metric but actually be training more effectively.

The paper's diagnostic framework goes beyond correlation. By tracking both MI and entropy simultaneously, it enables practitioners to classify training regimes (Figure 1): high-MI, high-entropy is the desired Diverse Reasoning regime; high-entropy, low-MI is Template Collapse (the invisible failure mode); low-entropy, high-MI is Compressed Reasoning (deterministic but input-faithful); and low-entropy, low-MI is complete Low-Entropy Collapse. This classification provides a richer picture of training health than any single metric could, and the paper demonstrates that different interventions move the training trajectory through this space along different axes (Figure 13): entropy tuning moves primarily horizontally (changing $H(Z|X)$ ), while SNR-Aware Filtering moves diagonally (increasing both $I(X; Z)$ and performance).

Why this matters beyond this paper. If the paper's findings generalize — and the breadth of environments, algorithms, and model scales tested in Table 4 suggests they may — then MI-based monitoring should become standard practice for agent RL training, replacing or supplementing entropy as the primary process-level health indicator. This is a methodological contribution that changes how experiments are run and evaluated, not just what algorithm is used. The fact that the MI proxy is computable from standard rollout data with no additional models or inference passes (in-batch cross-scoring, Section 2.3) removes the practical barrier to adoption.

An incremental shift with fundamental implications. The MI proxy itself is not algorithmically novel — in-batch cross-scoring for retrieval is a standard technique. What is novel is the recognition that this specific computation answers the right question (input dependence) rather than the wrong question (within-input diversity) for understanding reasoning quality under RL optimization. The paper's contribution is less the proxy design and more the demonstration that switching from entropy to MI as a monitoring target changes the entire interpretation of what constitutes healthy training.

Innovation 4: SNR-Aware Filtering as a Signal-Enhancement Intervention Orthogonal to Regularization

The paper's fourth contribution is a training intervention that targets the causal mechanism of template collapse at a different level than existing stabilization methods. SNR-Aware Filtering operates by selecting which data contributes to each update rather than by constraining the update's magnitude or direction.

What the field did before. Existing stabilization methods for LLM RL operate primarily by controlling the update itself: KL constraints limit how far the policy can move from a reference (Schulman et al., 2017; Stiennon et al., 2022), entropy regularization encourages the policy to remain diverse (Ouyang et al., 2022), clipping and trust regions prevent large parameter changes (Schulman et al., 2017), and gradient normalization reduces variance (GRPO, Shao et al., 2024). These methods all apply to the update after the data has been collected and the gradient has been computed. Some recent methods incorporate data selection — DAPO's filtering step (Yu et al., 2025) and rejection sampling approaches — but these typically filter based on reward magnitude or advantage sign, not on a property (reward variance) that specifically predicts gradient signal quality.

What makes this distinctive. SNR-Aware Filtering operates before the gradient is computed, at the data selection stage, and uses a criterion (within-prompt reward variance) that is specifically motivated by the SNR mechanism rather than by reward magnitude or policy performance. This is a different class of intervention from regularization — it is not about controlling the update's size or direction but about controlling the composition of the batch that produces the update. The distinction matters because regularization can only reduce the influence of noise; it cannot increase the influence of signal. Filtering can increase the effective SNR of the batch by removing data that would contribute primarily noise.

The paper demonstrates that this distinction has practical consequences. SNR-Aware Filtering improves performance on top of DAPO — an algorithm that already includes its own filtering/acceptance mechanisms (Table 4, +5.1% on Sokoban, +2.1% on FrozenLake, +2.8% on MetaMathQA) — suggesting that DAPO's filtering operates on a different axis (likely selecting based on advantage quality rather than reward variance as an SNR proxy). The paper also shows that trajectory-level filtering (selecting high-reward trajectories within each prompt) helps but underperforms prompt-level filtering (Table 7), because within a naturally low-RV prompt, artificially creating variance by sub-selecting trajectories does not recover genuine task-discriminative signal — it amplifies noise.

Boundary conditions as a feature, not a limitation. The paper explicitly identifies when SNR-Aware Filtering fails: when the ratio $\text{Std}(\text{RV}) / \text{Mean}(\text{RV})$ is near zero (Table 8, FrozenLake GRPO with ratio 0.33 and filtering loss of −5.0%). This is a strength of the contribution because it provides a clear, computable diagnostic for when the method will help versus when it will hurt. Rather than claiming universal benefit, the paper characterizes the conditions under which the SNR mechanism operates and the intervention is effective. This is more valuable to practitioners than a blanket recommendation.

A refinement rather than a paradigm shift, but a well-motivated one. SNR-Aware Filtering does not replace existing stabilization methods — it complements them. The paper's framing of filtering as a "signal-enhancement mechanism rather than a noise-control mechanism" (Section 5.2) makes clear that it operates on a different axis than KL or entropy tuning. The innovation is not in the filtering algorithm itself (top- $p$ selection is standard) but in the criterion (reward variance as SNR proxy) and the theoretical motivation (the SNR mechanism establishing why this specific criterion addresses the root cause of template collapse). This is an incremental algorithmic contribution with a strong mechanistic justification — a combination that is relatively rare in the LLM training literature, where methods are often proposed with empirical motivation alone.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper evaluates on seven diverse environments: Sokoban (procedurally generated grid puzzles with irreversible box-pushing dynamics), FrozenLake (navigation with stochastic transitions and sparse rewards), MetaMathQA (multi-step math QA with diminishing rewards across retries, drawn from the MetaMathQA dataset), Countdown (single-turn arithmetic reasoning from the numbers game), SearchQA (multi-turn search and question-answering from the RLLM framework), WebShop (interactive e-commerce navigation), and DeepCoder (code synthesis from input-output specifications, drawing from PrimeIntellect, TACO, and LiveCodeBench). Evaluations use a fixed set of 512 validation prompts per environment, decoded with temperature $T = 0.5$ using stochastic sampling. For Coreference environments (Sokoban, FrozenLake, MetaMathQA, Countdown), training runs for up to 400 rollout–update iterations with early stopping triggered when (i) reward variance drops below 10% of the baseline variance for 5 consecutive iterations, or (ii) validation success rate remains below 1% for 5 consecutive evaluation checkpoints.
Base model(s). The primary model is Qwen2.5-3B. The paper also tests Qwen2.5-0.5B, Qwen2.5-1.5B, Qwen2.5-7B, Qwen2.5-3B-Instruct, Llama3.2-3B, and the vision-language model Qwen2.5-VL-3B (with both text and image inputs). The choice of Qwen2.5-3B as the primary model reflects a mid-scale representative that is "representative of the capabilities of many contemporary LLMs" (Section 4.1) while being small enough for extensive ablation sweeps. The wider model spread (0.5B to 7B, plus a different family and a vision-language variant) tests whether findings generalize across scales, types, and modalities.
Metrics. The primary performance metric is task success rate — the fraction of validation prompts for which the agent achieves its goal (e.g., solving the puzzle, reaching the target, answering correctly). The paper also tracks mutual information proxy values (Retrieval-Acc, MI-ZScore-EMA, MI-Est, and variants from Table 1) computed from in-batch cross-scoring of reasoning traces against prompts, measuring input dependence $I(X; Z)$ . Conditional entropy $H(Z|X)$ and marginal entropy $H(Z)$ are logged in parallel via the scorer-based proxies: $H(Z|X) \approx -\frac{1}{PG} \sum_{i,k} \text{matched}_{i,k}$ and $H(Z) \approx -\frac{1}{PG} \sum_{i,k} \text{marginal}_{i,k}$ . Reward variance $\widehat{\text{Var}}(R|X)$ is computed per prompt as the sample variance of episode returns across the $G$ trajectories, used both as the filtering criterion and as a diagnostic. For the gradient analysis (Figure 3), task gradient norm ( $\|g_{\text{task}}\|$ ) and regularization gradient norm ( $\|g_{\text{reg}}\|$ ) are measured per reward-variance bucket. The paper also computes Spearman correlations between candidate diagnostics (MI proxies, entropy proxies) and final task performance across training runs (Figure 8).
Baselines. The paper evaluates SNR-Aware Filtering against a suite of RL algorithms and intervention strategies:
- No filtering baseline: Standard RL training (PPO, DAPO, GRPO, Dr. GRPO) without any SNR-based prompt filtering.
- PPO (Schulman et al., 2017): Proximal Policy Optimization with GAE ( $\gamma = 1.0, \lambda = 1.0$ ), asymmetric clipping ( $\epsilon_{\text{low}} = 0.2, \epsilon_{\text{high}} = 0.28$ ), and entropy regularization coefficient $\beta = 0.001$ .
- DAPO (Yu et al., 2025): A recent algorithm that includes its own filtering/acceptance step targeting stable training.
- GRPO (Shao et al., 2024): Group Relative Policy Optimization with advantage normalization.
- Dr. GRPO (Liu et al., 2025): A critical variant of GRPO that targets collapse-like failure modes.
- Entropy tuning: Sweeping the entropy regularization coefficient to control within-input diversity.
- KL tuning: Sweeping the KL divergence penalty coefficient to control deviation from the reference policy.
- Top-k filtering: A fixed-count alternative to top- $p$ that retains exactly $\lfloor \rho P \rfloor$ prompts regardless of variance distribution.
- Trajectory-level filtering: Retaining all prompts but sub-selecting the top-8 and bottom-8 trajectories per prompt by reward.
Generation budget / compute accounting. Each training iteration collects $K = P \times G = 128$ trajectories per environment. The default configuration uses prompt batch size $P = 8$ with group size $G = 16$ trajectories per prompt. When SNR-Aware Filtering is applied, the rollout budget remains 128 trajectories — only the organization into prompt groups varies. The paper sweeps alternative $(P, G)$ configurations in Table 5 while holding total trajectories fixed at 128: $128 \times 1$ (no filtering possible, since $G \geq 2$ is needed for variance), $64 \times 2$ , $32 \times 4$ , and $8 \times 16$ . Under filtering with keep rate $\rho = 0.9$ , fewer prompt groups enter gradient computation, reducing per-step wall-clock time by 26–41% compared to the unfiltered baseline. The variance computation itself adds <0.1% of iteration time. The actor learning rate is $1 \times 10^{-6}$ and critic learning rate is $1 \times 10^{-5}$ . Effective minibatch size scales with $\rho$ to keep optimization step size comparable.
Cross-validation / statistical protocol. The paper uses a fixed train/validation split: up to 400 training iterations on the training set, with evaluation on 512 held-out validation prompts per environment. Early stopping criteria are applied uniformly across all runs. For the Spearman correlation analysis (Figure 8), the paper computes correlations across runs with varying entropy regularization strength, KL constraint strength, and SNR-Aware Filtering keep rate. The MI proxy trajectories in Figures 5 and 14 track metrics continuously throughout training, showing the evolution of input dependence alongside task performance. The reward-variance quartile ablation (Table 6) uses a controlled intervention: prompts are sorted by within-prompt RV, divided into four equal-sized quartiles, and four separate training runs are conducted — each updating exclusively on one quartile, with all other settings fixed.

Main Quantitative Results

Template Collapse Occurs Reliably Across Training Configurations (Section 4.2)

Headline finding: Across all training configurations, RL-trained agents consistently develop reasoning that is fluent but input-agnostic — $I(X; Z)$ (measured by Retrieval-Acc) declines significantly while $H(Z|X)$ remains elevated, a divergence that entropy-based monitoring cannot detect. This holds without SNR-Aware Filtering and across multiple environments and RL algorithms.

Training dynamics under different interventions (Figure 5). The paper tracks three metrics during training on a representative environment (the figure caption does not specify which; from context it appears to be Sokoban): task success rate, Retrieval-Acc (the discrete MI proxy), and reasoning entropy. Under no filtering:

Retrieval-Acc declines early and sharply while reasoning entropy spikes — the hallmark of template collapse. The MI proxy serves as an early warning signal, degrading significantly before task performance shows visible decline.
With entropy regularization ( $\beta = 0.003$ ), KL regularization ( $\lambda = 0.0003$ ), and top- $p$ SNR-Aware Filtering ( $\rho = 0.9$ ), the MI proxy remains substantially higher throughout training. Top- $p$ filtering best preserves both task performance and reasoning diversity (as measured by sustained Retrieval-Acc and controlled entropy).

MI proxy dynamics across variants (Figure 14, Appendix D.1). Six alternative MI proxy metrics — MI Estimate, MI Z-Score (EMA), Retrieval Above Chance, Retrieval Accuracy @4, Retrieval Accuracy @8, and Conditional Entropy — all exhibit the same qualitative pattern: under no filtering, MI degrades sharply as training progresses; under the three interventions (entropy reg, KL reg, top- $p$ filtering), MI retention is substantially improved, with top- $p$ filtering consistently maintaining the highest proxy values throughout the 200 training steps shown.

Behavioral signature: reasoning length decline (Figure 7). Across eight diverse environments — including spatial agents, logic puzzle agents, visual agents, and math agents drawn from existing evaluations — reasoning length declines monotonically as training progresses. This systematic compression is the behavioral manifestation of template collapse: as agents converge toward reusable templates, they produce shorter, more formulaic reasoning outputs. This complements the MI-based diagnostics by showing that template collapse has a directly observable behavioral signature beyond the information-theoretic metrics.

SNR-Aware Filtering Consistently Improves Performance (Section 4.3)

Headline findings: SNR-Aware Filtering produces consistent improvements in both input dependence (MI proxy) and task performance across all four core environments, multiple RL algorithms, model scales, types, and modalities. The gains are achieved at no additional compute cost — the total rollout budget is fixed at 128 trajectories per iteration, and filtering actually reduces per-step wall-clock time.

Comparison of filtering strategies (Figure 6). Across four environments — Sokoban, FrozenLake, MetaMathQA, and Countdown — three filtering strategies are compared at the same training budget:

Top- $p$ (nucleus-style) filtering consistently achieves the highest success rates throughout training. The adaptive selection automatically adjusts to the variance distribution, rejecting entire batches when most prompts carry weak signal.
Top- $k$ filtering (fixed-count) underperforms top- $p$ because it retains a fixed fraction of prompts regardless of signal quality — potentially including low-quality updates when signal is uniformly poor.
No filtering shows the weakest performance across all environments, confirming that indiscriminate updates on all prompts systematically degrades learning.

Comprehensive results matrix (Table 4). The paper evaluates SNR-Aware Filtering across four tasks (Sokoban, FrozenLake, MetaMathQA, Countdown) under a wide range of experimental variants. Each cell reports baseline peak performance with the filtering delta in parentheses. Key results:

PPO baseline (Qwen2.5-3B):

Sokoban: 12.9% (+16.0) — filtering more than doubles success rate
FrozenLake: 67.0% (+10.9)
MetaMathQA: 92.6% (+0.6) — near-ceiling on this task
Countdown: 97.9% (+0.0) — at ceiling
Average: 67.6% (+6.9)

Algorithm variants (all Qwen2.5-3B):

DAPO: Average 67.4% (+2.9) — filtering provides gains on top of DAPO's built-in stabilization
GRPO: Average 67.5% (+3.7) — despite GRPO's advantage normalization which should address variance, filtering still helps
Dr. GRPO: Average 55.8% (+0.8) — smallest gain but still positive, and notably Dr. GRPO's no-filter baseline on FrozenLake is only 23.2% (substantially below other algorithms), suggesting the algorithm has other failure modes that filtering cannot fully address

Model scale (PPO, Qwen2.5 variants):

0.5B: Average 14.0% (+5.5) — the biggest relative gain, with Sokoban jumping from 3.3% to 26.2% (+22.9)
1.5B: Average 47.6% (+4.1) — MetaMathQA gains +7.0, pushing past 87%
3B: As above
7B: Average 77.3% (+4.1) — consistent gains even at the largest scale tested, with MetaMathQA showing the largest improvement (+11.7, from 84.0% to 95.7%)

Model type:

Qwen2.5-3B-Instruct: Average 73.4% (+4.1) — the instruct-tuned variant benefits comparably to the base model
Llama3.2-3B: Average 73.6% (+5.3) — filtering transfers to a different model family, with Sokoban showing the largest gain (+18.8)

Modality (Qwen2.5-VL-3B):

Text input only: Average 34.5% (+29.8) — FrozenLake jumps from 16.0% to 69.5% (+53.5)
Vision input: Average 42.3% (+35.8) — FrozenLake jumps from 19.5% to 79.0% (+59.5)

The vision-language results are particularly striking: SNR-Aware Filtering nearly quadruples success rates on FrozenLake for the VL model under both input modalities, suggesting that multi-modal agents may be especially susceptible to template collapse and especially responsive to signal-quality filtering.

Batch configuration sweep (Table 5). With a fixed total rollout budget of 128 trajectories, the paper sweeps $(P, G)$ configurations on Sokoban (Qwen2.5-3B):

$128 \times 1$ (no filtering possible): 23.6% task performance, 89.8s step time, 201.80 GB VRAM
$64 \times 2$ : No-filter 18.8% vs. filtered ( $\rho = 0.9$ ) 27.3% (+8.6), step time reduced by 29%
$32 \times 4$ : No-filter 24.2% vs. filtered 27.4% (+3.2), step time reduced by 41%
$8 \times 16$ (default): No-filter 15.6% vs. filtered 23.6% (+8.0), step time reduced by 26%

Configurations with $G \geq 4$ and filtering match or outperform the $128 \times 1$ baseline while reducing per-step time. The $8 \times 16$ no-filter baseline performs notably worse than $128 \times 1$ (15.6% vs. 23.6%), confirming that the group-sampling setup without filtering is disadvantaged — the SNR-Aware Filtering recovers and exceeds the performance lost by the repartitioning of the budget. VRAM differences across configurations are negligible (<0.5 GB variation).

MI Diagnoses Collapse Better Than Entropy Across All Interventions (Section 5.1)

Headline finding: MI-family metrics achieve positive Spearman correlations with final task performance (+0.39 for Trajectory MI-ZScore), while entropy-family metrics show near-zero or negative correlations (−0.11 to −0.14, Figure 8). MI therefore predicts performance more reliably than entropy, and entropy can point in the wrong direction.

Spearman correlation analysis (Figure 8). Across training runs with varying entropy regularization strength, KL constraint strength, and Top- $p$ filtering kept mass, the paper computes Spearman correlations between each candidate diagnostic and final task success rate:

MI-family metrics (all positive):
- Trajectory MI-ZScore: +0.39 (highest)
- Retrieval Accuracy: +0.29 (estimated from bar height)
- MI-ZScore-EMA: positive (exact value not quoted, but visible in figure)
Entropy-family metrics (all near-zero or negative):
- Reasoning Entropy: −0.11 to −0.14 range
- Conditional Entropy: similarly negative

The paper states that MI "predicts performance twice as reliably as entropy does" (the +0.39 vs. −0.11 comparison), and that "entropy actually points in the wrong direction."

Training trajectories under different interventions (Figure 13). The paper visualizes how three intervention families move the policy through the (MI, Performance) and (Entropy, Performance) diagnostic spaces. For each setting, two checkpoints are chosen (steps 10 and 400) and connected into a trajectory with arrows pointing to later steps. Color intensity indicates weaker to stronger intervention.

Entropy coefficient sweep: As the entropy coefficient increases from 0 to 0.1, the trajectory in the (MI, Performance) space moves primarily vertically (increasing entropy while MI and performance show limited movement). Entropy tuning shifts $H(Z|X)$ substantially without moving $I(X; Z)$ into the high-performance regime. Pushing entropy too high leads to instability and performance collapse.
KL coefficient sweep: Similar pattern — KL constraint anchors the policy near its reference distribution without boosting input dependence. The trajectory in the (MI, Performance) space shows more limited movement than the entropy sweep.
SNR-Aware Filtering sweep (top- $p$ from 1.0 down to 0.4) : The trajectory traces a monotonic improvement in both MI and task success. As filtering becomes more aggressive (stronger color), the checkpoint moves toward higher MI and higher performance. This is the intervention that successfully pushes the model into the high- $I(X; Z)$ , high-performance regime that entropy and KL tuning cannot reach.

Format validity cannot substitute for content-sensitive diagnostics (Figure 12). The paper tests whether a coarser signal — whether the model's output follows the required format (e.g., proper reasoning/action tags) — could serve as a collapse indicator without the overhead of MI estimation. The result: format validity is largely decoupled from collapse. Runs maintain near-perfect format validity while exhibiting low MI, confirming that structural correctness and semantic input-dependence are separate dimensions. This reinforces the need for content-sensitive diagnostics.

The SNR Mechanism: Causal Evidence and Boundary Conditions (Section 5.2)

Headline finding: The SNR framing makes a concrete causal claim — template collapse is a gradient-level consequence of low reward variance — and the paper stress-tests this with four converging experimental tests. All four support the mechanism.

Quartile ablation provides direct causal evidence (Table 6). On Sokoban (Qwen2.5-3B, $P = 8, G = 16$ ), prompts are sorted by within-prompt reward variance and divided into four quartiles (Q1 = highest RV, Q4 = lowest RV). Four separate training runs are conducted, each updating on only one quartile (keeping 25% of prompts per step):

Q1 (highest RV, range 4.4–5.6): Task performance 21.1%, MI Proxy 0.95, Entropy 2.02
Q2 (RV range 1.5–4.2): Task performance 19.5%, MI Proxy 0.93, Entropy 1.53
Q3 (RV range 0.0–0.2): Task performance 10.7%, MI Proxy 0.81, Entropy 1.41
Q4 (lowest RV, range 0.0–0.1): Task performance 11.0%, MI Proxy 0.73, Entropy 1.87

Task performance and MI degrade monotonically from Q1 to Q4, establishing the causal chain: reward variance → gradient quality → input-dependent reasoning. Combined with Theorem H.2 ( $\|g_{\text{task}}\| \leq \sqrt{\text{RV}}$ ), this links the theoretical bound to empirical outcomes through direct manipulation of the proposed causal variable.

Controlled noise injection weakens MI (Figure 9). In FrozenLake, the paper varies environment stochasticity (the probability of random action execution, from 0% to 100%) and measures success rates for both top- $p$ filtering and no filtering. Key findings:

Median success rates for both filtering and no-filtering decrease as stochasticity increases.
SNR-Aware Filtering maintains a clear advantage from 0% to 50% stochasticity.
The gap closes at 80–100% stochasticity, where high transition noise weakens reward variance as an informative signal proxy — even high-effort prompts yield noisy, undifferentiable rewards, so RV loses its discriminative power.

This is the expected consequence of the SNR chain: additional noise inflates within-prompt return variance in a signal-free way, diluting the advantage estimates that task gradients depend on. The filter's advantage attenuating at very high noise is itself informative — when the environment is so stochastic that RV loses its connection to signal quality, filtering cannot help and should not help, which is exactly what the mechanism predicts.

Prompt-level filtering outperforms trajectory-level filtering (Table 7). On Sokoban (Qwen2.5-3B), the paper disentangles whether the gains from SNR-Aware filtering come from selecting discriminative prompts or from discarding hard/noisy trajectories. Three conditions are compared:

No filter: 8/8 prompts used, 128 trajectories per update, task performance 12.9%, MI Proxy 0.83
Prompt-level RV filtering ( $\rho = 0.9$ ): 3.2/8 prompts used, 50.6 trajectories per update, task performance 23.6%, MI Proxy 1.80
Trajectory-level filtering: 8/8 prompts used, 64 trajectories per update (top-8 and bottom-8 retained per prompt), task performance 16.8%, MI Proxy 0.20

Trajectory-level filtering improves over no filtering (16.8% vs. 12.9%), confirming that improving per-prompt SNR by sub-selecting trajectories provides some benefit. However, prompt-level filtering substantially outperforms it (23.6% vs. 16.8%), despite using fewer total trajectories per update (50.6 vs. 64). The paper's interpretation: "within a naturally low-RV prompt, forcing within-prompt variance by sub-selecting trajectories amplifies noise. Selecting prompts that naturally produce discriminative signals is more effective."

When does SNR-Aware Filtering help? Predictable from RV statistics (Table 8). The paper identifies a practical diagnostic for whether filtering will be effective: the ratio $\text{Std}(\text{RV}) / \text{Mean}(\text{RV})$ , computable from a single rollout batch:

Sokoban, 14B: Ratio 1.29, filter $\Delta$ = +4.6% — high ratio, effective filtering
Sokoban, 3B ( $P = 32, G = 4$ ): Ratio 1.16, filter $\Delta$ = +3.2% — high ratio, effective filtering
FrozenLake, 3B (GRPO): Ratio 0.33, filter $\Delta$ = −5.0% — low ratio, filtering hurts

When the ratio is high (>1.0), the per-prompt RV distribution is bimodal — some prompts clearly carry signal while others are purely noise — and filtering cleanly separates signal from noise. When the ratio is near zero (<0.5), all prompts carry similar variance and filtering discards data uniformly with no benefit. This ratio serves as a cheap pre-training diagnostic to determine whether SNR-Aware Filtering is likely to help for a given task-algorithm-configuration combination.

Adaptive filtering behavior over training (Figure 10). The paper tracks two metrics over the course of training: the effective kept ratio $\rho_{\text{eff}}$ and the zero-variance prompt count. As training progresses:

The number of zero-variance prompts rises — more prompts produce trajectories with identical rewards as the policy converges.
The effective kept ratio falls — the filter automatically becomes more selective as signal quality degrades.

This adaptive tightening is precisely what a fixed strategy like top- $k$ with constant $k$ cannot replicate: it would continue absorbing gradient budget from uninformative prompts even as signal quality deteriorates. The top- $p$ filter responds to the collapsing variance distribution by concentrating updates on the shrinking pool of still-informative prompts.

Reward distribution collapse over training (Figure 11). Tracking prompt-level reward distributions across early, mid, and late training in Sokoban reveals a systematic shift: the hard portion shrinks as the policy improves, the mixed portion expands, and overall prompt-level variance collapses toward the late stages. This distribution-level signature mirrors the gradient-level story — late training is a regime where reward variation has been compressed to the point that gradient updates carry progressively less task-discriminative information. The filter's automatic tightening response (Figure 10) is the appropriate adaptation to this distribution-level shift.

RV is largely orthogonal to entropy and response length. The paper reports Spearman correlations: reward variance correlates weakly with conditional entropy (−0.14) and response length (0.12), while correlating strongly with task reward (0.63). RV therefore "targets a distinct axis of update quality rather than surface statistics, making it a complementary control knob to KL and entropy regularization" (Section 5.2). This explains why entropy-based stabilizers cannot prevent template collapse — they operate on axes that are largely independent of the reward variance that drives the SNR imbalance.

Ablation Studies and Robustness Checks

Filtering metric comparison (Appendix C, Table 9). On Sokoban (Qwen2.5-3B), the paper ablates the choice of filtering metric while holding the filtering strategy (top- $k = 0.5$ ) fixed. Results relative to the no-filter baseline (Task Perf 0.17, MI Proxy 0.54, Entropy 2.76, Unstable training — indicated by a crossmark):

Reward Variance filtering: Task Perf 0.38 (+0.20), MI Proxy 0.84 (+0.29), Entropy 1.64 (−1.12), Stable ✓
Reward Sum filtering: Task Perf 0.24 (+0.07), MI Proxy 0.80 (+0.26), Entropy 4.18 (+1.42), Unstable ✗
Entropy filtering: Task Perf 0.20 (+0.02), MI Proxy 0.41 (−0.14), Entropy 2.20 (−0.56), Unstable ✗
Entropy Variance filtering: Task Perf 0.23 (+0.06), MI Proxy 0.70 (+0.16), Entropy 2.94 (+0.18), Unstable ✗
Length filtering: Task Perf 0.16 (−0.02), MI Proxy 0.91 (+0.36), Entropy 1.65 (−1.10), Unstable ✗

The critical finding: Length filtering achieves the highest MI Proxy (0.91) but the lowest task performance (0.16) and unstable training. This confirms that high MI alone is not sufficient — the quality of the filtering criterion matters for translating input dependence into task success. Entropy-based filters either suppress MI or fail to prevent collapse. Only reward variance filtering simultaneously achieves high task performance, high MI retention, and training stability.

Keep strategy direction (Appendix C, Table 9). Under the same top- $k = 0.5$ configuration, the paper compares keeping the largest-variance prompts vs. the smallest-variance prompts:

Keep Largest: Task Perf 0.44, MI Proxy 0.89, Entropy 1.47, Stable ✓
Keep Smallest: Task Perf 0.29 (−0.15), MI Proxy 0.47 (−0.42), Entropy 5.31 (+3.84), Stable ✓

The asymmetry is pronounced: keeping high-variance groups yields substantially higher task performance and MI, while keeping low-variance groups degrades both and massively inflates entropy. This supports the SNR hypothesis that high-variance rollouts contain more informative training signal and validates that the filtering direction (high variance → keep) is the correct one.

Sampling strategy interaction with filtering (Appendix C, Table 9). Under fixed reward-variance filtering, the paper sweeps sampling strategies relative to the top- $p = 1.0$ baseline (Task Perf 0.17, MI Proxy 0.54, Entropy 2.76, Unstable):

Top- $p = 0.9$ : Task Perf 0.38 (+0.20), MI Proxy 0.84 (+0.29), Entropy 1.64 (−1.12), Stable
Top- $p = 0.5$ : Task Perf 0.29 (+0.12), MI Proxy 0.83 (+0.29), Entropy 1.88 (−0.88), Stable
Min- $p = 0.05$ : Task Perf 0.42 (+0.25), MI Proxy 0.67 (+0.13), Entropy 1.64 (−1.12), Stable
Min- $p = 0.2$ : Task Perf 0.45 (+0.27), MI Proxy 0.36 (−0.18), Entropy 3.01 (+0.26), Stable
Top- $k = 0.25$ : Task Perf 0.22 (+0.05), MI Proxy 0.86 (+0.32), Entropy 1.28 (−1.48), Stable
Top- $k = 0.5$ : Task Perf 0.44 (+0.27), MI Proxy 0.89 (+0.35), Entropy 1.47 (−1.29), Stable

The key finding: more aggressive filtering does not universally improve performance. Min- $p = 0.2$ achieves the highest task performance (0.45) but with severely degraded MI Proxy (0.36, down from baseline 0.54), suggesting it achieves success through a different mechanism than preserving input-dependent reasoning. The interaction between sampling aggressiveness and MI retention is non-trivial and non-monotonic — filtering helps up to a point, beyond which it can suppress MI while still improving task performance, a regime the paper does not fully characterize.

Format validity vs. MI (Figure 12). The paper tests whether format validity — a coarse, easily computed signal of whether the model outputs well-formed reasoning/action tags — can serve as a collapse indicator. The result: format validity is largely decoupled from input dependence. Runs maintain near-perfect format validity (>95%) while exhibiting a wide range of MI proxy values (from near 0 to >0.8), including runs with very low MI that would be classified as template-collapsed. This confirms that structural correctness and semantic input-dependence are separate dimensions and that format-based checks provide a false sense of security — reasoning can be syntactically well-formed yet semantically input-agnostic.

GRPO normalization amplifies noise at low RV (Appendix N). The paper proves formally that GRPO's advantage normalization (dividing by $\sqrt{\text{RV}(x)}$ ) causes the gradient estimator's variance floor to scale as $\text{RV}(x)^{-1}$ (Proposition N.1). Prompts with small reward variance suffer disproportionately noisy updates under GRPO-style normalization. This provides theoretical grounding for the empirical finding (Table 4) that GRPO shows filtering gains (+3.7% average) despite already incorporating variance normalization — the normalization does not solve the SNR problem; it may even exacerbate it for low-RV prompts.

RV statistics as a pre-training diagnostic (Table 8). Beyond the ratio analysis discussed above, the table provides per-setting RV statistics that characterize when filtering helps. For Sokoban (both 14B and 3B settings), RV Mean is >2, RV Std is >2.8, and RV Max exceeds 6 — a wide and high-variance distribution that allows filtering to cleanly separate signal-rich from signal-poor prompts. For FrozenLake GRPO, RV Mean is only 0.54, RV Std is 0.18, and the entire distribution is compressed (Min 0.22, Max 0.76) — all prompts carry similarly low variance, making filtering equivalent to random subsampling.

Critical Assessment

Claim 1: "Template collapse occurs when reasoning appears diverse within inputs but becomes input-agnostic across inputs."

What the experiments demonstrate. The paper provides converging evidence for this claim: (a) Training dynamics (Figure 5) show Retrieval-Acc (the discrete MI proxy) declining from approximately 0.8 to nearly 0 while conditional entropy remains elevated or spikes upward, confirming the specific pattern of high $H(Z|X)$ with low $I(X; Z)$ . This is validated across six alternative MI proxy variants (Figure 14), all showing the same qualitative degradation. (b) The behavioral manifestation — reasoning length decline across eight environments (Figure 7) — provides an independent, non-information-theoretic signature of template collapse consistent with the MI-based diagnosis. (c) The classification of training regimes in Figure 1 is empirically realized in the intervention trajectories (Figure 13), where no-filter runs move toward the low-MI, high-entropy quadrant characteristic of Template Collapse.

What is not demonstrated. The paper defines template collapse as a specific information-theoretic signature (high $H(Z|X)$ , low $I(X; Z)$ ) and provides extensive evidence that this signature reliably emerges during RL training. However, the paper does not demonstrate that the content of the reasoning actually becomes templated in a human-interpretable sense — it could be that reasoning remains input-dependent in ways the MI proxy fails to capture, or that the proxy's decline reflects changes in the model's calibration rather than genuine loss of input-conditioned reasoning. The reasoning length decline in Figure 7 is suggestive but not conclusive — shorter reasoning could reflect genuine efficiency improvements rather than template drift. Qualitative examples of template-collapsed reasoning are not provided in the main text (though they may exist in the codebase or supplementary materials). The behavioral link between the MI proxy decline and actual degradation in reasoning content remains somewhat indirect — the paper relies primarily on the correlation between MI and task performance to validate that the proxy captures meaningful reasoning quality, but this is a performance-based validation rather than a direct content analysis.

Claim 2: "Mutual information correlates with final performance much more strongly than entropy, making it a more reliable proxy for reasoning quality."

What the experiments demonstrate. Figure 8 provides clear evidence: Trajectory MI-ZScore achieves Spearman correlation +0.39 while Reasoning Entropy shows −0.11. The sign difference is the crucial finding — entropy points in the opposite direction from MI. Figure 13 further shows that intervention sweeps move the policy through the diagnostic spaces along different axes: entropy and KL tuning primarily shift $H(Z|X)$ without moving $I(X; Z)$ , while SNR-Aware Filtering moves both $I(X; Z)$ and performance along a positive trajectory.

Caveats and limitations. The correlation analysis pools across different types of interventions (entropy coefficient, KL coefficient, filtering keep rate), which conflates two potential sources of variation: differences in the underlying training algorithm and differences in the metrics' intrinsic relationship to performance. A cleaner analysis would compute correlations within each intervention type separately, as the relationship between MI and performance might differ under entropy tuning vs. filtering. The paper does not report whether the correlations are statistically significant (no p-values or confidence intervals), and the sample size for the correlation analysis is not explicitly stated — it appears to be based on the runs shown in Figure 13, which may be a relatively small number. Furthermore, the correlation is computed at a single training step (the final checkpoint), not across the entire training trajectory. An online monitoring metric should ideally predict eventual performance early in training; the paper shows MI declining before performance in Figure 5, but does not compute early-step MI-to-final-performance correlations, which would be the most relevant validation for the "early warning signal" claim.

Claim 3: "Template collapse is explained by an SNR mechanism: low reward variance weakens task gradients while input-agnostic regularization terms dominate."

What the experiments demonstrate. The evidence for the SNR mechanism is multi-layered and unusually rigorous for an empirical ML paper: (a) The gradient norm decomposition (Figure 3) directly shows task gradient norms scaling with reward variance while regularization gradient norms remain flat across RV buckets, under both PPO and GRPO. This is the crucial empirical observation that motivates the mechanism. (b) The quartile ablation (Table 6) provides causal evidence — training exclusively on high-RV prompts produces both better MI and better performance than training on low-RV prompts, establishing that RV is causally upstream of reasoning quality, not just correlated with it. (c) The controlled noise injection (Figure 9) shows that increasing environment stochasticity — which should degrade reward variance as a signal — predictably weakens the filter's advantage, consistent with the SNR mechanism's prediction that RV's value as a signal proxy degrades under high noise. (d) The formal bounds (Appendix H) prove that task gradient magnitude is upper-bounded by $\sqrt{\text{RV}(x)}$ and that the gradient estimator's SNR is bounded by $\sqrt{G} \cdot \sqrt{\text{RV}(x)} / \sigma(x)$ , providing theoretical grounding for the empirical patterns.

Caveats and limitations. The gradient norm decomposition (Figure 3) measures norms of gradient components, which is a coarse summary statistic. Two gradient vectors with the same norm can point in entirely different directions; the SNR mechanism's core claim is about the direction of the update (task-discriminative vs. input-agnostic), not just its magnitude. The paper does not measure gradient alignment — does the task gradient actually point in directions that increase $I(X; Z)$ , and does the regularization gradient point in directions that decrease it? This is a gap between the theoretical framing and the empirical measurements. The quartile ablation (Table 6) is compelling but confounds RV with other prompt properties — high-RV prompts might simply be easier or more informative for reasons unrelated to reward variance per se. A cleaner causal test would be to artificially inflate or deflate reward variance for the same set of prompts (e.g., by adding noise to rewards or by modifying the reward function) and measure the effect on MI and performance. The formal bounds establish that low RV limits task gradient magnitude but do not prove that this limit is the binding constraint in practice — the actual task gradient could be much smaller than the bound for other reasons, and the degree to which the bound is tight depends on the alignment between the score function and advantages, which the paper does not measure.

Claim 4: "SNR-Aware Filtering consistently improves both input dependence and task performance across tasks, algorithms, model scales, and modalities."

What the experiments demonstrate. Table 4 provides a remarkably comprehensive evaluation: 4 tasks × 4 algorithms × 4 model scales + 2 model types + 2 modalities = a large experimental matrix with consistently positive filtering deltas in most cells. The gains are substantial in several settings: +16.0% on Sokoban (PPO, 3B), +59.5% on FrozenLake (Qwen2.5-VL-3B, vision input), +22.9% on Sokoban (Qwen2.5-0.5B). The batch configuration sweep (Table 5) shows that the gains are robust to how the fixed 128-trajectory budget is partitioned across prompt groups. The filtering metric ablation (Table 9) shows that reward variance is superior to alternative criteria (entropy, length, reward sum). The keep direction ablation confirms asymmetry: keeping high-variance prompts helps; keeping low-variance prompts hurts.

Caveats and limitations. Despite the breadth, there are notable gaps and boundary conditions that temper the "consistently improves" claim:

The negative result in Table 8: FrozenLake GRPO shows a filtering loss of −5.0% with a Std/Mean RV ratio of 0.33. The paper frames this as a predictable boundary condition, which is scientifically honest, but it means the method is not universally beneficial — practitioners must check this ratio before deploying filtering, and the method provides no benefit (and active harm) in low-RV-variance regimes.
Close-to-ceiling tasks: MetaMathQA and Countdown baselines are already at 92.6% and 97.9% respectively, leaving minimal room for improvement. The +0.6 and +0.0 deltas are consistent with ceiling effects rather than filtering failing — but they also mean the method's practical value is limited on tasks where the base policy is already strong.
Dr. GRPO baseline on FrozenLake: Only 23.2% — dramatically below other algorithms (PPO: 67.0%, DAPO: 66.8%, GRPO: 70.9%). The filtering delta for Dr. GRPO on FrozenLake is only +0.6, suggesting that when the base algorithm is performing pathologically poorly, filtering cannot rescue it. This is a boundary condition worth noting: SNR-Aware Filtering amplifies signal from informative prompts but cannot create signal where the base algorithm is fundamentally failing.
No comparison to simple baselines: The paper does not compare SNR-Aware Filtering to the simplest possible improvement: training for more iterations with the same total compute. If filtering discards data (keeping only 30–50% of prompts at $\rho = 0.9$ , as the effective kept ratio in Figure 10 suggests), it is reducing the number of gradient updates per unit of rollout compute — a comparison to "no filtering but train for proportionally more iterations" would establish whether the gains come from better update quality or simply from being more selective about which data enters the gradient.
The filtering budget tradeoff is not fully characterized: SNR-Aware Filtering requires $G \geq 2$ trajectories per prompt, which repartitions the fixed rollout budget. Table 5 shows that the $8 \times 16$ configuration (default) has a worse no-filter baseline (15.6%) than $128 \times 1$ (23.6%) or $32 \times 4$ (24.2%). Filtering recovers and exceeds the $128 \times 1$ baseline, but this means filtering is partly compensating for the disadvantage of the group-sampling setup. The net benefit of filtering compared to a well-tuned $128 \times 1$ baseline is smaller than the +16.0 headline number suggests — the $32 \times 4$ filtered result (27.4%) vs. $128 \times 1$ (23.6%) represents a +3.8% gain, which is more modest.
Single training framework (veRL/HybridFlow): All experiments use the same training stack, which limits claims about generalizability to other RL training frameworks (e.g., OpenRLHF, TRL, custom implementations) where gradient computation, advantage estimation, and regularization may differ.
The keep rate $\rho = 0.9$ is not systematically tuned: The paper uses $\rho = 0.9$ throughout most experiments but does not report a sensitivity analysis across $\rho$ values (beyond the top- $p$ sweep in Figure 13 which uses a different evaluation setup). The optimal $\rho$ likely depends on the task, algorithm, and training stage, but the paper does not provide guidance on how to set it beyond the general Std/Mean RV diagnostic.
Lack of confidence intervals or statistical testing: All results in Table 4 report point estimates of peak success rate with deltas but no measure of variance across random seeds. Multi-turn agent RL is known to be high-variance; some of the reported gains might fall within the noise range, particularly the smaller deltas (e.g., +0.6, +0.8). The quartile ablation (Table 6) and filtering metric comparison (Table 9) likewise lack error bars, making it difficult to assess whether differences are statistically reliable.

Claim 5: "The MI proxy serves as an early warning signal, detecting reasoning degradation that entropy-based metrics miss entirely."

What the experiments demonstrate. Figure 5 (left panel) shows Retrieval-Acc declining substantially before task success rate shows visible degradation — this is the "early warning" pattern. Figure 5 (right panel) shows reasoning entropy remaining high or increasing during the same period where MI is declining, confirming that entropy misses the degradation that MI captures.

What is not demonstrated. The "early warning" claim requires quantifying how much earlier MI declines relative to performance — is it 10 steps, 50 steps, 100 steps? The paper does not compute lead time or formalize an early warning statistic. The claim also requires demonstrating that MI-based early stopping or intervention triggers would actually improve outcomes — a simple experiment would be: train with no filtering, but stop training (or switch interventions) when MI drops below a threshold, and compare final performance to either no-intervention or late-intervention baselines. The paper does not run such an experiment, so the practical value of MI as an early warning signal remains aspirational. Additionally, the "early decline" pattern is shown for one environment (presumably Sokoban, though the figure caption does not specify) — whether MI provides early warning consistently across environments, algorithms, and model scales is not systematically evaluated.

6. Limitations and Trade-offs

The Cost of Difficulty Estimation Is Not Accounted for in Performance Gains

SNR-Aware Filtering requires computing per-prompt reward variance from at least G = 2 trajectories per prompt each iteration — a group-sampling setup that repartitions the fixed 128-trajectory rollout budget into fewer unique prompts. The paper acknowledges this partitioning requirement (Section 4.3, Table 5) but the interaction between the group-sampling overhead and the filtering gains is more subtle than the headline deltas suggest.

The consequence. The paper's claims about improvement magnitude are implicitly benchmarked against an unfiltered baseline using the same group-sampling configuration. However, Table 5 reveals that the unfiltered baseline degrades substantially under group sampling: on Sokoban (Qwen2.5-3B), the standard 128 × 1 configuration achieves 23.6% task performance, while the 8 × 16 unfiltered baseline drops to 15.6% — a loss of 8.0 percentage points just from repartitioning the budget. SNR-Aware Filtering on 8 × 16 recovers to 23.6%, matching the 128 × 1 no-filter performance. The net benefit of filtering — compared to the best possible unfiltered configuration — is closer to 4 percentage points (27.4% on 32 × 4 filtered vs. 23.6% on 128 × 1), not the +16.0 that Table 4 reports against the 8 × 16 unfiltered baseline. A practitioner reading the headline numbers without consulting Table 5 would substantially overestimate filtering's absolute benefit.

What evidence exists. Table 5 provides the direct comparison across batch configurations. The 128 × 1 row represents the strongest no-filter baseline (since no prompt-sharing is required), and the filtered configurations only marginally exceed it. The paper does not discuss this "net vs. gross" distinction in the main text — the Table 4 deltas are all computed within a fixed (P, G) configuration, not against the best-in-class no-filter setup. The variance computation itself is cheap (<0.1% of iteration time), but the opportunity cost of group sampling — the loss of prompt diversity from sampling fewer unique prompts per iteration — is the real overhead, and it is not separated from the filtering benefit in the main results.

Mitigation status. The paper does not address this accounting issue. The headline Table 4 reports deltas against configuration-matched baselines, not against the best available no-filter configuration. A fair accounting would report filtering gains relative to the 128 × 1 no-filter baseline as a lower bound on practical improvement, alongside the configuration-matched deltas. The paper suggests per-task tuning of the keep rate ρ (Section 7, Limitations) but does not discuss tuning the (P, G) partition itself as a hyperparameter that interacts with filtering effectiveness. The practical recommendation — use G ≥ 4 with filtering (Table 5) — is implicit but not stated as guidance.

The Method Degrades or Fails When Reward Variance Is Uniformly Low

SNR-Aware Filtering operates on the assumption that within-prompt reward variance is a reliable proxy for gradient signal quality — specifically, that the batch contains a mix of high-variance (signal-rich) and low-variance (noise-dominated) prompts that filtering can separate. When this assumption fails, the method provides no benefit or actively hurts.

The consequence. In environments or training regimes where within-prompt reward variance is uniformly low across all prompts, filtering becomes equivalent to random subsampling — it discards data without improving the signal-to-noise ratio of the remaining batch. The paper's own evidence shows this concretely: on FrozenLake with GRPO (Table 8), the Std/Mean RV ratio is 0.33, indicating a compressed variance distribution where all prompts carry similarly low variance, and filtering produces a −5.0% degradation in task performance. This is not a minor edge case — it is a predictable failure mode for any environment with sparse or noisy rewards where trajectories from the same prompt tend to receive identical or nearly identical returns. The paper's formal bound (Appendix H, Theorem H.3) shows that the SNR of the gradient estimator is upper-bounded by √G · √RV(x) / σ(x), where σ²(x) is the irreducible reward noise. When σ(x) is large relative to √RV(x) — which occurs in stochastic environments with sparse rewards like FrozenLake — reward variance loses its connection to signal quality entirely.

This failure mode is particularly concerning because the conditions that cause it (stochastic transitions, sparse rewards, binary success/failure outcomes) are common in agent RL benchmarks. The Std/Mean RV ratio diagnostic (Table 8) provides a way to detect this condition before training, but the paper does not evaluate whether this diagnostic predicts filtering effectiveness across the full task suite — it is demonstrated only for the Sokoban and FrozenLake cases where filtering helps and hurts respectively.

What evidence exists. Table 8 provides the direct evidence: FrozenLake GRPO shows the lowest Std/Mean RV ratio (0.33) and the only large negative filtering delta (−5.0%). Figure 9 provides converging evidence: as environment stochasticity increases from 0% to 100%, the filtering advantage over no-filtering shrinks and eventually closes — at 80–100% stochasticity, even high-effort prompts produce undifferentiable rewards, and RV loses its discriminative power. The quartile ablation (Table 6) also indirectly supports this: Q3 and Q4 (the lowest RV quartiles) produce task performance around 10–11%, only marginally above what the unfiltered baseline achieves on Sokoban. The paper acknowledges the boundary condition explicitly (Section 5.2): "When the ratio is near zero, all prompts carry similar RV and filtering discards data uniformly."

Mitigation status. The paper provides the Std/Mean RV ratio as a diagnostic (Table 8) and recommends computing it from a single rollout batch before deploying filtering. However, this is a binary diagnostic — it tells you whether filtering is likely to help or hurt, but provides no guidance for environments in the intermediate regime where the ratio is moderate (0.5–1.0) and the filtering benefit is uncertain. The paper does not explore whether alternative filtering criteria (e.g., advantage-based variance, trajectory-level entropy, or learned SNR estimators) could recover signal discrimination in low-RV environments where raw reward variance fails. The method's sensitivity to this condition limits its applicability as a drop-in training improvement — practitioners must verify the RV distribution before adopting it, and the paper provides no fallback strategy for environments that fail the check.

The Method Has Not Been Evaluated on Open-Ended Generation or Tasks Without Scalar Rewards

All seven environments in the experimental testbed (Sokoban, FrozenLake, MetaMathQA, Countdown, SearchQA, WebShop, DeepCoder) share a critical structural property: each episode terminates with a scalar reward signal that can be computed automatically — from grid-based success conditions, exact-match answer checking, or unit tests. The SNR mechanism and SNR-Aware Filtering both depend fundamentally on within-prompt reward variance as the proxy for gradient signal quality.

The consequence. The method's applicability to tasks without clean scalar rewards is entirely unknown. Many important real-world applications of LLM agents — open-ended dialogue, creative writing, multi-step planning with subjective evaluation, negotiation, teaching — either lack automatic reward signals entirely or require learned reward models whose variance properties may differ substantially from the environment-generated rewards studied in the paper. If reward variance in these settings is driven primarily by reward model uncertainty rather than genuine trajectory quality differences, the SNR mechanism's mapping from RV to gradient signal quality may break down. Even in tasks with scalar rewards, if the reward function is shaped or dense (as opposed to the mixture of sparse and dense rewards in the paper's environments), the relationship between RV and gradient informativeness may differ — a prompt with high reward variance due to noisy shaped rewards may not actually provide discriminative signal about which reasoning paths are better.

The paper's formal framework (Appendix H, Theorem H.3) makes the dependence on reward noise σ(x) explicit: when σ(x) is large, the SNR bound degrades regardless of RV. In learned-reward settings, σ(x) could be substantial and non-stationary (the reward model's uncertainty may change over training), complicating the use of raw RV as an SNR proxy. The paper's negative result with entropy-based and length-based filtering criteria (Table 9) suggests that alternative signal proxies are not trivially available — the specific choice of reward variance as the filtering metric is load-bearing, not incidental.

What evidence exists. The paper's task suite (Table 3) covers a range of reward structures — dense (Sokoban, MetaMathQA, SearchQA, WebShop, DeepCoder), binary (FrozenLake, Countdown), and diminishing (MetaMathQA) — but all are environment-computed scalar signals. None involve learned reward models, human feedback, or multi-dimensional reward signals. The paper does not discuss or acknowledge the scalar-reward assumption as a limitation — this is a gap in the analysis rather than an explicitly bounded scope. The experiments provide no evidence about how the method would behave with noisy, learned, or multi-objective rewards.

Mitigation status. Not addressed. The paper does not discuss reward model settings, reward shaping implications, or alternative signal proxies for non-scalar or learned-reward environments. The Limitations section (Section 7) notes that "the method requires reward variance to be a reliable signal proxy, which degrades in sparse or noisy reward environments," but this refers to reward sparsity within the existing scalar-reward framework, not to the broader class of tasks without scalar rewards. A natural extension — using the learned critic's value variance or advantage variance as an SNR proxy in settings with learned reward models — is not explored.

Experimental Validation Is Limited to a Single Model Family (Qwen) and Base Architecture

The paper conducts its primary experiments using Qwen2.5 models (0.5B, 1.5B, 3B, 7B, 3B-Instruct, VL-3B) and one Llama variant (Llama3.2-3B). The training infrastructure uses the veRL/HybridFlow stack with specific PPO/GRPO configurations. While the model scale sweep (0.5B to 7B) and the inclusion of a vision-language variant partially address generalizability concerns, the scope remains narrow in important ways.

The consequence. Several aspects of the paper's findings could be model-family-specific. The Qwen2.5 architecture and pretraining recipe may produce different reasoning patterns, different sensitivity to regularization, or different reward variance properties than other widely-used model families (e.g., Gemma, Mistral, Yi, DeepSeek). The MI proxy's calibration depends on the model's own log-probability scoring — if different model families have different calibration properties (e.g., systematically overconfident or underconfident log-probabilities), the relationship between the proxy values and true input dependence could shift. The SNR mechanism's gradient norm decomposition (Figure 3) relies on the specific balance of task and regularization gradients, which depends on the model's gradient structure — if different model families have different gradient norms for the same task performance level, the quantitative relationship between RV and gradient dominance could change.

More subtly, the Qwen2.5 models were trained with specific data mixtures and alignment procedures (RLHF, instruction tuning) that may affect their propensity for template collapse. A model family with stronger pretraining on structured reasoning data might be more resistant to template collapse; a model family with weaker reasoning priors might be more susceptible. The paper's claim that template collapse "occurs reliably across training configurations" (Section 4.2) is based on the pattern holding across algorithms and scales within the Qwen family, not across model families with different training histories.

What evidence exists. Table 4 includes Llama3.2-3B with results broadly similar to Qwen2.5-3B (average performance 73.6% vs. 73.4% for Qwen2.5-3B-Instruct, filtering delta +5.3 vs. +4.1). This provides a one-data-point cross-family validation, but Llama3.2-3B shares architectural similarities with Qwen2.5 (both are dense transformer models of similar scale). The paper does not test decoder-only architectures with substantially different design choices (e.g., mixture-of-experts, different attention mechanisms, different tokenizers that segment reasoning tokens differently). The model scale sweep (0.5B → 7B) is within a single model family and a single order of magnitude; whether the findings hold at much larger scales (70B+) where reasoning capabilities and gradient properties may differ qualitatively is untested.

Mitigation status. Partial, through the inclusion of Llama3.2-3B (Table 4). However, this is a single cross-family comparison at one scale. The paper does not discuss the potential for model-family-specific effects or recommend validation on target model families before deploying the method. The limitations section does not mention this scope constraint. A robust generalizability claim would require at minimum 2–3 diverse model families at multiple scales, ideally including models with known differences in reasoning style or calibration properties.

The MI Proxy's Reliability as an "Early Warning Signal" Is Not Quantitatively Validated

The paper claims that the MI proxy "serves as an early warning signal, detecting reasoning degradation that entropy-based metrics miss entirely" (Section 4.2), and that "mutual information declines significantly before task performance degrades" (Section 2.2). This claim has substantial practical implications — it suggests practitioners should use MI as a trigger for early stopping, intervention switching, or hyperparameter adjustment. However, the paper's evidence for early warning capability is qualitative and based on a single training trajectory.

The consequence. Without quantitative characterization of the early warning property, practitioners cannot operationalize MI-based monitoring. Key questions are unaddressed: How much lead time does MI provide? (10 steps? 50 steps? Does it depend on the environment and algorithm?) Is the lead time sufficient to intervene usefully? What MI threshold should trigger an intervention, and is this threshold consistent across runs and tasks? Does the early warning pattern hold in all environments, or only in some? Figure 5 shows one trajectory where Retrieval-Acc declines before success rate — but the paper does not report whether this temporal ordering is statistically reliable across random seeds or whether the lead time is practically meaningful. It is possible that MI and performance decline nearly simultaneously in some settings, or that MI provides only a few steps of warning — too little to be actionable.

The "early warning" framing also conflates two distinct claims: (1) MI is a leading indicator (it declines before performance does, with useful lead time), and (2) MI is a more sensitive indicator (it detects degradation that current metrics miss). The paper supports claim (2) robustly through correlation analysis (Figure 8) and divergence from entropy (Figure 5), but claim (1) is supported only by visual inspection of a single training curve. A metric that is "more sensitive" but declines simultaneously with performance is still diagnostically valuable (it captures a different dimension of health), but it is not an early warning signal in the operational sense.

What evidence exists. Figure 5 shows one training trajectory on an unspecified environment (likely Sokoban based on context) where Retrieval-Acc drops from approximately 0.8 to 0.2 over the first ~50 steps while success rate remains flat near zero. The paper does not report lead time (e.g., number of steps between MI crossing a threshold and performance crossing the same threshold), does not compute the cross-correlation function between MI and performance to assess temporal precedence, and does not provide this analysis for any environment beyond the one shown. The MI proxy trajectories in Figure 14 show six additional proxy variants under the same training setup, all exhibiting MI decline before the 200-step window shown, but without paired performance curves to assess lead time.

Mitigation status. Not addressed. The paper acknowledges no limitation around the early warning claim. The limitations section focuses on the SNR mechanism's assumptions, the method's bounded applicability, and the need for per-task tuning (Section 7), but does not discuss the gap between qualitative observation and quantitative validation of MI as an actionable early warning metric. A proper validation would require: (1) computing lead time distributions across multiple random seeds and environments, (2) demonstrating that an MI-triggered intervention (e.g., activating SNR-Aware Filtering when MI drops below a threshold) improves final outcomes compared to fixed-schedule or no-intervention baselines, and (3) characterizing the false-positive rate (how often does MI decline without subsequent performance degradation?). None of these analyses appear in the paper or appendices.

Filtering Discards Data and May Narrow Exploration Coverage Over Extended Training

SNR-Aware Filtering removes low-variance prompts from each parameter update. Over the course of training, as the policy converges and more prompts drift toward zero reward variance (documented in Figures 10 and 11), the filter automatically becomes more selective — the effective kept ratio ρ_eff declines. The paper frames this adaptive tightening as a feature: it "concentrates gradient updates on the shrinking pool of still-informative prompts" (Section 5.2). However, this narrowing of the effective training distribution raises concerns about exploration coverage and long-term learning that the paper does not investigate.

The consequence. As the filter keeps fewer prompts per update, the policy receives gradient signal from an increasingly narrow slice of the prompt distribution. This creates two risks. First, prompts that are systematically filtered out (because they produce low reward variance under the current policy) may represent an important part of the task distribution — perhaps harder problems where the policy's performance is poor but uniform (all trajectories fail, producing zero variance), or edge cases where the environment's reward structure does not differentiate between good and bad behavior. By excluding these prompts from training, the policy may lose coverage of precisely the parts of the distribution where improvement is most needed. Second, the tightening feedback loop — the filter removes low-variance prompts → the policy updates only on the remaining high-variance prompts → the excluded prompts become even more low-variance (as the policy doesn't adapt to them) → the filter becomes even more aggressive — could lead to progressive narrowing of the effective training distribution, a form of distributional collapse that is distinct from but complementary to the template collapse the method addresses.

The paper acknowledges this risk in passing: "Aggressive filtering may narrow exploration coverage; the kept mass requires per-task tuning" (Section 7). However, no experiment characterizes the extent of coverage narrowing, measures its impact on out-of-distribution generalization, or explores mitigation strategies beyond tuning ρ.

What evidence exists. Figure 10 shows the effective kept ratio declining over training, confirming that the filter becomes more selective as the policy converges. Figure 11 shows the reward distribution collapsing — prompts shift from having diverse reward outcomes to having uniform reward structures, which is precisely what increases the zero-variance prompt count and triggers the filter's tightening. However, neither figure tracks the composition of kept vs. filtered prompts — are the filtered prompts systematically different (harder, more diverse, from a different part of the distribution) from the kept prompts? The quartile ablation (Table 6) shows that training exclusively on low-RV prompts degrades both MI and performance, confirming that low-RV prompts provide noisy signal — but this doesn't address whether completely excluding them is optimal versus including them at a lower weight or with a different learning rate. The paper does not evaluate a "soft filtering" variant (e.g., weighting prompts by RV rather than binary inclusion/exclusion) that could maintain coverage while downweighting noisy updates.

Mitigation status. Not addressed beyond the brief acknowledgment in Section 7. The paper does not measure whether filtering causes the policy to forget or degrade on low-RV prompts over extended training, does not compare hard filtering to soft weighting, and does not characterize the exploration-coverage tradeoff. This is a significant gap because the adaptive tightening documented in Figures 10–11 suggests the tradeoff becomes more severe over time — the very training regime where the paper recommends deploying the method. A practitioner concerned about maintaining broad capability coverage during long training runs would need to monitor prompt-level performance on filtered-out prompts separately, which the paper's evaluation protocol (validation on a fixed set of 512 prompts) does not capture if those prompts are from the systematically filtered portion of the distribution.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a diagnostic reframing of reasoning quality in agent RL rather than a new algorithm or architecture — and this is precisely what makes its impact potentially deeper than an incremental method improvement would be. By decomposing reasoning quality into within-input diversity (H(Z|X)) and cross-input distinguishability (I(X; Z)) and demonstrating that these axes move independently under standard training dynamics, the paper reveals that the field's default monitoring regime — track reward for outcome health and entropy for process health — is monitoring the wrong process metric. Entropy can remain high or increase while input dependence collapses, meaning practitioners have been systematically looking at a signal that points in the wrong direction (Spearman −0.11 to −0.14 for entropy vs. +0.39 for MI, Figure 8). This is not a claim that MI is slightly better — it is a claim that the standard diagnostic is actively misleading.

The magnitude of this shift is closest to a diagnostic paradigm correction: it does not overturn the RL training procedures themselves (PPO, GRPO, DAPO remain the workhorses), but it changes what "healthy training" means and how we would detect when it has gone wrong. The practical consequence is that future agent RL training should track MI proxies alongside (or instead of) entropy, and runs where MI declines while entropy remains stable should be treated as failing — even if reward and entropy look healthy. This is analogous to how the ML community learned that training loss is insufficient and validation metrics are necessary, or how the RL community learned that reward curves alone don't capture exploration quality. The paper's contribution is to identify which additional metric fills the gap.

The paper also resolves a latent tension in the empirical literature on LLM reasoning collapse. Prior work reported apparently contradictory findings: some studies found reasoning becoming more templated and less input-sensitive after RL (Wei et al., 2025; Yun et al., 2025), while others found that entropy regularization and KL constraints could maintain reasoning diversity (Ouyang et al., 2022; Xu et al., 2025). The paper's decomposition explains this: entropy-based interventions preserve H(Z|X) (within-input diversity), which prevents the most visible form of collapse, but they do not prevent I(X; Z) from declining — and different studies, testing on different environments and difficulty distributions, were implicitly measuring different axes of reasoning quality. The SNR mechanism provides the causal explanation for when collapse occurs (low within-prompt reward variance) and why existing stabilization methods cannot fully prevent it (they control the noise floor, not the signal strength).

The identification of verifier over-optimization and reward hacking as central challenges in RLHF has redirected substantial research effort toward reward model robustness. Similarly, this paper redirects attention toward the gradient signal-to-noise ratio as the binding constraint on reasoning quality in agent RL. The implication is that research effort should shift from developing stronger regularization schemes (which control ‖g_reg‖) toward improving the signal quality of individual updates (increasing ‖g_signal‖) — through better reward design, more informative environments, adaptive prompt selection, or learned signal-quality estimators. The paper's finding that DAPO — a state-of-the-art stabilization method — still benefits from SNR-Aware Filtering (Table 4, +2.9% average delta) suggests that even the best current methods leave substantial SNR headroom untapped.

Finally, the paper establishes mutual information as a first-class training diagnostic that is both theoretically grounded (via the Shannon decomposition and the SNR mechanism) and practically computable (via in-batch cross-scoring with no external models). This opens the door to MI-guided training interventions beyond filtering — for example, MI-based early stopping, MI-based hyperparameter scheduling, or MI-based prompt curriculum design — that were previously impractical because MI estimation was either too expensive or required external models. The lightweight proxy family (Table 1) makes MI tracking feasible as a standard training monitor, analogous to how gradient norm tracking became standard after the deep learning community recognized its diagnostic value.

Follow-Up Research This Work Enables

1. MI-triggered dynamic intervention switching rather than fixed-schedule filtering. The paper demonstrates that the MI proxy declines before task performance in at least some environments (Figure 5), but does not evaluate whether this early warning can trigger interventions automatically. A natural follow-up would train agents with no filtering initially, monitor MI, and activate SNR-Aware Filtering only when MI drops below a calibrated threshold — testing whether reactive filtering recovers performance comparable to filtering from the start while saving the data discarded by early filtering. This would convert the "early warning signal" claim from a qualitative observation into an operational protocol. The experiment requires: (1) establishing per-environment MI thresholds from pilot runs, (2) comparing reactive filtering to always-on filtering and no-filtering baselines across the seven environments in the paper's testbed, and (3) measuring whether reactive filtering achieves the same final performance as always-on filtering while using more of the available training data.

2. Learned SNR estimators that predict gradient informativeness without requiring G >= 2 trajectories per prompt. SNR-Aware Filtering requires group sampling (G >= 2) to compute reward variance, which forces a repartitioning of the fixed rollout budget and reduces the number of unique prompts seen per iteration. A lightweight predictor — perhaps a small linear head on top of the policy's hidden states, trained to predict whether a prompt will produce high-variance rollouts from its initial representation — could estimate SNR from a single trajectory per prompt, recovering the 128 × 1 budget efficiency while retaining filtering benefits. The training signal for this predictor would come from the reward variance computed on occasional group-sampled validation batches. The key metric: does a learned SNR filter match or exceed the performance of reward-variance-based filtering while using more diverse prompts per iteration? The paper's finding that reward variance correlates only weakly with entropy (−0.14) and response length (0.12) but strongly with task reward (0.63, Section 5.2) suggests SNR is not trivially predictable from surface statistics, making learned prediction a non-trivial challenge.

3. Stress-testing the SNR mechanism through controlled reward variance manipulation. The paper provides strong correlational and ablation evidence for the SNR mechanism (gradient decomposition in Figure 3, quartile ablation in Table 6, noise injection in Figure 9), but a direct causal test would manipulate reward variance for the same set of prompts while holding all else constant. For example: take a fixed set of prompts with naturally high reward variance, artificially reduce their variance by adding noise to rewards (collapsing the advantage estimates), and measure whether this intervention reproduces the MI decline predicted by the SNR mechanism. Conversely, take low-variance prompts and artificially inflate their variance (e.g., by scaling rewards or adding trajectory-dependent bonuses) to test whether this rescues MI. This would isolate RV as the causal variable rather than a correlate of other prompt properties. The experiment requires an environment where reward structure can be modified without changing the underlying task (Sokoban with configurable reward shaping would be ideal) and careful controls for whether modified rewards change the optimal policy.

4. Characterizing whether template collapse in the MI sense corresponds to human-judged template collapse in reasoning content. The paper defines template collapse information-theoretically (low I(X; Z), high H(Z|X)) and validates this definition through correlation with task performance. But does the MI proxy's decline actually correspond to reasoning becoming templated in a content sense — producing the same boilerplate phrases, reasoning structures, or action patterns across different inputs? Or does it capture a more subtle form of input-agnosticism that a human reader might not recognize as "templated"? A follow-up study would collect reasoning traces at different MI proxy levels, have human annotators judge whether the reasoning is input-specific or generic, and measure the alignment between human judgments and the MI proxy. This would establish whether the information-theoretic definition captures the intuitive notion of template collapse that practitioners care about. A negative result — MI proxy declining without humans perceiving template-like behavior — would suggest the proxy captures something real but different from "templating," possibly related to calibration or confidence rather than content; a positive result would validate the proxy as tracking a human-meaningful phenomenon.

5. MI-based prompt curriculum: actively selecting which prompts to train on based on their estimated contribution to I(X; Z) growth. SNR-Aware Filtering selects prompts within each batch based on reward variance — a proxy for the current gradient SNR. A more ambitious approach would select prompts strategically over the course of training to maximize the rate of MI improvement, analogous to curriculum learning but with an information-theoretic objective. Concretely: maintain a buffer of candidate prompts, periodically estimate each prompt's potential to increase I(X; Z) (e.g., by measuring how much the MI proxy improves after training on that prompt for a few steps), and prioritize prompts that are in the "zone of proximal development" — neither too easy (already high MI, low learning potential) nor too hard (near-zero RV, no gradient signal). The paper's quartile ablation (Table 6) and the Std/Mean RV diagnostic (Table 8) provide initial evidence that prompt selection matters, but a dynamic curriculum would go beyond static filtering to actively construct training batches that maximize reasoning quality improvement. This connects to the broader literature on data selection for language model training (e.g., DoReMi, DSIR) but with an information-theoretic objective specific to agent RL.

6. Extending the MI proxy framework to multi-agent and open-ended generation settings where scalar rewards are unavailable. The paper's MI proxy requires only in-batch cross-scoring of reasoning traces against prompts using the model's own log-probabilities — it does not depend on rewards, scalar outcomes, or external verifiers. This means the diagnostic component of the paper's contribution could generalize to settings where SNR-Aware Filtering cannot (because reward variance is undefined). Two natural extensions: (1) multi-agent RL, where a single agent's reasoning may become templated even as the collective behavior remains diverse, and where existing diagnostics are even more limited than in the single-agent case; (2) open-ended dialogue or creative generation, where no automatic reward exists but the MI proxy could still detect whether the model's responses are input-dependent or drifting toward generic boilerplate. For the multi-agent case, a follow-up would train cooperative agents (e.g., in a negotiation or collaborative planning task) and track per-agent MI to test whether template collapse propagates between agents (does one agent's collapse accelerate another's?) and whether per-agent filtering can prevent cascade effects. The paper explicitly notes that "how template collapse propagates in multi-agent RL remains open" (Section 7).

Practical Applications and Downstream Use Cases

1. Online health monitoring for production agent RL training pipelines. Organizations training LLM agents at scale (e.g., for customer support, code generation, or web navigation) currently monitor reward curves and occasionally entropy or response diversity. The paper's evidence that entropy is actively misleading (negative Spearman correlations with performance, Figure 8) while MI is positively predictive (+0.39) provides a direct upgrade path: add MI proxy computation to the training loop's logging (requiring only in-batch cross-scoring with negligible overhead, per Section 2.3) and flag training runs where MI declines while entropy remains stable for manual intervention. The practical benefit is earlier detection of silently failing runs, potentially saving days of GPU-time on training that looks healthy by traditional metrics but is producing agents with collapsed reasoning. A concrete adoption threshold: if MI drops below 50% of its peak value while success rate has not yet declined, trigger a checkpoint review or automatic ρ adjustment.

2. Filtering as a drop-in improvement for existing PPO/GRPO training stacks on mid-scale models. SNR-Aware Filtering requires no changes to the underlying RL algorithm — it is a data selection wrapper that can be added to any existing PPO, GRPO, or DAPO training loop. The implementation cost is minimal (reward variance computation from grouped rollouts, a top-p selection step) and the computational overhead is negative (filtering reduces per-step time by 26–41%, Table 5). For teams training agents on environments where the Std/Mean RV ratio exceeds ~1.0 (a cheap pre-training diagnostic), the paper's results suggest expected gains of +3% to +16% on task success rate (Table 4) at no additional wall-clock cost. The primary adoption barrier is the need for group sampling (G >= 2), which requires modifying the data collection pipeline if it currently uses G = 1 — but Table 5 shows that configurations with G = 4 and filtering match or exceed G = 1 no-filter baselines, so the transition is net-beneficial. The vision-language results (+35.8% average, Table 4) suggest multi-modal agent training — a rapidly growing application area — may see the largest gains.

3. MI-guided early stopping for cost-sensitive or high-stakes agent training. In settings where training compute is constrained (academic labs, startups) or where deploying a collapsed agent has high cost (safety-critical applications, customer-facing systems), the MI proxy's apparent ability to decline before task performance (Figure 5) could serve as an early stopping criterion that prevents wasted compute and unsafe deployments. A conservative protocol: train with no filtering, compute MI every N steps, and stop training if MI drops below a calibrated fraction of its peak and remains low for K consecutive checks — even if reward appears stable. Compare the performance of early-stopped agents to fully-trained agents that experienced template collapse, measuring both task success and worst-case failure modes. This is a zero-cost addition to existing training pipelines that could prevent the most severe cases of silent reasoning degradation. The paper's early stopping criteria already include reward-variance collapse detection, but this is an outcome metric; MI-based early stopping could trigger before variance collapse is detectable.

When to Prefer This Method

The paper explicitly identifies boundary conditions for SNR-Aware Filtering and articulates when it replaces versus complements existing methods. Decision framework:

Prefer SNR-Aware Filtering when: (1) the environment provides scalar rewards with non-trivial within-prompt variance — the ratio Std(RV) / Mean(RV) computed from a single rollout batch exceeds ~1.0 (Table 8), indicating a bimodal variance distribution where filtering can separate signal from noise; (2) you are already using group sampling (G >= 2) or can repartition your rollout budget without losing overall prompt diversity (Table 5 shows net benefit for G >= 4); (3) you observe declining MI alongside stable or increasing entropy during training (the signature of template collapse); (4) you are training mid-scale agents (0.5B–7B) where the gains appear consistent across model sizes (Table 4).
Prefer standard training (no filtering) when: (1) Std(RV) / Mean(RV) < 0.5 — reward variance is uniformly low across prompts, making filtering equivalent to random subsampling (FrozenLake GRPO, ratio 0.33, filtering loss of −5.0% in Table 8); (2) the environment has extremely sparse or binary rewards where most prompts produce zero variance regardless of reasoning quality; (3) you are near performance ceiling (e.g., MetaMathQA at 92.6%, Countdown at 97.9% in Table 4) where filtering provides marginal or zero benefit; (4) exploration coverage is paramount and you cannot risk the narrowing of effective training distribution that accompanies adaptive filtering (Figures 10–11).
SNR-Aware Filtering complements, rather than replaces, KL and entropy tuning. The paper explicitly demonstrates that filtering operates on a different axis — signal enhancement via prompt selection vs. noise control via regularization — and that filtering improves performance on top of state-of-the-art methods that already include stabilization mechanisms (DAPO: +2.9% average in Table 4). The practical implication: tune KL and entropy coefficients for stability as usual, then add SNR-Aware Filtering as an orthogonal knob for controlling update quality. Do not increase entropy regularization as a response to template collapse — the paper's entropy decomposition theorem (Appendix M) shows this can decrease I(X; Z) further by increasing H(Z|X) without increasing marginal diversity H(Z).