The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models

ArXiv: 2505.22617

🎯 Pitch

Reasoning language models trained with RL suffer a predictable performance ceiling because their policy entropy collapses to near zero early in training—and downstream performance is directly traded from this entropy via a precise exponential law. The authors prove this collapse is driven by a persistent positive covariance between token probability and advantage, then stop it with simple covariance-based regularization that lifts the entropy floor and yields over 6% average benchmark gains.

---

1. Executive Summary

This paper studies the collapse of policy entropy during reinforcement learning (RL) for reasoning language models, demonstrating that without intervention, policy entropy sharply declines to near zero early in training and imposes a predictable performance ceiling. Across 11 models spanning 4 families on math and coding benchmarks, the authors establish a quantitative entropy–performance transformation equation ($R = -a \exp(\mathcal{H}) + b$) showing that downstream performance is traded from entropy in a predetermined manner, with the ceiling fully predictable when entropy is exhausted ($\mathcal{H} = 0$, $R = -a + b$). Through theoretical derivation and empirical verification, the paper identifies that entropy change is driven by the covariance between action probability and advantage (Theorem 1)—a high-probability action with high advantage reduces entropy, while a rare action with high advantage increases it—and finds this covariance term remains positive throughout training, explaining the monotonic entropy decline. To address this, the paper introduces two entropy-control techniques, Clip-Cov (detaching gradients from a small fraction of high-covariance tokens) and KL-Cov (applying a KL penalty to top-covariance tokens), which lift the entropy floor and yield non-trivial performance improvements (e.g., +6.4% average across benchmarks for Qwen2.5-32B relative to GRPO), establishing that managing covariance-driven token updates is sufficient to escape the entropy bottleneck and sustain exploration during RL for LLM reasoning.

2. Context and Motivation

The Core Problem: RL for LLM Reasoning Hits an Entropy Wall

The paper addresses a specific, empirically pervasive obstacle in scaling reinforcement learning for reasoning language models: policy entropy collapse. When LLMs are trained with RL on verifiable reasoning tasks (mathematics, coding), their output token distribution becomes extremely peaked almost immediately — the model's uncertainty plummets to near zero within the first few hundred gradient steps. This manifests as the model becoming "overly confident," selecting the same high-probability tokens with near-certainty regardless of context. The consequence, documented in Figure 1 (left) and quantified in Figure 2, is that over 95% of the entropy drop and performance gain occur in the early training stage, after which the model plateaus with little to no further improvement.

This is not a minor inefficiency. The paper establishes through the empirical equation $R = -a \exp(\mathcal{H}) + b$ that downstream performance is directly "traded" from entropy — as entropy approaches zero, the performance ceiling becomes predetermined and unavoidable regardless of continued training. The model essentially exhausts its exploratory capacity before it can discover more sophisticated reasoning strategies that could lead to further gains. The authors locate this as the central bottleneck preventing the vision articulated by Silver & Sutton (2025) — that the training compute paradigm should shift from pretraining (imitation learning) to RL (learning from experience), a shift that requires sustained exploration to work at scale.

Why This Problem Matters

The significance is both practical and conceptual. On the practical side, the recent wave of reasoning-focused LLMs — OpenAI's o1, DeepSeek-R1, Kimi K1.5 — all rely on RL with verifiable rewards as the key mechanism for eliciting and improving chain-of-thought reasoning. If policy entropy inexorably collapses, the return on scaling RL compute is fundamentally bounded: adding more training steps beyond the initial entropy depletion yields marginal returns. As the authors put it (Section 2.6), the ceiling "greatly burdens the scalability of RL for LLM reasoning." This matters enormously for organizations deciding how to allocate their post-training compute budgets.

Conceptually, the problem challenges a foundational assumption in RL: that policies can continuously explore and improve given sufficient interaction with the environment. The maximum-entropy RL literature (Ziebart et al., 2008; Haarnoja et al., 2018) has long treated entropy regularization as essential precisely to prevent premature convergence to suboptimal deterministic policies. Yet, as the paper documents, standard entropy regularization techniques fail when applied to LLMs (Section 4.1) — small coefficients have negligible effect, large coefficients cause entropy explosion, and KL regularization towards a reference model preserves entropy but degrades performance (Figures 9, 10). This represents a genuine paradox: the methods designed to solve the exploitation-exploration trade-off do not transfer to the LLM regime, and the field lacks both an explanation for why and a remedy that works.

The problem also connects to an active debate about whether RL for LLMs merely "elicits latent behaviors already learned in pretraining" versus teaching genuinely new capabilities (Yue et al., 2025). The entropy collapse finding provides mechanistic evidence supporting the elicitation view — if the policy rapidly converges to a high-confidence distribution, it may simply be surfacing patterns the model already knows, unable to venture into genuinely novel solution spaces. The authors argue, however, that this is not an intrinsic limitation of RL but rather a consequence of the entropy mechanism specific to LLMs, which can and should be engineered past (Section 2.6).

Where Prior Approaches Fall Short

Maximum entropy RL from conventional applications does not transfer. In standard deep RL, adding an entropy bonus to the reward function (Mnih et al., 2016; Schulman et al., 2017b) or using soft policy iteration with explicit entropy maximization (Haarnoja et al., 2017, 2018) reliably maintains exploration. The paper tests these approaches directly (Section 4.1): adding entropy loss $\mathcal{L}_{\text{ent}} = \mathcal{L} - \alpha \mathcal{H}(\pi_\theta)$ to the policy objective is hypersensitive to the coefficient $\alpha$. At $\alpha = 0.0001$ or $0.001$, entropy still collapses. At $\alpha = 0.01$, entropy explodes. Even at $\alpha = 0.005$, where entropy stabilizes, performance does not improve over baselines. KL regularization toward a reference model (the PPO-style approach) preserves entropy but degrades task performance (Figure 10), suggesting the reference constraint impedes learning rather than guiding it. This explains why most contemporary works on RL for LLM reasoning — including Open-Reasoner-Zero, PRIME, DAPO — do not include entropy regularization terms (Section 4.1), despite their proven importance in other domains. The default approach in the field has effectively been to accept the collapse and live with the plateau.

The entropy dynamics of LLMs under RL were not understood. Prior to this work, there was no systematic study of why policy entropy behaves the way it does in LLM RL training. Works that did observe entropy patterns described them as incidental findings rather than objects of study. Yu et al. (2025) noted that raising PPO's upper clipping threshold ("clip-higher") modestly increased entropy, but did not analyze the underlying mechanism. He et al. (2025) mentioned entropy regularization's sensitivity but did not explain why it fails. Luo et al. (2025) and Yan et al. (2025) observed distinct entropy patterns when using different policy models or off-policy data, suggesting entropy behavior is not universal but context-dependent — yet no framework existed for understanding which aspects of the setup govern entropy dynamics. Perhaps most critically, no prior work had connected the step-wise change in policy entropy to the covariance between action probability and advantage, an insight the authors trace to an unpublished blog post by Liu (2025) and formalize rigorously in Section 3. This left the field with empirical observations ("entropy drops fast") but no explanatory mechanism, and consequently no principled way to intervene.

The "clip-higher" heuristic is partial and unstable. The primary existing technique for influencing entropy in GRPO-style training, raising PPO's upper clipping threshold $\epsilon$ from the default 0.2 to 0.28 (as in DAPO; Yu et al., 2025), does increase entropy by allowing more low-probability tokens to contribute to policy updates (Section 4.5). However, the paper shows this approach is unstable — it produces performance improvements early in training that later saturate and decline (Figure 12), and the resulting entropy curves are less stable than the proposed Clip-Cov and KL-Cov methods. The reason, the paper argues, is that clip-higher operates indirectly on entropy by expanding which tokens are included, rather than directly targeting the tokens that drive entropy collapse. It adds low-covariance tokens (low probability, high advantage, average covariance $\sim -0.03$) to the gradient, which dilutes rather than counteracts the influence of the high-covariance tokens that dominate entropy dynamics. This indirect approach cannot precisely control the entropy trajectory.

The predictability of RL performance was unexplored for LLM reasoning. While scaling laws for pretraining (Kaplan et al., 2020; Hoffmann et al., 2022) and for reward model overoptimization (Gao et al., 2022) are well-established, no prior work had demonstrated that RL performance on reasoning tasks follows a predictable functional form. Hilton et al. (2023) and Rybkin et al. (2025) studied scaling laws for RL on non-LLM models, but the entropy–performance predictability documented in Section 2.4 — where two coefficients $a, b$ fitted from the first 15% of training steps predict final performance with an average RMSE of 0.9% on math and 1.2% on code (Figure 5) — was not known. This predictability gap matters because, without it, RL training for reasoning models is a black box: practitioners cannot anticipate what performance a given model-data combination will achieve, nor whether continued training at increased compute will yield meaningful returns. The entropy–performance equation provides precisely this capability.

How This Paper Positions Itself

The paper positions itself not as proposing a new RL algorithm or a new reasoning benchmark, but as providing the mechanistic understanding and principled entropy-control tools necessary for the field to scale RL for LLM reasoning beyond the current plateau. This is a distinctly analytical contribution, following a trajectory familiar from other subfields where empirical success outpaces theoretical understanding: first comes the empirical demonstration (RL works for reasoning — DeepSeek-R1, o1), then comes the framework explaining how and why, which then enables the next generation of improvements.

The framing is explicit in the abstract and the tagline connecting Section 1 to Section 2: "our finding necessitates entropy management for continuous exploration toward scaling compute for RL." The paper argues that the entropy mechanism is the common thread linking the observed collapse, the predicta ble performance ceiling, and the failure of conventional entropy regularization. By deriving the entropy dynamics from first principles (Lemma 1, Theorem 1), the paper explains why the collapse is not a pathological failure mode but rather a predictable consequence of how policy gradient interacts with the softmax parameterization when the covariance between log-probability and advantage is positive — which it is, empirically, throughout training (Figure 8).

The proposed methods, Clip-Cov and KL-Cov, are direct implementations of this understanding. They do exactly what the theory says matters: limit the gradient contribution of tokens where the log-probability–advantage covariance is high. This is fundamentally different from prior approaches that add entropy bonus terms to the loss (which operates on the objective, not on the covariance structure of the gradient), enforce KL proximity to a static reference (which penalizes all deviations uniformly rather than targeting the entropy-destructive ones), or expand the clipping window (which adds low-covariance tokens rather than suppressing high-covariance ones). The philosophical stance is that entropy control should be covariance-aware and token-level, not uniform or global — a stance that follows naturally from the theoretical analysis but was not obvious before it.

The paper also positions its entropy–performance predictability as having direct practical value: it enables early stopping, predicting final performance of large models from small ones (via the log-linear relationship of coefficients $a, b$ with parameter count, Figure 7), and estimating the return on additional RL compute before committing to it (Section 2.6). This connects the work to the broader scaling laws literature while emphasizing that the predictability is contingent — it depends on the base model, the training data, and the absence of entropy intervention, and may not hold in different setups (off-policy data, different policy models; Section 2.6 discussion). The paper is careful not to claim universality where evidence is absent, explicitly calling for "a more in-depth analysis of the entropy behavior under different conditions."

In summary, this paper fills a specific and consequential gap: the field had a working recipe for RL-based reasoning improvement but no understanding of its fundamental limits, no explanation of why the recipe plateaus, and no principled way to push past the plateau. By establishing the entropy–performance tradeoff as predictable, deriving the covariance mechanism that drives it, and demonstrating that token-level covariance control breaks the bottleneck, the paper provides both the diagnostic framework and the treatment for what it identifies as the central challenge in scaling RL compute for reasoning LLMs.

3. Technical Approach

3.1 Reader Orientation

This paper builds a diagnostic framework and intervention toolkit for reinforcement learning on reasoning language models. The system being analyzed is not a new model architecture but rather the training dynamics of policy gradient RL applied to autoregressive LLMs — and the methods developed are lightweight modifications to the standard PPO/GRPO loss computation that prevent policy entropy from collapsing. The problem it solves is the phenomenon where LLM policies become extremely deterministic early in RL training, causing exploration to cease and performance to plateau at a predictable ceiling far below what the model might otherwise achieve. The solution takes the form of identifying the mathematical driver of entropy collapse — the covariance between action probability and advantage at the token level — and designing two gradient-level interventions (Clip-Cov and KL-Cov) that selectively suppress the contribution of high-covariance tokens, thereby maintaining entropy and enabling continued exploration without degrading performance.

3.2 Big-Picture Architecture (Diagram in Words)

The paper's technical approach has four major components, arranged in a logical pipeline from observation to intervention:

1. Empirical measurement apparatus — A standardized RL training protocol applied across 11 models and 4 model families, recording policy entropy and validation performance at every rollout step. This generates the data that establishes the entropy–performance predictability law ($R = -a \exp(\mathcal{H}) + b$).

2. Theoretical analysis of entropy dynamics — A mathematical derivation (Lemma 1, Proposition 1, Theorems 1–2) that connects step-wise entropy change to the covariance between log-probability and logit change, and further to action probability and advantage under policy gradient algorithms. This provides the mechanistic explanation for why entropy decreases monotonically.

3. Empirical validation of the covariance mechanism — Training runs where the group-wise covariance term and entropy difference are computed and tracked over time, confirming the theoretical prediction that covariance is positive throughout training and its magnitude matches the rate of entropy decline. This bridges the theory to the observed phenomenon.

4. Covariance-aware entropy control methods — Two techniques (Clip-Cov and KL-Cov) that modify the policy loss computation to selectively restrict gradient updates on tokens with the highest covariance values, implemented as minimal code changes to standard PPO/GRPO loss functions. These are evaluated on Qwen2.5-7B and Qwen2.5-32B against GRPO and clip-higher baselines.

Information flows as follows: prompts are sampled from the training distribution → the LLM generates responses autoregressively → a verifier assigns scalar rewards → the RL algorithm (GRPO, RLOO, REINFORCE++, or PRIME) computes token-level advantages → the policy loss is computed (modified by Clip-Cov or KL-Cov if entropy control is active) → model parameters are updated → the new policy's entropy is measured on the training batch → the cycle repeats. Entropy and validation accuracy are logged periodically, enabling both the curve-fitting analysis (Section 2) and the dynamics validation (Section 3.3).

3.3 Roadmap for the Deep Dive

• First, the formal problem setup and policy gradient framework — the RL objective, the advantage estimation variants, and the definition of policy entropy the paper uses throughout. This establishes common ground for all subsequent analysis.
• Second, the entropy–performance fitting methodology — how entropy and accuracy are measured, how the exponential function is fitted, how coefficients are extracted, and how predictability across time (early → late training) and across models (small → large, via coefficient extrapolation) is established. This is the empirical foundation that motivates the theoretical investigation.
• Third, the theoretical derivation of entropy dynamics — starting from the softmax parameterization (Lemma 1) and building through policy gradient logit changes (Proposition 1) to the full entropy change theorem (Theorem 1). The Natural Policy Gradient variant (Theorem 2) is included because it connects to the KL-constrained objectives used in PPO-style algorithms.
• Fourth, the experimental design for validating the theory — how covariance and entropy difference are computed in the bandit setting used for LLM RL, the normalization choices, and the difficulty-grouped analysis.
• Fifth, the failure of conventional entropy regularization — the experiments with entropy loss coefficients and KL penalties that demonstrate why the standard maximum-entropy RL toolkit does not work for LLMs, motivating the need for a new approach.
• Sixth, the Clip-Cov and KL-Cov mechanisms — the token-level covariance computation, the selection criteria, the modified loss functions, and the hyperparameter choices. This section includes the pseudocode that makes the methods concrete.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an analytical and methods paper whose core idea is that the entropy collapse observed in RL for LLM reasoning is mechanistically driven by positive covariance between action probability and advantage, and that intervening at the level of high-covariance tokens — rather than adding global entropy bonuses — is the principled way to maintain exploration.

---

Formal Problem Setup and Policy Gradient Framework

The paper considers RL on verifiable tasks — math and coding problems where a correctness signal is available through programmatic checking, eliminating reward hacking as a concern. The RL objective is standard:

$\max_\theta J(\theta) := \mathbb{E}_{\boldsymbol{x} \sim \mathcal{D}, \boldsymbol{y} \sim \pi_\theta(\boldsymbol{x})} \left[ r(\boldsymbol{y}) \right]$

where $\mathcal{D}$ is the training distribution of prompts $\boldsymbol{x}$, $\pi_\theta$ is the LLM parameterized by $\theta$, $\boldsymbol{y}$ is an autoregressively generated output sequence of $T$ tokens $\{y_1, \cdots, y_t, \cdots, y_T\}$, and $r(\boldsymbol{y})$ is the scalar verifier reward (typically 1 for a correct answer, 0 for incorrect).

What it computes: the expected reward under the policy's own distribution over completions, averaged across prompts. This is the canonical expected-return objective for episodic RL with a terminal reward only.

Why this form: the verifier provides only a final-answer reward (not per-step feedback), making this a sparse-reward, episodic RL problem. The expectation is over the policy's own generation distribution, meaning the objective changes as the policy changes — this is on-policy RL, and the data distribution shifts at every training step. This shift is precisely what makes entropy dynamics non-trivial: the policy is chasing a moving target defined by its own outputs.

The paper employs the Policy Gradient theorem (Williams, 1992) for gradient estimation:

$\nabla_\theta J(\theta) = \mathbb{E}_{\boldsymbol{x} \sim \mathcal{D}, \boldsymbol{y} \sim \pi_\theta(\boldsymbol{x})} \left[ \sum_{t=0}^{T} \nabla_\theta \log \pi_\theta(y_t | \boldsymbol{y}_{<t}) A_t \right]$

where $A_t$ is the advantage of generating token $y_t$ in context $\boldsymbol{y}_{<t}$, and $\boldsymbol{y}_{<t}$ is the prefix of all tokens before position $t$. The advantage is a scalar that indicates whether this token generation step led to a better-than-expected or worse-than-expected outcome.

What it computes: the gradient with respect to model parameters as a weighted sum of per-token log-probability gradients, where the weight on each token is its advantage. Tokens that contributed to successful trajectories (positive advantage) are reinforced; tokens that contributed to unsuccessful trajectories (negative advantage) are suppressed.

Why this form: the policy gradient theorem provides an unbiased estimate of the true gradient of expected return, requiring only the ability to sample from the current policy and evaluate the advantage of the sampled actions. For LLMs with massive action spaces (the token vocabulary), value function estimation over all actions is intractable; the policy gradient avoids this by only evaluating the log-probability gradient of the actually-chosen actions. The summation over tokens applies the theorem in the autoregressive setting, treating each token generation as a separate action in a sequential decision process.

Advantage estimation variants. Because the verifier provides only a terminal reward, the advantage $A_t$ must be estimated from the trajectory-level reward. The paper considers three methods:

For vanilla REINFORCE (Williams, 1992): $A_t = r(\boldsymbol{y})$ Every token in a trajectory shares the same advantage — the full trajectory reward. This is unbiased but has high variance because all tokens in a successful trajectory are reinforced equally, even if some were irrelevant or harmful.

For GRPO (Shao et al., 2024), which the paper uses most extensively: $A_t = \frac{r(\boldsymbol{y}) - \text{mean}(r(\boldsymbol{y_{1:K}}))}{\text{std}(r(\boldsymbol{y_{1:K}}))}$

where $K$ responses are sampled for each prompt and the advantage is the z-score of the trajectory reward within its group. This normalizes advantages per prompt, removing prompt-dependent difficulty effects (some prompts are harder than others, so raw reward magnitude is confounded by prompt difficulty).

What it computes: for each of the $K$ responses to the same prompt, the advantage subtracts the group mean reward (making it a relative signal — how much better or worse than the average attempt was this particular response) and divides by the group standard deviation (making the advantage scale-invariant across prompts of different difficulty). The result is a dimensionless scalar assigned identically to every token in the response.

Why this form: group-wise normalization solves two problems simultaneously. First, subtracting the group mean removes the prompt baseline — on a hard question where most responses are wrong, a correct response should be reinforced strongly even if its raw reward is only 1, while on an easy question where most responses are correct, a correct response should not be reinforced much because it was expected. Second, dividing by the standard deviation ensures consistent gradient magnitudes across prompts with different levels of variance (e.g., prompts where responses are uniformly good vs. uniformly bad vs. mixed). The alternative — using a learned value baseline — would require training a separate critic network, adding complexity and potential instability.

For PRIME (Cui et al., 2025), the advantage uses an implicit process reward model (PRM) that provides token-level value estimates, but the paper groups this with the other algorithms for the entropy–performance curve fitting without detailing the PRM computation.

PPO surrogate loss. To handle off-policy data and constrain policy updates (preventing destructively large single-step changes), the paper employs PPO's clipped surrogate objective (Schulman et al., 2017b):

$L(\theta) = \mathbb{E}_t \left[ \min\left( \frac{\pi_\theta(y_t | \boldsymbol{y}_{<t})}{\pi_{\theta_{\text{old}}}(y_t | \boldsymbol{y}_{<t})} A_t, \text{clip}\left( \frac{\pi_\theta(y_t | \boldsymbol{y}_{<t})}{\pi_{\theta_{\text{old}}}(y_t | \boldsymbol{y}_{<t})}, 1-\epsilon, 1+\epsilon \right) A_t \right) \right]$

where $\pi_{\theta_{\text{old}}}$ is the policy that generated the batch of responses (the behavioral policy used for data collection), $\pi_\theta$ is the current policy being updated, the ratio $\frac{\pi_\theta}{\pi_{\theta_{\text{old}}}}$ is the importance-sampling weight correcting for off-policy data, $\epsilon$ is the clipping threshold (default 0.2), and $\mathbb{E}_t$ denotes the empirical average over tokens in the batch.

What it computes: for each token, the loss is the minimum of two terms — the unclipped policy gradient term (proportional to the importance-sampling ratio times the advantage) and a clipped version of that term. If the ratio moves outside $[1-\epsilon, 1+\epsilon]$, the gradient is clipped to zero, preventing further movement away from the old policy. The min operation means that clipping only activates when it reduces the loss magnitude, which happens when the ratio moves in a direction that would increase the loss (reduce the advantage for positively-advantaged tokens, or increase it for negatively-advantaged tokens).

Why this form: the PPO clipped objective is a KL-constraint surrogate — it approximates a trust-region update without computing the actual KL divergence (which would require a second-order optimization). By clipping the importance-sampling ratio, it prevents the new policy from deviating too far from the policy that generated the data, which would invalidate the importance-sampling correction and lead to destructive gradient updates. The $\epsilon = 0.2$ default means the policy can at most double the probability of a positively-advantaged token or halve the probability of a negatively-advantaged one per update. The clip-higher baseline (Yu et al., 2025) raises $\epsilon$ to 0.28 for the upper clip only, allowing more reinforcement of low-probability positive-advantage tokens.

Definition of policy entropy. The paper measures the average token-level entropy of the policy on training data:

$\mathcal{H}(\pi_\theta, \mathcal{D}) = -\mathbb{E}_{\mathcal{D}, \pi_\theta} \left[ \log \pi_\theta(y_t | \boldsymbol{y}_{<t}) \right] = -\frac{1}{|\mathcal{D}|} \sum_{x \in \mathcal{D}} \frac{1}{|\boldsymbol{y}|} \sum_{t=1}^{|\boldsymbol{y}|} \mathbb{E}_{y_t \sim \pi_\theta} \left[ \log \pi_\theta(y_t | \boldsymbol{y}_{<t}, x) \right]$

where $\mathcal{D}$ is a batch of prompts randomly sampled from the training distribution, $|\boldsymbol{y}|$ is the length of a generated response, and the innermost expectation is over the model's token-level output distribution at each position. In practice, this is computed as the average negative log-probability of the actually-sampled token at each position across all tokens in all responses in the batch.

What it computes: the Shannon entropy of the policy's predictive distribution at each generation step, averaged over all steps and all prompts in the batch. A high value (e.g., 2–3 nats) means the model is uncertain across many possible next tokens; a low value (e.g., 0.1–0.5 nats) means the model is nearly deterministic, putting almost all probability mass on a single token.

Why this form: this is the standard information-theoretic entropy measure, and it directly quantifies the exploratory capacity of the policy — if entropy is high, the model generates diverse responses; if entropy is near zero, the model always produces the same output for a given prefix. The paper computes entropy on the training data rather than on a held-out set because the concern is about exploration during training — the policy needs to maintain uncertainty on the prompts it is actively learning from.

Training protocol. Across all experiments, the paper uses the "Zero" RL setting (DeepSeek-AI et al., 2025) — starting RL directly from base pretrained models without supervised fine-tuning. The veRL framework (Sheng et al., 2024) provides the distributed RL infrastructure. Key hyperparameters: policy learning rate $5 \times 10^{-7}$, implicit PRM learning rate $10^{-6}$ (for PRIME only), batch size 256, micro-batch size 128, 512 prompts per rollout with 8 sampled responses per prompt, reference KL coefficient 0 (no KL penalty by default), PPO $\epsilon = 0.2$. Prompts that receive all-correct or all-incorrect responses are filtered out (they provide no advantage signal under group normalization, since all responses have identical rewards).

---

The Entropy–Performance Fitting Methodology

The empirical law $R = -a \exp(\mathcal{H}) + b$ is established through a systematic fitting procedure across all model families and tasks.

Measurement protocol. During RL training, the paper logs two quantities every 4 rollout steps until convergence: (1) the policy entropy on the current training batch, computed as defined above; (2) the validation accuracy (pass@1) on the evaluation benchmarks (MATH500, AIME 2024, AMC, OlympiadBench, OMNI-MATH for math; Eurus-2-RL-Code and KodCode test sets for code). These $( \mathcal{H}, R )$ pairs form the data for curve fitting. For the 11-model sweep in Figure 2, training runs for 2400 gradient steps total, with entropy and performance recorded at 600 points per run.

Fitting procedure. For a given model and training run, all recorded $(\mathcal{H}, R)$ pairs are fit to the functional form:

$R = -a \exp(\mathcal{H}) + b$

using least-squares regression to determine the two coefficients $a > 0$ and $b$. The function is fit to all data points from the entire training run (200+ points across 800+ steps in the main experiments). The exponential form is chosen because it captures the observed concave shape: steep initial improvement as entropy drops from a high initial value, followed by a plateau as entropy approaches zero. The two-parameter form is the simplest that captures this saturation behavior — a linear fit $R = -a\mathcal{H} + b$ would fail because the relationship is strongly nonlinear, and a power-law $R = -a\mathcal{H}^c + b$ would add an unnecessary degree of freedom that overfits given the smooth empirical curves.

What the coefficients mean. Differentiating $R = -a \exp(\mathcal{H}) + b$ with respect to $\mathcal{H}$ yields:

$\frac{dR}{d\mathcal{H}} = -a \exp(\mathcal{H})$

where $dR/d\mathcal{H}$ is the rate at which the model converts a unit of entropy into downstream performance — the marginal efficiency of exploitation. At $\mathcal{H} = 0$ (entropy fully depleted), $R = -a + b$, which is the maximum validation score the model can achieve when the policy becomes deterministic. Thus, $a$ is the conversion rate coefficient (how efficiently entropy is traded for performance) and $b$ is the maximum theoretical performance (the ceiling at zero entropy).

What this fit enables: given measurements of $\mathcal{H}$ and $R$ from the early stage of a training run, fitting $a$ and $b$ from those early points yields a prediction of the entire entropy–performance trajectory, including the final performance $R_{\text{final}} = -a + b$. This is what Figure 5 demonstrates: fitting $a, b$ from only the first 36 training steps (15% of total steps) and extrapolating to predict the next 200 steps yields an average RMSE of 0.9% for math and 1.2% for code.

Why the exponential form: the exponential relationship implies that the marginal return on entropy consumption diminishes as entropy decreases. When entropy is high, consuming a unit of entropy yields a large performance gain ($\exp(\mathcal{H})$ is large); when entropy is low, the same entropy consumption yields a much smaller gain. This is consistent with an exploration-to-exploitation transition: early in training, the policy is exploring widely and quickly learns to rule out obviously wrong patterns (steep improvement); later, the policy is exploiting known patterns and has little entropy left to convert into new improvements (plateau). The exponential form also implies that the performance ceiling is approached asymptotically — it would take infinite entropy consumption to reach it exactly, which is why training plateaus at a finite value before entropy hits exactly zero.

Coefficient extrapolation across model sizes (Figure 7). The coefficients $a$ and $b$ are extracted for each model in the Qwen2.5 family (0.5B, 1.5B, 3B, 7B, 32B parameters, excluding embedding parameters) on both math and coding tasks. The resulting $(N_{\text{params}}, a)$ and $(N_{\text{params}}, b)$ pairs are fit to a log-linear function:

$a = \alpha_a \log(N_{\text{params}}) + \beta_a$ $b = \alpha_b \log(N_{\text{params}}) + \beta_b$

where $\alpha_a, \beta_a, \alpha_b, \beta_b$ are fit coefficients. The result (visible in Figure 7) is that $a$ and $b$ vary smoothly with model size at a log-linear rate — larger models can convert entropy into performance more efficiently (higher $a$) and achieve higher ceilings (higher $b$). This log-linear relationship mimics the observed scaling of loss with model size in pretraining scaling laws (Kaplan et al., 2020; Hoffmann et al., 2022), and it enables extrapolation: fitting the coefficient–parameter relationship from smaller models allows predicting the final RL performance of a larger model without training it.

Algorithm-independence of coefficients (Figure 6). The paper runs GRPO, RLOO, and PRIME on the same model (Qwen2.5-7B) and fits the $R$–$\mathcal{H}$ curves for each. The resulting curves and coefficients $a, b$ are nearly identical across algorithms despite their different advantage estimation methods (group normalization, leave-one-out baseline, implicit PRM). This indicates that the coefficients reflect intrinsic properties of the policy model and training data, not the specific RL algorithm. This is a critical finding for generality: it means the entropy–performance predictability is not an artifact of GRPO's particular advantage estimator but rather a consequence of the policy gradient dynamics interacting with the model's initial knowledge and the problem distribution.

Data-dependence of coefficients (Figure 14). Training Qwen2.5-7B on different datasets (GSM8K vs. Eurus-2-RL-Math) yields different $a, b$ values, confirming that the coefficients are not solely model-specific but depend on the interaction between model capabilities and data difficulty. This is expected: a model trained on harder data will have a lower ceiling $b$ and may convert entropy less efficiently (higher $a$ meaning each unit of entropy consumed yields less performance).

Instruct model fitting (Figure 15). The exponential form also fits when RL is applied to instruct-tuned models (Qwen2.5-Instruct variants), demonstrating that the entropy–performance tradeoff is not unique to base models. This is important because many RL recipes start from instruct models rather than base models.

---

Theoretical Derivation of Entropy Dynamics

The paper's theoretical analysis traces a path from the fundamental properties of softmax policies through the specific update rules of policy gradient algorithms to arrive at the entropy change equation. The derivation is adapted from an unpublished blog post by Liu (2025) and formalized with complete proofs in Appendix E.

Step 1: Entropy difference for any softmax policy (Lemma 1).

The starting point is the recognition that LLMs are softmax policies — their output distribution over the token vocabulary is parameterized by logits $z_{s,a}$ (also denoted $\theta_{s,a}$ in the tabular formulation):

$\pi_\theta(a | s) = \frac{\exp(z_{s,a})}{\sum_{a' \in \mathcal{A}} \exp(z_{s,a'})}$

where $s$ is the state (prompt + prefix) and $a$ is the action (next token). For a tabular softmax policy, each state–action pair has an independent logit parameter — this is a simplification relative to the neural network parameterization of LLMs, but it captures the essential structure because neural networks approximate this tabular mapping and the gradient flow through the softmax output layer follows the same form.

Lemma 1 states that, under a first-order Taylor approximation of the entropy change (valid when the parameter update is small, which holds for typical learning rates like $5 \times 10^{-7}$):

$\mathcal{H}(\pi_{\theta_{k+1}} | s) - \mathcal{H}(\pi_{\theta_k} | s) \approx -\text{Cov}_{a \sim \pi_{\theta_k}(\cdot | s)} \left( \log \pi_{\theta_k}(a | s), z_{s,a}^{k+1} - z_{s,a}^k \right)$

where $\mathcal{H}(\pi_{\theta_k} | s)$ is the entropy of the policy at step $k$ given state $s$, $z_{s,a}^k$ is the logit for action $a$ at step $k$, and $z_{s,a}^{k+1} - z_{s,a}^k$ is the change in that logit after one parameter update. The covariance is computed over the action distribution: for each action, weight the product of (log-probability minus its mean) and (logit change minus its mean) by the action's probability under the current policy.

What Lemma 1 computes: the approximate change in entropy at state $s$ after one gradient step, as the negative of a weighted sum over actions. Actions where high log-probability coincides with large positive logit change contribute positive covariance, which decreases entropy (multiplied by the negative sign). Actions where high log-probability coincides with large negative logit change contribute negative covariance, which increases entropy.

Why this form matters: Lemma 1 decomposes entropy change into a structure that can be analyzed in terms of the update rule. The logit change $z_{s,a}^{k+1} - z_{s,a}^k$ is determined by the RL algorithm, which is where policy gradient enters the picture. The covariance form reveals that entropy change is not about the average logit change or the average log-probability — it is specifically about their joint distribution. This means that two parameter updates with the same average logit change can have opposite effects on entropy depending on which actions receive the changes.

The proof (Appendix E.2) proceeds in three sub-steps. First, Taylor-expand $\mathcal{H}(\pi_{\theta_{k+1}} | s)$ around $\theta_k$ to first order: $\mathcal{H}_{k+1} \approx \mathcal{H}_k + \langle \nabla \mathcal{H}_k, z^{k+1} - z^k \rangle$. Second, compute the gradient of entropy with respect to logits: $\nabla_{\theta_{s,a}} \mathcal{H} = -\mathbb{E}_{a \sim \pi} [\log \pi(a|s) \cdot (\mathbf{1}\{a = a'\} - \pi(a'|s))]$, which follows from the derivative of the softmax log-probability (Lemma 2) and the fact that $\nabla_{\theta} \mathcal{H} = -\nabla_{\theta} \mathbb{E}[\log \pi]$. Third, the inner product $\langle \nabla \mathcal{H}, \Delta z \rangle$ simplifies to $-\text{Cov}(\log \pi, \Delta z)$ after algebraic manipulation that cancels the term involving the expectation of $\log \pi$ times the expectation of $\Delta z$. The covariance emerges because entropy is a convex function of logits — the second-order effect of how probability mass redistributes across actions is captured by the covariance structure at first order.

Step 2: Logit change under policy gradient (Proposition 1).

Proposition 1 connects Lemma 1 to the specific algorithm by deriving the logit change for vanilla policy gradient (using Eq. 2 for gradient estimation):

$z_{s,a}^{k+1} - z_{s,a}^k = \eta \cdot \pi_{\theta}(a | s) \cdot A(s, a)$

where $\eta$ is the learning rate and $A(s, a)$ is the advantage of taking action $a$ in state $s$.

What Proposition 1 computes: the update to each logit is proportional to the product of the action's probability under the current policy and its advantage. Common actions with high advantage get large positive logit increases (making them even more likely); common actions with negative advantage get large negative logit decreases (making them less likely); rare actions get small updates regardless of advantage because $\pi_\theta(a|s)$ is small.

Why this form: this follows directly from the policy gradient theorem and the softmax derivative. By Lemma 2 (Appendix E.1), $\frac{\partial \log \pi_\theta(a|s)}{\partial \theta_{s,a'}} = \mathbf{1}\{a = a'\} - \pi_\theta(a'|s)$. Plugging this into the policy gradient (Eq. 2), the gradient with respect to logit $\theta_{s,a}$ is:

$\nabla_{\theta_{s,a}} J(\theta) = \mathbb{E}_{a' \sim \pi} \left[ (\mathbf{1}\{a = a'\} - \pi_\theta(a|s)) \cdot A(s, a') \right] = \pi_\theta(a|s) A(s, a) - \pi_\theta(a|s) \cdot \mathbb{E}_{a' \sim \pi}[A(s, a')]$

The second term vanishes because the expectation of the advantage under the current policy is zero (Lemma 3: $\mathbb{E}_{a \sim \pi}[A(s, a)] = 0$ by definition of advantage as $Q - V$). Thus, $\nabla_{\theta_{s,a}} J = \pi_\theta(a|s) A(s, a)$, and the logit update is $-\eta$ times this gradient (the negative sign from gradient descent on the negative of the objective, or equivalently gradient ascent on the objective with learning rate $\eta$).

The key property: the logit update scales with $\pi_\theta(a|s)$. This means the policy gradient amplifies the updates of already-probable actions and suppresses updates of rare actions. This is not a design choice — it falls out of the softmax derivative (Lemma 2) and the fact that the gradient of $\log \pi$ with respect to the logit of the chosen action $a$ is $1 - \pi(a|s)$, while the gradient with respect to the logit of a different action $a' \neq a$ is $-\pi(a'|s)$. The $(1 - \pi)$ multiplier means that when an action is already high-probability, its own logit receives a large gradient; when an action is low-probability, its logit receives a small gradient from its own occurrences and only small negative gradients from other actions being chosen.

Step 3: Entropy change under policy gradient (Theorem 1).

Substituting Proposition 1 into Lemma 1 yields the main theoretical result:

$\mathcal{H}(\pi_{\theta_{k+1}} | s) - \mathcal{H}(\pi_{\theta_k} | s) \approx -\eta \cdot \text{Cov}_{a \sim \pi_{\theta_k}(\cdot | s)} \left( \log \pi_{\theta_k}(a | s), \pi_{\theta_k}(a | s) \cdot A(s, a) \right)$

What Theorem 1 computes: the entropy change after one policy gradient step is proportional to the covariance between an action's log-probability and the product of its probability and advantage. If actions with high log-probability (i.e., high probability before the update) also tend to have high $\pi \cdot A$ (i.e., been frequently chosen and received high advantage), then the covariance is positive and entropy decreases. If some low-probability actions have exceptionally high advantage, they could create negative covariance and increase entropy — but this requires the advantage to overcome the $\pi$ multiplier, which suppresses rare actions.

Why entropy decreases monotonically in practice: For covariance to be positive, the policy must assign higher-than-average log-probability to actions that also receive higher-than-average $\pi \cdot A$. At the start of RL training, the base model's output distribution is roughly calibrated — the model tends to be more confident on tokens where it is actually correct (Kadavath et al., 2022). This means that when the policy is confident (high $\pi$, high $\log \pi$), those tokens often appear in trajectories that turn out correct (positive $A$), yielding high $\pi \cdot A$. The covariance is positive, entropy drops. As training continues, the policy becomes more confident on tokens that led to success, further increasing $\pi$ for those tokens, which further increases $\pi \cdot A$ when advantage remains positive — a positive feedback loop that drives entropy to zero. The empirical validation (Figure 8, left) confirms this: covariance remains positive throughout training, and its magnitude closely tracks $-$entropy change ($-d(\mathcal{H})$).

The covariance structure also explains the difficulty-dependent behavior observed in Figure 8 (right). On easy prompts (high accuracy), the policy is well-calibrated: confident tokens reliably lead to correct answers, producing high positive covariance and rapid entropy depletion — the model quickly "locks in" on successful strategies. On hard prompts (low accuracy), the policy is poorly calibrated: confident tokens are not reliably associated with correct answers, advantage signals are noisier and often negative even for high-probability tokens, producing lower covariance and slower entropy change — the model remains uncertain because its confidence isn't rewarded. This graded response matches the intuition that easier problems allow faster exploitation while harder problems resist convergence because the policy hasn't yet identified reliable patterns.

Step 4: Natural Policy Gradient variant (Theorem 2).

For completeness, the paper also presents the NPG result (adapted from Liu, 2025; proof in Appendix E.4):

$\mathcal{H}(\pi_{\theta_{k+1}} | s) - \mathcal{H}(\pi_{\theta_k} | s) \approx -\eta \cdot \text{Cov}_{a \sim \pi_{\theta_k}(\cdot | s)} \left( \log \pi_{\theta_k}(a | s), A(s, a) \right)$

Under NPG, the logit change is $z^{k+1} - z^k = \eta \cdot A(s, a)$ — directly proportional to the advantage, without the $\pi_\theta(a|s)$ multiplier. This is because NPG preconditions the gradient by the inverse Fisher information matrix, which for softmax policies exactly cancels the $\pi$ factor in the standard policy gradient (Agarwal et al., 2021).

Why Theorem 2 matters for PPO: NPG is rarely used in LLM post-training due to its second-order optimization cost, but its connection to TRPO and PPO is fundamental — both approximate the NPG update by enforcing KL constraints on policy change. The PPO objective (Eq. 4) can be viewed as approximating an NPG step with a KL penalty, which means the NPG-form covariance (without the $\pi$ multiplier) is the relevant one for Clip-Cov and KL-Cov. This is why the paper uses the unweighted covariance formulation (Eq. 10) when implementing the entropy-control methods — it corresponds to the NPG/PPO regime where the $\pi$ pre-factor is absorbed by the trust-region constraint.

The takeaway from the theoretical analysis: entropy in softmax policies decreases when the policy's confidence is well-calibrated to advantage, which is exactly the situation at the start of RL training for LLMs. The decline continues because the policy gradient intrinsically amplifies updates to high-probability, high-advantage actions, creating a self-reinforcing cycle. To break the cycle, one must intervene not on the objective (adding entropy bonuses) but on the gradient-level mechanism by restricting the updates of the high-covariance tokens that drive the cycle. This theoretical insight directly motivates Clip-Cov and KL-Cov.

---

Empirical Validation of the Covariance Mechanism (Section 3.3)

To test Theorem 1 experimentally, the paper runs GRPO with policy gradient (on-policy learning without PPO surrogate) on Qwen2.5-7B and tracks two quantities throughout training: the step-wise entropy difference $d\mathcal{H} = \mathcal{H}_k - \mathcal{H}_{k-1}$ and the group-wise covariance term from Eq. 8.

Bandit setting for LLM RL. In the LLM RL setup, the state $s$ is the prompt $\boldsymbol{x}$ and the action $a$ is the entire response $\boldsymbol{y}$. This is a bandit formulation — there is no sequential decision-making within a response from the RL perspective because the advantage is assigned to the entire trajectory (or decomposed to tokens via the summation in Eq. 2). The covariance becomes:

$\text{Cov}_{\boldsymbol{y} \sim \pi_\theta(\cdot | \boldsymbol{x})} \left( \log \pi_\theta(\boldsymbol{y} | \boldsymbol{x}), \pi_\theta(\boldsymbol{y} | \boldsymbol{x}) \cdot A(\boldsymbol{y}, \boldsymbol{x}) \right)$

where $\log \pi_\theta(\boldsymbol{y} | \boldsymbol{x})$ is the sequence log-probability normalized by response length:

$\log \pi_\theta(\boldsymbol{y} | \boldsymbol{x}) = \frac{1}{|\boldsymbol{y}|} \left[ \sum_{t=1}^{|\boldsymbol{y}|} \log \pi_\theta(y_t | \boldsymbol{y}_{<t}, \boldsymbol{x}) \right]$

Why length normalization: without normalization, longer responses would have systematically lower log-probabilities (more tokens to sum over, each contributing a negative value), which would bias the covariance toward long responses regardless of their quality. Normalizing by length makes the log-probability comparable across responses of different lengths — it becomes the average per-token log-probability.

Group-wise covariance computation. For each prompt $\boldsymbol{x}$, the $K = 8$ sampled responses form a group. The log-probability and $A$ (computed as the group-normalized z-score reward) are computed for each of the $K$ responses, and the covariance is computed within this group (using the standard sample covariance formula). The group-wise covariances are then averaged across all prompts in the batch (256 prompts). This per-prompt grouping mirrors the advantage normalization — it ensures the covariance measures within-prompt alignment between confidence and success, not across-prompt confounds (e.g., some prompts having higher average confidence and higher average rewards because they are easier).

Result 1: Covariance and entropy difference are tightly coupled (Figure 8, left). The empirical curves of $\text{Cov}(\cdot)$ and $-d\mathcal{H}$ show "highly similar dynamics." Both start large and positive early in training, then decline gradually toward smaller positive values as training progresses. The correlation supports Theorem 1's prediction that entropy change is proportional to covariance, with the proportionality constant being the (negative) learning rate. The covariance remains positive throughout training — it never crosses zero into the regime where entropy would increase — which explains why entropy "decreases monotonically" across all experiments. The persistent positivity means that, on average, high-probability actions continue to be associated with positive advantage even late in training, though the association weakens as the policy converges and advantage signals become noisier (fewer mistakes to learn from).

Result 2: Covariance varies systematically with prompt difficulty (Figure 8, right). The paper splits training examples into three difficulty groups based on the accuracy of the $K = 8$ responses to each prompt: high accuracy (easy), medium accuracy, and low accuracy (hard). The covariance curves for these groups differ substantially:

• Easy prompts show the highest covariance, especially early in training. The policy is well-calibrated — when it's confident, it's usually right — so high-probability responses reliably receive positive advantage.
• Medium prompts show intermediate covariance. The policy has some signal but also noise — confidence is partially calibrated.
• Hard prompts show the lowest covariance, often close to zero. The policy's confidence is not predictive of success; high-probability responses are as likely to be wrong as right, so the covariance between log-probability and advantage is weak.

This difficulty-dependent behavior qualitatively explains the entropy–performance curves found in Section 2: easy problems are "exploited" quickly (high covariance → fast entropy drop → fast performance gain → early plateau), while hard problems resist exploitation (low covariance → slow entropy drop → slow performance gain → lower ceiling). The covariance analysis thus provides a mechanistic account of why the exponential $R$–$\mathcal{H}$ relationship emerges and why different difficulty distributions produce different coefficients $a, b$.

---

Failure of Conventional Entropy Regularization (Section 4.1)

Before introducing the proposed methods, the paper demonstrates why the standard maximum-entropy RL toolkit fails for LLMs, providing negative results that motivate the need for a fundamentally different approach.

Entropy loss experiment (Figure 9). The policy loss is modified to include an entropy bonus:

$\mathcal{L}_{\text{ent}} = \mathcal{L} - \alpha \mathcal{H}(\pi_\theta)$

where $\mathcal{L}$ is the original PPO/GRPO loss, $\alpha$ is the entropy coefficient, and $\mathcal{H}(\pi_\theta)$ is the batch-average policy entropy. Four values of $\alpha$ are tested: 0.0001, 0.001, 0.005, and 0.01.

Results: The method is extremely hyperparameter-sensitive:

• At $\alpha = 0.0001$ and $0.001$, entropy still collapses — the bonus is too small to counteract the covariance-driven entropy decline. The curves are nearly indistinguishable from the no-entropy-bonus baseline.
• At $\alpha = 0.01$, entropy explodes — the bonus dominates the loss, and the policy's uncertainty diverges rather than stabilizing. This is the classic failure mode of entropy regularization: when $\alpha$ is too large, the policy optimizes for entropy rather than reward.
• At $\alpha = 0.005$, entropy stabilizes at an intermediate level — but validation performance does not improve over the baseline. The model maintains exploration but doesn't convert it into better reasoning.

Why entropy loss fails for LLMs: the entropy term is a global average over all tokens and all prompts. It provides the same incentive to increase entropy on every token, regardless of whether that token's entropy is productive (enabling discovery of new reasoning strategies) or unproductive (adding noise to already-mastered patterns). In maximum-entropy RL for robotics or game-playing, the action space is small and all states benefit similarly from exploration. For LLMs with a 50,000+ token vocabulary and massively heterogeneous prompts (easy vs. hard, math vs. coding), a single global entropy bonus cannot differentiate between useful and wasteful uncertainty. Moreover, the covariance analysis shows that entropy decline is driven by a specific subset of high-covariance tokens — the entropy loss applies a uniform force against this decline, fighting the mechanism everywhere rather than targeting it where it matters.

KL regularization experiment (Figure 10). The policy loss is modified to include a KL penalty toward the initial (pre-RL) reference model:

$\mathcal{L}_{\text{KL}} = \mathcal{L} + \beta \mathbb{D}_{\text{KL}}(\pi_\theta \| \pi_{\text{ref}})$

where $\beta$ is the KL coefficient and $\mathbb{D}_{\text{KL}}$ is the Kullback-Leibler divergence between the current policy and the reference (base) model. Three values of $\beta$ are tested: 0.001, 0.005, and 0.01.

Results: KL regularization successfully preserves entropy — the entropy curves stabilize at non-zero values for all $\beta$ tested. However, validation performance degrades relative to the baseline, with larger $\beta$ producing larger performance drops. The KL penalty prevents the policy from moving toward any strategy not already present in the base model, effectively capping improvement at the base model's capability. This is the exactly wrong direction: RL is supposed to improve beyond the base model, but KL regularization constrains it to stay close to the base model. The method preserves entropy at the cost of preventing learning, which is why recent works on RL for LLM reasoning typically set the KL coefficient to zero (as the paper's default protocol does).

Why KL regularization fails for RL reasoning: in RLHF alignment (Ouyang et al., 2022), KL regularization toward the supervised fine-tuned model is essential because the reward model is a learned proxy that can be gamed — the policy might drift into regions where the reward model gives spuriously high scores but generations are nonsensical. The KL penalty prevents this reward hacking. But when rewards are verifiable (correct/incorrect math answers), there is no proxy to hack — the reward is ground truth. The only reason to stay near the reference model would be to preserve general capabilities, but for reasoning-specific RL, the goal is precisely to change the model's behavior on reasoning tasks. KL regularization thus fights the very purpose of the training.

Implication for entropy-control design: these negative results pin down the requirements for any successful entropy-control method for LLM RL. It must (1) be sensitive to which tokens lose entropy and why — not a global average; (2) preserve the ability to learn and improve beyond the base model — not constrain deviations; and (3) be stable and not hypersensitive to hyperparameters — avoid the entropy loss failure mode of small changes causing large entropy shifts. The covariance analysis provides exactly the token-level signal needed: target the tokens where covariance is high, because those are the ones driving the collapse.

---

Clip-Cov and KL-Cov Mechanisms (Section 4.2)

The paper proposes two methods that implement entropy control via covariance-aware token selection, modifying only a few lines of code in the standard PPO loss computation.

Token-level covariance computation (Eq. 10). For a batch of $N$ rollout tokens (each token $y_i$ is a specific token at a specific position in a specific response; $N$ is the total number of tokens across all responses in the batch), the token-wise centered cross-product is:

$\text{Cov}(y_i) = \left( \log \pi_\theta(y_i) - \frac{1}{N} \sum_{j=1}^{N} \log \pi_\theta(y_j) \right) \cdot \left( A(y_i) - \frac{1}{N} \sum_{j=1}^{N} A(y_j) \right)$

where $\log \pi_\theta(y_i)$ is the log-probability of token $y_i$ under the current policy (given its prefix context), and $A(y_i)$ is the token-level advantage. In the bandit formulation used in practice, $A(y_i) = A(\boldsymbol{y})$ is the same for all tokens in a response — the trajectory-level advantage — since per-token advantages are not estimated by GRPO.

What this computes: for each token, the deviation of that token's log-probability from the batch-mean log-probability, multiplied by the deviation of that token's advantage from the batch-mean advantage. The expectation of $\text{Cov}(y_i)$ over the batch equals the covariance in Theorem 2 (up to a constant factor from the batch size normalization). Tokens with high $\text{Cov}(y_i)$ are those where (a) the policy is much more confident than average on that token, AND (b) that token's response received much higher-than-average advantage. These are precisely the tokens driving entropy collapse per the theoretical analysis.

Why this formulation uses unweighted covariance (no $\pi$ multiplier): per the discussion of Theorem 2 vs. Theorem 1, this corresponds to the NPG/PPO regime. The covariance is between $\log \pi$ and $A$ directly, not $\log \pi$ and $\pi \cdot A$. The justification is that PPO's clipping and the KL constraint approximated by the surrogate loss effectively pre-multiply by the inverse Fisher information, canceling the $\pi$ factor, similar to how NPG does it analytically. In practice, this means the method targets tokens where log-probability and advantage are jointly high, without the additional suppression of low-probability tokens that the standard policy gradient's $\pi$ factor would cause.

Covariance distribution (Table 1). Analysis of Qwen2.5-7B at training step 1 reveals an extremely skewed distribution: the mean covariance across all tokens is only 0.003, but the top percentiles are orders of magnitude larger. The top 0.02% of tokens have mean covariance 5.654 (more than 1800× the average), the top 0.2% have mean 3.112, and the top 2% have mean 1.385. The top 50% of tokens have mean 0.152, which is 50× the overall mean but still dwarfed by the extreme right tail. This extreme skew means that a tiny fraction of tokens drives the bulk of the entropy decline — which is why Clip-Cov and KL-Cov can be effective while interfering with only $10^{-4}$ to $10^{-3}$ of tokens.

Clip-Cov method (Eqs. 11–12). The procedure has three steps:

1. Compute $\text{Cov}(y_i)$ for all $N$ tokens in the batch using Eq. 10.
2. Select a set $I_{\text{clip}}$ of tokens for gradient detachment. The selection is random among high-covariance tokens: from the set of tokens where $\text{Cov}(y_i)$ falls within the bounds $[\omega_{\text{low}}, \omega_{\text{high}}]$, randomly sample $\lfloor r \cdot N \rfloor$ tokens (without replacement). The bounds $\omega_{\text{low}}$ and $\omega_{\text{high}}$ are set "much higher than the average covariance ($>500\times$)". The ratio $r$ is the fraction of tokens to clip — in the main experiments, $r = 2 \times 10^{-4}$ (0.02% of tokens). So for a batch of, say, 1 million tokens, only 200 tokens are clipped.
3. Modify the PPO loss so that the gradient is zero for the selected tokens:

$L_{\text{Clip-Cov}}(\theta) = \begin{cases} \mathbb{E}_t \left[ \frac{\pi_\theta(y_t | \boldsymbol{y}_{<t})}{\pi_{\theta_{\text{old}}}(y_t | \boldsymbol{y}_{<t})} A_t \right], & t \notin I_{\text{clip}} \\ 0, & t \in I_{\text{clip}} \end{cases}$

The standard PPO clipped surrogate (Eq. 4) is used as the base loss for non-clipped tokens; the clipped tokens are simply detached (their contribution to the gradient is zero). The detach_() operation in PyTorch severs the gradient flow through those tokens' log-probabilities, preventing the policy from updating to reinforce (or suppress) those tokens.

What Clip-Cov does mechanistically: the highest-covariance tokens — those where the policy is most confidently aligned with positive advantage — are the primary drivers of entropy decline because, per Theorem 1/2, updating them further increases their probability, making the policy even more deterministic. By preventing gradient updates on these tokens, Clip-Cov stops the positive feedback loop at its source. The policy can still update other tokens (moderate and low covariance), learning from them normally, but the small set of tokens that would otherwise dominate and collapse the entropy are held fixed.

Why random selection within the high-covariance range: the randomness prevents the method from deterministically blocking the exact same tokens every step, which could create a systematic bias where certain tokens are permanently frozen. By randomly sampling a subset, each high-covariance token has a chance of being updated in some steps and blocked in others, allowing learning to proceed without the extreme entropy drain.

The bounds $\omega_{\text{low}}, \omega_{\text{high}}$: setting lower bound $\omega_{\text{low}} = 1$ and upper bound $\omega_{\text{high}} = 5$ (as in the experiments) means the method targets tokens in a specific high-covariance regime — above the bulk of the distribution (mean 0.003) but not the extreme outliers (mean 5.654 in the top 0.02%). This avoids the most extreme tokens (which might be too few to matter or might be pathological noise) and focuses on the "sweet spot" of tokens that are numerous enough to affect aggregate entropy but have large enough covariance to drive decline.

KL-Cov method (Eqs. 13–14). This method applies a KL penalty selectively to high-covariance tokens rather than detaching their gradients:

1. Compute $\text{Cov}(y_i)$ for all tokens as in Eq. 10.
2. Select the top-$k$ proportion of tokens by covariance: $I_{\text{KL}} = \{ i \mid \text{Rank}(\text{Cov}(y_i)) \leq k \cdot N \}$, where $k \ll 1$. In the main experiments, $k = 2 \times 10^{-3}$ for Qwen2.5-7B and $k = 2 \times 10^{-4}$ for Qwen2.5-32B.
3. Modify the loss to add a KL penalty term for the selected tokens:

$L_{\text{KL-Cov}}(\theta) = \begin{cases} \mathbb{E}_t \left[ \frac{\pi_\theta(y_t | \boldsymbol{y}_{<t})}{\pi_{\theta_{\text{old}}}(y_t | \boldsymbol{y}_{<t})} A_t \right], & t \notin I_{\text{KL}} \\ \mathbb{E}_t \left[ \frac{\pi_\theta(y_t | \boldsymbol{y}_{<t})}{\pi_{\theta_{\text{old}}}(y_t | \boldsymbol{y}_{<t})} A_t - \beta \mathbb{D}_{\text{KL}}(\pi_{\theta_{\text{old}}}(y_t | \boldsymbol{y}_{<t}) \| \pi_\theta(y_t | \boldsymbol{y}_{<t})) \right], & t \in I_{\text{KL}} \end{cases}$

where $\beta$ is the KL penalty coefficient (set to 1 in the main experiments), and the KL divergence is between the old (rollout) policy distribution at that token position and the current policy distribution. The KL is computed backward — $\mathbb{D}_{\text{KL}}(\pi_{\text{old}} \| \pi_\theta)$ — meaning it penalizes the current policy for assigning low probability to tokens that the old policy assigned high probability to.

What KL-Cov does mechanistically: for the selected high-covariance tokens, the policy receives a penalty if it moves away from the old (rollout) policy's distribution. This penalty counteracts the policy gradient's push to increase the probability of these tokens, serving as a brake on the entropy-depleting update. For tokens not in $I_{\text{KL}}$ (the vast majority), the standard PPO loss applies without modification. The method is analogous to per-token KL regularization, but applied only where needed — the high-covariance tokens — rather than globally. The KL coefficient $\beta = 1$ is much larger than the global KL coefficients tested in Section 4.1 (0.001–0.01), but because it applies to only $2 \times 10^{-4}$ to $2 \times 10^{-3}$ of tokens, the total KL penalty contribution to the loss is small relative to the policy gradient term.

Why backward KL ($\mathbb{D}_{\text{KL}}(\pi_{\text{old}} \| \pi_\theta)$): in standard PPO, the forward KL ($\mathbb{D}_{\text{KL}}(\pi_\theta \| \pi_{\text{old}})$) or symmetric KL is typically used for trust-region constraints. The backward KL has the property that it heavily penalizes the new policy for ignoring actions that the old policy thought were likely (mode-seeking behavior). For entropy control, this is exactly what's needed: the high-covariance tokens are ones the old policy was already confident on, and the penalty prevents the new policy from collapsing even further onto those tokens by maintaining some probability mass on other actions. Forward KL would penalize the new policy for trying novel actions, which is the opposite of what's intended.

Hyperparameter sensitivity and entropy controllability (Figure 13). The paper demonstrates that both methods provide a controllable "knob" for entropy:

• For Clip-Cov, increasing the clip ratio $r$ clips more tokens, preserving more entropy. The entropy curves shift upward monotonically with $r$.
• For KL-Cov, increasing the KL coefficient $\beta$ applies a stronger penalty on the selected tokens, preserving more entropy. The entropy curves shift upward monotonically with $\beta$.
• KL-Cov produces more stable entropy curves than Clip-Cov. Clip-Cov's random selection introduces stochasticity that causes entropy to fluctuate; KL-Cov's deterministic top-$k$ selection and continuous penalty produce smoother entropy trajectories. This makes KL-Cov potentially preferable for stabilizing training.

The key numbers that make this work: the fraction of intervened tokens is tiny — $2 \times 10^{-4}$ to $2 \times 10^{-3}$ — yet this small intervention radically changes the entropy trajectory (moving from near-zero entropy to maintaining $>10\times$ higher entropy than the baseline, as shown in Figure 12, left). This confirms the covariance distribution analysis (Table 1): a small number of outlier tokens disproportionately drive entropy dynamics, and targeting them is both necessary and sufficient.

Comparison with clip-higher (Section 4.5 discussion). The clip-higher baseline raises PPO's upper clipping threshold $\epsilon$ from 0.2 to 0.28 (affecting only positively-advantaged tokens moving above the ratio bound). The paper analyzes this as a coarse form of covariance-aware entropy control: by raising the upper threshold, clip-higher includes more low-probability tokens in the gradient update. Specifically, tokens with importance-sampling ratios in $[1.2, 1.28]$ that would have been clipped now contribute to the gradient. These tokens have low probability (otherwise their ratio wouldn't be high — they were unlikely under the old policy) and positive advantage (otherwise the upper clip wouldn't matter), so they have low covariance (low probability, high advantage: $\text{Cov} \sim -0.03$). Clip-higher thus dilutes the high-covariance tokens' influence by adding more low-covariance tokens to the gradient, rather than directly suppressing the high-covariance ones. This indirect mechanism is less stable and less precise than Clip-Cov/KL-Cov's direct targeting.

Implementation simplicity (Listing 11). The pseudocode in the paper shows that both methods can be implemented by modifying fewer than 10 lines of code in the standard PPO loss function. The core addition is computing the token-wise $\text{Cov}(y_i)$ (Eq. 10) and then either detaching gradients for selected tokens (Clip-Cov) or adding a KL term to the loss for selected tokens (KL-Cov). The selection hyperparameters ($r, \omega_{\text{low}}, \omega_{\text{high}}$ for Clip-Cov; $k, \beta$ for KL-Cov) are the only new knobs. This minimal implementation cost makes the methods immediately adoptable in existing RL-for-LLM pipelines (veRL, OpenRLHF, etc.) without architectural changes.

Experimental validation (Section 4.3, Table 2, Figure 12). The methods are tested on Qwen2.5-7B and Qwen2.5-32B trained on the DAPO-MATH dataset using GRPO. Training setup: 256 prompts per rollout step, 8 responses per prompt, temperature 1 for sampling, 8 policy updates per collected batch, prompts with all-correct/all-incorrect responses filtered, max generation length 8192. Test benchmarks: MATH500, AIME 2024, AIME 2025, AMC, OMNI-MATH, OlympiadBench, Minerva. Evaluation: temperature 0.6 for AIME/AMC (requiring multiple samples and majority voting to assess), greedy decoding for other benchmarks.

Key results from Table 2:

• Qwen2.5-7B: GRPO baseline averages 38.6% across 7 benchmarks. Clip-Cov improves to 40.4% (+1.8 points average). KL-Cov improves to 40.6% (+2.0 points average). Clip-higher is marginally better than GRPO (38.8%, +0.2 points) but less than the proposed methods.
• Qwen2.5-32B: GRPO baseline averages 45.8%. Clip-Cov improves to 50.3% (+4.5 points average). KL-Cov improves to 52.2% (+6.4 points average). The gains are substantially larger on the larger model. On the most challenging benchmarks, AIME24 and AIME25, KL-Cov achieves +15.0% and +14.6% over GRPO respectively (36.8% vs. 21.8% on AIME24; 30.8% vs. 16.2% on AIME25).
• Clip-higher shows instability on 32B: it achieves 47.2% average (+1.4 points) but degrades on MATH500 (77.2% vs. 84.2% for GRPO), suggesting the indirect entropy-control approach causes performance regression on some benchmarks.

Why larger models benefit more (Section 4.3 discussion): the paper hypothesizes that larger models "possess greater potential from pretraining" — they have more latent reasoning strategies that are not expressed by the base model's default (low-entropy) sampling distribution. Once the "exploration curse" is lifted by maintaining higher entropy, the 32B model can discover and reinforce these latent strategies, yielding larger gains than the 7B model, whose pretrained capacity is more limited. This connects back to the coefficient analysis (Figure 7): larger models have higher $b$ (higher ceiling), and entropy control raises the achievable fraction of that ceiling by preventing premature convergence.

Response length increases (Figure 12, middle): both Clip-Cov and KL-Cov cause the model to generate longer responses than vanilla GRPO. This is a natural consequence of maintained entropy: when the policy is less certain, it tends to generate more detailed reasoning chains (more tokens) rather than converging to short, confident outputs. The increased length is a sign of continued exploration — the model is trying more varied reasoning paths rather than locking into the shortest successful pattern.

4. Key Insights and Innovations

Innovation 1: Entropy Collapse Is a Deterministic, Predictable Tradeoff — Not a Pathological Failure

The most intellectually distinctive contribution of this paper is the reframing of entropy collapse from a training pathology to a deterministic, predictable, and quantifiable tradeoff between exploration and performance. Prior to this work, the rapid decline of policy entropy in LLM RL was either unnoticed (most RL-for-reasoning papers focused on final accuracy, not training dynamics), treated as an incidental nuisance to be worked around (Yu et al., 2025 noting it in passing and proposing clip-higher as a heuristic fix), or implicitly accepted by omitting entropy regularization altogether (Cui et al., 2025; Hu et al., 2025; Liu et al., 2025). The default stance was "entropy drops fast, and we don't know why, but standard entropy bonuses don't help, so let's ignore it."

This paper fundamentally shifts that stance by establishing that the entropy–performance relationship is not noise or an artifact — it is a lawful empirical regularity captured by the remarkably simple two-parameter exponential function $R = -a \exp(\mathcal{H}) + b$. The evidence for this is the sweep across 11 models from 4 families on both math and coding, where over 200 data points per run are fit with negligible error using only two coefficients (Figures 3, 4). The precision of the fit — not "roughly exponential" but "exactly exponential across all scales and tasks" — forces the conclusion that something deep about the RL dynamics is being captured.

What makes this genuinely novel rather than just a curve-fitting exercise is the predictive capability it unlocks. The finding that coefficients fitted from the first 15% of training steps predict final performance with 0.9–1.2% RMSE (Figure 5) transforms RL training from a black-box process into a forecastable one. This is conceptually analogous to what Kaplan et al. (2020) and Hoffmann et al. (2022) did for pretraining — converting an expensive empirical process into a predictable function of measurable quantities — but applied to the underexplored regime of RL dynamics rather than pretraining loss. The further finding that coefficients $a$ and $b$ vary log-linearly with model size (Figure 7) extends this predictability across model scales: one can train small models, extract their coefficients, and predict the RL performance ceiling of larger models without running the full training. This is not just a practical convenience; it structurally changes how one thinks about allocating RL compute, because the return on investment can be estimated before committing resources.

The predictability finding also carries a subtle but important philosophical implication about the nature of RL for LLM reasoning. If the entropy–performance trajectory is predetermined by the base model and training data (the coefficients are algorithm-independent, as shown in Figure 6; data-dependent, as shown in Figure 14), then RL in the absence of entropy intervention is primarily an elicitation process, not a learning process. The model is not discovering genuinely new capabilities; it is surfacing and reinforcing patterns already latent in its pretrained distribution, with the rate and ceiling of this surfacing dictated by the coefficients $a$ and $b$. This provides mechanistic support for the "elicitation vs. learning" debate (Yue et al., 2025) while also showing that the elicitation ceiling is not intrinsic to RL — it is imposed by entropy dynamics, and breaking those dynamics (as Clip-Cov and KL-Cov do) allows the model to exceed the predicted ceiling. The paper thus resolves a tension in the literature: RL can go beyond the base model, but only if entropy is actively managed; without management, it merely elicits with diminishing returns.

Innovation 2: The Covariance Mechanism Provides a Principled, Token-Level Explanation for When and Why Entropy Changes

The paper's second major conceptual advance is the theoretical and empirical demonstration that entropy dynamics in softmax policies are governed by a single, interpretable quantity: the covariance between log-probability and advantage (or log-probability and logit change, in the more general form). This is not a statistical correlation found post-hoc — it is derived from first principles (Lemma 1, Theorem 1) from the structure of the softmax parameterization and the policy gradient update rule, and then validated with close quantitative agreement between the covariance term and the measured entropy difference (Figure 8, left).

Why is this intellectually distinctive? Prior work on entropy in RL — including the maximum-entropy RL literature that this paper builds on (Ziebart et al., 2008; Haarnoja et al., 2018) — treated entropy as a quantity to be controlled (added to the reward, constrained by KL penalties) but not as a quantity whose dynamics could be analyzed from the structure of the algorithm. The standard approach was prescriptive: "add an entropy bonus to prevent premature convergence." This paper's approach is diagnostic: "the entropy change at each step equals (negative) the covariance between confidence and advantage, so to prevent collapse, address the covariance." This is a fundamentally different level of understanding — it explains not just that entropy should be maintained, but precisely which parameter updates cause it to decrease and therefore where to intervene.

The covariance mechanism also provides a unified explanation for a diverse set of empirical observations that were previously unexplained or explained only qualitatively. Why does entropy decrease monotonically? Because covariance is positive throughout training — high-probability actions tend to have positive advantage, especially early on when the base model is well-calibrated (Kadavath et al., 2022). Why do easy problems experience faster entropy collapse than hard problems? Because on easy problems, the policy's confidence is better calibrated to correctness, producing higher covariance (Figure 8, right). Why does entropy loss fail with small coefficients? Because a global entropy bonus does not counteract the specific high-covariance tokens driving the collapse; it applies a diffuse force everywhere, and the collapse is driven by a concentrated subset of tokens (Table 1: top 0.02% of tokens have >1800× the average covariance). Why does KL regularization preserve entropy but degrade performance? Because it constrains the policy globally to the reference distribution, preventing both the harmful high-covariance updates and the beneficial low-covariance updates that would improve reasoning. The covariance lens makes these previously puzzling results coherent.

The theoretical derivation itself, while building on Liu's (2025) unpublished blog post, is elevated here by rigorous formalization (complete proofs in Appendix E) and by the connection to the specific algorithms used in LLM RL (GRPO, PPO, NPG). The distinction between the standard PG form (Theorem 1: covariance with $\pi \cdot A$) and the NPG form (Theorem 2: covariance with $A$ alone) is not just a mathematical curiosity — it directly informs the design of Clip-Cov and KL-Cov, which use the NPG formulation because PPO's clipping approximates the inverse-Fisher preconditioning. This algorithmic awareness is what converts a theoretical insight into a practical intervention.

Innovation 3: Covariance-Aware Token-Level Intervention Is a New Class of Entropy-Control Mechanism

The proposed methods, Clip-Cov and KL-Cov, represent not just two new techniques but a new category of entropy-control strategy that is fundamentally distinct from both entropy regularization and KL regularization. The key conceptual move is to shift from objective-level intervention (adding entropy terms to the reward or loss) to gradient-level intervention (selectively blocking or penalizing updates to specific tokens based on their role in the entropy dynamics).

This shift is significant because the failure modes of objective-level entropy control for LLMs are structural, not merely a matter of finding the right hyperparameters. As Section 4.1 demonstrates, entropy loss fails because it applies uniformly to all tokens regardless of their covariance — it fights the entropy decline everywhere, including on tokens where the decline is harmless or even beneficial (tokens where the policy should become more confident because it has genuinely learned the correct pattern). KL regularization fails because it constrains the policy from moving anywhere, including toward improved reasoning strategies that differ from the base model. Neither approach has the selectivity needed to target the specific mechanism driving entropy collapse without also impeding learning.

Clip-Cov and KL-Cov achieve this selectivity by using the covariance itself as the selection criterion — exactly the quantity that Theorem 2 identifies as the entropy-change driver. This is an elegant conceptual closure: the theory tells you what to measure (covariance between log-probability and advantage), and the method intervenes precisely where that measurement is high. The tiny fraction of tokens intervened on ($2 \times 10^{-4}$ to $2 \times 10^{-3}$) is a direct consequence of the extreme skew in the covariance distribution (Table 1), which the methods discover and exploit rather than assume. This is not a hyperparameter that required extensive tuning to find — the skew is a property of the training dynamics that the theory predicts and the data confirms.

The comparison with clip-higher (Yu et al., 2025) is instructive for understanding the novelty. Clip-higher operates by adding low-covariance tokens (low probability, high advantage) to the gradient — it dilutes the influence of high-covariance tokens rather than directly suppressing them. The paper's analysis (Section 4.5) that clip-higher tokens have average covariance ~−0.03, while the targeted tokens in Clip-Cov/KL-Cov have covariances of 1–5+, reveals why the indirect approach is less stable and less effective: it doesn't remove the dominant entropy-depleting signal, it just adds noise around it. The instability of clip-higher on larger models (Table 2: it regresses on MATH500 for Qwen2.5-32B) is consistent with this interpretation — as the model gets larger and the covariance distribution becomes even more skewed, diluting the high-covariance signal becomes harder without also diluting the learning signal.

A subtle but important property of Clip-Cov and KL-Cov is that they are entropy-controllable but not entropy-prescribing. The methods provide a knob for adjusting how much entropy is maintained (Figure 13), but they do not enforce a target entropy value. This is philosophically different from maximum-entropy RL, which optimizes a specific entropy level via the coefficient $\alpha$ in $\mathcal{L} - \alpha \mathcal{H}$. The paper's methods say, essentially: "let the policy learn normally, but prevent the specific updates that cause runaway entropy collapse; the resulting entropy level will be whatever emerges from that constraint." This emergent-entropy approach avoids the need to specify the "right" entropy level — an open question the paper explicitly acknowledges — and instead focuses on removing the pathological mechanism. The fact that this approach yields both higher entropy and better performance (Table 2) suggests that the entropy level emerging from covariance-controlled learning is closer to optimal than either the collapsed entropy of vanilla GRPO or the artificially inflated entropy of entropy-regularized training.

Innovation 4: Entropy Collapse as the Bottleneck Connecting Pretraining Capacity to RL Scalability

The paper's findings collectively articulate a new diagnosis for why RL for LLM reasoning plateaus: it is not that the base model lacks latent reasoning capability, and it is not that the RL algorithm is fundamentally limited — it is that entropy collapse traps the policy in a narrow region of its output space before it can explore and reinforce the full range of strategies the base model could support. This diagnosis has implications that extend well beyond the specific methods proposed.

The most striking evidence for this diagnosis is the interaction between model size and the entropy-control methods' effectiveness (Table 2). On Qwen2.5-7B, Clip-Cov and KL-Cov improve average performance by ~2 points; on Qwen2.5-32B, they improve by ~5–6 points, with gains of +15.0% and +14.6% on the hardest benchmarks (AIME24, AIME25). This non-linear scaling of the benefit with model size suggests that larger models have disproportionately more latent reasoning capacity that remains inaccessible under entropy collapse. The base model's pretraining imbues it with diverse reasoning strategies, but without entropy control, RL converges to a small subset before it can discover and reinforce the others. The entropy bottleneck thus acts as a capacity gate: the effective RL-accessible capacity of a model is not its total pretrained capacity, but only the fraction that can be explored before entropy collapses.

This reframes the scaling narrative for RL compute. The "Bitter Lesson" (Sutton, 2019) and the recent push toward "learning from experience" (Silver & Sutton, 2025) argue that compute should shift from pretraining to RL because general methods that scale with computation ultimately outperform hand-crafted approaches. But if RL's effectiveness is bottlenecked by entropy collapse, then scaling RL compute (more training steps, more data) yields diminishing returns beyond the collapse point — as Figure 2 starkly shows: 94% of entropy loss and 93% of performance gain occur in the first 1/3 of training. The paper's entropy-control methods are thus not just performance improvements; they are enablers of the scaling paradigm itself. Without them, the vision of scaling RL compute is fundamentally capped; with them, the scaling curve can be extended, potentially opening a regime where larger models and more RL compute continue to yield meaningful improvements.

This diagnosis also connects to the elicitation-vs-learning debate in a nuanced way. The paper's evidence shows that the elicit-a-tion ceiling (the $b$ coefficient, performance at zero entropy without intervention) is not the hard upper bound — Clip-Cov and KL-Cov push performance beyond what the exponential fit would predict (Figure 12: the entropy-controlled models sustain higher entropy and achieve higher accuracy than the baseline's plateau). This means the baseline plateau is not the model's true capability limit; it is a self-imposed limitation created by the entropy dynamics. RL can teach beyond elicitation, but only if the entropy mechanism is managed to allow continued exploration. The paper thus provides both the diagnosis for why elicitation appears to be the limit in current practice, and the demonstration that it need not be.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The primary math training data is DAPO-MATH (Yu et al., 2025). For the broad entropy–performance fitting experiments (Section 2), training uses Eurus-2-RL-Math (Cui et al., 2025) for the Qwen family and Mistral-24B, and GSM8K (Cobbe et al., 2021) for other model families. Coding training uses AceCode (Zeng et al., 2025), Eurus-2-RL-Code (Cui et al., 2025), and KodCode (Xu et al., 2025). Evaluation benchmarks for math are MATH500 (Hendrycks et al., 2021), AIME 2024, AIME 2025 (Li et al., 2024), AMC (Li et al., 2024), OlympiadBench (He et al., 2024), OMNI-MATH (Gao et al., 2024), and Minerva (Lewkowycz et al., 2022). For code, the test splits of Eurus-2-RL-Code and KodCode are used. The difficulty of training data is adjusted per model family (Appendix A) to stabilize RL — more capable models receive harder data.

• Base model(s). Eleven models spanning four families are tested: Qwen2.5 at 0.5B, 1.5B, 3B, 7B, and 32B parameters (Qwen et al., 2025); Mistral-7B-v0.3, Mistral-Nemo-Base-2407, and Mistral-Small-3.1-24B-Base-2501 (Jiang et al., 2023); LLaMA3.2-3B and LLaMA3.1-8B (Meta, 2024); and DeepSeek-Math-7B-Base (Shao et al., 2024). This breadth is chosen to demonstrate that the entropy collapse phenomenon and the entropy–performance predictability are not artifacts of a single architecture or training recipe. The Qwen2.5 family receives the most detailed analysis because its consistent architecture across five scales enables coefficient extrapolation studies (Figure 7). For the entropy-control method evaluation (Section 4.3), Qwen2.5-7B and Qwen2.5-32B are used on DAPO-MATH. All RL training starts from base (non-instruct) models in the "Zero" setting (DeepSeek-AI et al., 2025).

• Metrics. The primary metric is validation accuracy (pass@1, expressed as a percentage of correctly solved problems). For AIME and AMC, where higher sampling budgets are common, results are reported as avg.@32 (average accuracy across 32 sampled solutions per problem, with majority voting or best-of-N selection). Policy entropy is computed as the batch-average token-level Shannon entropy: H = -(1/|batch|) * (1/|tokens|) * sum(log π_θ(token | prefix)), where the sum is over all tokens in all generated responses for a batch of prompts. The entropy metric is used for both diagnostic tracking and curve-fitting analysis but not as a training objective.

• Baselines. Four main baselines are compared: (a) GRPO (Shao et al., 2024) with standard PPO clipping (ε = 0.2), no entropy regularization, and reference KL coefficient set to zero — this is the default RL protocol used in most contemporary reasoning RL works; (b) GRPO with clip-higher (Yu et al., 2025), which raises the upper PPO clipping threshold to ε = 0.28 to allow more low-probability positively-advantaged tokens into the gradient update; (c) GRPO with entropy loss (Section 4.1), where the loss is L - α·H with various α values; and (d) GRPO with KL regularization (Section 4.1), where the loss is L + β·D_KL(π_θ || π_ref) with various β values. For the algorithm-independence analysis (Section 2.5), RLOO (Kool et al., 2019; Ahmadian et al., 2024) and PRIME (Cui et al., 2025) are also run as comparison points to GRPO.

• Generation budget / compute accounting. The paper does not use FLOPs as a compute metric. Instead, training steps (gradient updates) and rollout steps (data collection rounds) serve as the compute accounting. The standard protocol: 512 prompts per rollout step, 8 responses sampled per prompt (temperature 1.0), 8 policy updates performed on the collected batch (batch size 256, micro-batch size 128). The total generation budget per rollout step is therefore 4,096 complete responses. Training runs for 2,400 gradient steps in the broad sweep (Figure 2), and until convergence in the main fitting experiments (typically 800+ steps, with validation every 4 rollout steps). For the entropy-control experiments (Section 4.3), the protocol is identical except max generation length is set to 8192 tokens. The paper filters out prompts that receive all-correct or all-incorrect responses (these provide zero-variance advantage under group normalization).

• Cross-validation / statistical protocol. There is no explicit cross-validation or statistical significance testing reported. The entropy–performance fitting uses all data points from a single training run per model–task combination, with RMSE reported for the prediction experiments (Section 2.4: 0.9% average RMSE for math, 1.2% for code when predicting 200 future steps from only 36 early steps). The method evaluation (Table 2) reports a single run per configuration (no error bars or multiple seeds), which is a weakness — the variance of GRPO and related methods across random seeds is not characterized. The difficulty-group analysis (Figure 8, right) bins training examples based on rollout accuracy at the current step, which is an observational stratification rather than a controlled experiment.

---

Main Quantitative Results

The Entropy–Performance Relationship Across Model Families and Tasks (Section 2)

The paper's first major empirical claim is that policy entropy and validation performance follow a precise exponential relationship R = -a·exp(H) + b across a wide range of models, tasks, and algorithms.

Headline numbers: Figure 2 reports that across 11 model RL runs, 73% of the total entropy consumption and 76% of the total performance gain occur within the first 200 gradient steps (1/12 of the 2,400-step training). The first 800 steps (1/3 of training) account for over 93% of performance gains and 94% of entropy losses. More than 2/3 of the training steps yield marginal returns.

Fitting quality across models (Figures 3, 4). For all models and both math and coding tasks, the two-parameter exponential function R = -a·exp(H) + b fits the entire trajectory of over 200 (H, R) data points with high precision. The paper does not report R² values but the visual agreement in Figures 3 and 4 is essentially perfect — the fitted curve passes through the dense scatter of validation points at every entropy level. The function works across all four model families (Qwen2.5, Mistral, LLaMA, DeepSeek-Math), all model sizes (0.5B to 32B), and both task domains. This universality across architectures is stronger evidence than if the relationship held only for one model family.

Algorithm-independence (Figure 6). Training Qwen2.5-7B with GRPO, RLOO, and PRIME produces fitted curves with "nearly identical" coefficients a and b. The curves overlay each other closely in Figure 6, confirming that the entropy–performance tradeoff is not an artifact of GRPO's group-normalized advantage estimator. This is an important robustness check — it means the predictability is a property of the policy gradient dynamics interacting with the model and data, not a quirk of one algorithm.

Early-stage prediction (Figure 5). The practical payoff of the fitting is demonstrated by fitting a, b from only the first 36 training steps (approximately 15% of the total run) and using the fitted function to predict the subsequent 200 steps' validation performance. For the Qwen2.5 family on math, the average RMSE across all predicted steps is 0.9%, and the RMSE for the final performance prediction is 0.5%. For coding, the average RMSE is 1.2% and the final-performance RMSE is 1.9%. These are remarkably low errors — predicting final accuracy within ~1% from the earliest phase of training is a practically useful capability.

Coefficient scaling with model size (Figure 7). Extracting a, b for each Qwen2.5 model size and fitting log-linear relationships between parameter count and each coefficient reveals smooth scaling: both a and b increase log-linearly with model size (excluding embedding parameters). On math, the b coefficient (performance ceiling at zero entropy) ranges from approximately 0.25 for Qwen2.5-0.5B to approximately 0.50 for Qwen2.5-32B (reading from Figure 7). On code, the corresponding range is approximately 0.15 to 0.50. The a coefficient (conversion efficiency) also scales smoothly upward. This log-linear relationship mirrors pretraining scaling laws and enables extrapolation: one can predict the final RL performance of a larger model within the same family by training smaller models and extrapolating the coefficients.

Instruct models (Figure 15). The exponential relationship also holds when starting from instruct-tuned models rather than base models, albeit with different a, b values. The paper presents fitting results for Qwen2.5-Instruct models in Appendix D. This is a non-trivial extension — instruct models start with different entropy characteristics (often lower initial entropy due to supervised fine-tuning on narrow distributions), yet the exponential trade-off still governs their RL dynamics.

Data dependence (Figure 14). Training Qwen2.5-7B on different datasets (GSM8K vs. Eurus-2-RL-Math) produces different fitted curves with different a, b values. This confirms that the coefficients reflect the interaction between model capability and data difficulty, not solely intrinsic model properties. The paper does not systematically vary data difficulty to characterize the a, b surface in (model_size, data_difficulty) space, but the single data point establishes that the relationship is conditional on data.

---

Empirical Validation of the Covariance Mechanism (Section 3.3)

The paper tests Theorem 1's prediction that entropy change is proportional to the covariance between log-probability and π·A by tracking both quantities during training.

Headline result (Figure 8, left): The empirical curves of Cov(·) and -dH (negative entropy difference, i.e., the amount by which entropy decreased since the last measurement) show "highly similar dynamics" throughout training. Both start at relatively large positive values in early training, then gradually decline toward smaller positive values as training progresses. The curves are not merely correlated — they track each other in both overall trend and in the specific inflection points where entropy decline accelerates or decelerates. This close tracking validates the theoretical proportionality -dH ∝ Cov derived in Theorem 1, with the proportionality constant being the learning rate η.

Persistence of positive covariance: The paper emphasizes that Cov(·) remains positive throughout the entire training run — it never crosses zero into negative territory. This directly explains the empirical observation that policy entropy decreases monotonically across all experiments (Section 2.3). If covariance were to become negative at some point, entropy would increase, but this never happens under standard GRPO training without entropy intervention. The persistent positivity means that, on average across the training data, the policy's confidence remains at least partially calibrated to advantage — high-probability actions continue to be associated with above-average advantage — even as the policy converges.

Difficulty-dependent covariance (Figure 8, right): When training examples are binned by difficulty (based on rollout accuracy), the covariance curves separate substantially. Easy examples (high accuracy) show the highest covariance values, especially in early training. Medium examples show intermediate covariance. Hard examples (low accuracy) show the lowest covariance, remaining close to zero throughout training. This difficulty stratification provides a mechanistic explanation for why easy problems are "exploited" quickly (high covariance → rapid entropy drop → fast early performance gain) while hard problems resist exploitation (low covariance → slow entropy drop → slow learning). It also explains the shape of the exponential R–H curves: the steep initial phase is dominated by easy examples where covariance is high, and the slow plateau phase is dominated by hard examples where covariance is near zero.

Magnitude relationship: The paper does not report a quantitative correlation coefficient between Cov(·) and -dH (e.g., Pearson r), but the visual overlay in Figure 8 (left) suggests a strong correspondence. The proportionality constant appears to be approximately the learning rate, as predicted by Theorem 1, though the paper does not compute this explicitly or test whether multiplying Cov by the learning rate η = 5×10⁻⁷ produces the exact -dH magnitude (which would be a stricter test of the theory).

---

Failure of Conventional Entropy Regularization (Section 4.1)

The paper provides negative results for two standard entropy-control approaches, establishing that the problem requires a novel solution.

Entropy loss (Figure 9): Adding an entropy bonus L - α·H to the GRPO loss with coefficients α ∈ {0.0001, 0.001, 0.005, 0.01} produces the following outcomes:

• α = 0.0001 and 0.001: entropy still collapses to near zero; the curves are indistinguishable from the no-entropy-loss baseline. The bonus is too small relative to the covariance-driven entropy decline.
• α = 0.005: entropy stabilizes at an intermediate level (approximately 0.2–0.3 nats compared to near-zero for the baseline), but validation accuracy does not improve — it tracks the baseline or slightly underperforms it. The maintained exploration does not translate into better downstream performance.
• α = 0.01: entropy explodes upward (diverges to very high values, well above 1 nat), indicating the policy optimizes for entropy rather than reward. Validation accuracy presumably degrades sharply (the figure shows the entropy curve diverging but does not overlay accuracy for this case — this is a minor reporting gap).

The key finding: there is no α value tested where entropy loss simultaneously prevents collapse, maintains stable entropy, and improves validation performance. The method is either too weak to work, too strong to be stable, or stable but unhelpful.

KL regularization (Figure 10): Adding a KL penalty L + β·D_KL(π_θ || π_ref) with β ∈ {0.001, 0.005, 0.01} produces:

• All β values successfully stabilize entropy at non-zero levels (ranging from approximately 0.1 to 0.4 nats depending on β, compared to near-zero for the baseline).
• All β values degrade validation accuracy relative to the baseline, with larger β causing larger degradation. At β = 0.01, the accuracy drop is approximately 3–5 percentage points across most evaluation benchmarks (reading from Figure 10).
• The entropy preservation comes at the direct cost of preventing the policy from learning — the KL constraint keeps the model too close to the base distribution.

Implication: These negative results are not presented as tuning failures but as structural evidence that global, objective-level entropy intervention (applying the same entropy incentive to all tokens) is fundamentally mismatched to the mechanism driving LLM entropy collapse, which is concentrated in a tiny fraction of high-covariance tokens.

---

Clip-Cov and KL-Cov Performance (Section 4.3)

The main evaluation of the proposed entropy-control methods is on Qwen2.5-7B and Qwen2.5-32B trained on DAPO-MATH with GRPO.

Qwen2.5-7B results (Table 2, top section):

GRPO baseline averages 38.6% across 7 benchmarks (AIME24: 21.2%, AIME25: 9.6%, AMC: 58.7%, MATH500: 78.8%, OMNI-MATH: 27.9%, OlympiadBench: 40.7%, Minerva: 36.7%).

Clip-higher (Yu et al., 2025) achieves 38.8% average, a marginal +0.2 point improvement. Individual benchmark: AIME24 drops to 18.1% (−3.1 points from GRPO) while OlympiadBench rises to 43.3% (+2.6 points) — the method helps on some benchmarks and hurts on others, yielding a wash on average.

Clip-Cov achieves 40.4% average, a +1.8 point improvement over GRPO. Every benchmark except AMC improves (AMC drops marginally from 58.7% to 58.2%). Notable gains: AIME25 15.8% vs. 9.6% (+6.2 points), OlympiadBench 44.1% vs. 40.7% (+3.4 points), Minerva 41.1% vs. 36.7% (+4.4 points).

KL-Cov achieves 40.6% average, +2.0 points over GRPO. Every benchmark except Minerva improves or is flat (Minerva drops marginally from 36.7% to 38.2% — actually an improvement, re-reading: 38.2% vs. 36.7% is +1.5 points). Notable gains: AIME25 12.9% vs. 9.6% (+3.3 points), AMC 61.4% vs. 58.7% (+2.7 points), MATH500 80.8% vs. 78.8% (+2.0 points).

Qwen2.5-32B results (Table 2, bottom section):

GRPO baseline averages 45.8% (AIME24: 21.8%, AIME25: 16.2%, AMC: 69.7%, MATH500: 84.2%, OMNI-MATH: 35.2%, OlympiadBench: 43.6%, Minerva: 45.5%).

Clip-higher achieves 47.2% average, +1.4 points. However, this aggregate masks concerning instability: MATH500 drops from 84.2% to 77.2% (−7.0 points), a large regression on a benchmark where GRPO was already strong. Other benchmarks show strong gains — AIME24 35.6% vs. 21.8% (+13.8 points), AIME25 22.3% vs. 16.2% (+6.1 points) — but the unevenness suggests clip-higher is not a reliable entropy-control strategy at this scale.

Clip-Cov achieves 50.3% average, +4.5 points over GRPO. Notable gains: AIME25 22.7% vs. 16.2% (+6.5 points), OMNI-MATH 42.0% vs. 35.2% (+6.8 points), OlympiadBench 57.2% vs. 43.6% (+13.6 points — the single largest benchmark improvement in the table). MATH500 holds at 87.0% (+2.8 points).

KL-Cov achieves 52.2% average, +6.4 points over GRPO. This is the strongest method at this scale. Notable gains: AIME24 36.8% vs. 21.8% (+15.0 points), AIME25 30.8% vs. 16.2% (+14.6 points), AMC 74.5% vs. 69.7% (+4.8 points), OMNI-MATH 39.1% vs. 35.2% (+3.9 points), OlympiadBench 49.0% vs. 43.6% (+5.4 points). MATH500 is flat at 84.6% (matching GRPO's 84.2%). Every other benchmark improves.

Key patterns in these results:

1. Larger models benefit disproportionately from entropy control. The absolute improvement over GRPO is +1.8/2.0 points for 7B vs. +4.5/6.4 points for 32B. The relative improvement is even more dramatic on the hardest benchmarks: +15.0 points on AIME24 for KL-Cov on 32B vs. +1.4 points on 7B. This supports the paper's hypothesis that larger models have greater latent capacity that entropy collapse prevents from being explored.

2. KL-Cov outperforms Clip-Cov on the larger model but the two are comparable on the smaller model. On 7B, KL-Cov and Clip-Cov are within 0.2 points of each other (40.6% vs. 40.4%). On 32B, KL-Cov leads by 1.9 points (52.2% vs. 50.3%). This may reflect KL-Cov's more stable entropy curves (Figure 13) becoming more important at scale, or the continuous KL penalty providing finer-grained control than the binary clip/no-clip decision of Clip-Cov.

3. Clip-higher is unstable, not merely weaker. The regression on MATH500 for 32B (77.2% vs. 84.2%) is a genuine failure mode — the method hurts performance on a benchmark where GRPO was already strong while helping on harder benchmarks. In contrast, neither Clip-Cov nor KL-Cov causes any benchmark to drop below GRPO by a meaningful margin (AMC's 0.5-point drop for Clip-Cov on 7B is likely noise).

Training dynamics (Figure 12):

• Entropy (left panel): Vanilla GRPO entropy collapses to near zero early and stays there. Clip-Cov and KL-Cov maintain substantially higher entropy throughout training — for KL-Cov on 32B, the entropy remains over 10× higher than the baseline at the point where the baseline has plateaued. Clip-higher initially increases entropy but the curve becomes unstable and fluctuates.
• Response length (middle panel): Both Clip-Cov and KL-Cov produce longer responses than GRPO, with the gap widening over training. This is consistent with maintained exploration — the policy generates more varied, detailed reasoning chains rather than converging to the shortest successful pattern.
• Validation accuracy (right panel): Both proposed methods consistently outperform GRPO throughout training. The performance gap widens over time — GRPO plateaus early while Clip-Cov and KL-Cov continue to improve, albeit at a slowing rate. Clip-higher initially improves but then saturates and slightly declines, consistent with the instability noted in Table 2.

Entropy controllability (Figure 13):

For Clip-Cov, increasing the clip ratio r (the fraction of high-covariance tokens detached) produces monotonically higher entropy curves. Each tested ratio yields a distinct, well-separated entropy level. For KL-Cov, increasing the KL coefficient β similarly produces monotonically higher entropy curves, with KL-Cov producing "stabler entropy curves than Clip-Cov." The paper does not specify which ratios and coefficients correspond to the main Table 2 results (it specifies r = 2×10⁻⁴, k = 2×10⁻³ for 7B and 2×10⁻⁴ for 32B, β = 1), nor does it sweep to find the optimal entropy level for performance — the controllability is demonstrated but not optimized.

---

Ablation Studies and Robustness Checks

Entropy loss coefficient sensitivity (Figure 9): The entropy loss is hypersensitive to α. At α = 0.0001 and 0.001, entropy still collapses (the bonus is ineffective). At α = 0.01, entropy explodes (the bonus dominates). At α = 0.005, entropy stabilizes but performance does not improve. The transition from "no effect" to "explosion" spans only a factor of 10 in α, confirming the method is brittle and the effective window is small or nonexistent.

KL regularization coefficient sweep (Figure 10): KL penalty at β = 0.001, 0.005, 0.01 preserves entropy at all values tested but degrades validation performance, with degradation increasing monotonically with β. Even the smallest β that preserves entropy (0.001) does not improve over the baseline.

Algorithm-independence of entropy–performance fit (Figure 6): GRPO, RLOO, and PRIME produce "nearly identical" fitted curves when training the same model on the same data. This robustness check is important because it isolates the entropy–performance relationship from algorithmic details. If the relationship depended on, say, GRPO's group normalization, it would be less fundamental.

Instruct model fitting (Figure 15, Appendix D): The exponential function also fits when starting RL from instruct-tuned models rather than base models, demonstrating that the entropy–performance tradeoff is not specific to the "Zero" RL setting. The coefficients differ (instruct models likely start with different initial entropy and different latent reasoning capabilities), but the functional form holds.

Data-dependence of coefficients (Figure 14, Appendix C): Training the same model (Qwen2.5-7B) on different datasets (GSM8K vs. Eurus-2-RL-Math) shifts the fitted a, b values. This is expected and confirms that the coefficients reflect the model–data interaction, not just the model.

Difficulty-stratified covariance (Figure 8, right): The separation of covariance curves by prompt difficulty (high/medium/low accuracy) validates the mechanistic link between calibration quality and entropy dynamics. Easy prompts, where the base model is well-calibrated, produce high covariance and thus rapid entropy decline. Hard prompts, where calibration is poor, produce low covariance and thus slow entropy change.

Token-level covariance distribution (Table 1): The analysis at training step 1 reveals extreme skew: mean covariance across all tokens is 0.003, but the top 0.02% have mean 5.654 (over 1800× the average), the top 0.2% have mean 3.112, and the top 2% have mean 1.385. This extreme concentration is the empirical justification for why Clip-Cov and KL-Cov need to intervene on only 0.02%–0.2% of tokens — the entropy decline is driven by a tiny fraction of outlier tokens.

Clip-Cov ratio sweep (Figure 13, left): Varying the clip ratio produces monotonic control over the entropy level, with higher ratios preserving more entropy. The paper does not report validation performance at each ratio, only entropy curves — this is a gap, as the relationship between maintained entropy and final performance is precisely what one would want to optimize.

KL-Cov coefficient sweep (Figure 13, right): Similar to Clip-Cov, varying β produces monotonic entropy control, with KL-Cov yielding stabler (less noisy) entropy curves than Clip-Cov. Again, performance at each β is not reported, only entropy.

Comparison of methods within a single training run context: The paper does not ablate the specific choice of covariance bounds ω_low = 1 and ω_high = 5 for Clip-Cov, nor the specific choice of top-k selection vs. threshold-based selection for KL-Cov. It also does not test whether using the Theorem 1 formulation (covariance with π·A) rather than the Theorem 2 formulation (covariance with A alone) changes the methods' effectiveness — the NPG/PPO justification for using unweighted advantage is stated (Section 4.2) but not empirically validated with an ablation. There is no ablation testing whether random selection within the high-covariance range (Clip-Cov) is better or worse than deterministic selection of the top-k highest covariance tokens.

ReST-EM revision model (Appendix K, Figure 16) mentioned in the prior sections: This experiment appears in the prior sections' discussion of revision models and is not directly part of this paper's experiments. The entropy-control paper does not test ReST-EM or any rejection-sampling fine-tuning approach.

---

Critical Assessment

Does the paper demonstrate that entropy collapse imposes a predictable performance ceiling?

What was tested: The exponential fit R = -a·exp(H) + b is demonstrated across 11 models from 4 families on math and coding tasks using 3 RL algorithms. The fit quality is visually excellent. The early-stage prediction experiment (Figure 5) shows that coefficients from the first 15% of training predict the remaining trajectory with average RMSE of 0.9–1.2%.

What was not tested: The paper claims the ceiling is "fully predictable" when entropy is exhausted (H = 0, R = -a + b). However, training never reaches H = 0 — entropy asymptotically approaches zero, and the extrapolation to R = -a + b is an extrapolation from the fitted curve, not a direct measurement. The paper does not report the actual final performance vs. the predicted ceiling b - a to verify that the exponential extrapolation matches where training actually converges. More critically, the predictability is demonstrated only for training without entropy intervention. Once Clip-Cov or KL-Cov is applied (Section 4.3), the R–H relationship would necessarily change because entropy is being actively managed. The paper does not re-fit the curve under entropy-controlled training, so the claim that methods "break" the predictable ceiling is shown only by performance exceeding the baseline's plateau, not by demonstrating that the ceiling itself was exceeded relative to the fitted curve for that specific training run.

Verdict: The predictability is convincingly demonstrated for vanilla RL training without entropy intervention. The claim that this represents a "hard ceiling" that entropy control breaks is plausible but not quantitatively verified — one would need to compare the entropy-controlled final performance against the predicted ceiling -a + b for the same model–data combination to confirm that the ceiling was actually exceeded rather than simply pushed to a different a, b regime.

Does the covariance mechanism (Theorem 1) provide a valid mechanistic explanation for entropy collapse?

What was tested: The correlation between Cov(·) and -dH is shown for GRPO with policy gradient on Qwen2.5-7B (Figure 8, left). The curves are visually similar. The difficulty-dependent covariance analysis (Figure 8, right) shows that easy prompts have higher covariance.

What was not tested: The theoretical derivation (Theorem 1) assumes a tabular softmax policy — each state–action pair has an independent logit parameter. LLMs are not tabular; their logits are produced by a deep neural network that shares parameters across all state–action pairs. The paper does not discuss whether and how parameter sharing affects the first-order Taylor approximation used in Lemma 1. In a neural network, updating a parameter affects many state–action pairs simultaneously, potentially introducing cross-state covariance terms that the tabular derivation ignores. The empirical validation only tests the bandit formulation (prompt-level state, response-level action), which is coarser than the token-level theory — the covariance is computed over responses within a prompt group, not over tokens within a state. Furthermore, the proportionality constant between Cov and -dH should be the learning rate η per Theorem 1, but the paper does not verify this numerically — does -dH divided by Cov equal η? This would be a direct test of the theory. The paper also tests only one algorithm (GRPO with policy gradient, without PPO surrogate) for the dynamics validation, not the full PPO-clipped objective used in the main experiments, where the surrogate loss modification could alter the effective logit update beyond the simple η·π·A form.

Verdict: The covariance mechanism is qualitatively validated and provides a compelling explanatory framework. However, the empirical validation is weaker than it could be — the proportionality constant is not verified, the token-level prediction is not directly tested (the validation uses response-level covariance), and the gap between the tabular theory and the neural parameterization is not addressed. The claim that the mechanism "explains" the monotonic entropy decline is supported by the persistent positivity of covariance but would be strengthened by a quantitative match between predicted and actual entropy change magnitudes.

Do Clip-Cov and KL-Cov genuinely solve the entropy collapse problem?

What was tested: The methods are tested on Qwen2.5-7B and Qwen2.5-32B on math tasks (DAPO-MATH dataset) against GRPO and clip-higher baselines. Both methods preserve higher entropy and achieve better final performance (Table 2, Figure 12). Entropy controllability via hyperparameters is demonstrated (Figure 13).

What was not tested: Several important robustness dimensions are missing. (1) Single training protocol: The methods are evaluated with one specific training configuration (DAPO-MATH, GRPO, specific hyperparameters). There is no evidence that they work with other RL algorithms (RLOO, PRIME, REINFORCE++), other datasets (coding, which was part of the earlier entropy experiments), or other model families (Qwen2.5 only — the methods are not tested on Mistral, LLaMA, or DeepSeek-Math). (2) Single seed: The paper reports one run per configuration. GRPO and related methods have known variance across random seeds; without error bars or multiple runs, a +2.0 point improvement on 7B could be partially or entirely within run-to-run variance. (3) No optimal entropy analysis: The paper shows that entropy can be controlled but does not explore whether there is an optimal entropy level, or whether the methods' entropy levels are near-optimal. The relationship between the controlled entropy level and final performance is not characterized — this is a significant gap given the paper's framing of entropy as a resource to be managed, not eliminated. (4) Limited method ablations: The specific design choices (covariance bounds ω_low = 1, ω_high = 5; random vs. deterministic selection; Theorem 2 covariance vs. Theorem 1 covariance) are not ablated. It is unclear which aspects of the methods are essential and which are incidental. (5) No combination with other known techniques: The paper does not test whether Clip-Cov/KL-Cov interact synergistically or antagonistically with clip-higher, with length penalties, or with dynamic sampling temperatures. (6) No scaling beyond 32B: The log-linear coefficient extrapolation (Figure 7) suggests larger models would benefit more, but this is not tested — evaluating at 70B+ would substantially strengthen the scaling narrative.

Verdict: The methods show promising performance improvements over GRPO, particularly on the larger model where gains are substantial (+4.5 to +6.4 points average). However, the evidence base is narrow — one model family, one task domain, one algorithm, one seed per configuration. The claim that these methods "solve" the entropy collapse problem is premature; they are demonstrated to mitigate it in a specific regime and warrant broader validation. The instability of clip-higher on 32B (regressing on MATH500) is well-documented, making Clip-Cov/KL-Cov's stability a genuine advantage, but the single-seed limitation applies equally to this stability claim.

Does the entropy–performance predictability enable "predicting the performance of large models given small models" (Section 2.4)?

What was tested: The log-linear scaling of coefficients a, b with model size is shown for the Qwen2.5 family on math and coding (Figure 7). The fits are smooth.

What was not tested: The paper does not actually perform the extrapolation experiment — it does not fit a, b from, say, Qwen2.5-0.5B/1.5B/3B and then predict the 7B or 32B final RL performance, comparing the prediction to the actual result. The log-linear relationship is demonstrated, but its predictive accuracy is not quantified. This is analogous to showing a scaling law exists but not testing whether it successfully predicts held-out model sizes. The paper also does not test whether the coefficient–size relationship holds across model families (e.g., can you predict Mistral coefficients from Qwen coefficients scaled by parameter count?), which would be necessary for the claim of general predictability.

Verdict: The existence of a smooth scaling relationship is convincingly shown, but the predictive capability is an extrapolation from the data rather than a demonstrated result. This is a weaker claim than it appears — showing that coefficients scale log-linearly is necessary but not sufficient for prediction; one must actually make and verify predictions.

Missing experiments that would have strengthened the paper

Multi-seed evaluation of Clip-Cov and KL-Cov: Running 3–5 seeds per configuration and reporting mean ± std would address the single-seed concern and establish whether the observed improvements are statistically reliable. At minimum, the GRPO baseline should be run with multiple seeds to characterize its variance, so the reader can assess whether a +2.0 point improvement exceeds typical run-to-run noise.

Coding task evaluation for entropy-control methods: The entropy–performance fitting is shown for both math and coding (Figures 3, 4), but Clip-Cov and KL-Cov are only evaluated on math. Testing on coding would establish domain generality for the methods, not just for the diagnostic framework.

Non-Qwen model evaluation for entropy-control methods: The entropy–performance relationship is shown across Mistral, LLaMA, and DeepSeek-Math, but the proposed methods are only tested on Qwen2.5. The paper's framing implies the methods should work broadly, but this is not demonstrated.

Direct ceiling-exceedance test: For a model–data combination, run GRPO to completion, fit R = -a·exp(H) + b and compute the predicted ceiling -a + b. Then run Clip-Cov or KL-Cov on the same setup and check whether the final performance exceeds this predicted ceiling. This would directly test the paper's central narrative that entropy control breaks through the predictable performance ceiling.

Proportionality constant verification for Theorem 1: Compute -dH / Cov throughout training and check whether it equals the learning rate η (or is at least constant). A constant ratio would strongly validate the theory; a varying ratio would indicate that the first-order approximation or the tabular assumption breaks down.

Token-level (rather than response-level) covariance validation: Theorem 1 predicts entropy change at the token level. The bandit validation uses response-level covariance. A token-level analysis — computing covariance over tokens within a response rather than over responses within a prompt group — would directly test the theory at the resolution it was derived for. This is computationally feasible (it's how the methods themselves work) and would close the gap between theory and empirical validation.

6. Limitations and Trade-offs

Difficulty Estimation Cost Is Unaccounted for in the Entropy–Performance Predictability

The assumption or constraint. The entropy–performance fitting methodology ($R = -a \exp(\mathcal{H}) + b$) and the early-stage prediction capability (Section 2.4) rely on having access to validation accuracy measurements at multiple entropy levels throughout training. These measurements require running full evaluation on held-out benchmarks (MATH500, AIME, AMC, etc.) every 4 rollout steps. For the coefficient extrapolation across model sizes (Figure 7), one must train at least 2–3 smaller models to completion and extract their $a, b$ coefficients to fit the log-linear scaling relationship. The paper acknowledges, implicitly, that this is a measurement framework, not a cost-free prediction oracle — the "predictability" is conditional on having already collected the data needed to fit the coefficients.

The consequence. The practical value of predicting final performance from the first 15% of training steps (Figure 5: 0.9% RMSE for math, 1.2% for code) is limited by the fact that one must run that 15% of training for every model–data combination of interest before the prediction can be made. By the time 15% of training is complete (36 steps in the experiment), a substantial fraction of the total entropy drop has already occurred — recall from Figure 2 that 73% of entropy consumption happens in the first 200 steps. So the "early" prediction is made after the policy has already consumed most of its entropy, limiting the actionable window for intervention. Moreover, for the coefficient extrapolation across model sizes, one must train smaller models to completion first — this is not a free lunch; it is a measurement cost that must be amortized over many large-model training runs to be worthwhile. The paper does not discuss how many smaller models must be trained, at what cost, to achieve a given prediction accuracy for a target large model.

What evidence exists in the paper. Figure 5 demonstrates the prediction accuracy after 36 training steps. Figure 7 shows the log-linear coefficient scaling. Neither figure reports the computational cost of generating these predictions — the total GPU-hours for training the 4 smaller Qwen2.5 models to extract their coefficients, or the fraction of a full training run that 36 steps represents. The paper mentions that fitting requires "over 200 data points" (Section 2.4) but does not quantify the evaluation cost in FLOPs or wall-clock time relative to the training cost.

Mitigation status. The paper does not address this cost at all. It presents the predictability as a capability without discussing the investment required to achieve it. There is no proposal for reducing the data requirements (e.g., fitting from fewer measurement points, using only a subset of evaluation benchmarks), no analysis of how prediction accuracy degrades with fewer training steps observed, and no discussion of whether the log-linear coefficient scaling holds with only 2 smaller models rather than 4. This is an unacknowledged practical barrier to deploying the predictability framework in resource-constrained settings.

---

The Entropy–Performance Predictability Is Demonstrated Only Without Entropy Intervention

The assumption or constraint. The exponential fitting $R = -a \exp(\mathcal{H}) + b$ is established exclusively for training runs "without entropy intervention like entropy loss or KL regularization" (Section 2, Takeaway box). All curve-fitting experiments (Figures 3, 4, 6, 14, 15) use the default GRPO/RLOO/PRIME protocols with reference KL coefficient set to zero and no entropy bonus. The coefficients $a$ and $b$ are thus descriptive of a specific regime: unmanaged entropy collapse.

The consequence. Once entropy-control methods (Clip-Cov, KL-Cov) are applied, the $R$–$\mathcal{H}$ relationship necessarily changes — because entropy no longer follows the monotonic collapse trajectory that generated the original data. The paper does not re-fit the exponential function under entropy-controlled training, so it is unknown whether the relationship still holds, whether the functional form changes, or whether new coefficients $a', b'$ would be required. This means the paper's central narrative — "entropy collapse imposes a predictable performance ceiling, and entropy control breaks through it" — is only partially validated. The "ceiling" is predicted from the no-intervention fit ($R_{\text{ceiling}} = -a + b$), but the paper never compares this predicted ceiling against the actual final performance achieved by Clip-Cov/KL-Cov to verify that the ceiling was exceeded rather than simply shifted to a different $a, b$ regime. In other words, it is possible that entropy control changes the coefficients (higher $b$, different $a$) rather than breaking the exponential relationship itself, which would mean the ceiling still exists but at a higher level — a less dramatic but still important distinction.

What evidence exists in the paper. Section 4.3 shows that Clip-Cov and KL-Cov achieve higher final performance than GRPO without entropy intervention (Table 2), and Figure 12 shows that the entropy-controlled models sustain higher entropy and continue improving while GRPO plateaus. However, the paper does not fit $R = -a' \exp(\mathcal{H}) + b'$ for the entropy-controlled runs, does not report whether the entropy-controlled curves still follow an exponential form, and does not compare the empirical final performance of Clip-Cov/KL-Cov against the predicted ceiling $-a + b$ from the no-intervention fit for the same model–data combination.

Mitigation status. The paper acknowledges this limitation only indirectly. Section 2.6 notes that "other works that adopt different policy models or use off-policy data observed distinct entropy patterns. Therefore, this predictability is not arguably universal." This caveat refers to different setups (different models, off-policy data) rather than to the entropy-controlled regime, but the principle extends: changing the entropy dynamics changes the predictability. The paper calls for "a more in-depth analysis of the entropy behavior under different conditions" (Section 2.6) but does not perform this analysis for its own proposed methods — the one condition most directly relevant to its contribution.

---

Evaluation of Clip-Cov and KL-Cov Is Limited to a Single Model Family, Single Task Domain, Single Algorithm, and Single Seed

The assumption or constraint. The entropy-control methods are evaluated only on the Qwen2.5 family (7B and 32B) using GRPO on the DAPO-MATH math dataset (Section 4.3). The paper explicitly contrasts this with the broader entropy–performance fitting experiments, which span 4 model families (Qwen2.5, Mistral, LLaMA, DeepSeek-Math), 2 task domains (math, coding), and 3 RL algorithms (GRPO, RLOO, PRIME). The methods' evaluation is, by comparison, narrow.

The consequence. Several generalization questions are unanswered. First, domain generality: the entropy–collapse phenomenon is documented on both math and coding tasks (Figures 3, 4) with similar functional forms, but Clip-Cov and KL-Cov are tested only on math. If the methods exploit domain-specific properties of mathematical reasoning token distributions (e.g., formula tokens having different covariance characteristics than natural language tokens), they may not transfer to code generation, where syntax and semantics differ substantially. Second, model family generality: the Qwen2.5 family is known to have strong base reasoning capabilities; models from other families (Mistral, LLaMA, DeepSeek-Math) may exhibit different covariance distributions (the skew documented in Table 1 might be less extreme or differently structured), and the methods' efficacy could vary. Third, algorithm generality: the paper argues that the NPG/PPO formulation (Theorem 2) justifies using unweighted advantage in the covariance computation, but this justification is specific to PPO's trust-region approximation. Whether the methods work with simpler policy gradient algorithms (REINFORCE++, RLOO without PPO clipping) is untested. Fourth, statistical reliability: Table 2 reports a single training run per configuration. GRPO and its variants are known to exhibit non-trivial variance across random seeds due to the stochasticity of rollout sampling and the sensitivity of the group-normalized advantage estimator to specific prompt–response groupings. A +2.0 point average improvement on 7B (38.6% → 40.6% for KL-Cov) could plausibly fall within run-to-run variance; without error bars or multiple seeds, the statistical significance of the reported gains is unknown.

What evidence exists in the paper. Table 2 and Figure 12 constitute the entirety of the methods evaluation. The paper does not report multiple seeds, confidence intervals, or any statistical test. It does not evaluate on coding tasks, on non-Qwen models, or with non-GRPO algorithms for the method experiments. The broader entropy experiments (Sections 2 and 3) are more comprehensive, but they concern diagnosis, not intervention.

Mitigation status. The paper does not acknowledge this scope limitation explicitly for the method evaluation. Section 6 (Conclusion) speaks generally about RL scaling, and Section 2.6 discusses the non-universality of the entropy–performance predictability, but the specific narrowness of the Clip-Cov/KL-Cov experimental validation is not flagged as a limitation. This is a significant gap because the paper's practical recommendations (use Clip-Cov or KL-Cov to control entropy) implicitly assume broader applicability than the evidence supports.

---

The Theoretical Derivation Assumes a Tabular Softmax Policy, but LLMs Are Deep Neural Networks with Parameter Sharing

The assumption or constraint. Lemma 1 and Theorems 1–2 are derived under the assumption that the policy is a tabular softmax — each state–action pair $(s, a)$ has an independent logit parameter $\theta_{s,a} = z_{s,a}$ (Section 3.1, Lemma 1 statement: "Assume that policy $\pi_\theta$ is a tabular softmax policy, where each state-action pair $(s, a)$ is associated with an individual logit parameter $z_{s,a} = \theta_{s,a}$"). This assumption is necessary for the first-order Taylor expansion to decompose cleanly into per-state–action covariance terms, because it eliminates cross-parameter effects where updating one logit through shared network weights affects many other logits simultaneously.

The consequence. In a neural network, updating a single weight affects the logits of many tokens across many states simultaneously. The effective logit change for a given $(s, a)$ pair is not simply $\eta \cdot \pi_\theta(a|s) \cdot A(s, a)$ (Proposition 1); it includes contributions from gradient components flowing through shared parameters, which depend on the entire batch of training data and the network's architecture. The first-order Taylor approximation in Lemma 1 ignores these cross-state interactions — it assumes that the entropy change at state $s$ depends only on the logit updates for actions at state $s$, not on logit updates for actions at other states $s'$ that are coupled through shared parameters. For an LLM with billions of parameters shared across all tokens and prompts, this coupling could be substantial. The empirical validation (Section 3.3) partially addresses this by computing covariance at the response level (the bandit formulation), but this tests the aggregate relationship, not the per-state–action independence assumed in the derivation. It is possible that the aggregate covariance–entropy relationship holds even if the per-state mechanism is more complex, but the paper presents the theory as explaining the mechanism, not just the correlation.

What evidence exists in the paper. The empirical validation (Figure 8, left) shows that the response-level covariance term and $-d\mathcal{H}$ track each other closely. However, the paper does not test the stronger predictions of the tabular theory: whether the proportionality constant equals the learning rate, whether the relationship holds at the token level (as the theory predicts) rather than only at the response level (as validated), or whether removing the tabular assumption changes the predicted entropy change. The gap between the theory's assumptions and the experimental validation is not discussed — the paper presents the theory as if it directly describes the LLM training dynamics, when in fact it describes a simplified model of those dynamics.

Mitigation status. The paper does not acknowledge the tabular assumption as a limitation of the theoretical analysis. The derivations in Appendix E are mathematically rigorous within the tabular framework, but the step from "this holds for tabular softmax policies" to "this explains entropy dynamics in LLMs" is not bridged analytically. The empirical validation provides circumstantial support but does not test the specific mechanisms that the tabular assumption enables. This limitation is partially mitigated by the common practice in RL theory of deriving results for tabular cases and extending them empirically to function approximation settings (Agarwal et al., 2021), but the paper does not invoke this justification.

---

The Methods Intervene on Only ~0.02% of Tokens, but the Optimal Entropy Level for Performance Remains Uncharacterized

The assumption or constraint. Clip-Cov and KL-Cov are designed to be entropy-controllable (Figure 13) — the hyperparameters $r$ (clip ratio) and $\beta$ (KL coefficient) provide knobs for adjusting how much entropy is preserved. However, the paper sets these hyperparameters to fixed values ($r = 2 \times 10^{-4}$, $k = 2 \times 10^{-3}$ or $2 \times 10^{-4}$, $\beta = 1$) for the main evaluation (Section 4.3) without sweeping to find the entropy level that maximizes validation performance. The controllability demonstration (Figure 13) shows only entropy curves at different hyperparameter settings, not the corresponding validation accuracy curves.

The consequence. The paper cannot answer a practitioner's most basic question: "How much entropy should I target?" The methods provide the means to control entropy, but the objective — what entropy level is optimal for a given model–data combination — is left unspecified. The paper's framing implies that any increase in entropy above the collapsed baseline is beneficial because it prevents premature convergence, but this cannot be universally true — if it were, one would simply set the hyperparameters to maximize entropy (clip all high-covariance tokens, apply infinite KL penalty), which would clearly degrade performance by preventing all learning. There must exist an optimal trade-off between exploration (higher entropy) and exploitation (learning from high-covariance tokens), but the paper does not characterize where this optimum lies or how it varies with model size, task difficulty, or training stage. The fact that KL-Cov with the chosen hyperparameters substantially outperforms GRPO suggests the chosen entropy level is better than the collapsed baseline, but it does not demonstrate that it is near-optimal — larger entropy levels might yield even better performance, or slightly lower entropy might work better for some benchmarks.

The tiny fraction of intervened tokens ($10^{-4}$ to $10^{-3}$) that "totally change the entropy curve" (Section 4.5) also raises a sensitivity concern: if changing the intervention on 0.02% of tokens produces a large entropy shift, the inverse is true — small random fluctuations in which tokens fall into the high-covariance range could cause large entropy fluctuations, potentially destabilizing training in ways that a single-seed experiment would not capture.

What evidence exists in the paper. Figure 13 shows entropy curves for varying $r$ and $\beta$, but without the corresponding accuracy. Table 2 reports accuracy only for the single chosen hyperparameter setting. The paper states explicitly in Section 4.5: "It still remains open whether there exists an optimal entropy value to balance the exploration and training stability." This is an honest admission of a fundamental gap, but it means the methods are tools for entropy control rather than complete solutions for performance optimization — a distinction the paper's strong claims sometimes blur.

Mitigation status. The paper acknowledges this as an open question (Section 4.5) and does not attempt to solve it. The controllability demonstration (Figure 13) is a partial mitigation — it shows that the methods can systematically vary entropy, which is a prerequisite for finding the optimal level. But the absence of an accuracy sweep means that even the basic shape of the entropy–performance relationship under entropy-controlled training is unknown (is it monotonic? concave? does over-exploration hurt?). Future work is needed to characterize this relationship and potentially to develop adaptive schemes that adjust the intervention intensity based on training progress or per-prompt difficulty — directions the paper does not explore.

---

The Covariance-Driven Method Targets a Symptom (Entropy Decline) Rather than the Root Cause (Advantage Estimation Quality), and May Not Generalize to Better Verifiers

The assumption or constraint. The paper's theoretical analysis (Theorem 2) and the resulting methods (Clip-Cov, KL-Cov) identify the covariance between log-probability and advantage as the driver of entropy decline and intervene to suppress high-covariance token updates. However, the cause of high covariance is not addressed: why are high-probability actions consistently associated with positive advantage? The paper's answer (Section 3.2) is that the base model is well-calibrated — when it is confident, it tends to be correct. This means high covariance is a symptom of a good property (calibration) interacting with the policy gradient update rule in a harmful way (entropy collapse). The methods suppress the harmful effect (by blocking updates to well-calibrated high-confidence tokens) but do not address whether the calibration itself should be preserved or modified.

The consequence. If the advantage estimation were improved — for example, by using a learned process reward model that provides more accurate token-level advantage signals, or by using better baselines that reduce advantage variance — the covariance structure would change. With less noisy advantages, the correlation between confidence and correctness might weaken for some tokens (where the base model is confidently wrong) and strengthen for others (where it is confidently right). Clip-Cov and KL-Cov would still suppress high-covariance tokens, but the set of high-covariance tokens would shift, potentially in ways that make the intervention less effective or even harmful. The methods are designed for the specific advantage estimation regime of GRPO with group-normalized trajectory-level rewards — a regime characterized by high variance and the bandit assumption that all tokens in a correct response get equal positive advantage. In a regime with better per-token advantage signals (e.g., PRIME's implicit PRM, or process reward models trained with Monte Carlo rollouts), the covariance distribution might be less skewed, or the relationship between covariance and entropy change might differ, potentially rendering the methods unnecessary or requiring different hyperparameters.

More fundamentally, the methods' strategy of blocking updates to high-covariance tokens means deliberately not reinforcing well-calibrated confident correct predictions. These are precisely the tokens that the policy gradient should reinforce — they represent cases where the model knows what to do and does it correctly. The methods prevent this reinforcement to preserve exploration, but the cost is that the model learns less efficiently from its successes. The fact that performance improves despite this (Table 2) suggests that the exploration benefit outweighs the reinforcement cost, but this trade-off may not be favorable in all regimes — particularly if the base model's calibration is already strong and the remaining room for improvement comes from refining already-correct strategies rather than discovering new ones.

What evidence exists in the paper. The paper notes the calibration connection explicitly (Section 3.2): "At the early stage, the policy demonstrates high covariance on training data, implicating the policy's confidence is well-calibrated, thus can safely exploit trajectories with high confidence, strengthening belief and minimize entropy." The difficulty-stratified covariance analysis (Figure 8, right) confirms that easy prompts (where calibration is best) have the highest covariance. The paper does not test Clip-Cov/KL-Cov with alternative advantage estimation methods (PRIME, RLOO, learned baselines), does not analyze how the covariance distribution changes when advantage estimation improves, and does not consider whether calibration-preserving alternatives (e.g., selectively reducing confidence on confidently-wrong tokens rather than blocking updates on confidently-right tokens) might be more efficient.

Mitigation status. The paper does not address this limitation. The framing of entropy control as "breaking the bottleneck" treats the covariance mechanism as something to be overcome rather than a symptom whose underlying cause might be productively addressed. Section 4.5 discusses the philosophy of entropy control but only in terms of "several pivotal tokens are crucial for the entropy of LLMs" — it does not question whether the pivotal tokens should be pivotal, or whether the problem could be solved upstream by improving advantage estimation to reduce spurious covariance rather than downstream by blocking its effects. This is a trade-off the paper makes implicitly: intervene at the gradient level rather than improving the reward signal. Whether this is the right choice is an empirical question the paper does not investigate.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the conversation around RL for reasoning LLMs from an algorithm-design perspective toward an entropy-dynamics perspective. Before this work, the community's approach to entropy in LLM RL training was fragmented: some practitioners added entropy bonuses (with mixed results), most omitted entropy regularization entirely (accepting collapse as inevitable), and a few proposed heuristic fixes (clip-higher) without understanding the underlying mechanism. The field lacked a unified framework for thinking about why entropy behaves as it does and what to do about it. This paper provides that framework.

The shift is more diagnostic than paradigmatic — the paper does not introduce a new RL algorithm, a new model architecture, or a new training objective. Instead, it introduces a mechanistic lens through which existing algorithms and their failure modes can be understood. The key conceptual move is the demonstration that entropy dynamics in softmax policies under policy gradient are governed by a single interpretable quantity — the covariance between log-probability and advantage — and that this quantity is not merely correlated with entropy change but is its proximal cause under the first-order Taylor approximation (Theorem 1). This transforms entropy from a black-box quantity that happens to decrease during training into a predictable consequence of the interaction between the policy gradient update rule and the model's calibration (the degree to which confidence predicts correctness). The fact that covariance remains positive throughout training — because base LLMs are reasonably well-calibrated (Kadavath et al., 2022) — explains the universality of the collapse phenomenon across model families, model sizes, and RL algorithms (Sections 2.3, 2.5).

This diagnostic reframing resolves the apparent contradiction in the literature: why do some works find that entropy regularization helps (as in classic maximum-entropy RL; Haarnoja et al., 2018; Mnih et al., 2016) while others find it ineffective or harmful for LLM reasoning (He et al., 2025; the negative results in Section 4.1 of this paper)? The answer is that objective-level entropy bonuses fight the symptom (low entropy) globally, without addressing the distributional nature of the problem. Entropy collapse is driven by a tiny fraction of outlier tokens — the top 0.02% have over 1800× the average covariance (Table 1) — and a uniform entropy bonus is either too weak to counteract these outliers (small coefficients), too strong and destabilizing (large coefficients), or stable but misallocated (preserving entropy on tokens where collapse is harmless or even desirable). The covariance-based diagnosis explains why objective-level intervention fails structurally, not just empirically, and this explanation is what makes the proposed gradient-level intervention (Clip-Cov, KL-Cov) principled rather than another heuristic.

The paper also reconciles the tension between the "elicitation" and "learning" views of RL for LLMs (Yue et al., 2025). The empirical finding that the entropy–performance trajectory follows a predetermined exponential curve ($R = -a \exp(\mathcal{H}) + b$) with algorithm-independent coefficients (Figure 6) supports the elicitation view: in the absence of entropy intervention, RL primarily surfaces latent patterns already present in the base model, with diminishing returns as entropy is exhausted. However, the paper shows that this is not an intrinsic limitation of RL — it is a consequence of the entropy mechanism, not of the learning paradigm. Clip-Cov and KL-Cov push performance beyond the baseline plateau (Table 2, Figure 12), demonstrating that RL can teach beyond the base model's most accessible strategies, provided the entropy bottleneck is managed. The paper thus positions itself as showing both why elicitation appears to be the limit in current practice, and that it need not be.

The research directions this work makes more attractive include any that require understanding or controlling the exploration–exploitation tradeoff in LLM training. Covariance-aware intervention becomes a general design principle: rather than adding exploration bonuses to the objective, identify the specific parameter updates driving premature convergence and selectively constrain them. This principle could extend beyond RL to other training paradigms where policy or output distributions collapse undesirably (supervised fine-tuning on narrow distributions, iterative self-improvement loops, distillation). Conversely, the work makes naive entropy regularization less attractive as a research direction — the failure modes documented in Section 4.1 are structural, not a matter of finding the right coefficient. Research effort is better spent on understanding and intervening on the mechanisms driving collapse (as this paper does) than on tuning global entropy bonuses.

Follow-Up Research This Work Enables

Characterize the optimal entropy level for reasoning performance under covariance-controlled training. The paper demonstrates that Clip-Cov and KL-Cov can control entropy (Figure 13) but does not identify what entropy level maximizes downstream accuracy. A natural follow-up would sweep the clip ratio $r$ and KL coefficient $\beta$ across a wide range for a fixed model–data combination, measuring both the resulting entropy trajectory and the final validation accuracy, to map the entropy–performance surface under controlled conditions. The key question is whether the relationship is monotonic (more entropy always helps, up to a point), concave (an interior optimum), or task-dependent (different optimal entropy levels for math vs. coding, or for easy vs. hard benchmarks). This experiment would also test whether the exponential $R$–$\mathcal{H}$ relationship still holds under entropy-controlled training, or whether a new functional form emerges when covariance-driven collapse is suppressed. A strong follow-up would report the optimal $r$ and $\beta$ for Qwen2.5-7B and 32B on DAPO-MATH, and test whether these optima transfer to other model families (Mistral, LLaMA) without re-tuning.

Token-level empirical validation of Theorem 1 to close the gap between the tabular theory and neural network training. The paper's empirical validation of the covariance mechanism (Section 3.3) uses the bandit formulation — computing covariance at the response level rather than the token level. Theorem 1, however, predicts entropy change at the per-state (per-token-prefix) level. A direct token-level test would compute $\text{Cov}_{y_t \sim \pi_\theta(\cdot | \boldsymbol{y}_{<t}, \boldsymbol{x})}(\log \pi_\theta(y_t), \pi_\theta(y_t) \cdot A_t)$ for each generation step (using the token-level advantages from Eq. 2, approximated by the trajectory-level advantage for GRPO), and then check whether the per-token covariance aggregated over the batch equals $-d\mathcal{H} / \eta$, as Theorem 1 predicts. If the proportionality constant deviates systematically from the learning rate, the gap would quantify the effect of parameter sharing (which the tabular derivation ignores) on entropy dynamics. If the token-level covariance tracks $-d\mathcal{H}$ as closely as the response-level covariance does, it would strengthen the claim that the mechanism is genuinely understood at the resolution where the theory operates.

Apply covariance-aware entropy control to coding tasks and non-Qwen model families to establish domain and architecture generality. The entropy–performance fitting is demonstrated on both math and coding (Figures 3, 4), and across 4 model families (Qwen2.5, Mistral, LLaMA, DeepSeek-Math), but Clip-Cov and KL-Cov are evaluated only on Qwen2.5 on math (Section 4.3). A minimal follow-up would test Clip-Cov and KL-Cov on Qwen2.5-7B on a coding dataset (e.g., Eurus-2-RL-Code or KodCode) using the same hyperparameters as the math experiments, and on Mistral-7B or LLaMA3.1-8B on DAPO-MATH. The key measurements are: (a) whether the covariance distribution (Table 1) has a similar extreme skew in these new settings; (b) whether the same $r$ and $k$ hyperparameters produce entropy control without re-tuning; (c) whether the performance gains over GRPO (Table 2) generalize. A negative result — e.g., covariance skew being much less extreme on code, making Clip-Cov ineffective — would refine the understanding of when covariance-aware intervention is needed, not just that it works.

Test whether Clip-Cov and KL-Cov synergize with improved advantage estimation (process reward models). The paper's entropy-control methods are motivated by and tested with GRPO's trajectory-level group-normalized advantage estimator, which assigns identical advantage to every token in a response. This is a high-variance estimator, and the covariance it induces between confidence and advantage may partly reflect noise (confident tokens in responses that happen to be correct by chance) rather than genuine calibration. A natural extension would test Clip-Cov/KL-Cov with a learned process reward model (PRM) that provides token-level advantage estimates — e.g., the implicit PRM from PRIME (Cui et al., 2025) or a Monte Carlo rollout-trained PRM. The hypothesis is that better advantage estimation would reduce spurious covariance (confident tokens in correct-by-chance responses would receive lower advantage), potentially reducing the need for aggressive entropy control or shifting the optimal intervention ratio. A strong experiment would train Qwen2.5-7B with PRIME's advantage estimator, compare the covariance distribution against GRPO's (is the skew less extreme?), and test whether Clip-Cov with the same hyperparameters still improves performance or whether the benefit shrinks (indicating that part of GRPO's entropy collapse is driven by advantage noise, not just calibration).

Develop adaptive, difficulty-conditioned entropy control that varies intervention strength per prompt. The difficulty-stratified covariance analysis (Figure 8, right) shows that easy prompts have much higher covariance than hard prompts. This implies that uniform entropy control (the same $r$ or $k$ for all prompts) is suboptimal: easy prompts likely need more intervention (their high covariance drives fast entropy collapse) while hard prompts might benefit from less (their low covariance means they already explore adequately, and blocking updates could slow their already-difficult learning). A follow-up could implement per-prompt intervention ratios based on the prompt's current rollout accuracy or average covariance: for prompts with high accuracy, increase $r$ or $k$ to preserve more entropy; for prompts with low accuracy, reduce $r$ or $k$ to allow more exploitation. This connects the covariance-based entropy-control framework to the compute-optimal test-time scaling paradigm, where strategy is adapted to per-prompt difficulty. The experiment would compare uniform vs. difficulty-conditioned Clip-Cov/KL-Cov on DAPO-MATH, measuring both average accuracy and per-difficulty-bin accuracy. The hypothesis is that difficulty-conditioned control would improve hard-prompt performance without sacrificing easy-prompt gains.

Multi-seed statistical validation and scaling to larger models to establish reliability and extrapolation behavior. The paper reports single-seed results for Clip-Cov/KL-Cov, and the entropy–performance coefficient scaling (Figure 7) predicts larger models will benefit more from entropy control. A thorough follow-up would run 5 seeds each of GRPO, Clip-Cov, and KL-Cov on Qwen2.5-7B and 32B to quantify variance and establish statistical significance, then extend to Qwen2.5-70B (or a comparable large model) to test the extrapolation. If the log-linear coefficient scaling holds, the 70B model should show even larger absolute gains than the 32B (+6.4 points for KL-Cov over GRPO). If the gains saturate or reverse at 70B, it would indicate that the relationship between model size and the benefit of entropy control is not log-linear, and that there exists an optimal model scale for covariance-aware intervention — a finding that would substantially refine the paper's scaling narrative. This experiment is expensive but feasible given the availability of Qwen2.5-70B and distributed RL frameworks like veRL.

Practical Applications and Downstream Use Cases

Improving open-source RL training recipes for reasoning models. The most immediate practical application is integrating Clip-Cov or KL-Cov into existing RL-for-reasoning pipelines (veRL, OpenRLHF, DAPO). The implementation cost is minimal — the pseudocode in Listing 11 shows that the modifications require fewer than 10 lines of code added to the standard PPO loss computation. For teams training reasoning models using GRPO on math or code data, the methods offer a drop-in improvement: on Qwen2.5-32B, KL-Cov improves average accuracy across 7 benchmarks by 6.4 percentage points over GRPO (45.8% → 52.2%, Table 2), with gains concentrated on the hardest benchmarks (+15.0 points on AIME24, +14.6 points on AIME25). Given that the hyperparameters ($r = 2 \times 10^{-4}$, $k = 2 \times 10^{-4}$ to $2 \times 10^{-3}$, $\beta = 1$) are reported and the method requires no additional model components, compute overhead, or data, this is one of the lowest-cost–highest-impact modifications available for current reasoning RL pipelines. Teams should be aware of the single-seed limitation and validate on their specific model–data combination, but the risk–reward ratio strongly favors adoption.

Early stopping and compute allocation decisions using the entropy–performance predictability. The finding that the entropy–performance trajectory is predictable from the first 15% of training (Figure 5: 0.9% RMSE for math, 1.2% for code) enables practical early stopping in RL training. Organizations running RL on expensive large-model setups can monitor entropy and validation accuracy for the first ~50 training steps, fit $a$ and $b$, and estimate the final performance $-a + b$ as well as the marginal return on additional training steps (from the derivative $dR/d\mathcal{H} = -a \exp(\mathcal{H})$). If the predicted ceiling is only marginally above the current performance, or if the predicted gains from continued training are below a cost–benefit threshold, training can be terminated early, saving significant compute. For organizations training multiple model sizes within a family, the log-linear coefficient scaling (Figure 7) enables predicting the larger model's final RL performance from smaller models' fitted coefficients — a capability analogous to pretraining scaling laws but for RL post-training. A team could train Qwen2.5-0.5B/1.5B/3B to completion (relatively cheap), extract $a, b$ for each, fit the log-linear trend, and predict Qwen2.5-32B's RL ceiling before committing to the 32B training run. If the predicted ceiling is unsatisfactory, the team could adjust the training data (which shifts $a, b$, as shown in Figure 14) or switch model families without wasting large-model compute.

Guiding entropy-management hyperparameter selection with the covariance distribution. The extreme skew in the token-level covariance distribution (Table 1: top 0.02% of tokens have >1800× the mean covariance) provides a diagnostic for practitioners setting Clip-Cov or KL-Cov hyperparameters. Rather than blindly sweeping $r$ or $k$, one can compute the covariance distribution at training step 1 for the specific model–data combination, identify the percentile containing the extreme outliers (the right tail of the distribution), and set the intervention ratio to target that tail. For example, if the top 0.1% of tokens have covariance >5 while the remaining 99.9% have covariance <1, an intervention ratio of $10^{-3}$ would target the outlier regime without affecting the bulk of tokens. This data-driven hyperparameter selection could reduce the need for expensive sweeps and make the methods more transferable across model–data combinations. The paper's controllability demonstration (Figure 13) shows that the entropy level responds monotonically to the intervention hyperparameters, so practitioners can also use a small number of short training runs with different $r$ or $\beta$ to calibrate the entropy level they want, then commit to a full run with the chosen setting.

When to Prefer This Method

The paper positions Clip-Cov and KL-Cov against three alternatives, with clear conditions for preference:

Prefer Clip-Cov or KL-Cov over vanilla GRPO (no entropy intervention):

• When training a base LLM with GRPO on reasoning tasks where sustained exploration is needed to discover diverse reasoning strategies. The methods preserve entropy and prevent the performance plateau that vanilla GRPO hits after the early training stage (Figure 12, right: GRPO plateaus while Clip-Cov/KL-Cov continue improving).
• When training larger models (32B+), where the gap between GRPO and the entropy-controlled methods is largest (Table 2: +6.4 points average for KL-Cov on 32B vs. +2.0 points on 7B). Larger models have more latent reasoning capacity that entropy collapse prevents from being surfaced, so the return on entropy control scales with model size.
• When the training data includes hard problems that the model cannot quickly master, since the methods maintain the exploration needed for the model to eventually discover correct strategies for these problems (evidenced by the largest gains on the hardest benchmarks: +15.0 points on AIME24 for KL-Cov on 32B).

Prefer KL-Cov over Clip-Cov:

• When training stability is a priority. KL-Cov produces "stabler entropy curves" (Figure 13, caption) because it uses a continuous KL penalty rather than Clip-Cov's binary gradient detachment with random selection. The continuous penalty provides finer-grained control and less stochasticity in the entropy trajectory.
• At larger model scales. KL-Cov outperforms Clip-Cov by a wider margin on 32B (52.2% vs. 50.3% average, Table 2) than on 7B (40.6% vs. 40.4%), suggesting KL-Cov's stability advantage becomes more important as model size grows.

Prefer Clip-Cov over KL-Cov:

• When implementation simplicity is paramount. Clip-Cov requires only a detach_() call on selected tokens, while KL-Cov requires computing an additional KL divergence term. Clip-Cov is the minimal-code-change option (Listing 11: fewer added lines than KL-Cov).

Prefer Clip-Cov or KL-Cov over clip-higher (Yu et al., 2025):

• When the training involves larger models or challenging benchmarks where clip-higher's instability becomes problematic. The paper shows clip-higher regressing on MATH500 for Qwen2.5-32B (77.2% vs. GRPO's 84.2%, Table 2) while the proposed methods maintain or improve performance on all benchmarks.
• When precise entropy control is desired. Clip-higher operates indirectly by adding low-covariance tokens rather than targeting high-covariance ones (Section 4.5), making it a coarse knob. Clip-Cov and KL-Cov provide monotonic, well-separated entropy control via their hyperparameters (Figure 13).

Prefer Clip-Cov or KL-Cov over entropy loss or KL regularization (Section 4.1):

• In essentially all RL-for-reasoning scenarios where the goal is to improve task performance beyond the base model. Entropy loss is too hyperparameter-sensitive to be practically usable (Figure 9: the window between ineffective and explosive is <10× in coefficient), and KL regularization toward a reference model preserves entropy but degrades performance by constraining learning (Figure 10), making it counterproductive for reasoning improvement where the whole point is to move beyond the base model's distribution.

Caveats. The paper's method evaluation is limited to Qwen2.5 models on math tasks with GRPO, and to single-seed results (Table 2). The preference conditions above should be interpreted as hypotheses grounded in the paper's evidence rather than as universally validated recommendations. Practitioners using different model families, task domains, or RL algorithms should run a small-scale validation before adopting these methods for production training.