Diversity or Precision? A Deep Dive into Next Token Prediction

🎯 Pitch

Contrary to the belief that high entropy in a model's output distribution fuels exploration, imposing a precision-oriented prior during pre-training creates a superior launchpad for reinforcement learning. By actively penalizing low-ranking 'tail' tokens and sharpening probability around correct answers, rather than preserving broad diversity, the authors unlock significantly better reasoning performance after RL.

1. Executive Summary

This paper reinterprets next-token prediction as a single-step policy gradient optimization and introduces a generalized pre-training objective that reshapes the token-output distribution through a reward-shaping strategy — using a positive reward scaling factor (β) to control probability concentration on ground-truth tokens and a rank-aware mechanism (assigning asymmetric rewards to high-ranking vs. low-ranking negative tokens) to modulate local entropy — establishing that the pre-trained distribution critically shapes the exploration space for subsequent reinforcement learning. Training a family of dense and MoE models (1B–20B parameters) on 500B pre-training tokens followed by mid-training and RLVR on mathematical reasoning benchmarks, the authors find that precision-oriented priors (β < 0 for global entropy reduction, or penalizing tail tokens via λ̂ = −0.1) consistently yield superior downstream RL performance, with the low-entropy β = −0.25 configuration outperforming the high-entropy β = 0.50 baseline across all evaluation metrics (Avg@128, Cons@128, Pass@64) on the 4B dense, 10B-A0.5B MoE, and 20B-A1B MoE models. Contrary to the intuition that higher distribution entropy facilitates exploration, the paper demonstrates that a precision-focused pre-training prior provides a more effective initialization for RL, establishing that the diversity-precision trade-off must be explicitly managed during pre-training rather than deferred to subsequent RL stages — and that targeting tail-token suppression and ground-truth concentration is more beneficial than maintaining broad token diversity for end-to-end reasoning performance.

2. Context and Motivation

The fundamental question this paper tackles arises from a gap in how the field thinks about the relationship between pre-training and reinforcement learning (RL): what makes a good pre-trained model for subsequent RL fine-tuning? The standard answer has been simple — train the best next-token predictor you can, as measured by perplexity or downstream benchmarks, and then apply RL as a separate, subsequent stage. But this paper argues that this answer is incomplete because it ignores how the shape of the pre-trained token-output distribution — not just its accuracy — determines what the model can discover during RL.

The problem becomes visible through a concrete empirical pattern that recent work has documented (Wang et al., 2025; Zhu et al., 2025b; Cui et al., 2025; Gandhi et al., 2025): when LLMs engage in chain-of-thought reasoning, the uncertainty that drives exploration is concentrated in a small subset of forking tokens — pivotal decision points where the model must choose between qualitatively different reasoning paths. The majority of tokens, by contrast, exhibit low entropy. This means that the pre-trained model's output distribution at these few critical tokens disproportionately determines which reasoning trajectories the model can explore during RL. If pre-training assigns near-zero probability to a token that represents a promising reasoning step, RL may never discover it, regardless of how many rollouts are performed — the exploration space is effectively truncated by the pre-trained distribution.

The practical consequence is that two pre-trained models with identical perplexity can exhibit dramatically different downstream RL performance, because perplexity only measures aggregate predictive accuracy and is insensitive to whether the probability mass is appropriately distributed across the alternatives that matter for reasoning. This paper's central claim is that the precision-diversity trade-off must be explicitly managed during pre-training to shape the exploration space that RL inherits — and that the standard cross-entropy objective provides no knobs for doing so.

Why This Matters: The Pre-Training-to-RL Pipeline Is the Dominant Paradigm

The practical significance of this problem is hard to overstate. The dominant recipe for building state-of-the-art reasoning LLMs — exemplified by DeepSeek-R1 (Guo et al., 2025), Kimi K2, and Claude — follows a three-stage pipeline: (1) pre-train a base model on trillions of tokens, (2) optionally perform mid-training or supervised fine-tuning, and (3) apply RL with verifiable rewards (RLVR) to elicit long chain-of-thought reasoning. In this paradigm, Stage 1 determines the initial policy distribution from which all subsequent learning starts. If pre-training is misaligned with the needs of RL — for instance, by producing an overly flat distribution that fails to concentrate probability on promising tokens, or by being so sharp that it prematurely eliminates viable alternatives — then the entire downstream investment in RL compute is operating from a suboptimal initialization.

There is also a compelling theoretical bridge motivating this work. Recent efforts (Zelikman et al., 2024; Dong et al., 2025; Li et al., 2025; Xing et al., 2025) have explored applying RL objectives directly to pre-training corpora. Conceptually, next-token prediction can be reframed as a reasoning task: given a context, predict the next token, receiving a verifiable reward when correct. If the intermediate reasoning steps are omitted — i.e., the model generates the answer directly — this collapses to standard pre-training. This equivalence suggests that the pre-training objective is an RL objective (albeit with a specific, unexamined reward structure), and therefore that the choices made during pre-training (reward shaping, exploration, exploitation) should be informed by RL principles. The paper positions itself at this intersection: if pre-training is RL in disguise, then we should design pre-training objectives with the same intentionality about reward structure that we use in downstream RL.

Prior Approaches Fall Short Along Three Dimensions

The paper identifies specific limitations in how the field currently handles the pre-training-to-RL interface.

First, cross-entropy is treated as a fixed, unexamined constant. The standard pre-training objective maximizes log πθ(xt | st) — the log-probability of the ground-truth next token. This is universally used but rarely interrogated. As the paper formally derives (Section 2.2), cross-entropy implicitly encodes a specific reward structure: it assigns a reward of 1/πθ(xt | st) to the ground-truth token and exactly zero reward to all negative tokens. Suppression of alternatives happens only indirectly, through the softmax normalization constraint — when the probability of the correct token increases, competitors are forced to decrease proportionally. This reward structure offers no mechanism to independently control (a) how aggressively to concentrate probability on the ground truth versus maintaining diversity for plausible alternatives, or (b) whether to treat all negative tokens uniformly or to differentiate between high-probability competitors (which may represent genuine reasoning alternatives) and low-probability tail tokens (which represent noise).

Second, existing modifications to cross-entropy address narrow problems without the RL exploration lens. The literature on weighted cross-entropy variants — label smoothing and focal loss (Lin et al., 2018) — provides partial knobs. Label smoothing adds a uniform distribution over the vocabulary, encouraging diversity across all tokens equally. Focal loss down-weights the contribution of easy examples by multiplying the standard loss by (1 - πθ(xt | st))^γ, which has the effect of focusing training on tokens where the model is uncertain. But neither approach provides asymmetric control over the negative distribution, and neither is motivated by or evaluated against downstream RL performance. They are designed for and evaluated on static accuracy metrics, not on the quality of the exploration space they create for subsequent reinforcement learning.

Third, the field lacks any systematic framework for understanding how pre-training reward configuration affects RL outcomes. Even within the RL literature itself, there is an acknowledged gap. Studies on RL for reasoning (Cui et al., 2025; Wang et al., 2025) have observed that token-level entropy dynamics during RL are critical — for instance, that entropy collapse in early RL training can permanently damage reasoning capabilities. But these observations are made during RL, treating the pre-trained distribution as a fixed input. No prior work, to the authors' knowledge, has systematically varied the pre-training objective to study how it modulates RL behavior, nor proposed a general framework for designing pre-training objectives with downstream RL in mind. The field has been operating with an implicit assumption that "better pre-training accuracy → better RL initialization," and this paper challenges that assumption directly.

Where Conflicting Intuitions Exist

The paper explicitly sets itself against a conventional intuition: that higher entropy (more diversity) during pre-training should facilitate better RL exploration. The reasoning is superficially plausible — if RL needs to explore a wide space of reasoning paths, a flatter pre-trained distribution that assigns non-trivial probability to many alternatives should make more paths reachable. This intuition aligns with standard practices in RL, where exploration bonuses, entropy regularization, and stochastic policies are used to prevent premature convergence to suboptimal policies.

However, the paper presents evidence that this intuition is wrong for the specific setting of LLM pre-training followed by RLVR on reasoning tasks. The core finding — that precision-oriented priors (low entropy, aggressive ground-truth concentration, tail-token suppression) outperform diversity-oriented priors (high entropy, flat distributions, rewarding of high-ranking competitors) — suggests a more nuanced mechanism. One hypothesis the paper explores through its analysis of RL entropy dynamics (Figure 8) is that high-entropy pre-trained distributions actually experience more rapid entropy collapse during early RL, leading to premature convergence and shortened reasoning chains. In contrast, precision-oriented priors provide a more stable foundation where RL can progressively increase exploration without catastrophic forgetting or distribution collapse.

How This Paper Positions Itself

The paper positions itself as both a theoretical bridge and an empirical investigation. The theoretical contribution is the formal derivation in Section 2 showing that cross-entropy can be interpreted as a specific instance of policy gradient optimization (Equations 4–10), exposing its implicit reward structure and demonstrating that the objective can be generalized by modifying the reward function along two independent axes: positive reward scaling (controlling how much reward the ground-truth token receives as a function of the model's current certainty) and rank-aware negative shaping (assigning differential rewards to high-probability vs. low-probability negative tokens). This framework subsumes standard cross-entropy, label smoothing, and focal loss as special cases.

The empirical contribution is the first large-scale study — spanning five model scales (1B to 20B parameters), both dense and MoE architectures, and a complete pre-training → mid-training → RLVR pipeline on 500B+ tokens — of how different reward configurations during pre-training affect downstream RL performance on mathematical reasoning benchmarks. The paper does not propose a new RL algorithm or a new architecture. Instead, it treats the pre-training objective itself as a hyperparameter to be optimized for downstream RL performance, and provides concrete evidence about which settings of that hyperparameter work best.

The paper explicitly frames its contribution as providing a "more favorable exploration space for RL" (Section 1), rather than improving pre-training metrics per se. This distinguishes it from work that modifies pre-training for better perplexity or few-shot performance. The evaluation is end-to-end: what matters is not pre-training validation loss, but the reasoning accuracy achieved after RLVR — and the paper shows that pre-training objectives that produce comparable perplexity (Figure 1, Figure 2) can lead to substantially different RL outcomes (Figures 5–7).

Finally, the paper positions its findings as "contrary to the conventional intuition that higher distribution entropy facilitates effective exploration" (Section 1), directly challenging a widespread assumption in both the pre-training and RL communities. This counterintuitive result — that imposing a precision-oriented prior yields a superior exploration space — is the paper's most provocative claim and the one around which its broader significance revolves.

3. Technical Approach

3.1 Reader Orientation

The paper develops a generalized pre-training objective — a modified loss function for next-token prediction — that explicitly controls the trade-off between precision (aggressively concentrating probability on the correct token) and diversity (preserving probability mass for plausible alternatives). The core problem it solves is that standard cross-entropy provides no knobs for tuning this trade-off, and the default settings inherited from cross-entropy produce a suboptimal exploration space for subsequent reinforcement learning on reasoning tasks. The solution's shape is a reward-shaping framework that reinterprets cross-entropy as a policy gradient, then generalizes it by adding independent controls for positive token reward scaling (a parameter β that determines how aggressively to concentrate probability on the ground truth) and rank-aware negative token shaping (parameters λ̂ and λ̃ that assign different reward signals to high-probability vs. low-probability incorrect tokens, plus a hyperparameter k defining the boundary between those two regions).

3.2 Big-Picture Architecture (Diagram in Words)

The system is a training pipeline with one novel component — the generalized objective — embedded in a standard pre-training → mid-training → RLVR workflow. The major components are:

Base LLM (the stochastic policy πθ) — a transformer that takes a sequence of previous tokens as state st = {x₁, ..., x_{t-1}} and produces a probability distribution over the vocabulary V for the next token. Available in both dense and MoE architectures across five scales (1B, 4B dense; 5B-A0.3B, 10B-A0.5B, 20B-A1B MoE).
Generalized Reward Function r̄(st, at) — the paper's core contribution. This function replaces the implicit cross-entropy reward (which is zero for all incorrect tokens and 1/πθ(xt|st) for the correct one) with a parameterized reward that has three independent control knobs: β scales the positive reward, and λ̂, λ̃, and k shape the negative reward landscape by treating high-ranking and low-ranking incorrect tokens asymmetrically.
Pre-Training Objective — the policy gradient applied to single-token episodes (Equations 1–5), using the generalized reward instead of the cross-entropy-derived reward. This is applied to 500B tokens of general knowledge text.
Mid-Training — a standard continuation training stage on 100B tokens that increases reasoning content and extends sequence length to 16,384, applied identically regardless of the pre-training objective used.
RLVR (Reinforcement Learning with Verifiable Rewards) — an on-policy GRPO algorithm applied to mathematical reasoning tasks (700 steps at 8K sequence length, then continued at 16K), using the pre-trained/mid-trained model as the initial policy. The RL stage evaluates how the pre-training objective's shaping of the token distribution affects downstream reasoning performance.

Information flows as follows: a pre-training corpus → the model generates a token distribution → the generalized reward function scores each possible token (positive reward if it matches the ground truth, scaled by β and current probability; negative reward if it doesn't match, with different values for top-k and tail tokens) → the policy gradient updates the model → after 500B tokens, mid-training proceeds with standard cross-entropy → the resulting model enters RLVR on math problems → final evaluation measures reasoning accuracy.

3.3 Roadmap for the Deep Dive

First, the paper's formal reframing of next-token prediction as a single-step Markov Decision Process (Equations 1–5), because this is the conceptual foundation that makes reward shaping possible.
Second, the derivation showing how standard cross-entropy implicitly encodes a specific reward function (Equations 6–10), because this reveals what knobs cross-entropy lacks and why generalization is needed.
Third, the generalized reward function itself (Equations 11–13), which is the paper's technical contribution — the positive reward scaling factor β, the rank-aware negative shaping with λ̂, λ̃, and k, and how these combine to independently control global and local entropy.
Fourth, the concrete hyperparameter configurations tested and the design rationale behind each choice, since the paper's empirical contribution is the comparison of these configurations.
Fifth, the training pipeline (pre-training, mid-training, RLVR) and evaluation setup, because the paper's claims depend on the end-to-end workflow being identical except for the pre-training objective.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a theoretical reframing + empirical investigation paper whose core idea is that next-token prediction is a valid policy gradient problem (not just supervised learning), and therefore its reward function can be shaped using RL principles to control the diversity-precision trade-off in the pre-trained distribution.

Framing Next-Token Prediction as a Sequential Decision Process

The paper begins by reformulating autoregressive language modeling in the language of reinforcement learning. This is not itself novel — the LLM-as-policy analogy is widely used in RLHF — but the paper pushes it further by treating each individual token generation as a complete episode, which is a specific and consequential choice.

An autoregressive LLM is cast as a stochastic policy πθ where θ represents the model parameters. At each time step t, the model observes a state st, defined as the prefix of already-generated tokens X_{<t} = {x₁, x₂, ..., x_{t-1}}, and takes an action at by sampling a next token from the vocabulary V according to the policy distribution πθ(· | st). The training objective is to maximize the expected cumulative reward:

$J(θ) = \mathbb{E}_{τ ∼ π_θ} \left[ \sum_{t=1}^{n} r(s_t, a_t) \right]$

where τ = (s₁, a₁, s₂, a₂, ...) represents a trajectory sampled from the policy πθ, n is the sequence length, and r(st, at) is the scalar reward received for taking action at in state st.

What it computes: the expected total reward over complete sequences generated by the model's current policy. The expectation is taken over the distribution of trajectories induced by πθ, and the inner sum accumulates per-token rewards along each trajectory. The output is a scalar function of the model parameters θ that serves as the optimization target.

Why this form: this is the standard RL objective for episodic tasks. It captures that the model should maximize not just immediate predictive accuracy but the quality of the entire generated sequence. The policy gradient theorem (Equation 2) then provides an unbiased estimator of the gradient of this objective:

$\nabla_θ J(θ) = \mathbb{E}_{τ ∼ π_θ} \left[ \sum_{t=1}^{n} R(τ) \nabla_θ \log π_θ(a_t | s_t) \right]$

where R(τ) = Σ_{t'=1}^{n} r(s_{t'}, a_{t'}) is the total return of the trajectory. In practice, for variance reduction without introducing bias, R(τ) is replaced by the return-to-go G_t = Σ_{t'=t}^{n} r(s_{t'}, a_{t'}) (the sum of rewards from the current step onward), and a baseline b(st) is subtracted:

$\nabla_θ J(θ) = \mathbb{E}_{τ ∼ π_θ} \left[ \sum_{t=1}^{n} (G_t - b(s_t)) \nabla_θ \log π_θ(a_t | s_t) \right]$

Why the single-step collapse matters: the paper's crucial move is to treat the generation of a single token as a complete episode (Section 2.1, paragraph after Equation 3). This means the trajectory length is n = 1, so there is no sequential dependency — the "return-to-go" is simply the immediate reward r(st, at). The objective for a fixed state st simplifies to:

$J_t(θ | s_t) = \mathbb{E}_{a_t ∼ π_θ(· | s_t)} [r(s_t, a_t)]$

yielding the gradient:

$\nabla_θ J_t(θ | s_t) = \mathbb{E}_{a_t ∼ π_θ(· | s_t)} \left[ r(s_t, a_t) \nabla_θ \log π_θ(a_t | s_t) \right]$

This simplification is what enables the paper to derive the implicit reward of cross-entropy — because with a single-token episode, the reward r(st, at) must depend solely on the immediate state-action pair, with no future consequences. Standard teacher forcing (where the ground-truth prefix is provided as context regardless of the model's actual predictions) is therefore compatible with this framing, since each token prediction is evaluated independently against a known target.

Deriving Cross-Entropy's Implicit Reward Function

The paper's central theoretical move is to show that the standard cross-entropy objective — universally used in LLM pre-training — can be expressed exactly in the policy gradient form of Equation 5, which reveals the reward function that cross-entropy implicitly uses. This derivation (Equations 6–10) is the foundation for the generalized objective.

Standard LLM pre-training maximizes the log-likelihood of the ground-truth next token xt given the context st:

$J_{CE}(θ) = \log π_θ(x_t | s_t)$

The gradient of this objective is:

$\nabla_θ J_{CE}(θ) = \nabla_θ \log π_θ(x_t | s_t)$

What this gradient does directly: it increases the log-probability of exactly one token — the ground-truth xt — with a weight of exactly 1. It makes no explicit reference to any other token in the vocabulary.

The paper then rewrites this gradient as an expectation over the entire policy distribution πθ(· | st), using a standard trick from policy gradient derivations. First, the gradient is expressed in terms of probability mass rather than log-probability:

$\nabla_θ J_{CE}(θ) = \frac{1}{π_θ(x_t | s_t)} \nabla_θ π_θ(x_t | s_t)$

This uses the chain rule: ∇_θ log πθ = (1/πθ) ∇_θ πθ. Next, the indicator function \mathbb{1}(a_t = x_t) is introduced to expand the gradient over the full vocabulary:

$\nabla_θ J_{CE}(θ) = \frac{1}{π_θ(x_t | s_t)} \sum_{a_t \in \mathcal{V}} \mathbb{1}(a_t = x_t) \nabla_θ π_θ(a_t | s_t)$

The indicator is 1 when the candidate token at matches the ground truth xt, and 0 otherwise. By substituting \nabla_θ π_θ(a_t | s_t) = π_θ(a_t | s_t) \nabla_θ \log π_θ(a_t | s_t) (the reverse of the chain rule step), the expression becomes an expectation over the policy:

$\nabla_θ J_{CE}(θ) = \sum_{a_t \in \mathcal{V}} π_θ(a_t | s_t) \left[ \frac{\mathbb{1}(a_t = x_t)}{π_θ(a_t | s_t)} \nabla_θ \log π_θ(a_t | s_t) \right]$

$= \mathbb{E}_{a_t \sim π_θ(· | s_t)} \left[ \frac{\mathbb{1}(a_t = x_t)}{π_θ(a_t | s_t)} \nabla_θ \log π_θ(a_t | s_t) \right]$

Comparing this with the policy gradient form in Equation 5 reveals the intrinsic reward function that cross-entropy implicitly uses:

$r_{CE}(s_t, a_t) = \text{sg} \left( \frac{\mathbb{1}(a_t = x_t)}{π_θ(a_t | s_t)} \right)$

where sg(·) denotes the stop-gradient operator, meaning that the expression inside is treated as a constant when computing gradients — it does not contribute additional gradient terms through the reward channel. This is essential because in a true RL setting, the reward is external and not differentiated through.

What this reward function computes: when the sampled action at matches the ground-truth xt, the reward is 1 / πθ(xt | st) — the inverse of the model's current probability for the correct token. This means the reward is larger when the model was less confident in the correct answer (a low probability yields a high inverse), and smaller when the model was already confident (a high probability yields a low inverse). When the sampled action does not match the ground truth (at ≠ xt), the reward is exactly zero.

Why this form matters for understanding the implicit trade-off: the zero reward for negative tokens means cross-entropy does not actively penalize incorrect predictions. Instead, suppression of negative tokens happens indirectly through the softmax normalization constraint Σ_{a_t} π_θ(a_t | s_t) = 1. When the gradient update increases πθ(xt | st) via the positive reward, the probabilities of all other tokens are forced to decrease proportionally to maintain the sum-to-one constraint. This is an elegant mechanism — it avoids explicitly computing loss terms for every token in the vocabulary — but it also means that cross-entropy has no independent control over which negative tokens get suppressed and how aggressively. All negative tokens are treated identically: zero reward, with their probability decrease determined entirely by their current share of the probability simplex.

This derivation also establishes that the reward depends exclusively on information available at time step t — the ground-truth token xt is deterministically defined by the training corpus given the state st, satisfying the requirement from Equation 5 that the reward depend only on the immediate state-action pair. This justifies the single-step episode formulation.

The Generalized Reward Function: Three Independent Control Knobs

Having exposed cross-entropy's implicit reward structure, the paper proposes a generalized reward function r̄(st, at) with three independently adjustable components. The generalization proceeds along two axes: positive reward scaling (controlling how the ground-truth token is rewarded) and rank-aware negative shaping (controlling how incorrect tokens are differentially treated based on their position in the model's ranked probability distribution).

Positive Reward with a Scaling Factor β:

The modified positive reward for the ground-truth token is:

$\bar{r}_{\text{pos}}(s_t, a_t) = \text{sg} \left( \left( \frac{1}{π_θ(a_t | s_t)} \right)^{(1 - π_θ(a_t | s_t))^β} \right)$

where β is a hyperparameter controlling the reward scaling, and (1 - πθ(at | st))^β is the positive reward scaling factor.

What this computes: the standard cross-entropy positive reward 1/πθ is raised to the power of (1 - πθ)^β. When β = 0, the scaling factor becomes (1 - πθ)^0 = 1, so the exponent is 1, recovering standard cross-entropy exactly. When β < 0, the scaling factor (1 - πθ)^β is greater than 1 (since 1 - πθ is between 0 and 1, and a negative exponent yields values > 1), making the effective exponent larger than 1. This amplifies the reward — particularly for tokens where the model is already somewhat confident (higher πθ, smaller 1 - πθ, so (1 - πθ)^β becomes very large for negative β). The result is an aggressively concentrated distribution with low entropy. When β > 0, the scaling factor is less than 1, making the exponent fractional. This attenuates the reward, allowing the model to maintain a flatter distribution with higher entropy.

Why this specific form: the expression embeds the model's current confidence πθ into the reward scaling, making the modulation state-dependent — the degree of amplification or attenuation varies based on how certain the model currently is. This is critical because it means the reward shaping does not apply a uniform scaling across all tokens and contexts; it is more aggressive where the model is already uncertain (low πθ, high 1 - πθ) and less aggressive where the model is already certain. The paper contrasts this with the standard focal loss formulation (1 - πθ)^γ, noting that the proposed form is "equivalent to applying a state-dependent weight w_t = π_θ(x_t | s_t)^{1 - (1 - π_θ(x_t | s_t))^β} to the standard cross-entropy" (Section 4.1) — this connection to existing loss functions provides theoretical grounding.

Rank-Aware Negative Shaping with λ̂, λ̃, and k:

The modified negative reward for tokens that do not match the ground truth is:

$\bar{r}_{\text{neg}}(s_t, a_t) = \tilde{λ} \cdot \mathbb{1}(a_t \in K_t \land a_t \neq x_t) + \hat{λ} \cdot \mathbb{1}(a_t \notin K_t \land a_t \neq x_t)$

where K_t = \text{TopK}(π_θ(· | s_t), k) is the set of the top-k predicted tokens (by probability) at state st, λ̃ is the reward assigned to high-ranking negative tokens (those in the top-k), λ̂ is the reward assigned to low-ranking negative tokens (those outside the top-k), and \mathbb{1}(·) is the indicator function.

What this computes: the vocabulary is partitioned into three regions at each prediction step: (1) the single ground-truth token xt, which receives r̄_pos; (2) the top-k tokens by probability that are not the ground truth, which receive reward λ̃; and (3) all remaining tail tokens (outside the top-k and not the ground truth), which receive reward λ̂. The parameter k determines the boundary between "high-ranking" and "low-ranking" regions.

Why this form: the standard cross-entropy implicitly treats all negative tokens identically (zero reward). The rank-aware mechanism introduces asymmetric treatment: it can reward high-ranking negative tokens (λ̃ > 0) to preserve probability mass on plausible alternatives — preventing the model from becoming overconfident in the single ground truth — while simultaneously penalizing tail tokens (λ̂ < 0) to force the distribution to concentrate on the head and eliminate noise. Alternatively, it can penalize high-ranking competitors (λ̃ < 0) to force sharper discrimination between the ground truth and its nearest competitors, while leaving tail tokens unaffected (λ̂ = 0). The paper's experiments test λ̂ = −0.1, λ̃ = 0, k = 100 (suppress tail tokens, leave top-100 competitors unchanged) and λ̂ = 0, λ̃ = 0.1, k = 100 (reward top-100 competitors, leave tail tokens unchanged), which represent opposite strategies for managing local entropy.

The boundary k = 100 means that the top 100 tokens by probability receive different treatment from the remaining tens of thousands of tokens in the vocabulary. This is a practical choice reflecting the observation that in typical LLM token distributions, probability mass is heavily concentrated in the head — the top few hundred tokens account for the vast majority of the probability — so differentiating the head from the tail is both computationally tractable and semantically meaningful.

The Full Generalized Reward:

The complete reward function combines the positive and negative components:

$\bar{r}(s_t, a_t) = \bar{r}_{\text{pos}}(s_t, a_t) \cdot \mathbb{1}(a_t = x_t) + \bar{r}_{\text{neg}}(s_t, a_t) \cdot \mathbb{1}(a_t \neq x_t)$

What this computes: a per-token reward signal that replaces the standard cross-entropy reward in the policy gradient update (Equation 5). When β = 0, λ̃ = 0, and λ̂ = 0, this reduces exactly to standard cross-entropy — r̄_pos becomes 1/πθ (since the exponent is 1), and r̄_neg is identically zero. Any deviation from these baseline values modifies the reward landscape.

Why this decomposition into three parameters: the paper argues that diversity and precision are not opposite ends of a single spectrum but rather two dimensions that can be controlled independently. The β parameter controls global entropy — the overall peakedness or flatness of the distribution — by amplifying or attenuating the ground-truth reward uniformly. The λ̂, λ̃, and k parameters control local entropy — the relative allocation of probability mass among negative tokens — by selectively rewarding or penalizing different regions of the distribution. This separation enables experiments that isolate the effects of each type of control: for instance, one can test whether lowering global entropy alone improves RL outcomes without touching local entropy (by varying β while keeping λ̂ = λ̃ = 0), or whether reshaping the negative distribution matters independent of global concentration (by varying λ̂ and λ̃ while keeping β = 0).

Concrete Hyperparameter Configurations Tested

The paper tests five specific configurations of the generalized objective against the baseline (standard cross-entropy, β = 0, λ̃ = 0, λ̂ = 0):

Global low entropy (precision-oriented): β = −0.25, λ̃ = 0, λ̂ = 0. The negative β amplifies ground-truth rewards, aggressively concentrating probability mass on the correct token. Local entropy is not modified — all negative tokens receive zero reward as in standard cross-entropy.
Global high entropy (diversity-oriented): β = 0.50, λ̃ = 0, λ̂ = 0. The positive β attenuates ground-truth rewards, maintaining a flatter distribution with higher entropy. Again, local entropy is not modified.
Tail suppression (precision-oriented local): β = 0, λ̃ = 0, λ̂ = −0.10, k = 100. The global reward is standard cross-entropy, but low-probability tail tokens (outside the top-100) receive a penalty of −0.10, forcing the distribution to concentrate on the head. Top-100 competitors of the ground truth are unaffected.
Head rewarding (diversity-oriented local): β = 0, λ̃ = 0.10, λ̂ = 0, k = 100. The global reward is standard cross-entropy, but high-probability competitors (top-100 tokens that are not the ground truth) receive a reward of 0.10, preserving probability mass for plausible alternatives and preventing overconfidence.
Combined configurations: the paper also tests configurations that combine global and local controls, though these are evaluated primarily through the interaction effects visible in the training dynamics (Figures 1–2 show entropy and PPL trajectories for each configuration separately).

Why these specific values: the paper does not provide an extensive hyperparameter sweep rationale, but the chosen values represent symmetric perturbations around the baseline (β = −0.25 vs. β = 0.50 are roughly equidistant from zero in opposite directions; λ̂ = −0.10 and λ̃ = 0.10 are equal magnitude with opposite signs). This design enables clean comparisons: does pushing precision in either direction (global or local) outperform the baseline? Is the effect symmetric — does going toward diversity yield the opposite of going toward precision? The values are also chosen to be small enough that perplexity still converges to comparable low values across all configurations (as shown in Figures 1 and 2), ensuring that the comparison is between differently-shaped distributions of comparable predictive quality, not between well-trained and poorly-trained models.

Why is there no explicit term for the top-k correct tokens? An important subtlety: the reward function as written in Equation 12 only applies to at ≠ xt, so the top-k set K_t is implicitly the top-k tokens excluding the ground truth (since the ground truth is handled by r̄_pos). This means λ̃ rewards or penalizes the strongest competitors to the ground truth, not tokens that might include the ground truth itself. The boundary between "positive" and "high-ranking negative" treatment is clean: exactly one token (the dataset-defined ground truth) gets r̄_pos, the top-k other tokens get λ̃, and everything else gets λ̂.

The Training Pipeline: Pre-Training → Mid-Training → RLVR

The paper's empirical contribution depends on executing a complete training pipeline identically for all objective configurations, varying only the pre-training stage reward function. All other hyperparameters and data are held constant to isolate the effect of the pre-training objective on downstream RL performance.

Pre-Training Stage (Section 3.1, Section A):

Corpus: 500B tokens of general knowledge text. The data is deliberately curated to exclude synthetic long-reasoning traces to avoid confounding the observation of reasoning capability emergence during RL.
Architecture: both dense models (1B: 28 layers, 1536 hidden dim, 16 heads, 4 KV heads; 4B: 36 layers, 2560 hidden dim, 32 heads, 8 KV heads) and Mixture-of-Experts models (5B-A0.3B: 12 layers, 1024 hidden dim, 384 total experts, 12 active; 10B-A0.5B: 16 layers, 1536 hidden dim, 384 experts, 12 active; 20B-A1B: 24 layers, 1536 hidden dim, 384 experts, 12 active). MoE training uses an auxiliary-loss-free approach (Liu et al., 2024) — meaning expert load balancing is handled through architectural mechanisms rather than an explicit auxiliary loss term.
Optimizer: AdamW with weight decay 0.1 and gradient clipping at 1.0. Learning rate schedule: warmup-stable-decay with a global batch size of 16M tokens. During the stable 500B-token phase, the learning rate warms up over 2000 steps and stabilizes at 3 × 10^{-4}.
Sequence length: 4096 tokens during pre-training.
RoPE configuration: base frequency 1e4 (standard for pre-training context lengths).
The only variable across runs: the reward function configuration in the generalized objective (β, λ̂, λ̃, k). All other settings — data, architecture, optimizer, schedule — are identical.

Mid-Training Stage (Section 3.1, Section A):

Corpus: 100B additional tokens with approximately 5% synthetic data and significantly increased reasoning-oriented content (compared to the general-knowledge pre-training corpus). The synthetic component does NOT include long reasoning traces — the authors explicitly state they "deliberately exclude the synthetic long-reasoning data from all training stages" to cleanly observe when long chain-of-thought reasoning activates during RL.
Learning rate: decays from 3 × 10^{-4} to 3 × 10^{-5} over the 100B tokens.
Sequence length: extended to 16,384 tokens (from 4096 during pre-training) to support the long-context reasoning patterns that RL is expected to elicit.
RoPE adjustment: base frequency increased from 1e4 to 1e6, a standard practice for supporting longer contexts by reducing the decay rate of positional attention scores.
Objective: this stage uses standard cross-entropy for all model variants, regardless of the pre-training objective used. This is a critical design choice: the mid-training stage does not continue using the generalized objective. This means the paper measures the persistent effect of the pre-training distribution shape on the model's trajectory, rather than the effect of continuing to apply the shaped reward. If the pre-training objective matters, it matters because of the initial distribution it creates, not because the reward is applied throughout.

RLVR Stage (Section 3.1, Section B):

Algorithm: on-policy GRPO (Group Relative Policy Optimization) without KL regularization, following the recipe from Shao et al. (2024) and incorporating clip-higher and dynamic sampling strategies from Yu et al. (2025) to stabilize training.
Task domain: mathematical reasoning — the paper states "the emergence of long-reasoning capabilities is typically associated with these domains." The RL stage uses verifiable rewards (correctness of derived mathematical solutions), which is the standard RLVR paradigm exemplified by DeepSeek-R1.
Two sub-stages: an initial 700 steps with sequence length 8K, followed by continued training at sequence length 16K. This staged approach allows the model to first learn reasoning capabilities at moderate context lengths before scaling to longer chains.
Sampling: 16 outputs per prompt at temperature 1.0. Batch size 128. Constant learning rate 1 × 10^{-6} across both sub-stages.
Evaluation metrics: Avg@128 (average accuracy across 128 samples per problem), Cons@128 (majority voting accuracy across 128 samples), and Pass@64 (probability that at least one of 64 samples is correct, estimated using the unbiased estimator from Chen, 2021). Response length and policy entropy are also tracked throughout training (Figure 8) to reveal mechanistic differences between configurations.
Evaluation benchmarks (RL stage): AIME 2024, AIME 2025, AMC23, OlympiadBench, MATH-500, Minerva — all challenging mathematical reasoning datasets where long chain-of-thought reasoning is expected to improve performance.

Why this pipeline design isolates the pre-training effect: every model variant undergoes identical mid-training (standard cross-entropy on the same data) and identical RLVR (same algorithm, same hyperparameters, same math tasks). The only difference between, say, the β = −0.25 model and the β = 0.50 model is the reward function used during the 500B-token pre-training stage. Any difference in final RL performance must therefore be attributed to the downstream consequences of that pre-training distribution shape — whether through the exploration space it defines, the entropy dynamics it induces during RL, or the reasoning patterns it makes accessible.

Evaluation Design for Base Models (Pre- and Mid-Training)

The paper evaluates base models (pre-trained and mid-trained, before RL) on a comprehensive set of 19 benchmarks spanning five capability categories (Section 3.2):

General Knowledge: MMLU (4-shot CoT), MMLU-Pro (5-shot CoT), TriviaQA (5-shot), NaturalQuestions (5-shot)
Commonsense Reasoning: HellaSwag (0-shot), SIQA (0-shot), PIQA (0-shot), WinoGrande (0-shot), OpenBookQA (5-shot), CommonsenseQA (5-shot)
Logic Reasoning: ARC-Easy (0-shot), ARC-Challenge (0-shot), BBH (3-shot CoT)
Mathematics: GSM8K (4-shot CoT), MATH-500 (4-shot CoT), Minerva (4-shot CoT), OlympiadBench (0-shot)
Coding: HumanEval+ (0-shot), MBPP+ (3-shot)

For mathematics and coding tasks, the paper also reports Pass@k using the unbiased estimator:

$\text{Pass@k} = 1 - \frac{\binom{m - c}{k}}{\binom{m}{k}}$

where m is the total number of sampled responses per prompt, and c is the count of correct responses among those m samples.

What this computes: the probability that at least one correct solution is present in k independent attempts from the model, estimated without upward bias by solving k items without replacement from the m samples (where c are correct). The paper samples m = 128 responses at temperature 0.7 and top-p 0.95, reporting Pass@64.

Why this metric matters for the paper's argument: Pass@k measures the upper bound of the model's capability — its ability to produce a correct answer at all, not its tendency to do so on average. This is directly relevant to the exploration space argument: if a model has high Pass@k, it means the correct solution is somewhere in its output distribution, and the question is whether RL can learn to preferentially sample it. If Pass@k is low, the correct solution may not be reachable from the pre-trained distribution regardless of how well RL explores. The paper's Pass@k analysis (Figure 9, Section 3.6) examines whether different pre-training configurations affect this upper bound differently — and finds that precision-oriented priors (low entropy, tail suppression) yield higher Pass@k on math and coding tasks, suggesting they create a distribution where correct solutions are more reachable, even though the distribution is less diverse overall.

For base models, maximum output length is 4K for pre-trained models and 16K for mid-trained models (to accommodate the extended context window). For RL models, the maximum output length is 16K.

4. Key Insights and Innovations

Innovation 1: Reframing Pre-Training as a Reward-Shaping Problem for Downstream RL

The paper's most intellectually distinctive move is neither the specific reward function nor the empirical results — it is the conceptual reframing of pre-training as an RL-initialization design problem rather than a standalone accuracy-maximization problem. Before this work, the dominant paradigm treated pre-training and RL as sequential but independent stages: pre-train for the best possible perplexity using standard cross-entropy, then hand the resulting model to RL as a fixed starting point. The pre-training objective was a constant, not a variable to be optimized for downstream RL performance.

This paper breaks that separation by arguing — through the formal derivation in Section 2.2 — that cross-entropy already is a policy gradient with a specific, unexamined reward structure (zero reward for all incorrect tokens, inverse-probability reward for the correct one). Once this equivalence is established, the natural next question is: what reward structure during pre-training produces the best exploration space for RL? This shifts the optimization target from "minimize perplexity" to "shape the initial policy distribution to maximize RL's ability to discover good reasoning trajectories."

The significance of this reframing extends beyond the paper's specific findings. It establishes a new axis of optimization — the pre-training reward function — that the field had not previously considered as a design degree of freedom. Prior work on training-inference trade-offs (as in the companion paper referenced in the example) studied how to allocate compute between pre-training and inference, but took the pre-training objective as given. Prior work on weighted loss functions (focal loss, label smoothing) modified cross-entropy for static accuracy improvements, not for shaping the exploration manifold that RL inherits. This paper is the first to argue that the pre-training loss function should be designed with RL dynamics in mind, and that the standard cross-entropy configuration (β = 0, λ̂ = 0, λ̃ = 0) is an arbitrary point in a larger design space that happens to be suboptimal for downstream reasoning.

The theoretical bridge that enables this reframing — interpreting next-token prediction as a single-step MDP — is not itself the innovation (Ming et al., 2025, and others have noted this connection). The innovation is taking that bridge seriously enough to act on it: to systematically vary the reward function along two independent axes (positive token concentration and negative token asymmetry) and evaluate the downstream RL consequences at scale. This is a fundamental shift from treating pre-training as a solved problem whose only axis of improvement is "more data, bigger model" to treating it as an RL-informed design problem with unexplored hyperparameters.

Evidence tie-in: The paper demonstrates that models with comparable pre-training perplexity (Figures 1–2) exhibit substantially different RL performance (Figures 5–7), directly validating the claim that perplexity is insufficient to characterize the quality of a pre-trained model as an RL initialization. The 4× efficiency gap between the best and worst pre-training configurations on downstream reasoning — despite comparable pre-training loss — is the empirical signature of this reframing's validity.

Innovation 2: The Counterintuitive Finding That Precision-Oriented Priors Outperform Diversity-Oriented Priors for RL Exploration

The paper's most provocative empirical finding — and the one around which its broader significance revolves — is that imposing a precision-oriented prior during pre-training creates a better exploration space for RL than maintaining high entropy. This directly contradicts the widespread intuition that higher diversity in the pre-trained distribution should facilitate better RL exploration by keeping more reasoning paths accessible.

This finding is significant not merely as a "surprising result" but because it reveals a mechanism — observable in Figure 8 — that explains why the intuition fails. Models pre-trained with high global entropy (β = 0.50) experience more rapid entropy collapse during early RL training, accompanied by a sharp decrease in response length. By RL step 200, the high-entropy pre-trained model's policy entropy has already dropped below that of the low-entropy pre-trained model (β = −0.25), and its average response length has fallen by roughly 40%. In contrast, the low-entropy pre-trained model maintains a more stable entropy trajectory and shows a smooth, continuous increase in response length — the hallmark of progressive activation of long chain-of-thought reasoning.

What this suggests mechanistically is that the high-entropy pre-trained distribution is fragile under RL optimization pressure. When RL starts reinforcing certain reasoning patterns, the model's initially flat distribution collapses toward those patterns too aggressively, losing the very diversity that was supposed to help exploration. The precision-oriented prior, by contrast, provides a more structured starting point — the model is already confident about which tokens are correct, which means RL doesn't need to collapse entropy to achieve high reward on in-distribution patterns, freeing it to progressively explore longer reasoning chains.

This finding is a fundamental insight, not an incremental refinement. It changes the way researchers should think about the relationship between pre-training diversity and RL exploration: the goal is not maximal diversity, but well-structured precision that provides a stable foundation from which RL can progressively expand exploration without catastrophic distribution collapse. This is analogous to findings in other areas of deep learning — for instance, that sharp minima can generalize better than flat minima under certain conditions — where the relationship between a desirable property (flatness, diversity) and downstream performance is more nuanced than simple maximization.

Evidence tie-in: Figure 8 shows the entropy collapse phenomenon across 4B, 10B-A0.5B, and 20B-A1B models. Figures 5–7 show that β = −0.25 (low global entropy) consistently achieves higher Avg@128, Cons@128, and Pass@64 than β = 0.50 (high global entropy) across all model scales and RL steps throughout training — for instance, on the 4B dense model at step 1000, β = −0.25 achieves Cons@128 of 28.83 vs. 27.87 for β = 0.50 (Table 19 vs. Table 20), and on the 20B-A1B model at step 1000, β = −0.25 reaches Avg@128 of 36.06 vs. 32.41 for β = 0.50 (Table 29 vs. Table 30).

Innovation 3: Decomposing Diversity and Precision Into Independent, Orthogonal Control Axes

A key conceptual contribution is the insight that diversity and precision are not opposite ends of a single spectrum but two independent dimensions that can be controlled separately through the generalized reward function. The paper introduces two distinct mechanisms: a global entropy regulator (β) that controls how aggressively probability concentrates on the ground-truth token, and a local entropy shaper (λ̂, λ̃, k) that controls the relative allocation of probability mass among negative tokens by treating high-ranking and low-ranking candidates asymmetrically.

This decomposition is significant because prior modifications to cross-entropy — label smoothing, focal loss — conflated these two dimensions. Label smoothing adds uniform mass to all tokens, which increases global entropy but does so uniformly in a way that preserves probability mass on tail tokens (potentially noise). Focal loss down-weights easy examples, which affects global concentration but provides no mechanism for differentially treating high-probability competitors versus tail tokens. The paper's framework shows that these are special cases of a larger design space, and that the ability to control global and local entropy independently is what enables the discovery that precision-oriented global control and precision-oriented local control both outperform diversity-oriented alternatives — but through different mechanisms and with different effects on RL dynamics.

The empirical evidence supports that these are indeed independent dimensions: the tail-suppression configuration (λ̂ = −0.10, λ̃ = 0) and the global low-entropy configuration (β = −0.25) both outperform the baseline, but they exhibit different entropy dynamics during RL (Figure 8, right panels vs. left panels). The tail-suppression model maintains higher entropy during RL than the global low-entropy model, suggesting it achieves precision through a different mechanism — cleaning up the tail rather than aggressively spiking the ground truth — that leaves more room for RL to shape the head distribution. This demonstrates that the two control axes have genuinely different effects on downstream behavior, not just different implementations of the same underlying phenomenon.

This is an architectural innovation in how we think about pre-training objectives, not a metric-driven improvement. It provides a vocabulary (global vs. local entropy, positive scaling vs. rank-aware negative shaping) and a framework for future research to systematically explore the pre-training design space. The finding that both global and local precision-oriented strategies independently improve RL outcomes suggests that the beneficial effect is robust and not an artifact of a particular hyperparameter choice — and that future work could potentially combine them for further gains.

Evidence tie-in: Figures 1–2 show that β and (λ̂, λ̃) independently modulate PPL and entropy trajectories. Figures 5–7 show that β = −0.25 and (λ̂ = −0.10, λ̃ = 0) both outperform their respective diversity-oriented counterparts across all model scales. Figure 8 shows qualitatively different entropy dynamics for global vs. local precision strategies during RL.

Innovation 4: Identifying the Pre-Trained Distribution Shape as the Bottleneck for RL Reasoning Emergence

The paper provides diagnostic evidence that the shape of the pre-trained token distribution — not just its accuracy — is a primary bottleneck governing whether RL can elicit long chain-of-thought reasoning. This is a new diagnostic concept: the "exploration space quality" of a pre-trained model, measured not by what the model knows (perplexity, benchmark accuracy) but by what reasoning trajectories are reachable from its token-output distribution.

The key evidence for this claim comes from three converging observations:

First, the Pass@k analysis (Figure 9) shows that precision-oriented pre-training yields higher Pass@k on mathematics and coding tasks — meaning that correct solutions are present in the distribution at higher rates — even though the distribution has lower entropy overall. This suggests that precision-oriented priors don't just make the model more confident in the ground truth; they make the correct answer more findable among the model's top candidates, which is precisely what RL needs to discover good reasoning paths through exploration.

Second, the RL entropy and response length dynamics (Figure 8) reveal that models with diversity-oriented pre-training experience entropy collapse and reasoning suppression during early RL, while precision-oriented models show stable, progressive increases in reasoning depth. This diagnostic difference — visible during RL training, not just in final metrics — provides a mechanistic explanation for why the pre-training distribution matters, beyond just correlating initial and final performance.

Third, the consistency of the finding across model scales (1B to 20B), architectures (dense and MoE), and both global and local mechanisms suggests this is not a hyperparameter-specific artifact but a fundamental property of how pre-trained distributions interact with RL optimization on reasoning tasks. The paper's inclusion of both dense and MoE architectures at multiple scales is not just a robustness check — it demonstrates that the phenomenon is architecture-agnostic and scales with model capacity (Figure 3 shows that the advantage of precision-oriented priors grows with model size).

This diagnostic concept — that the pre-trained distribution's shape constrains RL exploration, and that precision provides a better constraint than diversity — is a new way of evaluating pre-trained models that could influence how the field designs and selects base models for reasoning-focused RL. It shifts the evaluation criterion from "how accurate is this model?" to "how good an exploration space does this model provide for RL?" — a distinction that matters precisely because accuracy and exploration-space quality are partially decoupled (as the paper shows).

Evidence tie-in: Figure 9 demonstrates that Pass@k is higher for precision-oriented priors. Figure 8 demonstrates the entropy-collapse diagnostic. Figure 3 and the scaling analysis in Section 3.3 show that precision-oriented advantages grow with model size, supporting the claim that this is a fundamental scaling property, not a small-model artifact.

Innovation 5: The Conceptual Link Between Pre-Training Reward Design and RL Over-Optimization Dynamics

The paper's analysis of RL training dynamics (Figure 8) reveals a previously undocumented phenomenon: the pre-training reward configuration determines the model's susceptibility to entropy collapse during RL, which in turn governs whether long chain-of-thought reasoning emerges or is suppressed. This connects two previously separate research threads — pre-training objective design and RL training stability — through a mechanistic channel (policy entropy dynamics during on-policy optimization).

What makes this a genuine insight rather than a restatement of the performance results is that it identifies entropy collapse as the mechanism linking pre-training configuration to downstream RL failure. The high-entropy pre-trained model (β = 0.50) doesn't just underperform — it exhibits a qualitatively different training trajectory: rapid entropy decline, response length collapse, and subsequent failure to recover reasoning depth. The low-entropy model shows stable, monotonic entropy and steady growth in response length. This diagnostic pattern is visible across all three evaluated model scales (4B, 10B-A0.5B, 20B-A1B) in Figure 8, making it a robust phenomenon.

This finding has implications beyond the paper's specific configuration. It suggests that RL training stability for reasoning is not solely a function of the RL algorithm and hyperparameters — the pre-trained initialization's distribution shape is a critical co-factor. This challenges the common practice of focusing RL stabilization efforts entirely on the RL stage (KL penalties, clipping, learning rate schedules) and suggests that pre-training choices can either predispose a model toward stable RL reasoning emergence or toward catastrophic entropy collapse.

The fact that the "local high-entropy" configuration (λ̃ = 0.1, rewarding top-100 competitors) does not exhibit the same entropy collapse as the "global high-entropy" configuration (β = 0.50) is particularly instructive. It suggests that the dangerous form of pre-training diversity is the one that dilutes the ground-truth signal (global entropy via β), not the one that preserves mass on plausible alternatives (local entropy via λ̃). This distinction — between healthy diversity that maintains alternative reasoning paths and harmful diversity that weakens the signal needed for stable RL optimization — is a conceptual contribution that could guide more nuanced approaches to pre-training objective design.

Evidence tie-in: Figure 8, comparing entropy and response length across configurations at all three model scales. The β = 0.50 panels show the distinctive entropy-collapse-and-response-shortening pattern absent from the β = −0.25 and λ̂ = −0.10 configurations. Tables 19–32 quantify the performance consequences.

5. Experimental Analysis

Evaluation Methodology

Dataset. Pre-training uses a curated corpus of 500B tokens focused on general knowledge, followed by a mid-training corpus of 100B tokens with approximately 5% synthetic data and increased reasoning-oriented content. Both stages deliberately exclude synthetic long-reasoning data. The RLVR stage uses mathematical reasoning tasks: AIME 2024, AIME 2025, AMC23, MATH-500 (Lightman et al., 2023), Minerva (Lewkowycz et al., 2022), and OlympiadBench (He et al., 2024) — all benchmarks where long chain-of-thought reasoning is expected to improve performance. For base model evaluation, 19 benchmarks span five categories: general knowledge (MMLU 4-shot CoT, MMLU-Pro 5-shot CoT, TriviaQA 5-shot, NaturalQuestions 5-shot), commonsense reasoning (HellaSwag 0-shot, SIQA 0-shot, PIQA 0-shot, WinoGrande 0-shot, OpenBookQA 5-shot, CommonsenseQA 5-shot), logic reasoning (ARC-Easy 0-shot, ARC-Challenge 0-shot, BBH 3-shot CoT), mathematics (GSM8K 4-shot CoT, MATH-500 4-shot CoT, Minerva 4-shot CoT, OlympiadBench 0-shot), and coding (HumanEval+ 0-shot, MBPP+ 3-shot).
Base model(s). The paper develops a family of LLMs following Qwen3 (Yang et al., 2025) architectures across five scales: 1B dense (28 layers, 1536 hidden dim), 4B dense (36 layers, 2560 hidden dim), 5B-A0.3B MoE (12 layers, 384 total experts, 12 active), 10B-A0.5B MoE (16 layers, 384 experts, 12 active), and 20B-A1B MoE (24 layers, 384 experts, 12 active). The choice spans both dense and MoE architectures at multiple scales to test whether findings are architecture-agnostic and scale with model capacity. MoE models use an auxiliary-loss-free approach (Liu et al., 2024). The 1B and 5B-A0.3B models undergo only pre-training; the 4B, 10B-A0.5B, and 20B-A1B models complete the full pre-training → mid-training → RLVR pipeline.
Metrics. For pre-training and mid-training stages: PPL (perplexity on held-out validation data) and distribution entropy (measured across the vocabulary at each prediction step) track training dynamics; benchmark-specific accuracy/EM scores and average scores aggregated across capability categories evaluate model performance. For mathematics and coding: Pass@k using the unbiased estimator Pass@k = 1 − binom(m−c, k) / binom(m, k) (Chen, 2021) with m = 128 samples at temperature 0.7 and top-p 0.95, reporting Pass@64 as the upper-bound capability metric. For RLVR stage: Avg@128 (average accuracy across 128 independent samples per problem), Cons@128 (majority voting accuracy — the most common answer among 128 samples), and Pass@64 (same estimator, sampling 128 responses per problem with 16K maximum output length). Additionally, RL training dynamics are monitored through policy entropy (the average entropy of the model's token-output distribution during generation) and response length (average number of tokens generated per problem).
Baselines. The paper's central comparison is between configurations of the generalized objective, with standard cross-entropy (β = 0, λ̃ = 0, λ̂ = 0) serving as the primary baseline. All configurations produce models trained on identical data with identical architectures, optimizers, and schedules — the only variable is the reward function during pre-training. The five tested configurations are: (1) β = −0.25, λ̃ = 0, λ̂ = 0 (global low entropy / precision-oriented), (2) β = 0.50, λ̃ = 0, λ̂ = 0 (global high entropy / diversity-oriented), (3) β = 0, λ̃ = 0, λ̂ = −0.10, k = 100 (tail suppression / precision-oriented local), (4) β = 0, λ̃ = 0.10, λ̂ = 0, k = 100 (head rewarding / diversity-oriented local), and (5) the baseline β = 0, λ̃ = 0, λ̂ = 0 (standard cross-entropy). There is no comparison to label smoothing, focal loss, or other weighted cross-entropy variants as separate baselines — the paper argues these are special cases of the generalized framework but does not implement them as independent experimental conditions.
Generation budget / compute accounting. The paper measures compute through training tokens: 500B for pre-training, 100B for mid-training, and up to 1000 RL steps with batch size 128 and 16 samples per prompt at each step. All model variants consume identical compute budgets at each stage — there is no generation-budget-based comparison (as in the companion paper's N-generations budget), since the research question is about the quality of the initialization, not the amount of inference compute. The fairness argument is that total FLOPs are matched across configurations at each stage, with the only difference being the reward function used during pre-training.
Cross-validation / statistical protocol. The paper does not employ cross-validation, statistical significance testing, or confidence intervals. Results are reported as point estimates from single training runs per configuration. The primary robustness mechanism is multi-scale replication: the key findings (precision-oriented priors outperform diversity-oriented priors, entropy collapse in high-entropy pre-training) are demonstrated across three model scales (4B, 10B-A0.5B, 20B-A1B) and two architectures (dense and MoE). The 1B and 5B-A0.3B models provide additional replication for the pre-training-only stage.

Main Quantitative Results

Pre-Training Dynamics: PPL and Entropy Under Different Reward Configurations

The paper first establishes that all reward configurations produce comparable final pre-training perplexity, confirming that the generalized objective modulates training dynamics without compromising predictive accuracy. As shown in Figures 1 and 2, PPL converges to similarly low values across all configurations for both dense (1B, 4B) and MoE (5B-A0.3B, 10B-A0.5B, 20B-A1B) architectures by 500B tokens. For the 1B dense model, final PPL values cluster around 2.2–2.4 across all configurations; for the 20B-A1B MoE, they cluster around 1.8–2.0. The key claim is that perplexity alone is insufficient to distinguish the configurations — the differences that matter for downstream RL are in the shape of the distribution, not its aggregate quality.

The parameter β functions as a potent global entropy regulator, with clear and consistent effects visible in Figures 1 and 2. Setting β = −0.25 significantly reduces entropy relative to the baseline: across the 1B dense model, final entropy at 500B tokens is approximately 2.2 for β = −0.25 versus approximately 2.6 for β = 0 and approximately 2.8 for β = 0.50. The same ordinal ranking — β = −0.25 produces lowest entropy, β = 0.50 produces highest entropy — holds across all model scales and architectures. This validates the claim from Section 2.3 that negative β amplifies ground-truth rewards, collapsing the distribution, while positive β maintains a flatter, higher-entropy distribution.

The rank-aware negative parameters λ̂ and λ̃ provide finer-grained, local entropy control, as shown in the right panels of Figures 1 and 2. The configuration λ̂ = −0.10, λ̃ = 0, k = 100 (tail suppression) produces entropy values very close to the baseline — within approximately 0.1 nats across all model scales — indicating that penalizing tail tokens reshapes the distribution locally without substantially changing global entropy. The configuration λ̃ = 0.10, λ̂ = 0, k = 100 (head rewarding) produces entropy values slightly above baseline for some models (e.g., approximately 2.6 vs. 2.5 for the 4B dense at 500B tokens) and near-identical for others, suggesting a modest local effect that preserves diversity among high-probability competitors.

A critical observation from Figures 1 and 2 is that all configurations produce stable, convergent training — there is no divergence, no loss spikes, and no training instability associated with any of the reward modifications. The generalized objective is a drop-in replacement for standard cross-entropy that preserves training stability while enabling distributional control.

Pre-Training Performance Scaling: Precision-Oriented Priors Show Superior Growth With Model Size

Figure 3 tracks the evolution of benchmark performance (average scores aggregated across all 19 evaluation benchmarks) for pre-trained models at intermediate checkpoints (125B, 250B, 375B, 500B tokens) and as a function of model scale. The headline finding: configurations that prioritize precision — either globally via β = −0.25 or locally via λ̂ = −0.10, λ̃ = 0 — demonstrate superior performance growth as model size increases, even when they do not show advantages at smaller scales.

For the global entropy comparison (Figure 3, left panels), the 1B dense model shows roughly equivalent performance across all three β configurations at 500B tokens (approximately 31–32 average score, as detailed in Tables 2–3). At the 4B dense scale, β = −0.25 achieves an average score of approximately 43.1 at 500B tokens versus 42.6 for the baseline and 42.4 for β = 0.50 (Table 4 vs. Table 5). For the 20B-A1B MoE, the gap widens: β = −0.25 reaches approximately 50.3 versus 49.8 for the baseline and 50.0 for β = 0.50 (Table 10 vs. Table 11). The key pattern is that the advantage of precision-oriented priors grows with model capacity — while β = −0.25 does not dominate at 1B, it establishes a clear lead by the 20B scale, and critically, it shows a steeper scaling slope between 375B and 500B tokens for the largest models.

For local entropy configurations (Figure 3, right panels), the pattern is similar: λ̂ = −0.10, λ̃ = 0 (tail suppression) matches or slightly exceeds the baseline across scales, with a growing advantage at larger model sizes. At 20B-A1B MoE, tail suppression achieves 50.1 average score at 500B tokens versus 49.8 for the baseline (Table 11). The head-rewarding configuration (λ̃ = 0.10, λ̂ = 0) performs comparably to the baseline across most scales, neither consistently underperforming nor outperforming.

The scaling analysis in the bottom panel of Figure 3 (model parameters vs. average score at 500B tokens) visually demonstrates that the β = −0.25 curve lies above both the β = 0 and β = 0.50 curves for both dense (1B → 4B) and MoE (5B → 10B → 20B) model families, with the gap visibly larger at 20B than at 5B. This supports the paper's claim that precision-oriented strategies "exhibit enhanced growth potential as model size increases" (Section 3.3).

The detailed benchmark-level results are provided in Tables 2–11 (pre-training) and Tables 12–17 (mid-training) in the Appendix, with bold highlighting indicating the best configuration at each checkpoint. At 500B pre-training tokens for the 20B-A1B MoE (Table 10 vs. 11), β = −0.25 achieves the highest scores on 8 of 19 individual benchmarks (including MMLU 41.67, TriviaQA 47.66, PIQA 78.35, WinoGrande 62.90), while baseline β = 0 leads on 5 benchmarks. This granular breakdown confirms that the precision advantage is not driven by a single benchmark category but reflects a broad improvement across knowledge and reasoning tasks.

Mid-Training Performance: Precision-Oriented Priors Maintain and Extend Their Advantage

Figure 4 tracks performance during the mid-training stage (0B → 100B mid-training tokens) on knowledge-average and reasoning-average scores for the 4B dense, 10B-A0.5B MoE, and 20B-A1B MoE models. All models in this stage use standard cross-entropy regardless of their pre-training configuration, so differences reflect persistent effects of the pre-training distribution shape.

The key finding: β = −0.25 consistently achieves the best performance across both knowledge and reasoning tasks at all three model scales. At 100B mid-training tokens for the 4B dense model (Figure 4, top-left panel), β = −0.25 achieves a knowledge average of approximately 41.4 versus 41.3 for the baseline and 41.0 for β = 0.50. The reasoning average gap is more pronounced: β = −0.25 reaches approximately 60.4 versus 60.4 for the baseline and 59.7 for β = 0.50. On the 20B-A1B MoE model, the knowledge and reasoning averages at 100B tokens show β = −0.25 reaching approximately 47.0 and 69.3 respectively, versus 47.2 and 67.8 for the baseline (Table 16 vs. Table 17) — here the advantage is clearer on reasoning (69.3 vs. 67.8, a 1.5 percentage point gap) than on knowledge.

For the local entropy parameters (Figure 4, right panels), the tail-suppression configuration (λ̂ = −0.10, λ̃ = 0) generally matches or slightly surpasses the standard CE baseline. At 100B tokens for the 10B-A0.5B MoE (Table 15), λ̂ = −0.10, λ̃ = 0 achieves reasoning average 57.8 versus 57.7 for the baseline and 57.7 for λ̃ = 0.10, λ̂ = 0 — essentially tied. For the 20B-A1B MoE (Table 17), tail suppression achieves reasoning average 68.0 versus 67.8 for the baseline and 69.0 for head rewarding. The head-rewarding configuration (λ̃ = 0.10, λ̂ = 0) shows "uncertainty in knowledge-intensive scenarios" (Section 3.4) with lower knowledge average at some scales (e.g., 46.8 vs. 47.2 for the 20B-A1B baseline at 100B tokens, Table 17) but comparable reasoning performance.

The mid-training results establish that the precision-oriented advantage observed during pre-training persists through a subsequent training stage using standard cross-entropy, confirming that the effect is durable and not dependent on continued application of the generalized objective.

RLVR Performance: The Central Finding — Precision-Oriented Priors Yield Superior Reasoning

Figures 5–7 and Tables 18–32 present the paper's central results: the RL training trajectories and final performance for models pre-trained with different reward configurations, evaluated on challenging mathematical reasoning benchmarks (AIME 2024, AIME 2025, AMC23, OlympiadBench, MATH-500, Minerva). These are the experiments that directly test the paper's core claim — that the pre-training reward configuration shapes the exploration space for downstream RL, and that precision-oriented priors provide a superior initialization.

Consistency of precision advantage across model scales and metrics. The headline finding is robust across all three model scales and all three evaluation metrics. At 1000 RL steps:

4B Dense Model (Figure 5, Tables 18–22): β = −0.25 achieves Avg@128 of 21.34 vs. 20.49 for baseline (β = 0) vs. 21.37 for β = 0.50; Cons@128 of 28.83 vs. 28.26 vs. 27.87; Pass@64 of 50.43 vs. 50.99 vs. 50.09 (Tables 19, 18, 20). While Avg@128 is comparable between β = −0.25 and β = 0.50, the Cons@128 and Pass@64 metrics favor β = −0.25 (28.83 vs. 27.87 Cons@128; 50.43 vs. 50.09 Pass@64). For local entropy, λ̂ = −0.10, λ̃ = 0 achieves Avg@128 of 21.25 vs. 20.49 baseline and 21.19 for λ̃ = 0.10, λ̂ = 0; Cons@128 of 27.79 vs. 28.26 vs. 28.04 (Tables 21, 18, 22). The tail-suppression configuration slightly trails the baseline on Cons@128 but leads on Avg@128.
10B-A0.5B MoE Model (Figure 6, Tables 23–27): β = −0.25 achieves Avg@128 of 16.18 vs. 14.37 for baseline vs. 14.49 for β = 0.50; Cons@128 of 22.53 vs. 22.43 vs. 22.49; Pass@64 of 50.75 vs. 47.52 vs. 49.59. The advantage is clearest in Avg@128 and Pass@64 — β = −0.25 leads baseline by 1.81 percentage points on Avg@128 and 3.23 points on Pass@64. For local entropy, λ̂ = −0.10, λ̃ = 0 achieves Avg@128 of 16.50 vs. 14.37 baseline vs. 14.74 for λ̃ = 0.10, λ̂ = 0; Cons@128 of 23.82 vs. 22.43 vs. 21.72; Pass@64 of 49.87 vs. 47.52 vs. 49.42. Tail suppression shows the strongest advantage on this model scale, with a 2.13 point Avg@128 lead over baseline.
20B-A1B MoE Model (Figure 7, Tables 28–32): β = −0.25 achieves Avg@128 of 36.06 vs. 34.79 for baseline vs. 32.41 for β = 0.50; Cons@128 of 43.06 vs. 43.73 vs. 42.19; Pass@64 of 67.54 vs. 65.55 vs. 68.03. The gap between β = −0.25 and β = 0.50 is 3.65 points on Avg@128 — the largest separation observed at any scale, confirming that the precision advantage scales with model capacity. For Pass@64, β = 0.50 slightly exceeds β = −0.25, but the average and consensus metrics consistently favor precision. For local entropy, λ̂ = −0.10, λ̃ = 0 achieves Avg@128 of 36.02 vs. 34.79 baseline vs. 34.13 for λ̃ = 0.10, λ̂ = 0; Cons@128 of 45.48 vs. 43.73 vs. 41.96 (Tables 31, 28, 32).

The performance trajectories reveal qualitative differences. Figures 5–7 show not just endpoint comparisons but entire training trajectories (every 100 steps from 100 to 1000). The β = −0.25 curves consistently lie above the β = 0.50 curves throughout training for Avg@128 and Cons@128 across all three model scales. The gap is not just present at convergence — it emerges early (within the first 200–300 RL steps) and is maintained or widens throughout training. This suggests that the precision-oriented pre-training initialization facilitates more effective learning from the very beginning of RL, not just a better final convergence point.

At the individual benchmark level, the precision advantage is most pronounced on the most challenging reasoning tasks. On AIME 2024 at step 1000 for the 20B-A1B MoE, β = −0.25 achieves Avg@128 of 13.38 vs. 12.14 for β = 0.50 (Tables 29 vs. 30). On AIME 2025, β = −0.25 achieves 52.32 vs. 11.25 for β = 0.50 — a dramatic difference of 41.07 percentage points on this highly challenging benchmark. (This exceptionally large gap for AIME 2025 on the 20B model is the most striking single-result in the paper, though the paper does not comment on it explicitly — it appears in Tables 29–30).

RL Training Dynamics: Entropy Collapse Explains the Performance Gap

Figure 8 provides the mechanistic explanation for the performance differences observed in Figures 5–7. By tracking policy entropy and response length throughout RL training for all configurations and model scales, the paper reveals qualitatively different training trajectories between precision-oriented and diversity-oriented priors.

For the global entropy comparison (Figure 8, left panels), the β = 0.50 (high entropy) configuration exhibits a distinctive failure mode: rapid entropy collapse during early RL training, accompanied by drastic response length reduction, followed by partial recovery. On the 4B dense model, β = 0.50 entropy drops from approximately 0.9 at step 0 to below 0.5 by step 100, while β = −0.25 entropy starts near 0.65 and declines smoothly to approximately 0.45 by step 1000. Simultaneously, β = 0.50 response length plunges from approximately 11,000 at step 0 to below 2,000 by step 100, then partially recovers to approximately 8,000–10,000 by step 1000 — but remains approximately 2,000 tokens shorter than β = −0.25 throughout. On the 10B-A0.5B MoE model, the pattern is even starker: β = 0.50 entropy drops from approximately 1.8 to below 0.4 by step 200, with response length collapsing from approximately 10,000 to near zero before slowly recovering. On the 20B-A1B MoE, β = 0.50 entropy starts near 1.1, drops to approximately 0.4 by step 200, while β = −0.25 starts near 0.65 and stabilizes near 0.5 — a much gentler decline.

In contrast, the β = −0.25 configuration shows stable, monotonic entropy trajectories with gradual decline and continuous, steady increases in response length — the hallmark of progressive activation of long chain-of-thought reasoning. On the 4B dense model, β = −0.25 response length increases from approximately 4,000 at step 0 to approximately 12,000 at step 1000. On the 10B-A0.5B MoE, it grows from approximately 5,000 to 10,000. On the 20B-A1B MoE, from approximately 3,500 to 7,500.

For the local entropy comparison (Figure 8, right panels), the configurations show more nuanced behavior. The tail-suppression configuration (λ̂ = −0.10, λ̃ = 0) maintains entropy trajectories very close to the baseline, with slightly higher entropy at later training steps on the 10B model. Response lengths for λ̂ = −0.10, λ̃ = 0 track the baseline closely or slightly exceed it. The head-rewarding configuration (λ̃ = 0.10, λ̂ = 0) shows intermediate behavior — neither the severe collapse of β = 0.50 nor the complete stability of β = −0.25 — with entropy declining somewhat faster than baseline but not catastrophically. This suggests that the entropy-collapse phenomenon is specifically linked to global entropy manipulation via β, while local entropy shaping via λ̂ and λ̃ provides more targeted distributional control without destabilizing RL dynamics.

The paper interprets these dynamics as the causal mechanism linking pre-training configuration to RL performance: the high-entropy pre-trained distribution creates a model that is fragile under RL optimization pressure — when RL begins reinforcing certain patterns, the model's initially flat distribution collapses too aggressively, losing the diversity it was supposed to preserve and suppressing the reasoning depth that requires maintained entropy at forking tokens. The precision-oriented prior, by front-loading confidence in ground-truth tokens, provides a stable foundation that doesn't need to collapse during RL, allowing progressive exploration and reasoning-length growth.

Pass@k Analysis: Precision-Oriented Priors Yield Higher Upper-Bound Capability

Figure 9 examines Pass@k curves for mathematics and coding tasks on base models (after mid-training, before RL) for the 4B dense, 10B-A0.5B MoE, and 20B-A1B MoE models. The key insight: precision-oriented priors yield higher Pass@k scores, particularly at large k, even though they produce lower-entropy distributions overall.

For the 4B dense model (Figure 9, top row), β = −0.25 achieves Pass@64 of approximately 60 on mathematics and approximately 70 on coding, versus approximately 57 and 65 for β = 0.50 and similar for baseline. The gap is most visible at k ≥ 16, suggesting that the precision-oriented distribution makes correct solutions more reachable when many samples are drawn — exactly the regime relevant for RL exploration. For local entropy, λ̂ = −0.10, λ̃ = 0 yields slightly higher Pass@k than baseline across most k on mathematics, while λ̃ = 0.10, λ̂ = 0 is comparable.

For the 10B-A0.5B MoE model (Figure 9, middle row), the pattern is less pronounced but still present: β = −0.25 and baseline are very close, with β = 0.50 slightly trailing on coding Pass@k. For local entropy, λ̂ = −0.10, λ̃ = 0 matches or slightly exceeds baseline on mathematics Pass@k.

For the 20B-A1B MoE model (Figure 9, bottom row), β = −0.25 achieves Pass@64 of approximately 70 on mathematics and approximately 85 on coding, versus approximately 65 and 78 for β = 0.50. The gap is substantial at all k ≥ 4 and widens with k.

The paper's interpretation (Section 3.6) is that "maximizing global diversity (high entropy) does not inherently yield higher Pass@k curves" and that "superior Pass@k scores in mathematics and coding tasks are achieved by prioritizing precision." Crucially, the paper argues that "this low-entropy setting does not lead to a collapse in output diversity" — meaning that the precision-oriented prior maintains sufficient variation to cover the solution space (high Pass@k) while concentrating probability mass more effectively than a flat distribution (resulting in higher per-sample accuracy in RL).

Ablation Studies and Robustness Checks

Effect of β across model scales: The paper implicitly ablates model scale by testing the same β values across five model sizes. As shown in Figure 3 (bottom panels), the advantage of β = −0.25 over β = 0 and β = 0.50 grows with model scale for both dense and MoE architectures. For the 1B dense model, the three configurations are nearly indistinguishable (average scores within ~1 point of each other at 500B tokens). By 20B-A1B MoE, β = −0.25 leads baseline by ~0.5 points and β = 0.50 by ~0.3 points on the pre-training average score — and the gap widens further during RL (Tables 28–30). This scaling trend is critical because it suggests the effect is not a small-model artifact but a fundamental property of how distribution shape interacts with model capacity and RL optimization. If the trend continues to larger scales, the precision advantage might be even more pronounced at the 70B–100B+ scales typical of production reasoning models.

Effect of λ̂ and λ̃ across model scales: The local entropy parameters show a more nuanced scaling pattern. Tail suppression (λ̂ = −0.10, λ̃ = 0) performs comparably to baseline at small scales (1B dense: Tables 2–3) and establishes a growing advantage at larger scales during RL (10B-A0.5B: Avg@128 16.50 vs. 14.37 baseline, Table 26 vs. 23; 20B-A1B: Avg@128 36.02 vs. 34.79 baseline, Table 31 vs. 28). Head rewarding (λ̃ = 0.10, λ̂ = 0) shows inconsistent scaling — it performs comparably to baseline on the 20B model during RL (Table 32 vs. 28: Avg@128 34.13 vs. 34.79) but trails notably on the 10B model at some steps. This asymmetry (tail suppression scales favorably; head rewarding does not) is a non-obvious finding: penalizing low-probability noise tokens scales better with model capacity than rewarding high-probability competitors.

Robustness across architectures: The paper tests both dense (1B, 4B) and MoE (5B-A0.3B, 10B-A0.5B, 20B-A1B) architectures with different configurations (layers, hidden dimensions, expert counts). The consistent pattern of β = −0.25 outperforming β = 0.50 across both architectures (Figures 5–7) demonstrates that the precision advantage is architecture-agnostic. The MoE models, which use an auxiliary-loss-free training approach (Liu et al., 2024), show the same qualitative patterns as dense models, suggesting the effect is not driven by expert routing dynamics or auxiliary loss interactions.

Robustness across training stages: The paper demonstrates that the precision advantage established during pre-training persists through mid-training with standard cross-entropy (Figure 4) and is maintained or amplified during RLVR (Figures 5–7). This three-stage persistence is critical: it rules out the possibility that the advantage is simply an artifact of the pre-training objective that would be washed out by subsequent training. The fact that mid-training — using standard cross-entropy on 100B additional tokens — does not eliminate the performance gap between configurations suggests that the distribution shape established during pre-training creates a basin of attraction that subsequent optimization does not easily escape.

Mid-training as an implicit ablation of continued reward shaping: Since mid-training uses standard cross-entropy for all models (Section 3.1), this stage serves as an ablation of whether continued application of the generalized objective is necessary for the benefit. The finding that precision-oriented pre-trained models maintain their advantage through 100B tokens of standard cross-entropy mid-training (Figure 4, Tables 12–17) demonstrates that the benefit is due to the initial distribution created during pre-training, not due to continued reward shaping.

Interaction between global and local entropy control: While the paper does not report full combinatorial sweeps of β, λ̂, and λ̃, the separate testing of global-only and local-only configurations provides indirect evidence about their independence. The fact that both β = −0.25 (global low entropy) and λ̂ = −0.10, λ̃ = 0 (local tail suppression) outperform the baseline — but through different RL dynamics (Figure 8: β = −0.25 shows lower entropy throughout RL; λ̂ = −0.10 maintains higher entropy than baseline) — suggests the two mechanisms are complementary rather than redundant. An open question not addressed by the paper is whether combining β = −0.25 with λ̂ = −0.10, λ̃ = 0 would yield further improvements, or whether there are diminishing returns or negative interactions.

Negative result: β = 0.50 (global high entropy) underperforms consistently: The paper frames this as a counterintuitive finding that challenges conventional wisdom. The consistency of the underperformance — across all model scales, architectures, metrics, and training stages — makes this a robust negative result. Critically, the underperformance is not due to poor pre-training convergence (Figures 1–2 show comparable PPL) but due to poor RL dynamics (Figure 8 shows entropy collapse and response length suppression). This negative result is arguably more informative than the positive result for β = −0.25, because it identifies a specific failure mode (entropy collapse during early RL) that can guide future research.

Negative result: Head rewarding (λ̃ = 0.10, λ̂ = 0) does not provide consistent benefits: Across model scales and evaluation stages, rewarding high-probability competitors to the ground truth (λ̃ = 0.10, λ̂ = 0, k = 100) produces mixed results — sometimes matching baseline, sometimes slightly underperforming, and showing "uncertainty in knowledge-intensive scenarios" (Section 3.4). This is notable because rewarding plausible alternatives might have been expected to preserve exploration diversity for RL. The fact that it does not help — and that penalizing tail tokens (λ̂ = −0.10) does help — suggests that the harmful tokens for RL exploration are not the strong competitors but the long-tail noise tokens. The paper does not deeply analyze why this is the case, leaving it as an open question.

Pass@k as a robustness check on the "diversity not eliminated" claim: The paper claims that low-entropy pre-training does not reduce output diversity in a way that harms solution coverage. Figure 9 provides the key evidence: β = −0.25 achieves higher Pass@k than β = 0.50 on mathematics and coding, meaning that correct solutions are present in the top-k samples at higher rates despite lower overall entropy. This is a counterintuitive robustness check: the precision-oriented distribution is both more peaked (lower entropy) and has better solution coverage (higher Pass@k), suggesting that probability mass is being shifted from noise tokens to correct-alternative tokens, improving the signal-to-noise ratio of the distribution without sacrificing diversity where it matters.

Response length as a diagnostic for reasoning activation: Figure 8 tracks response length throughout RL training as an implicit measure of whether the model is activating long chain-of-thought reasoning. Across all model scales, the configurations that achieve higher final reasoning scores (β = −0.25, λ̂ = −0.10) also show higher and more stable response lengths. The β = 0.50 configuration shows the distinctive pattern of rapid length collapse followed by partial recovery — but it never fully catches up to the β = −0.25 response length, suggesting that early RL training is a critical window where distribution shape determines whether reasoning patterns are reinforced or suppressed.

Critical Assessment

How Well Do the Experiments Support the Central Claims?

Claim: "The cross-entropy loss can be interpreted as a specific instance of policy gradient optimization." This is a theoretical claim, not an experimental one, and the derivation in Equations 6–10 is mathematically sound. However, the experiments do not directly test whether this interpretation is correct in any empirical sense — they test whether a generalization of cross-entropy based on this interpretation produces useful models. The derivation is a framing device that motivates the generalized objective, but the experimental results would be equally valid under a different theoretical motivation (e.g., "we modified the loss function and it worked better"). The strength of this claim rests on the derivation's internal logic, not on experimental validation.

Claim: "A precision-oriented pre-training prior provides a more effective initialization for RL than high-entropy distributions, leading to improved reasoning capabilities." This is the paper's central empirical claim, and it is well-supported by the evidence in Figures 5–7 and Tables 18–32. The β = −0.25 configuration consistently achieves higher Avg@128, Cons@128, and Pass@64 than β = 0.50 across all three model scales that complete the full pipeline, with the gap growing with model size. The claim is supported across six mathematical reasoning benchmarks and three complementary metrics (average accuracy, majority voting accuracy, and upper-bound capability).

However, the claim should be qualified in several important ways. First, the advantage is configuration-specific, not universal. While β = −0.25 (global low entropy) and λ̂ = −0.10, λ̃ = 0 (tail suppression) both outperform the baseline, the head-rewarding configuration (λ̃ = 0.10, λ̂ = 0) does not consistently outperform baseline and sometimes underperforms it. The paper's title asks "Diversity or Precision?" — but the answer is more nuanced than "precision beats diversity." It is specifically global precision via ground-truth amplification and local precision via tail-token suppression that help; local diversity via preserving high-probability competitors does not. The paper does not deeply analyze why some precision-oriented strategies work and others don't, which limits the generalizability of the "precision is better" conclusion.

Second, the advantage is most visible on the most challenging benchmarks (AIME, OlympiadBench) and less pronounced on MATH-500, where all configurations achieve relatively similar high performance (e.g., at 20B-A1B, MATH-500 Avg@128 is 75.42 for β = −0.25 vs. 72.67 for baseline vs. 68.51 for β = 0.50 — a 6.9 point gap, Tables 29, 28, 30). This suggests the precision advantage is most relevant for tasks that require extended reasoning chains, consistent with the paper's mechanism (precision enables stable entropy during RL, allowing long reasoning to emerge).

Third, the claim is demonstrated for mathematical reasoning only. The RL stage is exclusively mathematical tasks. Whether the precision advantage generalizes to coding RL, dialogue RL, or other RLVR domains is untested.

Claim: "Contrary to the conventional intuition that higher distribution entropy facilitates effective exploration, we find that imposing a precision-oriented prior yields a superior exploration space for RL." This is the most provocative framing, and the evidence is supportive but incomplete. The paper demonstrates that low-entropy pre-training leads to better RL outcomes, but it does not directly measure "exploration space" as an independent construct. The claim about exploration is inferred from three observations: (1) higher Pass@k for precision-oriented priors (Figure 9) suggests more reachable correct solutions; (2) more stable entropy dynamics during RL (Figure 8) suggest healthier exploration; and (3) better final RL performance (Figures 5–7) is the downstream consequence. However, the paper does not directly measure exploration diversity during RL (e.g., the number of distinct reasoning paths explored, the coverage of the solution space, or the rate at which novel correct solutions are discovered). The claim that the exploration space is "superior" is a causal interpretation of the performance results, not a directly tested hypothesis.

A stronger test would involve explicitly measuring exploration: for instance, tracking how many unique correct solutions are discovered across RL training, measuring the entropy at forking tokens specifically (rather than average token entropy), or analyzing whether precision-oriented priors lead to faster discovery of correct reasoning patterns early in RL training. The entropy curves in Figure 8 are aggregate metrics that don't isolate the behavior at the critical few forking tokens that the paper's motivation (Section 1) highlights as the locus of reasoning exploration.

Claim: "The parameter β serves as a potent global entropy regulator, while λ̂ and λ̃ facilitate local entropy fine-tuning." The evidence in Figures 1–2 strongly supports that β controls global entropy in the claimed direction (negative β reduces entropy, positive β increases it). The evidence for λ̂ and λ̃ as local entropy controls is partially supported: λ̂ = −0.10 produces entropy values very close to baseline (within ~0.1 nats), and the paper argues this reshapes the distribution locally without large global changes, but this claim is primarily based on the aggregate entropy metric rather than a detailed analysis of the token-rank distribution. A stronger demonstration would show, for instance, that probability mass moves from tails to head while the entropy of the top-k remains unchanged — but no such distributional analysis is provided.

Strengths of the Experimental Design

Multi-scale, multi-architecture replication. The paper's strongest experimental feature is that the same pattern — precision-oriented priors outperform diversity-oriented priors during RL — is demonstrated across three model scales (4B, 10B-A0.5B, 20B-A1B) and two architectures (dense and MoE) in the full pipeline, with additional replication at 1B and 5B-A0.3B for pre-training only. This level of replication is unusual in LLM training papers, where single-scale experiments are common due to computational cost. It substantially strengthens the robustness of the findings and supports the claim that the effect scales with model capacity.

Complete pipeline evaluation. By running the entire pre-training → mid-training → RLVR pipeline for each configuration, the paper avoids a common pitfall in pre-training studies: measuring only pre-training metrics and assuming they predict downstream performance. The finding that configurations with comparable pre-training perplexity produce substantially different RL performance (Figures 5–7) directly validates this design choice — if the paper had stopped at pre-training metrics, it would have concluded (incorrectly) that the configurations are essentially equivalent.

Diagnostic tracking of RL dynamics. The entropy and response length trajectories in Figure 8 are the paper's most valuable contribution beyond the headline performance numbers. They provide a mechanistic window into why the performance differences occur, moving beyond correlation to a plausible causal account. The entropy-collapse pattern for β = 0.50 is a specific, testable diagnostic that future work can use to evaluate pre-training configurations without running full RL to convergence.

Genuine Weaknesses and Missing Evidence

No statistical quantification of uncertainty. All results are reported as point estimates from single training runs per configuration (Tables 2–32). There are no error bars, confidence intervals, or multiple seeds. Given the inherent stochasticity in LLM pre-training (data ordering, dropout, initialization) and RL training (sampling temperature, reward noise), it is unclear whether the observed differences are statistically reliable, particularly the smaller gaps (e.g., the 0.5–1.0 percentage point advantage for β = −0.25 over baseline on some metrics). The replication across model scales provides qualitative robustness but not statistical rigor. For the RL stage, where 16 samples per prompt are drawn at temperature 1.0, there is substantial variance that could affect the Avg@128 and Cons@128 estimates — particularly for the most challenging benchmarks (AIME) where absolute accuracy is low (1–15%) and small sample counts of correct answers introduce high variance.

Missing hyperparameter sweeps. The paper tests exactly one value for each non-zero hyperparameter setting: β = −0.25, β = 0.50, λ̂ = −0.10, λ̃ = 0.10, k = 100. There is no exploration of intermediate values, more extreme values, or interaction effects between parameters. Critical questions are left unanswered: Is β = −0.25 near-optimal, or would β = −0.50 or β = −0.10 perform even better? Does the optimal β depend on model scale? Is the penalty λ̂ = −0.10 well-calibrated, or would a larger penalty (e.g., λ̂ = −0.50) help more? Would combining β = −0.25 with λ̂ = −0.10 yield additive benefits or saturate? The paper's title frames a binary choice ("Diversity or Precision?"), but the hyperparameter space is continuous, and the paper does not characterize the shape of the performance landscape within that space.

No comparison to label smoothing or focal loss as implemented baselines. The paper argues (Section 4.1) that label smoothing and focal loss are special cases of the generalized objective, but it does not include them as experimental baselines. A direct comparison showing that the proposed β and λ̂ configurations outperform label smoothing (a standard technique many practitioners already use) would strengthen the claim that the generalized objective provides benefits beyond existing methods. The absence of these baselines makes it difficult to assess whether the paper's contribution is the specific hyperparameter values or the framework itself.

Limited analysis of why certain precision strategies work. The paper demonstrates that β = −0.25 (global low entropy) and λ̂ = −0.10 (tail suppression) improve RL outcomes, but does not deeply analyze what these configurations do to the token distribution that makes RL more effective. Key missing analyses include: (1) How does the rank distribution of token probabilities change under each configuration? (2) Are the beneficial effects concentrated at specific token types (e.g., mathematical operators, logical connectives, numerical values)? (3) Does tail suppression primarily affect tokens that are genuinely noise, or does it inadvertently suppress rare-but-important tokens? (4) How do the differences in pre-training distribution manifest in the RL training data — do precision-oriented models produce higher-quality initial rollouts that create a better starting point for GRPO updates? Without this analysis, the paper's contribution is primarily empirical (this works) rather than scientific (this is why it works).

Single domain for RL evaluation. All RL experiments use mathematical reasoning tasks. The paper motivates this choice by noting that mathematics is where long chain-of-thought reasoning typically emerges, but it also means the findings may not generalize to other RLVR domains (code generation, tool use, dialogue). Mathematical reasoning has specific properties — deterministic correctness, well-defined reasoning steps, symbolic manipulation — that may interact with the pre-training distribution shape in ways that differ from other domains. A model pre-trained with precision-oriented priors might excel at math RL but underperform on creative writing RL, where high entropy and diversity might actually be beneficial.

No exploration of the pre-training data distribution's role. All experiments use the same 500B-token general knowledge corpus. The paper does not investigate whether the optimal β and λ̂ values depend on the pre-training data distribution — for instance, whether a code-heavy pre-training corpus would benefit from different reward shaping than a text-heavy corpus. This limits the generalizability of the specific hyperparameter recommendations.

The paper does not address the interaction with the RL algorithm choice. All RL experiments use on-policy GRPO without KL regularization. It is possible that the precision advantage is specific to this algorithm — for instance, a KL-regularized RL algorithm might handle high-entropy pre-trained models better by preventing the rapid entropy collapse observed in Figure 8 with β = 0.50. The paper does not ablate the RL algorithm or test whether the pre-training configuration ranking reverses under different RL setups.

Evaluation of "exploration space" is indirect. The paper's central narrative is about shaping the exploration space for RL, but the primary evidence for this claim is final RL performance and aggregate entropy dynamics. The paper does not directly characterize the exploration space — for instance, by measuring the diversity of generated reasoning paths, the coverage of correct solutions, the rate of discovering novel correct answers during RL, or the entropy specifically at forking tokens. The entropy metric in Figure 8 is averaged over all tokens and all contexts, which may obscure the behavior at the critical few decision points that the paper's own motivation highlights. This is a significant gap between the paper's conceptual framing and its empirical measurements.

6. Limitations and Trade-offs

6.1 Single RL Domain: All Findings Are Restricted to Mathematical Reasoning

The assumption or constraint. The paper's entire RL evaluation — the stage where the central claim about precision-oriented priors providing a better exploration space is tested — uses exclusively mathematical reasoning benchmarks (AIME 2024/2025, AMC23, MATH-500, Minerva, OlympiadBench). The authors motivate this by noting that "the emergence of long-reasoning capabilities is typically associated with these domains" (Section 3.1), but this choice means the paper's primary result is demonstrated for a single task family.

The consequence. Mathematical reasoning has specific structural properties — deterministic correctness, well-defined intermediate steps, symbolic manipulation, and highly skewed token distributions where key tokens (operators, variable names, numerical values) carry disproportionate importance — that may interact with the precision-oriented pre-training strategy in ways that do not generalize. A precision-oriented prior might be beneficial specifically because mathematical reasoning benefits from concentrated probability on a narrow set of operationally meaningful tokens, suppressing the syntactically valid but semantically irrelevant alternatives that a flat distribution would preserve. For domains where diversity is genuinely valuable — creative writing, open-ended dialogue, brainstorming, or code generation where multiple valid implementations exist — the same precision-oriented strategy might suppress useful alternatives and degrade performance. The paper provides no evidence either way.

What evidence exists in the paper. The pre-training and mid-training evaluation (Section 3.2, Figures 3–4, Tables 2–17) covers 19 benchmarks spanning general knowledge, commonsense reasoning, logic reasoning, mathematics, and coding, and shows that precision-oriented priors produce modest advantages on most categories. The Pass@k analysis (Figure 9) includes both mathematics and coding and shows precision benefits for both. However, these are base-model evaluations (pre-RL), and the paper's headline claim concerns RL exploration. There is no RL evaluation on coding, no RL evaluation on any non-reasoning task, and no discussion of whether the mechanism identified (entropy stability during RL) would manifest differently in non-mathematical domains.

Mitigation status. The paper does not acknowledge this as a limitation. It treats mathematical reasoning as the natural testbed for long chain-of-thought emergence without discussing whether the findings might be domain-specific. No future work is suggested for evaluating the approach on other RLVR domains.

6.2 No Statistical Quantification of Uncertainty: Single-Seed Results Without Confidence Intervals

The assumption or constraint. All experimental results in Tables 2–32 and Figures 1–9 are reported as point estimates from single training runs per configuration. There are no error bars, confidence intervals, standard deviations, or multiple training seeds for any metric at any stage (pre-training, mid-training, or RL). The paper's robustness argument relies entirely on replication across model scales rather than statistical rigor within each scale.

The consequence. LLM training at this scale (500B tokens, 1000 RL steps) involves substantial stochasticity from multiple sources — data ordering and batching during pre-training, random initialization, dropout, and particularly during RL where 16 samples per prompt are drawn at temperature 1.0. For the hardest benchmarks (AIME 2024/2025) where absolute accuracy at the 20B scale is 1–36% Avg@128, the number of correct samples per problem is small, and variance across runs could be high. Several of the paper's key comparisons involve gaps of 0.5–2.0 percentage points on aggregate metrics — for instance, at 4B dense RL step 1000, β = −0.25 achieves Cons@128 of 28.83 vs. 28.26 for baseline (Table 19 vs. 18), a difference of 0.57 percentage points. Without confidence intervals, it is impossible to assess whether gaps of this magnitude are statistically reliable or within the range of run-to-run variance. The dramatic AIME 2025 gap for the 20B model — β = −0.25 achieves Avg@128 of 52.32 vs. 11.25 for β = 0.50 (Tables 29 vs. 30, a 41.07 point gap) — is large enough to likely be robust, but the paper does not provide the statistical evidence that would confirm this.

What evidence exists in the paper. The paper provides no uncertainty quantification whatsoever. The multi-scale replication (consistent direction of effect across 4B, 10B-A0.5B, and 20B-A1B models) provides qualitative evidence that the precision-advantage pattern is not a single-seed fluke, but it cannot quantify the magnitude of uncertainty around any specific result. The detailed tables (18–32) report metrics at every 100 RL steps for each configuration, and the trajectories are relatively smooth (Figures 5–7), suggesting stable training — but smoothness of a single trajectory does not guarantee that re-running with a different seed would produce the same trajectory within the observed gap.

Mitigation status. The paper does not discuss this limitation. No future work is proposed to address it, and no explanation is offered for why multiple seeds were not used. The computational cost of multiple full-pipeline runs at this scale is a reasonable practical constraint, but this should be acknowledged transparently, and readers should be cautioned about the uncertainty in the point estimates — particularly for the smaller performance gaps that the paper uses to claim advantages for certain configurations.

6.3 Minimal Hyperparameter Exploration: A Single Value Tested Per Non-Zero Configuration

The assumption or constraint. The paper tests exactly one non-zero value for each hyperparameter in the generalized reward function: β = −0.25 (precision-oriented), β = 0.50 (diversity-oriented), λ̂ = −0.10 (tail suppression), λ̃ = 0.10 (head rewarding), and k = 100 (fixed). There is no sweep over intermediate values, more extreme values, or interaction effects between parameters. The paper's empirical claim that "precision is better than diversity" is supported only by the contrast between β = −0.25 and β = 0.50 — but these are two points in a continuous space.

The consequence. The paper cannot distinguish between several interpretations of its results. Is β = −0.25 near-optimal, or would β = −0.10 (weaker precision) perform better by providing some of the stability benefit without aggressive concentration? Would β = −0.50 (stronger precision) be even better, or would it overshoot into detrimental over-concentration? Does the optimal β depend on model scale — the paper's scaling analysis (Figure 3) shows the advantage growing with model size, but without a sweep it's unknown whether larger models benefit from more aggressive β or whether the same β = −0.25 remains appropriate. Similarly for λ̂: is −0.10 an appropriately calibrated penalty, or would −0.50 suppress important rare tokens? Does k = 100 provide a meaningful boundary between head and tail, or would k = 50 or k = 500 change the effect? And crucially: would combining β = −0.25 with λ̂ = −0.10 yield additive benefits, or do they operate through overlapping mechanisms such that combining them saturates? The paper's title frames a binary choice, but the hyperparameter space is high-dimensional and continuous, and the paper has characterized only three points in it (plus the origin).

What evidence exists in the paper. The experimental design is explicitly a comparison of specific configurations against a baseline, not a hyperparameter optimization study. Section 3.3 provides a rationale for the chosen values — they "allow us to isolate the specific effects of positive and negative reward signals" — but does not justify why these specific magnitudes were selected or discuss whether they are intended to be near-optimal or merely illustrative. The consistent direction of effect (both precision-oriented configurations outperform their diversity-oriented counterparts) provides some robustness, but without sweep data it is unknown whether the paper has identified the best point in the design space or simply two points on the correct side of the baseline.

Mitigation status. The paper does not discuss the lack of hyperparameter sweeps as a limitation. No future work is proposed to characterize the performance landscape more thoroughly. The claim that "imposing a precision-oriented prior yields a superior exploration space for RL" (Section 1, abstract) is presented as a qualitative finding about direction rather than a quantitative recommendation about magnitude, but the paper does not make this distinction clear, and readers may over-interpret the specific hyperparameter values (β = −0.25, λ̂ = −0.10) as "the" recommended settings.

6.4 Mid-Training and RL Use Standard Objectives: Only Pre-Training Benefits From the Generalized Objective

The assumption or constraint. The paper's generalized reward function is applied only during the 500B-token pre-training stage. Mid-training (100B tokens) uses standard cross-entropy for all models regardless of pre-training configuration (Section 3.1: "we perform mid-training on an additional 100B tokens, gradually decaying the learning rate"). The RLVR stage uses on-policy GRPO with a standard reward signal (verifiable correctness on mathematical problems), not the generalized reward function. This means the paper studies a one-time distributional intervention at the pre-training stage, not a sustained reward-shaping strategy throughout training.

The consequence. The paper's findings establish that the pre-training distribution shape has persistent downstream effects — which is a significant and non-obvious result — but they leave open the question of whether continued application of the generalized objective during mid-training or RL would provide additional benefits. Since mid-training uses 100B tokens of standard cross-entropy, this stage likely partially dilutes the distributional shaping from pre-training. The fact that precision advantages survive this dilution (Figure 4) is evidence of durability, but it also means the observed RL performance gaps may be lower bounds — a model that maintained precision-oriented reward shaping throughout all training stages might show even larger advantages. Conversely, if the primary mechanism is the initial distribution establishing a favorable basin of attraction for subsequent optimization, then continued shaping might be unnecessary or even counterproductive (analogous to how early-stage learning rate annealing matters more than late-stage). The paper cannot adjudicate between these possibilities.

What evidence exists in the paper. Section 3.1 explicitly states the design choice: mid-training uses standard cross-entropy and decaying learning rate; RL uses GRPO without KL regularization. The mid-training results (Figure 4, Tables 12–17) provide indirect evidence about the durability of the pre-training effect: the precision advantage is present but partially attenuated by 100B tokens of standard cross-entropy (e.g., at 20B-A1B after mid-training, Table 16, β = −0.25 reasoning average 69.28 vs. 67.76 for baseline — a gap of 1.52 points, smaller than the ~3.65-point Avg@128 gap observed after RL in Tables 29 vs. 28). However, no ablation tests whether applying the generalized objective during mid-training would preserve or amplify the advantage.

Mitigation status. The paper does not discuss this as a limitation. The design choice to apply the generalized objective only during pre-training cleanly isolates the effect the paper wants to study (initial distribution shape), which is methodologically sound, but the paper does not frame the resulting scope limitation — that we only know about one-shot pre-training interventions, not about sustained reward shaping — or discuss whether future work should explore continued application.

6.5 The Relationship Between Pre-Training Entropy Control and RL Exploration Is Inferred, Not Directly Measured

The assumption or constraint. The paper's central narrative is about "exploration space": that the pre-trained distribution "defines the model's behavioral trajectory and implicitly constrains its exploration space" (Section 1), and that precision-oriented priors "provide a more favorable exploration space for RL" (Section 3.5). However, the paper does not directly measure exploration during RL. The evidence for the exploration-space claim is indirect: (1) higher Pass@k for precision-oriented priors (Figure 9) suggests correct solutions are more reachable; (2) more stable entropy dynamics during RL (Figure 8) suggest healthier optimization; and (3) better final RL performance (Figures 5–7) is the downstream consequence.

The consequence. The causal chain the paper proposes — precision-oriented pre-training → better exploration space → more stable RL entropy dynamics → emergence of long chain-of-thought reasoning → higher benchmark accuracy — is plausible and internally consistent, but several links are untested. The paper does not measure exploration diversity during RL: how many distinct reasoning paths the model samples, how quickly it discovers correct solutions, whether it explores and abandons incorrect strategies, or the coverage of the solution space across RL training. The entropy metric in Figure 8 is averaged over all tokens and all contexts, which obscures the behavior at the critical forking tokens that the paper's own motivation (Section 1, citing Wang et al., 2025; Zhu et al., 2025b) identifies as the locus of reasoning exploration. The paper does not show, for instance, that β = −0.25 leads to higher entropy specifically at forking tokens while lowering entropy at non-critical tokens — which would directly support the exploration-space narrative. Instead, the entropy metric conflates these regimes.

Additionally, the Pass@k evidence (Figure 9) shows that precision-oriented priors have higher upper-bound capability, but Pass@k is a static property of the pre-trained distribution, not a dynamic measure of exploration during RL. A model might have high Pass@k (correct solutions exist in the distribution) but poor exploration (RL fails to discover those solutions because the distribution is too narrow around certain answers). The paper does not disentangle "better exploration" from "better starting point for exploitation."

What evidence exists in the paper. Figure 8 provides aggregate entropy trajectories; Figure 9 provides static Pass@k curves; Figures 5–7 provide final performance. None of these directly measure exploration behavior. The paper does not report metrics such as the number of unique correct solutions discovered during RL, the entropy at tokens annotated as "forking tokens," the rate of solution-space coverage over training, or any analysis of which tokens experience entropy changes under different pre-training configurations. The paper's claim about exploration space is a causal interpretation of the performance results, not a directly tested hypothesis with dedicated measurements.

Mitigation status. The paper does not acknowledge this gap between its conceptual framing and its empirical measurements. Section 3.5 interprets the RL performance differences as evidence about exploration space quality ("strategies that promote precision... enables the model to converge to higher-quality solutions, potentially providing a better exploration space for RL"). The word "potentially" flags the inferential nature of the claim, but this hedge is easily missed. The paper suggests no future work to directly measure exploration diversity or forking-token behavior.

6.6 No Direct Evidence on Computational Overhead or Training Stability Risks of the Generalized Objective

The assumption or constraint. The generalized reward function modifies the per-token reward computation during pre-training by adding the positive scaling factor (1 − πθ)^β and the rank-aware negative rewards λ̃ and λ̂. The paper reports that all configurations produce stable, convergent training (Figures 1–2), but it does not analyze the computational overhead of the generalized objective relative to standard cross-entropy, nor does it explore potential failure modes (e.g., whether more extreme hyperparameter values would cause training instability, gradient variance spikes, or numerical issues). The paper also does not discuss whether the rank-aware negative rewards require computing the top-k token set at each training step, which could add non-trivial overhead at large vocabulary sizes (typically 32K–256K tokens for modern LLMs).

The consequence. For practitioners considering adopting the generalized objective, the absence of overhead analysis creates uncertainty. The top-k operation, if computed exactly, requires sorting the model's full output distribution at every token position — for a batch size of 16M tokens, this could mean sorting 16M distributions of size 32K+ at each optimizer step. The paper does not specify whether an efficient approximation is used (e.g., approximate top-k via sampling or partial sorting) or whether the overhead is negligible relative to the transformer forward-backward pass. Without this information, practitioners cannot assess the cost-benefit trade-off of the method — if the generalized objective adds, say, 5% training overhead for a 1–3 percentage point RL improvement, the cost-effectiveness might or might not be favorable depending on deployment context. The paper also does not explore the sensitivity of training stability to the magnitude of the reward modifications: Figures 1–2 show stable training for the tested values, but would β = −1.0 cause gradient explosion? Would λ̂ = −1.0 cause numerical underflow in the softmax? Without stability boundaries, practitioners lack guidance on safe hyperparameter ranges.

What evidence exists in the paper. Section 3.3 and Figures 1–2 demonstrate that all tested configurations converge stably with comparable final PPL. This is evidence that the method is stable at the tested hyperparameter values, but does not characterize the stable region of the hyperparameter space. The paper provides no wall-clock time measurements, no FLOP counts comparing the generalized objective to standard cross-entropy, and no discussion of the computational cost of the top-k operation or the positive reward scaling factor computation. The training details in Section A specify batch size, learning rate, and optimizer settings but nothing about the per-step cost of the modified objective.

Mitigation status. The paper does not acknowledge this limitation. The focus is entirely on model quality outcomes, not on training efficiency or stability boundaries. No future work is suggested to characterize the computational overhead or stable hyperparameter ranges. Given the paper's practical framing — it aims to influence how pre-training objectives are designed — the absence of any cost or stability analysis is a notable gap for practitioners evaluating whether to adopt the method.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper changes the landscape by introducing a new axis of optimization — the pre-training reward function — that the field had previously treated as a fixed constant. Before this work, the dominant paradigm treated pre-training and RL as sequential but independent stages: pre-train with standard cross-entropy to maximize predictive accuracy, then apply RL as a separate process with its own optimization logic. The pre-training loss function was not a variable to be optimized; it was the unquestioned default. This paper demonstrates that this default is not just arbitrary — it is suboptimal for downstream RL on reasoning tasks, and that the space of possible pre-training reward configurations contains points that yield substantially better end-to-end performance.

The magnitude of this contribution is a conceptual reframing rather than a paradigm shift. The paper does not propose a new architecture, a new RL algorithm, or a new training pipeline. It keeps every component of the standard recipe intact and changes only the reward structure during pre-training. What makes this significant is that the field had not previously conceptualized the pre-training objective as a degree of freedom for optimizing downstream RL. The paper's formal derivation (Section 2.2) showing that cross-entropy implicitly encodes a specific, unexamined reward structure — zero reward for all incorrect tokens, inverse-probability reward for the correct one — makes this reframing concrete: if pre-training is already doing policy gradient optimization, then its reward function should be designed with the same intentionality used in downstream RL.

This reframing resolves a nagging tension that has existed in the literature but was rarely articulated explicitly. On one hand, the standard recipe says "train the best next-token predictor you can, then apply RL." On the other hand, practitioners have observed that pre-training perplexity and downstream RL performance are imperfectly correlated — two models with identical validation loss can exhibit different RL trajectories. The paper provides a mechanistic explanation for this gap: perplexity measures aggregate predictive accuracy but is insensitive to the shape of the token-output distribution, and that shape — particularly how probability mass is allocated among negative tokens and how aggressively it concentrates on the ground truth — governs the stability of RL optimization and the emergence of long chain-of-thought reasoning.

The paper also redirects research attention in specific ways. It makes improving pre-training objectives — not just scaling data and parameters — a first-class research direction for reasoning-focused LLM development. Prior scaling laws work (Hoffmann et al., 2022; Kaplan et al., 2020) treated the loss function as fixed and optimized the allocation of compute between model size and data quantity. This paper's results suggest that changing the loss function can provide efficiency gains orthogonal to scaling — a precision-oriented pre-training objective at a given model scale can outperform a diversity-oriented objective at the same scale, without increasing compute. This opens a new dimension in the "bitter lesson" debate: architectural and algorithmic innovations can matter, even at scale, when they target the interface between training stages rather than the model architecture itself.

It makes entropy dynamics during RL a diagnostic of first-order importance. The paper's most striking diagnostic finding — that pre-training entropy configuration determines whether RL experiences smooth entropy decline with reasoning growth or catastrophic entropy collapse with reasoning suppression (Figure 8) — establishes entropy trajectory as a key indicator of RL health. This suggests that future RL for reasoning should not only optimize for reward but should monitor and potentially regularize the entropy path, and that the pre-training stage should be designed with this entropy path in mind. It also provides a concrete lens for diagnosing why some pre-trained models benefit from RL while others degrade: the answer may lie in how the pre-trained distribution responds to on-policy optimization pressure.

It challenges the widespread intuition that higher diversity facilitates exploration. The paper's most provocative finding — that β = −0.25 (low entropy) consistently outperforms β = 0.50 (high entropy) — is not just a surprising empirical result but a conceptual challenge to a deeply held belief in the field. The standard story from both the pre-training literature (where label smoothing and entropy regularization are used to encourage diversity) and the RL literature (where exploration bonuses and stochastic policies prevent premature convergence) is that diversity is beneficial. This paper shows that, at least for the specific setting of pre-training for mathematical reasoning RL, the opposite is true: imposing a precision-oriented prior — aggressively concentrating probability on ground-truth tokens and suppressing tail noise — creates a more effective starting point for RL. This finding does not invalidate the value of diversity in all contexts, but it does establish that diversity must be carefully structured rather than indiscriminately maximized — the useful diversity is concentrated in the head of the distribution (plausible alternatives to the ground truth), while the tail represents noise that interferes with stable RL optimization. This is a more nuanced picture of the diversity-exploration relationship than the field has typically operated with.

Follow-Up Research This Work Enables

Characterizing the full hyperparameter landscape of the generalized objective. The paper tests exactly one non-zero value for each parameter (β = −0.25, β = 0.50, λ̂ = −0.10, λ̃ = 0.10, k = 100). The natural next step is a systematic sweep over a wider range: β ∈ {−0.50, −0.25, −0.10, 0, 0.10, 0.25, 0.50}, λ̂ ∈ {−0.50, −0.25, −0.10, −0.05, 0}, and k ∈ {10, 50, 100, 500, 1000}, plus combinatorial combinations of β and λ̂ to test for additive or interactive effects. The key questions are: Is there a monotonic relationship between β and RL performance (more negative β always better until some point of diminishing returns), or is there an optimal intermediate value? Does the optimal β depend on model scale — the paper's scaling analysis (Figure 3) suggests the precision advantage grows with scale, but does this mean larger models benefit from more aggressive β, or does the same β = −0.25 remain near-optimal? Does combining β = −0.25 with λ̂ = −0.10 yield additive benefits (since they operate through different mechanisms — global concentration vs. tail cleanup), or do they saturate? A strong follow-up would reproduce the full pipeline at one model scale (e.g., 4B dense, since it's the cheapest), sweep the hyperparameter grid, and produce a contour plot of downstream RL performance as a function of β and λ̂. This would transform the paper's qualitative finding ("precision is better") into a quantitative engineering tool.

Direct measurement of exploration diversity during RL as a function of pre-training configuration. The paper's central narrative is that precision-oriented pre-training provides a better exploration space for RL, but this claim is inferred from final performance and aggregate entropy dynamics rather than directly measured. A critical follow-up would instrument the RL training process to explicitly quantify exploration: track the number of unique correct solutions discovered per problem over training steps, measure the entropy specifically at forking tokens (identified by the method from Wang et al., 2025, or Zhu et al., 2025b — tokens where the distribution has multiple high-probability alternatives — separately from the entropy at low-uncertainty tokens), compute coverage metrics (what fraction of the space of possible correct reasoning paths is explored by step N), and analyze whether precision-oriented pre-training leads to faster discovery of correct solutions or better exploitation of discovered solutions. This would distinguish between competing mechanistic explanations: does β = −0.25 work because it makes correct solutions more reachable in the initial distribution (a static property), or because it creates more stable RL dynamics that allow progressive exploration (a dynamic property), or both? A strong follow-up would replicate the 4B dense model experiments with the three β configurations, add these exploration metrics, and produce a panel analogous to Figure 8 but with forking-token entropy and solution-space coverage as additional diagnostic traces.

Testing the generalization of the precision advantage to non-mathematical RLVR domains. The paper's RL evaluation is restricted to mathematical reasoning. The most important generalization test is whether the same precision-oriented priors benefit RL for code generation (where correctness is verifiable via unit tests, and the reasoning structure is similar — step-by-step algorithmic thinking) and for other reasoning domains where verifiable rewards exist (formal theorem proving, chemistry, physics problem-solving). A strong follow-up would replicate the pipeline with the same model architectures and pre-training configurations but apply RLVR on coding benchmarks (HumanEval+, MBPP+, LiveCodeBench) using execution-based rewards. The prediction from the paper's mechanism would be that mathematical reasoning and coding benefit similarly, since both involve structured multi-step reasoning where entropy stability at critical decision points matters. If the precision advantage fails to transfer to coding, that would suggest the mechanism is specific to mathematical token distributions (e.g., the unique vocabulary of mathematical notation) rather than a general property of reasoning. Additionally, testing on a domain where diversity is genuinely valuable — creative writing with human preference feedback, dialogue generation, or open-ended brainstorming — would establish the boundary conditions of the paper's recommendation. If precision-oriented priors harm performance in diversity-valued domains, that would refine the paper's message from "precision is better" to "precision is better for reasoning, diversity may be better for generation," which is a practically important distinction.

Investigating the interaction between pre-training reward configuration and RL algorithm choice. All RL experiments use on-policy GRPO without KL regularization. The entropy collapse observed for β = 0.50 (Figure 8) raises the question of whether this failure mode is specific to unregularized on-policy RL. KL-regularized RL algorithms (TRPO, PPO with KL penalty, or the KL-constrained variants used in RLHF) explicitly penalize large policy updates and might prevent the catastrophic entropy collapse seen in the β = 0.50 trajectories. If KL regularization rescues the high-entropy pre-trained model, then the ranking of pre-training configurations might reverse — diversity-oriented pre-training combined with KL-regularized RL might outperform precision-oriented pre-training. Conversely, if KL regularization helps all configurations equally, the precision advantage might persist. A strong follow-up would replicate the 4B dense RL experiments with three RL algorithms: GRPO without KL (baseline, as in the paper), GRPO with KL penalty to the pre-trained policy, and PPO with clipping. The key measurement is whether the performance gap between β = −0.25 and β = 0.50 narrows or reverses under KL-regularized RL. This would answer whether the paper's central finding is specific to unregularized GRPO or reflects a more fundamental property of how pre-trained distributions interact with policy optimization.

Pre-training data distribution as a moderator of the optimal reward configuration. All experiments use the same 500B-token general knowledge corpus. The composition of pre-training data — the proportion of code, math, natural language, multilingual text — likely interacts with the optimal β and λ̂ settings. A code-heavy pre-training corpus might have different token distribution statistics (more repetitive patterns, more deterministic next-token relationships) than a natural language corpus, potentially changing the benefit of precision-oriented priors. A strong follow-up would pre-train 1B dense models on curated corpora with varying proportions of code vs. natural language (e.g., 10/90, 50/50, 90/10 splits), each under three β configurations (−0.25, 0, 0.50), and measure both pre-training PPL and downstream RL performance on math and coding tasks. If the optimal β is different for code-heavy vs. text-heavy corpora, this would provide practical guidance for practitioners whose pre-training data mixtures differ from the paper's. It would also begin to characterize the pre-training objective as a function of the data distribution, moving from a universal recommendation to a context-dependent one.

Stress-testing the precision advantage at larger scales and with more extreme hyperparameters. The paper demonstrates the precision advantage up to 20B-A1B MoE parameters and 500B pre-training tokens. A natural stress test is: does the advantage hold at the 70B–100B+ scale that characterizes production reasoning models (DeepSeek-R1, Kimi K2)? And does it hold with more extreme hyperparameter values? The paper's β = −0.25 is a relatively mild modification — would β = −1.0 provide even stronger benefits at scale, or would it cause over-concentration that damages the model's ability to represent rare but important patterns? Conversely, would the β = 0.50 configuration — which performed worst in this paper — actually become competitive at much larger scales, where the model's greater capacity might better handle the flat distribution? A strong follow-up would train a single large model (70B+ parameters) on a smaller pre-training corpus (e.g., 100B tokens, to manage cost) with β values spanning a wider range, then evaluate RL performance. This stress test is important because the paper's scaling analysis (Figure 3) shows the precision advantage growing with model size, but only over the 1B–20B range — extrapolating this trend to 70B+ assumes monotonicity that may not hold.

Practical Applications and Downstream Use Cases

Reasoning-focused LLM training pipelines should treat the pre-training objective as a tunable hyperparameter rather than a fixed default. This is the most direct practical implication. Teams building reasoning models via the pre-training → RLVR pipeline should not assume that standard cross-entropy is the optimal pre-training objective. Instead, they should experiment with β values in the −0.50 to −0.10 range and tail-suppression parameters (λ̂ in the −0.20 to −0.05 range) on a smaller-scale proxy model (e.g., 1B–4B parameters) before committing to the full-scale pre-training run. The paper shows that at 20B-A1B MoE, β = −0.25 yields a 3.65 percentage point improvement in Avg@128 over β = 0.50 (36.06 vs. 32.41, Tables 29–30), and a 1.27 point improvement over the standard cross-entropy baseline (36.06 vs. 34.79, Tables 29 vs. 28). At production scales (100B+ parameters), if the scaling trend observed in Figure 3 continues, the absolute improvement in downstream reasoning accuracy could be worth several percentage points — a substantial gain for a modification that requires no additional compute, no architectural changes, and no pipeline restructuring, only a different reward computation during pre-training.

Pre-training checkpoint selection for RL should incorporate distributional diagnostics, not just validation perplexity. The paper shows that models with comparable perplexity can have substantially different RL performance. This means that validation loss — the standard criterion for checkpoint selection during pre-training and for deciding when to stop training — is insufficient for models destined for RL-based reasoning fine-tuning. A practical recommendation: during pre-training, periodically measure not just PPL but also the entropy of the token-output distribution (as in Figure 1), the Pass@k on held-out math and coding problems (analogous to Figure 9), and — if computational budget allows — a small-scale RL probe (e.g., 100 steps of RLVR on a subset of math problems) to assess how the current checkpoint responds to on-policy optimization. The entropy trajectory provides an early warning signal: if entropy is declining too rapidly or too slowly relative to the configurations that produced good RL outcomes in this paper, the pre-training hyperparameters might need adjustment. This adds a new dimension to the checkpoint selection problem that currently focuses exclusively on downstream benchmark accuracy.

On-device or edge deployment of reasoning models may benefit from precision-oriented pre-training to maximize per-parameter capability. The paper's FLOPs-matched trade-off analysis (which is implicit in its scaling comparisons) suggests that a smaller model pre-trained with precision-oriented priors can approach the RL reasoning performance of a larger model pre-trained with standard cross-entropy. At the 10B-A0.5B MoE scale with λ̂ = −0.10 (tail suppression), Avg@128 is 16.50 (Table 26) vs. 14.37 for the baseline (Table 23) — a ~15% relative improvement in reasoning accuracy from changing only the pre-training reward function. For resource-constrained deployments (on-device models, edge inference), where every parameter counts and additional inference-time compute is limited, this improvement is effectively free — it requires no additional model parameters, no additional inference FLOPs, and no change to the deployment infrastructure. The generalized objective simply produces a better-initialized model from the same pre-training compute budget. This makes the method particularly attractive for the growing ecosystem of small, specialized reasoning models targeting mobile and embedded deployment.

Self-improvement and iterative training pipelines should incorporate reward-shaped pre-training for the base model in each iteration. If an organization is running an iterative self-improvement loop — use the current model to generate training data, train a new model on that data, repeat — the paper's findings suggest that the base model for each iteration should be pre-trained with precision-oriented priors, not standard cross-entropy. The mechanism: each iteration of self-improvement involves a form of RL or RL-like optimization (filtering, re-ranking, or fine-tuning on model-generated solutions). If the base model for an iteration has a precision-oriented distribution, it may produce higher-quality initial solutions (due to higher Pass@k, as shown in Figure 9) and experience more stable optimization during the self-improvement step. This could improve the efficiency of the entire self-improvement loop — more useful data generated per iteration, faster convergence per iteration — beyond what would be achieved by simply scaling up the base model.

When to Prefer This Method

The paper does not articulate an explicit trade-off against named alternative pre-training objectives or position its method as preferable under specific, enumerated conditions. The generalized objective is proposed as a framework that subsumes standard cross-entropy, label smoothing, and focal loss as special cases, with the specific hyperparameter configurations tested (β = −0.25, λ̂ = −0.10) shown to outperform the standard cross-entropy baseline for downstream RL on mathematical reasoning. There is no experimental comparison against label smoothing, focal loss, or other weighted loss variants as implemented baselines, and the paper does not provide decision rules for when these alternatives might be preferable. The paper's practical recommendation is implicit: for the specific setting of pre-training models destined for RLVR on mathematical reasoning, precision-oriented reward configurations (β < 0 and/or λ̂ < 0) outperform diversity-oriented configurations (β > 0 and/or λ̃ > 0). The paper does not extend this recommendation to non-reasoning domains, non-RL downstream tasks, or other RL algorithms. No conditional decision matrix is warranted based on the paper's explicit claims and experimental scope.