RLP: Reinforcement as a Pretraining Objective

🎯 Pitch

Pretraining models without any ground-truth reasoning data, just by rewarding chain-of-thought that improves next-token prediction, already boosts a 1.7B model’s math-and-science average by 19% before any post-training. When scaled to a 12B hybrid model, this verifier-free dense reward raises accuracy from 43% to 61% using less than 0.13% of the data a compute-matched continuous pretraining would need.

1. Executive Summary

This paper introduces Reinforcement Learning Pre-training (RLP), a verifier-free pretraining objective that rewards chain-of-thought generation based on the information gain it provides for predicting future tokens — the increase in log-likelihood of the next token when conditioning on a sampled reasoning chain versus a no-think EMA baseline. Evaluated on QWEN3-1.7B-BASE across an eight-benchmark math-and-science suite, RLP lifts the overall average by 19% over the base model and by 17% over a compute-matched continuous pretraining baseline, with gains compounding after identical post-training to yield a 7–8% relative advantage. When scaled to the hybrid NEMOTRON-NANO-12B-V2, RLP improves the overall average from 42.81% to 61.32% — a 43% relative gain — while using only 0.125% of the data consumed by the continuously pretrained baseline, establishing that dense, verifier-free reinforcement signals can systematically induce reasoning capabilities during pretraining across architectures and model sizes.

2. Context and Motivation

The Core Problem: Pretraining and Reasoning Are Decoupled

The fundamental gap this paper identifies is a structural mismatch in how modern LLMs are trained versus how they need to operate. The dominant training paradigm follows a strict sequential order: first, pretraining with next-token prediction loss on vast amounts of text; then, much later — often as the very last stage — reinforcement learning is introduced during post-training to induce reasoning capabilities. The paper frames this as a question:

"While dominant, is this an optimal way of training?"

The answer the paper suggests is no, and the reasoning is grounded in how human cognition differs from token-by-token prediction. Human comprehension is characterized as "a parallel integration of input with prior knowledge" — we bring world knowledge and reasoning to bear while processing information, not as a separate post-hoc step. Current pretraining lacks any mechanism that explicitly encourages the model to pause, think, and integrate knowledge before predicting the next word. As the paper states:

"Current pretraining lacks such mechanisms, limiting the model's ability to reason and ground language in world knowledge during learning."

This creates a situation where state-of-the-art reasoning models (Guo et al., 2025; Yang et al., 2025) must rely entirely on post-training pipelines — supervised fine-tuning followed by reinforcement learning with human/verified feedback (RLHF) or verifiable rewards (RLVR) — to acquire capabilities that, the paper argues, should be cultivated earlier. The consequence is that the foundational pretraining phase, which consumes the vast majority of training compute and shapes the model's core representations, operates under an objective (next-token prediction) that is implicitly at odds with reasoning: it rewards statistical pattern matching, not deliberate inference.

Why This Gap Matters

The decoupling has several practical and theoretical implications that motivate this work:

1. Post-training is expensive and narrow. Methods like RLVR require task-specific verifiers, carefully curated datasets of solvable problems, or human feedback. These are costly to construct, limited in domain coverage, and fundamentally cannot scale to the breadth of data that pretraining consumes. The paper notes that such approaches are "often tied to the complexity of queries, limiting their scalability" (Appendix 12). If reasoning could be induced during pretraining on ordinary text, the reliance on expensive post-training pipelines would decrease.

2. Pretraining compute is underutilized. The pretraining phase represents the largest investment of computational resources in an LLM's lifecycle — typically trillions of tokens processed over weeks or months on thousands of GPUs. If this phase produces only a statistical next-token predictor with no explicit reasoning infrastructure, then a significant fraction of that compute is, in a sense, not being directed toward capabilities that ultimately matter for downstream tasks. The paper's perspective is that pretraining should actively shape the model's internal reasoning behavior, not just build a foundation that later stages must retrofit.

3. The reasoning gap compounds at scale. State-of-the-art models demonstrate that reasoning capabilities emerge more strongly with scale, but they also reveal that such capabilities require explicit training interventions (chain-of-thought prompting, RL with verifiers) to manifest. The paper implies that architectural scaling alone, under standard next-token objectives, leaves reasoning capacity latent rather than active — a kind of capability debt that must be paid off later.

4. Self-improvement loops are bottlenecked. If models can reason during pretraining — that is, generate internal thoughts that improve their own predictions — this opens the door to a virtuous cycle where the model's growing predictive ability feeds back into better reasoning, which in turn improves predictions further. Current pretraining has no mechanism for this kind of feedback because there is no reasoning channel to reinforce.

Where Prior Approaches Fall Short

The paper identifies several lines of prior work and explains why each is insufficient:

Standard next-token prediction pretraining. This is the dominant paradigm (Shannon, 1951; Bengio et al., 2003; Vaswani et al., 2017; Radford et al., 2018; Brown et al., 2020) and has produced all major LLMs. Its limitation is that it provides only an implicit, weak reasoning signal. The model learns to anticipate tokens from context, which can incidentally require reasoning (e.g., predicting the next step in a mathematical derivation), but the objective itself provides no incentive to explicitly think through intermediate steps. The model may learn to pattern-match the surface form of reasoning without developing robust internal deliberative processes.

Post-training RL with verifiable rewards (RLVR). Methods like DeepSeek-R1 (Guo et al., 2025) and related open-source efforts (Hugging Face, 2025; Hu et al., 2025; Luo et al., 2025; Liu et al., 2025a) apply RL after pretraining and SFT, using verifiable rewards (e.g., checking final answers against ground truth, compiling code and running tests). The paper identifies three key limitations of this paradigm for its purposes:

Domain restriction: These methods require tasks where correctness is mechanically verifiable — math problems with exact answers, code with unit tests. They cannot be applied to arbitrary text because most next-token predictions on general web text do not have verifiable ground-truth labels.
Late intervention: By the time RL is applied, the model has already spent enormous compute on next-token prediction without any reinforcement for reasoning. The representations and behaviors shaped during pretraining must be retrofitted rather than natively grown.
Scalability concerns: Verifiable reward datasets are small (thousands to millions of examples) compared to pretraining corpora (trillions of tokens). The reasoning patterns learned may not generalize beyond the narrow distribution of training problems.

Reinforcement Pre-training (RPT) (Dong et al., 2025). This is the most directly comparable prior work — indeed, the paper positions RLP explicitly in contrast to RPT. RPT applies RL to pretraining by rewarding correct next-token prediction after a CoT prefix, but the paper identifies critical weaknesses:

Sparse, binary rewards: RPT uses a binary signal — correct or incorrect next-token prediction — that provides no gradient for how much a thought helps. A thought that dramatically increases confidence in the correct token receives the same reward as one that barely pushes it over a threshold. This loses information about the quality of reasoning.
Token pre-selection via auxiliary models: RPT relies on an external "assistant" model to identify high-entropy tokens where thinking might help, and only applies RL at those positions. This (a) adds computational overhead during training, (b) couples the method to a separate model, and (c) limits where reasoning can be reinforced. The paper notes that "prior RPT demonstrations also depend on distilled checkpoints with strong prior reasoning ability, which clouds whether the method helps base models" (Section 1).
Limited scope: Because rewards are tied to binary correctness, RPT cannot be applied to tokens where correctness is ambiguous or continuous-valued — which is most tokens in natural language. The paper characterizes this as "ignoring the CoT content" in favor of a simplistic terminal signal.

Verifier-free post-training methods. Recent work explores removing the need for external verifiers in post-training RL. Yuan et al. (2024) uses iterative DPO where the model judges its own candidates; Liu et al. (2025b) trains with incentive RL on SFT corpora; Zhao et al. (2025) uses the model's own confidence as a reward. The paper acknowledges these but notes they are fundamentally post-training methods — they operate after SFT on curated reasoning data, not during pretraining on general text. As the paper states:

"Because these methods target post-training policies, direct comparisons are not well-posed" (Section 6).

Continuous pretraining on reasoning data. A pragmatic alternative is to simply continue pretraining the base model on high-quality reasoning corpora (e.g., math problems with solutions, scientific papers). Recent work (Wang et al., 2025; Gandhi et al., 2025) shows this can boost reasoning benchmarks. The paper treats this as its strongest baseline (denoted $M_{\text{CPT}}$ in tables) but demonstrates that even with compute-matched or data-matched budgets, RLP systematically outperforms continuous pretraining — by 17% on average for QWEN3-1.7B-BASE (Table 1) and more dramatically at scale (Table 2). The key point is that data quality alone cannot substitute for an objective that explicitly rewards the act of thinking.

How RLP Positions Itself

The paper frames RLP as occupying a novel position in the training pipeline: the last phase of pretraining, before any SFT or RLVR. This is deliberate. RLP does not replace the massive initial pretraining that builds core language capabilities from raw text. Instead, it is introduced at a point where the model already has substantial linguistic competence but has not yet been explicitly taught to reason. The paper conceptualizes this as:

"RLP reframes reinforcement learning for reasoning as a pretraining objective on ordinary text, bridging the gap between next-token prediction and the emergence of useful chain-of-thought reasoning" (Section 1).

This positioning enables several design choices that distinguish RLP:

Verifier-free by construction. Because the reward is the information gain — the log-likelihood ratio between the reasoned predictor and a no-think EMA baseline — it requires no external verifier, no ground-truth labels, and no task-specific checkers. The reward is computed entirely from the model's own outputs and the observed data. This makes RLP applicable to any text, including web crawl, academic papers, textbooks, and open-ended prose — precisely the kind of data used in pretraining.

Dense, position-wise signal. Unlike RPT's sparse binary reward at selected positions, RLP provides a continuous scalar reward at every position where a thought is sampled. The magnitude of the reward directly measures how much the thought improved next-token prediction, providing fine-grained credit assignment. This density enables efficient learning — the paper shows that RLP applied to only 170M tokens can outperform continuous pretraining on 6B tokens (a 35× data advantage for the baseline, Table 4).

Unified handling of reasoning and prediction. RLP uses a single network for both generating thoughts and predicting next tokens given those thoughts. There is no separate verifier model, value function, or reward model to train. The EMA baseline is a lagged copy of the same network, not a distinct architecture. This keeps the training loop simple — Algorithm 1 is a standard policy gradient loop with group-relative advantages — and avoids the complexity of multi-model RL systems.

Theoretical grounding in information theory. The paper provides two propositions (Section 2.3, Appendix 8) that connect the RLP objective to fundamental quantities:

Proposition 1: The expected reward equals the reduction in cross-entropy between the reasoned predictor and the no-think baseline. Maximizing the information-gain reward is equivalent to minimizing the expected predictive error attributable to the absence of reasoning.
Proposition 2: The CoT-conditioned objective is a lower bound on the log-likelihood improvement achievable by marginalizing over thoughts. This means that even though the model optimizes a single sampled thought, it is pushing toward a state where the expected output over many thoughts would be a better predictor. The bound is tight when the thought is deterministic or the predictor ignores it (the degenerate case).

These propositions provide interpretability: the reward is not an arbitrary RL signal but has a direct interpretation in terms of predictive improvement. They also provide tractability: maximizing the lower bound is computationally feasible because it involves sampling a single thought rather than marginalizing over all possible thoughts.

Empirical philosophy: test on base models, not distilled checkpoints. A recurring concern the paper raises about prior work (particularly RPT) is that evaluations on models already distilled from strong reasoning systems can obscure whether the method actually induces reasoning de novo. RLP is evaluated on base models — QWEN3-1.7B-BASE, which is a standard pretrained model with no prior RL for reasoning, and NEMOTRON-NANO-12B-V2, a hybrid architecture trained from scratch on diverse data. The paper emphasizes that RLP is "designed to shape thinking in base models by rewarding only those thoughts that measurably help next-token prediction" (Section 1), not to amplify pre-existing reasoning capabilities inherited from a teacher.

Relationship to the broader RL-for-reasoning landscape. The paper situates RLP within a growing recognition that RL is essential for advanced reasoning, but challenges the assumption that it must come at the end of training. By moving the RL signal earlier — into pretraining — RLP aims to make reasoning a native capability of the base model rather than a bolted-on augmentation. The paper's vision, articulated most clearly in the conclusion, is that:

"reinforcement pretraining [is] a principled and general alternative to likelihood-only training" (Section 7).

This is a significant departure from the prevailing "pretrain, then SFT, then RL" pipeline, and the paper's empirical results — particularly the persistence and compounding of gains after strong post-training (Tables 1 and 2) — are the primary evidence marshaled to support this vision.

` — this appears to be intentional formatting), and instructs the model to generate "the next few steps" rather than a complete solution. This keeps thoughts focused on local prediction improvement rather than long-range planning.

Why this prompt design: several elements are deliberate:

"Continue the text in the SAME style" ensures the thought is relevant to the prefix and maintains document coherence. This is important because RLP operates on diverse text — from math problems to web crawl to academic papers — and the thought must be stylistically appropriate to provide useful predictive information.
"Focusing on the next few steps" prevents the model from wasting thought budget on distant predictions that provide little immediate information. The reward is computed only for the next token $x_t$ , so thoughts should target information relevant to that specific prediction.
"Do not restate the question or add meta commentary" discourages boilerplate thinking that would not improve prediction. Meta-commentary ("I think the answer is...") doesn't help predict the actual next token of the document.

The prompt is applied via the system message in a chat-style interface, and the model generates thoughts autoregressively within the < /think> tags.

Completion Length and the Thought Budget

The length of the generated thought $|c_t|$ is a critical hyperparameter that directly controls how much computation the model spends on reasoning. The paper ablates completion lengths from 64 to 4096 tokens (Figure 2b, Appendix Table S.1):

Short thoughts underperform dramatically. At length 64, overall accuracy is only 11.50% (vs. 34.03% for the base model). This is worse than not thinking at all — thoughts of insufficient length actually degrade performance, likely because they provide misleading partial information.

The main jump occurs between 512 and 1024 tokens. At length 512, overall accuracy is 24.65%; at 1024, it jumps to 38.89% — a +14.24 point increase. Individual benchmarks show dramatic improvement: GSM8K goes from 46.58% to 75.13% (+28.55); MATH500 goes from 21.35% to 58.20% (+36.85).

Diminishing returns beyond 2048. At length 2048, overall is 42.17% (+3.28 over 1024). At 4096, overall is 42.21% (+0.04) — essentially no improvement while using roughly twice the thought budget (and thus twice the generation time). The paper defaults to 2048 as the preferred tradeoff.

Why length matters so much: the thought budget determines how much information the model can incorporate into its reasoning before predicting. Very short thoughts (64–256 tokens) are insufficient for the model to work through the context and identify what information is relevant. The regime change between 512 and 1024 tokens likely reflects a threshold where thoughts become long enough to engage in actual multi-step inference rather than surface-level pattern matching. Beyond 2048, the marginal benefit of additional thought tokens is near zero — the model has already extracted most of the available predictive information from the context.

KL Divergence Anchoring: Why It Doesn't Help

The paper investigates adding a token-level KL divergence penalty toward a fixed reference distribution — a common technique in RLHF to prevent the policy from diverging too far from the pretrained model. The ablation sweeps $\beta \in \{10^{-4}, 10^{-3}, 0\}$ (Figure 2c, Appendix Table S.1):

$\beta = 10^{-4}$ : Overall 41.35% (−0.82 vs. $\beta = 0$ )
$\beta = 10^{-3}$ : Overall 41.44% (−0.73 vs. $\beta = 0$ )
$\beta = 0$ : Overall 42.17%

What it computes: the KL penalty would add a term $\beta \cdot \text{KL}(\pi_\theta \| \pi_{\text{ref}})$ to the loss, penalizing the policy for deviating from some reference distribution.

Why it doesn't help: the paper attributes this to the self-stabilizing nature of the EMA baseline. In standard RLHF, KL anchoring is necessary because the reward model is an imperfect proxy for human preferences, and policies can exploit it through degenerate outputs. In RLP, the reward is directly tied to predictive improvement — there is no separate reward model to hack. The EMA baseline provides sufficient regularization by making rewards relative rather than absolute.

Additionally, the paper notes that KL anchoring "increases memory and step time" — it requires storing a reference model in memory and computing additional forward passes, adding overhead without benefit.

Algorithm 1: The Complete Training Loop

Algorithm 1 in the paper formalizes the entire procedure. Here is a step-by-step walkthrough of what happens in each training iteration:

Step 1: Snapshot the behavior policy. $\theta_{\text{old}} \leftarrow \theta$ — save the current parameters before sampling thoughts. These are the parameters that will be used to generate the rollouts, and they serve as the reference for computing importance ratios.

Step 2: Sample a minibatch. Draw $B$ examples $\{(x_{<t}^{(b)}, x_t^{(b)})\}_{b=1}^B$ from the dataset. Each example consists of a document prefix and the target next token at a randomly selected position.

Step 3: Generate thoughts. For each example $b$ , sample $G$ thoughts $c_t^{(b,i)} \sim \pi_{\theta_{\text{old}}}(\cdot \mid x_{<t}^{(b)})$ with each thought having at least one token. Thoughts are generated autoregressively at temperature 0.7 (specified in Appendix 10).

Step 4: Lazy initialization of EMA teacher. On the first batch, set $\phi \leftarrow \theta$ to initialize the EMA baseline.

Step 5: Compute baseline log-evidence. Using the EMA teacher, compute $S_{\text{EMA}}^{(b)} = \log \bar{p}_\phi(x_t^{(b)} \mid x_{<t}^{(b)})$ under teacher forcing (the ground-truth token is provided). No gradient is propagated through this computation.

Step 6: Compute reasoned log-evidence and rewards. For each thought $(b,i)$ , compute $S_{\text{pred}}^{(b,i)} = \log p_{\theta}(x_t^{(b)} \mid x_{<t}^{(b)}, c_t^{(b,i)})$ . The reward is $r^{(b,i)} = S_{\text{pred}}^{(b,i)} - S_{\text{EMA}}^{(b)}$ .

Step 7: Compute group-relative advantages. For each example $b$ , compute the group mean $\bar{r}^{(b)} = \frac{1}{G}\sum_j r^{(b,j)}$ and the corrected advantages $A^{(b,i)} = \frac{G}{G-1}(r^{(b,i)} - \bar{r}^{(b)})$ . The mean $\bar{r}^{(b)}$ is treated as a constant (stop-gradient).

Step 8: Compute importance ratios and clipped surrogate loss. For each thought token $u$ in each thought $(b,i)$ , compute the per-token importance ratio $\rho_u^{(b,i)}$ between the current policy and the behavior policy. Apply the clipped surrogate: for each thought, $L_{\text{clip}}^{(b,i)} = -\frac{1}{|c_t^{(b,i)}|}\sum_u \min(\rho_u \cdot \text{sg}(A^{(b,i)}), \text{clip}(\rho_u; 1-\epsilon_\ell, 1+\epsilon_h) \cdot \text{sg}(A^{(b,i)}))$ . Average over batch and groups.

Step 9: Update parameters. $\theta \leftarrow \theta - \eta \nabla_\theta L(\theta)$ where $\eta$ is the learning rate.

Step 10: Update EMA teacher. $\phi \leftarrow \tau\phi + (1-\tau)\theta$ with $\tau = 0.999$ .

Key design choices in the algorithm:

No gradient through rewards: Step 5 and 6 compute $S_{\text{EMA}}$ and $S_{\text{pred}}$ as scalar quantities detached from the computation graph. This ensures the reward is a proper RL signal, not a differentiable objective.
Group-relative normalization: Step 7 ensures that rewards are always relative to the group average, which adapts to varying difficulty across positions. A thought that scores $r = 5$ on an "easy" position (where the average is $\bar{r} = 4.8$ ) receives a smaller advantage than a thought that scores $r = 2$ on a "hard" position (where the average is $\bar{r} = 0.5$ ). This prevents the model from focusing only on positions where rewards happen to be large in absolute terms.
Clipping prevents destructive updates: Step 8's $\min$ operation ensures that a single large-advantage thought cannot cause an unbounded parameter change. This is especially important early in training when the policy is random and some thoughts may receive outlying rewards by chance.
EMA teacher updates after the policy: Step 10 updates the baseline after the policy update, not before. This means the baseline always lags by at least one step, maintaining the comparison gap needed for informative rewards.

Hyperparameter Summary

The paper reports key hyperparameters across multiple sections. Here is the consolidated configuration:

RLP training (Appendix 10):

Hardware: 32 H100 80GB SXM5 GPUs
Learning rate: constant $1 \times 10^{-6}$
Batch size: 512 (unique prompts per generation step)
Maximum context length: 2048 tokens
Number of rollouts: $G = 16$ (default for main experiments)
Thought completion length: 2048 tokens (default for main experiments)
Generation temperature: 0.7
KL coefficient: $\beta = 0$ (no KL anchoring)
EMA decay: $\tau = 0.999$
Training duration: 170M to 1B input tokens (depending on experiment)

Continuous pretraining baseline (Appendix 10):

Hardware: 32 H100 80GB SXM5 GPUs
Optimizer: AdamW with $\beta_1 = 0.9$ , $\beta_2 = 0.95$ , weight decay 0.1
Learning rate: maximum $1 \times 10^{-6}$ , minimum $1 \times 10^{-7}$
Batch size: 6M tokens
Context length: 8192 tokens
Parallelism: 2-way tensor and pipeline parallelism

Post-training (Appendix 10):

SFT: OpenThoughts dataset, filtered for examples with final answers (45,602 samples after filtering)
RLVR: MATH dataset (7,500 examples), 1 epoch, global batch size 1024, cosine annealing, initial learning rate $1 \times 10^{-6}$

Unspecified hyperparameters: the paper does not report values for $\epsilon_\ell$ and $\epsilon_h$ (the clipping thresholds in the surrogate objective), the optimizer used for RLP updates (presumably AdamW but not confirmed), or whether gradient accumulation is used to achieve the effective batch size. These are gaps that would need to be filled for exact replication.

Design Choices and Their Justifications

Why verifier-free instead of verifier-based? The central design decision of RLP is that the reward is computed from the model's own outputs and the observed data, without any external verifier. This choice is driven by the goal of operating on arbitrary pretraining text — most tokens in general web text do not have verifiably correct continuations. If RLP required a verifier, it would be restricted to domains with ground-truth labels (math, code), which would defeat the purpose of a pretraining objective.

Why information gain instead of binary correctness? The continuous log-likelihood ratio provides more information than a binary correct/incorrect signal. A thought that increases the probability of the correct token from 0.01 to 0.5 is more useful (in information-theoretic terms) than one that increases it from 0.49 to 0.51, even though the latter might cross an arbitrary correctness threshold. The continuous signal captures this nuance, providing richer credit assignment.

Why group-relative advantages instead of raw rewards? Raw rewards vary in scale across positions — some contexts are inherently more predictable than others. Group-relative advantages normalize within each group, removing position-level variance and focusing the gradient on the relative quality of thoughts for the same context. The Bessel correction $G/(G-1)$ ensures this normalization doesn't bias the estimator.

Why EMA baseline instead of a frozen reference? A frozen baseline would become increasingly stale as the model improves, making rewards trivially large and providing no useful gradient signal. The EMA baseline tracks the model's progress while maintaining a gap, ensuring rewards remain informative throughout training.

Why apply gradients only to thought tokens? The objective is to improve the thought policy — to generate thoughts that increase predictive evidence. The next-token predictor $p_\theta$ is not directly optimized; it improves indirectly because better thoughts lead to better conditioning. This separation is consistent with the RL framing: the thought is the action, the next-token prediction quality is the environment's response, and the policy is updated based on that response.

Why one token per document? Computational feasibility. Generating 2048-token thoughts with 16 rollouts for every token in a billion-token corpus would be prohibitively expensive (roughly $10^9 \times 16 \times 2048 = 3.3 \times 10^{13}$ generated tokens). Sampling one token per document amortizes the thought-generation cost while still providing sufficient reward events for learning.

4. Key Insights and Innovations

Innovation 1: Reasoning as a Pretraining Objective, Not a Post-Training Afterthought

The most fundamental conceptual move in this paper is temporal: it relocates reinforcement learning for reasoning from the end of the training pipeline to the pretraining phase itself. This is not an incremental shift — it inverts the dominant assumption that has governed LLM training for years.

The prior consensus. The standard pipeline — pretraining with next-token prediction, then supervised fine-tuning, then RL with verifiable rewards (RLVR) or human feedback (RLHF) — treats reasoning as a capability to be grafted onto a model that has already internalized language through likelihood-based training. This pipeline is so entrenched that it structures entire research programs: DeepSeek-R1 (Guo et al., 2025), Open-Reasoner-Zero (Hu et al., 2025), DeepScaleR (Luo et al., 2025), and ProRL (Liu et al., 2025a) all apply RL exclusively after SFT on curated reasoning datasets. The implicit assumption is that reasoning requires task-specific verifiers or curated solution traces, and therefore cannot be integrated into the unsupervised, domain-agnostic pretraining phase.

What RLP changes. The paper argues — and demonstrates — that reasoning can be induced earlier, during pretraining, using a reward signal that requires no external verifier, no ground-truth answers, and no curated reasoning data. The key enabling insight is that the quality of a thought can be measured by how much it improves next-token prediction. This reframes Chain-of-Thought generation from a post-hoc explanation mechanism into an exploratory action taken to reduce predictive uncertainty. Because next-token prediction is the universal pretraining task, this reward is available everywhere — on web crawl, academic papers, textbooks, and open-ended prose — not just on problems with verifiable answers.

Why this is a fundamental shift, not a small refinement. The standard pipeline implies that pretraining compute — which dwarfs post-training compute by orders of magnitude — is spent building a model that cannot yet reason. Reasoning capability is deferred to a later, narrower phase. RLP changes the basic economic equation of LLM training: it makes reasoning a first-class citizen of pretraining, shaped by the same massive data streams that teach language. The paper's headline result on NEMOTRON-NANO-12B-V2 crystallizes this: applying RLP for only 250M tokens (0.125% of the 20T-token pretraining budget) lifts the overall benchmark average from 42.81% to 61.32% — a 43% relative gain (Table 2). This is not a marginal improvement from a better RL recipe; it's evidence that a large fraction of the reasoning capability latent in the pretrained model was simply never elicited by the next-token objective alone.

The compounding evidence. Perhaps the most compelling argument for RLP as a foundational shift is that its gains survive and compound through strong post-training. When M_RLP and M_base both undergo identical SFT + RLVR pipelines, M_RLP maintains a 7–8% relative advantage on QWEN3-1.7B-BASE (Table 1) and a clear margin on NEMOTRON-NANO-12B-V2 (68.09% vs. 65.31%, Table 2). If RLP were merely teaching surface patterns that post-training would overwrite, these margins would vanish. Their persistence indicates that RLP shapes something deeper — a reasoning disposition that subsequent training refines rather than replaces.

Relationship to RPT (Dong et al., 2025). RPT also applies RL during pretraining, which might make RLP appear incremental. But the differences are qualitative. RPT uses sparse binary rewards (correct/incorrect next token) applied only to tokens pre-selected by an auxiliary entropy-filter model, and it "often relies on proxy-model filtering of 'easy' tokens" and "depend[s] on distilled checkpoints with strong prior reasoning ability" (Section 1). RLP, in contrast, is self-contained — one model, one objective, applicable uniformly to all positions in any text. The paper's matched comparison (Table 3) shows RLP outperforming RPT by 4% (token-matched) to 20% (FLOP-matched), confirming that the design differences translate to empirical gains.

Innovation 2: Information Gain as a Verifier-Free, Dense Reward for Thinking

The second distinctive contribution is the reward formulation itself. RLP defines the value of a thought as its information gain — the log-likelihood ratio between the reasoned predictor and a no-think baseline. This is a conceptual bridge between information theory and reinforcement learning that solves several problems simultaneously.

The prior landscape of reward design. In standard RL for reasoning, rewards come from external verifiers: checking final answers against ground truth (RLVR), comparing outputs to reference solutions, or soliciting human judgments (RLHF). These approaches share a common bottleneck — they require a source of truth outside the model, which restricts their applicability to domains where truth is well-defined and mechanically verifiable. Verifier-free post-training methods (Yuan et al., 2024; Liu et al., 2025b; Zhao et al., 2025) remove the external verifier but substitute model-internal judgments — the model evaluates its own outputs or uses its own confidence as reward. These are still fundamentally post-training approaches operating on curated data.

What information gain provides. RLP's reward $r(c_t) = \log p_\theta(x_t \mid x_{<t}, c_t) - \log \bar{p}_\phi(x_t \mid x_{<t})$ is neither external nor purely self-referential. The first term evaluates how well the model predicts the observed next token given the thought. The second term provides the counterfactual: how well the model would have predicted it without thinking. The observed token $x_t$ is the ground truth — it comes from the data, not the model. But the evaluation of predictive quality is entirely model-internal. This hybrid structure — data-grounded yet verifier-free — is what makes the reward applicable to arbitrary text.

Why density matters. Most RL signals in language domains are sparse: they arrive at the end of a complete output (a final answer, a compiled program, a human rating). RLP's reward arrives at every position where a thought is sampled. This density is crucial for efficient credit assignment — the model learns which parts of its thinking actually moved probability mass toward the correct continuation, not just whether the final answer was right. The ablation on completion length (Figure 2b, Appendix Table S.1) dramatically illustrates this: at 64 tokens, accuracy is worse than the base model (11.50% vs. 34.03%), presumably because insufficient thought length leads to noisy or misleading signals. At 2048 tokens, the dense reward has enough context to provide meaningful per-position credit, yielding 42.17%.

The theoretical architecture. The paper provides two propositions (Section 2.3, Appendix 8) that give the reward a clean information-theoretic interpretation. Proposition 1 proves that the expected reward equals the reduction in cross-entropy: $\mathbb{E}[r(c_t)] = \text{CE}(p^*, \bar{p}_\phi) - \text{CE}(p^*, p_\theta(\cdot \mid x_{<t}, c_t))$ . This means maximizing the reward is not arbitrary — it directly minimizes the predictive error attributable to the absence of reasoning. Proposition 2 proves that the CoT-conditioned objective is a lower bound on the improvement achievable by marginalizing over all possible thoughts. This gives the training procedure a principled status: even though it optimizes a single sampled thought per position, it is pushing toward a state where the model's expected output would be a better predictor.

Why this reframing matters beyond RLP. The information-gain framing suggests a general principle: any time a model can generate auxiliary computation (thoughts, subroutines, retrieved context) before making a prediction, the quality of that auxiliary computation can be measured by its impact on predictive accuracy. This principle could extend beyond CoT generation — to tool use (does calling this API improve my prediction?), retrieval (does this document help?), or multi-step planning (does this subgoal make the final answer more likely?). RLP demonstrates the principle in the specific case of chain-of-thought, but the conceptual framework is broader.

Innovation 3: The EMA Baseline as a Dynamic, Self-Stabilizing Counterfactual

A subtle but crucial innovation is the use of an Exponential Moving Average (EMA) of the model's own parameters as the no-think baseline. This design choice solves a problem that has plagued self-supervised RL approaches: how to maintain an informative comparison without an external reference.

The instability problem. If the baseline were frozen (e.g., the initial pretrained model), it would become increasingly stale as the model improves. Eventually, the reasoned predictor would be so much better than the frozen baseline that rewards would be trivially large and cease to provide useful gradient information — every thought would look brilliant. If the baseline tracked the model without lag, the log-likelihood ratio would collapse toward zero — no thought would appear to help because the baseline would immediately incorporate any improvements. Either extreme destroys the reward signal.

The EMA solution. The EMA baseline $\bar{p}_\phi$ is updated as $\phi \leftarrow \tau \phi + (1-\tau)\theta$ after each optimizer step, with $\tau = 0.999$ (Section 2.1). This means the baseline is a heavily smoothed, slightly lagged version of the current model — current enough to provide a meaningful comparison (it represents what the model would predict without thinking, at essentially its current capability level), but lagged enough to maintain a gap that thoughts can exploit. The paper characterizes this as a design that is "current enough to provide informative comparisons and intentionally lagged to mitigate reward hacking" (Section 2.1).

Empirical evidence of stability. The ablation on EMA decay $\tau$ (Appendix 12, Table S.6) shows a bell-shaped performance curve peaking at $\tau = 0.999$ . Lower values ( $\tau = 0.99$ , 0.995) yield weaker performance because the baseline updates too quickly and rewards collapse. Higher values ( $\tau = 0.9995$ ) also underperform, presumably because the baseline lags too far behind and comparisons become less informative. The paper notes that "across this range, training remained stable and we did not observe divergent or unstable behavior" — a non-trivial assertion for an RL procedure operating without external anchors.

Why this matters beyond RLP. The EMA baseline is an elegant instance of a broader principle: self-play with temporal separation. By making the model compete against a slightly older version of itself, RLP creates a curriculum that naturally becomes harder as the model improves — the baseline catches up, so the model must generate increasingly sophisticated thoughts to maintain positive rewards. This is reminiscent of self-play in games (AlphaGo, AlphaZero) but adapted to the continuous, non-adversarial domain of language modeling. The paper's explicit grounding of the EMA schedule in reward-hacking prevention (Section 2.1) is a diagnostic contribution: it identifies why self-referential RL signals can be stable and provides a concrete recipe (one-step lag, heavy smoothing) that other self-supervised RL methods could adopt.

Contrast with standard RLHF anchoring. In RLHF, stability is typically achieved through KL divergence penalties that anchor the policy to a frozen reference model. RLP's ablations (Figure 2c, Appendix Table S.1) show that adding KL anchoring degrades performance — $\beta = 10^{-4}$ drops overall accuracy by 0.82 points. The EMA baseline makes KL anchoring unnecessary because the temporal lag provides sufficient regularization. This is not just a hyperparameter finding; it reveals a qualitative difference between RLP's reward structure (which is self-stabilizing because it's grounded in data likelihood) and RLHF's reward structure (which requires external constraints because the reward model is imperfect and gameable).

Innovation 4: The Difficulty-Adaptive Credit Assignment Implicit in Group-Relative Advantages

The paper does not explicitly frame this as a "difficulty-aware" mechanism — unlike some prior work on test-time compute scaling — but the group-relative advantage formulation implicitly performs a form of difficulty-adaptive credit assignment that is worth surfacing as a distinct conceptual contribution.

The problem of position-varying reward scales. Different positions in text have different baseline predictability. A position where the next token is "the" after "I went to" is highly predictable; a position requiring domain knowledge ("The capital of Burkina Faso is") is less so. The raw information-gain reward $r(c_t)$ conflates two factors: the inherent difficulty of the prediction and the quality of the thought. A thought that achieves $r = 0.1$ on an unpredictable position may be far more valuable than one that achieves $r = 5.0$ on an easy position, but raw rewards would suggest the opposite.

The group-relative solution. By computing advantages relative to the mean reward within each group of $G$ thoughts sampled for the same position, RLP normalizes away position-level difficulty. The advantage $A^{(i)} = \frac{G}{G-1}(r^{(i)} - \bar{r})$ measures how much better (or worse) thought $i$ is compared to the average thought for this specific prediction. This means the model receives positive reinforcement for any thought that is better than its current average thinking on that particular context, regardless of the absolute scale of rewards.

Why this is a non-obvious design choice. A natural alternative would be to use raw rewards directly, or to normalize across the entire batch rather than within groups. Batch-level normalization would conflate position difficulty with thought quality — a thought on an easy position could receive a high advantage simply because the position is easy, not because the thought is good. Group-level normalization solves this by comparing thoughts only to other thoughts for the same prediction. The Bessel correction $G/(G-1)$ is a minor technical detail, but the conceptual move — making the reward relative to the model's current capability on that specific problem — is the key insight.

Evidence from the difficulty-robust performance. The paper does not provide a direct ablation comparing group-relative to batch-relative or raw-reward training. However, the consistent gains across diverse corpora (Table 4) — from structured SFT-style reasoning data to open-ended web crawl — provide indirect evidence that the credit assignment mechanism works across wildly varying difficulty distributions. The fact that RLP extracts useful reasoning signals from academic papers, math textbooks, and web crawl — each with very different baseline predictability — suggests that the group-relative normalization is effectively handling the domain shift in reward scales.

Connection to the broader RL literature. Group-relative advantages are a form of baseline subtraction for variance reduction, a standard technique in policy gradient methods (REINFORCE with baseline, A2C, PPO). But the paper's specific choice — an inclusive mean with Bessel correction, computed per-position rather than per-batch — is tailored to the structure of the problem. The thought space is combinatorial (each thought is a sequence of tokens), the prediction difficulty varies per position, and the group size $G$ is relatively small (16). The inclusive mean with correction ensures unbiasedness with low variance under these conditions. The paper's justification (Appendix 9) is primarily mathematical (proving monotonic improvement), but the practical significance is that it enables stable training on heterogeneous data without per-domain tuning — a requirement for any method that claims to be a general pretraining objective.

Innovation 5: The Demonstration That RL Gains Persist and Compound Through Post-Training

The final contribution is empirical rather than methodological, but it has significant implications for how the field thinks about training pipelines. The paper demonstrates that the benefits of RLP are not transient — they survive a strong post-training regimen (SFT + RLVR) and, in many cases, compound with it.

What "compounding" means concretely. In Table 1, M_base improves from 30.32% to 39.34% after post-training (a gain of 9.02 points). M_RLP improves from 36.03% to 42.51% (a gain of 6.48 points). The absolute gap between M_RLP+Post and M_base+Post is 3.17 points — larger than the base-model gap of 5.71 points would naively predict if post-training simply added a constant improvement. On specific benchmarks, the compounding is striking: on MMLU-Pro, M_base gains 9.68 points from post-training (28.17% → 37.85%) while M_RLP gains 7.78 points (34.62% → 42.40%), yet M_RLP+Post still leads by 4.55 points. On GPQA, M_base gains 5.68 points (25.25% → 30.93%) while M_RLP gains 5.05 points (28.28% → 33.33%).

Why this is not obvious. Reinforcement learning is known to be brittle — gains from one phase can be washed out or reversed by subsequent training. Catastrophic forgetting, reward hacking, and distribution shift can all cause RL-derived capabilities to degrade. The fact that RLP's gains not only survive but amplify through SFT + RLVR is evidence that RLP is teaching something foundationally useful — reasoning patterns that subsequent training can build upon rather than override. The paper frames this as RLP establishing "robust reasoning foundations that are not washed out by downstream alignment but instead compound with post-training" (Section 4).

The contrast with ReST^EM. Appendix K (not shown in the provided excerpt but referenced in the paper) describes an attempt to optimize the revision model using ReST^EM (a self-improvement method from Singh et al., 2024) that degraded performance. This negative result highlights that the compounding property is not guaranteed — it depends on the specific training recipe. RLP's compounding is therefore a finding about the compatibility of its induced reasoning with downstream alignment methods, not a universal property of any RL pretraining.

What this implies for training pipeline design. The compounding result challenges the standard assumption that pretraining and post-training are independent stages that can be optimized separately. If RLP-initialized models respond differently (and better) to post-training than standard models, then the optimal allocation of compute between pretraining, RL pretraining, SFT, and RLVR may be jointly determined — you cannot decide how much to spend on RLP without knowing how much you'll spend on downstream RL, because the two interact. This opens a research direction analogous to compute-optimal training (Hoffmann et al., 2022) but for the entire training pipeline including RL phases.

Evidence at scale. The NEMOTRON-NANO-12B-V2 results (Table 2) show the compounding effect at larger scale and with a different architecture. M_RLP improves from 42.81% to 61.32% (base stage), then to 68.09% after post-training. M_base+Post reaches 65.31%. The 2.78-point post-training gap on a hybrid Mamba-Transformer architecture confirms that the compounding effect is not specific to the QWEN3 transformer or to small-scale experiments.

5. Experimental Analysis

Evaluation Methodology

Dataset. The primary training corpus for RLP experiments spans six datasets across two families. The first family consists of SFT-style reasoning corpora: OmniMath (Gao et al., 2024), OpenThoughts (Guha et al., 2025), and Nemotron-Crossthink (Akter et al., 2025). The second family consists of general-purpose pretraining corpora: academic papers (ACAD), math textbooks (Math-Text), and open-ended web crawl QA pairs from Common Crawl (Web-Crawl). These are collectively referred to as the "diverse pretraining" (DPT) corpus. For post-training, the SFT stage uses the OpenThoughts dataset (filtered to 45,602 examples with final answers), and the RLVR stage uses the MATH dataset (Hendrycks et al., 2021b) with 7,500 examples across algebra, geometry, number theory, and precalculus.
Base model(s). Two primary base models are used: QWEN3-1.7B-BASE (Yang et al., 2025), a standard 1.7B-parameter Transformer, and NEMOTRON-NANO-12B-V2 (Nano, 2025), a 12B-parameter hybrid Mamba-Transformer. The QWEN3 model is chosen as a representative moderate-scale base model without prior reasoning-specific RL — the paper explicitly argues it "is representative of the capabilities of many contemporary LLMs" (Section 1). The NEMOTRON-NANO model is chosen to test scalability and architecture-agnosticism — it uses a fundamentally different backbone (State Space Models plus attention) and was trained from scratch on a different data mixture involving an ensemble of teacher models (DeepSeek-R1, Mixtral-8x22B, DeepSeek-V3, Phi-4, Nemotron-4-340B). A smaller scaling experiment also uses QWEN3-14B-BASE (Appendix 12, Table S.3). An early-checkpoint experiment uses NEMOTRON-NANO-12B-V2 at 4T tokens (20% of the full 20T pretraining budget, Appendix 12, Table S.4).
Metrics. The primary metrics are accuracy on a suite of benchmarks, organized into three aggregates. Math Avg covers four benchmarks: GSM8K (Cobbe et al., 2021), MATH-500 (Hendrycks et al., 2021c), Minerva Math (Lewkowycz et al., 2022), and AMC23, reported as Pass@1 average of 8 runs. Science Avg covers MMLU (Hendrycks et al., 2021a), MMLU-Pro (Wang et al., 2024), and GPQA-Diamond (Rein et al., 2024), reported as the average of greedy and Pass@1 scores from 4 runs (denoted Science Avg@1[4] when both are included). Overall is the macro-average of Math Avg and Science Avg. All evaluations use the NeMo-Skills framework (https://github.com/NVIDIA/NeMo-Skills), and grading functions are from the benchmark authors (e.g., Lightman et al., 2022 grading for MATH-500). For the RPT comparison (Table 3), simpler Math Avg and Science Avg aggregates (without @1[4] notation) are used, with Overall as their mean. Additional benchmarks reported in some experiments include AIME25, MMLU@1[4], MMLU-Pro@1[4], and GPQA@1[4].
Baselines. Several baselines are used across different comparisons. M_base: the unmodified pretrained model without any additional training. M_CPT (Continuous Pretraining): the base model further trained on the same tokens used for RLP, using standard next-token prediction loss (Appendix 10 details: Megatron-LM, AdamW with β1 = 0.9, β2 = 0.95, weight decay 0.1, learning rate 1e-6 to 1e-7, batch size 6M tokens, 8192 context length). Two variants are used: token-matched (M_CPT[170M], trained on the same number of input tokens as RLP) and FLOP-matched (M_CPT[6B], trained on 6B tokens to equalize total compute, Appendix 12). RPT (Reinforcement Pre-training) (Dong et al., 2025): a directly comparable RL-based pretraining method using sparse binary rewards and auxiliary entropy filtering, evaluated under both token-matched and FLOP-matched conditions (Table 3). Post-training baselines: M_base+Post, M_CPT+Post, and M_RLP+Post, where all models undergo identical SFT + RLVR pipelines.
Generation budget / compute accounting. Compute is measured in input tokens for standard pretraining, with the paper explicitly accounting for the additional FLOPs consumed by RLP's rollout phase. For FLOP-matched comparisons (Appendix 12), the total cost of RLP is computed as T_flop = (n × l_seq × bs × iters) + T_inp, where n = 16 rollouts, l_seq is the sequence length, bs is the batch size, and iters is the number of steps. This effectively yields a total computational budget that equates to approximately 6B tokens of continuous pretraining for a 170M-token RLP run — a 35× multiplier. The paper notes that this calculation "is a rigorous upper bound" because forward-pass FLOPs per generated token (~2N) are significantly less than forward+backward FLOPs per training token (~6N). RLP applies the reward to only one randomly selected token per document, so the number of reward-applied tokens is far less than the total input tokens. In contrast, RPT applies rewards to multiple pre-selected tokens per document, making the reward-token count substantially larger in token-matched comparisons — a point the paper flags as favoring RPT in the token-matched setting (Appendix 12).
Cross-validation / statistical protocol. The paper does not employ formal cross-validation for strategy selection (unlike some test-time compute scaling papers). Instead, it relies on direct evaluation on held-out benchmark suites, with consistency across multiple benchmarks and model scales serving as implicit validation. For the ablation sweeps (rollout count, completion length, KL weight), each configuration is evaluated on all benchmarks, and the best configuration is selected based on the overall average. The paper reports per-benchmark breakdowns in Appendix Table S.1, providing transparency about variance across tasks. Pass@1 metrics for math benchmarks are averaged over 8 runs, providing some statistical stability. However, the paper does not report confidence intervals, standard deviations, or the number of evaluation samples per benchmark, which limits formal statistical assessment.

Main Quantitative Results

RLP vs. Continuous Pretraining on QWEN3-1.7B-BASE (Table 1)

Headline result: M_RLP achieves an overall average of 36.03%, compared to M_base's 30.32% (+18.8% relative) and M_CPT's 30.85% (+16.8% relative). After identical post-training, M_RLP+Post achieves 42.51% vs. M_base+Post's 39.34% (+8.1% relative) and M_CPT+Post's 39.90% (+6.5% relative).

Base-model gains are broad and non-uniform. The largest absolute gains at the base stage are on GSM8K (+20.32 points, from 54.16% to 74.48%), MATH500 (+10.03 points, from 48.45% to 58.48%), and MMLU-Pro (+6.45 points, from 28.17% to 34.62%). Science Avg improves from 34.50% to 39.68% (+5.18 points), while Math Avg improves from 24.35% to 31.74% (+7.39 points). GPQA sees a more modest gain (+3.03 points, from 25.25% to 28.28%). Notably, some of the M_CPT baseline's scores are below M_base on certain benchmarks: MMLU drops from 50.08% (M_base) to 41.95% (M_CPT), and GPQA@1[4] drops from 27.52% to 24.75%. The paper notes this as evidence that continuous pretraining on reasoning data can introduce domain-specific degradation, whereas RLP generalizes more evenly.

Post-training compounds gains unevenly across benchmarks. The largest compounding effects are on science benchmarks. M_RLP+Post's Science Avg@1[4] is 45.76%, vs. M_base+Post's 41.35% (+4.41 points) and M_CPT+Post's 42.34% (+3.42 points). MMLU-Pro@1[4] shows a 4.77-point advantage over M_base+Post (41.30% vs. 36.53%). GPQA@1[4] shows a 3.45-point advantage (34.97% vs. 31.52%). On the math side, the post-training gap is narrower: Math Avg for M_RLP+Post is 36.03% vs. M_base+Post's 34.29% (+1.74 points). AIME25 — the hardest benchmark — shows M_RLP+Post at 7.05% vs. M_base+Post at 5.32% (+1.73 points), with M_CPT+Post at 5.89% in between. The overall advantage of M_RLP+Post over M_CPT+Post (42.51% vs. 39.90%, +2.61 points) is narrower than the base-stage advantage (36.03% vs. 30.85%, +5.18 points), indicating that while gains persist, post-training does partially close the gap — consistent with the idea that RLVR benefits all models but builds on stronger foundations from RLP.

Comparison to M_CPT on specific benchmarks is instructive. For MMLU, M_CPT degrades the base model (41.95% vs. 50.08%), while M_RLP improves it (56.14%). This suggests that standard continuous pretraining on the DPT mixture can harm some general-knowledge capabilities (perhaps through catastrophic forgetting or domain shift), whereas RLP's information-gain objective preserves or enhances them. However, the paper does not provide a detailed analysis of why this occurs, which is a gap in the experimental narrative.

Scaling RLP to NEMOTRON-NANO-12B-V2 (Table 2)

Headline result: M_RLP achieves an overall average of 61.32%, compared to M_base's 42.81% — a 43.2% relative improvement. This is achieved using only 250M tokens of RLP training (0.125% of the base model's 20T-token pretraining budget). After post-training, M_RLP+Post achieves 68.09% vs. M_base+Post's 65.31%, a +2.78-point margin.

Dramatic base-stage gains on science and math. Science Avg rises from 34.51% to 57.26% (+22.75 points), a 65.9% relative improvement. Math Avg rises from 61.38% to 65.33% (+3.95 points). The asymmetry — larger gains in science than math — contrasts with the QWEN3 results where both domains improved more evenly. This may reflect the NEMOTRON-NANO model having stronger initial math capabilities (61.38% vs. 24.35% for QWEN3), leaving less room for improvement, or differences in how the hybrid architecture benefits from RLP. MMLU nearly doubles: from 54.12% to 78.76% (+24.64 points). MMLU-Pro rises from 24.16% to 53.13% (+28.97 points). GSM8K rises from 72.31% to 85.98% (+13.67 points). However, AMC23 decreases from 70.63% to 57.19% (−13.44 points) — a notable negative result that the paper does not comment on. This suggests RLP can have regressive effects on specific benchmarks even as it improves aggregate performance, which is worth flagging as a potential replication concern.

Post-training effects. The gap narrows after post-training: M_RLP+Post's overall (68.09%) leads M_base+Post (65.31%) by 2.78 points. The science advantage persists (Science Avg 64.52% vs. 58.91%, +5.61 points), but the math advantage narrows (Math Avg 74.93% vs. 70.16%, +4.77 points). On AMC23, M_RLP+Post recovers to 75.00% vs. M_base+Post's 62.19% — an improvement even over the original M_base (70.63%), suggesting that post-training interacts favorably with RLP on this benchmark even though RLP alone degraded it. GPQA@1[4] shows an unusual pattern: M_RLP is at 48.86% (vs. M_base's 22.47%), but M_RLP+Post drops to 49.62% while M_base+Post rises to 52.15% — post-training actually narrows the gap from M_RLP's perspective. This could indicate that RLVR on MATH does not transfer well to GPQA for the RLP model, or that some of the base-stage gains on GPQA were not robust to subsequent training.

RLP vs. RPT: Token-Matched and FLOP-Matched Comparison (Table 3)

Headline result (token-matched): M_RLP achieves Overall 43.35% vs. M_RPT's 41.69% (+1.66 points, +4.0% relative), with Math Avg 49.62% vs. 47.50% (+2.12 points) and Science Avg 37.07% vs. 35.88% (+1.19 points).

Headline result (FLOP-matched): M_RLP achieves Overall 42.86% vs. M_RPT's 35.68% (+7.18 points, +20.1% relative), with Math Avg 45.95% vs. 36.66% (+9.29 points) and Science Avg 38.76% vs. 34.38% (+4.38 points).

Important methodological note on the comparison. The paper clarifies in Appendix 12 that the token-matched setting is actually favorable to RPT because RPT applies rewards to multiple pre-selected tokens per document, while RLP applies rewards to only one randomly selected token per document. Thus, RPT receives many more reward signals per input token. In the FLOP-matched setting, the number of target tokens (tokens for which rewards are computed) is equalized. Under this stricter comparison, RLP's advantage grows from +4.0% to +20.1% — suggesting that RPT's gains are partly attributable to the greater density of reward signals rather than the quality of those signals. The paper also notes that the FLOP-matched comparison "does not include the compute needed to pre-select tokens using an external LLM for RPT," which would further widen the gap in RLP's favor.

What drives the difference? The paper attributes RLP's advantage to three factors: (1) RLP's dense, continuous reward provides richer credit assignment than RPT's sparse binary signal; (2) RLP can operate on any token without pre-filtering, giving it broader coverage of the data distribution; and (3) RLP evaluates the content of the thought (via its impact on prediction) rather than just whether the final prediction is correct — RPT "ignoring the CoT content" (Section 1). However, without a controlled ablation that isolates each factor, it's difficult to attribute the advantage definitively.

RLP Across Diverse Corpora (Table 4)

Headline result: RLP produces consistent gains across all six corpus types tested, with Overall improvements ranging from +7.24 points (Nemotron-Crossthink: 43.36% vs. M_base's 36.12% from that specific experiment) to +6.21 points (Web-Crawl: 42.13% vs. M_base's 34.19%) relative to the base model. The base model's baseline for this experiment is 34.19% overall (the values differ slightly from Table 1 because this experiment uses 170M tokens of RLP training vs. the 1B tokens for Table 1's main results).

SFT-style corpora perform best. Nemotron-Crossthink yields the highest overall (43.36%), followed by Web-Crawl (42.13%), ACAD (41.71%), OpenThoughts (41.43% — note the text reports 40.66% for OpenThoughts overall in Table S.9 vs. 41.45% in Table 4, a discrepancy the paper does not address), OmniMath (41.43%), and Math-Text (41.62%). The gap between the best corpus (Nemotron-Crossthink, a curated reasoning dataset) and the worst (OmniMath, 41.43%) is only 1.93 points — remarkably narrow, suggesting that RLP's signal is largely corpus-agnostic.

General corpora are competitive with SFT corpora. Web-Crawl (42.13%) and ACAD (41.71%) both outperform OpenThoughts (40.66-41.45%) and are competitive with Nemotron-Crossthink. This is a significant finding because it challenges the assumption that RL for reasoning requires curated reasoning data. The paper emphasizes that "even purely non-reasoning general corpora such as web-crawl, RLP extracts a reasoning signal" (Section 5). The mechanism is presumably that web crawl contains implicit reasoning — explanations, arguments, step-by-step instructions — that the information-gain objective surfaces.

Science gains are more variable than math gains. Math Avg is relatively consistent across corpora, ranging from 46.35% (DPT) to 49.76% (Nemotron-Crossthink). Science Avg is more variable: 39.68% (DPT) to 42.54% (Nemotron-Crossthink). Science Avg@1[4] is most variable: 36.67% (DPT) to 37.78% (Nemotron-Crossthink). This suggests that RLP's science improvements are somewhat more sensitive to corpus quality than math improvements, consistent with the idea that mathematical reasoning patterns are more uniformly distributed across text types.

The FLOP-matched baseline is key. M_CPT[Nemotron-Crossthink, 6B] — continuous pretraining on 6B tokens of the same data — achieves Overall 38.04% (Table 4). This is below all RLP variants trained on only 170M tokens. The Math Avg for M_CPT[6B] is 43.90%, which is below the weakest RLP Math Avg (DPT: 46.35%). The Science Avg for M_CPT[6B] is 37.74%, below RLP's strongest (Nemotron-Crossthink: 42.54%) and roughly tied with RLP's weakest (DPT: 39.68%). This 35× data advantage for M_CPT that does not close the gap is compelling evidence that RLP's algorithmic structure — explicit reinforcement of useful thinking — provides a fundamentally more efficient learning signal than likelihood maximization.

The DPT[1B] result (Table 4, bottom row). RLP trained on 1B tokens of the general pretraining mix achieves Overall 40.90% — better than the 170M-token DPT variant (35.60%) but still below several 170M-token runs on curated data (Nemotron-Crossthink's 43.36%). This suggests that data quality matters (curated data outperforms general data at the same token budget), but also that RLP benefits from more data even when that data is less curated.

Compute-Matched Comparison: RLP Outperforms 35× More Data (Table 4)

This result is perhaps the most striking single finding in the paper and deserves separate emphasis. M_CPT[Nemotron-Crossthink, 6B] is a continuous pretraining run on 6B tokens of Nemotron-Crossthink data — 35× more data than the 170M tokens used for M_RLP[Nemotron-Crossthink, 170M]. Yet M_RLP achieves Overall 43.36% vs. M_CPT[6B]'s 38.04% — a 14.0% relative advantage. On Math Avg, the gap is 49.76% vs. 43.90% (+5.86 points). On Science Avg, it is 42.54% vs. 37.74% (+4.80 points).

The paper's FLOP accounting (Appendix 12) for this comparison: RLP's 170M input tokens × 16 rollouts × 2048 tokens per thought produces approximately the same total FLOPs as 6B tokens of standard training. The paper describes this as a "rigorous upper bound" because the forward-pass cost per generated token (~2N parameters) is less than the forward+backward cost per training token (~6N parameters), meaning the FLOP-equivalent budget for M_CPT could be even larger than 6B tokens.

This result directly addresses the concern that RLP's gains are "merely" due to more compute. Even when the baseline is given dramatically more data to compensate for RLP's rollout overhead, RLP's algorithmic advantage persists. The paper frames this as evidence that "RLP's gains stem from more efficient use of compute, not larger budgets" (Section 5).

Ablation Studies and Robustness Checks

Rollout count (G) from 4 to 32 (Figure 2a, Appendix Table S.1): Performance improves monotonically from G = 4 (Overall 41.38%) to G = 16 (Overall 42.17%), then slightly decreases at G = 32 (Overall 41.75%). The diminishing returns at G = 32, coupled with the increased computational cost, motivate the default choice of G = 16. The drop at G = 32 is driven mainly by GPQA (−2.13 points) and MATH500 (−1.85 points), while GSM8K is essentially flat (+0.14 points). This suggests that when the group size is too large, the diversity of thoughts may include some that are misleading, and the group-relative advantage may not sufficiently distinguish quality. Alternatively, at G = 32 with fixed batch size, the optimization may become noisier because the effective batch size for gradient computation changes. The paper does not investigate the mechanism behind the U-shaped curve.

Completion length from 64 to 4096 tokens (Figure 2b, Appendix Table S.1): This ablation reveals a phase transition between 512 and 1024 tokens. At length 64, Overall is 11.50% — worse than the 34.03% base model, confirming that too-short thoughts are actively harmful. At 128, Overall drops further to 9.96%. At 256, it recovers slightly to 14.08%. At 512, Overall is 24.65%. The jump to 1024 yields Overall 38.89% (+14.24 points), with GSM8K +28.55 and MATH500 +36.85. At 2048, Overall is 42.17% (+3.28 over 1024). At 4096, Overall is 42.21% (+0.04) — essentially flat. The paper defaults to 2048 as the preferred trade-off between reasoning quality and computational cost. This ablation is the strongest evidence that thought length is a critical determinant of RLP's effectiveness, and that there is a minimum threshold (roughly 1024 tokens) below which thinking provides insufficient information to improve predictions.

KL coefficient β (Figure 2c, Appendix Table S.1): KL anchoring does not help and slightly hurts performance. β = 0 yields Overall 42.17%. β = 10⁻⁴ yields 41.35% (−0.82). β = 10⁻³ yields 41.44% (−0.73). There are small isolated gains on some benchmarks (MMLU-Pro +1.43 at β = 10⁻⁴; AMC23 +1.88 at β = 10⁻³), but these are offset by broader declines (GSM8K −1.26 at 10⁻⁴, −2.82 at 10⁻³; GPQA −2.01 at 10⁻⁴, −1.51 at 10⁻³). The paper attributes the lack of benefit to the self-stabilizing nature of the EMA baseline, which provides sufficient regularization without an explicit KL penalty. The paper also notes that KL anchoring "increases memory and step time," suggesting a practical disincentive as well.

EMA decay rate τ from 0.99 to 0.9995 (Appendix 12, Table S.6): Performance forms a bell-shaped curve peaking at τ = 0.999 (Overall 39.54% — note this is from an earlier experimental configuration than the main 42.17% results). τ = 0.99 yields 38.82%, τ = 0.995 yields 39.21%, τ = 0.999 yields 39.54%, and τ = 0.9995 yields 39.20%. The paper reports that training was stable across all τ values. The peak at 0.999 is consistent with the design rationale: a baseline that is too fast (lower τ) collapses the reward signal, while one that is too slow (higher τ) becomes stale and provides less informative comparisons.

Domain specificity (Appendix 12, Table S.2): Training RLP on math-only, science-only, and combined (math + science) data reveals that multi-domain training provides the best overall average (43.36% vs. 42.21% for math-only and 42.36% for science-only). Math-only training yields strong math (48.23%) but weaker science (41.64%); science-only training yields strong science (39.65% @ 1[4], compared to 38.26% for math-only) but weaker math (49.17% vs. 49.76% for combined). The combined approach achieves the best of both, suggesting "complementary signals from multiple domains without diluting the benefits within each" (Appendix 12). This is a robustness check against the concern that RLP might overfit to a single reasoning domain.

Early vs. late RLP application (Appendix 12, Table S.4): Applying RLP to an early NEMOTRON-NANO-12B-V2 checkpoint (4T tokens, 20% of full pretraining) for 1B tokens improves Overall from 12.05% to 24.08% — a 2× improvement. Math Avg more than doubles from 21.93% to 50.14%. Science Avg@1[4] improves from 5.69% to 11.96%. While the final accuracy is much lower than the 20T-token results (Table 2), the relative gains are substantial, indicating that RLP can be effective even when the base model is relatively undertrained. The paper notes that "strongest final results come from applying RLP later in pretraining" but that early application can still yield large gains. This is a robustness check against the concern that RLP requires a well-trained base model to work.

Scaling to QWEN3-14B-BASE (Appendix 12, Table S.3): Applying RLP to a 14B-parameter model for 1B tokens on DPT improves Overall from 60.66% to 65.00% (+4.34 points). Math Avg improves from 65.77% to 68.32% (+2.55 points). Science Avg improves from 60.07% to 64.76% (+4.69 points). The gains are smaller in absolute terms than the 1.7B results (where Overall improved from 30.32% to 36.03%, +5.71 points), which is expected — a stronger base model has less room for improvement. The pattern of larger gains in science than math mirrors the NEMOTRON-NANO-12B-V2 results and may reflect ceiling effects in math for already-capable models.

Context length for CPT baseline (Appendix 12, Table S.5): To address concern that the 8K context length used for M_CPT might disadvantage it relative to the base model's 32K capability, the paper evaluates M_CPT trained at 32K context length. M_CPT(32K) achieves Overall 30.83% vs. M_CPT(8K)'s 32.90% — the longer context actually performs worse. Math Avg drops from 36.92% to 31.61% (−5.31 points). The paper attributes this to the DPT corpus consisting mostly of short documents where longer context provides no benefit but complicates optimization. This justifies the use of 8K as the stronger CPT baseline.

Perplexity after RLP training (Appendix 12, Table S.8): To verify that RLP does not degrade general language modeling, the paper reports perplexity (PPL) and negative log-likelihood (NLL) on Nemotron-Crossthink and Wikitext-103. M_RLP achieves lower PPL than M_base on both: Nemotron-Crossthink PPL drops from 2.91 to 2.36; Wikitext-103 PPL drops from 5.83 to 4.48. This is expected because the RLP reward is defined as log-likelihood improvement, so the policy update should improve prediction. The paper notes this as confirmation that RLP "optimizing for prediction accuracy" does not cause distribution shift away from natural language.

Thought trace analysis (Appendix 12, qualitative): A manual analysis of 50 randomly sampled thought traces reveals "focused, context-aware reasoning" characterized by "grammatical correctness and strong contextual relevance." The paper identifies three qualitative patterns: (1) context-aware reasoning that orients the model to the specific logical step required; (2) continuation-style formatting that mirrors the prefix's tone and notation; and (3) utility-driven anticipation where thoughts predict structural moves. An illustrative example is provided. This analysis is limited (50 samples, no inter-annotator agreement, no quantitative metrics), but serves to confirm that the information-gain objective does not produce degenerate or uninterpretable thinking — an important sanity check for any RL-on-language approach.

Does thinking actually help? (Section 2.5, Proposition 1): The paper provides a formal proof that the expected reward equals the reduction in cross-entropy. While this is a theoretical guarantee, the experimental evidence that RLP's benefits are real (not just internally consistent) comes from the benchmark improvements and the fact that M_RLP achieves lower perplexity on held-out text — thinking leads to genuinely better predictions.

Critical Assessment

The experiments are unusually comprehensive for a paper introducing a new pretraining objective, spanning two model families, two model sizes (1.7B and 12B, plus an additional 14B scaling point), six corpus types, and multiple compute-matched baselines. The key claims from the executive summary are largely supported, though with important boundaries that deserve explicit acknowledgment.

Does RLP lift overall average by 19% on QWEN3-1.7B-BASE? Yes — 36.03% vs. 30.32% = 18.8% relative improvement (Table 1). The 19% figure is correctly rounded. However, this gain is computed over the base model before any post-training. The base model's overall average of 30.32% is not a fixed ceiling — M_CPT achieves 30.85% on the same data, and M_base+Post (with no RLP or CPT) achieves 39.34%, which is higher than M_RLP's base-stage 36.03%. So the framing "19% lift" should be understood as "19% over the unmodified base model before post-training," not "19% over the best achievable result without RLP." The more relevant comparison is M_RLP+Post (42.51%) vs. M_base+Post (39.34%) — an 8.1% relative improvement — which is the actual deployment advantage if post-training is standard.

Does RLP outperform compute-matched continuous pretraining by 17%? The 17% figure (36.03% vs. 30.85% = 16.8%, Table 1) compares M_RLP against M_CPT trained on the same input tokens (170M or 1B, depending on the experiment). But M_CPT is not necessarily the strongest possible continuous pretraining baseline. The paper's own FLOP-matched M_CPT (Nemotron-Crossthink, 6B tokens) achieves 38.04% — higher than M_RLP's 36.03% (Table 1), though lower than the 43.36% that M_RLP achieves on the specific Nemotron-Crossthink data in that experiment. The appropriate comparison depends on whether you match input tokens, match total FLOPs, or match data quality. Across all these variants, RLP consistently leads — but the 17% figure specifically refers to the token-matched comparison in Table 1, and the gap narrows when post-training is applied (+7% relative over M_CPT+Post).

Does RLP improve the NEMOTRON-NANO-12B-V2 from 42.81% to 61.32%? Yes — Table 2 reports exactly these numbers, a 43.2% relative gain. However, this is M_RLP (which includes RLP on the intermediate 19.8T-token checkpoint) vs. M_base (the 20T-token continuously pretrained model). The M_base consumed more total pretraining (20T vs. 19.8T + 0.25B for RLP) — this is a near-compute-matched comparison favoring the baseline slightly. The M_RLP+M_post vs. M_base+Post gap is 68.09% vs. 65.31% — only a 4.3% relative improvement after post-training. So the headline 43% gain is accurate for the base stage but substantially attenuates after post-training. This is not a weakness — the paper is transparent about both numbers — but it contextualizes how much of the gain is "baked in" to the model before external verifiers are applied.

Does RLP use only 0.125% of the data consumed by the CPT baseline? The 0.125% figure is 250M tokens (RLP training) divided by 20T tokens (M_base total pretraining). This is a somewhat misleading comparison because: (1) M_base's 20T tokens include all of standard pretraining before the intervention point, while RLP is applied only at the end; and (2) M_RLP's total data exposure includes the 19.8T tokens of pretraining that the base model had already seen. A more accurate characterization: RLP adds 250M tokens of RL pretraining to a 19.8T-token base, while M_base adds 200B tokens (the final 1% of pretraining, roughly, extrapolating from the 20T total and the 19.8T checkpoint) of standard next-token training. The data efficiency claim is still valid — RLP achieves large gains with a small additional data budget — but the 0.125% framing overstates the efficiency by comparing against the entire pretraining budget rather than the marginal budget.

Do RLP gains persist and compound with post-training? Yes, this is one of the strongest and most consistent findings. Across both model families and all benchmark categories, M_RLP+Post > M_base+Post. The evidence is summarized in the Main Quantitative Results above. The compounding claim — that the gap widens after post-training — is more nuanced. For QWEN3-1.7B-BASE (Table 1), the M_RLP vs. M_base gap is +5.71 points, and the M_RLP+Post vs. M_base+Post gap is +3.17 points — actually narrower in absolute terms, though still positive. The "compounding" framing better describes the finding that gains don't vanish (which would happen if RLP taught surface patterns destroyed by subsequent training) than that they amplify multiplicatively. On specific benchmarks (e.g., MMLU-Pro: +6.45 base gap, +4.55 post gap), the absolute margin does persist.

Genuine weaknesses and limitations of the experimental design:

1. No formal statistical testing. The paper reports point estimates of accuracy without confidence intervals, standard deviations, or significance tests. On a 500-question test set (MATH500), a 2-point accuracy difference corresponds to 10 questions — statistical noise could account for some of the smaller reported margins. The Pass@1 average of 8 runs provides some stability, but only for math benchmarks; science benchmarks use 4 runs. Without standard errors, it's difficult to assess whether, e.g., the 0.04-point difference between completion lengths 2048 and 4096 is genuine saturation or noise.

2. Single post-training recipe. All post-training comparisons use one specific recipe (OpenThoughts SFT + MATH RLVR). It is possible that the post-training recipe is suboptimal for RLP-initialized models — or, conversely, that it is accidentally better-suited to RLP models — and a different recipe would change the gap. The paper doesn't ablate the post-training data, hyperparameters, or order, so we don't know whether the compounding effect is robust to post-training design choices.

3. No direct comparison to standard RLVR from the same base checkpoint. A natural question: how does RLP compare to simply applying RLVR (on MATH) directly to the base model, skipping RLP? The paper applies RLVR as part of post-training, but always after SFT. If RLVR alone (without SFT) achieves most of the gain that RLP provides, then RLP's contribution is less significant. The paper does not run this ablation. Similarly, there is no comparison between RLP and simply doing more SFT on the same data — could the gains be achieved by standard supervised training on reasoning traces?

4. The AMC23 regression on NEMOTRON-NANO-12B-V2 (Table 2). M_RLP scores 57.19% vs. M_base's 70.63% on AMC23 — a substantial drop of 13.44 points. The paper does not comment on this negative result. It could be a fluke, a genuine regression, or an interaction with the specific benchmark's answer format. Without investigation, it's unclear whether this is an isolated case or indicative of a broader failure mode where RLP degrades certain types of mathematical reasoning.

5. The GPQA@1[4] post-training inversion (Table 2). M_RLP achieves 48.86% on GPQA@1[4] vs. M_base's 22.47% — a massive base-stage gain. But after post-training, M_RLP+Post scores 49.62% (essentially flat) while M_base+Post jumps to 52.15% — now higher than M_RLP+Post. This suggests that some of RLP's GPQA gains may be fragile or that post-training interacts differently with the two models. The paper does not discuss this inversion.

6. No ablation on which tokens to apply RLP to. The paper applies RLP to one randomly selected token per document, but does not test whether alternative selection strategies (highest entropy, middle of document, positions with certain syntactic properties) affect performance. Given that RPT's token selection via entropy filtering was a key design element, it would be instructive to see whether RLP benefits from smarter selection or whether random selection is genuinely sufficient.

7. The FLOP accounting overestimates M_CPT's advantage but doesn't test extreme baselines. The 35× multiplier for FLOP matching is described as "rigorous upper bound." But an upper bound on M_CPT's compute budget means RLP is being compared against the best possible version of M_CPT that could be trained with those FLOPs — and still wins. This strengthens the result. However, the paper doesn't test what happens if M_CPT is given even more data or trained for longer. Could M_CPT eventually catch up to M_RLP with, say, 100× more data? The scaling trajectory of RLP vs. CPT is unknown.

8. The thought trace analysis is anecdotal. Fifty samples with no inter-annotator agreement, no comparison to baseline thoughts (e.g., from RPT), and no quantitative metrics (coherence, relevance, correctness) makes this a weak qualitative check. The paper could have computed automatic metrics like BLEU/ROUGE against the continuation, perplexity of the thought itself, or factuality checks.

9. No exploration of catastrophic forgetting on non-benchmark capabilities. The paper evaluates only on math and science benchmarks. There is no evaluation of whether RLP degrades performance on tasks it wasn't designed to improve — general knowledge, translation, summarization, code generation, or safety. The Wikitext-103 perplexity check (Appendix 12) is a weak proxy; lower perplexity on general text does not guarantee preserved capabilities on specific downstream tasks.

10. The RPT comparison uses a specific implementation. RPT is not an off-the-shelf method; the paper implements it following the original paper's description. Small implementation differences (token selection threshold, reward shaping, hyperparameters) could affect performance. The paper does not describe RPT implementation details, making it difficult to assess whether the comparison is fair. Similarly, the RPT model in the token-matched setting receives more reward signals than RLP (Appendix 12), which the paper flags as favoring RPT — but it also means the two methods are not being compared under identical conditions.

11. Missing baseline: supervised fine-tuning on thought-answer pairs. RLP generates thoughts and then rewards them based on information gain. A simpler alternative would be to collect high-quality thought-answer pairs (e.g., from a strong teacher model) and do standard SFT on those. If such data exists (e.g., from Nemotron-Crossthink itself), this would be a strong baseline: can explicit supervision on reasoning traces match or exceed RLP's implicit signal? The paper does not run this experiment.

12. The benchmark suite is math-and-science-heavy. All evaluated benchmarks are in math and science domains. While this aligns with the paper's framing as targeting "reasoning," it does not demonstrate that RLP improves general reasoning outside STEM domains — for example, legal reasoning, ethical reasoning, common-sense inference, or multi-step planning in narrative domains. The paper claims RLP yields "domain-agnostic" improvements, but the evidence is limited to domains where the "correct" continuation is relatively well-defined.

In summary, the experimental evidence strongly supports the paper's central claim that RLP induces reasoning capabilities during pretraining that improve benchmark performance, persist through post-training, and are more compute-efficient than equivalent continuous pretraining. The scaling results across architectures and model sizes are particularly compelling. However, the experimental design leaves several important questions open: whether gains generalize beyond STEM benchmarks, whether RLP can compete with supervised alternatives, whether the gains are robust to different post-training recipes, and whether the method has regressive effects on certain capabilities (as suggested by the AMC23 drop and the GPQA inversion). These are not fatal weaknesses — the paper is already unusually comprehensive — but they are genuine limitations that should inform how the results are interpreted.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Unaccounted For and Potentially Prohibitive

The assumption or constraint. RLP's central mechanism — generating chain-of-thought traces before predicting each next token — fundamentally requires generating tokens at inference time that standard pretraining does not. During training, this cost is explicit: for each of the 170M to 1B input tokens, RLP generates G = 16 rollouts of up to 2048 tokens each. The paper acknowledges that RLP's per-step training is approximately 2.25× slower than SFT/CPT (Appendix 12, Table S.7: 12.47 seconds per step for RLP vs. 5.54 seconds for SFT, with the same batch size of 512 and 32K context length). The paper frames this as acceptable because "RLP is drastically more data-efficient" — achieving in 170M tokens what requires 6B tokens of CPT (a 35× data advantage for the baseline).

The consequence. The data-efficiency argument obscures a deeper issue: the cost of generating thoughts at deployment time is not accounted for in any of the paper's benchmark evaluations. RLP is evaluated as a pretraining method — the model is trained with RLP, then evaluated using standard inference (greedy decoding or Pass@1 sampling) without generating additional thoughts at test time. This is an asymmetry: the model learned to reason during training but is not asked to reason during evaluation. The paper provides no evidence about whether the model's improved benchmark performance stems from (a) internalized reasoning patterns that now manifest without explicit thought generation, or (b) architectural changes that make the model a better next-token predictor but don't require ongoing reasoning. If it's (a), the model has genuinely learned to think in its forward pass — an implicit reasoning capability. If it's (b), the model has simply been optimized for better prediction without developing explicit, controllable reasoning. The distinction matters because if the improvement is primarily (b), the model may not generalize the reasoning behavior to out-of-distribution tasks as effectively as a model that continues to explicitly reason at test time.

More concretely, if a practitioner wanted to continue using explicit chain-of-thought at inference time (e.g., for complex multi-step problems where implicit reasoning may be insufficient), the cost model changes dramatically. RLP optimizes the thought policy during training, but there is no mechanism for the model to selectively deploy explicit reasoning at test time based on problem difficulty. The model is either run with thoughts (incurring the 2048-token generation overhead per prediction) or without them — there is no adaptive allocation strategy as found in test-time compute scaling work. The paper does not study how performance scales with explicit reasoning at inference time for RLP-trained models.

What evidence exists in the paper. Table S.7 (Appendix 12) provides the training-time slowdown factor. Figure 2b (ablation on completion length) shows that very short thoughts degrade performance, indicating that the training-time benefit of RLP depends on generating substantial thought traces. However, there is zero experimental data on inference-time thought generation after RLP training — the paper never evaluates the model with explicit CoT at test time, never compares greedy decoding to thought-augmented inference, and never measures the latency or FLOP overhead of deploying an RLP-trained model that continues to use thoughts. The thought trace analysis (Appendix 12) examines 50 training traces but does not test whether similar traces improve inference-time predictions.

Mitigation status. The paper does not address this limitation directly. It frames RLP as a pretraining objective that improves the base model, and all evaluations treat the resulting model as a standard next-token predictor at inference time. The implicit claim is that RLP's benefits are "baked into" the model weights and do not require ongoing thought generation. But this claim is never tested — the paper provides no ablation showing that the improved benchmark scores persist when the model is prohibited from any internal computation that resembles thought generation (which would be impossible to enforce). The paper also does not discuss whether the learned reasoning capability is accessible to prompt-based elicitation (e.g., whether adding "Let's think step by step" to the prompt further improves RLP-trained models beyond what it improves baseline models) or whether RLP effectively substitutes for explicit CoT at inference time. This is a significant practical gap: a practitioner adopting RLP does not know whether they should continue to use chain-of-thought prompting, use the model with greedy decoding, or develop a hybrid strategy.

Benchmark Coverage Is Narrowly Focused on STEM Reasoning with Verifiable Answers

The assumption or constraint. The paper evaluates RLP exclusively on math and science benchmarks: GSM8K, MATH500, Minerva, AMC23 (math), and MMLU, MMLU-Pro, GPQA (science). These are all domains where answers are objectively verifiable, multi-step logical deduction is rewarded, and reasoning patterns are relatively structured. The paper argues that RLP is "domain-agnostic" and "scalable to large, diverse corpora" (Section 5), citing its use on web crawl, academic papers, and textbooks. But the evaluation tells a narrower story: we only know that RLP improves performance on STEM reasoning benchmarks.

The consequence. There are two distinct concerns. First, domain transferability: RLP trains on diverse text (web crawl, academic papers) but is only tested on STEM benchmarks. We have no evidence about whether RLP improves performance on general natural language understanding (e.g., reading comprehension, summarization, translation), common-sense reasoning, code generation, safety judgments, or creative writing. The Wikitext-103 perplexity check (Appendix 12, Table S.8) shows that language modeling quality does not degrade — M_RLP achieves lower perplexity than M_base — but perplexity is a weak proxy for downstream task performance. A model could improve its next-token prediction accuracy (thus lowering perplexity) while becoming worse at tasks requiring abstract reasoning, factuality, or calibrated uncertainty.

Second, the verifiability confound: all evaluated benchmarks have clean, checkable answers. The information-gain reward r(c_t) = log p_theta(x_t | x_{<t}, c_t) - log bar{p}_phi(x_t | x_{<t}) is computed against the ground-truth next token from the training corpus. For math and science text, the "correct" next token is often unambiguous — there is typically one right continuation. For open-ended text (dialogue, opinion, narrative, poetry), the next token is one of many plausible continuations. RLP's reward prefers thoughts that increase the probability of the observed token, but in open-ended domains, the observed token is not uniquely correct — it's just what the author happened to write. A thought that correctly identifies the underlying intent but assigns high probability to a different-but-equivalent token would receive negative reward. This means RLP's signal may be systematically biased toward domains where the next token is highly constrained, and may actually penalize creative or diverse reasoning in open-ended domains. The paper does not discuss this tension.

What evidence exists in the paper. There is no evaluation on non-STEM benchmarks. The paper reports results on zero tasks outside the Math Avg and Science Avg categories shown in Tables 1–4 and Tables S.1–S.6, S.9. The scaling to NEMOTRON-NANO-12B-V2 (Table 2) and QWEN3-14B (Table S.3) uses the same benchmark suite. The corpus diversity experiment (Table 4) trains RLP on six different data types but evaluates all of them on the same STEM benchmarks — this tests whether training on diverse data still improves STEM scores, but does not test whether the diverse training improves non-STEM capabilities.

Mitigation status. The paper does not address this limitation or suggest that future work should evaluate on broader task families. It frames the STEM benchmark suite as sufficient to demonstrate reasoning improvement, and the claim of "domain-agnostic" pretraining augmentation (Section 5, final sentence) is supported only by corpus diversity on the training side, not task diversity on the evaluation side. A practitioner seeking to deploy RLP for a general-purpose language model would need to run their own evaluations on non-STEM tasks to determine whether the method helps or hurts those capabilities — the paper provides no guidance.

The Continuous Pretraining Baseline Is Potentially Weakened by Context Length and Data Mixture Choices

The assumption or constraint. The paper's strongest and most frequent comparison is between RLP and continuous pretraining (CPT) — M_CPT in Tables 1, 2, 4, and S.5. For these comparisons to be fair, M_CPT must represent the best achievable performance using standard next-token prediction on the same or equivalent data. Several design choices raise questions about whether this bar is set at the right height.

The consequence. If M_CPT underperforms due to suboptimal hyperparameters rather than an inherent limitation of next-token prediction, then RLP's reported advantages over CPT would overstate the methodological contribution. The specific concerns:

Context length mismatch. The base QWEN3-1.7B-BASE model supports a 32K context window, but M_CPT is trained with an 8K context length (Appendix 10). The paper addresses this with an ablation showing that M_CPT(32K) actually performs worse than M_CPT(8K) (Table S.5: Overall 30.83% vs. 32.90%). The explanation is that the DPT corpus contains mostly short documents, so longer context provides no benefit while complicating optimization. This is a reasonable defense, but it means the CPT baseline was trained under a hyperparameter choice (8K context) that may not be optimal in general — if a different data mixture or training recipe had been used, the 32K context might have been beneficial. Since RLP also operates at 2048 context length (Appendix 10), the comparison is between two methods both using shorter-than-native context windows, which makes it harder to generalize the finding to full-context training regimes.

Data mixture quality. M_CPT is trained on the same datasets used for RLP (DPT, Nemotron-Crossthink, etc.), but the paper does not explore whether CPT on higher-quality data — e.g., a larger, cleaner corpus of reasoning traces — could close the gap. The paper argues (Section 5) that "even with high quality reasoning data, RLP consistently outperforms CPT by a significant margin," pointing to the 8% advantage on Nemotron-Crossthink. But Nemotron-Crossthink is the same data used for both methods — it's not obvious that it represents the ceiling for what CPT can achieve. If, for example, CPT on a corpus of step-by-step solutions with explicit CoT annotation (akin to the data used for post-training SFT) could match RLP's performance, then RLP's advantage is not about RL vs. NTP but about the form of the training signal — which could potentially be replicated with better data curation.

No SFT-on-reasoning-traces baseline. Relatedly, the paper never compares RLP to a supervised baseline where the model is fine-tuned on thought-answer pairs (i.e., the same CoT traces that RLP generates, but treated as supervised targets rather than RL actions). If such a baseline were competitive with RLP, it would suggest that the RL mechanism is not essential — the benefit comes from exposing the model to reasoning traces, not from the information-gain reward structure. The paper does not address this alternative hypothesis.

What evidence exists in the paper. Table 4 shows that M_CPT trained on 6B tokens of Nemotron-Crossthink (35× more data than RLP) achieves Overall 38.04%, well below M_RLP's 43.36% on the same data with only 170M tokens. This is the strongest evidence that CPT's limitations are not purely a data volume issue. Table S.5 shows that 32K context CPT underperforms 8K context CPT on this data mixture, supporting the context length choice. However, neither result addresses the possibility that a different CPT recipe — different learning rate schedule, different data mixture, inclusion of explicit reasoning traces as targets — could substantially increase the CPT performance ceiling.

Mitigation status. The paper partially addresses the context length concern with the ablation in Table S.5, showing that the 8K context is the stronger of the two options for their specific data. The data quality concern is partially addressed by the Nemotron-Crossthink results — if a high-quality reasoning corpus can't make CPT competitive even with 35× more data, the gap is not easily explained by data quality alone. The absence of an SFT-on-reasoning-traces baseline is not addressed or acknowledged. A practitioner evaluating whether to adopt RLP would need to determine whether their specific CPT recipe (which may differ in data, context length, learning rate schedule, and model architecture) provides a stronger baseline than the paper's M_CPT — the paper does not provide enough experimental variation in the CPT configuration to answer this confidently.

The Method Has No Mechanism for Selective or Adaptive Thought Deployment

The assumption or constraint. RLP trains the model to generate a chain-of-thought of fixed length (2048 tokens in the default configuration) and then predicts the next token given that thought. The thought is generated at a randomly selected token per document, and all thoughts have the same length. There is no mechanism for the model to learn when to think, how long to think, or whether thinking is even needed at a given position. During training, the model always thinks at the random position; during evaluation, it never thinks (in the paper's setup). This creates a fundamental mismatch between the training objective (which rewards explicit reasoning) and the deployment usage (which is standard next-token prediction without reasoning).

The consequence. There are several downstream implications:

No difficulty-adaptive reasoning. Unlike test-time compute scaling methods that allocate more reasoning budget to harder problems, RLP treats all positions identically. The ablation on completion length (Figure 2b) shows that thought length strongly affects performance — 64-token thoughts degrade performance below the base model (Overall 11.50% vs. 34.03%), while 2048-token thoughts are optimal. But the model doesn't learn to choose thought length based on the context; length is a global hyperparameter set by the practitioner. If RLP were used at inference time with explicit CoT (which the paper does not evaluate), every prediction would incur the full 2048-token generation cost regardless of whether the position is predictable (e.g., the token "the" after "I went to") or genuinely requires reasoning (e.g., a mathematical derivation step).

No learned "skip-thinking" behavior. Because RLP always applies at the randomly selected position, the model is never trained to recognize when thinking is unnecessary. A model that has learned to think on every training example may develop a dependence on explicit reasoning that makes it less efficient at inference time — potentially requiring thought generation even for trivial predictions. Conversely, the paper's evaluation setup (greedy decoding without thoughts) suggests that the benefit is implicitly internalized, but without ablations comparing thought-augmented vs. thought-free inference for the RLP model, we don't know whether the internalized benefit is equivalent to the explicit reasoning benefit.

Fixed thought budget is arbitrary. The choice of 2048 tokens as the default thought length comes from the completion length ablation (Figure 2b), which shows diminishing returns beyond 2048 (at 4096, Overall improves by only +0.04 points). But this is a single global setting. Different domains, different positions within a document, and different difficulty levels might benefit from different thought budgets. A math problem might need 2048 tokens, while predicting the next word in a news article might need 128. RLP provides no mechanism for adapting the budget per-example.

What evidence exists in the paper. The completion length ablation (Figure 2b, Appendix Table S.1) provides the primary evidence that thought length matters and that too-short thoughts are harmful. The rollout count ablation (Figure 2a) shows that more parallel thoughts help up to G = 16. But there is no experiment that varies thought length or rollout count per example — all experiments use a fixed global setting. The paper also does not evaluate inference-time thought generation for RLP models, so we don't know whether the optimal inference-time thought length matches the training-time thought length, or whether thought generation at inference provides additional benefits beyond the implicit reasoning improvement.

Mitigation status. The paper does not address this limitation. It does not discuss adaptive thought budgeting, does not evaluate the model with explicit CoT at inference time, and does not explore whether RLP-trained models can learn to modulate their thinking based on context. This is a significant gap for a method that claims to "bring the core spirit of reinforcement learning — exploration — to the last phase of pretraining" (Section 1, Abstract): exploration is most valuable when it can be deployed selectively, and RLP provides no mechanism for selectivity. A practitioner would need to develop their own strategy for deciding whether and how much explicit reasoning to use at inference time with an RLP-trained model — the paper provides no guidance.

The FLOP-Matched and Token-Matched Comparisons Do Not Cover Downstream Inference Costs

The assumption or constraint. The paper's central efficiency claim is that RLP achieves better benchmark performance than continuous pretraining when matched for compute (FLOPs) or training tokens. The FLOP accounting (Appendix 12) computes RLP's training cost as the sum of input tokens processed and rollout tokens generated, and then gives M_CPT an equivalent budget (resulting in a 35× data advantage for CPT in the Nemotron-Crossthink comparison). This accounting is careful about training-time costs but entirely omits inference-time costs — both during evaluation (which is standard greedy decoding or Pass@1 sampling without thoughts) and during any potential future deployment that might use explicit reasoning.

The consequence. The training-time efficiency comparison is valid for the specific scenario the paper evaluates: a practitioner who trains a model, then deploys it with standard next-token prediction. In this scenario, RLP's training overhead (the 2.25× per-step slowdown, the 35× FLOP multiplier relative to input tokens) is a one-time cost that can be amortized over many inference queries. For high-volume deployments, the amortized cost might be negligible.

However, this accounting breaks down in two scenarios that the paper does not discuss:

If explicit reasoning is used at inference time. Suppose the RLP model is deployed with ongoing CoT generation — perhaps because the implicit reasoning improvement is insufficient for the hardest queries, or because the practitioner wants to use the thought capability that RLP explicitly trained. In this case, every inference call would cost significantly more than a standard model call (generating a 2048-token thought before predicting each next token). The 35× training-time FLOP multiplier would then be dwarfed by the per-query inference overhead, making the total cost much higher than the paper's accounting suggests.
If inference volume is low. For applications with relatively few queries (e.g., a research model evaluated periodically, or a model fine-tuned for specific tasks), the training-time cost cannot be amortized over many queries, and the 2.25× per-step slowdown directly translates to longer training time for equivalent performance. The paper's efficiency claims assume that training cost is the dominant consideration, which is true for large-scale production models but not for all use cases.

Additionally, the paper's FLOP accounting gives M_CPT a "rigorous upper bound" (Appendix 12) — meaning it assumes the maximum possible CPT cost within the FLOP budget — but the CPT baseline is not necessarily compute-optimal. If CPT were trained with a different learning rate schedule, batch size, or data mixture that converged faster, the FLOP-matched comparison might look different. The paper does not explore whether CPT could be made more compute-efficient with hyperparameter tuning optimized specifically for the FLOP-matched budget.

What evidence exists in the paper. Table S.7 (Appendix 12) provides the per-step training time comparison. The FLOP accounting is detailed in Appendix 12. The completion length ablation (Figure 2b) gives some indication of inference-time cost if thoughts were used — a 2048-token thought represents roughly 2048 additional forward passes at inference time. But the paper does not report inference latency, inference FLOPs, or total cost-of-ownership for an RLP-trained model deployed at scale.

Mitigation status. The paper is transparent about the training-time cost and provides the numbers needed for a practitioner to estimate total cost in their specific deployment scenario. It does not discuss inference-time cost because all evaluations use standard (thought-free) inference. The paper's implicit position is that RLP is a pretraining method whose cost should be compared to other pretraining methods, not to inference-time methods. This is a reasonable framing, but it means that the "efficiency" claims (4×, 35×, etc.) should be understood as training-time efficiency only, and a full lifecycle cost analysis — including any inference-time thought generation that a practitioner might choose to use — would require additional measurements that the paper does not provide.

RLP Has Not Been Shown to Work When Applied Earlier in Pretraining or Without a Sufficiently Capable Base Model

The assumption or constraint. All main experiments apply RLP to either a fully pretrained base model (QWEN3-1.7B-BASE, which completed the standard pretraining pipeline) or an intermediate checkpoint that is already 19.8T tokens into a 20T-token pretraining run (NEMOTRON-NANO-12B-V2). The paper frames this as "the last phase of pretraining" (Abstract) — RLP is introduced at a point where the model already has substantial linguistic and (implicit) reasoning capabilities, not from random initialization.

The consequence. This constrains the scope of the "RLP as a pretraining objective" claim. The paper's title is "RLP: Reinforcement as a Pretraining Objective," and the abstract states that RLP "brings the core spirit of reinforcement learning — exploration — to the last phase of pretraining." But the evidence only covers late-stage pretraining intervention. We do not know:

Whether RLP works from scratch. Could RLP be applied from random initialization? The information-gain reward requires the model to have some ability to predict next tokens — a randomly initialized model predicts essentially uniformly, so the information gain r(c_t) = log p_theta(x_t | x_{<t}, c_t) - log bar{p}_phi(x_t | x_{<t}) would be near zero regardless of thought quality. The reward signal would be uninformative and provide no learning gradient. RLP almost certainly cannot be used from scratch, which means it is not a full replacement for next-token prediction pretraining but rather a phase within it.
The minimum base model capability required. The paper provides one early-checkpoint experiment (Appendix 12, Table S.4): applying RLP to NEMOTRON-NANO-12B-V2 at 4T tokens (20% of the 20T budget) yields a 2× improvement (Overall 12.05% → 24.08%). This is a large relative gain but from a very low base. The Math Avg more than doubles (21.93% → 50.14%), suggesting that even a relatively weak base model can benefit. But this is still a model with 4T tokens of pretraining — far from random initialization. We don't know the shape of the curve: does RLP's effectiveness increase monotonically with base model capability, or is there a threshold below which the information-gain signal becomes too noisy?
Whether RLP can replace rather than augment CPT. All experiments apply RLP in addition to standard pretraining, not instead of a portion of it. Could the 1B tokens used for RLP on QWEN3-1.7B-BASE have been used more effectively if they were allocated to RLP earlier in training instead of to standard NTP? The paper does not study allocation of a fixed total pretraining budget between NTP and RLP phases — all comparisons are between NTP-only and NTP+RLP, not between different schedules that include RLP.

What evidence exists in the paper. The 4T-token early-checkpoint result (Table S.4) is the only experiment that varies the intervention point. It shows that RLP provides large relative gains even at 20% of full pretraining, but does not establish the trend across different intervention points or the minimum viable base model capability. The paper does not experiment with applying RLP at 1T, 8T, or 12T tokens to map out the relationship between base model strength and RLP benefit.

Mitigation status. The paper partially addresses this with the 4T-token experiment, noting that "while our strongest final results come from applying RLP later in pretraining, these findings indicate that RLP can already yield large gains when the model has seen only a small fraction of the standard pretraining budget" (Appendix 12). But "a small fraction" is still 4T tokens on a 12B-parameter model — an enormous amount of compute by any standard. The paper does not claim that RLP works from scratch or that it can replace early-stage pretraining, and its framing as "the last phase of pretraining" is consistent with the experimental evidence. A practitioner seeking to integrate RLP into their training pipeline would need to determine the optimal intervention point through their own experimentation; the paper establishes that late-stage intervention works and that earlier intervention (at 20% completion) also works, but does not provide a full characterization of the tradeoff curve between base model capability and RLP benefit.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper makes a specific, falsifiable challenge to one of the most entrenched assumptions in LLM training: that reinforcement learning for reasoning belongs exclusively at the end of the pipeline, after supervised fine-tuning. By demonstrating that a verifier-free, information-gain reward can systematically induce reasoning during pretraining — on arbitrary text, without curated answers or external verifiers — RLP opens the door to reconceptualizing the pretraining phase as an active reasoning curriculum rather than a passive next-token prediction exercise. This is not yet a paradigm shift (the method is demonstrated only in late-stage pretraining, not from scratch, and only on STEM benchmarks), but it is a strong reframing with paradigm-shift potential if subsequent work generalizes the findings.

What RLP resets in the conversation. Prior to this work, the field's default mental model was: pretraining builds statistical knowledge; post-training adds reasoning. RLP shows that these phases can overlap, and that the overlap is not merely possible but efficient — the 170M-token RLP run outperforming a 6B-token CPT run (Table 4, 35× data advantage for the baseline) is hard to explain unless the information-gain reward is extracting a fundamentally different (and more valuable) learning signal per token than next-token likelihood. This reframes the economic question: if a small RL phase during pretraining can substitute for a large volume of standard training, when should practitioners switch from NTP to RLP, and for how long?

Reconciling prior contradictions. The paper resolves a tension that has been building in the RL-for-reasoning literature. On one side, post-training RL (DeepSeek-R1, Open-Reasoner-Zero, etc.) showed that explicit reinforcement dramatically improves reasoning but required verifiable rewards and curated data, limiting applicability to narrow domains. On the other side, verifier-free RL methods (Yuan et al., 2024; Liu et al., 2025b; Zhao et al., 2025) removed the external verifier but were still post-training methods operating on curated data, and prior reinforcement pretraining (RPT, Dong et al., 2025) used sparse binary rewards that the paper shows underperform RLP's dense continuous signal by up to 20% (Table 3, FLOP-matched). RLP shows that the apparent tradeoff — "verifier-free OR domain-general, but not both" — was an artifact of method design, not a fundamental constraint. The key enabling insight is that the information-gain reward r(c_t) = log p_theta(x_t | x_{<t}, c_t) - log bar{p}_phi(x_t | x_{<t}) is simultaneously verifier-free (it requires only the observed token, not a correctness judgment) and domain-agnostic (the observed token exists in all text). This should accelerate work on self-supervised RL objectives for language modeling more broadly.

Directions that become more attractive. The paper makes verifier-free reinforcement learning at scale a credible research program. Prior work in this space was either theoretical (Han et al.) or limited to specific post-training domains. RLP provides a concrete recipe — EMA baseline, group-relative advantages, clipped surrogate on thought tokens — that is shown to work on web crawl, academic papers, and textbooks, not just curated reasoning data. This lowers the barrier for researchers who want to explore RL objectives during pretraining but were deterred by the complexity of verifier construction.

Directions that become less attractive. The paper's strong results with random token selection (one random token per document) and the absence of benefit from KL anchoring (Figure 2c) suggest that the research energy spent on token-selection heuristics (as in RPT's entropy-based filtering) and KL regularization (as in standard RLHF) may be misallocated for self-supervised RL objectives. If the reward is genuinely self-stabilizing — grounded in data likelihood rather than a learned reward model — then complex regularization and filtering may be unnecessary complexity rather than essential safeguards. Similarly, the finding that lookahead-style RL (ReST^EM, Appendix K) degrades revision model performance suggests that iterative on-policy self-improvement is not automatically beneficial and that simpler, offline RL approaches (like RLP's single-stage policy gradient) may be more robust.

The unresolved "why now" question. A critical caveat: the paper demonstrates that RLP works when applied to already-competent base models (QWEN3-1.7B-BASE after full pretraining; NEMOTRON-NANO-12B-V2 at 19.8T/20T tokens). There is no evidence that RLP would work from random initialization or early in pretraining. The information-gain reward requires that bar{p}_phi be a reasonably good predictor — otherwise, the difference between reasoned and no-think log-likelihoods is dominated by noise. This means RLP, as currently formulated, is likely a late-pretraining amplifier rather than a full pretraining objective. The paper's title ("Reinforcement as a Pretraining Objective") is therefore slightly aspirational — what RLP demonstrates is more precisely "reinforcement as a late-stage pretraining objective." Whether it can be pushed earlier is a key open question.

Follow-Up Research This Work Enables

Characterizing the minimum viable base model capability for RLP to be effective. The paper provides one early-intervention data point (NEMOTRON-NANO-12B-V2 at 4T tokens, Table S.4) showing large relative gains from a low absolute base (Overall 12.05% → 24.08%). But we don't know the shape of the curve: at what point during pretraining does the information-gain signal become informative enough to drive learning? A systematic study applying RLP at 1%, 5%, 10%, 25%, 50%, and 100% of total pretraining tokens — with FLOPs held constant across intervention points — would reveal whether there is a threshold below which RLP provides no benefit (because bar{p}_phi is too weak) or even degrades performance (because noisy rewards reinforce spurious patterns). The finding that 64-token completions harm performance (11.50% vs. 34.03% base, Figure 2b) suggests that insufficient reasoning capacity can be worse than none — the same may apply to insufficient base capacity. This experiment would directly test whether RLP is a replacement for, or a complement to, early-stage next-token pretraining, and would inform pipeline design for practitioners.

Evaluating RLP on non-STEM reasoning and generation tasks. The paper's entire evaluation suite is math and science benchmarks with verifiable answers. This leaves open the central question for a method claiming "domain-agnostic" improvements: does RLP improve reasoning in domains where the correct continuation is not uniquely determined? A natural follow-up would evaluate RLP-trained models on: (a) reading comprehension (e.g., SQuAD, NarrativeQA) to test whether improved next-token prediction translates to better understanding; (b) code generation (HumanEval, MBPP) where correctness is verifiable but the reasoning patterns differ from math; (c) multi-step planning (e.g., AlfWorld, WebArena) requiring action sequences rather than tokens; (d) open-ended generation with human evaluation (e.g., summarization, dialogue coherence). The paper's own theory (Proposition 1: expected reward equals CE reduction) predicts that RLP should help wherever next-token prediction accuracy correlates with task performance. But the theory also suggests a boundary: in domains where the observed token is merely one of many valid continuations, RLP's reward may penalize creative or diverse reasoning by rewarding only thoughts that increase probability of the specific training-set token. Evaluating on open-ended generation tasks would test this tension directly.

Combining RLP with explicit chain-of-thought at inference time. The paper evaluates RLP only with standard (thought-free) inference, leaving the model's explicit reasoning capability untested. A crucial follow-up experiment: compare RLP-trained models using (a) standard greedy decoding, (b) explicit CoT at inference time with the same < /think> prompt used during training, (c) explicit CoT with varying thought budgets (128, 512, 2048, 4096 tokens), and (d) a difficulty-adaptive strategy that decides whether to think based on some proxy (e.g., initial prediction entropy, similar to RPT's token selection but at inference time). The key question: does RLP training make explicit reasoning more effective than it would be for a baseline model? If so, RLP's value extends beyond the implicit reasoning baked into the weights — it also improves the model's capacity for on-demand, controllable reasoning. If not (i.e., explicit CoT helps the baseline model as much as the RLP model), then RLP's benefit is purely an architectural optimization that doesn't generalize to controllable reasoning. The paper's thought trace analysis (Appendix 12) shows that RLP generates coherent thoughts during training, but never tests whether those thoughts would improve inference-time predictions. This experiment is the natural bridge between RLP-as-pretraining and RLP-as-reasoning-amplifier.

Investigating the mechanism: is RLP teaching genuine reasoning or surface statistical patterns? The paper provides ablation-level evidence that RLP improves benchmark accuracy, but does not isolate why. A focused mechanistic study could: (a) train probes on the model's hidden states at various layers to decode whether RLP models develop intermediate representations that correspond to logical steps, intermediate quantities, or domain-specific concepts that baseline models do not; (b) test whether RLP models are more robust to adversarial perturbations of the input (e.g., permuting sentences, injecting irrelevant distractors) that would break surface-level statistical patterns but leave underlying reasoning intact; (c) evaluate on counterfactual reasoning tasks where the correct answer contradicts statistical co-occurrence patterns from pretraining (e.g., "If dogs could fly, where would they build nests?"); (d) measure whether the information gain of a thought correlates with its causal contribution — by intervening to replace the thought with a random or empty one and measuring the effect on prediction. If RLP primarily improves surface-level prediction accuracy without genuine reasoning transfer, it would still be practically useful (lower perplexity, better benchmarks) but would not represent the "independent thinking behavior" the abstract claims. If it does transfer to counterfactual and causal reasoning, the mechanism is deeper.

Stress-testing the EMA baseline design space. The EMA baseline (τ = 0.999, updated after each optimizer step) is a central but relatively unexplored component. The paper's ablation (Table S.6) sweeps τ across {0.99, 0.995, 0.999, 0.9995} and finds a peak at 0.999, but doesn't explore: (a) different update schedules (e.g., updating the EMA every K steps rather than every step, which would change the effective lag); (b) different initialization strategies (the paper lazily initializes φ ← θ on the first batch — what if the baseline were initialized from a different checkpoint, or warmed up with standard NTP before RLP begins?); (c) asymmetric architectures (the paper uses an identical architecture for π_θ, p_θ, and bar{p}_phi — could a smaller, cheaper baseline model provide similar-quality counterfactuals at lower cost?); (d) the interaction between EMA τ and training duration (with longer training, the optimal τ may shift because the model's rate of improvement changes). This experiment is important because the EMA baseline is the only source of "counterfactual pressure" in RLP — without it, the reward is undefined. Understanding its sensitivity would inform both theoretical analysis (is there an optimal lag schedule derived from the model's improvement rate?) and practical deployment (can we make the baseline cheaper without degrading reward quality?).

Scaling RLP to larger models and longer RLP phases. The paper demonstrates RLP at 1.7B, 12B, and 14B parameters, with RLP phases of 170M–1B tokens. The natural scaling question: does RLP's benefit increase, decrease, or plateau as model size grows? The paper's results hint at diminishing returns in absolute terms — the 1.7B model gains +5.71 points from RLP (Table 1, M_RLP vs. M_base), the 14B model gains +4.34 points (Table S.3), and the 12B model gains +18.51 points (Table 2, though this is not apples-to-apples due to different base model strengths). But these are single data points with different RLP budgets. A systematic scaling study at 1B, 3B, 7B, 13B, and 30B+ parameters, with RLP budget fixed in FLOPs (not tokens), would reveal whether there is a scaling law for RLP analogous to Chinchilla for NTP. Equally important: what happens when the RLP phase is extended? The paper's longest RLP run is 1B tokens — but if RLP is truly more data-efficient than NTP, applying it for 10B, 100B, or 1T tokens might yield disproportionate gains. The completions-length ablation (Figure 2b) suggests that the thought budget might need to scale with RLP duration if the model learns to use longer thoughts for more sophisticated reasoning. This experiment would help determine whether RLP is a one-time "reasoning injection" (applied briefly at the end of pretraining) or a sustained phase that should occupy a significant fraction of the total training budget.

Practical Applications and Downstream Use Cases

Cost-efficient pretraining for reasoning-heavy model families. For organizations training LLMs from scratch — particularly those targeting STEM, code, or analytical applications — the paper provides an actionable recipe to reduce total pretraining cost. The NEMOTRON-NANO-12B-V2 result (Table 2: 43% relative improvement with RLP applied to only 0.125% of the pretraining data budget) suggests a concrete pipeline: pretrain with standard NTP to ~95–99% of the total budget, then switch to RLP for the final 0.1–1% using a diverse pretraining corpus (DPT). The 2.25× per-step slowdown during the RLP phase is negligible when amortized over the full pretraining run — if RLP occupies 1% of steps at 2.25× cost, the total training time increases by only ~1.25%. In exchange, the resulting model achieves benchmark scores that would require either (a) a substantially larger model, (b) significantly more pretraining data, or (c) more aggressive post-training — each of which is far more expensive than the RLP overhead. For a team training a 70B-parameter model on 15T tokens, replacing the final 150B tokens of NTP with 1B tokens of RLP could yield reasoning improvements comparable to (based on extrapolation from the 12B results) training an additional 3–5T tokens of NTP — a 20–50× FLOP savings.

Improving data efficiency for domain-specific model adaptation. The corpus diversity experiment (Table 4) shows that RLP extracts useful reasoning signals from web crawl, academic papers, and textbooks — data that is widely available and cheap to collect. For organizations fine-tuning models for specialized domains (legal analysis, medical reasoning, financial modeling), RLP offers a way to induce domain-specific reasoning during continued pretraining without requiring expensive human-annotated reasoning traces or verifiable reward functions. The procedure would be: take a base model, assemble a corpus of domain-relevant text (legal opinions, medical case studies, financial reports), and apply RLP for a few hundred million tokens. The paper's results suggest that even 170M tokens of RLP on ACAD (academic papers) lifts the overall STEM benchmark average from 34.19% to 41.71% (Table 4), and while this is evaluated on STEM rather than the domain of adaptation, the mechanism — information-gain reward for predictive thinking — should transfer to any domain where coherent reasoning patterns exist in the text. The key advantage over supervised fine-tuning on reasoning traces is that RLP requires no annotation — the reward is computed automatically from the model's own predictions and the existing text.

Stronger base models for post-training pipelines. The paper's compounding result — that RLP's gains survive and amplify through SFT + RLVR — has direct implications for organizations that invest heavily in post-training. Current state-of-the-art reasoning models (DeepSeek-R1, OpenAI o-series) apply extensive RLVR on top of SFT starting from a strong base model. RLP can be integrated upstream of this pipeline: apply RLP before SFT, producing a base model that already has active reasoning dispositions. The paper's QWEN3-1.7B results (Table 1) show that M_RLP+Post achieves 42.51% vs. M_base+Post's 39.34% — a +3.17 point absolute advantage after identical post-training. For a production reasoning model where every point matters (e.g., competitive coding, mathematical olympiad, scientific research assistance), this is a significant margin. Moreover, the fact that the gains are largest on the hardest benchmarks (AIME25, MMLU-Pro, GPQA) suggests RLP is particularly valuable for pushing the frontier of reasoning capability rather than routine tasks. A team aiming to win benchmarks like AIME or GPQA could adopt RLP as a standard pre-SFT step.

Enabling reasoning capabilities in smaller, deployment-efficient models. The QWEN3-1.7B results (Table 1: 36.03% overall for M_RLP, comparable to some larger models' base performance) suggest that RLP can help close the gap between small models and their larger counterparts on reasoning tasks. For on-device deployment scenarios (mobile, edge, embedded) where model size is constrained by memory and latency, RLP offers a way to maximize reasoning capability per parameter: train the largest model that fits the hardware budget, then apply RLP to induce reasoning that would normally require a model 2–5× larger. This is especially relevant as on-device reasoning becomes a priority (voice assistants, code completion in IDEs, real-time translation). The paper's FLOP-matched comparison (Table 4, M_RLP on 170M tokens outperforming M_CPT on 6B tokens) suggests that the efficiency gain from RLP is not merely about trading training FLOPs for inference FLOPs — it produces a model that is genuinely more capable per unit of compute, which matters regardless of whether the constraint is training or deployment.

When to Prefer This Method

The paper explicitly positions RLP against two alternatives: continuous pretraining (CPT) on the same data with next-token prediction loss, and Reinforcement Pre-training (RPT) with sparse binary rewards. It does not position RLP against post-training-only RL methods (RLVR, RLHF) because RLP is a pretraining-phase intervention — the comparison is "RLP + post-training" vs. "standard pretraining + post-training," not "RLP vs. post-training RL." The following decision rules are grounded in the paper's specific findings:

Prefer RLP over CPT when the goal is to maximize reasoning benchmark performance under a fixed compute budget for continued pretraining. The paper's strongest evidence is Table 4: M_RLP on 170M tokens of Nemotron-Crossthink achieves Overall 43.36% vs. M_CPT on 6B tokens (35× more data, equal FLOPs) at 38.04%. If you are doing continued pretraining on a general or reasoning-heavy corpus and care about downstream STEM reasoning, RLP provides better performance per FLOP than NTP. The advantage is largest when the base model is already reasonably capable (late-stage pretraining) and when the corpus contains heterogeneous text (RLP handles diversity better than CPT, which can regress on some benchmarks — e.g., MMLU dropping from 50.08% to 41.95% under M_CPT in Table 1).
Prefer RLP over RPT when you want a self-contained method that does not require an auxiliary model for token selection and that maximizes reasoning gains per reward-applied token. Under FLOP-matched conditions with equalized reward-token counts (Table 3, bottom rows), RLP outperforms RPT by 20.1% relative in Overall (+7.18 points absolute). Under token-matched conditions (which favor RPT because it applies rewards to more tokens per document), RLP still leads by 4.0% relative. RLP's verifier-free, continuous reward avoids RPT's dependence on binary correctness signals, making it applicable to text domains where correctness is ambiguous (most of the web). Use RPT only if you have a specific reason to prefer sparse binary rewards — e.g., your application has a natural correctness criterion and you want compatibility with existing binary-reward RL infrastructure.
Prefer standard CPT over RLP when the training infrastructure cannot accommodate reinforcement learning (policy gradients, EMA teacher, rollout generation) or when the additional implementation complexity is not justified by the expected gain. RLP requires maintaining a separate EMA model in memory, generating and scoring 16 rollouts of 2048 tokens each per batch, and implementing a clipped surrogate objective — all of which add engineering complexity over standard NTP training. If the deployment scenario involves limited engineering resources, or if the target domain is known to be well-served by standard pretraining (e.g., factual knowledge tasks where reasoning is less critical), the ~7–8% relative post-training advantage from RLP may not justify the implementation cost.
The paper does NOT provide evidence to prefer RLP over (a) supervised fine-tuning on high-quality reasoning traces (the baseline is not tested), (b) applying RLVR directly to the base model without RLP (not tested), (c) using RLP from random initialization or early in pretraining (likely infeasible based on the method's requirements, and not tested before 4T tokens on a 12B model), or (d) using explicit chain-of-thought at inference time with an RLP-trained model (not evaluated). These are open questions that a practitioner would need to resolve through their own experiments — the paper establishes RLP as a strong method within its evaluated scope but does not provide evidence of superiority over these untested alternatives.