Reinforcement Pre-Training

ArXiv: 2506.08007

🎯 Pitch

What if language models earned a reward for every token they get right? By turning next-token prediction into a reasoning task with a simple correct/incorrect reward signal, RPT boosts hard-token accuracy and matches a 2.3× larger model’s performance—all using unlabeled text alone.

1. Executive Summary

This paper introduces Reinforcement Pre-Training (RPT), a new scaling paradigm that reframes next-token prediction as a reasoning task trained via reinforcement learning with verifiable rewards — specifically, the model generates chain-of-thought reasoning before predicting each token and receives a binary reward based on whether the prediction matches the ground-truth continuation from the pre-training corpus itself (a prefix-matching reward against byte-level token boundaries). Using DeepSeek-R1-Distill-Qwen-14B as the base model and OmniMATH as the pre-training corpus, RPT improves next-token prediction accuracy consistently across difficulty levels — elevating hard-token accuracy from 20.4% to 23.8% — while matching the language modeling performance of a significantly larger model (R1-Distill-Qwen-32B, ~2.3× more parameters). RPT also yields stronger downstream performance: zero-shot accuracy on MMLU-Pro rises from 68.9% to 71.1%, and RL fine-tuning on out-of-domain tasks achieves a higher ceiling (58.3% vs. 52.7%), establishing that RL-based pre-training generalizes beyond its mathematics-focused training corpus only when the base model already possesses foundational reasoning capabilities into which RPT can further incentivize deliberative next-token inference.

2. Context and Motivation

The Core Problem: Pre-Training and Reinforcement Learning Operate in Separate Paradigms

The fundamental tension this paper addresses is a structural mismatch between how large language models are pre-trained and how they are ultimately refined. The dominant pre-training paradigm — next-token prediction (NTP) — treats language modeling as a supervised learning problem: the model sees a context and is trained via maximum likelihood to reproduce the ground-truth next token from the training corpus. This objective is extraordinarily scalable because it requires no labeled data beyond the raw text itself, and it has driven the empirical success of models from GPT-3 to Llama to DeepSeek. However, it trains the model to produce tokens through what is essentially pattern recognition — learning statistical correlations between contexts and continuations — rather than through deliberative reasoning about why a particular token should follow.

The post-training paradigm — reinforcement learning, whether from human feedback (RLHF) or with verifiable rewards (RLVR) — operates on a fundamentally different principle: the model generates outputs, receives feedback on their quality, and is updated to maximize that feedback signal. This approach demonstrably improves reasoning, alignment, and task-specific performance (Ouyang et al., 2022; Jaech et al., 2024; Guo et al., 2025). But it suffers from a critical scalability limitation: it requires labeled data with verifiable answers (in the case of RLVR) or human preference judgments (in the case of RLHF), both of which are expensive to obtain at web scale. The result is that RL is confined to the post-training stage — applied to a narrow slice of curated data after the bulk of learning has already occurred — rather than being integrated into the primary knowledge acquisition phase.

This gap matters for several reasons the paper identifies, both explicitly and implicitly:

• Missed opportunity for foundational reasoning. If RL strengthens reasoning capabilities, applying it only at post-training means the model spends the vast majority of its training compute (during pre-training) learning correlations without explicit incentives to reason. Any reasoning capabilities that emerge from NTP pre-training are byproducts, not direct training targets. RPT asks: what if we incentivized reasoning from the very beginning, on every token, across the entire corpus?

• Capability ceiling from pre-training objective mismatch. When a model is pre-trained via NTP and then fine-tuned via RL, there is an objective gap: the pre-training loss function (cross-entropy) and the fine-tuning loss function (expected reward) optimize fundamentally different things. The pre-trained representations may not be optimally structured for the RL phase to build upon. By aligning the pre-training objective with RL from the start, RPT aims to minimize this gap — a hypothesis the paper tests empirically in its RL fine-tuning experiments (Section 4.3, Table 2).

• Inference-time reasoning patterns are learned late. Models like DeepSeek-R1 (Guo et al., 2025) demonstrate that chain-of-thought reasoning can be dramatically improved through RL at the post-training stage. But this reasoning capability is acquired from a relatively small set of verifiable math and code problems. RPT proposes that similar reasoning patterns — hypothesis generation, self-critique, exploration of alternatives — can be learned across all tokens in a general corpus, not just on specially structured problem-answer pairs.

Why This Problem Is Important

The significance of bridging pre-training and RL operates on multiple levels:

Theoretical significance: rethinking what pre-training is. The NTP objective has been so successful that it is often treated as synonymous with language model pre-training itself. RPT challenges this equivalence by proposing that NTP can be reframed not as a supervised learning task but as a reasoning problem with verifiable outcomes. Under this reframing, every token in a corpus becomes a miniature RL environment: the context is the state, the model's chain-of-thought plus prediction is the action, and the ground-truth next token provides the reward signal. This is a conceptual shift with deep implications — it suggests that the scalability of NTP (deriving training signal from raw text) and the effectiveness of RL (incentivizing deliberative behavior) are not fundamentally opposed but can be unified through a clever reformulation of the task itself.

Practical significance: scaling RL to the web. The most immediate practical bottleneck RPT addresses is data scarcity for RL. RLVR (Lambert et al., 2025) requires question-answer pairs with verifiable answers — math problems with ground-truth solutions, code with unit tests, formal reasoning tasks with known conclusions. These datasets are small relative to the trillions of tokens used in pre-training. RLHF (Ouyang et al., 2022) requires human preference data, which is expensive and difficult to scale. RPT sidesteps both limitations: the reward signal is the correctness of the predicted next token, which is always available for any text in any corpus. The paper is explicit that this "transforms the vast, unannotated text data typically used for next-token prediction into a massive dataset for general-purpose RL" (Section 1). If RPT works as claimed, it means RL can be applied at the scale of the entire web — a fundamentally new capability.

Economic and research implications. The scaling analysis in Section 4.2 (Figure 5) suggests that RPT's next-token prediction accuracy follows a power-law relationship with training compute ($R^2$ values of 0.989–0.997 across difficulty levels). This positions RPT as a potential new scaling axis alongside model size and data quantity. If these scaling trends hold at larger scales, organizations making pre-training compute investments would need to consider not just how much to train but with what objective — NTP or RPT — as a first-order decision affecting return on compute.

Where Prior Approaches Fall Short

The paper identifies specific limitations across several strands of prior work:

Standard next-token prediction is purely correlational. NTP pre-training maximizes $\sum_t \log P(x_t | x_{<t}; \theta)$. This encourages the model to learn whatever statistical regularities in the data help predict the next token — which can include genuine understanding but can equally include surface-level patterns, memorized sequences, and brittle correlations. The model is never explicitly required to explain its prediction or to verify that it makes sense given the broader context. The paper's Table 1 makes this concrete: R1-Distill-Qwen-14B under standard NTP achieves only 20.43% accuracy on hard tokens, despite being a capable reasoning model when explicitly prompted to think. The gap between what the model can do (when prompted to reason) and what it does do (when predicting directly) is precisely the gap RPT aims to close.

RLVR is domain-specific and data-limited. RLVR (Lambert et al., 2025) has shown impressive results for mathematical reasoning (DeepSeek-R1, Guo et al., 2025) and code generation, but it is fundamentally constrained by the availability of verifiable problems. The OmniMATH dataset used in this paper contains 4,428 problems — a large math dataset by RL standards, but minuscule compared to the trillion-token corpora used for NTP pre-training. This limitation means RLVR cannot serve as a general-purpose pre-training method; it can only enhance specific skills on specific data distributions.

RLHF relies on learned reward models susceptible to hacking. RLHF (Ouyang et al., 2022) replaced rule-based rewards with learned reward models trained on human preferences, enabling RL on open-ended generation where ground-truth answers don't exist. But learned reward models introduce their own problems: they can be exploited (reward hacking), they require expensive human annotation, and their quality degrades as the model's output distribution shifts during training. RPT's use of a rule-based, intrinsic reward (does the prediction match the corpus continuation?) avoids both the annotation cost and the reward hacking vulnerability. The paper explicitly positions this as an advantage: "the use of direct, rule-based reward signals inherently minimizes the risk of reward hacking often associated with complex, learned reward models" (Section 1).

Quiet-STaR (Zelikman et al., 2024) — the closest prior work. The paper identifies Quiet-STaR as "the most relevant work" (Section 5). Quiet-STaR also trains models to generate rationales before next-token prediction, using a helpfulness-based reward: rationales that increase the probability of the correct next token receive positive reinforcement. However, the paper identifies a critical vulnerability in this approach: "The helpfulness-based reward tends to be hacked by repeating the target token in the generated rationale, where the shortcut potentially harms the model." In other words, the model can learn to generate rationales that simply include the correct token as a signal to itself (e.g., "...the next word is probably 'cat'..."), inflating the reward without engaging in genuine reasoning. RPT's reward design — a binary correctness reward based on the output prediction, not the shift in probability — eliminates this shortcut. The model only gets credit if its final boxed prediction matches the ground truth, regardless of what its rationale says.

The pre-training/post-training objective gap is underexplored. While the field has extensively studied how to improve pre-training (scaling laws, data curation, architecture design) and post-training (RLHF, RLVR, instruction tuning) separately, there has been relatively little work on aligning the two phases. The paper's RL fine-tuning experiment (Section 4.3) provides evidence that this gap matters: when the R1-Distill-Qwen-14B baseline is subjected to continual NTP training on the same OmniMATH corpus before RL, its downstream RLVR performance collapses from 52.7% to 13.0% (Table 2). This dramatic degradation suggests that the representations learned under NTP become actively less suitable for RL-based fine-tuning when pushed further in the NTP direction — a striking finding that underscores the importance of objective alignment.

How This Paper Positions Itself

The paper's positioning can be understood along three dimensions:

As a unification of pre-training scale and RL effectiveness. The central claim is not that RPT is a better NTP method or a better RL method in isolation, but rather that it dissolves the distinction between pre-training and RL for language modeling. Every token becomes a mini RL problem; RL becomes the pre-training objective itself. This is a fundamentally different stance from prior work that treats RL as something you do after pre-training on a carefully curated subset of data. The paper's title — "Reinforcement Pre-Training" — is deliberately provocative in this sense, asserting that RL can and should be the pre-training paradigm, not just a post-training refinement.

As building on, not replacing, existing infrastructure. Importantly, RPT does not require new data, new model architectures, or new reward design for each domain. It uses the same web-text corpus as NTP, the same transformer architecture, and the same GRPO algorithm (Guo et al., 2025) as post-training RL. What changes is the framing of the task — from "predict the next token" to "reason about what the next token should be, then predict it, and be judged on correctness." This minimal modification means RPT can be adopted within existing training pipelines. The paper uses the verl library (Sheng et al., 2024) and vllm for inference — standard RL infrastructure — and applies standard RL algorithms (GRPO, on-policy rollouts with $G=8$ responses per context).

As an initial empirical demonstration, not a solved problem. The paper is candid about its limitations (Section 6): experiments are on a 14B model, the pre-training corpus is mathematical rather than general-domain, and training is initialized from a reasoning model (R1-Distill-Qwen-14B) rather than a base model. These choices are pragmatic — initializing from a model that already has basic reasoning capabilities avoids the cold-start problem of teaching a base model to generate coherent chain-of-thought from scratch — but they mean the paper's results should be understood as a proof of concept rather than a demonstration of RPT at the frontier of scale and domain diversity. The scaling curves in Figure 5, while encouraging, span a relatively narrow compute range (from approximately $10^{20}$ to $10^{21}$ FLOPs) and are fit to only six data points. The $R^2$ values are high, but this is a local fit; whether the power-law relationship holds at much larger scales is an open question the paper explicitly delegates to future work.

As part of a broader shift toward reasoning-centric pre-training. RPT can be seen as part of a emerging research direction that questions whether NTP — for all its empirical success — is the optimal way to extract understanding from text. Works like Quiet-STaR (Zelikman et al., 2024) and the various "think before you speak" approaches share the intuition that models should learn to reason about tokens, not just predict them. RPT's distinctive contribution within this landscape is the argument that RL, not supervised learning, is the right mechanism for incentivizing this reasoning — because RL directly optimizes for correct outcomes rather than for matching a teacher distribution. The paper does not claim that RPT makes NTP obsolete; rather, it offers RPT as a complementary or alternative scaling paradigm whose relative benefits may grow as models and datasets scale.

3. Technical Approach

3.1 Reader Orientation

This paper describes a reinforcement learning system that pre-trains a 14B-parameter language model by having it generate chain-of-thought reasoning before predicting each next token, then rewarding it only when the prediction matches the ground-truth continuation from the actual text corpus. The problem it solves is the objective mismatch between standard next-token prediction pre-training (supervised learning on correlations) and downstream reinforcement learning fine-tuning (reward maximization on outcomes) — RPT dissolves this distinction by turning every token in a pre-training corpus into a miniature RL problem with an intrinsic, verifiable reward signal derived from the corpus itself.

3.2 Big-Picture Architecture (Diagram in Words)

The RPT system has six major components that operate in a loop:

1. Pre-training corpus (OmniMATH) — a collection of 4,428 mathematical documents providing raw text from which contexts and ground-truth continuations are extracted.
2. Token-level data filter — a preprocessing step that uses a small proxy model (DeepSeek-R1-Distill-Qwen-1.5B) to compute the entropy of the top-16 next-token predictions at each position, filtering out low-entropy (easily predictable) tokens so RPT focuses training compute on challenging positions.
3. Base language model (DeepSeek-R1-Distill-Qwen-14B, call it $\pi_\theta$) — the model being trained, which already possesses basic chain-of-thought reasoning capabilities from its R1 distillation.
4. On-policy rollout generator — for each context, the current model $\pi_\theta$ generates $G = 8$ independent reasoning trajectories, each containing a chain-of-thought reasoning sequence followed by a final boxed prediction for the next token.
5. Prefix-matching reward verifier — a deterministic, rule-based function that compares each prediction's byte sequence against the ground-truth continuation from the corpus, awarding a reward of 1 for an exact prefix match at a valid token boundary and 0 otherwise.
6. GRPO (Group Relative Policy Optimization) updater — the RL algorithm that uses the group of $G$ rollouts per context to compute advantage estimates and update the model parameters $\theta$ to maximize expected reward.

Information flows as follows for a single training step: a context $x_{<t}$ is sampled from OmniMATH → the model generates $G = 8$ reasoning-and-prediction trajectories → the prefix-matching verifier checks each final boxed prediction against the ground-truth continuation $x_{\geq t}$, producing binary rewards → GRPO uses these within-group rewards to compute relative advantages and update the model parameters → the updated model becomes the new $\pi_\theta$ for the next iteration. This loop runs for 1,000 training steps with a batch size of 256 contexts, meaning approximately 256,000 distinct token positions are trained on.

3.3 Roadmap for the Deep Dive

• First, the formal definition of the next-token reasoning task (Section 3.1), because it defines what the model must produce and establishes the input-output specification that all subsequent components depend on.
• Second, the prefix-matching reward function (Section 3.2, Equation 3), because it is the objective signal driving all learning — understanding what constitutes a correct prediction and why byte-level matching with token-boundary validation is necessary.
• Third, the GRPO training objective (Section 3.2, Equation 4), because it defines how the reward signal is translated into parameter updates via on-policy reinforcement learning with group-based advantage estimation.
• Fourth, the token-level data filtering strategy (Section 3.3), because it determines which tokens receive training compute — a critical design choice that biases RPT toward harder, more reasoning-intensive predictions.
• Fifth, the full training configuration — base model initialization, hyperparameters, prompt template, rollout and extraction mechanisms — because these concrete details make the system reproducible and reveal implicit design decisions (e.g., using R1-Distill-Qwen-14B rather than a standard base model, the choice of $G = 8$ rollouts, the dynamic sampling strategy after step 500).

3.4 Detailed, Sentence-Based Technical Breakdown

This is a systems and methodology paper whose core idea is that next-token prediction can be reframed as a reinforcement learning problem where the model generates explicit reasoning before predicting and receives a binary correctness reward — transforming raw text into an RL training environment without requiring external annotations.

The Next-Token Reasoning Task

Standard next-token prediction treats language modeling as maximum-likelihood estimation: given a context $x_0, x_1, ..., x_{t-1}$, the model directly outputs a probability distribution over the vocabulary and is trained via cross-entropy to maximize the probability of the ground-truth token $x_t$. The model never explicitly articulates why it thinks $x_t$ should follow — it learns implicit statistical associations between contexts and continuations.

RPT replaces this with a two-stage generation process. For each position $t$ in the training corpus, the model $\pi_\theta$ receives the prefix $x_{<t}$ as context and must generate:

1. A chain-of-thought reasoning sequence $c_t$ — an arbitrary-length sequence of tokens where the model explores hypotheses, weighs alternatives, reflects on structural cues, and articulates its reasoning about what token should come next.
2. A final prediction $y_t$ — extracted as the content inside the last \boxed{} following the special token response in the generated output.

The combined output is denoted $o_t = (c_t, y_t)$, where $o_t \sim \pi_\theta(\cdot \mid x_{<t})$.

What the model actually generates in a single rollout. The model starts from the prompt template (described in Appendix D, Table 10) which wraps the context in formatting instructions and explicitly asks the model to "reason step by step to find the most probable next token as the final answer, and enclose it in \boxed{}." The model then autoregressively generates tokens — typically several hundred to several thousand tokens of reasoning — before outputting the special token response followed by its final prediction inside \boxed{}. The system extracts the boxed content as $y_t$ via string parsing. The reasoning sequence $c_t$ is everything before this extraction point. The model is free to use any reasoning patterns — the examples in Table 4 and Table 11 show it engaging in brainstorming, alternative consideration, self-correction, hypothesis testing, and reflection on structural context like markdown formatting.

What makes this a "reasoning task" rather than a generation task. The crucial distinction is that the model is judged on the correctness of its final answer, not on the quality or coherence of its reasoning chain. The paper never provides supervision on what constitutes good reasoning — the model must discover effective reasoning strategies purely through the reward signal on $y_t$. This is exactly the same structure as mathematical reasoning with RLVR: the model generates a chain-of-thought and a final answer, and only the final answer is verified. By importing this structure into pre-training, RPT turns every token position into a miniature reasoning problem where the "question" is the context and the "answer" is the next token.

Why this task design matters. The paper positions next-token reasoning as the mechanism that makes RL scaling possible at pre-training scale. Standard NTP cannot be trained with RL because there is no action to evaluate — the model outputs a distribution, not a discrete choice that can be checked against ground truth. Next-token reasoning solves this by making the model's prediction an explicit, extractable action (the boxed token) that can be verified against the corpus. The reasoning chain $c_t$ serves as the model's "private workspace" — it is generated but not directly scored, allowing the model to allocate variable computation (more tokens of thought) to harder next-token decisions. The paper explicitly connects this to inference-time scaling: "the internal reasoning process during pre-training effectively allows the model to allocate more 'thought' or computational effort to each prediction step, akin to a form of inference-time scaling applied at training time for each token."

The Prefix-Matching Reward Function

Once the model generates $y_t$ (the extracted boxed prediction), the system must determine whether that prediction is correct. The reward function is defined in Equation 3 of the paper:

$r_t^i = \begin{cases} 1 & \text{if } \bar{y}_t^i = \bar{x}_{\geq t}[1 : l] \text{ and } l \in \mathcal{L}_{\text{gt}} \\ 0 & \text{otherwise} \end{cases}$

where $\bar{y}_t^i$ is the byte sequence of the $i$-th prediction, $\bar{x}_{\geq t}$ is the byte sequence of the ground-truth continuation starting from position $t$, $l$ is the byte length of the prediction, $\bar{x}_{\geq t}[1:l]$ denotes the first $l$ bytes of the ground-truth continuation, and $\mathcal{L}_{\text{gt}}$ is the set of cumulative byte lengths corresponding to valid token boundaries in the ground-truth sequence.

What this equation computes. For each of the $G = 8$ generated predictions, the system converts both the prediction string $y_t^i$ and the ground-truth continuation $x_{\geq t}$ (which may contain multiple tokens) to their raw byte sequences, then checks two conditions: (1) the prediction's byte sequence is an exact prefix of the ground-truth continuation's byte sequence — meaning the predicted bytes match the beginning of what actually follows in the corpus; and (2) the length $l$ of the matched prefix corresponds to a complete token boundary in the ground-truth tokenization — meaning the model predicted one or more whole tokens, not a partial byte sequence that would split a token. If both conditions hold, the reward is 1. Otherwise, it is 0.

Why byte-level matching with token-boundary validation. The paper motivates this design by noting that next-token predictions can span multiple tokens or involve out-of-vocabulary tokens (Section 3.2, footnote 2). A simple string match — checking if the predicted text equals the ground-truth next token — would fail in two common cases. First, the model might correctly predict multiple tokens (e.g., predicting "size and" when the ground-truth next token is "size" followed by "and"), which should arguably receive credit. Second, the model might predict a token that is equivalent but tokenized differently due to whitespace or subword boundaries. Byte-level prefix matching with cumulative token boundary validation solves both: the model gets credit if its prediction forms a valid prefix of the continuation at the byte level that ends exactly at a token boundary. This means predicting "size" (which matches token boundary length $l_1$ in $\mathcal{L}_{\text{gt}}$) gets reward 1, predicting "size and" (which matches cumulative length $l_1 + l_2$) gets reward 1, and predicting "size a" (which does not match any cumulative boundary length) gets reward 0 even though it is a correct prefix of the bytes — because it would split the token "and" mid-byte-sequence.

Why binary reward. The paper explicitly contrasts this with prior work's "helpfulness-based" rewards (Quiet-STaR, Zelikman et al., 2024) that reward rationales based on how much they increase the probability of the correct next token. The binary correctness reward, combined with extraction from \boxed{}, eliminates a specific reward hacking vulnerability: in Quiet-STaR, the model can learn to generate rationales that simply contain the target token as a hint to itself (e.g., "...the next word is probably 'cat'..."), inflating the helpfulness reward without genuine reasoning. In RPT, the reward is based solely on the extracted final answer — whatever the model writes in its reasoning chain is irrelevant to the reward as long as the boxed answer is correct. This forces the model to use the reasoning chain for its actual epistemic purpose (figuring out the right answer) rather than as a mechanism to manipulate the reward signal. The binary nature of the reward — 1 for correct, 0 for incorrect — further eliminates gradient manipulation strategies that continuous reward signals might enable.

What alternatives were explored. Appendix A briefly mentions that the authors investigated three alternative reward designs: (1) first-token matching only (reward 1 if the first predicted token matches the ground-truth next token, ignoring subsequent tokens); (2) a dense reward scheme where correct predictions get reward 1 and incorrect predictions get the language model probability of that incorrect token as a smaller positive reward; and (3) conditional dense rewards applied only when at least one rollout in the group was correct. The paper reports that these alternatives "generally achieved performance comparable to the prefix matching reward" and concludes that "the reinforcement pre-training framework is relatively robust to these particular modifications in the reward signal." This robustness is notable — it suggests that the key innovation is the task structure (reasoning before predicting, with verifiable outcomes) rather than the precise form of the reward function.

What constitutes the ground-truth continuation. The ground-truth continuation $x_{\geq t}$ is simply the rest of the document from position $t$ onward, as it appears in the OmniMATH corpus. There is no external annotation — the corpus provides both the context (everything before position $t$) and the ground truth (everything from position $t$ onward). This is the critical property that enables RPT to scale: every token in every document automatically provides a verifiable RL training instance.

Relationship to standard RLVR rewards. In standard RLVR (Lambert et al., 2025), the reward comes from an external verifier that checks the model's output against a known answer (e.g., checking if the final answer to a math problem matches the solution). RPT's reward is structurally identical — it is a deterministic, rule-based function checking the model's output against a ground truth — but the ground truth is intrinsic to the data rather than externally annotated. This is the conceptual move that enables scaling: external annotations are scarce; intrinsic ground truths are available for every token.

The GRPO Training Objective

RPT uses the GRPO (Group Relative Policy Optimization) algorithm introduced by Guo et al. (2025) to update the model parameters. The training objective is formalized in Equation 4:

$J_{\text{RPT}}(\theta) = \mathbb{E}_{(x_{<t}, x_{\geq t}) \sim \mathcal{D}, \{o_t^i\}_{i=1}^G \sim \pi_\theta(\cdot \mid x_{<t})} \left[ r_t^i \right]$

where $\mathcal{D}$ is the set of all context-continuation pairs $\{(x_{<t}, x_{\geq t})\}_{t=1}^T$ extracted from the training corpus, $\pi_\theta$ is the current model, $o_t^i = (c_t^i, y_t^i)$ is the $i$-th generated response (reasoning chain plus prediction), $G = 8$ is the number of responses sampled per context, and $r_t^i \in \{0, 1\}$ is the prefix-matching reward.

What this equation computes. For each training step, the system samples a batch of contexts from $\mathcal{D}$, generates $G$ independent responses per context using the current model $\pi_\theta$, computes the reward for each response using the prefix-matching verifier, and then updates $\theta$ to increase the probability of generating responses that received reward 1 while decreasing the probability of responses that received reward 0. The expectation is over both the sampling of contexts from the corpus and the sampling of responses from the model, meaning the objective is a Monte Carlo estimate that becomes more accurate as more contexts and more rollouts are used.

How GRPO translates rewards into parameter updates. GRPO is an on-policy RL algorithm that compares responses within the same group to compute advantages. For a given context $x_{<t}$, the $G = 8$ generated responses form a group. The mean reward of the group is computed, and each response's advantage is its reward minus the group mean. Responses with above-average reward get positive advantages (the model is encouraged to generate them more often), and responses with below-average reward get negative advantages (the model is discouraged from generating them). The model parameters are then updated via a policy gradient that scales the log-probability of each response by its advantage. Because the advantage is computed relative to the group mean, GRPO does not require a learned value function (critic) — the group itself provides the baseline.

Why on-policy RL with group-based advantages. On-policy RL means that the responses used for training must be generated by the current model $\pi_\theta$, not by a previous version. As the model improves, the distribution of generated responses shifts, and the RL updates must track this shift. This is computationally expensive — each training step requires generating $G \times \text{batch\_size}$ new responses — but it is necessary for stable policy gradient optimization. Group-based advantage estimation (rather than using a learned value function) eliminates the need to train a separate critic network and avoids the instability that can arise from inaccurate value estimates. The choice of $G = 8$ represents a tradeoff: more rollouts per context would give better advantage estimates (reducing variance) but would increase the computational cost of each training step proportionally.

The full RL update procedure for one training step. First, 256 contexts are sampled from the training corpus. For each context, the model generates 8 responses (2,048 total generations per step) using a sampling temperature of 0.8. The prefix-matching verifier computes rewards for all 2,048 responses. Within each group of 8, advantages are computed as $r_t^i - \text{mean}(\{r_t^1, ..., r_t^8\})$. The GRPO algorithm then computes the policy gradient — the gradient of the expected reward with respect to $\theta$ — using these advantages, and updates the model parameters with a learning rate of $1 \times 10^{-6}$, Adam optimizer with $\beta = (0.9, 0.999)$, weight decay of 0.01, and a PPO mini-batch size of 256. The KL penalty coefficient is set to zero, meaning there is no explicit constraint on how far the updated policy can diverge from the previous policy. The entropy loss coefficient is also set to zero, following the "exact on-policy reinforcement learning" setting of Hao et al. (2025).

Dynamic sampling after step 500. Starting from training step 500, the paper activates "dynamic sampling to boost training efficiency" (Yuzz et al., 2025). While the exact mechanism is not detailed in the paper, the referenced DAPO work (Yuzz et al., 2025) describes a technique where the number of rollouts per question is dynamically adjusted based on recent performance — allocating more samples to contexts where the model is struggling and fewer to contexts where it consistently succeeds. This is conceptually similar to the compute-optimal test-time scaling in the reference example, but applied during training rather than inference. The paper uses a total of 1,000 training steps, with the first 500 using static $G = 8$ sampling and the remaining 500 using dynamic sampling.

The training length specification. The maximum prompt length is 4,096 tokens and the maximum response length is 8,192 tokens, for a combined maximum sequence length of roughly 12,288 tokens per rollout. At 8 rollouts per context and 256 contexts per batch with 1,000 steps, the total number of rollouts generated during training is approximately $256 \times 8 \times 1000 = 2,048,000$, though dynamic sampling after step 500 may alter this total. Each rollout involves generating a reasoning chain (typically hundreds to thousands of tokens) plus a final boxed prediction, meaning the total inference compute during training is measured in billions of generated tokens.

Token-Level Data Filtering via Entropy Thresholding

Not all next-token prediction decisions require reasoning. Many tokens in a corpus are highly predictable from local context alone — function words, common collocations, predictable completions of formulaic phrases. Training RPT on these easy tokens would waste compute on positions where the model already knows the answer without reasoning, and might even encourage degenerate reasoning patterns (generating elaborate justifications for obvious predictions).

To address this, the paper introduces a preprocessing step that filters the training data to include only tokens above a difficulty threshold. The procedure works as follows:

Proxy model for difficulty estimation. DeepSeek-R1-Distill-Qwen-1.5B — a much smaller model (1.5B parameters vs. the 14B being trained) — is used as a proxy to assess token difficulty. For each position $t$ in the training corpus, the proxy model processes the context $x_{<t}$ and outputs a probability distribution over the next token. The system then computes the entropy of the top-16 most probable next tokens:

$H(p) = -\sum_{i=1}^{16} p_i \log p_i$

where $p_i$ is the probability assigned to the $i$-th most likely next token by the proxy model.

Why entropy measures difficulty. Entropy quantifies the proxy model's uncertainty about what token comes next. Low entropy means the proxy model is confident — the probability mass is concentrated on one or a few tokens — indicating an easy prediction where the answer is obvious from local context. High entropy means the proxy model is uncertain — probability is spread across many plausible tokens — indicating a difficult prediction where reasoning might help discriminate between alternatives. The choice of top-16 (rather than full vocabulary) entropy is a practical truncation: the tail of the distribution often contains many tokens with near-zero probability that contribute negligibly to entropy but would inflate the computational cost of the calculation.

Filtering by entropy threshold. The paper applies entropy thresholds to categorize token positions into difficulty levels. While the exact threshold used during training is not explicitly stated as a single number, the evaluation in Section 4.1 uses thresholds of 0.5, 1.0, and 1.5 to define easy, medium, and hard splits respectively. During training, the system "filter[s] out low-entropy positions, prioritizing training on challenging tokens that require greater computational effort to predict." The implication is that only positions with proxy model entropy above some threshold (likely a value between 0.5 and 1.5) are included in the training set, though the paper does not specify the exact training threshold.

Why use a small proxy model. Using R1-Distill-Qwen-1.5B as the proxy rather than the 14B model being trained serves two purposes. First, it is computationally efficient — the 1.5B model can process the entire corpus at a fraction of the cost of the 14B model. Second, it provides a reasonably calibrated difficulty signal: if even a small reasoning model is uncertain about a token, the 14B model likely also faces a non-trivial prediction problem. Using the 14B model itself would be more accurate but would require forward passes through a 14B-parameter model for every token position in the corpus — a substantial cost before training even begins.

Relationship to the pre-training corpus. The OmniMATH dataset contains 4,428 mathematical documents. After applying the entropy-based filter, the effective training set is a subset of token positions from these documents — specifically, only positions where the proxy model's top-16 entropy exceeds the threshold. The paper does not report the total number of training tokens after filtering, but given that mathematical text contains many formulaic tokens (operators, variable names in standard patterns) that would be filtered out, the effective training set is likely substantially smaller than the raw token count of OmniMATH.

Why this filtering is necessary for RPT but not for standard NTP. Standard NTP trains on all tokens because the training signal (cross-entropy loss) is always informative — even easy tokens provide gradient information that improves the model's probability calibration. RPT, by contrast, generates explicit reasoning chains for each training instance. If most instances are trivially easy, the model learns to generate vacuous reasoning (e.g., "this is obviously a period because the sentence is ending") that does not generalize to genuinely difficult predictions. By filtering to hard tokens only, RPT ensures that the reasoning chains it generates are necessary for the task, making the RL signal more meaningful.

Training Configuration: Model, Prompts, Rollouts, and Extraction

Base model initialization. All RPT experiments start from DeepSeek-R1-Distill-Qwen-14B, not from a standard pre-trained base model like Qwen2.5-14B. This is a consequential design choice: R1-Distill-Qwen-14B has already been fine-tuned (via distillation from the full DeepSeek-R1) to generate chain-of-thought reasoning when prompted. Starting from this model ensures that the model can produce coherent reasoning from the very first RPT training step. If training were initialized from a standard base model that has never generated chain-of-thought, the initial rollouts would be gibberish or simple repetitions, and the RL signal might never get traction — a cold-start problem that the paper explicitly defers to future work (Section 6: "investigating RPT training from a standard base language model would provide further insights into its foundational impact").

Why 14B and why R1-Distill. The 14B scale is large enough to exhibit meaningful reasoning capabilities (the base model achieves 36.1% on SuperGPQA and 68.9% on MMLU-Pro in reasoning mode, Table 3) but small enough that a research team can run 1,000 steps of on-policy RL within reasonable compute constraints. Using the R1-Distill variant rather than Qwen2.5-14B directly provides two capabilities: the model already understands the \boxed{} extraction format from its R1 training, and it already generates chain-of-thought when prompted to "reason step by step." These pre-existing capabilities are not guaranteed in a standard base model.

Prompt template. The paper uses prompt template "v0" for the main experiments (Appendix D, Table 10):

Complete the given text under '### Context' by predicting the next token, and wrap it in '\boxed{}'. Please reason step by step to find the most probable next token as the final answer, and enclose it in \boxed{} (note: the token may begin with a space, e.g., \boxed{ para} or \boxed{ =}; do not use \text{}).

### Context
{prompt_content}

This template instructs the model to (1) reason step by step, (2) produce a final prediction in \boxed{}, and (3) handle the edge case where the predicted token begins with a space (common in subword tokenization where tokens like " para" or " =" include leading whitespace). The explicit note about spaces is important because standard LaTeX rendering would strip leading spaces from \boxed{ }, causing extraction failures for space-prefixed tokens.

Why this specific template. The paper evaluated seven template variants (v0 through v6, Table 10) and found substantial variation in initial performance: v0 achieved 3.0% Random@1 and 8.5% Pass@8, while v6 achieved 6.0% and 19.0% respectively. Despite v6's superior initial performance, the main experiments use v0 — the paper notes this explicitly ("Notice that the reinforcement pre-training experiments in Section 4 used the 'v0' prompt template") and delegates prompt optimization to future work. This means the reported RPT results represent a lower bound on what could be achieved with better prompting, since even the initial model's reasoning quality depends on prompt design.

Rollout generation parameters. During each training step, $G = 8$ rollouts are generated per context using a sampling temperature of 0.8. Temperature 0.8 strikes a balance: it introduces enough stochasticity for the model to explore different reasoning paths and predictions (exploration is necessary for RL), but it is low enough that the generated rollouts are mostly coherent rather than random. The maximum generation length per rollout is 8,192 tokens for the response (reasoning plus prediction), with a separate 4,096-token limit for the prompt (context plus template). This means each full rollout can be up to roughly 12,288 tokens.

Prediction extraction. After each rollout is generated, the system extracts the prediction $y_t^i$ by "directly extract[ing] the full sequence inside the last \boxed{} following the special token response as the model prediction for the next token." This extraction rule has several implications. Using the last \boxed{} means the model can use earlier boxed expressions in its reasoning (e.g., for intermediate calculations or hypotheses) without those being mistaken for the final prediction. Requiring the \boxed{} to follow the response token ensures that the model has a clear delineation between reasoning and answer — the model knows that response signals the transition to final-answer mode. If no valid \boxed{} is found after response, the extraction fails and that rollout presumably receives reward 0 (since no valid prediction can be verified).

What makes this extraction rule effective. This design mirrors the answer extraction format used in mathematical reasoning RL (DeepSeek-R1, Guo et al., 2025), where models are trained to output their final answer in \boxed{} after a reasoning chain. By reusing this convention, RPT inherits the answer-formatting behavior that R1-Distill models already possess, avoiding the need to train the model to output a new delimited format. The response token serves as an explicit "I'm done thinking, here's my answer" marker, which structures the generation into a clean reasoning-then-prediction sequence.

Optimization hyperparameters. The full set of RL hyperparameters (Appendix B, Table 5) specifies: actor gradient clipping at 0.2 (standard for PPO-style algorithms, preventing overly large policy updates), batch size of 256 contexts, PPO mini-batch size of 256 (meaning no sub-batching within a step), learning rate of $1 \times 10^{-6}$, Adam optimizer with $\beta = (0.9, 0.999)$ and weight decay 0.01, zero KL penalty coefficient, and zero entropy loss coefficient. The zero KL penalty is notable — in standard RLHF, a KL penalty is used to prevent the policy from diverging too far from the reference model (typically the pre-trained checkpoint). Its absence here means RPT allows the model to move arbitrarily far from the initial R1-Distill-Qwen-14B policy, relying on the binary reward signal and limited training steps to prevent catastrophic degradation.

The FLOPs accounting. The paper reports RPT's scaling curves in terms of "RL Compute (FLOPs)" on the x-axis of Figure 5, spanning approximately $10^{20}$ to $10^{21}$ FLOPs. These FLOPs include both the forward passes for generating rollouts (8 rollouts × 256 contexts × average response length per step) and the backward passes for computing gradient updates. The paper does not provide a detailed FLOPs breakdown, but the scaling analysis in Section 4.2 uses six checkpoints (steps 100, 200, 400, 800, 1000, and 1200) to fit the power-law relationship between compute and next-token prediction accuracy.

Why 1,000 total training steps. The paper does not explicitly justify this number, but several factors likely contribute. At a batch size of 256 contexts, 1,000 steps means training on approximately 256,000 distinct token positions (some may be revisited due to the finite size of the filtered OmniMATH dataset). This is sufficient to observe clear improvements over the baseline (Table 1) while staying within practical compute limits for a research experiment. The scaling analysis extends to 1,200 steps to provide additional data points for curve fitting, suggesting that further training might yield additional gains — an open question for future scaling work.

4. Key Insights and Innovations

Innovation 1: Reframing Pre-Training as a Reinforcement Learning Problem with Intrinsic Rewards

The paper's most fundamental conceptual move is dissolving the long-standing boundary between pre-training (supervised next-token prediction) and post-training (reinforcement learning with external rewards) by recognizing that the pre-training corpus itself contains a universal, intrinsic reward signal: whether the model can correctly predict the continuation. This is not an incremental improvement to NTP or to RL — it is a category-level reframing that asks: what if RL is the pre-training objective, applied at web scale, with every token serving as its own verifiable training instance?

To appreciate the shift, consider the field's dominant assumptions before this work. Pre-training has been synonymous with maximum-likelihood estimation via cross-entropy loss on raw text — a paradigm so entrenched that "pre-training" and "next-token prediction" are used almost interchangeably in the literature (Hoffmann et al., 2022; Kaplan et al., 2020). RL, meanwhile, has been confined to post-training on curated data: human preference judgments for alignment (RLHF, Ouyang et al., 2022) or question-answer pairs with verifiable solutions for reasoning (RLVR, Lambert et al., 2025). These two phases used fundamentally different objectives, different data, and different optimization algorithms, and the field largely accepted this split as inherent — pre-training learns from text, RL learns from feedback, and never the twain shall meet.

RPT challenges this split by observing that the next-token prediction task is already a reinforcement learning problem in disguise. A context is a state. The model's prediction is an action. The ground-truth continuation provides a verifiable outcome — does the prediction match or not? The only missing ingredient is making the model's prediction an explicit, extractable action rather than an implicit probability distribution. RPT supplies this via the next-token reasoning format: the model generates chain-of-thought and outputs a boxed answer, and the corpus continuation serves as the ground truth for binary reward calculation. Critically, this reward is not externally annotated — it is inherent in the text itself. Every token in every document automatically defines a complete RL training instance, making RL scalable to the entirety of the web corpus without requiring a single human label.

This insight has theoretical import beyond the specific RPT implementation. It implies a unified perspective on language model training: the distinction between pre-training and post-training is an artifact of objective choice, not a fundamental necessity. One could imagine a continuum of RL-based language model training where the reward density varies — from dense, token-level intrinsic rewards (RPT) to sparse, task-level verifiable rewards (RLVR) to learned reward models for open-ended generation (RLHF) — all operating within the same optimization framework. This unifies what were previously separate research subfields and suggests that the next generation of language models might be trained end-to-end with RL from the earliest stages, blurring the pre-train/fine-tune boundary entirely.

The evidence anchoring this reframing is the emergence of reasoning behavior from a purely outcome-based reward signal. The model is never told how to reason — it receives no supervision on its chain-of-thought quality, no demonstrations of good reasoning, and no dense rewards for plausible intermediate steps. Yet Figure 6 and Table 4 show that RPT-14B develops distinctive reasoning patterns qualitatively different from the problem-solving patterns of the base model from which it was initialized: a 161.8% increase in hypothesis-driven reasoning ("probably," "something like") and a 26.2% increase in deductive reasoning ("therefore," "consequently") compared to the R1-Distill baseline applied to explicit problem-solving. These patterns emerge purely because they help the model make correct predictions — the RL signal alone discovers that deliberative inference is instrumentally useful for next-token accuracy. This is a non-trivial finding: it demonstrates that RL with binary outcome rewards can bootstrap complex cognitive behaviors without explicit reasoning supervision, a result with implications far beyond language modeling.

Innovation 2: Identifying and Closing the Pre-Training/Post-Training Objective Gap as a First-Class Problem

The paper makes a diagnostic contribution that is separable from the RPT method itself: it provides concrete evidence that the objective mismatch between NTP pre-training and RL fine-tuning is not merely a theoretical inconvenience but a measurable, practically significant barrier to effective downstream training. This transforms the objective gap from an acknowledged-but-ignored property of the training pipeline into a problem worth solving directly.

The diagnostic evidence comes from Table 2's "Continual NTP training" baseline. When R1-Distill-Qwen-14B undergoes additional NTP training on the same OmniMATH corpus used for RPT, its subsequent RLVR fine-tuning performance collapses from 52.7% (the baseline without continual NTP) to 13.0%. This is not a small degradation — it is a collapse of nearly 40 percentage points, essentially destroying the model's ability to benefit from RL. The mechanism is instructive: additional NTP training pushes the model's representations further toward maximizing next-token likelihood, which optimizes for a fundamentally different objective than maximizing downstream task rewards. The representations that are optimal for predicting the next token given a context are not the representations that are optimal for generating chain-of-thought and boxed answers that match external verifiers. By doubling down on NTP, the model becomes less suitable for RL, not more.

This finding has significant implications for how the field thinks about pre-training. The dominant assumption has been that better language models (as measured by perplexity or next-token accuracy) are better foundations for downstream fine-tuning — an assumption baked into scaling laws research (Kaplan et al., 2020; Hoffmann et al., 2022) that treats pre-training loss as a sufficient statistic for downstream capability. RPT's continual NTP baseline complicates this picture: it suggests that pre-training loss and downstream RL transferability can be anti-correlated in certain regimes. A model with slightly worse next-token prediction accuracy (because it was trained with RPT rather than NTP) can be dramatically better at absorbing RL fine-tuning, because its pre-training objective already aligns with the RL paradigm.

RPT addresses this gap not by adding a post-hoc alignment step but by making the pre-training objective itself RL-compatible from the start. By training with GRPO on correctness-based rewards during pre-training, RPT-14B arrives at the fine-tuning stage already optimized for the RL objective family. The result (Table 2): RPT-14B starts higher before RL (56.3% vs. 51.2%) and reaches a higher ceiling after RL (58.3% vs. 52.7%), demonstrating both better initialization and better trainability. The 5.6 percentage point improvement in final RL performance, while not enormous in absolute terms, represents the entire gap between the 14B R1-Distill model and the 32B model on some benchmarks — suggesting that objective alignment during pre-training can substitute for significant model scale during fine-tuning.

Innovation 3: Scaling Laws for Reinforcement Pre-Training as a New Axis of Compute Scaling

The paper introduces the concept that RL-based pre-training follows its own scaling laws — distinct from, and potentially complementary to, the well-known scaling laws for model size and data quantity in NTP pre-training. This is significant not primarily for the specific power-law coefficients reported but for establishing that "training compute allocated to RL-based pre-training" is a legitimate, measurable scaling dimension that obeys predictable mathematical relationships.

The field has extensively characterized how NTP pre-training loss scales with compute, parameters, and tokens (Kaplan et al., 2020; Hoffmann et al., 2022), yielding power-law relationships that guide billion-dollar training decisions. But no analogous relationship existed for RL-based pre-training — it was not even clear that RL pre-training would exhibit smooth scaling behavior, given the well-known instability and variance of RL training. Figure 5 provides the first evidence that it does: next-token prediction accuracy under RPT follows $P(C) = A \cdot C^\alpha + P^*$, with $R^2$ values of 0.989 to 0.997 across difficulty levels over a compute range of approximately $10^{20}$ to $10^{21}$ FLOPs.

What makes this a conceptual innovation rather than just a curve-fitting exercise is the implication that compute allocation between NTP and RPT pre-training may be an optimizable design choice, analogous to how Hoffmann et al. (2022) showed that compute allocation between model size and data quantity is optimizable. If RPT scaling curves continue to hold at larger scales, organizations making pre-training investments would face a new type of decision: given a fixed compute budget, what fraction should be spent on NTP pre-training (to build broad knowledge) versus RPT pre-training (to incentivize reasoning patterns)? The paper does not answer this question — its experiments initialize from an already-trained model — but it establishes the conceptual and methodological foundation for asking it.

The difficulty-stratified scaling curves in Figure 5 add nuance that makes this innovation more than a simple "RPT scales" claim. The fact that accuracy improves across all three difficulty levels (easy: $R^2 = 0.995$, medium: $R^2 = 0.997$, hard: $R^2 = 0.989$) with similar functional forms suggests that RPT's benefits are not concentrated in a particular difficulty regime — the model learns to reason better about all token predictions, not just the hardest ones. This distinguishes RPT from approaches like the entropy-based filtering described in Section 3.3, which explicitly focus training on hard tokens. Even though the training data is filtered, the learned reasoning capability transfers to easy predictions as well, where the baseline model already performs adequately — a form of positive transfer that suggests the reasoning patterns learned on hard tokens generalize downward.

The paper's honesty about the limitations of this scaling analysis (Section 6: "the current pre-training corpus predominantly consists of mathematical documents") is itself part of the innovation's framing. The scaling laws are presented not as definitive laws but as evidence that RL-based pre-training is a coherent scaling paradigm worth investigating at larger scale and broader domain coverage — a call to the community to treat this as a new axis to explore rather than a solved problem.

Innovation 4: Intrinsic Reward Robustness — Binary Correctness Rewards as a Defense Against Reward Hacking

The paper makes a specific empirical finding with significant design implications: that using binary, rule-based correctness as the only reward signal — with no reward shaping, no dense intermediate feedback, and no learned reward model — is sufficient to train next-token reasoning, and may actually be preferable to richer reward signals because it eliminates a class of reward hacking strategies. This is a counterintuitive result in a field that often assumes denser rewards lead to faster or better learning.

The theoretical vulnerability RPT addresses is most clearly exemplified by Quiet-STaR (Zelikman et al., 2024), which the paper identifies as its closest prior work. Quiet-STaR uses a "helpfulness-based" reward: a generated rationale receives positive reward if it increases the probability the model assigns to the correct next token. The vulnerability is that the model can learn to generate rationales that contain the correct token as a self-hint (e.g., writing "...the next word is probably 'cat'..." in the rationale), which inflates the helpfulness reward without any genuine reasoning about why 'cat' is correct. The reward is hacked: the signal that was meant to incentivize understanding instead incentivizes a trivial lexical trick.

RPT's reward design — binary 1/0 based solely on whether the extracted boxed answer matches the corpus continuation — eliminates this vulnerability by construction. Whatever the model writes in its chain-of-thought is irrelevant to the reward. The model can still write "...the next word is probably 'cat'..." in its reasoning, but doing so does not guarantee a correct answer — the model must still correctly predict the token in the boxed output. And if the model can predict correctly without genuine reasoning, the RL signal provides no incentive to generate vacuous reasoning, because vacuous reasoning does not increase the reward. The result is that the reasoning chain becomes genuinely instrumental to the task: the model uses it to deliberate, explore alternatives, and verify hypotheses because those cognitive operations help arrive at correct predictions, not because they manipulate the reward signal.

This insight is backed by two pieces of evidence. First, the qualitative analysis in Table 4 and Table 11 shows reasoning chains that are genuinely deliberative — weighing alternatives, reflecting on structural context, considering edge cases — rather than simple pattern-matching or self-hinting. Second, Appendix A reports that alternative reward designs (dense rewards, conditional dense rewards, first-token-only matching) performed comparably to the binary prefix-matching reward, suggesting that the core benefit comes from the task structure (reason → predict → verify) rather than from fine-grained reward engineering. This robustness is itself a finding: it implies that practitioners adopting RPT do not need to invest heavily in reward shaping, because the verifiable outcome signal dominates whatever nuances the reward function might introduce.

The broader implication is a design principle for RL-based training systems: when a verifiable outcome signal exists, prefer binary rewards over learned or shaped rewards, even if the latter seem intuitively more informative. The binary reward acts as a natural regularizer against reward hacking by making the objective transparent and ungameable — the model cannot fool a rule-based verifier that checks for exact match against ground truth. This principle extends beyond RPT to any domain where ground-truth verification is possible (code execution, mathematical proof checking, factual verification against databases), and it echoes findings from the broader RL literature about the dangers of reward misspecification.

Innovation 5: Demonstrating That Reasoning Patterns Can Emerge from Outcome-Only RL Without Any Reasoning Supervision

The paper provides a specific and striking piece of evidence about how complex cognitive behaviors can be acquired: the model learns to reason — not because it is taught what good reasoning looks like, but because reasoning is instrumentally useful for achieving correct outcomes under the RL objective. This is a finding about the power of RL as a learning mechanism that goes beyond the specific RPT method.

The evidence is Figure 6 and the accompanying analysis in Section 4.5. RPT-14B, which was initialized from R1-Distill-Qwen-14B and then trained with binary correctness rewards on next-token reasoning, develops reasoning patterns that are qualitatively different from the reasoning patterns of its initialization model when that model is applied to explicit problem-solving. The shift is not subtle: hypothesis-driven reasoning increases by 161.8%, deduction increases by 26.2%, while breakdown-oriented reasoning (decomposing problems into subproblems) — the dominant pattern in the base model's problem-solving — is substantially less prominent in RPT's next-token reasoning. These are different cognitive strategies emerging from different task demands, and they emerge purely from outcome-based RL without any explicit instruction about which reasoning strategies to employ.

Why this matters beyond the specific number. It demonstrates that RL with verifiable rewards does not merely amplify existing behaviors — it can discover and amplify new behavioral strategies that are better suited to the task at hand. The base model was trained (via distillation from DeepSeek-R1) to solve explicit math problems by breaking them down into steps. When faced with next-token reasoning — a different task requiring prediction of what word follows a given context — the RL process discovers that hypothesis generation and deductive inference are more effective strategies than step-by-step decomposition, and it shifts the model's behavior accordingly. The RL signal acts as a search process over reasoning strategies, not just a reinforcement mechanism for existing ones.

This has deep implications for how we think about training reasoning capabilities. The dominant approach to improving LLM reasoning has been to provide reasoning demonstrations — either via supervised fine-tuning on chain-of-thought examples (Wei et al., 2022) or via distillation from reasoning models (DeepSeek-R1-Distill variants). These approaches teach the model what good reasoning looks like by example. RPT's finding suggests an alternative path: teach the model that reasoning leads to correct outcomes by creating an environment where reasoning is instrumentally necessary for reward. The model then discovers for itself what reasoning strategies work, potentially finding patterns that human demonstrators might not think to provide.

The qualitative example in Table 4 makes this concrete. The model's reasoning process is not a clean, textbook-style chain-of-thought. It is messy, iterative, and recursive — it considers alternatives ("Alternatively, it could be..."), doubts itself ("Wait, perhaps looking at how the initial document is structured"), reflects on meta-level features ("It's using markdown with headers"), and explores dead ends before converging. This resembles human deliberation more than the polished reasoning found in training demonstrations. Yet it works — the model arrives at correct predictions — and the RL process discovers and preserves this messy-but-effective strategy because it produces outcomes, not because it matches a template.

The paper's positioning of this finding as a contrast between "next-token reasoning" and "problem-solving" (Figure 6) is itself conceptually significant. It suggests that different types of cognitive work — predicting what comes next in a text versus solving a structured problem — benefit from different reasoning strategies, and that RL can adaptively discover which strategies suit which task. This opens the door to training regimes where models develop diverse reasoning repertoires tailored to different task demands, rather than applying a one-size-fits-all chain-of-thought template.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The primary pre-training corpus is OmniMATH (Gao et al., 2024), containing 4,428 competition-level mathematical problems and solutions sourced from AoPS Wiki and AoPS forum. For language modeling evaluation, a held-out validation set of 200 samples from OmniMATH is used (Section 4.1). For reinforcement fine-tuning experiments, the authors randomly sample questions with verifiable answers from Skywork-OR1 (He et al., 2025), using 256 examples for training and 200 for testing, filtered to include only challenging instances as identified by R1-Distill-Qwen-32B (Section 4.3). Zero-shot evaluation uses MMLU-Pro (Hendrycks et al., 2020) — a comprehensive multi-task understanding benchmark — and SuperGPQA (Du et al., 2025) — a large-scale benchmark of graduate-level reasoning questions spanning 285 disciplines (Section 4.4).

• Base model(s). All main experiments use DeepSeek-R1-Distill-Qwen-14B as the starting checkpoint. The authors argue this model "serves as a good starting point for reinforcement learning due to its basic reasoning capabilities" (Section 3.3). A DeepSeek-R1-Distill-Qwen-1.5B model serves as the proxy for token-level entropy-based data filtering. For the language modeling comparison in Figure 4, R1-Distill-Qwen-32B is included as a larger baseline. Qwen2.5-14B is included in Table 1 as the base model underlying R1-Distill-Qwen-14B.

• Metrics. The primary metric is next-token prediction accuracy — the fraction of token positions where the model's boxed prediction exactly matches the ground-truth continuation at a valid token boundary, as evaluated by the prefix-matching reward function (Equation 3). For reinforcement fine-tuning, the metric is downstream task accuracy on the 200-instance Skywork-OR1 test set. For zero-shot evaluation, the metric is accuracy on MMLU-Pro and SuperGPQA under a multiple-choice question format, following the evaluation protocols of Ma et al. (2025) and Zhou et al. (2025). For scaling analysis, the metric is next-token prediction accuracy stratified by difficulty (easy, medium, hard, defined by entropy thresholds of 0.5, 1.0, and 1.5 respectively). Reasoning pattern analysis (Section 4.5) uses keyword-based categorization into six pattern types: transition, reflection, breakdown, hypothesis, divergent thinking, and deduction.

• Baselines. The paper compares against several distinct baselines. For language modeling (Section 4.1, Table 1): (1) Standard next-token prediction: Qwen2.5-14B and R1-Distill-Qwen-14B selecting the token with highest probability directly, without generating reasoning. (2) Next-token reasoning without RPT: R1-Distill-Qwen-14B prompted to generate chain-of-thought before predicting (the "reasoning mode" baseline that uses the same prompt template as RPT but without RL training). (3) R1-Distill-Qwen-32B in standard NTP mode, to assess whether RPT-14B can match a ~2.3× larger model. For reinforcement fine-tuning (Section 4.3, Table 2): (4) R1-Distill-Qwen-14B with RLVR fine-tuning but no RPT pre-training. (5) Continual NTP training: R1-Distill-Qwen-14B further trained on the same OmniMATH corpus using standard next-token prediction objective, then fine-tuned with RLVR. For zero-shot tasks (Section 4.4, Table 3): (6) R1-Distill-Qwen-14B in standard NTP mode, (7) R1-Distill-Qwen-32B in standard NTP mode, and (8) R1-Distill-Qwen-14B in reasoning mode. The continual NTP baseline (5) is particularly important because it isolates the effect of the RL objective from the effect of simply seeing more data.

• Generation budget / compute accounting. For RPT training, compute is measured in total RL training FLOPs, encompassing both forward-pass inference (generating $G=8$ rollouts per context, each up to 8,192 response tokens) and backward-pass gradient updates. The scaling analysis (Figure 5) uses six checkpoints at steps 100, 200, 400, 800, 1000, and 1200 to map FLOPs to accuracy. The paper does not provide a precise FLOPs-per-step breakdown, but the x-axis of Figure 5 spans approximately $10^{20}$ to $10^{21}$ FLOPs. For evaluation, the generation budget differs by task: language modeling evaluation uses a single generation per token (extracting the boxed prediction); reinforcement fine-tuning evaluation uses a maximum of 32,000 tokens per validation instance with temperature 0.6; zero-shot evaluation uses up to 12,288 tokens with temperature 0.8. For the initial prompt template experiments (Appendix D, Table 8), Random@1 and Pass@8 (with 8 rollouts) are reported.

• Cross-validation / statistical protocol. The paper does not employ cross-validation for its main results. Language modeling accuracy is reported on a single held-out validation set of 200 samples from OmniMATH (Section 4.1). Reinforcement fine-tuning results are reported on a single 200-instance test set from Skywork-OR1 (Section 4.3). Zero-shot results are reported on the full test sets of MMLU-Pro and SuperGPQA without cross-validation (Section 4.4). The scaling curves in Figure 5 are fit using six data points from different training checkpoints, with $R^2$ values reported as the goodness-of-fit measure. Error bars or confidence intervals are not reported for any experimental result. The limited evaluation set sizes (200 for language modeling, 200 for RL fine-tuning) mean the reported accuracies have non-trivial sampling variance that is not quantified — a change of ±2 percentage points on a 200-sample set could occur purely from random seed variation.

Main Quantitative Results

Language Modeling Performance

The headline language modeling result appears in Table 1: RPT-14B achieves consistently higher next-token prediction accuracy than all baselines across all difficulty levels. On easy tokens (entropy > 0.5), RPT-14B reaches 45.11% compared to 41.60% for R1-Distill-Qwen-14B in standard NTP mode (a 3.51 percentage point improvement) and 41.90% for Qwen2.5-14B. On medium tokens (entropy > 1.0), RPT-14B achieves 33.56% vs. 29.46% (R1-Distill-Qwen-14B NTP) and 30.03% (Qwen2.5-14B). On hard tokens (entropy > 1.5), RPT-14B reaches 23.75% vs. 20.43% and 20.65% respectively.

What makes this result non-trivial is the comparison against next-token reasoning without RPT. When R1-Distill-Qwen-14B is prompted to generate chain-of-thought before predicting (the "Next-token reasoning" row in Table 1), its accuracy collapses — achieving only 3.31% on easy, 1.66% on medium, and 1.41% on hard splits. This means the base model, even though it was distilled from DeepSeek-R1 and can generate reasoning when asked, is dramatically worse at next-token reasoning than at direct next-token prediction. Something about the reasoning process, when applied to next-token prediction without RL training, actively degrades performance. RPT recovers and surpasses the direct-prediction baseline by training the model to make its reasoning productive rather than counterproductive.

Figure 4 strengthens the case by showing that RPT-14B matches the language modeling performance of R1-Distill-Qwen-32B — a model with approximately 2.3× more parameters — where RPT-14B achieves roughly 34% average next-token prediction accuracy compared to R1-Distill-Qwen-32B's approximately 32% (read from the bar chart; the paper reports these as "average next-token prediction accuracy across data of different difficulty levels"). The 14B R1-Distill baseline sits at roughly 30%. This demonstrates that RPT can compensate for significant model scale differences in language modeling accuracy — a 14B model with RPT training edges out a 32B model without it — though this comparison should be interpreted cautiously since the 32B model uses standard NTP, not RPT.

Important caveat on Table 1 baselines. The "Next-token reasoning" row for R1-Distill-Qwen-14B is evaluated using the same prompt template (v0) and extraction procedure (\boxed{}) as RPT-14B, but without any RL training. This baseline's extremely low accuracy (1.41–3.31%) reveals a critical detail: the R1-Distill model, despite its reasoning capabilities on explicit problem-solving tasks, cannot effectively transfer those reasoning patterns to next-token prediction without specialized training. The model either fails to produce valid boxed predictions, produces predictions in the wrong format, or generates reasoning that leads to incorrect conclusions. This baseline is essential for establishing that RPT's improvements come from the RL training process, not from the prompt template or reasoning format itself.

Scaling Properties of Reinforcement Pre-Training

Figure 5 presents the scaling analysis, modeling next-token prediction accuracy as a power-law function of training compute: $P(C) = A \cdot C^\alpha + P^*$. The key finding is that accuracy monotonically improves with increased RL compute across all three difficulty levels, with high coefficients of determination: $R^2 = 0.995$ (easy), $R^2 = 0.997$ (medium), and $R^2 = 0.989$ (hard). At the final checkpoint (1,200 steps, corresponding to the rightmost point on each curve), easy-token accuracy reaches roughly 50%, medium-token accuracy reaches roughly 38%, and hard-token accuracy reaches roughly 28% (values read approximately from the log-scale plot).

The power-law fit implies that each additional unit of training compute yields diminishing but predictable returns — the slope of the log-log curve decreases as compute increases, indicating that early training steps produce larger per-step improvements than later steps. This is consistent with the shape of scaling laws observed in NTP pre-training (Kaplan et al., 2020; Hoffmann et al., 2022), suggesting that RPT training follows similar fundamental dynamics despite using a fundamentally different objective (RL rather than maximum likelihood).

What the scaling analysis does not tell us is whether these curves would continue to improve or would plateau at higher compute budgets. The x-axis spans only approximately one order of magnitude ($10^{20}$ to $10^{21}$ FLOPs), and the fit uses only six data points — insufficient to distinguish between a power law that continues indefinitely and one that asymptotes. The $R^2$ values are high, but with only six points, even a poor functional form can achieve high $R^2$. A true scaling law would require at least two to three orders of magnitude in compute to establish the functional form reliably, as was done in Kaplan et al. (2020) and Hoffmann et al. (2022).

Additionally, the paper reports $R^2$ for the fitted curves but does not report the fitted parameters ($A$, $\alpha$, $P^*$) or their uncertainties. Without these values, the reader cannot assess whether the fitted exponents are plausible (typical scaling exponents for language modeling loss are in the range of -0.05 to -0.1) or whether the asymptotic performance $P^*$ is within a reasonable range (e.g., is $P^*$ approaching 100% accuracy, or some lower ceiling?). These omissions limit the analysis's value for practitioners who might want to extrapolate to larger compute budgets.

Reinforcement Fine-Tuning with RPT

Table 2 reports the core finding for downstream RL training. Before RL fine-tuning, RPT-14B starts at 56.3% accuracy on the Skywork-OR1 test set, compared to 51.2% for the R1-Distill-Qwen-14B baseline — a 5.1 percentage point advantage before any task-specific training. After RLVR fine-tuning (15 epochs on 256 training examples), RPT-14B reaches 58.3% while the baseline reaches 52.7% — a 5.6 percentage point advantage in final performance. This means the initial gap is largely preserved through fine-tuning, with RPT-14B showing a slightly larger improvement during fine-tuning (+2.0 points) compared to the baseline (+1.5 points).

The most striking result in Table 2 is the continual NTP training baseline. When R1-Distill-Qwen-14B undergoes additional standard next-token prediction training on the same OmniMATH corpus (i.e., comparing "standard NTP on the same data" versus "RPT on the same data"), its pre-RL accuracy collapses to 10.7% and its post-RL accuracy only recovers to 13.0%. This is a catastrophic degradation — the model becomes substantially worse at the downstream RL task after further NTP training, and RL fine-tuning cannot recover the lost performance. The comparison isolates the effect of the training objective: the same corpus, the same amount of data, but NTP training destroys downstream RL capability while RPT training enhances it.

This is the strongest empirical evidence in the paper for the claim that the pre-training objective matters for downstream RL. It demonstrates not just that RPT is better than NTP for subsequent RL (which could be explained by RPT being "more similar" to the fine-tuning task), but that additional NTP training is actively harmful — it makes the model worse at RL than if it had received no additional training at all. This finding has practical implications for multi-stage training pipelines: if a model will ultimately be fine-tuned with RL, continuing to train it with NTP on more data may be counterproductive.

Limitations of the fine-tuning experiment. The RL fine-tuning uses only 256 training examples and is evaluated on 200 test instances — very small by modern standards. The 15-epoch training regime likely involves substantial overfitting on such a small dataset. The paper does not report training curves, so we cannot see whether RPT-14B's advantage emerges early in fine-tuning or only after many epochs. Additionally, the Skywork-OR1 dataset, while filtered for challenging instances, represents a specific distribution of reasoning problems — we cannot know whether the RPT advantage generalizes to other downstream tasks or is specific to math-adjacent reasoning problems that resemble the OmniMATH pre-training corpus.

Zero-Shot Performance on End Tasks

Table 3 reports zero-shot accuracy on general-domain benchmarks. Under the reasoning mode (generating chain-of-thought before answering), RPT-14B achieves 39.0% on SuperGPQA and 71.1% on MMLU-Pro. This outperforms R1-Distill-Qwen-14B in reasoning mode (36.1% and 68.9%, respectively) by margins of 2.9 and 2.2 percentage points. It also substantially outperforms R1-Distill-Qwen-32B in standard NTP mode (37.2% and 56.5%), with particularly large gains on MMLU-Pro (71.1% vs. 56.5%, a 14.6 percentage point improvement).

The comparison between RPT-14B (reasoning mode) and R1-Distill-Qwen-14B (standard NTP mode) is even larger: 39.0% vs. 32.0% on SuperGPQA (+7.0 points), and 71.1% vs. 48.4% on MMLU-Pro (+22.7 points). However, this comparison conflates the effect of RPT training with the effect of using reasoning mode versus direct prediction — a more appropriate baseline is R1-Distill-Qwen-14B in reasoning mode, where the gains are much more modest (+2.9 and +2.2 points respectively).

Per-category breakdowns (Appendix C, Tables 6 and 7). The detailed results show that RPT-14B's gains are not uniform across subjects. On SuperGPQA (Table 6), RPT-14B improves over R1-Distill-Qwen-14B (reasoning mode) in some categories — Engineering (+6.0 points), Chemistry (+9.5 points), Economics (+0.5 points) — but performs worse in others — Management (-3.5 points), Literature and Arts (-0.5 points), Law (-0.5 points). This mixed pattern suggests that RPT's benefits are domain-dependent, with STEM fields showing more consistent improvements than humanities. On MMLU-Pro (Table 7), RPT-14B shows gains in Chemistry (+3.0 points), Math (+5.5 points), and Business (+6.5 points), but slight regressions in Economics (-3.0 points), Biology (-0.5 points), and Psychology (-2.5 points). The overall improvement is driven by large gains in a few subjects rather than uniform improvement across all categories.

This domain dependence is notable because OmniMATH — the RPT pre-training corpus — is a mathematical dataset. RPT is essentially math-specific pre-training that the paper evaluates on general-domain benchmarks. The fact that improvements appear primarily in math-adjacent categories (Math, Chemistry, Engineering) while performance holds steady or slightly degrades in others (Law, Literature) raises the question of whether RPT is learning genuinely general reasoning patterns or primarily math-specific patterns that transfer only to structurally similar domains. The paper does not discuss this pattern.

Next-Token Reasoning Pattern Analysis

Figure 6 and the associated analysis in Section 4.5 characterize the qualitative differences between RPT-14B's next-token reasoning and R1-Distill-Qwen-14B's problem-solving. The most dramatic difference is in hypothesis-driven reasoning: RPT-14B shows a 161.8% relative increase in usage of hypothesis patterns (keywords like "probably," "something like") compared to the problem-solving baseline. Deduction patterns increase by 26.2% (keywords like "therefore," "consequently," "logically"). In contrast, problem-solving relies much more heavily on breakdown patterns (decomposing problems into sub-steps) — the dominant strategy in the base model's reasoning when solving explicit math problems.

These numbers come from a lightweight keyword-based analysis on 200 sampled responses. The pattern groups and their associated keywords are listed in Appendix E, Table 9. For example, hypothesis patterns are identified by the presence of "probably" or "something like"; deduction patterns by "summarize," "conclusion," "therefore." This methodology has significant limitations — keyword matching is a coarse proxy for actual cognitive patterns, and the categorization is exclusive (each response is assigned to one dominant pattern) even though real reasoning involves multiple pattern types simultaneously. The paper does not report inter-annotator agreement (since there are no human annotators — the classification is purely keyword-based), nor does it validate that the keyword categories actually correspond to the intended reasoning patterns.

Despite these methodological limitations, the analysis provides suggestive evidence that RPT training changes how the model reasons, not just how accurately it predicts. The qualitative example in Table 4 reinforces this: the model's reasoning is iterative ("Wait, perhaps in the original, the next part was an example or an explanation"), explores alternatives ("Alternatively, it could be..."), and integrates multiple levels of context (semantic content, markdown structure, token-level whitespace). This pattern cannot be easily faked through keyword inclusion and suggests genuine deliberation rather than superficial pattern matching.

Table 11 (Appendix F) provides three additional case studies that support this interpretation. In Case 1, the model works through a mathematical comparison problem, generating and validating multiple hypotheses about what token follows ("One way to find a common denominator...," "Alternatively, it could be...," "Wait, but 18 x 24? That doesn't make sense") before converging on the correct prediction. In Case 2, the model engages with incomplete-sentence prediction, reflecting on what the original answer "was supposed to be" and cross-referencing with its knowledge of standard mathematical proofs. In Case 3, the model tackles a geometry problem involving circle scaling, checking coordinate representations and unit consistency before predicting. These examples are qualitatively different from the polished, linear chain-of-thought typically found in math problem-solving demonstrations — they show backtracking, doubt, and iterative refinement.

Ablation Studies and Robustness Checks

Prompt template variation (Appendix D, Table 8). The paper evaluates seven prompt templates (v0 through v6) on initial next-token reasoning performance before any RL training. The results show substantial sensitivity: Random@1 ranges from 3.0% (v0) to 6.0% (v6), and Pass@8 ranges from 8.5% (v0) to 19.0% (v6). This means the best template more than doubles Pass@8 performance relative to the worst — a large effect that the paper acknowledges but does not optimize for the main experiments. The main RPT training uses v0 (the worst-performing template) rather than v6, which the paper explicitly notes: "We leave prompt engineering based on other template variants for future work, which tends to improve the final performance." This means the reported RPT results likely represent a lower bound on achievable performance, and a well-optimized prompt could yield substantially higher numbers. However, it also means that some of RPT's gains over baselines might be attributable to suboptimal prompt choices in the baselines rather than to the RL training itself.

Reward function design (Appendix A). The paper reports investigating three alternative reward designs beyond the prefix-matching reward: (1) first-token matching only (reward based solely on whether the first predicted token matches the ground-truth next token); (2) dense reward (1 for correct prediction, language model probability for incorrect predictions as a smaller positive reward); and (3) conditional dense reward (dense rewards applied only to groups where at least one rollout was correct). The paper states these "generally achieved performance comparable to the prefix matching reward," suggesting RPT is robust to reward function design choices within the tested variations. However, no quantitative comparisons are provided — no table, no figure, no specific numbers — making it impossible to assess whether "comparable" means identical, within 1%, or within 5%. This is a significant omission for a paper whose central contribution is an RL-based training method. The reward function is the primary mechanism by which the training objective is defined, and understanding its sensitivity is crucial for reproducibility and practical adoption.

Entropy-based data filtering (Section 3.3, validated in Section 4.1). The filtering strategy uses R1-Distill-Qwen-1.5B as a proxy to compute top-16 next-token entropy at each position and filter out low-entropy tokens. The paper validates this approach indirectly through the difficulty-stratified evaluation in Table 1, which shows that RPT improves accuracy at all difficulty levels (easy, medium, hard as defined by entropy thresholds). However, there is no ablation comparing filtered vs. unfiltered training — we cannot know whether filtering is necessary for RPT's success or whether training on all tokens would yield similar or better results. If the model could learn next-token reasoning on easy tokens first before progressing to harder ones (a curriculum learning effect), filtering might actually harm performance. Conversely, if easy tokens lead to degenerate reasoning patterns (generating elaborate justifications for obvious predictions), filtering might be essential. The paper provides no evidence either way.

Base model initialization choice (implicit ablation through continual NTP baseline). The continual NTP training baseline in Table 2 serves as an implicit ablation of the RL objective versus the NTP objective, holding the training data constant. The result — NTP training on OmniMATH destroys downstream RL capability while RPT training enhances it — is the strongest evidence that the RL objective, not the data, drives the improvements. However, this is not a clean ablation of initialization: RPT initializes from R1-Distill-Qwen-14B and trains with RL; the continual NTP baseline initializes from the same model and trains with NTP. A proper ablation would also test RPT training initialized from a standard base model (Qwen2.5-14B) to determine whether R1-Distill's pre-existing reasoning capability is necessary. The paper explicitly identifies this as future work (Section 6: "investigating RPT training from a standard base language model would provide further insights into its foundational impact").

KL penalty coefficient set to zero (Appendix B, Table 5). The paper uses a KL penalty coefficient of 0, meaning there is no constraint on how far the policy can diverge from the initial R1-Distill-Qwen-14B policy. This is unusual for RL fine-tuning, where KL penalties are standard for preventing reward hacking and preserving general capabilities. The paper does not provide an ablation with non-zero KL penalty, so we cannot assess whether the zero-KL choice is important for performance or merely convenient. The absence of KL regularization is a potential stability concern — without it, the model could theoretically drift far from its initialization, potentially losing capabilities that are not reinforced by the binary reward signal (e.g., factual knowledge, linguistic competence). The paper does not evaluate whether RPT training degrades performance on tasks unrelated to next-token prediction, which would be necessary to assess the cost of zero-KL training.

Entropy loss coefficient set to zero (Appendix B, Table 5). Following Hao et al. (2025), the paper sets the entropy loss coefficient to 0 for "exact on-policy reinforcement learning." This means there is no bonus for maintaining response diversity — the model is free to collapse to a deterministic policy if that maximizes reward. The paper does not ablate this choice or report whether response diversity degrades over training. If the model learns to always output the same reasoning pattern regardless of context, this would limit generalization; the qualitative diversity in Table 11's examples suggests this may not be happening, but no quantitative diversity metric is reported.

Dynamic sampling after step 500 (Section 3.3). The paper activates dynamic sampling (Yuzz et al., 2025) starting from step 500, which adjusts the number of rollouts per context based on recent model performance. This is a training efficiency technique, not a core component of RPT. However, no ablation compares static vs. dynamic sampling, making it impossible to determine whether dynamic sampling improves, degrades, or simply accelerates training. This is particularly relevant for the scaling analysis in Figure 5, where the transition from static to dynamic sampling at step 500 means the relationship between compute and accuracy is not generated under a single, uniform training regime — the scaling curve conflates algorithmic improvements from dynamic sampling with genuine scaling of the base RPT algorithm.

Prompt template v0 vs. v6 (Appendix D, Table 8). While framed as an exploration of prompt sensitivity, this effectively serves as an ablation of instruction clarity. Template v0 uses bare instructions within ### Context markers; v6 uses more conversational framing ("You are a helpful assistant..."), code-block delimiters, and explicit listing of potential tokens. The large performance gap (Pass@8: 8.5% vs. 19.0%) demonstrates that the base model's next-token reasoning ability is highly prompt-dependent even before RL training — a finding with practical implications for prompt design in reasoning tasks. However, because the main experiments use v0 rather than the best-performing template, the reported RPT results may underestimate what a more careful prompt engineering effort could achieve.

Model scale comparison (Figure 4). The comparison between RPT-14B and R1-Distill-Qwen-32B serves as an implicit ablation of whether RPT can substitute for model scale. The finding that RPT-14B matches the 32B model on next-token prediction accuracy suggests that RPT training on a 14B model can compensate for approximately 2.3× in parameter count. However, this comparison is limited by the fact that the 32B model uses standard NTP rather than RPT — a fairer comparison would be RPT-14B vs. RPT-32B to determine whether RPT's benefits scale with model size, or RPT-14B vs. a 32B model that received some other form of reasoning-enhanced training.

Critical Assessment

Does RPT improve next-token prediction accuracy? Yes, the evidence is clear and multi-faceted. Table 1 shows consistent improvements across all difficulty levels (+3.5, +4.1, and +3.3 percentage points on easy, medium, hard tokens respectively vs. R1-Distill-Qwen-14B standard NTP). Figure 4 shows RPT-14B matching a 32B model. Figure 5 shows monotonic improvement with additional training compute. The scaling $R^2$ values are high (0.989–0.997). The baseline comparisons are appropriate — standard NTP, reasoning-without-RL, and larger models — and the improvements are directionally consistent across all comparisons.

However, the absolute magnitude of improvement deserves scrutiny. The gain over R1-Distill-Qwen-14B standard NTP is 3–4 percentage points across difficulty levels. Compared to the standard NTP baseline's accuracy of 20–42%, this represents roughly a 10–20% relative improvement — meaningful but not transformative. The comparison against next-token reasoning without RL (1.4–3.3% accuracy) is more dramatic, but this baseline is pathologically bad — the base model essentially cannot do next-token reasoning at all without RPT training, so the relevant baseline for assessing RPT's value is standard NTP, not reasoning-without-training. RPT's contribution is making next-token reasoning work (rising from ~3% to 24–45%), which is impressive, but the practical question is whether next-token reasoning via RPT is better than not reasoning at all (standard NTP). The answer is yes, by modest margins on the 14B scale tested.

Does RPT generalize beyond math? This is the most significant unverified claim. RPT is trained on OmniMATH — a mathematical corpus — and evaluated on MMLU-Pro and SuperGPQA, which span diverse domains. The zero-shot improvements in Table 3 (+2.9 on SuperGPQA, +2.2 on MMLU-Pro in reasoning mode) are real but small, and the per-category breakdowns (Tables 6, 7) show that improvements concentrate in math-adjacent categories while some non-STEM categories show slight regressions. The paper does not claim that OmniMATH-trained RPT is a general-domain pre-training method — Section 6 explicitly acknowledges the math-specific corpus as a limitation — but the zero-shot evaluation on general benchmarks implicitly tests a generalization claim. The modest and uneven improvements suggest that RPT's benefits are at least partially domain-specific, with stronger transfer to domains that share structural features with mathematical reasoning (formal reasoning, quantitative analysis) and weaker transfer to domains requiring different reasoning types (legal analysis, literary interpretation). A true test of RPT's generality would require training on a general-domain corpus (e.g., FineWeb, C4) and evaluating on a similarly broad set of tasks.

Does RPT improve RL fine-tuning? Table 2 provides evidence, but the experiment has several weaknesses. First, the sample size is extremely small: 256 training examples and 200 test examples. On a set this small, the difference between 52.7% and 58.3% (5.6 percentage points, or about 11 more correct answers out of 200) could be influenced by the specific random split. No cross-validation or multiple random seeds are reported. Second, the Skywork-OR1 dataset consists of reasoning problems with verifiable answers — structurally similar to the OmniMATH corpus used for RPT pre-training. The RPT advantage might reflect domain similarity rather than a general improvement in RL trainability. Testing RL fine-tuning on a completely unrelated domain (e.g., code generation, dialogue, summarization) would provide stronger evidence that RPT's benefits are objective-driven rather than data-driven. Third, the "Continual NTP training" baseline is trained on the same OmniMATH corpus, making it a fair comparison for objective choice, but it is unclear whether the NTP training hyperparameters were tuned or simply set to defaults. If the NTP training was suboptimal (wrong learning rate, wrong batch size, catastrophic forgetting), the collapse to 10.7% might reflect poor experimental design rather than a fundamental property of NTP.

The catastrophic collapse from continual NTP (Table 2: from 51.2%→10.7%). This is the paper's most striking single result, but it raises questions the paper does not answer. What mechanism causes additional NTP training on math texts to destroy downstream RL capability? The paper offers no analysis — no probing of what changed in the model's representations, no diagnosis of whether the model lost general reasoning ability or specifically lost the ability to generate chain-of-thought. One plausible mechanism is catastrophic forgetting of the R1-distilled reasoning format: the NTP training teaches the model to predict next tokens directly without \boxed{} formatting, so when subsequently asked to reason with RLVR, the model has forgotten or suppressed the \boxed{} extraction convention. If this is the case, the 13.0% post-RLVR accuracy might simply reflect the fact that the model rarely produces extractable answers, not that it has lost reasoning ability. Without analyzing why continual NTP fails, we cannot assess whether the failure generalizes or is an artifact of format mismatch.

Do the scaling laws in Figure 5 establish RPT as a reliable scaling paradigm? The scaling analysis is the weakest part of the paper empirically, despite being central to the paper's positioning as a "new scaling paradigm." Six data points over one order of magnitude in compute, fit to a three-parameter power law, with no out-of-distribution validation and no reported parameter uncertainties — this does not meet the evidentiary standard for establishing a scaling law. The $R^2$ values are high, but with only six points and three free parameters, even a linear function could achieve respectable $R^2$ over a narrow range. The figures from Kaplan et al. (2020) and Hoffmann et al. (2022) that established NTP scaling laws spanned multiple orders of magnitude in compute and validated the functional form through extrapolation to models of different sizes. RPT's scaling analysis is an encouraging preliminary result that justifies further scaling studies — it does not, by itself, establish that RPT follows a predictable power-law relationship that can guide resource allocation decisions.

Missing experiments that would substantially strengthen the paper. Several experiments are conspicuously absent. (1) RPT training from a standard base model (Qwen2.5-14B rather than R1-Distill-Qwen-14B): this would test whether RPT's reasoning improvements require a pre-existing reasoning capability or can be bootstrapped from scratch. (2) Training on a general-domain corpus: this would test the central claim that RPT can scale RL to web text, not just math. (3) Larger-scale RL fine-tuning experiments: testing RPT's transfer to RLVR on multiple downstream tasks with larger datasets would establish robustness. (4) Ablation of data filtering: comparing filtered vs. unfiltered training would determine whether the entropy-based filtering is necessary or merely accelerates training. (5) Diversity metrics: tracking whether the model's reasoning patterns collapse to a few templates over training would address concerns about zero-KL, zero-entropy training. (6) Evaluation on held-out capabilities: testing whether RPT training preserves or degrades the model's performance on tasks unrelated to next-token prediction (e.g., translation, summarization, factual recall) would characterize the specificity of RPT's effects. (7) Prompt optimization: since prompt template v6 doubles initial Pass@8 over v0 (19.0% vs. 8.5%), training RPT with v6 rather than v0 could yield substantially better results — this is a "free" improvement the paper leaves on the table.

Do the reasoning pattern results (Figure 6, Tables 4, 11) support the claim that RPT incentivizes "deeper understanding"? The qualitative evidence is suggestive and aligns with the paper's narrative. The examples show iterative, self-critical reasoning that goes beyond surface-level pattern matching. However, the quantitative analysis is thin — keyword matching on 200 responses, with no human validation of pattern categories, no inter-annotator agreement, and no comparison to what the model's reasoning looks like after equivalent NTP training. The 161.8% increase in hypothesis patterns is a large relative number, but without knowing the absolute base rates, it could reflect an increase from 5% to 13% of responses — meaningful but not dominant. The paper does not establish a causal link between specific reasoning patterns and prediction accuracy; we do not know whether the increase in hypothesis/deduction patterns causes the accuracy improvements or is merely correlated with them. This remains at the level of suggestive qualitative analysis rather than rigorous behavioral characterization.

Overall assessment of experimental evidence. The paper provides clear evidence that RPT improves next-token prediction accuracy relative to standard NTP on mathematical text, and suggestive evidence that these improvements transfer to RL fine-tuning and zero-shot reasoning tasks. The scaling analysis is preliminary but encouraging. The catastrophic failure of continual NTP on OmniMATH is a genuinely important finding that deserves further investigation, though its interpretation is limited by the lack of mechanistic analysis. The paper's main limitation is not in what it shows but in the scope of what it tests: one model size (14B), one domain (math), one initialization (R1-Distill), one reward design (prefix-matching binary), one prompt template (v0), and relatively small evaluation sets. The claimed generality — "a scalable and general-purpose approach to RL pre-training" — extends beyond what the experiments demonstrate, and the paper's Section 6 limitations acknowledge this candidly. The experiments establish RPT as a promising proof of concept whose generality and scalability remain to be validated.

6. Limitations and Trade-offs

The Pre-Training Corpus Is Mathematics-Specific, Making Generality Claims Unverified

The assumption or constraint. The paper positions RPT as a "scalable method to leverage vast amounts of text data for general-purpose RL" (Abstract) and a "scaling paradigm to advance language model pre-training" (Section 1). However, all RPT experiments use the OmniMATH dataset — 4,428 competition-level mathematical documents — as the pre-training corpus. The paper acknowledges this explicitly in Section 6: "the current pre-training corpus predominantly consists of mathematical documents; future work will explore its efficacy on broader, general-domain text."

The consequence. The central claim — that RPT can scale RL to web-text corpora — is not empirically tested. Mathematical text has structural properties that make next-token reasoning unusually well-suited to RPT's design: tokens are semantically dense (each symbol carries substantial meaning), continuations are logically constrained rather than stylistically conventional, and alternative completions are often demonstrably wrong rather than merely less probable. General web text — dialogue, narration, opinion, description — has fundamentally different properties: many plausible next tokens may be acceptable, correctness is often a matter of convention rather than logic, and reasoning may not be necessary for accurate prediction (the most probable next word in "I went to the store to buy some ___" is determined by collocation statistics, not deductive inference). RPT's binary reward design assumes a well-defined ground truth — the exact continuation from the corpus — but in general text, this ground truth is only one of many acceptable continuations. The model might correctly predict "milk" but be penalized because the corpus says "bread," teaching it that valid predictions are wrong. Whether RPT remains effective when "correctness" becomes ambiguous is unknown.

What evidence exists in the paper. The zero-shot evaluation on MMLU-Pro and SuperGPQA (Table 3) provides indirect evidence of domain transfer, but it tests downstream task performance, not next-token prediction accuracy on general text. The per-category breakdowns (Tables 6, 7) show that RPT's gains are uneven, with improvements concentrated in math-adjacent categories (Math +5.5, Chemistry +3.0 on MMLU-Pro) while some non-STEM categories show slight regressions (Law -0.5 on SuperGPQA, Economics -3.0 on MMLU-Pro). This pattern is consistent with domain-specific rather than general improvement. No experiment evaluates RPT's next-token prediction accuracy on general-domain text (e.g., Wikipedia, books, news), which is the direct test of whether the method works as claimed.

Mitigation status. The paper does not attempt to mitigate this limitation experimentally. Section 6 delegates it entirely to future work: "We would like to scale up the training corpus, including data size, and domain coverage. Large-scale general Internet data can be utilized during reinforcement pre-training." This is an appropriate acknowledgment, but it means the paper's central claim of generality remains a hypothesis, not a demonstrated result.

Difficulty Estimation for Token Filtering Uses a Proxy Model and Its Necessity Is Not Ablated

The assumption or constraint. RPT's training pipeline includes a preprocessing step (Section 3.3) that uses DeepSeek-R1-Distill-Qwen-1.5B — a separate, smaller model — to compute top-16 next-token entropy at each position in the corpus, filtering out low-entropy tokens so that RPT focuses training compute on "challenging tokens that require greater computational effort to predict." This filtering is described as an integral part of the RPT setup, but its relationship to RPT's success is never isolated.

The consequence. The filtering introduces two unverified dependencies. First, the choice of proxy model matters. Different proxy models (different sizes, different architectures, different training distributions) would produce different entropy estimates and therefore different filtered training sets. The paper provides no sensitivity analysis — we do not know whether RPT's results depend on the specific proxy model used or would transfer to other reasonable proxies. Second, and more fundamentally, it is unknown whether filtering is necessary at all. If RPT were trained on all tokens (both high-entropy and low-entropy), the model might learn next-token reasoning on easy tokens before progressing to harder ones — a natural curriculum. Alternatively, training on easy tokens might teach degenerate reasoning patterns (generating elaborate justifications for obvious predictions) that harm performance. Without an unfiltered baseline, neither hypothesis can be evaluated. The paper's claim that filtering "prioritiz[es] training on challenging tokens" implies this prioritization is beneficial, but no evidence supports this outside the indirect difficulty-stratified evaluation in Table 1, which shows RPT improves accuracy at all levels — evidence for the method's effectiveness, not for filtering's necessity.

What evidence exists in the paper. No experiment compares filtered vs. unfiltered RPT training. The entropy-based data filtering is described in Section 3.3 and its difficulty thresholds are used for evaluation stratification in Section 4.1 (entropy > 0.5, 1.0, 1.5 defining easy, medium, hard splits), but the training-time filtering threshold is not specified, and there is no ablation removing the filter. The paper evaluates RPT's accuracy on tokens of all difficulty levels (Table 1) but trains only on filtered data — we cannot know which tokens received RPT training and which did not, making it impossible to distinguish "RPT improves accuracy on easy tokens" from "RPT was trained only on hard tokens but transfers to easy ones."

Mitigation status. Not addressed. The filtering is presented as a design choice, not as a hypothesis to be tested. The paper does not discuss alternative filtering strategies (different proxy models, different entropy thresholds, different filtering criteria), does not provide an unfiltered baseline, and does not list filtering ablation as future work in Section 6. This is a significant gap because filtering is a preprocessing step that discards training data — if it is unnecessary or harmful, RPT's data efficiency claims are overstated; if it is essential, RPT's practicality depends on access to a suitable proxy model and a reliable entropy calibration procedure that the paper does not specify.

Training Is Initialized from a Reasoning Model, Leaving the Cold-Start Problem Unsolved

The assumption or constraint. All RPT experiments initialize from DeepSeek-R1-Distill-Qwen-14B, a model that has already been fine-tuned (via distillation from the full DeepSeek-R1) to generate chain-of-thought reasoning when prompted. The paper explicitly states this is a pragmatic choice: "R1-Distill-Qwen-14B serves as a good starting point for reinforcement learning due to its basic reasoning capabilities" (Section 3.3). Section 6 acknowledges this as a limitation: "RPT training is initialized from a reasoning model; investigating RPT training from a standard base language model would provide further insights into its foundational impact."

The consequence. This initialization choice means RPT is not demonstrated as a pre-training method in the standard sense — it is closer to continued training or specialization of an already-capable reasoning model. The cold-start problem — how to get RL training started when the model cannot yet generate coherent reasoning chains or correctly formatted boxed predictions — is entirely avoided by starting from a model that already possesses these capabilities. The consequence is that RPT's applicability to training from scratch (or from a standard base model like Qwen2.5-14B, Llama-3, etc.) is completely unknown. If initialized from a base model, the initial rollouts would likely be unstructured text without valid \boxed{} extractions, yielding zero reward for all responses. In a GRPO setting with group-based advantages, if all G rollouts receive zero reward, the advantage for each is zero (all equal to the group mean), and no learning occurs. The model would never escape this zero-reward trap without some mechanism to bootstrap initial reasoning capability — the exact mechanism that R1-Distill initialization provides but that is unavailable in true pre-training from scratch.

What evidence exists in the paper. None, by design. The paper provides no RPT training run from a non-reasoning initialization. The closest evidence is the catastrophic failure of continual NTP training on OmniMATH (Table 2: 51.2% → 10.7% pre-RL accuracy, 52.7% → 13.0% post-RL accuracy), which demonstrates that the R1-Distill initialization is fragile — additional NTP training destroys the very reasoning capability that RPT relies on. This fragility raises the stakes for the cold-start question: if the reasoning capability needed to start RPT can be destroyed by further training, how would it ever emerge from a model that lacks it entirely?

Mitigation status. The paper acknowledges this as future work (Section 6) but does not propose a bootstrapping mechanism. Several approaches could be tested: (1) a warmup phase using supervised fine-tuning on reasoning demonstrations to establish basic chain-of-thought and boxed-output formatting; (2) a curriculum that starts with extremely easy tokens where even simple heuristics can produce correct boxed predictions, providing non-zero reward to get the RL signal started; (3) reward shaping during early training to provide partial credit for format compliance even when predictions are wrong. None of these are explored. Until the cold-start problem is solved, RPT remains a method for enhancing already-reasoning models rather than a true pre-training paradigm.

The FLOPs-Matched Comparison with Larger Models Is Absent, Leaving the Scaling Claim Uncalibrated

The assumption or constraint. The paper claims RPT is a "new scaling paradigm" (Abstract) and presents scaling curves in Figure 5 showing that next-token prediction accuracy improves with RL compute. However, the paper does not perform a FLOPs-matched comparison between RPT training and the standard alternative — using the same total compute budget to do more NTP training, or to train a larger model with NTP, or to do extended inference-time reasoning. Without such comparisons, the reader cannot assess whether RPT is an efficient way to spend compute or merely a way that happens to improve performance.

The consequence. The scaling claim has ambiguous practical import. Consider a practitioner with a fixed compute budget for improving a 14B model's language modeling capability. They could: (a) run RPT training for 1,000 steps (the main experiment in this paper), (b) do additional NTP training on more data for the same FLOPs, (c) distill from a larger model, or (d) simply use a 32B model with standard NTP (which Figure 4 shows achieves comparable next-token prediction accuracy to RPT-14B — ~32% vs. ~34%). The paper provides no evidence that option (a) is superior to options (b), (c), or (d) in a FLOPs-matched sense. The 14B vs. 32B comparison in Figure 4 is suggestive but not FLOPs-matched — we do not know the training compute that produced R1-Distill-Qwen-32B, nor the RPT training compute, so we cannot compare total compute invested vs. accuracy achieved. RPT might be computationally expensive relative to the improvement it provides — the RL training requires generating 8 rollouts × 256 contexts × average response length per step, which is substantially more inference compute per training token than standard NTP, which requires only a single forward pass per token.

What evidence exists in the paper. Figure 4 shows RPT-14B achieving ~34% average accuracy vs. R1-Distill-Qwen-32B's ~32%, but without FLOPs accounting for either model's training. Figure 5 shows RPT accuracy improving with RL compute, but there is no NTP baseline on the same plot to compare slopes — we cannot see whether additional NTP compute would improve accuracy faster, slower, or equivalently. The paper's closest attempt at an objective comparison is Table 2's continual NTP baseline, but this shows NTP destroying downstream RL capability rather than competing with RPT on next-token prediction — it answers a different question (objective alignment for transfer) rather than the efficiency question (compute per unit of next-token accuracy improvement).

Mitigation status. Not addressed. The paper does not position FLOPs-matched comparison as necessary, nor does it list it as future work. Section 6 proposes "establish[ing] scaling laws for reinforcement pre-training to guide the scaling of large language models" but does not mention the need for comparative scaling laws that pit RPT against NTP at equal compute. Without such comparisons, "scaling paradigm" claims remain aspirational — RPT is a method that scales, but whether it scales better than alternatives is unknown.

Evaluation Uses Small Test Sets with No Statistical Quantification, Limiting Reliability of Claims

The assumption or constraint. The paper's main results are evaluated on very small datasets: 200 held-out samples from OmniMATH for language modeling (Section 4.1), 200 test instances from Skywork-OR1 for RL fine-tuning (Section 4.3), and the full MMLU-Pro and SuperGPQA test sets for zero-shot evaluation (Section 4.4). No confidence intervals, standard deviations, statistical significance tests, or multiple random seeds are reported for any result. The paper does not describe how the 200-sample validation and test sets were created (random split? fixed split? any stratification?), making it impossible to assess whether the reported differences could arise from sampling noise.

The consequence. The headline comparisons rest on differences that may not be statistically reliable at the reported sample sizes. On a 200-sample test set, the standard error of an accuracy estimate is approximately $\sqrt{p(1-p)/200}$. For the RL fine-tuning result (Table 2), the difference between RPT-14B (58.3%) and the baseline (52.7%) is 5.6 percentage points. At $p \approx 0.55$, the standard error is roughly $\sqrt{0.55 \times 0.45 / 200} \approx 3.5$ percentage points — meaning the observed difference is less than two standard errors from zero. A simple binomial test would not reject the null hypothesis of equal accuracy at conventional significance levels. Similarly, for the language modeling evaluation (Table 1), with 200 samples split across three difficulty levels (each containing some fraction of the 200), the per-level sample sizes are even smaller. The 3.5 percentage point improvement on easy tokens (45.11% vs. 41.60%) could arise from roughly 7 more correct predictions out of a subsample that might contain only 100–150 tokens — well within sampling noise. The zero-shot evaluations on MMLU-Pro and SuperGPQA use larger test sets (thousands of questions), making the 2–3 percentage point improvements there more likely to be statistically meaningful, but the per-category breakdowns (Tables 6, 7) again involve small per-category samples where noise dominates.

What evidence exists in the paper. No statistical quantification whatsoever. No error bars on any figure (Figures 4, 5). No confidence intervals in any table (Tables 1, 2, 3, 6, 7, 8). No mention of random seeds, dataset splits, or reproducibility procedures. The scaling analysis in Figure 5 reports $R^2$ values, which quantify goodness-of-fit for the power-law curve but say nothing about the uncertainty of individual accuracy measurements. The prompt template comparison (Table 8) reports Pass@8 and Random@1 without indicating how many contexts were evaluated, making the reliability of those percentages unclear.

Mitigation status. Not addressed. The paper does not discuss evaluation reliability as a concern, does not report multiple evaluation runs or cross-validation, and does not list improved evaluation methodology as future work. For a paper making quantitative claims about accuracy improvements, this is a significant omission. At minimum, reporting bootstrapped confidence intervals for the main comparisons (Table 1, Table 2, Table 3) would allow readers to assess which differences are likely to replicate and which are consistent with noise. At 200 test samples, several of the paper's central numerical claims — including the exact magnitude of RPT's improvement over baselines — should be treated as approximate estimates rather than precisely measured quantities.

The Revision-Free Nature of RPT Means It Cannot Recover from Initial Reasoning Errors Within a Rollout

The assumption or constraint. RPT's training procedure generates $G = 8$ independent reasoning trajectories per context, each producing a single chain-of-thought and a single final prediction. There is no mechanism for the model to revise its prediction based on intermediate verification, to backtrack within a reasoning chain when it detects an inconsistency, or to generate multiple candidate predictions and select among them. The reward signal is binary and based solely on the final boxed answer — if the reasoning chain leads to an incorrect conclusion, that entire rollout receives zero reward regardless of whether the reasoning was mostly correct but derailed at the last step.

The consequence. This is a fundamental structural constraint, not an implementation detail. In standard RLVR for math problem-solving (e.g., DeepSeek-R1, Guo et al., 2025), the model generates a single long chain-of-thought and can self-correct within that chain — if it notices an error mid-reasoning, it can backtrack and try a different approach, all within a single rollout that ultimately produces a correct answer and receives positive reward. RPT's next-token reasoning task does not permit this because the "correct answer" is a single token (or short token sequence), and the reasoning chain is generated before the answer is known to the model. The model cannot verify its prediction against the ground truth during reasoning (the ground truth is hidden), so it cannot learn mid-rollout error correction in the same way. The RL signal operates at the trajectory level — entire rollouts are reinforced or suppressed — rather than at the step level. This means RPT optimizes for reasoning strategies that tend to produce correct answers on average but may not optimize for the ability to detect and correct errors within a single reasoning episode, which is a hallmark of advanced reasoning behavior.

What evidence exists in the paper. The qualitative examples in Table 4 and Table 11 do show the model engaging in self-doubt and alternative consideration ("Wait, but 18 x 24? That doesn't make sense," "Alternatively, it could be..."), suggesting that some form of internal self-correction emerges even under trajectory-level RL. However, this self-correction is not directly incentivized — it emerges only to the extent that it helps produce correct final predictions. The paper does not measure how often the model's reasoning contains errors that go uncorrected, how often the model changes its mind during reasoning, or how often the final boxed answer differs from an intermediate hypothesis stated in the reasoning chain. The reasoning pattern analysis (Figure 6) shows a 161.8% increase in hypothesis patterns and a 26.2% increase in deduction, but does not analyze error detection or correction patterns specifically.

Mitigation status. Not addressed as a limitation. The paper frames the next-token reasoning format as a feature — it enables verifiable RL on raw text — rather than as a constraint on the type of reasoning that can be learned. However, this tradeoff is fundamental: the format that makes RPT scalable (extract a single boxed answer, verify against ground truth) is also what prevents RPT from training the model on multi-step verification, backtracking from dead ends, or iterative refinement within a single reasoning episode. Future work could explore whether allowing the model to output multiple candidate predictions and receive credit for the best one, or providing step-level rewards based on intermediate reasoning quality, could recover some of these capabilities while maintaining scalability. The paper's Section 6 does not identify this as a direction for future investigation.

7. Implications and Future Directions

How This Work Changes the Landscape

RPT introduces a category-level reframing of language model pre-training, not an incremental improvement to existing methods. The shift is conceptual rather than algorithmic: next-token prediction, long treated as a supervised learning problem, is recast as a reinforcement learning problem where the model generates explicit reasoning before predicting and receives a binary correctness reward derived from the corpus continuation itself. This dissolves a boundary the field has largely taken for granted — that pre-training learns from unlabeled text via maximum likelihood, while RL learns from labeled feedback on curated data — by revealing that the pre-training corpus itself contains a universal, intrinsic reward signal: whether the model can correctly predict what comes next.

The magnitude of the shift should not be overstated given the experimental scope. RPT is demonstrated on a single 14B model, trained on a mathematics-specific corpus (OmniMATH), initialized from an already-reasoning checkpoint (R1-Distill-Qwen-14B), and evaluated on relatively small test sets. This is a proof of concept, not a demonstration at frontier scale. However, the conceptual move has genuine novelty: it is the first work to show that RL with binary, rule-based rewards — the same family of algorithms used in DeepSeek-R1 for post-training reasoning enhancement — can be applied at the token level across an entire pre-training corpus without external annotation, and that doing so improves next-token prediction accuracy while producing qualitatively different reasoning patterns. The paper's closest prior work, Quiet-STaR (Zelikman et al., 2024), used a different reward mechanism (helpfulness-based rather than correctness-based) that the paper convincingly argues is vulnerable to a specific form of reward hacking absent in RPT. By demonstrating that binary correctness rewards work and are robust to reward function variations (Appendix A), RPT establishes a simpler, more principled baseline for future work than the reward-shaping approaches that preceded it.

The paper reconciles a tension that has been implicit but unresolved in the literature: the objective gap between pre-training and reinforcement fine-tuning is not merely a theoretical inconvenience but a measurable, practically significant barrier. Table 2's continual NTP training baseline provides the clearest evidence for this claim in the paper. When R1-Distill-Qwen-14B is further trained with standard next-token prediction on OmniMATH — the same data used for RPT — its downstream RLVR performance collapses from 52.7% to 13.0%, and RL fine-tuning cannot recover it. This is not a small degradation; it is catastrophic, and it demonstrates that additional NTP training can be actively harmful to subsequent RL capability. Prior to this result, the field's default assumption — implicit in the universal practice of NTP pre-training followed by RL fine-tuning — was that better language models (as measured by perplexity) are better foundations for downstream training. RPT provides counterevidence: representations optimized for next-token likelihood can become less suitable for RL-based fine-tuning when pushed further in the NTP direction. This finding, if it replicates across models and domains, has significant implications for how multi-stage training pipelines are designed.

The work makes several research directions more attractive:

• RL-based pre-training at scale becomes a legitimate research program rather than a theoretical curiosity. Before RPT, there was no demonstration that RL could be applied to raw text pre-training in a way that improved downstream performance. Now there is a concrete method, an open-source implementation (based on the verl library), and preliminary scaling curves suggesting power-law behavior. Researchers can build on this directly.

• Studying the pre-training/post-training objective gap becomes a first-class empirical question rather than an acknowledged-but-ignored property of training pipelines. The continual NTP baseline in Table 2 provides a template for measuring this gap: train a model with NTP on the same data used for an alternative objective (RPT), then compare downstream RL transfer. This diagnostic can be applied to any proposed pre-training objective, not just RPT.

• Token-level difficulty filtering as a design dimension gains empirical motivation. RPT's entropy-based filtering (Section 3.3) is presented as a pragmatic choice without ablation, but the idea that pre-training compute should be concentrated on tokens where reasoning is necessary — rather than spread uniformly across all tokens — is a novel resource allocation principle that could generalize beyond RPT to other compute-intensive pre-training methods.

Some research directions become less attractive, or at least require re-examination:

• Reward shaping for RL-based language model training is called into question. The paper's Appendix A reports that alternative reward designs (dense rewards, conditional dense rewards, first-token-only matching) performed comparably to the simple binary prefix-matching reward. If binary correctness is sufficient — and may actually be preferable because it eliminates reward hacking pathways — then the substantial effort invested in designing sophisticated reward functions for language model RL (in Quiet-STaR, in RLHF reward modeling, in various intrinsic motivation schemes) may be misallocated relative to simply making the task structure produce verifiable outcomes.

• The assumption that more NTP pre-training is always beneficial — implicit in the race toward ever-larger pre-training runs — is challenged. If additional NTP training can destroy downstream RL capability (Table 2), then organizations planning to fine-tune models with RL may want to stop NTP pre-training earlier and switch to RL-based objectives, rather than maximizing NTP tokens at all costs.

• Prompt engineering for reasoning tasks is shown to matter enormously for next-token reasoning specifically, with template v6 achieving 19.0% Pass@8 vs. v0's 8.5% (Table 8) — a more than 2× difference. This suggests that careful prompt design is a high-leverage investment for any method involving prompted reasoning, and that papers reporting reasoning results without prompt optimization may be substantially underestimating achievable performance.

Follow-Up Research This Work Enables

RPT training from a standard base model (e.g., Qwen2.5-14B) rather than from an already-reasoning R1-Distill checkpoint. This is the most critical missing experiment. RPT's current results depend on initializing from a model that already generates chain-of-thought and understands the \boxed{} extraction format. Training from a base model would test whether the RL signal alone can bootstrap both reasoning capability and format compliance, or whether a cold-start mechanism (e.g., a supervised warmup phase on reasoning demonstrations, or a curriculum that begins with extremely easy tokens where even simple heuristics produce correct boxed predictions) is necessary. A negative result — RPT fails to learn from a base model because all rollouts receive zero reward — would clarify that RPT is a fine-tuning method for reasoning models, not a pre-training method in the standard sense. A positive result — RPT bootstraps reasoning from scratch — would substantially strengthen the claim that RL with intrinsic rewards can replace NTP as a general-purpose pre-training objective.

RPT training on a general-domain corpus (e.g., FineWeb, C4, or The Pile) with evaluation on diverse next-token prediction and downstream tasks. The paper's central claim — that RPT can scale RL to web-text corpora — is untested. OmniMATH is a mathematics corpus where next-token continuations are logically constrained; general web text has fundamentally different properties (multiple acceptable continuations, convention-driven rather than logic-driven token choice, stylistic variation). A replication on general-domain data would need to address whether the binary prefix-matching reward remains appropriate when the corpus continuation is only one of many valid options, or whether a softer reward (partial credit for semantically plausible but non-matching predictions) becomes necessary. The experiment would measure next-token prediction accuracy on held-out general text (not just math) and downstream performance on a broad benchmark suite (not just MMLU-Pro and SuperGPQA) to determine whether RPT's benefits are domain-specific or genuinely general.

Ablation of entropy-based token filtering: RPT trained on all tokens vs. RPT trained only on high-entropy tokens, with matched compute budgets. The paper never isolates the effect of its filtering strategy. Training RPT on unfiltered data would test two competing hypotheses: (1) filtering is necessary because easy tokens produce degenerate reasoning patterns (elaborate justifications for obvious predictions) that harm the model's reasoning quality, or (2) filtering is unnecessary or harmful because easy tokens provide a natural curriculum (the model learns basic next-token reasoning on easy cases before progressing to harder ones). A strong experiment would run three conditions — unfiltered, entropy threshold = 0.5, and entropy threshold = 1.5 — and measure both next-token prediction accuracy and reasoning pattern quality (using the keyword-based analysis from Section 4.5, but validated with human annotation) across all difficulty levels. If filtering proves unnecessary, RPT's data efficiency claims are weakened (it requires preprocessing and discards data); if filtering proves essential, RPT inherits a dependency on a suitable proxy model for difficulty estimation.

FLOPs-matched comparison between RPT and additional NTP pre-training on the same corpus, measuring both next-token prediction accuracy and downstream RLVR transfer. The paper's scaling curves (Figure 5) show RPT accuracy improving with RL compute, but provide no comparison to how much NTP training on the same FLOPs would improve accuracy. A matched comparison would allocate an identical compute budget to three conditions: (a) RPT training for 1,000 steps (the current experiment), (b) NTP training on OmniMATH for equivalent FLOPs (NTP requires only a single forward pass per token, no rollout generation, so this would involve training on substantially more data), and (c) a hybrid approach (e.g., 500 steps NTP followed by 500 steps RPT). The comparison would answer the paper's implicit efficiency question: does RPT provide better return on compute than simply doing more of what already works (NTP)? The continual NTP baseline in Table 2 already provides a partial answer for downstream RL transfer (NTP is catastrophic), but the efficiency question for next-token prediction accuracy — the direct metric RPT optimizes — remains open.

Mechanistic analysis of why continual NTP training destroys downstream RL capability (Table 2: 51.2% → 10.7% pre-RL, 52.7% → 13.0% post-RL). This is the paper's most striking single result, but the paper offers no diagnosis of the mechanism. Plausible hypotheses include: (1) format forgetting — the model loses the \boxed{} extraction convention, so post-NTP RLVR rollouts rarely produce extractable answers; (2) representation collapse — NTP training pushes the model's internal representations toward surface-level statistical patterns that are poorly suited for chain-of-thought generation; (3) capability interference — NTP training on math text reinforces direct-answer patterns that compete with the deliberative reasoning patterns needed for RLVR. A mechanistic study would probe the model after NTP training to measure: format compliance rate (what fraction of rollouts produce valid \boxed{} extractions?), reasoning chain length and diversity (are rollouts shorter or more repetitive?), and hidden-state representations (do probing classifiers detect degradation in features relevant to reasoning?). Distinguishing between format forgetting (a superficial problem fixable by better prompting) and capability degradation (a fundamental problem) would determine whether the NTP→RL gap is a serious obstacle or an avoidable implementation issue.

Scaling RPT to larger models (e.g., 32B, 70B) and larger corpora with multi-order-of-magnitude compute ranges to establish reliable scaling laws. The paper's scaling analysis (Figure 5) uses six data points over roughly one order of magnitude in compute — insufficient to distinguish a power law from a saturating curve or to estimate asymptotic performance. A scaling law study modeled on Hoffmann et al. (2022) would train RPT at multiple model scales (e.g., 1.5B, 7B, 14B, 32B) with multiple compute budgets per scale, spanning at least two to three orders of magnitude in FLOPs, and would fit scaling laws for both next-token prediction accuracy and downstream task performance. Such a study would answer: does RPT's benefit over NTP increase, decrease, or stay constant with model scale? Is there a compute threshold below which RPT is not worth the overhead? What is the asymptotic accuracy ceiling for next-token prediction under RPT? Without these answers, the claim that RPT is a "scaling paradigm" remains an aspiration backed by suggestive but insufficient evidence.

Practical Applications and Downstream Use Cases

Improving small-model performance for on-device or low-latency deployment. Figure 4 demonstrates that RPT-14B (~34% average next-token prediction accuracy) matches or slightly exceeds R1-Distill-Qwen-32B (~32%) on language modeling accuracy — a ~2.3× larger model. If this pattern generalizes beyond mathematics, RPT could enable 14B-parameter models deployed on consumer hardware (laptops, phones) to achieve accuracy that would otherwise require a 32B model requiring datacenter GPUs. The practical workflow would be: take an existing reasoning-capable small model (e.g., a distilled variant like R1-Distill-Qwen-7B or 14B), apply RPT on a domain-relevant corpus (technical documentation for a coding assistant, medical literature for a clinical tool, legal texts for a contract analysis system), and deploy the resulting model with improved next-token prediction accuracy that would otherwise require a larger, more expensive model. The 3–4 percentage point accuracy improvement over standard NTP (Table 1) is modest in absolute terms but meaningful when it enables a smaller model to cross a usability threshold that previously required larger scale.

Pre-training for RL-heavy downstream pipelines where the objective gap matters. Table 2 provides the clearest practical signal: if a model will ultimately be fine-tuned with RLVR (for math tutoring, code generation with execution feedback, formal verification tasks), continuing to pre-train it with NTP on domain data may be actively harmful. Organizations building RL-fine-tuned systems should consider replacing late-stage NTP pre-training with RPT on domain corpora. The specific benefit quantified in the paper is a 5.6 percentage point improvement in final RLVR accuracy (58.3% vs. 52.7%) at the cost of 1,000 RPT training steps on 4,428 math documents — a relatively modest compute investment compared to the initial pre-training cost of the base model. The diagnostic value of the continual NTP baseline is also practically useful: teams can run a small-scale version of this experiment (NTP vs. RPT on their domain data, followed by RLVR on their downstream task) to determine whether their pipeline suffers from the objective gap before committing to a full RPT training run.

Data generation for self-improvement loops where reasoning quality matters. RPT's training process generates millions of reasoning trajectories (256 contexts × 8 rollouts × 1,000 steps ≈ 2 million rollouts) as a byproduct of RL training. These trajectories contain the model's chain-of-thought for next-token prediction across diverse contexts, including both successful (reward = 1) and unsuccessful (reward = 0) reasoning attempts. The successful trajectories could be filtered and used as supervised fine-tuning data for the next generation of models — a self-improvement loop where RPT both trains the current model and generates training data for future models. Unlike standard NTP pre-training, which produces no intermediate reasoning data, RPT's rollout generation is a natural source of reasoning demonstrations that can be mined for quality and reused. The paper does not explore this, but the infrastructure is already in place: the GRPO training loop generates, scores, and discards rollouts; saving the high-reward rollouts and using them for supervised fine-tuning of a subsequent model would be a straightforward extension.

When to Prefer This Method

The paper does not articulate an explicit tradeoff between RPT and named alternatives with specific conditions for choosing one over another. The experiments compare RPT against standard NTP and against next-token reasoning without RL, but the paper frames RPT as a new paradigm to be explored rather than as a replacement for existing methods that should be preferred under certain conditions. The Section 6 limitations acknowledge that RPT has been demonstrated only on mathematical text with reasoning-model initialization, and the future work directions focus on scaling up and generalizing rather than on defining deployment decision rules. Producing a "prefer RPT when X, prefer NTP when Y" matrix would impose a level of prescriptive specificity that the paper itself does not provide and that the experimental evidence cannot yet support given the narrow domain and model scale tested.