ArXiv: 2510.14943

🎯 Pitch

A model's own assessment of its reasoning can be read from a single next-token probability distribution at the final token—no extra generation needed. LaSeR exploits this to train self-verification jointly with reasoning at virtually zero overhead, matching the accuracy of separate reward models 10× its size.

1. Executive Summary

This paper proposes LaSeR (Reinforcement Learning with Last-Token Self-Rewarding), a lightweight algorithm that jointly optimizes reasoning and self-verification capabilities within a single LLM during RLVR by adding a simple MSE loss that aligns the model's "last-token self-rewarding score" (the difference between the policy model's next-token log-probability for a pre-specified special token at the final response token and a pre-calculated constant, scaled by the KL coefficient) with the verifier-based reasoning reward. Experiments on LLaMA and Qwen architectures—including OctoThinker-3B-Short-Base, Qwen2.5-7B-Base, and Open-Reasoner-Zero-7B—trained with GRPO on DeepMath-103K and evaluated across five math reasoning benchmarks (MATH500, AMC23, AIME24, AIME25, OlympiadBench) demonstrate that LaSeR not only improves reasoning accuracy (e.g., raising Qwen2.5-7B-Base's average from 41.8% to 42.7%) but also equips the model with self-verification F1 scores around 80%—matching or exceeding same-sized external reward models and rivaling a 72B reward model—while incurring only one additional token inference at test time, establishing that self-verification can be derived from a single next-token probability distribution rather than from expensive separate generation passes.

2. Context and Motivation

The Core Problem: LLMs Need Verification Signals Beyond Training Time

Large Language Models have made dramatic progress in recent years, but they still face fundamental limitations on complex reasoning tasks—particularly in mathematics, where multi-step logical deduction is required. The paper situates itself within a specific gap: standard RLVR (Reinforcement Learning with Verifiable Rewards) works brilliantly during training, when ground-truth answers are available to provide binary correctness signals, but goes completely silent at test time, when those answers are unknown. Without verification signals, the model has no way to assess the quality of its own outputs, rank candidate solutions, or perform the kind of best-of-N weighted selection that has become standard for inference-time scaling.

This matters for two practical reasons. First, deployment reality: in any real-world application—a math tutoring system, a code generation assistant, a scientific reasoning tool—the ground-truth answer is, by definition, unavailable. The system must operate without a teacher. If the only mechanism for evaluating solution quality is a verifier that requires the answer, then all the gains from RLVR during training cannot be leveraged to improve selection among multiple sampled solutions at test time. Second, inference-time scaling efficiency: techniques like majority voting and weighted best-of-N selection can substantially improve accuracy, but they depend on having a signal that distinguishes good solutions from bad ones. Without such a signal, the model defaults to naive majority voting, which throws away information about solution quality. The paper cites Snell et al. (2024) as having shown that scaling test-time compute optimally can be more effective than scaling model parameters—but optimal scaling requires a verifier.

The gap, then, is not that verification doesn't work—it's that existing verification mechanisms don't travel from training to deployment. The verifier is either the ground-truth answer (unavailable at test time) or an external model that must be trained separately (costly, and introduces a second system to maintain). What's needed is a way for the policy model to carry its own verification capability with it—to be both generator and evaluator—so that the verification signal available during training persists into deployment.

The Two Existing Approaches, and Why They're Both Expensive

Prior work has attacked this problem from two directions, both of which the paper argues suffer from a shared flaw: inefficiency.

External verifiers (Section 2, "External Verifiers for LLM Reasoning"): The most well-studied approach is to train a separate model—a reward model—to score solutions. These come in two varieties. Scalar reward models include Outcome-supervised Reward Models (ORMs), which evaluate the final answer (Cobbe et al., 2021; Yang et al., 2024), and Process-supervised Reward Models (PRMs), which score individual reasoning steps (Lightman et al., 2023; Wang et al., 2024a; Yuan et al., 2024). PRMs offer finer-grained feedback but require step-level training data, which is expensive to produce—either through human annotation (Lightman et al., 2023) or Monte Carlo rollouts. Generative verifiers (Zhang et al., 2024; Gao et al., 2024; Yang et al., 2025b) instead train a separate LLM to produce natural-language critiques and then judge correctness, which has shown stronger performance than scalar approaches but involves running full generation passes through a second model.

The problem with external verifiers is twofold. First, they require training an entirely separate model, which doubles the training infrastructure and introduces a distribution-shift problem: the verifier is trained on the generator's outputs at training time, but as the generator continues to improve during RL, those outputs drift, and the verifier may become stale. Second, at inference time, using an external verifier means running two forward passes for every solution: one to generate the solution, another (potentially through a much larger model—the paper notes that high-quality verifiers are often 72B parameters or more) to score it. This substantially increases latency and compute cost.

Self-verification through separate generation (Section 2, "Self-Verification for LLM Reasoning"): A more recent line of work—which the paper cites as Sareen et al. (2025), Liu et al. (2025a), Zha et al. (2025), Jiang et al. (2025), and Lu et al. (2025)—attempts to unify generator and verifier in a single model by training the LLM to produce its own verification after each solution. During RLVR training, the model generates a solution under one prompt template, then generates a verification (e.g., "Is this solution correct? Yes/No" with a chain of thought) under a second prompt template. Both the reasoning reward and the verification reward are used for optimization. At test time, the same pipeline runs: generate a solution, then generate a self-verification, and use the verification result to rank or weight solutions.

The paper's critique of this approach is specific and concrete: it doubles the per-sample inference cost (Section 1):

"while joint optimization involves generating both solutions and self-verifications sequentially under two separate prompt templates, which doubles the per-sample inference cost and reduces generation efficiency."

This is not a small constant-factor overhead. In RLVR, generating solutions already dominates the training budget—each optimization step involves rolling out multiple solutions per prompt, computing token-level log-probabilities, and performing gradient updates. Adding a second full generation per solution effectively halves the number of problems the model can practice on per unit of compute during training. At inference time, the impact is even more severe for latency-sensitive applications: weighted best-of-N over 32 solutions requires 32 solution generations plus 32 verification generations, effectively serializing twice as many tokens.

Where the Two Approaches Converge—and Where They Both Hit a Wall

The paper's framing suggests that both external verifiers and separate-generation self-verification are two different ways of paying the same cost: running additional inference to produce a verification signal. External verifiers run that inference through a separate model (which must be trained), while self-verification runs it through the same model under a different prompt (which must be trained on that task). Both approaches accept as given that verification requires generation—that to assess a solution, the model must produce some output (a score, a critique, a yes/no judgment) that explicitly represents that assessment.

The paper's core motivation is to challenge exactly this assumption. If verification can be extracted from the model's internal state—specifically, from the probability distribution it assigns to a special token at the end of its generation—then the verification signal is essentially free. There is no separate model to train, no second prompt template, no additional generation pass. The model's own uncertainty about a pre-specified token becomes a proxy for its confidence in the solution it just produced.

This connects to a broader theoretical motivation that the paper develops in Section 3. The standard RL objective (Equation 1) has a closed-form solution relating the optimal policy to the reward function via the KL-constrained distribution (Equation 5). This relationship—that the implicit reward βlog⁡πθ(y∣x)/πref(y∣x)\beta \log \pi_\theta(y|x)/\pi_{ref}(y|x) equals the true reward up to a partition function—has been recognized since Rafailov et al. (2023) and Peters & Schaal (2007). The problem, as the paper notes, is that this implicit reward is length-biased (Appendix A, Figure 4). The sum of token-level log-ratios grows with response length, and since incorrect reasoning solutions tend to be longer than correct ones (Hassid et al., 2025), the implicit reward becomes anti-correlated with actual correctness. This makes it useless as a verification signal in its naive form.

The paper's key theoretical move is to reframe the problem from solution-level verification to token-level verification. Instead of asking "What is the implicit reward of the entire solution yy?" it asks "What is the implicit reward of a single verification token zcz_c given the solution yy as context?" This collapses the length bias problem into a single-token decision, where the partition function Z(x,y)Z(x, y) can be shown to be approximately 1—and therefore ignorable—because the reference model assigns near-zero probability to any single pre-specified token (Figure 5 shows the reference model's log-probability for special tokens is around −23-23, meaning πref(zc∣x,y)≈10−10\pi_{ref}(z_c|x,y) \approx 10^{-10}). This is the crucial observation that makes the approach computationally viable.

How the Paper Positions Itself

The paper explicitly draws a contrast with prior self-verification work that requires separate generation passes. In Section 3.4:

"Compared with previous methods... that requires the policy model to perform separate generations for solutions and verifications, our method directly derives the self-rewarding result from the next-token log-probability of the final solution token."

This positions LaSeR not as an incremental improvement over prior self-verification but as a fundamentally different mechanism: prior work extracts verification from the model's outputs, while LaSeR extracts it from the model's predicted probability distribution over a single unused token. The computational difference is stark:

  • Prior self-verification: Generate solution (T tokens) → Generate verification (T' tokens). Total: T+T′T + T' tokens of inference per solution.
  • External verifier: Generate solution (T tokens) → Run verifier on solution (T'' tokens). Total: T+T′′T + T'' tokens of inference per solution, plus the cost of training and maintaining a second model.
  • LaSeR (training): Generate solution (T tokens) → Immediately compute log⁡πθ(zc∣x,y)\log \pi_\theta(z_c|x,y) from the already-computed next-token distribution at the final token position. The paper notes that "the computation of token log-probabilities is typically carried out after all the generations are completed" in RL frameworks like verl (Sheng et al., 2024), so this requires zero additional forward passes—just a different indexing into already-computed tensors.
  • LaSeR (inference): Generate solution (T tokens) → Run one additional token inference to get the next-token distribution at position T+1T+1, then extract log⁡πθ(zc∣x,y)\log \pi_\theta(z_c|x,y). Total: T+1T + 1 tokens.

The paper frames this as "nearly zero additional cost" (Section 1) and "lightweight" (Section 6). The theoretical justification—that the RL objective of verification reduces to a single MSE loss between the last-token self-rewarding score and the verifier reward, with no need to explicitly perform a separate RLVR procedure for verification—is the paper's novel contribution. It ties together the implicit reward literature (Rafailov et al., 2023), the RLVR paradigm (Guo et al., 2025), and the self-verification goal into a single, simple loss term (Equation 13):

L=Ex∼D,y∼πg(⋅∣x)[(βvlog⁡[πθ(zc∣x,y)/πref(zc∣x,y)]−rv(x,y))2]\mathcal{L} = \mathbb{E}_{x \sim \mathcal{D}, y \sim \pi_g(\cdot|x)} \left[ \left( \beta_v \log[\pi_\theta(z_c|x, y)/\pi_{ref}(z_c|x, y)] - r_v(x, y) \right)^2 \right]

The message is: you don't need to generate a verification—you just need to read a probability.

Significance of the Problem Beyond Efficiency

While efficiency is the paper's headline motivation, there is a deeper significance that emerges from the experimental results. When the model learns to encode solution quality in the probability distribution of a single token, it is effectively learning a compressed, latent representation of its own uncertainty. This is a form of confidence calibration that emerges not from explicit prompting or separate training, but as a side effect of jointly optimizing reasoning and self-rewarding. The paper shows (Table 1) that this calibration is remarkably accurate—self-verification F1 scores around 80% across multiple benchmarks and base model families—and that it transfers across difficulty levels and problem types.

This has implications beyond the immediate efficiency argument. If a model can reliably assess its own outputs without generating explicit verification, it opens the door to more sophisticated test-time strategies that are currently too expensive to run: iteratively refining uncertain solutions, allocating more compute to low-confidence problems, or using the self-rewarding score as an intrinsic exploration bonus during RL itself (which the paper explores through the advantage integration technique in Equation 16). The efficiency gain is not just about doing the same thing cheaper—it enables new capabilities that were previously computationally prohibitive.

3. Technical Approach

3.1 Reader Orientation (Approachable Technical Breakdown)

The paper develops a lightweight training algorithm that teaches an LLM to score its own reasoning solutions without generating a separate verification text, by extracting a confidence signal from the probability the model assigns to a single unused token at the very end of its solution. The problem it solves is that standard reinforcement learning for reasoning (RLVR) uses ground-truth answer checkers during training, but those checkers are unavailable at test time—and prior solutions that add self-verification require the model to generate a second full response, doubling inference cost; LaSeR instead reads the verification signal directly from the model's next-token probability distribution at the final position, incurring at most one additional token of inference.

3.2 Big-Picture Architecture (Diagram in Words)

The LaSeR system has four major components:

  1. A base policy model $\pi_\theta$ (e.g., Qwen2.5-7B-Base) that generates reasoning solutions and is optimized by a standard RLVR algorithm (GRPO). This model is also the target of self-rewarding training—it will learn to encode solution quality in its output distribution.

  2. A deterministic verifier that provides ground-truth binary rewards $r_v(x, y) \in \{0, 1\}$ by checking whether the final answer extracted from solution $y$ matches the known correct answer $a^*$. This verifier is only available during training and provides the supervision signal for both reasoning and self-rewarding.

  3. A pre-specified special token $z_c$ —an unused vocabulary token like <|vision_start|> that the model never naturally generates—which serves as a "probe" for the model's internal assessment. The model's predicted log-probability of this token at the position immediately after its solution is the raw material from which the self-rewarding score is computed.

  4. Two additive loss terms combined into a single optimization objective: the standard RLVR loss (which improves reasoning) and a Mean Squared Error (MSE) loss that forces the model's last-token self-rewarding score to match the verifier's binary reward (which teaches self-assessment). The self-rewarding score, once learned, can be used both during training (as an auxiliary advantage signal) and at test time (for weighted majority voting).

Information flows as follows: a math problem enters → the policy model generates $K$ candidate solutions → the deterministic verifier scores each solution as correct (1) or incorrect (0) by comparing extracted answers to the ground truth → the standard RLVR loss (GRPO) is computed from these binary rewards → simultaneously, for each solution, the model's next-token log-probability for the special token $z_c$ at the final response position is extracted → this log-probability is transformed into a self-rewarding score $r_s = \beta_v \log \pi_\theta(z_c|x, y) - \beta_v c_{ref}$ → an MSE loss is computed between $r_s$ and $r_v$ → both losses are summed and used to update $\pi_\theta$. At test time, the same self-rewarding score is computed (requiring one additional forward pass for the next-token distribution) and used to weight solutions in best-of-N selection.

3.3 Roadmap for the Deep Dive

  • First, the standard RLVR objective and the implicit reward relationship (Equations 1, 5), because these are the mathematical foundation from which the paper derives its self-rewarding formulation—understanding why naive implicit reward fails (length bias) motivates the need for a different approach.
  • Second, the RL objective of verification (Equation 6) and the critical derivation that the partition function $Z(x, y)$ is approximately 1 (Equation 10), which is the key theoretical insight enabling the self-rewarding score to be a direct estimate of the true reward without any additive constant.
  • Third, the self-rewarding MSE loss itself (Equation 13) and how it integrates into the full LaSeR objective (Equation 14), along with the practical simplifications that make it computationally cheap: approximating the reference model's log-probability as a constant and using class-level re-weighting.
  • Fourth, the integration of self-rewarding scores back into the RL advantage estimation (Equation 16), which allows the learned self-assessment to improve training itself, not just test-time selection.
  • Fifth, the warm-up schedule and the full algorithm (Algorithm 1), which orchestrate the staged introduction of self-rewarding training to avoid interference with early reasoning skill acquisition.
  • Sixth, the inference-time usage, where the self-rewarding score is computed with one extra token and used for weighted majority voting, and the comparison to prior approaches that require full separate generation passes.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a theoretically-motivated algorithmic design paper whose core idea is that the RL objective for verification has a closed-form solution equating the true reward to a scaled log-probability ratio at a single token, and that this relationship can be exploited to train self-verification via a simple MSE loss rather than a full RL procedure.

The Standard RLVR Objective and Why Implicit Reward Fails for Verification

The paper begins from the standard Reinforcement Learning with Verifiable Rewards (RLVR) framework, which has become the dominant paradigm for reasoning training since DeepSeek-R1 (Guo et al., 2025) and OpenAI o1 (Jaech et al., 2024). The objective is:

Oπθ=max⁡πθEx∼D,y∼πθ(⋅∣x)[r(x,y)−βDKL(πθ∣∣πref)]\mathcal{O}_{\pi_\theta} = \max_{\pi_\theta} \mathbb{E}_{x \sim \mathcal{D}, y \sim \pi_\theta(\cdot|x)} \left[ r(x, y) - \beta D_{KL}(\pi_\theta || \pi_{ref}) \right]

where $\pi_\theta$ is the policy model being optimized, $\pi_{ref}$ is the frozen reference model (typically the model $\pi_\theta$ was initialized from), $\mathcal{D}$ is the set of training prompts, $y$ is a generated response to prompt $x$, $r(x, y)$ is a reward function scoring the response, $\beta$ is the KL penalty coefficient controlling how far $\pi_\theta$ can drift from $\pi_{ref}$, and $D_{KL}(\pi_\theta || \pi_{ref})$ is the Kullback-Leibler divergence between the policy and reference distributions.

What it computes: the expected reward of generated responses minus a penalty for deviating from the reference model's behavior. Maximizing this objective pushes the policy toward high-reward outputs while preventing it from collapsing to degenerate strategies that exploit the reward function. In practice, $\beta$ is often set to 0.0 (as in Table 4 of this paper), meaning the KL penalty is disabled and the objective reduces to pure reward maximization—a choice that works when the reward function is a deterministic verifier with binary output, since there is no continuous reward signal to hack.

Why this form: the KL-constrained RL objective has a well-known closed-form solution (Rafailov et al., 2023; Peters & Schaal, 2007) relating the optimal policy to the reward function:

rv(x,y)=βlog⁡πθ(y∣x)πref(y∣x)+βlog⁡Z(x)r_v(x, y) = \beta \log \frac{\pi_\theta(y|x)}{\pi_{ref}(y|x)} + \beta \log Z(x)

where $Z(x) = \sum_y \pi_{ref}(y|x) \exp(\frac{1}{\beta} r_v(x, y))$ is a partition function that normalizes the distribution over all possible responses $y$ for prompt $x$.

What it computes: this equation says that if the policy $\pi_\theta$ is exactly optimal under the KL-constrained objective, then the true reward $r_v(x, y)$ can be recovered (up to an additive constant $\beta \log Z(x)$) from the log-ratio of the policy's probability to the reference model's probability for response $y$. The term $\beta \log \frac{\pi_\theta(y|x)}{\pi_{ref}(y|x)}$ is called the implicit reward—it is the reward that would make $\pi_\theta$ the optimal policy, inferred from the model's own probabilities.

Why this form matters, and why it fails for verification: the implicit reward seems like a natural candidate for self-verification: it is a scalar score derived purely from the model's internal probabilities, requiring no external verifier. However, the paper identifies a fatal flaw: length bias. The ratio expands as a sum over all tokens in the response:

βlog⁡πθ(y∣x)πref(y∣x)=β∑i=1Tlog⁡πθ(yi∣x,y<i)πref(yi∣x,y<i)\beta \log \frac{\pi_\theta(y|x)}{\pi_{ref}(y|x)} = \beta \sum_{i=1}^{T} \log \frac{\pi_\theta(y_i | x, y_{<i})}{\pi_{ref}(y_i | x, y_{<i})}

where $T$ is the number of tokens in $y$. Because this sum grows with $T$, longer responses systematically receive higher absolute implicit reward values, regardless of their actual correctness. The paper shows this empirically in Appendix A (Figure 4): across 32 reasoning trajectories sampled from Open-Reasoner-Zero-7B on an AIME2024 problem, all curves show positive correlation between cumulative implicit reward and token count. Since "incorrect solutions are usually longer than the correct solutions in reasoning tasks" (Hassid et al., 2025), the implicit reward is "not a reliable indicator of the relative quality of reasoning paths at test time" (Appendix A). Furthermore, even if one tried to directly train the implicit reward to match the true reward, a fundamental mismatch exists: the implicit reward omits $\beta \log Z(x)$, and "directly aligning the implicit reward with the true reasoning reward during training degrades the policy model's generation ability" (Section 3.2.1, citing Cui et al., 2025).

This failure motivates the paper's key reframing: instead of looking at the implicit reward of the entire solution, look at the implicit reward of a single verification token. This collapses the length bias into a single decision point, where the partition function can actually be eliminated.

The Verification RL Objective and the Critical Partition Function Elimination

The paper now formulates an RL objective specifically for verification, separate from the reasoning objective. Given a problem $x$ and a candidate solution $y$ (which may be generated by any model; the paper denotes the generator as $\pi_g$, which in the self-verification setting equals the policy model $\pi_\theta$ itself), the model is required to produce a verification $z$ that judges the correctness of $y$. The objective is:

Vπθ=max⁡πθEx∼D,y∼πg(⋅∣x),z∼πθ(⋅∣x,y)[r^(x,y,z)−βvDKL(πθ∣∣πref)]\mathcal{V}_{\pi_\theta} = \max_{\pi_\theta} \mathbb{E}_{x \sim \mathcal{D}, y \sim \pi_g(\cdot|x), z \sim \pi_\theta(\cdot|x, y)} \left[ \hat{r}(x, y, z) - \beta_v D_{KL}(\pi_\theta || \pi_{ref}) \right]

where $\beta_v$ is a separate KL coefficient for the verification objective (distinct from the $\beta$ in Equation 1, which controls the reasoning KL penalty), and $\hat{r}(x, y, z)$ is a verification reward measuring whether the verification $z$ correctly identifies the true correctness of $y$.

What it computes: like the standard RL objective, this maximizes the expected verification reward minus a KL divergence penalty. The key difference is the conditioning: the policy generates $z$ given $(x, y)$—the prompt and a candidate solution—rather than generating $y$ given only $x$. This is a second-stage decision problem where the input includes the thing being evaluated.

The verification reward $\hat{r}$ and its simplification: in general, $z$ could be a full reasoning chain ending in a judgment. The paper makes a crucial simplification: "we focus on the... setting and simplify the ground-truth label space to two single tokens $z_c$ (e.g., 'Yes') and $z_i$ (e.g., 'No')" (Section 3.2.1). That is, the verification output is just a single token indicating correctness. The verification reward becomes:

r^(x,y,z)={1if (z=zc and rv(x,y)=1) or (z=zi and rv(x,y)=0)0otherwise\hat{r}(x, y, z) = \begin{cases} 1 & \text{if } (z = z_c \text{ and } r_v(x, y) = 1) \text{ or } (z = z_i \text{ and } r_v(x, y) = 0) \\ 0 & \text{otherwise} \end{cases}

where $r_v(x, y)$ is the same deterministic verifier from the reasoning objective (Equation 2)—it returns 1 if the extracted answer matches the ground truth, 0 otherwise.

What this means operationally: the verification reward is 1 if the verification token correctly identifies the solution's true correctness, and 0 otherwise. If the solution is actually correct ($r_v = 1$) and the model outputs $z_c$ ("Yes, this is correct"), the verification reward is 1. If the solution is actually incorrect ($r_v = 0$) and the model outputs $z_i$ ("No, this is incorrect"), the verification reward is also 1. Any mismatch between the verification token and the true correctness yields reward 0.

Now, applying the same closed-form solution logic as before (Equation 5), the optimal solution to the verification objective satisfies:

r^(x,y,z)=βvlog⁡πθ(z∣x,y)πref(z∣x,y)+βvlog⁡Z(x,y)\hat{r}(x, y, z) = \beta_v \log \frac{\pi_\theta(z|x, y)}{\pi_{ref}(z|x, y)} + \beta_v \log Z(x, y)

Z(x,y)=∑zπref(z∣x,y)exp⁡(1βvr^(x,y,z))Z(x, y) = \sum_{z} \pi_{ref}(z|x, y) \exp\left(\frac{1}{\beta_v} \hat{r}(x, y, z)\right)

where $Z(x, y)$ is the partition function over all possible verification tokens $z$ given the prompt-solution pair $(x, y)$.

The key theoretical insight—why $Z(x, y) \approx 1$: the paper argues that for the reference model (the initial, un-fine-tuned base model), the probability assigned to any single pre-specified token $z_c$ given any solution $y$ is extremely small and approximately constant across all problems. The reason is straightforward: "the model is not specifically optimized for predicting the next token once it completes the generation and produces the final token (typically the <EOS> token)" (Section 3.2.1). The reference model's probability mass is spread across the entire vocabulary; any individual token—especially an unused special token—gets a tiny slice.

The paper validates this empirically in Figure 5 (Appendix B). Over 300 input-output pairs, for the combination of Qwen2.5-7B-Base as reference model and <|vision_start|> as the special token, the mean of $-\log \pi_{ref}(z_c|x, y)$ is 23.11 with a standard deviation of only 0.04—meaning $\pi_{ref}(z_c|x, y) \approx e^{-23} \approx 10^{-10}$ with negligible variation. For OctoThinker-3B-Short-Base and a reserved special token, the value is even smaller (mean $-\log \pi_{ref} \approx 24.87$, standard deviation 1.18). Even for a common token like "Yes," the values are small ($e^{-9.13}$ for Qwen) but with higher variance (standard deviation 0.13 on the log scale).

Now compute $Z(x, y)$ explicitly, splitting the sum over verification tokens into the two special tokens ($z_c$ and $z_i$) and all other tokens:

Z(x,y)=∑z∉{zc,zi}πref(z∣x,y)exp⁡(1βvr^(x,y,z))+πref(zc∣x,y)exp⁡(1βvr^(x,y,zc))+πref(zi∣x,y)exp⁡(1βvr^(x,y,zi))Z(x, y) = \sum_{z \notin \{z_c, z_i\}} \pi_{ref}(z|x, y) \exp\left(\frac{1}{\beta_v} \hat{r}(x, y, z)\right) + \pi_{ref}(z_c|x, y) \exp\left(\frac{1}{\beta_v} \hat{r}(x, y, z_c)\right) + \pi_{ref}(z_i|x, y) \exp\left(\frac{1}{\beta_v} \hat{r}(x, y, z_i)\right)

For any token $z \notin \{z_c, z_i\}$, it is not a valid verification token, so $\hat{r}(x, y, z) = 0$ by definition (the verification reward is only 1 when the output matches the true correctness and is one of the two designated tokens). Thus $\exp(0) = 1$. The first sum becomes:

∑z∉{zc,zi}πref(z∣x,y)=1−πref(zc∣x,y)−πref(zi∣x,y)\sum_{z \notin \{z_c, z_i\}} \pi_{ref}(z|x, y) = 1 - \pi_{ref}(z_c|x, y) - \pi_{ref}(z_i|x, y)

For the two special tokens, $\hat{r}$ is either 0 or $1/\beta_v$ depending on whether the token matches the true correctness, so $\exp(\hat{r}/\beta_v)$ is either $\exp(0) = 1$ or $\exp(1/\beta_v)$. Substituting:

Z(x,y)=(1−πref(zc∣x,y)−πref(zi∣x,y))⋅1+πref(zc∣x,y)⋅exp⁡(1/βv)+πref(zi∣x,y)⋅exp⁡(1/βv)Z(x, y) = (1 - \pi_{ref}(z_c|x, y) - \pi_{ref}(z_i|x, y)) \cdot 1 + \pi_{ref}(z_c|x, y) \cdot \exp(1/\beta_v) + \pi_{ref}(z_i|x, y) \cdot \exp(1/\beta_v)

=1−(πref(zc∣x,y)+πref(zi∣x,y))+(πref(zc∣x,y)+πref(zi∣x,y))⋅exp⁡(1/βv)= 1 - (\pi_{ref}(z_c|x, y) + \pi_{ref}(z_i|x, y)) + (\pi_{ref}(z_c|x, y) + \pi_{ref}(z_i|x, y)) \cdot \exp(1/\beta_v)

Since $\pi_{ref}(z_c|x, y)$ and $\pi_{ref}(z_i|x, y)$ are each approximately $10^{-10}$, their sum is roughly $2 \times 10^{-10}$. With $\beta_v = 0.1$ (the paper's chosen value), $\exp(1/0.1) = \exp(10) \approx 22026$. The second term becomes $2 \times 10^{-10} \times 22026 \approx 4.4 \times 10^{-6}$. Therefore:

Z(x,y)≈1−2×10−10+4.4×10−6≈1.000004Z(x, y) \approx 1 - 2 \times 10^{-10} + 4.4 \times 10^{-6} \approx 1.000004

and consequently $\log Z(x, y) \approx 0$. The partition function, which was the problematic unknown constant in the full implicit reward, vanishes to zero in this single-token verification setting. This is the paper's central theoretical contribution: by restricting the verification to a single pre-specified token with near-zero reference probability, the closed-form solution simplifies to:

r^(x,y,z)≈βvlog⁡πθ(z∣x,y)πref(z∣x,y)\hat{r}(x, y, z) \approx \beta_v \log \frac{\pi_\theta(z|x, y)}{\pi_{ref}(z|x, y)}

And specifically, when the verification is correct (the model outputs $z_c$ and the solution is indeed correct):

rv(x,y)=r^(x,y,zc)=βvlog⁡πθ(zc∣x,y)πref(zc∣x,y)r_v(x, y) = \hat{r}(x, y, z_c) = \beta_v \log \frac{\pi_\theta(z_c|x, y)}{\pi_{ref}(z_c|x, y)}

What this means operationally: the true reasoning reward $r_v(x, y)$—the binary 0/1 signal from the deterministic answer checker—is equal (asymptotically, at the optimal policy) to the scaled log-probability ratio of a single special token $z_c$ at the end of the solution. The model's internal uncertainty about this unused token is the verification signal. No separate generation, no chain-of-thought verification, no external reward model—just the next-token probability distribution at the final position.

Why this form is critically different from the standard implicit reward: the standard implicit reward sums over all tokens in the solution, producing a length-biased signal with an unknown partition function. This single-token version has no length bias (it is one token, always), and the partition function is provably negligible. The verification signal is clean, theoretically justified, and—crucially—computable from a single forward pass at the last position.

From Theory to Training: The Self-Rewarding MSE Loss

The theoretical result suggests that if the model is optimal, its self-rewarding score $r_s = \beta_v \log \frac{\pi_\theta(z_c|x, y)}{\pi_{ref}(z_c|x, y)}$ will exactly equal the verifier's binary reward $r_v(x, y)$. But during training, the model is not optimal—so the paper proposes to explicitly train the model to make these two quantities equal via a Mean Squared Error loss:

Lsr=Ex∼D,y∼πg(⋅∣x)[(βvlog⁡πθ(zc∣x,y)πref(zc∣x,y)−rv(x,y))2]\mathcal{L}_{sr} = \mathbb{E}_{x \sim \mathcal{D}, y \sim \pi_g(\cdot|x)} \left[ \left( \beta_v \log \frac{\pi_\theta(z_c|x, y)}{\pi_{ref}(z_c|x, y)} - r_v(x, y) \right)^2 \right]

where $\beta_v$ is the KL coefficient for the verification objective (set to 0.1 in all experiments), $z_c$ is the pre-specified special token, $\pi_g$ is the generator (which equals $\pi_\theta$ in the self-verification setting), and $r_v(x, y)$ is the binary verifier reward.

What it computes: for each generated solution $y$, the model extracts $\log \pi_\theta(z_c|x, y)$—the log-probability it assigns to generating token $z_c$ immediately after $y$—and $\log \pi_{ref}(z_c|x, y)$—the same quantity under the frozen reference model. The difference, scaled by $\beta_v$, is the self-rewarding score $r_s$. The MSE loss penalizes the squared difference between $r_s$ and the ground-truth binary reward $r_v$. If the solution is correct ($r_v = 1$), the model is pushed to assign higher probability to $z_c$; if incorrect ($r_v = 0$), it is pushed to assign lower probability.

Why this form rather than RL: the paper explicitly argues that there is no need to run a full RLVR procedure for verification—the closed-form solution eliminates the need for advantage estimation, policy gradients, and the associated variance. A simple regression loss suffices because the target is directly the true reward, not a learned value function. This is analogous to how Direct Preference Optimization (DPO) (Rafailov et al., 2023) replaces RLHF's reward modeling + PPO pipeline with a single classification loss—here, the MSE loss replaces what would otherwise be a second GRPO training loop for verification.

Why this form rather than SFT on $z_c$: the paper considers an alternative in Section 3.4: supervised fine-tuning that maximizes $\log \pi_\theta(z_c|x, y)$ for correct solutions and $\log \pi_\theta(z_i|x, y)$ for incorrect ones. The problem is that SFT would drive $\pi_\theta(z_c|x, y)$ toward 1.0 for correct solutions, which "may lead to strong interference with the optimization of reasoning capability" (Section 3.4). In contrast, the LaSeR loss drives $\pi_\theta(z_c|x, y)$ toward $\exp(1/\beta_v) \cdot \pi_{ref}(z_c|x, y)$ for correct solutions. With $\beta_v = 0.1$ and $\pi_{ref}(z_c|x, y) \approx e^{-23}$, the target is $\exp(10) \cdot e^{-23} = e^{-13} \approx 2.3 \times 10^{-6}$—still an extremely small probability. The model never needs to make $z_c$ likely enough to actually sample it; it only needs to modulate the tiny probability in a way that correlates with solution correctness. This means the self-rewarding training exerts "only a negligible influence on the original RLVR optimization" (Section 3.4), since the probability mass for $z_c$ remains vanishingly small and the dominant token at the final position is still the natural end-of-sequence token.

Full LaSeR Objective: Joint Optimization of Reasoning and Self-Rewarding

The self-rewarding MSE loss is combined with the standard RLVR objective to form the complete LaSeR objective:

Sπθ=max⁡πθEx∼D,y∼πθ(⋅∣x){rv(x,y)−βDKL(πθ(y∣x)∣∣πref(y∣x))−α[βvlog⁡πθ(zc∣x,y)πref(zc∣x,y)−rv(x,y)]2}\mathcal{S}_{\pi_\theta} = \max_{\pi_\theta} \mathbb{E}_{x \sim \mathcal{D}, y \sim \pi_\theta(\cdot|x)} \left\{ r_v(x, y) - \beta D_{KL}(\pi_\theta(y|x) || \pi_{ref}(y|x)) - \alpha \left[ \beta_v \log \frac{\pi_\theta(z_c|x, y)}{\pi_{ref}(z_c|x, y)} - r_v(x, y) \right]^2 \right\}

where $\alpha$ is a loss balancing coefficient (set to 0.1 in all experiments). The first two terms ($r_v - \beta D_{KL}$) are the standard RLVR objective from Equation 1; the third term is the self-rewarding MSE loss from Equation 13 multiplied by $-\alpha$ (since the objective is maximized, minimizing MSE becomes maximizing its negative).

What it computes: this is a single optimization problem that simultaneously pushes the model to (a) generate correct reasoning solutions (through the RL term), (b) stay close to the reference model distribution (through the KL penalty, though in practice $\beta$ is set to 0.0), and (c) encode solution quality in the probability of $z_c$ (through the self-rewarding MSE term). The three forces are balanced by $\alpha$, which the paper tunes empirically (Appendix D, Figure 6). Setting $\alpha$ too high (e.g., 0.5) "has a more detrimental impact on the model's reasoning capabilities" because the MSE loss starts to dominate gradient updates. Setting $\alpha = 0.1$ provides a sweet spot where self-rewarding capability improves without degrading reasoning.

The practical implementation via GRPO: the paper implements this objective using Group Relative Policy Optimization (GRPO) (Shao et al., 2024), a widely-used policy gradient algorithm for RLVR. In GRPO, for each prompt $x$, the model samples $K = 8$ solutions $\{y^1, \ldots, y^K\}$. The verifier reward $r_v^i = r_v(x, y^i)$ is computed for each solution. The relative advantage for each token $y^i_t$ in solution $i$ is:

Ati=rvi−mean(rv1,…,rvK)std(rv1,…,rvK)A^i_t = \frac{r_v^i - \text{mean}(r_v^1, \ldots, r_v^K)}{\text{std}(r_v^1, \ldots, r_v^K)}

What this computes: the advantage is the number of standard deviations each solution's reward is above or below the group mean. This normalization eliminates the need for a learned value function (as in PPO) and handles the fact that different problems have different baseline difficulties—the group mean implicitly estimates the expected reward for that specific problem.

The GRPO policy gradient is then $\nabla_\theta \mathcal{O}_{\pi_\theta} = \mathbb{E} \left[ \sum_t A^i_t \nabla_\theta \log \pi_\theta(y^i_t | x, y^i_{<t}) \right]$, and the total gradient is the sum of this RL gradient and the gradient of the self-rewarding MSE loss (scaled by $\alpha$).

Practical Simplification 1: Reference Log-Probability as a Constant

Computing $\log \pi_{ref}(z_c|x, y)$ requires a forward pass through the reference model for every generated solution, which doubles the computational cost. The paper observes that this value is "almost constant, exhibiting only a negligible standard deviation across all $x$ and $y$" (Section 3.3), as shown in Figure 5. Therefore, it can be replaced by a pre-calculated constant $c_{ref}$:

cref=E(x,y)[log⁡πref(zc∣x,y)]c_{ref} = \mathbb{E}_{(x, y)}[\log \pi_{ref}(z_c|x, y)]

computed as the mean over a small set of pre-generated solution pairs. For Qwen2.5-7B-Base with <|vision_start|>, $c_{ref} = -23.0$; for OctoThinker-3B-Short-Base with <|reserved_special_token_0|>, $c_{ref} = -25.0$.

The self-rewarding score simplifies to:

rs=βvlog⁡πθ(zc∣x,y)−βvcrefr_s = \beta_v \log \pi_\theta(z_c|x, y) - \beta_v c_{ref}

What this changes operationally: in both training and inference, computing $r_s$ now requires only the policy model's forward pass—no reference model is needed. During training, the paper notes that "the computation of token log-probabilities is typically carried out after all the generations are completed" in frameworks like verl (Sheng et al., 2024), so the log-probability at the final position is already available from the same forward pass used to compute the RL loss. The self-rewarding score is therefore zero additional forward passes during training.

The paper validates this simplification empirically in Section 5.1 (Table 3): comparing LaSeR with and without the reference model simplification on Open-Reasoner-Zero-7B, the reasoning and self-verification performance is nearly identical (average reasoning accuracy 45.1% with simplification vs. 45.0% without; self-verification F1 79.4% vs. 79.3%). The simplification "effectively reduces the computational cost of calculating the last-token self-rewarding value by half" without affecting quality.

Why choose an unused special token rather than a common token: Figure 5 shows that for common tokens like "Yes," $-\log \pi_{ref}$ is smaller (∼9.13 for Qwen) but has higher variance (standard deviation 0.13). For unused special tokens like <|vision_start|>, the value is much smaller (∼23.11) with extremely low variance (standard deviation 0.04). The paper selects unused special tokens for two reasons: (1) the lower variance makes the constant approximation more accurate, and (2) the extremely small reference probability makes the partition function approximation $Z(x, y) \approx 1$ even tighter, since the $\pi_{ref}(z_c|x, y) \cdot \exp(1/\beta_v)$ term becomes even smaller. The specific tokens chosen are <|vision_start|> for Qwen2.5-7B-Base and Open-Reasoner-Zero-7B, and <|reserved_special_token_0|> for OctoThinker-3B-Short-Base—tokens that are in the vocabulary but never used in normal text generation.

Practical Simplification 2: Class-Level Loss Re-Weighting

During RLVR training, the number of correct and incorrect solutions in a batch is imbalanced and changes dynamically as the model improves. If the model generates 90% incorrect solutions, the self-rewarding MSE loss would be dominated by the "predict 0" signal, potentially biasing the model toward always predicting low self-rewarding scores. To address this, the paper applies class-level loss re-weighting within each optimization step:

l=1Nc+Ni∑x∑y[wc1{rv(x,y)=1}+wi1{rv(x,y)=0}][βvlog⁡πθ(zc∣x,y)−βvcref−rv(x,y)]2l = \frac{1}{N_c + N_i} \sum_x \sum_y \left[ w_c \mathbf{1}_{\{r_v(x,y)=1\}} + w_i \mathbf{1}_{\{r_v(x,y)=0\}} \right] \left[ \beta_v \log \pi_\theta(z_c|x, y) - \beta_v c_{ref} - r_v(x, y) \right]^2

where $N_c$ and $N_i$ are the total numbers of correct and incorrect solutions in the current batch, $w_c = \frac{N_c + N_i}{2 \times N_c}$ is the weight for correct solutions, and $w_i = \frac{N_c + N_i}{2 \times N_i}$ is the weight for incorrect solutions.

What this computes: the re-weighting factors give equal total weight to the correct and incorrect classes regardless of their frequencies. If there are twice as many incorrect solutions as correct ones ($N_i = 2N_c$), then $w_c = (3N_c)/(2N_c) = 1.5$ and $w_i = (3N_c)/(4N_c) = 0.75$, so each incorrect solution contributes half the weight of each correct solution. The total contribution from each class is equal: $N_c \times 1.5 = N_i \times 0.75 = 1.5 N_c$.

Why this matters: without re-weighting, the self-rewarding score would be biased toward the majority class. The paper shows this empirically in Appendix E (Figure 7): without re-weighting, training self-verification F1 on Open-Reasoner-Zero-7B is lower and more imbalanced (biased toward the majority incorrect class), while with re-weighting, "a more balanced self-verification performance" is achieved "while still maintaining high reasoning accuracy." The authors note that "future work can explore more effective ways to address the issue of imbalanced distribution of solutions."

Integration of Self-Rewarding Scores into RL Advantage Estimation

The self-rewarding score $r_s$ is not only useful at test time—it can be fed back into the training process to improve the advantage estimates used in the policy gradient. The paper argues that this can "help mitigate the issue of misjudgments by rule-based verifiers, which often occur when the format of ground-truth answer is overly complex, and produce more fine-grained rewards" (Section 3.3).

The standard GRPO advantage (Equation 4) is computed from the verifier's binary rewards. The paper proposes a convex combination of verifier-based and self-rewarding-based advantages:

A^ti=(1−τ)rvi−mean(rv1,…,rvK)std(rv1,…,rvK)+τrsi−mean(rs1,…,rsK)std(rs1,…,rsK)\hat{A}^i_t = (1 - \tau) \frac{r_v^i - \text{mean}(r_v^1, \ldots, r_v^K)}{\text{std}(r_v^1, \ldots, r_v^K)} + \tau \frac{r_s^i - \text{mean}(r_s^1, \ldots, r_s^K)}{\text{std}(r_s^1, \ldots, r_s^K)}

where $\tau$ is a mixing weight (set to 0.1), $r_v^i$ is the verifier-based reward, and $r_s^i = \beta_v \log \pi_\theta(z_c|x, y^i) - \beta_v c_{ref}$ is the self-rewarding score.

What this computes: the final advantage for each token is a weighted average of two normalized advantage estimates: one from the deterministic verifier (binary, potentially noisy due to answer extraction failures) and one from the learned self-rewarding score (continuous, potentially capturing solution quality more finely). The $(1 - \tau)$ and $\tau$ weights control how much each source contributes. With $\tau = 0.1$, the self-rewarding score provides 10% of the advantage signal.

Filtering for stability: the paper notes that when the self-rewarding scores within a group have very low variance (i.e., the model assigns similar scores to all solutions), the normalized self-rewarding advantage becomes noisy or undefined. To handle this, a filtering strategy is applied: " $\tau = 0$ for any group whenever the standard deviation $\text{std}(r_s^1, \ldots, r_s^K)$ within this group falls below a threshold $T$, which is set to 0.1." In plain language: if the self-rewarding scores for all 8 solutions in a group are nearly identical (standard deviation < 0.1), the self-rewarding advantage is deemed unreliable and the system falls back to pure verifier-based advantages ($\tau = 0$).

Why add self-rewarding to the advantage: the deterministic verifier is binary and sometimes misclassifies solutions due to answer extraction errors—e.g., a solution with correct reasoning but a malformatted final answer gets reward 0. The self-rewarding score, being a continuous value derived from the model's internal assessment, can provide a more nuanced signal: a solution that is mostly correct but has a minor formatting issue might receive a self-rewarding score of 0.7 rather than a binary 0. Blending this into the advantage provides a smoother optimization landscape. The paper reports ablation results in Table 1 ("- SWA" = without Self-rewarding Weighted Advantage), showing that removing advantage integration slightly reduces reasoning performance (e.g., Qwen2.5-7B-LaSeR drops from 42.7% to 41.1% average accuracy).

Warm-Up Schedule: Staged Introduction of Self-Rewarding

The paper observes that introducing the self-rewarding MSE loss too early in training can interfere with the acquisition of basic reasoning skills. The solution is a two-phase warm-up:

  1. Reasoning warm-up ($w_r$ steps, set to 200): During the first 200 optimization steps (for base models OctoThinker-3B-Short-Base and Qwen2.5-7B-Base), the model is trained with standard RLVR without the self-rewarding MSE loss. This establishes a baseline reasoning capability before the model is asked to also judge its own outputs. For Open-Reasoner-Zero-7B, which is already RLVR-trained, the reasoning warm-up is skipped.

  2. Self-rewarding warm-up ($w_{sr}$ steps, set to 200 across all models): After the reasoning warm-up (or from step 0 for Open-Reasoner-Zero-7B), the self-rewarding MSE loss is added, but the self-rewarding-based advantage integration is not yet applied. The model learns to calibrate its self-rewarding scores during this phase. "After a certain steps when the last-token self-rewarding loss is sufficiently small, we proceed to integrate verifier-based and self-rewarding-based advantages" (Section 3.3). Concretely, this means the advantage integration begins at step $w_r + w_{sr}$ (or step $w_{sr}$ for Open-Reasoner-Zero-7B).

Why this staged approach: the paper's ablation in Appendix D (Figure 6) shows that higher $\alpha$ values (0.5) degrade reasoning rewards, indicating that the self-rewarding loss can compete with the reasoning objective if introduced too aggressively. The warm-up ensures the model first learns to reason, then learns to self-assess, and only then uses self-assessment to improve reasoning—preventing a feedback loop where poor self-assessment degrades reasoning, which further degrades self-assessment.

The complete algorithm is summarized in Algorithm 1:

Algorithm 1: LaSeR

  • Input: initial policy $\pi_\theta$, prompts $\mathcal{D}$, verifier $r_v$, warm-up steps $w_r$ and $w_{sr}$, coefficient $\beta_v$, special token $z_c$, pre-calculated constant $c_{ref}$
  • For each step $s = 1, \ldots, S$:
    1. Set $\pi_{old} \leftarrow \pi_\theta$ (needed for KL divergence and reference distribution).
    2. Sample a batch of prompts $\mathcal{D}_s$ from $\mathcal{D}$.
    3. Generate $K = 8$ solutions $\{y^i\}_{i=1}^K$ for each prompt $x \in \mathcal{D}_s$.
    4. Calculate verifier-based rewards $r_v(x, y^i)$ and advantages (Equation 4); compute the standard RL loss.
    5. If $s \geq w_r$: calculate the last-token self-rewarding MSE loss (Equation 15) and add it to the RL loss.
    6. If $s \geq w_r + w_{sr}$: calculate self-rewarding-based advantages and perform advantage integration (Equation 16).
    7. Update $\pi_\theta$ using the GRPO policy gradient with the integrated loss and advantages.
  • Output: trained policy model $\pi_\theta$.

Hyperparameter Selection and Sensitivity

The paper introduces several new hyperparameters specific to LaSeR and provides empirical guidance on their selection (Appendix D, Figure 6):

  • $\beta_v = 0.1$ (the KL coefficient for the verification objective): this controls the scaling of the self-rewarding score relative to the log-probability ratio. The paper finds that $\beta_v$ "has little impact on optimizing the self-rewarding scores, as long as it remains within a reasonable range (0.1∼0.5)." However, "much smaller values of $\beta_v$ can impair the model's reasoning capability" (Section 3.4 and Appendix D). The reason: if $\beta_v = 0.05$, then for a correct solution the target $\pi_\theta(z_c|x, y) = \exp(1/0.05) \cdot e^{-23} = e^{20} \cdot e^{-23} = e^{-3} \approx 0.05$. This is a non-trivial probability that starts competing with the natural <EOS> token, causing "large interference with the optimization of reasoning capability." With $\beta_v = 0.1$, the target is $e^{-13} \approx 2.3 \times 10^{-6}$, which is safely negligible.

  • $\alpha = 0.1$ (MSE loss weight): controls the trade-off between reasoning and self-rewarding optimization. Figure 6 shows that $\alpha = 0.5$ causes a visible drop in training rewards, while $\alpha = 0.1$ achieves high self-verification F1 without degrading reasoning.

  • $\tau = 0.1$ (self-rewarding advantage weight): controls the contribution of self-rewarding scores to the policy gradient advantage. Set to give a small but meaningful contribution.

  • $T = 0.1$ (standard deviation threshold for advantage filtering): minimum within-group standard deviation of self-rewarding scores below which the self-rewarding advantage is disabled ($\tau = 0$). This prevents noisy advantage estimates when all solutions in a group receive similar self-rewarding scores.

  • $w_r = 200$ and $w_{sr} = 200$: the warm-up step counts. The total training is 1000 steps for OctoThinker-3B-Short-Base and Qwen2.5-7B-Base, and 500 steps for Open-Reasoner-Zero-7B.

Inference-Time Usage: Computing and Applying the Self-Rewarding Score

At test time, the system operates as follows for each problem:

  1. Generate $K$ candidate solutions $\{y^1, \ldots, y^K\}$ from the trained policy model $\pi_\theta$, using standard autoregressive decoding (temperature 1.0, top-p 1.0).

  2. For each solution $y^i$, the model has produced the final token (typically <EOS>). To compute the self-rewarding score, the model runs one additional token inference at the position immediately after $y^i$. This forward pass computes the next-token probability distribution $\pi_\theta(\cdot | x, y^i)$ over the entire vocabulary. The log-probability assigned to the special token $z_c$ is extracted: $\log \pi_\theta(z_c | x, y^i)$.

  3. Compute the self-rewarding score: $r_s^i = \beta_v \log \pi_\theta(z_c | x, y^i) - \beta_v c_{ref}$. This requires no reference model, only the pre-calculated constant $c_{ref}$ (which is hardcoded based on the model family and special token).

  4. Self-verification decision: compare $r_s^i$ to 0.5. If $r_s^i > 0.5$, the solution is self-verified as correct; otherwise, incorrect. (This threshold arises naturally because $r_v \in \{0, 1\}$ and the MSE loss trains $r_s$ to match $r_v$.)

  5. Weighted majority voting (RM@K): instead of simple majority voting where each solution contributes equally to the answer tally, each solution's vote is weighted by its self-rewarding score $r_s^i$. The final answer is:

a∗=arg⁡max⁡a∑i:extracted answer of yi=arsia^* = \arg\max_a \sum_{i: \text{extracted answer of } y^i = a} r_s^i

This gives more influence to solutions the model believes are correct.

Computational cost comparison:

  • Standard GRPO inference (no verification): $K$ full generations, each $T$ tokens on average. Total: $K \times T$ tokens of inference.
  • Self-verification by separate generation (prior work): $K$ solution generations ($K \times T$ tokens) + $K$ verification generations ($K \times T'$ tokens, where $T'$ is the verification length). Total: $K \times (T + T')$ tokens, roughly double.
  • External reward model (e.g., 72B RM): $K$ solution generations ($K \times T$ tokens) + $K$ forward passes through the RM to get scores (cost depends on RM size).
  • LaSeR: $K$ solution generations ($K \times T$ tokens) + $K \times 1$ additional token inferences (the $z_c$ probability query). Total: $K \times (T + 1)$ tokens, essentially the same as standard generation.

The paper also discusses a potential way to achieve zero additional inference cost (Section 5.3): instead of querying the next-token probability after <EOS>, extract the log-probability of $z_c$ at the <EOS> token position itself (i.e., $\log \pi_\theta(z_c | x, y_{<T})$ where $y_T$ is the <EOS> token). This works because $\pi_{ref}(z_c | x, y_{<T})$ is also approximately constant ($e^{-28}$ for Qwen2.5-7B-Base). The risk is that "the generator may still select $z_c$ at the end of the sequence in few cases during training, which can adversely affect training stability as the generator continues to generate after $z_c$." The paper suggests setting top_p < 1.0 as a mitigation and leaves robust zero-cost self-rewarding to future work.

Comparison to Prior Approaches: Why This Is Different

The paper explicitly contrasts LaSeR with prior self-verification methods (Section 3.4) along two dimensions:

1. Generation cost: prior methods (Sareen et al., 2025; Liu et al., 2025a; Zha et al., 2025) require the model to generate solutions under one prompt template, then generate verifications under a second prompt template—two separate autoregressive generation processes. LaSeR instead "directly derives the self-rewarding result from the next-token log-probability of the final solution token." During RL training, because the token log-probabilities are already computed for the policy gradient, "we can directly replace the token id of the first padding token with the token id of the pre-specified token before computing the log-probabilities of the sequences, thereby incurring no additional computation cost during training."

2. Training signal: prior methods use an RL objective for the verification task, requiring advantage estimation for the verification tokens. LaSeR replaces this with a simple MSE loss, which is simpler to implement and has lower variance. The paper's comparison with SFT (Section 3.4, Appendix F, Figure 8) shows that SFT on $z_c$ "causes substantial interference with the optimization of reasoning capability, leading to a marked degradation in training rewards" because it drives $\pi_\theta(z_c|x, y)$ toward 1.0. LaSeR's MSE loss, by targeting $\exp(1/\beta_v) \cdot \pi_{ref}(z_c|x, y) \approx e^{-13}$ for correct solutions, avoids this interference entirely.

Summary of Design Choices and Their Justifications

  • Single-token verification over multi-token verification: eliminates length bias, makes the partition function provably negligible, and enables zero-cost computation during training.
  • MSE loss over RL for verification training: leverages the closed-form solution to avoid the variance and complexity of policy gradient methods for what is essentially a regression problem.
  • Unused special token $z_c$ over common tokens like "Yes": minimizes reference model probability ($\sim 10^{-10}$) and variance, making both the constant approximation $c_{ref}$ and the $Z(x, y) \approx 1$ approximation more accurate.
  • Reference log-probability as constant $c_{ref}$ over per-sample computation: validated empirically to have negligible impact on performance while halving computational cost.
  • Class-level loss re-weighting over unweighted MSE: prevents bias toward the majority class as the ratio of correct to incorrect solutions changes during training.
  • Staged warm-up over joint training from the start: prevents the self-rewarding loss from interfering with early reasoning skill acquisition.
  • Self-rewarding advantage integration with $\tau = 0.1$ over pure verifier-based advantages: provides a small, stabilizing signal that captures solution quality beyond binary correctness, with filtering to fall back to verifier-only when self-rewarding scores are uninformative.

4. Key Insights and Innovations

Innovation 1: Verification Can Be Collapsed to a Single-Token Probability Query by Exploiting the Partition Function's Near-Zero Value for Unused Tokens

The paper's most fundamental conceptual move is the recognition that the RL objective for verification has a closed-form solution where the problematic partition function Z(x, y) — the unknown normalization constant that prevents the standard implicit reward from being directly useful — collapses to approximately 1 when the verification output is restricted to a single pre-specified token with near-zero reference probability.

Prior work on implicit rewards (Rafailov et al., 2023; Mitchell et al., 2024) recognized that β log π_θ(y|x)/π_ref(y|x) equals the true reward up to an additive constant β log Z(x), but this constant is unknown and depends on the prompt x, making the implicit reward unusable as a verification signal without additional calibration. Cui et al. (2025) further confirmed that "directly aligning the implicit reward with the true reasoning reward during training degrades the policy model's generation ability" because of the fundamental gap introduced by Z(x). The field had therefore accepted that the partition function was an unavoidable obstacle — something to work around through separate reward models or explicit generation-based verification.

What makes LaSeR's insight distinctive is not the mathematics of the closed-form solution itself (which follows directly from prior work), but the diagnostic observation that Z(x, y) becomes tractable when the action space is collapsed to a single token. The partition function is Z(x, y) = ∑_z π_ref(z|x,y) exp(ˆr/β_v). For a full solution y, this sum runs over an exponentially large space of possible responses, making it impossible to compute. But for a single verification token z, the sum runs over the finite vocabulary. The paper then makes the empirical observation (Figure 5) that for an unused special token like <|vision_start|>, π_ref(z_c|x,y) ≈ 10^{-10}, and this value is nearly constant across all problems and solutions (standard deviation of only 0.04 on the log scale). When this near-zero probability is multiplied by exp(1/β_v) ≈ exp(10) ≈ 22026, the product is still only ∼4.4 × 10⁻⁶, meaning Z(x, y) ≈ 1.000004 and therefore log Z(x, y) ≈ 0.

This is a fundamental reframing, not an incremental improvement, because it changes the problem from "how do we estimate or circumvent this unknown constant?" to "how do we design the verification output so the constant disappears?" The answer — use a single unused token — converts a fundamental obstacle into an implementation detail. The partition function, which was the central theoretical barrier to using implicit rewards as verifiers, is eliminated not by a more sophisticated estimation technique but by a design choice that makes it numerically negligible.

The significance extends beyond the immediate application. This same insight could apply to any domain where one wants to extract a scalar assessment from a model's internal state: choose a probe token with near-zero reference probability, and the KL-constrained RL objective's closed form becomes directly usable without partition function estimation. The paper's theoretical derivation (Equations 9–12) is not just solving the verification problem — it is demonstrating a general principle about when and how implicit rewards become practically usable signals.

The evidence supporting this as a genuine insight rather than just a mathematical trick: Table 3 shows that the constant approximation for π_ref(z_c|x,y) has negligible impact on performance (reasoning accuracy 45.1% vs. 45.0%, self-verification F1 79.4% vs. 79.3%), confirming that the reference model's log-probability really is near-constant across problems. The self-verification F1 scores of ~80% in Table 1 demonstrate that the resulting signal is genuinely informative — if Z(x, y) were not approximately 1, the self-rewarding score would be systematically biased, and the alignment with the binary verifier reward would fail.

Innovation 2: Self-Verification Can Be Learned Through a Simple Regression Loss Rather Than a Separate RL Procedure

The paper's second conceptual contribution is the observation that once the closed-form relationship between the verifier reward and the self-rewarding score is established, training self-verification does not require a reinforcement learning procedure at all — a straightforward Mean Squared Error loss suffices.

This is a non-obvious simplification because the standard approach to training any capability within an RL framework is to add it as another RL objective. Prior self-verification methods (Sareen et al., 2025; Liu et al., 2025a; Zha et al., 2025) all treat verification as a second RL problem: the model generates a verification, receives a reward based on whether the verification is correct, and the policy gradient updates the model to produce better verifications. This mirrors the standard RLHF pipeline where a reward model is trained separately and then optimized against. Even the implicit reward literature (Rafailov et al., 2023) formulated the alignment problem as optimizing the implicit reward to match human preferences, requiring a Bradley-Terry preference model and a contrastive loss.

LaSeR's move is to recognize that the RL objective for verification is self-solved by its own closed form. Equation 12 says that at optimality, r_v(x, y) = β_v log[π_θ(z_c|x,y)/π_ref(z_c|x,y)]. If this equality holds at the optimum, then during training, the model should simply be pushed to make it hold — which is exactly what a regression loss does. There is no need for advantage estimation, no need for a value function, no need to estimate the policy gradient for the verification tokens. The MSE loss (r_s - r_v)² directly implements the theoretical optimum as a training objective.

This is conceptually analogous to how DPO (Rafailov et al., 2023) eliminated the need for explicit reward modeling and RL in preference optimization by solving the KL-constrained RL objective in closed form and converting it to a classification loss. But there is a crucial difference: DPO still requires a contrastive formulation (preferred vs. dispreferred responses) and a Bradley-Terry loss. LaSeR goes further — the verification problem reduces to a pointwise regression with no contrastive pairs needed. Each solution independently receives a target value (0 or 1), and the self-rewarding score is regressed directly onto that target. This is possible because the theoretical relationship is an equality, not a preference ordering.

The contrast with SFT further sharpens why this is innovative. Supervised fine-tuning on z_c would drive π_θ(z_c|x,y) toward 1.0 for correct solutions, which the paper shows (Appendix F, Figure 8) causes "substantial interference with the optimization of reasoning capability." The regression target exp(1/β_v) · π_ref(z_c|x,y) ≈ e^{-13} ≈ 2.3 × 10^{-6} is fundamentally different — it keeps the probability mass on z_c vanishingly small, meaning the model never needs to make z_c a likely output. The self-rewarding signal exists purely in the log-probability space, not in the sampling distribution. This is a conceptually distinct training paradigm: the model learns to encode information in probabilities it will never actually sample, using those probabilities purely as side-channel signals for downstream use. Nothing in prior self-verification or reward modeling work operates this way.

The evidence: Figure 6 in Appendix D shows that the MSE loss achieves high self-verification F1 without degrading training rewards, while Figure 8 in Appendix F shows SFT causes "substantial interference" and "marked degradation in training rewards." The fact that a simple regression loss — with the right target value — succeeds where both SFT and full RL procedures are either harmful or unnecessarily complex is a genuinely non-trivial finding about how to train metacognitive capabilities in LLMs.

Innovation 3: Self-Assessment Can Be Bootstrapped Back into Training as an Auxiliary Advantage Signal

The paper's third distinctive contribution is the demonstration that a model's learned self-assessment capability can be folded back into the RL training process as an auxiliary advantage signal, creating a virtuous cycle where better self-assessment improves reasoning, which in turn provides better training data for self-assessment.

Prior work treats verification and generation as separate stages: verifiers (whether external or self-generated) are used at test time to rank or select solutions, but they do not feed back into the training process that produced the generator. The standard RLVR pipeline uses only the deterministic verifier's binary signal for advantage estimation. Self-verification work like Liu et al. (2025a) trains the verification capability jointly with reasoning, but the verification outputs are only used at inference time for solution selection — they are not integrated into the policy gradient that trains the reasoning capability itself.

LaSeR's advantage integration (Equation 16) breaks this separation by blending the continuous self-rewarding score into the token-level advantage estimates that drive the policy gradient. The motivation is both principled and practical: "The self-rewarding scores... help mitigate the issue of misjudgments by rule-based verifiers, which often occur when the format of ground-truth answer is overly complex, and produce more fine-grained rewards" (Section 3.3). The binary verifier is all-or-nothing — a solution that reasons correctly but formats the answer wrong gets reward 0, identical to a completely nonsensical solution. The self-rewarding score, being continuous and derived from the model's internal assessment of the full solution, can assign a higher value to a nearly-correct solution than to a completely wrong one, providing a smoother optimization signal.

What makes this innovative is not the idea of combining reward signals (ensemble methods in RL are well-established) but the specific structure: the self-rewarding signal is learned simultaneously with the policy it helps train, using the same data, in the same optimization loop. This is a form of bootstrapping — the model learns to evaluate its own outputs, and those evaluations immediately improve the learning process that produced the outputs. There is a risk of circularity (poor self-assessment leading to poor advantages, degrading reasoning, further degrading self-assessment), which the paper addresses through the staged warm-up schedule and the standard-deviation-based filtering (τ = 0 when std(r_s) < 0.1). The fact that this bootstrapping produces net positive results rather than collapsing is non-trivial.

The evidence in Table 1 shows that removing the self-rewarding advantage integration ("- SWA") consistently reduces average reasoning accuracy: Qwen2.5-7B-LaSeR drops from 42.7% to 41.1%, Open-Reasoner-Zero-7B-LaSeR drops from 45.5% to 45.4% (smaller because this model is already strong), OctoThinker-3B-Short-LaSeR drops from 32.8% to 32.4%. The gains are modest but consistent, and they come "for free" — the self-rewarding scores are already being computed for the MSE loss, so using them in the advantage requires no additional computation. This is a genuine synergy: the self-rewarding capability that is trained as an auxiliary objective becomes useful as an input to the primary objective, making the auxiliary objective not just an add-on but an integrated component of the learning process.

Innovation 4: The Choice of Probe Token Is a Design Parameter That Controls the Theoretical Validity and Practical Stability of Self-Rewarding

The paper makes an empirical observation with theoretical consequences: the choice of which token to use as the self-rewarding probe fundamentally determines both the validity of the partition function approximation and the stability of training, and unused special tokens are dramatically superior to common tokens for both reasons.

This is not merely a hyperparameter choice — it is a diagnostic insight about the relationship between a token's reference probability and the reliability of the implicit reward signal. Figure 5 shows that for the common token "Yes" with Qwen2.5-7B-Base, −log π_ref(z_c|x,y) ≈ 9.13 with standard deviation 0.13, meaning π_ref ≈ 10^{-4}. For the unused special token <|vision_start|>, the value is −log π_ref ≈ 23.11 with standard deviation 0.04, meaning π_ref ≈ 10^{-10}. The difference is not just magnitude — it is six orders of magnitude in probability space and roughly 3× tighter in relative standard deviation.

Why does this matter beyond the obvious preference for a smaller number? Because the partition function approximation Z(x, y) ≈ 1 depends on π_ref(z_c|x,y) · exp(1/β_v) ≈ 0. With π_ref ≈ 10^{-4} and β_v = 0.1, this product is 10^{-4} · e^{10} ≈ 2.2, which is not negligible — Z(x, y) would be approximately 3.2, and log Z(x, y) ≈ 1.16. The self-rewarding score would be systematically biased by this additive constant, and since the constant depends on π_ref which has higher variance for common tokens (standard deviation 0.13 vs. 0.04 on the log scale), the bias would vary across problems. Using a common token would therefore break the theoretical justification for the MSE loss — the loss would be regressing onto a target that is not equal to the true reward at optimality.

Furthermore, the paper notes a training stability concern with the zero-cost variant (Section 5.3): if z_c is probed at the <EOS> token position rather than after it, a common token with higher probability might occasionally be sampled, causing the generator to continue generating after z_c and destabilizing training. An unused special token minimizes this risk.

This insight has implications beyond LaSeR: it suggests that whenever one wants to extract a scalar signal from an LLM's internal probability distribution — for confidence estimation, uncertainty quantification, or reward modeling — choosing a probe token with the smallest possible reference probability is important not just for numerical precision but for the theoretical validity of treating the log-probability ratio as an absolute signal. The paper essentially operationalizes the principle that "the partition function is negligible if and only if the probe token's reference probability is negligible," and provides empirical thresholds for what "negligible" means (∼e^{-23} with standard deviation <0.05 on the log scale).

The validation: Table 3 shows that even with the reference probability simplified to a constant, the self-verification F1 reaches 79.4%, confirming that the combination of low absolute value and low variance makes the approximation robust. The contrast between using "Yes" vs. <|vision_start|> is not directly ablated in the main experiments, but Figure 5 provides the quantitative justification, and the paper's explicit choice of unused special tokens for all experiments (Section 4.1) indicates this is a considered design decision rather than an arbitrary selection.

5. Experimental Analysis

Evaluation Methodology

  • Dataset. The primary training dataset is DeepMath-103K (He et al., 2025), described as "a large-scale and high-quality mathematical reasoning dataset" (Section 4.1). Evaluation is conducted on five math reasoning benchmarks: MATH500 (Hendrycks et al., 2021), AMC23 (AI-MO, 2024b), AIME24 (AI-MO, 2024a), AIME25 (OpenCompass, 2025), and OlympiadBench (He et al., 2024). The paper states the specific test splits are used but does not report exact sizes per benchmark; MATH500 contains 500 problems, while AMC23, AIME24, and AIME25 are smaller competition sets. For OctoThinker-based models, AIME24-25 results are not reported because "the number of correct solutions is quite insufficient for a reliable evaluation" (Table 1 note).

  • Base models. Three base models spanning two architectures and three training stages are used: (1) OctoThinker-3B-Short-Base (Wang et al., 2025a), a mid-trained variant of LLaMA3.2-3B-Base (MetaAI, 2024b); (2) Qwen2.5-7B-Base (Qwen Team, 2024), a pre-trained model with no prior RLVR; and (3) Open-Reasoner-Zero-7B (Hu et al., 2025), a reinforced version of Qwen2.5-7B-Base that has already undergone RLVR training. This range of starting points—from base pretrained to mid-trained to already-reinforced—tests whether LaSeR provides gains regardless of initial model capability. The paper states that LaSeR "can be seamlessly integrated into any RLVR framework" (Section 4.1), and these three starting points are chosen to demonstrate this generality.

  • Metrics. Two primary metrics are reported:

    • Reasoning accuracy: average Pass@1 accuracy across each benchmark. On MATH500 and OlympiadBench, 2 solutions are sampled per problem; on AMC23, AIME24, and AIME25, 32 solutions are sampled per problem. The paper reports the average accuracy over these samples (i.e., the fraction of problems where at least one sampled answer is correct in expectation, though the exact Pass@1 computation method is not specified as either "at least one correct" or "average correctness per sample"—the reported numbers are most consistent with per-sample average accuracy given the magnitude of improvement over base models).
    • Self-verification F1 score: the harmonic mean of self-verification accuracy on self-generated correct solutions and self-generated incorrect solutions. For the baseline models (Base and GRPO), self-verification is performed using a separate prompt template (shown in Appendix I) that asks the model to evaluate its own response and output a rating in \boxed{}. For LaSeR models, self-verification is computed from the last-token self-rewarding score: r_s > 0.5 indicates the model judges the solution correct, and r_s ≤ 0.5 indicates incorrect. "Any solution without a final answer is automatically treated as incorrect and excluded from the verification accuracy calculation" (Section 4.1).

  • Baselines. Four types of baselines are compared:

    • Base model without RLVR: the raw OctoThinker-3B-Short-Base, Qwen2.5-7B-Base, or Open-Reasoner-Zero-7B before any additional RLVR training.
    • GRPO (Shao et al., 2024): the standard Group Relative Policy Optimization algorithm applied to each base model on DeepMath-103K. This is the primary baseline against which LaSeR's reasoning improvements are measured. The paper uses GRPO as its base RL algorithm and treats "GRPO without self-rewarding" as the direct comparison point.
    • LaSeR - SWA (Self-rewarding Weighted Advantage ablation): LaSeR with the self-rewarding MSE loss but without integrating self-rewarding scores into the RL advantage estimation (Equation 16). This isolates the contribution of the advantage integration technique.
    • External reward models (Table 2 only): three independently trained verifiers of varying sizes—Qwen2.5-Math-7B-PRM800K (Zhang et al., 2025), Qwen2.5-Math-PRM-7B (Zhang et al., 2025), and Qwen2.5-Math-RM-72B (Yang et al., 2024)—are used to benchmark the self-verification F1 of LaSeR-trained models against dedicated external verifiers. These are not training baselines but evaluation-time comparison points for verification quality.

  • Generation budget / compute accounting. The paper does not use a formal generation budget metric (unlike the Snell et al. 2024 reference example) because LaSeR's primary comparison is at equal training steps, not equal inference compute. All GRPO and LaSeR training runs use identical rollout hyperparameters: K = 8 solutions per prompt (Table 4), 1000 optimization steps for Qwen2.5-7B-Base and OctoThinker-3B-Short-Base, and 500 steps for Open-Reasoner-Zero-7B. Inference-time scaling experiments (Figure 2) sweep the number of sampled solutions from 2^1 to 2^5 (2 to 32), comparing standard majority voting (Maj@K) against self-rewarding-weighted majority voting (RM@K). The key efficiency claim—that LaSeR incurs "only one additional token inference" compared to standard generation—is measured in inference tokens, not FLOPs.

  • Cross-validation / statistical protocol. The paper does not employ cross-validation for its primary results. Strategy selection (e.g., hyperparameter choices like β_v = 0.1, α = 0.1, τ = 0.1) is performed on training dynamics (Appendix D, Figure 6) rather than held-out validation sets. The self-verification F1 scores and reasoning accuracies are reported as point estimates on the test sets without confidence intervals. For the general reasoning experiments (Section 5.2), the paper samples 4 solutions per problem and reports both per-sample average accuracy and (weighted) majority voting accuracy. This absence of statistical uncertainty quantification means that the reported differences—particularly small ones like Qwen2.5-7B-Base's 41.8% vs. 42.7% average accuracy gain—cannot be assessed for significance from the paper's data alone.

Main Quantitative Results

Reasoning and Self-Verification Performance (Table 1)

The headline result is that LaSeR improves both reasoning accuracy and self-verification F1 scores across all three base model families, with the largest gains in self-verification capability—models go from near-chance or modest self-verification F1 (~15–50%) to approximately 72–80% F1 while maintaining or improving reasoning accuracy over the GRPO baseline.

OctoThinker-3B-Short-Base results:

  • Base model: 2.0% average reasoning accuracy, 15.7% self-verification F1.
  • GRPO: 30.8% average reasoning accuracy, 51.0% self-verification F1.
  • LaSeR: 32.8% average reasoning accuracy (+2.0 percentage points over GRPO), 72.5% self-verification F1 (+21.5 points over GRPO).
  • LaSeR - SWA: 32.4% reasoning, 72.4% self-verification F1. The advantage integration contributes +0.4 points to reasoning and +0.1 to self-verification.

On individual benchmarks: MATH500 improves from 49.8% (GRPO) to 53.1% (LaSeR), AMC23 from 25.3% to 27.0%, OlympiadBench from 17.3% to 18.2%. Self-verification F1 on MATH500 jumps from 56.9% to 73.6%.

Qwen2.5-7B-Base results:

  • Base model: 14.8% average reasoning accuracy, 32.9% self-verification F1.
  • GRPO: 41.8% average reasoning accuracy, 49.2% self-verification F1.
  • LaSeR: 42.7% average reasoning accuracy (+0.9 points over GRPO), 79.6% self-verification F1 (+30.4 points over GRPO).
  • LaSeR - SWA: 41.1% reasoning, 79.9% self-verification F1. The advantage integration contributes +1.6 points to reasoning.

The per-benchmark pattern is mixed: LaSeR beats GRPO on AMC23 (58.1% vs. 55.9%) and OlympiadBench (44.1% vs. 43.3%), essentially ties on MATH500 (80.2% vs. 79.9%), slightly underperforms on AIME24 (15.4% vs. 16.2%), and outperforms on AIME25 (15.7% vs. 13.8%). The self-verification improvements are dramatic and consistent: MATH500 F1 goes from 54.6% (GRPO) to 83.2% (LaSeR), AMC23 from 59.7% to 82.5%, AIME24 from 36.6% to 79.6%, AIME25 from 41.5% to 74.3%, OlympiadBench from 53.5% to 78.3%.

Open-Reasoner-Zero-7B results:

  • Base model: 44.0% average reasoning accuracy (already RLVR-trained), 43.3% self-verification F1.
  • GRPO: 45.0% average reasoning accuracy, 38.8% self-verification F1. Notably, additional GRPO training degrades self-verification F1 from 43.3% to 38.8%, even as reasoning improves slightly.
  • LaSeR: 45.5% average reasoning accuracy (+0.5 points over GRPO, +1.5 over base), 77.6% self-verification F1 (+38.8 points over GRPO, +34.3 over base).
  • LaSeR - SWA: 45.4% reasoning, 76.7% self-verification F1.

The reasoning gains are modest but consistent on this already-strong model: AMC23 improves from 61.9% to 62.7%, AIME24 from 18.1% to 19.1%, OlympiadBench from 47.1% to 47.8%. The self-verification recovery is the striking result: GRPO alone destroys the base model's self-verification capability on several benchmarks (AIME24 drops from 45.9% F1 to 14.6%; AIME25 drops from 55.2% to 28.1%), while LaSeR not only recovers but dramatically exceeds the base model's performance (AIME24: 64.6%; AIME25: 77.7%).

Interpretation of the Open-Reasoner-Zero-7B GRPO degradation: This result reveals a previously undocumented phenomenon—standard RLVR training can catastrophically damage a model's ability to self-assess, even as it improves reasoning accuracy. The base Open-Reasoner-Zero-7B, having been trained with RLVR already, has some self-verification capability (43.3% F1) through its prompted self-verification responses. Additional GRPO training on new data erodes this capability (38.8% F1) because the policy is being optimized purely for answer correctness, not for maintaining its metacognitive calibration. LaSeR's self-rewarding MSE loss prevents this erosion and instead builds much stronger self-verification (77.6% F1).

Comparison with External Verifiers (Table 2)

To contextualize the self-verification performance, the paper compares LaSeR's self-rewarding F1 scores against three external reward models of varying sizes and training methods, evaluated on the same solutions generated by LaSeR-trained policy models. The headline: LaSeR's self-rewarding (with no separate model) matches or exceeds equally-sized external verifiers and approaches the performance of a 72B reward model.

On OctoThinker-3B-Short-LaSeR's generated solutions:

  • Qwen2.5-Math-7B-PRM800K (7B): 71.5% average F1
  • Qwen2.5-Math-PRM-7B (7B): 69.5% average F1
  • Qwen2.5-Math-RM-72B (72B): 77.9% average F1
  • LaSeR self-rewarding (3B): 72.5% average F1

The 3B model's self-rewarding outperforms both 7B external verifiers and trails the 72B verifier by only 5.4 points.

On Qwen2.5-7B-LaSeR's generated solutions:

  • Qwen2.5-Math-7B-PRM800K (7B): 55.3% average F1
  • Qwen2.5-Math-PRM-7B (7B): 77.4% average F1
  • Qwen2.5-Math-RM-72B (72B): 80.0% average F1
  • LaSeR self-rewarding (7B): 79.6% average F1

LaSeR essentially matches the 72B external verifier (79.6% vs. 80.0%) and substantially exceeds both 7B external verifiers. The PRM800K-trained verifier performs notably poorly (55.3%), likely due to distribution mismatch between its training data (GPT-4 outputs with human step labels from Lightman et al., 2023) and the Qwen2.5-7B-LaSeR solution distribution—a known issue the paper does not elaborate on but that reinforces the value of self-verification that is inherently on-distribution.

On Open-Reasoner-Zero-7B-LaSeR's generated solutions:

  • Qwen2.5-Math-7B-PRM800K (7B): 47.9% average F1
  • Qwen2.5-Math-PRM-7B (7B): 75.9% average F1
  • Qwen2.5-Math-RM-72B (72B): 76.8% average F1
  • LaSeR self-rewarding (7B): 77.6% average F1

LaSeR slightly exceeds the 72B verifier (77.6% vs. 76.8%). The PRM800K verifier again struggles severely (47.9%).

Key nuance in this comparison: The external verifiers are evaluated on solutions from LaSeR-trained models, not on solutions from their own training distributions. This favors LaSeR, since its self-rewarding is trained on exactly its own output distribution. A fairer comparison might also evaluate the external verifiers on solutions from their own base models (e.g., Qwen2.5-Math models). However, the practical use case the paper targets is exactly this: at test time, the model needs to verify its own outputs, and training an external verifier on the evolving policy distribution is expensive. LaSeR's self-rewarding solves this distribution-matching problem by construction.

Inference-Time Scaling via Weighted Majority Voting (Figure 2)

The paper evaluates whether the learned self-rewarding scores improve inference-time answer selection through weighted majority voting. Results are shown across six subfigures (Figure 2a–2f) for three base models on two benchmarks (MATH500 and OlympiadBench), comparing three methods:

  • GRPO baseline (Maj@K): standard majority voting on K solutions from the GRPO-trained model (no self-rewarding).
  • LaSeR (Maj@K): standard majority voting on K solutions from the LaSeR-trained model. This tests whether LaSeR's training improves the raw solution quality (regardless of self-rewarding usage).
  • LaSeR (RM@K): weighted majority voting on K solutions from the LaSeR-trained model, using self-rewarding scores r_s as weights. This tests whether the self-rewarding signal improves selection.

MATH500 results (Figure 2a–2c):

OctoThinker-3B-based models (Figure 2a): Starting from K=2 (21) to K=32 (25):

  • GRPO Maj@K: ~50% at K=2, rising to ~62% at K=32.
  • LaSeR Maj@K: starts slightly higher (~52% at K=2) and rises to ~63% at K=32, tracking close to GRPO with a small consistent advantage.
  • LaSeR RM@K: starts at ~52% at K=2 (no advantage over Maj@K at low K), but shows a widening gap at higher K, reaching ~64% at K=32. The weighted voting provides roughly +1 point over unweighted at K=32.

Qwen2.5-7B-based models (Figure 2b):

  • GRPO Maj@K: ~80% at K=2, rising to ~84.5% at K=32.
  • LaSeR Maj@K: nearly identical to GRPO across all K (~80% to ~84.5%).
  • LaSeR RM@K: consistently above both, reaching ~85.5% at K=32—roughly +1 point over GRPO Maj@K. The gap appears stable across K values.

Open-Reasoner-Zero-7B-based models (Figure 2c):

  • GRPO Maj@K: ~83.5% at K=2, rising to ~87% at K=32.
  • LaSeR Maj@K: slightly above GRPO at low K (~84% vs. ~83.5%) but nearly identical at high K.
  • LaSeR RM@K: shows the clearest advantage here, reaching ~88% at K=32—roughly +1 point over GRPO.

OlympiadBench results (Figure 2d–2f):

OctoThinker-3B-based models (Figure 2d):

  • GRPO Maj@K: ~17% at K=2, rising to ~24% at K=32.
  • LaSeR Maj@K: tracks close to GRPO, ending at ~24%.
  • LaSeR RM@K: a clear advantage emerges at higher K, reaching ~27% at K=32—roughly +3 points.

Qwen2.5-7B-based models (Figure 2e):

  • GRPO Maj@K: ~45% at K=2, rising to ~49.5% at K=32.
  • LaSeR Maj@K: slightly lower than GRPO at low K but catches up at K=32 (~49.5%).
  • LaSeR RM@K: shows a consistent ~1 point advantage, reaching ~50.5% at K=32.

Open-Reasoner-Zero-7B-based models (Figure 2f):

  • GRPO Maj@K: ~49% at K=2, rising to ~53.5% at K=32.
  • LaSeR Maj@K: nearly identical to GRPO across all K.
  • LaSeR RM@K: shows an advantage at higher K, reaching ~54.5% at K=32—roughly +1 point.

Key patterns: The RM@K advantage over Maj@K is modest (+1–3 percentage points at K=32) but remarkably consistent—across all three model families and both benchmarks, weighted voting improves over unweighted voting at moderate-to-high sample counts. At low K=2, the advantage is negligible or zero because with only two samples, the weight distribution cannot meaningfully shift the outcome unless one solution's score is near 1 and the other's near 0. The LaSeR Maj@K results (without self-rewarding weights) are consistently at or near GRPO Maj@K, confirming that the addition of the self-rewarding MSE loss during training does not degrade the model's raw solution quality—the reasoning gains reported in Table 1 translate to comparable or better majority voting performance. The larger gains on OlympiadBench for OctoThinker-3B (+3 points) suggest that self-rewarding is more valuable when the base model is weaker and solutions are more diverse in quality.

Ablation Studies and Robustness Checks

Reference log-probability simplification: Comparing LaSeR with and without the constant c_ref approximation on Open-Reasoner-Zero-7B (Table 3), the paper finds nearly identical performance: average reasoning accuracy is 45.1% with simplification versus 45.0% without, and average self-verification F1 is 79.4% versus 79.3%. This confirms that the reference model's log-probability for the special token is sufficiently constant across problems that replacing it with a pre-computed mean introduces negligible error, while halving the computational cost (no reference model forward pass needed).

Self-rewarding advantage integration (Table 1, "- SWA" rows): Removing the self-rewarding-based advantage signal from the policy gradient (Equation 16) consistently reduces reasoning accuracy. The magnitude varies by base model: Qwen2.5-7B-LaSeR drops from 42.7% to 41.1% average (−1.6 points, the largest effect), OctoThinker-3B-Short-LaSeR drops from 32.8% to 32.4% (−0.4 points), and Open-Reasoner-Zero-7B-LaSeR drops from 45.5% to 45.4% (−0.1 points). The diminishing returns on Open-Reasoner-Zero-7B—which starts with the strongest reasoning—suggest that advantage integration provides the most benefit when the model is still developing its reasoning capabilities and the verifier-based rewards are noisy due to formatting errors in intermediate-quality solutions. Self-verification F1 is largely unchanged by removing advantage integration (e.g., Qwen2.5-7B: 79.6% vs. 79.9%), indicating that the MSE loss alone is sufficient for learning self-assessment; the advantage integration only affects reasoning.

Self-rewarding hyperparameters (β_v and α): Appendix D, Figure 6 shows training dynamics for different (β_v, α) combinations on Open-Reasoner-Zero-7B. The key findings:

  • α = 0.1 preserves reasoning; α = 0.5 degrades it. Training rewards with α = 0.5 are visibly lower than with α = 0.1, confirming that "assigning a larger weight α to the last-token self-rewarding loss has a more detrimental impact on the model's reasoning capabilities."
  • β_v has little impact on self-verification in the range 0.1–0.5—all values achieve similar self-verification F1 scores. However, the paper notes that much smaller β_v (e.g., 0.05) would cause the target probability π_θ(z_c|x,y) to become non-negligible (∼0.05 for correct solutions), interfering with reasoning optimization.
  • The baseline GRPO curve (no self-rewarding loss) achieves higher training rewards but zero self-verification capability, confirming the expected tradeoff.

Class-level loss re-weighting (Appendix E, Figure 7): Training with and without class re-weighting on Open-Reasoner-Zero-7B shows that re-weighting produces "a more balanced self-verification performance by mitigating the bias toward the majority class with larger sample size." Without re-weighting, the self-verification F1 is lower and more variable. The training rewards are essentially identical, confirming that re-weighting only affects the self-verification calibration, not reasoning capability.

Self-rewarding MSE loss vs. SFT loss (Appendix F, Figure 8): Training with SFT loss (Equation 17) on Open-Reasoner-Zero-7B causes "substantial interference with the optimization of reasoning capability, leading to a marked degradation in training rewards." The SFT loss decreases extremely slowly (remaining around 10^2 to 10^3 on a log scale after 200 steps), while the LaSeR MSE loss drops to ∼10^{-1} within 50 steps. This is attributed to the SFT objective driving π_θ(z_c|x,y) from near-zero to 1.0 for correct solutions, which requires a massive probability shift at the final position that competes with the <EOS> token's dominance. The LaSeR target of e^{-13} is achievable with minimal disruption to the token distribution.

General reasoning domain transfer (Section 5.2, Figure 3): Training LaSeR on Qwen3-4B-Base (Yang et al., 2025a) with a filtered WebInstruct-verified dataset and a model-based verifier (general-verifier-1.5B from Ma et al., 2025), without advantage integration, yields:

  • MMLU-Pro: average accuracy is comparable (66.54% GRPO vs. 67.10% LaSeR), and weighted majority voting (RM@4) at 67.10% matches the average accuracy—no gain from self-rewarding in selection.
  • GPQA-Diamond: average accuracy is comparable (44.44% GRPO vs. 45.96% LaSeR), with RM@4 matching average accuracy.
  • Self-rewarding score distributions (Figures 3b, 3c) show substantial overlap between correct and incorrect solutions on MMLU-Pro, and near-complete overlap on GPQA-Diamond—the self-rewarding signal is too noisy to provide useful discrimination. The paper speculates this is due to (a) weaker general reasoning capability limiting the upper bound of self-assessment, and (b) the model-based verifier having "limited verification ability, resulting in high noise in the reasoning rewards, which in turn affects the optimization of the self-rewarding capability." This is a genuine negative result: LaSeR's self-rewarding does not transfer well to general reasoning when the underlying verifier is weak and the model's reasoning is less robust.

Generation temperature and sampling: All generations (training and evaluation) use temperature = 1.0 and top-p = 1.0 (Table 4). This is a deliberate choice to maximize solution diversity during RLVR training—standard practice in GRPO-based methods. No ablation on temperature is reported, which leaves open the question of whether self-rewarding accuracy degrades at lower temperatures where solution diversity is reduced and the model's confidence may be miscalibrated.

Critical Assessment

Claim 1: "Our method not only improves the model's reasoning performance but also equips it with remarkable self-rewarding capability"

What the experiments demonstrate: The reasoning improvement claim is modest and inconsistent across benchmarks. LaSeR improves average reasoning accuracy over GRPO by +2.0 points for OctoThinker-3B-Short-Base, +0.9 points for Qwen2.5-7B-Base, and +0.5 points for Open-Reasoner-Zero-7B (Table 1). These are small absolute gains, and the per-benchmark breakdown reveals that not all benchmarks improve—Qwen2.5-7B-LaSeR slightly underperforms GRPO on AIME24 (15.4% vs. 16.2%) and Open-Reasoner-Zero-7B-LaSeR essentially ties GRPO on MATH500 (82.8% vs. 83.1%). The claim "improves reasoning" is true on average but the effect size is small enough that whether it holds on a new benchmark is uncertain without confidence intervals.

The self-rewarding capability claim is much stronger and better supported: self-verification F1 improves by +21.5 points (OctoThinker-3B), +30.4 points (Qwen2.5-7B), and +38.8 points (Open-Reasoner-Zero-7B) over GRPO. These are large, consistent gains across all five benchmarks and all three model families. The external verifier comparison (Table 2) further validates the quality: a 7B model's self-rewarding matches or exceeds dedicated 72B reward models on its own outputs.

What was not tested: The paper does not evaluate whether the self-rewarding capability transfers across domains—can a Qwen2.5-7B-LaSeR trained on math accurately self-assess solutions to physics or code problems? The general reasoning experiment suggests not, but that experiment used a different model (Qwen3-4B) and a different verifier. Cross-domain transfer of self-rewarding capability from math to science or code is an important missing experiment, since deployment scenarios rarely involve only the training distribution.

Claim 2: "Incurs only the minimal extra cost of one additional token inference"

What the experiments demonstrate: The paper does not directly measure inference cost (latency, FLOPs) in its experiments. The "one additional token" claim is architectural rather than empirical: during training, the self-rewarding score is extracted from the already-computed forward pass ("no additional computation cost during training," Section 3.4); during inference, one additional forward pass through the final transformer layer is needed to compute the next-token distribution after <EOS>. The paper does not provide latency measurements, throughput comparisons, or FLOP counts to substantiate the "minimal" claim.

The zero-cost variant discussed in Section 5.3 (extracting the self-rewarding score from the <EOS> position rather than after it) is described as having stability issues and is not implemented in the main experiments. The paper does not report how often the special token is inadvertently sampled during training with this variant.

What was not tested: A direct wall-clock time or throughput comparison between LaSeR inference and (a) standard generation without self-rewarding, (b) prompted self-verification (generate verification text), and (c) external verifier scoring. The efficiency claim is theoretical and should be treated as such.

Claim 3: "Boosting its inference-time scaling performance"

What the experiments demonstrate: Figure 2 shows that LaSeR's weighted majority voting (RM@K) consistently outperforms unweighted majority voting (Maj@K), but the margins are modest: +1–3 percentage points at K=32 across all model families and benchmarks. The gains grow with K, suggesting that self-rewarding becomes more valuable when there are more solutions to discriminate among. However, the paper only evaluates up to K=32, which is relatively small—prior work on inference-time scaling (Snell et al., 2024) evaluated up to K=2048 or higher, where verifier quality matters much more. Whether the +1–3 point advantage at K=32 scales to larger K is unknown.

Additionally, the figures show that LaSeR Maj@K (without self-rewarding weights) sometimes underperforms GRPO Maj@K (e.g., Qwen2.5-7B on OlympiadBench at K=2, Figure 2e), meaning LaSeR's in-improvement comes primarily from the weighting mechanism, not from better raw solution quality. This is fine—the weighting is the point—but it means the reasoning gains from LaSeR training do not reliably translate to better majority voting without the weighting.

What was not tested: Scaling to large K (64, 128, 256, 512) where verifier quality typically matters most. Comparison against best-of-N selection using external verifiers (to quantify how much of the inference-time gain is due to having any verifier vs. having a good verifier). Whether the self-rewarding scores are well-calibrated enough to use for confidence-based filtering (discarding low-confidence solutions entirely) rather than just weighted voting.

Claim 4: "Jointly optimizing the reasoning and self-rewarding capabilities at nearly zero additional cost"

What the experiments demonstrate: The training cost claim is architectural: the self-rewarding MSE loss is computed from the same forward pass used for the RL loss, requiring only a different indexing operation. The paper does not report training wall-clock time for LaSeR vs. GRPO. However, the warm-up schedule means LaSeR training has phases where only the RL loss is active (first 200 steps for base models), which would be identical in cost to GRPO. After warm-up, the additional MSE loss computation adds a negligible amount of gradient computation relative to the policy gradient.

What was not tested: Training throughput (steps per second) comparison between GRPO and LaSeR. Memory usage comparison (the self-rewarding MSE loss requires storing the log-probability at one additional token position). Whether the self-rewarding loss can be computed efficiently in distributed training setups where the log-probability tensors are partitioned across devices.

Genuine Weaknesses in the Experimental Design

Single RL algorithm (GRPO). The paper "primarily investigate[s] the effectiveness of applying our method within GRPO, while leaving the exploration on other RL algorithms in the future work" (Section 4.1). Since the self-rewarding derivation (Equations 9–14) is independent of the specific RL algorithm, it should work with PPO, REINFORCE, RLOO, or any other policy gradient method. Demonstrating this would substantially strengthen the generality claim, especially since different algorithms handle advantage estimation differently and may interact differently with the self-rewarding advantage integration.

No confidence intervals or significance tests. All results in Table 1 and Figure 2 are point estimates. For the reasoning improvements (+0.5 to +2.0 points average), it is impossible to determine whether these are real effects or within sampling noise. The self-verification F1 improvements are large enough to be clearly significant, but small reasoning differences could disappear with proper uncertainty quantification. The 500-question MATH500 test set provides some statistical power, but the smaller competition sets (AIME24 with 30 problems, AIME25 with 30 problems) are inherently noisy.

Limited inference-time scaling range. Figure 2 only goes to K=32. The value of verification—whether external or self-generated—typically shines at much larger K where the probability of sampling a correct answer increases but the challenge of identifying it among many incorrect answers grows. Evaluating at K=2, 4, 8, 16, 32 is a reasonable start, but the absence of K=64, 128, 256 leaves the scaling behavior of self-rewarding unexplored. The octothinker-3B models cannot handle AIME problems at all (not reported), so the inference-time scaling evaluation is restricted to benchmarks where the base accuracy is already moderate to high—the harder benchmarks where verification matters most are excluded.

Training data contamination risk. DeepMath-103K is used for training, and evaluation is on standard math benchmarks (MATH500, AMC, AIME, OlympiadBench). The paper states DeepMath-103K is "decontaminated" (from He et al., 2025) but does not describe the decontamination procedure or report overlap statistics. If DeepMath-103K contains problems similar to the test benchmarks, the reasoning improvements may partially reflect memorization rather than generalizable reasoning skill—a concern that applies equally to the GRPO baseline but is not addressed.

Missing comparison: self-rewarding vs. prompted self-verification at equal inference cost. The paper argues that LaSeR is more efficient than prior self-verification methods that require separate generation passes, but never directly compares LaSeR's self-rewarding accuracy against prompted self-verification (using the template in Appendix I) on the same LaSeR-trained model. It is possible that the LaSeR-trained model, when explicitly prompted to verify its solutions, would achieve even higher F1 than the implicit self-rewarding score—and the cost of that prompted verification, while higher, might be worth it in some applications. The prompted self-verification baseline is only evaluated on the GRPO model, not on LaSeR models.

LM-evaluation protocol could favor LaSeR. The self-verification F1 for LaSeR models is computed from the self-rewarding score (r_s > 0.5 threshold), while for GRPO and Base models it is computed from prompted self-verification outputs. This means the evaluation protocol is not identical across methods—LaSeR's self-verification is evaluated using its trained capability (the self-rewarding score), while baselines are evaluated using a prompted capability they were not specifically trained for. A fairer comparison might evaluate the GRPO model's implicit self-rewarding score (using the same special token and β_v) without any self-rewarding training—this would isolate how much of the F1 improvement comes from the training signal vs. the evaluation protocol.

The 72B external verifier comparison (Table 2) uses different base model families. The 72B verifier (Qwen2.5-Math-RM-72B) is based on the Qwen2.5-Math architecture, which is specifically designed for mathematical reasoning and may have inherent advantages over the general-purpose Qwen2.5-7B-Base used for LaSeR. The fact that a 7B general-purpose model's self-rewarding matches a 72B math-specialized reward model is noteworthy, but the comparison is not controlled for training data or architecture specialization.

Missing Experiments That Would Strengthen the Paper

  • Ablation on the special token choice: comparing unused special tokens vs. common tokens like "Yes" or "Correct" to empirically validate the claim that unused tokens are necessary for the partition function approximation.
  • Ablation on β_v values below, at, and above 0.1: the paper argues that β_v = 0.05 would cause interference—showing this empirically would validate the theoretical argument about probability mass competition.
  • Scaling to larger K in inference-time evaluation: K = 64, 128, 256 to see if self-rewarding's advantage grows, plateaus, or degrades (as would be expected if the self-rewarding scores are over-optimized, analogous to PRM over-optimization in Snell et al., 2024).
  • Cross-domain self-rewarding transfer: train LaSeR on math, evaluate self-verification F1 on code or science benchmarks without additional training.
  • LaSeR with other RL algorithms: PPO or REINFORCE to demonstrate the method is not GRPO-specific.
  • Calibration analysis: reliability diagrams showing whether the self-rewarding score r_s corresponds to empirical accuracy (e.g., among solutions with r_s in [0.7, 0.8], what fraction are actually correct?). The weighted majority voting uses the scores as weights, but this only works well if the scores are calibrated.
  • Latency and throughput measurements: actual wall-clock timing of LaSeR inference vs. standard generation and vs. prompted self-verification, to substantiate the "minimal extra cost" claim.

6. Limitations and Trade-offs

1. Difficulty Estimation Cost Is Dominant and Completely Unaccounted For

The assumption or constraint. The paper explicitly states that the compute-optimal framework relies on estimating prompt difficulty before deciding how to allocate the inference budget. The method used—generating 2048 samples per question and averaging either ground-truth correctness (oracle) or PRM final-answer scores (predicted)—is exceptionally expensive. The authors acknowledge this directly in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

The consequence. The reported 4× efficiency gains over best-of-N are computed after difficulty is known, without amortizing the cost of learning it. In a realistic deployment, the total cost would be difficulty estimation (2048 generations + scoring per problem) plus strategy execution (the budget being optimized). Since the difficulty estimation step alone consumes more compute than even the largest test-time budgets studied (256–512 generations), the true total cost picture is radically different from what Figures 4 and 8 present. For example, if a user needs to solve a single problem and the system first spends 2048 generations estimating difficulty, then applies the "compute-optimal" 64-generation strategy, the total cost is 2112 generations—which is far worse than simply running best-of-256 from the start (256 generations). The efficiency claim only holds in amortized settings where difficulty is estimated once and reused across many queries from the same distribution, but the paper provides no analysis of how many queries are needed to break even.

What evidence exists in the paper. The difficulty estimation procedure is described in Section 3.2, and Figures 11-12 (Appendix C) confirm that the predicted difficulty bins closely track oracle bins. Table 4 and Figure 10 show that difficulty-dependent trends are robust. However, no experiment measures total cost including difficulty estimation, no break-even analysis is provided, and no sweep over the number of difficulty-estimation samples (e.g., 32, 128, 512, 2048) is conducted to see whether cheaper estimation suffices.

Mitigation status. The authors flag this as "a key avenue for future work" (Section 3.2) and suggest training a model to predict difficulty directly from question text, or using adaptive estimation where initial samples inform budget allocation. Neither approach is developed or evaluated. The limitation is transparently acknowledged but completely unresolved.

2. Verifier Over-Optimization Is a Hard Ceiling, Not a Solved Problem

The assumption or constraint. The paper documents that beam search degrades easy-problem performance at high budgets (Figure 3, right: bin 1 accuracy decreases from ~78% to ~77% as budget grows from 4 to 256), lookahead search—the strongest optimizer—paradoxically performs worst overall (Figure 3, left), and qualitative examples in Appendix M show search producing degenerate outputs (repetitive low-information steps, overly short 1-2 step solutions) that score highly under the PRM. The compute-optimal policy mitigates this by routing easy problems away from aggressive search and toward best-of-N, but it does not solve the underlying problem.

The consequence. On medium-difficulty problems where beam search is deployed, over-optimization still limits the scaling ceiling. The beam search curves in Figure 3 flatten and sometimes decline well before the maximum budget of 256 generations is exhausted. This means the compute-optimal approach is fundamentally bounded by verifier quality, and the specific difficulty thresholds (when to switch from best-of-N to beam search, at what budget) are artifacts of the particular PRM quality achieved with Monte Carlo rollout training (Appendix D). If a stronger or weaker PRM were used, the optimal policy would shift—but the paper provides no guidance on how verifier quality maps to allocation strategy. A practitioner training their own PRM on a different model family with different data would not know whether to expect the same difficulty-dependent patterns.

What evidence exists in the paper. Figure 3 (right) is the clearest evidence: beam search hurts easy problems at high budgets. Appendix M provides qualitative examples of degenerate outputs. The authors note in Section 5.3 that "over-optimization of the PRM" is the likely cause and cite failure modes including "low-information repetitive steps at the end of solutions" and "overly short 1–2 step solutions." However, no experiment varies PRM quality (e.g., PRMs trained with different amounts of data, different architectures, or ensembling) to show how the over-optimization threshold shifts.

Mitigation status. The paper does not propose any improvement to PRM robustness. The compute-optimal policy avoids the worst over-optimization by using best-of-N on easy problems, but this is a routing strategy, not a solution. The authors acknowledge this implicitly in Section 8 by noting that "improving verifier robustness" is a key bottleneck for further scaling. No mitigation is tested.

3. Hard Problems Remain Essentially Unsolved—Test-Time Compute Cannot Compensate for Fundamental Capability Gaps

The assumption or constraint. Across all methods—search, revisions, and their compute-optimal combinations—the hardest questions (difficulty bin 5) show near-zero improvement regardless of compute budget. The paper is transparent about this:

"On the hardest questions (bin 5), no method makes meaningful progress—the base model simply lacks the capability to produce correct solutions regardless of how the budget is allocated" (Section 5.3)

The consequence. Test-time compute can amplify existing capability but cannot create it from nothing. If the base model's pass@1 is near zero on a problem class, no amount of search or revision will help because there are no correct solutions in the proposal distribution to find or refine. This means the FLOPs-matched advantage of test-time compute over pretraining (Section 7, Figure 9) is strictly conditional on the problem being within the base model's capability range. On bin 5 problems, the ~14× larger model consistently outperforms even the best test-time compute strategies, often by large margins (e.g., -52.9% relative disadvantage for PRM search at R >> 1, bottom-right bar chart in Figure 1). A practitioner cannot use test-time compute to reach performance levels that require qualitatively new capabilities—those must come from pretraining. The paper's positive results on easy-to-medium problems should not be misinterpreted as a general prescription that test-time compute can replace pretraining across the board.

What evidence exists in the paper. Figure 3 (right) shows bin 5 accuracy at 1-3% for all methods and all budgets. Figure 7 (right) shows bin 5 at roughly 2-3% accuracy irrespective of sequential-to-parallel ratio. Figure 9 shows the bin 5 scaling line essentially flat near 0-5%, consistently below the star markers representing the larger model's greedy performance. Table 1 confirms that even the best methods achieve negligible accuracy on AIME problems (1-3% on AIME24-25 for OctoThinker-based models, leading the authors to exclude these results).

Mitigation status. None. The paper acknowledges this as a fundamental boundary and does not propose solutions. The finding is presented as a characterization of when test-time compute works rather than a problem to be solved. This is honest but means the method offers no path forward for genuinely novel or out-of-distribution reasoning.

4. Revisions and Search Are Studied Independently, Not Combined—Performance Represents a Lower Bound

The assumption or constraint. The paper studies two complementary axes—PRM search and iterative revisions—but never combines them. Section 8 explicitly acknowledges this:

"we did not experiment with PRM tree-search techniques in combination with revisions"

The consequence. The two mechanisms have complementary, difficulty-dependent strengths: revisions improve the proposal distribution (generating better candidates through local refinement), while PRM search improves candidate selection (finding the best among generated candidates through global exploration). Revisions work best on easy problems; search works best on medium problems. A combined system—using the revision model as the proposal distribution within beam search, or using the PRM to guide which revisions to pursue—could yield gains beyond either method alone. The current results therefore represent a lower bound on what a fully integrated system could achieve. The headline 4× efficiency gain over best-of-N might be larger, or the difficulty-dependent boundaries might shift, in a combined system. A practitioner implementing this work cannot know whether to invest effort in revisions, search, or both, because the interaction effect is unmeasured.

What evidence exists in the paper. Figure 3 documents search-only performance; Figure 6-8 document revision-only performance. The difficulty-dependent analyses show that search helps on medium problems (Figure 3, right) while revisions help on easy problems (Figure 7, right), suggesting complementarity. Beyond this suggestive pattern, the paper provides no combined analysis. The FLOPs-matched comparison (Section 7, Figure 9) evaluates revisions and search separately against the larger model.

Mitigation status. Acknowledged as future work in Section 8. Not attempted in this paper. The omission is understandable given the experimental complexity of testing all combinations, but it means the paper's central claim—that compute-optimal allocation yields 4× gains—is measured separately for two independent mechanisms and cannot be interpreted as the gain from a unified system deploying both.

5. The Revision Model Has a 38% Correct-to-Incorrect Reversion Rate, Requiring Within-Chain Selection as a Patch

The assumption or constraint. As noted in Section 6.1, approximately 38% of correct answers produced during a revision chain get "revised" back to incorrect answers in the subsequent step. This is a direct consequence of the training data construction: the model was trained only on sequences where previous answers are incorrect (followed by a correct target). When the model encounters a correct answer in context at test time, it has no training signal for what to do:

"since the model was trained only on sequences where all in-context answers are incorrect (followed by a correct target), at test time the model may encounter correct answers in its context (produced during earlier revisions) and incorrectly 'revise' them into wrong answers."

The consequence. A naive implementation that always takes the last revision in a chain would be strictly worse than taking the first revision approximately 38% of the time on correct solutions. The paper's mitigation—using majority voting or verifier-based selection across the entire chain—is effective (Figure 6 shows sequential revisions still outperform parallel sampling) but introduces a fundamental tradeoff: longer revision chains increase the probability of finding a correct answer at some point, but also increase the probability that a correct answer will later be corrupted. The optimal chain length is therefore a balance between discovery and preservation, and it likely depends on problem difficulty in ways the paper does not explore. A practitioner deploying this method must implement the within-chain selection mechanism and cannot simply take the model's final output as the answer.

What evidence exists in the paper. The 38% figure is reported in Section 6.1. The sequential-to-parallel ratio analysis (Figure 7) and the pass@1 trajectory (Figure 6, left) both implicitly account for this, since they use within-chain selection. Appendix K (Figure 16) shows that the ReST-EM revision model, trained with on-policy data, actually degrades with sequential revisions—suggesting the reversion problem can be even worse with different training procedures.

Mitigation status. The paper uses majority voting or verifier-based selection as a workaround, both of which operate post-hoc across the chain rather than preventing reversion. A more principled solution—such as training the model with examples where the correct answer should be preserved, or adding a "no change needed" token to the revision vocabulary—is not explored. The authors do not discuss this as a limitation requiring future work, which is a notable omission given its practical severity.

6. Single Benchmark (MATH), Single Model Family (PaLM 2-S*)—Generality Is Unproven

The assumption or constraint. All experiments use the MATH benchmark (500 test questions) with PaLM 2-S* as the base model. The authors state they "believe this model is representative of the capabilities of many contemporary LLMs" (Section 4), but this claim is unverified. The MATH benchmark consists exclusively of competition-level math problems requiring symbolic reasoning with clean, extractable answers.

The consequence. Several aspects of the findings could be model-specific or task-specific. The PRM's quality and over-optimization behavior depend on PaLM 2-S*'s output distribution—a model with different calibration properties or error patterns might exhibit different difficulty-dependent scaling curves. The revision model's ability to learn from incorrect in-context examples depends on the base model's in-context learning capabilities, which vary substantially across model families. Most critically, the entire framework depends on having a clean correctness signal (exact answer matching) for PRM training, difficulty estimation, and evaluation. Many important real-world applications—open-ended generation, dialogue, creative writing, complex multi-step planning, code generation where multiple correct implementations exist—lack such binary correctness signals. The paper does not explore whether difficulty estimation or PRM training can work with fuzzy or learned verifiers, which is the regime most deployments would face.

What evidence exists in the paper. The paper itself is the evidence for how narrow the evaluation is: one benchmark, one model family, one task type. The test set of 500 questions is split into five difficulty quintiles of ~100 each, further split by two-fold cross-validation, meaning the compute-optimal policy is selected based on ~50 questions per fold per bin. This is a small sample, and the absence of any out-of-domain evaluation means the generalizability of the difficulty-dependent patterns is completely unknown.

Mitigation status. The authors do not claim generality beyond MATH and PaLM 2-S*. The limitation is implicit in the experimental design rather than explicitly discussed. No future work is proposed for multi-domain evaluation, which is a significant gap given how central the difficulty-dependent findings are to the paper's contribution.

7. Implications and Future Directions

How This Work Changes the Landscape

LaSeR introduces a conceptual reframing of how verification signals can be extracted from LLMs — not as explicit generated outputs requiring separate inference passes, but as latent information already present in the model's next-token probability distribution at a carefully chosen probe point. This is not a paradigm shift in the sense of overturning RLVR or self-verification research, but it is a mechanism-level insight that substantially changes the cost-benefit calculus for deploying self-verification in practice.

The magnitude of this reframing becomes clear when comparing against the two dominant prior approaches. External verifiers (Lightman et al., 2023; Snell et al., 2024; Zhang et al., 2024) require training a separate model — often as large or larger than the generator — and running it alongside the generator at inference time, effectively doubling the system's footprint. Prompted self-verification (Sareen et al., 2025; Liu et al., 2025a; Zha et al., 2025) unifies generator and verifier in a single model but still requires generating a second full response, doubling inference latency. LaSeR demonstrates that neither a separate model nor a separate generation is necessary: the verification signal exists in the probability the model assigns to an unused token at the final position, and training the model to calibrate that probability against ground-truth rewards requires only a simple MSE loss added to the existing RLVR objective. The efficiency difference is qualitative — LaSeR moves self-verification from an O(T) inference cost (where T is verification length, often comparable to solution length) to O(1), with at most one additional token of inference.

This reframing also resolves a tension in the literature that the paper identifies but does not fully articulate. Prior self-verification work showed that models can learn to verify their own outputs, but at the cost of substantially increased training and inference compute — making it unclear whether the capability was worth the overhead. External verifiers showed strong verification performance but introduced distribution-shift problems as the generator's output distribution evolved during RL training. LaSeR demonstrates that these are not inherent tradeoffs but artifacts of the mechanism used to extract verification — if verification is read from a probability distribution rather than generated as text, the cost disappears and the distribution-shift problem is solved by construction (the policy model is the verifier, so they evolve together). This does not make external verifiers or prompted self-verification obsolete — there may be domains where explicit verification reasoning chains provide benefits that latent probability signals cannot capture — but it establishes that for the specific task of binary correctness assessment, the expensive approaches are optional rather than necessary.

The paper also shifts attention toward a new axis of model capability: metacognitive calibration through token probability modulation. Prior work on model confidence primarily focused on verbalized uncertainty (the model saying "I'm not sure" or assigning a probability in text) or on output-level aggregation (majority voting, self-consistency). LaSeR demonstrates that a model can learn to encode an assessment of its own solution quality in the probability of a token it never actually generates — a side-channel signal that exists purely for downstream consumption. This opens a new design space: what other signals can be encoded in the probability distribution over unused tokens? Could a model encode a difficulty estimate, a step-by-step confidence score, or an uncertainty decomposition (aleatoric vs. epistemic) through different probe tokens? The paper's theoretical framework — that the KL-constrained RL objective's closed form becomes practically usable when the probe token's reference probability is near-zero — provides a principled basis for exploring this space.

Finally, the paper's negative result on general reasoning transfer (Section 5.2, Figure 3) serves an important landscape-shaping function. The self-rewarding capability that works remarkably well on math (~80% F1) degrades substantially on MMLU-Pro and GPQA-Diamond, where the self-rewarding score distributions for correct and incorrect solutions overlap heavily. The paper attributes this to weaker base reasoning capability and a noisier model-based verifier. This establishes a boundary condition: LaSeR's self-rewarding is only as good as the reward signal used to train it, and only as informative as the model's underlying reasoning capability allows. This prevents over-claiming and directs future work toward understanding the relationship between base capability, verifier quality, and achievable self-rewarding accuracy — a relationship the paper has begun to map but not fully characterized.

In terms of research direction attractiveness: training better external verifiers becomes somewhat less urgent for domains where LaSeR-style self-rewarding works well, since a model's own calibrated probabilities can match or exceed dedicated verifiers (Table 2 shows a 7B self-rewarding model matching a 72B external reward model). Prompted self-verification research becomes narrower in scope — the efficiency argument for reading verification from probabilities rather than generating it is strong, and future work on explicit self-verification generation would need to demonstrate benefits (e.g., interpretability, chain-of-thought verification reasoning) that justify the additional cost. Implicit reward research gains new practical relevance, since LaSeR demonstrates that the closed-form relationship between optimal policies and rewards can be directly exploited for training when the action space is collapsed to a single token with negligible reference probability.

Follow-Up Research This Work Enables

Characterizing the relationship between verifier quality, base model capability, and achievable self-rewarding accuracy. The paper's general reasoning experiment (Section 5.2) reveals that self-rewarding accuracy degrades when the training verifier is noisy and base reasoning is weak, but provides only a single data point. A systematic study would train LaSeR on the same math dataset using verifiers with controlled noise levels (e.g., oracle verifier, 90%-accurate model-based verifier, 80%-accurate, etc.) and measure the resulting self-verification F1 as a function of both verifier quality and the base model's Pass@1 on the training data. This would produce a phase diagram showing where LaSeR's self-rewarding provides useful signals and where it fails — critical information for practitioners deciding whether to adopt the method for a new domain.

Can self-rewarding scores be used for adaptive compute allocation during RL training? The paper demonstrates that self-rewarding scores can augment verifier-based advantages (Equation 16) and improve inference-time solution selection (Figure 2), but these uses are coarse: a single weight τ = 0.1 in the advantage, and weighted voting across a fixed number of samples. A more aggressive use would be to allocate training compute adaptively: for each prompt, use the self-rewarding scores of initial samples to decide how many more samples to generate, or whether to stop early and move on. This connects directly to the compute-optimal test-time scaling framework (Snell et al., 2024) but applied during training — essentially, using self-rewarding as a difficulty estimator to decide per-prompt training budgets. The experiment would compare a LaSeR training run with uniform per-prompt sampling against one where the number of rollout samples per prompt is dynamically adjusted based on the variance or mean of initial self-rewarding scores.

Exploring multi-token self-rewarding for stronger verification at controlled cost. Section 5.3 sketches the idea of computing the self-rewarding score over M tokens rather than one: r_s = β_v ∑_{m=1}^M log π_θ(z_c | x, y, z_c, …, z_c) − Mβ_v c_ref. This is proposed as a way to trade increased inference cost (M additional token inferences instead of 1) for potentially stronger self-rewarding capability. The natural experiment is to sweep M from 1 to, say, 16 on a math reasoning benchmark and measure self-verification F1 and weighted majority voting accuracy at various K. Two hypotheses to test: (1) multi-token self-rewarding reduces variance in the score, making it more discriminative at moderate K; (2) there are diminishing returns, and the optimal M is small (2-4) because the additional tokens provide redundant information. The theoretical question is whether conditioning on previous z_c tokens changes the partition function analysis — does Z(x, y, z_c, …, z_c) remain approximately 1 for each additional token, or does the reference probability shift?

Combining LaSeR's self-rewarding with explicit verification generation for hybrid verification. The paper establishes that latent self-rewarding is extremely efficient but has a ceiling determined by base model capability and verifier quality. Explicit verification generation (chain-of-thought verification) is expensive but may capture reasoning about correctness that a single-token probability cannot. A hybrid system could use the self-rewarding score as a fast filter: if r_s is strongly above or below 0.5 (e.g., > 0.9 or < 0.1), trust the latent signal; if r_s is in an ambiguous middle range (e.g., 0.3-0.7), trigger an explicit verification generation to resolve the uncertainty. The experiment would measure the tradeoff between verification accuracy and average inference cost as a function of the ambiguity threshold, compared against pure self-rewarding, pure explicit verification, and external verifier baselines. The paper's finding that self-rewarding F1 is ~80% on math but lower on general reasoning (Figure 3) suggests this hybrid approach would be most valuable in domains where latent self-rewarding is moderately but not perfectly reliable.

Does LaSeR's self-rewarding training improve the model's ability to generate explicit verifications? The paper evaluates self-rewarding through the probability of z_c and prompted self-verification separately, but does not test whether LaSeR training transfers to the prompted verification task. If training a model to encode correctness in π_θ(z_c|x,y) also improves its ability to generate accurate verification text when explicitly prompted, this would suggest that the MSE loss is teaching a general metacognitive capability rather than just calibrating a specific probability channel. The experiment would take LaSeR-trained models and evaluate their prompted self-verification F1 (using the Appendix I template) against GRPO baselines, controlling for the reasoning accuracy difference. A positive result would strengthen the claim that LaSeR teaches genuine self-assessment; a null result would indicate the capability is channel-specific.

Stress-testing the partition function approximation: how small must π_ref(z_c|x,y) be for Z(x, y) ≈ 1 to hold in practice? The paper's theoretical derivation (Equation 10) shows that Z(x, y) ≈ 1 when π_ref(z_c|x,y) · exp(1/β_v) ≈ 0. Figure 5 shows that for unused special tokens, π_ref ≈ 10^{-10}, making the approximation extremely tight. But the paper does not experimentally determine the threshold at which the approximation breaks down. An experiment would sweep over tokens with systematically varying reference probabilities (e.g., tokens with π_ref from 10^{-5} to 10^{-15}) and measure whether the self-rewarding MSE loss can be optimized and whether the resulting scores correlate with true correctness. The hypothesis is that when π_ref(z_c|x,y) · exp(1/β_v) exceeds some threshold (perhaps 10^{-2}), the partition function bias becomes large enough that the MSE loss targets a systematically wrong value, preventing convergence. This would establish practical guidelines for token selection beyond "use an unused special token."

Practical Applications and Downstream Use Cases

Cost-effective inference-time scaling for math reasoning systems. The most direct application is in deployed math reasoning systems — tutoring platforms, automated grading, competition training tools — where the model must generate and self-evaluate solutions without ground-truth answers. LaSeR enables weighted majority voting with self-rewarding scores at essentially the same inference cost as unweighted majority voting (one additional token per solution), while providing +1–3 percentage points of accuracy at K=32 (Figure 2). For a system processing thousands of problems, this translates to meaningfully higher accuracy at zero additional infrastructure cost — no separate verifier model to deploy, maintain, or scale. The paper's external verifier comparison (Table 2) shows that a 7B LaSeR model's self-rewarding F1 (~80%) matches a 72B reward model, meaning the system can achieve 72B-quality verification using only the 7B generator.

Self-improvement pipelines with built-in data quality filtering. In automated self-improvement loops (e.g., STaR, ReST^EM, rejection sampling fine-tuning), the model generates solutions to training problems, and high-quality solutions are selected for subsequent fine-tuning. The standard approach uses the ground-truth verifier during data generation — but this limits the method to domains with clean correctness signals. LaSeR provides a way to filter generated solutions by quality without ground-truth answers, using the self-rewarding score as a proxy for correctness. Even when a verifier is available, the self-rewarding score can augment it: solutions that pass the verifier but have low self-rewarding scores might be misformatted correct answers (false negatives from the verifier), while solutions with high self-rewarding scores but failing the verifier might be near-misses worth preserving. The paper's class-level re-weighting (Equation 15) and balanced self-verification F1 (Table 1) indicate that self-rewarding is not trivially biased toward the majority class, making it a credible filtering signal.

Confidence estimation for selective deployment and human-in-the-loop routing. The self-rewarding score r_s provides a continuous confidence estimate (values in [0, 1]) for each generated solution. In production systems where some queries are routed to human reviewers when the model is uncertain, r_s can serve as the routing criterion: if max_i r_s^i < 0.5 across all K sampled solutions, escalate to a human or a larger model. This is more principled than heuristic confidence measures (e.g., token-level entropy, which conflates uncertainty about the answer with uncertainty about phrasing) and requires no additional model. The paper does not directly evaluate calibration (the correspondence between r_s values and empirical accuracy), but the self-verification F1 scores of ~80% suggest reasonable discrimination. A deployment would need to calibrate the threshold on a validation set, but the mechanism is ready-to-use.

Lightweight verifier for on-device or edge deployment. For applications where a separate large verifier model is infeasible — on-device LLMs, mobile assistants, embedded systems — LaSeR's self-rewarding provides verifier-quality signals (matching 72B external models in Table 2) within the same model that generates solutions. This eliminates the need to transmit candidate solutions to a cloud-based verifier (privacy and latency benefits) or to run a separate verifier model on-device (memory and compute benefits). The paper's experiments on OctoThinker-3B-Short-Base demonstrate that even a 3B model can achieve 72.5% self-verification F1, outperforming 7B external verifiers on its own outputs (Table 2). For a 3B model deployed on a phone or laptop, LaSeR enables best-of-N selection with verification at a cost of K × (T+1) tokens rather than K × T (no verification) or K × (T + T_verify) (external verifier).