Does Your Reasoning Model Implicitly Know When to Stop Thinking?

ArXiv: 2602.08354

🎯 Pitch

Surprisingly, large reasoning models implicitly know when to stop generating reasoning chains, but this self-awareness is hidden by standard sampling. The authors release it with SAGE, a confidence-guided early-stopping strategy that slashes token usage (e.g., from 3,871 to 2,141 tokens on MATH-500) while jointly boosting accuracy through reinforcement learning.

---

1. Executive Summary

This paper studies how large reasoning models (LRMs) can be made to generate shorter, more accurate chains of thought by exploiting their implicit ability to recognize when to stop thinking—a capability the authors argue is obscured by standard sampling paradigms. Using DeepSeek-distilled Qwen models and Qwen3-8B on MATH-500, AIME, AMC23, OlympiadBench, and Minerva, the work introduces SAGE (Self-Aware Guided Efficient Reasoning) , a step-wise decoding strategy that terminates reasoning chains when the model assigns high cumulative confidence (Φ) to an end-of-response token (operating as a confidence-guided early-stopping mechanism layered over beam search), and SAGE-RL, a hybrid rollout scheme that mixes SAGE-generated trajectories into group-based reinforcement learning objectives like GRPO and GSPO. SAGE-RL-tuned models achieve simultaneous improvements in accuracy and token efficiency across six benchmarks—for instance, SAGE-GRPO on DS-7B yields a 43.4 token-efficiency score on MATH-500 (an 83.1% relative gain over the base model) while reducing average response length from 3,871 to 2,141 tokens—establishing that LRMs can learn compact reasoning patterns through RL without explicit length penalties, only when the rollout phase exposes them to confidence-pruned efficient trajectories alongside standard random samples.

2. Context and Motivation

The Core Problem: LRMs Think Too Long Without Good Reason

The reasoning revolution in large language models—exemplified by OpenAI's o1/o3, DeepSeek-R1, and their successors—has been powered by a simple mechanism: let the model "think" for longer by generating extended chains of thought (CoTs) before producing an answer. Reinforcement learning from verifiable rewards (RLVR), using algorithms like GRPO and GSPO, trains models to produce these long reasoning traces by rewarding them with binary 0/1 signals based solely on whether the final answer is correct. Since the model faces no penalty for verbosity, it learns a conservative strategy: keep generating tokens, exploring every possible angle, backtracking, and re-verifying, because any additional reasoning step might catch an error and turn a wrong answer into a correct one. The training objective says nothing about efficiency—only correctness matters.

This has produced models with genuinely impressive reasoning capabilities. DeepSeek-R1, QwQ-32B, and Qwen3 can solve competition-level mathematics problems that were out of reach for pre-reasoning-era LLMs. However, the success has come with an uncomfortable side effect that the paper identifies as the central problem: these models produce enormous amounts of redundant tokens that contribute nothing to correctness. The paper terms this the "overthinking" phenomenon.

The scale of the redundancy is striking. The paper cites concrete evidence from the recent literature:

• Balachandran et al. (2025) observe that DeepSeek-R1 produces responses nearly 5× longer than Claude 3.7 Sonnet on AIME 2025 while achieving comparable accuracy. The extra length is pure waste—it doesn't convert into better answers.

• Hassid et al. (2025) show that on the same benchmarks (AIME and HMMT), the shortest responses from QwQ-32B actually outperform randomly sampled ones by 2 percentage points, using 31% fewer tokens. This is a critical observation: it means that within the model's own distribution of generated solutions, the concise ones are often better—but they aren't the ones the model tends to produce.

• Shrivastava et al. (2025) sharpen this finding further: on AIME 2025, in 72% of problems where the model generated both correct and incorrect answers, the longer response was more likely to be incorrect than the shorter one.

These findings collectively establish a pattern that challenges the core assumption behind test-time scaling: once chain-of-thought length crosses some threshold, further length inflation is uncorrelated with—or even inversely related to—correctness. The model is generating tokens that don't help, and in many cases, the model's own best work is hiding inside relatively compact reasoning chains that it fails to select.

Why This Problem Matters: Efficiency Is Not a Luxury

The overthinking problem has practical consequences that make it more than an academic curiosity:

Computational cost. Long CoTs consume enormous FLOPs at inference time. For real-time applications—chatbots, coding assistants, interactive tutoring systems—every unnecessary token adds latency that degrades user experience. For batch processing—evaluating models on benchmarks, generating training data, running self-improvement pipelines—every unnecessary token costs money. The paper quantifies this in terms of token efficiency, defined as pass@1 / response length, which directly measures how much correctness you get per token spent.

The length-accuracy tradeoff is not a fixed curve. Current RLVR-trained models have internalized the lesson that "more tokens = safer," but the evidence cited by the paper shows this is a learned behavior, not an inherent limitation. The model can produce correct answers with far fewer tokens; it just doesn't know to stop. This implies the problem is solvable through better inference strategies or improved training—it's not a fundamental capacity ceiling.

Self-improvement pipelines need clean data. When LRMs generate solutions that are used to fine-tune themselves (as in STaR, ReST$^{EM}$, or rejection sampling), the quality and efficiency of those solutions matters doubly. Redundant reasoning traces don't just waste inference compute—they contaminate the training data with verbose, meandering patterns that the next generation of the model will learn to imitate.

The deployment bottleneck. The paper's findings in Section 7 (comparing FLOPs-matched test-time compute vs. pretraining in the prior-exec-summary-cited analysis) establish that, for problems within a model's capability range, smarter inference can substitute for larger models. But this substitution only works if the inference is actually efficient. A model that produces 4× the necessary tokens isn't realizing the potential gains that the compute-optimal framework promises.

The Gap: Where Prior Approaches Fall Short

The paper identifies three broad families of prior work on efficient reasoning, each with specific limitations that motivate the SAGE approach:

Training-Free Methods: Surface-Level Fixes

These approaches attempt to improve reasoning efficiency without modifying model weights—through prompt engineering (telling the model to "be concise"), early-exit mechanisms (stopping generation when confidence is high), or post-hoc pruning of already-generated chains. Examples include Chain-of-Draft (Xu et al., 2025a), which instructs models to write minimal working notes; TokenSkip (Xia et al., 2025), which trains models to produce "skip" tokens that compress reasoning; and various best-of-N pruning strategies that select the shortest correct answer from a pool of samples (Xie et al., 2023; Liao et al., 2025).

The paper argues these methods share a fundamental weakness: they cannot fundamentally resolve the issue of redundant reasoning because they don't change the model's underlying tendency to overthink. Prompt engineering approaches are "heavily contingent upon the model's instruction-following capability" (Section A.2), and in practice "the observed improvements in experiments are typically modest or insignificant." You can tell a model to be concise, but if its training has taught it that verbosity is safety, the instruction won't override deep-learned behavior.

Post-hoc selection methods (like best-of-N with length filtering) exploit the fact that shorter correct answers exist in the distribution, but they're wasteful: you have to generate N long chains to find the one short correct one. The paper's core insight is that this isn't a selection problem—it's a generation problem. The model should produce the short correct chain directly, not require wasteful oversampling to find it.

Offline Training Methods: The Data Scarcity Problem

These methods fine-tune models on curated datasets of concise chains of thought, often using distillation from a larger teacher model to produce the training data. Examples include supervised fine-tuning with variable-length CoT data (Yu et al., 2024; Kang et al., 2025), approaches that insert "compression" tokens during data generation (Qiao et al., 2025; ConCISE), and methods that train models to recognize when they can skip reasoning steps (Liu et al., 2024).

The paper identifies two critical weaknesses:

First, obtaining high-quality short chains of thought is genuinely difficult. You can't simply truncate long chains—that produces incoherent partial solutions. You need reasoning chains that are both complete (reaching the correct answer) and compact (without unnecessary steps), and generating such data at scale requires expensive human annotation or careful automated construction.

Second, and more fundamentally, offline training limits exploration on difficult problems. The paper states this explicitly: "the offline training paradigm tends to limit the model's exploration ability on difficult problems" (Section A.2). If you distil a teacher's short solutions into a student model, the student learns to imitate those specific efficient reasoning patterns—but it doesn't learn how to discover efficient reasoning patterns for novel problems. For hard problems where the correct reasoning path isn't obvious, this limitation can hurt accuracy.

The paper considers but rejects the straightforward approach of simply distilling SAGE's outputs into the model via SFT: "since distillation depends on a strong teacher model, we are concerned that self-distillation will limit the upper boundary of the model's reasoning capability."

Online Training Methods with Length Penalties: The Reward Hacking Trap

The most directly competitive approaches use reinforcement learning with explicit length penalties in the reward function. Examples include methods that subtract a token-count penalty from the correctness reward (Kimi Team, 2025a; Yeo et al., 2025; Cheng et al., 2025), approaches that optimize for the best accuracy under a fixed token budget (Yi et al., 2025; Hou et al., 2025; Qi et al., 2025), and sophisticated schemes that truncate reasoning steps and evaluate sub-chain rewards (Dai et al., 2025; Yue et al., 2025).

The paper identifies a fundamental tension in all of these: "explicit or implicit integration of length compression into the optimization objective may impair the model's reasoning capabilities" (Section A.2). The RL training process receives two competing signals—"be correct" and "be short"—and navigating this tradeoff requires careful reward engineering that can easily lead to training instability or reward hacking.

Consider the difficulty-aware methods like AdaptThink (Zhang et al., 2025) or GRPO-LEAD, which assign different length budgets based on problem difficulty. These work by telling the model "this is an easy problem, so use fewer tokens." But the difficulty assessment itself may be imperfect, and the hard constraint on length can prevent the model from exploring solution strategies that require more steps but turn out to be correct. The paper reports that AdaptThink achieves the highest token compression on some benchmarks, but "this high level of conciseness restricts the model's ability to explore different solution strategies," leading to accuracy degradation.

Similarly, GFPO (Shrivastava et al., 2025) uses oversampling—generating many candidate solutions and selecting short correct ones for training—which the paper notes is computationally expensive compared to SAGE-RL's approach, since it requires generating many long chains whose tokens are largely wasted.

The Overarching Gap: No One Has Asked Whether the Model KNOWS When to Stop

The paper's key observation—which distinguishes it from all prior work—is that none of these methods investigate whether LRMs possess an implicit capability to recognize the right time to terminate thinking. All prior approaches impose efficiency from the outside: through instructions, through curated data, through reward penalties, or through post-hoc selection. None ask: "Does the model itself, if given appropriate exploration, naturally gravitate toward concise reasoning chains that it terminates with confidence?"

This is the gap the paper fills. The framing is important: rather than treating length compression as something to be enforced, the paper investigates whether efficient reasoning is a capability that can be unleashed—a latent potential that exists within the model's probability distribution but is hidden by current sampling paradigms.

How This Paper Positions Itself

The paper's positioning has several components that together form a coherent research narrative:

From observation to mechanism. The paper starts with the empirical puzzle identified by prior work (short responses in pass@k outperform longer ones), digs deeper with their own metric (RFCS—the Ratio of the First Correct Step, showing that correct answers typically appear early in reasoning chains even though the model continues generating), and then asks the mechanistic question: why does the model continue past the point where the correct answer is already determined?

The implicit knowledge hypothesis. The paper's central claim is that LRMs implicitly know when to stop thinking. This knowledge is encoded in the model's probability distribution—when a correct, complete reasoning chain is being generated, the cumulative log-probability (Φ) of that path is high, and the end-of-response token has high relative probability within the candidate set. However, standard sampling strategies (random sampling, greedy decoding) fail to select these paths because they don't aggregate confidence signals across the full reasoning chain—they make local, token-by-token decisions based on next-token probability (φ) alone. This is the key mechanistic insight: next-token probability and cumulative-path probability can diverge, and current sampling only sees the former.

A training-free method that enables training-based learning. SAGE itself is a training-free decoding strategy—it requires no model fine-tuning. But the paper doesn't stop there. The key design choice is to use SAGE not as a standalone inference method but as a rollout strategy within reinforcement learning. By mixing SAGE-generated trajectories into the RL training process, SAGE-RL allows the model to learn efficient reasoning patterns from its own best behavior, without needing explicit length penalties, curated data, or any modification to the training objective. The RL objective remains purely correctness-based; efficiency is learned because the SAGE rollouts provide positive examples of correct-but-concise reasoning that the policy model learns to imitate through the advantage estimation mechanism.

Contrast with prior RL-based approaches. The paper explicitly contrasts SAGE-RL with methods that add length penalties to the reward: "instead of modifying the optimization objective, we optimize the sampling process to enable the policy model to directly learn the efficient reasoning chains uncovered by SAGE via the advantage estimation of RLVR." This distinction matters because it avoids the reward hacking and training instability that plague length-penalty methods. The RL training sees no length signal at all—it only sees correctness—but the samples it learns from are biased toward efficient reasoning chains.

Computational pragmatism. The paper also positions SAGE-RL as computationally lightweight relative to alternatives. Compared to GFPO's oversampling approach, SAGE-RL requires no additional parallel sampling—the SAGE trajectories are generated as part of the normal rollout batch. Compared to S-GRPO and VSRM's repeated rollouts for sub-chain reward estimation, SAGE-RL performs no extra rollouts. The only overhead is the SAGE decoding step itself, which the paper argues is manageable (2 out of 8 samples per group).

The scope claim. The paper is careful about its scope: it does not claim to have solved the overthinking problem. Rather, it claims to have demonstrated that (1) efficient reasoning capability exists latently in LRMs, (2) this capability can be surfaced through confidence-guided search, and (3) RL can incorporate these efficient patterns into pass@1 inference. The results show consistent but moderate accuracy gains combined with substantial efficiency improvements, not transformative capability jumps—which is appropriate for a method that changes how the model thinks rather than whether it can think.

3. Technical Approach

3.1 Reader Orientation

The paper builds a two-part system for producing shorter, more accurate reasoning chains from large reasoning models (LRMs): (1) a training-free decoding strategy called SAGE that performs step-wise search guided by the model's own cumulative confidence scores to find compact, correct solutions, and (2) a reinforcement learning integration called SAGE-RL that mixes these SAGE-generated efficient trajectories into the standard RLVR rollout process so the model learns to produce concise reasoning without any explicit length penalty. The core problem this system solves is that current LRMs generate substantial redundant reasoning tokens that don't improve correctness, but the model's own probability distribution contains shorter, high-confidence paths to correct answers that are systematically overlooked by standard sampling; the solution's shape is a confidence-guided search that "unleashes" this latent efficient reasoning capability, followed by RL that internalizes it into the model's default behavior.

3.2 Big-Picture Architecture (Diagram in Words)

The system has four major components connected in a pipeline that spans both inference-time decoding and training-time optimization:

1. Base Reasoning Model ($\pi_\theta$): An LRM such as DeepSeek-R1-Distill-Qwen-1.5B, DeepScaleR, Qwen3-8B, or DeepSeek-R1-Distill-Qwen-7B. This model generates reasoning steps token-by-token. It serves as the "proposal distribution" from which all reasoning paths originate, and it produces the log-probabilities that drive the confidence scoring.

2. Confidence Scoring Function ($\Phi$): The average cumulative log-probability of a reasoning path up to the current token, computed as $\Phi(\mathbf{y}_k) = \frac{1}{k} \sum_{i=1}^{k} \log \pi_\theta(y_i | \mathbf{y}_{<i}, \mathbf{x})$. This is the key metric that distinguishes SAGE from standard beam search: it evaluates whole-path confidence rather than next-token probability. It is computed on-the-fly during generation from the model's own output logits.

3. SAGE Decoding Algorithm (Sections 3–5): A step-wise reasoning chain exploration procedure that, at each reasoning step, generates $2m$ candidate continuations from each of $m$ active beams, scores them with $\Phi$, retains the top $m$, and terminates when any candidate reaches an end-of-response token. This produces $r$ complete reasoning chains per query. It operates at the granularity of full reasoning steps rather than individual tokens, and it requires no model fine-tuning.

4. SAGE-RL Integration (Section 6): A modification to the rollout phase of RLVR algorithms (GRPO, GSPO) where $r$ of the $G$ responses in each training group are generated via SAGE instead of standard random sampling. The remaining $G - r$ responses use default random sampling. The RL objective remains purely correctness-based; the SAGE rollouts provide positive examples of efficient reasoning that the policy learns to imitate through advantage estimation.

Information flows as follows: a query enters the system → the SAGE algorithm expands $m$ reasoning-step beams in parallel, scoring with $\Phi$ and retaining top candidates → when $r$ candidates reach an end-of-response token, SAGE returns $r$ completions → during RL training, these $r$ completions are mixed with $G - r$ randomly sampled completions → the standard RLVR objective (GRPO or GSPO) updates the model weights based on correctness rewards and group-relative advantages → over training, the model's pass@1 output distribution shifts toward the efficient reasoning patterns discovered by SAGE.

3.3 Roadmap for the Deep Dive

• First, the formal definitions of $\Phi$ and $\phi$ (Equations 1–2), since all of SAGE's search decisions depend on these scoring functions, and understanding their difference is essential to grasping why SAGE works.
• Second, the token-wise reasoning path exploration algorithm (TSearch, Section 3), which is the original form of SAGE operating at token granularity—this motivates the move to step-wise exploration and establishes the core principles before simplification.
• Third, the analytical experiments that validate SAGE's mechanism (Section 4), covering why $\Phi$ prevents length collapse while $\phi$ causes it, and why exploration width drives capability convergence—these are not separate experiments but rather the empirical foundation for understanding what SAGE is doing.
• Fourth, the step-wise SAGE algorithm itself (Section 5.1), which simplifies TSearch by operating at reasoning-step granularity and removing the tolerance parameter based on the insights from Section 4.
• Fifth, the SAGE-RL integration (Section 6), covering the hybrid rollout strategy, the mathematical modification to GRPO/GSPO objectives (Equations 9–15), and the training dynamics that explain why mixing SAGE trajectories works.
• Sixth, the hyperparameter analysis (Appendix D.2) and time complexity analysis (Appendix D.4), which together establish the practical tradeoffs in deploying SAGE and the reasoning behind the default settings ($m=2$, $r=2$, $G=8$).

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an algorithm design and empirical analysis paper whose core idea is that LRMs possess latent knowledge of when to terminate reasoning, and that this knowledge can be surfaced through cumulative-confidence-guided search, then internalized through reinforcement learning with hybrid rollouts.

---

Confidence Functions: $\Phi$ and $\phi$

The paper defines two related but distinct scoring functions on which all of SAGE's behavior depends. Understanding their difference is central to understanding the method.

Definitions

Given a query $\mathbf{x}$ and a prefix $\mathbf{y}_{<k} = (y_1, y_2, \ldots, y_{k-1})$ previously generated by the language model $\pi_\theta$, the next-token log-probability for token $y_i$ is:

$\phi(y_i; \mathbf{y}_{<i}) = \log \pi_\theta(y_i \mid \mathbf{y}_{<i}, \mathbf{x})$

where $\phi(y_i; \mathbf{y}_{<i})$ is the log-probability that the model assigns to the $i$-th token given the query and all preceding tokens. This is the standard per-token score available from any autoregressive language model during generation.

The average cumulative log-probability up to generation step $k$ is then:

$\Phi(\mathbf{y}_k) = \frac{1}{k} \sum_{i=1}^{k} \phi(y_i; \mathbf{y}_{<i})$

where $\mathbf{y}_k = (y_1, y_2, \ldots, y_k)$ is the sequence of $k$ generated tokens and $\Phi(\mathbf{y}_k)$ is the length-normalized total log-probability of that sequence.

What $\Phi$ computes: the arithmetic mean of the log-probabilities of every token in the prefix, producing a single scalar that estimates how confident the model is in the entire path taken so far. Because it averages rather than sums, it does not penalize longer sequences simply for being longer—a 100-token path with per-token log-probabilities of -1.0 and a 50-token path with per-token log-probabilities of -1.0 receive the same $\Phi$ score. However, a 100-token path where the early tokens have high probability (-0.5) but later tokens have low probability (-2.0) will receive a lower average than a compact path where all tokens have moderate probability (-1.0). The normalization by $k$ means $\Phi$ measures average quality per token, not total sequence quality.

Why this form: the alternative would be the un-normalized cumulative log-probability $\sum_{i=1}^{k} \phi(y_i; \mathbf{y}_{<i})$, which is the standard beam search score. The sum-length-normalized form corrects a well-known bias: un-normalized scores systematically favor shorter sequences in beam search (because each additional token adds a negative log-probability), causing beam search to produce degenerate short outputs unless length penalties are applied. By averaging, $\Phi$ makes scores comparable across different lengths. This is critical for SAGE's termination mechanism: it allows the algorithm to compare a path that reaches the end-of-response token at step 10 against a path still generating at step 10 without mechanically favoring either.

The Operational Distinction

The paper's key mechanistic insight is that $\Phi$ (whole-path confidence) and $\phi$ (next-token probability) can diverge significantly at the moment when a reasoning chain should terminate. Specifically, when a compact, correct reasoning chain reaches its natural conclusion, the model's next-token probability for the end-of-response token $\langle\text{response}\rangle$ may be low relative to other plausible continuations (the model could keep reasoning, and it assigns moderate probability to various continuations), yet the cumulative path probability $\Phi$ for that chain including the end-of-response token is high relative to alternative paths (because all the preceding reasoning steps were high-confidence). This divergence is exactly what the paper demonstrates empirically in Section 4.2.

Standard sampling strategies (greedy, random, vanilla beam search) rely on $\phi$ (next-token probability) for their decisions: greedy always picks the single highest-probability next token, beam search retains the top-m paths by cumulative log-probability (which is equivalent to $k \cdot \Phi$ but without the length normalization, hence biased), and random sampling follows $\phi$ as a categorical distribution. All of these can miss the moment when the model should stop because they evaluate local probability rather than global path quality.

---

Token-Wise Reasoning Path Exploration (TSearch)

The paper first presents TSearch, a token-level algorithm that establishes the core principles, before simplifying it to step-level SAGE in Section 5.1.

Token-Level Expansion

The algorithm maintains $m$ candidate sequences (beams) in parallel and expands them token-by-token until a maximum step budget $T_{\max}$ is reached. At each timestep $i$:

Step 1: Generate candidate next tokens. For each of the $m$ active beams $\mathbf{y}_{i-1}^{(j)}$, where $j \in [1, m]$, select the top $2m$ most probable next tokens according to the per-token score $\phi$:

$\mathcal{T}^{(j)} = \text{Top}_{2m}(\{y_i \mid y_i \in \mathcal{V}\}; \phi(\cdot; \mathbf{y}_{i-1}^{(j)}))$

where $\mathcal{V}$ is the model's vocabulary and $\text{Top}_{2m}$ ranks all candidate tokens by their $\phi$-scores (next-token log-probabilities) and returns the $2m$ highest. The factor of $2m$ (rather than $m$) is a design choice that provides a broader candidate window for end-of-response detection while still pruning aggressively.

Step 2: Form candidate sequences. From each parent beam, extend by each of its top $2m$ tokens to form $2m^2$ candidate sequences:

$\hat{Y}_i = \{\mathbf{y}_i^{(j,k)} \mid j \in [m], k \in [2m]\}$

where each candidate is constructed as $\mathbf{y}_i^{(j,k)} = \mathbf{y}_{i-1}^{(j)} \circ y_i^{(j,k)}$, with $y_i^{(j,k)}$ being the $k$-th best token from $\mathcal{T}^{(j)}$.

Step 3: Select top-m beams for next iteration. Score all $2m^2$ candidates with the cumulative confidence score $\Phi$ and retain the $m$ highest:

$Y_i = \text{Top}_m(\{\mathbf{y}_i^{(j,k)} \mid j \in [m], k \in [2m]\}; \Phi)$

What this computes operationally: at each token position, the algorithm generates $2m$ continuations per beam, scores the resulting $2m^2$ partial sequences by their average per-token log-probability so far, and keeps the $m$ most confident paths. This is beam search with three modifications: (1) scoring by length-normalized $\Phi$ rather than cumulative log-probability, (2) expanding $2m$ candidates per beam rather than the full vocabulary (a pruning step that keeps computation manageable), and (3) the termination mechanism described next.

Exploration Termination with Tolerance

The termination condition is the central innovation of TSearch. The key entity is the end-of-response token, denoted $\langle\text{response}\rangle$. The algorithm checks at each expansion step whether any candidate sequence ends with this token.

The Tolerance accept rank Ratio (TR) is defined as $h / (2m)$, where $h \in \{1, 2, \ldots, 2m\}$ is a hyperparameter representing the rank threshold. Given the required number of returned completions $r \in \{1, 2, \ldots, m\}$:

• If a candidate sequence $\mathbf{y}_i^{(j,k)}$ ends with $\langle\text{response}\rangle$ and the $\langle\text{response}\rangle$ token ranks within the top $h$ most probable tokens in $\mathcal{T}^{(j)}$ (i.e., the model is sufficiently confident at the local level that this is the right token), then this candidate is added to the completion set $\mathcal{O}$.

• If the $\langle\text{response}\rangle$ token appears but ranks below the threshold $h$, the candidate is discarded—the model's local confidence in stopping is too low.

• When $|\mathcal{O}| \geq r$, the entire process terminates, and the $r$ completions are returned.

• If $i = T_{\max}$ (the maximum step budget is exhausted) and fewer than $r$ completions have been found, the algorithm falls back to selecting the top $r - |\mathcal{O}|$ sequences from the final beam set $Y_{T_{\max}}$ by $\Phi$ score, ensuring exactly $r$ completions are always returned.

Why this form: the TR parameter provides a knob that controls the tradeoff between early termination and thorough reasoning. When $\text{TR} = 1$ (i.e., $h = 2m$), termination occurs immediately upon any $\langle\text{response}\rangle$ token appearing—the most aggressive early-stopping behavior. When $\text{TR} < 1$, termination requires that the end-of-response token rank highly within the local probability distribution, providing a safeguard against premature stopping when the model is uncertain. The paper's key empirical finding (Section 4.2) is that when TSearch uses $\Phi$ for beam selection, $\langle\text{response}\rangle$ consistently ranks first at the moment it appears, making TR effectively irrelevant—this observation is what enables the simplification to step-wise SAGE in Section 5.1.

Greedy Answer Sampling

After TSearch produces $r$ reasoning chains $\{\mathbf{t}_i\} \subset \mathcal{O}$ for a query $\mathbf{x}$, the final answers are generated greedily: $\pi_\theta(\mathbf{a}_i \mid \mathbf{x}, \mathbf{t}_i)$ where $\mathbf{t}_i$ is the reasoning chain and $\mathbf{a}_i$ is the answer extracted greedily conditioned on the chain. This means the reasoning chains are generated via the confidence-guided search, but once a chain is selected, the answer portion is generated deterministically by taking the most likely tokens under the model (temperature 0). This ensures that the final answer benefits from the reasoning chain's structure without introducing additional variance.

TSearch Variant Without $\Phi$ (Ablation)

For ablation, the paper defines TSearch w/ $\phi$, which replaces the beam selection at Step 3 with a greedy per-token criterion:

$Y_i = \{\mathbf{y}_i^{(j,k)} \mid (j,k) = \arg\text{Top}_m(\{y_i^{(j,k)} \mid j \in [m], k \in [2m]\}; \phi(\cdot; \mathbf{y}_{i-1}^{(j)})))\}$

Here, instead of scoring full candidate sequences by $\Phi$, the algorithm simply keeps the $m$ beams whose single new token has the highest $\phi$ score, regardless of the preceding path's confidence. This is equivalent to running $m$ independent greedy continuations in parallel with no cross-sequence comparison beyond the per-step token ranking.

What this ablation tests: the hypothesis that $\Phi$ (whole-path confidence) is necessary and $\phi$ (next-token confidence) is insufficient for identifying efficient reasoning chains. The results in Section 4.1 confirm this dramatically: TSearch w/ $\phi$ suffers severe performance collapse as exploration width increases, while TSearch w/ $\Phi$ improves both accuracy and conciseness.

---

Analytical Validation: Why $\Phi$ Works and $\phi$ Fails

The paper's Sections 4.1–4.3 are not separate experimental studies but rather the mechanistic validation of the TSearch design. They establish three observations that together explain why SAGE works and motivate its simplification to step-wise form.

Observation 1: High-Confidence Paths Lead to Efficient Reasoning (Section 4.1, Figure 4)

Using DeepSeek-R1-Distill-Qwen-7B on a 100-question subset of MATH-500 with a 10k token budget and $\text{TR} = 1$:

• TSearch w/ $\Phi$: As exploration width $m$ increases (from 1 to larger values), pass@1 increases while response length decreases. The model is finding shorter but better reasoning chains. At $m=4$, accuracy reaches approximately 92% while response length drops to roughly 2,213 tokens (vs. roughly 84% and 3,126 tokens for random sampling).

• TSearch w/ $\phi$: As $m$ increases, accuracy collapses sharply, following the steep decline in response length. The $\phi$-guided search terminates reasoning prematurely but at wrong moments—it finds short chains, but they're incorrect because the algorithm is selecting beams based on local token probabilities that don't reflect global path quality.

What this establishes: $\Phi$ is not just a convenient scoring function—it is necessary for distinguishing genuinely efficient reasoning chains from arbitrary short chains. The $\phi$ variant demonstrates what happens when you try to terminate early without whole-path confidence: the model stops at syntactically plausible but logically wrong points. The $\Phi$ variant shows that when you rank candidates by whole-path confidence, the paths that reach $\langle\text{response}\rangle$ early are disproportionately the correct ones.

Observation 2: $\Phi$ Paths Lead to Confident $\langle\text{response}\rangle$ Tokens (Section 4.2, Table 1, Figure 5)

This observation explains the role of the TR parameter and justifies its removal in SAGE.

Experiment: vary TR for both TSearch variants at $m=4, r=1$. For TSearch w/ $\Phi$, accuracy is essentially constant at ~92% regardless of TR (1.0, 0.75, 0.50). Response length is also essentially constant (~2,213, ~2,221, ~2,212 tokens). For TSearch w/ $\phi$, accuracy drops sharply with increasing TR: 0.89 at TR=0.50, 0.82 at TR=0.75, 0.79 at TR=1.00.

Why this happens (Figure 5): When evaluated by $\Phi$, the $\langle\text{response}\rangle$ token consistently ranks first within the candidate set $\mathcal{T}^{(j)}$ at the moment it appears—the policy is highly confident at the local level that stopping is correct. As exploration width increases, this rank remains at 1 for $\Phi$-guided search. In contrast, for $\phi$-guided search, the rank ratio of $\langle\text{response}\rangle$ gradually increases (worsens) as $m$ grows—the model becomes increasingly uncertain about whether $\langle\text{response}\rangle$ is the right next token, even though the beam happened to include it.

What this establishes: When beams are selected by whole-path confidence ($\Phi$), the paths that survive to produce $\langle\text{response}\rangle$ are precisely those where the model is also locally confident about stopping. The two signals are correlated when $\Phi$ is used for selection but uncorrelated when $\phi$ is used. This means TR is redundant for $\Phi$-guided search—you can set TR=1 (accept any $\langle\text{response}\rangle$ token) without risking premature termination, because only genuinely complete reasoning chains will reach $\langle\text{response}\rangle$ with high $\Phi$ scores in the first place.

The practical implication: This observation enables the simplification to SAGE in Section 5.1, where TR is removed entirely and termination occurs simply upon detecting $\langle\text{response}\rangle$ in any candidate. The $\Phi$ selection implicitly filters out premature $\langle\text{response}\rangle$ tokens.

Observation 3: Scaling Exploration Drives Capability Convergence (Section 4.3, Figure 7, Figure 15)

Using TSearch w/ $\Phi$ with a large token budget ($T_{\max} = 32{,}768$), the paper sweeps $m$ (exploration width) from 0 to larger values on DS-1.5B and DeepScaleR across MATH-500 and AMC23.

Finding: As $m$ increases:

• Pass@1 monotonically improves and approaches a plateau—the model is reaching the boundary of its intrinsic efficient-reasoning capability.
• Response length monotonically decreases and approaches a plateau—the model finds progressively shorter chains, but there's a minimum length below which correctness would require tokens the model can't compress.
• Token efficiency (pass@1 / response length) increases sharply then converges—the model is squeezing more correctness out of fewer tokens.

What this establishes: The efficient reasoning capability is not an artifact of a particular $m$—it's a latent property of the model that is progressively uncovered as exploration increases. The convergence behavior implies there's a well-defined "efficient reasoning frontier" for each model-benchmark pair. Stronger post-training (DeepScaleR vs. DS-1.5B) leads to steeper improvement, suggesting that RL training enhances the model's ability to leverage exploration for finding efficient paths.

Why this matters for SAGE-RL: This observation justifies using SAGE as a rollout strategy in RL training. If SAGE can consistently uncover efficient reasoning chains (which it does, with convergence at moderate $m$), then mixing those chains into RL training gives the policy model positive examples of efficient reasoning without needing to run SAGE at test time. The RL training teaches the model to generate these efficient chains directly (pass@1), rather than requiring confidence-guided search to find them.

---

Step-Wise SAGE: The Practical Algorithm

Based on the three observations above, the paper simplifies TSearch to SAGE by making two changes (Section 5.1):

Change 1: Step-Wise Instead of Token-Wise Expansion

The token-level expansion in Equation 5 is replaced with reasoning-step-level expansion. At each step $i$, each of the $m$ candidate sequences is extended by one full reasoning step $\mathbf{r}$ (a sequence of tokens ending at a natural step boundary, identified by "\n\n" delimiters), until the maximum reasoning step limit $T_{\max}$ is reached:

$\mathbf{y}_i^{(j,k)} = \mathbf{y}_{i-1}^{(j)} \circ \mathbf{r}_i^{(j,k)}, \quad \mathbf{r}_i^{(j,k)} \in \mathcal{R}^{(j)}$

where $\mathcal{R}^{(j)} = \{\mathbf{r}_i^{(j,1)}, \mathbf{r}_i^{(j,2)}, \ldots, \mathbf{r}_i^{(j,2m)}\}$ is a set of $2m$ reasoning steps independently sampled from the policy $\pi_\theta$ conditioned on the query $\mathbf{x}$ and the prefix $\mathbf{y}_{i-1}^{(j)}$ using standard random sampling (temperature = 1.0, top-p = 1.0). Each "step" is a complete chunk of reasoning text generated by the model in one sampling pass, terminated by the model's own "\n\n" delimiter or the end of the maximum generation length.

Why step-wise: Token-level expansion is computationally expensive—every token triggers a beam expansion and scoring operation. Step-level expansion amortizes this cost: the model generates complete reasoning paragraphs in parallel (using standard batched sampling, which vLLM optimizes heavily), and only $2m$ forward passes are needed per step rather than $2m$ per token. This makes SAGE practical on available hardware (the paper uses 8 GPUs). The step-wise granularity also aligns with the natural structure of chain-of-thought reasoning: reasoning decisions happen at the paragraph level, and individual token choices within a step are less informative about overall reasoning quality than the step-level structure.

Change 2: Simplified Termination

Based on Observation 2 (Section 4.2), the TR parameter is removed entirely. The termination condition becomes simply: if any candidate sequence $\mathbf{y}_i^{(j,k)}$ ends with $\langle\text{response}\rangle$, add it to the completion set $\mathcal{O}$. When $|\mathcal{O}| \geq r$, terminate. The fallback when $i = T_{\max}$ remains the same: select top $r - |\mathcal{O}|$ by $\Phi$.

Why this works: Because $\Phi$-guided beam selection ensures that only genuinely complete, high-confidence reasoning paths survive to the point where $\langle\text{response}\rangle$ appears. Any path that reaches $\langle\text{response}\rangle$ prematurely (before the reasoning is actually complete) will have accumulated low $\Phi$ scores from the incomplete reasoning steps and will have been pruned in earlier beam selection steps.

Computational Properties

With exploration width $m$, SAGE generates $2m$ reasoning steps in parallel at each expansion step, then scores $2m^2$ candidates. This has approximately $2m$ times the space complexity of Degrade SAGE (which generates only 1 step per expansion) but the same theoretical time complexity when unlimited parallelism is available (since all $2m$ steps can be generated simultaneously). In practice, with vLLM's memory-constrained serving on 8 GPUs, the paper reports that inference time increases with $m$, with a transition point around $m=2$ beyond which growth accelerates (Figure 14a). This motivates setting $m=2$ as the default, balancing exploration breadth against computational cost.

Degrade SAGE (Ablation Baseline)

For controlled comparison, the paper defines Degrade SAGE: directly sample one reasoning step at each iteration (no beam, no exploration) until $\langle\text{response}\rangle$ appears or $T_{\max}$ is reached. This is essentially standard autoregressive generation with early stopping when the model produces an end-of-response token. Comparing SAGE against Degrade SAGE isolates the effect of the exploration mechanism: both terminate early, but SAGE explores multiple candidate continuations at each step while Degrade SAGE commits to a single path.

The results in Figure 8 (Section 5.2) show that SAGE consistently outperforms Degrade SAGE at all step budgets. Under constrained step budgets, SAGE achieves higher pass@1 because it stops thinking earlier and thus produces more complete CoTs (less truncation). Under ample step budgets, a stable performance gap remains because SAGE's exploration finds reasoning chains that are both shorter and more likely to be correct than the single path Degrade SAGE commits to.

SAGE Inference Scaling Trends (Section 5.2, Figure 8)

The paper characterizes how SAGE(2,1) behaves as the maximum step budget scales:

• On stronger models (DeepScaleR) and harder datasets (AMC23): SAGE achieves larger pass@1 gains compared to Degrade SAGE. Stronger models have a higher capability ceiling, so exploration uncovers reasoning paths that genuinely improve correctness, not just compression.

• On weaker models (DS-1.5B) and simpler datasets (MATH-500): SAGE achieves larger response length reductions. Weaker models suffer from more severe overthinking (they generate many redundant tokens even when the correct answer was reachable early), so the efficiency gains are larger.

• From the dataset perspective: On easy datasets where the model already solves most problems correctly, SAGE focuses on compression—finding shorter chains that maintain accuracy. On hard datasets where many problems are near the model's capability boundary, SAGE focuses on accuracy—finding correct chains that single-path sampling would miss.

This adaptive behavior emerges naturally from the $\Phi$-guided search: on easy problems, many paths lead to correct answers, and $\Phi$ selects the shortest high-confidence one; on hard problems, correct paths are rare, and $\Phi$ identifies them when they exist by aggregating confidence signals across the full reasoning chain.

---

SAGE-RL: Integrating Efficient Reasoning into RL Training

Section 6 describes how SAGE is incorporated into reinforcement learning.

The Core Idea: Hybrid Rollout

Standard RLVR (GRPO or GSPO) samples a group of $G$ responses from the current policy for each training query. SAGE-RL modifies only the rollout phase: within each group of $G$ responses, $r$ responses are generated by SAGE(m, r) and the remaining $G - r$ by standard random sampling.

The paper uses $G = 8$, $m = 2$, $r = 2$: in each training batch, 2 out of 8 responses per question come from SAGE's confidence-guided search, and 6 come from default random sampling.

Why this proportion: The 2/8 ratio is a design choice that balances two competing needs. Too few SAGE trajectories (e.g., 1/8) would provide insufficient signal for the policy to learn efficient reasoning patterns. Too many SAGE trajectories (e.g., 4/8 or 8/8) would reduce the diversity of the rollout distribution, potentially limiting the model's exploration of alternative solution strategies. The paper's hyperparameter analysis (Appendix D.2, Table 5) shows that increasing $r$ from 1 to 2 has limited effect on final performance, while increasing $m$ from 1 to 2 yields substantial gains—suggesting exploration width matters more than the number of SAGE trajectories per group.

Mathematical Modification (Equations 9–15)

The SAGE-RL objectives are straightforward modifications of GRPO and GSPO where the expectation over rollouts is split into two components:

SAGE-GRPO (Equation 10):

$\mathcal{J}_{\text{SAGE-GRPO}}(\theta) = \mathbb{E}_{x \sim \mathcal{D}, \{y_i\}_{i=1}^{G} \sim \pi_{\theta_{\text{old}}}(\cdot|x)} \left[ \frac{1}{G} \left( \underbrace{\sum_{i=1}^{r} \frac{1}{|y_i|} \sum_{t=1}^{|y_i|} \min\left(w_{i,t}(\theta) \hat{A}_{i,t}, \text{clip}(w_{i,t}(\theta), 1-\epsilon, 1+\epsilon) \hat{A}_{i,t}\right)}_{\text{SAGE(m,r)}} + \underbrace{\sum_{i=r+1}^{G} \frac{1}{|y_i|} \sum_{t=1}^{|y_i|} \min\left(w_{i,t}(\theta) \hat{A}_{i,t}, \text{clip}(w_{i,t}(\theta), 1-\epsilon, 1+\epsilon) \hat{A}_{i,t}\right)}_{\text{Random Sampling}} \right) \right]$

where $w_{i,t}(\theta) = \frac{\pi_\theta(y_{i,t} | x, y_{i,<t})}{\pi_{\theta_{\text{old}}}(y_{i,t} | x, y_{i,<t})}$ is the token-level importance ratio and $\hat{A}_{i,t} = \frac{r(x, y_i) - \text{mean}(\{r(x, y_i)\}_{i=1}^{G})}{\text{std}(\{r(x, y_i)\}_{i=1}^{G})}$ is the group-relative advantage (Equation 11), which is identical across all tokens in the same sequence.

What this computes: the first sum (indices 1 to $r$) computes the standard GRPO token-level clipped surrogate objective over the SAGE-generated trajectories; the second sum (indices $r+1$ to $G$) computes the same objective over the randomly sampled trajectories. Both sums contribute equally to the gradient (each is divided by $G$), and the advantage estimates $\hat{A}_{i,t}$ are computed over the full group of $G$ responses, meaning the SAGE trajectories participate in determining the group mean and standard deviation used for advantage normalization.

SAGE-GSPO (Equation 13) follows the same structure but with the sequence-level importance ratio $s_i(\theta)$ (Equation 15) replacing the token-level ratio:

$s_i(\theta) = \left(\frac{\pi_\theta(y_i \mid x)}{\pi_{\theta_{\text{old}}}(y_i \mid x)}\right)^{1/|y_i|} = \exp\left(\frac{1}{|y_i|} \sum_{t=1}^{|y_i|} \log \frac{\pi_\theta(y_{i,t} \mid x, y_{i,<t})}{\pi_{\theta_{\text{old}}}(y_{i,t} \mid x, y_{i,<t})}\right)$

where $s_i(\theta)$ is the geometric mean of per-token probability ratios across the entire sequence. GSPO uses this scalar per sequence rather than per-token ratios, and the objective becomes:

$\mathcal{J}_{\text{SAGE-GSPO}}(\theta) = \mathbb{E} \left[ \frac{1}{G} \left( \sum_{i=1}^{r} \min\left(s_i(\theta) \hat{A}_i, \text{clip}(s_i(\theta), 1-\epsilon, 1+\epsilon) \hat{A}_i\right) + \sum_{i=r+1}^{G} \min\left(s_i(\theta) \hat{A}_i, \text{clip}(s_i(\theta), 1-\epsilon, 1+\epsilon) \hat{A}_i\right) \right) \right]$

with the same group-based advantage (Equation 14): $\hat{A}_i = \frac{r(x, y_i) - \text{mean}(\{r(x, y_i)\}_{i=1}^{G})}{\text{std}(\{r(x, y_i)\}_{i=1}^{G})}$.

Why SAGE-GSPO slightly outperforms SAGE-GRPO (Appendix D.1): The paper hypothesizes that GSPO's sequence-level importance sampling is more stable when the rollout distribution includes SAGE-generated trajectories. In SAGE-GRPO, the per-token probabilities of SAGE trajectories under $\pi_{\theta_{\text{old}}}$ may be lower than those of randomly sampled trajectories (because SAGE selects beams based on $\Phi$, which aggregates across tokens, rather than greedily maximizing per-token $\phi$). This increases the likelihood of clipping during token-level importance sampling when $w_{i,t}(\theta)$ falls outside $[1-\epsilon, 1+\epsilon]$. GSPO's sequence-level ratio $s_i(\theta)$ averages over the entire sequence, smoothing out per-token variance and reducing clipping.

The Learning Mechanism: Why Hybrid Rollout Works

The paper describes the learning mechanism in terms of advantage estimation and regularization:

Advantage amplification. SAGE-generated trajectories are more frequently correct and shorter than randomly sampled trajectories from the same policy. In the group-based advantage calculation (Equations 11, 14), these SAGE trajectories receive positive advantages (they're above the group mean), while many random trajectories receive zero or negative advantages. The GRPO/GSPO objective then increases the probability of tokens in the SAGE trajectories relative to the old policy, while decreasing the probability of tokens in poor random trajectories.

Implicit regularization toward efficiency. Crucially, there is NO explicit length penalty in the reward—the reward $r(x, y_i)$ is purely binary 0/1 based on answer correctness. The efficiency learning occurs because SAGE trajectories are both correct and concise, and the RL optimization increases their probability. The model learns to associate the reasoning patterns in SAGE trajectories (compact, confident, terminating early) with positive advantage, and over training shifts its pass@1 distribution toward these patterns.

Entropy reduction. Figure 9 (right) shows that SAGE-RL exhibits more significant entropy reduction than standard RLVR during training. As the policy increasingly assigns high probability to the efficient reasoning patterns discovered by SAGE, it becomes more deterministic—less likely to explore long, meandering reasoning chains. This is not an explicit penalty but an emergent consequence of the model learning to favor specific high-reward trajectories.

KL divergence increase. Figure 9 also shows higher KL divergence between the current policy and the reference (old) policy under SAGE-RL compared to standard RLVR. This indicates that the SAGE trajectories induce larger policy updates—the model is being pushed toward reasoning patterns that differ substantially from its original (verbose) distribution. This is consistent with the idea that efficient reasoning requires the model to unlearn its RLVR-acquired bias toward "keep generating tokens to be safe."

Training Configuration

The paper uses verl (Sheng et al., 2024) with the following hyperparameters, identical between SAGE-RL and baseline RLVR for fair comparison: global batch size 32 across 8 GPUs, 600 training steps, Adam optimizer with learning rate $1 \times 10^{-6}$, cosine warmup for first 50 steps, sampling temperature $T = 1.0$, KL regularization with $\beta = 0.001$, entropy coefficient $\gamma = 0.001$, maximum context length 9,216 tokens (with 1,024 reserved for the prompt). Training data is the English subset of DAPO (Yu et al., 2025) plus MATH problems at difficulty levels 3–5 (Zheng et al., 2025b), totaling approximately 20,000 problems.

What these hyperparameters control: the KL penalty $\beta = 0.001$ prevents the policy from diverging too far from the reference model in a single update, which is critical for stability when SAGE trajectories may have different token distributions. The entropy coefficient $\gamma = 0.001$ adds a small bonus for maintaining token-level diversity, which counters the entropy reduction observed in SAGE-RL and prevents the policy from collapsing to a deterministic mode too quickly.

Why Not Distillation? (Section A.2)

The paper explicitly considers and rejects directly distilling SAGE outputs into the model via supervised fine-tuning. The stated reason: "since distillation depends on a strong teacher model, we are concerned that self-distillation will limit the upper boundary of the model's reasoning capability." In self-distillation, the model would learn to imitate its own SAGE-generated reasoning chains, but those chains are bounded by the model's current capability at generation time. RL, by contrast, uses SAGE rollouts as positive examples within a broader exploration process—the randomly sampled trajectories ensure the model continues exploring solution strategies beyond what SAGE produces, and the RL optimization process can discover even more efficient patterns than those in the initial SAGE trajectories. The gradual entropy reduction and KL increase during SAGE-RL training (Figure 9) suggest a progressive refinement rather than a one-shot imitation.

Computational Comparison with Alternatives (Section A.2)

The paper positions SAGE-RL as computationally lightweight relative to several alternatives:

• GFPO (Shrivastava et al., 2025): requires oversampling—generating many candidate solutions per query and selecting short correct ones. SAGE-RL generates the same group size $G=8$ as standard RLVR; the SAGE overhead is in the decoding algorithm, not in generating extra samples.

• S-GRPO (Dai et al., 2025) and VSRM (Yue et al., 2025): require repeated rollouts to estimate sub-chain rewards for truncated reasoning steps. SAGE-RL performs no extra rollouts beyond the standard group generation.

• Length-penalty methods: require careful reward engineering to balance correctness and conciseness signals, which can cause training instability or reward hacking. SAGE-RL's reward remains purely correctness-based, avoiding this source of instability entirely.

The paper reports that SAGE-RL exhibits "no significant difference in training stability compared with vanilla RLVR" (Section A.2), which is a direct consequence of preserving the original optimization objective unchanged.

---

Hyperparameter Sensitivity Analysis (Appendix D.2)

The paper ablates the two key SAGE-RL hyperparameters using DS-1.5B: exploration width $m$ and number of SAGE rollouts per group $r$. The results in Table 5 compare SAGE(1,1)-GRPO, SAGE(2,1)-GRPO, and SAGE(2,2)-GRPO.

Exploration width $m$: Increasing from $m=1$ to $m=2$ yields substantial gains. On MATH-500, SAGE(2,1)-GRPO achieves 84.2% pass@1 vs. 84.0% for SAGE(1,1)-GRPO, but the response length drops significantly (2,952 vs. 3,416 tokens). On AIME 2025, the improvement is more dramatic: 25.7% vs. 24.8% pass@1 with similar length reduction. The training dynamics (Figure 12) show that SAGE(1,1)-GRPO exhibits much milder entropy reduction and KL increase compared to SAGE(2,1)-GRPO, and remains closer to vanilla GRPO behavior—suggesting that $m=1$ provides insufficient exploration to surface efficient reasoning patterns distinct from random sampling.

Number of SAGE rollouts $r$: Increasing from $r=1$ to $r=2$ has limited effect. SAGE(2,2)-GRPO achieves 84.8% vs. 84.2% for SAGE(2,1)-GRPO on MATH-500, with similar response length (~2,915 vs. ~2,952). The entropy and KL curves nearly overlap (Figure 12). The paper explains this as diminishing returns: "rollouts with similar reasoning trajectories offer minimal additional information" because both SAGE trajectories in the 2-rollout case tend to be drawn from the same high-confidence region of the model's distribution.

Practical default: SAGE(2,2) is used throughout the main experiments as a balanced configuration, though $r=1$ would be nearly as effective and computationally cheaper.

4. Key Insights and Innovations

Innovation 1: Reframing Overthinking from a Behavioral Flaw to a Sampling Artifact

The dominant narrative in the field treated length inflation in reasoning models as a learned behavior—an unfortunate side effect of RLVR training that rewards correctness without penalizing verbosity, causing models to adopt a "keep generating tokens to be safe" strategy. Prior work accepted this premise and proposed fixes that imposed efficiency externally: length penalties in the reward (Kimi Team, 2025a; Cheng et al., 2025), budget-constrained optimization (Hou et al., 2025; Qi et al., 2025), or difficulty-aware length assignments (Zhang et al., 2025; AdaptThink). Underlying all of these is the assumption that the model needs to be taught to be concise—that efficiency is a capability to be added, not recovered.

This paper performs a fundamental reframing. Through the RFCS metric (Ratio of the First Correct Step, Figure 3), it demonstrates that correct answers routinely appear early in reasoning chains—the model already produces concise, correct reasoning—but then continues generating hundreds of redundant tokens. In the example from Figure 2, the model derives the correct answer using only 500 tokens yet continues with an additional 452 tokens before terminating. This is not a model that doesn't know how to be concise; it's a model whose sampling mechanism prevents it from acting on its own knowledge.

The conceptual move is to treat the long-chain problem not as a capability deficit but as a selection failure under current sampling paradigms. The model's probability distribution contains compact, high-quality reasoning chains; standard decoding strategies simply fail to identify and prefer them. This shifts the research question from "how do we teach models to be efficient?" to "why can't models access their own efficient reasoning capability, and how do we surface it?" The answer—the Φ vs. ϕ distinction (Observations 1–3 in Section 4)—reveals that next-token probability alone is insufficient for recognizing when a reasoning chain is complete, and that cumulative-path confidence captures this signal.

This is a genuinely distinctive framing that resolves the apparent contradiction in prior work: training-free methods fail because they don't change the model's tendency to produce long chains (they only try to select among what's generated), while RL-with-length-penalty methods risk degrading capability because they fight against the model's learned behavior rather than surfacing its latent competence. SAGE-RL succeeds by neither fighting nor accepting the overthinking behavior—it bypasses it by showing the model its own best work and letting RL do the rest.

---

Innovation 2: The Φ–ϕ Divergence as a Diagnostic for When Reasoning Models Should Stop

No prior work on efficient reasoning—not the training-free methods, not the length-penalty RL approaches, not the budget-constrained optimization methods—had identified a concrete, measurable signal within the model's own probability distribution that distinguishes genuinely complete reasoning chains from prematurely truncated ones. The field operated with heuristics: make the model produce fewer tokens, penalize long outputs, set a budget. These all treat termination as an external constraint.

The paper's central diagnostic insight is that cumulative-path confidence (Φ) and next-token confidence (ϕ) systematically diverge at the moment when a compact reasoning chain should end, and that this divergence explains both why standard sampling fails and why confidence-guided search can succeed. The empirical demonstration in Section 4.2 is crisp: for Φ-guided search, the end-of-response token consistently ranks first in the local probability distribution at the moment it appears (Figure 5). For ϕ-guided search, the rank ratio of the end-of-response token degrades as exploration widens—the model becomes increasingly uncertain about whether it should stop, despite the beam having produced a structurally complete chain. The result is the Table 1 contrast: Φ-guided search is insensitive to TR (accuracy stays at ~92% whether TR is 1.0 or 0.50), while ϕ-guided search's accuracy collapses without tight TR filtering (0.79 at TR=1.0 vs. 0.89 at TR=0.50).

This is not just a convenient scoring choice—it's a mechanistic finding about how reasoning models encode completion knowledge. The model's local token predictions reflect immediate syntactic and semantic continuation possibilities; its cumulative-path probabilities reflect global coherence. When a reasoning chain is genuinely complete, these signals align—the path is globally confident and the model locally recognizes the end-of-response token as appropriate. When a chain is prematurely truncated (as in ϕ-guided search), the path score is low because the reasoning is incomplete, and the local end-of-response signal is correspondingly uncertain because the model "knows" it's not done.

The significance of this finding extends beyond SAGE itself. It provides a principled diagnostic for evaluating when reasoning models should stop—any method that proposes early termination should demonstrate that it's stopping at points where Φ and the end-of-response rank are jointly high, not just where a budget is exhausted or a cheap confidence threshold is crossed. This is a conceptual contribution that future work on efficient reasoning can build on, independent of SAGE's specific implementation.

---

Innovation 3: RL as a Mechanism for Internalizing Efficient Reasoning Patterns Without Explicit Efficiency Objectives

The standard approach to teaching models efficient behavior through RL is to modify the reward function—add a length penalty, set a token budget, incorporate a conciseness bonus. This creates a multi-objective optimization problem where the model must balance correctness against conciseness, and the literature is replete with examples of this tradeoff going wrong: models learn to output "I don't know" or produce degenerate short answers that technically satisfy the length constraint while abandoning reasoning entirely. Length-penalty methods are fundamentally fighting the model's RLVR-acquired instinct to "think longer to be safer."

SAGE-RL's innovation is to achieve efficiency learning without any efficiency signal in the objective function. The RL reward remains purely binary correctness (Equations 10, 13); the length compression emerges entirely from the distribution of rollouts that the model learns from. By mixing SAGE-generated trajectories (which are correct and compact) with standard random rollouts (which are correct and verbose), the group-based advantage estimation naturally assigns higher advantages to the SAGE trajectories—they're above the group mean because they achieve the same correctness reward with fewer tokens, while verbose trajectories that are also correct have only average advantages. The gradient signal pushes the policy toward the efficient trajectories' token distribution without any explicit "be shorter" term.

This is conceptually similar to how RLHF with a good reward model can teach helpfulness without an explicit "be helpful" feature in the reward—the signal comes from the relative ranking of trajectories, not from a decomposed reward function. SAGE-RL applies the same principle to efficiency: the SAGE trajectories demonstrate that correctness can be achieved with fewer tokens, and the advantage mechanism makes the model prefer those patterns.

The evidence that this works without degrading reasoning capability is important. Figure 9 shows that SAGE-RL achieves both higher pass@1 and shorter response length compared to standard RLVR, with the improvement in both metrics continuing throughout training. The entropy reduction (model becomes more confident in its efficient patterns) and KL divergence increase (model moves away from its original verbose distribution) track together smoothly, unlike the instability that often accompanies length-penalty methods. The difficulty-level analysis (Figure 13, Appendix D.3) shows that SAGE-RL's gains are concentrated on hard problems—where efficient reasoning patterns are most valuable and where length-penalty methods are most likely to hurt—confirming that the model is learning to be selectively concise rather than uniformly short.

This is a methodological contribution: hybrid-rollout RL as a general technique for transferring desirable properties from a training-time search procedure to a pass@1 model, without modifying the reward function. The specific instantiation uses SAGE for conciseness, but the framework could incorporate any search or decoding strategy whose outputs exhibit properties one wants the policy to internalize.

---

Innovation 4: The "Unleashing" Paradigm for Model Capability

The paper's title asks: "Does Your Reasoning Model Implicitly Know When to Stop Thinking?" The framing is deliberate and non-obvious. In most ML research, when a model exhibits undesirable behavior (verbosity, in this case), the default assumption is that the model lacks the capability to behave otherwise—it needs to be taught, fine-tuned, or constrained. The paper inverts this: the capability exists, but it's obscured by the sampling mechanism, not absent from the model.

This "unleashing" paradigm has precursors—for example, the insight that LLMs "know more than they can say" and that prompting can surface latent knowledge—but its application to reasoning process efficiency is novel. Prior work on overthinking (Chen et al., 2025b) identified that LRMs generate redundant tokens, but stopped at diagnosis. SAGE provides the constructive demonstration: by giving the model an expanded exploration space guided by cumulative confidence, it consistently finds compact, correct reasoning chains that standard sampling misses, and this behavior converges (Section 4.3, Figure 7) toward a well-defined efficiency frontier.

The paradigm has practical implications beyond this paper's methods. It suggests that in general, when a model exhibits suboptimal behavior under standard decoding, the first question should be: "Is this behavior latent in the model's distribution, and if so, what decoding strategy would surface the better behavior?" rather than "How do we retrain the model to fix this?" This is a research-methodology contribution—a shift in diagnostic posture that could apply to other model behaviors (hallucination, uncertainty expression, calibration) beyond reasoning efficiency.

The convergence analysis in Section 4.3 (Figure 15) is particularly significant here because it transforms the claim from anecdotal ("sometimes the model produces short correct answers") to systematic ("as exploration increases, the model's efficient reasoning capability asymptotically approaches a stable frontier"). This convergence implies that efficient reasoning is not a lucky sampling artifact but a stable property of the model's probability landscape—there exist high-Φ basins of attraction corresponding to compact solution paths, and SAGE's search discovers them.

The paper is appropriately careful about scope: this unleashing only works because the capability exists. On the hardest problems (difficulty bin 5 in the FLOPs-matched analysis context), no amount of search helps because there are no correct reasoning paths in the distribution to find. This boundary condition—efficient reasoning can be surfaced where the model has the underlying capability, but cannot be created where it doesn't—is a crucial qualification that distinguishes unleashing from magic.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper evaluates on six mathematical reasoning benchmarks: MATH-500 (Lightman et al., 2023, 500 test problems), AIME 2024 and AIME 2025 (Art of Problem Solving, 2024, competition-level problems), AMC23 (Mathematical Association of America, 2023), OlympiadBench (Chaoqun et al., 2024, olympiad-level problems), and Minerva (Lewkowycz et al., 2022). Training data for RL consists of the English subset of DAPO (Yu et al., 2025) plus MATH problems at difficulty levels 3–5 (Zheng et al., 2025b), totaling approximately 20,000 problems.

• Base model(s). Four widely adopted LRMs are used: DeepSeek-R1-Distill-Qwen-1.5B (DS-1.5B), DeepSeek-R1-Distill-Qwen-7B (DS-7B) (DeepSeek-AI, 2025), DeepScaleR (Luo et al., 2025b, a post-trained version of DS-1.5B), and Qwen3-8B (Yang et al., 2025a). These span a range of scales (1.5B to 8B parameters) and post-training extents, allowing investigation of how SAGE-RL interacts with model capability.

• Metrics. Three primary metrics are reported: Pass@1 (the fraction of problems where the model's single generated answer matches the ground truth), response length (LEN) (the average number of tokens in the complete output, including reasoning and answer sections), and token efficiency (TE) (computed as Pass@1 / LEN, reported in units of 10⁻³, measuring correctness per token). For OlympiadBench, Minerva, and MATH-500, results are averaged over 8 runs (mean@8) to control for sampling variance; for the smaller AIME and AMC benchmarks, results are averaged over 32 runs (mean@32).

• Baselines. The paper compares against five categories of prior methods: (1) Training-free efficiency methods: LC-R1 (Cheng et al., 2025), which optimizes length compression; (2) Budget-constrained methods: ThinkPrune-2k (Hou et al., 2025), which prunes reasoning within a 2,048-token budget, and AdaptThink (Zhang et al., 2025), which assigns difficulty-dependent thinking patterns; (3) Efficient reasoning fine-tuning: Efficient Reasoning (Arora et al., 2025), which trains models for reasoning conciseness, and GRPO-LEAD (Zhang et al., 2025), which uses difficulty-aware length penalties; (4) Standard RLVR: vanilla GRPO (Shao et al., 2024) and GSPO (Zheng et al., 2025a) with no efficiency modifications, serving as the direct baselines for SAGE-RL since the only difference is the rollout phase; (5) The base models themselves with standard random sampling (temperature = 1.0, top-p = 0.95), reported as the starting point for all comparisons.

• Generation budget / compute accounting. During evaluation, all methods use a maximum generation length of 32,768 tokens (consistent with DeepSeek-R1 and Hou et al., 2025), with temperature = 1.0 and top-p = 0.95. For SAGE decoding experiments (Section 5.2, Figure 8), the step budget (number of reasoning-step expansions before forced termination) is systematically scaled to characterize inference-time scaling behavior. For RL training, a group size of $G = 8$ responses per query is used, with SAGE-RL generating $r = 2$ via SAGE(2,2) and $G - r = 6$ via standard random sampling. All RL experiments train for 600 steps with a global batch size of 32 across 8 GPUs, using the Adam optimizer with learning rate $1 \times 10^{-6}$, cosine warmup for 50 steps, and a maximum context length of 9,216 tokens (1,024 reserved for the prompt). Training compute is not directly compared across methods since SAGE-RL's overhead is in the decoding algorithm rather than in additional rollouts.

• Cross-validation / statistical protocol. No cross-validation is used for strategy selection—SAGE's hyperparameters ($m = 2$, $r = 2$) are fixed based on the sensitivity analysis in Appendix D.2. For evaluation, results are averaged over multiple independent sampling runs (mean@8 or mean@32) to reduce variance from stochastic generation. All RL experiments use identical hyperparameters between SAGE-RL and baseline RLVR for fair comparison, isolating the effect of the rollout modification. The paper does not report confidence intervals or statistical significance tests.

Main Quantitative Results

The experimental results are organized across two axes: (1) the performance of the training-free SAGE decoding strategy compared to baselines, and (2) the performance of SAGE-RL-tuned models compared to both base models and prior efficient-reasoning methods. The training-free results establish that SAGE can surface efficient reasoning capability at inference time; the SAGE-RL results demonstrate that this capability can be internalized into pass@1 behavior through RL.

Training-Free SAGE Decoding: TSearch Analysis (Sections 4.1–4.3, Figures 4–7, Table 1)

TSearch w/ Φ vs. TSearch w/ φ (Figure 4, Section 4.1). Using DS-7B on a 100-question subset of MATH-500 with a 10k token budget and TR = 1, TSearch w/ Φ achieves the following as exploration width $m$ increases: accuracy improves from approximately 84% (at $m=1$, comparable to random sampling) to roughly 92% (at $m=4$), while response length simultaneously decreases from approximately 3,126 tokens (random baseline) to roughly 2,213 tokens. In contrast, TSearch w/ $\phi$ (which replaces cumulative-path scoring with per-token greedy selection) sees accuracy collapse from roughly 84% to well below 80% as $m$ increases, tracking a sharp decline in response length toward well under 2,000 tokens. The failure mode is clear: $\phi$-guided search terminates reasoning prematurely at incorrect points, while $\Phi$-guided search identifies genuinely compact yet correct chains.

TR sensitivity analysis (Table 1, Section 4.2). With TSearch(4,1), varying TR from 1.00 to 0.50 produces virtually no change in accuracy for TSearch w/ Φ (~92%, ~92%, ~91%) or response length (~2,213, ~2,221, ~2,212 tokens). For TSearch w/ $\phi$, accuracy drops sharply as TR becomes more permissive: 0.89 at TR = 0.50, 0.82 at TR = 0.75, 0.79 at TR = 1.00, while response length correspondingly drops from 2,609 to 2,129 tokens. This establishes that $\Phi$-guided search needs no tolerance filtering—the end-of-response token consistently ranks first when it appears (Figure 5)—while $\phi$-guided search requires aggressive filtering to avoid premature termination, and even then underperforms.

Exploration convergence (Figures 7 and 15, Section 4.3). Under a 32,768-token budget with TSearch w/ Φ, as $m$ scales from 0 to larger values on DS-1.5B and DeepScaleR across MATH-500 and AMC23: pass@1 increases monotonically and approaches a plateau (the model's efficient-reasoning frontier), response length decreases monotonically and plateaus, and token efficiency improves sharply then converges. DeepScaleR exhibits steeper improvement than DS-1.5B (greater post-training enhances the ability to leverage exploration), and the convergence pattern is consistent across both models and both benchmarks, establishing that efficient reasoning is a stable latent property rather than a sampling artifact.

SAGE vs. Degrade SAGE (Figure 8, Section 5.2)

SAGE(2,1) is compared against Degrade SAGE (single-path reasoning with early termination) under increasing step budgets on MATH-500 (mean@4) and AMC23 (mean@16). Under constrained step budgets, SAGE outperforms Degrade SAGE in pass@1 because it stops thinking earlier (producing more complete CoTs before truncation). Under ample step budgets, a stable performance gap persists: SAGE's exploration uncovers reasoning chains that are both shorter and more likely correct than the single path Degrade SAGE commits to. For example, on DeepScaleR on AMC23, SAGE achieves pass@1 above 70% at moderate budgets while Degrade SAGE remains below 68%. The random-sampling baselines for DeepScaleR (red dashed line) and DS-1.5B (blue dashed line) at 32,768 tokens provide reference points: SAGE consistently exceeds both baselines in either accuracy, conciseness, or both, depending on the model-dataset combination.

Difficulty-dependent behavior. On stronger models (DeepScaleR) and harder datasets (AMC23), SAGE delivers larger pass@1 gains. On weaker models (DS-1.5B) and easier datasets (MATH-500), SAGE delivers larger response length reductions. This adaptive specialization emerges from the Φ-guided search mechanism: on easy problems where many paths lead to correctness, Φ selects the shortest high-confidence one; on hard problems where correct paths are rare, Φ identifies them when they exist.

SAGE-RL Main Results (Table 2, Table 4, Figures 9–10)

The comprehensive results across 4 models × 6 benchmarks × up to 9 methods are presented in Table 4 (Appendix D.1); Table 2 in the main text shows a representative subset across 4 benchmarks. The headline findings are organized by model:

DS-1.5B results (Table 4). SAGE-RL variants (SAGE-GRPO, SAGE-GSPO) achieve the best or second-best token efficiency across all six benchmarks while consistently improving pass@1 over the base model. On AIME 2025, SAGE-GSPO yields pass@1 of 27.1% (a +6.2 percentage point gain over the base model's 20.9%) with response length compressed from 11,669 to 7,167 tokens—a token efficiency improvement of 111.1%. On AMC23, SAGE-GSPO achieves 68.3% pass@1 (+8.2 points) with length reduction from 8,250 to 5,278 tokens. The baselines that achieve the strongest compression (AdaptThink compresses DS-1.5B on MATH-500 from 4,882 to 2,563 tokens) do so at the cost of accuracy: AdaptThink drops pass@1 by 2.8 points on MATH-500 and shows similar degradation on AIME 2024 and OlympiadBench. In contrast, SAGE-RL improves both accuracy and conciseness simultaneously. Vanilla GRPO and GSPO moderately improve reasoning capability (e.g., GRPO achieves 28.3% on AIME 2024 vs. 25.1% base) but leave substantial token redundancy (response lengths of 8,767 and 8,604 tokens respectively), resulting in token efficiency scores that are significantly lower than efficient-reasoning baselines and SAGE-RL.

DeepScaleR results (Table 4). As a model that has already undergone extensive RL post-training (Luo et al., 2025b), DeepScaleR shows more modest gains from further fine-tuning. Nevertheless, SAGE-GRPO achieves 88.8% pass@1 on MATH-500 (+2.8 points over the 86.0% base, +1.2 points over GRPO's 87.6%) with length reduction from 3,805 to 3,117 tokens. On Minerva, SAGE-GRPO achieves 41.4% pass@1 (+2.8 points over base) with length reduction from 5,184 to 3,817 tokens—roughly twice the token efficiency gain of GRPO (10.9 vs. 9.21, representing a 45.6% vs. 23.6% relative improvement). ThinkPrune-2k, the only other method evaluated on DeepScaleR, achieves strong token efficiency on MATH-500 (28.0) but at the cost of a 3.5-point accuracy drop (82.5% vs. 86.0%).

DS-7B results (Table 4). SAGE-GRPO achieves 93.0% pass@1 on MATH-500 (+1.4 points over base, +1.0 over GRPO) with response length nearly halved from 3,871 to 2,141 tokens—a token efficiency improvement of 83.1% over the base model and the best TE of any method at 43.4. On AIME 2024, SAGE-GRPO achieves 55.3% pass@1 (+3.4 points over base, +2.8 over GRPO-LEAD, +2.8 over GRPO) with length compressed from 11,305 to 6,422 tokens. The efficient-reasoning baselines show an interesting pattern: LC-R1, AdaptThink, and Efficient Reasoning all achieve dramatic length compression (to ~2,000–2,400 tokens on MATH-500) but at the cost of substantial accuracy degradation (2.1–4.3 points below the base model's 91.6%). GRPO-LEAD partially mitigates this tradeoff—achieving 89.5% pass@1 on MATH-500 with length 2,752—but still underperforms SAGE-GRPO in both accuracy and conciseness.

Qwen3-8B results (Table 4). As the strongest base model (94.4% on MATH-500, 73.2% on AIME 2024), Qwen3-8B presents the hardest test for efficiency methods—the base model already achieves high accuracy, and overthinking is severe (average response length of 5,640 tokens on MATH-500, more than 2.5× the SAGE-GRPO-tuned DS-7B). Vanilla RLVR partially reduces length through a training artifact (sequences padded to fixed batch length during training encourage shorter generations) but this mechanism limits reasoning improvement or causes declines: GRPO drops pass@1 on MATH-500 from 94.4% to 93.6%, on AIME 2024 from 73.2% to 72.8%. SAGE-GRPO recovers to 95.0% on MATH-500 and 73.5% on AIME 2024 while compressing length from 5,640 to 3,015 tokens on MATH-500 and from 15,920 to 8,975 tokens on AIME 2024. SAGE-GSPO achieves the best token efficiency across nearly all benchmarks on Qwen3-8B, including a 105.3% TE improvement on MATH-500 (34.3 vs. 16.7 base) and 114.7% on OlympiadBench (8.59 vs. 4.00 base).

Training dynamics (Figure 9). The training curves for SAGE-GRPO and SAGE-GSPO vs. their vanilla RLVR counterparts, evaluated every 10 steps on MATH-500 under an 8,192-token budget, show that SAGE-RL exhibits: (1) more pronounced pass@1 improvement throughout training; (2) greater response length reduction; (3) more significant entropy decrease (the policy becomes more confident in generating efficient patterns); and (4) higher KL divergence from the reference model (the policy moves further from its original verbose distribution). These trends are consistent across both GRPO and GSPO, with GSPO-tuned models showing elevated entropy values overall (a known effect of sequence-level importance sampling; Zheng et al., 2025a).

RFCS analysis (Figure 10). On MATH-500, SAGE-GRPO-tuned models show substantially reduced RFCS(<1)—the proportion of samples containing redundant steps after the correct answer first appears—compared to Figure 3's base model statistics. Simultaneously, RFCS(avg) increases markedly, indicating that the models more frequently terminate thinking immediately after producing the correct answer. This provides direct behavioral evidence that SAGE-RL teaches the model to stop at the right moment.

Difficulty-level analysis (Figure 13, Appendix D.3). Comparing SAGE-GRPO-DS-1.5B against GRPO-DS-1.5B across MATH-500 difficulty levels 1–5: SAGE-GRPO converges faster at all difficulty levels and achieves comparable performance to GRPO on levels 1–3. The critical divergence occurs on levels 4–5, where SAGE-GRPO achieves substantially superior pass@1 and markedly lower response length, with the length reduction trend continuing even after GRPO has converged. This indicates that SAGE-RL's gains are concentrated on harder problems—consistent with the broader pattern in Table 4 where SAGE-RL shows larger improvements on challenging benchmarks (AIME 2024, AIME 2025, OlympiadBench, Minerva) than on relatively easier ones (MATH-500, AMC23).

Ablation Studies and Robustness Checks

Exploration width $m$ in SAGE-RL (Table 5, Figure 12): Increasing $m$ from 1 to 2 produces substantial gains across all benchmarks for DS-1.5B: SAGE(2,1)-GRPO achieves 28.5% on AIME 2024 vs. 28.3% for SAGE(1,1)-GRPO, but with response length of 7,308 vs. 7,979 tokens—the primary gain is efficiency rather than accuracy. The training dynamics (Figure 12) show that SAGE(1,1)-GRPO exhibits dramatically milder entropy reduction and KL divergence increase compared to SAGE(2,1)-GRPO, with both curves remaining close to vanilla GRPO behavior—indicating that $m=1$ provides insufficient exploration to surface reasoning patterns distinct from random sampling.

Number of SAGE rollouts $r$ per group (Table 5, Figure 12): Increasing $r$ from 1 to 2 yields marginal additional gains. SAGE(2,2)-GRPO achieves 84.8% on MATH-500 vs. 84.2% for SAGE(2,1)-GRPO, with nearly identical response length (~2,915 vs. ~2,952 tokens). The entropy and KL curves for $r=1$ and $r=2$ nearly overlap (Figure 12), confirming that additional SAGE rollouts within the same group provide minimal new information since both trajectories are drawn from the same high-confidence region.

GRPO vs. GSPO as the base RL algorithm (Table 4, Figure 9): SAGE-GSPO slightly outperforms SAGE-GRPO in response length reduction and overall token efficiency, particularly on Qwen3-8B (34.3 vs. 31.5 TE on MATH-500; 18.0 vs. 16.3 TE on AMC23). The paper hypothesizes that GSPO's sequence-level importance sampling is more stable when the rollout distribution includes SAGE trajectories, since per-token probabilities under $\pi_{\theta_{\text{old}}}$ may be lower for SAGE trajectories (causing clipping in GRPO's token-level ratio), while GSPO's geometric mean over the sequence smooths this variance. Both SAGE-GRPO and SAGE-GSPO substantially outperform their vanilla counterparts.

Comparison with length-penalty and budget-constrained baselines (Table 4): The consistent pattern across models and benchmarks is that methods achieving the strongest length compression (AdaptThink, LC-R1, ThinkPrune-2k) do so at the cost of accuracy degradation, while vanilla RLVR methods (GRPO, GSPO) achieve moderate accuracy gains but leave substantial token redundancy. SAGE-RL is the only approach that systematically achieves both accuracy improvement and significant length reduction. This pattern is particularly clear on DS-7B, where the four efficient-reasoning baselines (LC-R1, AdaptThink, Efficient Reasoning, GRPO-LEAD) all reduce pass@1 on MATH-500 (by 1.8–4.3 points) while SAGE-GRPO increases it by 1.4 points.

SAGE inference scaling vs. step budget (Figure 8): The comparison between SAGE(2,1) and Degrade SAGE under increasing step budgets serves as an implicit ablation of the exploration mechanism: removing the beam search and Φ-guided selection (collapsing to Degrade SAGE) consistently reduces performance at all budgets, with the gap persisting even under ample budgets where token count is no longer the bottleneck. This confirms that the gains come from reasoning chain quality improvement (finding better paths) rather than merely earlier termination.

Critical Assessment

Does the paper demonstrate that "LRMs implicitly know when to stop thinking"?

The evidence is substantial but narrower than the phrase suggests. What the paper actually demonstrates is more specific: when the model is given an expanded exploration space guided by cumulative-path confidence (Φ), it consistently identifies reasoning chains that terminate earlier and are more likely to be correct than those found by standard sampling. The "knowing when to stop" is thus a property of the model's probability distribution—there exists a subset of high-confidence paths that reach the correct answer compactly—rather than a property of the model's default generation behavior. The distinction matters because an "implicitly knows" claim might suggest the model would stop early if only it weren't for some superficial obstacle; the reality is that the model's learned tendency to overgenerate is deeply embedded in its token-level probability distribution, and surfacing the efficiency requires a non-trivial search procedure.

The RFCS analysis (Figure 3) provides the cleanest evidence for the knowledge claim: in over half of correct responses across all tested models, the correct answer appears well before the model stops generating. This establishes that the model can produce compact reasoning—it just doesn't select those paths under standard decoding. The convergence analysis (Figure 7, Figure 15) strengthens this by showing that the efficiency gain is systematic and stable, not a lucky sampling artifact.

However, several limitations constrain the generality of this claim:

• The analysis is limited to mathematical reasoning. Whether LRMs possess analogous "stop-thinking" knowledge for other reasoning domains (code generation, logical deduction, scientific reasoning) is untested. Mathematical reasoning has a particularly clean structure (step-by-step derivations with clear intermediate states) that may make cumulative confidence particularly informative. For more open-ended reasoning tasks, the Φ signal might not separate complete from incomplete reasoning as cleanly.

• The "knowledge" is identified by a specific scoring function. The paper demonstrates that Φ (average cumulative log-probability) is effective—but doesn't investigate whether other confidence measures (entropy-based, semantic-similarity-based, learned classifiers) would reveal the same or stronger signals. The claim is really "Φ surfaces efficient reasoning capability," not necessarily "the model's implicit knowledge is uniquely captured by Φ."

• No analysis of why the knowledge exists. The paper doesn't investigate whether the efficient reasoning capability is present in the base pre-trained model or emerges during RLVR training. If it's the latter (which seems likely—the model learned to reason during RLVR, and compact reasoning chains are rewarded because they're correct), then the "implicit" knowledge is a byproduct of the training process, not a pre-existing capability.

• The convergence claim (Observation 3) is based on two models on two datasets. While the pattern is consistent across DS-1.5B and DeepScaleR on MATH-500 and AMC23, this is a thin empirical base from which to claim a universal "convergence behavior." Stronger evidence would require testing across a broader range of model scales (the gap between 1.5B and 7B is large and unexplored for this analysis) and reasoning domains.

Does SAGE "unleash" efficient reasoning capability?

Yes, with qualifications. The training-free SAGE results (Figures 4, 8; Table 1) convincingly demonstrate that cumulatively-confident search finds shorter, more accurate reasoning chains than standard sampling at the same inference budget. The TR analysis (Table 1) provides a clean mechanistic explanation: Φ-guided search's insensitivity to the tolerance parameter shows that the end-of-response signal is naturally strong on the paths SAGE selects, meaning the algorithm isn't forcing early termination but rather discovering paths that naturally terminate with high confidence.

The Degrade SAGE comparison (Figure 8) strengthens the unleashing interpretation by showing that simply terminating early (without exploration) is insufficient—the exploration mechanism is necessary to identify which reasoning paths are worth committing to. This is an important negative result: "early stopping when the model says stop" isn't enough; you need to explore multiple continuations and select the one where the model's own confidence signals align.

However, the computational cost of this unleashing is significant and underdiscussed. Figure 14(a) shows that SAGE's inference time increases with exploration width, particularly beyond $m=2$ where the growth rate accelerates. The paper's default setting of $m=2$ is chosen as "the transition point between the slow-growth and fast-growth regions"—a pragmatic compromise, not a principled optimum. For deployment scenarios where inference latency is critical, SAGE's overhead (generating $2m$ reasoning steps in parallel at each expansion, then scoring $2m^2$ candidates) may be prohibitive, and the paper's claim that SAGE "achieves the same time complexity as Degrade SAGE" (Appendix D.4) only holds with unlimited parallelism; on the paper's own 8-GPU setup, SAGE incurs higher inference-time cost.

More fundamentally, the unleashing is not free even in accuracy terms. The paper doesn't investigate whether SAGE misses correct solutions that standard sampling would find—does the Φ-guided pruning at each step ever discard reasoning paths that would have led to correct answers but had temporarily low cumulative confidence? This is the classic precision-recall tradeoff in beam search pruning, and without an analysis of false negatives (correct paths pruned by SAGE that Degrade SAGE or random sampling would have found), the claim of "unleashing" is incomplete. Table 3's comparison with beam search (Section 4, Appendix B) suggests SAGE outperforms vanilla beam search, but that comparison is based on a special accuracy metric (at least one correct answer among returned sequences) and a small 100-question subset—not a rigorous false-negative analysis.

Does SAGE-RL teach models to internalize efficient reasoning patterns?

This is the paper's strongest empirical contribution. The evidence is consistent across four models (from 1.5B to 8B parameters), six benchmarks (from MATH-500 to AIME 2025), and two base RL algorithms (GRPO and GSPO), with clear advantages over both efficient-reasoning baselines and vanilla RLVR. The key pattern—SAGE-RL simultaneously improves accuracy and reduces length while competing methods trade one for the other—is remarkably consistent and establishes that hybrid-rollout RL achieves something distinct from simply adding a length penalty.

Several specific aspects of the evidence are particularly compelling:

• The training dynamics (Figure 9) tell a clear story. SAGE-RL's greater entropy reduction (more confident, less random generations) and higher KL divergence (bigger policy changes) compared to vanilla RLVR are exactly what you'd expect if the model is learning to favor a specific subset of high-reward trajectories—the efficient ones discovered by SAGE. These curves are smooth and progressive, not erratic, suggesting stable learning without reward hacking.

• The RFCS analysis (Figure 10) provides behavioral evidence, not just metric evidence. The reduction in RFCS(<1) (fewer redundant steps after the correct answer) and increase in RFCS(avg) (stopping closer to the moment of correctness) directly shows that the model learned when to stop, not just that shorter outputs can be correct.

• The difficulty-level analysis (Figure 13) addresses a key concern. If SAGE-RL were simply making the model output shorter answers indiscriminately, performance on hard problems would suffer. Instead, SAGE-RL achieves its largest gains on levels 4–5—the hardest problems—where efficient reasoning matters most. This suggests genuine improvement in reasoning quality, not just conciseness.

• The cross-model pattern is informative. On weaker models (DS-1.5B), SAGE-RL delivers larger efficiency gains (the overthinking problem is worse). On stronger models (Qwen3-8B), it maintains accuracy while dramatically reducing length (5,640 → 3,015 tokens on MATH-500). On extensively post-trained models (DeepScaleR), gains are smaller but still positive, showing that SAGE-RL adds value even after substantial RL. No model shows degradation from SAGE-RL, which is noteworthy given how sensitive RL training can be to rollout distribution changes.

However, several concerns limit the strength of the conclusions:

The 2/8 SAGE rollout ratio is arbitrary and unanalyzed. The paper uses $r=2$ SAGE trajectories per group of $G=8$ and ablates $r=1$ vs. $r=2$ (showing similar results), but never explores what happens with $r=4$, $r=8$, or $r=0$ (SAGE during training but not at evaluation—would the model still learn efficiency?). The sensitivity analysis is too narrow to establish that 2/8 is near-optimal. The paper's argument that SAGE(2,1) and SAGE(2,2) have nearly identical entropy/KL curves suggests diminishing returns, but without testing higher $r$, we can't know whether more SAGE trajectories would accelerate learning or cause overfitting to SAGE's specific reasoning patterns.

The comparison with length-penalty baselines uses open-source implementations with potentially different training configurations. The paper states that all RLVR methods (GRPO, GSPO, SAGE-GRPO, SAGE-GSPO) use identical hyperparameters for fair comparison, but the non-RLVR baselines (LC-R1, ThinkPrune, AdaptThink, Efficient Reasoning, GRPO-LEAD) are evaluated using their released models or reported results—which may have been trained with different data, different hyperparameters, or different computational budgets. Table 4 doesn't control for total training FLOPs across method families, making the comparison somewhat apples-to-oranges. The fact that SAGE-RL consistently outperforms these baselines is suggestive but not definitive—a perfectly fair comparison would require reimplementing each baseline under the exact same training regime.

No analysis of SAGE-RL's sample efficiency. The paper trains all methods for a fixed 600 steps and reports final performance. But does SAGE-RL converge faster? Figure 9 suggests it might (the SAGE-RL curves rise more steeply early in training), but this isn't quantified. If SAGE-RL achieves its gains with fewer training steps, that's an important practical advantage; if it requires the full 600 steps to surpass baselines, the benefit is purely in final quality, not training efficiency.

The SAGE-inference cost is not amortized into the SAGE-RL training budget. SAGE decoding is more expensive than standard random sampling (Figure 14a), and this cost is borne at every training step for 2 out of 8 rollouts. The paper does not compare total training FLOPs or wall-clock time between SAGE-RL and baselines. If SAGE rollouts take 2–3× longer than random sampling rollouts, then SAGE-RL's 600-step training run may be substantially more expensive than vanilla RLVR's—a cost that should be factored into the efficiency calculation.

Missing baselines. The paper doesn't compare against several natural alternatives: (a) simply truncating all training trajectories to a fixed length and training with standard RLVR; (b) using SAGE only for data generation (distillation) rather than mixed-rollout RL; (c) using a learned early-exit classifier rather than Φ-guided search. The distillation comparison is particularly important given the paper's stated concern about self-distillation "limiting the upper boundary of reasoning capability"—this claim is asserted but never empirically tested. A direct SFT-on-SAGE-outputs baseline would either validate this concern or reveal it as unnecessary caution.

The training data is relatively small (20,000 problems) and domain-specific (mathematics). It's unclear whether SAGE-RL's benefits would persist with larger, more diverse training sets or with training beyond 600 steps. The convergence curves in Figure 9 appear to still be improving at step 600, suggesting further gains might be possible with longer training, but also raising the question of whether the SAGE-RL advantage would persist or diminish.

Overall assessment

The paper's central empirical claims hold up well: (1) Φ-guided search consistently identifies shorter, more accurate reasoning chains than standard sampling (supported by Figures 4, 8; Table 1); (2) mixing these chains into RL training produces models that are both more accurate and more concise than baselines trained with standard RLVR or explicit efficiency penalties (supported by Tables 2 and 4; Figure 9); (3) the efficiency gains come from learning to terminate reasoning at appropriate moments, not from indiscriminate shortening (supported by Figures 10 and 13). The consistency across models, benchmarks, and RL algorithms is the paper's strongest asset—patterns that replicate across this many settings are unlikely to be artifacts.

The primary weaknesses are: (1) the computational cost of SAGE decoding during training is unaccounted for, making the true efficiency comparison ambiguous; (2) the comparison with non-SAGE baselines does not control for training FLOPs or implementation details; (3) several natural baselines (SFT distillation, fixed-length truncation, learned early-exit classifiers) are missing; and (4) the hyperparameter analysis is too narrow to establish that the chosen settings ($m=2$, $r=2$, $G=8$) are near-optimal or robust to variation in training data scale, training duration, or model architecture. These limitations don't invalidate the findings but do narrow the scope of what has been conclusively demonstrated: SAGE-RL works well for mathematical reasoning with DeepSeek-family and Qwen models under the specific training configuration tested. Broader claims about "LRMs implicitly knowing when to stop" or SAGE-RL as a general technique are plausible hypotheses motivated by this work but not yet established.

6. Limitations and Trade-offs

The Difficulty Estimation Problem Is Bypassed, Not Solved

The assumption or constraint. SAGE identifies efficient reasoning chains by exploring 2m candidate continuations per reasoning step and comparing them via cumulative-path confidence (Φ). This fundamentally requires the model to have correct reasoning paths in its distribution at non-trivial probability—SAGE can surface compact correct reasoning, but it cannot create reasoning capability where none exists. The paper is transparent about this boundary: on the hardest problems (difficulty bin 5), the base model's pass@1 is near zero, and no amount of exploration helps. This is the same fundamental limitation identified in the prior-exec-summary's FLOPs-matched analysis: "test-time compute can amplify existing capability but cannot create it."

The consequence. SAGE provides no mechanism for distinguishing problems it can solve compactly from problems it cannot solve at all. On problems outside the model's capability range, SAGE will either exhaust its step budget T_max without finding r completions (triggering the fallback to top-Φ selection among remaining beams) or will accept reasoning chains that reach the end-of-response token with high confidence but wrong answers—confidence and correctness are correlated but not identical, and the paper provides no calibration analysis of this relationship. In a deployment setting, this means SAGE cannot predict its own failure: it will confidently return wrong answers on hard problems, with no uncertainty signal to trigger escalation to a larger model or human review.

What evidence exists in the paper. The difficulty-level analysis in Appendix D.3 (Figure 13) shows that SAGE-GRPO converges faster than GRPO at all difficulty levels but the pass@1 gains are concentrated on levels 4–5; on levels 1–3, performance is essentially equivalent between SAGE-GRPO and GRPO. The convergence curves in Figure 13 also show that SAGE-RL does not magically solve the hardest problems—pass@1 on level 5 remains modest—but rather improves efficiency on problems the model can already solve. The paper does not measure SAGE's false-positive rate (high-confidence wrong answers) at any difficulty level, nor does it analyze whether SAGE's confidence scores Φ correlate with answer correctness strongly enough to serve as a rejection criterion.

Mitigation status. Not addressed. The paper does not propose or evaluate any mechanism for detecting when SAGE is operating beyond the model's capability, nor does it analyze the calibration of Φ as a correctness predictor. This is a critical gap for practical deployment: without a confidence threshold or difficulty estimator, a system using SAGE would produce wrong answers with unjustified confidence on out-of-capability problems, exactly the failure mode that makes LLM deployment risky in high-stakes settings.

---

The Computational Cost of SAGE Decoding Is Not Factored Into SAGE-RL's Efficiency Gains

The assumption or constraint. SAGE decoding is more computationally expensive than standard random sampling. At each expansion step with exploration width m, SAGE generates 2m reasoning steps in parallel and scores 2m² candidate sequences with Φ, compared to standard sampling which generates exactly 1 completion per query. The paper acknowledges this cost implicitly in Appendix D.4: "under this memory-limited setting, SAGE incurs higher inference-time cost compared to Degrade SAGE." Figure 14(a) quantifies this: "once the exploration width exceeds 2, the growth rate of inference time accelerates further."

The consequence. The headline efficiency metrics in Tables 2 and 4—token efficiency (TE = pass@1 / response length) and percentage improvements over baselines—measure output efficiency (correctness per output token) but completely ignore input efficiency (the compute required to generate those tokens). SAGE-RL's training process requires running SAGE decoding on 2 out of 8 rollouts per training query, for 600 steps, on batches of 32 queries, across 8 GPUs. The total training FLOPs for SAGE-RL are strictly greater than for vanilla RLVR (same number of rollouts G=8, but some rollouts use more expensive decoding), yet the paper compares final pass@1 and response length without controlling for training cost. A practitioner choosing between SAGE-RL and simply training vanilla RLVR for more steps (to match total FLOPs) receives no guidance—the paper's fixed-600-step comparison does not establish that SAGE-RL is more FLOP-efficient than longer standard training.

Furthermore, the paper's argument that SAGE-RL-tuned models recover inference efficiency at deployment (Figure 14b, showing 28.7% reduced normalized inference time) is valid only for pass@1 inference—the model generates shorter responses, so per-query latency drops. But this gain must be amortized against the extra training cost of SAGE-RL. For a model deployed on millions of queries, the training cost is negligible; for a model trained for a specific evaluation or low-volume use case, the training overhead may dominate.

What evidence exists in the paper. Figure 14(a) reports average per-sample runtime of SAGE under varying m on the paper's 8-GPU hardware, showing clear growth with exploration width. Figure 14(b) shows normalized inference time reduction for SAGE-GRPO-tuned models at pass@1. The paper does not report: (1) total training wall-clock time for SAGE-RL vs. vanilla RLVR; (2) total training FLOPs for SAGE-RL vs. vanilla RLVR; (3) the cost of computing Φ scores during training (which requires storing and averaging per-token log-probabilities across 2m² candidate sequences per expansion step); or (4) a FLOPs-matched comparison of SAGE-RL vs. RLVR trained for proportionally more steps.

Mitigation status. Partially addressed through the inference-time analysis (Figure 14), but the training-cost gap is unexamined. The paper frames SAGE-RL as computationally lightweight relative to alternatives like GFPO (which requires oversampling) and S-GRPO/VSRM (which require repeated rollouts), but this comparison is to other efficiency methods, not to the fundamental baseline of "train standard RLVR for longer." Future work would need FLOPs-controlled experiments to establish that SAGE-RL's efficiency gains are not achievable through simpler means with equivalent compute.

---

The Training-Free SAGE and Training-Based SAGE-RL Results Are Not Directly Comparable

The assumption or constraint. The paper presents two distinct claims: (1) SAGE, a training-free decoding strategy, surfaces efficient reasoning chains at inference time (Sections 4–5); and (2) SAGE-RL, a training-based method, teaches models to generate efficient chains at pass@1 (Sections 6–7). The natural synthesis would be: SAGE-RL-tuned models, at pass@1, achieve comparable efficiency to what training-free SAGE achieves at inference time—demonstrating successful internalization. The paper never makes this comparison.

The consequence. The paper provides no evidence about whether SAGE-RL actually captures the full efficiency potential that SAGE demonstrates at inference time. Consider the DS-7B results: training-free SAGE(2,1) achieves approximately 92% pass@1 with ~2,213 tokens on MATH-500 (Table 1, under 10k token budget; note this is on a 100-question subset, not the full 500). SAGE-GRPO-tuned DS-7B achieves 93.0% pass@1 with 2,141 tokens on the full MATH-500 (Table 4). These numbers appear comparable—suggesting SAGE-RL does recover SAGE's efficiency—but the comparison conflates different evaluation sets, different token budgets, and different decoding strategies. The paper never runs training-free SAGE on the SAGE-RL evaluation protocol, or SAGE-RL-tuned models with SAGE decoding (which might be even more efficient), leaving open the question of whether SAGE-RL leaves efficiency on the table relative to what SAGE demonstrates is possible.

This gap matters for practitioners deciding between deployment strategies. If training-free SAGE at inference time achieves comparable or better efficiency than SAGE-RL-tuned pass@1, the case for expensive RL training weakens—why train when you can decode? Conversely, if SAGE-RL achieves only partial internalization (say, 70% of the length reduction that inference-time SAGE would achieve), the paper's framing of "teaching the model efficient reasoning" overstates the degree of learning.

What evidence exists in the paper. The paper runs training-free SAGE evaluation primarily on a 100-question MATH-500 subset (Figures 4, 7; Table 1; Table 3) and occasionally on AMC23 (Figure 8), using different token budgets and configurations from the full-benchmark SAGE-RL evaluation. The SAGE-RL evaluation uses standard random sampling at temperature 1.0 (not SAGE decoding) with 32,768 max tokens across the full benchmarks. These are different experimental protocols, and the paper does not provide a unified comparison table. Figure 8 compares SAGE vs. Degrade SAGE at inference time and marks random-sampling baselines as reference lines, but does not plot SAGE-RL-tuned model performance on the same axes.

Mitigation status. Not addressed. The paper treats the training-free and training-based results as separate contributions, which is reasonable for an initial paper introducing both, but the absence of head-to-head comparison is a significant gap. A practitioner cannot determine from the current results whether they should deploy SAGE at inference time, SAGE-RL for pass@1, or both together.

---

The Analysis Is Confined to Mathematical Reasoning with Two Model Families

The assumption or constraint. All experiments use mathematical reasoning benchmarks (MATH-500, AIME, AMC23, OlympiadBench, Minerva) and models from two families: DeepSeek-R1-Distill-Qwen (1.5B, 7B, and the DeepScaleR post-trained variant) and Qwen3-8B. The paper provides no evidence about SAGE or SAGE-RL on code generation, logical reasoning, scientific QA, multi-step planning, or any domain where reasoning chains have different structural properties than mathematical derivations. The paper's justification is implicit—mathematical reasoning is where overthinking is most studied and where RLVR training has been most successful—but this narrowness limits the claims' generality.

The consequence. Several aspects of SAGE's mechanism may be specific to mathematical reasoning:

• Step boundaries are well-defined. SAGE segments reasoning chains by "\n\n" delimiters, which works for mathematical CoTs where paragraphs correspond to logical steps. Code generation, dialogue, or creative writing lack this clean structure, and step-wise expansion may be inappropriate or require different segmentation strategies.

• Correctness is binary and verifiable. The paper trains SAGE-RL with a rule-based 0/1 reward derived from answer matching. For domains where correctness is graded (partial credit), multi-dimensional (code correctness + efficiency + style), or subjective (dialogue quality), the reward signal's granularity changes, and the advantage estimation that drives efficiency learning may behave differently.

• Overthinking manifests as length inflation of a specific kind. In mathematical reasoning, overthinking produces redundant verification, backtracking, and exploration of dead ends. In other domains, "overthinking" might manifest as excessive hedging, unnecessary background exposition, or socially conditioned verbosity—patterns that SAGE's Φ-guided pruning might not identify as efficiently.

• The Φ signal may be more or less informative in other domains. Mathematical reasoning chains have a clear monotonicity property (each step builds on previous steps, and confidence should accumulate as the solution progresses). For domains with non-monotonic reasoning (e.g., creative brainstorming where early steps may be exploratory and low-confidence, or debate where the model argues multiple sides before concluding), cumulative-path confidence may not separate complete from incomplete reasoning as cleanly.

What evidence exists in the paper. None. The paper does not discuss domain generalization, does not speculate about applicability to non-mathematical reasoning, and does not cite prior work suggesting that the phenomena it identifies (overthinking, Φ–ϕ divergence, efficient reasoning convergence) have been observed in other domains. This is a scope limitation acknowledged by silence rather than explicit statement.

Mitigation status. Not addressed. The paper makes no claims about other domains and provides no roadmap for extending SAGE to them. This is appropriate for a paper introducing a new method, but it means all conclusions—including the central "LRMs implicitly know when to stop thinking" claim—should be understood as "LRMs implicitly know when to stop thinking on mathematical reasoning problems of the type studied here." Whether this capability exists in other reasoning domains, with other model architectures (e.g., non-Qwen, non-DeepSeek models), or at other scales (the gap between 1.5B and 8B leaves open questions about larger models) is unknown.

---

The Hybrid Rollout Ratio (r/G) Is Underanalyzed, and the Risk of Policy Collapse Is Unexplored

The assumption or constraint. SAGE-RL uses a fixed ratio of r=2 SAGE-generated trajectories to G-r=6 randomly sampled trajectories per training group, with G=8 total. The sensitivity analysis (Appendix D.2, Table 5) explores only r=1 vs. r=2 (both at m=2) and m=1 vs. m=2 (both at r=1 or r=2). It does not explore: r=4, r=8, or r=0 during training with SAGE at evaluation; training dynamics when the ratio changes over the course of training; or whether the optimal ratio is model-dependent.

The consequence. Two distinct risks are unexamined:

Risk 1: Under-internalization (too few SAGE trajectories). If r is too small, the policy may not receive enough signal to shift its distribution toward efficient reasoning patterns. The comparison of r=1 vs. r=2 shows minimal difference, but this does not establish that r=2 is sufficient—it only shows that doubling from 1 to 2 doesn't help. The optimal r might be larger, and the paper cannot distinguish "r=2 is optimal" from "the benefit of SAGE trajectories saturates at r=1" because the function is sampled at only two points.

Risk 2: Policy collapse (too many SAGE trajectories). If r approaches G, the rollout distribution becomes dominated by SAGE trajectories, which are drawn from a systematically different part of the model's probability landscape (high-Φ, compact reasoning paths). The policy might collapse to a narrow mode where it only generates SAGE-like reasoning chains, losing the exploration diversity that enables RL to discover novel solution strategies. Figure 9 shows that SAGE-RL exhibits greater entropy reduction than vanilla RLVR—if this reduction continues or accelerates with larger r, the policy might become deterministic enough to harm generalization. The paper does not test whether SAGE-RL-tuned models maintain their performance gains when evaluated on out-of-distribution problems (different benchmarks, different difficulty levels, different reasoning styles) that were not represented in training.

The paper's stated concern about self-distillation "limiting the upper boundary of the model's reasoning capability" (Section A.2) applies equally to SAGE-RL with high r: if the policy learns to imitate SAGE trajectories too closely, it might lose access to reasoning patterns that SAGE's Φ-guided pruning discards but that are actually useful for some problems. The paper never empirically bounds this risk.

What evidence exists in the paper. Only the r=1 vs. r=2 comparison in Table 5 and Figure 12, which shows essentially overlapping entropy and KL curves. This narrow sweep cannot characterize the function performance = f(r/G). The paper's analysis of training dynamics (Figure 9) shows entropy decreasing and KL increasing under SAGE-RL—trends consistent with effective learning but also with potential overfitting. No evaluation on held-out reasoning domains or difficulty levels outside the training distribution is provided.

Mitigation status. Not addressed. The paper treats r=2, G=8 as fixed hyperparameters justified by the limited sensitivity analysis and the claim that "rollouts with similar reasoning trajectories offer minimal additional information" (Appendix D.2). This justification is speculative (it assumes both SAGE trajectories per group are similar, which may be true for m=2 but not for larger m), and the broader question of how the SAGE-to-random ratio affects policy diversity, generalization, and the risk of collapse is unexamined. A practitioner implementing SAGE-RL on a new model or domain would need to perform their own sensitivity analysis over r, m, and G to determine safe operating parameters.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a diagnostic reframing, not a paradigm shift. It doesn't propose a new training objective, a new model architecture, or a new scaling law. Instead, it changes how the field should think about the overthinking problem in reasoning models: from a behavioral flaw to be corrected through external constraints (length penalties, budgets, difficulty-based gating) to a sampling artifact that obscures a latent capability the model already possesses.

The magnitude of this reframing is significant but bounded. It doesn't invalidate prior work on efficient reasoning—methods like AdaptThink and ThinkPrune achieve real compression—but it recontextualizes them as workarounds for a selection failure rather than solutions to a capability deficit. A model that needs to be told "this is an easy problem, so use fewer tokens" is a model whose sampling mechanism isn't surfacing its own best work. SAGE demonstrates that the model's best work is already there in the probability distribution, and the primary obstacle is that standard decoding (greedy, random, beam search) evaluates paths using next-token probability ϕ rather than cumulative-path confidence Φ. This is a specific, falsifiable claim about mechanism, not just an empirical observation.

The paper resolves a tension that has been building in the efficient-reasoning literature. Prior work showed that (a) shorter responses in pass@k often outperform longer ones (Hassid et al., 2025; Shrivastava et al., 2025), yet (b) training models to be shorter via length penalties frequently degrades reasoning capability. These findings appeared contradictory: if short chains are better, why does optimizing for shortness hurt? SAGE provides the resolution: short chains are better when they're the right short chains—the ones where cumulative confidence is high and the end-of-response token is locally probable. Arbitrary short chains (produced by ϕ-guided early stopping or length-penalty training) are worse because they truncate reasoning before it's complete. The distinction between "short because the reasoning is done" and "short because we forced it to stop" is the key mechanistic insight, and it explains why prior methods face an accuracy-efficiency tradeoff that SAGE-RL avoids.

The paper also redirects research attention in two specific ways:

Toward the proposal distribution, not the reward function. Most prior work on efficient reasoning modifies the training objective—add a length penalty, set a token budget, incorporate a conciseness bonus. SAGE-RL achieves efficiency gains with a purely correctness-based reward, demonstrating that what you train on (the rollout distribution) can be as important as what you optimize for. This opens a broader research direction: hybrid-rollout RL as a general technique for transferring desirable properties from a search or decoding procedure into a model's pass@1 behavior without reward engineering. The paper shows this works for reasoning conciseness; the same principle could apply to other properties (faithfulness, uncertainty calibration, safety) where a training-time search can surface desirable behaviors that the model doesn't produce under standard sampling.

Toward confidence signals beyond next-token probability. The paper's Φ scoring function is remarkably simple—the arithmetic mean of per-token log-probabilities—yet it produces qualitatively different search behavior than standard beam search's cumulative log-probability or greedy decoding's per-token selection. This suggests that the information needed for better decoding decisions is already present in the model's outputs, just not utilized by standard algorithms. The field has focused heavily on improving model weights (through better pretraining, RLVR, distillation) while treating decoding as a solved problem (sample, rank by probability, pick the best). SAGE demonstrates there's substantial headroom in improving decoding strategies, particularly for tasks where "knowing when to stop" is central to performance.

What becomes less attractive: complex reward engineering for efficiency (the paper shows a simpler approach works better), budget-constrained optimization that requires per-problem difficulty estimation (SAGE adapts naturally without explicit difficulty labels), and offline distillation of short chains (the paper's concern about self-distillation limiting capability, while untested, raises a valid theoretical concern that hybrid-rollout RL avoids).

---

Follow-Up Research This Work Enables

Directly testing the "implicit knowledge" claim across model scales, architectures, and reasoning domains. The paper demonstrates the Φ–ϕ divergence and efficient-reasoning convergence on two model families (DeepSeek-distilled Qwen, Qwen3) at 1.5B–8B parameters on mathematical reasoning. A strong follow-up would replicate the TSearch analysis from Section 4 on: (a) larger models (DeepSeek-R1-70B, Qwen3-235B, or API-access models like o3-mini), to test whether the efficient reasoning frontier shifts with scale or saturates; (b) architecturally distinct reasoning models (e.g., models trained with different RLVR variants, or models from the Kimi-K2 or Claude reasoning families), to test whether the phenomenon is an artifact of GRPO-based training; (c) non-mathematical reasoning domains with verifiable correctness—code generation (HumanEval, LiveCodeBench) where unit tests provide binary rewards, or logical reasoning (FOLIO, ProofNet) where deduction chains have clear step structure. The key measurements would be: does Φ-guided search consistently identify shorter correct reasoning chains? Does the Φ–ϕ rank divergence at termination (Figure 5) replicate? Does the convergence behavior (Figure 7) appear with similar exploration widths? Negative results in any domain would bound the claim's generality; positive results across diverse settings would elevate it from an interesting empirical finding to a broadly applicable principle.

A FLOPs-controlled comparison of SAGE-RL against training standard RLVR for more steps. The paper compares SAGE-RL and vanilla RLVR at a fixed 600 training steps (Tables 2, 4) but does not account for the additional computational cost of SAGE decoding during training (Figure 14a). The critical unanswered question is: if you take the extra FLOPs that SAGE-RL spends on SAGE rollouts and instead use them to train vanilla RLVR for more steps (or with a larger group size G), do you recover the same efficiency gains? A clean experiment would: (a) measure the total training FLOPs for SAGE(2,2)-GRPO vs. GRPO at the paper's 600-step configuration; (b) train GRPO for proportionally more steps (or with larger batch/group size) to match total FLOPs; (c) compare pass@1, response length, and token efficiency across the full benchmark suite. If SAGE-RL still outperforms FLOPs-matched RLVR, the method has a genuine efficiency advantage. If the gap closes or reverses, SAGE-RL's benefit is primarily in wall-clock training time (achieving good results faster, not necessarily better), which is still valuable but is a different claim. The paper's current evidence cannot distinguish these scenarios.

The distillation comparison the paper avoids: SFT on SAGE-generated trajectories vs. SAGE-RL. Section A.2 argues against distilling SAGE outputs via supervised fine-tuning because "self-distillation will limit the upper boundary of the model's reasoning capability." This claim is untested. A direct experiment would: (a) generate SAGE(2,2) trajectories on the same 20,000-question training set used for SAGE-RL; (b) fine-tune the base model on these trajectories using standard SFT (next-token prediction); (c) compare the resulting model against SAGE-GRPO and GRPO on the full benchmark suite. If the SFT model achieves comparable efficiency and accuracy to SAGE-RL, the paper's RL framework is unnecessarily complex—simple distillation would suffice. If the SFT model shows degraded performance on hard problems (consistent with the "limits exploration" concern), the claim is validated and the advantage of RL-based internalization is empirically established. This experiment is straightforward to run and would directly test one of the paper's key methodological claims.

Characterizing the risk of policy collapse under high SAGE-to-random ratios. The paper uses r=2 SAGE trajectories per group of G=8 (a 25% ratio) and sweeps only r=1 vs. r=2 in the sensitivity analysis. The entropy and KL curves in Figure 12 show nearly identical behavior for r=1 and r=2, but this doesn't establish safety at higher ratios. A systematic sweep of r/G from 0 (pure RLVR) to 1.0 (pure SAGE-generated rollouts) at fixed m and G would characterize: (a) at what ratio does pass@1 on in-distribution benchmarks peak? (b) at what ratio does the model's entropy drop to a level indicating mode collapse? (c) how does generalization to held-out benchmarks (e.g., train on MATH+DAPO, evaluate on AIME) degrade as r increases? If performance peaks at some intermediate ratio and declines toward pure SAGE, the paper has identified an optimal "exploration-exploitation" balance for rollout composition. If performance monotonically improves with r (up to some saturation point), SAGE-RL can be made more aggressive. The risk assessment is particularly important for practitioners who might be tempted to maximize SAGE trajectories to accelerate efficiency learning.

Extending SAGE to learned early-exit classifiers and comparing against Φ-based search. The paper's Φ scoring uses a simple average of log-probabilities—a training-free heuristic that works well but may not be optimal. A natural extension would train a lightweight classifier (a small MLP or linear probe on top of the frozen base model's hidden states) to predict whether a partial reasoning chain, if completed, will lead to a correct answer. This classifier could replace or augment Φ in the beam selection step, potentially providing a more calibrated confidence signal. The experiment would compare: (a) SAGE with Φ scoring (current method); (b) SAGE with a learned confidence classifier trained on the same data used for SAGE-RL; (c) SAGE with both signals combined. Key measurements: pass@1, response length, and the correlation between the classifier's confidence and actual correctness (calibration). If a learned classifier outperforms Φ, it would demonstrate that the model's internal representations contain stronger completion signals than its output log-probabilities, opening a direction for representation-based reasoning control.

Stress-testing SAGE-RL's robustness to domain shift and adversarial problems. The paper evaluates on mathematical benchmarks that are thematically similar to the training data. A critical robustness test would evaluate SAGE-RL-tuned models on: (a) mathematical problems from significantly different distributions (e.g., problems requiring non-standard notation, multi-modal problems with diagrams described in text, or problems in different languages); (b) problems specifically designed to trigger overthinking (e.g., problems where an obvious-but-wrong shortcut exists, requiring the model to resist premature termination); (c) problems where the correct answer requires unusually long reasoning (to test whether SAGE-RL-tuned models have lost the ability to generate extended chains when needed). If SAGE-RL maintains its accuracy gains without reverting to overthinking on these stress tests, the learned efficiency is robust. If accuracy degrades on problems requiring extended reasoning, the efficiency learning has overshot—the model has learned a bias toward conciseness that impairs capability on problems where longer reasoning is genuinely necessary. The difficulty-level analysis in Figure 13 shows SAGE-RL maintains or improves performance on hard problems, but this is within the MATH-500 distribution; out-of-distribution testing would provide stronger evidence.

---

Practical Applications and Downstream Use Cases

Cost-efficient batch inference for mathematical reasoning at scale. For organizations running large-scale mathematical problem evaluation (e.g., automated grading systems, math tutoring platforms generating solution explanations, or RL training pipelines generating rollouts), the primary cost driver is output tokens. A SAGE-GRPO-tuned DS-7B model reduces average response length from 3,871 to 2,141 tokens on MATH-500 (Table 4)—a 44.7% reduction—while simultaneously improving pass@1 from 91.6% to 93.0%. On the harder AIME 2024, length drops from 11,305 to 6,422 tokens (43.2% reduction) with pass@1 improving from 51.9% to 55.3%. For a batch pipeline processing 100,000 queries, this translates to roughly 170 million fewer output tokens generated (on MATH-500-difficulty problems) or nearly 500 million fewer tokens (on AIME-difficulty problems), directly proportional to inference cost savings on pay-per-token APIs or reduced GPU-hours for self-hosted deployment. Critically, these savings come with improved accuracy, not a tradeoff—unlike length-penalty methods that compress at the cost of correctness. Organizations currently using best-of-N or majority voting to improve accuracy while accepting the associated length cost could instead deploy SAGE-RL-tuned models at pass@1 with both better accuracy and lower per-query token consumption.

Latency-sensitive deployments where reasoning models currently time out. Real-time applications—interactive tutoring systems that provide step-by-step math guidance, coding assistants that reason about errors before suggesting fixes, or customer support systems that need to verify logical constraints—are sensitive to both tail latency and average response time. Current LRMs like DeepSeek-R1 produce responses averaging 11,000–16,000 tokens on competition math problems (Table 4, base model lengths for DS-7B and Qwen3-8B on AIME 2024), which at typical inference speeds of 50–100 tokens/second translates to 2–5 minutes of generation time—unacceptable for interactive use. SAGE-RL-tuned DS-7B reduces AIME 2024 response length to 6,422 tokens (SAGE-GRPO) or 6,889 tokens (SAGE-GSPO), cutting generation time roughly in half while improving accuracy. On the relatively easier MATH-500, DS-7B drops from 3,871 to 2,141 tokens—approximately a 1.7-second reduction at 100 tok/s, bringing total generation time into the range acceptable for real-time interaction. The paper's analysis of normalized inference time (Figure 14b) estimates a 28.7% reduction in average latency even on the easier benchmarks; on harder benchmarks where base model lengths are longer, the absolute time savings are substantially larger.

Training data generation for self-improvement pipelines with quality-efficiency requirements. When using LRMs to generate training data for self-improvement (rejection sampling, STaR-style iterative refinement, or distillation into smaller models), both the correctness and the conciseness of generated solutions matter. Long, meandering reasoning chains contaminate the training data with verbose patterns that the student model will imitate, perpetuating the overthinking problem in the next generation. SAGE-RL-tuned models serve as superior data generators: they produce solutions that are simultaneously more likely to be correct and substantially shorter than those from standard RLVR-trained models. For DS-7B on MATH-500, SAGE-GRPO generates solutions at 93.0% pass@1 with average length 2,141 tokens, compared to base model solutions at 91.6% pass@1 with 3,871 tokens—the SAGE-RL generator produces higher-quality data using 45% fewer tokens per example. For a pipeline generating 10,000 training examples, the SAGE-RL generator saves approximately 17 million output tokens while increasing the number of correct examples from 9,160 to 9,300. These efficiency gains compound across iterations of a self-improvement loop: each generation uses fewer resources, and the student model learns from cleaner (shorter, higher-quality) demonstrations.

Deployment of reasoning capability on resource-constrained edge devices. The paper demonstrates that SAGE-RL-tuned models can match or exceed the accuracy of larger base models while using far fewer tokens. While the paper doesn't directly compare across model sizes, the token efficiency metric provides guidance: SAGE-GRPO-tuned DS-1.5B achieves token efficiency of 3.54 on AIME 2025 (26.5% pass@1 / 7,479 tokens), while the base DS-7B achieves 2.96 (37.1% pass@1 / 12,540 tokens). The 1.5B model's pass@1 is lower, but its token efficiency is higher—it extracts more correctness per token. For an edge deployment where a 7B model doesn't fit in device memory but a 1.5B model does, SAGE-RL tuning makes the smaller model substantially more capable per token of output, and the shorter response lengths (7,479 vs. 12,540 base 7B, or 2,915 vs. 4,882 base 1.5B on MATH-500) reduce the time the device spends generating, which matters for battery-powered or thermally constrained hardware. This use case is speculative—the paper doesn't evaluate on-device inference latency or memory—but the token efficiency improvements directly translate to reduced time spent in autoregressive generation, which is often the dominant cost in edge deployments.