On the Optimal Reasoning Length for RL-Trained Language Models

🎯 Pitch

Length penalties can actively destroy a model’s ability to learn reasoning from scratch, but for models that already know how to reason, the same penalties can make them more efficient—until you push too far and trigger under-thinking.

1. Executive Summary

This paper studies how output length affects the efficiency–performance tradeoff in RL-trained language models, comparing several length control methods—RLOO-LP, ALP, and DRPO—on two models with different prior reasoning capabilities: Qwen3-1.7B-Base and DeepSeek-R1-Distill-Qwen-1.5B, across four math benchmarks (AIME 2024, AIME 2025, AMC, MATH-500). The core finding is that the length–performance relationship is model-dependent: Qwen3-1.7B-Base, which must acquire reasoning from scratch during RL, shows a monotonically increasing trend where any length penalty hurts accuracy, while DeepSeek-R1-Distill, which inherits reasoning from distillation, exhibits a non-monotonic relationship with an optimal intermediate length. Extending the theoretical framework of Ghosal et al. (2025) to RL-trained policies, the paper identifies two failure modes—long outputs increase dispersion (spreading probability mass away from correct answers even though the distribution's center remains accurate), and short outputs lead to under-thinking (shifting the distribution's center away from correct answers while also increasing dispersion)—establishing that length penalties can be harmful during reasoning acquisition, whereas models with pre-existing capabilities benefit from appropriate length control only when tuned to avoid both failure regimes simultaneously.

2. Context and Motivation

The Core Problem: Length Inflation in RL-Trained Reasoning Models

The paper addresses a tension that has emerged at the center of modern LLM reasoning research: reinforcement learning dramatically improves reasoning capabilities, but it also causes models to produce excessively long chain-of-thought outputs, inflating computational costs during both training and inference. This tension is not merely an engineering inconvenience — it touches on a fundamental question about whether the reasoning process itself is becoming inefficient or whether longer outputs genuinely reflect deeper, more careful reasoning that happens to be expensive.

The problem crystallized with the release of models like OpenAI's o1 (OpenAI, 2024) and DeepSeek-R1 (DeepSeek-AI, 2025), which demonstrated that RL post-training — particularly with methods like GRPO (Shao et al., 2024) and DAPO (Yu et al., 2025) — enables models to solve problems far beyond the reach of standard instruction-tuned LLMs. However, these gains came with a steep cost: RL-trained models spontaneously learn to generate chains of thought that can span thousands or even tens of thousands of tokens per problem. This phenomenon, sometimes called overthinking (Chen et al., 2025), means that a single inference call can consume orders of magnitude more compute than a standard LLM response, straining deployment budgets and making real-time applications impractical.

The key word in the previous sentence is "spontaneously." These models are not explicitly trained to produce long outputs — the behavior emerges from the RL optimization process itself. When the only reward signal is answer correctness, the optimization discovers that allocating more tokens to reasoning increases the probability of reaching correct answers. The result is a kind of compute arbitrage: the model effectively converts inference tokens into accuracy improvements, but without any mechanism to account for the cost of those tokens. This is the gap the paper identifies: while we now have methods to penalize long outputs, we lack a principled understanding of what output length actually optimizes for the efficiency–performance tradeoff, and whether the answer depends on the model's prior capabilities.

Why This Problem Matters

The practical stakes are substantial and multi-faceted:

Training cost amplification. RL training requires sampling multiple responses per prompt (typically 16 or more in GRPO-style methods) to compute group-relative advantages. When those responses balloon to 8,000–16,000 tokens each, the computation required for a single training step grows proportionally. A training run that might take 24 GPU-hours with short responses can easily consume 576+ GPU-hours (the budget used in this paper) when outputs lengthen. For organizations training reasoning models at scale, length inflation directly multiplies infrastructure costs.

Inference cost in deployment. Even after training, the resulting model produces long outputs at inference time. A reasoning model that generates 10,000 tokens to answer a math problem costs roughly 10–100× more per query than a standard model generating a few hundred tokens. This makes reasoning models economically unviable for many production applications, particularly those requiring low latency or high throughput. The issue is compounded by the fact that many queries in practice are relatively easy and do not require extended reasoning — yet the model produces verbose outputs regardless.

The self-reinforcing nature of length increase. Several works have observed that RL training dynamics cause output length to continuously increase throughout training (the paper reproduces this in Figure 6). Without intervention, there is no natural equilibrium — the model keeps finding ways to spend more tokens for marginal accuracy gains. This creates a moving target for deployment planning: the inference cost of a model checkpoint depends on when training was stopped, not on any stable property of the learned policy.

Theoretical significance. Beyond the practical concerns, the length–performance relationship raises a deeper question: does thinking more always help? The paper by Ghosal et al. (2025), titled "Does Thinking More Always Help? Mirage of Test-Time Scaling in Reasoning Models," provides a theoretical framework suggesting the answer is no — that there exists an optimal level of extended reasoning beyond which performance degrades due to increased variance in the output distribution. However, Ghosal et al. analyze this phenomenon under test-time interventions (e.g., appending "Wait" tokens to force the model to continue generating). Whether the same non-monotonicity emerges naturally in the policies learned through RL training — and how length control methods interact with this dynamic — remained an open question until this work.

This paper's primary motivation is to close that gap: to determine whether the optimal reasoning length for RL-trained policies follows a monotonic or non-monotonic pattern, and to understand what mechanisms govern the relationship.

Prior Approaches and Their Limitations

The response to length inflation in the community has been the development of length control methods — techniques that explicitly penalize long outputs during RL training. These methods can be grouped into three families, each with distinct mechanisms and limitations:

1. Length-based reward shaping (RLOO-LP, ALP). The most direct approach is to modify the reward function to penalize longer responses. RLOO-LP (Arora & Zanette, 2025) applies a length penalty only to correct responses, normalizing by per-prompt statistics so that difficult problems (which genuinely require longer reasoning) are penalized less heavily. ALP (Xiang et al., 2025) adapts the penalty strength based on per-prompt accuracy, penalizing length more aggressively on easy problems where verbose reasoning is clearly wasteful. These methods are simple to implement but introduce a new hyperparameter — the penalty strength — that must be tuned. More importantly, they operate on a fundamentally unvalidated premise: that length and performance trade off monotonically. If the true relationship is non-monotonic, then a uniform penalty that shrinks all outputs will push performance below the optimum for some problems while potentially helping others — and the net effect may be harmful rather than helpful. The existing papers proposing these methods do not systematically investigate this assumption.

2. Length-based weighting and filtering (DRPO, GFPO). Rather than modifying the reward, these methods modify the training data or objective weighting to favor shorter outputs. DRPO (Li et al., 2025a) assigns higher weights to shorter correct responses within the DisCO framework (Li et al., 2025b), ensuring that learning signals remain positive regardless of length while biasing optimization toward concise generations. GFPO (Shrivastava et al., 2025) samples larger groups per problem and filters out long responses before training, training only on the shortest correct outputs. The filtering approach is theoretically appealing because it avoids modifying the reward landscape entirely — it simply excludes data that would otherwise incentivize verbosity. However, the paper's reproduction attempt (Appendix B.4.1) reveals a critical limitation: GFPO failed to reduce output length in their experiments, with length actually increasing compared to the no-penalty DAPO baseline (Figure 5). The authors hypothesize that filtering out long incorrect responses (which often contain useful negative signal) prevents the model from learning to suppress verbose reasoning patterns. This negative result highlights that length control mechanisms interact with training dynamics in non-obvious ways.

3. Objective normalization differences (GRPO Sample Avg vs. DAPO Token Avg). Even without explicit length penalties, the choice of loss normalization affects how length variance influences training. GRPO's "Sample Avg" normalization divides the loss by each response's individual length (1/|y|), giving equal per-token weight to short and long responses. DAPO's "Token Avg" normalization divides by the total token count across all responses in a batch, effectively up-weighting tokens from longer responses. These normalization differences are not typically presented as "length control methods," but they have a substantial effect on training dynamics: the paper shows (Appendix C.1, Table 1) that when within-prompt length variance is high — as is the case for Qwen3-1.7B-Base (CV = 1.10) — Sample Avg normalization causes unstable gradient updates because the 1/|y| weighting amplifies differences between short and long responses in the same group. This instability manifests as training runs where output length begins to decrease during training, accompanied by declining validation scores (Figure 3). The crucial point is that prior work on GRPO and DAPO did not systematically analyze how these normalization choices interact with the base model's pre-existing length distribution or reasoning capabilities.

The Unifying Gap: No Understanding of Optimal Length

Taken together, the prior work on length control shares a common blind spot: none of the existing methods are grounded in an understanding of what output length actually optimizes for performance. The implicit assumption — sometimes explicit, as in the framing of "overthinking" — is that shorter is always better for efficiency, and the only question is how much accuracy to sacrifice for conciseness. This paper challenges that assumption head-on.

Ghosal et al. (2025) provide the theoretical foundation for why the assumption might be wrong. Their stylized Gaussian model shows that the expected reward exhibits a non-monotonic relationship with policy variance: small increases in variance improve reward by better covering the reward peak, while excessive variance dilutes probability mass away from it. Empirically, they find that appending "Wait" tokens to artificially extend generation increases output entropy, and that beyond an optimal point, this entropy increase correlates with performance degradation. However, their analysis is limited to test-time interventions on a fixed policy — they manipulate generation length by changing the decoding process, not by training different policies. The critical open question, which this paper addresses, is:

When RL training itself produces policies with different output lengths (through different length penalty methods), does the resulting distribution of policies exhibit the same non-monotonic pattern? And how do the mechanisms identified by Ghosal et al. — specifically, variance-driven performance degradation — manifest in RL-trained policies?

This question matters because test-time interventions and RL training-induced length variation operate on fundamentally different time scales. A test-time intervention changes how a fixed policy samples; RL training changes the policy parameters themselves. The relationship between length and performance in trained policies therefore reflects not just sampling variance but also shifts in the policy's central tendency (what answers it prefers) and its precision (how consistently it produces those answers).

How This Paper Positions Itself

The paper explicitly frames itself as an empirical investigation into the optimal reasoning length for RL-trained language models, building on the theoretical framework of Ghosal et al. (2025) but extending it from test-time analysis to the training-time policy space. Its contributions are diagnostic rather than methodological: it does not propose a new length control method, but rather provides the first systematic comparison of existing methods that reveals why they succeed or fail depending on the model.

The paper's position can be understood as addressing three nested questions:

Descriptive: What is the empirical relationship between output length and accuracy for RL-trained policies? Is it monotonic or non-monotonic, and does the answer depend on the base model?
Mechanistic: If non-monotonicity exists, what causes it? Specifically, is the degradation at long output lengths driven by the same variance-based mechanism identified in test-time interventions, or does it reflect different dynamics unique to trained policies?
Prescriptive: Given the answers to (1) and (2), how should practitioners choose length control strategies based on their model's characteristics and goals?

The paper's experimental design directly reflects these questions. The comparison between Qwen3-1.7B-Base (which starts without reasoning capabilities) and DeepSeek-R1-Distill-Qwen-1.5B (which inherits reasoning from distillation of DeepSeek-R1) is deliberate: it tests whether the length–performance relationship depends on whether reasoning is being acquired during RL training or refined from an existing base. The use of multiple length control methods with varying penalty strengths (RLOO-LP with α ∈ [0.1, 1.0], ALP with β ∈ [1e−6, 1e−3], DRPO with λ ∈ [0.02, 0.5]) is not about finding the "best" method, but about sampling a range of points along the length–performance curve to characterize its shape.

The decomposition into mode accuracy and dispersion metrics (Figure 2) is the paper's key analytical contribution beyond Ghosal et al. While Ghosal et al. use entropy as a single summary statistic of dispersion, this paper separates two sources of error that have different implications for length control: (1) the distribution's center being wrong (low mode accuracy), which suggests the model fundamentally does not know the correct answer, and (2) the distribution being spread out (high entropy, low mode share), which suggests the model knows the correct answer but produces it inconsistently. The finding that these two failure modes dominate at opposite ends of the length spectrum — under-thinking at short lengths, dispersion at long lengths — provides a unified framework for understanding why length control requires careful tuning rather than simple maximization or minimization.

In summary, the paper positions itself as filling the critical gap between theoretical understanding (Ghosal et al.'s test-time analysis of non-monotonicity) and practical methodology (the growing toolbox of length penalty methods). Its core claim is not that any particular method is best, but that the choice and tuning of length control must be conditioned on whether the base model is acquiring reasoning or refining it — and that getting this wrong can be actively harmful, as the monotonic case (Qwen3-1.7B-Base) demonstrates.

3. Technical Approach

3.1 Reader Orientation

This paper constructs a comparative experimental framework to characterize how output length relates to reasoning accuracy in RL-trained language models, rather than proposing a new method for length control. The system being built is an analysis pipeline — a set of controlled training runs, evaluation protocols, and decomposition metrics — that reveals that the optimal reasoning length depends fundamentally on whether the base model is acquiring reasoning from scratch during RL training or refining pre-existing reasoning capabilities inherited from distillation.

3.2 Big-Picture Architecture (Diagram in Words)

The analysis pipeline has five interconnected components, arranged in a sequential workflow:

First, two base models with contrasting prior reasoning capabilities are selected: Qwen3-1.7B-Base (a pretrained model with no special reasoning distillation, representing the "acquiring reasoning from scratch" case) and DeepSeek-R1-Distill-Qwen-1.5B (distilled from DeepSeek-R1, representing the "refining existing reasoning" case).

Second, each model undergoes RL training using a modified DAPO configuration with one of several length control methods — RLOO-LP, ALP, DRPO, or baselines Sample Avg (GRPO) and Token Avg (DAPO) — each applied at multiple penalty strengths. These methods differ fundamentally in how they penalize long outputs (reward shaping, objective weighting, or normalization), producing trained policies that span a wide range of output lengths.

Third, trained policies are evaluated on four mathematical reasoning benchmarks (AIME 2024, AIME 2025, AMC, MATH-500) by sampling multiple responses per problem (64 for AIME/AMC, 16 for MATH-500) and measuring average accuracy and average output token count. Each (method, penalty strength) combination produces a single data point in the length–accuracy plane.

Fourth, the length–accuracy relationship is characterized by examining whether these data points follow a monotonic or non-monotonic pattern, separately per model and per benchmark. This answers the descriptive question: what shape does the efficiency–performance frontier take?

Fifth, for models exhibiting non-monotonicity, the accuracy is decomposed into three interpretable metrics — mode accuracy, answer entropy, and mode share — computed from the distribution of answers across the multiple samples per problem. This decomposition separates whether performance degradation stems from the model's central tendency being wrong (under-thinking) versus its distribution being too spread out (dispersion), following and extending the framework of Ghosal et al. (2025).

Information flows linearly: model selection → RL training with length control → multi-sample evaluation → length–accuracy characterization → dispersion-based decomposition. The key analytical innovation is not any single component, but the synthesis: using multiple length control methods as an "instrument" to sweep the length variable, then decomposing the resulting accuracy into distribution-level properties to identify why different length regimes succeed or fail.

3.3 Roadmap for the Deep Dive

First, the RL training setup and DAPO modifications, since this is the shared foundation for all experiments. I will explain why the authors modified DAPO's batch size and precision configurations, what instabilities these modifications address, and how these choices affect the validity of length comparisons. This is essential because the RL training dynamics themselves interact with output length — understanding the training pipeline is prerequisite to interpreting the length–accuracy curves.
Second, the length control methods, since these are the "treatments" applied to explore different output length regimes. I will detail the mathematical form of each penalty (RLOO-LP, ALP, DRPO), how it modifies the training objective, and why different methods can produce policies with similar lengths but different accuracy — a distinction that matters when interpreting whether length causes or merely correlates with performance changes.
Third, the evaluation protocol, since the metrics (accuracy and output length) define what is being optimized and measured. I will explain the multi-sample evaluation strategy, why temperature and top-p settings matter for measuring dispersion, and how aggregate statistics (mode accuracy, entropy, mode share) are computed from per-problem sample distributions. This is the bridge between raw model outputs and the interpretable length–accuracy curves.
Fourth, the decomposition framework, since this is the paper's primary analytical contribution beyond Ghosal et al. (2025). I will define each metric precisely, explain what failure mode it diagnoses, and show how comparing trends in mode accuracy vs. entropy vs. mode share across the length spectrum reveals whether degradation is driven by central tendency shifts or dispersion increases. This requires distinguishing what the model "knows" (mode accuracy) from how consistently it expresses that knowledge (mode share/entropy).
Fifth, the robustness checks, since the paper anticipates several alternative explanations for its findings. I will explain the wall-clock time comparison (ruling out that short outputs underperform merely because fewer tokens are processed during training), the extended context length evaluation (ruling out response truncation as the cause of long-output degradation), and the batch size / precision ablation experiments (ruling out that observed patterns are artifacts of training instability rather than length effects).

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical analysis paper whose core idea is that the relationship between output length and reasoning accuracy in RL-trained policies depends on whether the base model starts with prior reasoning capabilities, and that this relationship is governed by two distinct failure modes — under-thinking at short lengths and dispersion-driven degradation at long lengths — that can be diagnosed by decomposing accuracy into measures of central tendency and spread.

RL Training Setup and DAPO Modifications

Base configuration. All models are trained on the DAPO-Math-17K dataset using the verl framework (a reinforcement learning library for LLMs). The objective follows the GRPO (Group Relative Policy Optimization) algorithm with the KL penalty removed (β = 0), meaning the optimization does not constrain the policy to stay close to the reference model. This is significant because removing the KL penalty allows the policy to drift further from the reference distribution, potentially enabling larger changes in output length and reasoning style — exactly the regime where length control methods are most needed.

Group-relative advantages are computed by normalizing rewards across the 16 sampled responses per prompt, specifically by subtracting the mean reward and dividing by the standard deviation. This per-prompt normalization ensures that the advantage signal reflects how much better or worse a particular response is relative to other attempts at the same problem, rather than relative to the overall reward distribution across different problems. This is important for reasoning tasks where some problems are inherently harder (and yield lower raw rewards) — the normalization prevents the optimization from simply avoiding hard problems.

The optimizer is AdamW with a learning rate of $1 \times 10^{-6}$ , weight decay of 0.1, and 10 warmup steps. The prompt batch size is 64, with 16 responses sampled per prompt (yielding 1,024 total generations per training step). Generation uses temperature 1.0 — this is a critical design choice because high-temperature sampling during training encourages exploration of diverse reasoning strategies, which contributes to the length variance that the length control methods aim to manage. Lower temperatures would constrain the output distribution and potentially mask the length–performance relationship that the paper studies.

DAPO-style dynamic sampling is enabled to filter out prompts with zero gradient signal. This mechanism identifies prompts where all 16 sampled responses are either all correct (providing no contrastive signal) or all incorrect (also providing no contrastive signal) and excludes them from that training step. This prevents the optimization from wasting compute on batches that provide no learning signal, which is particularly important when training on hard math problems where the pass rate may be very low.

Maximum response length limits. The maximum response length is set to 8K tokens for Qwen3-1.7B-Base and 16K tokens for DeepSeek-R1-Distill-Qwen-1.5B. These limits are chosen such that fewer than 5% of rollouts exceed the cap at the start of training. This is a deliberate design choice: if the cap were too tight, a substantial fraction of responses would be truncated, preventing the model from completing its reasoning and artificially suppressing length. If the cap were too loose, the model could generate arbitrarily long outputs, consuming excessive compute and potentially diverging. The 5% threshold represents a compromise that allows natural length growth while preventing unbounded expansion. The different caps for the two models reflect their different starting behaviors: DeepSeek-R1-Distill, which already reasons at length due to distillation, requires a higher cap to accommodate its natural output distribution.

Training duration. Training is performed for approximately 576 GPU-hours (8 GPUs × 72 hours). Results are reported at 640 steps for Qwen3-1.7B-Base and 480 steps for DeepSeek-R1-Distill-Qwen-1.5B. These step counts correspond to a common wall-clock cutoff: under the 8-GPU setup, the slowest run (typically one with minimal length penalty, producing the longest outputs and therefore requiring the most generation time per step) reaches these steps within the allotted 72-hour training window. This equal-wall-clock design is important because it ensures that comparisons between methods are fair in terms of total computational investment, not just step counts. A method that produces shorter outputs might complete more steps in the same time budget — the wall-clock constraint means that the paper measures performance per budget, not per step.

Distributed training. The training uses FSDP (Fully Sharded Data Parallelism) for model parallelism and vLLM for rollout generation. FSDP shards model parameters across GPUs, enabling training of models that would not fit on a single device. vLLM is used specifically for the rollout (generation) phase because it provides high-throughput inference optimized for serving. The separation between training engine (FSDP) and rollout engine (vLLM) introduces a subtle but important complication that the paper addresses in its batch size and precision modifications.

Batch Size Configuration: Why 64/64 Beats 512/32

The paper makes a deliberate departure from DAPO's default configuration that proves critical for stable training. The default DAPO setup uses a generation prompt batch size of 512 but a PPO mini-batch size of 32, meaning that after generating 512 × 16 = 8,192 responses, training performs 512/32 = 16 gradient updates on mini-batches of 32 prompts. This is an off-policy configuration: by the time the later mini-batches are processed, the rollout data was generated by an older version of the policy (from before the earlier gradient updates), creating a mismatch between the data distribution and the current policy.

The paper observes that under this 512/32 setting, training becomes unstable: response length begins to decrease during training, accompanied by a decline in validation scores on MATH-500 and AIME 2024 (Appendix A.3, Figure 3). This is precisely the opposite of what one would expect from RL training on reasoning — normally, RL increases both length and accuracy as the model learns to reason more extensively. The declining length and accuracy suggest that the optimization is not learning reasoning but rather some degenerate behavior induced by the off-policy updates.

The authors investigate the root cause by measuring the absolute difference between token probabilities computed by the rollout engine (vLLM) and the training engine (FSDP) . Under the 512/32 setting, this probability difference increases substantially over training (Figure 3, top right), reaching values around 0.0055. This growing mismatch indicates that the rollout engine and training engine are computing different log-probabilities for the same tokens, likely due to numerical precision differences compounded by repeated gradient updates on stale rollout data.

The fix is elegantly simple: set the PPO mini-batch size equal to the generation batch size, specifically 64 prompts per batch with 16 responses each, and process all 64 prompts in a single gradient update. This eliminates off-policy updates entirely — every gradient update uses data generated by the current policy, with no intervening policy changes between rollout and training. Under this 64/64 configuration, response length consistently increases throughout training, and validation performance remains stable. The probability difference between rollout and training engines remains small (Figure 3, top right, lower curve), confirming that the mismatch is the cause of the instability.

The choice of 64 rather than 32 (a smaller batch that would also be on-policy) is driven by efficiency: 64 prompts × 16 responses = 1,024 generations per step provides sufficient data for stable gradient estimates while maximizing the utilization of the 8-GPU setup. Larger batches would improve gradient estimation but reduce the number of training steps within the fixed time budget; smaller batches would increase steps but risk noisier gradients.

Why this matters for the length–performance analysis. The batch size configuration directly affects whether length control methods are evaluated on stable or unstable training runs. If the underlying training is unstable (as in the 512/32 setting), the observed length–accuracy relationship could reflect training pathology rather than genuine properties of the learned policy. By fixing the stability issue first, the paper ensures that subsequent comparisons between length control methods reflect real differences in policy quality, not artifacts of training divergence. This is a critical but often overlooked aspect of experimental design in RL for LLMs — the paper's explicit documentation of the instability and its resolution strengthens the credibility of its empirical findings.

Precision and Truncated Importance Sampling

The choice of numerical precision and importance sampling clipping represents another departure from default configurations, and the paper shows that different models require different settings for stable training.

Qwen3-1.7B-Base: BF16 with TIS. For Qwen3-1.7B-Base, training in BF16 (Brain Floating Point 16) with Truncated Importance Sampling (TIS) successfully mitigates the probability mismatch between rollout and training engines. TIS clips the importance sampling ratio — the ratio between the current policy's probability and the rollout policy's probability for a given token — to a maximum value (the paper uses a clipping threshold of c = 2.0, though this is mentioned in Figure 4's legend rather than the main text). This clipping prevents individual tokens with large policy ratios from dominating the gradient update, which is particularly important when the rollout and training policies diverge due to numerical differences.

All experiments on Qwen3-1.7B-Base completed without instability under this configuration, so the authors did not explore FP16 alternatives due to computational resource constraints. This is a practical limitation — it is possible but unverified that FP16 with TIS would also work, and vice versa that BF16 without TIS would fail.

DeepSeek-R1-Distill-Qwen-1.5B: FP16 without TIS. The DeepSeek-R1-Distill model exhibits different stability characteristics. The authors initially attempted BF16 with TIS, but training diverged in both of two independent runs (Appendix A.4). Following Qi et al. (2025), who demonstrated that FP16 substantially reduces the probability mismatch between rollout and training engines, the authors switched to FP16 (16-bit floating point) . Figure 4 (Appendix A.4) confirms that FP16 dramatically reduces the absolute probability difference between rollout and training engines compared to BF16 — from values around 0.025–0.030 in BF16 to below 0.005 in FP16.

An ablation comparing FP16 with TIS versus FP16 without TIS showed no noticeable difference in probability difference, suggesting that the improved numerical stability of FP16 alone is sufficient, without the need for importance sampling clipping. For simplicity, the authors chose to train without TIS in FP16.

Residual instability: the ALP case. Even with FP16, training stability is not guaranteed. The paper documents a specific failure case: ALP with β = 1e−4 in FP16 without TIS exhibited instability, where the probability difference temporarily spiked to a large value, causing training to fail (Figure 4). The authors interpret this as evidence that while FP16 substantially reduces but does not eliminate the probability mismatch — under certain conditions (specific length penalty configurations interacting with the training dynamics), the mismatch can still spike and cause divergence. They restarted training from scratch with the same configuration, and the second run completed successfully, which is reported as the result. This means that the ALP β = 1e−4 data point in the length–accuracy curves represents a run that required a restart, introducing a potential selection effect (successful runs are reported; failed runs are not).

Why precision matters for the length–performance interpretation. The sensitivity to numerical precision is not merely an engineering detail — it directly affects the interpretation of length control methods. If a particular length penalty configuration destabilizes training, the resulting policy may be degenerate not because of the length penalty's effect on learning, but because the RL optimization itself failed. The paper's care in documenting which configurations required restarts and which were stable increases transparency, but the fact that some configurations are inherently more fragile than others suggests that length control methods interact with optimization stability in ways that are not yet understood. This is an implicit finding of the paper: the practical deployability of length control methods depends not only on their effect on the length–performance frontier but also on their robustness to numerical issues in distributed RL training.

Length Control Methods: The "Treatments" Applied

The paper compares five approaches to length control, spanning three distinct mechanisms: reward shaping (RLOO-LP, ALP), objective weighting (DRPO), and loss normalization (GRPO Sample Avg, DAPO Token Avg). Each method produces a different point on the length–accuracy curve by biasing the RL optimization toward shorter or longer outputs. Understanding these methods is essential because they are not interchangeable — they affect the training dynamics in fundamentally different ways, and a given output length achieved through one method may correspond to very different accuracy than the same length achieved through another method.

RLOO-LP (REINFORCE Leave-One-Out with Length Penalty)

RLOO-LP (Arora & Zanette, 2025) is the most direct length control method: it subtracts a length-based penalty from the reward for correct responses. The method builds on the RLOO advantage estimator and applies a reward shaping term.

REINFORCE Leave-One-Out advantage estimation. RLOO samples $n$ responses $\{y_1, \ldots, y_n\}$ for a prompt $x$ and estimates the advantage for each response $y_i$ as:

$\hat{A}(y_i, x) = R(y_i, x) - \frac{1}{n-1} \sum_{j \neq i} R(y_j, x)$

where $\hat{A}(y_i, x)$ is the estimated advantage for response $y_i$ , $R(y_i, x)$ is the reward for response $y_i$ , and the second term is the mean reward of all other responses (excluding $y_i$ ).

What it computes: for each response, RLOO compares its reward to the average reward of the other $n-1$ responses to the same prompt. If the response is better than average, the advantage is positive; if worse, negative. The leave-one-out construction (excluding $y_i$ from the baseline) ensures the advantage estimator is unbiased — the baseline for response $i$ is computed independently of response $i$ 's own reward, preventing a downward bias that would occur if $y_i$ were included in its own baseline.

Why this form over standard advantage estimation: standard GRPO uses the mean of all $n$ responses (including $y_i$ ) as the baseline: $\hat{A}(y_i) = R(y_i) - \frac{1}{n}\sum_j R(y_j)$ . When $n$ is small (here $n = 16$ ), including $y_i$ in its own baseline introduces a bias that shrinks the advantage estimates, reducing the effective learning signal. RLOO's leave-one-out correction removes this bias at the cost of computing $n$ separate baselines (one per response, each excluding a different element). This is more computationally expensive but produces more accurate advantage estimates, which matters when the reward function is modified (as with length penalties) because biased advantage estimates could distort the effect of the penalty.

The reward shaping function. RLOO-LP defines the reward for a response as:

$R(x, y) = \mathbb{1}\{y = y^*(x)\} \cdot (1 - \alpha \cdot f(\text{len}(y)))$

where $\mathbb{1}\{y = y^*(x)\}$ is 1 if the response's final answer matches the ground truth and 0 otherwise, $\alpha \in [0, 1)$ is a hyperparameter controlling penalty strength, and $f(\text{len}(y))$ is a normalized length score.

What it computes: if the answer is incorrect, the reward is simply 0 — no length penalty is applied to incorrect responses. If the answer is correct, the reward starts at 1.0 and is reduced by $\alpha \cdot f(\text{len}(y))$ , where $f(\text{len})$ quantifies how long the response is relative to other correct responses for the same prompt. A response of average length gets $f(\text{len}) \approx 0.5$ (the sigmoid midpoint), so with $\alpha = 0.2$ , it loses about 0.1 from the maximum reward, receiving ~0.9. A very long response gets $f(\text{len}) \approx 1$ , losing the full $\alpha$ penalty. A very short response gets $f(\text{len}) \approx 0$ , receiving the full 1.0 reward.

Why this form over uniform length penalty: penalizing only correct responses is crucial — penalizing incorrect responses for being long would create a perverse incentive to produce short incorrect answers, which are easier to generate but not useful. By confining the penalty to correct responses, RLOO-LP creates a tradeoff: the model can earn the correctness reward by being right, but among correct answers, it earns higher reward by being concise. This preserves the incentive to prioritize correctness while nudging the distribution toward shorter correct solutions.

The per-prompt length normalization function.

$f(\text{len}(y)) = \sigma\left(\frac{\text{len}(y) - \text{mean}(x)}{\text{std}(x)}\right)$

where $\sigma(\cdot)$ is the sigmoid function, $\text{len}(y)$ is the token length of response $y$ , and $\text{mean}(x)$ and $\text{std}(x)$ are the mean and standard deviation of correct response lengths for prompt $x$ , estimated online during training.

What it computes: the raw length is z-scored using per-prompt statistics of correct responses, then squashed through the sigmoid to produce a value in $(0, 1)$ . A response exactly at the mean length of correct responses for that prompt gets $f(\text{len}) = \sigma(0) = 0.5$ . A response one standard deviation longer gets $\sigma(1) \approx 0.73$ , and one standard deviation shorter gets $\sigma(-1) \approx 0.27$ .

Why this form over raw token count: without per-prompt normalization, a length penalty applied uniformly would systematically penalize the model on difficult problems (which genuinely require longer reasoning) while being lenient on easy problems. Per-prompt normalization ensures that "long" is defined relative to what is typical for that specific prompt — if a problem naturally requires 5,000 tokens to solve, a 5,000-token response is not penalized (it's at the mean), while if a problem can be solved in 500 tokens, a 5,000-token response is strongly penalized (it's many standard deviations above the mean). The online estimation of $\text{mean}(x)$ and $\text{std}(x)$ means these statistics adapt as training progresses and the model's output distribution changes.

Hyperparameters swept in the paper. The paper evaluates $\alpha \in \{0.1, 0.2, 0.4, 0.8, 1.0\}$ . When $\alpha = 0$ , the penalty vanishes and RLOO-LP reduces to standard RLVR with the RLOO advantage estimator. When $\alpha = 1.0$ , a correct response of maximum relative length receives zero reward (the same as an incorrect response), creating the strongest possible pressure toward conciseness. The range from 0.1 to 1.0 spans from very weak to very strong penalties, allowing the paper to trace out how different degrees of length pressure affect the trained policy's length and accuracy.

ALP (Adaptive Length Penalty)

ALP (Xiang et al., 2025) differs from RLOO-LP in two key ways: (1) it uses a linear rather than sigmoid-based length penalty, and (2) it adapts the penalty strength based on per-prompt accuracy, penalizing length more aggressively on easy problems where verbose reasoning is clearly unnecessary.

The reward function is:

$R(x, y) = \mathbb{1}\{y = y^*(x)\} - \beta \cdot \text{len}(y) \cdot \max\left(\text{acc}(x), \frac{1}{G}\right)$

where $\mathbb{1}\{y = y^*(x)\}$ is 1 for correct answers and 0 for incorrect ones, $\beta$ is a global hyperparameter controlling overall penalty strength, $\text{len}(y)$ is the token length of the response, and $\text{acc}(x)$ is the online estimated accuracy for prompt $x$ (the fraction of sampled responses that are correct). $G$ is the group size (16 responses per prompt in this paper), so $1/G = 1/16 \approx 0.0625$ .

What it computes: for an incorrect response, the reward is $0 - \beta \cdot \text{len}(y) \cdot \max(\text{acc}(x), 1/G)$ , which is strictly negative — incorrect responses are penalized for their length, and the penalty is larger when the problem is easy (high acc(x)). For a correct response, the reward is $1 - \beta \cdot \text{len}(y) \cdot \max(\text{acc}(x), 1/G)$ , which can be positive or negative depending on how long the response is and how easy the problem is. The $\max(\text{acc}(x), 1/G)$ term ensures a minimum penalty floor — even when accuracy is zero (no sampled responses are correct), the model still receives a penalty proportional to $\beta \cdot \text{len}(y) / G$ , preventing unbounded length inflation on problems the model cannot yet solve.

Why this form over RLOO-LP: ALP's key insight is that length penalties should be adaptive to problem difficulty. On easy problems (acc(x) close to 1.0), the model clearly knows how to solve them and verbose output is pure waste — the penalty multiplier is near 1.0, strongly compressing output length. On hard problems (acc(x) close to 0), the model is still learning and needs to explore long reasoning chains — the penalty multiplier is near $1/G$ , providing only weak length pressure that allows extended exploration. This adaptivity prevents the model from prematurely compressing its reasoning on problems it cannot yet solve compactly. RLOO-LP's per-prompt normalization provides some adaptivity (difficult problems tend to have longer mean correct lengths), but it does not explicitly condition on whether the problem is easy or hard in absolute terms.

A critical difference from RLOO-LP: ALP penalizes incorrect responses, not just correct ones. The penalty term $-\beta \cdot \text{len}(y) \cdot \max(\text{acc}(x), 1/G)$ applies to all responses. This means incorrect responses receive a strictly negative total reward (the correctness indicator is 0, and the length penalty term is positive and subtracted), while in RLOO-LP incorrect responses receive exactly 0 reward regardless of length. The effect is that ALP more aggressively discourages long incorrect responses — the model learns that producing verbose wrong answers is worse than producing concise wrong answers. This could potentially accelerate learning by forcing the model to either get the right answer or keep its incorrect attempts short, but it also risks penalizing the exploratory long chains of thought that might eventually lead to correct answers.

Hyperparameters swept in the paper. The paper evaluates $\beta \in \{1\mathrm{e}{-6}, 1\mathrm{e}{-5}, 1\mathrm{e}{-4}, 3\mathrm{e}{-4}, 1\mathrm{e}{-3}\}$ . The range spans four orders of magnitude, reflecting the sensitivity of the linear penalty to the scale of $\text{len}(y)$ — with response lengths in the thousands, even $\beta = 10^{-3}$ can produce penalty terms on the order of $10^{-3} \times 5000 \times 1.0 = 5$ , which is large relative to the correctness reward of 1. At $\beta = 10^{-6}$ , the penalty on a 5,000-token response is only 0.005, essentially negligible.

Instability note. As discussed in the precision section, ALP with $\beta = 1\mathrm{e}{-4}$ in FP16 without TIS exhibited training instability (probability difference spike), requiring a restart. This is notable because it occurred at a moderate penalty strength, not at an extreme — the interaction between ALP's reward shaping and the RL optimization appears to have narrow stability margins even under the improved FP16 configuration.

DRPO (Decoupled Reward Policy Optimization)

DRPO (Li et al., 2025a) takes a fundamentally different approach: rather than modifying the reward function, it modifies the optimization objective by assigning length-based weights to correct responses within the DisCO (Discriminative Constrained Optimization) framework (Li et al., 2025b). The key property is that these weights are normalized only among correct responses, ensuring that the learning signal for correctness remains positive regardless of length — the model always experiences a positive gradient toward any correct answer, but the strength of that gradient is larger for shorter correct answers.

DisCO objective (the base framework). DisCO defines a score function that is the average log-likelihood of a response (intuitively, how probable the model considers the response per token):

$s_\theta(o, q) = \frac{1}{|o|} \sum_{t=1}^{|o|} \log \pi_\theta(o_t \mid q, o_{<t})$

where $s_\theta(o, q)$ is the score for response $o$ given prompt $q$ , $|o|$ is the number of tokens in the response, $\pi_\theta(o_t \mid q, o_{<t})$ is the policy's probability of token $o_t$ given the prompt and previous tokens, and the sum is divided by the response length to produce a per-token average log-probability.

What it computes: the score measures the model's average confidence in the response — a high score means the model assigns high probability to each token in the response, averaged over all tokens. The length normalization (dividing by $|o|$ ) is critical: without it, longer responses would systematically have lower (more negative) average log-probabilities simply because they contain more tokens, even if the model is equally confident in each token. This parallels the tension between Sample Avg and Token Avg normalization that the paper analyzes in Appendix C.1.

The DisCO objective maximizes:

$\max_\theta \mathbb{E}_q \left[ \sum_{o \in C_q} s_\theta(o, q) - \tau \log \sum_{o' \in W_q} \exp\left(\frac{s_\theta(o', q)}{\tau}\right) \right]$

where $C_q$ is the set of correct responses for prompt $q$ , $W_q$ is the set of incorrect responses, and $\tau$ is a temperature parameter controlling the sharpness of the contrast between correct and incorrect responses.

What it computes: the first term maximizes the average score of correct responses — the model is encouraged to assign high probability to all correct answers. The second term is a $\tau$ -softmax over incorrect responses: $- \tau \log \sum_{o' \in W_q} \exp(s_\theta(o', q) / \tau)$ penalizes the model for assigning high probability to any incorrect response. As $\tau \to 0$ , this approaches a hard max over incorrect responses (the model is penalized for the single most confident incorrect answer); as $\tau \to \infty$ , it approaches the average score over all incorrect responses. This contrastive structure forces the model to not just increase probability on correct answers but also actively suppress probability on incorrect ones.

Why this form over standard policy gradient: standard policy gradient (GRPO/DAPO) uses a single advantage estimate per response and updates the policy proportionally to that advantage. If all correct responses happen to be long in a particular batch, the gradient will push toward long responses regardless of length. DisCO separates the within-correct and within-incorrect optimization: the model tries to make all correct responses more likely and all incorrect responses less likely, with the temperature $\tau$ controlling the tradeoff. This separation means that length enters the optimization only through the length-based weights (in DRPO) or not at all (in standard DisCO), rather than through the correlation between length and correctness in a particular batch.

DRPO's length-based weighting. DRPO introduces weights $\omega(o)$ for correct responses:

$\omega(o) = \exp\left(\frac{1 - |o|/C}{\lambda}\right)$

where $|o|$ is the response length, $C$ is the maximum response length (the generation cap: 8K or 16K tokens), and $\lambda > 0$ is a regularization parameter.

What it computes: the weight decays exponentially with response length. When $|o| = 0$ (degenerate, not reachable), $\omega(o) = \exp(1/\lambda)$ . When $|o| = C$ (maximum length), $\omega(o) = \exp(0) = 1.0$ . Since $\lambda > 0$ and $|o| \leq C$ , the argument $(1 - |o|/C)/\lambda$ is always non-negative, so $\omega(o) \geq 1$ . The weight is larger for shorter responses: a response of length $C/2$ gets weight $\exp(0.5/\lambda)$ , while a response of length $C$ gets weight 1.0. The weight ratio is $\exp(0.5/\lambda)$ between the half-length and full-length responses.

Why this form: the exponential weighting ensures that shorter responses receive smoothly increasing weight without ever reaching zero for any response. The normalization by $C$ (the maximum length) makes the weight independent of absolute length scale — a 4,000-token response in a model with an 8K cap gets the same weight as a 2,000-token response in a model with a 4K cap (both are at 50% of their respective caps). The regularization parameter $\lambda$ controls the sharpness of the length preference: small $\lambda$ (e.g., 0.02) produces extreme weighting where a half-length response gets $\exp(25)$ ≈ 7.2 × 10^10 times more weight; large $\lambda$ (e.g., 0.5) produces mild weighting where the same ratio is only $\exp(1)$ ≈ 2.7.

The DRPO objective incorporates these weights into the DisCO objective:

$\max_\theta \mathbb{E}_q \left[ \frac{\sum_{o \in C_q} \omega(o) \cdot s_\theta(o, q)}{\sum_{o \in C_q} \omega(o)} - \tau \log \sum_{o' \in W_q} \exp\left(\frac{s_\theta(o', q)}{\tau}\right) \right]$

What it computes: the first term is now a weighted average of correct response scores, where shorter responses receive higher weights. The denominator $\sum_{o \in C_q} \omega(o)$ normalizes the weights so they sum to 1 within the correct set. This means the optimization tries to maximize a score that is more heavily influenced by short correct responses than long correct responses — but crucially, all correct responses still receive positive learning signal (their scores appear in the weighted sum with positive weight). There is no subtraction or penalty; the model is never told "short is good, long is bad" — only "short correct is better than long correct."

Why this form over explicit reward penalties (RLOO-LP, ALP): explicit penalties create a hard tradeoff where being correct but long can have lower reward than being incorrect but short. This can create perverse optimization landscapes where the model learns to produce short wrong answers because they score higher than long correct ones. DRPO avoids this by never making correctness-negative: the weight for any correct response is positive, so being correct is always better than being incorrect regardless of length. The length preference is a "tiebreaker" among correct responses, not a penalty that can override correctness.

Hyperparameters swept in the paper. The paper evaluates $\lambda \in \{0.02, 0.05, 0.1, 0.2, 0.5\}$ . The range from 0.02 to 0.5 spans from extreme to mild length preference. As $\lambda \to \infty$ , all weights approach $\exp(0) = 1$ , recovering the standard DisCO objective with no length preference.

Interaction with length variance. The paper's Appendix C.1 identifies an important interaction: DRPO inherits DisCO's per-sample length normalization (the $1/|o|$ in the score function), making it vulnerable to the same instability as GRPO Sample Avg when within-prompt length variance is high. On Qwen3-1.7B-Base (within-prompt CV = 1.10, Table 1), DRPO exhibits large negative length bias and unstable training, similar to Sample Avg (GRPO). This means that DRPO's length-weighting mechanism, while theoretically sound, does not solve the underlying normalization instability — it operates on top of a baseline that is already fragile for models with high length variance.

GRPO Sample Avg and DAPO Token Avg (Baselines Without Explicit Penalties)

The two baseline methods — Sample Avg (GRPO) and Token Avg (DAPO) — do not apply explicit length penalties but differ in how they normalize the policy gradient loss, which implicitly affects how length variation influences training.

GRPO Sample Avg normalization. The GRPO objective (Shao et al., 2024) normalizes the loss for each response by that response's individual length:

$J_{\text{GRPO}}(\theta) = \mathbb{E}\left[ \frac{1}{G} \sum_{i=1}^{G} \frac{1}{|y_i|} \sum_{t=1}^{|y_i|} \min\left(\rho_{i,t} \hat{A}_{i,t}, \text{clip}(\rho_{i,t}, 1-\epsilon, 1+\epsilon) \hat{A}_{i,t}\right) \right]$

where $G$ is the number of responses per prompt (16), $|y_i|$ is the number of tokens in response $i$ , $\rho_{i,t}$ is the importance sampling ratio for token $t$ in response $i$ , and $\hat{A}_{i,t}$ is the advantage estimate for that token.

What it computes: the inner sum computes a per-token clipped policy gradient for each response, then divides by the response length $|y_i|$ . This means that each response contributes equally to the gradient regardless of length — a 500-token response and a 5,000-token response with the same average per-token advantage contribute the same total gradient magnitude. The outer sum averages over the $G$ responses, giving each response equal weight in the batch.

Why this form: Sample Avg is the standard sequence-level normalization from the original GRPO paper. It treats each response as one independent data point, preventing long responses from dominating the gradient due to having more tokens. However, this normalization is problematic when within-prompt length variance is high: a 500-token response gets per-token gradients scaled by 1/500 = 0.002, while a 5,000-token response gets per-token gradients scaled by 1/5000 = 0.0002. If these two responses are in the same batch (same prompt), the shorter response's gradient contribution per token is 10× larger, even if the per-token advantage estimates are similar. This makes the gradient estimate highly sensitive to which responses happen to be short, contributing to instability.

DAPO Token Avg normalization. The DAPO objective (Yu et al., 2025) normalizes by the total token count across all responses in the batch:

$J_{\text{DAPO}}(\theta) = \mathbb{E}\left[ \frac{1}{\sum_{i=1}^{G} |y_i|} \sum_{i=1}^{G} \sum_{t=1}^{|y_i|} \min\left(\rho_{i,t} \hat{A}_{i,t}, \text{clip}(\rho_{i,t}, 1-\epsilon, 1+\epsilon) \hat{A}_{i,t}\right) \right]$

What it computes: the double sum accumulates per-token gradients across all $G$ responses without per-response length normalization. The entire sum is then divided by the total number of tokens across all responses, $\sum_i |y_i|$ . This means that a 5,000-token response contributes 10× as much to the gradient as a 500-token response — longer responses receive proportionally more weight because they contain more tokens.

Why this form over Sample Avg: DAPO's Token Avg normalization is designed to prioritize longer (and typically more correct) responses. If correct responses tend to be longer than incorrect ones (which is common in reasoning tasks), Token Avg up-weights the correct responses, effectively creating an implicit correctness-aware weighting. This can accelerate learning by focusing gradient updates on the responses most likely to contain reasoning the model should emulate. However, it also means that the gradient is sensitive to the total token count, which can vary substantially across batches. The paper's Appendix C.1 shows that Token Avg exhibits lower length bias (difference between correct and incorrect response lengths) and more stable training compared to Sample Avg, particularly on Qwen3-1.7B-Base where within-prompt length variance is high (CV = 1.10).

Why this comparison matters: the contrast between Sample Avg and Token Avg is not just an engineering detail — it represents two different philosophies about how length should enter the optimization. Sample Avg treats length as a nuisance variable to be normalized away (each response gets equal weight regardless of length). Token Avg treats length as informative signal (longer responses get more weight). The paper's finding that Sample Avg is unstable on Qwen3-1.7B-Base but stable on DeepSeek-R1-Distill (which has lower length variance) suggests that the right normalization depends on the base model's output characteristics, not just the optimization algorithm.

GFPO (Group Filtered Policy Optimization): The Failed Reproduction

The paper's Appendix B.4 documents an attempt to evaluate GFPO (Shrivastava et al., 2025) that was abandoned due to inability to reproduce the method's claimed length reduction effect. This negative result is instructive for understanding the challenges of length control.

How GFPO is supposed to work. GFPO samples $G$ responses per prompt (larger than the training group) and retains only the shortest correct responses for policy updates. Specifically, for each prompt, GFPO identifies the set $C_x$ of correct responses, sorts them by length, and keeps only the $k$ shortest ones. Training then proceeds using only these filtered responses. The intuition is that by excluding long correct responses from training, the model never receives gradient signal that would push it toward verbose reasoning — only concise correct reasoning is reinforced.

The attempted reproduction. The authors implemented the simplest GFPO variant: $G = 16$ responses per prompt, retaining $k = 8$ shortest outputs for training. This was applied to both Qwen3-1.7B-Base and DeepSeek-R1-Distill-Qwen-1.5B.

What actually happened. Figure 5 (Appendix B.4.1) shows that the average output length increased throughout training for both models under GFPO, particularly in the later stages, compared to the DAPO baseline without any length penalty. The effect was even more pronounced on the validation dataset than on the training dataset. The paper notes that GFPO appears to produce shorter outputs than DAPO in early training, but clarifies this is an artifact of the logging implementation — the average length is computed after filtering for the shortest outputs, not across all sampled responses.

Why the authors think it failed. The hypothesis offered is that GFPO's filtering mechanism, which excludes long outputs from training, also prevents the model from learning from long incorrect responses. In RL training for reasoning, long incorrect responses serve an important function: they provide negative examples that teach the model what not to do. If a long incorrect response contains a promising approach that goes wrong at step 7, seeing it in training (with negative advantage) teaches the model to avoid that particular error. By filtering out all long responses — both correct and incorrect — GFPO may deprive the model of this corrective signal, allowing verbose but flawed reasoning patterns to persist and even intensify because there is no gradient signal pushing against them.

Implications for length control design. This failed reproduction — which the paper documents rather than hiding — suggests a design principle that is not obvious from the methods that succeeded: effective length control must maintain a gradient signal against long incorrect responses, not just avoid reinforcing long correct ones. RLOO-LP achieves this by leaving incorrect response rewards at zero (so they are penalized by the advantage baseline if correct responses are shorter). ALP achieves this by explicitly penalizing incorrect response length. DRPO achieves this by keeping incorrect responses in the contrastive term, where long incorrect responses are actively pushed away from. GFPO's filtering violates this principle by removing the negative signal entirely, and the empirical result is counterproductive.

Because of this failure, GFPO is excluded from the main experimental comparisons, though the paper includes the training curves in Figure 5 for transparency. This is an example of the paper's commitment to reporting negative results that inform the overall understanding of length control, even when they do not produce publishable positive findings.

Evaluation Protocol

The evaluation protocol is designed to capture not just the accuracy of the trained policies but also the distributional properties that enable the decomposition analysis in Section 3.2.2.

Benchmarks. Evaluation is performed on four mathematical reasoning benchmarks: AIME 2024, AIME 2025, AMC, and MATH-500. These vary in difficulty and problem count: AIME (American Invitational Mathematics Examination) contains very difficult competition problems with small problem sets (30 problems each for 2024 and 2025); AMC (American Mathematics Competitions) contains moderate-difficulty problems; MATH-500 (from Hendrycks et al., 2021) contains 500 competition-level problems spanning diverse topics. The paper reports results per-benchmark to show that the length–performance patterns are consistent across difficulty levels and problem domains, not an artifact of a particular test set.

Sampling strategy. Following the evaluation protocol of DeepSeek-R1 (DeepSeek-AI, 2025), the paper samples multiple responses per problem and reports mean accuracy (not pass@1). The number of samples depends on the benchmark size: 64 responses per problem for AIME 2024, AIME 2025, and AMC, which have fewer problems and therefore benefit from more samples to reduce variance; 16 responses per problem for MATH-500, which has enough problems (500) that the cross-problem averaging provides sufficient statistical power with fewer samples per problem.

Why multiple samples: single-sample evaluation (pass@1) would measure only whether the model's most likely output is correct, which conflates central tendency (where the distribution is centered) with precision (how concentrated it is). By sampling multiple responses and reporting mean accuracy (pass@k, effectively), the evaluation captures the full distribution: a model that produces the correct answer 80% of the time scores 0.8, while a model that produces the correct answer 60% of the time scores 0.6, even if both have the same single most-likely answer. This is essential for the dispersion analysis, which requires distinguishing between models that "know" the answer (high mode accuracy) but are inconsistent (low mode share) versus models that don't know the answer at all (low mode accuracy).

Generation parameters. Evaluation uses temperature 0.6 and top-p 0.95, with a context size of 32K tokens. The temperature 0.6 is lower than the training temperature (1.0), reflecting the standard practice of using lower temperature for evaluation to reduce variance and improve reliability. The top-p 0.95 provides nucleus sampling that truncates the low-probability tail of the distribution, preventing the model from sampling extremely unlikely tokens that would produce degenerate outputs. The 32K context size is large enough to accommodate even the longest responses without truncation (the training caps are 8K and 16K), though the paper later verifies this with a 64K evaluation in the robustness checks.

Prompt template consistency. The paper uses the same prompt template for evaluation as for training, ensuring that the model sees inputs in the same format it was trained on. This is a standard but important design choice: changing the prompt format between training and evaluation can cause distribution shift that degrades performance independently of the model's reasoning capabilities, which would confound the length–performance analysis.

Evaluation framework. Evaluation is conducted using the EvalScope framework with vLLM for efficient batch inference. EvalScope is a standardized evaluation harness that handles prompt formatting, answer extraction, grading, and metric computation, reducing the risk of implementation errors that could bias comparisons between methods.

Metrics reported. For each (method, penalty strength, benchmark) combination, the paper reports:

Score (accuracy): the fraction of sampled responses that are correct, averaged over problems. This is the primary performance metric.
Average output tokens: the mean number of tokens generated per response, computed across all sampled responses for all problems in the benchmark. This is the primary efficiency metric.

These two numbers place each trained policy as a single point in Figure 1's length–accuracy plane, enabling the visual characterization of whether the relationship is monotonic or non-monotonic.

The paper's primary analytical contribution beyond Ghosal et al. (2025) is the decomposition of accuracy into three interpretable metrics that separately quantify the distribution's central tendency (where it is centered) and its dispersion (how spread out it is). This decomposition is applied per problem from the multiple sampled responses, then averaged across problems within each benchmark.

The intuition. Suppose we sample 64 responses for a single math problem and observe that 40 of them give answer A (correct), 20 give answer B (incorrect), and 4 give answer C (incorrect). The accuracy is 40/64 = 0.625. But this single number conflates two facts: (1) the model's preferred answer is A, which is correct — the distribution is centered on the right answer; (2) the model is inconsistent — it only produces the correct answer 62.5% of the time. The decomposition separates these: mode accuracy tells us that the most frequent answer is correct; mode share tells us that only 62.5% of responses agree on that answer; and entropy quantifies the overall spread across all three distinct answers. A different policy might have 64/64 responses all giving A (accuracy 1.0, mode share 1.0, entropy 0.0 — perfectly centered and perfectly precise). A third policy might have 10/64 giving A (correct), with 54 responses spread across many wrong answers (accuracy 0.156, mode accuracy still correct if A is the most frequent despite having only 10 votes, low mode share, high entropy — knows the answer but very inconsistent).

The paper defines and computes three metrics from the set of $k$ sampled responses per problem (where $k = 64$ for AIME/AMC and $k = 16$ for MATH-500):

Mode accuracy. Mode accuracy is 1 if the most frequently occurring final answer among the $k$ samples is the correct answer, and 0 otherwise. It answers the question: "Does the model's best guess tend to be correct?"

Formally, for a problem with correct answer $y^*$ and sampled answers $\{y_1, \ldots, y_k\}$ , let $\text{mode}(\{y_1, \ldots, y_k\})$ be the most frequently occurring answer (ties broken arbitrarily, though the paper does not specify the tie-breaking procedure). Then:

$\text{Mode Accuracy} = \mathbb{1}\{\text{mode}(\{y_1, \ldots, y_k\}) = y^*\}$

What it measures: whether the peak of the model's output distribution is aligned with the correct answer. A model can have high mode accuracy even with low overall accuracy — if the correct answer appears 20 times and 10 different wrong answers each appear 4 times, the mode is correct (mode accuracy = 1) even though only 20/64 = 31.25% of responses are correct. This metric captures "does the model know the answer?" independent of how consistently it produces it.

Why this metric matters for the length–performance analysis: if the performance degradation at long output lengths is caused by increased dispersion (the variance-based mechanism of Ghosal et al., 2025), we would expect mode accuracy to remain high while accuracy drops — the model still "knows" the answer but produces it inconsistently. If the degradation is caused by the distribution shifting away from the correct answer (a fundamentally different failure mode), we would expect mode accuracy to drop along with accuracy — the model no longer "knows" the correct answer at all.

Answer entropy. Answer entropy quantifies the diversity of distinct final answers among the $k$ samples, treating each unique answer as a category. For a problem where the $k$ samples produce $m$ distinct answers with frequencies $p_1, p_2, \ldots, p_m$ (where $p_i$ is the fraction of samples giving answer $i$ ), the entropy is:

$H = -\sum_{i=1}^{m} p_i \log(p_i)$

What it computes: if all $k$ samples give the same answer, there is one category with $p_1 = 1.0$ , and $H = -1.0 \cdot \log(1.0) = 0$ — zero entropy, complete consensus. If the $k$ samples are evenly split among $m$ distinct answers ( $p_i = 1/m$ for each), then $H = -m \cdot (1/m) \cdot \log(1/m) = \log(m)$ , which grows with $m$ . Higher entropy means more diversity in the model's outputs — it is considering more distinct possible answers.

Why entropy over simpler metrics: entropy captures both the number of distinct answers and the evenness of their distribution. A model that produces 63 copies of answer A and 1 copy of answer B has lower entropy than one that produces 32 copies of A and 32 copies of B, even though both have 2 distinct answers. This matters because the failure mode at long output lengths might be that the model produces many different wrong answers (high entropy) or that it splits nearly evenly between the correct answer and one plausible wrong answer (medium entropy, but low accuracy). The paper uses entropy as a continuous measure that can be correlated with output length.

Mode share. Mode share is the fraction of sampled responses that match the most frequent answer:

$\text{Mode Share} = \frac{\text{count}(\text{mode})}{k}$

What it computes: if 40 out of 64 responses give the same answer (regardless of whether that answer is correct), the mode share is 40/64 = 0.625. Mode share captures how concentrated the distribution is around its peak — low mode share means the model is indecisive, producing many different answers with similar frequency. High mode share means the model consistently produces the same answer.

Why mode share complements entropy: entropy and mode share both measure dispersion but emphasize different aspects. A bimodal distribution (e.g., 32 copies of A, 32 copies of B) has high entropy (0.693) and low mode share (0.5). A flat distribution over 10 answers (6–7 copies each) also has high entropy (~2.30) and similarly low mode share (~0.1). However, the bimodal case reflects the model being torn between two plausible answers, while the flat case reflects the model having no strong preference. Mode share alone cannot distinguish these, but combined with entropy (and mode accuracy) the pattern becomes interpretable: bimodal with correct answer as one mode → knows the answer but has a competing hypothesis; flat distribution → genuine uncertainty.

Combining the metrics for diagnosis. The paper's key insight is that by plotting mode accuracy, entropy, and mode share alongside overall accuracy as functions of output length (Figure 2), two distinct failure patterns emerge:

Dispersion-driven degradation (long outputs): mode accuracy remains stable or even increases, while entropy increases and mode share decreases. Interpretation: the model's distribution is moving closer to the correct answer (mode accuracy improves) but simultaneously becoming more spread out (higher entropy, lower mode share). The model "knows" the correct answer better but produces it less consistently, and the net effect is that accuracy declines because the increased dispersion outweighs the improved central tendency. This matches the variance-based mechanism of Ghosal et al. (2025): as the policy variance increases, probability mass dilutes away from the correct answer even though the mean remains well-positioned.
Under-thinking (short outputs): mode accuracy is low, mode share is low, and entropy is high. Interpretation: the distribution is both centered away from the correct answer and highly dispersed. The model does not "know" the answer — its reasoning is insufficient to reliably identify the correct solution, and as a result it produces inconsistent guesses. Short outputs prevent the model from completing the multi-step reasoning chains needed to converge on correct answers, leading to poor central tendency and high dispersion simultaneously.

These two patterns have fundamentally different implications for length control. Dispersion-driven degradation suggests the solution is to reduce variance — either through length penalties that prevent the policy from becoming too diffuse, or through techniques that sharpen the output distribution without shortening it (e.g., lower sampling temperature). Under-thinking suggests the solution is to allow longer outputs — the model needs more tokens to complete its reasoning. The optimal length regime is the one where mode accuracy has saturated (the model consistently centers on correct answers) but entropy has not yet begun to increase substantially (the distribution remains concentrated). This is precisely the non-monotonic relationship observed for DeepSeek-R1-Distill: performance peaks at moderate output lengths where mode accuracy is high and dispersion is moderate, degrading on both sides for different reasons.

Extension beyond Ghosal et al. (2025). Ghosal et al. (2025) analyze a stylized one-dimensional Gaussian model where the policy is $\pi(y|x) = \mathcal{N}(\mu_\pi, \sigma^2_\pi)$ and the reward function is $r(y) = \mathcal{N}(\mu_r, \sigma^2_r)$ . Under this model, the expected reward is:

$\mathbb{E}_{y \sim \pi}[r(y)] = \frac{1}{\sqrt{2\pi(\sigma^2_\pi + \sigma^2_r)}} \exp\left(-\frac{(\mu_\pi - \mu_r)^2}{2(\sigma^2_\pi + \sigma^2_r)}\right)$

This expected reward exhibits a non-monotonic relationship with $\sigma^2_\pi$ : small increases in variance improve reward by better covering the reward peak (reducing the effective distance $(\mu_\pi - \mu_r)^2 / (\sigma^2_\pi + \sigma^2_r)$ ), while excessive variance dilutes probability mass away from the peak (the $1/\sqrt{\sigma^2_\pi + \sigma^2_r}$ prefactor dominates).

In this stylized model, variance $\sigma^2_\pi$ captures only dispersion — the spread of outputs around the policy mean. The policy mean $\mu_\pi$ is assumed to be optimized separately (by the RL training shifting $\mu_\pi$ toward $\mu_r$ ). Ghosal et al. empirically validate this by measuring entropy of output distributions under test-time interventions and finding that entropy increases with artificially extended generation.

The paper's key extension is to recognize that in RL-trained policies with different length penalties, both $\mu_\pi$ and $\sigma^2_\pi$ change, not just $\sigma^2_\pi$ . Different length penalty configurations train policies with different output distributions — not just different variances, but different means (what the model tends to say) and different higher-order properties. The decomposition into mode accuracy (a proxy for $\mu_\pi$ alignment) and mode share / entropy (proxies for $\sigma^2_\pi$ ) allows the paper to separately diagnose whether a length regime shift has moved the distribution's center (mode accuracy change) or its spread (entropy/mode share change). This is essential because the appropriate intervention differs: if the problem is low mode accuracy, the fix is more reasoning (longer outputs, weaker length penalties); if the problem is high dispersion, the fix is sharper concentration (shorter outputs or lower sampling temperature), but only after ensuring mode accuracy remains high.

Why the decomposition framework is the paper's core analytical contribution: without it, Figure 1 would show that length and performance relate non-monotonically for DeepSeek-R1-Distill, but not why. The decomposition answers the "why" by showing that long outputs increase dispersion while preserving central tendency, and short outputs degrade both central tendency and dispersion. This diagnostic capability is what makes the paper actionable: it tells practitioners which failure mode their model is exhibiting, and therefore which direction to move the length control — longer if mode accuracy is low, shorter if dispersion is high but mode accuracy is high. Without this decomposition, a practitioner observing sub-optimal performance would not know whether to increase or decrease the length penalty.

Robustness Checks

The paper anticipates and addresses three alternative explanations for its findings: (1) that short outputs underperform merely because they process fewer training tokens, (2) that long outputs underperform merely because they get truncated by the evaluation context limit, and (3) that the observed patterns are artifacts of the specific normalization and batch configuration choices. The robustness checks vary in formality but collectively strengthen the causal interpretation of the length–performance relationship.

Wall-Clock Time Comparison

The confound. When training with a fixed number of steps (640 for Qwen3-1.7B-Base, 480 for DeepSeek-R1-Distill), methods that produce shorter outputs complete each training step faster because they generate fewer tokens. This means that within the 576 GPU-hour budget, short-output methods may complete more training steps than long-output methods. If short-output methods underperform when compared at equal step counts (as in Figure 1), one might argue they simply haven't been trained long enough — given more steps, they might catch up.

The test. Figure 7 (Appendix D.1.1) compares all methods at equal wall-clock time (576 GPU-hours) rather than equal steps. Under this comparison, the reported step counts differ — methods with stronger length penalties complete more steps within the time budget because each step is faster.

The result. The non-monotonic relationship for DeepSeek-R1-Distill persists under the equal-wall-clock comparison. Methods with strong length penalties that produce short outputs still underperform, even though they have been trained for more steps. This rules out the hypothesis that short-output underperformance is simply a matter of insufficient training tokens — even with additional steps (and therefore more total tokens processed over the course of training), the policies trained under strong length pressure do not reach the performance of those with moderate or no length penalties.

Why this matters: it confirms that the performance degradation in the short-output regime reflects a genuine limitation of the learned policy — under-thinking — rather than an artifact of comparing policies at different stages of convergence. The model trained under strong length pressure does not simply need more time; it has converged to a different, inferior policy that produces short but incorrect reasoning.

Extended Context Length Evaluation

The confound. The evaluation uses a 32K token context window. If some responses exceed 32K tokens, they get truncated — the model's generation is cut off mid-sentence, potentially omitting the final answer or critical reasoning steps. If long-output methods produce many responses that hit this limit, their measured accuracy would be artificially low not because the model's reasoning is poor, but because the evaluation infrastructure discards the end of its output.

The test. Figure 8 (Appendix D.1.2) evaluates DeepSeek-R1-Distill with a 64K token context length, doubling the evaluation limit. If truncation were the cause of long-output degradation, extending the context should improve accuracy for the rightmost points in the length–accuracy curve, potentially eliminating or reducing the non-monotonicity.

The result. The non-monotonic pattern persists with the 64K context length. The shape of the curve is essentially unchanged from the 32K evaluation (Figure 1, bottom). Furthermore, Figure 9 (Appendix D.1.2) compares truncation rates between 32K and 64K contexts across all methods: the rates are very similar, indicating that most responses that exceed 32K tokens also exceed 64K tokens. This means truncation is not the primary driver — the model genuinely produces outputs so long that even a 64K window does not accommodate them.

Why this matters: it rules out the most straightforward alternative explanation for the right-side decline in Figure 1. The performance degradation at long output lengths is not an artifact of evaluation infrastructure — it reflects a genuine property of the model's outputs, which become less accurate even when fully observed. Combined with the dispersion analysis, the interpretation is that long outputs increase the variance of the answer distribution, causing accuracy to decline even though the central tendency (mode accuracy) remains stable or improves.

Implicit Confound Checks in the Main Results

Beyond the explicit robustness checks in the appendix, several aspects of the main experiments serve as implicit controls:

Comparing multiple length control methods: if the length–performance relationship were an artifact of a particular method's mechanism (e.g., RLOO-LP's reward shaping creates a specific distortion), we would expect different methods to show different patterns. The fact that RLOO-LP, ALP, and DRPO (despite their fundamentally different mechanisms) fall along the same non-monotonic curve for DeepSeek-R1-Distill (Figure 1, bottom) suggests that the relationship is driven by output length itself, not by method-specific artifacts.
Benchmark consistency: the non-monotonic pattern for DeepSeek-R1-Distill appears across all four benchmarks (AIME 2024, AIME 2025, AMC, MATH-500) despite their different difficulty levels, problem counts, and mathematical domains. This rules out the possibility that the pattern is specific to a particular test set or difficulty range.
Model-dependent pattern emergence: the fact that Qwen3-1.7B-Base shows a monotonic pattern while DeepSeek-R1-Distill shows a non-monotonic pattern, using the same training pipeline and evaluation protocol, serves as an internal control. If the non-monotonicity were an artifact of the experimental setup (e.g., an evaluation bug that penalizes long outputs), it would likely affect both models. The model-dependence of the pattern supports the interpretation that it reflects genuine differences in the trained policies.
Dispersion analysis cross-validation: the decomposition in Figure 2 (and extended Figure 10 in the appendix) shows that the dispersion-driven degradation pattern (stable mode accuracy, increasing entropy, decreasing mode share) is consistent across all four benchmarks for DeepSeek-R1-Distill. This cross-benchmark consistency strengthens confidence that the pattern is real rather than a statistical artifact of the small sample sizes in individual benchmarks.

These implicit checks, combined with the explicit robustness experiments, create a multi-layered validation that the paper's core findings — monotonic length–accuracy for reasoning acquisition, non-monotonic for reasoning refinement, and the two distinct failure modes — are genuine properties of RL-trained language model policies rather than artifacts of the experimental design.

4. Key Insights and Innovations

Innovation 1: The Length–Performance Frontier Depends on Whether the Model Is Acquiring or Refining Reasoning

The paper's most fundamental contribution is not a method, a metric, or a training recipe — it is the diagnostic discovery that the shape of the length–performance relationship is model-dependent in a principled, predictable way. Prior work on length control (Arora & Zanette, 2025; Xiang et al., 2025; Li et al., 2025a; Shrivastava et al., 2025) implicitly assumed a universal tension: longer outputs cost more compute, so the goal is to find a penalty strength that trades off conciseness against accuracy, uniformly across models and tasks. This paper shows that assumption is wrong — or rather, it is only conditionally right. The tradeoff curve itself changes shape depending on whether the base model enters RL training already possessing reasoning capabilities (from distillation) or must acquire them from scratch.

The empirical signature is Figure 1: Qwen3-1.7B-Base produces a monotonically increasing curve where any length penalty — any reduction in output length — hurts accuracy. DeepSeek-R1-Distill-Qwen-1.5B produces a non-monotonic curve with a clear optimum at intermediate lengths, where both stronger and weaker penalties degrade performance relative to the peak. This is not a small effect that might vanish with better hyperparameter tuning — the monotonic-vs-non-monotonic distinction is qualitative and robust across four benchmarks, multiple penalty methods, and explicit robustness checks (equal wall-clock time, extended context length). The paper demonstrates that applying length penalties to a model that is still acquiring reasoning (Qwen3-1.7B-Base) is actively harmful — it prevents the exploratory long chain-of-thought outputs that the RL process needs to discover correct reasoning strategies.

What makes this genuinely novel is that it reframes length control from an optimization problem (find the right penalty coefficient) into a diagnostic question (first determine whether your model is in the acquisition or refinement regime, then decide whether to apply length pressure at all). The distinction matters enormously in practice: a team training a reasoning model from a base pretrained checkpoint should not use length penalties until the model has converged to reasonable accuracy, while a team fine-tuning an already-capable reasoning model should tune penalties to the non-monotonic optimum. The failure mode from getting this wrong is not just suboptimal efficiency — it is convergence to a fundamentally inferior policy that never learns to reason properly.

This reframing is not incremental. Prior work treated length control methods as universal tools; this paper shows their applicability is conditioned on a property of the training setup (prior reasoning capability) that was not previously recognized as relevant. The finding that the monotonic case exists at all — that there are realistic models and tasks where length penalties should be zero — is a conceptual reversal of the prevailing "shorter is better for efficiency" assumption.

Innovation 2: Decomposing the Non-Monotonic Regime into Two Distinct Failure Modes — Dispersion vs. Under-Thinking — With Opposite Remedies

The second major contribution is the decomposition framework that explains why the non-monotonic curve has the shape it does, and — critically — provides a diagnostic toolkit that tells a practitioner which direction to move the length penalty when performance is suboptimal. This is the paper's primary analytical advance beyond Ghosal et al. (2025).

Ghosal et al. (2025) demonstrated that test-time interventions that extend generation (appending "Wait" tokens) increase output entropy, and that beyond an optimal point, this entropy-driven dispersion degrades performance — a variance-based non-monotonicity in a fixed policy's sampling distribution. Their framework unifies the right side of the curve (long outputs → high dispersion → lower accuracy) but does not address the left side, because test-time interventions only push length upward from the policy's natural output length. They have no mechanism for studying what happens when length is artificially compressed.

This paper extends that framework to the full length spectrum by using length penalties during training to sweep the length variable in both directions. The key analytical move is separating dispersion (how spread out the answer distribution is) from central tendency (whether the distribution's peak aligns with the correct answer) via three metrics computed from multi-sample evaluation: mode accuracy, answer entropy, and mode share (Figure 2). This separation reveals that the left and right sides of the non-monotonic curve are governed by fundamentally different mechanisms:

Right side (long outputs, overthinking): mode accuracy remains stable or even improves — the model's "best guess" is increasingly likely to be correct. But simultaneously, entropy increases and mode share decreases — the distribution spreads out across more distinct answers. The model knows the right answer but produces it less consistently. The net accuracy decline occurs because the dispersion effect (more wrong answers sampled) outweighs the central tendency improvement (the right answer is more likely to be the mode). This matches Ghosal et al.'s variance-based mechanism but, crucially, is shown to occur in RL-trained policies during standard generation, not just under test-time interventions.
Left side (short outputs, under-thinking): mode accuracy is low, mode share is low, and entropy is high. The distribution is both centered away from the correct answer and highly dispersed — the model does not know the answer, and its guesses are inconsistent. Short outputs prevent the model from completing the multi-step reasoning chains needed to converge on the correct solution.

The diagnostic implication is direct and actionable. If a practitioner observes suboptimal performance on a reasoning model, they compute these three metrics and diagnose the failure mode: low mode accuracy with high dispersion → under-thinking → weaken length penalties to allow longer outputs; high mode accuracy but low mode share → dispersion-driven degradation → strengthen length penalties or lower sampling temperature to sharpen the distribution. Without this decomposition, an observed decline from the peak is ambiguous — the practitioner doesn't know whether to increase or decrease the penalty, and guessing wrong pushes performance further from the optimum. With the decomposition, the direction is unambiguous.

This is more than a measurement refinement of Ghosal et al. (2025). It is a conceptual completion of the length–performance theory. Ghosal et al. explained why thinking more can hurt; this paper explains both why thinking more can hurt and why thinking less can hurt, and provides the diagnostic framework to distinguish which regime a model is in. The fact that these two failure modes have opposite remedies — one requires longer outputs, the other requires shorter outputs — makes the decomposition not just interpretable but operationally necessary.

Innovation 3: The Discovery That Effective Length Control Must Maintain Gradient Signal Against Long Incorrect Responses — A Negative Result with Design Implications

The paper's third contribution is a negative result — the failed reproduction of GFPO (Shrivastava et al., 2025) — that, combined with the success patterns of the other methods, reveals a design principle for length control that was not previously articulated: length control methods must maintain a negative gradient signal against long incorrect responses, not just avoid reinforcing long correct ones.

The logic emerges from comparing what the successful methods (RLOO-LP, ALP, DRPO) do that GFPO does not. RLOO-LP leaves incorrect response rewards at zero, so they receive negative advantage when correct responses are shorter — the indirect penalty from the advantage baseline pushes against long wrong answers. ALP explicitly penalizes incorrect response length via the $-\beta \cdot \text{len}(y) \cdot \max(\text{acc}(x), 1/G)$ term. DRPO keeps incorrect responses in the contrastive term where they are actively pushed away from, regardless of length. All three ensure that if a model produces a verbose incorrect answer, it experiences a gradient that says "don't do that."

GFPO, by contrast, filters out long responses entirely before training — both correct and incorrect. The model never sees verbose errors during training, so it never receives gradient signal that would suppress the flawed reasoning patterns that produce them. The result (Appendix B.4.1, Figure 5) is that output length increases over training compared to the no-penalty baseline, exactly the opposite of the intended effect. The paper hypothesizes that by excluding long incorrect responses, GFPO removes the negative examples that teach the model what verbose failure looks like, allowing verbose but incorrect reasoning strategies to persist and intensify.

This finding is significant because it is not obvious from the design of any individual method. RLOO-LP's authors did not motivate their design in terms of maintaining negative signal on incorrect responses; they focused on penalizing correct-response length. ALP's authors motivated their design in terms of adaptivity to problem difficulty, not negative-signal preservation. DRPO's authors motivated their design in terms of decoupling length preference from correctness weighting. It is only through the comparative lens — and specifically through the failure of GFPO, which violates the principle — that the common requirement becomes visible.

The practical implication is that future length control methods should be evaluated not just on whether they reduce output length, but on whether they do so without eliminating the learning signal from incorrect verbose outputs. A filtering-based approach that seems intuitively appealing (just don't train on long responses) is counterproductive because it removes information the model needs. This is a design constraint that the field can now incorporate into method development, and it was discovered through careful documentation of a negative result — a practice that is rare and valuable in the current publication culture.

Innovation 4: Exposing the Interaction Between Loss Normalization, Length Variance, and Training Stability as a First-Class Factor in Length Control

The paper's fourth contribution is the systematic demonstration that the choice of loss normalization — GRPO's Sample Avg (per-response normalization) versus DAPO's Token Avg (global token normalization) — interacts with the base model's output length variance to determine training stability, and that this interaction is a hidden confound in length control comparisons that prior work did not recognize.

The evidence in Appendix C.1 (Table 1, Figure 6) is straightforward but its implications are profound. Qwen3-1.7B-Base, starting from a pretrained checkpoint without reasoning capabilities, produces outputs with high within-prompt length variance (coefficient of variation = 1.10). Under Sample Avg normalization, where each response is weighted by $1/|y|$ , this variance creates large differences in per-token gradient scaling between short and long responses in the same batch — a 500-token response gets 10× the per-token weight of a 5,000-token response. The result is unstable training, declining validation scores, and the emergence of a large negative length bias (incorrect responses become longer than correct ones). Under Token Avg normalization, where all tokens are weighted equally, the instability disappears — the model trains smoothly and length bias is smaller.

DeepSeek-R1-Distill, which inherits reasoning and produces outputs with much lower within-prompt length variance (CV = 0.45), trains stably under both normalizations. The variance is low enough that the per-response weighting differences in Sample Avg do not destabilize the optimization.

This finding reframes loss normalization as a stability prerequisite for length control, not merely a modeling choice. If a team applies Sample Avg normalization to a base model with high length variance (likely any model that has not been distilled from a reasoning model), the training may fail independently of the length control method — the optimization diverges for reasons unrelated to the length penalty. Comparing length control methods under such unstable training would produce meaningless results: any observed differences could reflect training pathology rather than genuine method effects. The paper's own choice to switch from the default DAPO 512/32 configuration to 64/64 was motivated by observing exactly this instability (Appendix A.3, Figure 3), and the subsequent training stability enabled the clean length–accuracy curves in Figure 1.

The broader implication is that the base model's output characteristics — specifically, the variance of its output length distribution — should inform the choice of training configuration before length control is even attempted. This is a meta-level insight: length control research has focused on what penalty to apply, but this paper shows that whether you can apply any penalty stably depends on the interaction between the base model and the optimization algorithm's normalization scheme. This interaction was not discussed in the original GRPO, DAPO, RLOO-LP, ALP, or DRPO papers, making it a genuinely novel finding with direct practical consequences for anyone attempting to reproduce or extend length control research.

5. Experimental Analysis

Evaluation Methodology

Dataset. All experiments use the DAPO-Math-17K dataset for training, a mathematics reasoning dataset with 17,000 training problems. Evaluation is conducted on four mathematical reasoning benchmarks: AIME 2024 (30 problems), AIME 2025 (30 problems), AMC (American Mathematics Competitions, problem count not specified in the paper), and MATH-500 (500 competition-level problems from Hendrycks et al., 2021). These benchmarks span a range of difficulty from competition-level to advanced challenge problems, allowing the paper to test whether length–performance patterns generalize across difficulty levels.
Base model(s). The paper uses two models at different points in the reasoning capability spectrum: Qwen3-1.7B-Base (Qwen Team, 2025), a 1.7-billion-parameter pretrained language model with no special reasoning distillation, representing the "acquiring reasoning from scratch" case; and DeepSeek-R1-Distill-Qwen-1.5B (DeepSeek-AI, 2025), a 1.5-billion-parameter model distilled from the larger DeepSeek-R1 reasoning model, inheriting chain-of-thought reasoning capabilities before RL training begins. The models are chosen at similar parameter scales (1.5B–1.7B) to control for capacity differences, isolating the effect of prior reasoning capabilities on length–performance dynamics.
Metrics. The paper reports four primary metrics:
- Score (accuracy): the fraction of sampled responses whose final answer matches the ground truth, averaged over all problems in the benchmark. Following the DeepSeek-R1 evaluation protocol, 64 responses are sampled per problem for AIME 2024, AIME 2025, and AMC, while 16 responses are sampled per problem for MATH-500.
- Average output tokens: the mean number of tokens generated per response, computed across all sampled responses for all problems in the benchmark. This is the primary efficiency metric, and when plotted against score, it forms the length–performance frontier in Figure 1.
- Mode accuracy: for a given problem with $k$ sampled responses, mode accuracy is 1 if the most frequent final answer among the $k$ samples is the correct answer, and 0 otherwise. This metric captures whether the peak of the model's output distribution aligns with the ground truth, independent of how consistently the model produces that answer.
- Answer entropy: for a problem where the $k$ sampled responses produce $m$ distinct answers with relative frequencies $p_1, \ldots, p_m$ , the entropy is $H = -\sum_{i=1}^m p_i \log(p_i)$ . Higher entropy indicates greater diversity in the model's outputs — more distinct answers being considered.
- Mode share: the fraction of the $k$ sampled responses that match the most frequent answer, i.e., $\text{count}(\text{mode})/k$ . This metric captures how concentrated the output distribution is around its peak, with low mode share indicating indecisiveness.
The decomposition metrics (mode accuracy, entropy, mode share) are computed per problem from the multi-sample responses, then averaged across problems within each benchmark. Together, they separate two sources of error: whether the distribution is centered on the wrong answer (low mode accuracy) versus whether it is spread across many answers (high entropy, low mode share).
Baselines. The paper compares against two baseline methods that do not apply explicit length penalties but differ in their loss normalization:
- Sample Avg (GRPO) (Shao et al., 2024): the original GRPO objective that normalizes the policy gradient loss by each response's individual length (1/|y|), giving equal per-response weight regardless of token count.
- Token Avg (DAPO) (Yu et al., 2025): the DAPO objective that normalizes by the total token count across all responses in a batch, giving proportionally more weight to longer responses.
Additionally, a Pre-RL baseline is reported in Figure 1 and related figures, representing the model's performance before any RL training, shown as a single point in the length–accuracy plane.
Generation budget / compute accounting. The paper controls for total computational investment in two complementary ways:
- Equal training steps: all models are trained for a fixed number of steps (640 for Qwen3-1.7B-Base, 480 for DeepSeek-R1-Distill-Qwen-1.5B) with identical batch sizes (64 prompts × 16 responses = 1,024 generations per step). This ensures that all methods process the same number of training prompts and generate the same number of rollout responses, controlling for the "data volume" axis of compute.
- Equal wall-clock time: the paper also reports an equal-wall-clock comparison (Appendix D.1.1, Figure 7) where all models are trained for the full 576 GPU-hour budget. Since methods that produce shorter outputs complete each training step faster, the reported step counts differ under this comparison — methods with stronger length penalties complete more steps. This controls for the "time budget" axis of compute, ruling out the confound that short-output methods underperform at equal steps simply because they have processed fewer training tokens.
The paper does not measure or compare FLOPs directly between methods. Instead, output length serves as the proxy for inference cost: longer outputs require more generation compute, both during training (rollout generation) and at inference time. The length–accuracy curves in Figure 1 can be read as efficiency–performance frontiers, where moving left represents lower compute cost and moving up represents higher accuracy.
Cross-validation / statistical protocol. The paper does not employ formal cross-validation for hyperparameter selection or statistical significance testing. Instead, it uses a hyperparameter sweep design: each length control method is evaluated at multiple penalty strengths (RLOO-LP: $\alpha \in \{0.1, 0.2, 0.4, 0.8, 1.0\}$ ; ALP: $\beta \in \{1\mathrm{e}{-6}, 1\mathrm{e}{-5}, 1\mathrm{e}{-4}, 3\mathrm{e}{-4}, 1\mathrm{e}{-3}\}$ ; DRPO: $\lambda \in \{0.02, 0.05, 0.1, 0.2, 0.5\}$ ), and all resulting policies are plotted together to characterize the empirical length–performance relationship rather than to identify a single "best" configuration. This sweep-based approach means that the paper's claims about the shape of the relationship (monotonic vs. non-monotonic) are supported by the consistency of the pattern across many independently trained policies with different hyperparameters, not by statistical inference on a single comparison. The robustness of the non-monotonic pattern is further validated by its replication across four benchmarks (Figure 1, bottom) and two evaluation context lengths (32K in Figure 1, 64K in Figure 8). However, the paper does not report confidence intervals, error bars, or formal significance tests for any comparison, meaning the reliability of individual data points — as opposed to the overall shape of the curves — is not statistically quantified.

Main Quantitative Results

The Length–Performance Frontier: Monotonic vs. Non-Monotonic Patterns

The paper's central empirical result emerges from Figure 1, which plots score against average output tokens for every (method, penalty strength) combination across all four benchmarks, separately for each model. This is not a comparison of individual methods but a characterization of the frontier — the set of achievable (length, accuracy) pairs — for each model.

Qwen3-1.7B-Base (Figure 1, top row). The observed pattern is monotonically increasing: longer outputs consistently correspond to higher accuracy, with no evidence of a performance peak at intermediate lengths. Across all four benchmarks and the aggregate "Average" panel (which averages across benchmarks), the trend is clear: methods that produce shorter outputs (stronger length penalties) achieve lower scores, while methods that produce longer outputs (weaker or no length penalties) achieve higher scores.

Quantitatively, reading from the "Average" panel, the Pre-RL baseline achieves approximately 0.15 score at approximately 2,500–3,000 output tokens. The best-performing policies — Token Avg (DAPO) and RLOO-LP with weak penalties ( $\alpha = 0.1, 0.2$ ) — reach approximately 0.27–0.30 score at approximately 8,000–12,000 output tokens. In contrast, the strongest length penalty configurations (RLOO-LP $\alpha = 1.0$ , ALP $\beta = 1\mathrm{e}{-3}$ ) produce outputs in the 1,000–2,000 token range with scores around 0.04–0.10, substantially below even the Pre-RL baseline. The monotonic relationship is particularly visible in the MATH-500 panel, where Token Avg (DAPO) achieves roughly 0.63–0.65 at roughly 2,500–3,000 tokens, while the strongest ALP penalty ( $\beta = 1\mathrm{e}{-3}$ ) produces outputs under 1,000 tokens with scores near 0.40.

Critically, Sample Avg (GRPO) and DRPO appear as outliers in the Qwen3-1.7B-Base panels. These methods achieve lower scores than other methods at comparable output lengths. For example, in the AIME 2024 panel, Sample Avg (GRPO) and DRPO points cluster around 2,500–4,000 output tokens with scores of approximately 0.06–0.10, while Token Avg (DAPO) and weak RLOO-LP points at similar or slightly longer lengths achieve 0.10–0.14. The paper explains this in Appendix C.1: Sample Avg and DRPO exhibit unstable training on Qwen3-1.7B-Base due to its high within-prompt length variance (CV = 1.10), causing large negative length bias (incorrect responses tend to be longer than correct ones) and degraded performance. This means the frontier for Qwen3-1.7B-Base is not a single clean curve but a family of curves, where the maximum achievable score at a given length depends on whether the training configuration (normalization scheme, penalty type) is stable.

DeepSeek-R1-Distill-Qwen-1.5B (Figure 1, bottom row). The observed pattern is non-monotonic: there exists an optimal intermediate output length where accuracy peaks, with performance declining for both shorter and longer outputs. This pattern is consistent across all four benchmarks and the aggregate "Average" panel.

Reading from the "Average" panel, the Pre-RL baseline achieves approximately 0.44–0.46 score at approximately 2,000–3,000 output tokens (the Pre-RL point appears at the left end of the distribution, reflecting the model's naturally concise outputs before RL training). As RL training increases output length, performance improves, peaking at approximately 0.50–0.52 score at approximately 4,000–6,000 output tokens — this peak is occupied by moderate penalty configurations such as RLOO-LP $\alpha = 0.2$ , ALP $\beta = 1\mathrm{e}{-4}$ , and DRPO $\lambda = 0.1, 0.2$ . Beyond the peak, as output length continues to increase (primarily driven by methods without length penalties: Token Avg/DAPO, Sample Avg/GRPO, and very weak penalties), performance declines. At the rightmost points (approximately 10,000–15,000 output tokens), scores fall to approximately 0.44–0.48, below the peak and in some cases below the Pre-RL baseline.

The non-monotonicity varies in sharpness across benchmarks. In the MATH-500 panel, the peak is modest but visible: the best policies achieve roughly 0.82–0.83 at roughly 1,500–2,500 tokens, while the longest-output policies (Sample Avg/GRPO at approximately 3,500 tokens, DAPO at approximately 2,800 tokens) achieve roughly 0.78–0.80. The drop is approximately 2–4 percentage points from peak to rightmost point. In the AMC panel, the non-monotonicity is more pronounced: scores peak at roughly 0.64–0.66 at roughly 2,500–3,500 tokens, then decline to roughly 0.56–0.58 at roughly 6,000–8,000 tokens — a drop of roughly 6–8 percentage points. In the AIME 2024 and AIME 2025 panels, the absolute scores are lower (peaks around 0.32–0.36 and 0.24–0.26, respectively) and the non-monotonicity is still visible but compressed by the overall difficulty of these benchmarks.

Importantly, unlike Qwen3-1.7B-Base, Sample Avg (GRPO) and DRPO do not appear as outliers for DeepSeek-R1-Distill. Both methods fall along the same general trend as the other methods, consistent with the paper's Appendix C.1 analysis that DeepSeek-R1-Distill's lower within-prompt length variance (CV = 0.45) prevents the normalization-related instability that affects these methods on Qwen3-1.7B-Base. The fact that Sample Avg and DRPO produce some of the longest outputs (rightmost points in Figure 1, bottom) and achieve below-peak scores is consistent with the non-monotonic interpretation — they are in the over-thinking regime — rather than indicating training instability.

Consistency of the pattern across methods. A striking feature of Figure 1, bottom, is that points from fundamentally different methods — RLOO-LP (reward shaping), ALP (adaptive reward shaping), DRPO (objective weighting), Sample Avg (GRPO loss normalization), and Token Avg (DAPO loss normalization) — all fall along a single consistent curve in the length–accuracy plane for DeepSeek-R1-Distill. This convergence suggests that output length itself, not the specific mechanism used to control it, is the primary determinant of accuracy. If different methods produced different accuracy at the same length (e.g., if RLOO-LP systematically achieved higher accuracy than ALP at the same output length), it would indicate that length penalties distort the optimization in method-specific ways beyond simply compressing output. The fact that the methods largely overlap on the same curve strengthens the causal interpretation: length is the active variable, and the penalty method matters primarily through its effect on length, not through side effects on the policy's reasoning quality.

Decomposition Analysis: Diagnosing the Two Failure Modes

The paper's second major quantitative result is the decomposition of accuracy into mode accuracy, answer entropy, and mode share for DeepSeek-R1-Distill-Qwen-1.5B, presented in Figure 2 (AMC and MATH-500) and Figure 10 in Appendix D.2 (all four benchmarks). This analysis reveals that the non-monotonic pattern in Figure 1 (bottom) is driven by two distinct failure mechanisms operating at opposite ends of the length spectrum.

Dispersion-driven degradation at long output lengths. In the right portion of each panel in Figure 2 (points with longer average output tokens, typically corresponding to methods without length penalties or with very weak penalties), a consistent pattern emerges across benchmarks:

Mode accuracy remains stable or even increases slightly. In the AMC panel (Figure 2, second row), mode accuracy for the rightmost points is approximately 0.78–0.82, comparable to or higher than the 0.72–0.78 range at the peak. In the MATH-500 panel (fourth row), mode accuracy at the rightmost points is approximately 0.87–0.89, slightly higher than the 0.85–0.87 range at the peak.
Answer entropy increases substantially. In the AMC panel, entropy rises from approximately 1.65–1.70 at the peak to approximately 1.80–1.85 at the rightmost points. In the MATH-500 panel, entropy rises from approximately 0.55–0.60 at the peak to approximately 0.65–0.70 at the rightmost points.
Mode share decreases. In the AMC panel, mode share drops from approximately 0.67–0.68 at the peak to approximately 0.63–0.64 at the rightmost points. In the MATH-500 panel, mode share drops from approximately 0.86–0.87 at the peak to approximately 0.83–0.84 at the rightmost points.

The interpretation is clear: as outputs lengthen beyond the optimum, the model's best guess (mode) remains correct or even becomes slightly more likely to be correct, but the distribution simultaneously spreads out — the model produces a wider variety of distinct answers, reducing the consistency with which it outputs the correct one. The net effect on accuracy (score) is negative because the increased probability mass on wrong answers (due to higher dispersion) outweighs any improvement in the mode's correctness. This matches the variance-based mechanism proposed by Ghosal et al. (2025): increasing policy variance dilutes probability mass away from the correct answer even when the distribution's center is well-positioned.

Under-thinking at short output lengths. In the left portion of each panel (points with shorter average output tokens, corresponding to methods with strong length penalties), a different pattern emerges:

Mode accuracy is low. In the AMC panel, the leftmost points (strongest length penalties: RLOO-LP $\alpha = 1.0$ , ALP $\beta = 1\mathrm{e}{-3}$ ) show mode accuracy around 0.72–0.74, substantially below the 0.78–0.82 at the peak. In the MATH-500 panel, mode accuracy drops to approximately 0.85 for strong penalties, compared to 0.87–0.89 at the peak.
Answer entropy is high. In the AMC panel, entropy at the leftmost points is approximately 1.75–1.80, comparable to or higher than at the peak (1.65–1.70). In the MATH-500 panel, entropy is approximately 0.65–0.70 at the leftmost points, higher than the 0.55–0.60 at the peak.
Mode share is low. In the AMC panel, mode share drops to approximately 0.64–0.65 at the leftmost points, lower than the 0.67–0.68 at the peak. In the MATH-500 panel, mode share drops to approximately 0.83–0.84, lower than the 0.86–0.87 at the peak.

The interpretation is that short outputs cause a dual failure: the distribution's center shifts away from correct answers (low mode accuracy), and the distribution remains spread out (high entropy, low mode share). The model neither knows the answer nor produces consistent outputs. This is consistent with the "under-thinking" hypothesis: when output length is artificially compressed, the model cannot complete the multi-step reasoning chains needed to reliably arrive at correct answers, resulting in poor and inconsistent reasoning.

The peak as the intersection of two forces. The optimal intermediate length (approximately 4,000–6,000 tokens in aggregate for DeepSeek-R1-Distill) represents the point where mode accuracy has saturated — the model's reasoning is sufficiently developed to center its distribution on correct answers — but entropy has not yet begun its steep increase. To the left of this point, mode accuracy is still rising as length increases (the model is still "learning to think"); to the right, mode accuracy has plateaued (the model already "knows how to think") but dispersion is increasing (the model is "over-thinking" and considering too many alternatives). The non-monotonicity in overall accuracy (score) arises because the marginal benefit of improved central tendency diminishes as mode accuracy saturates, while the marginal harm of increased dispersion accelerates.

This decomposition is the paper's key evidentiary contribution. Without it, Figure 1 (bottom) would establish that non-monotonicity exists but not why. The decomposition provides the mechanistic explanation and, critically, the diagnostic framework for determining whether a given suboptimal policy needs more length (if mode accuracy is low) or less length (if mode accuracy is high but dispersion is high).

Training Dynamics: Length Bias and Stability

The paper's Appendix C.1 (Figure 6, Table 1) provides quantitative evidence on training dynamics that explains the performance differences between normalization schemes and between models.

Table 1: Coefficient of variation of output lengths (first 10 training steps, Token Avg).

Model	Overall CV	Within-prompt CV	Between-prompt CV
Qwen3-1.7B-Base	1.53	1.10	0.45
DeepSeek-R1-Distill-1.5B	0.68	0.45	0.50

The key statistic is the within-prompt CV: 1.10 for Qwen3-1.7B-Base versus 0.45 for DeepSeek-R1-Distill-Qwen-1.5B. This means that for a given prompt, Qwen3-1.7B-Base produces responses whose lengths vary by more than their mean on average — the standard deviation of lengths within a prompt group exceeds the mean length. Under Sample Avg normalization, where each response is weighted by $1/|y|$ , this variance causes per-token gradient magnitudes to differ by factors of 10× or more between short and long responses in the same batch, destabilizing training. DeepSeek-R1-Distill's within-prompt CV of 0.45 is less than half of Qwen3-1.7B-Base's, explaining why Sample Avg is stable on DeepSeek-R1-Distill but not on Qwen3-1.7B-Base.

Length bias evolution (Figure 6). The "Normalized Length Bias" panels in Figure 6 show the difference between mean lengths of correct and incorrect responses, normalized by the overall mean length: $(\text{correct} - \text{wrong}) / \text{mean}$ . A negative value indicates that incorrect responses tend to be longer than correct ones — the model is spending more tokens on wrong answers.

For Qwen3-1.7B-Base (Figure 6, top right), Sample Avg (GRPO) and DRPO exhibit large negative length bias that grows over training, reaching approximately −0.8 by step 200 and remaining deeply negative through step 600. This means incorrect responses are roughly 80% longer than correct ones on average, relative to the overall mean. Token Avg (DAPO), RLOO-LP ( $\alpha = 0.2$ ), and ALP ( $\beta = 1\mathrm{e}{-4}$ ) show much smaller length bias, ranging from approximately −0.4 to 0.0, with Token Avg maintaining the most neutral bias.

For DeepSeek-R1-Distill-Qwen-1.5B (Figure 6, bottom right), all methods show relatively small length bias (approximately −0.6 to −0.45), and the bias is more stable over training. Sample Avg and DRPO do not diverge substantially from the other methods, consistent with their stable training on this model.

The implication is that the poor performance of Sample Avg and DRPO on Qwen3-1.7B-Base (visible in Figure 1, top) is directly linked to their large negative length bias: the model learns to produce verbose incorrect answers while being concise on correct ones, a pathological pattern that both degrades accuracy and wastes compute on wrong reasoning.

Ablation Studies and Robustness Checks

Wall-clock time comparison (training budget): One might hypothesize that short-output methods underperform in Figure 1 simply because a fixed step count provides fewer total training tokens for shorter outputs. Figure 7 (Appendix D.1.1) compares all DeepSeek-R1-Distill methods at equal wall-clock time (576 GPU-hours) rather than equal steps. The non-monotonic pattern persists across all four benchmarks, with methods producing short outputs (strong length penalties) still underperforming despite having completed more training steps. For example, in the aggregate "Average" panel, the leftmost points still cluster below 0.45 score despite having received more training steps than methods at the peak (~0.50–0.52). This rules out insufficient training as the explanation for short-output underperformance.
Extended evaluation context length (truncation confound): A competing explanation for the right-side decline in Figure 1 (bottom) is that long outputs exceed the 32K evaluation context limit, causing truncation that artificially lowers accuracy. Figure 8 (Appendix D.1.2) replicates the DeepSeek-R1-Distill length–accuracy curves with a 64K context length. The non-monotonic pattern remains essentially unchanged — the rightmost points still show accuracy below the peak, with scores approximately 0.44–0.48 at 10,000–15,000 tokens compared to ~0.50–0.52 at the peak. Figure 9 further shows that truncation rates at 32K and 64K are very similar across methods, indicating that most truncated responses exceed even the 64K limit. Truncation is therefore not the driver of long-output degradation.
Training stability under different batch size configurations: Appendix A.3 (Figure 3) documents the discovery that DAPO's default 512/32 batch configuration (512 prompt batch size, 32 mini-batch size) causes training instability for Qwen3-1.7B-Base, manifesting as decreasing response length and declining validation scores. Switching to a 64/64 configuration (both batch sizes equal) eliminates off-policy updates and stabilizes training. This ablation was performed before the main experiments, ensuring that the length–performance patterns reported in Figure 1 reflect stable training rather than optimization pathology. The 512/32 instability was not observed in the original DAPO paper (which used different models and tasks), making this a novel finding about the interaction between batch configuration and base model characteristics.
Precision and truncated importance sampling (TIS) for DeepSeek-R1-Distill: Appendix A.4 (Figure 4) compares three configurations: BF16 with TIS, FP16 with TIS, and FP16 without TIS. BF16 with TIS diverged in both of two independent runs for DeepSeek-R1-Distill. FP16 dramatically reduced the probability mismatch between rollout and training engines (from ~0.025–0.030 to below 0.005 absolute difference), enabling stable training. FP16 with and without TIS showed no noticeable difference in probability mismatch, so FP16 without TIS was used for all DeepSeek-R1-Distill experiments. However, one configuration — ALP with $\beta = 1\mathrm{e}{-4}$ in FP16 without TIS — still exhibited instability, requiring a restart (the successful second run is reported). This documents that even under the improved FP16 configuration, certain length penalty strength and method combinations can destabilize training, though the instability is not deterministic (the second run of the same configuration succeeded).
GFPO reproduction failure: Appendix B.4.1 (Figure 5) documents the attempt to evaluate GFPO (Shrivastava et al., 2025) with $G = 16$ , $k = 8$ . On both Qwen3-1.7B-Base and DeepSeek-R1-Distill-Qwen-1.5B, GFPO increased output length compared to the DAPO baseline, particularly in later training stages. Training output length under GFPO rose to approximately 8,000 tokens for Qwen3-1.7B-Base and above 16,000 tokens for DeepSeek-R1-Distill by the end of training, exceeding the DAPO baseline in both cases. This negative result — the method does not produce the length reduction claimed in the original work — is informative: it suggests that filtering out long responses (both correct and incorrect) removes negative gradient signal against verbose errors, allowing verbose patterns to intensify.
Effect of loss normalization on length bias (implicit ablation): While not presented as a formal ablation, the comparison between Sample Avg (GRPO) and Token Avg (DAPO) across the two models (Figure 1, Figure 6, Table 1) constitutes a de facto ablation of the normalization scheme's interaction with base model length variance. The finding is clear: Sample Avg is unstable on Qwen3-1.7B-Base (high length variance) but stable on DeepSeek-R1-Distill (low length variance), while Token Avg is stable on both. This interaction was not previously documented and has direct practical implications for choosing training configurations based on the base model's output characteristics.

Critical Assessment

Claim 1: "The relationship between output length and performance differs qualitatively across models: Qwen3-1.7B-Base shows a monotonically increasing trend, while DeepSeek-R1-Distill exhibits a non-monotonic relationship with an optimal intermediate length."

What was tested. Figure 1 presents length–accuracy scatter plots for both models across four benchmarks, with each point representing a different (method, penalty strength) configuration. For Qwen3-1.7B-Base, the visual trend is clearly upward-sloping — longer outputs are associated with higher accuracy, with no visible inflection point or decline at the right end. For DeepSeek-R1-Distill, the visual trend shows a peak at intermediate lengths and lower accuracy at both shorter and longer outputs.

Does this support the claim? Partially, with important qualifications about what "monotonically increasing" means in context.

For DeepSeek-R1-Distill, the evidence for non-monotonicity is reasonably convincing. The pattern appears across all four benchmarks (Figure 1, bottom) and persists under robustness checks (equal wall-clock time in Figure 7, extended context in Figure 8). The peak-to-rightmost decline is modest — roughly 2–8 percentage points depending on the benchmark — but it is consistent across data points from different methods, making it unlikely to be noise. The decomposition analysis (Figure 2) provides a coherent mechanistic explanation (dispersion-driven degradation) that aligns with prior theoretical work (Ghosal et al., 2025), adding plausibility.

For Qwen3-1.7B-Base, the claim of monotonicity is harder to assess. The key issue is that the rightmost points in Qwen3-1.7B-Base's Figure 1 panels correspond to the longest outputs the training pipeline produced — roughly 8,000–12,000 tokens in aggregate. The claim that the relationship is "monotonically increasing" is based on observing no decline within this range. But this is fundamentally different from demonstrating that the relationship would remain monotonic if outputs could reach, say, 15,000 or 20,000 tokens. If the same dispersion-driven degradation mechanism operates in all models, Qwen3-1.7B-Base would also eventually exhibit non-monotonicity — it simply has not reached that regime because its base output lengths (without distillation-induced reasoning) are naturally shorter, and the training budget (576 GPU-hours) may not have been sufficient to push it into the over-thinking regime.

The paper's implicit argument is that Qwen3-1.7B-Base is still in the "acquiring reasoning" phase where longer outputs are strictly beneficial, and that the monotonic pattern reflects this. But this is an interpretation of a truncated curve, not a demonstrated asymptotic property. The training runs for Qwen3-1.7B-Base with the weakest length penalties (Token Avg/DAPO, RLOO-LP $\alpha = 0.1$ ) produce outputs around 8,000–12,000 tokens — these are the rightmost points. We cannot know from the data whether extending training further would push these points rightward along a continued upward slope or whether they would eventually curve downward. The paper does not report training curves showing whether output length and accuracy for these methods had plateaued by step 640 or were still increasing, which would help distinguish "still rising" from "converged to a monotonic frontier."

A fairer characterization of the evidence would be: within the length range explored (up to ~12,000 tokens for Qwen3-1.7B-Base), the relationship appears monotonic, while for DeepSeek-R1-Distill, the explored range (up to ~15,000 tokens) contains the non-monotonic peak. This is consistent with the acquisition-vs-refinement hypothesis, but does not prove that Qwen3-1.7B-Base would remain monotonic at arbitrary lengths.

Claim 2: "Long outputs increase dispersion, and short outputs lead to under-thinking."

What was tested. Figure 2 (and extended Figure 10) decomposes accuracy into mode accuracy, entropy, and mode share for DeepSeek-R1-Distill, showing that mode accuracy remains stable at long lengths while entropy increases and mode share decreases (dispersion), and that all three metrics degrade at short lengths (under-thinking).

Does this support the claim? The dispersion claim is well-supported for DeepSeek-R1-Distill. The pattern — stable/increasing mode accuracy coupled with increasing entropy and decreasing mode share — is visible in all four benchmarks (Figure 10) and is consistent with the Ghosal et al. (2025) framework. The magnitude of the entropy increase is modest (e.g., from ~1.65 to ~1.85 on AMC, from ~0.55 to ~0.70 on MATH-500) but directionally consistent across all points in the long-output regime.

The under-thinking claim is supported but with thinner evidence. At the short-output end, the number of data points with substantially compressed lengths is small — only the strongest penalty configurations (RLOO-LP $\alpha = 1.0$ , ALP $\beta = 1\mathrm{e}{-3}$ , DRPO $\lambda = 0.02$ ) produce outputs in the 1,000–2,000 token range for DeepSeek-R1-Distill. The under-thinking diagnosis (low mode accuracy, high entropy, low mode share) is based on these few configurations. This is partly inherent to the experimental design: the length penalties sweep a range, and the extreme values naturally have fewer configurations near them. But it means the characterization of the "short-output regime" relies on a small number of data points, making it more sensitive to method-specific artifacts (e.g., perhaps ALP $\beta = 1\mathrm{e}{-3}$ degrades performance through a mechanism other than pure length compression, and a different method achieving the same length would perform better).

Additionally, the under-thinking decomposition is only performed for DeepSeek-R1-Distill. For Qwen3-1.7B-Base, the monotonic relationship means that all short-output configurations perform poorly, but the paper does not present mode accuracy / entropy / mode share decompositions that would confirm whether this poor performance is also "under-thinking" (low mode accuracy + high dispersion) or a different failure mode. Since Qwen3-1.7B-Base is hypothesized to be in the reasoning acquisition phase, its short-output failures might have a different signature — e.g., mode accuracy might be low but entropy might also be low (the model is confidently wrong), which would indicate a different mechanism than the DeepSeek-R1-Distill short-output regime. The paper does not check this, so the "under-thinking" label for Qwen3-1.7B-Base's poor short-output performance remains an interpretation rather than a demonstrated finding.

Claim 3: "Length penalties may hinder reasoning acquisition, while properly tuned length control can improve efficiency for models with strong prior reasoning."

What was tested. The claim has two parts: (1) length penalties are harmful when the model is acquiring reasoning (Qwen3-1.7B-Base), and (2) length penalties can help when applied in moderation to a model with prior reasoning (DeepSeek-R1-Distill). For (1), Figure 1 (top) shows that any method producing shorter outputs than the no-penalty baseline (Token Avg/DAPO) achieves lower accuracy. For (2), Figure 1 (bottom) shows that moderate penalties (RLOO-LP $\alpha = 0.2$ , ALP $\beta = 1\mathrm{e}{-4}$ ) achieve higher accuracy than no-penalty methods while producing shorter outputs, improving efficiency.

Does this support the claim? For part (1), the evidence is strong but the causal interpretation is not airtight. Figure 1 (top) clearly shows that shorter outputs correlate with lower accuracy for Qwen3-1.7B-Base. However, the claim that length penalties hinder reasoning acquisition implies a causal mechanism: the penalties prevent the exploratory long chain-of-thought that RL needs to discover reasoning strategies. The paper does not directly test this mechanism — it observes the correlation and offers a plausible interpretation. An alternative interpretation is that the length penalties on Qwen3-1.7B-Base produce shorter outputs and degrade performance through independent mechanisms (e.g., the reward shaping distorts the optimization landscape in ways that harm learning irrespective of length), and the correlation between length and accuracy is spurious. The fact that different penalty methods (RLOO-LP, ALP, DRPO) with different mechanisms all show the same trend weakens this alternative — if the degradation were method-specific, we would expect different methods at the same length to show different accuracy — but does not eliminate it, since all methods share the property of penalizing long outputs.

The stronger test would be to train Qwen3-1.7B-Base with no length penalty until it reaches reasonable accuracy (matching or exceeding DeepSeek-R1-Distill's starting accuracy), and then apply length penalties to see whether the relationship transitions from monotonic to non-monotonic. This would directly demonstrate that the monotonic-vs-non-monotonic distinction is driven by reasoning maturity rather than by some fixed property of the model architecture. The paper does not perform this experiment, leaving the acquisition-vs-refinement interpretation as a plausible but unverified hypothesis.

For part (2), the evidence is convincing within the constraints of the experimental design. Figure 1 (bottom) clearly shows that moderate length penalties achieve higher accuracy than no-penalty methods at substantially shorter output lengths. For the aggregate "Average" panel, the peak at ~0.50–0.52 score at ~4,000–6,000 tokens represents both higher accuracy and shorter outputs than the no-penalty methods at ~0.44–0.48 score at ~10,000–15,000 tokens. This is a genuine efficiency gain: the moderate-penalty policies achieve better performance with less compute.

However, the paper does not quantify the efficiency gain in standardized terms (e.g., "4× fewer tokens for equivalent accuracy"), and the gains are benchmark-dependent. On MATH-500, the peak (~0.82–0.83 at ~1,500–2,500 tokens) represents only a modest improvement over no-penalty methods (~0.78–0.80 at ~2,500–3,500 tokens) — roughly 30–40% fewer tokens for a 2–5 percentage point accuracy gain. On AMC, the gains are larger (6–8 percentage points at roughly half the token count). The paper does not report whether these differences are statistically significant or whether the optimal penalty strength generalizes across benchmarks.

General Strengths of the Experimental Design

Multi-method convergence. The strongest feature of the experimental design is that multiple fundamentally different length control methods (reward shaping, objective weighting, loss normalization) produce points that lie along the same length–accuracy curve for DeepSeek-R1-Distill. This convergence across method types substantially strengthens the interpretation that length itself, not method-specific side effects, is the primary determinant of accuracy. If the paper had tested only one length control method, it would be unclear whether the observed pattern reflected the method's mechanism or the length–performance relationship itself.

Transparency about training instabilities. The paper's documentation of the GFPO reproduction failure (Appendix B.4.1), the batch size instability (Appendix A.3), the precision sensitivity (Appendix A.4), and the ALP restart case demonstrates an unusual commitment to reporting what went wrong alongside what worked. These negative results contain genuinely useful information about the fragility of RL training for reasoning and the design constraints on length control methods.

Decomposition framework. The mode accuracy / entropy / mode share decomposition provides a mechanistic explanation for the non-monotonic pattern that goes beyond curve-fitting. It shows not just that accuracy declines at long lengths, but what kind of errors cause the decline. This is the kind of analysis that turns an empirical observation into an actionable diagnostic.

Weaknesses and Limitations

Small number of data points at extremes. For DeepSeek-R1-Distill, the short-output regime is characterized by only a handful of configurations (roughly 3–5 points per benchmark with lengths below 3,000 tokens). The long-output regime is better populated (7–10 points), but the "dispersion-driven degradation" interpretation relies on comparing the rightmost points to the peak — a comparison of two clusters, not a continuous trend with dense sampling. More granular penalty strengths (especially at the strong-penalty end) would strengthen confidence in the shape of the curve.

No statistical quantification. The paper reports no confidence intervals, no error bars, no significance tests, and no measure of variance across training runs with different random seeds. Individual data points in Figure 1 represent single training runs. The ALP $\beta = 1\mathrm{e}{-4}$ case demonstrates that the same configuration can fail on one run and succeed on another (Appendix A.4), indicating that run-to-run variance exists. Without error bars or multiple seeds, we cannot assess how much of the scatter in Figure 1 represents genuine differences between methods versus random variation in training outcomes. This is particularly concerning for claims about the precise location of the optimal length and the magnitude of the accuracy decline at long outputs.

Single training budget. All experiments use a fixed training budget of 576 GPU-hours. The paper argues that the non-monotonic pattern in DeepSeek-R1-Distill is not an artifact of insufficient training (Figure 7), but the converse question is not addressed: would the monotonic pattern in Qwen3-1.7B-Base become non-monotonic with substantially more training? If reasoning acquisition eventually saturates and the model enters a refinement phase, the length–performance relationship should transition from monotonic to non-monotonic. Testing this would require training Qwen3-1.7B-Base for substantially longer (perhaps 2,000+ steps) and observing whether a peak emerges. The fixed budget limits the paper's ability to distinguish "this model will always show monotonic behavior" from "this model shows monotonic behavior within the explored training horizon."

The GFPO reproduction was minimal. The paper tested only one GFPO configuration ( $G = 16$ , $k = 8$ ) and observed that length increased. It is possible that different GFPO hyperparameters (larger $G$ , different $k$ , different filtering criteria) would successfully reduce length. The original GFPO paper (Shrivastava et al., 2025) may have used configurations or base models where the filtering mechanism works as intended. The paper's conclusion that GFPO fails "because it removes negative signal from long incorrect responses" is a hypothesis, not a demonstrated mechanism — it could equally be that the specific $G = 16$ , $k = 8$ configuration was poorly chosen, or that GFPO interacts badly with the DAPO training setup in ways unrelated to negative signal preservation.

No decomposition for Qwen3-1.7B-Base. The paper's central diagnostic framework (mode accuracy, entropy, mode share) is only applied to DeepSeek-R1-Distill in the main text (Figure 2) and appendix (Figure 10). For Qwen3-1.7B-Base, the length–performance relationship is characterized only by the aggregate score in Figure 1 (top). This is a missed opportunity: applying the same decomposition to Qwen3-1.7B-Base would reveal whether its short-output failures have the same signature as DeepSeek-R1-Distill's under-thinking (low mode accuracy + high dispersion) or a different signature, which would strengthen or weaken the acquisition-vs-refinement interpretation. The paper's narrative implies that Qwen3-1.7B-Base is still "acquiring reasoning" and thus suffers from a different type of short-output failure than DeepSeek-R1-Distill's under-thinking, but this is not empirically demonstrated.

Single domain (mathematical reasoning). All experiments use mathematical reasoning benchmarks. The paper's claims about "reasoning acquisition" and "optimal reasoning length" are implicitly claims about mathematical reasoning specifically, not reasoning in general. Mathematical reasoning has distinctive properties — answers are discrete and verifiable, reasoning chains follow logical deduction, and longer outputs often correspond to more explicit step-by-step derivations — that may not generalize to other reasoning domains (code generation, logical reasoning, scientific explanation, debate). The paper does not discuss domain specificity as a limitation.

No latency or wall-clock analysis at inference. The paper uses "average output tokens" as the sole efficiency metric, which is a proxy for FLOPs but ignores latency. Sequential generation cannot be parallelized, so a policy producing 4,000 tokens takes roughly half the wall-clock time of one producing 8,000 tokens. But the paper does not report inference latency, and all policies are compared on the basis of token count rather than time-to-answer. For deployment scenarios where latency matters (interactive assistants, real-time systems), the optimal length in tokens may not correspond to the optimal length in user-perceived time if there are overhead costs that are independent of output length (prompt processing, model loading, etc.).

Missing Experiments That Would Strengthen the Paper

Multiple training seeds per configuration. Running even 2–3 seeds for the key configurations (Token Avg/DAPO, a moderate-penalty method, and a strong-penalty method) would provide error bars on the length–accuracy curves and reveal whether the non-monotonic pattern is robust to training stochasticity. The ALP instability case (Appendix A.4) demonstrates that training outcomes can vary across runs.
Extended training for Qwen3-1.7B-Base. Training Qwen3-1.7B-Base with no length penalty beyond 640 steps (perhaps to 1,500–2,000 steps) would test whether the monotonic pattern persists or whether a non-monotonic peak eventually emerges as reasoning capabilities saturate. This would directly test the acquisition-vs-refinement hypothesis.
Decomposition analysis for Qwen3-1.7B-Base. Applying the mode accuracy / entropy / mode share metrics to Qwen3-1.7B-Base would reveal whether its failure modes match DeepSeek-R1-Distill's under-thinking signature or differ, providing evidence for or against the paper's narrative about reasoning acquisition.
A length-neutral baseline that varies sampling temperature. The paper controls length through training-time penalties that shape the policy. An alternative approach is to train a single policy (e.g., Token Avg/DAPO) and vary length at inference time through generation parameters (temperature, top-p, max tokens). If the same non-monotonic pattern emerged from inference-time length variation, it would suggest that the relationship is a property of the learned policy's internal structure, not a training artifact. If it did not, it would suggest that training-time penalties produce qualitatively different policies at different lengths, not just the same policy compressed.
A model with intermediate prior reasoning. The paper compares two extremes: a base pretrained model (no reasoning) and a distilled reasoning model (strong prior reasoning). A model with moderate prior reasoning — e.g., a base model that has undergone some instruction tuning on reasoning tasks but not full distillation — would help establish whether the transition from monotonic to non-monotonic is gradual or sharp, and whether there is a threshold level of prior reasoning below which length penalties are always harmful.
Correlation between length and step count in reasoning chains. The paper interprets output length as a proxy for "amount of reasoning," but longer outputs could simply contain more verbose wording rather than more reasoning steps. Analyzing the number of distinct reasoning steps (e.g., lines of equation manipulation, logical inferences) versus filler text would validate whether the dispersion at long lengths is caused by more reasoning exploration or by stylistic verbosity.

6. Limitations and Trade-offs

The assumption or constraint. The paper's central narrative — that the length–performance relationship is monotonic when a model is acquiring reasoning but non-monotonic when it is refining pre-existing reasoning — rests on observing Qwen3-1.7B-Base's length–accuracy curve over the range up to ~8,000–12,000 output tokens and interpreting the absence of a downturn as evidence that the curve would remain monotonic at any length. The paper acknowledges this implicitly by not discussing whether Qwen3-1.7B-Base's curve would eventually turn downward, stating only that "Qwen3-1.7B-Base shows a monotonically increasing trend" (Section 3.2.1) and hypothesizing that "Qwen3-1.7B-Base must acquire reasoning patterns through longer exploratory outputs" (Section 3.2.1).

The consequence. The paper cannot distinguish between two fundamentally different interpretations of its own data. Interpretation A: Qwen3-1.7B-Base is in a genuinely different regime where longer is always better, and no amount of additional training would produce a non-monotonic peak. Interpretation B: Qwen3-1.7B-Base would also exhibit non-monotonicity if training continued long enough — its current rightmost points simply have not reached the dispersion-dominated regime that DeepSeek-R1-Distill already occupies. If Interpretation B is correct, the "acquisition versus refinement" distinction is not a qualitative difference between models but a description of where each model sits on the same universal non-monotonic curve at a given training budget. The practical implication flips: rather than never applying length penalties to base models, practitioners would need to determine whether their model has reached the non-monotonic peak yet, and apply penalties only after that point. The paper's prescriptive claim — "length penalties can be harmful during reasoning acquisition" — would need to be reframed as "length penalties are harmful before the model reaches the non-monotonic peak," which is a statement about training stage, not model type.

What evidence exists in the paper. The training curves in Appendix C.1, Figure 6 show that for Qwen3-1.7B-Base, output length under Token Avg (DAPO) is still increasing at step 640 — the curve has not flattened. This means the model has not converged to a stable output length, and its length–accuracy relationship at step 640 represents a snapshot of a still-evolving policy, not an equilibrium property. The paper does not report whether accuracy was still increasing, plateauing, or declining at step 640 for any configuration on Qwen3-1.7B-Base. Without this information, the claim that the relationship is "monotonically increasing" cannot be separated from the claim that "training has not yet reached the regime where non-monotonicity would appear."

Mitigation status. Not addressed. The paper treats the monotonic/non-monotonic distinction as a model property rather than a training-stage property and makes prescriptive recommendations based on this interpretation without testing extended training for Qwen3-1.7B-Base. Section 4 mentions only that "a promising direction for future work is developing methods that automatically find the optimal length regime," which does not address the question of whether the same model would transition from monotonic to non-monotonic with additional training.

Single Training Run Per Configuration With No Statistical Quantification

The assumption or constraint. Every data point in Figure 1 — the paper's central empirical result — represents a single training run of one (method, penalty strength, model) combination. The paper does not report confidence intervals, error bars, standard deviations, or any form of statistical testing. The ALP instability case documented in Appendix A.4 demonstrates that training outcomes are not deterministic: "ALP with β = 1e−4 in FP16 without TIS exhibited instability: the probability difference temporarily spiked to a large value, causing training to fail. We restarted training from scratch with the same configuration, and the second run completed successfully, which we report as our result." This means that for at least one configuration, the outcome depended on which of two runs is selected.

The consequence. We cannot assess how much of the scatter in Figure 1 represents genuine differences in the length–performance tradeoff (e.g., RLOO-LP achieving higher accuracy than ALP at the same output length because of a better penalty mechanism) versus random variation across training runs (the same configuration would produce a different point if re-run with a different seed). This matters for the paper's specific claims: if the apparent peak in DeepSeek-R1-Distill's non-monotonic curve is produced by a single run that happened to be lucky, the "optimal intermediate length" may not exist with the precision the paper implies. More broadly, the lack of error quantification means that the paper's recommendations ("use moderate length penalties for distilled models, avoid them for base models") are based on point estimates of unknown reliability.

The ALP case further introduces a selection effect: configurations that failed were restarted until they succeeded, and only successful runs are reported. If some configurations are inherently more prone to failure, the reported results for those configurations represent a biased subset of runs — the ones that happened to survive. This could artificially inflate the apparent performance of fragile methods relative to robust ones.

What evidence exists in the paper. The ALP instability case (Appendix A.4, Figure 4) is explicit evidence of run-to-run variability. The BF16 divergence for DeepSeek-R1-Distill ("training diverged in both of two independent runs") demonstrates that some configurations fail entirely, further confirming that training outcomes are stochastic. Figure 1 contains visible scatter — for example, in the DeepSeek-R1-Distill MATH-500 panel, points at roughly 1,800–2,500 output tokens range from approximately 0.78 to 0.83 in score, a spread of 5 percentage points at similar lengths. Without error bars, we cannot determine whether this spread is noise or signal.

Mitigation status. Not addressed. The paper does not discuss the lack of statistical quantification as a limitation, does not report multiple seeds for any configuration, and does not include error bars in any figure. The multi-method convergence in Figure 1 (bottom) — points from different methods falling along a consistent curve — provides qualitative evidence that the overall shape is not purely noise, but it does not substitute for formal quantification of uncertainty at individual points.

All Results Are on Mathematical Reasoning Only, Yet Conclusions Are Framed as General

The assumption or constraint. The paper's title, abstract, and conclusion use the unqualified term "reasoning" — "On the Optimal Reasoning Length for RL-Trained Language Models," "length penalties may hinder reasoning acquisition," "LLMs have achieved remarkable progress in complex reasoning tasks" — but all experiments are conducted exclusively on mathematical reasoning benchmarks (AIME 2024, AIME 2025, AMC, MATH-500) using a mathematics-specific training dataset (DAPO-Math-17K). The paper does not discuss whether its findings are expected to transfer to other reasoning domains.

The consequence. Mathematical reasoning has distinctive properties that may make length–performance dynamics domain-specific. Math problems have well-defined, verifiable correct answers (enabling the binary reward signal that drives RL training); their reasoning chains involve formal symbol manipulation and step-by-step deduction; and longer outputs in math tend to correspond to more explicit derivations with intermediate equations. Other reasoning domains — code generation (where correctness is determined by test cases), logical reasoning over natural language premises, multi-step planning, scientific explanation, debate and argumentation — differ in whether "reasoning length" maps cleanly to "reasoning quality" in the same way. In code generation, for instance, a verbose solution with many redundant steps may be objectively worse than a concise one even if both pass tests, because code quality matters beyond mere correctness. In logical reasoning, longer outputs might reflect the model getting lost in irrelevant detail rather than exploring productive reasoning paths. The paper's central diagnostic framework — mode accuracy, entropy, and mode share — depends on having multiple samples that produce discrete, comparable final answers (e.g., "42" vs. "43"). For open-ended generation tasks or domains where answers are structured objects (code, plans, paragraphs), computing a "mode" may be ill-defined or require fuzzy matching that introduces its own complications.

The paper's claim that the optimal length control strategy "depends on the model's pre-existing reasoning capabilities" (Section 1) may not hold if the relationship between length and performance is domain-specific in addition to (or instead of) model-specific. A model that exhibits non-monotonicity on math might show monotonic improvement on code generation with the same length penalties, or vice versa.

What evidence exists in the paper. None. All four evaluation benchmarks (AIME 2024, AIME 2025, AMC, MATH-500) and the training dataset (DAPO-Math-17K) are mathematical. The paper does not include experiments on any non-math reasoning task and does not discuss domain specificity as a limitation or scope boundary. The only variety is within mathematics — the benchmarks span different difficulty levels and competition formats — but they all test the same underlying skill (mathematical problem-solving with discrete final answers).

Mitigation status. Not addressed. The paper does not mention domain generalization as a limitation, does not suggest that findings may be math-specific, and does not include non-math evaluations. The framing throughout implies that "reasoning" is a unitary construct and that findings about mathematical reasoning apply to reasoning in general.

The Under-Thinking Failure Mode Is Diagnosed From Very Few Data Points at Extreme Length Penalties

The assumption or constraint. The paper's identification of "under-thinking" as the failure mode at short output lengths for DeepSeek-R1-Distill is based on the leftmost points in Figure 2 — configurations with the strongest length penalties: RLOO-LP α = 1.0, ALP β = 1e−3, DRPO λ = 0.02, and possibly the next-strongest tier (α = 0.8, β = 3e−4, λ = 0.05). For DeepSeek-R1-Distill across all four benchmarks, this amounts to roughly 3–6 data points per benchmark in the regime where output length is substantially compressed below the optimum (roughly 1,000–2,500 tokens compared to the peak at roughly 4,000–6,000 tokens in aggregate, Figure 1, bottom).

The consequence. The characterization of "under-thinking" — that short outputs cause both the distribution's center to shift away from correct answers (low mode accuracy) and the distribution to spread out (high entropy, low mode share) — is based on a small number of extreme penalty configurations. If these specific configurations degrade performance through mechanisms other than pure length compression, the under-thinking diagnosis may conflate the effect of short length with the effect of aggressive reward shaping. For example, ALP with β = 1e−3 penalizes incorrect response length very strongly (the penalty term dominates the correctness reward for long incorrect responses), which could create optimization pathologies independent of output length. Similarly, RLOO-LP with α = 1.0 reduces the reward for a maximally long correct response to zero — making it indistinguishable from an incorrect response in reward space — which could fundamentally distort what the model learns to treat as "good" beyond simply compressing length.

The small number of short-output data points also means the paper cannot characterize whether the relationship between length and the decomposition metrics is continuous (mode accuracy and mode share decline smoothly as length decreases) or discontinuous (there is a threshold below which reasoning collapses abruptly). If the relationship is discontinuous, the "optimal length" is not a point on a smooth curve but a cliff edge, with very different practical implications for how carefully length penalties must be tuned.

What evidence exists in the paper. Figure 2 and Figure 10 show the short-output regime (left portions of each panel) for DeepSeek-R1-Distill. The leftmost 2–3 points in the AMC and MATH-500 panels clearly show lower mode accuracy, higher entropy, and lower mode share compared to the peak, but the separation between these extreme points and the peak is sparse — there are few intermediate penalty strengths that would trace out the transition. For Qwen3-1.7B-Base, no decomposition is performed at all (Section 3.2.2 applies only to DeepSeek-R1-Distill), so we cannot determine whether its short-output failures match the under-thinking signature or reflect a different mechanism.

Mitigation status. Not addressed. The paper treats the under-thinking diagnosis as established for all short-output regimes without discussing the sparsity of data at the low-length extreme or the possibility that extreme penalty strengths introduce method-specific artifacts beyond length compression. The paper does not suggest evaluating intermediate penalty strengths that would provide denser coverage of the short-output regime or running additional experiments to isolate the effect of length from the effect of penalty mechanism at extreme settings.

Difficulty Estimation and Adaptive Length Allocation Are Not Addressed, Limiting Practical Deployment

The assumption or constraint. The paper's prescriptive conclusion — "length control should be tuned to avoid both under-thinking from overly short outputs and dispersion from overly long outputs" (Section 4) — implies that a practitioner can identify the optimal length regime for their specific model and task. However, the paper provides no method for doing this without running the full hyperparameter sweep and decomposition analysis that the paper itself performed (training dozens of policies with different penalty methods and strengths, evaluating on held-out benchmarks, and computing mode accuracy, entropy, and mode share). The paper explicitly acknowledges this gap: "A promising direction for future work is developing methods that automatically find the optimal length regime without manual hyperparameter search" (Section 4).

The consequence. The paper demonstrates that optimal length control exists but provides no practical path to finding it for a new model or task. A practitioner wanting to apply these insights faces a chicken-and-egg problem: to determine whether their model is in the acquisition or refinement regime (and thus whether to apply length penalties at all), they would need to train multiple policies with different penalty strengths and examine the length–accuracy curve — but this is exactly the expensive sweep the paper argues should be avoided. The decomposition framework (mode accuracy, entropy, mode share) is diagnostic after multiple policies have been trained and evaluated, not predictive before training. A practitioner cannot look at an untrained base model and determine whether it will exhibit monotonic or non-monotonic behavior.

Furthermore, the optimal length almost certainly varies per-prompt. The paper's own related work discussion of ALP (Xiang et al., 2025) notes that it adapts penalty strength based on per-prompt accuracy, implicitly acknowledging that a uniform penalty is suboptimal because easy problems need less reasoning than hard ones. The paper's aggregate analysis across all problems in a benchmark masks this per-prompt heterogeneity — the "optimal intermediate length" in Figure 1 is an average across problems of varying difficulty, and the true per-problem optimum may be quite different. A policy trained with a uniform moderate penalty might be simultaneously under-thinking on hard problems and over-thinking on easy ones, with the aggregate curve showing a misleading "optimum" that satisfies neither.

What evidence exists in the paper. The paper's entire experimental design requires the full sweep to characterize the length–performance relationship. There is no experiment testing whether a simpler procedure — e.g., monitoring mode accuracy and entropy during training and adjusting the penalty online — can find the optimum without the full sweep. The paper does not report per-difficulty results (e.g., breaking down the AMC or MATH-500 benchmarks by problem difficulty) that would reveal whether the optimal length is difficulty-dependent. The Pre-RL baseline in Figure 1 shows that DeepSeek-R1-Distill's starting accuracy varies substantially across benchmarks (~0.22–0.26 on AIME vs. ~0.78–0.82 on MATH-500, bottom row of Figure 1), but the optimal length is reported only in aggregate, not stratified by the model's pre-training accuracy on specific problem subsets.

Mitigation status. Acknowledged as future work but not addressed experimentally. The paper's conclusion states this as a direction for future research without proposing even a preliminary method. The gap between "we have characterized the optimal length regime for these two specific models on these four benchmarks" and "a practitioner can find the optimal length regime for their model and task" is substantial and unaddressed.

Length Is Treated as the Sole Determinant of Efficiency, Ignoring Latency and the Serial Nature of Generation

The assumption or constraint. The paper uses "average output tokens" as the sole metric for computational cost, implicitly assuming that the cost of generating a response scales linearly with its token count and that all tokens cost the same. This ignores the serial dependency inherent in autoregressive generation: generating 4,000 tokens sequentially takes approximately twice the wall-clock time of generating 2,000 tokens, regardless of how many GPUs are available. The paper acknowledges this only indirectly, by noting that "longer outputs increase computational costs during both training and inference" (Section 1) without distinguishing latency (wall-clock time) from throughput (total FLOPs).

The consequence. The paper's characterization of the "efficiency–performance tradeoff" conflates two distinct resources: total computational work (FLOPs, proportional to tokens generated) and latency (wall-clock time, proportional to tokens generated but with a constant overhead for prompt processing and model loading). For batch inference scenarios where throughput is the binding constraint, token count is a reasonable cost metric. But for interactive applications where latency determines user experience, a policy that produces 4,000 tokens at 80% accuracy may be strictly worse than one that produces 8,000 tokens at 85% accuracy if both must complete within a fixed time budget — the shorter policy leaves time on the table that could have been used for more reasoning. Conversely, in latency-constrained settings (e.g., a 2-second response time limit), the optimal policy may be one that generates the best possible answer within the token budget that fits in 2 seconds, which is a different optimization problem than maximizing accuracy per token.

Additionally, the paper's training configurations use different maximum response length caps (8K for Qwen3-1.7B-Base, 16K for DeepSeek-R1-Distill). These caps directly affect the maximum possible latency per query. The paper justifies these choices based on the fraction of rollouts exceeding the cap at the start of training (<5%), but does not discuss the latency implications of deploying models with such high maximum generation lengths in production.

What evidence exists in the paper. The paper reports only average output tokens, with no latency measurements, no time-to-answer distributions, and no analysis of how generation time scales with output length in the specific inference setup used (vLLM with the reported GPU configuration). The evaluation protocol (64 samples per problem for AIME/AMC, 16 for MATH-500) is designed to measure accuracy, not to simulate a deployment scenario with latency constraints. The "efficiency" gains claimed — e.g., that moderate penalties achieve higher accuracy at shorter output lengths than no-penalty methods — are efficiency gains in tokens, not necessarily in wall-clock time.

Mitigation status. Not addressed. The paper does not discuss latency as a dimension of the efficiency–performance tradeoff, does not report time measurements, and does not analyze how the optimal length might shift under different latency constraints. The term "efficiency" throughout the paper refers exclusively to token count.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the conversation around length control in RL-trained reasoning models from a tooling question (which penalty method to use) to a diagnostic question (whether your model even needs length control at all, and if so, in which direction). Prior work on RLOO-LP, ALP, DRPO, and GFPO implicitly assumed that length inflation is a universal problem to be solved — that shorter is always better for efficiency, and the only question is how aggressively to compress outputs while preserving accuracy. This paper demonstrates that this assumption is conditionally wrong: for models acquiring reasoning from scratch (Qwen3-1.7B-Base), any length penalty is actively harmful, preventing the exploratory long chain-of-thought that RL needs to discover reasoning strategies. For models refining existing reasoning (DeepSeek-R1-Distill-Qwen-1.5B), length penalties can help, but only within a narrow intermediate range — and the direction of the optimal adjustment depends on which side of the non-monotonic peak a given policy occupies.

This is not an incremental refinement of length control methodology. It is a reframing of the problem that changes which question practitioners should ask. Before this work, the question was: "What penalty strength minimizes output length while preserving accuracy?" After this work, the prior question becomes: "Is my model in the reasoning acquisition or refinement regime?" The answer determines whether length control should be applied at all, and the paper provides the first empirical evidence that the answer depends on a measurable property of the training setup — the base model's prior reasoning capabilities — rather than on universal hyperparameter tuning.

The paper also resolves a latent tension in the literature. Ghosal et al. (2025) established theoretically (and empirically under test-time interventions) that thinking more can hurt — that variance-driven dispersion degrades performance beyond an optimal length. But their framework only addressed the right side of the length spectrum (overthinking) and only under artificial test-time manipulation (appending "Wait" tokens). This left open the question of whether RL-trained policies naturally enter the overthinking regime, and what happens on the left side (underthinking). This paper closes both gaps: it shows that standard RL training without length control indeed pushes DeepSeek-R1-Distill into the dispersion-dominated overthinking regime (Figure 1, bottom, rightmost points), and that the left side of the curve is governed by a distinct failure mode — underthinking — where both central tendency and dispersion are degraded (Figure 2). The decomposition into mode accuracy and dispersion metrics provides the diagnostic toolkit that distinguishes these two failure modes, which have opposite remedies: more length for underthinking, less length for dispersion. This turns Ghosal et al.'s theoretical insight into an operationally actionable framework for training.

The failed GFPO reproduction (Appendix B.4.1) is a landscape-changing negative result in its own right. It reveals a design constraint that was not previously articulated: effective length control must maintain gradient signal against long incorrect responses, not just avoid reinforcing long correct ones. Methods that filter out long responses entirely (GFPO) remove the negative examples that teach the model what verbose failure looks like, allowing verbose reasoning patterns to intensify rather than shrink. This constraint was not obvious from the design of any individual successful method — RLOO-LP's authors did not motivate their approach in terms of negative signal preservation, nor did ALP's or DRPO's. It is only through the comparative lens, and specifically through the failure of a method that violates the constraint, that the common requirement becomes visible. This reframes length control design as a constrained optimization problem where the constraint — preserve negative gradient signal on long incorrect responses — is as important as the objective.

The paper also exposes the interaction between loss normalization, base model length variance, and training stability as a first-class factor in length control (Appendix C.1). The finding that GRPO's Sample Avg normalization is unstable on models with high within-prompt length variance (Qwen3-1.7B-Base, CV = 1.10) but stable on models with low variance (DeepSeek-R1-Distill, CV = 0.45) means that the choice of training configuration must be conditioned on the base model's output characteristics before length control is even attempted. This interaction was not documented in the original GRPO, DAPO, or any of the length penalty papers, making it a novel finding with immediate practical consequences for anyone attempting to reproduce or extend this line of work.

In terms of which research directions become more or less attractive: more attractive are investigations into the transition between acquisition and refinement regimes (when does a model cross from monotonic to non-monotonic?), diagnostic tools that can determine which regime a training run is in without a full hyperparameter sweep, and length control methods that explicitly preserve negative signal on incorrect responses. Less attractive are one-size-fits-all length penalty proposals that do not condition on model characteristics or training stage, and filtering-based approaches that remove long responses from training without ensuring that negative gradient signal on verbose errors is preserved through alternative mechanisms.

Follow-Up Research This Work Enables

Characterizing the acquisition-to-refinement transition with extended training. The paper's central finding — monotonic length–accuracy for Qwen3-1.7B-Base, non-monotonic for DeepSeek-R1-Distill — is interpreted as a consequence of prior reasoning capability, but the paper cannot distinguish this from a training-stage effect because Qwen3-1.7B-Base's training was stopped at 640 steps while output length was still increasing (Figure 6). A direct follow-up would train Qwen3-1.7B-Base with no length penalty (Token Avg/DAPO) for 2,000+ steps and measure whether the length–accuracy curve eventually transitions from monotonic to non-monotonic — and if so, at what accuracy level or training step the peak emerges. If the transition occurs at a specific accuracy threshold (e.g., when MATH-500 score exceeds ~0.50), it would suggest that the acquisition/refinement distinction is better characterized by the model's current capability level than by its pretraining provenance, enabling a capability-conditional length control policy that applies penalties only after the model crosses the threshold. If the transition never occurs, it would validate the paper's interpretation that base pretrained models and distilled reasoning models occupy genuinely different length–performance regimes.

Inference-time length variation as a test of whether training-time penalties produce qualitatively different policies. The paper controls output length through training-time penalties that shape the learned policy. An alternative approach — train a single policy (e.g., Token Avg/DAPO) with no length penalty, then vary output length at inference time through generation parameters (max tokens, temperature, top-p) — would test whether the non-monotonic pattern in Figure 1 reflects a property of the learned policy's internal structure or is an artifact of training different policies. If setting a max output token limit at inference time on the no-penalty policy reproduces the same non-monotonic curve (with underthinking at very low limits and peak accuracy at intermediate limits), it would suggest that the policy has learned a reasoning strategy whose quality depends on length regardless of how that length is achieved. If it does not — if the inference-time-limited policy achieves different accuracy than a training-time-penalized policy at the same output length — it would indicate that training-time penalties produce qualitatively different policies (e.g., learning to reason more efficiently per token rather than simply truncating the same reasoning process). This experiment could be run on the already-trained DeepSeek-R1-Distill policies from this paper, requiring only additional evaluation runs with different max token settings.

Developing online diagnostics for failure mode detection during training. The paper's decomposition framework (mode accuracy, entropy, mode share) is applied post-hoc to fully trained policies, requiring multi-sample evaluation on a held-out benchmark. A practical next step is to develop online proxies that can be computed during training from the rollout data itself — without requiring a separate evaluation run — to detect whether the current policy is in the underthinking regime (low mode accuracy, high entropy) or the dispersion regime (high mode accuracy, high entropy). For example, tracking the mode accuracy and entropy of the 16 sampled responses per prompt during RL rollouts (averaged over a moving window of recent training steps) could provide a real-time signal of which failure mode dominates, enabling automated adjustment of the length penalty strength without a separate hyperparameter sweep. A strong follow-up would implement this online diagnostic, connect it to a simple PID controller that adjusts the penalty strength to maintain mode accuracy above a threshold while minimizing entropy, and demonstrate that it converges to near-optimal length without manual tuning on both Qwen3-1.7B-Base (where the controller should set the penalty to zero) and DeepSeek-R1-Distill (where it should converge to a moderate penalty).

Stress-testing the negative signal hypothesis through controlled GFPO variants. The paper hypothesizes that GFPO's failure is caused by removing gradient signal on long incorrect responses, but this is inferred from a single configuration ( $G = 16$ , $k = 8$ ). A systematic follow-up would test this hypothesis by designing GFPO variants that selectively filter: (a) filter only long correct responses while keeping all incorrect responses (testing whether removing positive reinforcement for verbose correctness is sufficient), (b) filter only long incorrect responses while keeping all correct responses (testing whether specifically removing negative signal on verbose errors causes length inflation), and (c) filter both but add a length-based penalty on incorrect responses in the remaining data (testing whether the negative signal can be provided through reward shaping rather than data inclusion). If variant (a) succeeds at reducing length while variant (b) causes inflation, it would confirm the negative signal hypothesis. If variant (c) succeeds, it would provide a practical fix for GFPO-style filtering methods. These experiments would use the same DAPO-Math-17K dataset and training setup as the paper, with Qwen3-1.7B-Base as the primary testbed since it exhibits the highest length variance and largest negative length bias.

Domain transfer: does the optimal length generalize from math to code, logic, or science? The paper's findings are exclusively on mathematical reasoning. A natural extension would replicate the core experiment — comparing Qwen3-1.7B-Base and DeepSeek-R1-Distill-Qwen-1.5B with the same length penalty sweep — on a non-math reasoning domain such as code generation (e.g., LiveCodeBench or HumanEval with test-case-based binary rewards), logical reasoning (e.g., ARC or FOLIO), or multi-step scientific reasoning (e.g., GPQA). The key question is whether the monotonic/non-monotonic distinction is model-dependent (as the paper claims) or domain-dependent as well. If DeepSeek-R1-Distill shows non-monotonicity on code but not on logic, it would indicate that the optimal length regime depends on the interaction between model capabilities and task structure, requiring per-domain length tuning. If the same non-monotonic pattern appears across all domains for DeepSeek-R1-Distill and the same monotonic pattern appears across all domains for Qwen3-1.7B-Base, it would strengthen the paper's claim that prior reasoning capability is the primary determinant. The evaluation would require adapting the mode accuracy / entropy / mode share decomposition to domains where answers are structured objects (code, proofs) rather than discrete tokens, likely requiring fuzzy matching or test-case-based correctness grouping.

Per-difficulty length allocation: does the optimal length vary systematically with problem difficulty? The paper's aggregate length–accuracy curves mask per-problem heterogeneity — the "optimal intermediate length" in Figure 1 is an average across problems of varying difficulty, and the true per-problem optimum may be quite different. A follow-up would stratify the evaluation benchmarks by problem difficulty (either using the benchmarks' built-in difficulty labels where available, or by the Pre-RL model's pass@1 rate on each problem, analogous to the difficulty binning in the prior technical approach paper) and measure whether the optimal output length shifts systematically. The prediction from ALP's design (which adapts penalty strength based on per-prompt accuracy) is that easy problems have a shorter optimal length while hard problems benefit from longer outputs. If confirmed, it would suggest that the ideal length control method is not a fixed penalty but a difficulty-conditioned one, where the penalty strength varies per problem based on estimated difficulty. This would connect directly to the literature on compute-optimal test-time scaling, framing output length as a resource to be allocated adaptively across problems rather than uniformly compressed.

Practical Applications and Downstream Use Cases

Training reasoning models from base checkpoints: disable length penalties until reasoning capability emerges. The paper's most directly actionable finding for training practitioners is that applying length penalties to a base pretrained model (Qwen3-1.7B-Base) degrades performance at every penalty strength tested — the point in Figure 1 (top) with the highest accuracy is Token Avg/DAPO with no explicit length penalty. For teams training reasoning models from standard pretrained checkpoints (as opposed to distilled reasoning models), this implies that length control should be disabled entirely during the initial phase of RL training. The exploratory long chain-of-thought that RL spontaneously produces is not waste to be compressed but a necessary substrate for discovering reasoning strategies. Premature length pressure prevents this exploration and leads to convergence to policies that are both short and wrong — the underthinking regime where mode accuracy and dispersion are both poor (Figure 2, left portions). The practical recipe is: start RL training with Token Avg normalization (not Sample Avg, which the paper shows is unstable on high-variance base models) and no length penalty, monitor output length and validation accuracy, and only consider introducing length penalties after accuracy has plateaued, at which point the model may have transitioned into the refinement regime where moderate penalties can improve efficiency.

Fine-tuning distilled reasoning models: tune length penalties to the non-monotonic peak using mode accuracy as a guide. For teams fine-tuning models that already possess reasoning capabilities — such as DeepSeek-R1-Distill variants or other reasoning-distilled models — the paper provides a specific tuning strategy. Rather than blindly sweeping penalty strengths and picking the one with the best validation accuracy (which conflates underthinking and overthinking failures), practitioners can use the decomposition framework to diagnose which side of the peak their current policy occupies. If mode accuracy on a validation set is substantially lower than the best observed mode accuracy across penalty configurations, the policy is likely underthinking — reduce the penalty strength to allow longer outputs. If mode accuracy is high but mode share is low and entropy is high, the policy is likely overthinking — increase the penalty strength or lower the sampling temperature to sharpen the distribution. The paper's Figure 2 shows that for DeepSeek-R1-Distill, the peak performance on AMC corresponds to mode accuracy ~0.78–0.82, entropy ~1.65–1.70, and mode share ~0.67–0.68, providing concrete reference values for what a well-tuned policy looks like on competition math — though these values would need re-calibration for different models and domains. The practical gain is that a diagnostically-informed tuning process avoids the trial-and-error of sweeping penalties in the wrong direction (e.g., strengthening penalties when the model is already underthinking, which would further degrade performance).

Cost-aware model selection: choosing between training a distilled model with length control versus a base model without. The paper's comparison between Qwen3-1.7B-Base and DeepSeek-R1-Distill-Qwen-1.5B — two models of similar parameter count but different reasoning provenance — provides a concrete case study in the cost–capability tradeoff for reasoning model deployment. DeepSeek-R1-Distill achieves substantially higher accuracy than Qwen3-1.7B-Base across all benchmarks (compare the y-axis scales in Figure 1: DeepSeek-R1-Distill's AIME 2024 scores peak at ~0.32–0.36 while Qwen3-1.7B-Base's peak at ~0.14–0.16), and its non-monotonic length–accuracy curve means that moderate length penalties can simultaneously improve accuracy and reduce inference cost (peak at ~4,000–6,000 tokens vs. ~10,000–15,000 for no-penalty methods). For a deployment with a fixed inference budget, the choice between these two model classes depends on whether the accuracy of the base model after RL training is sufficient: if it is, training the base model with no length penalty is simpler and avoids the tuning complexity; if higher accuracy is needed, the distilled model with tuned length penalties delivers better performance per token. The paper's numbers provide a concrete basis for this decision: on MATH-500, Qwen3-1.7B-Base with Token Avg reaches ~0.63–0.65 at ~2,500–3,000 tokens, while DeepSeek-R1-Distill with moderate RLOO-LP (α = 0.2) reaches ~0.82–0.83 at ~1,500–2,500 tokens — roughly 30% higher accuracy at roughly 30% fewer tokens, but requiring the upfront cost of obtaining a distilled model and tuning the penalty.

Budget-constrained RL training: prioritizing stable normalization over sophisticated length penalties. For teams with limited compute budgets who can only afford a small number of training runs, the paper's stability analysis (Appendix C.1) provides a clear prioritization. Before experimenting with any length penalty method, ensure that the training configuration is stable for the specific base model: use Token Avg (DAPO) normalization rather than Sample Avg (GRPO) if the base model has high output length variance (which is likely for non-distilled models), set the mini-batch size equal to the generation batch size to avoid off-policy updates (the 64/64 configuration rather than 512/32), and use FP16 precision with potential TIS for models that exhibit BF16 instability (as documented for DeepSeek-R1-Distill in Appendix A.4). The paper shows that getting these fundamentals wrong can cause training to silently fail — producing policies that appear to converge but have actually entered a degenerate regime of decreasing length and declining accuracy (Figure 3). The cost of diagnosing and fixing such failures after the fact can exceed the cost of the length penalty sweep itself, so front-loading stability validation is the highest-return investment for practitioners new to RL-based reasoning training.