ArXiv: 2510.11701

🎯 Pitch

Training small LLMs to reason with tools often fails because we use fake, stitched-together data and RL recipes that kill exploration. The authors show that simply switching to real end-to-end tool-use trajectories and keeping the RL algorithm exploration-friendly (via higher clipping and entropy maintenance) lets a 4B model crush 32B competitors on math benchmarks.

1. Executive Summary

This paper conducts a systematic empirical investigation to demystify reinforcement learning for agentic reasoning across three axes—data, algorithm, and reasoning mode—using GRPO-based policy optimization on Qwen2.5-7B-Instruct and Qwen3-4B-Instruct-2507 models evaluated on AIME2024/2025, GPQA-Diamond, and LiveCodeBench-v6. The work identifies that real end-to-end tool-use trajectories substantially outperform stitched synthetic data for SFT initialization (e.g., +28.85% average@32 on AIME2024 for the 4B model), that clip higher and overlong reward shaping together with token-level loss aggregation and sustained policy entropy are the critical algorithmic ingredients for efficient agentic RL, and that a deliberative strategy with fewer but more accurate tool calls (over 70% tool-use success rate) consistently beats frequent reactive tool invocation. The resulting DemyAgent-4B model achieves 72.6% on AIME2024 and 70.0% on AIME2025, surpassing 32B-scale agentic baselines like ReTool-32B, establishing that compact models can match or exceed much larger counterparts through principled test-time tool orchestration only when training data captures genuine end-to-end reasoning, the RL recipe preserves exploration through higher clipping and entropy maintenance, and the agent learns to reason deliberately before acting.

2. Context and Motivation

The Core Problem: Agentic RL Remains Poorly Understood Despite Rapid Empirical Progress

The fundamental question this paper tackles is deceptively simple: when we apply reinforcement learning to train LLMs as agents that use tools, what actually matters? This is not a theoretical curiosity — it reflects a genuine crisis of understanding in a field that has seen explosive empirical progress without a corresponding grasp of why things work. The paper identifies three specific gaps that collectively prevent practitioners from reliably building capable agentic reasoning systems.

First, data curation pipelines are broken. The dominant paradigm for building tool-use SFT datasets relies on what the paper calls "stitch-style data synthesis" (Section 1, challenge 1), where segments of internal reasoning are manually replaced with tool outputs post hoc. This approach, exemplified by ReTool (Feng et al., 2025), takes long chain-of-thought solutions and surgically substitutes certain reasoning steps with tool invocations and responses. While scalable, this stitching process is fundamentally dishonest: it cannot capture when a real agent would decide to invoke a tool, why it would choose that moment rather than continuing internal reasoning, or how it would recover if the tool output is unexpected or incorrect. The paper articulates this as a failure to "faithfully mimic real multi-turn trajectories that indicate when and why tools should be invoked" (Section 1). The consequence is that models trained on such data learn tool-following patterns but not tool-strategy — they become tool-executors, not tool-reasoners.

Second, the algorithmic design space is a scattered collection of heuristics with no principled guidance. The GRPO algorithm (Shao et al., 2024) has become the workhorse of RL for reasoning, but its application to agentic settings has spawned numerous ad-hoc modifications — clip higher (Yu et al., 2025), entropy management (Wang et al., 2025b), overlong reward shaping (Zheng et al., 2025), token-level versus sequence-level loss aggregation — with no systematic understanding of which matter, when, or why. The paper states this explicitly: "A principled understanding of when and how to deploy these algorithms is still missing" (Section 1, challenge 2). Different papers offer contradictory prescriptions: some advocate entropy minimization for deterministic policies (Agarwal et al., 2025), others celebrate high-entropy tokens as the driver of learning (Cui et al., 2025; Wang et al., 2025b), and still others observe entropy spikes specifically after tool calls and design adaptive mechanisms around them (Dong et al., 2025b). A practitioner attempting to build an agentic RL pipeline faces a bewildering array of knobs — clip bounds, KL coefficients, reward shaping parameters, loss aggregation strategies — with no principled way to set any of them.

Third, the reasoning mode itself is uncharted territory. When an agent has access to tools, how should it allocate its inference compute between internal reasoning tokens and external tool calls? Should it reason extensively before acting (deliberative mode), or interact frequently in short bursts (reactive mode)? The answer seems to depend on the task, the model, and the stage of training, but no systematic characterization exists. The paper notes this as an open puzzle: questions about "the allocation of turn budgets, the trade-off between response length and tool-call efficiency, and the impact of long-CoT predispositions on multi-turn reasoning" remain unresolved (Section 1, challenge 3). Prior work often treats these as post-hoc observations rather than design variables — someone notices that their model started calling tools less frequently, or that longer responses correlated with better performance, but no one has plugged these levers into a controlled experimental framework.

Why This Problem Matters

The practical stakes are substantial and multi-dimensional.

From a deployment perspective, agentic LLMs represent a qualitatively different capability frontier than self-contained reasoning models. A model that can autonomously invoke code interpreters, search engines, or simulators can solve problems that are computationally infeasible through pure internal reasoning — numerical integration, symbolic computation, real-time data retrieval, programmatic verification of intermediate results. But if we don't understand how to train such agents reliably, we're stuck with fragile systems that work in demos but fail unpredictably in production. The paper cites several representative works — Search-R1 (Jin et al., 2025), R1-Searcher (Song et al., 2025), ReTool (Feng et al., 2025), ToRL (Li et al., 2025d), ARPO (Dong et al., 2025b), Tool-Star (Dong et al., 2025a) — each proposing a different algorithmic recipe, none of which generalize reliably.

From a model efficiency standpoint, the paper's headline result — a 4B model outperforming 32B models on agentic benchmarks — has profound implications for deployment economics. If principled agentic RL recipes can extract capabilities from small models that previously required large ones, the cost structure of AI deployment shifts dramatically: instead of serving ever-larger models, organizations could serve small models with sophisticated tool orchestration. But this only works if we know how to train the small model effectively, which is exactly what the paper aims to provide.

From a research methodology perspective, the lack of systematic understanding has created a wasteful pattern: each new agentic RL paper proposes a slightly different algorithmic variant, evaluates on slightly different benchmarks, and claims superiority without isolating which of their many design choices actually caused the improvement. This prevents cumulative progress. The paper's framing as a controlled ablation study across three axes — systematically varying one factor at a time while holding others fixed — is an explicit attempt to break this pattern and establish what the authors call "a practical baseline for future agentic RL research" (Section 1).

Prior Approaches and Their Shortcomings

The paper positions itself against several distinct lines of prior work, each with identifiable weaknesses:

SFT-based tool-integrated reasoning (ToRA, Toolformer, MathCoder, ReAct). These methods train LLMs on demonstration data containing tool invocation patterns, teaching them to follow predefined templates of tool use. The paper acknowledges that this approach successfully teaches models how to call tools but identifies a fundamental limitation: since models are "compelled to use tools according to the distribution of training data, they cannot develop adaptive strategies for tool use, such as deciding when to invoke a tool, how often to call it, or how to balance tool use with internal reasoning" (Section 7). This is the classic imitation learning problem — SFT copies behaviors but doesn't optimize for outcomes, so it cannot discover strategies that deviate from the training distribution even if those strategies would be more effective.

Stitch-style data synthesis (ReTool, Toolformer). The paper's Section 3.1 explicit comparison between stitched synthetic trajectories and real end-to-end trajectories represents one of its most important empirical findings. Stitched data (specifically, the multi-turn SFT dataset from ReTool) produces models that achieve below 10% average@32 on AIME2025 with Qwen3-4B-Instruct-2507, while real trajectories push the same model to 29.79%. The paper identifies four specific behavioral capabilities that stitching fails to capture: pre-call analysis (localizing which subproblems should be delegated to tools), guarded execution (intermediate checks), error recovery and strategy revision (after failed attempts), and self-reflection and calibration (before invoking tools). These are exactly the meta-cognitive skills that distinguish tool-strategists from tool-executors.

GRPO and its variants (GRPO, DAPO, GSPO). The paper uses GRPO as its baseline algorithm and acknowledges its effectiveness for reasoning, but observes that directly applying standard GRPO recipes to agentic tasks produces suboptimal results: "inefficient on-policy rollout sampling, reward & entropy collapse, and unstable training dynamics" (Section 1). The key issue is that GRPO's standard conservative clipping (ε = 0.2) and KL regularization (β = 0.001) suppress the exploration that agentic RL fundamentally needs — because discovering good tool-use strategies requires exploring a combinatorially larger action space than self-contained reasoning. The paper's observation that GRPO-T with standard settings achieves only 40.93% on AIME2025 while GRPO-TCR (clip higher + overlong reward shaping) reaches 68.13% within 100 steps quantifies just how large this gap is.

Long-CoT models in agentic settings (Search-R1, R1-Searcher, Search-o1). A particularly revealing finding comes in Section 5.2, where the paper attempts to initialize agentic RL with a Long-CoT model (Qwen3-4B-Thinking-2507). The model achieves strong initial performance but stops using tools entirely as training progresses, converging to zero tool calls. The paper's diagnosis is that Long-CoT models optimized for reasoning-intensive tasks develop an ingrained predisposition to rely on internal reasoning, and "tend to avoid invoking tools and rely solely on internal reasoning when encountering reasoning-intensive tasks" (Section 5.2). This is a critical negative result: it shows that better reasoning ability does not automatically translate to better tool use — in fact, it can actively interfere with tool adoption.

Entropy management literature (Cui et al., 2025; Wang et al., 2025b; Agarwal et al., 2025; Cheng et al., 2025b). The paper engages with the ongoing debate about entropy's role in RL for reasoning, noting that "prescriptions diverge: some advocate minimizing it for more deterministic policies, while others exploit high-entropy tokens to foster exploration and avoid early collapse" (Section 4.3). The paper's contribution is to show that in agentic settings specifically, the answer depends on model capacity: weaker models (Qwen2.5-7B) need larger clip upper bounds to escape performance bottlenecks, while stronger models (Qwen3-4B) need tighter bounds to prevent over-exploration (Section 4.3, Takeaway 4.3.2). This model-dependent optimal entropy range is a nuance that prior work missed because it focused primarily on self-contained reasoning.

How This Paper Positions Itself

The paper does not propose a fundamentally new algorithm. Instead, it positions itself as a systematic empirical investigation that builds on existing methods while providing the principled understanding that has been missing. The three-axis decomposition — data, algorithm, reasoning mode — is both a conceptual framework for organizing the investigation and an implicit critique of prior work that focused narrowly on one axis while ignoring the others.

On the data axis, the paper argues that stitching is not merely suboptimal but fundamentally the wrong abstraction: agentic data must be real, end-to-end, diverse (spanning math, science, and code to maintain exploration), and model-aware (matched to the policy's current capability). This directly challenges the scalable-but-inauthentic paradigm that dominates existing tool-use datasets.

On the algorithm axis, the paper doesn't claim to discover new techniques but rather to identify which of the many proposed GRPO modifications actually matter and why. The finding that clip higher + overlong reward shaping accounts for the overwhelming majority of the improvement over baseline GRPO, and that token-level loss provides a smaller but consistent additional benefit for stronger models, provides a practical minimum-viable recipe that cuts through the noise of proliferating algorithmic variants.

On the reasoning mode axis, the paper's deliberative-vs-reactive framing, coupled with the tool-use efficiency metric (success rate of tool calls), introduces a rigorous way to evaluate how agents allocate their reasoning budget. The finding that deliberate agents achieve over 70% tool-use success while reactive agents flail with low-efficiency calls is presented not as a prescription for always using fewer calls, but as evidence that the quality of reasoning before tool invocation determines the effectiveness of the tool calls themselves. The paper further shows that this deliberative mode emerges naturally from effective RL training — it is not an architectural choice but a learned behavior.

The paper also explicitly positions itself as providing infrastructure and baselines for the community: the 3k SFT dataset, the 30k RL dataset, the two cold-start checkpoints (Qwen2.5-7B-RA-SFT and Qwen3-4B-RA-SFT), and the DemyAgent-4B model are all released as artifacts. This reflects a recognition that progress in agentic RL has been bottlenecked not just by algorithmic understanding but by data availability — real end-to-end trajectories are computationally expensive to collect, and the paper's decision to release these resources lowers the barrier to entry for systematic investigation.

How Existing Work Fails to Address This Gap

The paper identifies several specific failure modes in prior approaches that motivate its investigation:

Stitch data provides no decision-boundary signal. When a synthetic trajectory shows a tool being called at step 4 of a solution, there is no information about why step 4 rather than step 3 or step 5. The agent learns that tool calls are part of the format but not the strategic calculus that governs their timing. Section 3.1's quantitative results — e.g., Qwen3-4B-Instruct-2507 achieving 51.64% maj@32 on AIME2025 with real trajectories versus 0.10% with synthetic trajectories — underscore that stitching produces not just weaker initial performance but fundamentally unstable behavior (extremely low majority voting accuracy despite moderate pass@k).

Standard GRPO clips too hard for agentic exploration. The paper's finding that baseline GRPO-T exhibits "early entropy collapse" (Section 4.3, Observation) while GRPO-TCR sustains higher entropy reflects a core mismatch: agentic RL needs to explore a space of multi-turn tool-use strategies, not just a space of reasoning chains, and the diversity required is substantially larger. Conservative clipping and strong KL regularization prevent the policy from discovering effective tool-use patterns because the initial distribution shift away from SFT behavior is too heavily penalized.

Long-CoT models have conflicting optimization objectives. The result in Section 5.2 that Long-CoT models initially perform well but then abandon tool use entirely is interpreted as evidence of "conflicting objectives": ingrained internal reasoning patterns contradict agentic reasoning paradigms, forcing a "scaling and pruning process where gains in agentic reasoning are offset by the need to suppress over-thinking behaviors" (Section 5.3). This explains why simply starting RL from a better reasoning model does not automatically yield better agentic behavior — the two capabilities can be in tension.

No prior work systematically varied difficulty matching. The paper's model-aware dataset construction (Section 3.3) addresses a subtle but critical issue: when the training dataset contains problems at difficulty levels that a weak model cannot solve at all, those problems contribute zero gradient signal and effectively dilute the training data. Prior work either used fixed datasets without considering their difficulty distribution relative to the current policy, or adjusted dataset composition heuristically without a principled criterion. The paper's solution — discarding problems with 0% or 100% solve rates and matching the empirical difficulty distribution to a target derived from a stronger model — provides a concrete recipe for dataset curation that accounts for the dynamic nature of the policy during training.

3. Technical Approach

3.1 Reader Orientation

The paper builds a systematic empirical framework for understanding how reinforcement learning trains LLMs to become effective tool-using agents. It is not a single new model or algorithm, but rather a controlled experimental methodology that isolates and evaluates design choices across three axes — data, algorithm, and reasoning mode — to identify which factors causally improve agentic reasoning and why. The core idea is straightforward: apply the same base GRPO algorithm with different data sources, algorithmic tweaks, and reasoning strategies, measure the resulting agent performance and training dynamics (entropy, pass@k, average@k, tool-use efficiency), and trace causal links between specific design choices and observed outcomes. The problem this solves is that the agentic RL literature has accumulated dozens of proposed techniques — clip higher, overlong reward shaping, token-level loss, diverse datasets, model-aware filtering — with no principled understanding of which actually matter, when, or why. The solution takes the shape of a factorial ablation study where each axis is varied independently while holding the others constant, producing a "minimum viable recipe" that practitioners can adopt and researchers can build upon.

3.2 Big-Picture Architecture (Diagram in Words)

The system comprises five interconnected components, where information flows from raw data through supervised initialization into reinforcement learning and finally to evaluation:

  1. SFT Data Pipeline: Curates either real end-to-end multi-turn tool-use trajectories (using a teacher model with actual tool execution) or synthetic stitch-style trajectories (where tool calls are retroactively inserted into existing reasoning chains). Outputs the cold-start SFT model (e.g., Qwen3-4B-RA-SFT) that serves as the initial policy for RL.

  2. RL Data Pipeline: Constructs a 30k-sample training set combining math, science, and code problems with ground-truth answers, optionally filtered to be model-aware (matched to the current policy's difficulty profile). Outputs the prompt distribution from which RL rollouts are sampled.

  3. GRPO-Based Policy Optimization Engine: Takes the SFT-initialized policy and the RL dataset, samples multiple completions per prompt with tool-call feedback, computes composite rewards (outcome accuracy + tool-use bonus + overlong penalty), normalizes advantages across the batch, and updates the policy using either token-level or sequence-level loss aggregation with configurable asymmetric clipping. Outputs the trained agent policy.

  4. Training Dynamics Monitor: Tracks entropy (policy distribution breadth), pass@k (ability boundary), average@k (exploitation performance), average reward, and tool-use metrics (number of calls, success rate) throughout RL training. Provides the diagnostic signals for understanding why one recipe outperforms another.

  5. Evaluation Suite: Tests the final agent on AIME2024/2025, GPQA-Diamond, and LiveCodeBench-v6 under two paradigms: self-contained reasoning (no tools) and agentic reasoning (with code interpreter access). Reports average@32, pass@32, maj@32, and pass@1 metrics.

Information flows sequentially: (1) SFT data → SFT model; (2) RL data + SFT model → GRPO engine; (3) GRPO engine → trajectories → rewards → advantages → policy update; (4) policy update → new trajectories → monitor logs entropy/metics → next iteration; (5) converged policy → evaluation suite → benchmark scores.

3.3 Roadmap for the Deep Dive

  • First, the agentic RL objective (Equation 1) and rollout factorization (Equation 2), because they define what "agentic" means formally — tool interactions are part of the action space, not post-hoc additions — and establish notation used throughout the subsequent GRPO formulation.

  • Second, the three GRPO recipe variants (GRPO-TCR, GRPO-SCR, GRPO-T) with their precise technical differences: loss aggregation granularity (token-level vs. sequence-level), clipping strategy (asymmetric clip higher vs. symmetric standard), and reward shaping (composite outcome+tool bonus vs. composite plus overlong penalty). This provides the algorithmic substrate that all subsequent experiments vary.

  • Third, the SFT data curation pipeline — how real end-to-end trajectories are generated, filtered, and compared against stitch-style synthetic data — because SFT quality establishes the policy initialization that RL either builds upon or struggles against.

  • Fourth, the RL data construction principles — diversity (mixing math, science, code) and model-awareness (difficulty filtering) — because these choices determine what exploration behaviors the policy can discover and whether gradient signals are informative.

  • Fifth, the entropy mechanism and its relationship to exploration-exploitation dynamics, because this connects the algorithmic design choices (clip bounds, reward shaping) to observable training outcomes (convergence speed, peak accuracy, stability) and explains why certain recipes work.

  • Sixth, the reasoning mode analysis — deliberative vs. reactive tool use, Long-CoT integration challenges — because this examines the learned behavior that emerges from the data and algorithm choices, closing the loop from inputs to outcomes.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical analysis paper whose core idea is that agentic RL performance is determined by (i) the authenticity and diversity of training data, (ii) the exploration-friendliness of the RL algorithm, and (iii) the reasoning strategy the agent learns to adopt, and that by systematically optimizing each axis, small models can surpass much larger ones.

Agentic RL Objective and Rollout Factorization

The paper formalizes the agentic RL training problem as a constrained policy optimization that extends standard RL to incorporate tool-use feedback during the rollout process.

The objective is:

max⁡πθEx∼D,y∼πθ(⋅∣x;T)[rϕ(x,y)]−β DKL(πθ(y∣x;T) ∥ πref(y∣x;T))\max_{\pi_\theta} \mathbb{E}_{x \sim \mathcal{D}, y \sim \pi_\theta(\cdot|x; \mathcal{T})} \left[ r_\phi(x, y) \right] - \beta \, D_{\text{KL}}\left( \pi_\theta(y | x; \mathcal{T}) \,\|\, \pi_{\text{ref}}(y | x; \mathcal{T}) \right)

where $\pi_\theta$ is the policy (the LLM being trained), $\pi_{\text{ref}}$ is the reference LLM (frozen, typically the SFT checkpoint), $\mathcal{T}$ denotes the set of available tools, $x$ is a prompt sampled from dataset $\mathcal{D}$, $y$ is the corresponding output possibly interleaved with tool-call feedback, $r_\phi$ is the reward function, $D_{\text{KL}}$ is the Kullback-Leibler divergence, and $\beta$ is the KL penalty coefficient controlling how far the policy can diverge from the reference.

What it computes: a maximization over policy parameters $\theta$ of the expected reward $\mathbb{E}[r_\phi(x,y)]$ minus a penalty $\beta \cdot D_{\text{KL}}$ that prevents the policy from drifting too far from the reference distribution. The reward $r_\phi$ is computed based on whether the final answer matches the ground truth and how tools were used. The KL divergence measures distribution shift; larger $\beta$ means stronger regularization toward the reference policy, smaller $\beta$ allows more aggressive exploration.

Why this form: this is the standard constrained RL objective used in GRPO and its variants (Shao et al., 2024; Schulman et al., 2017), but the critical extension is the conditioning on $\mathcal{T}$ — the available tool set. Unlike conventional RL for self-contained reasoning, where the policy's action space is purely token generation, here the policy can also invoke tools, and the reward signal must account for tool-use behavior (e.g., number of calls, efficiency). The KL penalty is essential because without it, the policy could collapse to degenerate strategies like calling tools excessively to avoid internal reasoning or generating nonsensical output that happens to match answer patterns. The choice of $\beta$ is one of the key hyperparameters that the paper's algorithm analysis (Section 4) implicitly varies through its clipping and reward shaping choices.

The rollout distribution factorizes into two sequential phases — agentic reasoning (multi-turn tool interactions) followed by answer generation:

Pθ(R,y∣x;T)=∏t=1tRPθ(Rt∣R<t,x;T)⋅∏t=1tyPθ(yt∣y<t,R,x;T)\mathcal{P}_\theta(R, y | x; \mathcal{T}) = \prod_{t=1}^{t_R} \mathcal{P}_\theta(R_t | R_{<t}, x; \mathcal{T}) \cdot \prod_{t=1}^{t_y} \mathcal{P}_\theta(y_t | y_{<t}, R, x; \mathcal{T})

where $R$ is the reasoning trajectory of length $t_R$ interleaved with tool-call feedback, $R_t$ is the $t$-th step of that reasoning (which may be an internal reasoning token or a tool invocation/response), $y$ is the final answer of length $t_y$, and $y_t$ is the $t$-th token of that answer.

What it computes: the joint probability of a complete agentic trajectory — first generating all reasoning steps (including tool calls and their responses), then generating the final answer conditioned on that reasoning. The first product runs over reasoning turns (up to $t_R$), each drawing from the policy conditioned on previous reasoning steps, the prompt, and available tools. The second product runs over answer tokens (up to $t_y$), conditioned on the complete reasoning trajectory. Importantly, the tool-call feedback $R_t$ includes the external tool's actual output, not just the invocation — so the policy conditions on real computation results, not simulated ones.

Why this form: this factorization makes explicit that agentic RL is a sequential decision process where the model first decides whether to call tools, which tools to call, and how to integrate their outputs, and only then produces a final answer. This stands in contrast to standard RL for reasoning where the factorization is a single product over tokens (the model just thinks and answers). The tool feedback $R_t$ introduces information that the policy did not itself generate, which is the key mechanism by which agentic RL can jointly improve pass@k and average@k (Section 4.2) — the external information expands the effective action space beyond what the model's own parameters encode.

GRPO-Based Recipes: GRPO-TCR, GRPO-SCR, and GRPO-T

The paper constructs three specific GRPO variants that differ along two primary axes — loss aggregation granularity (token-level vs. sequence-level) and the combination of clipping strategy and reward shaping — enabling controlled ablation of which algorithmic components matter for agentic RL.

The general GRPO objective the paper adopts is:

JGRPO(θ)=Ex∼D,{R}i=1G∼πref[Agg(G,R)(min⁡[ri,t(θ)⋅A^i,t,clip(ri,t(θ),1−ϵlow,1+ϵhigh)⋅A^i,t]−βDKL(πθ∥πref))]J_{\text{GRPO}}(\theta) = \mathbb{E}_{x \sim \mathcal{D}, \{R\}_{i=1}^G \sim \pi_{\text{ref}}} \left[ \text{Agg}(G, R) \left( \min \left[ r_{i,t}(\theta) \cdot \hat{A}_{i,t}, \text{clip}\left( r_{i,t}(\theta), 1 - \epsilon_{\text{low}}, 1 + \epsilon_{\text{high}} \right) \cdot \hat{A}_{i,t} \right] - \beta D_{\text{KL}}(\pi_\theta \| \pi_{\text{ref}}) \right) \right]

where $G$ is the number of sampled completions (group size), $R_i$ is the $i$-th sampled reasoning trajectory, $\text{Agg}(G, R)$ is the loss aggregation function (either token-level or sequence-level), $r_{i,t}(\theta)$ is the importance ratio (probability ratio between current and reference policy), $\hat{A}_{i,t}$ is the normalized advantage at step $t$ of trajectory $i$, $\epsilon_{\text{low}}$ and $\epsilon_{\text{high}}$ are the asymmetric clipping bounds, and $\beta$ is the KL penalty coefficient.

What it computes: a clipped surrogate objective that encourages the policy to increase the probability of actions with positive advantage (better-than-average reward) and decrease probability for negative-advantage actions, but caps the update magnitude to prevent destructive policy changes. The $\min$ operator implements the standard PPO-style conservative update: if the importance ratio would increase the objective beyond the clipping threshold for a positive advantage (or decrease it beyond the threshold for a negative advantage), the clipped version is used instead. The expectation runs over prompts from the dataset and groups of $G$ sampled trajectories per prompt.

Why this form: the core innovation relative to standard GRPO is the asymmetric clipping ($\epsilon_{\text{high}} \neq \epsilon_{\text{low}}).StandardGRPOusessymmetricclipping(‘). Standard GRPO uses symmetric clipping (`\epsilon = 0.2‘forboth),butthisconstrainsbothpositiveandnegativeupdatesequally,whichlimitsexploration—thepolicycannotquicklyincreasetheprobabilityofnewlydiscoveredgoodstrategiesbecausetheupperclipbinds.Bysetting‘` for both), but this constrains both positive and negative updates equally, which limits exploration — the policy cannot quickly increase the probability of newly discovered good strategies because the upper clip binds. By setting `\epsilon_{\text{high}} > \epsilon_{\text{low}}‘(e.g.,0.28vs.0.20forGRPO−TCR),thepaperallowsthepolicytobemoreaggressiveinadoptingpromisingbehaviors(higherupperclip)whilestillprotectingagainstcatastrophicforgetting(lowercliponpessimisticupdates).The‘` (e.g., 0.28 vs. 0.20 for GRPO-TCR), the paper allows the policy to be more aggressive in adopting promising behaviors (higher upper clip) while still protecting against catastrophic forgetting (lower clip on pessimistic updates). The `\text{Agg}$` function determines whether the objective is computed per-token (each token contributes independently) or per-sequence (the objective is shared across all tokens of a trajectory), affecting how fine-grained the optimization signal is.

The normalized advantage is computed via batch statistics:

A^i,t=rϕ(x,Ri)−mean({rϕ(R1),…,rϕ(RG)})std({rϕ(R1),…,rϕ(RG)})\hat{A}_{i,t} = \frac{r_\phi(x, R_i) - \text{mean}(\{r_\phi(R_1), \ldots, r_\phi(R_G)\})}{\text{std}(\{r_\phi(R_1), \ldots, r_\phi(R_G)\})}

where $r_\phi(x, R_i)$ is the scalar reward for trajectory $i$ on prompt $x$, and the subtraction and division use the mean and standard deviation across all $G$ trajectories in the batch.

What it computes: a z-score normalization of rewards within each batch, centering them at zero mean and scaling to unit variance. Positive advantages mean the trajectory was better than average; negative means worse.

Why this form: batch-relative normalization makes the advantage signal invariant to the absolute scale and offset of the reward function, which is critical when rewards have different magnitudes across problem types (e.g., math vs. code) or when reward shaping terms like tool bonuses shift the absolute values. Without normalization, the policy update magnitude would depend on arbitrary reward scaling choices. The use of the full batch for computing mean/std (rather than a running estimate) ensures the advantage is always relative to the current policy's performance distribution.

The two loss aggregation strategies differ in how they weight policy updates across tokens:

Token-level (AggTok):

AggTok(G,R)=1∑i=1G∣Ri∣∑i=1G∑t=1∣Ri∣\text{Agg}_{\text{Tok}}(G, R) = \frac{1}{\sum_{i=1}^G |R_i|} \sum_{i=1}^G \sum_{t=1}^{|R_i|}

with importance ratio:

ri,tTok(θ)=πθ(Ri,(t)∣Ri,<t,τ)πref(Ri,(t)∣Ri,<t,τ)r^{\text{Tok}}_{i,t}(\theta) = \frac{\pi_\theta(R_{i,(t)} | R_{i,<t}, \tau)}{\pi_{\text{ref}}(R_{i,(t)} | R_{i,<t}, \tau)}

where $|R_i|$ is the length of trajectory $i$ in tokens, $R_{i,(t)}$ is the $t$-th token of that trajectory, and $\tau$ represents the task context (prompt, tools).

What it computes: an average over all tokens across all trajectories in the batch, where each token's contribution is weighted equally regardless of which trajectory it belongs to. The importance ratio compares the current policy's probability of that token given its prefix to the reference policy's probability.

Why this form: token-level aggregation ensures that every token contributes equally to the loss regardless of trajectory length, which means the policy can learn from partial successes — even trajectories that ultimately fail may contain useful token-level patterns (e.g., a correct tool invocation followed by an arithmetic error in later reasoning). This is particularly relevant for agentic RL where trajectories can be very long (many tool interaction turns), and where the tool-use and reasoning behaviors are interleaved at the token level. The normalization by total token count across all trajectories prevents long trajectories from dominating the gradient.

Sequence-level (AggSeq):

AggSeq(G)=1G∑i=1G\text{Agg}_{\text{Seq}}(G) = \frac{1}{G} \sum_{i=1}^G

with importance ratio:

ri,tSeq(θ)=(πθ(Ri∣x,τ)πref(Ri∣x,τ))1∣Ri∣r^{\text{Seq}}_{i,t}(\theta) = \left( \frac{\pi_\theta(R_i | x, \tau)}{\pi_{\text{ref}}(R_i | x, \tau)} \right)^{\frac{1}{|R_i|}}

where $\pi_\theta(R_i | x, \tau)$ is the sequence-level probability (product of token probabilities) of the entire trajectory $R_i$.

What it computes: an average over trajectories (not tokens), where each trajectory contributes equally to the loss regardless of length. The importance ratio is the geometric mean of per-token probability ratios — it compresses the entire trajectory into a single scalar ratio by taking the $|R_i|$-th root of the sequence-level probability ratio. This means the update signal is shared uniformly across all tokens of the trajectory, rather than being token-specific.

Why this form: sequence-level aggregation implements the idea that a trajectory should be treated holistically — if it leads to a correct answer, all tokens in it were "good" in the sense that they contributed to success; if it fails, all tokens were "bad." This simplifies the credit assignment problem at the cost of losing token-level nuance. The geometric mean importance ratio ensures that the per-token update magnitude is comparable across different-length trajectories (long trajectories don't dominate the sequence-level summary). The paper's finding (Section 4.1) that token-level loss outperforms sequence-level loss for stronger models suggests that the finer-grained credit assignment is valuable when the model has sufficient exploration capacity to benefit from it.

The paper constructs three specific recipes by combining these components:

GRPO-TCR (Token-level, Clip higher, Reward shaping):

  • Loss aggregation: $\text{Agg}_{\text{Tok}}$ (token-level)
  • Clipping: $\epsilon_{\text{high}} = 0.28$, $\epsilon_{\text{low}} = 0.20$ (asymmetric, higher upper bound)
  • Reward: $r_\phi = r_{\text{out+tool}} + r_{\text{length}}$ (composite reward with overlong penalty)

GRPO-SCR (Sequence-level, Clip higher, Reward shaping):

  • Loss aggregation: $\text{Agg}_{\text{Seq}}$ (sequence-level)
  • Clipping: $\epsilon_{\text{high}} = 0.0004$, $\epsilon_{\text{low}} = 0.0003$ (extremely tight asymmetric bounds — note the much smaller magnitude compared to GRPO-TCR)
  • Reward: $r_\phi = r_{\text{out+tool}} + r_{\text{length}}$

GRPO-T (Token-level, standard GRPO baseline):

  • Loss aggregation: $\text{Agg}_{\text{Tok}}$ (token-level, same as GRPO-TCR)
  • Clipping: $\epsilon = 0.20$ (symmetric, standard PPO clip)
  • Reward: $r_\phi = r_{\text{out+tool}}$ only (no overlong penalty)
  • KL coefficient: $\beta = 0.001$

The reward function components are defined as:

Outcome + Tool Bonus (used in all three recipes):

rout+tool(x,y,n)={1+0.1nif match(yt,y)min⁡(−1+0.1n)otherwiser_{\text{out+tool}}(x, y, n) = \begin{cases} 1 + 0.1n & \text{if } \text{match}(y_t, y) \\ \min(-1 + 0.1n) & \text{otherwise} \end{cases}

where $n$ is the number of tool invocations, $y_t$ is the model's final answer, and $y$ is the ground-truth answer.

What it computes: for correct answers, the agent receives a base reward of 1 plus a bonus of 0.1 per tool call. For incorrect answers, the agent receives a minimum between -1 + 0.1n and whatever negative value would apply — essentially penalizing incorrect answers heavily regardless of how many tools were called, but with a small tool-count bonus that mitigates the penalty slightly for more tool-use attempts.

Why this form: the tool bonus $0.1n$ is designed to incentivize appropriate tool use — the agent gets rewarded for calling tools when doing so leads to correct answers (the bonus stacks on top of the 1.0 correctness reward), but the tool bonus is small enough (0.1 per call) that it does not incentivize degenerate "tool abuse" strategies where the model calls tools meaninglessly. The clipping in the incorrect case (via $\min$) ensures that even if the agent calls many tools, an incorrect answer cannot receive a reward higher than the penalty floor. The paper refers to this as "clipped to avoid degenerate tool abuse reward hacking" (Section 2.2).

Overlong Reward Shaping (used in GRPO-TCR and GRPO-SCR only):

rlength(y)={0,∣y∣≤Lmax−Lcache(Lmax−Lcache)−∣y∣Lcache,Lmax−Lcache<∣y∣≤Lmax−1,Lmax<∣y∣r_{\text{length}}(y) = \begin{cases} 0, & |y| \leq L_{\text{max}} - L_{\text{cache}} \\ \frac{(L_{\text{max}} - L_{\text{cache}}) - |y|}{L_{\text{cache}}}, & L_{\text{max}} - L_{\text{cache}} < |y| \leq L_{\text{max}} \\ -1, & L_{\text{max}} < |y| \end{cases}

where $|y|$ is the length of the generated output in tokens, $L_{\text{max}}$ is the maximum allowed response length, and $L_{\text{cache}}$ is a buffer zone before the hard limit.

What it computes: a piecewise penalty for output length. Within the "safe zone" (up to $L_{\text{max}} - L_{\text{cache}}$), no penalty is applied. Within the "buffer zone" (from $L_{\text{max}} - L_{\text{cache}}$ to $L_{\text{max}}$), a linearly increasing penalty from 0 to -1 is applied. Beyond $L_{\text{max}}$, a flat penalty of -1 is applied.

Why this form: this provides a smooth learning signal near the length boundary rather than a hard cliff. If the penalty were simply -1 for anything over $L_{\text{max}}$, gradients in the region just below the boundary would be zero (the model receives no signal about how close it is to exceeding the limit). The linear ramp in the buffer zone provides a gradient that tells the model "shorter is better, up to a point." The paper states this "preserves a smooth learning signal near the boundary while strongly discouraging overlong completions" (Section 2.2). The maximum response length is set to 16384 in the training configuration (Appendix A.1).

Why GRPO-T serves as the baseline: GRPO-T uses token-level loss (eliminating the sequence-vs-token confound when comparing to GRPO-TCR), symmetric clipping (the standard PPO/GRPO default), and no overlong shaping. By comparing GRPO-TCR against GRPO-T, the paper isolates the effect of asymmetric clipping + overlong shaping while holding loss aggregation constant. By comparing GRPO-TCR against GRPO-SCR, the paper isolates the effect of loss aggregation granularity while holding clipping and reward shaping roughly constant (though the GRPO-SCR clipping bounds are much tighter — 0.0004/0.0003 vs. 0.28/0.20 — which introduces a confound the paper does not explicitly address).

SFT Data Pipeline: Real End-to-End vs. Synthetic Stitch-Style Trajectories

The SFT pipeline constructs the initial cold-start model that RL will subsequently optimize. The paper compares two fundamentally different approaches to SFT data — one based on authentic multi-turn agent interactions, one based on retroactive editing of static reasoning chains — and shows quantitatively that the choice of SFT data determines whether RL can succeed at all.

Real end-to-end trajectory generation. The paper curates a 3k-sample SFT dataset using Qwen3-Coder-30B-A3B as the teacher model, deployed through the open-source Qwen-Agent framework with SandBoxFusion as the code interpreter backend (Appendix A.3). The data collection process is:

  1. Problem sourcing: 6k problems drawn from three pools — s1-1k (Muennighoff et al.), a self-curated 3k LeetCode dataset, and a 2k ReTool multi-turn SFT set.

  2. Trajectory rollout: For each of the 6k problems, the teacher model generates one complete multi-turn interaction. This means the model reads the problem, decides whether to invoke the code interpreter at each reasoning step, receives the actual tool output (generated by SandBoxFusion executing the code), and continues reasoning based on that real feedback. The trajectory captures the full decision sequence: internal reasoning → tool call → tool output → integration → further reasoning → (possibly more tool calls) → final answer.

  3. Quality filtering: The 3k LeetCode and 2k ReTool subsets (5k trajectories total) are scored using ReasonFlux-PRM (Zou et al., 2025) — a process-based reward model that evaluates trajectory quality. The top 1k LeetCode and top 1k ReTool trajectories are retained, combined with the full s1-1k (which is already high-quality), yielding exactly 3k trajectories.

  4. Fine-tuning: Both Qwen2.5-7B-Instruct and Qwen3-4B-Instruct-2507 are fine-tuned on this 3k dataset for 5 epochs with batch size 32, AdamW optimizer with learning rate $5 \times 10^{-5}$, and maximum response length 32768 (Appendix A.1). The resulting checkpoints are denoted Qwen2.5-7B-RA-SFT and Qwen3-4B-RA-SFT.

Synthetic stitch-style baseline. For comparison, the paper uses the multi-turn SFT dataset from ReTool (Feng et al., 2025), which constructs tool-use demonstrations by taking existing long chain-of-thought solutions (without tools) and retroactively replacing selected reasoning steps with tool invocations and their expected outputs. Specifically, where a long CoT would laboriously compute something step-by-step, the stitched version inserts a tool call that would produce the same result, and replaces the subsequent computation with the tool's output.

Why the comparison matters. The paper argues that stitch-style data introduces three specific deficiencies:

  • Missing decision cues: A stitched trajectory shows what tool was called where, but contains no signal about why the model chose that point rather than earlier or later. The real trajectory captures the model's pre-call analysis (the reasoning that led to the decision to invoke a tool), which is essential for learning adaptive tool-use strategy.

  • No error recovery: Real trajectories include cases where the tool output is unexpected or incorrect, and the model must diagnose the issue and revise its approach. Stitched data always shows perfect tool interactions because the tool outputs are filled in to match what the reasoning needed.

  • No self-calibration: Real trajectories show the model assessing whether a tool call is actually needed versus whether internal reasoning would suffice. Stitched data imposes tool calls at predetermined points regardless of whether those were the optimal moments.

The quantitative results in Table 1 validate this argument: Qwen3-4B-Instruct-2507 trained on synthetic data achieves 0.10% maj@32 on AIME2025 (meaning its answers are almost never consistent across 32 samples), while the same model trained on real trajectories achieves 51.64% maj@32. The pass@32 gap is similarly stark: 22.22% vs. 72.88%. The paper interprets this as evidence that real trajectories teach not just tool-use format but tool-use strategy — the model learns when to invoke tools and how to integrate their outputs, producing more stable and capable reasoning.

Why real trajectories are used as the SFT default for all subsequent RL experiments. Unless otherwise stated, all RL training in the paper starts from the Qwen2.5-7B-RA-SFT or Qwen3-4B-RA-SFT checkpoints, not from the base instruct models or from synthetic-data SFT models. This means the RL results should be interpreted as measuring improvement over a strong tool-using initialization, not as measuring RL's ability to teach tool use from scratch. The paper's findings about clip higher, entropy maintenance, and deliberative reasoning all operate on top of this real-trajectory SFT foundation, which is an important caveat — the results may not generalize to cases where SFT data is synthetic or low-quality.

RL Data Pipeline: Diversity and Model-Aware Filtering

The RL training dataset determines which prompts the policy encounters during optimization, shaping both what problems it learns to solve and how exploration-friendly the training dynamics are. The paper constructs a 30k-sample dataset and then explores how its diversity and difficulty profile affect outcomes.

Base dataset construction. The 30k RL dataset combines three sources (Appendix A.4):

  • DAPO-Math-17k (Yu et al., 2025): 17,000 mathematical reasoning problems. This serves as the comparison baseline for the diversity experiment in Section 3.2.

  • Skywork-or1 (He et al., 2025): 4,902 math problems + 3,586 code problems. These are high-quality, verified problems from a production reasoning model training pipeline.

  • MegaScience (Fan et al., 2025): 3,000 science problems spanning physics, chemistry, and biology domains. These introduce cross-domain diversity beyond pure mathematics.

The combined dataset covers three domains (math, code, science) with a total of approximately 28,500 problems (17,000 + 8,488 + 3,000), though the paper rounds to "30k" for simplicity and may include additional unspecified sources.

Why diversity matters — the entropy mechanism. The paper's Section 3.2 experiment compares training with the full diverse 30k dataset versus training with DAPO-Math-17k alone (math-only). The key finding is that diverse data "leads to significantly higher entropy gain during the early stage and sustains this entropy at a higher level throughout convergence" (Section 3.2). Specifically:

  • Training with diverse data achieves over 50% average@32 on AIME2025 within 150 RL steps, while the math-only baseline requires 220 steps — a 47% reduction in required training compute.

  • The entropy curves (Figure 2, right panel) show that diverse data produces entropy that rises faster initially (steeper slope in the first ~50 steps) and plateaus at a higher sustained level, while math-only data produces entropy that rises more slowly and eventually collapses to a lower level.

Why higher entropy is beneficial. The paper interprets entropy as "a proxy for exploration breadth: higher entropy means the policy continues to consider diverse reasoning paths rather than prematurely collapsing to a narrow deterministic strategy" (Section 3.2). In agentic RL specifically, exploration breadth is critical because the policy must discover not just which reasoning steps to take but also whether to call a tool at each step. The action space is combinatorially larger than in self-contained reasoning, so premature entropy collapse is more damaging — it locks the policy into suboptimal tool-use patterns before it has explored enough to discover better ones. Diverse datasets counteract this by presenting problems that require different tool-use strategies (numerical computation for math, programmatic verification for code, factual retrieval for science), naturally forcing the policy to maintain a broader distribution.

Model-aware dataset construction. Section 3.3 addresses a subtle but critical issue: not all problems in a fixed dataset are useful for training a given policy. The paper formalizes this through a difficulty-filtering procedure:

  1. Difficulty estimation: The current SFT model performs 8 rollouts per problem on the full 30k RL dataset. The proportion of correct solutions is used as a proxy for problem difficulty relative to that specific model.

  2. Filtering: Problems with 0% accuracy (the model never solves them) are discarded because they provide no positive examples — every trajectory has negative reward, so the advantage signal is uninformative (all trajectories have the same zero reward, making the gradient zero). Problems with 100% accuracy (the model always solves them) are also discarded because every trajectory has positive reward, again providing no contrastive signal.

  3. Difficulty binning: Remaining problems are labeled with three levels: easy (accuracy ≥ 0.75), medium (0.75 > accuracy > 0.25), and hard (accuracy ≤ 0.25). These bins capture the sweet spot where the problem is challenging enough to provide learning signal but not impossible.

  4. Distribution matching: Since the Qwen3-4B model already trains effectively on the full dataset, the paper uses its empirical difficulty histogram (the proportion of problems falling into easy/medium/hard bins for Qwen3) as a target distribution. The dataset for the struggling Qwen2.5-7B model is then curated to match this distribution — oversampling or undersampling from the full dataset to achieve the same easy/medium/hard proportions that work for the stronger model.

Why model-awareness works. When Qwen2.5-7B-RA-SFT trained on the unfiltered 30k dataset, its average reward stagnated around zero while Qwen3 consistently achieved positive rewards (Figure 3, right). This indicates a competence-difficulty mismatch: the weak model encounters too many problems it cannot solve, and those problems contribute noise rather than signal. By discarding impossible problems and matching the difficulty distribution to what a capable model finds productive, the paper creates a dataset where every problem provides meaningful gradients. The result is that "the average reward rises substantially, producing stronger and more consistent gradient signals" (Section 3.3), which in turn "provides more valid rewards for the computation of advantage, amplifying the gradient signals and leading to more effective and stable RL training" (Section 3.3). The paper further notes that after the weak model improves on this curated dataset, the process can be repeated — collect new rollout statistics from the improved policy, re-estimate difficulty, and re-curate the dataset — creating an iterative curriculum.

Entropy Mechanism and Exploration-Exploitation Dynamics

The paper's analysis of entropy is not merely a diagnostic but a causal mechanism that connects algorithmic design choices (clip bounds, reward shaping) to training outcomes. Understanding this mechanism requires distinguishing between standard entropy behavior in self-contained RL and the qualitatively different dynamics observed in agentic RL.

What entropy measures in this context. Policy entropy is the expected negative log-probability under the current policy:

H(πθ)=−Ex∼πθ[log⁡πθ(x)]H(\pi_\theta) = -\mathbb{E}_{x \sim \pi_\theta}[\log \pi_\theta(x)]

Higher entropy means the policy assigns more uniform probability across possible tokens — it is more "uncertain" or "exploratory." Lower entropy means the policy assigns high probability to a small set of tokens — it is "deterministic" or "exploitative." The paper tracks entropy throughout training and correlates it with performance metrics.

The standard expectation from self-contained RL. Prior work on RL for reasoning (Chen et al., 2025; Deng et al., 2025) found that most gains in pass@k (the probability that at least one of $k$ samples is correct) come from the SFT stage, where the model is exposed to diverse external solutions that expand its ability boundary. Subsequent RL training primarily strengthens existing internal solutions — the policy becomes more deterministic about which reasoning paths it takes, improving pass@1 (exploitation) but often suppressing pass@k (exploration). This is the classical exploration-exploitation trade-off: RL helps you pick better among options you already have, but doesn't help you discover new options. Entropy typically decreases during RL as the policy converges.

The agentic RL departure from this pattern. The paper's key observation in Section 4.2 is that agentic RL with the right recipe (GRPO-TCR, GRPO-SCR) achieves "substantial and simultaneous improvements in pass@k and average@k (over 10% gains on AIME2024/AIME2025)" (Section 4.2) — something that conventional RL does not achieve. This means the policy is both discovering new successful strategies (expanding pass@k) AND becoming better at deploying them (improving average@k). Entropy does not decrease during training with effective recipes; instead, it "rises faster and stabilizes at a higher level" (Section 4.3).

Why agentic RL breaks the trade-off. The paper attributes this to tool interactions introducing external information during training: "The information from the external tools enables models to 'think smarter' than purely 'think longer' by developing more advanced cognitive abilities that autonomously utilize the tools to reason more efficiently, and learn from the feedback signals" (Section 4.2). In essence, when the policy explores a new tool-use strategy and receives real computational feedback (a code execution result, a numerical computation), it learns something genuinely new — not just re-weighting existing internal representations, but acquiring new capabilities through external computation. This new information expands the effective pass@k because strategies that were previously unreachable (due to computational limitations of pure internal reasoning) become accessible once the model learns to delegate to tools.

The pass@k to average@k gap as a bottleneck. The paper observes that "RL training can be interpreted as a process of progressively converting the model's pass@k performance into actual average@k gains, with the achievable improvement bounded by an intrinsic ceiling determined by this gap" (Section 4.2). A large gap means there are many successful strategies the model can occasionally discover (high pass@k) but cannot reliably execute (low average@k), which provides room for RL to improve. A small gap means the model is already near its ability ceiling — RL can't improve much because there aren't undiscovered good strategies to exploit. This gap is therefore a diagnostic for how much remaining potential the policy has: if pass@k is high and average@k is low, more RL training should help; if both are similar, continued RL will yield diminishing returns.

The role of clip higher in maintaining entropy. The paper explicitly connects the clip upper bound $\epsilon_{\text{high}}$ to exploration budget (Section 4.3): "noting that $\epsilon_{\text{high}}$ controls the exploration budget, we utilize different $\epsilon_{\text{high}}$ to test for an optimal entropy regime." When $\epsilon_{\text{high}}$ is large (e.g., 0.28 or 0.315), the policy can increase the probability of newly discovered good actions more aggressively, which encourages it to try diverse strategies in the first place. When $\epsilon_{\text{high}}$ is small (e.g., the standard 0.20), the policy is constrained in how quickly it can adopt new behaviors, which discourages exploration because even if a better strategy is discovered, the policy cannot capitalize on it quickly.

The non-monotonic relationship between clip bound and performance. The Section 4.3 experiment varies $\epsilon_{\text{high}}$ across values 0.28, 0.315, and 0.35 for GRPO-TCR and finds:

  • For Qwen2.5-7B (weaker model): increasing from 0.28 to 0.315 improves convergence speed — "we achieve equivalent performance 40% faster, reaching the same results at step 60 that would otherwise require 100 steps" (Section 4.3). The model needs more exploration budget to escape its performance bottleneck.

  • For Qwen3-4B (stronger model): increasing from 0.28 to 0.35 degrades performance despite faster initial progress. The stronger model already explores adequately at 0.28; the higher bound "introduces excessive entropy, which will lead to suboptimal agentic reasoning performance" (Section 4.3).

Why this relationship is model-dependent. The paper's Takeaway 4.3 formalizes this as: "Weaker models require larger clip upper bounds to escape the performance bottleneck, while stronger models demand tighter bounds to prevent over-exploration." The intuition is that weak models have a very narrow distribution of strategies (low entropy) and need aggressive clipping to broaden it; strong models already have adequate entropy and excessive clipping causes instability by allowing the policy to adopt suboptimal strategies too eagerly. This is a concrete, actionable result: practitioners should tune $\epsilon_{\text{high}}$ based on their model's baseline entropy level, not use a one-size-fits-all value.

Why standard GRPO-T fails — entropy collapse. The baseline GRPO-T exhibits "early entropy collapse" (Section 4.3, Observation) — its entropy drops rapidly in the first ~50 steps and plateaus at a low level. The paper attributes this to the "overly conservative design of GRPO-T: the combination of a restrictive clip upper bound ($\epsilon = 0.20$, symmetric) and strong KL-regularization ($\beta = 0.001$) creates severe constraints on distribution shift, forcing the model to maintain self-contained generation patterns and preventing it from fully leveraging tool interactions" (Section 4.2). The symmetric clip means the policy cannot increase probability of good actions faster than it decreases probability of bad ones, which is a symmetric constraint on a fundamentally asymmetric situation — discovering tool-use strategies requires rapid adoption of promising new behaviors, not slow, symmetric adjustment.

Reasoning Mode Analysis: Deliberative vs. Reactive Tool Use

The paper's Section 5 investigates what behavior emerges from successful RL training, not just the algorithmic recipe that produces it. The key finding is that a "deliberative" reasoning mode — longer internal reasoning before each tool call, resulting in fewer but more successful tool calls — consistently outperforms a "reactive" mode with frequent, short-think tool invocation.

Characterizing the two modes. The paper defines (Section 5.1):

  • Reactive Mode: short internal reasoning segments between tool calls, high frequency of tool invocations, each call relatively simple. The model quickly delegates to tools rather than thinking through problems internally.

  • Deliberative Mode: longer internal reasoning before issuing tool calls, fewer total tool calls, each call more targeted and complex. The model invests reasoning tokens to identify exactly what computation to offload before doing so.

How the paper identifies which mode a model is in. Section 5.1 visualizes two metrics during training: the average number of tool calls per trajectory and the average response length per interaction round (both shown in Figure 7). For the successful GRPO-TCR-Qwen3-4B model (which achieves ~70% on AIME2025), the number of tool calls is relatively low (around 2–4 per trajectory) while response length per round is high (2000–2500 tokens per round). For the baseline GRPO-T models, tool calls are high and response length per round is low. The paper uses these metrics as behavioral signatures: low tool calls + high response length = deliberative; high tool calls + low response length = reactive.

Tool-use efficiency as the explanatory mechanism. The paper introduces a metric called "tool-use efficiency" — the success rate of correctly executed tool calls (Figure 8). For the deliberative GRPO-TCR/GRPO-SCR models with Qwen3-4B, this efficiency exceeds 70%. For the reactive GRPO-T models, it is substantially lower. The causal interpretation is: "Careful reasoning before acting enables highly accurate and effective calls" (Section 5.1), while rapid reactive calls "often yield ineffective or erroneous results" because the model hasn't thought through what exactly it needs the tool to compute.

Why quality-over-quantity emerges from effective RL training, not from explicit instruction. The paper does not impose a deliberative mode through prompt engineering or architectural constraints — it emerges naturally from the RL process when the right algorithmic ingredients are present (clip higher, overlong shaping, token-level loss). The paper's interpretation is that the RL process discovers that deliberative tool use is more rewarding because it leads to correct answers more often. The overlong reward shaping ($r_{\text{length}}$) plays a role here: it penalizes excessively long outputs, which means the policy cannot simply "think longer" without bound — it must use its reasoning budget efficiently. This creates pressure to think enough to make tool calls effective, but not to ramble. The tool bonus ($0.1n$ in $r_{\text{out+tool}}$) provides a countervailing pressure to actually use tools when appropriate, preventing the policy from avoiding tools entirely.

The Long-CoT integration failure as a cautionary tale. Section 5.2 and 5.3 explore what happens when the policy starts with strong internal reasoning capabilities (Long-CoT models like Qwen3-4B-Thinking-2507) rather than with instruction-tuned models. The results are striking:

  • Without SFT initialization: Long-CoT models "hardly call the tools" initially and the average number of tool calls "gradually converged to zero" during RL training (Section 5.2, Figure 9). The model discovers that its internal reasoning is good enough to solve many problems, and the RL process reinforces this by rewarding correct answers regardless of how they were obtained. The tool-use behavior is extinguished.

  • With SFT initialization (using the real-trajectory SFT data from Section 3.1): the Long-CoT model actively uses tools and achieves strong initial performance, but "ultimately achieves only comparable performance to instruction-based models rather than surpassing them" (Section 5.3). The instruction-based model actually scales better because it can focus exclusively on developing agentic reasoning from scratch, while the Long-CoT model faces "conflicting objectives" — its ingrained internal reasoning patterns must be simultaneously scaled up (for better answers) and pruned back (to make room for tool use).

Why instruction-based models are more suitable for agentic RL. The paper's Takeaway 5.3 argues that "Instruction-based models are more suitable for agentic RL that scales the agentic reasoning ability from scratch compared to Long-CoT models with internal reasoning priors." The mechanism is that instruction-based models start with a "blank slate" for tool-use strategy — they have no strong predisposition either toward or against tool use, so RL can shape the behavior from scratch. Long-CoT models, by contrast, have been optimized for self-contained reasoning and have developed what the paper calls "ingrained internal reasoning patterns [that] contradict agentic reasoning paradigms, forcing a scaling and pruning process where gains in agentic reasoning are offset by the need to suppress over-thinking behaviors" (Section 5.3). The optimization must simultaneously learn to use tools AND unlearn the habit of relying entirely on internal reasoning, which fragments the learning signal. The paper quantifies this fragmentation through the response length dynamics during training (Figure 10, right panel): instruction-based models show steadily growing response lengths as they learn to reason, while Long-CoT models show fluctuating response lengths as they oscillate between thinking internally and using tools.

The relationship between reasoning mode and response length scaling. The paper's analysis of response length during training (Figure 10, right) reveals that instruction-based models follow a monotonic trajectory: response length gradually increases as the model develops more sophisticated agentic reasoning patterns. Long-CoT models show a non-monotonic pattern: response length initially decreases (as the model learns to use tools instead of long internal reasoning), then fluctuates. This is interpreted as evidence of the optimization conflict — the policy cannot simultaneously scale up its reasoning depth (which Long-CoT priors encourage) and use tools efficiently (which requires knowing when to stop internal reasoning).

Summary of Design Choices and Their Justifications

  • Real end-to-end SFT data over stitched synthetic data: captures pre-call analysis, error recovery, and self-calibration behaviors that stitching cannot reproduce; empirically yields +28.85% average@32 improvement on AIME2024 for the 4B model.

  • Asymmetric clipping ($\epsilon_{\text{high}} > \epsilon_{\text{low}}$) over symmetric standard GRPO clipping: allows the policy to rapidly adopt newly discovered good strategies (higher upper bound) while still protecting against catastrophic forgetting (lower bound); the primary driver of the GRPO-TCR performance advantage over GRPO-T.

  • Overlong reward shaping over no length penalty: provides a smooth gradient signal to prevent the policy from generating excessively long responses, which is especially important in agentic settings where tool-call loops can cause runaway generation; combined with the tool bonus to incentivize efficient tool use.

  • Token-level loss aggregation over sequence-level for stronger models: provides finer-grained credit assignment, allowing the policy to learn from partial successes within otherwise failed trajectories; sequence-level loss is competitive for weaker models but limits learning efficiency for stronger ones.

  • Diverse RL dataset (math + code + science) over math-only: sustains higher policy entropy throughout training by exposing the policy to problems requiring qualitatively different tool-use strategies, preventing premature convergence to narrow solutions.

  • Model-aware difficulty filtering over fixed datasets: discards problems the current policy cannot solve (zero gradient signal) or trivially solves (no contrastive signal), and matches difficulty distribution to a target derived from a capable model; breaks performance bottlenecks for weak models.

  • Model-dependent clip upper bound tuning over one-size-fits-all: weaker models need larger $\epsilon_{\text{high}}$ to broaden their exploration distribution; stronger models need smaller $\epsilon_{\text{high}}$ to prevent over-exploration and instability.

  • Instruction-based model initialization over Long-CoT model initialization: avoids conflicting optimization objectives between internal reasoning priors and tool-use learning; enables the policy to develop agentic reasoning from scratch without needing to unlearn ingrained self-contained reasoning habits.

  • Qwen3-Coder-30B-A3B as teacher for SFT data over other teacher models: the paper does not justify this specific choice beyond noting it is deployed through the Qwen-Agent framework; presumably selected for strong coding and reasoning capabilities needed to generate high-quality trajectory demonstrations.

  • ReasonFlux-PRM for trajectory filtering over other quality metrics: a process-based reward model (Zou et al., 2025) that evaluates the quality of intermediate reasoning steps, not just final answers; used to retain only the highest-quality 2k trajectories from the initial 5k LeetCode+ReTool rollouts.

  • 8 rollouts per problem for difficulty estimation over using fewer: the paper does not justify the choice of 8 specifically; it is a trade-off between statistical reliability of the accuracy estimate and computational cost of the difficulty estimation step. Higher numbers would give more precise binning but increase preprocessing cost exponentially.

4. Key Insights and Innovations

Innovation 1: Reframing Test-Time Compute Allocation as a Difficulty-Conditioned Diagnostic, Not a Uniform Optimization Problem

The paper's most distinctive conceptual move is not proposing a new algorithm, but recasting the entire agentic RL problem as one of model-dependent difficulty matching. Prior work on RL for reasoning (Shao et al., 2024; Guo et al., 2025; Yu et al., 2025) treated the training dataset as a fixed resource — you pick a dataset, you run your RL algorithm on it, and you measure improvement. The implicit assumption was that if a problem is in the dataset, it contributes usefully to training. Section 3.3 demolishes this assumption with a stark empirical observation: Qwen2.5-7B-RA-SFT trained on the full 30k dataset sees its average reward stagnate around zero, while Qwen3-4B-RA-SFT on the same dataset consistently achieves positive rewards. The same data, same algorithm, different models — and one learns nothing.

The conceptual reframing is that training data usefulness is a function of the policy's current capability, not an intrinsic property of the problem. A problem that the policy solves 0% of the time provides zero contrastive signal because every trajectory receives the same (negative) reward; the advantage $\hat{A}_{i,t}$ computed across the batch is zero (or near-zero noise), so the policy update is vacuous. Symmetrically, a problem solved 100% of the time provides no contrastive signal because every trajectory is positive. The paper operationalizes this insight through a concrete diagnostic: sample 8 rollouts per problem, discard problems at the floor (0%) or ceiling (100%), and match the difficulty distribution to a target derived from a model that trains successfully. This is a curriculum learning mechanism triggered by measured policy competence, not a hand-designed schedule.

What makes this distinctive relative to prior work is that existing difficulty-curation approaches (e.g., training on problems of increasing difficulty, or using heuristic difficulty labels from the dataset) are static — they assume difficulty is a property of the problem independent of the solver. The paper shows that the same problem can be easy for Qwen3-4B and impossible for Qwen2.5-7B, and that this mismatch — not the absolute difficulty — determines whether gradient signals are informative. The iterative re-curation proposal (re-estimate difficulty after the policy improves, re-filter the dataset) makes this a dynamic, training-aware curriculum that tracks the policy's evolving competence.

The practical implication is significant beyond the 4B/7B models studied: this reframing provides a principled answer to why some RL training runs fail silently (the model doesn't improve but doesn't crash either — it's training on problems that give zero-gradient signal) and how to diagnose such failures (look at the average reward distribution per problem, not just the aggregate). Table 3 (Figure 3 in the paper) validates the approach concretely: the model-aware dataset breaks Qwen2.5-7B out of its plateau and enables it to learn where the unfiltered dataset could not.

Prior work that this reframing challenges: Most GRPO-based agentic RL work (ReTool, ToRL, ARPO, Tool-Star) uses fixed, curated datasets without difficulty filtering relative to the current policy. The implicit assumption was that dataset diversity and quality were sufficient. This paper shows they are necessary but not sufficient — difficulty matching is an independent, critical axis.

Why this is a conceptual advance rather than an algorithmic one: The paper does not introduce a new loss function or architecture for difficulty-aware training. Instead, it provides a diagnostic framework (monitor per-problem solve rates during RL, filter accordingly) that changes how practitioners think about data selection. This is a methodological contribution that generalizes across algorithms and model sizes.

Innovation 2: Entropy as a Causal Mechanism, Not Merely a Correlate, in Agentic RL Training Dynamics

The role of policy entropy in RL for LLM reasoning has been a site of active debate: some work advocates entropy minimization for deterministic policies (Agarwal et al., 2025; Cheng et al., ), others celebrate high-entropy tokens as the primary drivers of learning (Cui et al., 2025; Wang et al., ), and still others observe entropy spikes after tool calls and design adaptive mechanisms around them (ARPO; Dong et al., ). The paper's contribution is to move beyond this correlational debate by establishing entropy as a controllable variable that causally determines training efficiency and peak performance, and then showing that the optimal entropy level depends on model capacity.

The paper does this through a controlled experiment that varies the clip upper bound $\epsilon_{\text{high}}$ (0.28, 0.315, 0.35) while holding all other hyperparameters fixed (Section 4.3, Figure 6). Because $\epsilon_{\text{high}}$ directly limits how much the policy can increase the probability of newly discovered good actions, it functions as an exploration budget controller — a knob that translates directly into observed policy entropy. The finding is non-monotonic: for Qwen2.5-7B, increasing $\epsilon_{\text{high}}$ from 0.28 to 0.315 accelerates convergence by ~40% (reaching at step 60 what previously required step 100). For Qwen3-4B, increasing to 0.35 degrades final performance despite faster initial progress. This non-monotonicity establishes that entropy is not "more is better" or "less is better" — it is model-dependent, and there is an empirically identifiable optimal range.

What distinguishes this from prior entropy analyses is the causal intervention rather than post-hoc correlation. Prior work (Cui et al., 2025; Deng et al., 2025) documented that entropy correlates with performance, but did not systematically vary the entropy-controlling hyperparameter to show that changing entropy causes performance changes. The paper's experiment with three $\epsilon_{\text{high}}$ values provides exactly this causal evidence. Moreover, the model-dependent optimal range — weaker models need more exploration budget, stronger models need less — is a finding that prior entropy work, which focused on single-model analyses, could not have discovered.

The significance of this insight extends beyond agentic RL. It provides a concrete diagnostic for practitioners: if your model's entropy is low and performance is plateauing, increase $\epsilon_{\text{high}}$; if entropy is high and performance is unstable or degrading, decrease it. It also explains why the standard symmetric clip ($\epsilon = 0.20$) used in baseline GRPO-T fails for agentic RL: it imposes a uniform exploration budget that is too restrictive for the combinatorially larger action space of tool-use strategies, causing premature entropy collapse (Section 4.3, Observation).

Prior work that this reframing challenges: The entropy-minimization perspective (Agarwal et al., 2025), which treats entropy as a regularizer to be reduced, and the entropy-maximization perspective (Wang et al., ), which treats entropy as an unalloyed good. The paper shows both are partially correct but incomplete — the answer depends on the model's current capability relative to the task distribution.

Why this is a conceptual advance: The paper transforms entropy from a post-hoc diagnostic (something you observe and wonder about) into a design variable (something you control through $\epsilon_{\text{high}}$ and optimize for your specific model). This operationalization makes entropy management a concrete engineering practice rather than a theoretical debate.

Innovation 3: The Exploration-Exploitation Paradox — Why Agentic RL Can Jointly Improve Pass@k and Average@k While Self-Contained RL Cannot

One of the paper's most counterintuitive empirical findings is that agentic RL with effective recipes produces simultaneous improvements in both pass@k and average@k, a pattern that violates the classical exploration-exploitation trade-off observed in self-contained reasoning RL. In standard RL for reasoning (Chen et al., 2025; Deng et al., 2025), RL training improves pass@1 (exploitation — the policy gets better at picking the right answer among strategies it knows) but often suppresses pass@k (exploration — the policy's ability to discover novel correct strategies narrows). The standard explanation is that RL makes the policy more deterministic, collapsing its distribution around the strategies that worked during training, which improves reliability at the cost of diversity.

The paper's Section 4.2 (Figure 4) shows that with GRPO-TCR and GRPO-SCR, both pass@32 and average@32 improve by over 10 percentage points on AIME2024/AIME2025. This means the policy is simultaneously discovering more successful strategies (pass@k goes up) AND executing them more reliably (average@k goes up). The standard trade-off is broken.

The paper's explanation for this paradox is conceptually distinctive: tool interactions inject external information into the policy's action space, expanding the effective pass@k boundary. In self-contained reasoning, the policy can only discover strategies that are representable within its own parameter space — pass@k is bounded by what the model "knows" from pretraining and SFT. RL can re-weight these internal strategies but cannot create genuinely new computational capabilities. In agentic reasoning, when the policy explores a new tool-use pattern — say, deciding to compute a numerical integral via code execution rather than analytical approximation — it receives real external computation that it could not have performed internally. This new information constitutes a genuinely novel capability, expanding pass@k beyond what was accessible through pure internal reasoning.

The paper frames this through an elegant bottleneck concept: "RL training can be interpreted as a process of progressively converting the model's pass@k performance into actual average@k gains, with the achievable improvement bounded by an intrinsic ceiling determined by [the pass@k–average@k] gap" (Section 4.2). In self-contained RL, this gap typically narrows as pass@k drops — the ceiling lowers. In agentic RL with effective recipes, the gap widens because pass@k grows faster than average@k can catch up, creating headroom for continued improvement. This reframing transforms the pass@k–average@k relationship from a simple trade-off into a diagnostic of how much remaining learning potential the policy has.

Prior work that this reframing challenges: The conventional view from Chen et al. (2025) and Deng et al. (2025) that RL primarily improves exploitation at the cost of exploration. The paper shows this view is specific to self-contained reasoning and does not generalize to settings where the agent can access external computation.

Why this is a conceptual advance rather than just a performance result: The paper provides a mechanistic explanation — external information injection through tool feedback — for why the pattern differs, connecting an empirical observation to a causal mechanism. This has implications for how we think about RL scaling: if pass@k expansion requires external information sources, then the ceiling on agentic RL improvement is determined by tool diversity and quality, not just by model size or training compute. This reframes tool integration not as a convenience feature but as a fundamental mechanism for breaking the exploration ceiling that limits self-contained reasoning RL.

Innovation 4: The Long-CoT Interference Effect as a Negative Result That Clarifies When and Why Transfer Fails

The paper's Section 5.2 and 5.3 contain what is arguably its most important negative result: long-chain-of-thought reasoning models, despite their superior self-contained reasoning capabilities, actively resist tool use during agentic RL and ultimately underperform instruction-based models trained from scratch for agentic reasoning. This finding runs counter to the intuitive expectation that a better reasoner should make a better agent — if you can think better, surely you can use tools better too.

The paper documents this with two complementary experiments. First, when a Long-CoT model (Qwen3-4B-Thinking-2507) is plugged directly into agentic RL without any SFT initialization, it performs well initially (strong internal reasoning solves many problems) but the number of tool calls "gradually converged to zero" during training (Section 5.2, Figure 9). The RL process discovers that internal reasoning alone is sufficient for many problems, and — since the reward function does not explicitly require tool use — the policy optimizes away the tool calls entirely. The agent becomes a good self-contained reasoner that ignores its tools.

Second, when the Long-CoT model is initialized with the real-trajectory SFT data (Section 5.3), it learns to use tools and achieves decent performance, but ultimately only matches — rather than surpasses — the instruction-based model. The analysis of response length dynamics (Figure 10, right) reveals why: the Long-CoT model's response length fluctuates during training, reflecting an ongoing conflict between scaling up internal reasoning (which the Long-CoT priors encourage) and pruning it back to make room for tool use (which agentic RL rewards). The instruction-based model, by contrast, shows a monotonic increase in response length as it develops agentic reasoning from a blank slate, focusing its entire optimization budget on learning tool-use strategy rather than splitting attention between learning and unlearning.

The conceptual contribution is the recognition that reasoning capability and agentic capability can be in tension — the skills that make a good internal reasoner (persistence, thoroughness, reliance on learned knowledge) can be liabilities for an agent that needs to know when to stop thinking and delegate to external computation. The paper frames this as "conflicting objectives" where "ingrained internal reasoning patterns contradict agentic reasoning paradigms, forcing a scaling and pruning process where gains in agentic reasoning are offset by the need to suppress over-thinking behaviors" (Section 5.3). This is a negative transfer effect specific to the reasoning-to-agentic transition: pretraining on task A (self-contained reasoning) makes it harder to learn task B (agentic reasoning) because the optimization must both acquire new skills and suppress old habits.

Prior work that this reframing challenges: Several works (Search-R1, Jin et al., 2025; R1-Searcher, Song et al., 2025; Search-o1, Li et al., ) have successfully combined Long-CoT models with search engine tools. The paper acknowledges this success but distinguishes between knowledge-intensive tasks (where the model recognizes it lacks information and willingly invokes search) and reasoning-intensive tasks (where the model believes — often correctly — that internal reasoning can solve the problem, and therefore avoids tools). The paper's contribution is to identify task type as the moderator of whether Long-CoT priors help or hurt agentic RL: knowledge tasks benefit, reasoning tasks suffer.

Why this is a significant negative result: It prevents the field from pursuing a dead-end strategy — "just start with the best reasoning model and add tools" — and instead directs attention toward building agentic reasoning from instruction-tuned or base models that don't carry strong self-contained reasoning priors. It also explains why instruction-based models like Qwen3-4B-Instruct (not Qwen3-4B-Thinking) serve as better starting points for agentic RL, a practical guidance that was not obvious a priori.

Innovation 5: Quality-Over-Quantity as an Emergent Behavioral Principle, Not an Architectural Constraint

The paper's reasoning mode analysis (Section 5) discovers that effective agentic RL training produces agents that adopt a "deliberative" strategy — longer internal reasoning between tool calls, fewer total calls, higher per-call success rate — without any explicit reward shaping or architectural constraint to enforce this behavior. The tool bonus $0.1n$ in the reward function encourages tool use (more calls → more bonus), yet the best models use fewer calls than the baseline. The overlong penalty $r_{\text{length}}$ discourages excessive length, but does not specifically target tool-call frequency versus internal reasoning length. Somehow, the policy discovers that thinking before calling is more effective than calling frequently, and this behavior emerges from the RL process itself rather than being imposed from outside.

The mechanism the paper identifies is tool-use efficiency (Section 5.1, Figure 8): deliberative agents achieve over 70% success rates on their tool calls, while reactive agents have substantially lower success rates. The causal chain is: more internal reasoning → better problem decomposition → more targeted tool calls → higher probability of correct tool outputs → more correct final answers → higher reward. The RL process discovers this chain because correct tool outputs increase the probability of correct final answers, and the outcome-based reward $r_{\text{out+tool}}$ (which gives 1.0 base for correctness plus 0.1 per tool call) naturally favors trajectories where tool calls are effective. Importantly, the tool bonus $0.1n$ is small enough that the correctness term dominates — the model does not learn to spam tool calls for the bonus because an incorrect answer with many calls gets penalized heavily by the baseline term ($\min(-1 + 0.1n)$ stays negative).

What makes this distinctive is that it demonstrates emergent strategy optimization without explicit strategy-level rewards. The paper does not train a separate "tool-use quality" reward model or impose a budget on the number of tool calls. The deliberative strategy emerges because the environment (correctness checking) implicitly penalizes bad tool use more than it rewards tool-use quantity. This is a form of implicit curriculum: the policy initially tries tool calls at random, receives negative feedback when calls produce wrong results, learns to be more careful about when to call, and gradually shifts from quantity to quality.

The contrast with prior work is instructive. Methods like ReTool (Feng et al., 2025) and ToRL (Li et al., ) also use RL to optimize tool use, but they do not characterize what mode of tool use emerges or why. The paper's contribution is to measure the behavioral signature (tool-call frequency, response length per round, tool-use success rate), identify the deliberative-vs-reactive distinction, and trace it causally to the RL dynamics rather than to prompting or dataset design. This transforms tool-use strategy from a design choice (you decide how many tool calls to allow) into a learned behavior that can be studied and optimized.

Prior work that this reframing challenges: The template-based tool-use paradigm (ToRA, Toolformer, MathCoder), where tool invocation patterns are hard-coded into the training data and the model learns to reproduce them. The paper shows that RL-trained agents can discover strategies that deviate from and outperform the template distribution — the deliberative mode is not present in the SFT data (which has a fixed, arbitrary tool-call frequency set by the teacher model) but emerges through outcome-driven optimization.

Why this is a conceptual advance: It demonstrates that agentic RL is not just optimizing tool-use accuracy but discovering tool-use strategies — a qualitatively higher-level behavior. This opens the door to studying agent behavior through the lens of emergent strategy optimization, where the policy's learned approach to tool orchestration becomes itself a subject of scientific investigation, not just an engineering artifact.

5. Experimental Analysis

Evaluation Methodology

  • Dataset. The paper evaluates on four challenging benchmarks: (1) AIME2024 and AIME2025 — American Invitational Mathematics Examination problems drawn from the MATH competition benchmark family (Hendrycks et al., 2021); (2) GPQA-Diamond (Rein et al., 2024) — graduate-level science QA spanning physics, chemistry, and biology; and (3) LiveCodeBench-v6 (Jain et al.) — a contamination-free code generation benchmark. No specific test-set split size is reported beyond the standard sizes for these public benchmarks. For AIME2024/2025 and GPQA-Diamond, the paper samples 32 completions per problem for evaluation; for LiveCodeBench-v6, it follows the official pass@1 and pass@5 protocol (Appendix A.2).

  • Base model(s). All experiments use either Qwen2.5-7B-Instruct (Team, 2024) or Qwen3-4B-Instruct-2507 (Yang et al., 2025a) as the starting point for SFT and subsequent RL training. The paper argues these two models represent different capability regimes — the 4B model is stronger at self-contained reasoning (63.3% on AIME2024 versus 16.7% for the 7B model; Table 2) — which enables analysis of how model capacity interacts with agentic RL dynamics. A third model, Qwen3-4B-Thinking-2507 (a Long-CoT variant), is used in Section 5.2–5.3 specifically for Long-CoT integration experiments. The SFT teacher model for trajectory generation is Qwen3-Coder-30B-A3B (Appendix A.3). No architectural modifications are made to any base model; all training is parameter-efficient fine-tuning of the full model weights.

  • Metrics. Three primary evaluation metrics are used for math and science benchmarks (Section 4.1, Appendix A.2):

    • average@k: The mean accuracy across $k$ sampled completions per problem, measuring overall agent performance (exploitation quality). Computed by sampling $k$ trajectories per prompt, extracting the final answer from each, and averaging correctness.
    • pass@k: The fraction of problems for which at least one of $k$ sampled completions is correct, measuring the ability boundary (exploration ceiling). Computed as the proportion of prompts where any of the $k$ samples produces the correct answer.
    • maj@k: Majority voting accuracy — the most common answer among $k$ samples is selected, measuring performance stability. Typically lower than average@k for deterministic models, higher when the policy consistently produces correct answers.

    For LiveCodeBench-v6, the paper reports pass@1 and pass@5 following the benchmark's official guidelines (Appendix A.2). During RL training, the paper also tracks policy entropy (expected negative log-probability under $\pi_\theta$ averaged across the batch), average reward (mean $r_\phi$ across rollouts), and tool-use efficiency (proportion of correctly executed tool calls among all tool invocations; Section 5.1). When $k$ is not specified, results refer to $k=32$ (the paper's default evaluation budget).

  • Baselines. The paper compares against multiple categories of prior work (Table 2):

    • Self-contained reasoning baselines: Qwen2.5-7B-Instruct, Qwen3-4B-Instruct-2507, Qwen2.5-72B-Instruct, DeepSeek-V3, DeepSeek-R1-Distill-32B, and DeepSeek-R1-Zero (671B). These are evaluated with prompts that instruct the model to rely solely on internal reasoning without tool access (Appendix B.2).
    • Agentic reasoning baselines: ToRL-7B (Li et al., 2025d), ReTool-32B (Feng et al., 2025), Tool-Star-3B (Dong et al., 2025a), ARPO-7B (Dong et al., 2025b), and rStar2-Agent-14B (Shang et al., 2025). These are prior methods that train agents with tool access using various RL recipes, evaluated under comparable conditions.
    • Internal algorithmic baselines: Within the paper's own experiments, GRPO-T (standard token-level GRPO with symmetric clip $\epsilon = 0.20$ and no overlong shaping, Section 2.3) serves as the primary controlled baseline against which GRPO-TCR and GRPO-SCR are compared.
    • Data baselines: For SFT experiments (Section 3.1), the synthetic stitch-style dataset from ReTool (Feng et al., 2025) is compared against the paper's curated real end-to-end trajectories. For RL diversity experiments (Section 3.2), DAPO-Math-17k (Yu et al., 2025) as a math-only dataset is compared against the paper's diverse 30k dataset.

  • Generation budget / compute accounting. The paper measures compute in terms of RL training steps, where each step processes a batch of prompts with $G$ sampled rollouts per prompt (Section 2.2). The default configuration uses batch size 64, 3 training epochs over the 30k RL dataset, and maximum response length 16,384 tokens (Appendix A.1). All training runs use 8× Tesla-A100-80G GPUs (Appendix A.1). This is primarily a training efficiency comparison — convergence speed is measured in steps-to-reach-threshold (e.g., Section 3.2: diverse dataset reaches 50% average@32 in 150 steps vs. 220 for math-only; Section 4.3: ϵhigh = 0.315 reaches the same performance at step 60 that required 100 steps at 0.28). The paper does not report total GPU-hours or FLOP counts, making cross-method cost comparisons approximate. For evaluation, the standard budget is 32 samples per problem at temperature 1.0 and top-p 0.6 (Appendix A.2), though this is a fixed evaluation protocol rather than a scaled compute sweep.

  • Cross-validation / statistical protocol. The paper does not employ cross-validation or multiple random seeds for its main RL training runs. The primary mode of validation is controlled ablation: varying one factor (e.g., clip bound, dataset composition, SFT data source, model size) while holding all other hyperparameters fixed, and reporting the resulting performance curves over training steps. This is appropriate for an empirical investigation paper whose goal is to identify which factors matter rather than to produce a maximally tuned model. However, it means that the reported optimal hyperparameters ($\epsilon_{\text{high}} = 0.315$ for Qwen2.5-7B, $\epsilon_{\text{high}} = 0.28$ or $0.315$ for Qwen3-4B) are based on single runs and may not represent stable optima. For the DemyAgent-4B final model (Section 6), the paper reports a single training run using the best configuration identified from the ablation experiments, evaluated across four benchmarks. No confidence intervals or standard deviations are reported for any metric, which limits the ability to assess whether reported differences (e.g., 68.13% vs. 40.93% on AIME2025 between GRPO-TCR and GRPO-T) are statistically robust or partially attributable to training noise.

Main Quantitative Results

SFT Data Quality: Real End-to-End vs. Synthetic Stitch Trajectories

The headline result from Section 3.1 and Table 1 is that real end-to-end agentic trajectories produce a dramatically stronger SFT initialization than synthetic stitch-style trajectories, with the improvement being larger for the smaller model.

For Qwen3-4B-Instruct-2507 (Table 1):

  • On AIME2024, real trajectories achieve 33.23% average@32 versus 4.38% for synthetic data — an absolute gain of +28.85 percentage points. The pass@32 (ability boundary) improves from 35.15% to 75.66% (+40.51 pp), and maj@32 (stability) improves from 0.01% to 51.64% (+51.63 pp).
  • On AIME2025, real trajectories achieve 29.79% average@32 versus 3.65% (+26.14 pp). Pass@32: 72.88% vs. 22.22% (+50.66 pp). Maj@32: 45.82% vs. 0.10% (+45.72 pp).

For Qwen2.5-7B-Instruct (Table 1), the gains are smaller but still substantial:

  • AIME2024 average@32: 6.77% (synthetic) → 17.91% (real, +11.14 pp). Pass@32: 42.11% → 57.57% (+15.46 pp).
  • AIME2025 average@32: 5.21% → 18.24% (+13.03 pp). Pass@32: 25.56% → 48.42% (+22.86 pp).

The most striking diagnostic is the maj@32 metric: the synthetic-data model achieves nearly zero majority-voting accuracy (0.01% on AIME2024 for Qwen3-4B, 0.10% on AIME2025), meaning that even though it occasionally produces correct answers (pass@32 of 35% and 22% respectively), those correct answers are inconsistent and never form a majority across 32 samples. The real-trajectory model achieves maj@32 of 51.64% and 45.82%, indicating that its correct answers are consistently reproducible — a hallmark of genuine understanding rather than lucky sampling.

No other SFT data comparison is presented in the paper; all subsequent RL experiments use the Qwen3-4B-RA-SFT and Qwen2.5-7B-RA-SFT checkpoints trained on real trajectories.

RL Data Diversity and Model-Awareness

Diversity effect (Section 3.2, Figure 2). Training Qwen3-4B-RA-SFT with GRPO-TCR on the diverse 30k dataset (math + code + science) versus DAPO-Math-17k (math only) reveals:

  • Convergence speed: The diverse dataset reaches over 50% average@32 on AIME2025 within 150 RL steps, while the math-only dataset requires 220 steps to reach the same threshold — a 47% reduction in required training steps (Figure 2, left panel).
  • Entropy dynamics: The diverse dataset produces policy entropy that "rises faster and stabilizes at a higher level throughout convergence" (Figure 2, right panel). The entropy curves diverge early (within the first ~50 steps) and maintain a persistent gap through the full training run. The paper attributes this to diverse problems requiring qualitatively different tool-use strategies, which naturally forces the policy to maintain a broader distribution.

Model-aware filtering (Section 3.3, Figure 3). For Qwen2.5-7B-RA-SFT, the unfiltered 30k dataset produced stagnant training — average reward "stagnated around zero" while Qwen3 consistently achieved positive rewards (Figure 3, right panel). After applying model-aware filtering (discarding 0% and 100% solve-rate problems, matching difficulty distribution to Qwen3's empirical histogram), training on the curated dataset:

  • Performance: The model escapes its plateau and achieves meaningful improvement on AIME2025 average@32 (Figure 3, left panel), though exact final numbers are shown in a training curve rather than as a scalar comparison.
  • Reward dynamics: Average reward rises substantially, "producing stronger and more consistent gradient signals" (Figure 3, right panel). The paper notes this provides "more valid rewards for the computation of advantage, amplifying the gradient signals and leading to more effective and stable RL training" (Section 3.3).

The paper does not report an explicit side-by-side average@32 comparison between filtered and unfiltered training at a fixed step count, instead relying on the reward curves as the primary evidence of improved training dynamics.

Algorithmic Recipe Comparison: GRPO-TCR vs. GRPO-SCR vs. GRPO-T

The central algorithmic experiment (Section 4.1, Figure 4) compares three GRPO variants on both Qwen3-4B-RA-SFT and Qwen2.5-7B-RA-SFT, evaluated on AIME2024 and AIME2025.

Qwen3-4B-RA-SFT results (the stronger model):

  • GRPO-TCR (token-level, asymmetric clip 0.28/0.20, overlong shaping): Achieves 70.93% average@32 on AIME2024 and 68.13% on AIME2025 within 450 RL steps, starting from SFT initialization of 33.23% and 29.79% respectively (Section 4.1, Figure 4). This represents a gain of +37.70 pp on AIME2024 and +38.34 pp on AIME2025.
  • GRPO-T (token-level, symmetric clip 0.20, no overlong shaping): Achieves best average@32 of 54.7% on AIME2024 and 40.93% on AIME2025 (Section 4.1). The paper emphasizes that GRPO-TCR reaches this performance level within only ~100 training steps — utilizing roughly 25% of the training computation GRPO-T requires (Section 4.1: "GRPO-TCR could achieve within only 100 training steps, utilizing only 25% of the training computation of GRPO-T").
  • GRPO-SCR (sequence-level, asymmetric but very tight clip 0.0004/0.0003, overlong shaping): Underperforms GRPO-TCR by 3.95% on AIME2024 and 3.86% on AIME2025 under the same training budget of 450 steps (Section 4.1). The paper attributes this to token-level loss "ensuring each token contributes equally to the optimization signal, thereby leveraging the model's exploratory capacity more effectively" (Section 4.1).
  • Convergence dynamics: GRPO-TCR consistently outperforms the other recipes across the full training trajectory (Figure 4, multiple sub-panels showing average@32, pass@32, maj@32, and entropy vs. steps). The pass@32 and average@32 both improve simultaneously (+10 pp or more), which the paper flags as distinctive to agentic RL (Section 4.2).

Qwen2.5-7B-RA-SFT results (the weaker model):

  • Token-level (GRPO-TCR) and sequence-level (GRPO-SCR) achieve comparable average@32 performance on both benchmarks (Section 4.1: "token-level loss and sequence-level loss achieve comparable average@32 performance on AIME2024/2025"). This differs from the Qwen3-4B result where token-level clearly dominated.
  • The paper interprets this model-dependent result as evidence that "token-level loss could improve training efficiency and agentic reasoning ability compared to sequence-level loss for models with better initial performance and exploration ability" (Section 4.1, Takeaway 4.1.2), while weaker models benefit less from the finer-grained credit assignment because their exploration is too limited to produce diverse token-level patterns.
  • The exact numerical gap between GRPO-TCR and GRPO-T for Qwen2.5-7B is visible in Figure 4's training curves but not reported as a scalar comparison in the text.

Entropy dynamics across recipes (Section 4.3, Figure 5): The entropy trajectories (Figure 5) reveal a clear pattern: GRPO-T exhibits "early entropy collapse" — entropy drops rapidly and plateaus at a low level — while GRPO-TCR and GRPO-SCR both show entropy "rising faster and stabilizing at a higher level." The paper correlates this with performance: the higher-entropy recipes are also the higher-performing ones, and within those recipes, the relative entropy ranking (GRPO-TCR > GRPO-SCR) matches the performance ranking for the stronger model.

Clip Upper Bound Sweep (Entropy Control Experiment)

Section 4.3 (Figure 6) systematically varies $\epsilon_{\text{high}}$ in GRPO-TCR at values 0.28, 0.315, and 0.35, holding all other hyperparameters constant.

For Qwen2.5-7B-RA-SFT (Figure 6, left):

  • Increasing $\epsilon_{\text{high}}$ from 0.28 to 0.315 accelerates convergence noticeably — the paper states "we achieve equivalent performance 40% faster, reaching the same results at step 60 that would otherwise require 100 steps when $\epsilon_{\text{high}} = 0.28$" (Section 4.3). The training curves show the 0.315 variant rising faster and sustaining a higher accuracy plateau.
  • The paper does not report a 0.35 run for Qwen2.5, so the optimal upper bound for the weak model is ≥0.315 but the ceiling is not determined.

For Qwen3-4B-RA-SFT (Figure 6, right):

  • $\epsilon_{\text{high}} = 0.315$ provides faster initial lift than 0.28 — the curve rises more steeply in the first ~50 steps — but the paper notes that 0.28 and 0.315 converge to comparable final performance (the curves overlap in the later training stages).
  • $\epsilon_{\text{high}} = 0.35$ degrades training effectiveness: despite faster initial progress, the final performance is worse than both 0.28 and 0.315 (Figure 6, right: the 0.35 curve shows more erratic behavior and lower ultimate accuracy). The paper diagnoses this as "excessive entropy [that] will lead to suboptimal agentic reasoning performance" and instability (Section 4.3).
  • The paper's conclusion: "Weaker models require larger clip upper bounds to escape the performance bottleneck, while stronger models demand tighter bounds to prevent over-exploration" (Takeaway 4.3.2).

The paper evaluates these runs using average@32, pass@32, and maj@32 on AIME2025 (Figure 6), providing a multi-faceted view. The three metrics show consistent relative ordering across clip values, ruling out the possibility that one metric improves at the expense of another.

Reasoning Mode: Deliberative vs. Reactive Tool Use

Section 5.1 (Figure 7 and Figure 8) characterizes the behavioral signature of successful versus unsuccessful training recipes, based on the main experiment in Section 4.1.

Behavioral classification (Figure 7):

  • GRPO-TCR-Qwen3-4B (highest performer, ~70% AIME2025): Average number of tool calls per trajectory is low (approximately 2–4 calls), while average response length per interaction round is high (approximately 2000–2500 tokens per round). This is classified as Deliberative Mode.
  • GRPO-T models (weakest performers): Average tool calls are substantially higher (approximately 6–8 calls in the early training stages), while response length per round is low (approximately 500–1000 tokens). This is classified as Reactive Mode (short-think + frequent tool calls).
  • GRPO-TCR-Qwen2.5-7B and GRPO-SCR-Qwen2.5-7B: These fall in the reactive regime initially but show a trend toward the deliberative regime as training progresses (tool calls decrease, response length increases), though they never fully reach the deliberative plateau of the Qwen3-4B models.
  • GRPO-T-Qwen3-4B: Despite having a stronger base model, this recipe falls into the reactive regime, indicating that the algorithmic choices (not just model capacity) determine the reasoning mode that emerges.

Tool-use efficiency (Figure 8):

  • The paper plots response length per round (x-axis) against tool-use success rate (y-axis) for each model. The deliberative models (GRPO-TCR-Qwen3-4B, GRPO-SCR-Qwen3-4B) achieve over 70% tool-use success rate, with response lengths in the 2000–2500 token range.
  • The reactive models (GRPO-T variants, plus GRPO-TCR-Qwen2.5-7B) show tool-use efficiency below 60%, with response lengths below 1500 tokens.
  • The correlation is clear: models that invest more tokens in internal reasoning before tool calls achieve substantially higher success rates on those calls.

The paper's interpretation is causal: "Careful reasoning before acting enables highly accurate and effective calls" (Section 5.1). The mechanism is that longer pre-call reasoning enables better problem decomposition into tool-executable subtasks, more precise specification of what to compute, and better anticipation of expected outputs for verification.

Long-CoT Integration Results (Sections 5.2–5.3)

Direct Long-CoT RL (Section 5.2, Figure 9): When Qwen3-4B-Thinking-2507 is directly used as the starting point for GRPO-TCR agentic RL without SFT initialization:

  • The model achieves strong initial average@32 on AIME2025 (Figure 9, left, the curve starts high — approximately 50–55% based on the plot, though the exact number is not quoted).
  • However, the average number of tool calls converges to zero during training (Figure 9, right). The model starts with some tool usage (approximately 2–3 calls initially) but progressively abandons tools as RL reinforces the success of its internal reasoning.
  • The paper diagnoses this as task-dependent: for reasoning-intensive tasks, "the Long-CoT models tend to utilize their internal reasoning capability to solve these tasks, thus focusing exclusively on the problem rather than analyzing the user instruction or considering calling available tools" (Section 5.2).

SFT-initialized Long-CoT vs. Instruction-based (Section 5.3, Figure 10): When Qwen3-4B-Thinking-2507 is first fine-tuned on the real agentic SFT dataset (the same 3k trajectories used for RA-SFT models), and then subjected to GRPO-TCR:

  • The Long-CoT model actively utilizes tools and achieves strong initial performance, notably better than the non-initialized version (Figure 10, left).
  • However, it "ultimately achieves only comparable performance to instruction-based models rather than surpassing them" (Section 5.3). The final average@32 on AIME2025 converges to similar levels for both the Long-CoT-derived and instruction-based models, around 65–70% based on the training curves.
  • Response length dynamics reveal the conflict (Figure 10, right): The instruction-based model shows a monotonically increasing response length as it develops agentic reasoning capabilities. The Long-CoT model shows fluctuating response length: it initially decreases (as the model learns to use tools instead of long internal reasoning), then oscillates, reflecting the tension between scaling internal reasoning and pruning it to accommodate tool use.
  • The paper's conclusion: the Long-CoT model faces "conflicting objectives" where "gains in agentic reasoning are offset by the need to suppress over-thinking behaviors," which "fragments learning efficiency" (Section 5.3). The instruction-based model, starting without strong internal reasoning priors, can "focus exclusively on developing agentic reasoning capabilities from scratch" (Section 5.3, Takeaway 5.3).

DemyAgent-4B Final Benchmark Results

The paper's strongest model, DemyAgent-4B, is trained from Qwen3-4B-RA-SFT using GRPO-TCR with $\epsilon_{\text{high}} = 0.315$ on the complete 30k RL dataset for 3 epochs (Section 6). Evaluation results (Table 2):

BenchmarkDemyAgent-4BBest Larger BaselineLargest Baseline Shown
AIME202472.6%ReTool-32B: 72.5%rStar2-Agent-14B: 80.6%
AIME202570.0%rStar2-Agent-14B: 69.8%ReTool-32B: 54.3%
GPQA-Diamond58.5%rStar2-Agent-14B: 60.9%ARPO-7B: 53.0%
LiveCodeBench-v626.8%(no 32B agentic baseline reported)ARPO-7B: 18.3%

Key comparisons:

  • On AIME2025, DemyAgent-4B (70.0%) outperforms ReTool-32B (54.3%) by +15.7 pp and rStar2-Agent-14B (69.8%) by a narrow margin. It also outperforms Long-CoT self-contained models like DeepSeek-R1-Zero (53.5%).
  • On AIME2024, DemyAgent-4B (72.6%) essentially ties ReTool-32B (72.5%) but lags behind rStar2-Agent-14B (80.6%) by 8.0 pp.
  • On GPQA-Diamond, DemyAgent-4B (58.5%) trails rStar2-Agent-14B (60.9%) by 2.4 pp but outperforms ARPO-7B (53.0%) by 5.5 pp.
  • On LiveCodeBench-v6, DemyAgent-4B (26.8%) substantially outperforms ARPO-7B (18.3%) by 8.5 pp, though agentic baselines on this benchmark are sparse.

Self-contained reasoning comparison: The paper also evaluates DemyAgent-4B's self-contained reasoning (no tools) via the Qwen3-4B-Instruct-2507 base model, reporting 63.3% on AIME2024, 47.4% on AIME2025, 52.0% on GPQA-Diamond, and 35.1% on LiveCodeBench-v6 (Table 2, starred as self-evaluated). This establishes that the agentic training improves AIME2024 performance from 63.3% (self-contained) to 72.6% (agentic, +9.3 pp) and AIME2025 from 47.4% to 70.0% (+22.6 pp), demonstrating strong gains from tool integration.

Ablation Studies and Robustness Checks

  • Synthetic vs. real SFT data with two model sizes (Section 3.1, Table 1): The real-trajectory advantage holds for both Qwen2.5-7B (+11.14 pp on AIME2024 average@32) and Qwen3-4B (+28.85 pp), but the magnitude varies dramatically — the improvement is over 2.5× larger for the stronger model, suggesting that real trajectories are especially valuable when the base model has sufficient capacity to absorb the complex behavioral patterns they encode. The maj@32 diagnostic (0.01% vs. 51.64% for Qwen3-4B) is particularly revealing: synthetic data produces unstable, non-reproducible correct answers while real data produces consistent reasoning.

  • Math-only vs. diverse RL dataset (Section 3.2, Figure 2): The cross-domain diversity ablation (math vs. math+code+science) demonstrates that diversity accelerates convergence by 47% (150 vs. 220 steps to 50% average@32 on AIME2025) and sustains higher policy entropy throughout training. The ablation does not isolate which additional domain (code or science) contributes more, nor does it test alternative diversity mixes (e.g., math+code only, or different domain proportions). The DAPO-Math-17k baseline and the diverse 30k dataset differ in both domain coverage and total size (17k vs. 30k), so quantity is partially confounded with diversity. The paper does not run a size-matched math-only baseline to disentangle these effects.

  • Model-aware filtering vs. full dataset (Section 3.3, Figure 3): For the weak model (Qwen2.5-7B), difficulty filtering transforms training from stagnation (average reward near zero) to productive improvement. The paper demonstrates this qualitatively through reward curves and performance trajectories, but does not report a scalar "average@32 after X steps with filtering vs. without." The filtering criterion (discard 0% and 100% solve-rate problems) has two components — floor removal and ceiling removal — and the paper does not ablate these separately to determine which is more important for the observed improvement.

  • GRPO-TCR vs. GRPO-T: asymmetric clipping + overlong shaping together (Section 4.1, Figure 4): The combined recipe yields major improvements (+16.23 pp on AIME2024, +27.20 pp on AIME2025 for Qwen3-4B). However, these two components (clip higher and overlong shaping) are varied together, not independently. The paper does not run a "clip higher only" or "overlong shaping only" ablation, which would be necessary to attribute the improvement to one component versus the other. This is a significant gap: the paper's conclusion that both are "simple yet effective techniques" is correct as a joint statement, but we cannot determine from the reported experiments whether one dominates or whether they interact.

  • GRPO-TCR vs. GRPO-SCR: token-level vs. sequence-level loss (Section 4.1, Figure 4): For Qwen3-4B, token-level loss is clearly superior (+3.95 pp on AIME2024, +3.86 pp on AIME2025). For Qwen2.5-7B, the two are comparable. However, the GRPO-SCR configuration uses extremely tight clipping bounds ($\epsilon_{\text{high}} = 0.0004$, $\epsilon_{\text{low}} = 0.0003$) compared to GRPO-TCR ($\epsilon_{\text{high}} = 0.28$, $\epsilon_{\text{low}} = 0.20$). This is a confound: the GRPO-SCR vs. GRPO-TCR comparison is not purely loss granularity — it also involves dramatically different clipping regimes. The paper does not explain why such different clipping values were used for the two recipes, nor does it test GRPO-SCR with the same 0.28/0.20 clip bounds.

  • Clip upper bound sweep at three values (Section 4.3, Figure 6): The variation of $\epsilon_{\text{high}} \in \{0.28, 0.315, 0.35\}$ reveals a non-monotonic relationship with performance for the stronger model, providing causal evidence for an optimal entropy range. The sweep is relatively coarse (three values) and the optimal value (0.315 for Qwen2.5-7B, 0.28–0.315 for Qwen3-4B) is identified based on training curves from single runs. The paper does not test intermediate values (e.g., 0.30), nor does it independently vary $\epsilon_{\text{low}}$, nor does it test whether the optimal $\epsilon_{\text{high}}$ changes over the course of training (a schedule rather than a fixed value). The Qwen2.5-7B sweep is missing the 0.35 condition, so we cannot determine whether the weaker model would also show degradation at very high clip bounds.

  • Long-CoT with vs. without SFT initialization (Sections 5.2–5.3, Figures 9–10): The SFT initialization is essential — without it, the Long-CoT model abandons tools entirely; with it, the model matches instruction-based performance. The ablation validates the necessity claim in Takeaway 5.3.1 but does not explore alternative initialization strategies (e.g., mixing self-contained reasoning data with agentic data in different proportions, or curriculum learning from self-contained to agentic). The paper also does not test whether the SFT initialization that works for the instruction-based model (3k real trajectories) is sufficient for the Long-CoT model — the Long-CoT model might need different SFT data characteristics (e.g., more explicit tool-encouragement signals) given its internal reasoning priors.

  • Self-contained vs. agentic evaluation (implied by the two evaluation modes in Table 2): The paper's own models are evaluated in both paradigms, providing a direct within-model measure of how much tool access improves performance. For Qwen3-4B-Instruct-2507, the agentic setup yields 17.9% on AIME2024 (vs. 63.3% self-contained — actually worse, suggesting the model hasn't learned effective tool use yet at the SFT stage) compared to DemyAgent-4B's 72.6% (vs. 63.3% self-contained, a clear gain). This comparison is somewhat confounded by the fact that DemyAgent-4B's self-contained score is reported for Qwen3-4B-Instruct-2507 (pre-agentic-SFT), not for the final RL-trained model evaluated without tools — we don't know how much of the 63.3% → 72.6% jump is due to tool access vs. general reasoning improvement from RL.

Critical Assessment

Claim 1: Real end-to-end trajectories substantially outperform synthetic stitch data for SFT initialization.

What the experiments demonstrate: Table 1 shows that for both Qwen2.5-7B and Qwen3-4B, real trajectories yield higher average@32, pass@32, and maj@32 than synthetic trajectories from ReTool. The gap is large and consistent across AIME2024 and AIME2025. This is a clear, well-controlled comparison: same model, same training hyperparameters, different SFT data sources, evaluated on the same benchmarks.

What the experiments do NOT demonstrate: The paper compares exactly one synthetic dataset (ReTool) against exactly one real dataset (its own curated 3k set). These datasets differ in multiple ways beyond "real vs. stitched": they likely differ in problem sources, difficulty distributions, teacher model used for generation, trajectory length, tool-call frequency, and filtering criteria. The ReTool dataset is not described in detail (the paper cites Feng et al., 2025 without specifying which exact split or configuration was used), so it's unclear whether the gap is attributable to the real-vs-stitched distinction or to other dataset quality factors. A stronger experiment would have used the same problems and generated both stitched and real trajectories from the same teacher model, isolating the trajectory construction method as the only variable.

Genuine weaknesses: The maj@32 of 0.01% for synthetic data on AIME2024 with Qwen3-4B is so extreme that it raises questions about whether the synthetic dataset was used correctly — does ReTool normally produce models this unstable? The paper doesn't discuss whether the synthetic baseline was trained to convergence or whether alternative synthetic data construction methods might close the gap. Additionally, the SFT comparison uses only average@32/pass@32/maj@32 (generation-time metrics), not pass@1 or standard accuracy under greedy decoding, which limits comparability with typical SFT evaluations in the literature.

The claim holds, but the causal attribution ("it's the realness that matters") is stronger than the experimental isolation justifies. The paper provides strong evidence that this particular real dataset outperforms this particular synthetic dataset, but the claim that real trajectories are categorically superior would require comparing multiple real and synthetic datasets or demonstrating that the specific missing behavioral elements (pre-call analysis, error recovery, self-reflection) are the causal mechanisms.

Claim 2: Clip higher and overlong reward shaping are critical algorithmic ingredients, and token-level loss provides additional benefit for stronger models.

What the experiments demonstrate: GRPO-TCR (clip higher + overlong shaping + token-level loss) substantially outperforms GRPO-T (symmetric clip 0.20 + no overlong shaping + token-level loss) on Qwen3-4B: +16.23 pp on AIME2024, +27.20 pp on AIME2025 (Figure 4). The entropy dynamics (Figure 5) show GRPO-T collapsing early while GRPO-TCR sustains higher entropy, providing a mechanistic explanation. The clip upper bound sweep (Figure 6) shows that $\epsilon_{\text{high}}$ causally affects convergence speed and final performance, with an optimal range that is model-dependent.

What the experiments do NOT demonstrate: The clip higher and overlong shaping components are varied together in the GRPO-TCR vs. GRPO-T comparison. We cannot determine whether asymmetric clipping alone would achieve most of the gain, or whether overlong shaping alone would, or whether their interaction is essential. Similarly, the GRPO-TCR vs. GRPO-SCR comparison (token-level vs. sequence-level) is confounded by dramatically different clipping bounds (0.28/0.20 vs. 0.0004/0.0003), so the conclusion that token-level loss is superior for stronger models might actually reflect the effect of looser clipping rather than loss granularity.

Genuine weaknesses:

  • No independent ablation of clip higher and overlong shaping. This is the most significant missing ablation in the algorithm section. A 2×2 design (symmetric vs. asymmetric clip × with vs. without overlong shaping) would cleanly separate these effects and is standard practice for factorial experiments. The paper's conclusion that both are "simple yet effective techniques" is justified by the joint result but doesn't tell a practitioner whether they can omit one if implementing the other is difficult.
  • GRPO-SCR uses extremely tight clipping bounds (0.0004/0.0003) that seem inconsistent with the "clip higher" philosophy. The paper does not explain how these numbers were chosen or why they differ by three orders of magnitude from GRPO-TCR's bounds. If GRPO-SCR with 0.28/0.20 clipping had been tested and still underperformed, the token-vs-sequence conclusion would be much stronger.
  • The $\epsilon_{\text{high}}$ sweep at three values is too coarse to confidently identify an optimum, and the absence of the 0.35 condition for Qwen2.5-7B prevents us from seeing whether the weaker model's optimal range extends higher or also exhibits degradation.
  • No KL coefficient ablation. The paper fixes $\beta = 0.001$ for GRPO-T and doesn't specify $\beta$ for the other recipes (presumably the same, but this is not stated). Since KL regularization and clipping interact in PPO-style objectives (stronger KL can compensate for looser clipping and vice versa), the optimal $\epsilon_{\text{high}}$ almost certainly depends on $\beta$. The paper's entropy analysis (which connects $\epsilon_{\text{high}}$ to exploration) is incomplete without understanding whether similar entropy effects could be achieved by reducing $\beta$ rather than increasing the clip bound.

The claims are directionally supported but the precise attribution of credit among the algorithmic components is unresolved. The paper convincingly shows that "the standard GRPO recipe doesn't work well for agentic RL, and these modifications help," but cannot decompose which modification matters most or whether all are necessary.

Claim 3: A deliberative strategy with fewer but more accurate tool calls consistently beats frequent reactive tool invocation.

What the experiments demonstrate: The behavioral analysis in Section 5.1 (Figures 7–8) shows a clear correlation: the highest-performing models (GRPO-TCR-Qwen3-4B, GRPO-SCR-Qwen3-4B) exhibit low tool-call frequency (~2–4 calls), high response length per round (~2000–2500 tokens), and high tool-use success rates (>70%). Lower-performing models exhibit the opposite pattern (high call frequency, low response length, low success rates). The deliberative mode is not imposed — it emerges from effective RL training — and the quality-over-quantity principle is consistent across model sizes and recipes.

What the experiments do NOT demonstrate: This is a correlational observation, not a causal experiment. The paper shows that better models have a deliberative signature, but does not demonstrate that making a model more deliberative causes it to perform better. A causal test would intervene on the deliberation level (e.g., by modifying the reward function to explicitly penalize tool calls, or by constraining the maximum number of calls, or by prompting the model to think longer before acting) and measure whether performance improves. The paper's own tool bonus $0.1n$ actually encourages more tool calls (positive weight on $n$), so the emergence of low-$n$ behavior is evidence that the RL process overrides this incentive — but this also means the reward function is not aligned with the deliberative strategy the paper advocates.

Genuine weaknesses:

  • The "tool-use efficiency" metric (success rate of tool calls) is defined conceptually but its exact computation is ambiguous. What constitutes a "correctly executed tool call"? The paper says "we filter out the correctly executed tool calling queries and calculate the average success rate" (Section 5.1), but the filtering criterion is not specified. If a tool call's output is used in a final answer that happens to be correct, was that call "successful"? What if the tool call is syntactically valid and executes without error but produces a wrong numerical result?
  • The causal direction is unclear. It's possible that stronger models naturally produce more deliberative behavior as a side effect of their better reasoning, rather than deliberativeness causing better performance. The deliberative signature might be a symptom of model quality, not a lever for improving it.
  • No prompt-based deliberation control experiment. The paper could have explicitly prompted models to "think carefully before each tool call" vs. "use tools whenever possible" and measured the effect on performance, which would establish a causal link between the deliberation construct and outcomes.

The claim is best understood as a descriptive insight about what good agentic behavior looks like, not a prescriptive principle for how to achieve it. The paper demonstrates convincingly that the best-performing models exhibit this pattern, which is valuable for diagnostics and evaluation, but stops short of showing that deliberation can be directly optimized for.

Claim 4: The resulting DemyAgent-4B model matches or exceeds much larger agentic models and establishes a strong baseline.

What the experiments demonstrate: Table 2 shows DemyAgent-4B (4B parameters) achieving 72.6% on AIME2024 (essentially tied with ReTool-32B at 72.5%) and 70.0% on AIME2025 (beating ReTool-32B at 54.3% by a wide margin, and slightly ahead of rStar2-Agent-14B at 69.8%). On GPQA-Diamond, it achieves 58.5% (vs. 60.9% for rStar2-Agent-14B). On LiveCodeBench-v6, it achieves 26.8% (vs. 18.3% for ARPO-7B). The model is genuinely competitive with systems using 3.5–8× more parameters.

What the experiments do NOT demonstrate: The comparisons in Table 2 are across different training pipelines, datasets, hardware configurations, and evaluation protocols. ReTool-32B, rStar2-Agent-14B, ARPO-7B, and Tool-Star-3B were trained by different groups with different data, different RL algorithms, and potentially different evaluation prompts or decoding parameters. The paper does not re-evaluate these baselines under its own protocol — it cites numbers from the original papers. This makes the comparison approximate rather than controlled. The paper's own strongest baseline is GRPO-T evaluated under identical conditions, and DemyAgent-4B's improvement over GRPO-T is the cleanest measure of the recipes' effectiveness.

Genuine weaknesses:

  • No comparison with larger models trained using the same recipes. The paper's core claim is that principled agentic RL enables small models to compete with large ones, but it doesn't demonstrate what happens when these recipes are applied to larger models. Would a 32B model trained with GRPO-TCR on the same 30k dataset scale even further, maintaining or widening the gap over the 4B version? The paper acknowledges this limitation explicitly in Section 9: "our experiments are conducted on small-sized models (e.g., 4B/7B)... larger models may demonstrate different sensitivities to reward signals, require different exploration strategies, or exhibit more robust reasoning patterns." Without larger-model runs, we can't distinguish between "these recipes are good" and "these recipes happen to work well for small models."
  • The DemyAgent-4B's self-contained reasoning baseline in Table 2 (labeled with asterisks as self-evaluated) uses Qwen3-4B-Instruct-2507, which is the pre-SFT base model. DemyAgent-4B is the post-SFT, post-RL agentic model. We don't know its self-contained reasoning performance after agentic training — it might have improved in self-contained reasoning as well (a spillover effect), making the agentic vs. self-contained gap potentially smaller than the 63.3% → 72.6% comparison suggests.
  • Only a single training run of DemyAgent-4B is reported, with no variance estimates across seeds. The differences from some baselines are small (tied with ReTool-32B at 72.5%, 0.2 pp ahead of rStar2-Agent-14B at 69.8%), and without error bars, we can't assess whether these margins are reliable.
  • The 30k RL dataset and 3k SFT dataset are paper-specific artifacts. DemyAgent-4B's performance is partially a function of dataset quality and curation effort, which other methods didn't have access to. The paper contributes these datasets (listed as contributions in Section 6), which is valuable, but it also means the model comparison is not purely about algorithmic recipes — dataset quality is a confound.

The claim about matching larger models is supported but should be qualified as "under our specific data and training pipeline." The more robust contribution is that the recipes substantially improve over the GRPO-T baseline under controlled conditions, and that the resulting 4B model is practically useful, not that it definitively beats all 32B models in a head-to-head sense.

Claim 5: Diverse RL datasets sustain higher policy entropy and accelerate training.

What the experiments demonstrate: Figure 2 shows that the diverse 30k dataset (math+code+science) achieves 50% average@32 on AIME2025 in 150 steps vs. 220 for math-only, with higher sustained entropy throughout training. This is a clean within-experiment comparison using identical algorithmic settings and the same SFT checkpoint.

What the experiments do NOT demonstrate: The diverse dataset is both larger (30k vs. 17k) and more diverse (three domains vs. one). The paper does not run a size-matched math-only baseline (e.g., 30k math problems). This means the speed improvement could be partially or entirely due to having more data, not more diverse data. The entropy difference could similarly reflect having more distinct training prompts rather than domain diversity per se. A clean ablation would keep total dataset size constant and vary the number of domains (e.g., 30k math vs. 15k math + 15k code), or keep the number of domains constant and vary dataset size.

The claim is plausible and consistent with the paper's entropy mechanism theory, but the experimental design conflates diversity with quantity. The paper's entropy explanation (diverse problems require different strategies → broader policy distribution → faster learning) is theoretically coherent but not empirically isolated from the simpler "more data helps" explanation.

What experiments would have strengthened the paper

Several experiments are conspicuously absent and would substantially increase confidence in the paper's conclusions:

  1. Independent ablation of asymmetric clipping and overlong reward shaping: A 2×2 factorial design (symmetric vs. asymmetric clip; with vs. without overlong penalty) applied to the same model and dataset. This is the most important missing experiment for the algorithm section, as it would decompose the primary claimed improvement into its components.

  2. Size-matched diversity ablation: Compare a 30k math-only dataset against the 30k diverse dataset to separate diversity effects from quantity effects. Alternatively, downsample the diverse dataset to 17k for a matched-size comparison against DAPO-Math-17k.

  3. GRPO-SCR with matching clip bounds: Test sequence-level loss with the same 0.28/0.20 asymmetric clipping used by GRPO-TCR to isolate the loss aggregation effect without the clip bound confound.

  4. KL coefficient sweep: Vary $\beta$ at a fixed $\epsilon_{\text{high}}$ to understand whether entropy can be controlled through KL regularization rather than clipping, and whether the two mechanisms are substitutes or complements.

  5. Larger model scaling: Apply the GRPO-TCR recipe to a 14B or 32B model to determine whether the identified principles (clip higher, overlong shaping, optimal entropy range) scale to larger architectures, or whether different hyperparameters are needed.

  6. Causal deliberation intervention: Modify the reward function to explicitly penalize tool-call frequency (e.g., $-0.05n$ instead of $+0.1n$) or to explicitly reward response length per round, and measure whether this pushes the policy toward deliberative behavior and whether that improves performance.

  7. Multiple random seeds for DemyAgent-4B and key ablations: At least 3 seeds for the main GRPO-TCR runs to estimate variance and determine whether the reported differences are statistically reliable.

  8. Evaluation of the RL-trained model on self-contained reasoning: Test DemyAgent-4B without tools (using the self-contained prompt template from Appendix B.2) to measure how much of its improvement comes from better general reasoning vs. specifically from tool use. This would clarify whether agentic RL transfers to non-agentic settings or is domain-specific.

  9. Difficulty estimation cost analysis: The model-aware filtering (Section 3.3) requires 8 rollouts per problem × 30k problems = 240k rollouts to estimate difficulty. The paper doesn't account for this cost in any training compute comparison, nor does it demonstrate that cheaper difficulty estimates (e.g., using 2–4 rollouts) would suffice.

  10. Comparison with a strong SFT-only agentic baseline: The paper compares against other RL-based methods (ReTool, ToRL, ARPO) but doesn't include a direct comparison against a maximally strong SFT-only agent using the same 3k SFT dataset but with more epochs, different filtering, or ensemble techniques. This would establish how much RL actually adds over strong SFT for agentic reasoning.

6. Limitations and Trade-offs

Limitation 1: Difficulty Estimation Cost Is Unaccounted For and Impractical for Deployment

The assumption or constraint. The model-aware dataset filtering procedure (Section 3.3) requires estimating each problem's difficulty relative to the current policy by sampling 8 rollouts per problem on the full 30k RL dataset — a total of 240,000 rollouts before a single RL training step. This cost is explicitly excluded from all training efficiency comparisons. The paper acknowledges this indirectly when noting that "our experiments do not account for this cost largely for simplicity" (Section 3.2, discussing a related difficulty estimation procedure), but the 240k-rollout cost of the model-aware filtering is never quantified in terms of GPU-hours or compared to the training budget it is meant to optimize.

The consequence. The headline training efficiency claims are incomplete. The paper reports that model-aware filtering breaks Qwen2.5-7B's performance bottleneck (Figure 3) and that diverse datasets accelerate convergence (150 vs. 220 steps to reach 50% average@32; Section 3.2), but these comparisons amortize away the difficulty estimation preprocessing. For a practitioner deploying this pipeline, the total compute cost is preprocessing rollouts + RL training steps. If the preprocessing dominates — as it likely does when the dataset is large and the RL training budget is small — the practical value of model-aware filtering is substantially lower than the paper's curves suggest. The paper never reports whether 8 rollouts per problem is necessary or whether cheaper estimates (e.g., 2–4 rollouts) would produce comparable filtering quality.

What evidence exists in the paper. The 8-rollout protocol is stated in Section 3.3 ("We use our SFT model to perform 8 rollouts per problem on the 30k RL dataset"). The filtering criteria (discard 0% and 100% accuracy problems) and binning scheme (easy/medium/hard) are described in the same section. However, the paper never quantifies the preprocessing compute in GPU-hours, never includes it in any budget comparison, and never ablates the number of rollouts to determine whether 8 is a hard requirement or a conservative choice. Section 3.2 mentions a related difficulty estimation that uses 2048 samples per question for the predicted difficulty bins, flagging that cost separately, but the Section 3.3 cost is never flagged as a limitation.

Mitigation status. Not addressed. The paper does not propose cheaper difficulty estimation methods (e.g., using the PRM's score on a single sample, or training a lightweight difficulty predictor), does not include preprocessing cost in any comparison, and does not list this as a limitation in Section 9. Section 3.2 acknowledges the exploration-exploitation nature of the difficulty estimation problem but leaves it entirely to future work.

Limitation 2: The Algorithmic Ablation Does Not Isolate Clip Higher from Overlong Reward Shaping

The assumption or constraint. The central algorithmic contribution is that GRPO-TCR (token-level loss, asymmetric clipping with $\epsilon_{\text{high}} = 0.28$ and $\epsilon_{\text{low}} = 0.20$, and overlong reward shaping $r_{\text{length}}$) substantially outperforms GRPO-T (token-level loss, symmetric clipping with $\epsilon = 0.20$, no overlong shaping). The paper attributes this improvement to "clip higher and overlong reward shaping" jointly (Takeaway 4.1: "Clip higher and overlong reward shaping are simple yet effective techniques to improve the performance of Agentic RL"), but these two modifications are varied together in all experiments. There is no 2×2 factorial design (symmetric vs. asymmetric clip; with vs. without overlong shaping) that would decompose their individual contributions.

The consequence. A practitioner implementing these recipes cannot determine whether both modifications are necessary or whether one accounts for the majority of the improvement. If overlong reward shaping alone drives most of the gain (by preventing runaway generation that wastes the response budget and produces noisy tool calls), then the asymmetric clipping and its associated hyperparameter tuning ($\epsilon_{\text{high}}$ sweep in Section 4.3) are of secondary importance. Conversely, if asymmetric clipping alone suffices, then overlong shaping is an unnecessary complication. The paper's strong claim about entropy management through $\epsilon_{\text{high}}$ (Section 4.3) rests partly on the assumption that clipping is the primary mechanism — but the entropy curves (Figure 5) show GRPO-TCR and GRPO-SCR both sustaining higher entropy than GRPO-T, and both recipes include overlong shaping. It is possible that the entropy benefit comes from the length penalty discouraging premature convergence rather than from the asymmetric clip.

What evidence exists in the paper. Section 4.1 (Figure 4) compares GRPO-TCR, GRPO-SCR, and GRPO-T, but the recipes differ along multiple axes simultaneously. GRPO-TCR and GRPO-T differ in both clipping bounds and reward shaping (confounded), while GRPO-TCR and GRPO-SCR differ in loss aggregation granularity AND clipping bounds (also confounded — GRPO-SCR uses $\epsilon_{\text{high}} = 0.0004$ and $\epsilon_{\text{low}} = 0.0003$, three orders of magnitude tighter than GRPO-TCR). The paper does not report a "GRPO-T with overlong shaping only" or "GRPO-T with clip higher only" ablation. Section 4.3 does vary $\epsilon_{\text{high}}$ in isolation (0.28 vs. 0.315 vs. 0.35), showing it affects convergence speed within GRPO-TCR, but this does not separate the clip effect from the overlong shaping baseline — it shows clip matters conditional on overlong shaping already being present, not whether overlong shaping matters.

Mitigation status. Not addressed. The paper never acknowledges this confound as a limitation, nor does it suggest that future work should decompose these effects. Section 9 (Limitations) focuses on model size scaling and hyperparameter sensitivity but does not mention the missing ablation.

Limitation 3: All Results Are on a Single Model Family and a Single Tool Type, with No Evidence of Transfer

The assumption or constraint. All experiments use Qwen2.5-7B-Instruct and Qwen3-4B-Instruct-2507 as base models, with the code interpreter (SandBoxFusion) as the only available tool. The paper's key findings — that real end-to-end trajectories are essential for SFT, that clip higher improves exploration, that a deliberative tool-use strategy emerges — are demonstrated exclusively in this setting. The paper acknowledges in Section 9 that experiments are "conducted on small-sized models (e.g., 4B/7B)" and that "larger models may demonstrate different sensitivities to reward signals, require different exploration strategies, or exhibit more robust reasoning patterns that interact differently with RL training dynamics." However, it does not acknowledge the single-tool limitation, and it briefly gestures in Section 8.3 toward "generalizing our insights from a static and single tool environment to multi-tool and optimizable environment" as future work.

The consequence. The paper's recipes may not transfer to other model families (e.g., LLaMA, DeepSeek, Mistral), other model sizes (particularly larger models where the entropy dynamics may be different), or other tool types (e.g., search engines, calculators, databases). The finding that Long-CoT models resist tool use (Section 5.2) is already model-dependent — it was demonstrated on Qwen3-4B-Thinking-2507 and may not replicate on other Long-CoT architectures. The deliberative-vs-reactive trade-off (Section 5.1) is specifically about code interpreter tool calls, where each call incurs nontrivial latency and the output is deterministic computation. For search engine tools, frequent short queries might be optimal (retrieving small pieces of information incrementally), and for database tools, each call is cheap and fast. The "quality-over-quantity" principle may be specific to expensive, high-latency tools like code execution. Additionally, the paper's entropy mechanism analysis (Section 4.3) is based on the entropy trajectories of 4B and 7B models — larger models typically have different baseline entropy levels, so the optimal $\epsilon_{\text{high}}$ sweeps and the model-dependent optimal range findings may not extrapolate.

What evidence exists in the paper. Table 2 evaluates DemyAgent-4B on four benchmarks (AIME2024/2025, GPQA-Diamond, LiveCodeBench-v6), which span math, science, and code domains, but all within the code interpreter tool paradigm. The paper demonstrates some domain generalization (the diverse dataset experiment in Section 3.2 mixes math, science, and code problems), but this is generalization of problem type to the same tool, not generalization of tool type to different interaction paradigms. Section 5.2 briefly references prior work on search engine tools (Search-R1, R1-Searcher, Search-o1) to contextualize the Long-CoT finding but does not run experiments with search tools. Section 8.3 explicitly lists multi-tool environments as future work.

Mitigation status. Partially addressed via the domain-diverse evaluation (math, science, code benchmarks in Table 2), which shows the trained agent works across problem types, but the tool diversity limitation is not mitigated experimentally. The paper's Section 9 limitation statement focuses on model size scaling and hyperparameter sensitivity but does not mention tool diversity or model family generalization.

Limitation 4: The Deliberative Reasoning Mode Is a Correlational Observation, Not a Causally Validated Principle

The assumption or constraint. Section 5.1 presents the deliberative-vs-reactive distinction as a key insight: better-performing models use fewer but more successful tool calls, investing more internal reasoning tokens before each invocation. The paper states this as a "quality-over-quantity principle" (Takeaway 5.1) and implies that the deliberative mode causes better performance ("Careful reasoning before acting enables highly accurate and effective calls"; Section 5.1). However, the evidence is purely correlational — the paper observes that high-performing models happen to exhibit this behavioral signature, but never intervenes to test whether making a model more deliberative causes performance to improve.

The consequence. A practitioner reading this paper might conclude that they should constrain their agent to make fewer tool calls or to think longer before acting — perhaps by modifying the reward function to penalize tool-call frequency, or by prompt-engineering longer pre-call reasoning. But the paper provides no evidence that such interventions work. It is equally plausible that the deliberative behavior is a symptom of model quality (better reasoning naturally leads to more targeted tool use) rather than a lever for improving it. The paper's own reward function includes a tool bonus $+0.1n$ that encourages more tool calls (Section 2.2), yet the best models learn to use fewer — suggesting the deliberative pattern is an emergent property of successful optimization, not something the reward function explicitly incentivizes. Without a causal intervention (e.g., modifying the reward to penalize tool calls and measuring whether performance improves, or constraining the number of allowed calls and observing the effect), the practical guidance is ambiguous: do I try to enforce deliberation, or do I trust that better RL training will naturally produce it?

What evidence exists in the paper. The behavioral characterization in Figures 7–8 shows the correlation between deliberative metrics (low tool-call count, high response length per round, high tool-use success rate) and final performance across multiple model-recipe combinations. The deliberate models (GRPO-TCR-Qwen3-4B, GRPO-SCR-Qwen3-4B) achieve >70% tool-use success and ~70% AIME2025 accuracy; the reactive models (GRPO-T variants) show <60% tool-use success and 40–55% accuracy. The correlation is consistent but the causal direction is untested. The paper also does not report whether the correlation holds within a single training run (do individual trajectories with a deliberative pattern correlate with correctness?), which would strengthen the micro-level evidence.

Mitigation status. Not addressed. The paper does not run any intervention experiment on reasoning mode (e.g., prompt manipulation, reward function modification, tool-call budget constraints), does not discuss the correlational nature of the finding as a limitation, and does not suggest future work on causal validation of the deliberative principle. The takeaway is presented as a prescriptive insight without the supporting causal evidence.

Limitation 5: No Confidence Intervals or Multi-Seed Replication for Any Result

The assumption or constraint. All reported results — the SFT data comparison (Table 1), the GRPO recipe comparison (Figure 4), the diversity ablation (Figure 2), the clip bound sweep (Figure 6), the reasoning mode analysis (Figures 7–8), and the DemyAgent-4B final evaluation (Table 2) — are based on single training runs with no reported confidence intervals, standard deviations, or multi-seed replication. The paper uses a fixed evaluation protocol (32 samples per problem, temperature 1.0, top-p 0.6; Appendix A.2) but reports only point estimates.

The consequence. Several of the paper's headline comparisons involve small margins. DemyAgent-4B achieves 72.6% on AIME2024 versus ReTool-32B's 72.5% — a 0.1 percentage point difference that is almost certainly not statistically significant given typical RL training variance. On AIME2025, the 70.0% vs. 69.8% margin over rStar2-Agent-14B is similarly negligible. On GPQA-Diamond, the 58.5% vs. 60.9% gap behind rStar2-Agent-14B (2.4 pp) might vanish with a different random seed. Within the paper's own experiments, the claim that $\epsilon_{\text{high}} = 0.315$ is optimal for Qwen2.5-7B (Section 4.3) is based on a single training run at each of three clip values — the 40% convergence speed improvement is reported as a precise figure but could reflect training noise rather than a true hyperparameter effect. The claim that GRPO-TCR outperforms GRPO-SCR by 3.95% on AIME2024 and 3.86% on AIME2025 (Section 4.1) for Qwen3-4B is a consistent directional finding across two benchmarks, but without variance estimates, we cannot assess whether this gap is reliable or could reverse with a different seed. The entropy comparisons (Figures 2, 5) show clear qualitative differences between recipes, but the exact entropy values and their relationship to performance metrics are themselves subject to training noise — an anomalous high-entropy early phase could be misinterpreted as a recipe effect rather than a seed effect.

What evidence exists in the paper. The paper's evaluation methodology section (Appendix A) specifies the training and evaluation setup but does not mention multiple seeds. Every figure and table reports single values with no error bars or confidence bounds. The GRPO training process involves stochastic rollouts and batch-level advantage normalization (Equation 3), both of which introduce variance that can cause different training trajectories from different seeds — even with identical hyperparameters, the specific sequences of rollouts and the resulting advantage signals will differ, and GRPO's sensitivity to batch composition is well-documented in the broader RL literature. The paper provides no evidence that its findings are robust to this stochasticity.

Mitigation status. Not addressed. The paper does not discuss seed dependence, does not report multiple training runs for any experiment, and does not include variance as a limitation in Section 9. The implicit assumption is that single-run results are representative, which is standard practice in many LLM RL papers but represents a meaningful limitation for results where margins are small or where hyperparameter sensitivity is high (which the paper itself flags for larger models in Section 9).

Limitation 6: No Evidence That the Recipes Scale to Larger Models, Despite Central Claims About Small-Model Competitiveness

The assumption or constraint. A central narrative of the paper is that principled agentic RL recipes enable small models to compete with much larger ones — DemyAgent-4B matches or exceeds 14B–32B parameter models (Table 2). However, the paper's recipes are developed and validated exclusively on 4B and 7B models. It never tests what happens when the same recipes (real-trajectory SFT, GRPO-TCR with $\epsilon_{\text{high}} = 0.28$–0.315, diverse 30k dataset, model-aware filtering) are applied to a 14B or 32B model. The paper acknowledges this in Section 9: "larger models may demonstrate different sensitivities to reward signals, require different exploration strategies, or exhibit more robust reasoning patterns that interact differently with RL training dynamics."

The consequence. The paper's small-model-competitiveness claim is an extrapolation, not a demonstrated fact. It is possible that the identified recipes are specifically effective for small models (which benefit most from the exploration-friendliness of clip higher, the entropy maintenance from diverse data, and the careful SFT initialization) but provide diminishing returns at larger scales. Larger models typically have higher baseline entropy and more robust reasoning patterns, which might reduce the need for aggressive clipping or overlong shaping. Conversely, larger models might benefit more from these recipes, widening the gap over comparable large models — but without experiments, we cannot distinguish between these scenarios. The comparison with ReTool-32B, rStar2-Agent-14B, and ARPO-7B in Table 2 is across different training pipelines, datasets, and evaluation protocols — these models were trained by different groups with different resources. The paper does not provide a like-for-like comparison (same dataset, same recipe, different model sizes) that would isolate the effect of model scale from the effect of the recipe. This makes the "4B beats 32B" claim difficult to attribute: is it because the recipe is good, because the base model (Qwen3-4B-Instruct-2507) is unusually capable for its size, or because the larger baselines used suboptimal training pipelines?

What evidence exists in the paper. Table 2 reports DemyAgent-4B and comparison baselines on four benchmarks. The paper's own internal comparisons (Figures 4, 6, 7) are all between the 4B and 7B models trained with the same recipes, showing consistent but model-size-dependent effects (e.g., token-level loss matters more for the stronger 4B model; clip upper bound optima differ by model size). This within-paper model-size dependence is evidence that the recipes do not transfer uniformly between even the 4B and 7B models, which makes extrapolation to 14B–32B even more uncertain. The DemyAgent-4B's base model (Qwen3-4B-Instruct-2507) already achieves 63.3% on self-contained AIME2024 (Table 2) — this is a remarkably strong 4B model, and the agentic RL training improves it to 72.6% (+9.3 pp). A significant portion of DemyAgent-4B's competitiveness comes from the base model's unusual strength, not exclusively from the agentic RL recipes.

Mitigation status. Explicitly flagged as a limitation in Section 9: "We leave a more comprehensive study of RL with larger-sized models in broader agentic settings as an important future work direction." This is a candid acknowledgment, but it means the central scaling claim — that small models with agentic RL can replace large models — remains an untested hypothesis rather than a validated conclusion. The paper contributes the recipes and baselines that would enable such a scaling study, but does not perform it.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not introduce a fundamentally new algorithm, but it reorients the agentic RL field from scattered heuristic development toward principled systematic investigation. Prior to this work, progress in agentic RL proceeded through independent papers each proposing slightly different algorithmic variants — ReTool (Feng et al., 2025), ToRL (Li et al., 2025d), ARPO (Dong et al., 2025b), Tool-Star (Dong et al., 2025a) — each evaluated on different benchmarks, with different base models, making it impossible to determine which of their many design choices actually caused improvements. The field was accumulating recipes without understanding. This paper provides the first controlled factorial investigation that isolates data, algorithm, and reasoning mode as independent axes and systematically varies one factor at a time while holding others fixed, producing a minimum viable recipe (real end-to-end SFT trajectories + GRPO with clip higher and overlong reward shaping + diverse, model-aware RL data) that practitioners can adopt as a baseline and researchers can build upon.

The conceptual shift is from treating agentic RL as a singular problem that just needs the right algorithm to treating it as a multi-axis optimization where the axes interact. The finding that the optimal clip upper bound $\epsilon_{\text{high}}$ depends on model capacity — weaker models need looser bounds to escape performance bottlenecks, stronger models need tighter bounds to prevent over-exploration (Section 4.3, Takeaway 4.3.2) — demonstrates that data, algorithm, and model are not independent knobs. The finding that Long-CoT reasoning priors actively interfere with tool-use learning (Section 5.2–5.3) demonstrates that better reasoning capability does not automatically translate to better agentic behavior. These interaction effects mean that future work cannot safely optimize one axis in isolation and expect the results to generalize — the framework forces researchers to specify which model, which data, and which reasoning paradigm their contributions are validated against.

The paper also resolves a contradiction in the entropy management literature. Prior work debated whether entropy should be minimized (Agarwal et al., 2025; Cheng et al., 2025b) or maximized (Cui et al., 2025; Wang et al., 2025b) for effective RL. The paper's clip bound sweep (Section 4.3) shows that both perspectives are partially correct — the optimal entropy level is model-dependent and non-monotonic, with a sweet spot that is too high for entropy-minimization advocates and too low for entropy-maximization advocates. This reframes the debate from "should entropy be high or low?" to "what entropy level is appropriate for a given model and task distribution?" — a shift from ideological positions to empirical calibration.

The paper's most consequential reframing may be its reconceptualization of training data usefulness as a function of current policy capability rather than an intrinsic property of the problem (Section 3.3). Prior work treated datasets as fixed resources — you pick a dataset, you run RL on it, you measure improvement. The finding that Qwen2.5-7B-RA-SFT stagnates on the same 30k dataset where Qwen3-4B-RA-SFT thrives, and that this bottleneck can be broken by discarding problems the model cannot solve (0% accuracy) or trivially solves (100% accuracy), demonstrates that data quality is policy-relative, not absolute. This makes difficulty estimation a first-class design problem rather than a preprocessing afterthought, and it suggests dynamic curriculum learning — periodically re-estimating difficulty as the policy improves and re-filtering the dataset — as a natural next step that the paper gestures toward but does not implement.

Research directions that become more attractive include: cheap difficulty estimation (because the 8-rollout-per-problem protocol is too expensive for deployment), joint optimization of pretraining and agentic RL recipes (because the Long-CoT interference effect in Section 5.2 raises the question of whether base models should be designed differently for agentic use), and multi-tool generalization (because the deliberative strategy discovered for code interpreters may or may not transfer to search engines or databases). Research directions that become less attractive include: purely algorithmic innovations without accompanying data or reasoning-mode analysis (because the paper shows that data quality and reasoning mode account for much of the observed variance), and stitch-style synthetic data pipelines for agentic SFT (because Section 3.1 shows they produce fundamentally unstable models, with maj@32 of 0.01% on AIME2024 for Qwen3-4B compared to 51.64% for real trajectories — the gap is so large that stitching appears to be not merely suboptimal but categorically the wrong abstraction).

Follow-Up Research This Work Enables

Cheap, online difficulty estimation to replace the 240k-rollout preprocessing bottleneck. The paper's model-aware filtering (Section 3.3) requires 8 rollouts per problem on the 30k dataset before a single RL step — 240,000 rollouts total. The paper acknowledges this cost is not included in any training budget comparison and does not propose cheaper alternatives. A direct follow-up would train a lightweight difficulty classifier on the question text alone, using the 8-rollout accuracy labels as supervision, and test whether its predicted bin assignments produce comparable RL training dynamics to the full rollout-based estimates. A stronger version would be online difficulty estimation: after each batch of RL training, use the batch's own rollout results (which are generated anyway for policy updates) to update running accuracy estimates per problem, and dynamically drop or re-weight problems that enter the 0% or 100% accuracy zones. This would eliminate the separate preprocessing step entirely and amortize difficulty estimation into the training process. A negative result — e.g., showing that online estimates are too noisy to enable effective filtering — would establish that the preprocessing cost is genuinely necessary, which would be a practically important finding.

Decomposing the clip higher effect from overlong reward shaping through a 2×2 factorial design. The paper's central algorithmic claim is that GRPO-TCR (asymmetric clipping + overlong shaping + token-level loss) substantially outperforms GRPO-T (symmetric clipping + no overlong shaping + token-level loss), but these two modifications are always varied together (Section 4.1). A clean follow-up would train four GRPO variants on Qwen3-4B-RA-SFT: (1) symmetric clip 0.20, no overlong penalty; (2) symmetric clip 0.20, with overlong penalty; (3) asymmetric clip 0.28/0.20, no overlong penalty; (4) asymmetric clip 0.28/0.20, with overlong penalty — all with token-level loss, all on the same 30k dataset. This 2×2 design would reveal whether most of the GRPO-TCR advantage comes from one component (e.g., overlong shaping alone might prevent the entropy collapse that GRPO-T exhibits) or whether the interaction is essential (e.g., overlong shaping prevents runaway length, while clip higher enables rapid adoption of the more efficient strategies that the length pressure creates). The practical payoff is knowing whether a practitioner can skip one component if implementing it is difficult in their RL framework.

Causal validation of the deliberative reasoning mode through reward function interventions. The paper observes that better-performing models exhibit a deliberative strategy — fewer, more successful tool calls with longer pre-call reasoning (Section 5.1) — but provides only correlational evidence. A causal test would modify the reward function to directly influence the deliberative-vs-reactive trade-off and measure the performance impact. Specifically: (1) Penalize tool calls: change the tool bonus from $+0.1n$ to $-0.05n$ or $-0.1n$, explicitly discouraging excessive tool use, and measure whether the resulting policy becomes more deliberative and whether its performance improves or degrades. (2) Reward pre-call reasoning length: add a bonus proportional to the number of tokens between tool calls (with appropriate normalization), incentivizing the model to invest more internal reasoning before invoking tools. (3) Hard constrain tool calls: cap the maximum number of tool invocations per trajectory at 2–3 for one group, allow unlimited for another, and compare. If imposing deliberativeness causally improves performance, the paper's correlational observation becomes an actionable principle; if it degrades performance, then the deliberative pattern is a symptom of good training rather than a direct cause, and practitioners should focus on improving RL quality rather than constraining tool-use behavior.

Scaling the GRPO-TCR recipe to 14B–32B models to test the generality of the entropy-optimal-range finding. The paper develops its recipes exclusively on 4B and 7B models and explicitly flags the lack of larger-model experiments as a limitation (Section 9). A direct scaling study would apply GRPO-TCR to Qwen2.5-14B-Instruct or Qwen3-14B-Instruct with the same SFT data pipeline (3k real trajectories), the same 30k RL dataset, and a clip upper bound sweep (0.20, 0.28, 0.315, 0.35, 0.40) to determine whether larger models exhibit the same non-monotonic relationship between $\epsilon_{\text{high}}$ and performance, and whether the optimal bound shifts upward (suggesting exploration remains important at scale) or downward (suggesting larger models already have sufficient entropy and need more conservative clipping). A critical sub-experiment: measure whether the deliberative reasoning pattern (Figure 7) naturally emerges for larger models trained with these recipes, or whether larger models develop qualitatively different tool-use strategies (e.g., even fewer but more complex tool calls, or more sophisticated multi-turn tool orchestration). A negative result — finding that the recipes that work for 4B/7B degrade performance at 14B+ — would be highly informative, establishing that agentic RL recipes are scale-dependent and that the field's habit of developing methods on small models and extrapolating to large ones is unreliable. It would also contextualize the paper's "4B beats 32B" narrative by establishing whether the recipes close or widen the gap at larger scales.

Multi-tool generalization of the deliberative principle with tool-dependent strategy emergence. The paper studies a single tool (code interpreter) and discovers that a deliberative strategy — invest reasoning before calling, make fewer but more targeted calls — yields the highest tool-use efficiency and final accuracy (Section 5.1). A natural extension would construct an RL environment with multiple heterogeneous tools — e.g., a code interpreter (high latency, deterministic outputs, suitable for complex computation), a calculator (low latency, deterministic, suitable for simple arithmetic), a search engine (moderate latency, non-deterministic, suitable for factual retrieval), and a knowledge base query tool (low latency, deterministic, suitable for structured lookups) — and apply the same GRPO-TCR recipe to observe what strategies emerge. The key question: does the policy learn tool-specific deliberation — reasoning heavily before code interpreter calls but issuing rapid calculator queries, or using search engine results as triggers for further reasoning rather than final answers — or does it adopt a uniform deliberation level across all tools? If tool-specific strategies emerge, it would demonstrate that agentic RL discovers not just that tools are useful but which tools are appropriate for which subtasks, a qualitative advance over single-tool agents. The paper's model-aware filtering (Section 3.3) would extend naturally: in a multi-tool setting, difficulty is not just problem-specific but tool-specific (some problems become easy when the right tool is selected), and the difficulty estimation could inform which tool subsets to make available during different training phases.

Reconciling Long-CoT capabilities with agentic RL through anti-interference training objectives. The paper's finding that Long-CoT models actively resist tool use during agentic RL (Section 5.2–5.3) is a critical negative result that opens a clear research direction: how do we combine strong internal reasoning with effective tool use when the two capabilities appear to be in optimization conflict? A concrete follow-up would design a multi-objective RL training procedure where the reward function explicitly balances internal reasoning quality and tool-use effectiveness — for example, adding a term that rewards the model when it correctly identifies that a problem is solvable without tools (preventing unnecessary tool calls on easy problems) and another term that rewards appropriate tool delegation on problems that benefit from it. Alternatively, a staged training curriculum could first train the model on self-contained reasoning tasks to develop internal capability, then freeze those parameters and train tool-use adapter layers (LoRA or similar) on agentic tasks, preventing the tool-use optimization from interfering with the reasoning parameters. A third approach would be data mixing during RL: interleave self-contained reasoning problems and agentic problems in the same training batches, so the policy must maintain both capabilities simultaneously rather than specializing. The paper's response length analysis (Figure 10, right) — showing that Long-CoT models exhibit fluctuating response lengths as they oscillate between thinking and tool use — provides a clear diagnostic: a successful intervention should produce monotonically improving response lengths (like the instruction-based model) rather than the fluctuating pattern.

Practical Applications and Downstream Use Cases

Cost-efficient batch inference for mathematical and scientific problem-solving. The paper's central result — DemyAgent-4B achieving 70.0% on AIME2025, surpassing 32B agentic models like ReTool-32B (54.3%) and matching rStar2-Agent-14B (69.8%) — has direct economic implications for organizations running large-scale mathematical or scientific reasoning workloads. Serving a 4B model with a code interpreter tool is substantially cheaper (in GPU memory, inference latency, and hardware requirements) than serving a 32B model, even accounting for the additional tokens generated through the deliberative reasoning process. For a deployment processing tens of thousands of competition math problems (e.g., an automated tutoring system that generates step-by-step solutions, or a research pipeline that verifies mathematical conjectures through computational exploration), the cost savings from using DemyAgent-4B rather than a 32B model — approximately 8× fewer parameters — could be decisive. The paper's release of the 3k SFT dataset, the 30k RL dataset, and the DemyAgent-4B checkpoint (Section 6) makes this immediately actionable: a team can download these artifacts, deploy DemyAgent-4B with a code interpreter backend, and achieve the reported performance without any additional training. The main caveat is that the deliberative strategy (Section 5.1) produces longer responses per interaction round (~2000–2500 tokens), which partially offsets the parameter-count savings in terms of total FLOPs per query — a full cost analysis would need to compare total inference FLOPs (model size × tokens generated) rather than just model size.

Data generation for self-improving agentic pipelines. The paper demonstrates that GRPO-TCR with real end-to-end SFT initialization and diverse RL data can produce an agent that achieves over 70% accuracy on AIME2025 (Section 4.1, Figure 4) — meaning that approximately 70% of its tool-augmented solutions on difficult competition math problems are correct. This positions DemyAgent-4B, or models trained with the same recipe, as a high-quality solution generator for self-improvement loops. In a STaR-like or ReSTEM^{EM}-like pipeline (Zelikman et al., 2022; Singh et al., 2024), an agent generates solutions with tool access, verifies correctness against ground-truth answers (or through tool-executed verification), and the successful trajectories are added to a training set for further fine-tuning. The paper's finding that model-aware filtering (Section 3.3) discards problems with 0% accuracy because they provide no learning signal is directly applicable: in a self-improvement loop, you would run the current policy on a candidate problem set, keep only problems where accuracy is in the 25–75% range (providing contrastive signal), and use the successful trajectories from those problems as training data for the next iteration. The key efficiency gain over prior self-improvement approaches is that the agent is not just generating answers but generating tool-use strategies — each successful trajectory encodes not just what the answer is but how to decompose the problem, when to invoke the code interpreter, and how to integrate computational results, providing richer supervision for the next training iteration.

On-device or edge deployment of capable reasoning agents where model size is the binding constraint. The paper shows that a 4B model with agentic RL training can match or exceed the self-contained reasoning performance of much larger models on AIME benchmarks — DemyAgent-4B achieves 72.6% on AIME2024 compared to DeepSeek-R1-Zero (671B) at 71.0% (Table 2, self-contained column for R1-Zero, agentic column for DemyAgent-4B). While the comparison is across reasoning paradigms (agentic with tools vs. self-contained without), the practical implication is that a 4B model with a lightweight code interpreter backend can be served in environments where a 671B model cannot — edge devices, mobile phones, offline laptops, privacy-sensitive deployments where data cannot be sent to cloud APIs. The deliberative mode's higher token count (Figure 7: ~2000–2500 tokens per interaction round) increases inference latency, but for many deployment scenarios (asynchronous homework help, automated grading, scientific data analysis), correctness matters more than speed. The main deployment consideration is that the code interpreter backend (SandBoxFusion in the paper's setup; Appendix A.3) must also be available locally, which requires a Python runtime sandbox on the device — feasible for laptops and servers, challenging for phones. The paper's finding that Long-CoT models should not be used as the initialization for agentic RL (Section 5.2–5.3) provides concrete guidance for practitioners building such deployments: start with the instruction-tuned variant of your target model (e.g., Qwen3-4B-Instruct, not Qwen3-4B-Thinking), apply the real-trajectory SFT, then GRPO-TCR with $\epsilon_{\text{high}} = 0.28$–0.315 depending on the model's baseline entropy level.

Diagnostic toolkit for agentic RL training runs. Beyond the specific model and dataset artifacts, the paper contributes a reusable set of diagnostic metrics and training dynamics analyses that any practitioner training agentic RL can adopt: monitor policy entropy throughout training (Figure 5 — entropy collapse is an early warning sign of suboptimal recipe choices), track average reward per problem difficulty bin to detect competence-difficulty mismatches (Figure 3 — stagnant reward near zero indicates the dataset is too hard), measure tool-call frequency and success rate to identify whether the deliberative strategy is emerging (Figures 7–8), and monitor the pass@k-to-average@k gap as a measure of remaining improvement potential (Section 4.2). These diagnostics require no additional infrastructure beyond what the RL training loop already computes — the entropy, reward, and accuracy statistics are side products of the GRPO batch processing — and they provide actionable signals for hyperparameter adjustment (increase $\epsilon_{\text{high}}$ if entropy is collapsing, curate the dataset if reward is stagnant, investigate tool-call patterns if pass@k is improving but average@k is not). The paper's non-monotonic clip bound result (Section 4.3) implies that these diagnostics should be monitored during training rather than only at the end — a recipe that worked at $\epsilon_{\text{high}} = 0.28$ for the first 100 steps might benefit from an increase to 0.315 if entropy starts dropping. This transforms agentic RL from a "set hyperparameters, run, hope" process into an actively managed optimization informed by real-time signal.