PivotRL: High Accuracy Agentic Post-Training at Low Compute Cost

ArXiv: 2603.21383

🎯 Pitch

Training LLM agents usually forces a brutal trade-off: cheap SFT craters out-of-domain performance, while full RL burns compute on thousands of uninformative trajectory steps. PivotRL breaks this trade-off—achieving 4× fewer rollout turns than end-to-end RL and +10% OOD accuracy over SFT—by restricting on-policy rollouts exclusively to 'pivot' turns where the model genuinely wavers, and rewarding any functionally valid action, not just exact string matches.

---

1. Executive Summary

This paper introduces PivotRL, a novel post-training framework for long-horizon agentic tasks that combines the compute efficiency of supervised fine-tuning with the out-of-domain accuracy of end-to-end reinforcement learning by operating on existing SFT trajectories. Trained on four agentic domains—conversational tool use, software engineering, terminal control, and web browsing—using Qwen3-30B-A3B-Thinking-2507 as the base model, PivotRL relies on two key mechanisms: pivot filtering (selecting only informative intermediate turns where sampled actions exhibit mixed success/failure outcomes under the reference policy, rather than training on all turns) and functional-equivalent reward (assigning credit via domain-specific verifiers that accept any locally acceptable action instead of demanding strict string matching with the SFT demonstration). Across the four domains, PivotRL achieves +4.17% higher average in-domain accuracy than same-data SFT and +10.04% higher OOD accuracy in non-agentic tasks including math, science QA, and competitive coding. On SWE-Bench Verified, PivotRL attains competitive accuracy with end-to-end RL while requiring 4× fewer rollout turns (~133K vs. ~542K), establishing that local on-policy optimization from expert-prefix states can substitute for full-trajectory environment interaction only when rollouts are concentrated at informative decision points and evaluated with permissive local credit.

2. Context and Motivation

The Core Problem: Agentic Post-Training Is Caught Between Two Unsatisfactory Extremes

The fundamental challenge this paper addresses is a tension that anyone training LLMs for agentic tasks confronts immediately: how do you teach a model to execute long, multi-step interactions with tools and environments without either (a) destroying its general capabilities through catastrophic forgetting, or (b) spending prohibitive amounts of compute on full-trajectory reinforcement learning?

This tension arises from the nature of agentic tasks themselves. Unlike single-turn question answering or code generation, agentic tasks unfold over many turns of model-environment interaction. A conversational assistant might call APIs across dozens of turns to complete a user request. A coding agent might search a repository, open files, edit code, run tests, and interpret results over 10–30 tool-calling steps. A terminal agent might execute a sequence of bash commands, inspecting outputs and adjusting its plan after each one. A browsing agent might issue search queries, navigate results, and synthesize information across multiple pages. In all these cases, the model must not only produce correct individual actions but also maintain coherent state, recover from errors, and adapt its strategy as environmental feedback arrives.

The post-training pipeline for these capabilities has historically been dominated by two approaches, each with severe drawbacks:

Supervised Fine-Tuning (SFT) is cheap but fragile. Given a dataset of expert trajectories $\mathcal{D}_{\text{sft}} = \{\tau_i\}_{i=1}^N$, where each trajectory is a sequence of states and demonstrated actions $\tau = (s_0, a_0^*, \dots, s_T, a_T^*)$, SFT trains the model to maximize the likelihood of the demonstrated actions at each step:

$\mathcal{L}_{\text{sft}}(\theta) = -\mathbb{E}_{\tau \sim \mathcal{D}_{\text{sft}}, (s_t, a_t^*) \sim \tau} \left[ \log \pi_\theta(a_t^* \mid s_t) \right]$

This is computationally efficient: the data is collected once, offline, and each gradient update only requires the tokens in the demonstration, not any fresh environment interaction. However, SFT suffers from a well-documented failure mode: out-of-domain (OOD) degradation. When fine-tuned on a narrow distribution of agentic tasks, the model's performance on unrelated tasks—mathematical reasoning, scientific question answering, competitive programming—collapses. The paper reports catastrophic drops: after training on terminal-domain data, SFT causes AIME25 accuracy to plummet from 86.04 to 21.56 (Table 3), a -64.48 percentage point regression. This is not a minor drift; it is near-total capability destruction. The phenomenon has been studied extensively (Chu et al., 2025; Luo et al., 2025b; Li et al., 2024) and is attributed to SFT's tendency to overfit the narrow demonstration distribution, effectively overwriting the model's broader knowledge.

End-to-End Reinforcement Learning (E2E RL) preserves capabilities but is computationally prohibitive. In E2E RL, the model generates complete trajectories from scratch—starting from the initial environment state and interacting turn-by-turn until the task completes or fails—and receives a reward signal at the end. These on-policy rollouts are used to compute policy gradients, typically via GRPO (Group Relative Policy Optimization):

$\mathcal{J}_{\text{GRPO}}(\theta) = \mathbb{E}_{s \sim \mathcal{D}_{\pi_{\text{old}}}, \{a_i\}_{i=1}^G \sim \pi_{\text{old}}(\cdot \mid s)} \left[ \frac{1}{G} \sum_{i=1}^G \min\left(w_i(\theta) \hat{A}_i, \text{clip}(w_i(\theta), 1-\epsilon, 1+\epsilon) \hat{A}_i \right) - \beta \ \text{KL}(\pi_\theta(\cdot \mid s) \ \| \ \pi_0(\cdot \mid s)) \right]$

where $\hat{A}_i = \frac{r_i - \frac{1}{G}\sum_{j=1}^G r_j}{\text{std}(\{r_j\}_{j=1}^G) + \epsilon_{\text{std}}}$ is the group-normalized advantage. E2E RL typically yields better in-domain accuracy than SFT while retaining OOD capabilities, because the on-policy data distribution keeps the model's broader knowledge alive (Chen et al., 2025; Shenfeld et al., 2025). The problem is the cost: every parameter update requires generating fresh, complete trajectories from the current policy, each involving many turns of environment interaction. For SWE-Bench, a typical training trajectory spans 12–25 tool-calling turns. At a batch size of 512 (16 prompts × 32 generations), a single training step consumes thousands of environment rollouts. The paper reports that reaching 32.67% on SWE-Bench Verified requires approximately 542K cumulative rollout turns for E2E RL (Section 4.2, Appendix A.2). This makes E2E RL impractical for organizations without massive compute budgets or for rapid experimentation cycles.

Why This Tension Matters

The practical stakes are enormous. Agentic LLMs are now being deployed at production scale for coding assistants, customer service automation, and complex workflow orchestration. NVIDIA's own Nemotron-3-Super model (Team, 2026) uses agentic post-training as a core pipeline stage (Table 5). For such deployments, the choice between SFT and E2E RL is not academic: SFT means faster iteration but degraded general capabilities (a coding agent that forgets basic math is unacceptable in an integrated product), while E2E RL means better quality but slower development cycles and higher infrastructure costs. The paper frames this as a compute-quality Pareto frontier that existing methods cannot escape.

Beyond the immediate cost-quality tradeoff, there is a deeper data utilization problem. Organizations collect large datasets of expert trajectories—either from human demonstrations, stronger models, or synthetic pipelines—to bootstrap agentic capabilities. SFT uses these trajectories inefficiently: it only trains the model to reproduce exactly one action per decision point, ignoring the vast space of functionally equivalent actions that would also be correct. If a coding agent's expert trace shows search("function definition") but the model generates grep "function definition" src/—a semantically identical action—SFT penalizes this as an error. Meanwhile, E2E RL uses the expert data only indirectly (if at all), preferring to learn from scratch through trial and error, which discards the hard-won knowledge encoded in those trajectories.

There is also a theoretical dimension. Rajaraman et al. (2020) showed that the suboptimality of offline imitation (behavior cloning) grows quadratically with task horizon. For a task with horizon $T$, the error compounds as $O(T^2 \epsilon)$ where $\epsilon$ is the single-step error rate. On long-horizon agentic tasks ($T$ = 10–30 turns), even a modest per-step error rate leads to near-guaranteed failure of the overall trajectory. The theoretical fix is interaction: DAgger-style algorithms (Ross et al., 2011) interleave learning with expert corrections on the learner's own state distribution, reducing the compounding error. E2E RL achieves something similar through on-policy exploration, but at the cost of full-trajectory rollouts. The paper positions PivotRL as a middle ground: it uses expert states (from the SFT dataset) to provide the "corrective" on-policy data without needing full trajectories, effectively achieving the benefit of interaction at a fraction of the cost.

Prior Attempts and Their Shortcomings

The paper identifies three categories of prior work that attempt to bridge SFT and RL, each with limitations that motivate PivotRL's design:

1. Naive Local RL from Expert Trajectories. The simplest attempt is to convert SFT trajectories into RL episodes by conditioning the model on intermediate states from the expert trace and then rewarding actions that exactly match the demonstration. Specifically, sample a state $s_t$ from an SFT trajectory, generate on-policy actions $a \sim \pi_\theta(\cdot \mid s_t)$, and assign:

$r_{\text{strict}}(s, a) = \mathbf{1}[a = a^*(s)]$

This approach—which the paper calls "naive local RL"—addresses the offline nature of SFT by introducing on-policy sampling, but the authors' preliminary experiments reveal that it fails to improve OOD accuracy relative to standard SFT on the same data (Table 4: 57.34 vs. 58.44 for SFT on $\tau$2-Bench). The paper traces this failure to two specific bottlenecks:

• Uninformative turns dominate. Under GRPO's group-normalized advantage, if all $G$ sampled actions at a turn either succeed or fail uniformly, the normalized advantage $\hat{A}_i$ evaluates to zero for every action in the group, producing no meaningful gradient update (Proposition 3.1). Empirically, 71% of randomly sampled turns produce exactly this—zero learning signal—meaning most of the rollout budget is wasted on states where the model is already consistently correct or consistently wrong.

• Exact-match credit is overly restrictive. In generative action spaces (tool calls, shell commands, search queries), countless actions are functionally equivalent to the demonstration without being string-identical. The paper quantifies this via a miss rate: $\Pr[r_{\text{strict}}(s, a) = 0 \mid r_{\text{func}}(s, a) = 1]$, where $r_{\text{func}}$ is a permissive verifier that accepts any functionally correct action. When many actions are functionally acceptable but not exact matches, the strict reward throws away valid positive examples, reducing the effective sample size for RL and introducing noise into the advantage estimates.

The equivalence of 57.34 (naive local RL) and 58.44 (same-data SFT) on $\tau$2-Bench is damning: simply converting SFT data into RL format without addressing which states to train on and how to assign credit yields no improvement over behavior cloning, despite the additional compute spent on on-policy rollouts.

2. End-to-End RL with Full Trajectory Rollouts. The alternative is to abandon expert trajectories entirely and train through complete environment interaction. This has been successful—OpenHands (Wang et al., 2025b), SWE-Gym (Pan et al., 2025), and various coding agents achieve strong SWE-Bench results through E2E RL—but the compute cost is extreme. As established, reaching 32.67% on SWE-Bench Verified requires ~542K rollout turns. For a production pipeline generating hundreds of training trajectories per update across multiple agentic domains, this cost multiplies rapidly. The paper does not argue that E2E RL is ineffective; it argues that E2E RL's cost prevents it from being the default choice for most practitioners, creating a gap for methods that achieve comparable accuracy with fewer environment interactions.

3. Hybrid Methods That Combine SFT and RL. Several prior works attempt to combine the strengths of SFT and RL. Uchendu et al. (2023) and Hester et al. (2018) use expert demonstrations to jump-start RL training, often by pre-training the policy with SFT and then fine-tuning with RL. Setlur et al. (2026) condition on partial trace prefixes to reuse computation. Ming et al. (2026) repurpose SFT tokens directly as rollout rewards. These methods share a common philosophy: use offline data to accelerate online learning. However, none of them systematically address the turn-selection problem—which states in a long trajectory are worth spending rollout budget on—or the credit-assignment problem in the context of generative action spaces where exact matching is too strict. PivotRL differentiates itself by treating both problems as first-class design decisions with principled justifications (theoretical results in Section 3.2) rather than ad-hoc adjustments.

The Two Bottlenecks in Detail

The paper's motivating observations (Section 2, final paragraph) are worth examining closely because they directly justify PivotRL's two core mechanisms:

Bottleneck 1: Homogeneous rollout groups produce zero advantage. Under GRPO, the normalized advantage for action $i$ in a group of size $G$ is:

$\hat{A}_i = \frac{r_i - \frac{1}{G}\sum_{j=1}^G r_j}{\text{std}(\{r_j\}_{j=1}^G) + \epsilon_{\text{std}}}$

If all rewards are identical—either all 1 (every sampled action succeeds) or all 0 (every sampled action fails)—then $r_i - \frac{1}{G}\sum r_j = 0$ for all $i$, and $\hat{A}_i = 0$. The policy gradient update vanishes. This is the mathematical expression of the intuition that you cannot learn from a state where the model already "knows what to do" (all correct) or "has no good options" (all incorrect). The policy needs contrast: some actions that work and some that don't, so that it can learn to discriminate between them.

The paper's finding that 71% of randomly sampled turns produce zero signal is striking. On a long-horizon agentic trajectory of 15 turns, this means approximately 10–11 turns contribute nothing to learning if you sample uniformly. The remaining 4–5 turns carry the entire training burden. But because uniform sampling doesn't distinguish between informative and uninformative turns, most of the rollout budget (and therefore most of the training compute) is wasted on states that produce no gradient.

This observation is the direct motivation for pivot filtering: instead of training on all turns, profile each turn offline under the reference policy (by sampling actions and scoring them with the verifier), and retain only those with nonzero reward variance—turns where sampled actions produce mixed success/failure outcomes. These "pivot" turns concentrate the training budget on states that actually generate learning signals.

Bottleneck 2: Exact-match reward discards functionally correct actions. In an agentic action space, the set of acceptable actions at a state $s$, denoted $\mathcal{M}(s)$, is typically large. For a conversational assistant, many different API call sequences might satisfy a user's request. For a coding agent, many different tool invocations might correctly locate and fix a bug. The SFT demonstration $a^*(s)$ is just one element of $\mathcal{M}(s)$. If the reward function only credits exact string matches:

$r_{\text{strict}}(s, a) = \mathbf{1}[a = a^*(s)]$

then for any $a \in \mathcal{M}(s)$ where $a \neq a^*(s)$, the reward is zero. This has two damaging effects:

• Reduced effective sample size. If $\mathcal{M}(s)$ contains many actions but only $a^*(s)$ gets credit, the expected reward under the reference policy is $\pi_0(a^*(s) \mid s)$, which might be very small. Most sampled actions that are functionally correct receive zero reward, reducing the number of positive examples in each rollout group and making the advantage estimates noisy.

• Wrong credit signal. Exact matching penalizes valid variations, teaching the model to reproduce the demonstration verbatim rather than learn the underlying task structure. This is behavior cloning in disguise: the RL objective collapses to SFT on the single demonstrated action, because only that action has nonzero probability of receiving reward.

The paper quantifies this via the miss rate—the probability that an action is functionally correct but fails the exact-match test. A high miss rate means the strict reward is systematically misclassifying good actions as bad, corrupting the policy gradient.

How PivotRL Positions Itself

PivotRL is not a new RL algorithm or a new architecture. It is a framework for converting existing SFT trajectories into an efficient RL training pipeline by addressing the two bottlenecks described above. Its key design insight is that you do not need full-trajectory rollouts to get the benefits of on-policy RL—you only need on-policy sampling at specific, carefully chosen decision points, and you need a reward function that recognizes the broader space of acceptable actions rather than demanding exact imitation.

The paper explicitly draws on theoretical foundations to justify both mechanisms (Section 3.2):

• Proposition 3.1 and Theorem 3.2 formalize why mixed-outcome turns are uniquely valuable: the natural gradient norm of the statewise reward objective scales directly with reward variance, meaning that states with higher variance produce stronger per-sample learning signals. This is not a heuristic—it is a property of the KL-regularized GRPO objective along the exponential-tilt path $\pi_{s,\beta} \propto \pi_0 e^{r(s,\cdot)/\beta}$.
• Theorem 3.3 shows that functional reward (assigning credit to any $a \in \mathcal{M}(s)$) produces a policy that shifts probability mass toward acceptable actions while preserving the reference policy's conditional distribution within both the acceptable set and its complement. This means the model's relative preferences among task-unrelated actions remain unchanged, explaining why PivotRL retains OOD performance while SFT destroys it.

The paper's framing is deliberately pragmatic: PivotRL is designed to slot into existing post-training pipelines that already have SFT trajectory datasets. It does not require new data collection, new reward models, or new infrastructure beyond what is needed for E2E RL. It repurposes the SFT data that organizations already possess, converting it from a behavior-cloning resource into an RL resource by selectively sampling on-policy actions at pivot states and evaluating them with domain-specific verifiers.

This positioning is validated by the paper's deployment context: PivotRL is used as a production-scale post-training stage in NVIDIA's Nemotron-3-Super model (Table 5), where it handles the agentic verticals while other RL environments handle reasoning and chat. The claim is not that PivotRL replaces SFT or E2E RL entirely, but that it occupies a previously empty point on the Pareto frontier: near-E2E-RL accuracy at near-SFT compute cost, achieved by being selective about where and how to spend the on-policy rollout budget.

3. Technical Approach

3.1 Reader Orientation

PivotRL is a turn-level reinforcement learning algorithm that converts existing SFT trajectory datasets into an efficient on-policy training pipeline for long-horizon agentic tasks. It solves the tension between SFT's OOD degradation and E2E RL's prohibitive compute cost by doing two things: (1) spending rollout budget only on pivot states—intermediate trajectory turns where sampled actions produce mixed success/failure outcomes—and (2) evaluating those rollouts with functional-equivalent reward that accepts any locally correct action rather than demanding strict string matching with the SFT demonstration.

3.2 Big-Picture Architecture (Diagram in Words)

PivotRL has five major components that operate in sequence:

1. SFT Trajectory Dataset — a pre-existing collection of expert trajectories $\mathcal{D}_{\text{sft}} = \{\tau_i\}$, where each trajectory is a sequence of states and demonstrated actions $(s_0, a_0^*, \dots, s_T, a_T^*)$. This is the same data that would be used for standard supervised fine-tuning.

2. Turn Extraction — every assistant turn from every trajectory is extracted into a flat dataset of pivot candidates $\mathcal{D}_{\text{cand}} = \{(s_t, a_t^*)\}$. Each item pairs a state (the full interaction history up to that turn) with the expert's demonstrated action.

3. Offline Turn Profiling and Filtering — before any RL training begins, each candidate turn is profiled under a frozen reference policy $\pi_0$. For each state $s$, $K$ actions are sampled from $\pi_0(\cdot \mid s)$, scored with a domain-specific functional verifier $r_{\text{func}}$, and the empirical reward variance $\hat{\sigma}^2(s)$ and mean $\hat{\mu}(s)$ are computed. Turns with zero reward variance (uniformly solved or uniformly failed) are discarded, and turns with reward mean above a threshold $\lambda_{\text{diff}}$ are also removed. The surviving turns form the pivot set $\mathcal{D}_{\text{pivot}} = \mathcal{D}_{\text{adv}}$.

4. Local On-Policy Rollout — during training, states are sampled from $\mathcal{D}_{\text{pivot}}$. At each sampled state $s$, the current policy $\pi_{\theta_{\text{old}}}$ generates $G$ actions $\{a_i\}_{i=1}^G$. Each action is executed locally to obtain a verifier score $r_{\text{func}}(s, a_i)$. Critically, this is a single-turn execution—the model does not continue the trajectory beyond scoring this one action.

5. GRPO-style Optimization — the $G$ actions and their verifier scores are used to compute group-normalized advantages $\hat{A}_i$, and the policy is updated via a clipped importance-weighted objective with a KL penalty toward the reference policy.

Information flows as follows: expert trajectories → all assistant turns extracted → each turn profiled offline with reference policy and verifier → uninformative turns discarded → remaining pivot states stored → during training, pivot states sampled → current policy generates single-turn actions → verifier scores each action → group-normalized advantages computed → policy updated.

3.3 Roadmap for the Deep Dive

• First, the turn extraction and candidate dataset construction—how a long multi-turn trajectory is decomposed into individual states for local RL. This establishes the data structure that everything else operates on.

• Second, the offline turn profiling and filtering mechanism—how pivot states are identified and why the reward variance and mean thresholds are the right criteria. This is the mechanism that addresses Bottleneck 1 (uninformative turns), and I will walk through Proposition 3.1 and Theorem 3.2 to explain why reward variance matters theoretically.

• Third, the functional-equivalent reward function and the local rollout procedure—how credit is assigned to sampled actions at pivot states. This addresses Bottleneck 2 (overly strict exact-match credit), and I will walk through Theorem 3.3 to explain why functional reward preserves OOD performance.

• Fourth, the GRPO optimization objective and how it differs from standard E2E RL—the clipped surrogate loss, the KL penalty, and the group normalization. This is where the actual policy update happens.

• Fifth, the full training loop (Algorithm 1) connecting all components.

• Sixth, domain-specific instantiations—how turn definitions, data construction, and verifier design differ across conversational tool use, software engineering, terminal control, and web browsing, since these choices determine the practical scope of the method.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a method and empirical analysis paper whose core idea is that you can achieve near-E2E-RL accuracy with near-SFT compute cost by being selective about which states you train on and how you assign credit at those states.

---

Turn Extraction and Candidate Dataset Construction

Every long-horizon agentic trajectory is a sequence of alternating environment observations and model responses. For a trajectory $\tau$, the decomposition at assistant decision boundaries yields:

$\tau = (s_0, a_0^*, s_1, a_1^*, \dots, s_T, a_T^*)$

where $s_t$ is the full interaction history from the beginning of $\tau$ up to, but not including, the $t$-th assistant action, and $a_t^*$ is the demonstrated assistant completion at that state.

What this means operationally: At turn $t$, the state $s_t$ contains everything the model has seen so far—the initial user instruction, all previous environment observations, and all previous assistant actions. The demonstrated action $a_t^*$ is what the expert (whether human, stronger model, or synthetic pipeline) chose to do next. This is the standard formalism for turn-level training: actions are entire model completions at call boundaries, not individual tokens.

Turn extraction flattens every trajectory in the SFT dataset into a single collection:

$\mathcal{D}_{\text{cand}} = \{(s_t, a_t^*)\}_{\tau \in \mathcal{D}_{\text{SFT}}, t=0,\dots,T}$

where $\mathcal{D}_{\text{SFT}}$ is the original SFT trajectory dataset, $\tau$ ranges over all trajectories, and $t$ ranges over all assistant turns within each trajectory.

What this produces: A flat dataset of state-action pairs, where each pair represents one decision point in one trajectory. For a dataset of $N$ trajectories with average horizon $T+1$, this yields approximately $N \times (T+1)$ candidate training examples. In the $\tau$2-Bench domain, the paper uses 281,774 trajectories (Appendix A.2); at roughly 5–15 turns each, this produces millions of candidate states.

Why this decomposition matters: Standard SFT trains on exactly these pairs, maximizing $\log \pi_\theta(a_t^* \mid s_t)$—the model learns to reproduce the expert's exact action at each decision point. PivotRL repurposes the same pairs differently: the state $s_t$ becomes the conditioning context for on-policy sampling, and $a_t^*$ becomes one (of potentially many) reference points for defining the acceptable action set. The key difference is that while SFT only uses $a_t^*$ as a positive training target, PivotRL uses $s_t$ as an exploration launch point and evaluates whatever actions the current policy generates against a broader functional correctness criterion.

A subtle point about action granularity: Throughout the paper, an "action" is the full assistant completion at a model-call boundary—not an individual token. Depending on the domain, this completion might be a conversational tool-use turn (natural language plus API calls), a coding-agent tool invocation (e.g., search("query")), a bash command, or a search/browsing step. Natural-language-only assistant turns are also considered actions when the benchmark includes them. This turn-level granularity is coarser than token-level RL (which would assign credit to each generated token) and matches the natural decision boundaries in agentic interaction.

---

Offline Turn Profiling and Pivot Filtering

This is the mechanism that addresses Bottleneck 1: the observation that 71% of randomly sampled turns produce zero learning signal under GRPO. The solution is to profile every candidate turn before training and retain only those likely to generate nonzero advantages.

Step 1: Reference policy sampling. For each candidate state $s$ in $\mathcal{D}_{\text{cand}}$, the paper samples $K$ local rollouts from a frozen reference policy $\pi_0$ (typically the policy used to initialize PivotRL):

$a^{(k)} \sim \pi_0(\cdot \mid s), \quad k = 1, \dots, K$

What this means: The reference policy $\pi_0$ is the model before any PivotRL training—often the base model or an SFT-initialized checkpoint. For each candidate state, we ask: "if we sample $K$ actions from the current model at this state, what pattern of successes and failures do we see?" This is purely an offline profiling step; no gradient updates occur. The paper does not specify $K$ explicitly in the main text, but Appendix A.2 references profiling details in the domain-specific sections.

Step 2: Verifier scoring. Each sampled action is scored using the domain-specific functional verifier $r_{\text{func}}$ (detailed in the next subsection). This produces a batch of $K$ binary rewards $\{r_{\text{func}}(s, a^{(k)})\}_{k=1}^K \in \{0, 1\}$.

Step 3: Compute empirical statistics. From these $K$ rewards, the paper computes:

$\hat{\mu}(s) = \frac{1}{K} \sum_{k=1}^K r_{\text{func}}(s, a^{(k)})$

$\hat{\sigma}^2(s) = \frac{1}{K} \sum_{k=1}^K \left(r_{\text{func}}(s, a^{(k)}) - \hat{\mu}(s)\right)^2$

where $\hat{\mu}(s)$ is the empirical success rate at state $s$ under the reference policy and $\hat{\sigma}^2(s)$ is the empirical reward variance.

What these statistics capture: $\hat{\mu}(s) \approx 1$ means the model almost always succeeds at this state (it's an "easy" turn). $\hat{\mu}(s) \approx 0$ means the model almost always fails (it's a "hard" turn, or the model has no good options). $\hat{\sigma}^2(s) > 0$ means the model sometimes succeeds and sometimes fails—actions at this state produce mixed outcomes.

Step 4: Filter to form the pivot set. The retained training set $\mathcal{D}_{\text{adv}}$ is:

$\mathcal{D}_{\text{adv}} = \left\{(s, a^*) \in \mathcal{D}_{\text{cand}} : \hat{\sigma}^2(s) > 0, \ \hat{\mu}(s) < \lambda_{\text{diff}}\right\}$

where $\lambda_{\text{diff}}$ is a difficulty threshold. The first filter $\hat{\sigma}^2(s) > 0$ removes turns where the model is either uniformly correct or uniformly incorrect. The second filter $\hat{\mu}(s) < \lambda_{\text{diff}}$ concentrates training on mixed-outcome turns that are still difficult (success rate below threshold), preventing the model from spending rollout budget on states it already mostly solves.

Why varianace matters theoretically—Proposition 3.1: The paper proves a simple but crucial fact:

Proposition 3.1 (Only mixed-outcome turns produce nonzero group-normalized updates). Let $s$ be a fixed state and let $\{r_i\}_{i=1}^G$ be the binary rewards of a rollout group at $s$. If all rewards are identical, then the normalized advantages in Eq. (1) are zero for every $i$. Equivalently, only rollout groups with positive reward variance can contribute a nonzero group-normalized update.

What this proves: Under GRPO's advantage normalization, if every sampled action at a state gets the same reward—whether all 0 or all 1—then $\hat{A}_i = 0$ for all $i$, and the policy gradient update vanishes. The reason is mechanical: the group-normalized advantage subtracts the group mean and divides by the standard deviation. If all rewards are identical, the standard deviation is zero and each $r_i - \frac{1}{G}\sum r_j = 0$. The $\epsilon_{\text{std}}$ term in the denominator prevents division by zero but still produces advantages of zero.

What this means for training: Any rollout budget spent on a uniformly solved or uniformly failed state is wasted—it contributes nothing to the gradient. By filtering out states with $\hat{\sigma}^2(s) = 0$ (which in practice means all $K$ sampled rewards are identical), PivotRL ensures that every state in the training set has a chance of producing nonzero advantages. The empirical finding that 71% of randomly sampled turns produce zero signal explains why naive local RL (which samples uniformly) performs no better than SFT: most of its compute is spent on gradient-free updates.

Why variance matters theoretically—Theorem 3.2: The deeper theoretical result shows that reward variance is not just a binary filter criterion but the fundamental scale of the learning signal. The theorem considers the statewise expected reward objective:

$J_s(\pi) = \mathbb{E}_{a \sim \pi}[r(s, a)]$

and a specific KL-regularized path—the exponential-tilt distribution:

$\pi_{s,\beta}(a) = \frac{\pi_0(a \mid s) \ e^{r(s,a)/\beta}}{\sum_{b \in \mathcal{A}(s)} \pi_0(b \mid s) \ e^{r(s,b)/\beta}}$

where $\beta > 0$ is the KL penalty coefficient. This is the idealized minimizer of the statewise objective $-J_s(\pi) + \beta \ \text{KL}(\pi \ \| \ \pi_0)$.

The theorem then defines a quantity called the population GRPO score:

$\gamma_{s,\beta} := -\mathbb{E}_{a \sim \pi_{s,\beta}} \left[ \frac{r(s, a) - \mathbb{E}_{a' \sim \pi_{s,\beta}} [r(s, a')]}{\sqrt{\text{Var}_{a' \sim \pi_{s,\beta}} (r(s, a'))}} \ \partial_\beta \log \pi_{s,\beta}(a) \right]$

What this quantity represents: $\gamma_{s,\beta}$ is a theoretical proxy for the magnitude of the GRPO update at state $s$ along the KL-regularized path $\pi_{s,\beta}$. The expectation is over actions drawn from the idealized KL-regularized policy; the term inside brackets is the product of the normalized advantage (the same form as $\hat{A}_i$ in GRPO) and the derivative of the log-policy with respect to $\beta$, which measures how the policy changes as the KL penalty is varied.

The theorem then establishes:

$\gamma_{s,\beta} = \frac{1}{\beta^2} \ \|\nabla_{\text{nat}} J_s(\pi_{s,\beta})\|^2_{F,\pi_{s,\beta}} = \frac{\sqrt{\text{Var}_{a \sim \pi_{s,\beta}}(r(s, a))}}{\beta^2}$

where $\nabla_{\text{nat}} J_s(\pi_{s,\beta})$ is the natural gradient of $J_s$ under the Fisher information metric.

What this proves: The population GRPO score (and therefore the magnitude of the idealized GRPO update) scales directly with the reward standard deviation $\sqrt{\text{Var}(r(s,a))}$. At fixed $\beta$, states with larger reward variance induce larger natural gradient norms and larger population GRPO scores. The natural gradient norm itself equals the reward standard deviation (the first equality), and the GRPO score along the KL path is exactly $\sigma / \beta^2$.

Why the natural gradient connection matters: The natural gradient is the steepest descent direction under the Fisher metric, which accounts for the geometry of the probability distribution rather than Euclidean parameter space. The fact that its norm equals the reward standard deviation means that variance in outcomes is the only thing that determines the magnitude of the theoretically optimal per-state update. It's not a heuristic correlation; it's an identity for the exponential-tilt path.

The proof sketch (Appendix B.1): The proof proceeds in two steps. First, it computes the natural gradient of $J_s$ at $\pi$ as $\nabla_{\text{nat}} J_s(\pi) = r(s, \cdot) - \mathbb{E}_\pi[r(s, \cdot)]$—the centered reward function. Its Fisher norm is therefore exactly the variance. Second, it differentiates $\log \pi_{s,\beta}(a) = \log \pi_0(a \mid s) + r(s,a)/\beta - \log Z_s(\beta)$ with respect to $\beta$, showing that $\partial_\beta \log \pi_{s,\beta}(a) = -\frac{1}{\beta^2}(r(s,a) - \mathbb{E}_{\pi_{s,\beta}}[r(s,\cdot)])$. Substituting into the definition of $\gamma_{s,\beta}$ yields $\sigma / \beta^2$.

Practical implication of the theory: The theory justifies the choice to filter for states with high reward variance as a principled way to maximize per-sample learning signal. It's not just that uniform-reward states produce zero advantage (Proposition 3.1); it's that among states with nonzero variance, those with higher variance produce proportionally stronger updates (Theorem 3.2). This explains why PivotRL outperforms random turn selection: pivots concentrate the training budget on the most informative decision points, where each sampled rollout contributes maximally to changing the policy.

The difficulty threshold $\lambda_{\text{diff}}$: After filtering for nonzero variance, PivotRL further filters for $\hat{\mu}(s) < \lambda_{\text{diff}}$. This removes states where the model already succeeds most of the time (high $\hat{\mu}$) even though there is still some variance. The intuition is that states with high success rates provide limited room for improvement—the model is already mostly correct—so spending rollout budget there has diminishing returns. The paper does not specify the exact value of $\lambda_{\text{diff}}$ in the main text (it is domain-specific), but the ablation in Table 6 shows the effect: "low-reward-mean pivots" ($\mathcal{D}_{\text{adv}}$) achieve 63.81 on $\tau$2-Bench versus 59.68 for "random pivots" ($\mathcal{D}_{\text{cand}}$), a +4.13 improvement from the additional difficulty filter.

The pivot terminology: Throughout the paper, "pivot" refers specifically to the filtered subset used for training. A "pivot candidate" is any extracted turn; a "pivot" is a turn that survived filtering. The name evokes the idea of a fulcrum point: these are the decision points where the policy is poised between success and failure, and where training can most effectively shift it toward better outcomes.

---

Functional-Equivalent Reward and Local Rollout

This is the mechanism that addresses Bottleneck 2: the observation that strict exact-match reward discards functionally correct actions, reducing effective sample size and corrupting the credit signal.

The strict reward baseline. The simplest way to convert an SFT demonstration into an RL reward is:

$r_{\text{strict}}(s, a) = \mathbf{1}[a = a^*(s)]$

where $a^*(s)$ is the expert's demonstrated action at state $s$. This rewards the model only when it exactly reproduces the demonstration, character by character. The paper's preliminary experiments (Section 2) show that this "naive local RL" approach achieves 57.34 on $\tau$2-Bench, which is slightly worse than same-data SFT at 58.44. This confirms that simply adding on-policy sampling with exact-match credit does not improve over behavior cloning.

Why strict reward fails. In generative action spaces, the set of locally acceptable actions $\mathcal{M}(s)$ is typically much larger than the singleton $\{a^*(s)\}$. For a coding agent, grep "pattern" file.py, find . -name "*.py" -exec grep "pattern" {} \;, and search("pattern") might all be functionally correct ways to locate code. For a conversational assistant, different API call sequences might satisfy the same user intent. The strict reward only credits one of these, assigning zero reward to all other valid actions.

The paper quantifies this failure mode via the miss rate:

$\text{miss rate} = \Pr[r_{\text{strict}}(s, a) = 0 \mid r_{\text{func}}(s, a) = 1]$

where $r_{\text{func}}$ is the permissive functional verifier. A high miss rate means many actions that are functionally correct fail the exact-match test. When the miss rate is high, the strict reward systematically undercounts successes, making the RL signal noisy—the model receives zero reward for good actions that happen to differ from the demonstration, and it cannot distinguish between genuinely bad actions and valid variations.

The PivotRL reward function. Instead of exact matching, PivotRL assigns reward based on functional acceptability:

$r_{\text{func}}(s, a) = \mathbf{1}[a \in \mathcal{M}(s)]$

where $\mathcal{M}(s) \subseteq \mathcal{A}(s)$ is the set of locally acceptable actions under a domain-specific verifier.

What this means: For any state $s$, the verifier defines a set of actions that are "good enough" at this turn. An action receives reward 1 if it belongs to this set, regardless of whether it matches the demonstration. An action receives reward 0 otherwise. This is still a binary reward (0 or 1), but the "1" region is expanded from a single point to a set.

What makes this functional rather than exact: The verifier does not compare the sampled action to the demonstration. It evaluates the action on its own terms using domain knowledge: does the tool call have the right schema? Does the bash command achieve the intended effect? Does the search query retrieve relevant results? The verifier is a lightweight programmatic check, not a learned reward model—it encodes rules about what makes an action acceptable at a particular decision point.

The verifier design varies by domain (Appendix A.2):

• $\tau$2-Bench (conversational tool use): The verifier performs output-schema validation, normalized string similarity, and equivalence-based LLM-as-judge scoring over the tool call and its immediate effect. It checks whether a sampled tool call is semantically interchangeable with the demonstrated call—not whether the strings match.

• SWE-Bench Verified (software engineering): The verifier uses a deliberately coarse signal: it matches tool-call names only. It checks whether the model selected the correct type of operation (e.g., search, open, edit, run) without attempting to score tool arguments or patch quality. The paper explicitly states this is a "coarse local signal"; final task success is determined only by the full SWE-Bench evaluation harness.

• Terminal-Bench (terminal control): The verifier combines output-schema validation, normalized string similarity, and equivalence-based LLM-as-judge scoring over the command and its immediate effect, asking whether the sampled command is "locally interchangeable" with the demonstrated command.

• BrowseComp (web browsing): The verifier checks whether the browsing step (search query, result opening, evidence gathering) is functionally appropriate for the multi-hop question-answering task. Specific verifier details are lighter in the paper for this domain.

Why this reduces to EXACT matching as a special case: If $\mathcal{M}(s) = \{a^*(s)\}$ (the acceptable set contains only the demonstrated action), then $r_{\text{func}} = r_{\text{strict}}$. The functional reward is a strict generalization: it can always be made as restrictive as exact matching, but can be relaxed to accept more actions when domain knowledge supports it.

The local rollout procedure. At each selected pivot state $s$, PivotRL samples $G$ actions from the current policy:

$\{a_i\}_{i=1}^G \sim \pi_{\theta_{\text{old}}}(\cdot \mid s)$

Each action is then executed in the environment to obtain its functional reward. The execution is local: the model generates one action, the environment processes it, the verifier scores the outcome, and then the process stops. The model does not continue the trajectory from the resulting state. This is the key efficiency gain over E2E RL: rather than generating and executing a full trajectory of length $T$ (requiring $T$ model calls and $T$ environment interactions per sample), PivotRL only requires 1 model call and 1 environment interaction per sample.

The paper uses $G$ (rollout group size) as a hyperparameter. In the SWE-Bench comparison (Appendix A.2), PivotRL uses a batch size of 1024 split as 64 prompts × 16 generations per prompt, meaning $G = 16$.

What happens to the demonstrated action $a^*(s)$ during RL? It is not directly used in the reward computation. The demonstrated action serves only as an anchor for the pivot selection process (it appears in $\mathcal{D}_{\text{cand}}$ and $\mathcal{D}_{\text{adv}}$) and as context for the verifier in some domains (e.g., Terminal-Bench's LLM-as-judge compares the sampled command to the demonstrated command). During the RL update itself, the only signal is the verifier score on the sampled actions.

Why functional reward preserves OOD performance—Theorem 3.3: This is the theoretical result that explains PivotRL's strong OOD retention. The theorem analyzes the regularized objective that PivotRL approximately minimizes:

$\mathcal{L}_{\text{func},\beta}(\pi) = \mathbb{E}_{s \sim d} \left[ -\mathbb{E}_{a \sim \pi(\cdot \mid s)}[r_{\text{func}}(s, a)] + \beta \ \text{KL}(\pi(\cdot \mid s) \ \| \ \pi_0(\cdot \mid s)) \right]$

where $r_{\text{func}}(s, a) = \mathbf{1}[a \in \mathcal{M}(s)]$, $\beta > 0$ is the KL penalty coefficient, $d$ is a fixed state distribution (the distribution induced by sampling from $\mathcal{D}_{\text{pivot}}$), and $\pi_0$ is the reference policy.

What this objective does: It encourages the policy to put high probability on actions in $\mathcal{M}(s)$ (minimizing $-\mathbb{E}[r_{\text{func}}]$), while penalizing deviation from the reference policy (the KL term). The $\beta$ parameter trades off these two forces: small $\beta$ prioritizes reward maximization and allows large KL divergence; large $\beta$ prioritizes staying close to $\pi_0$.

For each state $s$, define:

$\rho(s) = \pi_0(\mathcal{M}(s) \mid s)$

the total probability mass that the reference policy assigns to acceptable actions at state $s$. Define:

$q_\beta(s) = \frac{\rho(s) e^{1/\beta}}{(1 - \rho(s)) + \rho(s) e^{1/\beta}}$

which is the optimal total mass on $\mathcal{M}(s)$ after KL regularization.

The theorem then states: $\mathcal{L}_{\text{func},\beta}$ has a unique minimizer $\pi_\beta^*$ such that, for each state $s$ ($d$-almost surely):

$\pi_\beta^*(\mathcal{M}(s) \mid s) = q_\beta(s) \geq \rho(s)$

with strict inequality whenever $0 < \rho(s) < 1$.

What this means: The optimal policy under functional-reward RL increases the total probability mass on acceptable actions—from $\rho(s)$ under the reference policy to $q_\beta(s)$ under the trained policy. The increase is larger when $\beta$ is smaller (weaker KL penalty) and when $\rho(s)$ is further from 0 or 1 (more room to shift mass). This is the "in-domain improvement": the model gets better at producing acceptable actions at the states it was trained on.

The crucial second part: Among all distributions satisfying the condition $\pi(\mathcal{M}(s) \mid s) = q_\beta(s)$, the minimizer $\pi_\beta^*(\cdot \mid s)$ is the unique minimizer of $\text{KL}(\pi(\cdot \mid s) \ \| \ \pi_0(\cdot \mid s))$. Moreover, it preserves the reference policy's conditional distribution within both the acceptable set and its complement:

$\frac{\pi_\beta^*(a \mid s)}{\pi_\beta^*(b \mid s)} = \frac{\pi_0(a \mid s)}{\pi_0(b \mid s)} \quad \text{for } a, b \in \mathcal{M}(s)$

$\frac{\pi_\beta^*(a \mid s)}{\pi_\beta^*(b \mid s)} = \frac{\pi_0(a \mid s)}{\pi_0(b \mid s)} \quad \text{for } a, b \in \mathcal{M}(s)^c$

What this means in plain language: The trained policy reallocates probability mass between the acceptable set and the unacceptable set (shifting mass from the complement into $\mathcal{M}(s)$), but it does not reorder actions within either set. If action $a$ was twice as likely as action $b$ under the reference policy, and both are in $\mathcal{M}(s)$, then $a$ remains twice as likely under the trained policy. The same holds for any two actions in the complement $\mathcal{M}(s)^c$. The relative preferences among actions that serve the same functional role (or are both irrelevant to the current task) are preserved exactly.

Why this explains OOD retention: In a large language model, any given action (a specific tool call, a specific natural language response) is typically relevant to only a small number of tasks. For a state $s$ drawn from the pivot training distribution (which comes from a specific agentic domain), the acceptable set $\mathcal{M}(s)$ contains actions relevant to that task. The complement $\mathcal{M}(s)^c$ contains all other actions—actions relevant to math problems, science questions, coding tasks, translation, and everything else the model was pretrained on. Theorem 3.3 says that the relative ordering among all these task-unrelated actions is preserved. The model does not forget that find the derivative of... is a better response to a calculus question than list files in directory—because both are in $\mathcal{M}(s)^c$ for the tool-use state $s$, and their relative probability ratio is unchanged.

Contrast with SFT: Standard SFT trains the model to maximize $\log \pi_\theta(a^*(s) \mid s)$. This directly pushes probability mass toward the single demonstrated action $a^*(s)$ and pushes mass away from all other actions, including those in $\mathcal{M}(s)^c$ that are irrelevant to the current task. There is no guarantee that SFT preserves the relative ordering among task-unrelated actions. In fact, the empirical evidence (Table 2, average OOD change of -9.83 for SFT vs. +0.21 for PivotRL) suggests exactly the opposite: SFT catastrophically reshuffles the model's broader knowledge, while the KL-regularized, functional-reward optimization of PivotRL preserves it.

The exact form of the minimizer (Appendix B.2, Eq. 16):

$\pi_\beta^*(a \mid s) = \begin{cases} \frac{q_\beta(s)}{\rho(s)} \pi_0(a \mid s), & a \in \mathcal{M}(s) \\ \frac{1 - q_\beta(s)}{1 - \rho(s)} \pi_0(a \mid s), & a \notin \mathcal{M}(s) \end{cases}$

This is a block-rescaled version of the reference policy: multiply all probabilities in $\mathcal{M}(s)$ by $q_\beta(s) / \rho(s)$ (which is $\geq 1$) and all probabilities in $\mathcal{M}(s)^c$ by $(1 - q_\beta(s)) / (1 - \rho(s))$ (which is $\leq 1$). The rescaling is uniform within each block, so the relative ordering is preserved.

Practical caveat: Theorem 3.3 analyzes the exact minimizer of the idealized regularized objective, not the finite-sample GRPO update that PivotRL actually implements. The connection is that GRPO with a KL penalty approximately follows the gradient of this objective. The theorem provides the qualitative insight—functional reward + KL regularization = block-rescaling that preserves relative preferences—even though the actual training trajectory is noisy and approximate.

---

The GRPO Optimization Objective

PivotRL's parameter update uses the standard GRPO (Group Relative Policy Optimization) objective, adapted to the local, turn-level setting. At each training step, for each state $s$ in the minibatch, the objective is:

$\mathcal{J}_{\text{PivotRL}}(\theta) = \mathbb{E}_{\substack{s \sim \mathcal{D}_{\text{pivot}} \\ \{a_i\}_{i=1}^G \sim \pi_{\theta_{\text{old}}}(\cdot \mid s)}} \left[ \underbrace{\frac{1}{G} \sum_{i=1}^G \min\left(w_i(\theta) \hat{A}_i, \ \text{clip}(w_i(\theta), 1-\epsilon, 1+\epsilon) \hat{A}_i\right)}_{\text{clipped surrogate loss}} - \underbrace{\beta \ \text{KL}(\pi_\theta(\cdot \mid s) \ \| \ \pi_0(\cdot \mid s))}_{\text{KL penalty}} \right]$

What each term means:

• $\mathcal{D}_{\text{pivot}}$ is the filtered pivot set (in practice, $\mathcal{D}_{\text{pivot}} = \mathcal{D}_{\text{adv}}$).
• $\pi_{\theta_{\text{old}}}$ is the policy at the start of the current optimization step—frozen for the duration of the step.
• $G$ is the rollout group size: the number of actions sampled per state.
• $w_i(\theta) = \frac{\pi_\theta(a_i \mid s)}{\pi_{\theta_{\text{old}}}(a_i \mid s)}$ is the importance sampling weight. It measures how much the current policy's probability of action $a_i$ has changed relative to the policy that generated the sample. When $\theta = \theta_{\text{old}}$, all $w_i = 1$.

The group-normalized advantage $\hat{A}_i$: For the $G$ actions sampled at state $s$, each receives a functional reward $r_i = r_{\text{func}}(s, a_i) \in \{0, 1\}$. The advantage for action $i$ is:

$\hat{A}_i = \frac{r_i - \frac{1}{G} \sum_{j=1}^G r_j}{\text{std}(\{r_j\}_{j=1}^G) + \epsilon_{\text{std}}}$

where $\text{std}(\{r_j\})$ is the sample standard deviation of the $G$ rewards and $\epsilon_{\text{std}}$ is a small constant to prevent division by zero.

What this computes: $\hat{A}_i$ is a normalized score: it measures how much better action $i$ is than the average action in its group, in units of the group's standard deviation. If action $i$ succeeded ($r_i = 1$) while most others failed, it receives a large positive advantage. If it failed while most others succeeded, it receives a large negative advantage. If all actions had the same outcome, all advantages are zero (Proposition 3.1).

The min and clip operations: The clipped surrogate loss is:

$\min\left(w_i(\theta) \hat{A}_i, \ \text{clip}(w_i(\theta), 1-\epsilon, 1+\epsilon) \hat{A}_i\right)$

• If $\hat{A}_i > 0$ (action was better than average), the objective encourages increasing $\pi_\theta(a_i \mid s)$, but the gradient is clipped when $w_i(\theta) > 1 + \epsilon$, preventing the policy from changing too much in one update.
• If $\hat{A}_i < 0$ (action was worse than average), the objective encourages decreasing $\pi_\theta(a_i \mid s)$, but the gradient is clipped when $w_i(\theta) < 1 - \epsilon$.

The $\epsilon$ parameter is a small positive number that controls how far the new policy can diverge from the old policy in a single step. The paper does not specify the exact value of $\epsilon$ in the main text; it is a standard GRPO hyperparameter typically set to 0.2.

The KL penalty: $\beta \ \text{KL}(\pi_\theta(\cdot \mid s) \ \| \ \pi_0(\cdot \mid s))$ is an estimate of the Kullback-Leibler divergence between the current policy and the reference policy $\pi_0$, weighted by $\beta$. This term penalizes divergence from the reference policy, serving a dual purpose: (1) it prevents the policy from drifting too far from its initialization, which would cause the importance sampling weights to become unreliable; and (2) it implements the regularization that Theorem 3.3 studies, which is the mechanism for preserving the reference policy's relative action ordering and thus OOD performance.

The reference policy $\pi_0$ is typically the policy at the start of PivotRL training—the same policy used for offline pivot profiling. The KL penalty is computed per-state as an average over the sampled actions.

How this objective differs from standard E2E RL GRPO: The mathematical form is identical to the GRPO objective in Eq. (2) of Section 2. The difference is in what constitutes a "trajectory" and a "reward":

• In E2E RL, the state $s$ is encountered during a full end-to-end trajectory, the action $a_i$ is a complete model turn within that trajectory, and the reward $r_i$ is the final trajectory success signal (often sparse, assigned only at the last turn and propagated back).
• In PivotRL, the state $s$ is sampled directly from $\mathcal{D}_{\text{pivot}}$ (an expert-intermediate state), the action $a_i$ is a single-turn sample conditioned on that state, and the reward $r_i$ is the local functional verifier score for that single action.

This means PivotRL avoids the credit assignment problem that plagues sparse-reward E2E RL: because the reward is local (assigned immediately at each turn based on the verifier), there is no need to propagate credit backward through a multi-turn trajectory. The model learns directly whether an action was good at the specific decision point where it was taken.

What the objective does not do: PivotRL does not use a learned reward model. All rewards come from programmatic verifiers. This is both a strength (no reward model training cost, no reward hacking risk) and a limitation (the verifier must be designed for each domain, and its quality bounds the quality of the RL signal).

The batch composition: A training minibatch consists of $B$ states sampled from $\mathcal{D}_{\text{pivot}}$. For each state, $G$ actions are sampled from $\pi_{\theta_{\text{old}}}$, yielding $B \times G$ total action samples. The objective is averaged over all $B$ states and all $G$ actions per state. In the SWE-Bench configuration, $B = 64$ and $G = 16$, yielding 1024 total action samples per batch.

---

The Full Training Loop

Algorithm 1 in the paper (Section 3.1) describes the complete PivotRL procedure. I will now walk through it in an operational narrative, connecting all the components described above.

Prerequisites: Before PivotRL training begins, the practitioner needs:

1. Expert trajectories $\mathcal{D}_{\text{sft}}$: a dataset of multi-turn agentic trajectories with demonstrated actions at each assistant turn. These are the same trajectories that would be used for SFT.

2. A reference policy $\pi_0$: typically the base model or an SFT-initialized checkpoint. This policy is frozen—it is never updated during PivotRL training. It serves as the starting distribution for pivot profiling and as the KL penalty target.

3. An initial policy $\pi_\theta$: initialized to $\pi_0$ (or a nearby checkpoint). This policy is updated during training.

4. A domain-specific verifier $V$: a function that takes a state $s$ and a sampled action $a$ and returns a binary score $r_{\text{func}}(s, a) \in \{0, 1\}$. Different domains use different verifiers (tool-call matching, command equivalence, etc.).

5. Hyperparameters: rollout group size $G$, difficulty threshold $\lambda_{\text{diff}}$, GRPO-specific parameters ($\epsilon$, $\beta$, $\epsilon_{\text{std}}$), optimizer settings, and number of training steps $K_{\text{train}}$.

Phase 1: Turn extraction (offline, done once). Every assistant turn is extracted from every trajectory in $\mathcal{D}_{\text{sft}}$. The result is:

$\mathcal{D}_{\text{cand}} = \{(s_t, a_t^*)\}_{\tau \in \mathcal{D}_{\text{sft}}, t=0,\dots,T}$

For a dataset with $N$ trajectories of average horizon $H = T+1$, this yields $N \times H$ candidate states. In the $\tau$2-Bench domain, this is millions of candidates; in Terminal-Bench, approximately 20,000 after additional deduplication.

Phase 2: Offline profiling (offline, done once). For each candidate state $s$ in $\mathcal{D}_{\text{cand}}$:

1. Sample $K$ actions from the frozen reference policy: $a^{(k)} \sim \pi_0(\cdot \mid s)$ for $k = 1, \dots, K$.
2. Execute each action in the environment and score with the verifier: $r^{(k)} = r_{\text{func}}(s, a^{(k)})$.
3. Compute empirical statistics: $\hat{\mu}(s) = \frac{1}{K} \sum_k r^{(k)}$, $\hat{\sigma}^2(s) = \frac{1}{K} \sum_k (r^{(k)} - \hat{\mu}(s))^2$.
4. Retain the state if $\hat{\sigma}^2(s) > 0$ AND $\hat{\mu}(s) < \lambda_{\text{diff}}$.

The result is $\mathcal{D}_{\text{pivot}} \leftarrow \mathcal{D}_{\text{adv}}$, the filtered pivot set.

What this profiling costs: Each candidate state requires $K$ single-turn model generations and $K$ environment interactions (for verifier scoring). For millions of candidates and $K$ in the tens or hundreds, this is substantial compute. However, it is a fixed, one-time cost that is amortized over all training steps. The paper does not include this cost in the training compute budgets (Figures 1 and the E2E comparisons), instead treating it as a preprocessing step. This is analogous to how SFT requires one-time dataset construction—except PivotRL's preprocessing involves model sampling rather than just file I/O. The paper acknowledges this implicitly by framing difficulty estimation as an area for future work.

Phase 3: RL training loops (online). For $k = 1, \dots, K_{\text{train}}$:

1. Sample a minibatch of $B$ states from $\mathcal{D}_{\text{pivot}}$. The states are drawn uniformly (or with some batch-composition strategy) from the pivot set.

2. Generate on-policy rollouts: For each state $s_b$ in the minibatch:

   • Sample $G$ actions from the current policy: $\{a_{b,i}\}_{i=1}^G \sim \pi_{\theta_{\text{old}}}(\cdot \mid s_b)$.
   • Execute each action in the environment to obtain its functional reward: $r_{b,i} = r_{\text{func}}(s_b, a_{b,i})$.

3. Compute advantages: For each state $s_b$, compute the group-normalized advantage for each of its $G$ actions:

   $\hat{A}_{b,i} = \frac{r_{b,i} - \frac{1}{G}\sum_{j=1}^G r_{b,j}}{\text{std}(\{r_{b,j}\}_{j=1}^G) + \epsilon_{\text{std}}}$

4. Update policy parameters: Compute the PivotRL objective from Eq. (7):

   $\mathcal{J}_{\text{PivotRL}}(\theta) = \frac{1}{B} \sum_{b=1}^B \left[ \frac{1}{G} \sum_{i=1}^G \min\left(w_{b,i}(\theta) \hat{A}_{b,i}, \ \text{clip}(w_{b,i}(\theta), 1-\epsilon, 1+\epsilon) \hat{A}_{b,i} \right) - \beta \ \text{KL}(\pi_\theta(\cdot \mid s_b) \ \| \ \pi_0(\cdot \mid s_b)) \right]$

   where $w_{b,i}(\theta) = \pi_\theta(a_{b,i} \mid s_b) / \pi_{\theta_{\text{old}}}(a_{b,i} \mid s_b)$.

   Take a gradient step: $\theta \leftarrow \theta - \eta \nabla_\theta \mathcal{J}_{\text{PivotRL}}(\theta)$.

5. Update the old policy: Periodically (or every step), set $\theta_{\text{old}} \leftarrow \theta$ so that the next batch uses the updated policy for sampling. The paper does not specify the exact frequency; in standard GRPO, $\theta_{\text{old}}$ is typically updated every step.

What happens to $\pi_0$ during training? It is never updated. The reference policy remains frozen throughout all $K_{\text{train}}$ steps. The KL penalty references this fixed target, so the policy is always pulled back toward its initialization. This is a critical design choice: if $\pi_0$ were updated, the KL penalty would drift, potentially allowing the policy to wander further and lose its OOD knowledge.

What prevents the policy from collapsing to the reference? The reward term in the objective pushes the policy away from $\pi_0$ toward higher-reward actions. The $\beta$ parameter trades off this push against the KL pull. If $\beta$ is too large, the KL penalty dominates and the policy barely changes. If $\beta$ is too small, the policy drifts far from $\pi_0$ and OOD degradation occurs. The paper uses the same $\beta$ as standard GRPO and does not report additional tuning.

Convergence and checkpoint selection: The paper selects the "best checkpoint over training" based on validation accuracy for the in-domain benchmarks (Table 6 note). For the SWE-Bench E2E comparison, PivotRL reaches peak accuracy of 32.67% at step 130 with ~133K cumulative rollout turns (Appendix A.2).

Cumulative rollout turn accounting: Each training step generates $B \times G$ single-turn rollouts. Over $K_{\text{train}}$ steps, the total rollout turns is $B \times G \times K_{\text{train}}$. For SWE-Bench PivotRL: batch size 1024 ($B=64$, $G=16$), 130 steps, total = $64 \times 16 \times 130 = 133,120$ rollout turns. For E2E RL: batch size 512 ($B=16$, $G=32$), each trajectory has 12–25 turns, total over 72 steps ≈ 542,000 rollout turns. The ~4× reduction comes from two factors: PivotRL uses single-turn rollouts (1 turn per sample vs. 12–25), and PivotRL reaches comparable accuracy in fewer total training samples.

---

Domain-Specific Instantiations

While PivotRL's core algorithm is domain-agnostic, the practical instantiation—what constitutes a turn, how data is generated, and how the verifier works—differs across the four agentic domains. These choices determine the method's applicability scope.

$\tau$2-Bench (Conversational Tool Use):

• Data: 281,774 trajectories across 838 domains, generated using a synthetic pipeline similar to ToolAce (Liu et al., 2025b), Kimi-K2 (Team, 2025b), and DeepSeek-V3.2 (DeepSeek-AI, 2025b).
• Action definition: The full assistant turn at the model-call boundary, which may contain natural language, tool calls, or both.
• Pivot candidates: Every assistant turn in every trajectory.
• Verifier: Output-schema validation, normalized string similarity, and equivalence-based LLM-as-judge scoring over the tool call and its immediate effect. The LLM judge compares the sampled tool call to the demonstrated call and determines functional equivalence.
• Training base model: Qwen3-30B-A3B-Thinking-2507.
• Key result: PivotRL achieves 63.81 vs. 58.44 for same-data SFT and 44.35 for the base model (Table 1).

SWE-Bench Verified (Software Engineering):

• Data: Internal trajectory dataset from OpenHands (Wang et al., 2025b), OpenCode (Anomaly, 2026), and Codex (OpenAI, 2026) on tasks from SWE-Gym (Pan et al., 2025) and R2E-Gym (Jain et al., 2025), using MiniMaxAI/MiniMax-M2.5. Final filtered pivot set: 87,718 samples.
• Action definition: The next assistant tool call in the coding trace (non-error tool-call actions only).
• Verifier: Tool-call name matching only. Does not score arguments or patch quality. The paper explicitly calls this "a deliberately coarse local signal" (Appendix A.2), because final task success depends only on the full SWE-Bench evaluation harness (which checks whether the final patch resolves the GitHub issue).
• Evaluation: mean@3 with the OpenHands harness.
• Key nuance: PivotRL achieves 32.67% on SWE-Bench Verified, which is competitive with E2E RL (also 32.67% at the matched point in Figure 1) but lower than SFT's 37.40%. The paper explains this as PivotRL being a compute-quality tradeoff: it achieves comparable accuracy to E2E RL (the stronger training paradigm) at much lower cost, but does not match SFT's peak performance on this particular benchmark. This is the only domain where SFT outperforms PivotRL in absolute terms (Table 1).
• E2E RL comparison details: PivotRL trains with batch size 1024 (64 prompts × 16 generations) at 1 turn/sample, reaching 32.67% at step 130 (~133K cumulative rollout turns). E2E RL trains with batch size 512 (16 prompts × 32 generations) at 12–25 turns/trajectory, reaching the same 32.67% at step ~72 (~542K cumulative rollout turns). Both use the same number of compute nodes.

Terminal-Bench (Terminal Control):

• Data: Resolved trajectories from Qwen/Qwen3-Coder-480B-A35B-Instruct and moonshotai/Kimi-K2-Instruct using Terminus-1 and Terminus-2 agents. Final dataset: approximately 20,000 samples after pivot filtering and command deduplication.
• Action definition: The next bash command produced at the model-call boundary.
• Pivot candidates: Every assistant bash action. Additional command-deduplication step applied to the low-reward-mean set to improve diversity.
• Verifier: Asks whether the sampled command is "locally interchangeable" with the demonstrated command. Combines output-schema validation, normalized string similarity, and equivalence-based LLM-as-judge scoring over the command and its immediate effect.
• Key result: PivotRL achieves 20.00 vs. 13.75 for SFT and 5.42 for base (Table 1). This domain shows the most severe SFT OOD degradation: after terminal-domain SFT training, AIME25 drops from 86.04 to 21.56 (Table 3).

BrowseComp (Web Browsing):

• Data: Multi-hop question-answer dataset with browsing trajectories generated using DeepSeek-V3.2 and an online search engine. Final dataset: 13,215 samples.
• Action definition: The next browsing step at a model-call boundary—issuing a search query, opening a result, or taking the next evidence-gathering action.
• Pivot candidates: Each search-related assistant step.
• Verifier: Not detailed in the paper at the level of the other domains, but follows the functional-equivalence pattern checking whether the browsing step is appropriate for the multi-hop QA task.
• Key result: PivotRL achieves 11.30 vs. 1.50 for SFT and 2.50 for base (Table 1). This is the largest relative gain over SFT (nearly 10×) and the only domain where SFT actually degrades from the base model's accuracy.

Summary of domain-specific patterns: Across all four domains, PivotRL outperforms SFT in three and ties or nearly ties in one (SWE-Bench, where SFT wins on absolute accuracy but PivotRL preserves OOD). The gain is largest where SFT performs worst relative to the base model (BrowseComp, Terminal-Bench), suggesting that PivotRL's advantage is most pronounced when the training data provides a weak behavior-cloning signal—exactly the regime where functional-equivalent reward and pivot selection matter most.

---

Design Choices and Their Justifications (Summary)

• Offline pivot filtering over uniform sampling: Uniform sampling wastes 71% of rollout budget on zero-advantage states. Filtering for mixed-outcome turns ensures every training state can produce nonzero gradients (Proposition 3.1), and Theorem 3.2 shows that higher-variance states produce proportionally stronger updates.

• Functional-equivalent reward over exact-match reward: Exact matching discards functionally correct actions (high miss rate), reducing effective sample size and corrupting the credit signal. Functional reward accepts any locally acceptable action. Theorem 3.3 proves that this choice, combined with KL regularization, shifts probability toward acceptable actions while preserving the reference policy's conditional distribution elsewhere, explaining OOD retention.

• Local (single-turn) rollout over full-trajectory rollout: Full trajectories are expensive ($T$ environment interactions per sample). Local rollout at pivot states requires only 1 interaction per sample, reducing total rollout turns by ~4× on SWE-Bench while achieving comparable accuracy.

• GRPO with KL penalty over unregularized policy gradient: The KL penalty prevents policy drift and—via Theorem 3.3—is the mechanism that preserves OOD performance. Without it, the policy would maximize reward without constraint, likely causing the same OOD degradation as SFT.

• Binary verifier reward over continuous reward: Binary rewards (0/1) simplify advantage computation and avoid calibration issues, at the cost of coarse signal. The theoretical results (Theorem 3.2, Theorem 3.3) are derived for binary rewards but generalize to continuous rewards with appropriate normalization.

• Turn-level granularity over token-level: Operating at model-call boundaries aligns with natural decision points in agentic interaction. Token-level credit would be finer-grained but would require more complex credit assignment and larger rollout budgets.

• Frozen reference policy over periodically updated reference: A moving reference policy would allow unbounded drift. The fixed $\pi_0$ ensures a stable KL penalty target and preserves the pivot profiling statistics (since pivots are profiled under $\pi_0$).

4. Key Insights and Innovations

Innovation 1: Reframing SFT-to-RL Conversion as a Data-Centric Filtering Problem Rather Than an Algorithmic One

The dominant approach to bridging SFT and RL in agentic domains has been to combine them sequentially—pretrain with behavior cloning, then fine-tune with on-policy RL (Uchendu et al., 2023; Hester et al., 2018)—or to modify the RL objective to incorporate demonstration data. These approaches treat the algorithm as the unit of innovation: design a better loss function, a better credit assignment scheme, a better mixing ratio between offline and online data.

PivotRL makes a fundamentally different move: it identifies the data selection problem as the bottleneck, not the optimization algorithm. The paper's diagnostic finding—that 71% of randomly sampled turns from SFT trajectories produce zero learning signal under GRPO—is the pivotal insight that reorients the entire problem. This is not an algorithmic failure (GRPO is correct; the gradient should be zero when all outcomes are identical). It is a data allocation failure: the training pipeline is spending compute on states that physics (Proposition 3.1) says cannot produce a gradient.

The field's default assumption has been that more interaction data is better—that the path from SFT to RL involves adding on-policy rollouts, period. PivotRL's reframing says: you already have enough data in your SFT trajectories; you just need to be selective about which states you use for RL. The innovation is not a new optimizer or a new reward function; it's the recognition that the SFT-to-RL bridge is primarily a filtering architecture: profile candidate states offline, retain only those where the reference policy exhibits mixed outcomes, and spend the entire RL budget there.

This is a conceptual shift from "RL needs environment interaction" to "RL needs informative decision points, and expert trajectories already contain those points if you know how to find them." It transforms SFT data from a passive resource (demonstrations to imitate) into an active resource (a map of the state space that tells you where the model is uncertain). The offline profiling step (computing reward variance under the reference policy) is essentially building a uncertainty map over the expert trajectory distribution—a diagnostic tool that no prior work in agentic RL had systematically deployed.

Significance beyond performance: This reframing has implications beyond the 4× reduction in rollout turns. It suggests that the bottleneck in agentic post-training is not the amount of environment interaction but the quality of state selection. A natural corollary is that improving pivot filtering—through better variance estimation, adaptive difficulty thresholds, or learned state-value functions—could yield further gains without any changes to the RL algorithm. This redirects research attention from optimizer design to data curation, a shift with parallels to the pretraining literature's recognition that data quality often matters more than model architecture.

Evidence: Table 4 shows that removing pivot filtering alone (keeping functional reward, switching to random turn selection $\mathcal{D}_{\text{cand}}$) drops $\tau$2-Bench accuracy from 63.81 to 59.68—a 4.13-point loss. Figure 3 shows that $\mathcal{D}_{\text{adv}}$ maintains substantially higher per-batch reward variance throughout training compared to random selection, directly validating the theoretical claim that pivots sustain informative gradient signals deeper into optimization. The monotonic improvement from random turn selection to variance-filtered to variance-plus-difficulty-filtered (Table 6: 59.68 → 63.81) demonstrates that filtering quality, not just filtering presence, drives gains.

---

Innovation 2: Establishing, with Theoretical Justification, That Functional Credit Assignment Preserves OOD Performance Through Block-Rescaling of the Action Distribution

The paper's second conceptual contribution is a precise characterization of why functional-equivalent reward combined with KL regularization preserves out-of-domain capabilities, while SFT destroys them. This is not merely an empirical finding (though the numbers are striking: +10.04% OOD retention advantage). It is a theoretical diagnosis that changes how we understand the relationship between reward design and generalization in LLM fine-tuning.

Prior work on OOD degradation in SFT (Chu et al., 2025; Luo et al., 2025b) had established the phenomenon—fine-tuning on narrow task distributions causes catastrophic forgetting—and prior work on RL (Chen et al., 2025; Shenfeld et al., 2025) had established that on-policy training mitigates it. But the mechanism was poorly understood. Why does RL preserve OOD capabilities while SFT destroys them? Is it the on-policy sampling? The KL penalty? The reward structure?

PivotRL's Theorem 3.3 isolates the answer. The optimal policy under functional-reward RL is a block-rescaled version of the reference policy: multiply all probabilities in the acceptable action set $\mathcal{M}(s)$ by $q_\beta(s)/\rho(s)$ (a factor $\geq 1$) and all probabilities in the complement by $(1-q_\beta(s))/(1-\rho(s))$ (a factor $\leq 1$). Within each block, the relative ordering is preserved exactly:

$\frac{\pi_\beta^*(a \mid s)}{\pi_\beta^*(b \mid s)} = \frac{\pi_0(a \mid s)}{\pi_0(b \mid s)}$

for any two actions both in $\mathcal{M}(s)$ or both outside it.

This is a structural guarantee that SFT cannot provide. SFT maximizes $\log \pi_\theta(a^*(s) \mid s)$, which pushes mass onto the single demonstrated action and away from everything else with no ordering constraint. The model can (and empirically does) completely reshuffle its relative preferences among task-unrelated actions—hence AIME25 dropping from 86.04 to 21.56 after terminal-domain SFT (Table 3). PivotRL's update, by contrast, is conservative: it increases the total budget allocated to acceptable actions while leaving the internal ranking of unacceptable actions untouched. If "solve a calculus problem" was ranked higher than "list directory contents" under the reference policy, it remains higher under the trained policy.

What makes this distinctive as an idea: It reframes OOD retention from an optimization artifact (something that happens to work better with RL) to a structural consequence of reward design. The key variable is not whether you use RL—it's whether your reward function defines a set of acceptable actions rather than a single target. Exact-match reward ($r_{\text{strict}}$) with KL regularization would not produce this block-rescaling property because the acceptable set is a singleton ($\mathcal{M}(s) = \{a^*(s)\}$), and Theorem 3.3's ordering preservation within the complement still holds, but the mass shift is onto a single point, effectively reducing to SFT-like behavior. The functional reward's expansion of $\mathcal{M}(s)$ is what creates room for the policy to improve on the task while maintaining its broader knowledge structure.

Significance beyond performance: This result provides a design principle for reward functions in LLM fine-tuning: define rewards over sets of acceptable outputs, not individual targets. It connects to broader themes in AI alignment—specifically, the idea that preserving a model's prior over irrelevant actions while steering it toward desirable ones is a form of minimal-intervention alignment that avoids unnecessary capability destruction. The theorem also implies that the KL penalty coefficient $\beta$ directly controls the OOD retention guarantee: larger $\beta$ means $q_\beta(s)$ is closer to $\rho(s)$, meaning less mass shift overall and stronger preservation of the reference policy. This gives practitioners a tunable knob for the OOD/in-domain tradeoff.

Evidence: Table 2 shows the aggregate effect: average OOD change of +0.21 for PivotRL vs. -9.83 for SFT, with no single benchmark dropping more than -3.12 under PivotRL vs. -64.48 for SFT (AIME25, terminal domain). The per-domain breakdown in Table 3 shows this pattern is consistent across all four training domains—PivotRL preserves OOD accuracy while SFT causes broad regression. The ablation in Table 4 demonstrates that removing functional reward (using $r_{\text{strict}}$ instead) drops accuracy from 63.81 to 57.34, providing direct evidence that the functional-verifier design, not just the pivot filtering, drives the gain over naive local RL.

---

Innovation 3: Diagnosing Verifier Over-Optimization Through the Lens of Group-Normalized Advantage Structure, and Pivot Filtering as a Mitigation

The paper makes a diagnostic contribution that connects reward variance at individual states to the fundamental structure of the GRPO gradient. This is not just a filtering heuristic—it is an analytic insight into a hidden failure mode in local RL from expert data.

The key observation (Proposition 3.1) is deceptively simple: if all actions sampled at a state receive identical rewards, the GRPO advantage is zero, and the gradient update vanishes. This is mechanically true by the definition of group normalization. But the paper does not stop at stating this fact. It uses it diagnostically: 71% of randomly sampled turns exhibit exactly this property, meaning the majority of a naive local RL training budget is spent on gradient-free updates. This explains why naive local RL performs no better than SFT despite the additional compute—it is literally not learning from most of its samples.

What makes this diagnosis innovative is that it reframes the "wasted compute" problem from one of insufficient exploration (the standard RL narrative) to one of task structure. The reason most turns produce zero-advantage groups is not that the policy isn't exploring enough—it's that at those states, the task is either trivially easy (the model always gets it right) or impossible (the model always gets it wrong) under the current policy. More exploration won't help at the easy states (the policy is already optimal there) or the impossible states (no amount of local sampling will produce a correct action). The problem is not "how do we explore better" but "how do we identify the states where exploration can actually produce a learning signal."

This connects to a broader pattern that the reference paper from the example also identified: verifier over-optimization. In the PaLM 2-S* paper, beam search over-optimized the PRM on easy problems, degrading performance. In PivotRL, the analogous phenomenon is subtler: uniform turn sampling over-optimizes the training distribution, spending compute on states where the policy already has deterministic behavior. The fix in both cases is not to change the optimizer but to change which states the optimizer sees. The compute-optimal test-time paper did this via difficulty-conditioned strategy selection; PivotRL does it via offline variance-based filtering.

What distinguishes this from standard RL exploration: Standard approaches to sparse-reward problems in RL involve curiosity-driven exploration (rewarding novel states), count-based bonuses, or entropy regularization. These all try to encourage the policy to visit more diverse states during rollouts. PivotRL does something different: it fixes the state distribution (to the expert trajectory distribution) and then filters for states that are inherently informative under the current policy. This is closer to active learning or uncertainty sampling than to exploration: you have a fixed pool of candidate states, and you select the subset where the model's behavior is most variable.

Significance: This diagnosis suggests that many RL pipelines for agentic tasks may be leaving performance on the table not because their algorithms are wrong but because their state sampling strategies are inefficient. The 71% figure is an empirical measurement for one model on one set of benchmarks, but the conceptual point—that group-normalized advantage with binary rewards produces zero signal at any state with deterministic outcomes—is universal. Any GRPO-based local RL pipeline that samples states uniformly from trajectories will waste budget on deterministic states unless it incorporates some form of variance-based filtering. This makes pivot selection a transferable architectural pattern, not a domain-specific trick.

Evidence: The ablation in Table 4 shows the direct effect: removing pivot filtering (using $\mathcal{D}_{\text{cand}}$ instead of $\mathcal{D}_{\text{adv}}$) drops $\tau$2-Bench accuracy from 63.81 to 59.68. Figure 3 visualizes the mechanism: under random sampling, per-batch reward standard deviation collapses quickly during training, indicating that most sampled states are becoming deterministic (all actions succeeding or all failing). Under pivot sampling ($\mathcal{D}_{\text{adv}}$), the reward standard deviation stays higher throughout training, meaning the model continues to encounter states where its actions have mixed outcomes and can therefore learn.

---

Innovation 4: Demonstrating That Single-Turn Local Rollouts at Expert-Intermediate States Can Replace Full-Trajectory RL, with an Explicit Compute-Quality Pareto Frontier

The paper's final conceptual contribution is an empirical finding with immediate practical stakes: local RL at carefully chosen pivot states achieves competitive accuracy with full-trajectory RL at a fraction of the environment interaction cost, establishing a new point on the compute-quality Pareto frontier.

This is not obvious a priori. The standard argument for full-trajectory RL is that it exposes the model to compounding errors—mistakes at early turns propagate to later states, and the model must learn to recover from its own distribution shift. Local RL at expert-intermediate states bypasses this: the model never sees the consequences of its own suboptimal actions because it's always conditioned on expert history. One might expect this to produce a "brittle" policy that performs well when the trajectory is on the expert path but collapses when it deviates—the classic DAgger motivation (Ross et al., 2011).

PivotRL's empirical result challenges this intuition, at least for the SWE-Bench domain. The model trained on single-turn rollouts from expert-intermediate states achieves 32.67% accuracy, matching E2E RL at the same accuracy, with ~4× fewer rollout turns and ~5.5× less wall-clock time (Figure 1). This suggests that for these tasks, the critical learning signal comes not from experiencing the consequences of mistakes over multiple turns but from learning to discriminate between good and bad actions at the specific decision points where the model is uncertain—exactly the pivot states.

What makes this a conceptual contribution rather than just a speedup: It changes our understanding of where the value of on-policy interaction comes from in agentic RL. The field has assumed that the primary benefit of E2E RL is the on-policy state distribution—that training on states the model actually visits (rather than expert states) is what prevents compounding errors and preserves OOD capabilities. PivotRL's results suggest an alternative: the benefit comes from on-policy action sampling at states where credit assignment is informative, and expert-intermediate states are a sufficient proxy for the on-policy state distribution if you select them carefully.

This has significant implications for how agentic RL systems are architected. If local RL at pivot states can match full-trajectory RL, then the expensive environment-simulation infrastructure needed for long-horizon rollouts can potentially be replaced with lighter-weight, single-turn execution environments. The compute savings compound: beyond the reduced rollout turns, you avoid the engineering complexity of managing multi-turn environment state, the latency of long trajectories, and the challenge of credit assignment over extended horizons.

Caveats and the nature of the contribution: The paper does not claim that local RL universally dominates full-trajectory RL. The SWE-Bench comparison is a single-domain result, and SFT actually achieves higher absolute accuracy (37.40) than both PivotRL (32.67) and the E2E RL baseline at the matched compute point. The innovation is not "PivotRL beats everything" but "local RL occupies a previously unexplored point on the compute-quality Pareto frontier: competitive with E2E RL at much lower cost, and competitive with SFT on OOD retention." This is a characterization of a tradeoff space, not a claim of dominance.

The explicit Pareto framing—plotting accuracy against cumulative rollout turns and wall-clock time (Figure 1)—is methodologically significant. It provides a template for future comparisons of agentic training methods: report not just final accuracy but the full cost-accuracy curve, so practitioners can make informed decisions based on their compute budgets.

Evidence: Figure 1 plots the tradeoff directly. At the matched accuracy of 32.67%, PivotRL requires ~133K rollout turns vs. ~542K for E2E RL. Table 5 shows that PivotRL operates at production scale in Nemotron-3-Super, where it improves agentic benchmarks substantially over the SFT baseline (e.g., SWE-Bench from 12.87 to 61.33) while operating alongside other RL environments for reasoning and chat—a practical validation of the Pareto frontier concept in a large-scale deployment.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper uses four agentic benchmarks, each corresponding to one training domain: τ2-Bench (Barres et al., 2025) for conversational tool use, SWE-Bench Verified (Jimenez et al., 2024) for software engineering, Terminal-Bench (Team, 2025c) for terminal control, and BrowseComp (Wei et al., 2025) for web browsing. Additionally, eight out-of-domain benchmarks are used for OOD evaluation: IFBench, AIME25, MATH500, LiveCodeBench, Scicode, MMLU-Pro, MMLU-ProX, and WMT24++. Training data consists of expert trajectories generated via synthetic pipelines, stronger models, or human demonstrations (domain-specific details in Appendix A.2), with dataset sizes ranging from approximately 13,000 samples (BrowseComp) to 281,774 trajectories (τ2-Bench). The SWE-Bench training set uses an internal trajectory dataset from OpenHands, OpenCode, and Codex on tasks from SWE-Gym and R2E-Gym, filtered to 87,718 pivot samples.

• Base model(s). All experiments start from Qwen3-30B-A3B-Thinking-2507, a 30B-parameter mixture-of-experts model with 3B active parameters. The paper states this model is representative of contemporary open-weight LLM capabilities. For the Nemotron-3-Super production deployment (Table 5), the base model is NVIDIA's in-house architecture, with PivotRL operating as one stage in a multi-stage post-training pipeline that includes RL environments for reasoning and chat alongside the agentic PivotRL environments.

• Metrics. The primary metric is accuracy (percentage points) — the fraction of evaluation instances where the model's final answer or trajectory outcome is correct according to the benchmark's evaluation harness. For in-domain tasks, accuracy is computed per-benchmark and averaged across domains. For OOD tasks, the metric is change (Δ) relative to the base model — the difference between the post-trained model's accuracy and the base model's accuracy on each OOD benchmark, with negative values indicating degradation. For the SWE-Bench E2E RL comparison (Figure 1), the metric is accuracy plotted against cumulative rollout turns and cumulative wall-clock time during training, enabling a compute-quality Pareto analysis. SWE-Bench specifically uses mean@3 evaluation with the OpenHands harness.

• Baselines. The paper uses four baselines across different experiments: (1) Base model (Qwen3-30B-A3B-Thinking-2507) — the pretrained model without any agentic post-training, evaluated zero-shot or few-shot depending on the benchmark; (2) Same-data SFT — standard supervised fine-tuning on the identical expert trajectories used for PivotRL training, with the same base model, prompts, and data (Section 4: "For every SFT–PivotRL comparison, the base model, prompts, and expert trajectories are identical"); (3) E2E RL — end-to-end reinforcement learning with full-trajectory on-policy rollouts using GRPO, trained with a batch size of 512 (16 prompts × 32 generations) at 12–25 turns per trajectory on SWE-Bench (Appendix A.2); (4) Naive local RL — the baseline described in Section 2 that samples states uniformly from SFT trajectories and uses exact-match reward $r_{\text{strict}}(s, a) = \mathbf{1}[a = a^*(s)]$, evaluated in the ablation study (Table 4). This last baseline specifically isolates the effect of PivotRL's two mechanisms: removing pivot filtering and functional reward reduces to naive local RL.

• Generation budget / compute accounting. Test-time compute during evaluation is not a variable grid-sweep parameter in this paper (unlike the best-of-N scaling studies); instead, evaluation uses a fixed generation strategy per benchmark (e.g., mean@3 for SWE-Bench, single-pass for other benchmarks). The critical compute accounting is for training cost, measured in two ways: (1) Cumulative rollout turns — the total number of environment interactions during training, where one "turn" is one model action generation and execution. For PivotRL, each training sample is a single-turn rollout, so total turns = number of training samples × $G$ (group size). For E2E RL, total turns = sum of trajectory lengths across all training episodes. (2) Cumulative wall-clock time — measured on the same number of compute nodes. For the SWE-Bench comparison (Appendix A.2), PivotRL uses batch size 1024 (64 prompts × 16 generations) at 1 turn/sample, reaching ~133K cumulative rollout turns; E2E RL uses batch size 512 (16 prompts × 32 generations) at 12–25 turns/trajectory, reaching ~542K cumulative rollout turns at the matched accuracy point. Both methods use the same number of compute nodes. The paper also reports production-scale results (Table 5) where PivotRL is one stage in a larger pipeline, but does not break down the per-stage compute budget for that deployment.

• Cross-validation / statistical protocol. The paper does not report standard cross-validation, confidence intervals, or error bars on any experimental results. The four training runs are single-domain (each model is trained on one agentic domain and evaluated on that domain plus the eight OOD benchmarks). For the SWE-Bench E2E comparison, the matched accuracy point (32.67%) is the peak accuracy achieved by PivotRL (step 130) and a specific checkpoint of E2E RL (step ~72) — it is not an average over seeds or splits. The ablation results (Table 4, Table 6) report single-run best-checkpoint accuracy. The production results (Table 5) report accuracy before and after the RL stage that includes PivotRL, again as single-point measurements without statistical characterization. This absence of statistical rigor is a meaningful limitation.

Main Quantitative Results

In-Domain Accuracy Across Four Agentic Domains (Table 1)

PivotRL improves over the base model by an average of +14.11 percentage points across the four domains, compared to +9.94 for same-data SFT — a +4.17 advantage for PivotRL over SFT. The domain-specific breakdown (Table 1):

• τ2-Bench: Base = 44.35, SFT = 58.44 (+14.09), PivotRL = 63.81 (+19.46). PivotRL outperforms SFT by +5.37.
• SWE-Bench Verified: Base = 19.07, SFT = 37.40 (+18.33), PivotRL = 32.67 (+13.60). SFT outperforms PivotRL by +4.73 — this is the only domain where SFT achieves higher absolute accuracy. The paper attributes this to the deliberately coarse verifier used for SWE-Bench (tool-call name matching only, no argument or patch-quality scoring), which limits the per-step RL signal quality compared to the full-trajectory behavior cloning signal of SFT.
• Terminal-Bench: Base = 5.42, SFT = 13.75 (+8.33), PivotRL = 20.00 (+14.58). PivotRL outperforms SFT by +6.25.
• BrowseComp: Base = 2.50, SFT = 1.50 (-1.00), PivotRL = 11.30 (+8.80). PivotRL outperforms SFT by +9.80. Notably, SFT actually degrades from the base model on this benchmark — the base model achieves 2.50 while SFT achieves 1.50, a -1.00 regression — making PivotRL's gain of +8.80 over base particularly dramatic.

The overall pattern: PivotRL outperforms SFT when SFT provides weak signal (BrowseComp, where SFT degrades from base) or moderate signal (Terminal-Bench, τ2-Bench), but underperforms SFT when SFT's behavior-cloning signal is strong (SWE-Bench, where SFT nearly doubles the base model's accuracy). This is consistent with the theoretical framework: PivotRL's functional reward thrives when exact-match imitation would be overly restrictive (BrowseComp's diverse search strategies) but provides weaker per-step signal when the demonstration actions are already near-optimal and the verifier is coarse (SWE-Bench's tool-call name matching).

Out-of-Domain Retention Across Eight Benchmarks (Table 2, Table 3)

The aggregated OOD results (Table 2) represent the paper's strongest empirical finding. Averaged across the four training runs:

• Base model average OOD accuracy: 66.62
• SFT average change (Δ): -9.83
• PivotRL average change (Δ): +0.21

PivotRL maintains essentially zero OOD degradation on average, while SFT causes nearly 10 percentage points of regression. The per-benchmark breakdown reveals the severity of SFT's damage:

• IFBench: SFT Δ = -11.46, PivotRL Δ = +0.82
• AIME25: SFT Δ = -19.72, PivotRL Δ = -1.20
• MATH500: SFT Δ = -8.51, PivotRL Δ = +0.35
• LiveCodeBench: SFT Δ = -7.76, PivotRL Δ = -0.17
• Scicode: SFT Δ = -11.39, PivotRL Δ = +1.90
• MMLU-Pro: SFT Δ = -2.99, PivotRL Δ = +0.31
• MMLU-ProX: SFT Δ = -7.16, PivotRL Δ = +0.12
• WMT24++: SFT Δ = -9.61, PivotRL Δ = -0.49

No single benchmark drops more than -3.12 under PivotRL (AIME25, terminal domain training), compared to SFT's worst regression of -64.48 (AIME25, also terminal domain). PivotRL shows positive OOD change on five of eight benchmarks (+0.82, +0.35, +1.90, +0.31, +0.12), suggesting it may provide marginal generalization benefits rather than merely avoiding degradation.

The per-domain OOD breakdown (Table 3) reveals that the domain of training strongly affects the severity of SFT's OOD regression:

• τ2-Bench training: SFT causes moderate regression across OOD benchmarks (worst: AIME25 at -9.79). PivotRL maintains near-zero change across all eight benchmarks (range: -0.43 to +1.85).
• SWE-Bench training: SFT causes more severe regression (LiveCodeBench at -9.25, IFBench at -13.24). PivotRL shows small positive changes on most benchmarks (+0.23 to +3.04), with only AIME25 showing -1.04.
• Terminal-Bench training: SFT causes catastrophic regression — AIME25 drops from 86.04 to 21.56 (-64.48), MATH500 from 98.05 to 63.55 (-34.50), WMT24++ from 36.97 to 6.31 (-30.66), Scicode from 36.83 to 9.39 (-27.44). PivotRL maintains near-baseline performance, with the worst drop being AIME25 at -3.12.
• BrowseComp training: SFT causes moderate regression. PivotRL maintains near-baseline performance across all benchmarks.

This domain-dependent severity of SFT's OOD destruction is notable: terminal-domain SFT is far more destructive than conversational-tool-use SFT, despite both using similar training data volumes (281,774 trajectories for τ2-Bench vs. ~20,000 samples for Terminal-Bench). The paper does not investigate why, but a plausible hypothesis is that terminal commands constitute a highly specialized action space (bash syntax, specific command patterns) that is very distant from the base model's general text distribution, so SFT overfitting to this narrow domain displaces a larger fraction of the model's general knowledge.

SWE-Bench Compute-Quality Comparison (Figure 1)

Figure 1 plots SWE-Bench Verified accuracy against cumulative rollout turns (Figure 1a) and cumulative wall-clock time (Figure 1b), comparing PivotRL and E2E RL starting from the same base model:

• Accuracy-matched comparison: Both methods reach 32.67% accuracy. PivotRL requires ~133K cumulative rollout turns; E2E RL requires ~542K — a ~4.1× reduction. In wall-clock time on the same number of compute nodes, PivotRL requires ~5.5× less time.
• Training dynamics: PivotRL reaches its peak accuracy (32.67%) at step 130. The E2E RL curve is not fully shown beyond the matched point, but the paper reports that E2E RL reaches the same accuracy at step ~72. The trajectory shapes differ meaningfully: PivotRL shows a rapid initial accuracy increase (the curve rises steeply in the first ~50K rollout turns) followed by gradual improvement; E2E RL shows slower initial progress (accuracy is lower at equivalent rollout-turn counts) but continues improving with more turns.
• Wall-clock advantage (5.5×) is larger than rollout-turn advantage (4×): This gap arises because each PivotRL rollout turn is a single model generation and single environment interaction, while each E2E RL rollout turn is one step in a multi-turn trajectory that requires maintaining environment state across turns. The 5.5× wall-clock reduction factors in not just the reduced number of turns but also the reduced per-turn overhead of managing long-horizon environment interactions.
• Limitation of this comparison: SFT achieves 37.40% on SWE-Bench — higher than both PivotRL (32.67%) and the E2E RL baseline at the matched compute point. The paper's narrative is that PivotRL achieves competitive accuracy with E2E RL at lower cost, not that it achieves state-of-the-art accuracy. The Pareto framing acknowledges that SFT occupies a different point on the frontier (higher accuracy at lower compute cost, but catastrophic OOD degradation). A complete comparison would plot all three methods on an accuracy-vs-OOD-retention plane, but the paper does not provide this.

Production-Scale Deployment (Table 5)

Table 5 reports agentic benchmark accuracy during the Nemotron-3-Super post-training pipeline, where PivotRL covers the agentic environments while other RL environments handle reasoning and chat:

• τ2-Bench: 48.00 (after SFT) → 64.00 (after PivotRL stage)
• SWE-Bench Verified: 12.87 → 61.33
• Terminal-Bench 1.1 Core: 23.33 → 34.17
• BrowseComp: 13.03 → 25.04

These gains are substantially larger than the single-domain experiments in Table 1, but the comparison is not controlled: Nemotron-3-Super's RL stage includes PivotRL environments and other RL environments (reasoning, chat) simultaneously, making it impossible to attribute the full improvement to PivotRL alone. The paper presents Table 5 as evidence of production viability rather than a controlled evaluation of PivotRL's contribution. The OOD impact of this combined RL stage is not reported.

Ablation Studies and Robustness Checks

Pivot filtering ablation (Table 4): Removing pivot filtering and using all candidate turns (labeled "$-\text{Pivot filtering}$" in Table 4, equivalent to $\mathcal{D}_{\text{cand}}$ + functional reward) reduces τ2-Bench accuracy from 63.81 to 59.68. This 4.13-point drop validates that spending rollout budget on informative states rather than all states is a significant contributor to PivotRL's gain. The filtered configuration ($\mathcal{D}_{\text{adv}}$) retains only mixed-outcome, low-reward-mean states; the unfiltered configuration ($\mathcal{D}_{\text{cand}}$) includes states where the model is uniformly correct or uniformly incorrect, which produce zero-advantage groups and waste compute (Proposition 3.1).

Functional reward ablation (Table 4): Removing functional reward and using strict exact-match reward (labeled "$-\text{Functional reward}$" in Table 4, equivalent to $\mathcal{D}_{\text{cand}}$ + strict reward) reduces τ2-Bench accuracy to 57.34. This is slightly below same-data SFT at 58.44, confirming that naive local RL with exact-match credit not only fails to improve over SFT but can underperform it. The 57.34 vs. 63.81 gap (6.47 points when moving from strict reward + all turns to functional reward + pivot turns) demonstrates that both mechanisms are necessary for the full gain.

Turn selection granularity on τ2-Bench (Table 6): The paper compares three pivot selection strategies on τ2-Bench:

• Same-data SFT: 58.44
• Random pivots ($\mathcal{D}_{\text{cand}}$): 59.68 — already improves over SFT (+1.24), showing that local on-policy RL with functional reward provides benefit even without filtering, though the gain is modest.
• Low-reward-mean pivots ($\mathcal{D}_{\text{adv}}$): 63.81 — the full PivotRL filtering improves by an additional +4.13 over random pivots.

The monotonic improvement (58.44 → 59.68 → 63.81) demonstrates that more selective filtering yields larger gains, consistent with the theoretical prediction that higher-variance states produce stronger gradient signals (Theorem 3.2). The per-subdomain breakdown shows the pattern holds across τ2-Airline, τ2-Retail, and τ2-Telecom, with low-reward-mean pivots consistently outperforming random pivots.

Training dynamics: reward variance over time (Figure 3): Figure 3 plots per-batch reward standard deviation (the dispersion of verifier scores within each batch) throughout RL training for the two pivot selection strategies. Under random sampling ($\mathcal{D}_{\text{cand}}$), the reward standard deviation collapses quickly — starting around 0.48 and dropping to approximately 0.35 by step 100, then continuing to decline. Under pivot sampling ($\mathcal{D}_{\text{adv}}$), the reward standard deviation starts higher (around 0.50) and declines more slowly, remaining above the random-sampling curve throughout training. This directly validates the design hypothesis: pivot filtering maintains higher-variance rollout groups, which produce larger group-normalized advantages and therefore stronger per-sample gradient updates (Theorem 3.2). The collapse under random sampling explains why naive local RL plateaus: as training progresses and the policy improves, an increasing fraction of randomly sampled states become deterministic (uniformly solved), producing zero-advantage groups and contributing nothing to the gradient. Pivot states, by construction, remain mixed-outcome longer because they are selected from regions where the policy was uncertain.

Training dynamics: accuracy over time (Figure 2): Figure 2 shows τ2-Bench validation accuracy throughout training. The $\mathcal{D}_{\text{adv}}$ pivot set yields the highest terminal accuracy and the steepest optimization trajectory, outperforming both random sampling and same-data SFT. The SFT baseline is plotted as a horizontal reference line (58.44), showing that PivotRL with $\mathcal{D}_{\text{adv}}$ crosses above SFT accuracy relatively early in training and continues improving, while random sampling converges to a level only slightly above SFT (59.68).

Scalability to larger models (Table 5): The production-scale results with Nemotron-3-Super demonstrate that PivotRL scales to much larger models (120B total parameters, 12B active) and operates effectively as part of a multi-stage post-training pipeline. However, the lack of a controlled comparison (PivotRL alone vs. SFT alone on Nemotron-3-Super) limits the robustness of this evidence. The gains in Table 5 (+16 on τ2-Bench, +48.46 on SWE-Bench) are substantially larger than the single-domain gains in Table 1, suggesting that either (a) PivotRL's benefits compound when combined with reasoning and chat RL, (b) the Nemotron-3-Super base model has different properties than Qwen3-30B, or (c) the additional RL environments contribute significantly to the agentic benchmark improvements.

Domain-specific filter design (Appendix A.2): The Terminal-Bench domain applies an additional command-deduplication step beyond the standard variance-and-difficulty filtering, "to improve diversity" in the final dataset of approximately 20,000 samples. The paper does not ablate this deduplication step, so its contribution relative to the standard pivot filtering is unknown.

Critical Assessment

Claim: PivotRL outperforms same-data SFT in in-domain accuracy by +4.17% on average

What was tested and what it demonstrates. The paper trains four separate models, each on one agentic domain, and compares PivotRL against SFT using identical base models and training trajectories. The +4.17 average advantage (Table 1) is genuine: PivotRL wins on three domains (τ2-Bench: +5.37, Terminal-Bench: +6.25, BrowseComp: +9.80) and loses on one (SWE-Bench: -4.73). However, the "average" framing obscures important heterogeneity. SWE-Bench is arguably the most prominent and rigorous agentic benchmark in the set — it is the standard evaluation for coding agents and the domain where E2E RL is the default training paradigm. PivotRL's underperformance on SWE-Bench (-4.73 vs. SFT, Table 1) is the paper's most significant negative result and is not adequately explained. The paper attributes it to the "deliberately coarse local signal" of the SWE-Bench verifier (tool-call name matching only), but this raises a question about the method's generality: if PivotRL requires a domain-specific verifier that can discriminate good from bad actions at each turn, and if such verifiers are hard to build for complex tasks (where tool-call names are insufficient), then the method's applicability is bounded by verifier quality in ways the paper does not systematically explore.

What weakens the claim. The SWE-Bench result undermines the universalizability of the +4.17 figure. If we weight domains by their prominence and difficulty, the picture shifts: on the hardest and most widely recognized benchmark, PivotRL loses to SFT. The paper frames this as a compute-quality tradeoff (PivotRL beats E2E RL on cost but loses to SFT on accuracy), but the abstract and introduction do not highlight this nuance, instead reporting the averaged +4.17 gain. A reader scanning only the introduction would not learn that SFT outperforms PivotRL on the benchmark that matters most to the agentic coding community.

Additional concern: single training run per domain. Each domain is trained once with PivotRL and once with SFT. There are no repeated runs, no error bars, and no significance tests. The BrowseComp delta (+9.80, from 1.50 SFT to 11.30 PivotRL) is the largest relative gain, but also the benchmark with the most extreme SFT degradation (SFT drops below the base model's 2.50). This combination of tiny base accuracy and SFT collapse means small absolute differences in the number of correctly solved instances can produce large percentage-point swings — statistical noise cannot be ruled out without uncertainty quantification.

Claim: PivotRL achieves +10.04% higher OOD accuracy than same-data SFT

What was tested and what it demonstrates. This is the paper's most robust finding. The OOD evaluation is comprehensive — eight benchmarks spanning math (AIME25, MATH500), science (Scicode), general knowledge (MMLU-Pro, MMLU-ProX), competitive coding (LiveCodeBench), instruction following (IFBench), and translation (WMT24++) — and the pattern is consistent across all four training domains. PivotRL's OOD retention is essentially perfect, with average change hovering near zero (+0.21, Table 2) and no single benchmark dropping more than -3.12 (AIME25, terminal domain). SFT's catastrophic forgetting is dramatic and consistent, with average -9.83 and worst-case -64.48 (AIME25, terminal domain, Table 3). The magnitudes make statistical significance concerns largely moot: PivotRL doesn't just "retain better" — it retains everything, at levels indistinguishable from the base model.

What weakens the claim. Two caveats. First, the paper does not report OOD performance for the E2E RL baseline — the comparison is PivotRL vs. SFT, not PivotRL vs. E2E RL on OOD retention. The paper's positioning (Section 1) claims to combine "the data efficiency of SFT with the generalization capabilities of E2E RL," but the OOD experiment only tests the SFT half of this comparison. Without a head-to-head OOD evaluation against E2E RL, the claim that PivotRL matches E2E RL's OOD retention is an extrapolation from Theorem 3.3 and the behavioral pattern, not an empirical demonstration. Second, the OOD evaluation is limited to reasoning, knowledge, and translation benchmarks — it does not test retention of safety alignment, instruction-following nuance, or other OOD capabilities that matter for deployment but are harder to benchmark. The claim of "OOD retention" is therefore scoped to the eight tested benchmarks, which are predominantly academic reasoning and knowledge tasks.

Claim: PivotRL achieves competitive accuracy with E2E RL while requiring 4× fewer rollout turns

What was tested and what it demonstrates. The SWE-Bench comparison (Figure 1) shows that at the matched accuracy point of 32.67%, PivotRL requires ~133K rollout turns versus ~542K for E2E RL. The cost reduction is real and well-measured: cumulative rollout turns are a clean, implementation-independent metric, and the wall-clock comparison (5.5× reduction) accounts for per-turn overhead differences. The finding that local RL at pivot states can match full-trajectory RL at lower cost challenges the assumption that E2E interaction is necessary for agentic RL.

What weakens the claim. The claim of "competitive accuracy" must be qualified: 32.67% is the accuracy of both methods at the matched point, but it is lower than SFT's 37.40% (Table 1) and lower than state-of-the-art SWE-Bench results from larger models trained with more extensive RL (not reported in the paper, but well-known in the field). PivotRL's advantage over E2E RL is purely on cost at equivalent (moderate) accuracy — it does not demonstrate that PivotRL can surpass E2E RL's peak accuracy with more compute. The paper does not push either method to saturation; the E2E RL curve could continue improving beyond 32.67% with additional training, and the paper does not show the full E2E RL scaling curve. The comparison is at a single matched point rather than a full scaling analysis (unlike the PaLM 2-S* test-time compute paper, which computes optimal scaling across many budget levels). This makes it a weaker demonstration of the Pareto frontier than it could be.

Additionally, the E2E RL baseline might not be optimally tuned — batch size 512 (16 prompts × 32 generations) with 12–25 turns per trajectory is one specific configuration. The paper does not report hyperparameter sweeps for either method. PivotRL's 4× advantage could shrink or grow under different E2E RL configurations.

Claim: PivotRL's design is theoretically grounded, with Theorem 3.2 explaining pivot selection and Theorem 3.3 explaining OOD retention

What was tested and what it demonstrates. The theoretical results (Section 3.2) provide clean, interpretable explanations for why reward variance matters (Theorem 3.2: natural gradient norm equals reward standard deviation) and why functional reward preserves OOD performance (Theorem 3.3: block-rescaling preserves relative ordering within action subsets). These are genuine contributions to understanding the mechanisms at play.

What weakens the claim. The theory is not directly tested — the paper does not verify that the empirical policy updates follow the block-rescaling pattern predicted by Theorem 3.3, nor does it measure whether the natural gradient norm tracks with reward variance during GRPO training (as Theorem 3.2 would predict for the idealized KL path). The theorems analyze the minimizer of a regularized objective (Eq. 11) and the population GRPO score along an exponential-tilt path, neither of which is exactly what PivotRL's finite-sample, clipped-GRPO updates implement. The connection between theory and practice is qualitative and analogical, not quantitative and verified. This is common in ML papers, but the paper's presentation ("we substantiate our methodology with a lightweight theoretical analysis") does not overclaim the theory's precision. The ablation results (Table 4) provide behavioral validation — removing pivot filtering hurts, removing functional reward hurts more — but do not test the specific theoretical mechanisms.

Missing experiments that would strengthen the paper

1. PivotRL vs. E2E RL on OOD retention. The paper claims to match E2E RL's generalization but never demonstrates this empirically. A direct OOD comparison between PivotRL and E2E RL (both trained on SWE-Bench to equivalent in-domain accuracy) would close the most significant gap between the claims and the evidence.

2. Difficulty estimation cost amortization analysis. The offline pivot profiling step (sampling $K$ actions per candidate state, scoring with the verifier) is computationally expensive — for millions of candidate states and $K$ in the tens or hundreds, this preprocessing could rival the cost of the RL training itself. The paper does not report profiling cost or include it in any compute budget. A practical deployment analysis would amortize this cost over the number of training steps (since profiling is done once and reused) and compare total cost (profiling + training) against E2E RL's cost. Without this, the 4× rollout-turn reduction is an upper bound, not a realized deployment gain.

3. Sensitivity to $\lambda_{\text{diff}}$ and $K$ (profiling hyperparameters). The paper does not sweep the difficulty threshold $\lambda_{\text{diff}}$ (which controls how many low-reward-mean states are retained) or the profiling sample count $K$ (which controls the accuracy of the variance estimates). The choice of $\mathcal{D}_{\text{adv}}$ over $\mathcal{D}_{\text{cand}}$ matters (Table 6), but whether further gains are possible with different thresholds or whether the current choice is near-optimal is unknown.

4. PivotRL with a stronger SWE-Bench verifier. The paper attributes PivotRL's SWE-Bench underperformance to the coarse verifier (tool-call name matching only). An experiment with a richer verifier (e.g., argument-matching, LLM-as-judge over tool-call quality) would test whether PivotRL can close the gap with SFT on SWE-Bench. If a better verifier enables PivotRL to match or exceed SFT's 37.40% while retaining OOD performance, that would significantly strengthen the method's case for complex coding tasks.

5. Multi-domain combined training. All experiments train on a single agentic domain and evaluate OOD on non-agentic benchmarks. A natural deployment scenario is training on multiple agentic domains simultaneously. Does PivotRL's OOD retention hold when the model is trained on τ2-Bench + SWE-Bench + Terminal-Bench + BrowseComp simultaneously? Does SFT's catastrophic forgetting compound across domains? The single-domain experiments demonstrate the mechanism, but the production deployment (Table 5, Nemotron-3-Super) uses multi-domain training without a published OOD evaluation, leaving the practical regime unexplored.

6. Comparison to other SFT-to-RL bridge methods. The paper's only comparison to prior SFT-to-RL hybrid methods is the naive local RL baseline ($r_{\text{strict}}$, no filtering, Table 4). It does not implement or compare against Uchendu et al. (2023), Hester et al. (2018), Setlur et al. (2026), or Ming et al. (2026) — the methods it cites as related work (Section 5.2). Without these comparisons, it is unclear whether PivotRL's gains come from the pivot filtering idea specifically or from the general benefit of on-policy local RL (which prior methods also provide in different forms).

Summary of evidential strength

• In-domain accuracy advantage over SFT: Supported with qualifications. The average +4.17 figure is genuine but domain-dependent; PivotRL underperforms SFT on SWE-Bench, the most prominent benchmark. Statistical uncertainty is unreported.
• OOD retention advantage over SFT: Strongly supported. The effect is large, consistent across four training domains and eight OOD benchmarks, and the magnitudes (SFT: -9.83 average, -64.48 worst; PivotRL: +0.21 average, -3.12 worst) are statistically unambiguous even without formal significance tests.
• Compute reduction vs. E2E RL: Supported at a single accuracy point. The ~4× rollout-turn reduction at 32.67% SWE-Bench accuracy is well-measured, but the comparison is at one matched point rather than across a scaling sweep, and PivotRL's peak accuracy lags behind SFT on this benchmark.
• Theoretical justification: Provides qualitative insight consistent with empirical patterns, but is not directly validated through quantitative mechanism experiments.
• Production viability: Demonstrated in a large-scale deployment (Nemotron-3-Super, Table 5), but without controlled comparison to isolate PivotRL's contribution from the other RL environments in the pipeline.

6. Limitations and Trade-offs

1. Offline Pivot Profiling Cost Is Unaccounted For in Headline Compute Savings

The assumption or constraint. PivotRL requires an offline profiling step before RL training begins: every candidate turn from the SFT trajectories is sampled $K$ times under the frozen reference policy $\pi_0$, each sampled action is executed in the environment and scored with the verifier, and the empirical reward mean and variance are computed (Section 3.1, Eq. 4). Only turns passing the variance and difficulty filters ($\hat{\sigma}^2(s) > 0$, $\hat{\mu}(s) < \lambda_{\text{diff}}$) are retained for training. The paper explicitly acknowledges that this cost is excluded from the headline comparisons:

"our experiments do not account for this cost largely for simplicity" (Section 3.2)

The consequence. For large SFT datasets, this profiling cost can be enormous. In the $\tau$2-Bench domain, the paper uses 281,774 trajectories (Appendix A.2). At a conservative estimate of 10 turns per trajectory, that yields ~2.8 million candidate states. If $K = 100$ (a plausible number for reliable variance estimation, though the paper does not specify $K$), profiling requires ~280 million single-turn model generations and environment interactions — likely exceeding the cost of the RL training itself. For the SWE-Bench domain (87,718 pivot samples after filtering, Appendix A.2), the number of candidate states before filtering is presumably much larger. The profiling cost is a fixed up-front investment that is amortized over training steps, but it is not amortized in any of the paper's cost calculations. The ~4× rollout-turn reduction relative to E2E RL (Figure 1, ~133K vs. ~542K turns) omits the profiling turns entirely. If profiling requires, say, 500K rollouts across all candidate states, the true total compute advantage shrinks or disappears.

Furthermore, profiling must be redone whenever the reference policy changes — for example, if the base model is updated, if the SFT initialization changes, or if the verifier is refined. This makes PivotRL less attractive for rapid experimentation cycles where models and data evolve frequently.

What evidence exists in the paper. The paper provides no measurement of profiling cost. Neither $K$ (the number of profiling samples per candidate state) nor the total profiling rollout turns are reported. The SWE-Bench comparison (Figure 1) plots cumulative rollout turns during RL training only, starting from the filtered pivot set. The paper does not state what fraction of candidate states survive filtering in each domain (only the final pivot set sizes: 87,718 for SWE-Bench, ~20,000 for Terminal-Bench). The gap between candidate set size and pivot set size is unknown, making it impossible to estimate profiling cost from the reported numbers.

Mitigation status. The paper does not attempt to reduce profiling cost through more efficient estimation (e.g., using learned difficulty predictors, adaptive sampling, or smaller $K$). It flags this as future work in Section 3.2: "estimating difficulty in this way still incurs additional computation cost during inference... we leave further optimizations of this step as an important avenue for future work." In practice, the profiling step represents an exploration-exploitation tradeoff: compute spent assessing state informativeness versus compute spent on RL training. The paper does not explore this tradeoff or provide guidance on choosing $K$ or $\lambda_{\text{diff}}$.

---

2. PivotRL Requires Per-Domain Verifier Engineering That Is Not Validated for Complex Tasks

The assumption or constraint. PivotRL's functional reward $r_{\text{func}}(s, a) = \mathbf{1}[a \in \mathcal{M}(s)]$ depends entirely on the quality of a domain-specific verifier that defines the set of locally acceptable actions $\mathcal{M}(s)$. The verifier is a hand-crafted programmatic function (Section 3.1, Eq. 6), not a learned model. For SWE-Bench, the verifier is deliberately coarse: it matches tool-call names only, ignoring arguments and patch quality (Appendix A.2). For $\tau$2-Bench, it uses schema validation and LLM-as-judge scoring. For Terminal-Bench, it uses command interchangeability checks. Each domain requires a custom verifier implementation — there is no general recipe beyond "use domain knowledge."

The consequence. PivotRL's performance is upper-bounded by its verifier's accuracy. On SWE-Bench, where the verifier only checks tool-call names, PivotRL achieves 32.67% versus 37.40% for SFT (Table 1) — a 4.73-point deficit. The paper attributes this to the verifier's coarseness, but this is the central tension: a more precise verifier would require more engineering effort to build, while a coarse verifier limits the RL signal quality. The method offers no guidance on how to construct verifiers for new domains, how to validate their quality, or what level of granularity is sufficient. For tasks where functional equivalence is hard to define programmatically — open-ended dialogue, creative writing, strategic planning — building a verifier may be as difficult as solving the task itself. In such domains, PivotRL's performance would collapse to something close to the naive local RL baseline (57.34 on $\tau$2-Bench with strict reward, Table 4).

What evidence exists in the paper. The SWE-Bench result is the clearest demonstration of this limitation: PivotRL underperforms SFT, and the paper explicitly blames the verifier. The ablation on $\tau$2-Bench (Table 4) shows that replacing functional reward with strict exact-match reward drops accuracy from 63.81 to 57.34 — a 6.47-point gap that directly measures the value of a well-designed verifier in one domain. However, the paper does not provide any ablation on verifier quality for SWE-Bench (e.g., testing an argument-matching verifier vs. the name-only verifier) to quantify how much of the SFT-PivotRL gap is attributable to verifier coarseness versus other factors. Without this, it is unknown whether a richer SWE-Bench verifier would close the gap, narrow it, or leave it unchanged.

Mitigation status. The paper acknowledges this as future work in Section 6: "In future work, we plan to extend our framework to incorporate non-programmatic verifiers, such as LLM-as-a-judge frameworks and process reward models." However, the paper provides no experimental evidence that LLM-as-a-judge verifiers would improve SWE-Bench results — this is a forward-looking aspiration, not a demonstrated extension. The current method stands or falls on the practitioner's ability to engineer a domain-specific verifier, and the paper offers no tools or principles to guide that engineering beyond the four examples provided.

---

3. SWE-Bench Is the Only Domain Where PivotRL Fails to Outperform SFT, and the Explanation Is Incomplete

The assumption or constraint. The paper frames PivotRL as generally superior to same-data SFT in in-domain accuracy (+4.17% average, Table 1). This average is computed across four domains, but SWE-Bench — arguably the most prominent and rigorous agentic benchmark in the set — shows PivotRL losing to SFT by 4.73 points (32.67 vs. 37.40). The paper attributes this to the "deliberately coarse local signal" of the SWE-Bench verifier (Appendix A.2) and positions PivotRL's SWE-Bench performance as a compute-quality tradeoff against E2E RL (Figure 1) rather than against SFT.

The consequence. SWE-Bench is the standard evaluation for agentic coding and the domain where the paper's central claim — that PivotRL combines the efficiency of SFT with the accuracy of E2E RL — is most consequential. The fact that PivotRL underperforms SFT on this benchmark undermines the generality of the method. If a practitioner's primary use case is agentic coding (which dominates current industry interest in agentic LLMs), PivotRL offers worse in-domain accuracy than SFT, and the paper's value proposition shifts entirely to OOD retention and compute savings relative to E2E RL. But as discussed in Limitation 4, OOD retention relative to E2E RL is never measured. The paper's abstract and introduction report the averaged +4.17 gain without qualifying that SWE-Bench is a loss, which overstates the method's generality.

The explanation for the SWE-Bench underperformance is plausible but unverified. If the verifier is the bottleneck, the paper should demonstrate that improving it closes the gap. If the gap remains even with a richer verifier, then PivotRL's turn-level local RL paradigm may be fundamentally limited on tasks where per-turn action quality cannot be assessed independently of full-trajectory outcomes — a regime where behavior cloning (SFT) genuinely dominates. The paper does not distinguish between these two possibilities.

What evidence exists in the paper. Table 1 shows the raw numbers: PivotRL 32.67 vs. SFT 37.40 vs. Base 19.07 on SWE-Bench Verified. The paper's narrative in Section 4.2 reframes this comparison: "SWE-Bench is a natural comparison point because E2E RL is the standard training method for software-engineering agents... PivotRL reaches comparable accuracy to E2E RL with ~4× fewer rollout turns." This shifts the frame from "PivotRL vs. SFT" (where PivotRL loses) to "PivotRL vs. E2E RL on compute" (where PivotRL looks favorable). The framing is not dishonest — the paper reports both comparisons — but a casual reader of the abstract would not learn that PivotRL loses to SFT on the most important benchmark.

Mitigation status. The paper does not investigate why SWE-Bench is the outlier. No experiment tests whether a richer verifier (e.g., argument-level matching, LLM-as-judge over tool-call quality) improves PivotRL's SWE-Bench accuracy. No analysis examines whether certain types of SWE-Bench instances benefit from PivotRL while others benefit from SFT. The paper's future work section (Section 6) mentions LLM-as-a-judge verifiers but does not commit to resolving the SWE-Bench gap specifically.

---

4. OOD Retention Relative to E2E RL Is Claimed but Never Measured

The assumption or constraint. The paper's central positioning (Section 1) is that PivotRL combines "the data efficiency of SFT with the generalization capabilities of E2E RL." The OOD experiments (Tables 2–3) demonstrate that PivotRL preserves OOD performance relative to the base model (average change +0.21), while SFT causes catastrophic degradation (average change -9.83). This establishes PivotRL's advantage over SFT on OOD retention. However, the paper never runs E2E RL on the same OOD benchmarks, so it never measures whether PivotRL's OOD retention is equivalent to, better than, or worse than E2E RL's.

The consequence. The claim that PivotRL matches E2E RL's generalization is an extrapolation from Theorem 3.3 (which shows that functional-reward RL preserves the reference policy's relative ordering on task-unrelated actions) and from the behavioral pattern that PivotRL avoids SFT's catastrophic forgetting. But Theorem 3.3 does not predict whether E2E RL with full-trajectory rollouts would preserve OOD capabilities better, worse, or equivalently to PivotRL — it is a statement about functional reward specifically, not about on-policy training in general. E2E RL might preserve OOD capabilities through a different mechanism (the on-policy state distribution keeps the model's broader knowledge active) that could be stronger or weaker than PivotRL's block-rescaling preservation. Without a direct comparison, the claim that PivotRL has "the generalization capabilities of E2E RL" is unsupported.

This matters for practitioners. If E2E RL provides superior OOD retention to PivotRL (e.g., E2E RL shows a +2.0 OOD improvement while PivotRL shows +0.21), then the choice between PivotRL and E2E RL involves a genuine three-way tradeoff: SFT offers best in-domain accuracy and worst OOD retention at lowest cost; PivotRL offers moderate in-domain accuracy and good OOD retention at moderate cost; E2E RL offers best OOD retention (hypothetically) at highest cost. The paper cannot characterize this tradeoff because the E2E RL OOD data does not exist.

What evidence exists in the paper. None. The OOD experiments (Tables 2–3) compare PivotRL and SFT only. The SWE-Bench E2E RL comparison (Figure 1) measures in-domain accuracy and training cost only — no OOD evaluation is reported for the E2E RL checkpoints. The production deployment (Table 5) uses PivotRL alongside other RL environments, making it impossible to attribute OOD effects to PivotRL specifically versus the combined pipeline. The paper cites Chen et al. (2025) and Shenfeld et al. (2025) as evidence that on-policy RL mitigates forgetting, but does not replicate or extend those findings to its own experimental setup.

Mitigation status. The paper does not acknowledge this gap. Section 8 of the example reference paper explicitly noted that PRM search and revisions were studied independently and never combined, flagging this as a limitation. PivotRL does not similarly note the absence of E2E RL OOD comparisons. The claim of "generalization capabilities of E2E RL" is treated as if it follows from the SFT comparison and the theory, but it does not.

---

5. Training Is Single-Domain; Multi-Domain Scaling and Interaction Effects Are Unexplored

The assumption or constraint. All controlled experiments train on a single agentic domain and evaluate OOD on non-agentic benchmarks (Section 4.1). Each of the four models in Tables 1–3 was trained exclusively on one domain's trajectories ($\tau$2-Bench or SWE-Bench or Terminal-Bench or BrowseComp), with no mixing of domains. The production deployment (Table 5) uses multi-domain training, but without a controlled comparison to single-domain results or to alternative multi-domain strategies.

The consequence. In practice, a production agentic model must handle multiple agentic domains simultaneously — a single model that can do conversational tool use, coding, terminal interaction, and web browsing. The paper provides no evidence about how PivotRL behaves in this multi-domain regime. Several failure modes are plausible:

• Negative transfer across domains: Training on terminal commands might interfere with coding ability (both involve structured command syntax) in ways that don't show up in OOD benchmarks (which test math, science, and general knowledge rather than adjacent agentic skills). The paper's OOD evaluation focuses on non-agentic tasks, not cross-domain agentic transfer.

• Pivot set overlap or conflict: When pivots from multiple domains are pooled, the filtering criteria (variance and difficulty thresholds) might select different proportions from different domains, leading to imbalanced training that over-optimizes one domain at the expense of others. The paper provides no guidance on how to set domain-specific thresholds or balance pivot sets in multi-domain training.

• KL penalty calibration across domains: The KL penalty coefficient $\beta$ is held constant within a single domain's training. In multi-domain training, different domains may require different KL strengths — a domain where the base model has strong prior knowledge (e.g., coding) might need a larger $\beta$ to prevent drift than a domain where the base model is weak (e.g., terminal control). The paper does not address this.

• OOD degradation might compound or interact: SFT's OOD degradation compounds across domains (training on $\tau$2-Bench already causes -9.83 average OOD regression; adding SWE-Bench training on top might make it worse). PivotRL's near-zero OOD change in single-domain training does not guarantee that multi-domain training remains near-zero — the block-rescaling analysis (Theorem 3.3) is per-state and does not address the cumulative effect of training on many different acceptable sets $\mathcal{M}(s)$ from different domains.

What evidence exists in the paper. The production deployment (Table 5) is the only multi-domain result, and it is uncontrolled. Nemotron-3-Super's RL stage includes PivotRL environments and other RL environments for reasoning and chat simultaneously. The table reports only in-domain agentic benchmark improvements — no OOD evaluation, no comparison to multi-domain SFT, and no comparison to multi-domain E2E RL. The single-domain results cannot be extrapolated to the multi-domain regime with confidence.

Mitigation status. The paper does not discuss multi-domain training as a limitation or provide any experimental guidance for it. The production deployment demonstrates that multi-domain PivotRL is possible and produces large in-domain gains (Table 5: SWE-Bench from 12.87 to 61.33), but the absence of controlled comparisons and OOD evaluation means the deployment serves as an existence proof rather than a characterization of the multi-domain tradeoff space.

---

6. Single Training Run Per Domain, No Statistical Characterization

The assumption or constraint. Every experimental result in the paper is a single training run. The four domain-specific models (Tables 1–3) are each trained once with PivotRL and once with SFT. The ablation (Table 4) reports the best checkpoint from a single run per configuration. The SWE-Bench E2E comparison (Figure 1) matches PivotRL's peak accuracy (step 130, 32.67%) to an E2E RL checkpoint (step ~72, also 32.67%) from single training runs. No standard deviations, confidence intervals, or significance tests are reported anywhere. The paper does not claim to have run multiple seeds, nor does it discuss the stability of the reported results.

The consequence. Without uncertainty quantification, the paper's quantitative claims cannot be assessed for statistical reliability. Several numbers raise concern:

• BrowseComp: Base = 2.50, SFT = 1.50, PivotRL = 11.30 (Table 1). This is the paper's largest relative gain (+9.80 over SFT), but the absolute accuracies are tiny. On a benchmark where the base model solves 2.5% of instances, a difference of a few correctly solved instances — driven by sampling noise in model generation, environment stochasticity, or training seed — can produce large percentage-point swings. Without error bars, it is unknown whether the 11.30 result is significantly above 1.50 or whether a different random seed would produce a different ordering.

• SWE-Bench matched comparison (Figure 1): PivotRL reaches 32.67% at step 130; E2E RL reaches 32.67% at step ~72. Both are single-run peak accuracies. The variance of these peaks across random seeds could be large — especially for PivotRL, where the pivot filtering step introduces additional randomness (which states survive filtering depends on the $K$ profiling samples). If PivotRL's peak accuracy varies by ±3 points across seeds, the matched comparison becomes much less precise.

• Ablation ordering (Table 4): Full PivotRL = 63.81, "-Pivot filtering" = 59.68, "-Functional reward" = 57.34. The -1.66 gap between the two ablated configurations (59.68 vs. 57.34) might not be significant if each has a standard error of 2 points. The paper cannot claim that functional reward matters more than pivot filtering based on these numbers without variance estimates.

• OOD averages (Table 2): The PivotRL average OOD change of +0.21 is the mean of changes across eight benchmarks and four training runs (32 data points). Some of these changes are positive, some negative, none large. The claim that PivotRL has "essentially perfect" OOD retention is supported by the magnitudes and consistency, but the precision of the estimate (±0.5? ±2.0?) is unknown.

What evidence exists in the paper. None. The paper does not report running multiple seeds, does not show error bars on any figure or table, and does not discuss result stability. This is a significant departure from standard empirical ML practice, where at minimum the variance across random seeds is reported for small-scale experiments, and ideally confidence intervals are provided for headline claims.

Mitigation status. The paper does not acknowledge the absence of statistical characterization as a limitation. It is possible that the computational cost of multiple training runs (each requiring full RL training on large datasets) was prohibitive, but the paper does not state this. The external validity of the results — particularly the specific numeric improvements — is therefore weaker than it appears, and practitioners should treat the reported accuracies as point estimates from single runs rather than as stable, reproducible quantities.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper reframes the central challenge of agentic post-training as a data selection and credit assignment problem rather than an algorithmic optimization problem, and in doing so opens a previously inaccessible point on the compute-quality Pareto frontier. The shift is conceptual but carries immediate practical consequences for how organizations architect their post-training pipelines.

The dominant narrative in agentic RL has been that on-policy environment interaction is the essential ingredient that separates RL from behavior cloning — that full-trajectory rollouts expose the model to the consequences of its own mistakes, preventing the compounding error that dooms offline imitation (Rajaraman et al., 2020). PivotRL challenges this narrative not by disputing the value of on-policy data but by relocating where that value comes from. The paper's diagnostic finding — that 71% of randomly sampled turns from expert trajectories produce zero learning signal under GRPO, and that spending rollout budget on those turns is wasted compute — reveals that the bottleneck is not the absence of interaction but the uniform allocation of interaction budget across uninformative states. The implication is that expert trajectories already contain the informative decision points; the challenge is identifying them, not generating new ones from scratch.

This reorients the field's attention from how to collect more interaction data to how to select the right states to interact from. The practical upshot is that organizations already sitting on large SFT trajectory datasets — which is nearly every team training agentic LLMs — possess an underutilized resource. PivotRL provides a recipe for converting those trajectories from passive demonstrations into active RL training data, without requiring new data collection infrastructure.

The paper also provides the first structural explanation for why some post-training methods destroy OOD capabilities while others preserve them. Theorem 3.3 isolates the mechanism: functional-equivalent reward combined with KL regularization produces a block-rescaling of the action distribution that shifts probability mass toward acceptable actions while exactly preserving the relative ordering of all other actions. This is not a heuristic correlation — it is a geometric property of the regularized objective's minimizer. The result reframes OOD retention from an empirical optimization artifact (something that RL "happens to do better") to a design consequence of reward function structure. It implies that the critical variable is not whether you use RL but whether your reward function defines a set of acceptable actions rather than a point target. This provides a principled vocabulary for discussing generalization in fine-tuned models and suggests a concrete design rule: whenever possible, define rewards over equivalence classes of acceptable outputs, not individual reference strings.

A secondary reframing concerns the role of the KL penalty in RL fine-tuning. The standard motivation for KL regularization is to prevent policy collapse and maintain generation diversity. Theorem 3.3 gives it a sharper interpretation: the KL penalty is the mechanism that enforces the block-rescaling structure, and the coefficient β directly controls how much probability mass shifts from the complement into the acceptable set. This makes β a tunable knob for the OOD/in-domain tradeoff — larger β means stronger preservation of the reference policy's full distribution, smaller β means more aggressive optimization on the training domain. This is not a new insight in the KL-regularized RL literature broadly, but PivotRL's domain (agentic tasks with large, structured action spaces) and its explicit connection to OOD retention make the tradeoff concretely measurable in ways that prior work on chat alignment does not.

Reconciling prior contradictions. The paper provides a unified explanation for why prior SFT-to-RL bridge methods (Uchendu et al., 2023; Hester et al., 2018; Ming et al., 2026) showed inconsistent results, sometimes matching RL and sometimes failing to improve over behavior cloning. The difference, PivotRL suggests, is whether those methods implicitly concentrated training on informative states. Naive local RL with uniform turn sampling (Table 4: 57.34, equivalent to SFT's 58.44) fails because it wastes 71% of its budget on deterministic states. Prior methods that happened to focus training on mixed-outcome decision points — perhaps through reward shaping, trajectory length effects, or domain-specific biases — would show stronger results. Methods that sampled states uniformly would show weak results. PivotRL provides the diagnostic tool (reward variance profiling) and the theoretical justification (Proposition 3.1, Theorem 3.2) to explain both outcomes within a single framework.

Research directions that become more attractive. The paper makes verifier design for local credit assignment a first-class research problem. Prior work on reward models focused primarily on outcome-level reward (e.g., RLHF reward models that score full responses) or process-level reward for reasoning chains (Lightman et al., 2023). PivotRL demonstrates that turn-level verifiers for agentic actions — programmatic functions that assess whether a single tool call, shell command, or search step is locally acceptable — can drive substantial gains even when they are coarse (SWE-Bench's tool-call name matching). This opens a design space: how fine-grained should a per-turn verifier be? What is the tradeoff between verifier precision and verifier construction cost? Can verifiers be learned rather than hand-crafted? The paper's future work explicitly identifies LLM-as-a-judge and process reward models as promising directions, but the more fundamental question is whether turn-level verifiability is a property of the task (some tasks admit clean per-step evaluation; others do not) or a property of the verifier engineering investment (with enough effort, any task can be decomposed into locally verifiable steps).

Research directions that become less attractive. PivotRL's empirical results suggest that naive full-trajectory E2E RL — training from scratch with sparse end-of-episode rewards on long-horizon agentic tasks — may be an unnecessarily expensive default. The paper does not argue that E2E RL is obsolete, but it demonstrates that much of E2E RL's benefit can be recovered by local RL at pivot states, at a fraction of the environment interaction cost. This shifts the burden of proof: future work proposing E2E RL for agentic tasks should compare against a pivot-filtered local RL baseline rather than (or in addition to) a pure SFT baseline, and should report the marginal benefit of full trajectories over expert-intermediate states. Methods that rely on full-trajectory credit assignment (e.g., RL with sparse trajectory-level rewards and no per-step verifier) become harder to justify if a per-step verifier can be constructed and pivot states can be identified.

The paper also casts doubt on the value of exact-match reward as a baseline for converting demonstrations into RL data. Naive local RL with strict string matching performs no better than SFT (57.34 vs. 58.44, Table 4), confirming that the gap between functional correctness and surface-form matching in generative action spaces is large enough to nullify the benefits of on-policy sampling. Future work that proposes to convert SFT data into RL episodes should justify its credit assignment scheme against a functional-equivalent baseline, not just against SFT or against no training.

---

Follow-Up Research This Work Enables

Directly measure whether PivotRL's OOD retention matches or exceeds E2E RL's. The paper claims PivotRL combines "the generalization capabilities of E2E RL" with the efficiency of SFT (Section 1), but the OOD retention experiments (Tables 2–3) compare PivotRL only against SFT, never against E2E RL. A direct comparison would train E2E RL on SWE-Bench (or any of the other three domains) to the same in-domain accuracy as PivotRL (32.67% on SWE-Bench, or a matched accuracy point on another domain) and then evaluate both on the eight OOD benchmarks. This would test whether Theorem 3.3's block-rescaling property (preserving relative ordering of task-unrelated actions) produces OOD retention that is equivalent to, better than, or worse than the retention produced by full-trajectory on-policy training. If E2E RL shows superior OOD retention, the tradeoff space becomes three-dimensional (in-domain accuracy, OOD retention, compute cost) rather than the two-dimensional framing the paper presents. If PivotRL shows equivalent or better OOD retention, the case for local RL over full-trajectory RL becomes substantially stronger.

Quantify the verifier quality ceiling: how much SWE-Bench accuracy does PivotRL recover with a richer per-turn verifier? The paper attributes PivotRL's SWE-Bench underperformance (32.67% vs. SFT's 37.40%, Table 1) to the deliberately coarse verifier that matches only tool-call names, not arguments or patch quality (Appendix A.2). This is a testable claim. An experiment would construct progressively richer SWE-Bench verifiers — tool-call name matching only (the current setup), tool-call name + argument schema matching, tool-call name + argument schema + LLM-as-judge over argument quality, and finally a verifier that runs the generated patch against the test suite — and measure PivotRL's accuracy at each verifier level. If accuracy monotonically increases with verifier richness and approaches or exceeds SFT's 37.40%, it confirms that verifier quality is the binding constraint and that investment in verifier engineering is the path to closing the SWE-Bench gap. If accuracy plateaus well below SFT's, it suggests a more fundamental limitation: that turn-level credit assignment, even with a perfect per-turn verifier, cannot capture the trajectory-level dependencies that matter for SWE-Bench task success. This would bound the scope of tasks for which local RL is sufficient.

Characterize the tradeoff between pivot profiling cost and training efficiency to find the deployment-relevant Pareto frontier. PivotRL's offline profiling step — sampling K actions per candidate state under the reference policy, scoring them with the verifier, and computing reward variance — is computationally significant but entirely unmeasured in the paper. A systematic study would vary K (the number of profiling samples per candidate state) across a range (e.g., K = 4, 8, 16, 32, 64, 128) and measure two outcomes: the quality of the retained pivot set (measured by how well the profiling-stage variance estimates predict actual training-stage reward variance, and by the final trained model's accuracy) and the total profiling cost (in rollout turns). The goal is to find the minimum K that produces pivot sets statistically indistinguishable from large-K pivot sets in downstream training performance. If small K (e.g., K = 8–16) suffices, the profiling cost becomes negligible relative to training, and the headline 4× rollout-turn reduction is real rather than an upper bound. If large K is required (e.g., K ≥ 64), the amortized profiling cost significantly erodes the compute advantage, and the method's practical value case depends on how many training steps the profiling cost is amortized over. This experiment would also reveal whether the 71% zero-signal-turn statistic (Section 2) is robust to the choice of K — an underpowered profiling step might misclassify informative turns as uninformative or vice versa.

Test pivot filtering with learned rather than programmatic verifiers, especially for domains without clean local correctness signals. The current method requires a domain-specific programmatic verifier, which limits applicability to tasks where such verifiers can be engineered. An extension would replace the hand-crafted verifier with a learned turn-level reward model — trained either on the SFT trajectories directly (supervised, distinguishing demonstrated actions from randomly sampled actions) or via LLM-as-a-judge annotations on sampled turn-level actions. The key question is whether a learned verifier can maintain sufficient accuracy to support the functional-reward mechanism without introducing reward hacking. The theoretical framework (Theorem 3.3) applies to any verifier that defines an acceptable set  M(s); the practical risk is that a learned verifier's errors would either (a) expand  M(s) too broadly, crediting genuinely bad actions and diluting the RL signal, or (b) contract  M(s) too narrowly, reproducing the exact-match reward problem. A strong follow-up would train a learned verifier on the τ2-Bench domain (where the programmatic verifier works well and can serve as a ground-truth reference), compare PivotRL accuracy with the learned verifier against the programmatic verifier, and measure whether the learned verifier's errors correlate with specific failure modes (e.g., over-crediting actions that are superficially similar to demonstrations but functionally wrong).

Multi-domain PivotRL with controlled OOD evaluation to test whether OOD retention degrades under cumulative training. The paper's single-domain experiments demonstrate near-perfect OOD retention when training on one domain at a time, but production deployments (Table 5) train on multiple agentic domains simultaneously. A controlled experiment would train PivotRL sequentially or jointly on all four domains (τ2-Bench, SWE-Bench, Terminal-Bench, BrowseComp) and evaluate both in-domain accuracy per domain and OOD accuracy on the eight non-agentic benchmarks, comparing against (a) SFT trained on the same multi-domain data, (b) E2E RL trained on the same multi-domain data, and (c) a weighted combination of the four single-domain PivotRL models. The hypothesis is that PivotRL's OOD retention might degrade under cumulative training because the block-rescaling property (Theorem 3.3) applies per-state, but when the model is trained on many different acceptable sets from many different domains, the KL penalty may not be sufficient to prevent the accumulated mass shifts from distorting the reference policy's global structure. If multi-domain PivotRL maintains near-zero OOD regression while multi-domain SFT's regression compounds, the case for PivotRL as the default agentic post-training method becomes very strong. If multi-domain PivotRL shows meaningful OOD degradation, then either the KL penalty coefficient β needs to be increased (trading off in-domain improvement) or domain-specific reference policies are needed.

Apply pivot filtering to other sparse-reward RL settings outside agentic tasks. The core mechanism — profiling candidate states for reward variance under a reference policy and training only on mixed-outcome states — is domain-agnostic. It could apply to any setting where training data consists of long trajectories with sparse or turn-level rewards, and where the majority of decision points may be deterministic under the current policy. Candidates include: multi-step mathematical reasoning (where intermediate steps are verifiable via a process reward model, and many steps may be uniformly correct or incorrect for a given model), dialogue systems (where turn-level user satisfaction signals are available but most turns are either trivially handleable or unhandleable), and robotic manipulation (where state-action trajectories contain many waypoints, only some of which are near the boundary of the policy's competence). An experiment would replicate the PivotRL pipeline — extract decision points, profile reward variance under the reference policy, filter for mixed-outcome states, train with local rollouts — in one of these domains and measure whether the 71% zero-signal statistic replicates and whether pivot filtering provides a similar efficiency gain over uniform state sampling. This would test the generality of the paper's central diagnostic and establish pivot filtering as a transferable architectural pattern rather than an agentic-domain-specific trick.

---

Practical Applications and Downstream Use Cases

Production agentic post-training pipelines with strict OOD safety requirements. The paper's most actionable finding for practitioners is that PivotRL enables substantial in-domain accuracy improvements (+14.11 average over base, Table 1) with essentially zero OOD degradation (+0.21 average change, Table 2). For any deployment where the model must serve both agentic and non-agentic use cases — a coding assistant that must also answer general knowledge questions, a customer service agent that must not lose mathematical reasoning capability — PivotRL eliminates the catastrophic forgetting risk that makes SFT-based agentic fine-tuning dangerous to deploy. The specific numbers: SFT averaged -9.83 OOD change, with worst-case drops of -64.48 on AIME25 after terminal-domain training (Table 3). A production team that currently avoids agentic SFT because of OOD degradation risk can adopt PivotRL as a drop-in replacement, using the same trajectory datasets they already possess, and expect the in-domain gains without the OOD penalties. The paper's deployment in Nemotron-3-Super (Table 5) demonstrates this at scale: the model's SWE-Bench accuracy improves from 12.87 to 61.33 while the model continues to serve as a general-purpose LLM.

Cost-efficient agentic fine-tuning for teams with limited RL infrastructure. The ~4× reduction in rollout turns relative to E2E RL (Figure 1, ~133K vs. ~542K) and the ~5.5× reduction in wall-clock time make PivotRL accessible to teams that cannot afford the environment-simulation infrastructure and GPU hours required for full-trajectory RL. A team with existing SFT trajectory data (generated via synthetic pipelines, stronger models, or human annotation) and a budget for moderate on-policy sampling can run the full PivotRL pipeline: offline pivot profiling (one-time cost, amortized over training), then GRPO training with single-turn rollouts. No multi-turn environment state management is required — the environment only needs to execute single actions and return verifier scores. This lower infrastructure barrier is particularly relevant for domains where full-trajectory simulation is expensive or unavailable (e.g., enterprise tool APIs that are rate-limited or have side effects), since PivotRL can train on partial rollouts from logged expert states without needing to re-execute full workflows.

Rapid-domain adaptation where SFT trajectories are available but OOD retention is critical. When a new agentic domain is added to an existing model's capabilities, SFT on that domain's trajectories alone causes OOD regression on all other capabilities. PivotRL enables adding the new domain without retraining the entire multi-domain RL pipeline. A team could: (1) collect expert trajectories for the new domain; (2) profile candidate turns with the existing reference policy; (3) train PivotRL on the filtered pivot set; (4) deploy the updated model with confidence that existing capabilities are preserved. The Terminal-Bench results illustrate the value: PivotRL improves Terminal-Bench from 5.42 to 20.00 while keeping AIME25 at 82.92 (vs. 21.56 for SFT, Table 3). This pattern — large in-domain gain, negligible OOD loss — makes PivotRL suitable for incremental capability addition in continuously deployed models.

Self-improvement data generation loops. The paper's theoretical framework and empirical results suggest that PivotRL could serve as the optimization engine in a self-improvement pipeline where the model generates its own training data. The loop would work as follows: (1) the model generates trajectories on new agentic tasks; (2) a functional verifier scores turn-level actions; (3) pivot filtering identifies mixed-outcome turns; (4) PivotRL trains on those turns; (5) the improved model generates higher-quality trajectories, which are scored and filtered, and the cycle repeats. The advantage over standard self-improvement via SFT on model-generated completions is that PivotRL's functional reward and KL regularization prevent the policy from collapsing to a narrow mode (the SFT failure) while still pushing it toward higher reward. The ~4× cost reduction relative to full-trajectory RL makes this loop feasible at scale, since each iteration requires only single-turn rollouts at pivot states rather than full trajectories. This application is speculative — the paper does not demonstrate a self-improvement loop — but it follows directly from the method's design: PivotRL is exactly the kind of update rule (on-policy, KL-regularized, functional-reward) that sustainable self-improvement requires, and the pivot filtering ensures that each iteration's training budget is spent on states where the model can actually learn.

---

When to Prefer This Method

The paper articulates a clear tradeoff between three post-training paradigms — SFT, E2E RL, and PivotRL (local RL with pivot filtering and functional reward) — and its experiments characterize the conditions under which each dominates. The decision rule is:

• Prefer PivotRL over SFT when OOD retention is a hard requirement (the model must preserve performance on non-agentic tasks including math, coding, science, and translation) AND a domain-specific functional verifier can be constructed that meaningfully expands the acceptable action set beyond a single demonstration. This condition holds for most production agentic deployments where the model serves general-purpose use cases alongside agentic ones, and where tool-call schema validation, command equivalence checks, or LLM-as-judge evaluation can approximate functional correctness at each turn. The evidence: PivotRL achieves +10.04% higher OOD accuracy than SFT across eight benchmarks (Table 2) while providing comparable or better in-domain accuracy on three of four tested domains (Table 1).

• Prefer SFT over PivotRL when in-domain accuracy on a specific benchmark is the sole optimization target and OOD degradation is acceptable (or irrelevant, e.g., for a single-purpose agent deployed in isolation) AND the SFT demonstrations provide strong behavior-cloning signal. This condition held for SWE-Bench in the paper's experiments, where SFT achieved 37.40% vs. PivotRL's 32.67% (Table 1), likely because the coarse SWE-Bench verifier (tool-call name matching only) limited PivotRL's per-step RL signal quality while SFT's full-trajectory imitation provided a stronger training target.

• Prefer PivotRL over E2E RL when environment interaction is expensive (long trajectories, rate-limited APIs, complex simulation infrastructure) AND a domain-specific functional verifier can be constructed. PivotRL achieves competitive accuracy with E2E RL on SWE-Bench (32.67% for both at the matched point in Figure 1) with ~4× fewer rollout turns and ~5.5× less wall-clock time, but this comparison is at a single accuracy point. The advantage is purely on cost at equivalent moderate accuracy; the paper does not demonstrate that PivotRL can match E2E RL's peak accuracy if both are pushed to saturation.

• Prefer E2E RL over PivotRL when (a) no satisfactory per-turn functional verifier can be constructed (the task requires trajectory-level credit assignment) OR (b) the marginal benefit of full on-policy state distributions over expert-intermediate states is large (e.g., tasks where the model's errors compound severely and recovery behavior must be learned from states the model actually visits, not expert states). The paper does not experimentally characterize this regime, but theory (Rajaraman et al., 2020) predicts that the gap between offline imitation and online interaction grows quadratically with horizon, suggesting that for very long-horizon tasks ($T \gg 30$ turns), full-trajectory RL may provide benefits that local RL cannot recover.

• Prefer PivotRL with a richer verifier over all alternatives (speculative, not demonstrated in the paper) when the verifier can accurately evaluate turn-level action quality — potentially through learned reward models, execution-based verification (running code against tests), or human feedback — and the target task admits decomposition into locally verifiable steps. Under these conditions, PivotRL would combine the in-domain accuracy of the strongest method (since rich verifiers provide strong per-step RL signal) with the OOD retention of KL-regularized RL and the compute efficiency of local rollouts. The paper's SWE-Bench result (32.67% with a name-only verifier vs. 37.40% for SFT) is a lower bound; the open question is whether a sufficiently rich verifier can push PivotRL above SFT on SWE-Bench, making this the dominant strategy for coding agents as well.