ArXiv: 2509.02547
🎯 Pitch
Reinforcement learning doesn't just align LLMs—it can install genuinely new reasoning behaviors, but only under tight conditions: high-fidelity verifiable rewards, compositional multi-step tasks, and a critical 'Goldilocks' zone of base model capability. This survey maps that phenomenon across planning, tool use, and self-improvement, revealing that smaller models can beat much larger proprietary systems when trained as autonomous agents in dynamic environments.
1. Executive Summary
This survey formalizes the emerging paradigm of Agentic Reinforcement Learning (Agentic RL)—a shift from treating LLMs as static single-turn sequence generators to conceptualizing them as autonomous decision-making agents embedded in partially observable, temporally extended environments—and proposes a comprehensive twofold taxonomy organized around core agentic capabilities (planning, tool use, memory, self-improvement, reasoning, and perception) and their application domains (search, code generation, mathematical reasoning, GUI navigation, and multi-agent systems). Synthesizing over five hundred recent works, the paper argues that reinforcement learning serves as the critical mechanism for transforming these capabilities from static heuristic modules into adaptive, robust behaviors—ranging from outcome-based reward RL for search agents (e.g., GRPO-trained query generation against live web APIs) to process-level RL for mathematical reasoning (e.g., step-wise PRM supervision interleaved with code execution in tool-integrated reasoning pipelines) to multi-agent RL for joint policy optimization (e.g., MAGRPO training decentralized agents under a shared objective). The survey consolidates the landscape of open-source environments, benchmarks, and frameworks, establishing that Agentic RL has enabled smaller models to rival or surpass much larger proprietary systems on domain-specific benchmarks—yet the field remains bounded by fundamental challenges in trustworthiness (reward hacking, hallucination, sycophancy), scaling up training computation and environments, and the unresolved mechanistic debate over whether RL merely amplifies pre-existing reasoning patterns or installs qualitatively new capabilities, with evidence suggesting that genuinely novel behaviors emerge primarily under conditions of high-fidelity verifiable rewards, compositional multi-step structure, and base models operating in an intermediate capability regime.
2. Context and Motivation
The Core Problem: A Fragmented Landscape Without a Unifying Framework
The fundamental problem this survey addresses is the absence of a unified conceptual and formal treatment of how reinforcement learning transforms large language models into autonomous agents. By early 2025, two largely disconnected research communities had emerged around the intersection of LLMs and RL, each with their own terminology, evaluation protocols, and assumptions—without a coherent framework connecting them.
The first community, which the paper terms "Synergy between RL and LLMs," focused on applying RL algorithms to improve or align LLMs as static text generators. The dominant paradigm here was Preference-Based Reinforcement Fine-Tuning (PBRFT) , exemplified by RLHF (Christiano et al., 2017; Ouyang et al., 2022) and DPO (Rafailov et al., 2023). In this framework, an LLM is optimized to produce single-turn outputs that align with human preferences or benchmark scores—the model generates a response to a prompt, receives a scalar reward (from a learned reward model or a preference pair), and is updated accordingly. Classic algorithms like PPO (Schulman et al., 2017), its critic-free variant GRPO (DeepSeek-AI et al., 2025), and direct preference methods like DPO dominated this space. The survey notes that this body of work produced considerable algorithmic sophistication—from variance-reduced advantage estimation (Yue et al., 2025) to online preference optimization (Hong et al., 2024)—but remained fundamentally constrained to degenerate, single-step Markov Decision Processes (MDPs) with horizon .
The second community, termed "LLM Agents," focused on building systems where LLMs function as autonomous decision-makers capable of reasoning, planning, invoking tools, maintaining memory, and interacting with environments over extended horizons. Research here produced a rich ecosystem of individual systems and capabilities: ReAct-style tool calling (Yao et al., 2023), retrieval-augmented generation (Fan et al., 2024), planning frameworks like plan-execute-reflect loops (Masterman et al., 2024), self-evolution mechanisms (Tao et al., 2024), multi-agent coordination (Aratchige & Ilmini, 2025; Hong et al., 2024), and multimodal integration for embodied and GUI tasks (Durante et al., 2024). However, as the survey notes, this work largely treated the LLM's internal parameters as frozen—the "agent" was a prompting architecture or a hand-designed pipeline wrapped around a static model, not an RL-optimized policy. Capabilities like planning, memory, or self-reflection were implemented as heuristic modules rather than learned behaviors.
The critical gap was that neither community spoke the same language. PBRFT researchers optimized LLMs as sequence generators but ignored sequential decision-making; agent researchers built sophisticated interactive systems but relied on brittle manual engineering rather than principled policy optimization. There existed no formal abstraction that could distinguish these paradigms, no taxonomy connecting particular RL algorithms to specific agent capabilities, and no systematic understanding of when and why RL succeeds or fails in transforming static modules into adaptive behaviors.
Why This Gap Matters: The Stakes for Practical and Theoretical Progress
The survey identifies several reasons why this fragmentation is consequential, grounding its argument in both practical deployment concerns and theoretical understanding.
Practical impact: The emerging landscape of production agentic systems. Between late 2024 and mid-2025, several high-profile commercial systems demonstrated that RL-trained agents could achieve capabilities far beyond those of mere prompted LLMs. OpenAI's Deep Research achieved 51.5% pass@1 on BrowseComp (Wei et al., 2025)—a benchmark explicitly designed to be difficult for non-agentic models—by combining RL-shaped web search policies with long-horizon reasoning and cross-source verification. DeepSeek-R1 (DeepSeek-AI et al., 2025) demonstrated that pure RL training with verifiable rewards could induce complex reasoning behaviors (verification, backtracking, self-correction) without explicit supervision for those behaviors. Kimi K2, Qwen QwQ-32B, and numerous other systems similarly showed that RL was the critical differentiator between a chatbot and an autonomous agent. The survey cites these systems directly (Section 3.2) as evidence that "tool-integrated reasoning is no longer a niche capability but a baseline feature of advanced agentic models."
Yet despite this explosion of practical results, the research community lacked a systematic understanding of how to build such systems. Each success story used different RL algorithms, different reward designs, different environment setups, and different evaluation protocols—making it impossible to compare approaches, identify general principles, or transfer insights across domains. The survey positions itself as providing the first unified map of this territory, giving researchers and practitioners a shared vocabulary and conceptual framework.
Theoretical significance: The mechanistic debate. The survey identifies a deeper theoretical question that the fragmentation obscured: does RL merely amplify existing reasoning patterns (the "amplifier" view), or does it install qualitatively new capabilities (the "new-knowledge" view)? This debate, discussed explicitly in Section 6.4, has profound implications for the entire enterprise of training LLMs. If RL is only an amplifier—reshaping the base model's output distribution to concentrate probability on already-reachable reasoning paths—then the primary bottleneck is pretraining, and RL is a refinement tool. If RL can install genuinely new computation (exploiting sparse outcome signals to learn patterns that next-token prediction could not), then RL is a fundamental capability-creating mechanism, potentially as important as pretraining itself.
Prior work had produced conflicting evidence. Studies like Yue et al. (2025) showed that large- pass@ analyses often find base models matching or surpassing their RL-tuned counterparts, suggesting RL primarily improves pass@1 through distribution reshaping rather than capability creation. Conversely, work on parity learning (Tsilivis et al., 2025) showed that RL enables generalization on problems where next-token training alone is statistically or computationally prohibitive. Other studies documented emergent cognitive behaviors—verification, backtracking, subgoal setting—absent in base models (Gandhi et al., 2025). The survey argues that resolving this debate requires precisely the formal framework it provides: by characterizing the state space, action space, transition dynamics, and reward structure of different problem settings (Section 2), researchers can systematically investigate under what conditions RL produces amplification versus genuine capability emergence.
The deployment economics question. The survey also contextualizes its importance through the lens of resource allocation. The emergence of reasoning models like OpenAI's o1 and o3 series (OpenAI et al., 2024; OpenAI Team, 2025) demonstrated that test-time computation—extended chains of thought, multi-step verification, tool interleaving—could dramatically improve performance on complex tasks. But these models were primarily products of closed-source research labs with massive compute budgets. The open question, which the survey's consolidated landscape of environments and frameworks (Section 5) aims to support, is whether smaller research groups and companies can systematically build such capabilities using open-source RL pipelines. The survey's explicit consolidation of frameworks—from general-purpose RL libraries (RLlib, Acme) to specialized agentic RL systems (AgentFly, Agent Lightning, AWorld)—is motivated by the practical need to democratize agentic training infrastructure.
Where Prior Approaches Fell Short
The survey identifies specific limitations that prevented prior work from addressing the fragmentation it describes.
1. Isolated capability studies without a unifying formal model. Prior surveys on LLM agents (Luo et al., 2025; Plaat et al., 2025; Li, 2025; Masterman et al., 2024) typically organized their analysis around individual capabilities—tool use, planning, memory, reasoning—but treated these as independent modules with separate methods, datasets, and evaluation criteria. The survey argues this misses the crucial insight that RL can jointly optimize these capabilities when they are formulated as interdependent components of a single policy. For instance, a search agent's decision about when to invoke a tool (tool use), what to retain in memory (memory), and how to decompose a complex query (planning) are not independent problems—they are facets of a single sequential decision process. Only by formalizing the entire system as a POMDP (Section 2.1) can one design reward functions, credit assignment mechanisms, and training curricula that optimize the interaction between capabilities rather than each in isolation.
2. Inconsistent terminology and evaluation protocols. The survey notes that "current studies often examine isolated capabilities, domains, or custom environments, with inconsistent terminology and evaluation protocols, making systematic comparison and cross-domain generalization difficult" (Section 1). For example, "planning" in the context of GUI agents meant something entirely different from "planning" in mathematical reasoning or code generation. RL algorithms were evaluated on different benchmarks with different metrics (pass@1, success rate, task completion, F1 score), making it impossible to assess whether an algorithmic innovation in one domain would transfer to another. The survey's dual taxonomy—one axis for capabilities, one for tasks, with an explicit dependency matrix in Table 9—provides the first systematic attempt to align these disparate threads.
3. The PBRFT-to-agentic transition was underway but untheorized. Between the release of DeepSeek-R1 in January 2025 and the survey's publication, a massive wave of research had begun applying the same RL algorithms (GRPO, PPO, DPO variants) to agentic tasks—but in an ad hoc, domain-by-domain manner. The survey catalogs this explosion: RL for search agents (DeepRetrieval, Search-R1, R1-Searcher), RL for code agents (DeepSWE, SWE-RL, o1-Coder), RL for GUI agents (GUI-R1, UI-R1, WebAgent-R1), RL for math agents (ARTIST, ToRL, DeepSeek-Prover), RL for embodied agents (VLN-R1, RLVLA, TGRPO). Each of these lines of work independently discovered similar patterns—the importance of verifiable rewards, the challenge of sparse credit assignment, the emergence of self-correction behaviors—but there was no synthesis connecting them. The survey positions itself as providing exactly this synthesis: by formalizing the MDP/POMDP distinction (Section 2) and mapping it onto specific capabilities (Section 3) and tasks (Section 4), it shows that these seemingly disparate research threads are instances of a common paradigm.
4. The gap between industrial and open-source capabilities. The survey explicitly notes that "most open-source models still fail on OpenAI's BrowseComp" (Section 4.1.2) and that closed-source systems (OpenAI Deep Research, Kimi-Researcher, Google Gemini's DeepResearch) are "markedly stronger." This gap is attributed not to fundamental algorithmic secrets but to "more powerful foundation models and the availability of more high-quality data." The survey's consolidation of open-source environments (Section 5.1) and frameworks (Section 5.2) is thus motivated by the practical goal of closing this gap—providing the infrastructure needed for the open-source community to reproduce and extend industrial-scale agentic RL.
How This Paper Positions Itself
The survey positions itself not as introducing a new method or algorithm, but as providing the conceptual infrastructure for the emerging field of Agentic RL. Its positioning rests on four pillars:
A formal foundation via MDP/POMDP abstraction. The paper's central theoretical contribution (Section 2) is the formal articulation of the distinction between PBRFT and Agentic RL through the lens of Markov decision processes. PBRFT is characterized as a degenerate MDP with state space , single-step horizon , deterministic transitions, and scalar outcome rewards . Agentic RL is characterized as a Partially Observable MDP (POMDP) with multi-step horizon , dynamic state transitions , hybrid action spaces , and composite reward functions combining sparse task rewards with dense sub-rewards. This formalization is not merely definitional—it has direct consequences for algorithm design, credit assignment, and evaluation that the survey systematically traces through subsequent sections.
The survey explicitly positions this formalism as addressing a coordination failure in the field: "By casting both preference-based RFT and Agentic RL as MDPs or POMDPs, we clarify the theoretical implications of treating LLMs either as static sequence generators or as interactive, decision-capable agents embedded within dynamic environments" (Section 2.1). This directly enables cross-domain comparison—a code agent navigating a repository, a GUI agent interacting with a desktop, and an embodied agent manipulating objects can all be understood as instances of the same POMDP formulation, differing only in their specific state representations, action spaces, and reward functions.
A capability-centered taxonomy that treats RL as the common optimizer. Unlike prior surveys that catalog capabilities descriptively, Section 3 is organized around the thesis that RL is the mechanism for transforming each capability from a static heuristic into an adaptive behavior. For planning, this means distinguishing between RL as an external guide (training value functions for MCTS without updating the LLM) versus RL as an internal driver (directly optimizing the LLM's policy for plan generation through environmental interaction). For memory, it means tracing the evolution from passive retrieval (RAG-style, where memory content and access patterns are fixed) through RL-guided retrieval (where RL controls when to query memory) to full RL-controlled memory management (where RL decides what to store, update, delete, and forget). For tool use, it means charting the progression from ReAct-style prompting (static patterns), through tool-integrated reasoning with outcome-only RL (emergent self-correction), toward long-horizon multi-turn TIR with process-level credit assignment.
This taxonomy is explicitly not exhaustive—the survey emphasizes that listed methods are "illustrative rather than exhaustive" and that coverage of related work is provided in Section 4. The goal is conceptual alignment: showing that the same RL principles (policy gradient optimization over trajectories, credit assignment across time steps, exploration-exploitation tradeoffs in action selection) apply across all capabilities, creating a unified research agenda.
A task-oriented counterpart that demonstrates breadth. Section 4 provides the empirical evidence for the taxonomy's claims by surveying how RL has been applied across seven major task domains: search and research agents, code agents, mathematical reasoning agents, GUI agents, vision agents, embodied agents, and multi-agent systems. Each subsection is structured to show the same progression: from RL-free methods (prompting, SFT) through early RL attempts (outcome-only rewards) to more sophisticated formulations (process rewards, hybrid approaches, emergent multi-turn strategies). The survey uses this structure to argue that this progression is not coincidental but reflects the fundamental advantage of Agentic RL: the ability to learn from environmental feedback rather than merely imitating static demonstrations.
The application-capability dependency matrix (Table 9) explicitly maps which capabilities are "core," "supporting," or "minimal" for each task domain. This is a deliberate positioning move: it shows that the capability taxonomy is not an abstract exercise but directly predicts which RL formulations will be most important for which applications. For instance, search agents depend heavily on planning and tool use (core) but minimally on perception—suggesting that the critical RL challenges for search are credit assignment across multi-turn queries and optimizing tool invocation timing. GUI agents depend heavily on planning, tool use, and perception—suggesting that multimodal RL formulations are essential. This mapping provides a research roadmap: it tells practitioners which capabilities to prioritize when building agents for a given domain.
An infrastructure consolidation that enables reproducibility. Section 5 explicitly positions the survey as a practical resource by cataloging environments (Section 5.1) and frameworks (Section 5.2) for Agentic RL. Unlike prior surveys that might list tools descriptively, this section is organized to guide selection: environments are categorized by task domain (web, GUI, coding/software engineering, domain-specific, simulated/game, general-purpose), and frameworks are categorized by type (agentic RL frameworks, RLHF/LLM fine-tuning frameworks, general-purpose RL frameworks) with explicit summaries of key features and resource links. The survey argues that this consolidation addresses a critical bottleneck: "the scarcity of interactive, adaptive environments and the difficulty of engineering effective reward signals" (Section 6.3).
The Broader Intellectual Context
The survey situates Agentic RL within a larger historical arc from traditional RL (applied to Atari games, robotics, board games) through LLM-specific RL (RLHF, DPO, GRPO) to the fully agentic setting. Figure 2 in the paper visually encodes this progression as an outward-expanding fan, with color-coded regions representing features specific to each paradigm. The key conceptual claim is that Agentic RL subsumes but extends prior paradigms: it inherits the policy optimization machinery of traditional RL (value functions, advantage estimation, exploration strategies), the alignment objectives of LLM RL (preference modeling, KL constraints, reward robustness), and adds the multi-turn, multi-tool, multi-modal complexity of interactive environments.
This positioning has direct consequences for the survey's scope boundaries, which it states explicitly: the primary focus is "how RL empowers LLM-based agents in dynamic environments," while out-of-scope topics include RL for human value alignment (harmful query refusal), traditional RL algorithms that are not LLM-based (e.g., multi-agent RL for StarCraft without language models), and RL for boosting pure LLM performance on static benchmarks (the PBRFT setting). These boundaries are not arbitrary—they follow from the POMDP formalization: if the environment does not involve multi-step interaction, partial observability, and dynamic state transitions, it falls outside the Agentic RL paradigm.
Finally, the survey positions its contribution as timely and necessary given the field's trajectory. The introduction notes that with the release of models like OpenAI o3—which possesses "both self-evolving reasoning capabilities and support for tool use"—researchers were "beginning to contemplate how to deeply integrate LLMs with downstream tasks through reinforcement learning methods" (Section 1). The survey thus frames itself as both a retrospective synthesis (organizing over 500 papers into a coherent taxonomy) and a forward-looking roadmap (identifying open challenges in trustworthiness, scaling, and mechanistic understanding that will shape the next generation of agentic systems).
3. Technical Approach
3.1 Reader Orientation
This paper is a survey—a synthesis and organizational framework—not a paper proposing a new algorithm or system; its core idea is that reinforcement learning serves as the critical mechanism for transforming static LLM capabilities (planning, tool use, memory, reasoning, self-improvement, perception) into adaptive, robust agentic behaviors, and that this transformation can be understood through a unified formal lens of Partially Observable Markov Decision Processes (POMDPs). The "system" it builds is conceptual: a taxonomy that connects formal definitions (Section 2) to capability modules (Section 3) to task domains (Section 4) to practical infrastructure (Section 5), enabling researchers and practitioners to systematically design, compare, and scale Agentic RL systems across the rapidly fragmenting landscape of LLM agent research.
The problem it solves is incoherence: prior work studied agent capabilities in isolation, used inconsistent terminology, applied different RL algorithms to different domains without understanding why some succeeded and others failed, and lacked a shared formal language to connect theoretical insights (e.g., credit assignment mechanisms) with practical design choices (e.g., reward function structure). The solution's "shape" is a twofold taxonomy—one axis organized around RL-optimizable agent capabilities, the other around application domains—grounded in a formal MDP/POMDP abstraction that makes precise the distinction between traditional preference-based RL fine-tuning (PBRFT) and the emerging agentic paradigm, with each capability and task analyzed through the lens of what specific RL formulation (reward structure, action space design, credit assignment strategy) makes it work.
3.2 Big-Picture Architecture (Diagram in Words)
The survey's conceptual architecture has five major components:
-
Formal Foundation (Section 2): A mathematical framework that defines both PBRFT and Agentic RL as MDP/POMDP tuples
⟨S, O, A, P, R, T, γ⟩, making explicit the differences in state spaces (single prompt vs. evolving world states), action spaces (pure text vs. text + structured tool actions), transition dynamics (deterministic vs. stochastic), reward functions (scalar outcome vs. composite step-level + task-level), and learning objectives (single-step expectation vs. discounted cumulative return). This foundation provides the shared vocabulary for all subsequent analysis. -
Capability Taxonomy (Section 3): A set of six core agentic modules—planning, tool use, memory, self-improvement, reasoning, and perception—each analyzed through the lens of how RL transforms it from a static heuristic into an adaptive policy. For each capability, the survey traces an evolutionary progression (e.g., for tool use: ReAct-style prompting → tool-integrated reasoning with outcome-only RL → long-horizon multi-turn TIR with process-level credit assignment) and identifies the specific RL mechanisms (external guidance vs. internal policy optimization, sparse vs. dense rewards, token-level vs. trajectory-level credit assignment) that govern each stage.
-
Task Domain Taxonomy (Section 4): A mapping of seven major application domains—search/research agents, code agents, mathematical reasoning agents, GUI agents, vision agents, embodied agents, and multi-agent systems—each surveyed chronologically and methodologically. Each domain section follows a consistent structure: RL-free methods (prompting, SFT), outcome-based reward RL (sparse success signals), process-based reward RL (dense intermediate supervision), and hybrid approaches (search-guided expert iteration, self-play). A dependency matrix (Table 9) explicitly maps which capabilities are core, supporting, or minimal for each domain.
-
RL Algorithm Toolkit (Section 2.7): A compact reference for the four canonical RL algorithm families—REINFORCE, PPO, DPO, GRPO—each presented with its mathematical objective, key mechanism (variance reduction via baselines, policy ratio clipping, implicit reward from preference pairs, group-relative advantage estimation), and a comparison table (Table 2) of popular variants organized by family, objective type (policy gradient, preference optimization), and key mechanism (e.g., "decoupled clip and dynamic sampling" for DAPO, "reference policy reset" for ProRL).
-
Infrastructure Compendium (Section 5): A consolidated catalog of environments (organized by domain: web, GUI, coding/SWE, domain-specific, simulated/game, general-purpose) and RL frameworks (organized by type: agentic RL frameworks, RLHF/LLM fine-tuning frameworks, general-purpose RL frameworks), each with key features and resource links, enabling practitioners to select appropriate training infrastructure.
Information flows through the survey as follows: the formal foundation (Section 2) establishes what Agentic RL is → the capability taxonomy (Section 3) explains what capabilities RL optimizes and how → the task taxonomy (Section 4) demonstrates where these optimized capabilities are deployed and what results they achieve → the infrastructure compendium (Section 5) provides with what tools practitioners can build Agentic RL systems → open challenges (Section 6) identifies what remains unsolved.
3.3 Roadmap for the Deep Dive
The technical breakdown below follows this order, designed to build from abstract formalism to concrete implementation:
-
First, the formal MDP/POMDP distinction between PBRFT and Agentic RL (Section 2.1–2.7), because all subsequent analysis depends on understanding what fundamentally changes when we move from single-turn text generation to multi-turn interactive decision-making—this establishes the mathematical vocabulary (state, action, transition, reward, objective) used throughout the paper.
-
Second, the four canonical RL algorithm families (REINFORCE, PPO, DPO, GRPO) and their key variants, because these are the optimization engines that drive every capability and task surveyed later—understanding their mechanisms (clipping, group-relative advantage, implicit rewards) is prerequisite to understanding why specific design choices (e.g., GRPO for search agents, PPO for embodied agents) are made.
-
Third, the capability taxonomy (planning, tool use, memory, self-improvement, reasoning, perception), because this is the survey's central organizational contribution—each capability is analyzed through the lens of how RL transforms it, with the evolutionary progression from static heuristic to adaptive policy traced in detail. The order within capabilities follows a logical dependency: planning (high-level strategy) first, then tool use (action execution), memory (state persistence), self-improvement (policy refinement), reasoning (deliberate computation), and perception (multimodal grounding).
-
Fourth, the two-way relationship between capabilities and algorithms: for each capability, we examine not just that RL is applied, but which specific RL formulation (reward structure, credit assignment granularity, on-policy vs. off-policy training) is appropriate and why—this connects the capability taxonomy back to the formal foundation.
3.4 Detailed, Sentence-Based Technical Breakdown
This is primarily a conceptual synthesis paper whose core contribution is a unified taxonomy and formal framework for understanding Agentic RL, supported by extensive literature survey across over 500 papers. The "technical approach" is the set of conceptual tools, formal definitions, and organizational principles the paper develops to make sense of the field.
3.4.1 The Formal MDP/POMDP Distinction: PBRFT vs. Agentic RL
The survey's foundational technical contribution is the formalization of the distinction between traditional Preference-Based Reinforcement Fine-Tuning (PBRFT) and Agentic RL through the lens of Markov Decision Processes. Both paradigms are cast as instances of a seven-element tuple ⟨S, O, A, P, R, T, γ⟩, but the elements take fundamentally different forms in each case, directly determining what algorithms are applicable, what behaviors can emerge, and what evaluation metrics are meaningful.
PBRFT as a degenerate MDP. The survey characterizes PBRFT—the dominant paradigm in RLHF, DPO, and related alignment methods—as a degenerate single-step MDP:
where the observation space O is omitted because the state is fully observed (no partial observability).
State space. The state space $S_{\text{trad}} = \{\text{prompt}\}$ consists of a single static prompt input. The survey states: "In the training process, each episode starts from a single prompt state $s_0$; the episode terminates immediately after the model emits one response." This means the state is not a function of previous actions or environmental dynamics—it is the input prompt, fixed at the start of the episode. There is no notion of an evolving world state, no accumulation of intermediate observations, and no partial observability.
What this implies operationally: the LLM never observes the consequence of its own output within the same episode. It produces a response, receives a reward, and the episode ends. There is no feedback loop where the model's generation changes the state it will observe next. This fundamentally limits what can be learned: the model can learn to produce outputs that correlate with high reward, but it cannot learn sequential strategies, cannot learn to recover from errors, and cannot learn to acquire information through interaction.
Action space. The action space $A_{\text{trad}}$ consists of "pure text sequences"—the LLM generates natural language tokens via autoregressive decoding. There are no structured actions, no tool invocations, no environment interactions. The action is the complete output sequence, treated as a single decision even though it is generated token-by-token.
What this implies operationally: the "action" in PBRFT is monolithic—the entire response is treated as one choice, with no intermediate feedback possible. The RL optimization operates on the final, complete output; credit cannot be assigned to individual tokens or reasoning steps without external annotation (process reward models, step-level verifiers). This is why PBRFT has historically required either (a) a learned reward model that scores the complete response (RLHF) or (b) preference pairs that implicitly encode relative quality (DPO).
Transition dynamics. The transition function is deterministic and trivial: $P(s_1 | s_0, a) = 1$, where $s_1$ is the terminal state. The survey explicitly notes "there is no uncertainty." Once the action (text response) is selected, the next state is deterministically the terminal state—the episode ends.
What this implies operationally: there is no exploration in the traditional RL sense. The model does not need to consider that taking action $a$ from state $s_0$ might lead to state $s_1$ where further decisions are required. The "world" consists of exactly one decision point. This makes the optimization problem conceptually simple (maximize expected reward for a single choice) but also means the learned policy cannot generalize to settings where decisions have delayed consequences.
Reward function. The reward function is a single scalar assigned to the complete response: $R_{\text{trad}}(s_0, a) = r(a)$, where $r: A \to \mathbb{R}$ is a scalar score. This score may come from a learned reward model (trained on human preference data), a rule-based verifier (comparing to ground-truth answers), or an AI feedback model. The survey notes that regardless of the implementation, "its core follows the equation"—a single scalar for the complete output, with no intermediate feedback.
What this implies operationally: credit assignment is global—every token in the response receives the same reward signal regardless of its actual contribution to the outcome. A brilliant reasoning step followed by a careless arithmetic error receives the same negative reward as a response that was wrong from the start. This sparsity fundamentally limits sample efficiency: the model must explore a combinatorially large space of token sequences with only a single scalar signal per complete sequence to guide learning.
Learning objective. The optimization objective is to maximize expected reward under the policy:
where $\pi_\theta$ is the LLM parameterized by $\theta$. No discount factor is needed because there is only one step. The survey describes this as "maximum-expected-reward sequence modeling."
Agentic RL as a POMDP. In contrast, Agentic RL is modeled as a Partially Observable Markov Decision Process:
where $O$ is the observation function mapping states to observations: $o_t = O(s_t)$. The horizon $T > 1$ and discount factor $0 < \gamma < 1$ are general.
State space. The state $s_t \in S_{\text{agent}}$ represents the full world state—this could include the content of web pages visited, the state of a code repository, the visual layout of a GUI, the positions of objects in an embodied environment, the conversation history, and any intermediate reasoning or computation. The LLM agent does not observe $s_t$ directly; it receives observations $o_t = O(s_t)$ which may be partial, noisy, or high-dimensional (screenshots, text transcripts, API responses).
What this implies operationally: the agent must reason under uncertainty. It does not know the full state of the world; it must use observations to infer what is true, plan actions to acquire missing information, and maintain internal state (memory) to track what it has learned. The partial observability is not a minor detail—it is the defining challenge of agentic settings and what distinguishes them from the fully-observed PBRFT setting.
Action space. The action space is hybrid, comprising two distinct subspaces:
$A_{\text{text}}$ is the space of free-form natural language tokens emitted via autoregressive decoding—this corresponds to the LLM's text generation capability and serves roles such as reasoning aloud, communicating with users, or producing intermediate analysis.
$A_{\text{action}}$ is the space of structured, environment-interactive behaviors, typically delimited in the output stream by special tokens ⟨action_start⟩ and ⟨action_end⟩. These actions may invoke external tools (e.g., call("search", "Einstein")), interact with an environment (e.g., move("north")), or manipulate software (e.g., click(button_id)). The survey notes that $A_{\text{action}}$ is "recursively constructed, such that an element $a \in A_{\text{action}}$ may itself represent a sequence $(a_1, ..., a_k)$ of primitive actions, thus unifying primitive and composite actions within the same space."
What this implies operationally: the agent must learn not only what to say but when and how to act. A single policy must jointly model language generation and environment interaction, deciding at each time step whether to produce text (reasoning, communication) or execute an action (tool invocation, state modification). This interleaving is what enables the emergent behaviors—self-correction based on tool feedback, adaptive query refinement, backtracking after failed actions—that the survey documents across task domains.
The functional distinction between the subspaces is semantic: $A_{\text{text}}$ defines outputs intended for interpretation without directly altering external state, while $A_{\text{action}}$ defines behaviors that either (i) acquire new information through tool invocations or (ii) modify the state of a physical or simulated environment.
Transition dynamics. The environment evolves under uncertainty:
What this equation computes: the probability distribution over next states given the current state and chosen action. When $a_t$ is a text action, the environment state typically does not change (though the agent's internal context—conversation history, working memory—does). When $a_t$ is a structured action, the environment transitions to a new state reflecting the action's effect (a web page loads, a robot arm moves, a code file is modified).
Why this form: this is the standard MDP transition formulation, but the hybrid action space makes it richer than standard RL settings. The stochasticity $\sim P$ is crucial—in real environments, tool calls may fail, web pages may load differently, and physical actions may have uncertain outcomes. A deterministic transition function would be inadequate for modeling these settings. The survey emphasizes that this "sequential formulation contrasts with the one-shot mapping of PBRFT, enabling policies that iteratively combine communication, information acquisition, and environment manipulation."
Reward function. The reward function is composite, supporting multiple granularities:
What this equation computes: a scalar reward signal that may be sparse (non-zero only on task completion, e.g., unit test passes, correct final answer), dense (step-level progress signals, e.g., compilation success, intermediate assertion passes), or learned (estimated by a reward model from environmental observations).
Why this form: the composite structure reflects a key design choice in Agentic RL that has no analogue in PBRFT. Purely sparse outcome rewards suffer from the credit assignment problem—in a 50-step trajectory, only the final step receives a non-zero reward, making it extremely difficult to learn which intermediate actions were good or bad. Purely dense rewards risk reward hacking—the agent optimizes for the intermediate signal rather than the true objective. The composite form allows practitioners to balance these concerns: use sparse task rewards to define the true objective, add dense sub-rewards to guide exploration, and zero-out rewards when neither applies to avoid providing misleading signals.
The survey explicitly acknowledges that different task domains use different instantiations: "unit-test passes" for code agents, "symbolic verifier success" for math agents, "API call success" for tool-use agents, and "task completion flags" for embodied agents.
Learning objective. The optimization objective is to maximize discounted cumulative reward:
What this equation computes: the expected sum of discounted rewards over a trajectory of length $T$, where the trajectory $\tau = (s_0, a_0, s_1, a_1, ..., s_{T-1}, a_{T-1})$ is sampled by rolling out the policy $\pi_\theta$ in the environment, and $\gamma$ is the discount factor controlling the tradeoff between immediate and future rewards.
Why this form: the discounted cumulative return is the standard RL objective for sequential decision-making. The discount factor $\gamma < 1$ has both mathematical (ensuring convergence in infinite-horizon settings) and practical (prioritizing near-term rewards over distant ones, which is often appropriate when outcomes become increasingly uncertain over long horizons) motivations. This contrasts with PBRFT's $\gamma = 1$, which is possible only because there is no temporal structure to discount.
The survey explicitly frames the shift in optimization: "PBRFT focuses on single-turn text quality alignment without explicit planning, tool use, or environmental feedback, while Agentic RL involves multi-turn planning, adaptive tool invocation, stateful memory, and long-horizon credit assignment, enabling the LLM to function as an autonomous decision-making agent."
3.4.2 The Four Canonical RL Algorithm Families
The survey devotes Section 2.7 to a compact but technically precise presentation of the four algorithm families that power both PBRFT and Agentic RL, with variants summarized in Table 2. This is not a methodological contribution but a reference framework—it establishes the algorithmic toolkit that the subsequent capability and task analyses draw upon.
REINFORCE: The Foundational Policy Gradient. The survey presents REINFORCE (Williams, 1992) as the simplest policy gradient algorithm, providing the theoretical foundation for more sophisticated methods. The objective gradient is:
where:
$a^{(i)} \sim \pi_\theta(a | s_0)$is the$i$-th sampled response from the policy,$R(s_0, a^{(i)})$is the reward received for that response,$b(s_0)$is a baseline function (commonly the value function$V(s_0)$) used to reduce variance,$N$is the number of samples used to estimate the gradient.
What it computes: an unbiased estimate of the policy gradient—the direction in parameter space that increases the probability of actions that led to higher-than-baseline rewards and decreases the probability of actions that led to lower-than-baseline rewards. The baseline $b(s_0)$ subtracts the expected reward, leaving the advantage $R - b$ which indicates whether a particular action was better or worse than average.
Why this form: the log-derivative trick ($\nabla_\theta \log \pi$) converts the expectation over actions into a differentiable form that can be estimated by sampling. The baseline reduces variance without introducing bias (because $\mathbb{E}[\nabla_\theta \log \pi \cdot b] = 0$ when $b$ does not depend on the action). Without a baseline, REINFORCE suffers from extremely high variance—gradient estimates can fluctuate wildly because the raw reward magnitude (e.g., 0 or 1) is used directly rather than being centered. The survey notes REINFORCE's drawbacks: "high variance in gradient estimates, sample inefficiency, sensitivity to learning rate and the lack of a critic (value estimator)."
Proximal Policy Optimization (PPO). PPO (Schulman et al., 2017) is presented as the dominant RL algorithm for LLM alignment due to its stability. Its core mechanism is policy ratio clipping—limiting how much the updated policy can differ from the old policy to prevent destructively large updates. The objective is:
where:
$\pi_\theta$is the current policy,$\pi_{\theta_{\text{old}}}$is the old (pre-update) policy,- The ratio
$\frac{\pi_\theta}{\pi_{\theta_{\text{old}}}}$measures the relative probability of action$a^{(i)}_t$under the new vs. old policy, $A(s_t, a^{(i)}_t) = R(s_t, a^{(i)}_t) - V(s_t)$is the estimated advantage (how much better the action was than the value function's prediction),$V(s_t)$is the learned value function (a critic network typically the same size as the policy network),$\epsilon$is the clipping hyperparameter (typically 0.1 or 0.2).
What it computes: for each action, if the advantage $A$ is positive (the action was good), the objective encourages increasing the probability ratio $\frac{\pi_\theta}{\pi_{\theta_{\text{old}}}}$, but clips it at $1 + \epsilon$ to prevent the new policy from becoming too different from the old one. If the advantage is negative, the objective encourages decreasing the ratio, but clips it at $1 - \epsilon$. The $\min$ operation ensures that the objective is a pessimistic bound—it uses whichever term (clipped or unclipped) gives the lower value, preventing the policy from exploiting the objective function by making large changes that the clipping would otherwise reward.
Why this form: straight policy gradient can produce destructively large updates—a single batch of high-reward trajectories can cause the policy to collapse to a narrow mode, losing diversity and making further learning impossible. The clipping mechanism provides a form of trust-region constraint that is simpler to implement than the original TRPO (Schulman et al., 2015) while achieving similar stability. The survey notes that recent variants "have concentrated on reducing the bias (Kazemnejad et al., 2024) or variance (Yue et al., 2025) in the advantage estimation" and "making improvements from the perspectives of stable policy update mechanisms (Liu et al., 2025) or mitigating sparse rewards (Dai et al., 2025)." The main drawback is reliance on a separate critic network, which "substantially increases the parameter count during training."
Direct Preference Optimization (DPO). DPO (Rafailov et al., 2023) bypasses explicit reward modeling entirely by casting preference alignment as a likelihood-based objective on human preference data. The loss is:
where:
$\mathcal{D} = \{(y_w, y_l)\}$is a dataset of preference pairs where$y_w$is the preferred (winning) response and$y_l$is the dispreferred (losing) response,$\pi_\theta$is the policy being optimized,$\pi_{\text{ref}}$is a frozen reference policy (usually the initial SFT model),$\beta$is a hyperparameter controlling the strength of the KL constraint,$\sigma$is the logistic sigmoid function.
What it computes: a binary cross-entropy loss where the model's own log-probability ratios (relative to the reference policy) serve as the implicit reward. The term $\beta \log \frac{\pi_\theta(y_w | x)}{\pi_{\text{ref}}(y_w | x)}$ is the implicit reward for the winning response—higher when the policy assigns higher probability to $y_w$ relative to what the reference policy would assign. Similarly for the losing response. The sigmoid converts the difference between these implicit rewards into a probability that $y_w$ is preferred, and the loss maximizes this probability.
Why this form: by directly optimizing preferences without an intermediate reward model, DPO avoids the complexity of reward model training, distribution shift, and reward hacking that can plague RLHF pipelines. However, the survey notes that its "performance is intrinsically tied to the quality and coverage of its static preference dataset"—it cannot improve beyond what the preference data covers, unlike online RL methods that can discover novel high-reward behaviors through exploration.
Variants are organized by their key innovation: $\beta$-DPO (dynamic KL coefficient), SimPO (uses average log probability as implicit reward, removing the reference model), IPO (avoids overfitting by regularizing toward a preference policy via log-likelihood ratio gap control), KTO (knowledge transfer from teacher models), ORPO (online preference optimization), Step-DPO (step-wise supervision extending DPO to multi-step reasoning), and LCPO (length-aware preference optimization with limited data).
Group Relative Policy Optimization (GRPO). GRPO (DeepSeek-AI et al., 2025) is the most recent addition and has catalyzed significant research interest following DeepSeek-R1's success. It eliminates PPO's critic network by computing advantages from relative rewards within a group of responses. The objective is:
where:
$G$is the group size—$G$responses are sampled for each prompt,- The advantage is estimated by group-relative normalization:
What this computes: for each response in a group of $G$ independently sampled responses to the same prompt, the advantage is the standardized reward—how many standard deviations above or below the group mean. This replaces the absolute value function $V(s)$ used in PPO with a relative comparison: a response that achieves reward 0.8 is "good" if the group mean is 0.5, but "bad" if the group mean is 0.95.
Why this form: by eliminating the critic network, GRPO reduces the parameter count during training (no separate value network) and simplifies the architecture. The group-relative advantage is inherently normalized (zero mean, unit variance within each group), which helps stabilize training across prompts with different reward scales. However, the survey notes a key vulnerability: "the group-based advantage estimation is vulnerable to high variance and low accuracy" when the group size is small or rewards are noisy. This has motivated variants like DAPO (decoupled clip, dynamic sampling), GSPO (sequence-level clipping based on sequence likelihood), Posterior-GRPO (rewards only successful reasoning processes), and Dr.GRPO (corrects bias in the GRPO optimization objective). Table 2 provides a comprehensive comparison of variants from the PPO, DPO, and GRPO families, each characterized by its objective type and key mechanism.
3.4.3 RL for Planning: External Guidance vs. Internal Policy Optimization
The survey identifies two distinct paradigms for integrating RL into agent planning, distinguished by whether RL operates on the LLM's parameters or on an auxiliary search component (Section 3.1).
RL as an External Guide for Planning. In this paradigm, the LLM's parameters are frozen. The LLM serves as a knowledge-rich action proposer—it generates candidate actions or plans, but does not learn from outcomes. RL is instead used to train an auxiliary value or heuristic function that evaluates the quality of proposed actions, guiding a classical search algorithm like Monte Carlo Tree Search (MCTS).
The survey cites RAP (Hao et al., 2023) and LATS (Zhou et al., 2024) as representative examples. The mechanism operates as follows: the LLM proposes possible next actions or reasoning steps given the current state; the RL-trained value function (or heuristic) scores each proposal; a search algorithm (MCTS) selects which branch to explore based on these scores; the search proceeds until a solution is found or a budget is exhausted. The RL training happens offline—the value function is trained on successful and failed trajectories collected from prior searches—and the LLM is never updated.
Planning without Search (Hong et al., 2025) extends this idea by using offline goal-conditioned RL to learn a language-based value critic that guides LLM reasoning without needing online search. The critic is trained to predict, given a partial reasoning state and a goal description, the expected reward of continuing from that state. At inference time, the critic scores candidate continuations and the LLM selects the highest-scoring one.
Why this design: external guidance separates the knowledge problem (what actions are possible?) from the evaluation problem (which actions are good?). The LLM handles the former—drawing on its pretraining to propose plausible actions—while RL handles the latter—learning from environmental feedback which proposals actually lead to success. This separation means the LLM's knowledge remains intact (no risk of catastrophic forgetting) and the RL component can be much smaller and faster to train.
The survey also notes a complementary direction: Learning When to Plan (Paglieri et al., 2025) formulates the decision of whether to invoke planning at all as an RL problem. The agent learns a meta-policy that, given a task, decides whether to reason quickly (fast, intuitive response) or deploy explicit planning (slower, more deliberate search). The reward function balances task accuracy against computational cost, training the agent to allocate planning effort efficiently.
RL as an Internal Driver of Planning. In this paradigm, the LLM's parameters are directly updated through RL. The LLM is cast as a policy model that generates plans, and RL-based feedback from environmental interaction is used to refine this policy.
The mechanism operates through trial and error: the LLM (as policy) produces a plan, executes it in the environment, receives a reward (task success, partial progress, or failure), and the policy parameters are updated via policy gradient to increase the probability of plans that led to high rewards. The survey cites ETO (Song et al., 2024) as an example using DPO on successful vs. failed trajectories—preference pairs are constructed from positive and negative planning trajectories, and the LLM is updated to prefer the successful ones.
VOYAGER (Wang et al., 2024) exemplifies a more sophisticated internal-driver approach through lifelong learning. The agent (in Minecraft) iteratively: generates a plan using the LLM, executes it, records the trajectory, and—if successful—extracts the trajectory as a reusable skill stored in a skill library. The skill library grows over time, and the LLM learns to compose previously acquired skills to solve increasingly complex tasks. Here, RL is not applied as a single optimization step but as a continuous curriculum: the agent's own successful behaviors become training data for future improvements.
RLTR (Li et al., 2025) introduces a specific formulation: it decouples planning from answer generation. The agent first produces a plan (a sequence of tool-use actions), then separately generates the answer based on the plan's execution. Tool-use rewards directly evaluate the action sequence quality—did the tools return useful information? Did the execution succeed?—enabling focused optimization of planning capability without requiring the final answer to be verifiable.
Why this design: internal-policy optimization allows the LLM to learn from its own mistakes. Unlike external guidance (where the LLM's proposals remain unchanged), RL-driven internal optimization means the LLM gets better at proposing good plans over time. This enables generalization: a plan that worked for one task may be partially transferable to similar tasks, and the LLM can learn the abstract properties of good plans (e.g., decompose into independent subgoals, verify intermediate results) rather than memorizing specific plan templates.
The survey identifies Planner-R1 (Zhu et al., 2025) as examining the effect of reward density on planning. It shows that "shaped, process-level rewards markedly improve learning efficiency and enable smaller models to attain competitive planning capability"—a key empirical finding linking the reward structure choice (Section 2.5) to the planning capability.
Prospective synthesis. The survey articulates a forward-looking vision: the distinction between external search and internal policy optimization should dissolve. An ideal agent would "internalize the structured search process itself, seamlessly blending intuitive, fast plan generation with deliberate, slow, deliberative reasoning." RL would optimize a meta-policy governing the deliberation process—learning when to explore alternative paths, how deeply to reason, and when to commit to an action—rather than just refining the final plan output.
3.4.4 RL for Tool Use: From Imitation to Outcome-Driven Optimization
The survey traces tool use through three evolutionary stages, each representing a different learning paradigm (Section 3.2).
Stage 1: ReAct-style Tool Calling (Imitation-Based). Early tool-use systems relied on either prompt engineering or supervised fine-tuning. In the prompt engineering approach (ReAct, Yao et al., 2023), few-shot exemplars demonstrate a "Thought-Action-Observation" cycle—the LLM sees examples of interleaving reasoning ("I need to find the population of Tokyo") with actions ("Search[Tokyo population]") and observations ("The population is 13.96 million"), and learns via in-context learning to produce similar patterns.
In the SFT approach, models are fine-tuned on datasets of tool-interaction trajectories. Toolformer (Schick et al., 2023) used a self-supervised objective: given a corpus of text, the model learns to predict where API calls should be inserted (e.g., "The population of Tokyo is [API: population(Tokyo)] 13.96 million"). AgentTuning (Zeng et al., 2024) and Agent-FLAN (Chen et al., 2024) fine-tuned on expert-generated collections of tool-use trajectories. The key limitation in both cases: the model learns to replicate static patterns rather than discovering optimal tool-use strategies. It cannot adapt when tools fail, cannot learn that certain tools work better in certain contexts, and cannot discover novel tool compositions.
Stage 2: Tool-Integrated RL (Outcome-Driven Optimization). This stage shifts the learning objective from imitation to performance optimization. RL is applied with outcome-based rewards—the agent receives a positive signal when the task is completed successfully and zero otherwise—and must discover effective tool-use strategies through exploration.
The survey highlights ToolRL (Qian et al., 2025) as foundational: starting from a base model without any imitation traces, RL training elicits emergent capabilities including "self-correction of faulty code, adaptive adjustment of invocation frequency, and the composition of multiple tools for complex sub-tasks." This is significant because it demonstrates that tool use can emerge from pure outcome optimization without explicit tool-use demonstrations—the RL signal alone is sufficient to guide the model toward discovering that invoking a code interpreter when stuck, or searching for information when uncertain, increases the probability of eventual success.
Subsequent work in this stage—OTC-PO (Wang et al., 2025), ReTool (Feng et al., 2025), AutoTIR (Wei et al., 2025), VTool-R1 (Wu et al., 2025), DeepEyes (Zheng et al., 2025), ARTIST (Singh et al., 2025), ToRL (Li et al., 2025)—employs RL policies that interleave symbolic computation (code execution, image editing, API calls) with natural-language reasoning within a single rollout. The mechanism: at each step, the policy can either generate reasoning text or execute a tool action; tool execution produces results that are fed back into the policy's context; the policy continues reasoning based on these results. The RL optimization operates over the entire interleaved trajectory, with rewards determined by final task success.
What makes this interleaving powerful: the agent can use reasoning to decide when to invoke a tool, use tool results to inform subsequent reasoning, and use reasoning to interpret and correct tool failures. This creates a tight feedback loop that pure SFT cannot capture—SFT can only teach the model to reproduce the specific tool-use patterns present in the training data, while RL can discover patterns that the training data didn't contain.
The survey notes that Lin & Xu (2025) provides theoretical grounding: they "theoretically prove that TIR fundamentally expands LLM capabilities beyond the 'invisible leash' of pure-text RL by introducing deterministic tool-driven state transitions." The intuition: pure text generation (even with chain-of-thought) operates in a purely autoregressive setting where each token is predicted based on previous tokens; tool use introduces deterministic, externally-verified computations that can break the autoregressive constraint (e.g., code execution provides ground-truth results that don't depend on the model's probability distribution).
The survey observes that this stage has produced mature commercial and open-source systems: "OpenAI's DeepResearch and o3, Kimi K2, Qwen QwQ-32B, Zhipu GLM Z1, Microsoft rStar2-Agent and Meituan LongCat routinely incorporate these RL-honed strategies."
Stage 3 (Prospective): Long-Horizon Multi-Turn TIR. The survey identifies the primary frontier as extending tool-integrated RL to tasks requiring many rounds of tool interaction where early decisions have delayed consequences. The bottleneck is temporal credit assignment: "Current RL approaches often depend on sparse, trajectory-level/outcome-based rewards, making it difficult to pinpoint which specific tool invocation in a long, interdependent sequence contributed to success or failure."
The challenge is illustrated by a hypothetical: a 30-step tool-use trajectory succeeds at step 30. Which of the first 29 steps were essential? Which were irrelevant? Which were actually harmful but happened to not prevent eventual success? Outcome-only rewards provide the same signal to every step, making exploration inefficient—the agent cannot distinguish between an action that was necessary for success and one that was merely harmless.
The survey cites early attempts at addressing this: GiGPO (Feng et al., 2025) introduces turn-level advantage estimation—rather than a single trajectory-level reward, the advantage is computed per interaction turn based on the trajectory prefix up to that turn. SpaRL (Wang et al., 2025) decomposes delayed rewards into per-step signals using a learned credit assignment model. However, the survey characterizes these as "early steps" and identifies the development of "more granular credit assignment mechanisms that can accurately guide the agent through complex decision chains without inadvertently punishing useful exploration or promoting reward hacking" as "a critical and largely unsolved problem."
3.4.5 RL for Memory: From Passive Retrieval to Active Management
The survey traces memory through four phases, with RL playing an increasingly central role (Section 3.3).
Phase 1: RAG-Style Memory (Passive, Predefined). Early memory systems treated memory as an external datastore with fixed management rules. MemoryBank (Zhong et al., 2024) used vector databases for storage with retrieval based on semantic similarity. MemGPT (Packer et al., 2023) implemented OS-like memory management with predefined policies. HippoRAG (Gutierrez et al., 2024) constructed knowledge graphs with heuristic-based access patterns. In all cases, the agent had no control over what to remember, when to retrieve, or how to update—these were hand-designed rules.
Phase 2: RL-Guided Retrieval. The first application of RL to memory focused on when to query the memory store, while the memory content itself remained static. The survey's representative example is Prospect (Tan et al., 2025), where an RL policy adjusts retrieval behavior through two mechanisms: "prospective reflection" (multi-level summarization—the agent summarizes recent interactions at different levels of abstraction, creating hierarchical memory representations) and "retrospective reflection" (reinforcing retrieval outcomes—if retrieving a particular memory led to a successful interaction, the policy is updated to make similar retrievals in the future).
Memory-R1 (Yan et al., 2025) introduced a more active model: a dedicated Memory Manager component learns, via PPO or GRPO, to perform structured operations (ADD, UPDATE, DELETE, NOOP—do nothing) on the memory store. The reward signal comes from downstream QA performance—if adding a piece of information to memory improves the agent's ability to answer questions, the ADD operation is reinforced. An Answer Agent then uses a Memory Distillation policy to reason over RAG-retrieved entries.
Phase 3: RL for Token-Level Memory (Trainable Controllers). This phase introduces agents that regulate their own memory states stored in token form, without relying on fixed external memory systems. The survey distinguishes two instantiations.
Explicit token memory: The memory is maintained as human-readable natural language. MemAgent (Yu et al., 2025) maintains a natural-language memory pool alongside the LLM. An RL policy determines, at each interaction segment, which tokens to retain or overwrite in this pool—effectively learning to compress long-context inputs into concise, informative summaries. The RL objective: maximize task performance while keeping the memory pool within a fixed size constraint. Similarly, ReSum (Wu et al., 2025) uses RL to produce turn-wise interaction summaries for ReAct agents, and Context Folding (Sun et al., 2025) applies RL to fold long contexts into compressed representations.
Implicit (latent) token memory: Memory is maintained in the form of latent embeddings that are not tied to human-readable text. MemoryLLM (Wang et al., 2024) introduces a fixed set of "memory tokens"—a small number of special embedding vectors that are prepended to the LLM's input. As the context evolves, these tokens are repeatedly retrieved (integrated into the LLM's forward computation via attention), updated (the LLM's output at the memory token positions becomes the new memory representation), and carry information across interaction turns. M+ (Wang et al., 2025) extends this by scaling to more memory tokens for long-context tracking. The key property: the memory tokens are machine-native—they can represent information in forms optimized for the LLM's internal computation rather than being constrained to human-interpretable formats.
MemGen (Zhang et al., 2025) extends this paradigm by using latent memory tokens not just for storage but for generating experiential knowledge—the memory tokens serve as carriers that can produce knowledge relevant to the current context, enabling the agent to draw on past experiences without explicit retrieval.
Phase 4 (Prospective): RL for Structured Memory. The survey identifies structured memory representations—temporal knowledge graphs (Zep, Rasmussen et al., 2025), atomic memory notes (A-MEM, Xu et al., 2025), hierarchical graphs (G-Memory, Zhang et al., 2025; Mem0, Chhikara et al., 2025)—as the next frontier. Currently, management of these structures (insertion, deletion, abstraction, linkage updates) uses handcrafted rules. The survey identifies "little work has explored the use of RL to dynamically control the construction, refinement, or evolution of such structured memory, making this an open and promising direction."
3.4.6 RL for Self-Improvement: Verbal, Internalized, and Iterative
The survey identifies three levels of RL-driven self-improvement, distinguished by whether the improvement mechanism operates only at inference time and whether gradient updates are involved (Section 3.4).
Level 1: Verbal Self-Correction (Inference-Only, No Gradient Updates). The agent generates an answer, linguistically reflects on potential errors, and produces a refined solution—all within a single forward pass without parameter updates. The survey terms this "verbal reinforcement learning" because the reflection process mimics RL's trial-and-error loop but operates entirely at the language level.
The mechanism in Self-Refine (Madaan et al., 2023): the LLM is prompted with three distinct templates—one for initial generation, one for generating feedback on the initial output, and one for refining based on feedback. The process iterates for a fixed number of rounds, with the LLM seeing its own previous output and the generated feedback as context for the refinement step.
The survey identifies three strategies for enhancing verbal self-correction:
(I) Multiple sampling: Generate multiple output rollouts and aggregate critiques or solutions. If-or-Else (Li et al., 2024) generates multiple candidate solutions, critiques each, and selects the one with the most consistent internal reasoning. UALA (Han et al., 2024) samples multiple reflection trajectories and uses consistency as a quality signal.
(II) Structured reflection workflows: Decompose the reflection process into dedicated stages rather than prompting for a monolithic self-critique. Chain-of-Verification (He et al., 2024) manually separates the process into "Retrieving, Rethinking, and Revising"—the agent first retrieves relevant facts, then rethinks its answer in light of those facts, then revises accordingly.
(III) External guidance: Ground reflection in verifiable, objective feedback by incorporating external tools. Self-Debugging (Chen et al., 2024) uses a code interpreter to execute the generated code and provide error messages, which the LLM then uses to debug. T1 (Kang et al., 2025) uses a mathematical calculator to verify intermediate computations. Step-wise reward models (Xiong et al., 2025) score individual reasoning steps for correctness, providing dense feedback for reflection.
Level 2: Internalizing Self-Correction (Gradient-Based Updates). While verbal self-correction is ephemeral (the improvement disappears after the session ends), this level uses RL to internalize reflective feedback into the model's parameters, permanently improving its ability to identify and correct errors.
The survey identifies two training approaches. DPO-based methods: KnowSelf (Qiao et al., 2025) uses DPO and RPO to enhance agents' self-reflection in text-based games. The preference data is constructed from the agent's own interaction trajectories—successful reflection-correction sequences are preferred over unsuccessful ones. Reflection-DPO (Patel et al., 2025) applies this in user-agent interaction, training agents to better infer user intent through reflective reasoning.
Critic-based methods: SWEET-RL (Zhou et al., 2025) and ACC-Collab (Estornell et al., 2025) train an external critic model to provide revision suggestions for the actor agent's actions. This separates the reflection capability from the acting capability—the critic learns to identify errors and suggest improvements, while the actor learns to incorporate those suggestions. The RL training can operate on both components: the critic is trained to produce suggestions that lead to improved outcomes, and the actor is trained to follow the critic's guidance effectively.
Level 3: Iterative Self-Training (Autonomous Bootstrapping). The most advanced level involves fully autonomous loops where the agent generates its own training data through interaction, learns from outcomes, and iteratively improves—without human-labeled data.
(I) Self-play and search-guided refinement: Analogous to AlphaZero. R-Zero (Huang et al., 2025) employs MCTS to explore a reasoning tree, using the search results to iteratively train both a policy LLM (the actor—generates candidate reasoning steps) and a value LLM (the critic—evaluates the quality of partial reasoning states). The policy proposes actions, MCTS explores consequences, successful trajectories are used to improve both components, and the improved components enable more effective search in the next iteration.
The ISC framework (Tian et al., 2024) operationalizes a cycle of "Imagination, Searching, and Criticizing"—the agent generates potential solution paths (imagination), uses search to explore them (searching), applies a critic to evaluate them (criticizing), and uses the critic's feedback to refine its reasoning strategy.
(II) Execution-guided curriculum generation: The agent creates its own problems and learns from verifiable outcomes. Absolute Zero (Zhao et al., 2025) proposes its own tasks, attempts solutions, verifies them via execution (e.g., code execution, mathematical calculation), and uses outcome-based rewards to refine its policy. Self-Evolving Curriculum (Chen et al., 2025) frames problem selection as a non-stationary bandit task—the agent learns to select problems at the right difficulty level to maximize learning gains, generating an adaptive curriculum.
TTRL (Zuo et al., 2025) applies this at test time: for a single problem, it uses execution-based rewards to rapidly fine-tune a temporary copy of the agent's policy, generates the answer with this specialized policy, and then discards the temporary copy. This is not permanent learning but task-specific adaptation.
(III) Collective bootstrapping: Multiple agents share experiences to accelerate learning. SiriuS (Zhao et al., 2025) constructs a repository of successful reasoning trajectories from multi-agent interactions, and uses this growing knowledge base to train individual agents. MALT (Motwani et al., 2025) uses a three-agent setup where agents observe each other's successes and failures.
Prospective: Meta-Evolution of Reflection. The survey envisions RL applied at a meta-level to optimize the reflection process itself. Rather than using a fixed reflection strategy (e.g., always do verbal self-correction followed by execution-guided verification), the agent learns a meta-policy that selects the appropriate reflection strategy based on task characteristics, available resources, and past experience—deciding when a quick verbal check is sufficient versus when costly execution-guided search is warranted.
3.4.7 RL for Reasoning: Fast vs. Slow, and the Integration Challenge
The survey frames reasoning through the dual-process theory (Kahneman, 2011): fast reasoning (System 1—intuitive, pattern-driven, low latency) and slow reasoning (System 2—deliberate, multi-step, higher accuracy). The key insight is that RL is the primary mechanism for inducing slow reasoning in LLMs (Section 3.5).
Fast reasoning: Most conventional LLMs operate in this mode—reasoning is implicitly encoded in next-token prediction. The model produces a response directly without explicit intermediate reasoning steps. The survey notes that this is "quick, intuitive, and pattern-driven" but "vulnerable to factual errors, biases, and shallow generalization." Research to improve fast reasoning focuses on internal mechanisms (e.g., confidence estimation to identify unreliable predictions; Lightman et al., 2023) and external verification frameworks, but these risk "algorithmic adaptivity issues and can easily fall into the complexity trap."
Slow reasoning: This mode explicitly produces intermediate reasoning traces—chain-of-thought, multi-step verification, reasoning-augmented RL. The survey characterizes the output structure: "clear exploration and planning structure, frequent verification and checking behaviors, and generally longer inference lengths and times." The canonical examples are OpenAI's o1 and o3 series, DeepSeek-R1, and methods incorporating dynamic test-time scaling or RL for reasoning.
RL's role in slow reasoning: The survey identifies RL as the mechanism for inducing slow reasoning behaviors. DeepSeek-R1 demonstrated that pure RL training with verifiable rewards (without explicit supervision for chain-of-thought structure) could cause the model to spontaneously produce extended reasoning traces including verification, backtracking, and self-correction. This is significant because it suggests these behaviors are not explicitly programmed but emerge from the optimization pressure to produce correct answers—the model discovers that "thinking longer" and "checking its work" increases the probability of reward.
The survey distinguishes several approaches to constructing slow reasoning through RL:
- Structured agentic search: Macro-o1, HuatuoGPT-o1, AlphaMath use MCTS-style search to explore reasoning trees, training the policy from search results.
- Long-CoT dataset construction: HiICL-MCTS, LLaVA-CoT, rStar-Math, ReasonFlux generate long chain-of-thought datasets that embody deliberative thinking patterns, which are then used for supervised training.
- Latent space reasoning: Some approaches (Zhang et al., 2025; Hao et al., 2024) perform reasoning in continuous latent representations rather than discrete token sequences, enabling parallel exploration of reasoning trajectories.
Prospective: Integrating slow reasoning into agentic settings. The survey identifies two key challenges. Training stability: "Reinforcement learning in agentic scenarios faces greater challenges in training stability, such as ensuring compatibility with diverse environments"—the reward structure and state representation must work across different tools, tasks, and domains. Overthinking: Slow reasoning models may produce unnecessarily long chains of thought for simple problems, wasting computation. The survey points to hybrid strategies (adaptive test-time scaling) that aim to combine the efficiency of fast reasoning with the rigor of slow reasoning—letting the model decide, based on task difficulty, whether to respond quickly or engage in extended deliberation.
3.4.8 RL for Perception: From Passive Recognition to Active Visual Cognition
The survey traces the application of RL to multimodal (vision-language) models, identifying a progression from passive perception to active cognition (Section 3.6).
From Passive Perception to Active Visual Cognition. Early work applied RL to vision-language models primarily for enhancing passive perception—producing extended CoT sequences that improve visual understanding. Visual-RFT (Liu et al., 2025) and Reason-RFT (Tan et al., 2025) directly adapted GRPO to the vision domain, incorporating vision-specific metrics like IoU (Intersection over Union, measuring bounding box overlap) as verifiable reward signals. STAR-R1 (Li et al., 2025) introduced partial rewards tailored for visual GRPO—rather than a single reward for the final answer, intermediate visual reasoning steps (e.g., correctly locating an object) receive partial credit.
Subsequent work—Vision-R1 (Huang et al., 2025), VLM-R1 (Shen et al., 2025), LMM-R1 (Peng et al., 2025), MM-Eureka (Meng et al., 2025)—developed specialized policy optimization algorithms for step-wise visual reasoning, demonstrating strong performance even on small 3B-parameter models.
The more recent and significant shift is toward active visual cognition—treating visual representations as manipulable and verifiable intermediate thoughts, not just as static inputs to be perceived. The survey identifies three mechanisms:
Grounding-driven active perception: The LVLM repeatedly looks back at the image during reasoning, anchoring each reasoning step to specific image regions via bounding boxes. GRIT (Fan et al., 2025) interleaves bounding-box tokens with textual CoT and uses GRPO with both verifiable rewards and bounding-box correctness as supervision. Ground-R1 (Cao et al., 2025) highlights evidence regions (via IoU-based rewards) prior to text-only reasoning. DeepEyes (Zheng et al., 2025) demonstrates that end-to-end RL can naturally induce grounding behaviors without explicit grounding supervision.
Tool-driven active perception: External tools—code snippets, runtime environments, visual manipulation operations—assist the model's cognitive workflow. VisTA (Huang et al., 2025) and VTool-R1 (Wu et al., 2025) teach models to select and use visual tools through RL. Pixel Reasoner (Su et al., 2025) expands the action space with operations like crop, erase, and paint, using curiosity-driven rewards to discourage premature termination of exploration. The reward encourages the agent to explore the image thoroughly rather than settling for a quick (potentially incorrect) answer.
Generation-driven active perception: The LVLM generates sketches or images interleaved with CoT reasoning, externalizing intermediate representations. Visual Planning (Xu et al., 2025) uses imagined image rollouts as CoT, with downstream task success as the reward. GoT-R1 (Duan et al., 2025) applies RL within a Generation-CoT framework, letting models discover semantic-spatial reasoning plans before producing the final image. T2I-R1 (Jiang et al., 2025) decouples the process into semantic-level CoT (high-level planning) and token-level CoT (patch-wise pixel generation), jointly optimizing both stages with RL.
The survey extends the analysis to audio (reasoning enhancement for audio-language models, fine-grained optimization for speech synthesis) and notes the applicability of similar RL principles across modalities.
3.4.9 The Application-Capability Mapping
The survey's organizational contribution culminates in Table 9, which provides a qualitative dependency matrix mapping six core capabilities (planning, tool-use, memory, self-improvement, reasoning, perception) onto seven task domains (search, code, math, GUI, vision, embodied, multi-agent systems). Each cell is marked as:
- Core (•): The capability is essential for the domain—without it, the agent cannot function effectively.
- Supporting (◦): The capability contributes to performance but is not strictly essential.
- Minimal (–): The capability plays a negligible role in the domain.
The matrix serves as a navigation aid and research roadmap. For example, search agents depend on planning and tool use (core) but minimally on perception—suggesting research should prioritize RL formulations for multi-turn query decomposition and tool invocation optimization. GUI agents depend on planning, tool use, and perception (all core)—requiring multimodal RL formulations that jointly optimize visual grounding and action selection. The matrix enables practitioners to identify which capabilities their domain requires and, by consulting the corresponding capability section, which RL formulations are appropriate.
4. Key Insights and Innovations
Innovation 1: The Formal POMDP Distinction as a Diagnostic Instrument Rather Than Merely a Definition
The most intellectually distinctive move in this survey is the decision to formalize the distinction between PBRFT and Agentic RL through the lens of Markov Decision Processes—and, crucially, to treat this formalization not as a passive definition but as an active diagnostic tool that explains why certain methods succeed or fail across domains. Prior surveys on LLM agents (Luo et al., 2025; Plaat et al., 2025; Li, 2025; Masterman et al., 2024) organized their taxonomies around capabilities or architectures, but none provided a formal language that connects the structure of a problem (its state space, action space, transition dynamics, reward function) to the algorithm design choices (credit assignment granularity, reward density, on-policy vs. off-policy training) needed to solve it. The survey's MDP/POMDP abstraction fills this gap.
What makes this a conceptual advance rather than a definitional exercise. Any survey can define terms; what distinguishes this one is the explanatory work the formalism performs across the paper. Consider the credit assignment problem—one of the most frequently cited challenges in agentic settings. The POMDP formulation makes precise why credit assignment is harder in Agentic RL than in PBRFT: in PBRFT, with and deterministic transitions, every token in the response receives the same reward signal regardless of its contribution, so credit assignment is trivial (there is only one decision). In Agentic RL, with and stochastic transitions, the reward at time depends on actions at through the state evolution , creating a temporal entanglement that requires explicit credit assignment mechanisms. This is not merely a definition—it directly predicts which algorithms will work and which will fail: outcome-only RL (REINFORCE, DPO with trajectory-level preferences) will struggle on long-horizon tasks because they cannot disentangle the contributions of individual actions, while process-level methods (step-wise reward models, turn-level advantage estimation) will be necessary. The survey traces exactly this prediction through Section 3: tool use moves from outcome-only RL to turn-level advantage estimation (GiGPO, SpaRL); reasoning moves from sparse outcome rewards to step-wise process supervision; memory moves from RL-guided retrieval to token-level credit assignment.
Comparison to prior frameworks. Before this survey, the dominant conceptual framework for understanding LLM agents was the "modules" approach—an agent was a collection of separate components (planning module, memory module, tool-use module) each with its own heuristics, datasets, and evaluation criteria. This framework could describe agents but could not diagnose failures or prescribe solutions. Why does a particular agent fail on long-horizon tasks? Under the modules view, the answer might be "its planning module is insufficient" or "its memory module needs improvement"—vague prescriptions that don't specify what kind of improvement. Under the POMDP view, the answer becomes precise: the reward function is too sparse, causing the credit assignment mechanism (e.g., GRPO's group-relative advantage) to provide uninformative gradients; the solution is to densify the reward signal (process rewards, sub-goal rewards) or to use a more sophisticated credit assignment algorithm (turn-level advantage, hierarchical RL). This shift from module-level to mechanism-level diagnosis is fundamental.
Significance beyond taxonomy. The survey explicitly treats the POMDP formalism as the foundation for the "mechanistic debate" in Section 6.4—whether RL merely amplifies existing reasoning patterns or installs new capabilities. This debate cannot be properly framed without the formal distinction between PBRFT (where the "amplifier" view is trivially true because the MDP has only one decision point—there are no new behaviors to discover, only distribution reshaping) and Agentic RL (where the "new-knowledge" view becomes testable because multi-step interaction creates the possibility of emergent strategies that no amount of distribution reshaping could produce). By tying the debate to specific structural properties of the MDP—compositional multi-step structure, high-fidelity verifiable rewards, base models in an intermediate capability regime—the survey converts what could be a philosophical argument into an empirically tractable research program.
Innovation 2: The Capability Taxonomy as a Theory of How RL Transforms Modules into Behaviors
The survey's capability taxonomy (planning, tool use, memory, self-improvement, reasoning, perception) is superficially similar to taxonomies in prior agent surveys. What distinguishes it is the thesis that each capability passes through the same evolutionary progression—from imitative/static (prompting, SFT) through outcome-optimized (sparse reward RL) to process-aware (dense reward RL, credit assignment mechanisms)—and that this progression is not coincidental but law-like: it reflects the fundamental advantage of RL over supervised learning, which is the ability to learn from environmental feedback rather than replicate demonstrations. This is a theoretical claim, not merely an organizational convenience.
The law-like progression. The survey documents isomorphic progressions across all six capabilities:
- Tool use: ReAct/SFT → outcome-only RL (ToolRL, ARTIST) → turn-level credit assignment (GiGPO, SpaRL).
- Planning: Prompting/SFT → external RL guidance (RAP, LATS) → internal policy optimization (ETO, VOYAGER, RLTR).
- Memory: Passive retrieval (RAG, MemoryBank) → RL-guided retrieval (Prospect, Memory-R1) → token-level RL control (MemAgent, MemGen).
- Self-improvement: Verbal self-correction (Self-Refine) → gradient-based internalization (KnowSelf, SWEET-RL) → autonomous bootstrapping (R-Zero, Absolute Zero, SiriuS).
- Reasoning: Fast reasoning (implicit in next-token prediction) → outcome-only RL inducing slow reasoning (DeepSeek-R1) → process-level RL with structured exploration (MCTS-guided training).
- Perception: Passive perception (Visual-RFT, Reason-RFT) → active grounding (GRIT, Ground-R1) → tool-driven and generation-driven active cognition (Pixel Reasoner, GoT-R1).
Why does this progression recur? The survey's answer is structural: supervised learning on static trajectories can only teach the agent to replicate what is in the data. RL with sparse outcome rewards can teach the agent to discover strategies that succeed, but suffers from credit assignment failure on long trajectories. RL with dense process rewards can teach the agent how to succeed—which intermediate steps contribute to the outcome—enabling more sample-efficient learning and better generalization. This is not a contingent empirical observation; it follows from the properties of the learning signals available at each stage.
What this reframing achieves. Prior work studied each capability in isolation, with different communities developing different methods at different stages of this progression. The tool-use community was largely at the outcome-only RL stage; the reasoning community had progressed to process-level RL with DeepSeek-R1's success; the memory community was transitioning from static retrieval to RL-guided management. The survey's taxonomy reveals that these are not separate research threads but instances of the same pattern—enabling cross-pollination. A credit assignment mechanism developed for tool use (turn-level advantage in GiGPO) could be applied to planning. A self-improvement technique from reasoning (R-Zero's search-guided refinement) could inform memory management. The taxonomy makes these connections visible.
Significance as a research roadmap. The progression is not complete for any capability—each has a "prospective" section identifying the next stage. For tool use: long-horizon multi-turn TIR. For planning: meta-policies that learn when to deliberate. For memory: RL for structured memory management. For self-improvement: meta-evolution of reflection strategies. For reasoning: integrating slow reasoning into agentic settings. For perception: from tool-driven to fully autonomous active cognition. These are not wishlists; they are predictions derived from the progression logic—if the pattern holds, each capability should reach the prospective stage as the necessary algorithmic infrastructure (credit assignment mechanisms, environment simulators, verifiable reward structures) matures.
Innovation 3: The Application-Capability Dependency Matrix as a Systematic Research Prioritization Tool
Table 9 in the survey—the qualitative dependency matrix mapping capabilities (rows) to applications (columns)—is, at first glance, a simple lookup table. Its intellectual contribution, however, is that it operationalizes the survey's thesis into a resource allocation tool. If the survey's core argument is that RL is the mechanism for transforming static capabilities into adaptive behaviors, then the matrix answers the practical question: which capabilities should a practitioner invest in optimizing for a given domain, and which can be left as heuristics?
How the matrix works as a prioritization engine. The matrix encodes both necessity (which capabilities are "core" vs. "supporting" vs. "minimal") and the difficulty of optimization (implied by the corresponding capability section's progression stage). For search agents, planning and tool use are core, while perception is minimal—this tells a practitioner: invest in RL formulations for multi-turn query decomposition and tool invocation optimization, but don't bother with visual grounding unless your search task involves images. Moreover, since tool use and planning are at the outcome-only RL stage in most search agent work, the matrix implicitly flags that process-level credit assignment (turn-level advantage, sub-goal rewards) is the next investment frontier.
For GUI agents, planning, tool use, and perception are all core—this is a substantially harder domain because it requires multimodal RL formulations that jointly optimize visual grounding and action selection. The matrix makes this difficulty visible: a GUI agent developer needs to solve the RL problems of three capabilities simultaneously, each at an early stage of the progression, whereas a code agent developer (where perception is minimal) can focus primarily on tool use and reasoning. This explains, retroactively, why code agents have progressed faster than GUI agents—they have fewer simultaneous RL optimization challenges.
Comparison to prior organizational schemes. Previous surveys organized by capability (e.g., "survey of planning in LLM agents") or by domain (e.g., "survey of code generation agents") but never mapped the two onto each other systematically. The consequence was that researchers in one domain would reinvent techniques already developed in another, failing to recognize that their "novel" credit assignment method for code generation was structurally identical to a method already used in search agents. The dependency matrix makes these structural identities explicit: if code agents and search agents both have "tool use" as a core capability, then advances in tool-use RL for code agents should transfer to search agents, and vice versa.
The matrix as a dynamic artifact. The survey presents the matrix as a snapshot, but the progressive logic of the capability taxonomy implies that the matrix is dynamic—as capabilities advance through their evolutionary stages, the matrix entries should be revised. If tool use reaches the prospective stage (long-horizon multi-turn TIR with robust credit assignment), the difficulty of building a search agent (which depends on tool use) decreases. If perception reaches the active-cognition stage, the difficulty of building a GUI agent (which depends on perception) decreases. The matrix thus serves not only as a current-state map but as a progress tracker—it shows which capability bottlenecks are currently limiting each application domain.
Innovation 4: The Mechanistic Debate Formulation as a Unifying Theoretical Question
Section 6.4 of the survey frames a specific question—does RL merely amplify existing reasoning patterns ("amplifier" view) or install qualitatively new capabilities ("new-knowledge" view)?—and positions it as the central theoretical puzzle that the entire field of Agentic RL must resolve. This is a significant intellectual contribution because it unifies disparate empirical observations under a single falsifiable question and provides the conceptual tools (the MDP/POMDP formalism, the capability progression stages, the reward structure taxonomy) needed to answer it.
Why this debate is important beyond academic philosophy. The answer to the amplifier-vs-new-knowledge question determines where to invest resources. If RL is primarily an amplifier, then the bottleneck is pretraining—invest in larger models, more diverse pretraining data, better next-token prediction. RL is a refinement tool for post-training alignment and inference-time optimization, but it cannot compensate for fundamental capability gaps. If RL can install new capabilities, then the bottleneck shifts—invest in better RL training infrastructure, more sophisticated reward design, richer environments, more granular credit assignment. A research lab betting on the amplifier view would allocate budget to scaling pretraining; a lab betting on the new-knowledge view would allocate to scaling RL training.
How the survey advances the debate. Prior to this survey, evidence on both sides was scattered across papers with incompatible experimental setups. The "amplifier" view was supported by studies showing that base models matched or surpassed RL-tuned models at high pass@ (Yue et al., 2025). The "new-knowledge" view was supported by studies showing that RL enabled generalization on problems where next-token training was prohibitive (Tsilivis et al., 2025) and induced emergent behaviors like verification and backtracking (Gandhi et al., 2025). The survey's contribution is not to resolve the debate but to provide the framework for resolving it systematically: by characterizing the conditions under which RL produces amplification vs. emergence—high-fidelity verifiable rewards, compositional multi-step structure, base models in an intermediate capability regime—Section 6.4 converts a binary yes/no question into a conditional research program. The question is no longer "does RL create new capabilities?" but "under what specific conditions (reward structure, task compositionality, base model capability) does RL create new capabilities, and under what conditions does it merely amplify?"
The debate as a lens on the survey's own contribution. The survey's entire architecture—the formal POMDP distinction, the capability progression stages, the reward structure taxonomy—can be seen as building the conceptual apparatus needed to answer this question. The POMDP formalism specifies what "new capability" means: a behavior that requires multi-step interaction to manifest and cannot be produced by any static distribution over the base model's outputs. The capability progressions specify how new capabilities emerge: through the shift from outcome-only to process-level RL. The reward structure taxonomy specifies what enables emergence: verifiable, compositional rewards that provide dense learning signals. The question itself organizes the survey's content into a coherent research agenda.
The evidence cited and its implications. The survey's mathematical reasoning case study provides the most concrete evidence: studies emphasizing pass@1 improvements (~2/3 of cited works) align with the amplifier view; studies showing pass@ frontier expansion (~1/3 of cited works) align with the new-knowledge view. The conditions for new-knowledge behavior—high-fidelity verifiable rewards, compositional multi-step tasks, intermediate-capability base models—are precisely the conditions that the POMDP formulation identifies as necessary for effective agentic learning. This alignment between theoretical prediction and empirical observation strengthens the case that the formalism is not merely definitional but explanatory.
5. Experimental Analysis
Evaluation Methodology
This paper is a survey, not an experimental paper. It does not conduct original experiments. Rather, it synthesizes experimental results from over five hundred prior works, organizing them by task domain and capability. The "evaluation methodology" is therefore the survey's approach to selecting, presenting, and contextualizing these results—not a description of a unified experimental protocol (which does not exist across the heterogeneous landscape of Agentic RL research).
Datasets and benchmarks. The survey covers a remarkably diverse set of benchmarks, organized by task domain in Section 5.1 (Table 10). For search agents, key benchmarks include BrowseComp (Wei et al., 2025b)—a challenging benchmark measuring the ability to locate hard-to-find information—and WebShop (Yao et al., 2022), a simulated e-commerce website with crowdsourced instructions. Deep Research systems are evaluated on report-generation quality and multi-source synthesis, though the survey notes that "most open-source models still fail on OpenAI's BrowseComp" (Section 4.1.2). For code agents, the survey cites HumanEval (Chen et al., 2021) with 164 hand-crafted Python problems measured by pass@k; MBPP (Austin et al., 2021) with 974 entry-level Python tasks; LiveCodeBench (Jain et al., 2025b) as a continuously updated, contamination-free benchmark; SWE-bench (Jimenez et al., 2024) for real-world GitHub issue resolution; and BigCodeBench (Zhuo et al., 2025) with 1,140 tasks requiring composition of multiple function calls. For mathematical reasoning, benchmarks include MATH-500, AIME24, AIME25 (for informal reasoning), and miniF2F (Zheng et al., 2022) and ProofNet (Azerbayev et al., 2023) (for formal theorem proving). For GUI agents, the survey covers OSWorld (Xie et al., 2024) with real-world computer tasks across operating systems, AndroidWorld (Rawles et al., 2025) with 116 tasks across 20 applications, and WebArena (Zhou et al., 2024b) as a self-hostable web environment. For embodied agents, benchmarks include ALFWorld (Shridhar et al., 2021) for text-based embodied tasks and Crafter (Hafner, 2022) for open-world survival. The survey explicitly does not impose a standardized evaluation protocol across these domains—indeed, the absence of such standardization is one of the fragmentation problems it identifies (Section 1).
Base model families. The survey catalogs an enormous range of base models, reflecting the field's rapid evolution and diversity. Frequently cited families include: Qwen2.5 variants (3B, 7B, 14B, 32B, 72B parameters, both base and instruct versions) used extensively in search agent RL (Search-R1, Research, DeepResearcher), code agent RL (DeepCoder-14B, AceCoder, CTRL), and math agent RL (rStar2-Agent); Llama-3 variants (3B, 8B, 70B parameters) used in code refinement (RLEF) and search agents; DeepSeek variants (DeepSeek-Coder, DeepSeek-R1-Distilled, DeepSeek-Prover) used in formal theorem proving and code generation; PaLM 2 models (referenced primarily in the separate analysis paper, not this survey); and proprietary systems including OpenAI's o1/o3 series, Kimi K2, and Google Gemini. The survey highlights a critical pattern: smaller models trained with Agentic RL (e.g., 7B Qwen2.5 for search, 14B for math reasoning) can rival or surpass much larger models—for instance, rStar2-Agent achieves 80.6% on AIME24 and 69.8% on AIME25 with a 14B model in "just 510 RL steps" (Section 4.3.1). The survey does not itself select a single model family for experiments, as it is not an experimental paper; rather, it identifies model choices as a variable of interest across studies.
Metrics. The survey reports metrics as they appear in the cited works, with no unified metric across domains—this heterogeneity is precisely the fragmentation the survey documents. Common metrics include: pass@1 (fraction of test problems where the first generated solution is correct, used in code generation, math reasoning, and search tasks); pass@k (fraction where at least one of k samples is correct, used to distinguish distribution reshaping from capability creation); task success rate (binary completion, used in GUI and embodied agents); accuracy on benchmark X (domain-specific evaluation, e.g., miniF2F proof success rate, SWE-bench Verified resolution rate); BrowseComp pass@1 (for deep research agents, where 51.5% represents the state-of-the-art from OpenAI Deep Research); and AIME scores (for competition-level math, reported as average pass@1). The survey explicitly notes that metric inconsistency across works makes "systematic comparison and cross-domain generalization difficult" (Section 1). It does not propose a new unifying metric but rather catalogs the existing landscape to make the fragmentation visible.
Baselines. The survey identifies several classes of baselines used across the literature. RL-free baselines: prompting-only methods (ReAct-style tool calling, chain-of-thought without fine-tuning), supervised fine-tuning (SFT) on static trajectory data (common in GUI and code domains), and vanilla VLM baselines (for vision and GUI tasks, where pre-trained vision-language models are used zero-shot). RL baselines within domains: majority voting (selecting the most common answer among N samples, without a learned verifier), best-of-N with outcome reward models (ORM), and best-of-N with process reward models (PRM). Cross-paradigm baselines: RL-free methods serve as baselines for RL-based methods, showing the improvement from adding environmental feedback to static imitation. For instance, in code generation, SFT on HumanEval serves as the baseline against which RL-trained models (DeepCoder-14B, AceCoder) are compared; in search, ReAct-style query generation serves as the baseline against which GRPO-trained query policies (Search-R1, DeepRetrieval) are compared. The survey does not itself establish new baselines but catalogs which baselines are standard in each domain.
Generation budget and compute accounting. The survey notes that compute is measured heterogeneously across works, with no standardized accounting. Common measures include: number of generations (samples from the policy, used in best-of-N and beam search comparisons); number of RL training steps (e.g., "510 RL steps" for rStar2-Agent); number of environment interactions (relevant for GUI and embodied agents); wall-clock training time on specific hardware (rarely reported consistently, but noted as a practical concern); and total FLOPs (used in comparisons that match pretraining and inference compute, though the survey focuses on the methodological taxonomy rather than FLOPs-matched analyses). The survey does not enforce a uniform budget measure across the works it cites, reflecting the field's current inconsistency.
Cross-validation and statistical protocols. The survey does not itself apply cross-validation; it catalogs protocols used in cited works. Notable examples include: two-fold cross-validation within difficulty bins for strategy selection in compute-optimal test-time scaling (cited from the separate analysis paper); train/validation/test splits standard in benchmarks (MATH-500 test split, HumanEval as a fixed test set); and LiveCodeBench's contamination-free design where problems are sourced from recent competitions to prevent training data leakage. The survey identifies the small size of many test sets as a limitation: "the test set of 500 questions, split into five difficulty quintiles of ~100 each... means the compute-optimal policy is selected based on ~50 questions per fold per bin" (from the prior executive summary's assessment, which applies to the separate analysis paper cited within the survey). For the survey itself, statistical rigor varies enormously across the 500+ cited works, and the survey does not impose a uniform standard—it documents the heterogeneity.
Main Quantitative Results
Since this is a survey synthesizing over 500 papers, "main quantitative results" refers to the headline findings that emerge across the literature when organized through the survey's taxonomic lens. These findings are drawn from Section 4 (task domain surveys) and contextualized by the capability taxonomy in Section 3.
RL for Search and Research Agents (Section 4.1)
Headline: RL-trained search policies dramatically outperform RL-free baselines on complex retrieval tasks, but open-source models still lag behind closed-source systems by wide margins.
The survey documents a consistent pattern across open-source search agent research: applying GRPO or PPO to query generation and multi-step retrieval produces substantial improvements over prompting-only baselines. DeepRetrieval (Jiang et al., 2025c), which frames one-shot query generation as a GRPO-trained policy, demonstrates that "directly rewarding recall and relevance against live search results" (Section 4.1.1) enables a 3B-parameter Qwen2.5 model to learn effective retrieval strategies without supervised query trajectories. Search-R1 (Jin et al., 2025a) extends this to multi-turn settings, integrating retrieved-token masking with outcome-based rewards to interleave query formulation and answer generation across multiple retrieval steps. The survey characterizes the mechanism: outcome-only RL with verifiable retrieval quality signals is sufficient to induce emergent search behaviors including query refinement based on previous results.
However, the headline finding is qualified by a sobering gap: "most open-source models still fail on OpenAI's BrowseComp" (Section 4.1.2). OpenAI's Deep Research achieves 51.5% pass@1 on BrowseComp (Section 4.1.2), while open-source systems lag substantially—the survey does not cite specific open-source BrowseComp scores, but notes the failure explicitly. This gap is attributed to proprietary systems' "more powerful foundation models and the availability of more high-quality data" (Section 4.1.2). The survey maps the progression: early open-source work (DeepRetrieval, Search-R1) established RL feasibility; subsequent work (R1-Searcher with two-stage cold-start PPO, AutoRefine with GRPO-trained refinement phases) improved multi-turn reasoning; the current frontier (ASearcher with large-scale asynchronous RL, WebDancer with human trajectory supervision) addresses scalability and long-horizon performance. Table 4 summarizes the key open-source methods with their base model families and resource links.
A notable methodological trend documented in the survey: the shift from live web API training to "internal knowledge" search to address cost and stability. ZeroSearch (Sun et al., 2025a) replaces live web retrieval with an LLM-distilled pseudo search engine, using curriculum RL to approach live-engine performance without real queries. SSRL (Fan et al., 2025b) performs entirely offline "self-search" during training, transferring seamlessly to online inference. The survey frames this as a practical necessity driven by "the API cost [being] too high and severely limit[ing] scalability" and "the document quality of real-time internet document searching [being] uncontrolled" (Section 4.1.1). The quantitative implication: offline self-search enables stable, scalable training that approaches online performance, though the survey does not cite a specific comparison figure.
RL for Code Agents (Section 4.2)
Headline: Outcome-only RL matches or exceeds SFT baselines on competitive programming benchmarks; process-level rewards accelerate convergence but require engineering effort.
The survey organizes its code agent results along a progression from single-round generation to iterative refinement to full software engineering (SWE). For single-round code generation (Section 4.2.1), the key comparison is SFT baselines vs. outcome-only RL. DeepCoder-14B (Luo et al., 2025c) applies GRPO training to a DeepSeek-R1-Distilled-Qwen-14B model, using a sparse Outcome Reward Model to prevent reward hacking on curated coding data. The survey reports that this approach enables a 14B model to reach competitive performance with much larger proprietary systems (the specific benchmark scores are cited in the referenced paper, not in the survey itself, consistent with the survey's synthesizing role). Similarly, Absolute Zero (Zhao et al., 2025a) demonstrates self-play RL without human data—generating coding tasks and using execution outcomes as verifiable rewards—bootstrapping reasoning from zero human demonstrations.
For process-level reward methods, the survey reports quantitative advantages in convergence speed. PRLCoder (Ye et al., 2025b) achieves "+10.5% pass-rate improvements over the base model" using process-supervised RL where "a teacher model mutates lines of reference solutions and assigns positive/negative signals based on compiler and test feedback" (Section 4.2.1). The survey interprets this as evidence that "fine-grained supervision at the line-level can guide code synthesis more effectively than outcome-only signals" (Section 4.2.1). StepCoder (Dou et al., 2024) decomposes compilation and execution into step-level signals for shaping, while PSGPO (Dai et al., 2025) leverages intermediate error traces as dense rewards.
For iterative code refinement (Section 4.2.2), the survey highlights RLEF (Gehring et al., 2025), which "reduces the number of attempts needed and improves competitive-programming performance relative to single-shot baselines" by grounding correction loops in real error messages while optimizing for ultimate pass rates. The survey notes that this multi-turn RL formulation is closer to real-world debugging workflows than single-shot generation.
For automated software engineering (Section 4.2.3), the headline result is the emergence of RL-trained models that achieve strong performance on SWE-bench-style evaluations. DeepSWE (Luo et al., 2025b) performs large-scale RL on software engineering missions using verified task completion as the sole reward, achieving "leading open-source results" (Section 4.2.3). Qwen3-Coder (Team, 2025b) performs "large-scale execution-driven reinforcement learning on long-horizon, multi-turn interactions across 20,000 parallel environments, yielding state-of-the-art performance on benchmarks like SWE-Bench Verified." The survey notes that these systems represent a qualitative shift from single-function generation to repository-level, multi-file software engineering.
The survey's Figure 7 (Benchmark Performance of RL-Enhanced Code and SWE Methods) provides a visual comparison, though the survey does not quote specific numbers from it—referring readers to the figure for detailed score comparisons.
RL for Mathematical Reasoning Agents (Section 4.3)
Headline: RL with verifiable rewards produces substantial gains on informal math benchmarks; formal theorem proving remains challenging, with expert iteration (ExIt) methods showing the most promise.
For informal mathematical reasoning (Section 4.3.1), the survey documents a consistent pattern: outcome-only RL with verifiable correctness signals (numeric answers, symbolic equality) enables substantial improvements over SFT baselines, and process-level rewards further accelerate learning. ARTIST (Singh et al., 2025) uses outcome-only rewards to train tool-integrated agentic reasoning, interleaving code execution with natural language reasoning, achieving strong performance with emergent self-reflection and context-aware CoT behaviors. ToRL (Li et al., 2025l) improves performance "by exploiting the scaling of tool-integrated reasoning RL and encouraging code execution behaviour" with emergent adaptive tool-use and self-correction.
The survey highlights a particularly striking quantitative finding from 1-shot RLVR (Wang et al., 2025r): "RL with only 1 example performs close to using a 1.2k-example dataset, and with 2 examples comes close to using the 7.5k MATH training dataset" (Section 4.3.1). The survey also notes the "post-saturation generalization" phenomenon: "test accuracy continues to improve even after the training accuracy on the single example approaches 100%"—a counterintuitive finding suggesting RL induces generalization beyond memorization of the training example.
For state-of-the-art results, rStar2-Agent (Shang et al., 2025a), a 14B-parameter model, achieves "average pass@1 scores of 80.6% on AIME24 and 69.8% on AIME25" in just 510 RL steps, trained with agentic RL using a Python execution environment and a novel GRPO-RoC algorithm that resamples on correct rollouts amid tool-noise.
ZeroTIR (Mai et al., 2025) investigates scaling laws: "a strong correlation between training computational effort and the spontaneous code execution frequency, the average response length, and the final task accuracy" (Section 4.3.1), suggesting that RL training compute directly predicts emergent tool-use behavior and performance.
For formal mathematical reasoning (Section 4.3.2), the survey documents a more challenging landscape. DeepSeek-Prover-v1.5 (Xin et al., 2025) demonstrated that "binary verifier feedback" alone can drive significant improvements on miniF2F and ProofNet when combined with effective exploration (RMaxTS, a variant of MCTS with intrinsic rewards for novel tactic states). DeepSeek-Prover-v2 (Ren et al., 2025) introduced subgoal decomposition with binary Lean feedback at the subgoal level for denser supervision. The survey presents the ExIt (expert iteration) paradigm as the most promising approach: search discovers valid proof trajectories, these are used as expert demonstrations for the apprentice policy, and the improved policy bootstraps subsequent search. STP (Dong & Ma, 2025) extends this with a conjecturer–prover self-play loop that generates an adaptive curriculum, reporting "large gains on LeanWorkbook and competitive results among whole-proof generation methods on miniF2F and ProofNet" (Section 4.3.2). The survey's Figure 8 provides a visual comparison of math method benchmarks.
RL for GUI Agents (Section 4.4)
Headline: RL transforms GUI agents from brittle, static-action replicators to adaptive, interactive decision-makers; interactive online RL in live environments substantially outperforms static SFT but requires robust infrastructure.
The survey's GUI agent subsection traces a clear quantitative progression. RL-free methods (vanilla VLMs, SFT on static trajectories) serve as baselines. InfiGUIAgent (Liu et al., 2025n) uses a two-stage SFT pipeline, and UI-AGILE (Lian et al., 2025) enhances SFT with continuous rewards and cropping-based resampling—but both "face the limitation of scarce human operation traces" (Section 4.4.1).
RL in static GUI environments uses pre-collected datasets with deterministic execution traces and rule-based outcome evaluation. GUI-R1 (Luo et al., 2025d) "adopts an R1-style reinforcement fine-tuning pipeline over a unified action schema, using simple format and correctness rewards to improve step-level action prediction with modest data" (Section 4.4.2). UI-R1 (Lu et al., 2025d) applies group-relative policy optimization to stabilize policy updates and improve exact parameter matching. The survey notes that these static-RL methods improve grounding and reasoning but cannot learn to recover from errors because "all changes to the web pages or visual elements occur exclusively in response to the agent's actions" (Section 5.1.1)—there is no environmental stochasticity to learn from.
RL in interactive GUI environments represents the current frontier. WebAgent-R1 (Wei et al., 2025f) conducts "end-to-end multi-turn reinforcement learning with asynchronous trajectory generation and group-wise advantages, improving success on diverse web tasks" (Section 4.4.3). MobileGUI-RL (Shi et al., 2025c) scales training on Android virtual devices with "trajectory-aware GRPO, a decaying efficiency reward, and curriculum filtering, improving execution efficiency and generalization while keeping the system practical for large rollout volumes." UI-Venus (Gu et al., 2025), fine-tuned via RFT with custom reward functions and a self-evolving trajectory framework, achieves "a new state-of-the-art performance in both UI grounding and navigation." The survey does not quote specific accuracy figures but characterizes these results as demonstrating that online RL in live environments substantially outperforms both static SFT and static RL. Table 7 systematically compares GUI agent methods by paradigm (vanilla VLM, SFT, RL) and environment complexity (static vs. interactive).
RL for Multi-Agent Systems (Section 4.7)
Headline: End-to-end multi-agent RL enables emergent coordination and division of labor, but faces severe scalability challenges from credit assignment complexity.
The survey traces a clear progression in multi-agent RL. RL-free multi-agent evolution methods (CAMEL, MetaGPT, MAD, MoA) use fixed coordination protocols without parameter updates—they serve as baselines for RL-based approaches. RL-driven optimization of non-parametric coordination modules (GPTSwarm, MaAS, G-Designer) keep agent parameters frozen while using RL to optimize communication topologies, routing policies, or workflow graphs. The survey notes that "rewards are typically delayed and sparse... requiring global-to-local credit decomposition or structural priors to avoid collapse" (Section 4.7.1).
RL-driven optimization of selected agent policies (Optima, DITS, MALT, ACC-Collab) updates only a subset of agents while keeping others frozen. The survey characterizes the credit assignment problem as "fundamentally semi-local"—rewards emerge from collective trajectories but gradients apply only to optimized agents. "This requires mechanisms such as role-conditioned DPO, local advantage estimation, or counterfactual baselines to prevent reward hijacking by non-updated agents" (Section 4.7.2). Empirically, "such partial optimization yields better sample efficiency than fully joint multi-agent training while still enabling the emergence of specialized roles" (Section 4.7.2).
End-to-end multi-agent RL (MAGRPO, MAPoRL, MLPO, ReMA, FlowReasoner, Chain-of-Agents, SPIRAL) jointly trains all agents under a shared or decentralized objective. MAGRPO (Liu et al., 2025l) "formalizes multi-LLM cooperation as a Dec-POMDP problem and introduces a multi-agent variant of GRPO, which enables joint training of LLM agents in MAS while maintaining decentralized execution" (Section 4.7.3). SPIRAL (Liu et al., 2025b) "presents a fully online, multi-turn, multi-agent self-play reinforcement learning framework for LLMs in zero-sum games, employing a shared policy with role-conditioned advantage estimation (RAE) to stabilize learning" and "demonstrates that gameplay fosters transferable reasoning skills that significantly improve mathematical and general reasoning benchmarks" (Section 4.7.3). The survey notes that MMedAgent-RL (Xia et al., 2025b) in the medical domain "significantly outperforms existing Med-LVLMs and achieves more human-like diagnostic behavior" through dynamically coordinated specialist agents with curriculum-guided learning.
The critical quantitative challenge: "sample complexity grows roughly linearly with the number of agents and quadratically with interaction depth" (Section 4.7.3). Algorithms mitigate this using centralized critics or value-shared baselines, but "achieving scalable credit decomposition remains a central open challenge" (Section 4.7.3). Table 8 summarizes multi-agent RL methods with their dynamic properties, training characteristics, and resource links.
Ablation Studies and Robustness Checks
Since this is a survey and not an experimental paper, "ablation studies" in the traditional sense (controlled experiments removing components to measure their impact) are not conducted by the survey itself. Rather, the survey catalogs findings from ablation-like comparisons that appear across the cited literature. What follows are the key comparative analyses the survey highlights, organized by the methodological axes they probe.
Outcome-only vs. process-level rewards (across domains): The survey documents a recurring pattern: process-level rewards accelerate convergence but at the cost of engineering complexity. For code generation, PRLCoder achieves "+10.5% pass-rate improvements over the base model" with process supervision (Section 4.2.1), while outcome-only methods like DeepCoder-14B require more training steps but achieve competitive final performance. For mathematical reasoning, process supervision through subgoal decomposition (DeepSeek-Prover-v2) improves upon outcome-only methods (DeepSeek-Prover-v1.5) specifically on complex multi-step proofs, but outcome-only methods suffice for problems where the final answer verifiability is high. The survey does not provide a unified quantitative comparison (the benchmarks differ across domains) but identifies this as a cross-cutting design choice with consistent qualitative effects.
RL-free vs. RL-based methods (across domains): The survey presents this as the most fundamental comparison, and it is strongly supported across every task domain. For search, RL-trained query policies outperform prompting-only and SFT baselines (Section 4.1). For code, RL-trained models surpass SFT baselines on competitive programming benchmarks and SWE tasks (Section 4.2). For math, RL with verifiable rewards substantially outperforms SFT on reasoning benchmarks (Section 4.3). For GUI, online RL in interactive environments outperforms both static SFT and static RL (Section 4.4). The survey does not aggregate these into a single quantitative claim (the metrics and baselines differ), but the pattern is consistent and central to the survey's thesis that "RL provides the critical mechanism for transforming these capabilities from static, heuristic modules into adaptive, robust agentic behavior" (Abstract).
Static vs. interactive RL environments (GUI agents, Section 4.4): The GUI agent literature provides a natural ablation: RL on static, pre-collected datasets vs. RL in live, interactive environments. The survey reports that interactive RL enables "robustness to stochastic transitions and long-horizon dependencies" (Section 4.4.3) that static RL cannot capture—specifically, error recovery behaviors (e.g., the agent tries an action, observes it fails, and adapts) require live environment feedback. MobileGUI-RL's trajectory-aware GRPO with curriculum filtering represents the state of the art for interactive mobile GUI training, while WebAgent-R1's asynchronous trajectory generation with group-wise advantages scales interactive web training.
GRPO variants (Section 2.7, Table 2): The survey catalogs the proliferation of GRPO variants, each addressing specific limitations of the base algorithm. DAPO (decoupled clip and dynamic sampling) addresses training instability in multi-turn settings (Golubev et al., 2025). GSPO (sequence-level clipping based on sequence likelihood) addresses token-level vs. sequence-level credit mismatch. Posterior-GRPO (rewards only successful reasoning processes) addresses reward hacking where the model learns to produce outputs that score well under the PRM but are incorrect. Dr.GRPO (corrects bias in the GRPO objective) addresses a theoretical bias in the optimization. The survey does not provide head-to-head comparisons of these variants (they are typically evaluated on different benchmarks), but their proliferation itself is a finding: the base GRPO algorithm has identifiable weaknesses that spawn targeted improvements.
Self-play and search-guided training (Sections 3.4, 4.3): The survey highlights a critical comparison between outcome-only RL and search-guided RL (expert iteration) for formal theorem proving. The ExIt paradigm (DeepSeek-Prover-v1.5, STP, Lean-STaR) provides an alternative to direct RL: "instead of directly optimizing against sparse verifier signals, ExIt performs search-guided data augmentation" where "valid proof trajectories discovered by search and checked by a verifier are reused as expert demonstrations in an imitation-learning loop" (Section 4.3.2). STP specifically demonstrates that "solely relying on expert iteration will quickly plateau due to the sparse positive rewards" and addresses this with a conjecturer–prover self-play curriculum (Dong & Ma, 2025). The survey presents this as evidence that the choice between direct RL and ExIt is not fundamental—both can work, but ExIt requires additional machinery to avoid plateaus.
Closed-source vs. open-source performance (Sections 4.1-4.4): Throughout the survey, a persistent finding is the gap between closed-source industrial systems and open-source research. OpenAI Deep Research achieves 51.5% pass@1 on BrowseComp; open-source systems "still fail" on this benchmark (Section 4.1.2). For code, commercial systems like Claude Code and OpenAI Codex "have emphasized preference-aligned fine-tuning and reinforcement learning" (Section 4.2.3) but training details are limited. The survey attributes this gap to "more powerful foundation models and the availability of more high-quality data" rather than algorithmic secrets, and positions its framework consolidation as an effort to close the gap.
Hardware and scaling practicalities (distributed throughout): The survey notes practical ablation-like findings related to infrastructure. For web-based RL training, the shift from live API search to offline self-search (ZeroSearch, SSRL) was driven by the finding that "the API cost is too high and severely limits scalability" and that "noisy information brings instability to the training process" (Section 4.1.1)—these are not formal ablations but de facto infrastructure ablations that practitioners will encounter. For GUI agents, training at scale required "massively parallel, fully asynchronous RL infrastructure" (ComputerRL, Section 4.4.3) and "Android virtual devices" (MobileGUI-RL)—the survey implicitly ablates the feasibility of single-machine vs. distributed training for interactive environments.
Critical Assessment
The survey is a synthesis paper, not an experimental paper. Therefore, a traditional critical assessment of its own experiments is inapplicable. Instead, what follows is an assessment of two questions: (1) Do the results the survey catalogs across the literature genuinely support its central claims about the role of RL in agentic systems? (2) What are the limitations of the survey's evidential basis—that is, what patterns in the cited literature weaken or qualify the conclusions the survey draws?
On the Claim That RL Is the Critical Mechanism for Transforming Static Modules into Adaptive Behaviors
This is the survey's central thesis. The evidence it marshals is extensive and consistent across domains: for every capability surveyed (planning, tool use, memory, self-improvement, reasoning, perception), moving from RL-free to RL-based methods produces performance improvements. For every task domain (search, code, math, GUI, vision, embodied, multi-agent), RL-based methods outperform RL-free baselines on the standard benchmarks. The survey documents this pattern across over 500 papers.
However, the evidence supports a narrower claim than the survey's rhetoric suggests. What the surveyed literature demonstrates is that RL applied with verifiable outcome rewards improves performance on tasks where the reward signal is well-defined (unit test passes, proof verification, task completion, numeric answer matching). This is a significant finding—it shows that outcome optimization is more effective than imitation learning for tasks with clear success criteria—but it is not the same as demonstrating that RL transforms "static modules into adaptive behaviors." The latter claim implies that RL induces qualitatively new capabilities (error recovery, adaptive strategy selection, meta-learning of when to use which tool). Whether RL actually does this, or whether it merely improves pass@1 by reshaping the output distribution over already-existing competent trajectories, is precisely the mechanistic debate the survey itself identifies as unresolved (Section 6.4).
The survey acknowledges this tension: "approximately 2/3 [of cited math reasoning works] primarily emphasize improvements in pass@1 accuracy, while about 1/3 explicitly report expanding pass@k frontiers" (Section 6.4). This is a crucial qualification that the survey's abstract and introduction—with their strong claims about "transforming" capabilities—do not adequately foreground. If two-thirds of the cited works primarily demonstrate distribution reshaping (the "amplifier" view), then the evidence for genuine capability creation (the "new-knowledge" view) is concentrated in a minority of studies under specific conditions. The survey identifies these conditions—"high-fidelity, often executable or formally checkable reward signals; compositional or multi-step structure where many partial trajectories are verifiably graded; and base models in the 'intermediate' regime"—but the claim that RL "transforms" capabilities is conditional on these conditions being met, which is not always emphasized in the survey's broader framing.
On the Capability Progression as a Law-Like Pattern
The survey argues that each capability passes through the same evolutionary progression: RL-free → outcome-only RL → process-level RL → prospective stage. This is a structural claim about the nature of agentic learning, not merely an organizational convenience. The evidence for this progression is strong for tool use (where the progression from ReAct to ToolRL to GiGPO is well-documented) and for reasoning (where the progression from implicit reasoning to DeepSeek-R1's outcome-induced CoT to process-level supervision is clear). For other capabilities, the evidence is sparser. For memory, RL-guided retrieval (Phase 2) and RL for token-level memory (Phase 3) are represented by only a handful of works each (Prospect, Memory-R1, MemAgent, MemGen), and the progression from Phase 2 to Phase 3 is more aspirational than empirically demonstrated. For perception, the "active visual cognition" stage is represented by recent preprints (GRIT, Pixel Reasoner, GoT-R1) that have not yet been extensively replicated or compared. The survey's claim that the progression is "law-like" should therefore be understood as a hypothesis supported by suggestive evidence in some capabilities, not as a fully validated empirical regularity across all capabilities.
On the Application-Capability Dependency Matrix (Table 9)
Table 9 is the survey's practical contribution—it maps which capabilities are core/supporting/minimal for each task domain. The matrix is constructed based on the survey authors' qualitative assessment of the literature rather than on quantitative ablation studies. This is not a weakness per se—a qualitative dependency matrix is a legitimate scholarly contribution—but the evidential basis is interpretive rather than empirical. Whether "perception" is truly "core" for GUI agents (as the matrix asserts) or whether an agent could succeed with text-based representations of GUI states is not empirically tested by the survey. The matrix should be understood as a set of hypotheses about capability dependencies that could be tested by future work, not as established facts.
The survey also does not provide quantitative evidence that the matrix is useful—that is, it does not demonstrate that practitioners who follow its recommendations (investing in the core capabilities for their domain) achieve better results than those who do not. This would require a controlled study across multiple teams, which is beyond the scope of a survey. The matrix's value is heuristic, not empirical.
On the RL Algorithm Comparison (Section 2.7, Table 2)
The survey's presentation of RL algorithms (REINFORCE, PPO, DPO, GRPO and their variants) is primarily definitional rather than comparative. Table 2 lists variants with their key mechanisms, but provides no head-to-head performance comparisons, no guidance on when to prefer PPO over GRPO or DPO over both, and no analysis of failure modes. This is a significant limitation for a survey that aims to guide practitioners: a researcher deciding between DAPO, GSPO, and Dr.GRPO for their code generation task receives no empirical basis for the choice from the survey. The algorithms are presented as a toolkit without a user manual.
This limitation reflects a genuine gap in the literature—very few papers systematically compare RL algorithms across multiple domains with controlled experimental setups—but the survey could have at least cataloged which algorithms are used in which domains and with what reported performance, providing a de facto empirical guide even in the absence of controlled comparisons.
On the Environment and Framework Catalog (Section 5)
Section 5 is one of the survey's most practically valuable contributions—it consolidates a fragmented landscape of environments and frameworks with resource links and key features. The limitation is that the catalog is descriptive rather than evaluative. The survey does not report which environments are most suitable for which research questions, which frameworks have the best performance/scalability, or which combinations of environments and frameworks have been successfully used together. A practitioner reading Section 5 knows that options exist but not which to choose. This reflects the survey's scope as a synthesis rather than a benchmarking effort, but it limits the section's practical utility.
On the Evidence for Scalability and Compute Requirements
The survey repeatedly references the computational demands of Agentic RL—the need for "massively parallel, fully asynchronous RL infrastructure" (ComputerRL), the challenge of API costs for web training (motivating ZeroSearch and SSRL), the quadratic scaling of sample complexity with agent count in multi-agent systems—but provides almost no quantitative data on actual compute requirements. How many GPU-hours does it take to train a DeepCoder-14B? What is the wall-clock time for WebAgent-R1's asynchronous trajectory generation? How do training costs compare between outcome-only and process-level reward methods? The survey's silence on these questions is a significant gap for practitioners, who need cost estimates to make resource allocation decisions. This gap reflects the broader literature's inconsistency in reporting compute requirements—the survey could have noted this as a specific deficiency and called for standardized reporting.
On the Generality of Findings Across Model Scales and Families
The survey documents that smaller models trained with Agentic RL can rival larger models—rStar2-Agent's 14B model achieving 80.6% on AIME24, DeepCoder-14B reaching competitive coding performance. However, the survey does not systematically analyze how RL benefits scale with model size. Do the same RL algorithms work equally well for 3B, 7B, 32B, and 70B models? Do process-level rewards matter more for smaller models (which may need denser signals) or for larger models (which may overfit to sparse signals)? The survey catalogs models of different sizes but does not extract scaling trends from the literature—a missed opportunity given the practical importance of model size decisions.
Summary Assessment
The survey's central thesis—that RL is a critical mechanism for improving LLM agent performance across capabilities and tasks—is well-supported by the sheer volume and consistency of cited results across domains. The more ambitious claims—that RL transforms static modules into adaptive behaviors, that capabilities follow a law-like progression, that the dependency matrix provides actionable guidance—are supported more by the survey's interpretive framework than by the empirical evidence directly, and should be understood as theoretical contributions (hypotheses about the structure of agentic learning) rather than as experimentally validated facts. The survey's most significant evidential limitation is its descriptive rather than comparative approach to algorithms and infrastructure; the most significant gap is the absence of quantitative compute requirements and scaling trends. These limitations are inherent to the survey format and the heterogeneity of the underlying literature, but they bound the strength of the conclusions that can be drawn.
6. Limitations and Trade-offs
6.1 The Survey Is Descriptive, Not Prescriptive — It Cannot Tell Practitioners Which Algorithm to Use for Their Specific Problem
The assumption or constraint. The survey positions itself as providing "conceptual infrastructure" (Section 1) and a "unified formal lens" (Section 3) for understanding Agentic RL, but it deliberately avoids making prescriptive algorithmic recommendations. Section 2.7 presents the four canonical RL algorithm families (REINFORCE, PPO, DPO, GRPO) and Table 2 catalogs their variants with key mechanisms—but the survey provides no head-to-head comparisons, no guidance on when to prefer one over another, and no analysis of failure modes that would inform algorithm selection. The survey acknowledges this implicitly in its scope definition: "Primary focus: how RL empowers LLM-based agents in dynamic environments" rather than "which RL algorithm is best for which task."
The consequence. A practitioner starting an Agentic RL project—training a search agent, a code agent, or a GUI agent—cannot use this survey to answer the most basic design question: should they use GRPO (as in Search-R1) or PPO (as in WebDancer) or DPO (as in KnowSelf)? The survey's approach of cataloging what was done without analyzing why it worked there and whether it would work elsewhere means the practitioner must replicate the decision process of each cited paper independently. If the field's fragmentation is a problem (as the survey argues), then a survey that reproduces this fragmentation at the algorithmic level—listing options without comparison—leaves the central practical challenge unaddressed.
What evidence exists in the paper. The evidence for this limitation is structural, not localized to a single figure. Table 2 lists 15+ GRPO variants without any performance data. Section 4 presents each task domain's RL methods as a chronological narrative ("first X, then Y, most recently Z") with reported metrics from each paper, but the metrics are incomparable across papers because they use different benchmarks, different base models, and different training budgets. There is no figure, table, or section that attempts to normalize results across algorithms. The survey's application-capability dependency matrix (Table 9) maps which capabilities are important for which tasks, but provides no mapping from capabilities to algorithms. A practitioner who concludes "my search agent needs planning and tool use (core)" still does not know whether to implement planning via GRPO (as in DeepRetrieval) or DPO (as in WebThinker) or PPO (as in R1-Searcher).
Mitigation status. The survey does not address this limitation. It acknowledges that the field lacks standardized evaluation protocols (Section 1: "Current studies often examine isolated capabilities, domains, or custom environments, with inconsistent terminology and evaluation protocols, making systematic comparison and cross-domain generalization difficult"), but it does not itself attempt to impose or construct such comparisons. The survey's contribution is organizational, not empirical—it provides a taxonomy for thinking about the field, not a decision procedure for acting within it. This is a legitimate scholarly contribution, but it bounds the survey's practical utility.
6.2 The Formal POMDP Abstraction Is Presented as a Unifying Framework, but the Survey Provides No Empirical Demonstration That Using It Improves Outcomes
The assumption or constraint. The survey's central intellectual move is formalizing PBRFT vs. Agentic RL through the MDP/POMDP distinction (Section 2). The abstraction is elegant: PBRFT is a degenerate MDP with T=1, deterministic transitions, and scalar outcome rewards; Agentic RL is a POMDP with T>1, stochastic dynamics, hybrid action spaces, and composite rewards. The survey argues this formalization "enables systematic comparison and cross-domain generalization" and "clarifies the theoretical implications of treating LLMs either as static sequence generators or as interactive, decision-capable agents." However, the survey provides no experimental evidence that using this formalism to design an Agentic RL system produces better results than not using it. The formalism is a conceptual tool, not a validated engineering methodology.
The consequence. A skeptic could argue that the POMDP formalism is a post-hoc rationalization—it describes what successful systems already do rather than providing novel design insights that improve systems. The survey itself documents that researchers in different domains independently discovered the same patterns (outcome-only RL → process-level RL, sparse rewards → dense rewards) without explicitly invoking POMDP theory. If the formalism is merely descriptive, its practical value is limited. If it is genuinely predictive—if using the POMDP lens to analyze a new task domain would lead to different and better design choices than domain-specific intuition—the survey provides no demonstration of this. The formalism's value is asserted, not empirically established.
What evidence exists in the paper. The survey's evidence for the formalism's utility is the consistency of patterns across domains—the observation that tool use, planning, memory, and reasoning all progress from RL-free to outcome-only RL to process-level RL (Section 3). But this consistency could equally be explained by convergent evolution (researchers in different domains independently discovering what works) rather than by the formalism's explanatory power. The survey does not present a case study where applying the POMDP lens revealed a non-obvious design choice that was subsequently validated. The formal definitions in Section 2 are precise and internally coherent, but their connection to the empirical patterns in Sections 3-4 is interpretive—the survey interprets the empirical patterns through the formal lens, but does not demonstrate that the formal lens generated novel predictions that were subsequently confirmed.
Mitigation status. The survey does not claim that the formalism has been empirically validated as a design methodology—it positions the formalism as "the foundation for systematic investigation" (Section 2) and a "shared vocabulary" (Section 1) rather than as a proven engineering framework. This is a reasonable scope for a survey, but it means the central intellectual contribution (the POMDP abstraction) remains at the level of a conceptual proposal rather than an empirically grounded theory of agentic learning.
6.3 Coverage of the Literature Is Extensively Cataloged but Inconsistently Evaluated—The Survey Does Not Distinguish Between Robust and Fragile Results
The assumption or constraint. The survey synthesizes "over five hundred recent works" (Abstract) and organizes them into the capability-task taxonomy. However, the survey's approach to evaluating the cited works is primarily aggregative rather than critical. It reports the claims of each paper largely at face value—what the paper says it achieved, on what benchmark, with what method—without systematically assessing the robustness of those claims. There is no discussion of which results have been replicated across independent groups, which benchmarks are known to be saturated or contaminated, which results have been contradicted by subsequent work, or which findings are sensitive to specific implementation details.
The consequence. The survey's extensive tables (Tables 2-8) and method listings create an impression of a mature, well-validated field, but the underlying reliability of individual results is highly variable. A paper claiming "+10.5% pass-rate improvements" (PRLCoder, Section 4.2.1) and a paper claiming "state-of-the-art performance" on a benchmark may have very different levels of robustness—the former might be a carefully controlled ablation, the latter might be a one-off result sensitive to random seed. The survey's flat presentation obscures these differences. A practitioner who uses the survey as a guide may invest in implementing a method whose reported results are fragile, non-reproducible, or applicable only under narrow conditions not disclosed in the original paper. The survey's comprehensiveness paradoxically makes it harder for practitioners to identify which results are reliable—everything is presented with equal weight.
What evidence exists in the paper. The evidence for this limitation is in the structure of the survey itself. Section 4 presents results as chronological narratives: "Method A achieved X on benchmark Y; Method B subsequently achieved X+δ; most recently, Method C achieved X+2δ." There is no systematic attempt to assess whether Method C's improvement over Method B is statistically significant, whether the benchmark is contaminated, whether the base model was comparable, or whether the training budget was equivalent. The survey occasionally notes limitations of specific works—for example, it observes that "most open-source models still fail on OpenAI's BrowseComp" (Section 4.1.2) and that closed-source systems "are markedly stronger"—but these observations are ad hoc rather than systematic. There is no section, table, or metric that characterizes the robustness of results across the surveyed literature.
Mitigation status. The survey does not attempt to address this. It acknowledges the heterogeneity of evaluation protocols (Section 1: "inconsistent terminology and evaluation protocols, making systematic comparison and cross-domain generalization difficult") but does not itself impose evaluation standards or flag particularly fragile results. The survey's contribution is breadth of coverage, not depth of critical assessment. This is a legitimate tradeoff for a survey of this scale, but it means the survey cannot be used as a reliability guide—practitioners must independently assess the robustness of cited results before adopting methods.
6.4 The Survey Provides No Quantitative Framework for Reasoning About Compute Costs, Making Resource Allocation Decisions Unguided
The assumption or constraint. The survey repeatedly references the computational demands of Agentic RL—"massively parallel, fully asynchronous RL infrastructure" (ComputerRL, Section 4.4.3), "the API cost is too high and severely limits scalability" (ZeroSearch motivation, Section 4.1.1), and the quadratic scaling of sample complexity in multi-agent systems (Section 4.7.3)—but provides essentially no quantitative data on actual compute requirements across the surveyed methods. How many GPU-hours does it take to train a DeepCoder-14B? What is the wall-clock time for WebAgent-R1's asynchronous trajectory generation on a standard cluster? How do training costs compare between outcome-only GRPO (costing only policy forward passes) and PPO with a separate critic network (costing additional critic forward/backward passes)? How does the cost of process-level reward annotation (e.g., step-wise human labels or automated verifier training) compare to the cost of additional policy training with sparse rewards? The survey does not address these questions.
The consequence. A practitioner deciding whether to adopt Agentic RL faces a resource allocation problem: given a fixed compute budget, should they invest in more pretraining, more SFT data, or RL training? The survey provides no quantitative basis for answering this question. The capability progression it documents (RL-free → outcome-only RL → process-level RL) implies increasing compute requirements (more complex reward signals, longer training, more environment interactions), but the magnitude of the increase is never quantified. Similarly, the survey's analysis of the mechanistic debate (Section 6.4) has direct resource implications—if RL is primarily an amplifier, investing in better pretraining may be more cost-effective than scaling RL; if RL creates new capabilities, the opposite may hold—but the survey provides no cost-effectiveness data to inform this decision. The practical consequence is that the survey's recommendations (e.g., "use process-level rewards for long-horizon tasks") are unmoored from the resource constraints that real deployments face.
What evidence exists in the paper. The survey's silence on compute costs is evident throughout. Section 5 catalogs environments and frameworks with key features but no performance benchmarks (e.g., throughput comparisons). Section 4 reports benchmark scores without training budgets (e.g., rStar2-Agent achieves 80.6% on AIME24 "in just 510 RL steps"—but what is the wall-clock time of 510 RL steps? How many GPUs? What model size?). The survey notes in Section 6.2 that "scaling RL training computation directly enhances the reasoning ability of LLM-based agents" and cites the Agent RL Scaling Law study showing "longer training horizons systematically improve tool-use frequency, reasoning depth, and overall task accuracy," but the cited scaling law relates training steps to performance, not compute cost to performance. The difference matters: 1000 GRPO steps on a 7B model have a very different cost than 1000 PPO steps on a 32B model.
Mitigation status. The survey does not address this gap. It flags compute scaling as an "open challenge" (Section 6.2: "scaling demands massive compute, making efficiency vital") and discusses efficiency-focused methods (POLARIS, Dynamic Fine-Tuning), but does not provide a systematic framework for reasoning about compute-resource tradeoffs. This limitation is partially inherent to the survey format—standardized cost reporting does not exist across the 500+ cited papers—but the survey could have noted this as a specific deficiency in the literature and called for standardized cost reporting as a community norm.
6.5 The Survey's Taxonomy and Framework Are Static—They Do Not Account for Capability Interactions, Transfer Effects, or the Dynamics of the Agentic RL Training Process
The assumption or constraint. The survey organizes its analysis around a static capability taxonomy (planning, tool use, memory, self-improvement, reasoning, perception) and a static application-capability dependency matrix (Table 9). Each capability is analyzed largely in isolation (Section 3.1–3.6), with cross-references to related capabilities but no formal model of how capabilities interact, conflict, or transfer during joint RL training. The dependency matrix marks capabilities as "core," "supporting," or "minimal" for each task domain, but this is a qualitative snapshot that does not capture dynamics—how the importance of a capability changes as the agent learns, how improving one capability might degrade another, or how capabilities learned in one domain might transfer to another.
The consequence. The survey's model of Agentic RL is additive—train each capability module with RL, and the agent improves. But real multi-capability training involves complex interactions that the survey does not address. Evidence from the cited literature itself shows this: Section 6.2 (Data Size) notes that "some [domain] pairings enhance each other, while others interfere and reduce performance" and that "model initialization also matters—instruction-tuned models generalize differently than raw ones." The Guru dataset shows that "math and code benefit from transfer, but domains like simulation or logic need dedicated training." These findings—cited within the survey—directly challenge the additive model implicit in the capability taxonomy. If training on code interferes with logic reasoning, then a practitioner cannot simply "add RL for tool use" and "add RL for reasoning" and expect additive gains—the capabilities interact in ways the taxonomy does not capture.
Similarly, the survey's progressive model (each capability advances through RL-free → outcome-only RL → process-level RL) assumes that the progression is independent for each capability. But in practice, improving tool use through process-level RL might accelerate or hinder the progression of planning, because the agent now has more reliable tools and may plan differently. The survey's framework provides no language for describing these capability-capability interactions during training.
What evidence exists in the paper. The survey itself provides evidence that interactions matter, even as its framework does not model them. Section 6.2's discussion of cross-domain RL (Li et al., 2025o) explicitly documents interference effects. The case study in Section 6.4 notes that new-capability behaviors emerge under conditions including "compositional or multi-step structure where many partial trajectories are verifiably graded"—this implies that the interaction between reasoning and tool use (compositional structure) and the interaction between self-improvement and environment design (verifiable partial trajectories) are what enable emergence, not any single capability in isolation. The survey's framework can describe these interactions post-hoc but cannot predict them.
Mitigation status. The survey does partially address this through its application-capability matrix and the prospective sections that envision integrated capabilities (e.g., the synthesis of deliberation and intuition in planning, Section 3.1; the integration of slow reasoning into agentic settings, Section 3.5). However, these are forward-looking aspirations rather than analytical tools. The survey acknowledges that "we did not experiment with PRM tree-search techniques in combination with revisions" (quoting a separate analysis paper but reflecting a broader pattern in the literature) and that combining search with revisions "could yield gains beyond either method alone." The framework's static nature is thus a limitation of the current state of the field—most work studies capabilities in isolation—but the survey's decision to mirror this fragmentation in its taxonomy, rather than providing a model of interactions, limits its ability to guide research toward integrated systems.
6.6 The Survey Does Not Engage With the Question of When Agentic RL Is the Wrong Approach—It Assumes RL Is the Solution Without Delineating Failure Regimes
The assumption or constraint. The survey's thesis is that "reinforcement learning serves as the critical mechanism for transforming these capabilities from static, heuristic modules into adaptive, robust agentic behavior" (Abstract). This framing treats RL as universally beneficial for agentic capabilities. The survey does not systematically address the question: when should a practitioner NOT use RL? When is SFT sufficient? When does RL fail to improve over carefully designed heuristics? When does RL actively degrade performance (reward hacking, catastrophic forgetting, mode collapse)?
The consequence. A practitioner who accepts the survey's framing at face value might conclude that RL should be applied to every capability in every agentic system. But the literature the survey itself cites contains counterexamples. Section 6.1 (Hallucination) notes that "outcome-driven RL, which rewards only the correctness of the final answer, can encourage agents to find spurious correlations or shortcuts" and that "RL-only post-training can worsen factuality." Section 6.1 (Sycophancy) observes that "RLHF can directly incentivize and exacerbate sycophantic tendencies." The ReST^EM experiment (Appendix K of the survey, cited in the separate analysis paper) shows that RL-based self-improvement can "substantially hurt performance." These are not minor caveats—they are documented failure modes where RL makes things worse. Yet the survey does not systematize these failure modes into guidance about when RL is contraindicated.
What evidence exists in the paper. The survey collects evidence of RL's downsides but does not organize it into actionable warnings. Section 6.1 (Trustworthiness) discusses security risks (reward hacking), hallucination amplification, and sycophancy—all ways RL can degrade agent behavior. The mechanistic debate (Section 6.4) acknowledges that RL may function as "primarily an amplifier that sharpens and reuses patterns already implicit in the pretrained model"—which implies that for tasks where the base model already performs well, RL may provide marginal benefit at substantial cost. The survey notes that "on easier, low-noise benchmarks or with coarse outcome-only rewards, RL predominantly acts as an amplifier." These observations, scattered across different sections, amount to a partial catalog of RL's limitations, but the survey never assembles them into a coherent framework for deciding whether to use RL at all.
Mitigation status. The survey partially addresses this through its characterization of the mechanistic debate and its discussion of trustworthiness challenges (Section 6.1), but these are framed as challenges to be overcome rather than as boundary conditions on RL's applicability. The survey's rhetorical commitment to RL as "the critical mechanism" for transforming agentic capabilities makes it difficult for the survey to simultaneously argue that there are circumstances where RL should not be applied. The survey acknowledges that RL's effects are "highly dependent on the training pipeline" (Section 6.1, Hallucination) and that "a structured approach combining SFT with a verifiable-reward RL process can mitigate" degradation—but this is a statement about training recipes, not about the fundamental decision of whether to apply RL. A practitioner reading the survey would benefit from explicit guidance: under conditions A, B, and C, RL is likely to improve performance; under conditions X, Y, and Z, RL is likely to degrade it or provide negligible benefit. The survey does not provide this.
7. Implications and Future Directions
How This Work Changes the Landscape
This survey changes the landscape not by proposing a new algorithm or model—it offers no training recipe, no benchmark score, no novel loss function—but by providing something the field has conspicuously lacked: a shared conceptual language and organizational framework that spans the entire heterogeneous space of Agentic RL research. The nature of this contribution is best understood as an exercise in intellectual infrastructure construction. Before this survey, the field of Agentic RL consisted of hundreds of papers operating in parallel but largely disconnected research threads—each using its own terminology, its own evaluation protocols, its own assumptions about what constitutes an "agent," and its own tacit understanding of what RL contributes. The survey's formal MDP/POMDP distinction (Section 2), capability taxonomy (Section 3), task-domain mapping (Section 4), and infrastructure consolidation (Section 5) together constitute a lingua franca that makes cross-domain comparison and synthesis possible for the first time.
The magnitude of the shift is best characterized as a reframing, not a paradigm shift. A paradigm shift would require overturning a previously dominant framework; but Agentic RL did not have a dominant framework to overturn—it had fragmentation. The survey's contribution is to impose order on chaos, to reveal that seemingly disparate research threads (GRPO-trained search agents, PPO-trained GUI agents, ExIt-trained theorem provers, MAGRPO-trained multi-agent systems) are instances of a common POMDP formulation differing only in their state representations, action spaces, and reward structures. This is reframing work in the classic sense: the same phenomena, once viewed through the survey's lens, reveal structural identities that were previously invisible. A researcher working on code agent RL can now recognize that their credit assignment problem is structurally identical to one faced by a search agent researcher; a memory system designer can see that their progression from static retrieval to RL-guided management mirrors the same progression occurring in tool use and planning.
Reconciliation of prior contradictions. The survey implicitly reconciles a tension that pervades the literature: why do some papers report dramatic RL improvements while others find RL degrades performance or provides negligible gains? The survey's answer, distributed across its capability taxonomy and the mechanistic debate in Section 6.4, is that RL's effectiveness is conditional on specific structural properties of the task and the training setup. High-fidelity verifiable rewards, compositional multi-step structure, and base models in an intermediate capability regime enable RL to produce genuine capability emergence; sparse outcome rewards, low-compositionality tasks, and base models near the capability ceiling or floor produce primarily distribution reshaping (the "amplifier" effect). This conditional account explains why RLHF can exacerbate sycophancy (Section 6.1) while outcome-only RL can induce sophisticated reasoning in DeepSeek-R1—the difference lies not in the algorithm family but in the reward structure, the task compositionality, and the base model's initial capability. The survey thus converts a confusing set of contradictory empirical claims into a coherent conditional framework.
Which research directions become more attractive. The survey's framework makes several research directions newly compelling. The most important is process-level credit assignment mechanisms for long-horizon tasks—the survey's capability progressions (Sections 3.2, 3.4, 3.5) repeatedly identify the transition from outcome-only to process-level RL as the key enabler of sophisticated agentic behavior, yet the current state of process-level methods (turn-level advantage estimation in GiGPO, step-wise decomposition in SpaRL) is nascent. The survey's formal analysis makes clear why this transition is necessary (the POMDP's temporal credit assignment problem cannot be solved by trajectory-level rewards alone) and provides a taxonomy of existing approaches organized by capability, enabling cross-pollination. Similarly, the survey's identification of verifier over-optimization—"the primary bottleneck preventing unbounded improvements from additional compute" (Section 6.1, building on the separate analysis paper)—directs attention toward robust reward design and away from merely scaling search or training steps.
The survey also makes certain directions less attractive by clarifying their fundamental limitations. Pure scaling of RL training steps without addressing credit assignment (Section 6.2) is revealed as a diminishing-returns strategy—the Agent RL Scaling Law (Mai et al., 2025) shows monotonic improvements, but the survey's analysis of reward sparsity implies these improvements will plateau without denser feedback. Offline, static-dataset RL for interactive tasks (e.g., GUI agents trained only on static trajectories, Section 4.4.2) is revealed as fundamentally incapable of producing error-recovery behaviors because the environment never provides stochastic feedback. The survey does not argue against these directions but clarifies their boundary conditions, enabling researchers to invest effort where the structural conditions for success are met.
The landscape after this survey. The survey's most significant practical effect on the field will likely be on how new Agentic RL papers are written, reviewed, and compared. A paper proposing a new credit assignment method for code agents can no longer present itself as solving a domain-specific problem; the survey's framework compels it to position itself within the broader capability progression (is this advancing tool-use RL from outcome-only to process-level? Is it addressing the same temporal credit assignment challenge identified for search agents and GUI agents?). A paper evaluating a new multi-agent RL algorithm can no longer claim novelty based solely on the number of agents or the task domain; the survey's application-capability matrix (Table 9) and the multi-agent RL taxonomy (Section 4.7.3) make clear which dimensions of the problem (credit assignment structure, communication protocol, training centralization) are the meaningful axes of comparison. This standardization of evaluation and comparison is, in itself, a form of progress—it reduces duplication, enables accumulation of knowledge, and directs attention toward genuinely open problems rather than reinvention of known solutions in new domains.
Follow-Up Research This Work Enables
A systematic benchmark for cross-domain credit assignment mechanisms. The survey identifies temporal credit assignment as the central bottleneck across capabilities (tool use, Section 3.2; planning, Section 3.1; self-improvement, Section 3.4; multi-agent, Section 4.7.3), but the literature currently evaluates credit assignment methods on domain-specific benchmarks (SWE-bench for code, BrowseComp for search, miniF2F for math) with incomparable metrics. The survey's formal POMDP framework provides the vocabulary for constructing a unified credit assignment benchmark: define tasks with controlled horizon length , controlled reward sparsity (fraction of steps with non-zero reward), controlled action-space compositionality (number of tools, tool interaction depth), and controlled partial observability. Methods like GiGPO's turn-level advantage, SpaRL's step-wise decomposition, and MAGRPO's role-conditioned advantage estimation could then be compared on the same axes. A strong follow-up would construct such a benchmark spanning 3–4 domains (code, search, GUI, embodied) with standardized metrics, measuring how each method's performance degrades as horizon length increases and reward density decreases. The survey's Table 9 provides the mapping from capabilities to domains needed to select representative tasks.
Testing the survey's capability progression hypothesis through longitudinal study of a single domain. The survey proposes that each capability passes through the same progression—RL-free → outcome-only RL → process-level RL → prospective stage—but this claim is based on cross-sectional evidence across domains at different stages of maturity. A direct test would take a single agentic task (e.g., a complex multi-tool code generation benchmark) and systematically apply each stage of the progression: first, establish the SFT baseline on static trajectories; second, apply outcome-only RL with execution-based rewards; third, implement process-level rewards (compilation checks, assertion passes, code-review critiques); fourth, measure whether the progression of improvements matches the survey's prediction that outcome-only RL provides gains over SFT but plateaus due to credit assignment, while process-level RL sustains improvement to longer horizons. The survey's identification of the mechanistic debate conditions (high-fidelity verifiable rewards, compositional multi-step structure, intermediate base model capability, Section 6.4) provides the criteria for selecting a task where the full progression should be observable.
Investigating capability interference and transfer during joint RL training. The survey's capability taxonomy treats capabilities largely in isolation, but Section 6.2 (Data Size) cites evidence that cross-domain RL produces both synergy and interference—"some pairings enhance each other, while others interfere and reduce performance" (Li et al., 2025o). The Guru dataset (Cheng et al., 2025) shows that "math and code benefit from transfer, but domains like simulation or logic need dedicated training." The survey's framework makes this phenomenon newly tractable: by formalizing each capability as an RL-optimizable module within the POMDP, one can design controlled experiments where capabilities are jointly trained in pairs (tool use + planning, reasoning + memory, perception + tool use) and measure both the main effect of joint training and the interaction effect (positive vs. negative transfer). A strong follow-up would systematically sweep pairs of capabilities on a fixed base model and task suite, measuring not only final performance but also the learning dynamics—does joint training accelerate or decelerate the progression of each capability through the RL-free → outcome-only → process-level stages? The survey's application-capability matrix (Table 9) predicts which capability pairs are most likely to interact (e.g., planning and tool use for search agents, both marked "core") and thus provides a hypothesis space for exploration.
Quantifying when Agentic RL is cost-ineffective relative to SFT or pretraining scaling. The survey documents that RL improves agent performance across domains but provides no quantitative framework for cost-effectiveness analysis (Section 6.4). A direct follow-up—analogous to the FLOPs-matched comparison in the separate analysis paper (not this survey)—would match total compute budgets across three conditions for a fixed task: (a) SFT on additional expert demonstrations, (b) outcome-only RL with verifiable rewards, and (c) process-level RL with intermediate supervision signals. For each condition, measure both final task performance and the compute cost (GPU-hours, environment interactions). The survey's analysis of the mechanistic debate predicts that RL should be most cost-effective relative to SFT under conditions of high-fidelity verifiable rewards and compositional multi-step structure; a negative result—finding that SFT with sufficient data matches RL at equal cost—would substantially qualify the survey's thesis that RL is "the critical mechanism for transforming these capabilities."
Developing structured memory RL with the survey's capability progression as a guide. The survey identifies RL for structured memory (temporal knowledge graphs, atomic memory notes, hierarchical graphs) as the prospective frontier for the memory capability (Section 3.3), noting that "little work has explored the use of RL to dynamically control the construction, refinement, or evolution of such structured memory." The survey's progression framework makes concrete what this unexplored direction requires: define structured memory operations (graph node insertion, edge creation, abstraction, pruning) as discrete actions within the POMDP action space , design reward signals that measure downstream task improvement from specific memory operations (analogous to how tool-use RL rewards tool invocations that improve task outcomes), and train an RL policy that decides when and how to modify the memory structure. A strong follow-up would implement this for one structured memory representation (e.g., a temporal knowledge graph as in Zep) on a multi-session QA benchmark, comparing RL-guided memory management against the handcrafted heuristics currently used, and measuring whether the learned policy discovers non-obvious memory strategies (e.g., proactively deleting outdated information, creating abstraction nodes that compress related facts).
Stress-testing the survey's framework: when does the capability progression fail? The survey's framework makes a strong implicit prediction—that capabilities reliably progress through RL-free → outcome-only RL → process-level RL—but its own evidence base includes counterexamples. The ReST^EM-trained revision model degraded with additional sequential revisions (separate analysis paper, Appendix K); hallucination can be amplified by outcome-only RL (Section 6.1); and certain domain pairings show interference rather than synergy (Section 6.2). A systematic stress test would identify, ex ante, conditions under which the progression should fail—for instance, tasks where the verifiable reward signal is systematically misaligned with the true objective (reward hacking), tasks where the base model's capability is near-zero (no competent trajectories to amplify), or tasks where the action space is so large that exploration never encounters positive reward (sparse reward starvation). The survey's formal POMDP framework provides the vocabulary for specifying these conditions, but the survey itself does not delineate the failure boundaries. A negative-results paper that deliberately constructs tasks meeting these failure conditions and demonstrates that RL either provides no improvement or degrades performance would significantly strengthen the framework by calibrating its domain of applicability.
Practical Applications and Downstream Use Cases
Structured research agent development using the survey's capability-task mapping. Organizations building deep research agents (search + synthesis + report generation) can use the survey's application-capability matrix (Table 9) to prioritize development effort. The matrix identifies planning and tool use as "core" for search agents, perception as "minimal." This directly informs engineering resource allocation: invest in RL formulations for multi-turn query decomposition (planning, Section 3.1) and tool invocation optimization (tool use, Section 3.2) using the PPO/GRPO-based methods cataloged in Section 4.1 (DeepRetrieval for query generation, AutoRefine for iterative refinement, ASearcher for long-horizon scaling), while deferring investment in visual understanding. The survey's infrastructure consolidation (Section 5.1) provides the environment selection guidance: WebArena and Mind2Web for web-based training, AgentGym for general capability improvement. The survey's finding that "most open-source models still fail on OpenAI's BrowseComp" (Section 4.1.2) while closed-source systems achieve 51.5% pass@1 provides a concrete performance target and gap analysis.
Cost-efficient code agent training by staging the capability progression. Organizations deploying LLMs for software engineering can adopt the survey's staged progression model (Section 4.2) rather than jumping directly to end-to-end SWE-bench RL. Stage 1: single-round code generation with outcome-only RL (DeepCoder-14B's approach: GRPO with sparse Outcome Reward Model to prevent reward hacking) on HumanEval and MBPP. Stage 2: iterative refinement with execution-feedback RL (RLEF's approach: grounding correction loops in real error messages) on competitive programming benchmarks. Stage 3: full software engineering with repository-level RL (DeepSWE's approach: verified task completion as sole reward) on SWE-bench. The survey's documentation that process-level rewards (PRLCoder's +10.5% pass-rate improvement) accelerate convergence but require engineering investment provides a cost-benefit framework for deciding when to advance to the next stage. The survey's framework consolidation (Section 5.2, Table 11) identifies VerlTool and AgentFly as suitable RL frameworks for tool-integrated code agent training.
GUI agent deployment with the survey as a training paradigm selection guide. Organizations deploying GUI automation agents face a choice between three training paradigms cataloged in Section 4.4: RL-free (vanilla VLMs or SFT on static trajectories, Section 4.4.1), RL in static environments (GUI-R1, UI-R1, Section 4.4.2), and RL in interactive environments (WebAgent-R1, MobileGUI-RL, UI-TARS, Section 4.4.3). The survey's analysis indicates that interactive RL enables error-recovery behaviors that static RL cannot capture, but at substantially higher infrastructure cost (massively parallel asynchronous training, Android virtual devices). For deployment scenarios where the cost of action errors is high (financial transactions, medical systems), the survey's finding that online RL agents learn robustness to stochastic transitions (Section 4.4.3) justifies the infrastructure investment. For scenarios where tasks are deterministic and human demonstration data is abundant, static SFT may suffice. The survey's environment catalog (Table 10: OSWorld, AndroidWorld, WindowsAgentArena) provides the implementation substrate; its framework catalog (Table 11: AgentFly for scalable asynchronous execution, ComputerRL for API-GUI hybrid interaction) provides the training infrastructure.
Multi-agent system design using the survey's credit assignment taxonomy. Organizations deploying multi-agent LLM systems (Section 4.7) must choose between three levels of RL integration: RL-free coordination (CAMEL, MetaGPT—Section 4.7, pre-RL), RL-optimized coordination modules (GPTSwarm, MaAS—Section 4.7.1, where only communication topologies are RL-trained), and end-to-end multi-agent RL (MAGRPO, MAPoRL—Section 4.7.3, where all agent policies are jointly optimized). The survey's analysis that "sample complexity grows roughly linearly with the number of agents and quadratically with interaction depth" (Section 4.7.3) provides a practical scaling law for deciding which tier is feasible given available compute. For a deployment with 2–3 agents performing shallow interactions (query → response → selection), end-to-end RL with MAGRPO's decentralized execution may be tractable. For a deployment with 10+ agents performing deep iterative coordination (debate, iterative refinement, hierarchical planning), the survey's analysis implies that RL-optimized coordination modules (training only the communication topology) or even RL-free protocols may be the only practically scalable option. The survey's documentation that partial policy optimization (training only a subset of agents, Section 4.7.2) "yields better sample efficiency than fully joint multi-agent training while still enabling the emergence of specialized roles" provides a middle-ground engineering strategy.
