ArXiv: 2601.12538
🎯 Pitch
Moving beyond static benchmarks, this survey reveals that LLM reasoning fundamentally fails in open-ended settings unless models are redesigned as autonomous agents that plan, act, and adapt through continuous environmental feedback. By defining a three-layer taxonomy—foundational, self-evolving, and collective—it shows how reasoning scales not via bigger models, but by bridging thought and action across dynamic interaction modes.
1. Executive Summary
This survey synthesizes and formalizes the paradigm of Agentic Reasoning, organizing it into a three-layer taxonomy—foundational single-agent capabilities, self-evolving adaptation, and collective multi-agent coordination—and analyzing each layer through the complementary lenses of in-context reasoning (scaling test-time interaction via structured orchestration, e.g., ReAct-style interleaving or tree-search planning) and post-training reasoning (internalizing strategies via reinforcement learning or supervised fine-tuning, e.g., GRPO-based policy optimization). The survey contextualizes these mechanisms across five application domains—mathematical exploration and vibe coding, scientific discovery, embodied robotics, healthcare and medicine, and autonomous web exploration—and reviews over 100 benchmarks that isolate core primitives like tool use, search, memory, and multi-agent collaboration. A central insight is that agentic reasoning marks a paradigm shift from passive sequence prediction to interactive, adaptive intelligence, establishing that reasoning scales through structured interaction—planning, feedback integration, and role specialization—only when systems are designed to bridge thought and action across environmental dynamics, optimization settings, and agent interaction modes.
2. Context and Motivation
The Core Problem: Reasoning Must Extend Beyond Static Inference
The fundamental gap this survey addresses is the disconnect between how large language models (LLMs) have traditionally been evaluated for reasoning and what reasoning actually demands in real-world, interactive settings. Standard LLM reasoning benchmarks—GSM8K, MATH, AIME—treat reasoning as a single-pass, closed-world prediction task. A model receives a fixed prompt, generates a chain of thought, and produces a final answer. The environment never changes, the model never acts, and there is no feedback loop. The paper explicitly frames this as insufficient:
"Conventional LLMs lack mechanisms to act, adapt, or improve in open-ended environments where information evolves over time."
This gap is not merely philosophical. It manifests concretely across virtually every domain the survey examines. In scientific discovery, a model that can solve static math problems cannot autonomously design experiments, interpret contradictory evidence, or revise hypotheses when new data arrives. In embodied robotics, a model that generates step-by-step plans in text cannot ground those plans in physical perception, recover from execution failures, or adapt to novel object configurations. In healthcare, a model that answers clinical exam questions cannot maintain longitudinal patient context across encounters, reconcile conflicting specialist opinions, or update recommendations when lab results change mid-consultation.
The survey argues that this transformation requires reframing reasoning as agentic: not a property of the model alone, but a property of the model-in-environment loop. The paper's definition is explicit on this point:
"Agentic reasoning positions reasoning as the central mechanism of intelligent agents, spanning foundational capabilities (planning, tool use, and search), self-evolving adaptation (feedback and memory-driven adaptation), and collective coordination (multi-agent collaboration), realizable through either in-context orchestration or post-training optimization."
Why This Gap Is Important: The Shift from Scaling Parameters to Scaling Interaction
The importance of bridging this gap stems from two converging trends that the survey identifies but does not always explicitly separate: the saturation of static reasoning benchmarks and the practical impossibility of solving open-ended tasks through pretraining alone.
Saturation of static benchmarks. The paper notes that widely used math datasets "are increasingly saturated, which makes it difficult to distinguish among modern high-performing models." When GSM8K and MATH cease to differentiate frontier models, the field needs richer evaluation paradigms. But more critically, these benchmarks never required agents to act—they only tested whether a model could produce a correct final answer under idealized conditions. This creates a dangerous illusion of progress: a model that scores 95% on MATH may be entirely incapable of formulating its own research questions, debugging its own erroneous derivations, or recognizing when its assumptions no longer hold.
The impossibility of pretraining-only solutions. The survey implicitly argues—and supports through its systematic review of over 100 benchmarks and dozens of application-domain systems—that certain reasoning capabilities simply cannot be acquired through pretraining on static text corpora, regardless of scale. Consider the concrete example from the paper's discussion of embodied agents: a model that has never interacted with a physical environment cannot learn to adjust its grip force based on haptic feedback, because that feedback only exists during interaction. Similarly, a model that has never collaborated with another agent cannot learn to negotiate task allocation, because the negotiation protocol emerges from the dynamics of multi-agent communication, not from observing fixed transcripts.
This has profound practical implications. If certain reasoning capabilities are interaction-gated—only learnable through experience—then the research community needs to shift investment from ever-larger pretraining runs toward architectures and training paradigms that support continual learning through interaction. The paper's taxonomy is designed to make this shift tractable by providing a vocabulary and organizational scheme for reasoning about how interaction improves reasoning, not just whether it does.
Theoretical significance: reasoning as a control problem. The paper formalizes this shift mathematically in Section 2.2, modeling agentic reasoning as a Partially Observable Markov Decision Process (POMDP) with an explicit internal reasoning variable. This is not cosmetic notation—it reflects a substantive claim: that reasoning should be analyzed using the tools of sequential decision-making under uncertainty, not just language modeling. The factorization of the policy into an internal thought component and an external action component:
establishes that agentic systems perform computation in a reasoning trace space before committing to actions. This separates the survey's approach from both pure language modeling (where there is no action) and pure reinforcement learning (where reasoning is often treated as an implicit, unobservable process). The survey argues that making reasoning explicit and architecturally first-class is necessary for building systems that can plan, reflect, and collaborate.
Where Prior Approaches Fall Short
The paper positions itself at the intersection of two established but largely disconnected literatures: LLM reasoning and AI agent architectures. Its contribution is not to identify a gap that no one has noticed, but to provide the unifying framework that reveals why existing work in each silo is insufficient.
Shortcoming 1: LLM reasoning research treats reasoning as a model-internal process. The survey acknowledges substantial prior work on chain-of-thought prompting, decomposition techniques, program-aided reasoning, and reinforcement post-training. These methods, it notes, "improve the internal expressiveness of reasoning traces but typically remain within static inference settings, where reasoning unfolds in a single forward pass without external interaction." The survey cites Huang and Chang (2022), Chen et al. (2025), Xu et al. (2025), and Ke et al. (2025) as representative of this paradigm.
The limitation is not that these methods are wrong—chain-of-thought prompting genuinely improves reasoning accuracy—but that they are architecturally incomplete. A model that reasons step-by-step about a math problem still cannot:
- Decide when to stop reasoning and start acting (the exploration-exploitation tradeoff)
- Update its reasoning strategy based on whether its actions succeeded or failed
- Coordinate its reasoning with other agents that have different information or capabilities
The paper's framework addresses this by showing that these are not separate problems—they are different manifestations of the same underlying requirement: reasoning must be situated in an environment, not abstracted from it.
Shortcoming 2: AI agent surveys treat reasoning as an implicit byproduct of architecture. The paper differentiates itself from surveys like Zhang et al. (2025), Lin et al. (2025), Fang et al. (2025), and Gao et al. (2025), which analyze agent architectures through the lens of reinforcement learning, tool-use modules, or self-evolution. The survey acknowledges these as valuable but argues that they "focus on reinforcement learning for agentic search and decision-making" and "emphasize self-evolving and lifelong agentic systems" without centering on the reasoning process itself.
Specifically, the paper argues that existing agent surveys often treat reasoning as something that emerges from putting the right modules together—a planner here, a memory module there, a tool-use interface. The survey's contribution is to flip this perspective: reasoning is not the output of good architecture; good architecture is the output of understanding what reasoning requires. The three-layer taxonomy (foundational, self-evolving, collective) is designed to reflect stages of reasoning sophistication, not just system components. Each layer introduces qualitatively new reasoning demands: planning and tool use test whether an agent can decompose goals and act on them; self-evolution tests whether an agent can learn from the consequences of its actions; collective reasoning tests whether multiple agents can coordinate their reasoning processes toward shared goals.
Shortcoming 3: The literature is fragmented, with conflicting findings explained by a missing variable—environmental dynamics. One of the survey's most important implicit arguments is that the apparent contradictions in the literature—debates about whether self-correction works, whether tree search helps, whether multi-agent debate improves accuracy—stem from a failure to condition on environmental dynamics and task difficulty. A method that works beautifully on a stable, closed-world math benchmark may fail catastrophically when the environment changes mid-episode. A self-reflection loop that improves code generation may degrade performance when the feedback signal becomes noisy in embodied settings.
The paper's taxonomy is designed to resolve this fragmentation by making environmental dynamics a first-class dimension of analysis. The three complementary scopes—foundational, self-evolving, collective—are not arbitrary categories; they correspond to progressively more dynamic and demanding environments. Foundational reasoning operates in stable environments where goals are clear and feedback is reliable. Self-evolving reasoning operates in environments that change over time, requiring adaptation. Collective reasoning operates in environments where other agents are part of the environment, introducing strategic complexity. By organizing methods along this dimension, the survey provides a framework for understanding when and why different approaches succeed or fail, rather than simply cataloging them.
How This Paper Positions Itself Relative to Existing Work
The survey's positioning can be understood through three deliberate choices about scope, structure, and contribution type.
Not a method paper—a synthesis and roadmap. The survey does not propose a new agent architecture, training algorithm, or benchmark. Its contribution is conceptual and organizational: it provides the vocabulary, taxonomy, and formal framework that the field needs to reason about agentic reasoning as a unified phenomenon rather than a collection of loosely related techniques. This is evident from the paper's structure: rather than organizing by method (e.g., "ReAct," "Tree of Thoughts," "Voyager"), it organizes by capability layer (foundational, self-evolving, collective) and optimization mode (in-context, post-training). This forces the reader to think in terms of what reasoning demands rather than what algorithms exist.
A reasoning-centric lens, not an architecture-centric one. The paper explicitly states this positioning in Section 2.1:
"Our survey provides a reasoning-centric lens on intelligent agency. We examine how foundational reasoning mechanisms, post-training adaptation, and long-term self-evolution jointly constitute the basis of agentic reasoning, illustrating the transition from static prediction to interactive, adaptive, and continually improving intelligence."
This is a methodological choice with consequences. An architecture-centric survey would organize around components (memory modules, planning modules, tool interfaces). The reasoning-centric survey instead asks: what does reasoning need to accomplish at each layer, and how do different implementations satisfy those needs? This makes the survey more useful for researchers trying to design new systems—they can identify which layer their problem falls into and understand the reasoning demands, rather than being told which off-the-shelf module to plug in.
Bridging two communities through a shared formal framework. The POMDP formulation in Section 2.2 is the survey's primary theoretical contribution. By modeling agentic reasoning as a sequential decision problem with explicit reasoning traces, it creates a common language for the LLM reasoning community (which thinks in terms of chains of thought and prompting strategies) and the RL/MARL community (which thinks in terms of policies, value functions, and credit assignment). This is not just academic formalism—it enables the survey to discuss in-context reasoning as inference-time search (maximizing over reasoning trajectories without parameter updates) and post-training reasoning as policy optimization (updating parameters to improve expected returns) within the same mathematical framework. The collective intelligence extension to Dec-POMDPs further shows that multi-agent reasoning is not a separate problem but a natural generalization of the single-agent case, with communication serving as an extension of the reasoning process rather than a separate mechanism.
Acknowledging what is not covered. The survey sets explicit scope boundaries: it covers reasoning-empowered agentic systems where reasoning drives adaptive behavior, analyzed through in-context and post-training optimization modes. It does not attempt to cover all of multi-agent reinforcement learning, all of tool-use, or all of LLM safety—these are referenced only insofar as they illuminate agentic reasoning specifically. This focused scope is a strength, not a limitation; it prevents the survey from becoming an unmanageable catalog and keeps the analysis centered on the paper's core thesis: that reasoning is the organizing principle for intelligent agency, and that understanding how reasoning scales through interaction is the central challenge for the next generation of AI systems.
3. Technical Approach
3.1 Reader orientation
What is being built: A conceptual framework—not a piece of software—that organizes the entire landscape of agentic reasoning methods into a coherent three-layer taxonomy, where each layer captures reasoning at a different level of environmental dynamism (from stable single-agent settings to evolving self-adaptive systems to collaborative multi-agent ecosystems), and each method within a layer is further categorized by whether it operates through in-context orchestration (inference-time prompting, search, and workflow design without parameter updates) or post-training optimization (reinforcement learning and supervised fine-tuning that internalize reasoning strategies into model weights).
What problem it solves: The survey addresses the fragmentation of the agentic reasoning literature by providing a unified organizational scheme that makes explicit what capabilities each layer demands, what optimization modes are available, and how methods relate to one another—enabling researchers to (1) locate their work within a larger intellectual landscape, (2) identify which reasoning demands their system addresses versus which it ignores, and (3) understand why a method that works in one setting (e.g., static math reasoning) fails in another (e.g., embodied control) in terms of the layer's environmental dynamics rather than superficial implementation differences.
3.2 Big-picture architecture (diagram in words)
The survey constructs its framework through four nested organizational structures:
-
The three-layer capability taxonomy: The outermost structure partitions all agentic reasoning by the environmental dynamics the agent must handle. Foundational agentic reasoning (Layer 1) operates in stable environments with clear goals and reliable feedback—the agent plans, uses tools, and searches, but the world does not change unpredictably. Self-evolving agentic reasoning (Layer 2) introduces environments that change over time, requiring the agent to integrate feedback, maintain persistent memory, and adapt its behavior across episodes without full retraining. Collective multi-agent reasoning (Layer 3) introduces strategic complexity: other agents are part of the environment, requiring role differentiation, communication protocols, and coordination mechanisms.
-
The two-mode optimization axis: Orthogonal to the layers, every method is classified by when and how reasoning strategies are optimized. In-context reasoning modifies behavior at inference time through structured prompting, search algorithms, or workflow orchestration without changing model parameters. Post-training reasoning uses supervised fine-tuning or reinforcement learning to internalize successful strategies into the model's weights, producing persistent behavioral changes that generalize beyond individual inference episodes.
-
The POMDP formal backbone (Section 2.2): Underpinning both dimensions is a mathematical framework that models agentic reasoning as a Partially Observable Markov Decision Process with an explicit reasoning trace variable. This formalism is not merely decorative—it provides the vocabulary for discussing why certain methods work in certain layers. The factorization of the policy into
$\pi_{\text{reason}}(z_t \mid h_t)$(internal deliberation) and$\pi_{\text{exec}}(a_t \mid h_t, z_t)$(external action) establishes that reasoning is computation performed before committing to actions, and that different methods differ in how they implement this computation—whether through in-context search over reasoning traces or post-training optimization of the reasoning policy. -
The application-benchmark mapping: The final organizational layer connects the taxonomy to concrete instantiations. Each of the five application domains (math/code, scientific discovery, embodied robotics, healthcare, autonomous web) is analyzed by identifying which foundational, self-evolving, and collective capabilities it demands, and the benchmark review (Section 7) is organized first by core mechanism (tool use, search, memory, multi-agent) and then by application domain—allowing the reader to trace a capability from the abstract taxonomy to a specific evaluation protocol.
Information flow: The taxonomy is not a pipeline with temporal ordering. Rather, it is a classification scheme: a researcher takes a method (say, Reflexion for code debugging), identifies that it belongs to the self-evolving layer (because it uses reflective feedback from execution traces to improve behavior), determines that it operates primarily through in-context reasoning (because the reflection is appended to the prompt rather than used to update parameters), and then understands its relationship to other methods in the same cell of the taxonomy (e.g., other reflective feedback methods like Self-Refine) and to methods in adjacent cells (e.g., parametric adaptation methods like ReST that also use feedback but internalize it through training).
3.3 Roadmap for the deep dive
-
First, the POMDP formalization (Section 2.2): This is the theoretical foundation—understanding the tuple
$\langle \mathcal{X}, \mathcal{O}, \mathcal{A}, \mathcal{Z}, \mathcal{M}, \mathcal{T}, \Omega, \mathcal{R}, \gamma \rangle$and the policy factorization is prerequisite to understanding why in-context reasoning is modeled as inference-time search and post-training reasoning as policy optimization. Without this, the classification scheme appears arbitrary. I will explain each component of the POMDP, define the reasoning trace space$\mathcal{Z}$as distinct from the action space$\mathcal{A}$, and walk through the policy factorization and its implications. -
Second, in-context reasoning as search (Section 2.2, continued): Once the POMDP vocabulary is established, I will explain how in-context reasoning methods—ReAct's interleaved thought-action loops, Tree of Thoughts' MCTS-style exploration, and structured workflow designs—are all special cases of search over reasoning traces
$\mathcal{Z}$with frozen parameters$\theta$, maximizing some heuristic value function. This reveals the common structure underlying apparently disparate methods. -
Third, post-training reasoning as policy optimization (Section 2.2, continued): I will then explain the complementary paradigm: how reinforcement learning (specifically GRPO) and supervised fine-tuning optimize the policy parameters
$\theta$to align behavior with long-horizon rewards. The GRPO objective (Equation 3) and its group-relative advantage (Equation 4) will receive the full four-part equation treatment. This shows how methods like DeepSeek-R1 and Search-R1 fit into the framework. -
Fourth, the three-layer taxonomy structure (Sections 3–5): With the optimization modes understood, I will walk through each layer—foundational, self-evolving, collective—explaining what environmental dynamics characterize the layer, what reasoning capabilities it demands, and how methods in that layer instantiate those capabilities through either in-context or post-training approaches. At each layer, I will call out specific representative methods from the survey's tables as concrete examples of how the abstract taxonomy maps to real systems.
-
Fifth, the self-evolving meta-learning loop (Section 2.2, conclusion): The survey introduces a cross-layer concept—the self-evolving agent as meta-learner—distinguishing between verbal evolution (updating textual guidelines), procedural evolution (creating new tools), and structural evolution (modifying source code). This framework unifies methods like Reflexion, Voyager, and AlphaEvolve under a common meta-update rule.
-
Sixth, the collective intelligence generalization (Section 2.2, conclusion): Finally, I will explain how the POMDP extends to Dec-POMDPs for multi-agent settings, where communication becomes part of the reasoning process—one agent's external action (a message) triggers another agent's internal reasoning chain. This generalization shows that multi-agent reasoning is not a separate problem but a natural extension, with the primary challenge shifting from single-agent planning to mechanism design (optimizing communication topology and incentive structures).
3.4 Detailed, sentence-based technical breakdown
This paper is a conceptual survey—it does not propose a new algorithm, train a model, or evaluate on a benchmark. Its technical contribution is the formal framework and taxonomy that organize and relate existing methods. The core idea is that agentic reasoning can be understood by (1) modeling the agent-environment interaction as a Partially Observable Markov Decision Process with an explicit reasoning trace variable, (2) analyzing all methods as either inference-time search over reasoning traces (in-context reasoning) or optimization of the reasoning policy through training (post-training reasoning), and (3) organizing methods into three layers based on the environmental dynamics they address, with each layer introducing qualitatively new reasoning demands that compound on the previous layers.
The POMDP Formalization of Agentic Reasoning
The survey models agentic reasoning mathematically in Section 2.2. The formalism serves three purposes: it provides a common vocabulary for discussing methods across the three layers, it explains why in-context and post-training approaches are complementary rather than competing, and it reveals that multi-agent reasoning is a natural generalization of single-agent reasoning rather than a separate problem.
The environment tuple. The survey defines the environment as a POMDP with the following components:
where $\mathcal{X}$ is the latent environment state space (the true state of the world, which the agent cannot directly observe—e.g., the ground-truth mathematical proof that would solve the problem, the actual positions of objects in a robot's workspace, the underlying diagnosis for a patient's symptoms). The observation space $\mathcal{O}$ contains what the agent can perceive at each timestep (e.g., the text of a math problem, a camera image from a robot, a patient's reported symptoms and lab results). The external action space $\mathcal{A}$ contains the actions the agent can take that affect the environment (e.g., invoking a search API, commanding a robot joint, ordering a diagnostic test). Crucially, the survey introduces a reasoning trace space $\mathcal{Z}$—this is the set of possible internal deliberations the agent can perform (e.g., chains of thought, planning steps, reflections on past failures). $\mathcal{M}$ is the agent's internal memory or context space, which summarizes the interaction history. $\mathcal{T}$ and $\Omega$ are the environment's transition and observation kernels (the probability distributions over next states and observations given current state and action). $\mathcal{R}$ is the reward function (e.g., correctness of the final answer, task completion, safety compliance), and $\gamma \in (0, 1)$ is the discount factor that weights future rewards relative to immediate ones.
What this tuple represents: The formalism encodes the core insight that an agent never sees the true state $\mathcal{X}$—it only receives observations $\mathcal{O}$ and must maintain an internal summary $\mathcal{M}$ of everything it has experienced so far. At each timestep $t$, the agent conditions on its history $h_t = (o_{\leq t}, z_{<t}, a_{<t})$—all past observations, reasoning traces, and actions. The history can be equivalently summarized by a memory state $m_t \in \mathcal{M}$. This means the agent's behavior depends not just on the current observation but on everything it has previously reasoned and done.
Why this form: The explicit separation of $\mathcal{X}$ (true state) from $\mathcal{O}$ (observations) captures partial observability—the agent's knowledge is always incomplete, which forces it to reason under uncertainty. The separation of $\mathcal{Z}$ (reasoning) from $\mathcal{A}$ (actions) captures the think-before-acting structure that distinguishes agentic reasoning from both pure language modeling (no actions) and pure reinforcement learning (where reasoning is often implicit). Without $\mathcal{Z}$, the formalism would collapse in-context reasoning and tool-use into the same category; with it, the survey can distinguish between methods that differ in how they compute reasoning traces (in-context search vs. post-training optimization) even when they produce the same actions.
The policy factorization. The survey factorizes the agent's policy—the distribution over its behavior—into two components:
where $\pi_{\text{reason}}(z_t \mid h_t)$ is the internal thought component that, given the history $h_t$, generates a reasoning trace $z_t$ (this could be a chain-of-thought, a plan, a reflection on past errors), and $\pi_{\text{exec}}(a_t \mid h_t, z_t)$ is the external action component that, given the history and the current reasoning trace, selects an action $a_t$ (this could be invoking a tool, moving a robot, or producing a final answer). The parameter vector $\theta$ controls both components.
What this factorization computes: Given the history $h_t$, the agent first samples a reasoning trace $z_t$ from $\pi_{\text{reason}}$, then samples an action $a_t$ from $\pi_{\text{exec}}$ conditioned on both the history and the reasoning trace. The objective is to maximize expected return: $J(\theta) = \mathbb{E}_{\tau}\left[\sum_{t \geq 0} \gamma^t r_t\right]$, where the expectation is over trajectories $\tau = (h_0, z_0, a_0, h_1, z_1, a_1, \ldots)$ sampled from the policy and the environment.
Why this factorization: It makes explicit that reasoning is computation performed before committing to actions, and that different agentic reasoning methods differ in how they implement $\pi_{\text{reason}}$. In-context reasoning methods keep $\theta$ frozen and optimize the reasoning trace $z_t$ at inference time through search (e.g., Tree of Thoughts explores multiple candidate reasoning paths before selecting one). Post-training reasoning methods update $\theta$ to improve the quality of reasoning traces generated by $\pi_{\text{reason}}$ across many episodes. This factorization also reveals that tool-use is not a separate category—it is an action $a_t$ selected by $\pi_{\text{exec}}$ based on the reasoning trace $z_t$, which means that improving tool-use requires either better reasoning (generating better $z_t$ that specify when and how to use tools) or better action selection (better $\pi_{\text{exec}}$ given a reasoning trace).
In-Context Reasoning as Inference-Time Search
The survey models in-context reasoning as search over the reasoning trace space $\mathcal{Z}$ with frozen parameters $\theta$. This subsection explains how the formalism captures methods like ReAct, Tree of Thoughts, and structured workflow designs.
The inference-time search objective. When parameters $\theta$ are frozen, the agent optimizes its behavior by searching for a reasoning trajectory $\tau$ that maximizes a heuristic value function:
where $\tau = (h_0, z_0, a_0, h_1, z_1, a_1, \ldots)$ is a complete trajectory of observations, reasoning traces, and actions, each node $u_t$ is a representation derived from the partial trajectory up to time $t$ (typically the pair $(h_t, z_t)$), and $\hat{v}_{\phi}$ is a heuristic evaluator or verifier parameterized by $\phi$ that scores how promising a partial reasoning state is. The $\arg\max$ selects the trajectory with the highest total score.
What this equation computes: It defines a search problem over the space of possible reasoning-action sequences. The agent explores different reasoning traces $z_0, z_1, \ldots$, computes their values using $\hat{v}_{\phi}$, and selects the sequence that maximizes cumulative value. The verifier $\hat{v}_{\phi}$ could be the language model's own probability assessments, a separately trained reward model, or environment feedback (e.g., execution results for code). This search is performed at inference time without changing $\theta$.
Why this form: It unifies diverse methods. ReAct performs greedy decoding over alternating thought $z$ and action $a$ tokens—equivalent to a depth-1 search with $\hat{v}_{\phi}$ being the model's next-token log-probability. Tree-of-Thoughts treats partial thoughts as nodes $u$ and uses MCTS-style rollouts to estimate $\hat{v}_{\phi}(u)$ through simulation followed by heuristic evaluation, then selects the highest-value path. Beam search methods keep the top-$k$ partial reasoning traces at each step based on $\hat{v}_{\phi}(u)$. The common structure—search over $\mathcal{Z}$ guided by $\hat{v}_{\phi}$—explains why these methods are grouped together in the in-context optimization mode despite different search algorithms.
Connection to the policy factorization. In terms of the policy $\pi_{\theta}(z_t, a_t \mid h_t)$, in-context reasoning corresponds to keeping $\pi_{\text{reason}}$ and $\pi_{\text{exec}}$ fixed but not sampling from them directly. Instead, the agent generates multiple candidate reasoning traces $z_t$ (e.g., by sampling from $\pi_{\text{reason}}$ at high temperature or by structured prompting), evaluates them using $\hat{v}_{\phi}$, and selects the best one before proceeding to action selection. This is a form of test-time compute scaling that does not require parameter updates.
Post-Training Reasoning as Policy Optimization
The survey models post-training reasoning as optimization of the policy parameters $\theta$ using reinforcement learning or supervised fine-tuning, with the goal of improving the quality of reasoning traces and actions across episodes.
The GRPO objective. For reinforcement learning, the survey highlights Group Relative Policy Optimization (GRPO) as the dominant method for reasoning tasks, citing its use in DeepSeek-R1 and related systems. The GRPO objective is:
where $q$ is a prompt sampled from the prompt distribution $P(\mathcal{Q})$, $G$ is the number of outputs sampled for each prompt (the group size), $\rho_i = \frac{\pi_{\theta}(y_i \mid q)}{\pi_{\theta_{\text{old}}}(y_i \mid q)}$ is the importance sampling ratio comparing the current policy's probability of output $y_i$ to the old policy's probability (used for off-policy correction), and $\hat{A}_i$ is the group-relative advantage (defined next). The $\min$ and $\text{clip}$ terms implement PPO-style conservative updates—if the policy change ratio $\rho_i$ is too large (greater than $1 + \epsilon$) or too small (less than $1 - \epsilon$), the advantage $\hat{A}_i$ is multiplied by the clipped ratio rather than the raw ratio, preventing the policy from changing too much in a single update. The $\beta D_{\text{KL}}(\pi_{\theta} \| \pi_{\text{ref}})$ term is a KL-divergence penalty that keeps the current policy close to a reference policy $\pi_{\text{ref}}$ (typically the pre-trained model), preventing catastrophic forgetting of general capabilities.
What this objective computes: For each prompt $q$, the agent samples $G$ outputs $\{y_i\}_{i=1}^G$ from the current policy, computes a reward $r_i$ for each (e.g., correctness of the final answer), and then constructs advantages $\hat{A}_i$ that measure how much better each output is relative to the group average. The objective increases the probability of outputs with positive advantage (better than average) and decreases the probability of outputs with negative advantage (worse than average), while the clipping and KL penalty prevent the policy from moving too far from its previous state.
Why this form: GRPO eliminates the need for a separate value network (a "critic" that estimates expected future reward), which is required in standard PPO. Instead, it constructs advantages entirely from within-group comparisons: each output's advantage is based on how its reward compares to the group mean. This matters for reasoning tasks because training a separate value network is computationally expensive and often unstable when rewards are sparse (e.g., only the final answer is scored, not intermediate steps). The group-relative approach makes GRPO more sample-efficient and simpler to implement for reasoning-specific post-training.
The group-relative advantage. The advantage $\hat{A}_i$ is computed as:
where $r_i$ is the reward for output $i$ (a single scalar, e.g., 1 if correct, 0 if incorrect), $\mu$ is the mean reward across the group of $G$ outputs for the same prompt, $\sigma$ is the standard deviation of rewards across the group, and $\delta > 0$ is a small constant for numerical stability (preventing division by zero when all outputs receive the same reward).
What this advantage computes: It normalizes the raw reward $r_i$ into a z-score relative to the group: an output with reward exactly equal to the group mean gets advantage 0, an output better than average gets positive advantage proportional to how many standard deviations above the mean it is, and an output worse than average gets negative advantage. The normalization by $\sigma$ ensures that the magnitude of advantages adapts to the difficulty of the prompt—when all outputs are similarly good (small $\sigma$), even small reward differences produce large advantages, encouraging the policy to discriminate finely; when outputs vary widely (large $\sigma$), advantages are smaller, preventing the policy from overfitting to a single lucky sample.
Why this form: The group-relative construction naturally handles the exploration-exploitation tradeoff in reasoning tasks. When the model is already good at a prompt (most outputs are correct, $\mu$ is high, $\sigma$ is low), the advantages are small, preventing unnecessary policy changes. When the model is uncertain (outputs vary in correctness, $\sigma$ is moderate), the advantages provide a meaningful signal for improvement. When the model consistently fails (all outputs incorrect, $\mu = 0$, $\sigma = 0$), the advantages are zero (or near-zero with $\delta$), preventing the policy from degrading by trying to optimize a meaningless signal. This adaptive normalization is particularly important for reasoning tasks where difficulty varies widely across prompts.
Relationship to SFT-based post-training. The survey notes that supervised fine-tuning (SFT) is also a post-training method—it updates $\theta$ to increase the probability of reasoning traces that led to correct outcomes in a curated dataset. However, SFT operates on fixed data (the model does not generate its own training examples) and does not use relative advantages—it simply maximizes the likelihood of demonstrated trajectories. This makes SFT simpler but also more brittle: the model can overfit to the specific patterns in the training data and fail to generalize to unseen prompts. GRPO and related RL methods address this by having the model generate its own outputs (on-policy data collection) and optimizing based on rewards rather than imitation, which can lead to more robust and adaptive strategies.
The Self-Evolving Meta-Learning Loop
The survey introduces a cross-layer concept in Section 2.2 (and elaborates in Section 4): self-evolving agents operate across episodes $k = 1, \ldots, K$, optimizing not just the reasoning within an episode but the agent system itself. The meta-update rule is:
where $S_k$ is the evolvable system state at episode $k$—this could be the agent's explicit memory contents, its library of learned tools, its source code, or its model parameters, depending on the evolution type. $\tau_k$ is the trajectory from episode $k$ (the sequence of observations, reasoning traces, and actions the agent experienced), and $F_k$ is the environmental feedback from that episode (rewards, execution errors, human corrections). The update function $U$ transforms the old system state into a new one using the experience and feedback.
What this rule computes: After each episode, the agent modifies itself—its memory, tools, code, or parameters—based on what happened. The modification is designed to improve performance on future episodes. The survey categorizes this evolution by the nature of $S$.
Why this form: It unifies three qualitatively different approaches under a common update rule. Verbal evolution (e.g., Reflexion) has $S_k$ as a textual reflection or set of guidelines. After an episode, the agent synthesizes what went wrong into a linguistic summary (e.g., "I failed because I didn't check the API documentation before calling the function"), and $U$ appends this summary to $S_k$ so it conditions future $\pi_{\text{reason}}$. Procedural evolution (e.g., Voyager) has $S_k$ as a library of executable skills or tools. After an episode, the agent may synthesize a new skill (e.g., a Minecraft crafting recipe) and $U$ adds it to the library, permanently expanding the action space $\mathcal{A}$. Structural evolution (e.g., AlphaEvolve) has $S_k$ as the agent's source code or architecture. $U$ is a mutation operator implemented by an LLM that edits the code, potentially changing the reasoning algorithm itself. The common structure—an update function that uses episode experience to modify persistent state—reveals that these methods differ in what they evolve (prompts, tools, code), not in the abstract principle of evolution through interaction.
Connection to memory and feedback. The survey's detailed treatment of self-evolution in Section 4 identifies two core mechanisms—feedback and memory—that instantiate the meta-update. Feedback provides the signal $F_k$ (either self-generated critique, verifier judgments, or validator binary outcomes). Memory provides the substrate $S_k$ for accumulating and organizing these signals across episodes. The survey's categorization of memory types (flat vs. structured, in-context vs. post-training control) corresponds to different choices of how $S_k$ is represented and how $U$ accesses and modifies it.
The Collective Intelligence Generalization
The survey extends the single-agent POMDP to a decentralized partially observable multi-agent setting (Dec-POMDP) for collective reasoning. The core generalization is adding a communication channel $\mathcal{C}$: each agent $i$'s observation $o^i_t$ includes messages $c^{-i}_{t-1}$ generated by other agents at the previous timestep. The joint policy $\pi$ is composed of individual policies $\pi^i$ for each agent $i = 1, \ldots, N$.
The key insight: Communication is not a separate mechanism from reasoning—it is an extension of the reasoning process. When agent $i$ generates a message (an action $a^i_t$ in the communication channel), agent $j$ receives it as part of its observation $o^j_{t+1}$ and conditions its reasoning $\pi^j_{\text{reason}}(z^j_{t+1} \mid h^j_{t+1})$ on it. This means that one agent's external action becomes another agent's reasoning prompt. The survey explicitly states:
"In agentic MARL, communication is not merely signal transmission but an extension of the reasoning process: one agent's external action can act as a prompt that triggers another agent's internal reasoning chain."
What this generalization implies: The challenge shifts from single-agent planning (optimizing $\pi_{\text{reason}}$ and $\pi_{\text{exec}}$ for one agent) to mechanism design: optimizing the communication topology (who talks to whom), the communication protocol (what format messages take), and the incentive structure (what rewards each agent receives) so that decentralized reasoning processes $\pi^i_{\text{reason}}$ collectively align toward a coherent global objective. The survey notes that frameworks like AutoGen and CAMEL represent static role-playing with fixed policies, while more recent methods like GPTSwarm and those trained via PPO/GRPO aim to optimize the joint reasoning distribution.
Why this form: The Dec-POMDP extension reveals that multi-agent reasoning is not a fundamentally different problem from single-agent reasoning—it is the same POMDP formalism applied to a setting where part of the environment (the other agents) is strategic. The Centralized-Training/Decentralized-Execution (CTDE) paradigm, where agents share information during training but act independently at inference time, emerges naturally from this formulation: during training, the joint policy can be optimized using global information (all agents' observations and rewards), but at inference time, each agent only has access to its own observation $o^i_t$ and communication channel $c^{-i}_{t-1}$, requiring it to reason about others' intentions and knowledge without full observability.
The Three-Layer Taxonomy: Environmental Dynamics and Reasoning Demands
With the formal framework established, the survey organizes methods into three layers, each characterized by the environmental dynamics the agent faces and the reasoning capabilities those dynamics demand. The layers are cumulative: each builds on the capabilities of the previous one.
Layer 1: Foundational Agentic Reasoning (Section 3). This layer addresses environments that are stable—the goal is clear, the rules of the environment do not change during an episode, and feedback is reliable when it arrives (e.g., a math problem with a correct answer that can be verified, a web navigation task with a known target page, a code generation problem with unit tests). The agent's primary reasoning demands are: (1) planning—decomposing the goal into subgoals and sequencing actions to achieve them, (2) tool use—deciding when and how to invoke external APIs, search engines, code interpreters, or robotic controllers to extend its capabilities, and (3) search—exploring the space of possible reasoning traces and actions to find high-quality solutions, either through in-context tree search (MCTS, beam search) or post-training policy optimization that internalizes search strategies.
The survey's tables in Section 3 organize representative systems by structure (sequential, tree, graph, hierarchical) and optimization mode. For planning, in-context methods like Tree of Thoughts (tree search over reasoning paths) and ReWOO (decoupled observation and reasoning) contrast with post-training methods that use RL to optimize planning policies (e.g., RAGEN, which models planning as an MDP and trains with dense feedback). For tool use, in-context methods like ReAct (interleaved reasoning and tool calls) and ART (retrieval-augmented demonstrations) contrast with post-training methods like Toolformer (self-supervised API call generation followed by SFT) and ToolRL (reinforcement learning for multi-tool policy learning).
Layer 2: Self-Evolving Agentic Reasoning (Section 4). This layer addresses environments that are dynamic—they change over time, either within an episode (the agent's actions alter the environment, requiring adaptation) or across episodes (the task distribution shifts, requiring cumulative learning). The agent's reasoning demands expand beyond Layer 1 to include: (1) feedback integration—processing evaluative signals to correct errors and refine strategies, whether through self-critique (reflective feedback), validator signals (binary success/failure), or parameter updates (parametric adaptation), (2) memory—maintaining persistent, structured records of past experiences that can be retrieved and used to inform future reasoning, and (3) self-evolving planning, tool use, and search—extending the foundational capabilities to improve over time, such as autonomously generating new tools when existing ones are insufficient (e.g., LATM, CRAFT) or refining search strategies based on accumulated experience (e.g., Reasoning Bank, Dynamic Cheatsheet).
The survey's categorization of memory in Section 4.2 is particularly detailed, distinguishing between in-context use of flat memory (e.g., MemGPT's cache-inspired long-term storage, Workflow Memory's procedural trace tracking), structured memory representations (e.g., knowledge graphs in MEM0 and Zep, hierarchical trees in MemTree), and post-training memory control (e.g., Memory-R1's dual-agent design where a Memory Manager learns via RL when to add, update, or delete entries). This categorization reflects the progression from passive storage to active, learned management.
Layer 3: Collective Multi-Agent Reasoning (Section 5). This layer addresses environments that are strategic—other agents are part of the environment, and their behavior depends on their own reasoning processes, which may be cooperative, competitive, or mixed-motive. The agent's reasoning demands expand further to include: (1) role differentiation—assigning specialized functions to agents based on task structure (e.g., leader/coordinator, worker/executor, critic/evaluator) and adapting these roles to domain-specific needs (e.g., software engineering roles like architect and reviewer, healthcare roles like triage agent and specialist), (2) collaboration and division of labor—designing communication protocols, coordination schemas, and task allocation mechanisms that enable agents to jointly solve problems more effectively than any individual could, and (3) multi-agent evolution—scaling self-evolution from individual agents to collective systems, where shared memory, communication protocols, and collaboration strategies co-evolve through interaction.
The survey's treatment of collaboration topology (Section 5.2.2) is particularly revealing of the taxonomy's analytical power. It distinguishes graph-based topology generation (learning communication graphs through continuous relaxation, pruning, or MCTS-based search) from policy-based topology generation (training agent-selection policies via RL with group-relative advantages or LLM-generated reward functions). This maps directly onto the in-context vs. post-training distinction: graph-based methods typically operate at system design time (analogous to in-context orchestration at the meta-level, where the topology is chosen before execution), while policy-based methods learn to route tasks during execution (analogous to post-training optimization, where the routing policy is internalized through RL).
How the layers compose. The survey's taxonomy is not just three independent categories—it is a cumulative structure. A collective multi-agent system (Layer 3) still needs foundational single-agent capabilities (Layer 2) for each agent to plan, use tools, and search effectively. It also benefits from self-evolving mechanisms (Layer 2) for each agent to improve through feedback and memory. The layers represent additional reasoning demands that compound: foundational reasoning addresses "how do I solve this problem?", self-evolving reasoning addresses "how do I get better at solving such problems?", and collective reasoning addresses "how do we solve this problem together, given that each of us has partial information and capabilities?". The survey's organization by layer—rather than by method type or application domain—reflects this cumulative structure, making it explicit that higher-layer systems depend on lower-layer capabilities.
The In-Context vs. Post-Training Axis: Orthogonal to the Layers
The survey applies the in-context vs. post-training distinction at every layer of the taxonomy. This is not a third dimension (which would create a 3D grid) but rather a cross-cutting classification that asks, for any given method at any layer: are the reasoning strategies specified at inference time through prompting, search, and workflow orchestration (in-context), or are they internalized into model parameters through training (post-training)?
At the foundational layer, in-context planning includes Tree of Thoughts (MCTS over reasoning paths with frozen weights) and ReWOO (decoupled reasoning from observation via prompt design). Post-training planning includes methods that use RL to optimize planning policies, such as RAGEN (which models planning as an MDP and trains with dense interaction feedback) and WebRL (which uses a self-evolving online curriculum with RL for web navigation planning).
At the self-evolving layer, in-context feedback integration includes Reflexion (appending self-critique to the prompt context) and Self-Refine (iterative critique and revision within a single inference episode). Post-training feedback integration includes ReST (critique-revision pairs used as supervised training data) and methods that use DPO or RLHF to align behavior with verifier scores.
At the collective layer, in-context collaboration includes manually crafted pipelines (e.g., MetaGPT's predefined agent roles and communication rules) and LLM-driven orchestration (e.g., MAS-GPT, where an LLM generates a multi-agent workflow conditioned on the user query). Post-training collaboration includes methods like MAGRPO (which extends GRPO to multi-agent settings with group-relative advantages computed across agents) and MAPoRL (multi-agent post-co-training with collaboration-aware verifiers).
Why this axis matters. The in-context vs. post-training distinction captures a fundamental tradeoff in how reasoning capabilities are acquired and deployed. In-context methods are flexible—they can be adapted to new tasks by changing prompts or search strategies without retraining—but they are computationally expensive at inference time (search over reasoning traces consumes test-time compute) and bounded by the base model's inherent capabilities. Post-training methods are efficient at inference time (the optimized policy generates good reasoning traces in a single forward pass) and can exceed the base model's initial capabilities, but they require substantial training data or environment interaction and are less flexible to novel tasks. The survey's framework makes this tradeoff explicit: a system designer can choose in-context orchestration for tasks where flexibility matters and post-training optimization for tasks where efficiency and capability depth matter, understanding that the two approaches are complementary within each layer.
Summary of Design Choices and Their Justifications
The survey's technical approach is characterized by several deliberate design choices:
Choice 1: POMDP formalism with explicit reasoning traces. The survey could have organized methods descriptively (grouping by superficial similarity) or by application domain. Instead, it chose a mathematical framework that reveals structural commonalities. Justification: the POMDP formalism makes explicit that in-context and post-training methods are solving the same underlying problem (optimizing behavior in a partially observable environment) through different computational mechanisms (search over traces vs. parameter optimization). This prevents the survey from treating them as unrelated and enables principled comparison.
Choice 2: Three-layer taxonomy based on environmental dynamics. The survey could have organized by method type (planning, memory, tool use) or by model architecture. Instead, it chose to organize by the environment's demands on reasoning—stable, dynamic, strategic. Justification: this organization explains why methods fail when moved to more demanding environments. A planning method that works in stable settings (Layer 1) may fail in dynamic settings (Layer 2) not because of a bug in the planning algorithm, but because it lacks the feedback integration and memory mechanisms that Layer 2 demands. The layer structure makes these dependencies explicit.
Choice 3: Orthogonal in-context vs. post-training axis. The survey could have treated this as a third layer dimension. Instead, it applies it as a cross-cutting classification across all layers. Justification: the optimization mode is independent of the environmental dynamics. A system at any layer can use in-context reasoning, post-training reasoning, or both. Keeping this axis orthogonal prevents the taxonomy from ballooning into a 6-category grid (3 layers × 2 modes) and allows methods that combine both modes (e.g., using in-context reflection to generate training data for post-training) to be discussed naturally.
Choice 4: Self-evolution unified under a meta-update rule. The survey could have treated feedback, memory, and capability evolution as separate topics. Instead, it unified them under the meta-update $S_{k+1} \leftarrow U(S_k, \tau_k, F_k)$, with feedback providing $F_k$ and memory providing $S_k$. Justification: this reveals that methods like Reflexion (verbal evolution through reflection), Voyager (procedural evolution through skill synthesis), and AlphaEvolve (structural evolution through code mutation) differ in what they evolve, not in the abstract principle of using episode experience to modify persistent state.
Choice 5: Communication as an extension of reasoning in collective settings. The survey could have treated multi-agent communication as a separate mechanism. Instead, it modeled it as one agent's action becoming another agent's reasoning prompt, extending the policy factorization naturally. Justification: this preserves the reasoning-centric lens—communication is not a protocol layered on top of reasoning, but a form of reasoning that crosses agent boundaries, with the same need for planning (what to communicate), feedback (was the communication effective?), and memory (what was communicated previously?).
Choice 6: Systematic classification of representative systems in tables rather than exhaustive catalogs. The survey uses tables (e.g., Tables 2–6) to classify representative systems by key properties (structure, format, tool use, memory type, etc.) rather than attempting to list every method. Justification: the survey's contribution is the classification scheme itself, not the enumeration of all existing methods. The tables demonstrate that the taxonomy can accommodate diverse methods and reveal patterns (e.g., that in-context planning methods tend to use tree or graph structures, while post-training planning methods tend to use sequential structures with RL) without overwhelming the reader with an exhaustive list.
Choice 7: Application domains analyzed through the three-layer lens. Rather than treating applications as a separate section disconnected from the taxonomy, the survey re-applies the three-layer structure within each application domain (Section 6). Justification: this shows that the taxonomy is not just a conceptual abstraction—it captures real structural differences in how systems are built across domains. An embodied agent needs foundational planning, self-evolving feedback from environment interaction, and (in multi-robot settings) collective coordination, just as a scientific discovery agent needs foundational tool use, self-evolving hypothesis refinement, and collective multi-agent literature review. The domain-specific analysis validates the taxonomy's generality.
4. Key Insights and Innovations
Innovation 1: Reasoning as a Function of Environmental Dynamics, Not Just Model Capability
The survey's most fundamental conceptual move is reframing reasoning not as a property of a model (how well it performs on a static benchmark) but as a relationship between the model and its environment. This is a qualitative shift in how the field has conceptualized reasoning progress. The dominant paradigm in LLM reasoning research—exemplified by Chain-of-Thought prompting, tree-of-thought search, and reinforcement post-training on math benchmarks—implicitly treats reasoning as an internal computation problem: given a fixed input, can the model produce the correct output? The environment is a static text prompt; the only variable is model capability.
The survey argues this framing is not just incomplete but actively misleading when applied to the systems it reviews. The core evidence is structural, not quantitative: the three-layer taxonomy itself is an empirical claim that reasoning demands change qualitatively as environmental dynamics change. Foundational reasoning (Layer 1) assumes stable environments with clear goals—a math problem isn't changing mid-solution. Self-evolving reasoning (Layer 2) introduces non-stationarity: the environment changes because the agent's own actions alter it, and feedback arrives asynchronously. Collective reasoning (Layer 3) introduces strategic complexity: other agents are part of the environment, and their behavior depends on their own (possibly conflicting) goals.
This is not an incremental refinement of existing LLM reasoning taxonomies; it is a fundamental reframing. Prior surveys (Huang and Chang, 2022; Chen et al., 2025) organized reasoning methods by technique type—prompting strategies, search algorithms, post-training methods—without conditioning on what the environment demands. The survey's insight is that technique-type organization obscures why methods fail when deployed in more demanding settings. A planning method that works perfectly on MATH may fail in WebArena not because the planning algorithm is flawed, but because WebArena requires the agent to handle partial observability and dynamic page content—environmental demands that Layer 1 methods are not designed to address. The taxonomy makes these dependencies explicit.
The significance extends beyond taxonomy. If reasoning capability is environment-relative rather than absolute, then benchmark design becomes critical in a way that is often overlooked. The survey's benchmark review (Section 7) implicitly argues that evaluating an agent on a static math dataset tells you almost nothing about whether it can reason in an embodied setting, because the environmental dynamics are fundamentally different. This implies that the field's heavy reliance on MATH and GSM8K as "reasoning benchmarks" has created a distorted picture of progress—models appear to be improving at reasoning when they are actually improving at a narrow subset of reasoning that happens to occur in stable, closed-world environments. The survey's framework provides the vocabulary for diagnosing this distortion.
Innovation 2: The In-Context vs. Post-Training Axis as a Unified Optimization Lens
The survey makes a distinctive intellectual move by treating the timing and mechanism of optimization—inference-time search versus training-time parameter updates—as the primary cross-cutting dimension across all reasoning layers, rather than treating them as separate research subfields. This is not just organizational convenience; it reveals a deep structural property of agentic reasoning systems that prior work had not made explicit.
Before this survey, in-context reasoning (prompting, search, workflow orchestration) and post-training reasoning (SFT, RL, preference optimization) were typically discussed by different communities using different vocabularies. The in-context literature talked about "prompt engineering" and "test-time compute scaling"; the post-training literature talked about "policy optimization" and "reward modeling." The survey's POMDP formalism (Section 2.2) shows that both are solving the same underlying optimization problem—maximizing expected return in a partially observable environment governed by the policy factorization $\pi_{\theta}(z_t, a_t \mid h_t) = \pi_{\text{reason}}(z_t \mid h_t) \cdot \pi_{\text{exec}}(a_t \mid h_t, z_t)$—through different computational mechanisms. In-context reasoning optimizes the reasoning trace $z_t$ directly at inference time through search over $\mathcal{Z}$ with frozen $\theta$; post-training reasoning optimizes $\theta$ through gradient-based updates using episode rewards.
What makes this framing significant is that it reveals the tradeoff structure between the two modes. In-context reasoning is flexible but computationally expensive at inference time and bounded by the base model's capability ceiling. Post-training reasoning is efficient at inference time and can push beyond the base model's initial capabilities, but it requires substantial training data and is less flexible to novel tasks. The survey's Tables 2–6 implicitly demonstrate this tradeoff: in-context methods dominate when tasks are diverse and adaptation must be rapid (e.g., ReAct for web navigation, where each website is different), while post-training methods dominate when tasks are repetitive and efficiency matters (e.g., ToolRL for multi-tool policy learning, where the toolset is stable but the optimal invocation strategy is complex).
The distinction between GRPO's group-relative advantage (Equation 4) and inference-time search (Equation 2) is the mathematical instantiation of this tradeoff. GRPO normalizes rewards within a group of outputs for the same prompt, making it adaptive to prompt difficulty but requiring multiple samples per prompt during training—it is a statistical optimization over many episodes. Inference-time search evaluates candidate reasoning traces against a heuristic verifier $\hat{v}_{\phi}$ for a single prompt—it is a computational optimization within a single episode. The survey's framework makes it natural to ask when each approach is appropriate, rather than treating them as competing paradigms.
This is a fundamental reframing, not an incremental one, because it provides a common language for discussing methods that were previously siloed. A researcher working on tree-search planning can now understand that their method is one way of implementing $\pi_{\text{reason}}$ through test-time computation, and that the same reasoning capability could alternatively be built through post-training if the task distribution is stable enough to amortize the optimization cost. Conversely, a researcher working on GRPO for reasoning can understand that their method is internalizing search strategies that could be replicated (less efficiently but more flexibly) through in-context tree search. The framework doesn't favor one approach—it clarifies when each is appropriate.
Innovation 3: Self-Evolution as a Unifying Principle Across Feedback, Memory, and Capability Expansion
The survey's treatment of self-evolution (Sections 4 and 5.3) makes a distinctive conceptual contribution by unifying three research threads that had previously been treated as independent: feedback mechanisms (reflection, critique, verifier-guided correction), memory architectures (flat, structured, post-training controlled), and capability expansion (self-evolving planning, tool creation, search refinement). The meta-update rule $S_{k+1} \leftarrow U(S_k, \tau_k, F_k)$ is not just notation—it is a diagnostic tool that reveals the common structure underlying apparently disparate methods.
Consider three systems that appear unrelated: Reflexion (verbal self-critique for code debugging), Voyager (skill library synthesis for Minecraft), and Memory-R1 (RL-trained memory manager for long-horizon QA). Under the survey's framework, they are all instances of the same meta-update pattern with different choices of $S$ and $U$. Reflexion evolves $S$ as textual guidelines through LLM-based synthesis of error logs; Voyager evolves $S$ as executable code skills through code generation and validation; Memory-R1 evolves $S$ as memory entries through reinforcement learning over memory operations. The fact that these methods differ in what they evolve and how they evolve does not obscure the fact that they share the same abstract architecture: an agent interacts with an environment, receives feedback, and modifies a persistent state to improve future performance.
What makes this unification significant is that it identifies a missing piece in prior work. Before this survey, feedback research and memory research were largely separate communities. Feedback papers asked "how can an agent learn from its mistakes?" Memory papers asked "how can an agent retain and retrieve past experience?" The survey's framework shows that these are not separate questions—feedback provides the signal $F_k$, memory provides the substrate $S_k$, and the meta-update $U$ is where they meet. A memory system without feedback is just storage; a feedback system without memory cannot accumulate improvement across episodes.
The categorization into verbal, procedural, and structural evolution is also a conceptual contribution with practical implications. It establishes a hierarchy of evolution sophistication: verbal evolution (changing prompts) is the simplest and most common, but it is limited by the expressiveness of natural language. Procedural evolution (creating new tools) expands the agent's action space but requires verification mechanisms to ensure tool correctness. Structural evolution (modifying source code) is the most powerful but also the most dangerous, as it can break the agent entirely. This hierarchy helps a system designer understand what they are committing to: if verbal evolution is insufficient, they can move to procedural evolution, but they must also invest in tool verification infrastructure.
The survey's discussion of multi-agent evolution (Section 5.3) extends this unification further by showing that collective self-evolution is not a separate problem but a generalization of single-agent self-evolution with the added complexity that $S$ may be shared across agents. The challenges of credit assignment (which agent contributed to success?), memory coherence (do all agents agree on what happened?), and strategy alignment (are agents evolving in compatible directions?) are natural consequences of applying the meta-update rule to a multi-agent setting, and the survey's framework makes these challenges explicit rather than treating them as emergent mysteries.
Innovation 4: Communication as an Extension of Reasoning in Multi-Agent Settings
The survey makes a subtle but important conceptual move in its treatment of multi-agent systems: it reframes communication not as a separate mechanism layered on top of reasoning, but as reasoning itself, extended across agent boundaries. This is crystallized in the observation that "one agent's external action can act as a prompt that triggers another agent's internal reasoning chain" (Section 2.2). The Dec-POMDP extension makes this formal: when agent $i$ generates a message $c^i_t$ as an action, agent $j$ receives it as part of its observation $o^j_{t+1}$ and conditions its reasoning policy $\pi^j_{\text{reason}}(z^j_{t+1} \mid h^j_{t+1})$ on it.
This reframing matters because it resolves a tension in the multi-agent literature. Prior work often treated communication as either an engineering problem (designing message formats and protocols) or a strategic problem (what to communicate to influence others' behavior). The survey's framework shows that these are the same problem viewed from different angles, and that the core challenge is not communication per se but joint reasoning: how do decentralized reasoning processes $\pi^i_{\text{reason}}$ collectively produce coherent, goal-directed behavior when each agent has only partial information?
The significance of this insight is most visible in the survey's treatment of collaboration topologies (Section 5.2.2). Graph-based topology generation (learning who communicates with whom) and policy-based topology generation (learning when and what to communicate) are both optimizing the same underlying quantity—the joint reasoning distribution—through different computational mechanisms. Graph-based methods optimize the structure of the reasoning network; policy-based methods optimize the content of reasoning within a given structure. The survey's framework makes it natural to ask whether these can be jointly optimized, which is exactly the direction that methods like GPTSwarm (which uses RL to optimize both the communication graph and the agent prompts) are beginning to explore.
This is an incremental but structurally important contribution because it connects multi-agent LLM research to the established Dec-POMDP and CTDE (Centralized Training with Decentralized Execution) frameworks in multi-agent reinforcement learning. Prior LLM-based multi-agent work often reinvented these concepts in ad hoc ways—designing communication protocols from scratch, manually specifying agent roles—without recognizing that decades of MARL research had formalized the same problems. The survey's POMDP extension provides a bridge: an LLM researcher reading about role-based collaboration in MetaGPT can now understand it as a special case of Dec-POMDP with fixed, pre-specified policies $\pi^i$, while an MARL researcher reading about MAGRPO's group-relative advantages can recognize the GRPO machinery adapted to multi-agent credit assignment. This cross-pollination is exactly what a good survey should enable.
5. Experimental Analysis
Evaluation Methodology
This paper is a conceptual survey—it does not conduct new experiments, train models, or evaluate on benchmarks. It synthesizes, organizes, and contextualizes results from over 100 existing benchmarks and hundreds of prior systems. The "experimental analysis" of a survey paper is therefore the analytical framework it provides for evaluating the experiments of others. The survey's contribution is not a set of quantitative results but a systematic lens for interpreting the experimental landscape: what has been measured, what has not, and what the measurements mean when viewed through the taxonomy of foundational, self-evolving, and collective reasoning across in-context and post-training optimization modes.
Dataset. Not applicable in the traditional sense. The survey reviews benchmarks spanning five application domains and seven core mechanism categories. For context: the tool-use benchmark review (Section 7.1.1) covers datasets ranging from single-turn evaluations like ToolQA (1,530 dialogues, 13 tools) and APIBench (16,450 instruction-API pairs, 1,645 APIs) to multi-turn, multi-tool suites like MTU-Bench (54,798 dialogues, 136 tools). The search benchmark review (Section 7.1.2) spans unimodal settings (WebWalker, InfoDeepSeek, WideSearch, RAVine) and multimodal settings (MMSearch, MM-BrowseComp, PaperArena, Video-BrowseComp). The memory and planning review (Section 7.1.3) covers long-horizon episodic memory (PerLTQA, LOCOMO, LONGMEMEVAL, MultiChallenge) and multi-session recall (MemSim, REALTALK, MemoryAgentBench). The multi-agent benchmark review (Section 7.1.4) spans game-based evaluations (MAgent, Pommerman, SMAC, MineLand), simulation-centric real-world assessments (SMARTS, Nocturne, IMP-MARL, REALM-Bench), and language-driven social reasoning tasks (LLM-Coordination, AVALONBENCH, MultiAgentBench). The application-level review (Section 7.2) further covers embodied agents (ALFWorld, OSWorld, AndroidArena), scientific discovery (DISCOVERYWORLD, ScienceWorld, ScienceAgentBench), medical agents (AgentClinic, MedAgentBench, EHRAgent), web agents (WebArena, VisualWebArena, Mind2Web), and general tool-use agents (GTA, NESTFUL, CodeAct).
Base model(s). The survey does not restrict itself to a single model family. The systems reviewed are built on diverse backbones: GPT-3, GPT-4, PaLM, Gemini, Claude, DeepSeek, LLaMA, Qwen, and various open-source models, spanning parameter scales from small on-device models to frontier-scale architectures. The survey's Section 6 application analysis makes this diversity explicit—for instance, embodied agents include systems built on vision-language-action models (Gemini Robotics, Octopus), web agents span from WebGPT (fine-tuned GPT-3) to modern RL-trained agents (WebRL, Agent-R1), and healthcare agents range from MedAgent-Pro (hierarchical diagnostic workflows on GPT-4 class models) to EHRAgent (executable code generation over structured EHR data). The survey's contribution is to organize these heterogeneous results into a coherent picture, not to standardize on a single evaluation platform.
Metrics. The survey does not define a unified metric—precisely because it argues that different reasoning layers and application domains demand different evaluation criteria. The benchmark review (Section 7) catalogs domain-specific metrics: tool-use benchmarks measure task completion accuracy, API selection F1, and plan validity; search benchmarks measure answer accuracy, retrieval precision, and interaction efficiency (number of steps to solution); memory benchmarks measure factual recall accuracy, multi-hop reasoning correctness, and temporal inference precision; multi-agent benchmarks measure win rates, task completion rates, coordination efficiency, and communication quality; application-level benchmarks measure end-to-end task success (e.g., ALFWorld's goal completion rate, WebArena's task success rate, AgentClinic's diagnostic accuracy). This diversity is not a weakness—it is the survey's empirical argument that agentic reasoning cannot be reduced to a single metric, because the reasoning demands at each layer (foundational, self-evolving, collective) are qualitatively different and require qualitatively different evaluation protocols.
Baselines. The survey's "baselines" are the classification dimensions themselves—the three-layer taxonomy and the in-context vs. post-training axis provide the comparative framework. When analyzing planning methods (Section 3.1), the survey compares in-context workflow design approaches (structured prompting, tree search) against post-training planning optimization (RL-based policy learning, SFT-based plan imitation). When analyzing feedback mechanisms (Section 4.1), it compares reflective feedback (inference-time self-critique) against parametric adaptation (training-time feedback internalization) and validator-driven feedback (binary outcome retry loops). When analyzing collaboration (Section 5.2), it compares manually crafted pipelines (fixed role assignments, pre-specified communication rules) against LLM-driven orchestration (dynamic workflow generation) and post-training topology optimization (graph-based and policy-based learned collaboration). These are not baselines in the traditional sense of "naive method vs. proposed method"; they are the comparative axes that a researcher should use to understand why one approach might outperform another in a given setting.
Generation budget / compute accounting. The survey does not standardize compute accounting across the reviewed work. This is a deliberate choice reflecting the reality that different papers measure computation differently: some count LLM API calls, some count total tokens generated, some count environment steps, and some report wall-clock time. The survey's Section 2.2 formalism implicitly provides a framework for thinking about compute: in-context reasoning methods consume test-time compute (search over $\mathcal{Z}$ at inference), while post-training methods amortize compute into training-time optimization of $\theta$. The survey's application analysis (Section 6) occasionally mentions compute costs qualitatively—for example, noting that embodied agents like Voyager require continuous environment interaction for skill library construction, while web agents like WebRL train with self-evolving online curricula that generate new tasks from unsuccessful attempts—but it does not attempt to reduce these diverse cost structures to a single metric. This is a legitimate limitation: a systematic compute-matched comparison across reasoning methods would be valuable but is outside the survey's scope.
Cross-validation / statistical protocol. Not applicable in the traditional sense, as the survey is a synthesis of existing results rather than an original empirical study. The survey does not apply cross-validation to select strategies, compute significance tests, or correct for multiple comparisons across the hundreds of benchmarks it reviews. It presents results as reported in original papers without systematic re-evaluation. This is standard for survey papers in AI—the contribution is organizational and conceptual, not empirical validation—but it means the survey's conclusions are only as reliable as the individual papers it synthesizes. A systematic meta-analysis with standardized evaluation protocols, as the survey itself implicitly calls for in Section 8 (open problems), would produce stronger quantitative conclusions but would be a fundamentally different type of paper.
Main Quantitative Results
This section does not report new experimental results. Instead, it characterizes the nature of the evidence base that the survey organizes, organized by taxonomy layer and optimization mode. The "results" of a survey paper are the patterns that emerge from systematic classification—what kinds of evidence exist at each layer, where the evidence is strong, and where it is thin.
Foundational Agentic Reasoning: Planning, Tool Use, and Search
Planning. The evidence base for foundational planning (Section 3.1) is the most mature of any layer. The survey's Table 2 classifies over 20 representative planning systems by structure (decomposed, sequential, tree, graph, hierarchical, reactive), format (natural language, formal language, programming language, visual features), and tool integration (none, retrieval, API, knowledge graph, object detector, world model). The pattern that emerges is that in-context planning methods dominate the language agent modality, while post-training planning methods are more common in visual and embodied modalities where grounding requires learned policies.
Specifically, in language-only settings (math reasoning, code generation, web navigation), the dominant paradigm is search-based planning at inference time: Tree of Thoughts (tree search over reasoning paths), ReWOO (decoupled reasoning from observation), RAP (reasoning as planning with world models), and various MCTS-based methods (AlphaZero-like tree search, Monte Carlo Tree Diffusion) all operate with frozen model weights. In visual and embodied settings (GUI agents, robotics), post-training methods are more prevalent: VisualPredictor learns abstract world models with neuro-symbolic predicates for planning; Embodied-R uses reinforcement learning to activate embodied spatial reasoning; and VIKI-R trains heterogeneous multi-agent coordination with hierarchical RL. This pattern is not random—it reflects the fact that language reasoning benefits from the strong priors of pretrained LLMs (making in-context search effective without additional training), while embodied reasoning requires learning to ground language in perception and motor control (making post-training optimization necessary).
The survey does not provide aggregated quantitative comparisons across these methods—no single benchmark tests all approaches. However, the classification structure itself is informative: a researcher designing a new planning system can consult Table 2 to see that if their system operates in the language modality with natural language reasoning traces and no tool integration, they are competing with a dense cluster of in-context methods; if they are operating in the visual modality with formal language representations and tool integration, they are in a sparser region of the design space.
Tool use. The tool-use evidence base (Section 3.2, Table 3) reveals a clear progression from in-context prompting to post-training mastery to orchestration-based integration. The survey classifies tool-use systems by integration stage (inference-time, post-training, orchestration), learning type (prompting, few-shot, SFT, RL, rule-based, search), and tool strategy (interleaved reasoning-action, self-generated API calls, multi-tool coordination, hierarchical orchestration).
The survey reports that in-context tool integration—exemplified by ReAct (interleaved reasoning-action loops), ART (retrieved multi-step demonstrations), and ChatCoT (chain-of-thought with tool calls)—provides broad coverage with minimal engineering but is bounded by context window length and the base model's inherent tool-use capabilities. Post-training methods—exemplified by Toolformer (self-supervised API call generation followed by SFT), ToolLLM (SFT on 16,000+ real-world APIs), and ToolRL (reinforcement learning for multi-tool policy learning)—achieve more robust and generalizable tool use but require substantial training data and compute. Orchestration-based methods—exemplified by HuggingGPT (planner-executor multi-tool coordination), OctoTools (hierarchical orchestration with standardized tool cards), and ToolChain* (A* search over tool decision trees)—scale to massive tool libraries (hundreds or thousands of APIs) by introducing explicit planning layers above individual tool calls.
A notable pattern from the classification: SFT-trained tool-use systems tend to be brittle to distribution shift, overfitting to specific API invocation patterns in the training data, while RL-trained systems show better generalization to unseen tools but are harder to train due to sparse rewards and credit assignment challenges over long tool-use sequences. This maps onto the survey's in-context vs. post-training tradeoff: SFT is a form of post-training that amortizes tool-use strategies into model weights at the cost of flexibility; RL is a form of post-training that learns more adaptive strategies but requires environment interaction; in-context methods preserve maximum flexibility but at higher inference cost.
Search. The agentic search evidence base (Section 3.3, Table 4) highlights a structural divergence between in-context retrieval-augmented reasoning and post-training search policy learning. In-context methods—ReAct (interleaved reasoning and search API calls), Self-Ask (decomposed querying), IRCoT (interleaved retrieval with chain-of-thought), Self-RAG (reflective retrieval with self-assessment)—rely on prompt engineering to guide retrieval behavior. They are flexible and deployable without training but suffer from instability across domains, as their effectiveness depends on prompt quality and the base model's ability to follow retrieval instructions.
Post-training methods—Search-R1 (RL-trained web search agent), Deep-Researcher (RL-trained multi-step web browsing), ReSearch (RL-optimized step-wise retrieval with verifier rewards), ReARTeR (RL-trained tool-cluster retrieval)—learn when and how to retrieve through outcome-driven rewards. The survey cites evidence that these methods exhibit emergent behaviors: Deep-Researcher demonstrates spontaneous plan formulation, cross-source corroboration, and self-reflection during web research tasks. ReSearch achieves adaptive retrieval during complex QA. These behaviors emerge from RL training but are difficult to engineer through prompt design alone.
The structure-enhanced search category (Agent-G, MC-Search, GeAR, ARG) represents a middle ground: systems that reason over knowledge graphs or multimodal sources using structured operations (graph queries, symbolic traversal) rather than free-form retrieval. The survey's classification reveals that these methods are particularly strong for tasks requiring factual precision and multi-hop reasoning, complementing the open-ended coverage of web search agents with structured, verifiable knowledge access.
Quantitative patterns absent from the survey. The survey does not report head-to-head accuracy comparisons across tool-use methods or search methods, because no unified benchmark exists that tests all approaches. This is a genuine gap in the experimental landscape that the survey's open problems section (Section 8) implicitly identifies: the field needs standardized evaluation protocols that span in-context and post-training methods at the same task, ideally with compute-matched comparisons. Without such benchmarks, claims about the superiority of one approach over another remain domain-specific and difficult to generalize.
Self-Evolving Agentic Reasoning: Feedback, Memory, and Capability Evolution
The evidence base for self-evolving reasoning (Section 4, Tables 5–6) is characterized by strong case-study evidence but limited systematic comparisons. The survey classifies dozens of systems, but most have been evaluated on different benchmarks, making it difficult to draw quantitative conclusions about which feedback or memory mechanism is "best."
Feedback mechanisms. Table 5 classifies 25+ systems by feedback stage (inference, training), feedback source (self-generated critique, AI verifier, binary validator, environment signals), and update target (trajectory, model parameters, output only). The survey's classification reveals three distinct feedback regimes with different performance characteristics:
Reflective feedback methods (Reflexion, Self-Refine, Constitutional AI, RLAIF) modify reasoning trajectories at inference time without parameter updates. The survey reports that these methods are effective for tasks where the base model can already generate reasonable outputs and improvement requires error correction rather than capability acquisition—for example, Reflexion improves code debugging by appending self-critique to the prompt context, and Self-Refine improves text generation through iterative critique-revision loops. Their limitation is that they cannot teach the model fundamentally new capabilities; they only help the model use existing capabilities more effectively.
Parametric adaptation methods (AgentTuning, ReST, ReFT, Distill-CoT) incorporate feedback into model parameters through additional training. The survey reports that these methods produce durable improvements that generalize beyond individual inference episodes—for example, ReST trains on critique-revision pairs to internalize self-correction behavior, and AgentTuning fine-tunes on high-quality agent trajectories to improve general agent capabilities. Their limitation is that they require curated training data and risk overfitting to specific feedback patterns.
Validator-driven feedback methods (ReZero, Retrials, CodeRL, SWE-bench) are the simplest but most scalable: they use binary success/failure signals to guide retry loops without modifying reasoning or parameters. The survey reports that these methods are particularly effective in domains with reliable validation (code execution against unit tests, embodied task completion signals), where they can achieve high success rates through repeated sampling alone. Their limitation is that feedback is non-diagnostic—failure does not teach the model why it failed or how to avoid similar failures in the future.
Memory systems. Table 6 classifies 30+ systems by setting (in-context, post-training), format (text, semantic, workflow, trajectory, graph, multimodal, implicit), and memory type (factual, experience). The survey's classification reveals an evolution from static buffers to reasoning-integrated memory architectures. Early systems (MemGPT, MemoryBank, LlamaIndex) treat memory as a passive retrieval store; recent systems (Amem, Evo-Memory, Reasoning Bank, Dynamic Cheatsheet) treat memory as an active component of the reasoning loop that is dynamically updated, reflected upon, and used to guide future decisions.
The survey highlights a particular trend: workflow and experience memory (Workflow Memory, ACE, Reasoning Bank, Sleep-time Compute, Dynamic Cheatsheet) represents a shift from storing factual knowledge to storing procedural knowledge—reusable reasoning patterns, successful strategies, and failure analyses. The quantitative evidence for this shift is anecdotal rather than systematic: individual papers report accuracy improvements on their respective benchmarks, but no meta-analysis compares workflow memory vs. factual memory across tasks. This is a significant gap: the survey's taxonomy suggests that procedural memory should be particularly valuable for self-evolving agents that encounter similar task structures repeatedly, but this hypothesis has not been tested across diverse benchmarks.
The post-training memory control category (MemAgent, Mem1, Memory-R1, Memory-as-Action, MemRL) is the most recent and least evaluated. These systems use reinforcement learning to train agents to manage their own memory—deciding what to store, when to retrieve, and what to discard. The survey reports that Memory-R1 achieves improved long-horizon QA performance through a dual-agent design where a Memory Manager learns RL-based memory operations, and MemAgent learns through the DAPO algorithm to maintain constant-size memory across conversations. However, these results are from individual papers on different benchmarks, and no systematic comparison exists across methods or against simpler baselines (e.g., fixed-size sliding window memory). The survey's framework makes the value of such comparisons explicit, but they remain future work.
Capability evolution. The survey's Section 4.3 on evolving foundational capabilities (self-evolving planning, tool use, search) catalogs methods but provides limited quantitative evidence. Self-evolving planning methods (SCA, SELF, SCoRe, TextGrad, RAGEN, DYSTIL) are evaluated on diverse benchmarks with no standardized comparison. Self-evolving tool-use methods (LATM, CRAFT, CREATOR, ToolMaker) demonstrate that agents can create new tools when existing ones are insufficient, but the survey does not report systematic evaluations of tool creation quality or the conditions under which tool creation improves task performance vs. degrades it. Self-evolving search methods (Dynamic Cheatsheet, Reasoning Bank, Evo-Memory) show that retrieval quality improves as memory accumulates, but the rate of improvement and the conditions for effective search evolution remain unquantified.
Collective Multi-Agent Reasoning: Collaboration, Communication, and Co-Evolution
The evidence base for collective reasoning (Section 5, Tables 7–8) is the most heterogeneous and least standardized of any layer. Multi-agent systems are evaluated on dozens of different benchmarks across game environments, simulation platforms, and application domains, making cross-system comparison extremely difficult.
Role taxonomy and domain specialization. The survey's classification of domain-specific roles (Section 5.1.2) is purely descriptive—it catalogs how different application domains (software engineering, finance, legal, education, healthcare, biomedicine, music) instantiate the generic roles of leader, worker, critic, memory keeper, and communication facilitator. The survey does not provide evidence that role specialization improves performance over homogeneous agent teams, because few papers run the relevant ablation (comparing a role-specialized system to the same system with homogeneous agents). This is a significant gap: the taxonomy assumes role specialization is beneficial, but the experimental evidence for this assumption is scattered and domain-specific.
Collaboration topology optimization. The survey's treatment of topology generation (Section 5.2.2) is the most quantitatively grounded part of the collective reasoning analysis, because topology optimization is an active research area with concrete benchmarks. Graph-based methods (GommFormer, G-designer, AgentPrune, AGP, AFlow, GPTSwarm, MaAS, DynaSwarm) are evaluated on multi-agent coordination benchmarks like SMAC, MPE, and custom LLM-agent benchmarks. Policy-based methods (MAGRPO, MHGPO, COPY, LGC-MARL, MAPoRL, COPPER, SIRIUS) are evaluated on reasoning and coordination tasks. However, the survey does not report aggregated comparisons—graph-based and policy-based methods are typically evaluated on different benchmarks, making it impossible to claim that one approach is generally superior.
The survey's key insight from this classification is structural rather than quantitative: the optimization target matters more than the optimization method. Topology optimization methods that learn communication graphs (who talks to whom) are solving a different problem than policy optimization methods that learn communication content (what to say and when). The survey's framework suggests that both are necessary—communication structure and communication content must be jointly optimized—but this joint optimization remains an open problem that the survey explicitly identifies (Section 8.4).
Multi-agent evolution. The survey's treatment of multi-agent self-evolution (Section 5.3) is the most forward-looking and least empirically grounded section. The survey catalogs systems that enable multi-agent co-evolution through interaction and feedback (Multi-Agent Evolve, CoMAS), reinforcement fine-tuning (MARFT, Stronger-MAS, MAPoRL), role specialization (MALT, MARS), and preference alignment (Alignment Waltz, preference-based MARL). However, the survey acknowledges that "multi-agent memory systems remain largely unexplored with respect to post-training and model adaptation" (Section 5.3.2) and that "extending [post-training techniques] to enable multi-agent teams to co-evolve their memory structures and management policies remains an open problem."
This is not a weakness of the survey—it is an accurate characterization of the field's immaturity. The survey's framework provides the vocabulary for designing experiments that would test multi-agent evolution hypotheses (e.g., does joint training of memory management policies improve collective performance more than independent training? under what conditions does role specialization emerge spontaneously vs. require explicit design?), but these experiments have not yet been run at scale.
Ablation Studies and Robustness Checks
This survey does not conduct original ablations. It reviews ablation patterns from the literature it synthesizes. The systematic patterns that emerge from the survey's classification can be understood as cross-study ablations—variations in method design that have been tested across different papers and benchmarks, revealing which design choices matter.
Feedback source ablation (reflective vs. parametric vs. validator-driven). The survey's classification of feedback mechanisms (Table 5) reveals that the choice of feedback source significantly impacts what kind of improvement is possible. Reflective feedback (inference-time self-critique) improves reasoning quality within episodes but does not produce durable capability gains. Parametric adaptation (training on feedback) produces durable gains but requires curated data and risks overfitting. Validator-driven feedback (binary retry loops) scales to large problem volumes with reliable validation but provides no diagnostic signal. The survey does not report a single ablation study that tests all three on the same task with the same base model—such a study would be highly informative but does not exist in the literature it reviews.
Memory structure ablation (flat vs. structured vs. post-training controlled). The survey's classification of memory systems (Table 6) suggests that structured memory (graphs, workflows, hierarchical trees) improves retrieval precision and reasoning coherence over flat memory buffers, particularly for multi-hop and long-horizon tasks. The survey cites specific comparisons: GraphRAG improves multi-hop information integration over dense retrieval; MEM0 and Zep report improved semantic grounding with knowledge graph memory over flat key-value stores; Workflow Memory reduces plan recovery failures by capturing procedural traces. However, no systematic ablation compares flat vs. structured vs. post-training-controlled memory across a diverse set of tasks with matched compute budgets. The survey's framework predicts that the optimal memory structure should depend on task characteristics (factual tasks benefit from graph memory, procedural tasks benefit from workflow memory, continuously changing tasks benefit from post-training control), but this hypothesis has not been systematically tested.
Collaboration topology ablation (manual vs. LLM-driven vs. learned). The survey's analysis of collaboration schemas (Section 5.2) compares manually crafted pipelines, LLM-driven orchestration, and post-training topology optimization. The survey reports that LLM-driven orchestration (e.g., MAS-GPT, MetaAgent) improves adaptability over manually crafted pipelines by dynamically generating workflows conditioned on task context. Post-training topology optimization (e.g., GPTSwarm, MaAS, DynaSwarm) further improves performance by learning communication structures from task rewards. However, the survey does not report a controlled ablation that isolates the effect of topology from the effect of agent capability—when a learned topology outperforms a manual topology, it is unclear whether the improvement comes from better communication structure or from the learning process itself improving agent policies.
In-context vs. post-training ablation. This is the most important ablation that the survey's framework calls for but that the existing literature largely does not provide. For any given reasoning capability—planning, tool use, search, feedback integration, memory management—the survey's taxonomy predicts that in-context methods should be more flexible but less efficient, while post-training methods should be more efficient but less flexible. However, few papers directly compare an in-context implementation and a post-training implementation of the same capability on the same task with the same base model and matched compute budgets. The survey's Section 8.2 (long-horizon agentic reasoning) implicitly identifies this gap by noting that RL-trained agents "rely on heavily engineered, domain-specific rewards and largely treat episodes independently," while in-context agents scale through test-time compute but "errors still compound rapidly in long tasks." A systematic in-context vs. post-training comparison would resolve this tradeoff quantitatively; the survey provides the conceptual framework for designing such a comparison but does not report one because it does not yet exist in the literature.
Multi-agent vs. single-agent ablation. Several individual papers within the survey's scope compare multi-agent and single-agent performance on specific tasks. The survey cites examples where multi-agent debate improves reasoning accuracy over single-agent generation, where role-based collaboration improves software development quality over monolithic agents, and where agent routing improves question-answering efficiency over single-agent retrieval. However, the survey does not attempt to meta-analyze these results—the benchmarks, tasks, and metrics are too heterogeneous. The survey's taxonomy predicts that multi-agent systems should outperform single-agent systems when tasks require diverse expertise or parallel exploration, but should underperform when coordination overhead dominates. Systematic testing of this prediction across diverse task types remains future work.
Negative results. The survey reports several negative results from the literature that are informative for understanding the boundaries of current methods:
-
GRPO/RL instability in tool-use environments. The survey notes that "advanced methods such as ARPO and DAPO extend [GRPO] to handle sparse rewards and improve stability in complex tool-use environments" (Section 2.2), implicitly acknowledging that standard GRPO can be unstable when tool-use actions have delayed or sparse rewards. This is a negative result: naive RL application to tool-use reasoning can fail.
-
ReST EM degradation. Section 4.2.3 notes that "the ReST EM experiment (Appendix K, Figure 16) further highlights the fragility of revision training. Attempting to optimize the revision model with RL-style training caused performance to degrade substantially with sequential revisions, likely because on-policy data collection amplified spurious correlations in the revision trajectories." This is a concrete negative result: the same RL approach that improves some reasoning tasks can catastrophically degrade others.
-
Verifier over-optimization in search. Section 3.3 implicitly references the over-optimization phenomenon documented in prior work (not the survey's own experiments, but part of the literature it synthesizes): beam search and aggressive optimization against a learned verifier can produce solutions that score highly but are incorrect, particularly on easy problems where the verifier's assessments are reliable but residual errors get amplified by adversarial optimization. This negative result motivates the adaptive allocation strategies that are central to the reasoning-as-control perspective.
-
Multi-agent communication overhead. The survey notes that "communication overhead can easily undermine MAS efficiency" (Section 5.1.1) and that "scaling to larger agent populations further introduces challenges in topology adaptation, coordination overhead, and safety" (Section 8.4). Several papers in the survey's scope report that adding more agents eventually degrades performance due to communication costs, though the survey does not aggregate these results quantitatively.
Critical Assessment
This is a conceptual survey, not an empirical study. The appropriate critical assessment focuses on whether the survey's analytical framework is supported by the evidence it reviews, whether the framework reveals genuine patterns or merely imposes structure on noise, and what experimental evidence would be needed to validate or refute the framework's central claims.
Does the three-layer taxonomy reflect real structural differences in reasoning demands, or is it an arbitrary categorization? The survey's core claim is that environmental dynamics—stable (foundational), dynamic (self-evolving), and strategic (collective)—define qualitatively different reasoning requirements. The evidence supporting this claim is the differential distribution of methods across layers. Foundational reasoning methods (Section 3) overwhelmingly use in-context planning, tool use, and search, reflecting the fact that stable environments with clear goals do not require the persistent adaptation mechanisms that characterize self-evolving methods. Self-evolving reasoning methods (Section 4) introduce feedback integration and memory architectures that are largely absent from foundational systems, reflecting the additional demands of non-stationary environments. Collective reasoning methods (Section 5) introduce role differentiation, communication protocols, and coordination mechanisms that are absent from single-agent systems, reflecting the strategic complexity introduced by other agents.
The survey does not, however, provide quantitative evidence that these layer distinctions are causally meaningful. It does not show that adding a feedback mechanism to a foundational system transforms it into a self-evolving system with measurably different behavior and failure modes. It does not show that removing communication from a collective system degrades it to single-agent performance. It does not test whether the boundaries between layers are sharp (a system is either foundational or self-evolving) or continuous (systems can be partially self-evolving). These causal tests would require controlled experiments that add or remove capabilities layer by layer and measure the impact—experiments that the survey does not report because the literature it reviews largely does not conduct them.
Does the in-context vs. post-training axis capture a genuine tradeoff, or is it a false dichotomy? The survey's orthogonal axis—classifying every method as either in-context or post-training—is a simplification that creates analytical clarity at the cost of nuance. Many real systems combine both modes: an agent might use post-training to learn a base policy and in-context reasoning to adapt that policy at test time. The survey acknowledges this (Section 2.2: "Together, they provide an actionable roadmap for designing agents") but does not provide a framework for analyzing hybrid systems that blend in-context and post-training optimization. This is a gap: the most powerful agentic reasoning systems in the survey's scope (e.g., self-evolving agents that both train on past experience and reflect in-context on current failures) operate at the intersection of the two modes, yet the survey's taxonomy treats them as separable.
The empirical support for the tradeoff is also indirect. The survey observes that in-context methods dominate in domains requiring flexibility (e.g., web navigation across diverse sites) and post-training methods dominate in domains with stable task distributions (e.g., tool-use policy learning for a fixed API set). This correlation is consistent with the tradeoff hypothesis but does not demonstrate causality. A rigorous test would compare an in-context and post-training implementation of the same reasoning capability on the same task distribution while varying the stability of the distribution—the survey does not report such a study because it does not exist in the reviewed literature.
Does the meta-update framework genuinely unify feedback, memory, and capability evolution, or is it too abstract to be useful? The survey's $S_{k+1} \leftarrow U(S_k, \tau_k, F_k)$ formalism is elegant—it reveals that Reflexion (verbal self-critique), Voyager (skill library synthesis), and Memory-R1 (RL-trained memory management) share the same abstract structure of experience-driven state modification. However, the formalism's explanatory power depends on whether it generates testable predictions beyond "these methods are similar because they all update state."
The survey's categorization of evolution into verbal, procedural, and structural provides some predictive leverage: it suggests that verbal evolution is the easiest to implement but most limited in expressiveness, procedural evolution expands capabilities but requires verification, and structural evolution is the most powerful but most dangerous. These predictions are intuitively plausible—changing prompts is safer than writing new tools, which is safer than rewriting source code—but the survey does not provide systematic evidence that methods at different evolution levels actually exhibit these tradeoffs in practice. A necessary test would compare a verbal evolution method, a procedural evolution method, and a structural evolution method on the same task, measuring both performance improvement and failure rates. Again, the survey does not report such a study because the literature does not contain one.
Does the survey's application-domain analysis validate the taxonomy, or does it merely demonstrate that the taxonomy can be applied? The survey's Section 6 re-applies the three-layer taxonomy within each application domain, showing that math exploration, scientific discovery, embodied robotics, healthcare, and web agents all exhibit foundational, self-evolving, and collective reasoning instances. This demonstrates the taxonomy's coverage (it can describe diverse systems) and consistency (the same layers appear across domains), but not its discriminative power (does the taxonomy help predict system behavior or guide system design?).
A stronger validation would show that the taxonomy predicts failures: systems designed for a lower layer should fail when deployed in higher-layer environments, and the failure modes should be predictable from the layer mismatch. For example, a foundational planning agent deployed in a healthcare setting should fail not just because it makes mistakes, but specifically because it lacks the feedback integration and memory mechanisms that self-evolving systems provide—its failures should be characterized by an inability to learn from errors across patients, not just by individual decision errors. The survey's application analysis hints at such patterns but does not systematically test them. The healthcare section notes that "persistent memory is essential for tracking medical or patient history and maintaining context across interactions" (Section 6.4.2), implying that foundational-only systems would fail at longitudinal care, but this is presented as a design observation rather than an experimentally tested hypothesis.
Genuine weaknesses.
-
No systematic quantitative comparisons across methods within any layer. The survey classifies hundreds of systems but never reports a table comparing their accuracy on a shared benchmark. This is partly because the field lacks shared benchmarks that span in-context and post-training methods across all three layers—the survey's open problems section identifies this gap—but it means the survey cannot answer the most basic question a practitioner would ask: "which method should I use for my task?"
-
Single-survey perspective, no formal meta-analysis. The survey synthesizes results as reported in original papers without correcting for publication bias, without re-evaluating results under standardized protocols, and without statistical aggregation. If the surveyed papers systematically overestimate performance (due to cherry-picking, benchmark overfitting, or unreported hyperparameter tuning), the survey inherits those biases. A rigorous meta-analysis would re-evaluate a subset of methods under controlled conditions; the survey does not attempt this.
-
In-context vs. post-training classification is sometimes forced. Many methods blend both modes—for example, an agent might use in-context reflection to generate training data for post-training (as ReST does), making it simultaneously in-context and post-training. The survey classifies such methods into one mode or the other based on their dominant mechanism, which simplifies the taxonomy but risks misrepresenting the method's full design.
-
No treatment of computational cost or deployment feasibility. The survey analyzes reasoning mechanisms in abstraction from their computational requirements, latency characteristics, or engineering complexity. A method that requires thousands of RL training episodes in a simulator may be academically interesting but practically infeasible for many applications; the survey's taxonomy does not distinguish between methods based on their deployment readiness. The open problems section acknowledges this implicitly (Section 8.1 on personalization, 8.2 on long-horizon reasoning) but the main analysis does not address it.
-
The self-evolving layer conflates within-episode adaptation and across-episode learning. Methods that adapt within a single episode (e.g., Reflexion's self-critique loop) and methods that learn across episodes (e.g., memory consolidation over many tasks) are grouped under the same "self-evolving" umbrella, but they place very different demands on memory architecture and credit assignment. The survey's intra-test-time vs. inter-test-time distinction (Section 5.3.1) partially addresses this but does not apply it systematically to the classification in Section 4.
Missing experiments that would strengthen the framework.
-
Compute-matched comparisons across in-context and post-training implementations of the same capability. For planning, tool use, search, and feedback integration, a study that allocates equal total FLOPs to an in-context method (spending compute at inference time) and a post-training method (spending compute at training time) would directly test the survey's central tradeoff hypothesis.
-
Layer ablation studies. A system with foundational + self-evolving + collective capabilities systematically stripped down to foundational-only, foundational + self-evolving, and full-stack configurations, tested on tasks that demand different layers, would validate whether the taxonomy's layers correspond to causally necessary capabilities.
-
Cross-domain transfer studies. A self-evolving memory architecture trained on one domain (e.g., code debugging) and tested on another (e.g., scientific discovery) would test the survey's implicit claim that self-evolving mechanisms are general across domains—do the same feedback and memory mechanisms transfer, or are they domain-specific?
-
Standardized multi-agent benchmarks that span role-based, communication-based, and co-evolution methods. The survey identifies that different multi-agent methods are evaluated on different benchmarks; a unified benchmark would enable quantitative comparison and potentially reveal which aspects of multi-agent design (roles, communication topology, training paradigm) matter most.
Where claims hold conditionally.
-
The three-layer taxonomy captures meaningful structure: Supported by the distribution of methods across layers, but only if one accepts that the survey's classification of individual methods is accurate. The condition is that the taxonomy is descriptive (it organizes existing work) rather than prescriptive (it does not predict which layer a new method will need).
-
In-context reasoning is more flexible but less efficient than post-training reasoning: Supported by the observed concentration of in-context methods in diverse-task domains and post-training methods in stable-task domains, but conditional on the assumption that these domain assignments reflect genuine performance advantages rather than historical accident or researcher preference.
-
Self-evolution requires both feedback and memory; neither alone is sufficient: Supported by the structure of systems in Section 4 (all self-evolving systems have both feedback and memory components), but conditional on the definition of "self-evolving"—a system with only feedback but no memory might still improve within an episode, and the survey would classify it as reflective rather than self-evolving.
-
Multi-agent reasoning is a generalization of single-agent reasoning with communication as an extended reasoning process: Supported by the Dec-POMDP formalism and the structural similarity between single-agent and multi-agent policy factorization, but conditional on accepting that communication is best modeled as part of reasoning rather than as a separate protocol layer. This is a modeling choice, not an empirically testable claim—it is useful insofar as it generates productive research directions, which the survey argues it does (Section 8.4).
6. Limitations and Trade-offs
Limitation 1: The Taxonomy Is Descriptive, Not Causally Validated
The assumption or constraint. The survey's central contribution is the three-layer taxonomy—foundational, self-evolving, collective—and the orthogonal in-context vs. post-training optimization axis. The paper presents this as an organizational framework for the literature, not as a causal model of reasoning. It does not claim that the layers correspond to necessary and sufficient capabilities, or that adding a layer's mechanisms to a lower-layer system transforms its behavior in predictable ways. The taxonomy is asserted, not tested.
The consequence. A practitioner trying to use this survey to design a system faces a fundamental uncertainty: does the taxonomy reflect genuine structural properties of agentic reasoning, or is it an arbitrary categorization that imposes pattern on noise? If the layers are causally meaningful, then a system designer who identifies their target environment as "dynamic" (Layer 2) knows they must invest in feedback integration and memory, because foundational-layer systems will systematically fail. If the layers are merely descriptive, then a system built with only foundational-layer mechanisms might work perfectly well in a dynamic environment—the taxonomy wouldn't predict failure, only describe that most existing systems in dynamic environments happen to include feedback and memory.
The survey implicitly claims causal significance when it maps applications to layers: healthcare agents are analyzed through self-evolving reasoning (Section 6.4.2) because "persistent memory is essential for tracking medical or patient history." This implies that a foundational-only healthcare agent would fail at longitudinal care—not just perform worse, but fail in a structurally predictable way. But the survey provides no experimental evidence for this failure mode. No study in the reviewed literature systematically strips self-evolving capabilities from a healthcare agent and measures the degradation. The prediction is plausible—it aligns with clinical intuition—but it is not empirically validated.
The same uncertainty applies to the in-context vs. post-training axis. The survey posits a tradeoff: in-context reasoning is more flexible but less efficient; post-training reasoning is more efficient but less flexible. This tradeoff is mathematically grounded in the POMDP formalism (computation at inference time vs. computation amortized into training), but the survey provides no experiment where the same reasoning capability is implemented both ways on the same task with matched compute budgets. The concentration of in-context methods in diverse-task domains (web navigation) and post-training methods in stable-task domains (tool-use policy learning) is a correlation—it is consistent with the tradeoff hypothesis but does not confirm it. The correlation could equally reflect historical accident (the web navigation community happened to develop in-context methods first) or researcher preference, rather than a genuine performance advantage driven by the tradeoff.
What evidence exists in the paper. The survey provides structural evidence—the distribution of methods across taxonomy cells—but no causal evidence. Tables 2–6 show that methods do cluster in certain regions of the taxonomy (e.g., in-context planning dominates language agents; post-training planning is more common in embodied agents), but the survey does not test whether these clusters reflect performance advantages or are artifacts of research community conventions. Section 5.1.2 catalogs domain-specific roles across seven application domains, but never compares a role-specialized multi-agent system to a homogeneous-agent system on the same task—the survey acknowledges this gap implicitly by not reporting such comparisons.
Mitigation status. The paper does not address this limitation. It presents the taxonomy as a given organizational scheme and does not discuss validation requirements. The open problems section (Section 8) does not identify taxonomy validation as a challenge. A researcher who wants causal confidence in the taxonomy would need to design layer-ablation studies—systems with capabilities systematically added or removed, tested across environments with varying dynamics—that the survey does not propose and the reviewed literature does not contain.
Limitation 2: The Survey Cannot Support Quantitative Method Selection Decisions
The assumption or constraint. The survey reviews over 100 benchmarks and hundreds of systems, but never reports a single table comparing methods by quantitative performance on a shared benchmark. This is not an oversight—it is a direct consequence of the field's fragmentation. The benchmarks reviewed in Section 7 are so heterogeneous in task definition, evaluation protocol, base model, and compute budget that aggregating results across them would be meaningless. The survey acknowledges this implicitly through its organizational choice: methods are classified by structural properties (reasoning trace format, memory type, optimization mode) rather than by accuracy numbers.
The consequence. A practitioner reading this survey to decide which planning method, tool-use approach, or memory architecture to adopt for their system receives no quantitative guidance. The survey can tell them that Tree of Thoughts and ReWOO are both in-context planning methods, and that RAGEN is a post-training planning method, but it cannot tell them which achieves higher task success rate, which requires less compute, or which is more robust to distribution shift. The survey can tell them that Reflexion and Self-Refine are both reflective feedback methods, but it cannot tell them whether one systematically outperforms the other on code debugging tasks.
This is a consequential limitation for a survey that positions itself as "actionable guidance for agentic systems" (abstract). The guidance is structural and conceptual—it helps a researcher understand what kind of method they might need—but it provides no empirical basis for choosing among methods of the same kind. The survey's classification tells you that you're in the "in-context tool-use" cell of the taxonomy; it does not tell you whether to use ReAct or ART or ChatCoT.
The limitation is compounded by the fact that the surveyed papers almost certainly overestimate their methods' performance due to standard sources of bias: benchmark overfitting (models may be indirectly tuned on test sets through iterative development), selective reporting (papers report the best of many hyperparameter configurations), and publication bias (negative results are less likely to be published). The survey inherits these biases but does not correct for them—it reports each paper's claimed results as-is. A meta-analysis that re-evaluated methods under standardized conditions would produce more reliable quantitative guidance, but the survey explicitly does not attempt this.
What evidence exists in the paper. The absence of quantitative comparisons is pervasive. Section 3.1 (planning) classifies over 50 methods into six categories but reports no accuracy numbers for any of them. Section 3.2 (tool use) reports that "RL has been shown to yield more robust, adaptive, and generalizable tool-use policies than SFT alone," but this is a qualitative summary—no table shows by how much, on which benchmarks, under what conditions. Section 4.1 (feedback) reports that reflective feedback methods "improve model reliability by modifying the reasoning process during inference," but does not quantify the improvement or compare it to parametric adaptation. Section 5.2.2 (topology optimization) discusses graph-based and policy-based methods but does not report their relative performance. The benchmark review in Section 7 catalogs what exists but makes no attempt to reconcile results across benchmarks.
Mitigation status. The paper partially acknowledges this limitation by providing extensive benchmark coverage (Section 7 catalogs dozens of evaluation protocols with their task formats and metrics) that a practitioner could use to design their own evaluation. Section 7.2 provides application-level benchmark descriptions that help a practitioner select relevant evaluation suites for their domain. But this is indirect—the survey tells you where to look for quantitative evidence, but does not synthesize that evidence into comparative conclusions. The open problems section (Section 8) does not identify the lack of cross-method quantitative comparison as a challenge, focusing instead on algorithmic and deployment challenges. A practitioner seeking quantitative guidance would need to follow the survey's references to original papers and conduct their own comparison—the survey does not do this work for them.
Limitation 3: The Self-Evolving Layer Conflates Within-Episode Adaptation and Across-Episode Learning
The assumption or constraint. Section 4 treats "self-evolving agentic reasoning" as a unified category encompassing both methods that adapt within a single inference episode (e.g., Reflexion's generate-critique-revise loop, Self-Refine's iterative improvement) and methods that learn across many episodes (e.g., Voyager's skill library accumulation, Memory-R1's trained memory management policy, ReST's training on critique-revision pairs). The survey's meta-update rule in Section 2.2—$S_{k+1} \leftarrow U(S_k, \tau_k, F_k)$—abstracts over both cases by indexing episodes by $k$, but this formalism does not distinguish between an "episode" as a single task instance and an "episode" as a complete task that the agent might encounter once or repeatedly.
The survey partially addresses this conflation in Section 5.3.1 by introducing the intra-test-time vs. inter-test-time distinction: intra-test-time evolution adapts during task execution; inter-test-time evolution consolidates adaptations across tasks. However, this distinction is introduced in the multi-agent evolution section rather than in the main self-evolution section (Section 4), and it is not applied systematically to the classification of single-agent feedback and memory systems.
The consequence. A practitioner trying to build a self-evolving system faces very different engineering requirements depending on whether they need within-episode adaptation or across-episode learning. Within-episode adaptation (e.g., self-correcting a faulty plan mid-execution) requires feedback mechanisms that operate at inference time with low latency—the agent cannot afford to run a full training loop after every action. Across-episode learning (e.g., accumulating a skill library over many Minecraft sessions) requires persistent memory that survives between episodes, mechanisms for credit assignment over long time horizons, and strategies for avoiding catastrophic forgetting of earlier skills.
The survey's unified treatment obscures these differences. A practitioner reading Section 4.1 (feedback) encounters Reflexion (within-episode self-critique) and ReST (across-episode training on critique-revision pairs) in the same section, with no discussion of how their deployment requirements differ. Reflexion can be deployed immediately on any model with no additional training; ReST requires curating a dataset of trajectories and running fine-tuning, which may take hours or days. The computational cost, latency profile, and infrastructure requirements are completely different, yet the survey treats them as variants of the same "agentic feedback" concept.
Similarly, Section 4.2 (memory) mixes systems like MemGPT (which extends context windows within a session by caching and retrieving past observations) with systems like Memory-R1 (which trains a dual-agent memory management policy through RL over many conversations). MemGPT is a drop-in inference-time augmentation; Memory-R1 requires substantial RL training on conversation datasets. The survey's classification by memory type (factual vs. experience) and format (flat vs. structured vs. post-training controlled) captures structural differences but not the deployment-readiness differences that matter most to a practitioner.
What evidence exists in the paper. The conflation is visible in the structure of Tables 5 and 6. Table 5 (feedback mechanisms) groups reflective feedback (inference-time) and parametric adaptation (training-time) under the same "Agentic Feedback Mechanisms" umbrella, with the only distinguishing column being "Update Target" (trajectory vs. model parameters). This column captures what is updated but not the cost, latency, or infrastructure implications of the update. Table 6 (memory) similarly groups in-context memory use and post-training memory control without a column for deployment complexity or training requirements.
Section 5.3.1 partially acknowledges the distinction by noting that intra-test-time evolution enables "correcting failures and refining strategies on the fly" while inter-test-time evolution enables "accumulation of persistent, generalizable capabilities over a lifelong interaction stream." But this discussion is buried in the multi-agent section and does not feed back into the single-agent self-evolution classification. A reader who only reads Sections 3–4 would not encounter this distinction at all.
Mitigation status. The paper makes a partial attempt to address this by introducing the intra/inter distinction in Section 5.3.1, but does not apply it retroactively to the main taxonomy in Section 4. A complete mitigation would restructure Section 4 to explicitly separate within-episode adaptation methods (reflective feedback, in-context memory) from across-episode learning methods (parametric adaptation, post-training memory control), with explicit discussion of the different infrastructure, data, and latency requirements of each. The survey does not attempt this restructuring. The open problems section (Section 8.2) discusses long-horizon credit assignment as a challenge but does not identify the within-episode vs. across-episode conflation as a limitation of the current taxonomy.
Limitation 4: No Accounting for Computational Cost, Latency, or Deployment Feasibility
The assumption or constraint. The survey analyzes reasoning mechanisms in abstraction from their computational requirements, wall-clock latency, memory footprint, or engineering complexity. The POMDP formalism in Section 2.2 defines optimality as maximizing expected return $J(\theta) = \mathbb{E}_{\tau}\left[\sum_{t \geq 0} \gamma^t r_t\right]$ without any constraint on the computational budget used to achieve that return. The taxonomy classifies methods by their reasoning structure (in-context vs. post-training, flat vs. structured memory, manual vs. learned collaboration) without considering how these choices translate to deployment costs.
This is not an oversight—it is a deliberate scope choice. The survey's goal is to organize the conceptual landscape of agentic reasoning, not to provide a cost-benefit analysis of deployment options. The POMDP formalism is standard in reinforcement learning and does not typically include compute budgets. However, the consequence for practitioners is significant.
The consequence. Methods that appear equivalent under the survey's taxonomy can have wildly different deployment feasibility. Consider two approaches to tool use: ReAct (in-context interleaving of reasoning and tool calls) and ToolRL (post-training RL for multi-tool policy learning). Under the survey's classification (Table 3), both are tool-use optimization methods—ReAct is "In-Context Integration," ToolRL is "Post-Training Integration." A practitioner reading the taxonomy would understand that they differ in when optimization occurs, but would not learn that:
- ReAct requires no training infrastructure and can be deployed immediately on any instruction-tuned LLM. Its cost is in inference-time latency and API call overhead—every tool invocation requires generating reasoning tokens and waiting for tool responses, which can make a single task take minutes.
- ToolRL requires substantial RL training infrastructure—environment simulation for tool interaction, reward design, distributed training for GRPO-style optimization over thousands of episodes. Training may take days or weeks of GPU time. However, once trained, the model can invoke tools in a single forward pass with low latency.
These are not subtle differences—they determine whether a method is viable for a given deployment scenario. A startup building a real-time chatbot cannot wait minutes per response and will prefer post-training. A research lab exploring a novel task distribution where tool use patterns are not yet stable will prefer in-context methods because retraining for each task variant is infeasible. The survey's taxonomy does not capture this tradeoff.
The same issue applies across the entire taxonomy. Self-evolving methods that use reflective feedback (Section 4.1.1) add inference-time latency proportional to the number of critique-revision cycles. Parametric adaptation methods (Section 4.1.2) add training-time compute but no inference-time overhead. Collective methods that use all-to-all agent communication (Section 5.2) scale quadratically in the number of agents, making them impractical beyond small teams. Methods that learn communication topologies (Section 5.2.2) add a training phase that itself requires running multiple agent configurations to evaluate performance. None of these cost structures are captured in the taxonomy.
What evidence exists in the paper. The survey provides no systematic treatment of computational cost. Individual system descriptions occasionally mention efficiency concerns: Section 5.1.1 notes that "communication overhead can easily undermine MAS efficiency" and that the communication facilitator role exists to "manage communication bandwidth." Section 3.2.1 mentions that GEAR uses a small language model for tool selection "to reduce costs." But these are qualitative asides, not systematic accounting.
The survey provides no table comparing methods by training FLOPs, inference FLOPs per task, wall-clock latency, memory footprint, or infrastructure requirements. The GRPO objective (Equation 3) involves sampling $G$ outputs per prompt—the survey notes that $G$ is typically 4–64, but does not discuss how the choice of $G$ trades off training cost against policy improvement. The inference-time search objective (Equation 2) involves evaluating many candidate trajectories—the survey does not discuss how the branching factor and search depth affect compute requirements.
Mitigation status. The paper does not address this limitation. The open problems section (Section 8) identifies challenges in robustness, trustworthiness, and efficiency, but "efficiency" is discussed in terms of multi-agent coordination overhead and long-horizon credit assignment, not in terms of the systematic cost analysis that a practitioner needs for method selection. The survey could have included a section on "practical deployment considerations" that mapped taxonomy categories to typical compute requirements, latency profiles, and infrastructure needs—it does not attempt this. A practitioner seeking cost guidance must follow references to individual papers and extract cost information themselves, which varies widely in availability and measurement methodology.
Limitation 5: The Application-Domain Analysis Demonstrates Coverage, Not Predictive Power
The assumption or constraint. Section 6 applies the three-layer taxonomy to five application domains—math exploration and vibe coding, scientific discovery, embodied robotics, healthcare and medicine, and autonomous web exploration. Within each domain, the survey identifies systems that exhibit foundational capabilities (planning, tool use, search), self-evolving capabilities (feedback, memory, capability refinement), and collective capabilities (multi-agent collaboration). The analysis follows a consistent template: for each domain, there is a subsection on foundational reasoning, a subsection on self-evolving reasoning, and a subsection on collective reasoning.
The survey presents this as evidence that the taxonomy is general—it can describe diverse systems across domains. The introduction frames the application analysis as "illustrating how different agentic reasoning mechanisms are instantiated and evaluated across domains." But there is a gap between demonstrating that the taxonomy can describe systems (coverage) and demonstrating that the taxonomy predicts system behavior or guides design (predictive power).
The consequence. A practitioner reading the application analysis learns that their domain contains examples of each reasoning layer, but does not learn whether those layers are necessary for success in their domain, or whether investing in a higher layer will improve performance relative to staying at a lower layer. For example, Section 6.4 (healthcare) describes clinical planning systems (foundational), self-evolving systems with memory and feedback (e.g., EHRAgent's feedback learning, MedAgentSim's experience histories), and multi-agent diagnostic collaboration (e.g., MDAgents, DoctorAgent-RL). But the survey does not report any study that compares a foundational-only diagnostic system to a self-evolving diagnostic system on the same clinical benchmark. A healthcare AI developer reading the survey learns that both types of systems exist, but does not learn whether adding self-evolution mechanisms to a foundational system improves diagnostic accuracy, reduces harmful errors, or enables new clinical workflows.
The same gap appears in every application domain. Section 6.1 (math exploration) describes foundational planning and tool-use in systems like AlphaGeometry and FunSearch, self-evolving feedback in program-search-based discovery, and collective multi-agent systems like multi-component polytope generation. But the survey does not report whether collective systems outperform foundational systems on math discovery tasks, or whether self-evolving feedback improves the rate of mathematical discovery over static tool-use.
The survey's implicit claim—that the three-layer taxonomy captures genuine stages of reasoning sophistication—predicts that higher-layer systems should outperform lower-layer systems on tasks requiring the corresponding capabilities. For example, a self-evolving system should outperform a foundational system on tasks where the environment changes over time, because the foundational system lacks mechanisms to adapt. The application analysis does not test this prediction anywhere. It catalogs what exists; it does not evaluate what works.
What evidence exists in the paper. The application-domain analysis (Section 6) is purely descriptive. It identifies systems and describes their reasoning mechanisms but never compares them quantitatively. There are no tables comparing system performance on domain-specific benchmarks. There are no ablation studies showing that adding self-evolving mechanisms to a foundational system improves performance. There are no failure analyses showing that foundational systems fail in characteristic ways that self-evolving systems avoid.
The benchmark review (Section 7) catalogs evaluation protocols and metrics for each domain, but the application analysis does not reference these benchmarks to compare systems. Section 7.2.4 describes medical benchmarks (AgentClinic, MedAgentBench, MedAgentsBench) that could be used to compare foundational and self-evolving diagnostic systems, but Section 6.4 does not report results from these benchmarks. Section 7.2.1 describes embodied benchmarks (ALFWorld, OSWorld) that could compare foundational planning to self-evolving skill acquisition, but Section 6.3 does not report such comparisons.
Mitigation status. The paper does not acknowledge this as a limitation. The application analysis is presented as a demonstration of the taxonomy's applicability and completeness—it shows that agentic reasoning mechanisms "are instantiated and evaluated across realistic domains and task settings" (Section 1). The survey does not frame the application analysis as a validation study and does not discuss what a rigorous validation would require. A more critical framing would acknowledge that the application analysis demonstrates descriptive coverage but leaves open the question of whether the taxonomy captures causally meaningful structure. The open problems section (Section 8) does not identify domain-specific evaluation of the taxonomy's predictive power as a challenge.
Limitation 6: The Formal Framework Is Presented but Not Used as an Analytical Tool Throughout the Survey
The assumption or constraint. Section 2.2 introduces a formal POMDP framework for agentic reasoning, including the environment tuple $\langle \mathcal{X}, \mathcal{O}, \mathcal{A}, \mathcal{Z}, \mathcal{M}, \mathcal{T}, \Omega, \mathcal{R}, \gamma \rangle$, the policy factorization $\pi_{\theta}(z_t, a_t \mid h_t) = \pi_{\text{reason}}(z_t \mid h_t) \cdot \pi_{\text{exec}}(a_t \mid h_t, z_t)$, the in-context search objective $\tau^{\star} \in \arg\max_{\tau} \sum_{t} \hat{v}_{\phi}(u_t)$, the GRPO post-training objective (Equations 3–4), the self-evolving meta-update $S_{k+1} \leftarrow U(S_k, \tau_k, F_k)$, and the Dec-POMDP extension to multi-agent settings. This formalism is the survey's primary theoretical contribution—it provides a unified mathematical language for discussing agentic reasoning across layers and optimization modes.
The consequence. The formalism is almost entirely absent from the subsequent sections. Sections 3–6, which constitute the bulk of the survey (over 40 pages of analysis), do not systematically use the POMDP vocabulary to analyze methods. The policy factorization is never referenced when discussing planning methods in Section 3.1—the survey does not, for example, explain that Tree of Thoughts implements $\pi_{\text{reason}}$ as MCTS over $\mathcal{Z}$ with a learned heuristic $\hat{v}_{\phi}$, while ReWOO implements $\pi_{\text{reason}}$ as a decomposition step followed by $\pi_{\text{exec}}$ tool invocation. The $\hat{v}_{\phi}$ verifier is never referenced in the tool-use section (Section 3.2) to explain how methods differ in their scoring of tool invocation decisions. The self-evolving meta-update is referenced qualitatively in Section 4 but not applied systematically—the survey does not, for each feedback method, specify what $S_k$, $U$, and $F_k$ correspond to.
The result is a disconnect between the survey's theoretical framing and its practical analysis. A reader who carefully works through the POMDP formalism in Section 2.2—understanding the distinction between $\mathcal{Z}$ and $\mathcal{A}$, the factorization of the policy, the inference-time search objective—arrives at Section 3 expecting these concepts to structure the subsequent discussion. Instead, they encounter a descriptive taxonomy organized by method type (planning, tool use, search) and format (in-context vs. post-training), with no systematic mapping back to the formal framework. The formalism is mentioned occasionally—the GRPO objective appears in discussions of post-training tool use (Section 3.2.2) and multi-agent training (Section 5.3.3)—but it is not used as the primary analytical lens.
This matters because the formalism is the survey's most distinctive intellectual contribution relative to other LLM reasoning and agent surveys. Prior surveys (Huang and Chang, 2022; Chen et al., 2025; Zhang et al., 2025) do not provide a unified POMDP formalism. The survey's formalism could have enabled precise comparisons—for example, showing that two methods differ only in their choice of $\hat{v}_{\phi}$ (learned reward model vs. execution feedback) or only in whether they optimize $\theta$ directly (post-training) or search over $\mathcal{Z}$ with frozen $\theta$ (in-context). The survey does not make these comparisons. It uses the formalism to define vocabulary in Section 2.2 and then largely abandons it.
What evidence exists in the paper. The formalism appears in Section 2.2 and is referenced in Section 3.2.2 (GRPO for tool-use RL) and Section 5.3.3 (multi-agent GRPO variants like MAGRPO and MHGPO). It is otherwise absent. A text search through Sections 3–6 for $\pi_{\text{reason}}$, $\pi_{\text{exec}}$, $\mathcal{Z}$, $\hat{v}_{\phi}$, $U(S_k, \tau_k, F_k)$, or "Dec-POMDP" reveals almost no usage beyond Section 2.2. The survey's Tables 2–6 classify methods by structure, format, tool strategy, and memory type—categories that could have been mapped to the formalism (e.g., "structure" could distinguish methods that use tree search over $\mathcal{Z}$ from methods that use sequential rollouts) but are not.
Mitigation status. The paper does not address this disconnect. The formalism is presented as the theoretical foundation but not maintained as an analytical tool. This may reflect the realities of survey writing—a rigorous formal analysis of every method in the survey would be extremely lengthy and might not add insight proportional to its cost. However, the current presentation creates an expectation that is not fulfilled. A reader who wants a formal understanding of how specific methods relate to the POMDP must do the mapping themselves, using the vocabulary from Section 2.2 but without guidance from the survey. A more integrated approach would have used the formalism to organize at least the classification tables—labeling each method by its instantiation of $\pi_{\text{reason}}$, $\pi_{\text{exec}}$, and $\hat{v}_{\phi}$—or provided formal case studies of representative methods in each section. The survey attempts neither.
7. Implications and Future Directions
How This Work Changes the Landscape
This survey does not propose a new algorithm, model, or benchmark. Its contribution is conceptual: it provides the first unified organizational framework for reasoning about agentic reasoning as a coherent phenomenon rather than a loose collection of techniques. The magnitude is best characterized as a reframing and synthesis, not a paradigm shift—the individual methods it catalogs already exist in the literature. What changes is how the field can think about those methods: their relationships, their complementary strengths, and the environmental conditions under which each is appropriate.
The shift from method-centric to capability-centric thinking. Before this survey, the dominant organizational scheme for LLM reasoning was by technique family: chain-of-thought prompting, tree search, self-correction, tool use, multi-agent debate. Each technique family had its own literature, its own benchmarks, and its own community. The survey replaces this with a capability-centric scheme: what does reasoning demand at each level of environmental dynamism? Foundational reasoning (stable environments) demands planning, tool use, and search. Self-evolving reasoning (dynamic environments) additionally demands feedback integration and persistent memory. Collective reasoning (strategic environments) additionally demands role differentiation, communication, and coordination. This reframing matters because it explains why methods fail when moved across environments—not because of bugs in the algorithm, but because the environment demands capabilities the method was not designed to provide.
A concrete example: the survey's taxonomy explains why chain-of-thought prompting (a foundational in-context method) works well on MATH but fails on WebArena. MATH is a stable environment with a clear goal and no state change during problem-solving. WebArena is a dynamic environment where page content changes with each action and the agent must maintain state across steps. The failure is not a limitation of chain-of-thought—it is a category error: applying a Layer 1 method to a Layer 2 environment. Before this survey, such failures were documented but not systematically explained. The taxonomy provides the vocabulary for the explanation.
Reconciling contradictions in the self-correction literature. The survey implicitly resolves a tension that has persisted in the field: some papers find that LLMs can self-correct their reasoning (Madaan et al., 2023; Shinn et al., 2023), while others find that self-correction is largely ineffective (Huang et al., 2023; Stechly et al., 2023). The survey's taxonomy explains this as a matter of environmental dynamics and feedback quality. Self-correction works when (1) the base model can already generate reasonable outputs (the problem is within its capability range), (2) the feedback signal is reliable (execution results, ground-truth verifiers), and (3) the environment is stable enough that correction within an episode translates to improved outcomes. Self-correction fails when the model cannot generate correct solutions at any nontrivial rate (difficulty bin 5 in the compute-optimal scaling literature), when the feedback signal is noisy or self-generated, or when the environment changes too rapidly for within-episode reflection to accumulate into improvement.
The survey does not test this resolution experimentally—it synthesizes it from the structure of the taxonomy. But the resolution is actionable: a researcher evaluating a new self-correction method should report difficulty-stratified results (does it help on easy, medium, and hard problems?) and feedback-source ablations (does it work with self-generated critique, verifier scores, and ground-truth labels?). The taxonomy makes these analyses natural rather than optional.
Research directions that become more attractive. (1) Systematic comparison of in-context and post-training implementations of the same reasoning capability, with matched compute budgets, to quantify the flexibility-efficiency tradeoff that the survey's orthogonal axis posits but does not validate. (2) Layer-ablation studies that systematically add or remove feedback, memory, and coordination mechanisms from agent systems and measure the impact on tasks with varying environmental dynamics, to test whether the taxonomy's layers are causally meaningful. (3) Unified benchmarks that span multiple layers—a benchmark suite where the same underlying task (e.g., question answering) is instantiated in a stable environment (static dataset), a dynamic environment (information changes over time, requiring memory), and a strategic environment (multiple agents with different information must collaborate), to test whether performance differences across layers are systematic. (4) Verifier robustness as a first-class research problem—the survey's classification of search methods and feedback mechanisms reveals that learned verifiers (PRMs, ORMs) are the primary bottleneck for both in-context reasoning (over-optimization during tree search) and post-training reasoning (reward hacking during RL). Research on verifier calibration, adversarial robustness, and ensemble methods becomes more attractive because it addresses a cross-cutting bottleneck rather than a method-specific limitation.
Research directions that become less attractive. (1) Incremental improvements to in-context prompting for static benchmarks (e.g., yet another variant of chain-of-thought that achieves +0.5% on GSM8K). The survey's framework makes clear that static benchmarks only test Layer 1 capabilities, and that progress on these benchmarks says little about agentic reasoning in dynamic environments. The field's overinvestment in static math reasoning benchmarks is not a path to general agentic intelligence. (2) Single-axis method comparisons without accounting for environmental dynamics or compute budget. A paper that claims "Method A outperforms Method B on benchmark X" without reporting difficulty-stratified results and compute-matched baselines provides limited insight under the survey's framework, because the comparison may reflect a mismatch between method capability and environment demands rather than a genuine algorithmic improvement.
Follow-Up Research This Work Enables
Causal validation of the three-layer taxonomy through capability ablation studies. The survey's central claim—that foundational, self-evolving, and collective reasoning represent structurally different capability demands—is asserted but not demonstrated experimentally. A strong follow-up would construct a single agent system that can be run in three configurations: foundational-only (planning + tool use + search, no memory or feedback), foundational + self-evolving (adding feedback integration and persistent memory), and full-stack (adding multi-agent coordination). This system would be evaluated on a benchmark suite where task instances are explicitly annotated with the environmental dynamics they require: stable tasks (static QA, single-step tool use), dynamic tasks (multi-turn interactions where state changes, tasks requiring learning from past failures across episodes), and strategic tasks (tasks requiring coordination with other agents with partial information). The prediction from the survey's taxonomy is that foundational-only systems should perform comparably to self-evolving systems on stable tasks but degrade sharply on dynamic tasks, and that the degradation mode should be predictable: not just lower accuracy, but specific failure patterns like inability to recover from errors across episodes or failure to adapt strategies based on accumulated experience. A study that confirms these predictions would transform the taxonomy from a descriptive scheme to a causally validated model of reasoning capability layers.
Systematic in-context versus post-training comparison on shared benchmarks with compute-matched budgets. The survey's orthogonal optimization axis posits that in-context reasoning (inference-time search with frozen parameters) and post-training reasoning (parameter optimization through RL or SFT) represent a fundamental tradeoff between flexibility and efficiency. This tradeoff is mathematically grounded in the POMDP formalism (Section 2.2) but has never been tested experimentally in a controlled setting. A strong follow-up would select three reasoning capabilities that can be implemented both ways—tool-use policy learning, search strategy optimization, and feedback integration—and implement both in-context and post-training versions sharing the same base model. Evaluation would use a common benchmark suite with tasks varying in distribution shift (to test flexibility) and compute budgets matched across conditions (total FLOPs, not just inference FLOPs, to account for training cost). The in-context version would spend its budget on test-time search over reasoning traces; the post-training version would amortize its budget into RL training. The prediction is that in-context versions should maintain higher accuracy under distribution shift (new tools, new search environments, new feedback types) because they do not overfit to training distributions, while post-training versions should achieve lower inference latency and higher accuracy under in-distribution evaluation because they amortize optimization into model weights. Quantifying this tradeoff—how much accuracy does post-training lose per unit of distribution shift, and how much latency does in-context reasoning add per unit of accuracy gain—would provide the quantitative guidance that the survey currently lacks.
Unified multi-method benchmark for self-evolving memory architectures. The survey classifies 30+ memory systems (Table 6) across dimensions of architecture (flat, hierarchical, heterogeneous), topology (centralized, distributed, hybrid), content (semantic, procedural, cognitive-phase), and management (forgetting, verification, learning-based control), but provides no quantitative comparison across these dimensions on shared tasks. A strong follow-up would construct a benchmark specifically designed to test memory system capabilities: long-horizon tasks requiring retention across episodes with controlled delays between relevant information and query time (to test retrieval precision), tasks with deliberately introduced contradictory information across episodes (to test conflict resolution and memory updating), tasks with varying information density (to test storage efficiency and forgetting policies), and tasks requiring information synthesis across multiple stored episodes (to test reasoning over memory rather than just retrieval from memory). A suite of representative memory architectures—one flat in-context memory (MemGPT-style), one structured graph memory (MEM0-style), one workflow memory (Workflow Memory-style), and one post-training controlled memory (Memory-R1-style)—would be evaluated on this benchmark with a shared base model. The survey's taxonomy predicts that flat memory should be sufficient for simple factual recall, structured memory should outperform flat memory on multi-hop and relational queries, workflow memory should excel on procedural tasks, and post-training controlled memory should show superior adaptability to changing task distributions. Testing these predictions would validate the taxonomy's finer-grained classification within the self-evolving layer and provide practitioners with evidence-based guidance for memory system selection.
Difficulty-conditioned evaluation of agentic reasoning methods across application domains. The survey's application analysis (Section 6) demonstrates that the three-layer taxonomy can describe systems across five domains, but does not show that the taxonomy predicts performance. A strong follow-up would take the insight from the compute-optimal test-time scaling literature (referenced in Section 2's discussion of reasoning under uncertainty) and apply it to agentic reasoning evaluation: for each application domain, stratify tasks by difficulty relative to base model capability (e.g., using the PRM score distribution or pass@1 rate to bin tasks into quintiles, following the methodology of Snell et al., 2024). Then evaluate representative systems from each taxonomy layer on each difficulty bin. The prediction is that foundational methods should perform well on easy tasks (where the model's base capabilities suffice) but hit a ceiling on hard tasks (where adaptation or collaboration are required), self-evolving methods should show the largest gains on medium tasks (where feedback and memory can push the model past its base capability), and collective methods should show gains on tasks requiring diverse expertise or parallel exploration. A negative result—finding that self-evolving methods provide no benefit over foundational methods even on hard dynamic tasks—would be extremely informative: it would suggest that the taxonomy's layers are not capability-enabling but merely descriptive of existing system designs. A positive result would transform the taxonomy into a practical tool for matching system architecture to task difficulty.
Verifier robustness as a cross-cutting bottleneck: a systematic characterization study. The survey's treatment of search (Section 3.3), feedback (Section 4.1), and post-training (Section 2.2, GRPO) all point to learned verifiers and reward models as the critical reliability bottleneck for agentic reasoning. In-context search methods over-optimize against verifier scores, finding solutions that score highly but are incorrect. Post-training RL methods suffer from reward hacking, where the policy exploits imperfections in the reward model. Reflective feedback methods fail when self-generated critique is unreliable. However, the verifier robustness problem has been studied primarily in the context of specific methods (e.g., PRM over-optimization in beam search for math reasoning) rather than as a cross-cutting phenomenon. A strong follow-up would systematically characterize verifier robustness across optimization methods: train a single verifier (e.g., a process reward model for mathematical reasoning) and then subject it to increasing optimization pressure through in-context beam search (varying beam width), best-of-N weighted selection (varying N), and post-training GRPO (varying training steps). At each optimization level, measure both verifier score and ground-truth accuracy to trace the over-optimization curve. The study would additionally test verifier training interventions—soft Monte Carlo labels versus binary correctness labels (Appendix E of the compute-optimal scaling paper), ensemble verification, adversarial training on search-generated solutions—to determine which most improve robustness. The survey's framework makes this study natural because it reveals that the same verifier underlies multiple reasoning mechanisms; improving verifier robustness would simultaneously improve in-context search, post-training RL, and reflective feedback.
Practical Applications and Downstream Use Cases
Agentic system design for domain-specific deployments. The survey's most immediately actionable contribution is its application-domain analysis (Section 6), which provides a template for mapping domain requirements to reasoning capabilities. A team building a healthcare AI system, for example, can consult Section 6.4 to identify that clinical decision-making demands: (1) foundational planning for diagnostic workflows (mapping symptoms to test recommendations, integrating guideline criteria into structured plans), (2) tool use for accessing medical knowledge bases, executing clinical calculators, and interpreting medical images, (3) self-evolving memory for maintaining patient-specific context across encounters and updating recommendations when new lab results arrive, and (4) in complex cases, multi-agent collaboration where specialist agents (e.g., radiologist, pathologist, primary care) coordinate to reach consensus diagnoses. The survey does not tell the team which specific method to use for each capability, but it tells them what capabilities they need and provides references to systems that have implemented each capability (e.g., MedAgent-Pro for hierarchical diagnostic planning, TxAgent for pharmacological tool integration, EHRAgent for structured EHR reasoning with feedback, MDAgents for adaptive multi-agent medical collaboration). This is more actionable than a generic "LLMs can be used for healthcare" recommendation and more systematic than ad hoc component selection.
Benchmark selection for method evaluation. The survey's benchmark review (Section 7) organizes evaluation suites by core mechanism and application domain, enabling researchers to select benchmarks that test the specific capabilities their method targets. A team developing a new memory architecture for long-horizon reasoning can consult Section 7.1.3 to identify relevant benchmarks: LOCOMO and LONGMEMEVAL for multi-session recall, MultiChallenge for memory-intensive conversation phenomena (retention, inference, editing, coherence), MemBench for factual and reflective memory across 60K episodes, and Evo-Memory for experience reuse in test-time learning. Without this organization, the team might default to generic LLM benchmarks (MMLU, GSM8K) that provide no signal about memory capability. The survey's categorization by mechanism type—tool use, search, memory and planning, multi-agent—makes it straightforward to identify which benchmarks isolate which capability, avoiding the common pitfall of evaluating a new memory system on a benchmark that primarily tests reasoning depth rather than retention and retrieval across long contexts.
Curriculum design for agent training pipelines. The survey's three-layer structure—foundational, self-evolving, collective—can be operationalized as a training curriculum for building increasingly capable agent systems. A research team aiming to build a general-purpose agent could structure development in three phases: Phase 1 (foundational) focuses on single-agent planning, tool use, and search in stable environments, using benchmarks like ToolQA, APIBench, and WebArena (single-task mode). Phase 2 (self-evolving) adds feedback integration and persistent memory, evaluated on benchmarks with dynamic state and long-horizon dependencies like MultiChallenge, MemBench, and ALFWorld. Phase 3 (collective) introduces multi-agent coordination, evaluated on benchmarks like MultiAgentBench, LLM-Coordination, and SMART-LLM. The survey's taxonomy provides the rationale for this ordering: capabilities at each layer depend on capabilities from lower layers. An agent that cannot plan and use tools (Layer 1) cannot benefit from feedback on its tool-use decisions (Layer 2). An agent that cannot maintain persistent memory of past interactions (Layer 2) cannot coordinate effectively with other agents (Layer 3), because coordination requires remembering past communications and commitments. The curriculum design follows directly from the taxonomy's cumulative structure, making the survey a practical development roadmap as well as a conceptual framework.
Investment prioritization for AI research organizations. The survey's identification of verifier robustness as a cross-cutting bottleneck has direct implications for how organizations should allocate research investment. Rather than funding separate teams to improve in-context search algorithms, post-training RL methods, and reflective feedback mechanisms—each of which would independently encounter the same verifier over-optimization ceiling—an organization could centralize investment in verifier robustness research, with the understanding that improvements to verifier calibration, adversarial training, and ensemble verification would simultaneously improve all downstream reasoning methods that depend on the verifier. The survey makes this argument implicitly through its classification structure: the same verifier (or reward model) appears as $\hat{v}_{\phi}$ in in-context search (Section 2.2), as the reward signal in GRPO post-training (Section 2.2), and as the critique source in validator-driven feedback (Section 4.1.3). Making this dependency explicit enables an organization to identify the highest-leverage investment: a 10% improvement in verifier reliability likely produces larger aggregate system improvement than a 10% improvement in any individual method that uses the verifier, because the verifier improvement compounds across all downstream uses. This is a concrete resource allocation insight that emerges from the survey's organizational structure but is not stated as a recommendation—practitioners can extract it directly from the taxonomy.
