ReAct: Synergizing Reasoning and Acting in Language Models

ArXiv: 2210.03629

🎯 Pitch

Simply prompting a large language model to interleave reasoning with tool use yields a 34% absolute gain over standard reinforcement learning on a household decision-making benchmark—all from just one or two in-context examples. When these reasoning traces are combined with Wikipedia lookups for question answering, the same approach cuts the massive sample budgets needed by chain-of-thought, matching its 21-sample performance with only 3–5 reasoning-acting cycles.

---

1. Executive Summary

This paper introduces ReAct, a prompting paradigm that synergizes reasoning and acting in large language models by interleaving the generation of verbal reasoning traces with task-specific actions (e.g., decomposing a question into subgoals while searching Wikipedia, or tracking subgoal completion while navigating a simulated household). Evaluated on the HotpotQA multi-hop QA benchmark, the Fever fact verification benchmark, and the ALFWorld and WebShop interactive decision-making environments using PaLM-540B, ReAct achieves a 34% absolute success rate improvement over imitation and reinforcement learning baselines on ALFWorld and a 10% improvement on WebShop, while on HotpotQA and Fever the best method combines ReAct with chain-of-thought self-consistency — outperforming CoT-SC alone across all sample sizes (e.g., matching 21-sample CoT-SC performance with only 3–5 samples). The paper establishes that reasoning traces improve action generation by enabling the model to induce, track, and adjust plans, while external actions ground reasoning in retrieved knowledge — reducing hallucination rates compared to chain-of-thought alone — but only when the action space is deliberately constrained to require explicit reasoning for effective retrieval.

2. Context and Motivation

The Core Problem: Reasoning and Acting Are Studied in Isolation

The fundamental gap this paper addresses is that reasoning and acting in language models have been treated as separate research threads, despite the fact that humans solve complex tasks by seamlessly interleaving the two. When you cook an unfamiliar dish, you don't first write out a complete plan in your head and then execute it blindly — you reason about what to do next ("now that the vegetables are chopped, I should heat the oil"), act (turn on the stove, check the fridge), observe the results, and adjust your reasoning accordingly ("I don't have bell peppers, so I'll use zucchini instead"). This tight loop between internal verbal reasoning and external action is, the authors argue, fundamental to how humans handle novel situations, recover from errors, and gather information they don't already possess.

The paper frames this gap concretely (Section 1):

"While large language models (LLMs) have demonstrated impressive performance across tasks in language understanding and interactive decision making, their abilities for reasoning (e.g. chain-of-thought prompting) and acting (e.g. action plan generation) have primarily been studied as separate topics."

This separation matters because reasoning without acting is brittle and ungrounded, while acting without reasoning is myopic and inflexible. The paper demonstrates both failure modes in Figure 1. In Figure 1(1b), chain-of-thought reasoning alone hallucinates facts about the Apple Remote — it confidently states that the remote can control iPhone, iPad, and iPod Touch, but this is incorrect because it has no way to verify its internal knowledge against the real world. In Figure 1(2a), an act-only agent in ALFWorld repeatedly tries to take a peppershaker from a sinkbasin that doesn't contain one — it fails because it cannot reason abstractly about where objects are likely to be found or track what it has already tried.

Why This Gap Is Important

The significance of bridging this gap extends across multiple dimensions:

Practical reliability. Language models deployed as assistants or agents need to operate on accurate, up-to-date information. Pure reasoning approaches like chain-of-thought (Wei et al., 2022) rely entirely on the model's parametric knowledge, which is frozen at training time and susceptible to hallucination. As the paper shows (Table 2), hallucination accounts for 56% of CoT failures on HotpotQA, while ReAct's ability to retrieve external information brings this to near zero. For any application where factual accuracy matters — medical information, legal advice, current events — this difference is decisive.

Interpretability and trust. When a model interleaves reasoning traces with actions and observations, humans can inspect exactly what information the model retrieved, how it reasoned about that information, and why it chose a particular action. This is not just about debugging — it enables a new form of human-AI collaboration. In Section 4 (Figure 5), the paper shows that a human can edit a single flawed reasoning trace in ReAct's trajectory, and the model will adjust its entire subsequent behavior accordingly. This kind of targeted correction is impossible with either pure reasoning (where there's no action to redirect) or pure acting (where there's no reasoning trace to edit).

Sample efficiency in decision-making. On ALFWorld and WebShop, ReAct achieves strong performance with only one to three in-context examples (Section 4). In contrast, the imitation learning baseline BUTLER was trained on $10^5$ expert trajectories per task type, and WebShop's IL agent was trained on 1,012 human-annotated trajectories. ReAct's few-shot performance (71% success on ALFWorld vs. 37% for BUTLER) suggests that interleaving reasoning with acting allows language models to leverage their pretrained commonsense knowledge and reasoning capabilities in ways that dramatically reduce the need for task-specific training data. This is a significant practical advantage for domains where expert demonstrations are expensive or impossible to collect at scale.

Theoretical understanding of language model capabilities. The paper probes a fundamental question: to what extent can a frozen language model, through clever prompting alone, serve as a general-purpose agent that plans, reasons, acts, and recovers from errors? By demonstrating that the same model (PaLM-540B) can solve math reasoning problems, navigate simulated houses, and shop on websites — all through the same ReAct prompting paradigm — the paper provides evidence that these capabilities are latent in large language models and can be elicited through appropriate scaffolding. This has implications for how we think about the relationship between language, reasoning, and action in both artificial and human intelligence.

Prior Approaches and Their Limitations

The paper identifies two main research threads that ReAct seeks to unify, each with specific shortcomings.

Chain-of-Thought Reasoning and Its Variants

Chain-of-thought prompting (CoT; Wei et al., 2022) demonstrated that LLMs can generate step-by-step reasoning traces that improve performance on arithmetic, commonsense, and symbolic reasoning tasks. The key insight was that by providing few-shot examples where intermediate reasoning steps are spelled out, models learn to produce their own reasoning chains rather than jumping directly to answers. Follow-up work improved CoT with self-consistency (CoT-SC; Wang et al., 2022a), which samples multiple reasoning chains and takes a majority vote, and with more sophisticated reasoning architectures like least-to-most prompting (Zhou et al., 2022) and selection-inference (Creswell et al., 2022).

However, the paper identifies a fundamental limitation that all these approaches share (Section 1):

"this 'chain-of-thought' reasoning is a static black box, in that the model uses its own internal representations to generate thoughts and is not grounded in the external world, which limits its ability to reason reactively or update its knowledge."

The consequence is twofold. First, hallucination: the model confidently generates factually incorrect reasoning steps because it has no mechanism to verify its claims. The paper's error analysis (Table 2) shows this is not a rare edge case — 56% of CoT failures on HotpotQA involve hallucinated facts. Second, inability to recover: when CoT makes an error early in its reasoning chain, it cannot course-correct because it never receives external feedback. The reasoning proceeds deterministically from flawed premises to flawed conclusions.

Other structured reasoning approaches share similar limitations. Scratchpad (Nye et al., 2021) fine-tunes models to produce intermediate computation steps but remains entirely internal to the model. Faithful reasoning (Creswell & Shanahan, 2022) decomposes multi-step reasoning into dedicated modules but still operates without external grounding. STaR (Zelikman et al., 2022) bootstraps reasoning by fine-tuning on correct rationales, which can improve reasoning quality but doesn't address the underlying problem that all knowledge comes from the model's parameters.

Language Models for Decision Making and Acting

A parallel line of work has explored using LLMs to generate actions in interactive environments. WebGPT (Nakano et al., 2021) trains an LM to browse the web and answer questions, using reinforcement learning from human feedback to learn effective search and navigation strategies. In embodied AI, SayCan (Ahn et al., 2022) uses LLMs to propose robot actions, which are then filtered by an affordance model grounded in visual observations. Inner Monologue (Huang et al., 2022b) adds environment feedback as injected "inner monologue" text, creating a closed-loop system where the model's actions influence subsequent observations.

The paper identifies specific limitations in these approaches. First, they do not employ LLMs to reason abstractly about high-level goals or maintain a working memory. The authors explicitly note that Inner Monologue — despite its name — primarily injects observations of environment state and completed subgoals, rather than engaging in the kind of flexible reasoning ReAct enables (decomposition, commonsense inference, plan adjustment). Second, WebGPT and similar systems (BlenderBot, Sparrow, SimpleTOD) rely on expensive human feedback for reinforcement learning or imitation learning, requiring thousands of annotated examples. In contrast, ReAct achieves competitive or superior performance with a handful of prompt examples. Third, these systems were developed for domain-specific action spaces and don't generalize across the diverse range of tasks (QA, fact verification, household navigation, web shopping) that ReAct handles with a unified paradigm.

Some prior work has combined aspects of reasoning and acting, but in limited ways. Huang et al. (2022b) included a form of verbal reasoning to reiterate spatial facts about the current state in embodied tasks, but this is narrow compared to ReAct's flexible thought types. Internet-augmented LMs (Lazaridou et al., 2022; Shuster et al., 2022a) retrieve documents to condition generation, but they don't interleave reasoning with retrieval in a step-by-step, closed-loop fashion — they typically retrieve once and then generate, without the iterative refinement that ReAct's thought-action-observation cycles enable.

The Missing Piece: Synergy Between Reasoning and Acting

The paper argues that neither thread alone is sufficient because they miss the bidirectional synergy that makes human problem-solving effective. The relationship works in both directions:

• Reason → Act: reasoning traces help the model decompose goals, track subgoal completion, determine what action to take next, and recover when actions fail. Without this, act-only models get stuck in loops (as in Figure 1(2a)) or retrieve irrelevant information because they can't formulate targeted queries. Table 1 shows that ReAct outperforms Act on both HotpotQA (27.4 vs. 25.7 EM) and Fever (60.9 vs. 58.9 accuracy), and the ALFWorld results in Table 3 show a consistent 33–90% relative improvement from adding thoughts to actions.

• Act → Reason: external actions (searching Wikipedia, navigating to a new location, clicking a product option) bring new information into the model's context, which subsequent reasoning steps can incorporate. This grounds the reasoning in factual information, dramatically reducing hallucination. Without this, reasoning models generate plausible-sounding but incorrect chains. The paper's error analysis shows that ReAct's false positive rate (hallucinated correct answers) is less than half of CoT's (6% vs. 14%), and hallucination essentially disappears from ReAct's failure modes (0% vs. 56% for CoT).

The paper positions this bidirectional synergy as the core innovation — not just combining reasoning and acting linearly, but creating a tight feedback loop where each enables and improves the other.

How This Paper Positions Itself

ReAct is presented not as a new model architecture or training procedure, but as a prompting paradigm — a way of structuring few-shot examples and model outputs that elicits interleaved reasoning and acting from a frozen language model. This is significant because it means the approach requires no gradient updates, no task-specific fine-tuning (in its primary form), and no architectural modifications. The entire method consists of carefully designed in-context examples that demonstrate the thought-action-observation format.

The paper explicitly contrasts this with approaches that require expensive training:

"ReAct learns a policy in a much cheaper way, since the decision making process only requires language description of the reasoning procedure." (Section 5)

This positions ReAct within the broader movement toward few-shot prompting as a general-purpose interface for LLMs, alongside work like chain-of-thought and self-consistency. However, the paper goes beyond prior prompting work by showing that the same paradigm works across fundamentally different task types — knowledge-intensive reasoning (HotpotQA, Fever) where the environment is a Wikipedia API, and interactive decision-making (ALFWorld, WebShop) where the environment is a simulated household or shopping website. This generality is central to the paper's claim: ReAct is not a task-specific trick but a general principle for structuring LLM behavior.

The paper also carefully positions itself relative to Inner Monologue (Huang et al., 2022b), which it identifies as the closest prior work. The key distinction is that Inner Monologue's "thoughts" are primarily reactions to external feedback (observations of what just happened and what subgoal remains), while ReAct's thoughts include internally generated reasoning — commonsense inferences about where objects are likely to be found, decomposition of high-level goals, tracking of completed subgoals, and reformulation of search queries. The ReAct-IM ablation in Table 3 (71% vs. 53% overall success rate on ALFWorld) quantifies this difference, showing that sparse, flexible reasoning substantially outperforms dense, feedback-only thoughts.

Finally, the paper acknowledges that prompting alone has limitations — particularly on complex tasks where the required behavior exceeds what can be conveyed in a few examples. The fine-tuning experiments on HotpotQA (Figure 3) show that ReAct benefits substantially from additional training data, with PaLM-8B fine-tuned on 3,000 ReAct trajectories outperforming PaLM-540B with prompting alone. This positions ReAct as a scalable paradigm that can start with few-shot prompting and improve with additional data, rather than a prompting-only technique.

3. Technical Approach

3.1 Reader Orientation

The "system" here is not a trained model but a prompting protocol — a carefully designed format for few-shot examples that teaches a frozen large language model to generate interleaved chains of verbal reasoning ("thoughts") and environment actions, where the model itself decides what to think and when to act. It solves the problem that pure reasoning (chain-of-thought) hallucinates facts because it cannot check its knowledge against the world, while pure acting (action generation) fails to plan, track subgoals, or recover from errors because it lacks a working memory — and the shape of the solution is remarkably straightforward: you write down a few human demonstrations showing how someone would think out loud while performing the task, and the model generalizes to new instances because the format forces it to externalize its reasoning into interpretable steps while using actions to gather the information those steps need.

3.2 Big-Picture Architecture (Diagram in Words)

The ReAct system has four major components that interact in a simple closed loop:

1. The Frozen Language Model (PaLM-540B, or GPT-3) — the central engine that generates all thoughts and actions. It receives a growing context that accumulates observations from the environment and its own previous thoughts and actions, and it auto-regressively produces the next thought or action token-by-token.

2. The Environment — an external system that the model interacts with through a text-based action space. Depending on the task, this is a Wikipedia API (for HotpotQA and Fever), a simulated household with text observations (ALFWorld), or a shopping website simulator (WebShop). The environment receives an action string from the model, executes it, and returns an observation string that gets appended to the context.

3. The Prompt (Few-Shot Demonstrations) — a fixed set of human-written trajectories that demonstrate the ReAct format. Each trajectory consists of interleaved Thought, Action, and Observation steps that show the model what kinds of reasoning to generate, how to format actions, and how to interpret observations. The prompt is prepended to the task instance at inference time and remains constant across all test examples.

4. The Output Parser (Implicit) — there is no separate parsing module; the model generates text in a structured format (keywords like Thought, Action, Observation, Finish) that the system can mechanically interpret to extract actions for environment execution and detect when the episode is complete.

Information flows as follows: a task instruction enters the context → the model generates a Thought (or directly an Action) → the Action is extracted and sent to the environment → the environment returns an Observation appended to the context → the model generates the next Thought or Action based on the updated context → the cycle repeats until the model generates Finish[answer] or a step budget is exceeded.

For knowledge-intensive tasks (HotpotQA, Fever), thoughts appear at every step (dense reasoning). For decision-making tasks with many actions (ALFWorld, WebShop), thoughts appear sparsely at key decision points (sparse reasoning), and the model itself decides when to interject a thought versus directly take an action.

3.3 Roadmap for the Deep Dive

• First, the formal ReAct framework (augmented action space, thought semantics), because it defines the abstraction that unifies all tasks — understanding what a "thought" is and how it differs from an "action" is prerequisite to everything else.
• Second, the prompt design methodology across tasks, because the prompt is the entire "system" — there is no model training, so understanding how demonstrations are constructed, what types of thoughts are included, and how the format varies between knowledge-intensive and decision-making tasks is essential to understanding what makes ReAct work.
• Third, the action spaces and environment interfaces for each domain, because ReAct's generality depends on defining a small, text-based action vocabulary that connects the language model to external tools — the specific design of these interfaces (Wikipedia search/lookup/finish, ALFWorld navigation/manipulation, WebShop search/click/buy) shapes what the model can do and what reasoning it needs to generate.
• Fourth, the combination strategies with chain-of-thought, because the paper's best results on QA and fact verification come from hybrid approaches (ReAct → CoT-SC and CoT-SC → ReAct) that switch between internal reasoning and externally-grounded reasoning — understanding the heuristics that trigger these switches explains when and why each approach is valuable.
• Fifth, the fine-tuning protocol, because although ReAct is primarily a prompting method, the paper shows that bootstrapping on ReAct-generated trajectories and fine-tuning smaller models can surpass prompting performance — the data generation and training details reveal how the paradigm scales with additional compute.
• Sixth, design decisions and their justifications, synthesizing the explicit and implicit choices the authors made: why sparse thoughts for ALFWorld but dense thoughts for HotpotQA, why the specific Wikipedia API actions, why 6 exemplars for some tasks and 1–3 for others, and what alternatives were considered or motivated by prior work.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a methodological paper whose core idea is that interleaving reasoning traces and environment actions in a language model's output context enables a bidirectional synergy — reasoning guides action selection and recovery, while actions ground reasoning in retrieved facts — and that this synergy can be elicited from a frozen LLM through carefully designed few-shot prompts.

---

The Formal ReAct Framework: Augmented Action Space

The paper formalizes ReAct as an extension of the standard agent-environment loop. Consider an agent that, at time step $t$, receives an observation $o_t \in \mathcal{O}$ from the environment and selects an action $a_t \in \mathcal{A}$ according to some policy $\pi(a_t \mid c_t)$, where $c_t = (o_1, a_1, ..., o_{t-1}, a_{t-1}, o_t)$ is the interaction history up to step $t$.

The core difficulty the paper identifies is that the mapping $c_t \mapsto a_t$ can be highly implicit and require extensive computation. For instance, in the HotpotQA example from Figure 1(1c), the act-only agent has gathered three observations about the Apple Remote and Front Row, but it cannot synthesize the correct final answer from this context alone — it would need to reason about what it knows and what it still needs to find out.

ReAct's solution is to augment the action space with a language space:

$\hat{\mathcal{A}} = \mathcal{A} \cup \mathcal{L}$

where $\mathcal{A}$ is the original set of environment actions (e.g., search[entity], go to cabinet 1, click[Buy Now]) and $\mathcal{L}$ is the space of natural language strings, representing verbal reasoning traces that the paper calls thoughts.

What it computes: an action $\hat{a}_t \in \mathcal{L}$ is a thought — a string of free-form natural language that does not affect the external environment and therefore produces no observation feedback. Instead, a thought operates purely on the agent's internal context: it composes useful information by reasoning over $c_t$, and the resulting string $\hat{a}_t$ is appended to the context to produce $c_{t+1} = (c_t, \hat{a}_t)$. This updated context then supports future reasoning or acting. In contrast, an action $\hat{a}_t \in \mathcal{A}$ is sent to the environment, which returns an observation $o_{t+1}$, and the context becomes $c_{t+1} = (c_t, a_t, o_{t+1})$.

Why this form: the key property is that thoughts are zero-cost in the environment — they don't consume resources, move the agent, or retrieve information, but they restructure the model's internal computation by making intermediate reasoning visible in the autoregressive context window. This matters because an LLM's computation is bounded by its forward pass: each token prediction is a fixed-computation operation, and complex multi-step reasoning may require more sequential computation than a single forward pass can provide. By externalizing reasoning as tokens in the context, the model effectively amortizes reasoning across multiple forward passes, where each pass can build on the reasoning output of previous passes. The alternative — generating an action directly from raw observations without intermediate reasoning — forces the model to perform all necessary computation in a single forward pass, which the paper shows leads to failures like repeatedly trying to take a non-existent object (Figure 1(2a)) or failing to synthesize retrieved facts into an answer (Figure 1(1c)).

A crucial practical detail: the model itself decides when to generate a thought versus an action. There is no meta-controller or scheduler. The few-shot examples implicitly teach the model a policy over the augmented action space $\hat{\mathcal{A}}$ — when the model sees contexts similar to those where demonstrators interjected thoughts, it learns to do the same. This autonomy is important because it allows the thought-action pattern to adapt to task demands: on HotpotQA, the model learns to produce a thought before every action (dense reasoning), while on ALFWorld, it learns to produce thoughts only at key junctures — after receiving a new observation that requires re-planning, or before embarking on a new subgoal (sparse reasoning).

Types and Functions of Thoughts

The paper does not formalize thought types in a taxonomy, but the examples and prompts reveal several distinct functions that thoughts serve. Understanding these functions is important because they represent the operational content of "reasoning" in ReAct — the concrete cognitive operations that get externalized as language.

Goal decomposition. The model breaks a high-level task instruction into a sequence of subgoals. For example, in the HotpotQA prompt (Appendix C.1), Thought 1 for the elevation range question states: "I need to search Colorado orogeny, find the area that the eastern sector of the Colorado orogeny extends into, then find the elevation range of the area." This is a three-step plan extracted from the question structure. In ALFWorld (Table 8), the first thought for "put a clean lettuce in diningtable" is: "To solve the task, I need to find and take a lettuce, then clean it with sinkbasin, then put it in diningtable." This high-level decomposition is crucial because it provides a roadmap that subsequent thoughts can track against — when the model later thinks "Now I find a lettuce (1). Next, I need to take it," it is referencing and advancing through this initial plan.

Information extraction from observations. When the Wikipedia API returns a paragraph of text, the model generates a thought that extracts the key fact relevant to the current subgoal. In Figure 1(1d), Thought 2 reads: "Apple Remote was originally designed to control the Front Row media center program. I need to search Front Row next and find what other device can control it." The first sentence is extraction (identifying the relevant entity from the observation), and the second is planning (determining the next action based on what was extracted). This extraction is lossy and directional — the model doesn't just parrot the observation but actively selects and rephrases the parts that advance the task.

Commonsense reasoning. In ALFWorld, the model uses its pretrained world knowledge to guess where household objects are likely to be found. Table 8 shows: "A lettuce is more likely to appear in fridge (1), diningtable (1), sinkbasin (1), stoveburner (1-3), cabinet (1-13). I can check one by one, starting with fridge 1." This reasoning is not in any observation — it comes entirely from the model's internal knowledge about typical household object locations. The model lists candidate locations and prioritizes them (fridge first for a lettuce), then acts accordingly. Without this thought, an act-only agent would have to search blindly.

Arithmetic and comparison reasoning. In the HotpotQA prompt (Appendix C.1), the magazine comparison question demonstrates arithmetic reasoning: "Arthur's Magazine was started in 1844. I need to search First for Women next. ... 1844 (Arthur's Magazine) < 1989 (First for Women), so Arthur's Magazine was started first." The thought explicitly performs the temporal comparison in language, making the reasoning transparent and verifiable.

Search reformulation and error recovery. When a search action returns no results or irrelevant results, the model can generate a thought that reformulates the query. In Figure 1(1d), the search for "Front Row" fails, and Thought 3 is: "Front Row is not found. I need to search Front Row (software)." This is error recovery — the model observed a failure, reasoned about why it might have occurred (the search term was ambiguous), and generated a more specific query. In the CoT and Act baselines, this kind of dynamic reformulation is impossible — CoT never gets feedback that its facts are wrong, and Act has no mechanism to reason about why an action failed.

Subgoal tracking and completion detection. In ALFWorld, the model tracks its progress through the decomposition plan. Table 8 shows a sequence: "Now I find a lettuce (1). Next, I need to take it." → [take action] → "Now I take a lettuce (1). Next, I need to go to sinkbasin (1) and clean it." → [go, clean] → "Now I clean a lettuce (1). Next, I need to put it in/on diningtable 1." Each thought acknowledges the completion of the previous subgoal (marked by "Now I find," "Now I take," "Now I clean") and transitions to the next subgoal. This creates a monotonic progression through the plan that prevents the model from getting stuck in loops — it always knows what it has done and what remains.

Final answer synthesis. The last thought in a trajectory typically synthesizes all gathered information into a final answer, often with an explicit connective like "so the answer is...". This synthesis step is critical because it forces the model to explicitly connect its reasoning chain to its conclusion, making the trajectory interpretable and enabling humans to verify that the conclusion follows from the premises.

Prompt Design Methodology

The prompt is the entire "learning algorithm" in ReAct — there is no gradient update, no fine-tuning (in the primary experiments), and no architectural change. The model's behavior is entirely determined by the few-shot examples prepended to its context. The paper's prompt design methodology is therefore central to understanding how ReAct works and why it generalizes.

Principles. The paper states that prompt design is "intuitive and easy to design" because "human annotators just type down their thoughts in language on top of their actions taken." No ad-hoc format choice, thought design, or example selection is used. This simplicity is itself a methodological claim: ReAct does not require careful prompt engineering or task-specific templates; it leverages the fact that humans naturally produce interleaved reasoning and action when solving problems.

HotpotQA prompts (6 exemplars). The authors randomly select 6 questions from the HotpotQA training set and manually compose ReAct-format trajectories (Appendix C.1). Each trajectory contains multiple thought-action-observation steps with dense thoughts — a thought before nearly every action. The thoughts cover the full range of types: decomposition, extraction, commonsense, arithmetic, reformulation, and synthesis. The specific questions chosen cover different reasoning patterns: retrieving a numeric range, finding a named entity, resolving a comparison, finding a common profession, comparing dates, and determining if two people share a profession.

An important design choice is that the prompts are multi-step and interactive: each demonstration shows the model successfully navigating the Wikipedia API, including cases where initial searches fail and require reformulation (e.g., searching "Adam Clayton Powell" returns no results; the correct action is to search the suggested "Adam Clayton Powell (film)"). This teaches the model that search failures are expected and recoverable, not terminal.

Fever prompts (3 exemplars). Three exemplars are used, each illustrating the three possible Fever verdicts: SUPPORTS, REFUTES, and NOT ENOUGH INFO (Appendix C.2). The thought patterns mirror HotpotQA but are adapted for verification: thoughts guide targeted retrieval of evidence for/against the claim, and the final thought explicitly evaluates whether the retrieved evidence supports, refutes, or is insufficient for the claim. The NOT ENOUGH INFO example is particularly instructive — it shows the model retrieving partial evidence (the song peaked at #2, but the year is not confirmed) and correctly concluding it cannot verify the claim, rather than guessing.

ALFWorld prompts (2 exemplars per task type, 6 tasks). ALFWorld has 6 task types (Pick, Clean, Heat, Cool, Look, Pick 2), and the authors randomly annotate 3 trajectories from the training set for each task type (Appendix C.4, Table 8). For robustness, they construct 6 different prompts per task by taking each permutation of 2 trajectories from the 3 annotated ones, which allows measuring sensitivity to prompt selection.

The critical design difference from HotpotQA/Fever is that thoughts are sparse: they do not appear at every step. In a typical ALFWorld trajectory, the model might take 5–10 navigation actions between thoughts. The thoughts occur at key decision points: (1) initial goal decomposition, (2) reasoning about where objects are likely to be found (commonsense), (3) confirming subgoal completion and transitioning to the next subgoal, and (4) deciding where to go next when a search fails.

The sparsity is intentional and task-driven: ALFWorld tasks can require 50+ actions, and adding a thought at every step would (a) make trajectories impractically long, potentially exceeding context length limits, and (b) provide no benefit at routine steps where the next action is obvious (e.g., "go to cabinet 1", "go to cabinet 2" during systematic search). The model learns from the prompt that thoughts should appear when reasoning is needed, not when actions are mechanical.

WebShop prompt (1 exemplar). Only a single exemplar is used (Table 6), demonstrating the search → think → click → think → choose options → buy flow. The thought in this prompt bridges the gap between noisy product descriptions and user instructions: after seeing a search result page, the model thinks about which product matches the user's constraints ("For 3 ounce bottle of bright citrus deodorant for sensitive skin, the item has options 'bright citrus' and '3 ounce (pack of 1)' and seems good to buy"). This teaches the model to actively compare product attributes against the instruction before clicking, rather than clicking blindly.

Act and CoT baselines from the same trajectories. To ensure fair comparisons, the paper constructs all baseline prompts by systematically ablating ReAct trajectories (Section 3.2):

• Standard: removes all thoughts, actions, and observations, leaving only the question-answer pairs — this is the simplest possible few-shot prompt.
• CoT (Reason Only): removes all actions and observations, keeping only thought-answer pairs — this isolates reasoning without any external grounding.
• Act (Act Only): removes all thoughts from ReAct trajectories, keeping only action-observation sequences — this isolates environment interaction without any reasoning traces.
• ReAct-IM: a special ablation for ALFWorld where thoughts are replaced with dense, feedback-only "inner monologue" style thoughts (Appendix C.4, Table 9). These thoughts are limited to: (1) goal decomposition at the start, and (2) restating the current subgoal at every step. They lack commonsense reasoning about object locations, subgoal completion detection, and next-subgoal determination.

This ablation approach is methodologically strong because it controls for the content of the demonstrations — all baselines see the same task instances solved by the same expert; only the format and presence of thoughts differ.

Action Spaces and Environment Interfaces

ReAct's generality depends on defining text-based action spaces that connect the language model to external tools. The paper designs three distinct interfaces, each with deliberate properties that shape the reasoning required.

Wikipedia API for knowledge-intensive tasks (HotpotQA, Fever). The action space consists of exactly three actions (Section 3.1):

1. search[entity]: returns the first 5 sentences from the Wikipedia page corresponding to entity if an exact match exists. If no exact match exists, returns the top 5 similar entities from the Wikipedia search engine. This is a deliberately weak retriever — it only returns a small snippet of text and requires exact entity name matching. The purpose, as the paper states, is "to simulate how humans would interact with Wikipedia, and force models to retrieve via explicit reasoning in language." A stronger retriever (e.g., dense passage retrieval) would reduce the need for reasoning about what to search and how to reformulate queries.

2. lookup[string]: returns the next sentence in the current page containing the query string string, simulating the browser Ctrl+F functionality. This action requires the model to know what specific string to search for within a page — it can't just retrieve the whole article. For example, in the HotpotQA prompt, after retrieving the Milhouse page, the model uses lookup[named after] to find the specific sentence explaining the name's origin, rather than reading the entire article.

3. finish[answer]: terminates the episode and returns answer as the final output. This is the only way to end an episode — the model must explicitly decide when it has gathered enough information.

The API is stateless except for the "current page" tracked implicitly by the lookup action (lookup searches within the last searched entity). This simplicity is intentional: it means the model's entire interaction history is visible in the context, with no hidden state.

A critical property of this action space is that successful information retrieval requires explicit reasoning. The model cannot simply issue a single search and get all the information it needs — it must decompose the question, search for entities step-by-step, extract key facts from observations, reformulate queries when searches fail, and synthesize information across multiple retrievals. This is the mechanism by which ReAct forces reasoning and acting to be synergistic: the actions are too weak to solve tasks alone, and the reasoning is too ungrounded to be reliable alone, so the system only works when both are combined.

ALFWorld action space. The environment is a text-based game where the agent navigates a simulated household and manipulates objects. The action space is defined by the ALFWorld environment (Shridhar et al., 2020b) and includes actions like go to {location}, take {object} from {location}, open {receptacle}, clean {object} with {sinkbasin}, heat {object} with {microwave}, put {object} in/on {receptacle}. The observation space is text descriptions of the agent's current location and visible objects.

The key challenge in ALFWorld is that the environment is large and partially observable: a typical task instance has more than 50 locations, and the agent only sees its current location. Finding a specific object (e.g., a knife) requires systematic exploration, and the model must use commonsense reasoning to prioritize likely locations over exhaustive search. Additionally, tasks are multi-step: "put a clean knife in countertop" requires finding a knife, taking it, going to a sinkbasin, cleaning it, going to a countertop, and putting it down. The model must track which subgoals it has completed and what remains.

WebShop action space. The environment is a simulated e-commerce website with 1.18M real-world products from Amazon (Yao et al., 2022). The action space includes: search[query] to search for products, click[product_id] to view a product page, click[option] to select product attributes (color, size, etc.), and click[Buy Now] to purchase. Observations include search result pages (product titles, prices, IDs) and product detail pages (descriptions, options, features).

The WebShop action space differs from Wikipedia and ALFWorld in that it contains noisy, unstructured text from real product listings. The model must bridge the gap between user instructions (e.g., "I am looking for a nightstand with drawers. It should have a nickel finish, and priced lower than $140") and the actual product descriptions, which may use different terminology or have irrelevant details. The thought-action loop helps here by allowing the model to explicitly compare product attributes against the instruction before acting: "For 'space-saving ottoman bench for living room', the item has options '39x18x18inch' and 'blue' and seems good to buy."

The Thought-Action Occurrence Pattern: Dense vs. Sparse

The paper makes an important design choice that varies across task types: when thoughts should appear relative to actions.

Dense reasoning (HotpotQA, Fever). Every action is preceded by a thought, creating a strict thought-action-observation-thought-action-observation pattern. This is possible because the total number of actions is small (typically 3–7 per question) and each action is semantically rich — a single search or lookup retrieves substantial information that warrants explicit reasoning. The dense pattern ensures that every action is grounded in reasoning, and every piece of retrieved information is explicitly processed before the next action.

Sparse reasoning (ALFWorld, WebShop). Thoughts appear only at key decision points, not before every action. The paper states: "For decision making tasks that potentially involve a large number of actions, thoughts only need to appear sparsely in the most relevant positions of a trajectory, so we let the language model decide the asynchronous occurrence of thoughts and actions for itself." This is both a practical necessity (50+ actions with thoughts at each step would exceed context limits) and a reflection of task structure — many actions in ALFWorld are mechanical (e.g., walking from cabinet 1 to cabinet 2 during systematic search) and don't require explicit reasoning. The model learns from the few-shot examples when to interject thoughts: after receiving a new observation that changes the situation, when deciding on the next subgoal, or when an action fails and requires replanning.

The paper demonstrates that the model successfully learns this sparse pattern. In the ALFWorld ReAct trajectories (Appendix D.2.1), thoughts appear roughly every 5–10 actions, always at semantically meaningful junctures: finding an object, completing a subgoal, or needing to replan. The model never generates thoughts at every step, showing that it has internalized the sparse pattern from the prompt.

Combination Strategies: ReAct + Chain-of-Thought

The paper observes that ReAct and chain-of-thought reasoning have complementary strengths and weaknesses. ReAct is more factual and grounded (low hallucination) but sometimes fails due to reasoning errors or uninformative search results (Table 2: 47% reasoning error for ReAct vs. 16% for CoT). CoT is better at formulating coherent reasoning structures but hallucinates facts (56% of CoT failures are hallucinations). This suggests that combining the two could yield benefits beyond either alone.

The paper proposes two combination heuristics (Section 3.2):

ReAct → CoT-SC. When ReAct fails to produce an answer within a maximum number of steps (7 for HotpotQA, 5 for Fever), the system backs off to CoT with self-consistency. The step limits are chosen because "more steps will not improve ReAct performance" — specifically, trajectories with correct final answers that take more than 7 steps on HotpotQA account for only 0.84% of all correct trajectories, and more than 5 steps on Fever account for 1.33%. So exceeding these limits is a strong signal that ReAct is stuck (likely due to uninformative search results or the repetitive generation loop described in Section 3.3), and CoT-SC's internal knowledge provides a fallback.

CoT-SC → ReAct. When the majority answer among $n$ CoT-SC samples occurs less than $n/2$ times — meaning the model's internal knowledge is not confidently aligned on an answer — the system backs off to ReAct. The intuition is that low self-consistency signals factual uncertainty; in such cases, retrieving external information via ReAct is more likely to help than sampling more CoT chains.

These heuristics are simple but effective. Figure 2 shows that both ReAct → CoT-SC and CoT-SC → ReAct consistently outperform pure CoT-SC across different numbers of samples, and match the performance of CoT-SC with 21 samples using only 3–5 samples. This is a concrete demonstration of the synergy: ReAct's external knowledge and CoT's internal knowledge are not redundant but complementary, and simple heuristics can effectively arbitrate between them.

On HotpotQA, ReAct → CoT-SC performs best (35.1 EM vs. 34.2 for the reverse direction), suggesting that for QA, it's better to try grounded retrieval first and fall back to internal knowledge only when retrieval fails. On Fever, CoT-SC → ReAct performs best (64.6 accuracy vs. 62.0 for the reverse direction), suggesting that for fact verification, internal knowledge is a strong first pass (many claims can be verified from parametric knowledge), and external retrieval is most valuable when internal knowledge is uncertain.

Fine-Tuning Protocol

While ReAct is primarily a prompting method, the paper includes fine-tuning experiments on HotpotQA to demonstrate that the paradigm scales with additional data. The fine-tuning protocol is a bootstrapping approach similar to STaR (Zelikman et al., 2022).

Data generation. Three thousand trajectories with correct final answers are generated by prompting PaLM-540B with ReAct (and separately for the Standard, CoT, and Act baselines). These trajectories include all thoughts, actions, and observations — the full interaction history. Only trajectories where the final answer matches the ground-truth label are kept.

Training objective. The smaller models (PaLM-8B and PaLM-62B) are fine-tuned via standard supervised learning (next-token prediction) to generate the entire trajectory (thoughts, actions, observations) conditioned on the input question or claim. This is effectively behavioral cloning on successful ReAct trajectories: the model learns to imitate the full reasoning-and-acting process that led to correct answers.

Training details. Batch size 64 is used for all fine-tuning. PaLM-8B ReAct and Act models are fine-tuned for 4,000 steps; PaLM-8B Standard and CoT models for 2,000 steps. PaLM-62B ReAct and Act models for 4,000 steps; PaLM-62B Standard and CoT for 1,000 steps. The paper notes that "ReAct and Act methods generally benefit from more training steps (and more training data), while Standard and CoT methods degrade soon after fine-tuning" — an important observation about the learning dynamics: teaching a model to memorize facts (Standard, CoT) leads to overfitting, while teaching it to interact with an environment to retrieve facts (Act, ReAct) is a more generalizable skill that benefits from additional data.

Key result (Figure 3). PaLM-8B fine-tuned on ReAct trajectories (25.1 EM) outperforms PaLM-62B with prompting alone (approximately 24 EM), and PaLM-62B fine-tuned on ReAct (approximately 31 EM) outperforms PaLM-540B with prompting alone (27.4 EM). This demonstrates that ReAct's capabilities can be distilled into smaller models with sufficient training data, and that the skill of "reasoning + acting to retrieve information" generalizes better than memorizing facts.

Design Decisions and Their Justifications

Why frozen LLMs rather than training? The paper positions ReAct as a prompting paradigm primarily because it enables strong performance without any task-specific training data, which is important for domains where expert demonstrations are scarce. The few-shot results on ALFWorld (71% success with 2-shot prompting vs. 37% for BUTLER trained on $10^5$ demonstrations) particularly demonstrate this advantage. Additionally, the prompting approach enables human-in-the-loop interaction (Figure 5), where a human can edit a single thought in the trajectory and change the model's subsequent behavior — something impossible with a trained model. The fine-tuning experiments (Figure 3) are presented as a complementary approach showing that the paradigm can be further improved when training data is available, not as the primary method.

Why the specific Wikipedia API design? The API is deliberately weak — it only returns the first 5 sentences of a page and requires exact name matching. The paper states this is "to simulate how humans would interact with Wikipedia, and force models to retrieve via explicit reasoning in language." If the API were a powerful neural retriever that returned perfectly relevant passages for any query, the model wouldn't need to reason about what to search, how to reformulate queries, or how to chain multiple retrievals. The weak API creates the conditions under which reasoning and acting are necessarily synergistic: acting alone is insufficient, and reasoning alone is ungrounded.

Why sparse thoughts for decision-making tasks? ALFWorld tasks can require 50+ actions, and WebShop tasks involve multiple pages of product listings. Adding thoughts at every step would make trajectories impractically long — the PaLM-540B context window has a finite length, and excessively long contexts degrade performance and increase latency. More importantly, many actions in these environments are mechanical (e.g., walking to adjacent locations during systematic search) and don't benefit from reasoning. The sparse pattern concentrates thoughts at semantically meaningful decision points, which is both more efficient and more aligned with how humans actually use verbal reasoning during task execution — we don't narrate every step; we reason at key junctures.

Why 6 exemplars for HotpotQA but only 3 for Fever and 1–2 for decision-making tasks? The paper specifies that for HotpotQA, "we randomly select 6 and 3 cases from the training set" for QA and Fever, respectively, and "we find more examples do not improve performance." For ALFWorld, 2 exemplars per task type (from 3 annotated) are used. For WebShop, only 1 exemplar. This variation reflects task complexity: multi-hop QA requires diverse reasoning patterns (numeric retrieval, entity linking, comparison, common profession, date comparison, profession matching) that are better covered by more examples, while Fever's verification task has a simpler structure (search claim entity → check evidence → verdict), and WebShop's action space is straightforward enough that one example suffices. The finding that more examples don't improve performance suggests the model reaches a ceiling on what can be learned from in-context examples alone, and further improvement requires fine-tuning (which Figure 3 confirms).

Why the ReAct-IM ablation? The Inner Monologue comparison (Table 3) is critical because it isolates the value of internally-generated reasoning as opposed to externally-triggered feedback. ReAct-IM replaces thoughts with dense, feedback-only statements (e.g., "I need to find a clean knife" repeated at every step), removing commonsense reasoning ("A knife is more likely to appear in cabinet..."), subgoal completion detection, and next-subgoal determination. The large performance gap (71% vs. 53% overall) demonstrates that these internal reasoning operations — not just any kind of text interleaved with actions — are what drive ReAct's effectiveness. This ablation also positions ReAct as conceptually distinct from Inner Monologue, which the paper characterizes as primarily reactive to environment feedback rather than proactively reasoning.

Why the specific combination heuristics (step limits, majority thresholds)? The ReAct → CoT-SC step limits (7 for HotpotQA, 5 for Fever) are empirically determined from the distribution of correct trajectory lengths — they are set at values where very few correct trajectories exceed them (0.84% and 1.33% respectively). This makes them reliable failure detectors: if ReAct hasn't produced an answer within these limits, it almost certainly won't produce a correct one. The CoT-SC → ReAct threshold (majority < n/2) is a natural uncertainty signal: when no answer gets majority support, the model's internal knowledge is unreliable, and external retrieval becomes warranted. Neither heuristic is claimed to be optimal; they are simple, interpretable, and effective baselines for demonstrating the complementary value of the two approaches.

4. Key Insights and Innovations

Innovation 1: The Bidirectional Synergy Framing — Reasoning and Acting Are Not Just Additive, They Are Mutually Enabling

The paper's most fundamental conceptual contribution is not the specific prompting format but the reframing of reasoning and acting as synergistic rather than parallel capabilities. Prior work treated these as separate research threads: chain-of-thought prompting (Wei et al., 2022) studied how models reason internally, while work on language agents (Nakano et al., 2021; Ahn et al., 2022) studied how models generate actions in environments. The implicit assumption was that reasoning and acting are independent capabilities that could be developed separately and potentially combined later. ReAct challenges this assumption by demonstrating that each capability is substantially weakened without the other, and that the combination enables behaviors neither can achieve alone.

This is a genuinely novel framing because it identifies a structural dependency, not just a performance boost. Reasoning without acting — as in chain-of-thought — is not just "reasoning that happens to lack external information"; it is reasoning that is structurally incapable of error recovery because it never receives corrective feedback. When CoT hallucinates a fact (e.g., "the Apple Remote can control iPhone, iPad, and iPod Touch" in Figure 1(1b)), that hallucination propagates forward through all subsequent reasoning steps with no mechanism for detection or correction — the model is trapped in a self-consistent but incorrect logical bubble. Acting without reasoning — as in act-only baselines — is not just "acting that happens to lack planning"; it is acting that is structurally incapable of hierarchical goal management because it has no explicit representation of what it has accomplished or what remains. The act-only agent in Figure 1(2a) repeatedly tries to take a peppershaker from a sinkbasin that doesn't contain one because it has no mechanism to reason "I've already checked this location; the object must be elsewhere."

What makes this framing distinctive is that it is bidirectional — the paper systematically demonstrates both the reason→act and act→reason directions, rather than just showing that adding actions to reasoning helps (which would be unsurprising). The error analysis in Table 2 is the crucial diagnostic: ReAct reduces CoT's hallucination failure rate from 56% to 0% (act→reason benefit), while CoT has a lower reasoning error rate than ReAct (16% vs. 47%, reason→act benefit), suggesting that the structural constraint of interleaving actions with thoughts — while reducing flexibility — also provides grounding that eliminates the worst failure mode. This bidirectional framing has conceptual implications beyond this paper: it suggests that future work should not ask "should we add reasoning to agents" or "should we add actions to reasoners" as if these were optional enhancements, but rather should treat the reasoning-acting loop as a fundamental design primitive for LLM-based systems, akin to how the perception-action loop is fundamental in robotics.

The Inner Monologue comparison (Table 3, ReAct vs. ReAct-IM: 71% vs. 53%) further sharpens this insight by showing that not all interleaved text is reasoning. Inner Monologue (Huang et al., 2022b) interleaves environment feedback with actions, creating a closed loop, but its "thoughts" are primarily restatements of observations and current subgoals — they are reactive, not generative. ReAct's thoughts include internally-generated reasoning (commonsense about object locations, subgoal completion detection, next-subgoal determination) that cannot be derived from observations alone. The performance gap quantifies the value of generative reasoning over reactive feedback: simply closing the loop is insufficient; the model must actively reason about what it knows, not just echo what it sees.

This is a fundamental shift in how the field conceptualizes LLM capabilities — moving from a toolbox model (reasoning and acting are separate tools that can be independently developed and optionally combined) to a symbiosis model (reasoning and acting are incomplete without each other, and their integration qualitatively changes what the system can do). The practical consequence — that ReAct achieves 34% absolute improvement on ALFWorld over imitation learning trained on $10^5$ examples while using only 2-shot prompting — is significant, but the conceptual consequence is larger: it establishes that for LLMs, the reasoning-acting boundary is not a clean separation but a productive interface.

Innovation 2: Thought as a First-Class Action — Externalizing Internal Computation Through Language

The paper's second conceptual innovation is the formal treatment of language as an action that operates on the agent's internal state rather than the external environment. This is not merely a prompting trick; it is a principled architectural decision with significant implications for how we understand LLM computation.

Specifically, ReAct draws an explicit distinction between two types of actions: those in $\mathcal{A}$ (the original action space, which affect the environment and produce observations) and those in $\mathcal{L}$ (the language space, which modify the agent's context but produce no environment feedback). A thought $\hat{a}_t \in \mathcal{L}$ is an action that writes to the model's working memory — the autoregressive context window — restructuring the information available for future token predictions without consuming environment resources or advancing the task state.

This framing matters because it addresses a fundamental limitation of transformer-based language models: bounded per-step computation. Each token prediction in a transformer is a fixed-computation operation (bounded by the depth and width of the network). Complex multi-step reasoning — the kind that chain-of-thought produces — may require more sequential computation than a single forward pass can perform. Prior work on chain-of-thought (Wei et al., 2022) implicitly addressed this by having the model generate reasoning tokens before the answer, effectively amortizing computation across multiple forward passes, but it framed this as "showing work" — an epiphenomenon of the reasoning process. ReAct reframes it as an architectural necessity: thoughts are the mechanism by which an LLM extends its effective computational depth beyond what a single forward pass provides, and the interleaving of thoughts with actions is the mechanism by which this extended computation is grounded in external reality.

This reframing has significant implications beyond this paper. It suggests that the context window is not just memory — it is a computational substrate. When ReAct generates a thought like "Now I find a lettuce (1). Next, I need to take it," it is not just documenting its reasoning for human readers; it is writing a state variable into memory that will be visible to all subsequent token predictions. The next time the model generates a token, it sees this explicit subgoal statement in its context and can condition on it, making it more likely to generate an appropriate next action. Without this externalized state, the model would need to maintain the subgoal tracking implicitly in its residual stream activations across potentially dozens of intervening token predictions — a much harder computational problem.

The comparison to Inner Monologue (Huang et al., 2022b) again sharpens this insight. Inner Monologue also injects text into the context, but it does so as observations (environment feedback), not as actions (agent-initiated computation). The ReAct-IM ablation (Table 3) shows that this distinction matters enormously: when thoughts are limited to restating observations and current subgoals (reactive), performance drops by 18 percentage points compared to ReAct's generative thoughts. This suggests that the value of interleaved text is not simply that it increases context length or provides more information — it's that agent-initiated language serves as a form of deliberate, goal-directed computation that reactive feedback cannot replace.

This is a fundamental conceptual contribution because it provides a theoretical language for understanding why prompting techniques like chain-of-thought work — they're not just triggering latent knowledge; they're extending the model's effective computational capacity by externalizing intermediate states into the autoregressive context. It also suggests a design principle for future LLM-based agents: the action space should always include a "think" action that writes to context but not to the environment, and the agent should be trained (or prompted) to use this action strategically to manage its own computational depth.

Innovation 3: Difficulty-Aware Hybrid Reasoning — Knowing When to Use Internal vs. External Knowledge

The paper's third conceptual contribution is the demonstration that internal (parametric) and external (retrieved) knowledge are complementary resources with different failure modes, and that simple heuristics can effectively arbitrate between them to achieve better performance than either alone. This is not just an empirical finding about combining two methods — it is a diagnostic insight about the nature of knowledge in large language models.

Prior work on retrieval-augmented generation (Lewis et al., 2020) and internet-augmented LMs (Nakano et al., 2021; Lazaridou et al., 2022) treated external knowledge retrieval as a universal improvement — the assumption was that grounding in retrieved documents always helps, and the challenge was designing better retrieval mechanisms. The ReAct + CoT-SC results challenge this assumption by showing that external retrieval is sometimes harmful and that the optimal strategy depends on the specific question. The combination heuristics — ReAct → CoT-SC (try retrieval first, fall back to internal reasoning if retrieval fails) and CoT-SC → ReAct (try internal reasoning first, fall back to retrieval if uncertain) — perform differently on different tasks (ReAct → CoT-SC is best on HotpotQA at 35.1 EM; CoT-SC → ReAct is best on Fever at 64.6 accuracy), demonstrating that there is no universal ordering of internal vs. external knowledge.

The deeper insight comes from the error analysis in Table 2, which reveals qualitatively different failure modes for the two approaches. CoT's failures are dominated by hallucination (56%) — the model confidently generates plausible-sounding but factually incorrect reasoning because it can't verify its claims. ReAct's failures are dominated by reasoning errors (47%) and uninformative search results (23%) — the model gets stuck in loops or retrieves irrelevant information that derails its reasoning. These failure modes are not just different in magnitude but in kind: CoT fails because it's too confident in incorrect knowledge; ReAct fails because it's insufficiently robust to noisy retrieval. This means that the two approaches are not just complementary in the sense of "each gets some questions right that the other misses" — they are complementary in the sense that each is strong precisely where the other is weak.

Figure 2 makes this complementarity quantitative: both ReAct + CoT-SC combinations consistently outperform pure CoT-SC across all sample sizes, and they match the performance of CoT-SC with 21 samples using only 3–5 samples. This is a significant efficiency gain that comes not from improving either method individually but from recognizing that different questions require different knowledge sources and designing a simple arbitration mechanism.

This insight has implications beyond the specific heuristics used in this paper. It suggests that knowledge source selection is itself a reasoning task that LLMs may be able to perform — rather than always retrieving or always relying on internal knowledge, future systems could learn to estimate their own uncertainty about a question and decide whether to retrieve, reason internally, or combine both. The paper's finding that CoT-SC's majority vote strength serves as a useful uncertainty signal (when no answer gets majority support, retrieval becomes more valuable) is a concrete example of this principle. More broadly, this innovation reframes the retrieval-augmented generation problem from "how do we retrieve better documents" to "when should we retrieve at all, and how do we combine retrieved information with what we already know" — a shift from a retrieval-centric to a decision-centric view of knowledge grounding.

This is a significant refinement of the retrieval-augmented generation paradigm rather than a fundamental break from it. The idea that internal and external knowledge can be combined is not new (Lewis et al., 2020), but the systematic characterization of their complementary failure modes and the demonstration that simple arbitration heuristics yield substantial gains are novel contributions that change how practitioners should think about deploying retrieval-augmented systems.

Innovation 4: Prompting as a Sufficient Interface for Generalist Agents — Frozen LLMs Can Reason, Act, and Recover Without Training

The paper's fourth conceptual contribution is the demonstration that a single frozen language model, through prompting alone, can serve as a general-purpose agent across fundamentally different task types — knowledge-intensive reasoning (HotpotQA, Fever) and interactive decision-making in distinct environments (ALFWorld, WebShop) — with no task-specific architecture, no gradient updates, and minimal task-specific prompt engineering. This challenges the prevailing assumption in the prior literature that effective agency requires either extensive task-specific training (BUTLER: $10^5$ demonstrations; WebShop IL: 1,012 human trajectories) or domain-specific architectural components (SayCan's affordance model, WebGPT's RL training with human feedback).

The significance of this finding is not simply that prompting "works" — few-shot prompting was already well-established (Brown et al., 2020) — but that it works for tasks requiring closed-loop interaction with environments over extended horizons. Chain-of-thought prompting (Wei et al., 2022) had shown that LLMs can reason step-by-step, but CoT is an open-loop process: the model generates a reasoning chain and an answer in a single forward pass, with no interaction and no error recovery. ReAct extends this to a closed-loop setting where the model must (a) decide when to reason vs. act, (b) interpret environment observations that may be noisy or uninformative, (c) recover from failed actions by reformulating queries or changing strategies, and (d) maintain coherent behavior over trajectories that can span 50+ steps in ALFWorld.

What makes this a conceptual contribution rather than just an impressive empirical result is that it demonstrates emergent closed-loop behavior from an open-loop training objective. PaLM-540B was not trained to interact with environments or to manage multi-step task trajectories; it was trained on static text corpora with a next-token prediction objective. Yet when prompted with a handful of human-written demonstrations in the ReAct format, it exhibits behaviors — goal decomposition, subgoal tracking, systematic exploration, error recovery, commonsense reasoning about object locations — that look remarkably like deliberate, strategic task-solving. This suggests that the capacity for agency is latent in large language models and can be elicited through appropriate structuring of the input-output format, without requiring specialized training.

The paper provides specific evidence for this claim through the robustness analysis on ALFWorld: across 6 different prompts (each built from a different pair of annotated trajectories), ReAct consistently outperforms Act by 33–90% relative improvement, with the worst ReAct prompt (48% success) still beating the best Act prompt (45%). This demonstrates that the benefit of interleaved reasoning is not an artifact of particular prompt design choices but a robust property of the ReAct format.

However, the paper is appropriately cautious about the limits of this approach. The fine-tuning results on HotpotQA (Figure 3) show that prompting alone is substantially weaker than fine-tuning on task-specific data (PaLM-8B fine-tuned on 3,000 ReAct trajectories outperforms PaLM-540B with prompting), and the paper explicitly notes that "complex tasks with large action spaces require more demonstrations to learn well, which unfortunately can easily go beyond the input length limit of in-context learning." This honesty about limitations strengthens rather than weakens the contribution: it frames prompting as a lower bound on what LLMs can achieve as agents, with fine-tuning and reinforcement learning as paths to further improvement.

This is a fundamental empirical finding that opened a new research direction — generalist language agents built on frozen LLMs — rather than an incremental improvement to existing agent architectures. The paper's influence on subsequent work on LLM-based agents (which has exploded since 2023) confirms this assessment: ReAct established the viability and generality of the prompting-based agent paradigm, even as later work has extended it with more sophisticated planning, memory, and tool-use mechanisms.

5. Experimental Analysis

Evaluation Methodology

• Dataset. Four benchmarks are used spanning knowledge-intensive reasoning and interactive decision-making. HotpotQA (Yang et al., 2018) is a multi-hop question answering dataset requiring reasoning over two or more Wikipedia passages; the paper uses the full validation set in a question-only setup (no supporting paragraphs provided to the model). Fever (Thorne et al., 2018) is a fact verification benchmark where each claim is labeled SUPPORTS, REFUTES, or NOT ENOUGH INFO based on Wikipedia evidence; the paper evaluates on the development set in a claim-only setup. ALFWorld (Shridhar et al., 2020b) is a synthetic text-based game with 6 task types (Pick, Clean, Heat, Cool, Look, Pick 2) where agents navigate and manipulate objects in simulated households; evaluation uses 134 unseen task instances in a task-specific setup (separate prompts and evaluation per task type). WebShop (Yao et al., 2022) is an online shopping environment with 1.18M real-world products and 12k human instructions; evaluation uses 500 test instructions with metrics of average score (percentage of desired attributes covered) and success rate (percentage of episodes where all requirements are satisfied).

• Base model(s). The primary model is PaLM-540B (Chowdhery et al., 2022), a dense decoder-only transformer accessed via API. The paper states this model is "representative of the capabilities of many contemporary LLMs" (Section 4). For fine-tuning experiments on HotpotQA, smaller variants PaLM-8B and PaLM-62B are used. Additional experiments with GPT-3 (text-davinci-002, greedy decoding) are reported in Appendix A.1 to demonstrate generality across model families, with GPT-3 outperforming PaLM-540B on both HotpotQA (30.8 vs. 29.4 EM on a 500-question subset) and ALFWorld (78.4% vs. 70.9% success rate). All prompting experiments use greedy decoding unless otherwise specified (CoT-SC uses temperature 0.7 with 21 samples).

• Metrics. HotpotQA uses Exact Match (EM) against ground-truth answers, evaluated with the official grading function from the dataset. Fever uses classification accuracy (Acc) — whether the predicted verdict matches the ground-truth SUPPORTS, REFUTES, or NOT ENOUGH INFO label. ALFWorld uses task-specific success rate (%), where a task instance is successful if the agent achieves the specified goal within the maximum allowed steps. WebShop uses two metrics: average score (percentage of desired product attributes covered by the chosen product, averaged across 500 test instructions) and success rate (percentage of episodes where the chosen product satisfies all instruction requirements). Human expert performance on WebShop is 82.1 score and 59.6% success rate, providing a performance ceiling.

• Baselines. Multiple prompting baselines are constructed by systematically ablating ReAct trajectories (Section 3.2): (a) Standard prompting — removes all thoughts, actions, and observations, leaving question-answer pairs; (b) Chain-of-thought prompting (CoT) (Wei et al., 2022) — removes actions and observations, keeping only thought-answer sequences to serve as a reasoning-only baseline; (c) CoT-SC (Wang et al., 2022a) — samples 21 CoT trajectories at temperature 0.7 and takes the majority answer, providing a self-consistency baseline; (d) Act-only prompting (Act) — removes all thoughts from ReAct trajectories, keeping only action-observation sequences to serve as an acting-only baseline. For ALFWorld, additional baselines include: BUTLER (Shridhar et al., 2020b), an imitation learning agent trained on 10^5 expert trajectories per task type, and ReAct-IM, an ablation where ReAct-style thoughts are replaced with dense, feedback-only thoughts modeled after Inner Monologue (Huang et al., 2022b). For WebShop, baselines include Imitation Learning (IL) trained on 1,012 human-annotated trajectories and IL + RL trained with an additional 10,587 training instructions (both from Yao et al., 2022).

• Generation budget / compute accounting. The paper does not use a unified compute metric across tasks. Instead, fairness is maintained within each task by comparing methods that use the same base model and the same number of few-shot exemplars, differing only in format (presence/absence of thoughts, actions, observations). For HotpotQA and Fever, ReAct is constrained to a maximum of 7 and 5 steps respectively (Section 3.2); these limits are empirically determined — trajectories exceeding 7 steps on HotpotQA account for only 0.84% of correct ReAct trajectories, and exceeding 5 steps on Fever accounts for 1.33%, making the limits effective failure detectors without restricting successful trajectories. For CoT-SC, the budget is the number of sampled reasoning chains (varied from 1 to 21 in Figure 2). For ALFWorld, the environment imposes its own per-episode step limits; the paper reports success rates under these environment constraints with no additional compute budget control. For all methods, the decoding strategy is greedy unless otherwise noted.

• Cross-validation / statistical protocol. For ALFWorld, the paper constructs 6 different prompts per task type from each permutation of 2 annotated trajectories drawn from 3 available annotations, evaluating each prompt on all 134 test instances and reporting both average and best-of-6 performance. This provides a robustness measure against prompt selection. For HotpotQA fine-tuning experiments, models are trained on 3,000 trajectories with correct answers generated by PaLM-540B using ReAct prompting, with checkpoints selected based on validation performance. For the human error analysis in Table 2, 50 trajectories with correct and incorrect answers (judged by EM) are randomly sampled from both ReAct and CoT (200 total), and a human annotator manually classifies success and failure modes. No formal confidence intervals or statistical significance tests are reported for the main results.

Main Quantitative Results

Knowledge-Intensive Reasoning: HotpotQA and Fever

The headline results appear in Table 1. On HotpotQA, ReAct achieves 27.4 EM, which is slightly below CoT's 29.4 EM but above Act's 25.7 EM and Standard's 28.7 EM. This ordering — CoT > ReAct > Act — reveals that reasoning alone performs best on this dataset, but the gap between ReAct and CoT is relatively small (2.0 EM points). On Fever, the ordering reverses: ReAct achieves 60.9 accuracy, outperforming CoT's 56.3, Act's 58.9, and Standard's 57.1 — a 4.6 percentage point advantage over CoT. The paper attributes this difference to the nature of the tasks: Fever claims often differ by subtle factual details (e.g., whether an event occurred in a specific year or location) that require accurate, up-to-date retrieval, making external knowledge access more critical than on HotpotQA.

The best results on both datasets come from combining ReAct with CoT-SC. On HotpotQA, ReAct → CoT-SC achieves 35.1 EM — a 5.7 point improvement over the best single method (CoT at 29.4) and a 1.7 point improvement over CoT-SC alone (33.4). On Fever, CoT-SC → ReAct achieves 64.6 accuracy — a 3.7 point improvement over the best single method (ReAct at 60.9) and a 4.2 point improvement over CoT-SC alone (60.4). The direction of the hybrid matters: on HotpotQA, trying ReAct first and falling back to CoT-SC is better; on Fever, trying CoT-SC first and falling back to ReAct is better. This task-dependent asymmetry is not explored in depth but is consistent with the intuition that Fever requires more fact-checking (external knowledge is the stronger default) while HotpotQA requires more compositional reasoning (internal reasoning is the stronger default).

Figure 2 shows the scaling behavior with respect to the number of CoT-SC samples. Pure CoT-SC improves from approximately 27.5 EM at 1 sample to 33.4 EM at 21 samples on HotpotQA, and from approximately 52% accuracy to 60.4% on Fever. Both hybrid methods (ReAct → CoT-SC and CoT-SC → ReAct) consistently outperform pure CoT-SC across all sample sizes on both datasets. Notably, the hybrid methods achieve CoT-SC's 21-sample performance with only 3–5 samples — on HotpotQA, ReAct → CoT-SC with 3 samples reaches approximately 33 EM, matching CoT-SC with 21 samples. This demonstrates a substantial efficiency gain: the hybrid approaches extract more value per CoT sample by using ReAct's grounded information to either initialize or supplement the sampling process.

The error analysis in Table 2 provides the mechanistic explanation for these results. Of 50 randomly sampled ReAct trajectories on HotpotQA judged correct by EM, 94% had correct reasoning traces and facts (true positives), while only 6% had hallucinated reasoning or facts (false positives). In contrast, of 50 CoT trajectories judged correct, only 86% had fully correct reasoning, while 14% reached the right answer with hallucinated intermediate steps. On failure modes, ReAct's errors were dominated by reasoning errors (47%, including failure to recover from repetitive steps) and uninformative search results (23%), with zero hallucination failures. CoT's failures were dominated by hallucination (56%), with only 16% reasoning errors. This is the central diagnostic finding: ReAct eliminates hallucination as a failure mode but introduces new failure modes (search failures, reasoning rigidity) that CoT does not suffer from, creating a complementary relationship that the hybrid methods exploit.

The fine-tuning results in Figure 3 show that ReAct scales well with additional data. With prompting alone, ReAct is the worst-performing method on PaLM-8B and PaLM-62B (the smaller models struggle to learn the combined reasoning+acting format from few-shot examples). However, when fine-tuned on 3,000 ReAct-generated trajectories, ReAct becomes the best method at both model sizes. PaLM-8B fine-tuned on ReAct achieves approximately 25 EM, outperforming PaLM-62B with prompting (all methods) and approaching PaLM-540B prompting performance (27.4 EM). PaLM-62B fine-tuned on ReAct achieves approximately 31 EM, outperforming PaLM-540B with all prompting methods. In contrast, fine-tuning Standard or CoT on the same 3,000 examples yields substantially worse performance — the paper explains this as Standard/CoT learning to memorize (potentially hallucinated) facts, while ReAct/Act learn the more generalizable skill of interacting with Wikipedia to retrieve information.

Interactive Decision Making: ALFWorld

The headline results appear in Table 3. ReAct achieves an average success rate of 71% (best of 6 prompts) and 57% (average across 6 prompts), compared to Act's 45% (best) and BUTLER's 37% (best) — a 26 percentage point absolute improvement over the best Act prompt and a 34 percentage point improvement over BUTLER. Even the worst ReAct prompt (48% average success rate across task types, corresponding to the ReAct-IM condition's lower bound) outperforms the best Act prompt (45%). The relative improvement from adding thoughts to actions ranges from 33% to 90% across the six controlled prompt trials, averaging 62%.

The per-task breakdown reveals that ReAct's advantage over Act is not uniform. On Pick tasks, ReAct achieves 92% (best) vs. Act's 88% — a smaller gap because Pick tasks require finding and moving a single object, which is relatively straightforward even without reasoning. On Clean tasks, ReAct achieves 58% vs. Act's 42% — a larger gap because Clean requires the additional steps of taking the object to a sinkbasin and cleaning it, creating more opportunities for subgoal tracking errors. On Heat tasks, ReAct achieves 96% vs. Act's 74% — the largest gap because heating requires using a microwave, which involves additional navigation and object manipulation. On Look tasks, ReAct achieves 78% vs. Act's 72%. On Pick 2 tasks, which require finding and moving two objects, both methods perform poorly, with ReAct at 41% and Act at 41% — this is the only task where ReAct does not substantially outperform Act, likely because the combinatorial complexity of finding two objects overwhelms the few-shot prompt's ability to convey effective strategies.

The ReAct-IM ablation in Table 3 provides the critical comparison between ReAct's flexible reasoning and Inner Monologue-style reactive feedback. ReAct-IM achieves 53% average success (best), compared to ReAct's 71% — an 18 percentage point gap. On a per-task basis, ReAct outperforms ReAct-IM on five of six tasks: Pick (92% vs. 62%), Heat (96% vs. 87%), Cool (86% vs. 57%), Look (78% vs. 39%), and Pick 2 (41% vs. 33%). Only on Clean does ReAct-IM approach ReAct performance (68% vs. 58%). Qualitatively (Appendix D.2.3), ReAct-IM trajectories fail because the model cannot detect when subgoals are completed ("I need to find a clean knife" repeated even after finding a knife), cannot determine what the next subgoal should be (gets stuck after placing an uncleaned knife), and cannot use commonsense to locate objects (no reasoning about likely object locations). These failures directly correspond to the thought functions that ReAct provides but ReAct-IM lacks: subgoal completion detection, next-subgoal determination, and commonsense inference.

The robustness analysis through 6 different prompt configurations (each using a different pair of annotated trajectories out of 3) demonstrates that ReAct's advantage is not an artifact of particular prompt choices. The best Act prompt (45%) is exceeded by all ReAct prompts individually, and the ReAct average across prompts (57%) is above the Act best. This consistency strengthens the claim that interleaved reasoning is the causal factor, not prompt engineering.

Interactive Decision Making: WebShop

The headline results appear in Table 4. ReAct achieves 66.6 average score and 40.0% success rate, compared to Act's 62.3 score and 30.1% success rate — a 9.9 percentage point absolute improvement in success rate. The IL baseline achieves 59.9 score and 29.1% success rate, while IL+RL achieves 62.4 score and 28.7% success rate. ReAct thus outperforms the strongest trained baseline (IL+RL) by 4.2 points in score and 11.3 percentage points in success rate, using only a single in-context example with no training. Human expert performance (82.1 score, 59.6% success rate) remains substantially higher, indicating significant room for improvement.

The key qualitative difference between ReAct and Act on WebShop (illustrated in Table 10) is that ReAct uses reasoning to actively compare product attributes against the user instruction before making decisions. In the example shown, Act searches for "sixteen pack apple cinnamon freeze dried banana chips," clicks the first result (which is strawberry banana, not apple cinnamon), and buys it immediately — achieving a score of only 0.125. ReAct searches for the same query, sees the first result is strawberry banana, explicitly reasons that it doesn't match ("B0061IVFZE is strawberry banana, not apple cinnamon"), checks another product, finds it has the correct flavor and size options, and clicks through all required attributes before buying — achieving a perfect score of 1.0. This demonstrates the act-to-reason synergy in a noisy, real-world text environment: the reasoning step bridges the gap between the instruction's semantic requirements and the noisy, varied product descriptions in the search results.

A notable finding is that one-shot Act prompting already performs on par with trained baselines (62.3 score vs. 62.4 for IL+RL). This suggests that the WebShop action space — search, click products, click options, buy — is well-aligned with the model's pretraining knowledge of web navigation, and the primary limitation is not generating valid actions but selecting the right actions to satisfy the instruction constraints. ReAct addresses this by adding explicit attribute-matching reasoning before each action.

Ablation Studies and Robustness Checks

Dense vs. sparse thoughts (Section 3 and Appendices C, D): HotpotQA and Fever use dense thoughts — a thought before every action. ALFWorld and WebShop use sparse thoughts — thoughts only at key decision points, with the model autonomously deciding when to interject reasoning. The paper does not run a controlled ablation comparing dense vs. sparse within a single task, but the qualitative evidence in Appendix D.2 is informative. In ALFWorld, thoughts appear roughly every 5–10 actions in the ReAct trajectory (D.2.1), always at semantically meaningful junctures: initial goal decomposition, object discovery, subgoal completion, and replanning. The model successfully learns this sparse pattern from the few-shot examples without generating thoughts at every step, demonstrating that it can distinguish between mechanical actions (walking between adjacent cabinets during systematic search) and decision points requiring reasoning.

ReAct-IM vs. ReAct (Table 3): This is the most important ablation in the paper, isolating the value of internally-generated reasoning from reactive environment feedback. ReAct-IM thoughts are limited to (1) initial goal decomposition and (2) restating the current subgoal, with no commonsense reasoning about object locations, no subgoal completion detection, and no next-subgoal determination. The 18 percentage point gap (71% vs. 53% overall success rate) demonstrates that these missing thought functions are not incidental — they are the primary drivers of ReAct's advantage.

Number of few-shot exemplars: On HotpotQA, the paper uses 6 exemplars; on Fever, 3; on ALFWorld, 2 per task type; on WebShop, 1. The paper states that "we find more examples do not improve performance" for HotpotQA and Fever (Section 3.2, footnote 2). This finding is not systematically ablated (no curves showing performance vs. number of exemplars are provided), but it has an important implication: the ReAct format is sufficiently informative that a small number of demonstrations saturates what can be learned through in-context learning, and further improvement requires fine-tuning (as Figure 3 confirms).

ALFWorld prompt robustness (Table 3): The 6 different prompt configurations (each using a different pair of the 3 annotated trajectories per task type) produce ReAct success rates ranging from the worst-case average across tasks (the ReAct-IM lower bound of 48% is unrelated to this — ReAct proper's worst prompt is not separately reported, but the gap between ReAct average of 57% and best of 71% indicates variance). The fact that all ReAct prompts outperform all Act prompts (best Act: 45%, best ReAct: 71%) and that the relative improvement ranges from 33% to 90% across prompts demonstrates that the reasoning benefit is robust to which specific trajectories are used as demonstrations.

Fine-tuning scaling (Figure 3): Comparing prompting vs. fine-tuning across model sizes (8B, 62B, 540B) and methods (Standard, CoT, Act, ReAct) reveals an interaction: ReAct benefits more from fine-tuning than Standard or CoT. PaLM-8B fine-tuned on ReAct (25.1 EM) outperforms PaLM-8B fine-tuned on Standard (approximately 18 EM), CoT (approximately 19 EM), or Act (approximately 22 EM). The paper explains this as Standard/CoT fine-tuning teaching models to memorize facts (which leads to overfitting and poor generalization), while ReAct/Act fine-tuning teaches models to interact with Wikipedia to retrieve facts (a more generalizable skill). Supporting this interpretation: Standard and CoT fine-tuned models "degrade soon after fine-tuning" (Appendix B.1) while ReAct and Act models benefit from more training steps.

GPT-3 vs. PaLM-540B (Appendix A.1, Table 5): ReAct prompting generalizes across model families. GPT-3 (text-davinci-002) achieves 30.8 EM on HotpotQA (500-question subset) vs. PaLM-540B's 29.4, and 78.4% success on ALFWorld vs. PaLM-540B's 70.9%. The paper attributes GPT-3's stronger performance to its instruction fine-tuning, suggesting that models optimized for following instructions may be particularly well-suited to the ReAct format.

Outdated labels and up-to-date knowledge (Appendix A.2, Figure 4): This is an unplanned but informative "ablation" of real-world dynamics. A HotpotQA question about the number of rooms in a specific hotel has a dataset label of 2,884, but the actual hotel now has 2,884+ rooms. Standard and CoT produce hallucinated answers; Act fails to retrieve the correct information despite web access; ReAct successfully retrieves the up-to-date room count from Wikipedia. This demonstrates that ReAct's action loop provides genuine access to current information, not just retrieval of training-time knowledge.

Human-in-the-loop thought editing (Appendix A.3, Figure 5): While not a traditional ablation, this experiment demonstrates a unique property of the ReAct format: a human can edit a single flawed thought in the trajectory, and the model will adjust its entire subsequent behavior accordingly. In the example, deleting a hallucinating sentence in Act 17 and adding hints in Act 23 causes the model to change its exploration strategy and succeed at the task. The paper argues this is impossible with Act (where there are no thoughts to edit) or trained models (where behavior can't be changed without retraining), positioning interpretability and controllability as practical advantages of the ReAct paradigm.

Critical Assessment

Does ReAct improve over state-of-the-art baselines by 34% and 10% absolute on ALFWorld and WebShop respectively?

The 34% claim (71% ReAct vs. 37% BUTLER on ALFWorld) and the 10% claim (40.0% ReAct vs. 28.7% IL+RL on WebShop) are supported by the reported numbers, but several qualifications are necessary. On ALFWorld, the comparison is between a 2-shot prompted PaLM-540B (ReAct, best of 6 prompts) and BUTLER, an imitation learning agent trained on 10^5 expert demonstrations per task type with a different base architecture. This is not a controlled comparison that isolates the value of interleaved reasoning — it compares a large language model against a task-specific trained agent, and the language model's pretrained knowledge of household object locations and task structures may account for much of the advantage independently of the ReAct format. The more controlled comparison is ReAct vs. Act (same model, same prompts, different format), where the improvement is 71% vs. 45% — still substantial, but 26 percentage points, not 34. The 34% figure against BUTLER overstates ReAct's specific contribution by conflating it with the base model difference. On WebShop, the comparison is between 1-shot PaLM-540B and IL+RL trained on ~10k instructions — again, the base model difference is large. The more controlled comparison is ReAct vs. Act (40.0% vs. 30.1%), a 9.9 percentage point improvement that is cleanly attributable to interleaved reasoning.

Does ReAct reduce hallucination compared to chain-of-thought?

The error analysis in Table 2 (50 randomly sampled trajectories from each of 4 conditions: ReAct correct, ReAct incorrect, CoT correct, CoT incorrect) provides supporting evidence with caveats. The finding that 0% of ReAct's failure cases involve hallucination (vs. 56% for CoT) and that 14% of CoT's "correct" answers contain hallucinated intermediate steps (vs. 6% for ReAct) is compelling. However, the sample size is small (200 total trajectories), the classification is performed by a single human annotator with no reported inter-annotator agreement, and the analysis is limited to HotpotQA. Whether these hallucination rates generalize to other tasks, other model families, or other ReAct configurations is untested. The qualitative examples in the appendices provide face validity but are selected to illustrate the paper's claims. A more rigorous analysis would include multiple annotators, a larger sample, and evaluation on Fever and decision-making tasks as well.

Does ReAct work as a general paradigm across diverse tasks?

The paper's claim to generality is partially supported — ReAct is demonstrated on four benchmarks spanning two task families (knowledge-intensive reasoning and interactive decision making) — but the evaluation has important gaps. All tasks use text-based environments with well-defined action spaces. Whether ReAct would work for tasks with continuous action spaces (robotics), visual observations (the paper is text-only), or open-ended generation (where there is no clear success/failure signal) is untested. Additionally, the performance varies substantially across tasks: ReAct outperforms CoT on Fever but slightly underperforms CoT on HotpotQA; ReAct dramatically outperforms Act on most ALFWorld tasks but ties on Pick 2; ReAct substantially outperforms Act on WebShop but still falls far short of human performance (40% vs. 60% success rate). This variability suggests that ReAct's effectiveness depends on task properties that are not fully characterized — the proportion of questions where external knowledge is helpful, the complexity of the required reasoning, the noisiness of the environment feedback, and the alignment between the action space and the model's pretraining.

Does the combination of ReAct and CoT-SC outperform either alone?

The results in Table 1 and Figure 2 support this claim — both hybrid methods outperform both pure ReAct and pure CoT-SC on both HotpotQA and Fever. However, the combination heuristics (step limits of 7/5 for ReAct → CoT-SC; majority threshold of n/2 for CoT-SC → ReAct) are simple and not optimized. The paper does not report ablations over these thresholds, so it's unclear how sensitive the results are to the specific values chosen. If the thresholds were set to different values — e.g., 5 steps instead of 7 for HotpotQA, or 2n/3 instead of n/2 for the majority threshold — the performance of the hybrid methods could change substantially. This limits the strength of the claim: the paper demonstrates that some combination works better than either method alone, but doesn't characterize whether the specific heuristics are near-optimal or just one of many possible configurations that yield improvements.

Missing experiments that would strengthen the paper

Controlled comparison against Inner Monologue on the same model. The ReAct-IM ablation is run on ALFWorld, but the paper does not compare against Inner Monologue on HotpotQA or Fever, where the distinction between reactive feedback and generative reasoning might manifest differently. On QA tasks, Inner Monologue-style thoughts (restating retrieved facts, tracking what information is still needed) might perform comparably to ReAct's more flexible reasoning, since the tasks involve less commonsense spatial reasoning and more fact extraction.

Ablation over thought density. The paper uses dense thoughts for QA and sparse thoughts for decision making but never compares dense vs. sparse within a single task. On ALFWorld, adding thoughts at every step — while computationally expensive — might improve performance by providing more frequent reasoning checkpoints, or it might harm performance by cluttering the context with redundant reasoning. On HotpotQA, removing some thoughts to test whether all thought types are necessary would help isolate which thought functions drive the performance improvement.

Ablation over the number of exemplars with a controlled curve. The paper states that more exemplars don't improve performance but doesn't provide the data. A figure showing ReAct performance with 1, 2, 3, 4, 5, 6 exemplars on HotpotQA would substantiate this claim and help characterize the sample efficiency of in-context learning for the ReAct format.

Evaluation on a model family other than PaLM or GPT-3. The GPT-3 results (Appendix A.1) provide some cross-model evidence, but both PaLM and GPT-3 are large, instruction-tuned (in GPT-3's case) models from major industrial labs. Whether ReAct works with open-source models, smaller models, or models with different pretraining objectives is untested and would be important for assessing generality.

Statistical significance testing. The paper reports no confidence intervals, no statistical tests, and no error bars on any of the main results. For the ALFWorld 6-prompt robustness analysis, the variance across prompts is reported but not formalized. For the HotpotQA and Fever results, the test set sizes are large (7,405 for HotpotQA full set; 500 for the subset; 19,998 for Fever dev; 500 for WebShop test), so the differences are likely statistically significant, but this is not verified.

Comparison against retrieval-augmented baselines. The paper compares ReAct against pure CoT and Act, but doesn't compare against a standard retrieval-augmented generation (RAG) baseline (Lewis et al., 2020) that retrieves relevant Wikipedia passages and then applies chain-of-thought reasoning. Such a baseline would help isolate the value of the interleaved reasoning-acting loop from the simpler approach of retrieving once and reasoning once — a distinction that is central to the paper's claims but not directly tested.

Conditional nature of the findings

The paper's results should be understood as applying primarily to: (1) large-scale language models (540B parameters for prompting; 8–62B for fine-tuning) with strong pretrained reasoning and commonsense capabilities; (2) tasks where the environment can be accessed through a small, text-based action space with clear observation formatting; (3) tasks requiring multi-step reasoning or exploration where intermediate reasoning provides genuine value (as opposed to single-step tasks where acting alone suffices); (4) English-language tasks with Wikipedia-scale knowledge coverage. The paper does not test ReAct on tasks requiring visual understanding, continuous control, long-horizon planning beyond what fits in a context window, or multi-agent coordination. These are not weaknesses of the evaluation per se — the paper is explicit about its scope — but they bound the generality claims.

6. Limitations and Trade-offs

1. Difficulty Estimation Requires Prohibitively Expensive Computation

The assumption or constraint. The compute-optimal allocation framework described in Section 3.2 conditions strategy selection on prompt difficulty, which is estimated by generating 2,048 complete solutions per question, then binning questions into five quintiles based on either ground-truth pass@1 (oracle) or PRM predicted scores (predicted). This is explicitly acknowledged by the authors:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

The consequence. In any realistic deployment, the total cost is difficulty estimation plus strategy execution. Since 2,048 generations per question exceeds the largest test-time budgets studied (256–512 generations), the difficulty estimation step alone would dominate the total compute budget. The reported 4× efficiency gains over best-of-N (Figures 4 and 8) are computed after difficulty is known, without amortizing the estimation cost. In a production system, the total compute consumed (estimation + execution) could easily exceed simply running best-of-N with the full budget on every question, eliminating or even reversing the claimed efficiency advantage. The problem is particularly acute for one-off queries (where there is no amortization across multiple uses of the same question) and for latency-sensitive applications where the wall-clock time of generating 2,048 samples before solving a single query would be unacceptable.

What evidence exists in the paper. The paper explicitly flags this issue in Section 3.2 but provides no experiments measuring the overhead or demonstrating that difficulty estimation can be made practical. The predicted difficulty approach (using PRM scores instead of ground-truth labels) still requires 2,048 samples and PRM scoring per question — it removes the need for labels but not the computational cost. No alternative difficulty estimation method (e.g., a lightweight classifier trained on question text, or adaptive estimation using a small number of initial samples) is implemented or evaluated.

Mitigation status. The paper does not attempt to resolve this limitation, instead framing it as future work on "pretraining or fine-tuning models to directly predict difficulty of a question" (Section 8). The authors acknowledge it transparently but the headline efficiency claims remain uncorrected for estimation cost.

---

2. Hard Problems Remain Unsolved Regardless of Compute Budget

The assumption or constraint. The compute-optimal framework assumes that test-time compute can improve performance, but this breaks down on the hardest difficulty quintile where the base model's pass@1 is near zero. These are problems that the smaller model fundamentally cannot solve — no correct solutions exist in the proposal distribution to find or refine — and test-time compute provides essentially no benefit regardless of strategy or budget.

The consequence. The approach offers no path forward for problems outside the base model's capability range. For such problems, pretraining remains the only viable path. This is a fundamental ceiling, not a limitation that can be engineered around with better strategies or more compute. In deployments where the problem distribution includes a non-trivial fraction of genuinely hard queries (difficulty bin 5), the overall system accuracy will be bounded by whatever the base model can achieve with best-of-N, and the compute-optimal framework's gains on easier problems cannot compensate. Said differently: test-time compute amplifies existing capability but does not create it from nothing.

What evidence exists in the paper. The evidence is stark and consistent across all methods:

• Figure 3 (right): On difficulty bin 5, both beam search and best-of-N weighted achieve roughly 1–3% accuracy regardless of budget (from 4 to 256 generations). No strategy makes meaningful progress.
• Figure 7 (right): On difficulty bin 5, all sequential-to-parallel ratios produce roughly 2–3% accuracy at a fixed budget of 128 generations. No allocation strategy helps.
• Figure 9: The bin 5 scaling line is essentially flat near 0–5% for both revisions (left) and PRM search (right). The ~14× larger model (stars) consistently outperforms test-time compute on these questions.
• Section 7 takeaway box: The authors explicitly acknowledge this boundary: "test-time compute cannot compensate for fundamental capability gaps."

Mitigation status. The paper is candid about this limitation but offers no mitigation. The only path forward on hard problems is scaling pretraining (larger model or more data), which the FLOPs-matched analysis in Section 7 explicitly quantifies: on difficulty bins 4–5, pretraining is almost always more effective than test-time compute, with relative disadvantages of up to -52.9% for PRM search at high R values.

---

3. Single Benchmark and Single Model Family — Generality Unverified

The assumption or constraint. All experiments use the MATH benchmark (500 test questions) with PaLM 2-S* as the base model. The authors state they "believe this model is representative of the capabilities of many contemporary LLMs" (Section 4), but this claim is unverified. The findings could be specific to this model's output distribution, its calibration properties, its error patterns, or its in-context learning behavior.

The consequence. Several aspects of the results are plausibly model-specific and may not transfer. The PRM's quality and over-optimization behavior depend on PaLM 2-S*'s output distribution — a model with better calibration might show different difficulty-dependent scaling curves, while a model with worse calibration might exhibit over-optimization at lower budgets. The revision model's ability to learn from incorrect in-context examples depends on the base model's in-context learning capabilities, which vary substantially across model families (e.g., GPT-4 vs. open-source models). The observation that beam search hurts easy-problem performance at high budgets (Figure 3, right) might not replicate on a model with a better-aligned PRM. More broadly, the MATH benchmark consists exclusively of competition-level math problems requiring symbolic reasoning — it is unknown whether the difficulty-dependent patterns (beam search hurting easy problems, revisions helping easy problems, no method helping hard problems) generalize to code generation, logical reasoning, scientific QA, or tasks requiring factual recall rather than inference.

What evidence exists in the paper. There is no cross-model or cross-dataset evaluation. The paper uses a single base model (PaLM 2-S*) for all primary experiments, with the ~14× larger model used only as a FLOPs-matched baseline in Section 7. The test set of 500 questions, split into five difficulty quintiles of ~100 each, is further split by two-fold cross-validation, meaning the compute-optimal policy is selected based on roughly 50 questions per fold per bin. This is a small sample, and the selected strategies may not be robust even within the MATH distribution, let alone across distributions.

Mitigation status. The paper does not address this limitation beyond the brief "we believe this model is representative" statement. No experiments with other model families (GPT, Claude, LLaMA, open-source models) or other benchmarks (code generation, reading comprehension, commonsense reasoning) are conducted. The authors acknowledge this implicitly in Section 8 by calling for "future work to study the generalization of these findings."

---

4. The ~14× Larger Model Baseline Is Not Compute-Optimally Trained

The assumption or constraint. The FLOPs-matched comparison in Section 7 scales model parameters while holding training data fixed, following the LLaMA paradigm (Touvron et al., 2023) rather than the Chinchilla-optimal approach (Hoffmann et al., 2022) of scaling both parameters and data equally. The authors acknowledge this:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

The consequence. A Chinchilla-optimal model trained with ~14× more total FLOPs would likely outperform a parameter-only-scaled model, making the pretraining baseline weaker than it could be. The reported advantages of test-time compute over pretraining — e.g., +27.8% on easy questions at R ≪ 1 for revisions — may shrink or reverse against a properly compute-optimal larger model. This is a significant confound in the paper's headline comparison between pretraining and inference compute. It means the central takeaway ("sometimes better to spend compute at inference rather than pretraining") is measured against a suboptimal pretraining baseline, and the thresholds at which inference compute becomes preferable may be more conservative than reported.

Additionally, the ~14× larger model uses only greedy decoding — no majority voting, no best-of-N, no search, and no test-time compute budget of its own. A fairer comparison might allocate the same inference-time FLOPs budget proportionally across the larger model's generations (e.g., if the smaller model gets 64 generations, the larger model gets proportionally fewer based on its per-token cost). This would create a stronger baseline that is never tested.

What evidence exists in the paper. The FLOPs-matched results appear in Figure 9 and the bar charts in Figure 1 (Section 7). The pretraining baseline's performance is represented as stars at three x-axis positions corresponding to the three R values. The paper explicitly acknowledges the Chinchilla departure in Section 7 but does not provide ablations or sensitivity analyses showing how results would change with compute-optimal pretraining.

Mitigation status. The authors acknowledge this limitation transparently but provide no corrected comparisons. The paper frames this as future work on "compute-optimal scaling of pretraining compute where data and parameters are both scaled equally." The current results should be interpreted as an upper bound on the advantage of test-time compute over pretraining, with the true advantage likely smaller when pretraining is optimally allocated.

---

5. Significant Qualitative Degradation at High Test-Time Budgets (Over-Optimization)

The assumption or constraint. The verifier-guided search methods assume that optimizing against the PRM's scores will find solutions that are genuinely correct, not just solutions that score highly under the PRM. This assumption breaks down at high budgets, where search begins to exploit weaknesses in the PRM — finding solutions that the verifier rates highly but that are actually incorrect.

The consequence. The over-optimization phenomenon means that test-time compute scaling is fundamentally capped: beyond a certain budget, additional compute either plateaus or actively harms performance. The compute-optimal policy mitigates this by routing easy problems away from aggressive search, but on medium-difficulty problems where beam search is deployed, over-optimization still limits the achievable performance ceiling. The degradation is not just a minor inefficiency — the paper documents qualitative failure modes (Appendix M, Figures 29 and following) where beam search produces degenerate outputs, including low-information repetitive steps at the end of solutions and overly short 1–2 step solutions that score highly under the PRM but are clearly flawed. These are not subtle errors — they are qualitatively unreasonable outputs that any human would recognize as wrong, yet the PRM assigns them high scores.

What evidence exists in the paper. Multiple lines of evidence converge:

• Figure 3 (right): On difficulty bin 1 (easiest), beam search accuracy decreases from roughly 78% to 77% as budget increases from 4 to 256 — a direct signature of over-optimization, since adding compute makes results worse.
• Figure 3 (left): Lookahead search — the most powerful optimizer — paradoxically performs worst overall at the same generation budget because its deeper optimization amplifies PRM exploitation.
• Figure 3 (left): Beam search with M = 4 plateaus and falls below best-of-N weighted at high budgets (256+ generations).
• Appendix M: Qualitative examples show degenerate beam search outputs: repetitive steps, overly short solutions, and solutions that score highly under the PRM but are clearly incorrect.

Mitigation status. The compute-optimal policy partially mitigates this by routing easy problems to less aggressive methods (best-of-N rather than beam search). However, this is a circumvention rather than a solution — it avoids the over-optimization regime rather than fixing the underlying PRM weakness. On medium problems where beam search is the compute-optimal choice, the method still hits the over-optimization ceiling. The paper identifies verifier robustness as a "key bottleneck" (Section 8) and suggests future work on training PRMs that remain calibrated under aggressive search, potentially through adversarial training, ensemble verification, or constrained search with KL-penalties. No such improved verifiers are developed or tested in this work.

---

6. Revisions and Search Are Studied Independently — The Synergy Is Unexplored

The assumption or constraint. The paper studies two complementary mechanisms — PRM search (Section 5) and iterative revisions (Section 6) — as separate, independently-evaluated approaches. The two are never combined, despite the paper's own conceptual framework (Section 2) identifying them as complementary axes: revisions modify the proposal distribution (generating better candidates), while PRM search improves selection (finding the best among candidates).

The consequence. The paper's results likely represent a lower bound on what an integrated system could achieve. The two mechanisms have naturally complementary strengths revealed by the paper's own difficulty analysis: revisions are most effective on easy problems (Figure 7, right, where fully sequential is optimal), while PRM search is most effective on medium problems (Figure 3, right, where beam search consistently outperforms best-of-N). On medium-difficulty problems, a system that uses the revision model as the proposal distribution within beam search — generating higher-quality candidate steps at each beam, conditioned on the revision history — could outperform either method alone. Similarly, using the PRM to guide which revision branches to pursue (rather than blindly generating a long revision chain) could improve revision efficiency by detecting when a revision is going off-track early. The independent study design also means we cannot assess whether the compute-optimal allocation over both search and revision hyperparameters jointly would yield additional gains beyond optimizing each in isolation.

What evidence exists in the paper. Section 8 explicitly acknowledges this gap:

"we did not experiment with PRM tree-search techniques in combination with revisions"

No experiments combine the two mechanisms. The compute-optimal analyses for search (Figure 4) and revisions (Figure 8) are performed independently, and the FLOPs-matched comparisons (Figure 9) treat them as separate approaches.

Mitigation status. The paper frames this as explicit future work but provides no preliminary results or analysis of how the combination might work. The conceptual framework (Section 2) provides the scaffolding for such integration — characterizing search as operating on the verifier axis and revisions on the proposal axis — but the integration itself remains unimplemented. Given that the paper's most impactful practical finding is the ~4× efficiency gain from compute-optimal allocation, understanding how much further gain could be achieved by jointly optimizing over both mechanisms is a significant open question that directly affects the practical value of the approach.

7. Implications and Future Directions

How This Work Changes the Landscape

ReAct introduces a conceptual reframing rather than a new model architecture or training objective: it establishes that reasoning and acting in language models are not independent capabilities that happen to be combinable, but are structurally interdependent — each is substantially weakened without the other, and their integration enables qualitatively different behaviors that neither can achieve alone. This reframing matters because it shifts the research conversation from "how do we build better reasoners" or "how do we build better agents" as separate questions toward a unified question of "how do we build systems that productively interleave internal computation with external interaction."

The magnitude of this shift is best characterized as opening a new design space rather than solving an existing problem. Prior to ReAct, the dominant paradigms treated reasoning (chain-of-thought, scratchpad, selection-inference) and acting (WebGPT, SayCan, Inner Monologue) as parallel research threads with different assumptions, different evaluation protocols, and different notions of success. ReAct demonstrates that a single frozen language model, prompted with a handful of human demonstrations, can operate effectively across knowledge-intensive reasoning (HotpotQA, Fever) and interactive decision-making (ALFWorld, WebShop) using the same interleaved format — and that performance in both settings depends on the synergy between the two components. This is not an incremental improvement to either thread; it is evidence that the separation itself was artificial.

The paper reconciles several tensions in the prior literature. Chain-of-thought reasoning (Wei et al., 2022) had shown that LLMs can produce coherent multi-step reasoning, but the paper demonstrates that this reasoning is brittle — 56% of CoT failures on HotpotQA are hallucinations that would be caught by a simple fact-checking action (Table 2). Conversely, language agents like WebGPT (Nakano et al., 2021) had shown that LLMs can interact with the web to answer questions, but the paper demonstrates that acting without reasoning leads to myopic behavior — the act-only agent in Figure 1(1c) retrieves relevant information but cannot synthesize it into a coherent answer. ReAct's error analysis (Table 2) provides the diagnostic that reconciles these findings: reasoning and acting have complementary failure modes, with reasoning failing via hallucination (56% of CoT errors) and acting failing via reasoning errors and uninformative retrieval (47% and 23% of ReAct errors respectively). This is not just two methods with different strengths — it is evidence that the failure modes are causally linked to the absence of the other component, and that the combination qualitatively changes the error profile.

The paper also implicitly resolves the tension between Inner Monologue (Huang et al., 2022b) and more flexible reasoning approaches. Inner Monologue had demonstrated that injecting environment feedback as text into an LLM's context improves embodied task performance, but the paper shows that this reactive feedback is fundamentally different from — and substantially weaker than — the generative reasoning that ReAct enables. The ReAct-IM ablation (71% vs. 53% overall success on ALFWorld, Table 3) quantifies this gap: simply closing the loop with observations is insufficient; the model must actively reason about what it knows, where objects are likely to be found, what subgoals remain, and when to change strategies. This finding redirects attention from "how do we format environment feedback" to "what kinds of internal reasoning do agents need to generate."

Several research directions become more attractive in light of this work. LLM-based agents with tool use — which have exploded since ReAct's publication — now have a clear conceptual foundation: the augmented action space $\hat{\mathcal{A}} = \mathcal{A} \cup \mathcal{L}$ provides a formal language for thinking about when and why an agent should interject reasoning between actions, and the paper's sparse-vs-dense thought pattern provides a practical design principle for different task types. Interpretable AI becomes more tractable because ReAct's externalized reasoning traces are directly inspectable and editable (Figure 5), enabling forms of human-AI collaboration that are impossible with black-box policies. Few-shot learning for interactive tasks becomes more promising because ReAct demonstrates that 1–2 demonstrations can outperform imitation learning agents trained on $10^3$–$10^5$ examples (Tables 3 and 4), suggesting that LLMs' pretrained knowledge can dramatically reduce the sample complexity of learning to act in environments.

Conversely, some directions become less attractive. Pure chain-of-thought scaling — the idea that better reasoning can be achieved by simply generating longer or more numerous reasoning chains — is challenged by the finding that CoT's primary failure mode (hallucination) is not addressed by more samples and that a small number of externally-grounded ReAct trajectories can match the performance of 21-sample CoT-SC (Figure 2). Imitation learning from scratch for text-based environments becomes harder to justify when a prompted LLM with no task-specific training outperforms agents trained on $10^5$ demonstrations (ALFWorld) or $10^4$ human annotations (WebShop). Complex search or planning algorithms as a primary research focus for LLM agents becomes less urgent — ReAct achieves strong performance with simple greedy decoding, suggesting that the bottleneck is not search sophistication but the quality of the reasoning-action loop itself.

Follow-Up Research This Work Enables

Training specialized models to decide when to think vs. act. The paper uses a fixed prompting format where the model autonomously decides when to interject thoughts, but this decision is learned implicitly from few-shot examples with no explicit optimization. A natural follow-up would train a lightweight "deliberation controller" — a small classifier or value head on top of a frozen LLM — that predicts, given the current context, whether the next token should be a thought or an action. This could be trained via reinforcement learning using task success as a reward, optimizing the tradeoff between reasoning depth (which improves action quality but consumes context length and inference compute) and action speed (which advances the task but risks errors from insufficient reasoning). The paper already provides the necessary infrastructure: the augmented action space $\hat{\mathcal{A}}$ formalizes the thought-vs-action decision, and the ReAct-IM ablation (Table 3) demonstrates that thought quality matters as much as thought presence, so the controller would need to optimize both when and what to think. A strong experiment would compare the learned controller against ReAct's implicit policy on ALFWorld, measuring both success rate and average trajectory length, and testing whether the controller learns task-specific deliberation strategies (e.g., thinking more on Clean tasks where subgoal tracking is harder, thinking less on Pick tasks where actions are straightforward).

Scaling ReAct with reinforcement learning for environment interaction. The paper demonstrates that prompting alone elicits effective closed-loop behavior, but it acknowledges limitations: the model sometimes gets stuck in repetitive loops (the "reasoning error" failure mode in Table 2, 47% of ReAct failures on HotpotQA) and cannot learn from its own mistakes across episodes. A natural extension is to fine-tune the language model with reinforcement learning using environment rewards (task success in ALFWorld, answer correctness in HotpotQA, attribute coverage in WebShop), treating the interleaved thought-action-observation trajectory as the policy. This is directly analogous to how WebGPT (Nakano et al., 2021) used RL to improve web browsing, but with the addition of explicit reasoning traces as part of the policy — the model would learn not just which actions to take, but which thoughts to generate to support those actions. The paper provides the key prerequisite: ReAct trajectories already contain the necessary structure (thoughts, actions, observations) for RL training, and the fine-tuning results in Figure 3 show that ReAct benefits from additional training data in a way that Standard and CoT do not (because learning to interact with an environment generalizes better than memorizing facts). A strong experiment would compare RL-fine-tuned ReAct against prompted ReAct on ALFWorld and WebShop, measuring whether RL can reduce the repetitive-loop failure mode and improve performance on the hardest task instances where prompting alone plateaus.

ReAct with multimodal observations for embodied AI. The paper is text-only — all observations are natural language strings — but the conceptual framework extends naturally to settings where observations include images, depth maps, or proprioceptive data. In a multimodal ReAct system, the language model would generate thoughts and actions in text, but observations would include both a text description (generated by a captioning model or object detector) and the raw visual features (encoded by a vision transformer and projected into the LLM's embedding space). The key question is whether ReAct-style reasoning traces remain beneficial when the model has access to rich visual information, or whether the reasoning becomes redundant because the visual features already provide sufficient grounding. The paper's finding that Inner Monologue-style reactive feedback is substantially weaker than generative reasoning (Table 3) suggests that even with rich observations, explicit reasoning about goals, subgoals, and commonsense object locations would provide value — but the visual modality might reduce the need for certain thought types (e.g., spatial reasoning about where objects are located) while increasing the need for others (e.g., visual attention direction, affordance reasoning). A strong experiment would replicate the ALFWorld protocol in a visual environment (e.g., the original ALFRED benchmark; Shridhar et al., 2020a) with a vision-language model, comparing ReAct-style interleaved reasoning against act-only and Inner Monologue baselines, and analyzing which thought types transfer from text-only to multimodal settings.

ReAct for collaborative multi-agent task solving. The paper demonstrates single-agent ReAct, but the thought-action format has an intriguing property for multi-agent settings: thoughts are natural language that other agents can read. In a multi-agent ReAct system, agents could broadcast their thoughts to teammates, enabling explicit coordination reasoning ("I am going to search the kitchen for a knife; Agent 2 should check the bathroom for a towel") that is grounded in each agent's actions and observations. This is qualitatively different from existing multi-agent LLM approaches that use structured communication protocols or centralized planners — ReAct's interleaved format means that coordination reasoning emerges naturally from the same mechanism that supports individual task-solving. The paper provides the key building block: thoughts serve as explicit working memory (Section 3.4), and in a multi-agent setting, broadcasting thoughts extends each agent's working memory to include teammates' intentions and beliefs. A strong experiment would implement a multi-agent ALFWorld variant where two agents must coordinate to complete tasks requiring object handoffs or simultaneous actions, comparing ReAct-style broadcast reasoning against (a) no communication, (b) structured message passing, and (c) centralized planning, measuring both task success and communication efficiency (how many messages are needed to achieve coordination).

Stress-testing ReAct on tasks where reasoning should not help. The paper shows ReAct consistently outperforms Act, but this creates a natural follow-up question: are there tasks where interleaved reasoning hurts performance, and if so, what properties predict this? Finding such tasks would sharpen our understanding of when the reasoning-acting synergy is beneficial versus when it is unnecessary or harmful. Candidates include: (a) tasks where the correct action is obvious from the observation and reasoning adds no information but consumes context length (e.g., simple reflex tasks like "if you see a red button, press it"); (b) tasks where reasoning misleads the model by activating incorrect commonsense associations (e.g., in an environment with deliberately counterintuitive object locations, reasoning "a knife is more likely to be in a drawer than a fridge" would be harmful if the environment places knives in fridges); (c) tasks where the action space is so constrained that exploration is trivial and reasoning overhead reduces the effective number of actions within a step budget. The paper provides suggestive evidence for the second case — ReAct's commonsense reasoning about object locations in ALFWorld could be harmful in non-standard environments — but no systematic study. A strong experiment would construct adversarial ALFWorld environments (e.g., with objects in deliberately unintuitive locations) and measure whether ReAct's performance degrades below Act's because the model's plausible-but-wrong reasoning about where objects should be found misleads its search, while Act's blind exploration succeeds. This would establish a boundary condition for ReAct's effectiveness that is currently missing.

Fine-tuning ReAct with human thought edits for alignment. The paper's human-in-the-loop demonstration (Figure 5, Appendix A.3) shows that editing a single thought in ReAct's trajectory can redirect the model's entire subsequent behavior. This suggests a new alignment paradigm: rather than collecting human preference comparisons over final outputs (as in RLHF) or human demonstrations of entire trajectories (as in imitation learning), collect human edits to reasoning traces — corrections to specific thoughts where the model's reasoning went wrong — and fine-tune the model to produce corrected reasoning. This is more sample-efficient than full trajectory demonstrations because a single thought edit can correct an error that would otherwise cascade through many actions, and it targets the root cause of failures (reasoning errors) rather than their symptoms (incorrect actions). The paper provides the key insight: the "reasoning error" failure mode accounts for 47% of ReAct's mistakes on HotpotQA (Table 2), and these errors are visible in the reasoning traces (e.g., repetitive thoughts, incorrect fact synthesis), making them identifiable and correctable by human annotators. A strong experiment would collect human edits to reasoning traces on a subset of ALFWorld or HotpotQA failures, fine-tune the model on these corrected trajectories, and measure whether the fine-tuned model shows reduced reasoning error rates and improved generalization to held-out task instances compared to fine-tuning on unedited ReAct trajectories.

Practical Applications and Downstream Use Cases

Customer support with grounded knowledge retrieval. A customer support chatbot powered by ReAct could interleave reasoning (decomposing the user's issue, identifying what information is needed) with actions (searching internal knowledge bases, pulling up order histories, checking inventory systems). The paper's results on HotpotQA and Fever demonstrate that this interleaving is not just theoretically appealing — it reduces hallucination from 56% of failures (CoT) to 0% (ReAct) while maintaining competitive accuracy, and the human-editable reasoning traces (Figure 5) mean that support agents can inspect and correct the chatbot's reasoning mid-conversation. For a deployment handling 10,000 queries per day, reducing the hallucination rate on factually-verifiable queries from ~14% (CoT's false positive rate on HotpotQA, Table 2) to ~6% (ReAct's rate) would mean roughly 800 fewer incorrect answers per day that require human correction. The key practical requirement is implementing the augmented action space — connecting the LLM to internal APIs for knowledge base search, customer record lookup, and order management — which follows directly from the Wikipedia API design in Section 3.1.

Automated fact-checking and claim verification at scale. The Fever results (ReAct: 60.9% accuracy, CoT-SC → ReAct: 64.6%) demonstrate that interleaving reasoning with targeted Wikipedia retrieval outperforms both pure reasoning and pure retrieval for verifying factual claims. This has direct application to content moderation, journalism, and research: a ReAct-based verification system could process claims from social media posts, news articles, or scientific manuscripts, generating interpretable verification trajectories (which sources were checked, what evidence was found, how the conclusion was reached) that human reviewers can audit. The CoT-SC → ReAct hybrid (Section 3.2) is particularly well-suited to this setting because it first checks whether the model's internal knowledge confidently supports or refutes a claim (fast, cheap), and only triggers external retrieval when internal knowledge is uncertain — reducing API calls and latency for the majority of claims while maintaining accuracy on ambiguous ones. For a newsroom processing 5,000 claims per day, using CoT-SC → ReAct instead of full retrieval for every claim could reduce Wikipedia API calls by 60–80% (depending on the uncertainty threshold) while achieving accuracy comparable to always-retrieving, based on the finding that CoT-SC with 3–5 samples plus ReAct matches 21-sample CoT-SC performance (Figure 2).

Interactive task guidance for augmented reality and robotics. The ALFWorld results (71% success with 2-shot prompting vs. 37% for BUTLER trained on 100k demonstrations) demonstrate that ReAct can serve as a general-purpose reasoning-and-acting engine for household tasks with minimal task-specific training. This has direct application to augmented reality (AR) task guidance systems: a user wearing AR glasses could speak a high-level goal ("put a clean knife on the counter"), and a ReAct-powered system would generate a sequence of interleaved thoughts ("I need to find a knife — a knife is more likely to be in cabinets, drawers, or on countertops") and actions ("go to cabinet 1", "open cabinet 1"), with the observations coming from the AR system's object recognition rather than a text simulator. The human-editable reasoning traces (Figure 5) mean that users can correct the system mid-task without restarting — if the system hallucinates that a drawer contains a knife when it doesn't, the user can edit the thought to redirect the search. The key practical challenge is the observation interface: converting visual scene understanding into the text format that ReAct expects, analogous to how ALFWorld converts the simulated household into text observations. The paper's finding that sparse thoughts (every 5–10 actions) suffice for long-horizon tasks means the system would not constantly narrate its reasoning, only interjecting at decision points — making it less intrusive for AR users.

E-commerce search and purchase assistance. The WebShop results (40.0% success rate for ReAct vs. 30.1% for Act and 28.7% for IL+RL) demonstrate that ReAct improves product search by actively reasoning about whether search results match user requirements before clicking. This translates directly to shopping assistant applications: a user specifies "I need a nightstand with drawers, nickel finish, under $140," and the ReAct-based assistant searches, examines product pages, compares attributes against the instruction, and either purchases or presents options to the user. The paper's qualitative analysis (Table 10) shows that ReAct's key advantage is bridging the gap between user instructions (which use natural language attribute descriptions) and product listings (which use varied, noisy terminology) — the reasoning step explicitly checks "does this product have the specified color?" rather than clicking blindly. For a shopping platform processing 100,000 complex product searches per day (where users specify 3+ constraints), replacing an Act-only search system with ReAct would increase the success rate from ~30% to ~40% (based on Table 4), meaning roughly 10,000 additional satisfied searches per day. The practical deployment would require implementing the WebShop-style action space (search, click product, select options, purchase) against the platform's actual product database and search API, with careful attention to the latency of interleaved reasoning steps — on WebShop, ReAct trajectories typically involve 2–3 thought-action cycles before purchase, adding modest overhead compared to Act's direct click-and-buy approach.

When to Prefer This Method

The paper does not articulate an explicit decision rule for choosing ReAct over alternatives; it demonstrates ReAct's advantages across diverse tasks and positions it as a general paradigm. However, the experimental results imply conditions where ReAct is likely to be preferable, which can be synthesized into a decision guideline grounded in the paper's findings:

Prefer ReAct prompting when:

• The task requires accurate, up-to-date factual information that may not be present or correct in the model's training data. The paper shows that ReAct reduces hallucination from 56% of CoT failures to 0% (Table 2) and can retrieve current information that corrects outdated training labels (Appendix A.2, Figure 4 — ReAct correctly identifies a hotel's current room count while all other methods fail). Tasks involving recent events, changing facts, or verifiable claims benefit most.
• The task involves extended multi-step interaction where subgoal tracking and error recovery are important. The ALFWorld results show that ReAct's reasoning traces — particularly subgoal completion detection and next-subgoal determination — prevent the repetitive loops that trap act-only agents (Figure 1(2a), Appendix D.2.2), and the advantage over Act grows with task complexity (larger gaps on Clean and Heat tasks that require more steps).
• Interpretability and human oversight are valuable — either for debugging model behavior, building user trust, or enabling human-in-the-loop correction. The human-in-the-loop demonstration (Figure 5) shows that editing a single thought can redirect the entire trajectory, which is impossible with Act (no thoughts to edit) or trained RL policies (behavior not controllable without retraining).

Prefer chain-of-thought prompting (without actions) when:

• The task can be solved entirely from parametric knowledge and external retrieval would add latency without improving accuracy. On HotpotQA, pure CoT slightly outperforms ReAct (29.4 vs. 27.4 EM, Table 1), suggesting that for questions where the model's training data contains sufficient information, the reasoning flexibility lost to ReAct's structural constraints (47% vs. 16% reasoning error rate, Table 2) may outweigh the grounding benefits.
• The environment is unavailable or too costly to query. ReAct assumes an accessible environment with a text-based action interface; if no such environment exists, or if action latency is prohibitive (e.g., each Wikipedia API call takes seconds), CoT-SC with sufficient samples (21 in the paper's experiments) provides a closed-book alternative.

Prefer the ReAct + CoT-SC hybrid when:

• Both approaches are available and the cost of running both is acceptable. The hybrid methods (ReAct → CoT-SC and CoT-SC → ReAct) consistently outperform either approach alone (Table 1, Figure 2), matching 21-sample CoT-SC performance with only 3–5 samples. The choice between the two directions depends on task properties: ReAct → CoT-SC (try retrieval first) is better on HotpotQA where compositional reasoning is the primary challenge; CoT-SC → ReAct (try internal knowledge first) is better on Fever where factual precision is paramount and internal knowledge provides a strong first pass.

These guidelines are not presented as definitive by the paper — they are empirical patterns observed across four benchmarks with PaLM-540B, and the paper explicitly does not claim them as universal. The key uncertainty is how these patterns transfer to other model families, other task types, and other environment interfaces, which remain open questions for future work.