Do As I Can, Not As I Say: Grounding Language in Robotic Affordances

ArXiv: 2204.01691

🎯 Pitch

A robot paired with a large language model goes from near-total failure to 74% execution success on long-horizon tasks, simply by multiplying the model’s semantic plan scores with value functions that check what the robot can actually do in its current environment. Because this “grounding” only requires swapping in a better language model to cut errors, the paper demonstrates a direct path where advances in LLMs automatically translate to more capable real-world robots.

1. Executive Summary

This paper introduces SayCan, a method that combines large language models with robotic affordance functions to enable a mobile manipulator to execute abstract, long-horizon natural language instructions in real-world environments. Evaluated across 101 tasks in office kitchen settings with a PaLM-based robot system, SayCan grounds the LLM's semantic knowledge about task procedures (the "Say" — scoring the likelihood that a skill makes progress toward a high-level instruction) in the robot's physical capabilities (the "Can" — value functions that capture the probability of successfully executing each skill from the current state), multiplying these two probabilities to select actions that are both useful and feasible. The grounded approach achieves an 84% planning success rate and 74% execution success rate, nearly doubling performance over ungrounded baselines — and the system's performance improves simply by swapping in a better LLM (PaLM 540B reducing errors by half compared to FLAN), establishing that language model advances can directly translate to robotics gains only when the model is grounded in the agent's affordances.

2. Context and Motivation

The Core Problem: LLMs Know What to Do, But Not Whether They Can Do It

The fundamental gap this paper addresses is the disconnect between a language model's semantic knowledge about task procedures and an embodied agent's physical capability to execute those procedures in a specific environment. Large language models (LLMs) trained on internet-scale text corpora encode rich knowledge about everyday tasks — how to clean a spill, how to prepare a meal, what objects are typically needed for certain activities. If a robot could tap into this knowledge, it could potentially interpret abstract, temporally extended natural language commands like "I just worked out, can you bring me a drink and a snack to recover?" without requiring explicit programming for every possible instruction variant.

However, as the paper frames the problem in Section 1, this knowledge exists in a vacuum:

"LLMs are not grounded in the physical world and they do not observe the consequences of their generations on any physical process. This can lead LLMs to not only make mistakes that seem unreasonable or humorous to people, but also to interpret instructions in ways that are nonsensical or unsafe for a particular physical situation."

The paper illustrates this with its opening example (Figure 1): a kitchen robot asked "I spilled my drink, can you help?" might receive an LLM-generated response like "You could try using a vacuum cleaner" — a narratively reasonable but physically impossible suggestion if no vacuum exists in the scene or if the robot lacks vacuuming capability. Even with prompt engineering to coerce the LLM into outputting a step-by-step plan, the model has no mechanism for determining whether each proposed step is actually executable given the robot's current state, its motor capabilities, and the objects present in the scene.

Why This Problem Matters

The significance is both practical and conceptual, spanning several dimensions:

Enabling general-purpose instruction following. For robots to be useful in unstructured human environments (homes, offices, hospitals), they must handle instructions that are abstract ("bring me something hydrating"), context-dependent ("throw away the one I just drank from"), and temporally extended ("restock the rice chips on the far counter"). Hand-engineering every possible instruction variant is combinatorially impossible. LLMs offer a path to zero-shot generalization — the robot can interpret novel phrasings and novel task combinations without explicit training on those specific commands. But this only works if the LLM's knowledge can be connected to physical execution.

The interaction challenge. The paper frames instructions as covering multiple axes of variation that real users exhibit (Section 5, Table 1): time-horizon (single primitives to 10+ step sequences), language complexity (structured commands to fully unstructured crowd-sourced requests), abstraction (verbs like "restock" and nouns like "fruit" that require semantic interpretation), and embodiment (instructions whose correct plan depends on the robot's current state and environment layout). An ungrounded LLM fails on all axes that require awareness of physical context.

A new architecture for robot learning systems. Implicitly, the paper addresses a structural question in robotics: where should "intelligence" live in a robotic control stack? Traditional approaches place reasoning either in a symbolic planning layer (with hand-crafted rules and predicates) or in end-to-end learned policies (which rarely handle long-horizon abstraction). SayCan proposes a hybrid: the LLM provides the open-ended semantic reasoning and plan decomposition, while learned value functions provide the physical grounding. This factorization means improvements to either component — better language models or better robotic skills — independently boost overall system performance, as the paper demonstrates by showing that swapping PaLM for FLAN reduces errors by half (Table 3).

Prior Approaches and Where They Fall Short

The paper identifies several lines of existing work, each with specific limitations that SayCan addresses:

1. Ungrounded LLM planning through prompt engineering alone. Concurrent work by Huang et al. (Section 7) used prompt engineering to extract temporally extended plans from LLMs for embodied agents, roughly corresponding to the "Generative" baseline in SayCan's experiments. The limitation is fundamental: while prompt engineering can constrain the LLM's output format (e.g., to produce numbered steps with specific action templates), it cannot ground those steps in the robot's physical reality. The LLM might propose picking up an object that is not visible, navigating to a location that is blocked, or ordering steps in a physically impossible sequence. The paper's experiments confirm this: the "No VF" (no value function) and "Generative" baselines achieve only 67% and 74% planning success respectively, compared to SayCan's 84% (Table 2), with the gap concentrated in instruction families that require physical awareness — embodiment tasks drop from 64% to 18% without affordances.

2. End-to-end language-conditioned behavioral cloning (BC) and reinforcement learning (RL). A substantial body of work has trained policies that directly map natural language instructions to low-level robot actions using imitation learning or RL (Section 7). The paper tests this approach as a baseline: "BC NL" feeds the full high-level instruction directly into the language-conditioned policy. The result is stark — 0% success rate across all tasks, including single primitives (Table 2). This is unsurprising: these policies are trained on atomic commands like "pick up the coke can" and have no mechanism for decomposing "throw away the apple and bring me a coke" into its constituent steps. Even the more sophisticated "BC USE" baseline, which projects the high-level instruction to the nearest known command using sentence embeddings, achieves 60% on single primitives but 0% on all other families. The failure mode is clear: end-to-end policies cannot handle instructions outside their training distribution, particularly those involving multiple sequential skills or abstract language.

3. Grounding through additional training inputs. Several prior works have attempted to ground language models by training them to accept sensor inputs (camera images, depth maps; see Section 7 references to VideoBERT, VisualBERT, ViLBERT) or by fine-tuning them with interactive data (rewards, ranking feedback). These approaches require the LLM to learn a joint representation of language and perceptual data, which is data-intensive and typically limited to the specific sensor modalities and embodiments used during training. SayCan avoids this by not modifying the LLM at all — it uses the LLM as a frozen, off-the-shelf scorer of text sequences and grounds it through a separate, independently trained set of value functions. This means the LLM can be swapped without retraining (enabling the PaLM-to-FLAN comparison) and the value functions can be trained with standard RL methods on the specific robot platform.

4. Classical task and motion planning (TAMP). Symbolic planners (STRIPS, hierarchical planners) can sequence high-level actions with feasibility guarantees, but require manually specified preconditions, effects, and world models — and typically operate over symbolic rather than perceptual states. Learned TAMP approaches relax some of these constraints but still struggle with open-domain natural language instructions that use abstract verbs and nouns far from the planner's predefined predicate vocabulary. SayCan's key advance is that the LLM handles the language-to-skill mapping (translating "recover from a workout" into finding water and an apple, as in Figure 5a) without any manually specified semantic rules, while the value functions handle the symbol-to-physical feasibility check.

5. Grounding through LLM representations in policy training. A popular approach has been to use pre-trained LLM embeddings as the language encoder for downstream policy networks (Section 7 references work by Hill et al., Lynch and Sermanet, Nair et al., and others). While this improves generalization to synonymous phrasings of the same skill ("grab" vs. "pick up"), it still trains the policy to execute individual atomic commands. The LLM's broader knowledge about task decomposition — that throwing away a can first requires finding it, picking it up, navigating to the trash, and placing it — remains unused. SayCan moves the LLM's role from encoding language to reasoning with language: the LLM actively decomposes high-level instructions into sequences of skills rather than passively providing embeddings.

How SayCan Positions Itself

The paper's positioning sits at a deliberate intersection of several research traditions:

A probabilistic factorization of language grounding and physical grounding. The core intellectual move (Section 3) is to factor the probability that a skill makes progress toward a high-level instruction — denoted $p(c_i | i, s, \ell_\pi)$ — into the product of two independent terms: $p(\ell_\pi | i)$ (the LLM's estimate that skill $\ell_\pi$ is a semantically appropriate next step for instruction $i$) and $p(c_\pi | s, \ell_\pi)$ (the value function's estimate that skill $\ell_\pi$ can be successfully executed from state $s$). Crucially, this factorization makes a specific assumption explicit:

"Assuming that a skill that succeeds makes progress on $i$ with probability $p(\ell_\pi | i)$ (i.e., its probability of being the right skill), and a skill that fails makes progress with probability zero"

This is a modeling choice — it assumes independence between semantic appropriateness and physical feasibility, and it assumes that a physically successful skill execution always constitutes progress if the skill was semantically appropriate. These are reasonable first-order approximations but carry implicit assumptions that the paper acknowledges implicitly through its failure analysis (e.g., when the LLM selects an inappropriate skill that the value function then incorrectly marks as feasible).

Not a new planning algorithm, but a new combination of existing components. SayCan does not propose novel methods for language modeling, reinforcement learning, or behavioral cloning. Instead, it proposes a specific architecture for combining these components: frozen LLM + pretrained value functions + multiplicative scoring + iterative replanning. This is both a strength (simplicity, modularity, upgradability) and a deliberate research contribution: showing that the interface between language and robotics components is itself a design problem with significant performance implications.

A zero-shot approach to long-horizon tasks. Critically, SayCan requires no additional training when given new high-level instructions, new combinations of skills, or even new language models. The robot's skill repertoire (Π) is fixed and trained independently; the LLM is queried at inference time with no fine-tuning. This positions SayCan in the emerging paradigm of foundation models for robotics — large models trained on broad data that can be adapted to specific embodiments without additional training.

A bridge between NLP and robotics that allows independent scaling. The paper explicitly demonstrates (Section 5.1, Table 3, and Appendix Table 6) that swapping in a better LLM improves end-to-end robotic task performance: PaLM-SayCan (540B parameters) achieves 84% planning success versus FLAN-SayCan's 70%. The paper frames this as a key insight:

"This is particularly exciting because it represents the first time we can see how an improvement in language models translates to a similar improvement in robotics. This result indicates a potential future where the fields of language processing and robotics can collaboratively improve each other and scale together."

This positioning anticipates a research trajectory where NLP advances — better reasoning, multilingual capabilities, chain-of-thought prompting — directly benefit robotic systems without requiring the robotics community to retrain perception or control stacks. The paper demonstrates this concretely: by simply appending chain-of-thought prompting to the existing prompt (Section 5.2, Table 4), SayCan gains the ability to handle negations and reasoning-requiring tasks that the base system fails on, with zero changes to the robot's skills or value functions.

3. Technical Approach

3.1 Reader Orientation

SayCan is a decision-making system for robots that combines a frozen large language model (the "Say" component) with learned robotic value functions (the "Can" component) to select and execute sequences of physical skills that accomplish abstract, long-horizon natural language instructions. The system solves the problem of grounding language model knowledge in physical reality: an LLM knows that cleaning a spill involves finding a sponge, but does not know whether a sponge is currently visible, whether the robot is holding something else, or whether the robot can physically reach the sponge — SayCan multiplies the LLM's semantic judgment with a physically-grounded feasibility estimate to produce plans that are both useful and executable, iteratively selecting one skill at a time, executing it, and replanning from the new state.

3.2 Big-Picture Architecture (Diagram in Words)

The SayCan system has four major components and a loop that connects them:

1. Skill Repertoire (Π) — a fixed set of atomic robot behaviors (e.g., "pick up the coke can," "go to the table," "put down the sponge"), each with a short natural language description $\ell_\pi$ and an associated policy $\pi$ for executing the behavior. These skills and their policies are trained independently via behavioral cloning (BC) or reinforcement learning (RL) before SayCan operates.

2. Affordance Functions (the "Can") — for each skill, a learned value function $p(c_\pi \mid s, \ell_\pi)$ that estimates the probability that skill $\ell_\pi$ can be successfully executed from the current state $s$ (where $c_\pi$ is a binary random variable indicating completion). These value functions are trained via temporal-difference RL with sparse rewards (1 for success, 0 otherwise), making them exactly the affordance probability that the skill is possible.

3. Large Language Model (the "Say") — a frozen, pre-trained LLM (primarily PaLM 540B) used as a scoring model, not a generative decoder. Given the high-level instruction $i$ and the history of previously selected skills, the LLM computes $p(\ell_\pi \mid i, \text{history})$ for each candidate skill $\ell_\pi$ — the probability that this skill is a semantically appropriate next step toward completing the instruction. The LLM is queried through prompt engineering that structures the interaction as a dialog between a human and a robot, with examples showing valid plans.

4. The Planner (the combination) — at each decision step, the planner computes a combined score for every skill $\pi \in \Pi$ as the product of the LLM's semantic probability and the affordance function's feasibility probability: $p_{\text{combined}} \propto p(\ell_\pi \mid i, \text{history}) \cdot p(c_\pi \mid s, \ell_\pi)$. The skill with the highest combined score is selected, executed by the robot (updating the physical state $s$), appended to the instruction history, and the process repeats until a termination token ("done") is selected.

The information flow is: instruction $i$ enters → LLM scores all skill descriptions against $i$ → affordance functions score the same skills against current state $s$ → scores are multiplied → argmax selects the winning skill → skill policy executes on the robot → state updates → winning skill description is appended to the history → LLM re-scores with updated history → repeat until "done."

3.3 Roadmap for the Deep Dive

• First, the formal problem statement and probabilistic factorization (Section 3.4, "Problem Statement and Probabilistic Factorization"), which defines precisely what SayCan computes and why the product of two independent probabilities is the right objective.
• Second, the LLM scoring mechanism (Section 3.4, "LLM Scoring: The 'Say' Component"), which explains how a frozen language model is used as a scoring function via prompt engineering with a fixed output vocabulary — including the prompt structure, how probabilities are extracted, and why scoring outperforms generative decoding with post-hoc projection.
• Third, the affordance functions (Section 3.4, "Affordance Functions: The 'Can' Component"), which covers how value functions are trained with RL to produce $p(c_\pi \mid s, \ell_\pi)$, how different skill families (pick, go-to, place) use different affordance implementations, and how raw Q-values are calibrated into probabilities.
• Fourth, the iterative planning loop (Section 3.4, "Iterative Planning and Execution"), which walks through the full Algorithm 1 — how skills are selected one at a time, how context is updated, and how termination is handled.
• Fifth, the skill training pipeline (Section 3.4, "Training the Low-Level Skills"), which describes how the individual policies and value functions are obtained — the BC-Z-based behavioral cloning, the MT-Opt-based reinforcement learning, the simulation-to-real transfer, and the multi-task training architecture.
• Sixth, the practical calibration and implementation choices (Section 3.4, "Practical Calibration and Implementation Choices"), which covers how raw value function outputs are normalized into well-behaved probabilities, how different skill types use different affordance sources, and how the system avoids degenerate behaviors like repeated execution of already-completed skills.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and architecture paper whose core idea is that the probability of a skill advancing a high-level instruction can be factorized into independent semantic and physical terms, enabling frozen LLMs and independently-trained robotic policies to be combined with no additional training.

Problem Statement and Probabilistic Factorization

SayCan formalizes the instruction-following problem as follows. The system receives a user-provided natural language instruction $i$ that can be "long, abstract, or ambiguous" (Section 3). It has access to a fixed set of skills $\Pi$, where each skill $\pi \in \Pi$ performs a short atomic task (e.g., picking up a specific object, navigating to a location, placing an object) and comes with two pieces of metadata: a short natural language description $\ell_\pi$ (e.g., "pick up the coke can") and an affordance function $p(c_\pi \mid s, \ell_\pi)$, which is the probability that the skill with description $\ell_\pi$ can be successfully completed from the current state $s$. Here $c_\pi$ is a Bernoulli random variable indicating successful completion.

The LLM provides $p(\ell_\pi \mid i)$ — the probability that skill description $\ell_\pi$ is a semantically valid next step for instruction $i$. However, the quantity the system actually needs is $p(c_i \mid i, s, \ell_\pi)$, the probability that executing skill $\ell_\pi$ from state $s$ successfully makes progress toward completing instruction $i$. The paper makes two modeling assumptions to factor this:

1. A skill that succeeds physically ( $c_\pi = 1$ ) makes progress on $i$ with probability $p(\ell_\pi \mid i)$ — i.e., if you can do the skill, its semantic probability determines whether it was the right skill.
2. A skill that fails physically ( $c_\pi = 0$ ) makes progress with probability zero — a failed execution cannot help.

Under these assumptions:

$p(c_i \mid i, s, \ell_\pi) \propto p(c_\pi \mid s, \ell_\pi) \cdot p(\ell_\pi \mid i)$

where $p(c_\pi \mid s, \ell_\pi)$ is the affordance (world-grounding — "can the robot do this?"), $p(\ell_\pi \mid i)$ is the LLM score (task-grounding — "is this the right thing to do?"), and the product gives the probability that the skill both can be done and should be done.

What it computes: this multiplication takes two independent assessments — the LLM's semantic judgment about task relevance and the robot's physical assessment about execution feasibility — and combines them into a single score for ranking candidate skills. For each skill $\pi$, the affordance function is evaluated once from the current state $s$, the LLM is queried once for its task relevance, the two scalar probabilities are multiplied, and the skill with the maximum product is selected.

Why this form: the factorization assumes conditional independence between skill appropriateness (driven by language understanding) and skill feasibility (driven by physical state). This independence is the key architectural insight — it means the LLM needs no access to sensor data and the value functions need no language understanding beyond their atomic skill descriptions. An alternative formulation that fed sensor data into the LLM (as in VisualBERT-style approaches) would require joint training and would make the system sensitive to the specific LLM used. A formulation that used only affordances (the "No LLM" baseline) would fail on abstract instructions that require semantic interpretation. A formulation that used only the LLM (the "No VF" baseline) would propose physically impossible actions. The multiplicative form ensures that a skill must be both semantically appropriate and physically feasible to be selected — if either probability is near zero, the product is near zero, which naturally filters out semantically plausible but physically impossible suggestions (like "use a vacuum" when no vacuum exists) and physically possible but semantically irrelevant ones (like picking up an apple when asked to clean a spill).

The paper notes that in the undiscounted, sparse reward case with a reward of 1.0 for success and 0.0 otherwise, the value function learned via RL exactly corresponds to an affordance function in the Gibsonian sense — it specifies "whether a skill is possible in a given state" (Section 2). This connection between RL value functions and affordances is not original to this paper, but the paper's insight is to use it as the physical grounding interface between language models and robot control.

LLM Scoring: The "Say" Component

The LLM component of SayCan computes $p(\ell_\pi \mid i, \text{history})$ — the probability that a given skill description $\ell_\pi$ is the semantically appropriate next step given the high-level instruction $i$ and the sequence of previously executed skills. This is done through a specific mechanism that treats the LLM as a scoring function over a fixed candidate set, not as a generative decoder.

Prompt engineering and response structure. The LLM is conditioned with a carefully constructed prompt (full prompt shown in Appendix D.3, Listing 1) that provides 17 examples of human-robot dialog. Each example follows this pattern:

Human: How would you [high-level instruction]?
Robot: I would: 1. [skill], 2. [skill], ..., N. done.

The prompt serves several functions simultaneously. It teaches the model the response format (numbered steps with "done" as a termination token). It demonstrates the mapping from abstract language to concrete skill sequences (e.g., "How would you bring me some snacks?" maps to finding and delivering specific snack items). It establishes the dialog framing ("How would you" / "I would") which the paper found to be "both more natural and performant" than imperative formats. It implicitly communicates the robot's available skills and objects by including them in the example sequences.

The paper reports an ablation in Appendix D.3 (Table 5) showing how planning success rate in a language-based simulator varies with the number of prompt examples:

• With 0 examples and a requirement to terminate: 10% planning success (the model rarely issues "done").
• With 1 example: 64% success (immediate jump, but occasional termination failures).
• With 4 examples: 82% success.
• With 17 examples (the full prompt used in experiments): 88% success.
• Without the termination requirement, 0 examples achieved 52% success, indicating that some task decomposition knowledge is "already imbued within the language model" even without examples.

The paper also notes that the structure of the examples matters: "Providing explicit numbers between steps (e.g., 1., 2., instead of combining skills with 'and then' or other phrases) improved performance, as did breaking each step into a separate line." Examples that overly feature specific objects create bias toward those objects in scoring.

Scoring mode vs. generative decoding. A critical design choice is that SayCan does not ask the LLM to generate a plan freely and then parse it. Instead, it uses the LLM's scoring interface — the model outputs the log-probability it assigns to each candidate completion from a fixed, pre-specified set. This is the same mechanism that underlies language model evaluation metrics like perplexity, but repurposed for decision-making.

Formally, the LLM represents a distribution over token sequences $p(w_k \mid w_{<k})$, where $w_k$ is a token that appears at position $k$. Given the prompt (which includes the instruction $i$ and the dialog history including previously selected skills $\ell_{\pi_{n-1}}, \dots, \ell_{\pi_0}$), the model can score any candidate continuation. SayCan constructs candidate continuations corresponding to each skill $\ell_\pi \in \ell_\Pi$ — specifically, it forms the string that would result from appending skill $\ell_\pi$ as the next numbered step in the "Robot: I would:" response format — and queries the LLM for the log-probability of that continuation. The probabilities are then normalized across the skill set: $p(\ell_\pi \mid i, \text{history})$ is computed as the softmax over these log-probabilities (or equivalently, the relative likelihood among all skill descriptions).

The scoring approach has three advantages over generative decoding. First, it guarantees that the selected action is always a valid skill from the repertoire — there is no risk of the LLM generating a syntactically malformed step or referencing a skill that does not exist, because the candidate set is fixed and pre-validated. Second, it produces explicit probabilities for every option, which can be meaningfully multiplied with affordance scores — a sampled or beam-searched generation would only provide a single sequence without calibrated likelihoods for alternatives. Third, it makes the system interpretable: the user can inspect the ranked list of candidate skills and their combined scores to understand why the robot chose a particular action (as shown in Figure 6, which visualizes the decision-making process).

Iterative querying with history. At each decision step $n$, the LLM is queried with the instruction $i$ and all previously selected skills appended to the history. This means the LLM sees the full dialog context:

Human: [instruction i]
Robot: I would: 1. [skill_0], 2. [skill_1], ..., n-1. [skill_{n-1}]

The next query asks the model to score continuations that append "n. [candidate_skill]" to this context. This iterative querying enables the LLM to condition its scoring on what has already been done — if the robot has already picked up a coke can and brought it to the user, the model should assign low probability to "pick up a coke can" again and high probability to "done" or to the next object in a multi-object instruction. The paper demonstrates this in Figure 6: after the robot has already brought one soda, the LLM correctly assigns higher probability to the second soda rather than repeating the first, and the affordance function prevents early termination until the task is complete.

Handling multiple language models. The paper emphasizes that the LLM used for planning (PaLM 540B, or FLAN 137B in the ablation) is separate from the language model used to embed skill descriptions for the low-level policies. The low-level BC and RL policies use a pre-trained Universal Sentence Encoder (USE) to embed skill descriptions into vectors that condition the policy networks. This separation allows SayCan to "utilize different language models well suited for different abstraction levels — understanding planning with respect to many skills as opposed to expressing specific skills more granularly" (Section 4). The planning LLM handles semantic reasoning about task decomposition; the embedding LLM handles conditioning low-level visuomotor control on specific atomic commands.

Affordance Functions: The "Can" Component

The affordance component of SayCan provides $p(c_\pi \mid s, \ell_\pi)$, the probability that skill $\pi$ with description $\ell_\pi$ can be successfully executed from the current state $s$. These affordance functions are derived from value functions learned via reinforcement learning, and they serve as the physical grounding that makes the LLM's semantic plans executable.

Why value functions equal affordances. In the undiscounted, sparse reward setting, the agent receives a reward of 1.0 at the end of an episode if the skill was successfully completed, and 0.0 otherwise. Under these conditions, the state-action value function $Q^\pi(s, a)$ represents the expected discounted sum of future rewards, which is exactly the probability of eventually succeeding from state $s$ when taking action $a$ and then following policy $\pi$. The state value function $v(s) = \max_a Q^\pi(s, a)$ represents the probability of success from state $s$ under the optimal action choice. This is the affordance property: $v(s)$ tells you "can this skill be completed from here?" in a principled, learned way that generalizes across states.

The paper formalizes this in its preliminaries (Section 2) by defining a Markov Decision Process $\mathcal{M} = (\mathcal{S}, \mathcal{A}, P, R, \gamma)$ with state space $\mathcal{S}$, action space $\mathcal{A}$, transition function $P$, reward function $R$, and discount factor $\gamma$. The Q-function is learned via approximate dynamic programming that minimizes:

$L_{TD}(\theta) = \mathbb{E}_{(s,a,s') \sim \mathcal{D}} \left[R(s, a) + \gamma \mathbb{E}_{a^* \sim \pi} Q^\pi_\theta(s', a^*) - Q^\pi_\theta(s, a)\right]$

where $\mathcal{D}$ is the dataset of states and actions (replay buffer), $\theta$ are the parameters of the Q-function, $s'$ is the next state after taking action $a$ in state $s$, $a^*$ is an action sampled from the policy $\pi$, and $\gamma$ is the discount factor.

What it computes: the temporal difference loss drives $Q^\pi_\theta(s, a)$ toward the expected sum of discounted future rewards from state $s$ and action $a$ under policy $\pi$. The first term $R(s, a) + \gamma \mathbb{E}_{a^* \sim \pi} Q^\pi_\theta(s', a^*)$ is the TD target — the immediate reward plus the discounted value of the next state under the policy. The second term $Q^\pi_\theta(s, a)$ is the current estimate. Their squared difference (in the implicit loss) is minimized to bring the Q-function into consistency with the Bellman equation.

Why this form: temporal difference learning is the standard approach for learning value functions from off-policy data because it bootstraps — it uses the learned value function's own estimate of the next state's value as part of the target, which propagates reward information backward through time without requiring complete trajectory rollouts or a known transition model. The sparse reward structure (1.0 for success, 0.0 otherwise) means the Q-function naturally calibrates to a probability scale between 0 and 1, making it directly interpretable as an affordance. An alternative approach like Monte Carlo returns (averaging full-episode outcomes) would require complete episodes and would have higher variance, while model-based planning would require learning an explicit world model of the environment dynamics.

Multi-task training for amortization. Rather than training a separate policy and value function for each of the 551 skills in the full repertoire, the paper uses multi-task BC and multi-task RL where a single model is conditioned on the language description of each skill. The language conditioning works as follows: each skill's text description (e.g., "pick up the coke can") is passed through a pre-trained Universal Sentence Encoder (USE), producing a fixed-size embedding vector. This embedding is concatenated with the robot action and non-image state features (such as gripper height) and used as conditioning input to the policy and value function networks. This means a single set of network parameters handles all skills, and the distinction between skills is carried entirely by the language embedding. The paper reports that this enables scaling to hundreds of skills without training hundreds of separate models.

Training protocol for RL policies (MT-Opt-based). The RL policies are trained using the MT-Opt framework in the Everyday Robots simulator, with RetinaGAN used for sim-to-real transfer to reduce the visual domain gap. The training process (detailed in Appendix C.2) works as follows:

1. Bootstrapping from demonstrations: simulation demonstrations provide initial success episodes to seed the replay buffer, preventing the RL algorithm from needing to discover successful behaviors through pure random exploration (which would be infeasible given the sparse reward).

2. Online data collection: 3000 CPU workers continuously collect episodes in simulation using the current policy, adding successful and unsuccessful trajectories to the replay buffer.

3. Target Q-value computation: an additional 3000 CPU workers compute target Q-values from the replay buffer, decoupling this computation from the gradient update step so the TPU can be used solely for model training.

4. Training: the model is trained on 16 TPUv3 chips for approximately 100 hours. The loss is a log loss (since rewards are 0 or 1, this is essentially binary cross-entropy on the Q-value predictions). Prioritized experience replay is used, with episode priority tuned to keep the replay buffer close to 50% success for each skill. The priority $p$ is defined as $p = 1 + 10 \cdot |0.5 - \bar{p}|$, where $\bar{p}$ is the average success rate of episodes in the replay buffer for that skill — this up-weights episodes from skills where success is rare (the buffer is heavily skewed toward failure) and down-weights episodes from skills where success is very common, maintaining a balanced training signal.

5. Image processing: the 640×512 input image is padded by 100 pixels left-right and 40 pixels top-down, then randomly cropped back to 640×512 to provide spatial augmentation. Standard random brightness and contrast augmentations are also applied.

6. Network architecture: the RL model uses an architecture similar to MT-Opt (shown in Appendix Figure 9). The camera image is processed through 7 convolutional layers. The language embedding, robot action, and non-image state features are concatenated and passed through fully connected layers, then tiled spatially and added to the convolutional volume. The combined features pass through 11 more convolutional layers. The output is gated through a sigmoid, constraining Q-values to [0, 1], which is consistent with the success-probability interpretation.

Training protocol for BC policies (BC-Z-based). The behavioral cloning policies are trained from human teleoperation demonstrations rather than RL (Section 4, Appendix C.2). The dataset consists of 68,000 teleoperated demonstrations collected over 11 months using a fleet of 10 robots, expanded with 276,000 autonomous episodes from learned policies that are success-filtered to yield 12,000 additional successful episodes. The training protocol:

1. Data filtering: human raters mark episodes as unsafe (robot collided with environment), undesirable (robot perturbed irrelevant objects), or infeasible (skill cannot be done or is already accomplished). Marked episodes are excluded from training.

2. Architecture: the BC model uses a ResNet-18 based architecture conditioned by FiLM layers, with the USE language embedding modulating the visual features. The 640×512 input image is downsampled to 256×320 for faster iteration (half-size images work well for BC but not for RL value functions, which require full-size images). Multiple fully connected layers are applied to the final visual features to output each action component separately: arm position (continuous, MSE loss), arm orientation (continuous, MSE loss), gripper command (discrete, cross-entropy loss), and termination action (discrete, cross-entropy loss). Each action component is weighted evenly in the loss.

3. Training: the BC model is trained on 16 TPUv3 chips for approximately 27 hours.

4. Image augmentations: random brightness, contrast, and spatial cropping are applied identically to the RL model.

Why both BC and RL? The paper uses BC policies for actual skill execution (they achieve higher success rates at the current stage of data collection) but uses RL value functions for the affordances. The BC policies do not produce value functions — they are direct action-to-observation mappings — so the affordance models must come from the RL training. The paper notes this explicit division: "While we find that the BC policies achieve higher success rates at the current stage of our data collection process, the value functions provided by the RL policies are crucial as an abstraction to translate control capabilities to a semantic understanding of the scene." This is a practical design trade-off: BC gives better execution, RL gives necessary affordance estimates, and SayCan's modular architecture allows mixing them.

Action space. The policies output seven components (Section 4): the six degrees of freedom of the end-effector pose (3D position and 3D orientation), gripper open/close commands, x-y position and yaw orientation delta for the mobile base, and a terminate action. This covers both manipulation and mobile navigation in a unified action space.

Evaluation-driven policy selection. Policies are continuously evaluated and graduated (Appendix C.3). A separate evaluation protocol tests skills in the Mock Office Kitchen, with 18–48 skills sampled per skill category (pick up, knock over, place upright, open/close drawers, move object near another). Object positions are randomized with one or two distractor objects. Episodes terminate after 50 actions or when the policy samples terminate. Human operators supervise multiple robots performing evaluation simultaneously, recording successes and failures. Models that outperform prior models on mock stations are promoted to testing in the real kitchen and eventually integrated into SayCan. Figure 11 in Appendix C.3 shows the progression of per-skill success rates over 11 months of continuous data collection, growing from 1 skill in April 2021 to hundreds of skills at publication time.

Iterative Planning and Execution

With the LLM scoring mechanism and affordance functions in place, SayCan's planning loop (Algorithm 1 in the paper) operates as follows:

Step-by-step execution of Algorithm 1.

1. Initialization (line 1): Set $n = 0$ (step counter) and $\pi = \emptyset$ (empty history of selected skills). The initial state $s_0$ is the current robot observation.

2. Loop (line 2): Continue while the previous selected skill $\ell_{\pi_{n-1}}$ is not "done." On the first iteration, no previous skill exists, so the loop always enters.

3. Score computation for all skills (lines 4–8): For every skill $\pi \in \Pi$ with its language description $\ell_\pi$:

   • Line 5: Compute $p_{\text{LLM}_\pi} = p(\ell_\pi \mid i, \ell_{\pi_{n-1}}, \dots, \ell_{\pi_0})$ — the LLM scores the skill description given the instruction and the full history of previously selected skills.
   • Line 6: Compute $p_{\text{affordance}_\pi} = p(c_\pi \mid s_n, \ell_\pi)$ — the affordance function evaluates the probability of successful execution from the current state $s_n$.
   • Line 7: Compute $p_{\text{combined}_\pi} = p_{\text{affordance}_\pi} \cdot p_{\text{LLM}_\pi}$ — multiply the two probabilities.
   • Line 8: Store the combined score in a set $C$.

4. Skill selection (line 10): $\pi_n = \arg\max_{\pi \in \Pi} C$ — select the skill with the maximum combined score.

5. Execution (line 11): Execute the selected skill's policy $\pi_n(s_n)$ in the environment. The robot physically performs the skill (moving its arm, navigating its base, manipulating objects), producing a new state $s_{n+1}$.

6. Increment and repeat (line 12): Set $n = n + 1$ and return to step 2, now with the selected skill description appended to the history.

7. Termination: The loop ends when a skill with description "done" is selected. The "done" skill has a small fixed affordance value (0.1, as discussed below) to ensure it can be selected when no other useful skills remain.

What this loop computes: it produces a sequentially-extended plan where each skill is chosen based on both its semantic fit to the original instruction and its physical feasibility at the moment of execution. This is not a plan-and-then-execute architecture — the robot executes skills immediately after selection and replans from the new state, which means that if the state changes during execution (due to skill success, partial failure, or environmental dynamics), the next skill selection automatically adapts.

Why this form: the iterative replanning addresses a key limitation of open-loop planning. If the system generated a full 10-step plan at the outset and then executed it blindly, any execution failure in step 3 would invalidate steps 4–10. By interleaving planning and execution, SayCan naturally handles the case where a skill succeeds (the affordance for that skill drops to near-zero because it is already done, so the system moves on) and the case where the environment changes. The paper explicitly notes this limitation in Section 8: "the system is not easily able to react to situations where individual skills fail despite reporting a high value, though this could potentially be addressed by appropriate prompting of the language model for a correction." The follow-up work by Huang et al. (2022, cited in Section 5.2) builds on SayCan to add exactly this closed-loop capability through "inner monologue" — feeding environment feedback back into the LLM.

History context management. The LLM is queried with all previously selected skills in order, which gives it access to what has been accomplished. This is crucial for instructions that involve multiple objects ("bring me a coke and an apple") — after delivering the coke, the LLM should know that it still needs to find and deliver the apple. The paper demonstrates this in Figure 15a: when asked to "bring me two different sodas," the LLM first scores the coke can highly, and after the coke is delivered, it correctly scores the pepsi can highly for the second step. The affordance function prevents early termination by giving a higher score to the second soda retrieval than to "done" — if the system terminated after one soda, the instruction would be incomplete.

Training the Low-Level Skills

The low-level skills that form SayCan's action repertoire are trained using a combination of behavioral cloning and reinforcement learning, with the specific architectures and training procedures detailed in Section 4 and Appendix C.

Language conditioning for policies. All low-level policies (both BC and RL) are conditioned on language through pre-trained sentence embeddings. A Universal Sentence Encoder (USE) is used to embed each skill's text description into a fixed-size vector. The USE is frozen during policy training — only the policy network parameters are updated. This means the policy network learns to map from "embedding of 'pick up the coke can'" to the appropriate visuomotor behavior, while the USE provides a semantic similarity structure that enables some generalization (e.g., "grab the coke" might have a similar embedding to "pick up the coke can"). However, as the paper emphasizes, the USE-based language conditioning handles only low-level skill specification — it does not provide the task decomposition or semantic reasoning that the planning LLM (PaLM) provides. The two language models operate at different levels of the abstraction stack.

RL policy architecture (Appendix C.1, Figure 9). The RL model uses a convolutional encoder-decoder architecture with language conditioning:

1. Visual encoder: The camera image (640×512 RGB) is processed through 7 convolutional layers to produce a spatial feature volume.

2. Language and state conditioning: The skill description is embedded by the USE language model. This embedding is concatenated with the robot action (from the previous timestep), the non-image state features (including gripper height), and a "time-to-completion" signal indicating how much of the previous action remains to be executed (enabling asynchronous control where inference happens while the robot is still moving).

3. Fusion: The concatenated conditioning vector goes through fully connected layers, then is tiled spatially (replicated across the spatial dimensions) and added to the convolutional volume from the visual encoder.

4. Deeper processing: The fused features pass through 11 additional convolutional layers.

5. Output gating: The final output is passed through a sigmoid activation, constraining the Q-value to $[0, 1]$, consistent with the success-probability interpretation.

This architecture is similar to MT-Opt with the modification that language embeddings replace the task-ID conditioning used in the original MT-Opt work, enabling zero-shot generalization to new skill phrasings.

BC policy architecture (Appendix C.1, Figure 10). The BC model uses a ResNet-18 backbone with FiLM conditioning:

1. Visual encoder: A ResNet-18 processes the downsampled input image (256×320, half the original resolution for faster training).

2. Language conditioning (FiLM): The USE embedding is used to produce FiLM parameters (scale and shift vectors) that modulate the activations at multiple layers of the ResNet. This is the same conditioning mechanism used in BC-Z, where it was shown to enable zero-shot generalization to new combinations of objects and actions.

3. Multi-head output: The final visual features are passed through separate fully connected layers for each action component — arm position (6 DoF, continuous, MSE loss), arm orientation (continuous, MSE loss), gripper open/close (discrete, cross-entropy loss), base movement (continuous x-y position and yaw orientation delta), and termination (discrete, cross-entropy loss). Each action component is weighted evenly in the total training loss.

The BC model does not receive the "time-to-completion" signal or the previous action, as these were found unnecessary for learning the BC policy from demonstrations. The BC model also does not produce Q-values — it directly outputs actions — which is why the affordance functions must come from the separately trained RL models.

Reward function and success labeling. Both RL and BC training rely on human-labeled success judgments. For each training episode (a robot performing a skill given a language command), human raters watch a video of the performance and answer whether the skill was accomplished successfully. The paper uses a majority-vote protocol: if two out of three raters agree the skill succeeded, the episode is labeled with a positive reward (1.0 for the reward function; included in BC training data). This sparse, binary success signal drives both the RL value function calibration (since Q-values converge to success probabilities) and the BC data filtering (unsuccessful episodes are excluded, though the paper notes that episodes are only excluded if marked as unsafe, undesirable, or infeasible — not simply unsuccessful).

Skill repertoire composition. The paper defines 551 skills across seven skill families and 17 objects, including picking, placing and rearranging objects, opening and closing drawers, navigating to various locations, and placing objects in specific configurations. For the experiments, the authors select a subset of skills that are "most amenable to more complex behaviors via composition and planning as well as those that have high performance at the current stage of data collection" (Section 4). The final skill set spans:

• Pick skills (e.g., "pick up the coke can," "pick up the apple") — multi-task BC policy with RL value functions, trained as described above.
• Go to skills (e.g., "go to the table," "find a sponge") — classical planning-based navigation stack (the paper assumes object locations are known), with affordance based on Euclidean distance to the target.
• Place skills (e.g., "put down the sponge") — scripted Cartesian motion planning policy; no learned affordance (always set to 1.0).
• Drawer manipulation skills (e.g., "open the drawer," "close the drawer") — BC policy with heuristic affordance (always possible when near the drawer).
• Terminate skill ("done") — fixed affordance of 0.1, always semantically valid (the LLM can choose to terminate at any step).

This heterogeneous implementation — mixing learned policies, classical planning, and heuristics — demonstrates the flexibility of the probabilistic interface: as long as each skill provides a probability of success (however computed), it can be incorporated into SayCan's planning framework.

Practical Calibration and Implementation Choices

The raw outputs of value functions, distance measures, and heuristics do not naturally produce well-calibrated probabilities in [0, 1] that can be meaningfully multiplied with LLM scores. The paper applies several calibration techniques to convert these raw values into usable affordance probabilities:

Calibration formula for pick skills. The trained RL value functions for pick skills produce Q-values that typically range between empirically observed minimum and maximum values for each skill. The paper normalizes these to [0, 1]:

$p_{\text{affordance}}^{\text{pick}} = \text{clamp}\left(\frac{v_{\text{pick}} - v_{\text{pick}}^{\text{min}}}{v_{\text{pick}}^{\text{max}} - v_{\text{pick}}^{\text{min}}}, 0, 1\right)$

where $v_{\text{pick}}$ is the raw value function output (the state value $v(s) = \max_a Q(s, a)$ found through cross-entropy method optimization), $v_{\text{pick}}^{\text{max}} = 0.5$ and $v_{\text{pick}}^{\text{min}} = 0.2$ are empirically determined thresholds, and the clamp function constrains the output to [0, 1].

What it computes: this is a min-max normalization that maps the raw value function output from its typical operating range [0.2, 0.5] onto the full [0, 1] probability interval. When the value is at or below 0.2 (the minimum observed for feasible skills), the affordance is 0 — the skill is deemed impossible. When the value is at or above 0.5 (the maximum observed for infeasible skills), the affordance is 1 — the skill is deemed certainly possible. Values in between are linearly interpolated.

Why this form: the raw Q-values are learned via RL with sparse rewards, but the training process does not guarantee calibration to [0, 1] — the Q-function might never output exactly 0 or 1 because the Bellman backup with discount factor $\gamma < 1$ and function approximation errors produces values that cluster in a narrower range. The min-max normalization with empirically chosen thresholds provides a pragmatic calibration that makes the affordance scores numerically compatible with the LLM's probability outputs (which are naturally in [0, 1] since they are softmax-normalized). The specific thresholds (0.2 and 0.5) are dataset- and robot-specific and would need recalibration for a different robot or environment.

Affordance for go-to skills (distance-based). For navigation skills, the affordance is based on the Euclidean distance $d$ (in meters) from the robot's current position to the target location:

$p_{\text{affordance}}^{\text{goto}} = \text{clamp}\left(\frac{d_{\text{goto}}^{\text{max}} - d}{d_{\text{goto}}^{\text{max}} - d_{\text{goto}}^{\text{min}}}, 0, 1\right)$

with $d_{\text{goto}}^{\text{max}} = 100$ and $d_{\text{goto}}^{\text{min}} = 0$. This produces a linearly decreasing probability as distance increases — the robot is most confident it can reach nearby locations and least confident about distant ones.

Affordance for place skills (constant). Place skills are assumed to always be feasible: $p_{\text{affordance}}^{\text{place}} = 1.0$. The paper justifies this by noting that "language is sufficient to understand place is only possible after a pick" — the LLM has learned from the prompt examples that placing an object only makes sense after picking it up, so the language model (rather than the affordance function) prevents inappropriate placement commands.

Affordance for terminate (small constant). The terminate action ("done") receives a fixed affordance of $p_{\text{affordance}}^{\text{terminate}} = 0.1$. This small value ensures that termination can be selected when all other skills have near-zero affordances (nothing is feasible) or when all task-relevant skills have been completed (the LLM assigns high probability to "done" and low probability to remaining skills). If the terminate affordance were zero, the system could get stuck in an infinite loop when no skills are feasible.

Preventing repeated execution of completed skills. The paper implements a practical mechanism to prevent SayCan from selecting a skill that "has already been performed or will have no effect" (Appendix D.2). When a skill is successfully executed, its affordance is manually capped to indicate that the reward has been received and the skill does not need to be repeated. This prevents behaviors like the robot picking up an object it is already holding, or navigating to a location it is already at, even if the raw value function would still assign a non-zero probability.

Object-location knowledge assumption. The paper explicitly notes that "since the focus of this work is mainly on planning, we assume the location of objects are known." Navigation skills map directly to the coordinates of known object locations using a classical planning-based navigation stack. This means SayCan does not need to solve the perception problem of searching for objects — it knows where "the coke can" is, and the navigation skill simply drives there. This assumption simplifies the affordance computation (go-to skills are purely distance-based) but limits the system's applicability to environments without known object locations. The paper acknowledges this as a scope limitation and suggests that object detection could be integrated into the affordance pipeline in future work.

Cross-validation for prompt engineering. The paper notes that "as SayCan as a whole requires affordances from a world embodiment, it is not straightforward to optimize this structure and tune parameters quickly." To address this, the authors built a "language-based simulator" that, given a high-level instruction and a ground-truth sequence of skills, outputs synthetic affordances consistent with the query and solution, along with distractor affordances for robustness testing. The simulator is used to verify that SayCan recovers the correct solution and to tune prompt engineering choices (number of examples, phrasing, structure) before deploying on the physical robot. This highlights an important practical aspect: the prompt engineering is optimized in a fast, simulated loop rather than through expensive physical trials.

Mixing policies and affordances from different sources. The flexibility of the probabilistic interface allows SayCan to mix policies from behavioral cloning (pick skills), classical planning (go-to, place skills), and heuristics (drawer skills, termination), with affordances from learned value functions, distance measures, and constants. This is deliberate: the paper states that "SayCan is capable of incorporating many different policies and affordance functions through its probability interface." The only requirement is that each skill provides a probability $p(c_\pi \mid s, \ell_\pi) \in [0, 1]$ of successful execution. This modularity means that as new skills are developed (e.g., through improved RL training, new sensors, or new hardware) or existing skills are improved, they can be added to the repertoire by simply adding their language description to the candidate set and their affordance function to the computation — no retraining of the LLM, no modification of other skills, and no change to the planning algorithm.

Skill selection granularity. As a practical note, the paper mentions that some skills from the full 551-skill repertoire are excluded because they are "not naturally part of long-horizon tasks" (e.g., "move object near object" and "knock object over") or because their performance is not yet consistent enough for reliable long-horizon chaining (e.g., drawer manipulation, which is demonstrated in Section 5.2 as a case study but not integrated into the main 101-instruction evaluation). The planning success rates for the integrated drawer tasks (Table 7 in Appendix E.3) show 100% planning but only 33% execution, with the primary failure being that the manipulation policy fails to open the drawer wide enough — an execution failure, not a planning failure.

4. Key Insights and Innovations

Innovation 1: Probabilistic Factorization of Semantic and Physical Grounding as a Zero-Shot Interface

The paper's most fundamental conceptual contribution is not a new planning algorithm, but rather the architectural decision to factor robotic instruction following into two independent probability terms multiplied together: $p(c_\pi \mid s, \ell_\pi)$ (physical feasibility from value functions) and $p(\ell_\pi \mid i)$ (semantic appropriateness from LLMs). What makes this framing distinctive is that it identifies the interface between language models and robotic controllers as itself a design problem — and proposes a specific mathematical interface (multiplicative combination of independent probabilities) that requires no joint training, no shared representations, and no modification of either component.

Before SayCan, the dominant paradigms for connecting language to robot behavior fell into two camps that each required tighter coupling. End-to-end approaches (Lynch and Sermanet, 2020; Nair et al., 2021; Hill et al., 2020) used LLM embeddings as inputs to downstream policy networks, which tied the language representation to the specific policy architecture and training data — you could not swap the LLM without retraining the policy. Multi-modal grounding approaches (VideoBERT, VisualBERT, ViLBERT) trained LLMs to accept sensor inputs directly, which required joint training data combining language and perception and bound the LLM to the specific sensor modalities of the training setup. Prompt-engineering-only approaches (Huang et al., 2022, concurrent with SayCan) extracted plans from LLMs with no physical grounding at all — the robot had no mechanism to determine whether proposed steps were executable.

SayCan's factorization is a fundamental shift in this landscape because it establishes that the connection between language knowledge and physical capability can be stateless — purely multiplicative, with each component computable independently. The LLM needs no access to sensor data (it only sees text: the instruction, the history, and skill descriptions). The value functions need no awareness of the high-level instruction (they only answer "can skill X succeed from this state?"). The two probability distributions are computed separately and combined through multiplication at decision time. This has three consequences that the paper demonstrates but which are architectural properties, not experimental findings:

First, modular upgrade paths. Because the factorization has no shared parameters or joint training, either component can be replaced independently. The paper demonstrates this concretely by swapping PaLM 540B for FLAN 137B and showing that planning success drops from 84% to 70% (Table 3) — a 14 percentage point degradation that required zero changes to the robot's skills, value functions, or affordance calibration. In the other direction, improving the underlying robotic skills (e.g., adding drawer manipulation in Section 5.2, or improving policy success rates over 11 months as shown in Appendix Figure 11) automatically improves system performance without touching the LLM or prompt. This property is not a minor convenience — it means the NLP and robotics components of the system can advance on independent research timelines and the combined system benefits from progress in either field, which is a fundamentally different development model than joint end-to-end training.

Second, zero-shot generalization across all axes of instruction variation. Because the LLM is queried at inference time (not fine-tuned on task-specific data), its broad semantic knowledge — including synonym handling, abstraction, common sense about task procedures, and even multilingual understanding — is available without any task-specific training. Table 2 shows that BC NL (end-to-end language-conditioned policy) achieves 0% across all instruction families, and BC USE (sentence embedding projection) achieves 60% on single primitives but 0% on everything else. SayCan achieves 84% planning on the same tasks because the LLM's pre-trained knowledge handles the language-to-skill mapping that end-to-end policies would need explicit training data to learn. The multilingual results (Appendix E.5, Table 8) are particularly striking: Chinese, French, and Spanish queries achieve identical planning rates to English with no additional engineering — the LLM's multilingual training data automatically provides this capability because the interface is text-only.

Third, interpretability as a direct consequence of the architecture. Because each skill receives an explicit combined score computed as a product of two interpretable components (the LLM's semantic probability and the affordance function's feasibility probability), the system's decision-making can be visualized and inspected (Figure 6, Appendix Figure 14). A human operator can see not only which skill was selected, but what the alternatives were and why they scored lower — was the LLM's semantic score low (the skill doesn't make sense for the instruction) or was the affordance low (the skill is physically infeasible)? This interpretability is not an add-on feature but a direct consequence of the factored architecture: the LLM and affordance scores are computed separately and can be exposed separately. In contrast, an end-to-end policy that maps directly from instruction and sensor data to actions produces a black-box decision with no decomposition into "what should I do" versus "what can I do."

The paper's factorization has an important limitation that the authors do not fully analyze: the assumption that semantic appropriateness and physical feasibility are conditionally independent. This assumption can fail in practice — for example, if the robot is holding an apple, the LLM might correctly assign low probability to "pick up an apple" (semantically inappropriate because the robot already has one) and the affordance might also be low (physically infeasible to pick up while holding something), but these two low scores are correlated through the underlying state, not independent. The paper's 16% planning failure rate (84% rather than 100%) likely includes cases where this independence assumption breaks down, but no specific failure analysis isolates these correlation-induced errors versus separate LLM errors and affordance errors (the paper reports 65% LLM failures, 35% affordance failures, but does not report how many failures involve both simultaneously). Despite this limitation, the factorization's practical success — enabling a working system on 101 real-world tasks with zero additional training — demonstrates that the independence assumption is a productive first-order approximation.

Innovation 2: Value Functions as the Physical Grounding Interface — Not Policies, Not Detectors, Not Models

The paper's second distinctive conceptual move is its choice of what provides the "Can" — not object detectors, not success classifiers, not explicit world models, but learned value functions from reinforcement learning. This is a specific and non-obvious choice that has significant architectural implications that the paper demonstrates but whose conceptual significance warrants explicit analysis.

Prior work on grounding language in physical contexts has used several alternative mechanisms. Object detection approaches (e.g., ViLD used in the paper's open-source tabletop implementation) ground language by detecting whether the objects mentioned in an instruction are present in the scene. Success classifiers trained on execution videos can predict whether a skill will succeed, but they are discriminative models that do not capture the value of intermediate states and do not provide a signal for which action to take, only whether a proposed action would work. Explicit world models (as in classical TAMP approaches) require hand-specified preconditions and effects for each action, which do not generalize to open-vocabulary objects or perceptual states.

Value functions are fundamentally different from all of these alternatives in two ways that make them particularly well-suited as the grounding interface:

First, value functions capture graded feasibility, not binary possibility. An object detector answers "is the coke can visible?" with a yes/no (or confidence score that the object exists). A value function answers "what is the probability of successfully completing 'pick up the coke can' from this state?" which implicitly encodes many factors beyond object presence: the robot's current arm configuration, whether the gripper is occupied, the distance to the object, the presence of obstacles, the robot's learned competence at this specific grasping task from similar states, and the downstream effects of this action on future task success. When the paper shows (Figure 2) that a value function for "pick up the red bull can" produces a high score when the can is visible and reachable, and a low score when the robot is navigating an empty space, the function is doing more than object detection — it is integrating visual perception, motor feasibility, and learned task competence into a single scalar. This means the value function automatically handles cases that would require explicit engineering in alternative approaches: if the robot is already holding an object, the pick affordance should be low not because the object isn't visible, but because grasping is physically impossible with an occupied gripper — a value function trained on execution outcomes naturally learns this.

Second, value functions are action-oriented rather than state-descriptive. A world model answers "what will happen if I take this action?" by predicting the next state. A value function answers "is this action worth taking?" by predicting the probability of eventual success. For grounding purposes, the second question is more directly useful because it collapses the complex dynamics of robotic interaction into a single decision-relevant scalar. The paper's probabilistic factorization only needs $p(c_\pi \mid s, \ell_\pi)$ — it does not need to predict what the world will look like after the action, only whether the action will succeed. This makes value functions a minimal-sufficient grounding signal: they provide exactly the information needed for the multiplicative combination with the LLM and no more.

The paper's use of RL-trained value functions specifically (rather than supervised success classifiers trained on execution outcomes) adds an additional property: value functions are trained with temporal-difference learning, which propagates reward information backward through time. This means the value function for a pick skill can learn that certain approach trajectories are better than others not because the approach itself is directly rewarded, but because trajectories that bring the gripper to the correct pre-grasp position make the subsequent grasp more likely to succeed. A supervised classifier trained only on final success/failure labels would only learn correlations between initial states and final outcomes, missing the sequential structure. The paper's use of MT-Opt with its distributed training infrastructure (3000 CPU workers for data collection, 3000 for target computation, 16 TPUv3 chips for training, prioritized experience replay balancing the buffer to 50% success per skill) provides the scale necessary to learn these value functions across hundreds of skills.

The paper also demonstrates the flexibility of this interface by using different sources of affordance for different skill families: learned RL value functions for pick skills, distance-based heuristics for go-to skills, constant 1.0 for place skills. This mixing would be inelegant if the affordance concept were tightly coupled to a specific learning method, but the paper's probabilistic interface treats them all identically — each skill provides a number in [0, 1] — so heterogeneous implementations coexist seamlessly. This is conceptually significant because it means the grounding interface can adopt the best available method for each skill type (learned where data exists, heuristic where physics is simple, constant where language provides sufficient constraint) without architectural fragmentation.

The limitation of this approach, which the paper acknowledges implicitly through its calibration procedure (thresholding raw Q-values with $v_{\text{pick}}^{\text{max}} = 0.5, v_{\text{pick}}^{\text{min}} = 0.2$), is that trained value functions are not naturally well-calibrated probabilities. The calibration requires domain-specific threshold selection and assumes that the value function's operating range is consistent across states and skills — an assumption that may break as the robot encounters novel states or as skills are updated. Future work on better-calibrated value functions (through techniques like distributional RL or explicit uncertainty estimation) would directly improve SayCan by reducing reliance on heuristic calibration.

Innovation 3: The LLM-as-Scorer Paradigm — Constrained Output Through Fixed-Vocabulary Likelihood Evaluation

A third conceptual move, which is easy to overlook because it appears as an implementation detail, is the decision to use the LLM as a scoring function over a fixed candidate set rather than as a generative decoder. This choice has profound implications for reliability, interpretability, and compatibility with the probabilistic interface that deserve recognition as a distinct innovation.

The dominant paradigm for using LLMs in planning at the time (and largely still) was generative: give the LLM a prompt describing the task, let it generate a plan as free-form text, and then parse the text to extract actionable steps. This is the approach in concurrent work by Huang et al. (2022), which roughly corresponds to SayCan's "Generative" baseline. The generative approach has an obvious failure mode: the LLM might produce text that is syntactically malformed, references unavailable skills or objects, uses unexpected formatting, or omits necessary steps — and the downstream parser must handle all of these edge cases. The paper's Generative baseline (Table 2) achieves 74% planning success compared to 84% for the scoring approach, with the gap attributable to these parsing and constraint failures.

The scoring approach avoids these failures entirely by constraining the LLM's output space to exactly the set of admissible skills. For each skill $\ell_\pi \in \ell_\Pi$, the system constructs a specific candidate continuation string (the text that would result from appending that skill as the next numbered step in the "Robot: I would:" format) and queries the LLM for the log-probability of that exact string. The LLM never has the opportunity to generate invalid text — it is asked "how likely is this specific continuation?" rather than "generate a continuation." This converts the problem from unconstrained text generation (where errors require post-hoc detection and correction) to constrained classification (where the model simply scores pre-validated options).

This is a fundamental shift in how LLMs are used for decision-making, not an incremental refinement. It has three consequences that the paper demonstrates:

First, it guarantees output validity with no parsing or post-processing. The selected skill is always an element of the fixed candidate set. There is no risk of the LLM inventing a skill name, referencing an object not in the repertoire, or formatting its response in a way the system cannot parse. This reliability is particularly important for robotic systems where executing an incorrectly-parsed action could be unsafe.

Second, it produces explicit, comparable probabilities for every option. The LLM assigns a likelihood to each candidate skill, not just the winner. These probabilities are meaningful numbers that can be multiplied with affordance scores — a generative model producing a single sampled output does not provide calibrated likelihoods for alternatives. This is what enables the multiplicative combination at the heart of SayCan: $p(\ell_\pi \mid i)$ must be a number for every $\pi$, and the scoring interface provides exactly that.

Third, it naturally handles iteration with history. When the robot selects and executes a skill, that skill's description is appended to the dialog history, and the LLM is re-queried with the expanded context. The scoring approach handles this seamlessly — the LLM simply scores skill descriptions against the updated prompt, and the probabilities shift based on what has already been done. A generative approach would need to either regenerate the entire plan from scratch (which could produce inconsistent plans across iterations) or implement a constrained decoding mechanism that is aware of previous steps.

The paper's ablation on prompt engineering (Appendix D.3, Table 5) reveals that the scoring approach requires careful prompt design to work well — with 0 examples, planning success is only 10% when termination is required, suggesting that the pre-trained LLM does not naturally understand the "numbered sequence ending with done" format without demonstration. However, with just 4 examples, performance jumps to 82%, indicating that the format is quickly learnable through few-shot prompting. This sensitivity to prompt design is a practical limitation but does not diminish the conceptual significance of the scoring paradigm itself.

The scoring approach has an important limitation that the paper does not discuss: it requires that all candidate skills can be enumerated and scored, which scales linearly with the size of the skill repertoire. For the 101-instruction evaluation, the skill set is small enough (on the order of tens of skills across objects and locations) that scoring all options is computationally trivial. For a robot with thousands of skills (combining hundreds of objects with dozens of action types), the scoring cost could become prohibitive. Solutions might involve hierarchical scoring (first select the action type, then the object) or using the LLM's generative capability to produce a shortlist of candidates that are then scored, but the paper's architecture does not address this scaling question.

Innovation 4: Empirical Demonstration That LLM Improvements Directly Improve Robotics — Without Retraining

The paper's most practically consequential finding is not a new method but an empirical result with systemic implications: swapping a better language model into SayCan (PaLM 540B replacing FLAN 137B) improves planning success from 70% to 84% and execution success from 61% to 74%, with zero changes to the robot's skills, policies, value functions, or environment (Table 3). The paper frames this as "the first time we can see how an improvement in language models translates to a similar improvement in robotics."

This result matters not because it's surprising that better language models produce better plans — that's expected — but because it demonstrates that the factorization architecture decouples NLP progress from robotics engineering. This is a systemic innovation, not an algorithmic one: it establishes a development model where the NLP community can advance language models (better reasoning, broader knowledge, multilingual capability, chain-of-thought prompting) and the robotics community can advance physical skills (more reliable grasping, more objects, more dexterous manipulation) on independent tracks, and the combined system improves from both.

The paper provides evidence across multiple axes of this decoupling. The LLM ablation (Appendix E.2, Table 6) shows monotonic improvement with model size: PaLM 8B achieves 38% planning (generative with USE projection), PaLM 62B achieves 72%, PaLM 540B achieves 74%, and FLAN 137B achieves 43%. The gap between PaLM 540B and FLAN (74% vs. 43% on planning with generative output) is particularly instructive — FLAN was specifically fine-tuned on an "instruction answering" dataset, which might naively seem like the right training objective for an instruction-following system, but PaLM's broader pre-training on a more diverse corpus produces better results. This suggests that the LLM's role in SayCan — understanding the semantic content of instructions, making common-sense inferences about task procedures, mapping abstract language to concrete objects — benefits from broad world knowledge more than from instruction-following fine-tuning specifically.

The chain-of-thought case study (Section 5.2, Table 4) demonstrates a different kind of NLP improvement: by adopting a prompting technique from Wei et al. (2022) that was developed for pure NLP reasoning tasks, SayCan gains the ability to handle negations ("bring me a fruit-flavored drink without caffeine") and reasoning-requiring tasks ("the snack is too small, bring me something more filling") that the base system could not solve. The chain-of-thought modification requires only changing the prompt — the LLM first generates an "Explanation:" in generative mode, then the explanation is included in the context for scoring mode. No robot components change. This demonstrates that advances in prompting techniques (not just model architecture or scale) directly benefit robotic systems under the SayCan architecture.

The multilingual capability (Appendix E.5, Table 8) makes the same point elegantly: Chinese, French, and Spanish queries achieve identical planning rates to English because the underlying LLM (PaLM) was trained on multilingual corpora. The robot's skills — their language descriptions, their value functions, their policies — are entirely English. The LLM handles the translation implicitly. No multilingual robotics engineering is required.

This finding has implications for research prioritization and resource allocation. It suggests that investment in better language models (whether through scale, training data, or prompting techniques) is directly investment in better robotic systems — as long as the architecture supports modular replacement. In a traditional end-to-end architecture where the language model is fine-tuned as part of the policy, upgrading the LLM would require retraining the entire policy on all physical interaction data. The factorization makes LLM upgrades essentially free (beyond the inference cost of the larger model). This changes the economic calculus for robotics research: when robotics and NLP improvements compound independently, both communities can specialize without the coordination overhead that joint training architectures require.

The limitation of this finding is that it has been demonstrated only on the specific task distribution (kitchen manipulation with 15 objects and 5 locations) and with two language model families (PaLM and FLAN). It remains to be seen whether the result generalizes to more diverse task domains, larger skill repertoires, or models from different architectural families (e.g., encoder-decoder vs. decoder-only). The paper's open-source tabletop implementation (Section 6) uses GPT-3 as the LLM and ViLD object detection for affordances, suggesting the architecture is model-agnostic, but no systematic LLM-comparison results are reported for that domain.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The evaluation uses 101 natural language instructions spanning 7 instruction families (Table 1). The instructions were developed from multiple sources: crowd-sourcing via Amazon Mechanical Turk, in-person requests from kitchen users, and inspiration from benchmarks such as ALFRED (Shridhar et al., 2020) and BEHAVIOR (Srivastava et al., 2022). The 7 families are: NL Single Primitive (15 instructions testing single-skill commands), NL Nouns (15 instructions with abstract noun references like "fruit" or "soda"), NL Verbs (15 instructions with abstract verbs like "restock" or "recycle"), Structured Language (15 instructions that explicitly spell out the plan, serving as an ablation of natural language understanding), Embodiment (11 instructions testing sensitivity to robot and environment state), Crowd-Sourced (15 instructions with unstructured, realistic phrasing), and Long-Horizon (15 instructions requiring 10+ step sequences). The instructions span multiple axes: time-horizon (from single primitives to multi-step sequences), language complexity (from structured commands to fully unstructured requests), and embodiment (instructions whose correct plan depends on whether the robot is already holding an object, whether objects are present, etc.).

• Base model(s). The primary LLM is PaLM 540B (Chowdhery et al., 2022), a 540-billion-parameter dense Transformer language model. Ablation experiments also test PaLM 62B, PaLM 8B, and FLAN 137B (Wei et al., 2021), which is a 137-billion-parameter model fine-tuned on an "instruction answering" dataset. The low-level robotic policies use a Universal Sentence Encoder (USE; Cer et al., 2018) for embedding skill descriptions, which is a smaller frozen language model separate from the planning LLM. PaLM was chosen as the primary model because "PaLM outperforms FLAN" on generative planning tasks (Appendix Table 6) despite FLAN being specifically fine-tuned for instruction following — the authors hypothesize that PaLM's broader and improved pre-training dataset compensates for the lack of instruction-specific fine-tuning.

• Metrics. Two primary metrics are used (Section 5). Plan success rate measures whether the sequence of skills selected by the model is correct for the instruction, regardless of whether the skills actually succeed during physical execution. Three human raters indicate whether the generated plan can achieve the instruction; if 2 out of 3 agree, the plan is marked successful. The paper notes that "for many instructions there may be multiple valid solutions" (e.g., bringing a sponge first or throwing away a can first), so raters are instructed to accept any valid ordering. Execution success rate measures whether the full SayCan system (LLM + affordance functions + skill policies) actually performs the instruction successfully on the physical robot. Three human raters watch the robot execution and answer "whether the robot achieves the task specified by the task string"; 2 out of 3 agreement marks success. Both metrics are reported per instruction family and in aggregate across all 101 tasks.

• Baselines. Five baselines are evaluated (Table 2). No VF (No Value Function): PaLM-SayCan with the affordance term removed, selecting the maximum LLM-score skill at each step. This isolates the contribution of physical grounding. Generative (Gen.): The LLM generates a plan freely (in the "Robot: I would: 1. ..., 2. ..." format), then each generated step is projected to the nearest skill in the repertoire via cosine similarity of USE embeddings. This corresponds to the approach in concurrent work by Huang et al. (2022) and tests whether scoring outperforms generative decoding with post-hoc projection. BC NL: The full high-level instruction $i$ is fed directly into the language-conditioned behavioral cloning policy — this represents standard RL or BC-based instruction following methods (Stepputtis et al., 2020; Lynch and Sermanet, 2020; Nair et al., 2021) and tests whether end-to-end policies can handle long-horizon instructions without decomposition. BC USE: The high-level instruction is projected to the nearest known command or command sequence via USE embedding cosine similarity (considering both individual skills and combinatorial sequences like "1. find coke can, 2. pick coke can"), then fed to the language-conditioned policy. This tests whether embedding-based retrieval can substitute for LLM-based planning. FLAN-SayCan: A full SayCan system using FLAN 137B as the LLM instead of PaLM, isolating the effect of language model quality on end-to-end robotic performance.

• Generation budget / compute accounting. SayCan does not use a generation budget in the sense of parallel sampling or beam width sweeping. At each decision step, the LLM scores all candidate skills (on the order of tens) — a single forward pass per skill per step. The affordance functions are evaluated once per skill per step from the current state. The total compute per instruction scales linearly with the number of planning steps (which varies from 1 for single-primitive tasks to 10+ for long-horizon tasks) times the number of skills in the repertoire. The paper does not report inference latency or FLOP counts, focusing instead on success rates as the primary efficiency metric. The practical computational cost is dominated by the LLM queries (PaLM 540B forward passes), which the paper does not quantify, though the scoring approach (evaluating log-probabilities of fixed continuations) is more efficient than autoregressive generation since it can be parallelized across candidates.

• Cross-validation / statistical protocol. No formal cross-validation or statistical significance testing is reported. The 101 evaluation instructions are fixed, and the primary comparisons are across methods (SayCan vs. baselines) and across LLMs (PaLM vs. FLAN) on this fixed test set. The paper does not report confidence intervals, standard deviations, or multiple evaluation runs per instruction. The evaluation is conducted in two physical environments (real office kitchen and mock kitchen) to test generalization across environments, but this tests spatial generalization of the skills and value functions, not statistical reliability of the planning evaluation. The human rating protocol (3 raters, 2/3 agreement) provides some robustness to individual rater variability, but the number of raters and their agreement rates are not reported beyond the threshold criterion.

Main Quantitative Results

Overall System Performance

PaLM-SayCan achieves a planning success rate of 84% and an execution success rate of 74% across all 101 instructions in the mock kitchen environment (Table 2, "PaLM-SayCan" columns). In the real kitchen (a different physical environment than the training setup), planning success drops by 3 percentage points (from 84% to 81%) and execution success drops by 14 percentage points (from 74% to 60%). The larger drop in execution compared to planning indicates that the primary degradation when transferring environments is in the reliability of the low-level skill policies, not in the LLM's planning ability — the affordance functions and policies are affected by domain shift (different lighting, background, object placement) while the LLM is environment-agnostic.

Comparison Against Ungrounded Baselines — The Necessity of Physical Grounding

The ablation of affordance functions demonstrates that physical grounding is essential for reliable planning. No VF (LLM scoring only, no affordances) achieves a planning success rate of 67% across all instructions, compared to SayCan's 84% — a 17 percentage point gap (Table 2). The Generative baseline (free-text generation with post-hoc projection) achieves a slightly better 74%, but still trails SayCan by 10 percentage points. These aggregate numbers mask substantial variation across instruction families, revealing where grounding matters most:

• Embodiment tasks: SayCan achieves 64% planning vs. No VF at 18% and Generative at 36%. This is the family where grounding is most critical — these instructions explicitly test whether the system understands the current robot and environment state (e.g., "Put the coke on the counter" when starting from different completion stages), and the 46 percentage point gap between SayCan and the No VF baseline isolates the affordance functions' contribution to state-awareness.

• NL Nouns: SayCan achieves 67% planning vs. No VF at 53% and Generative at 53%. These tasks require mapping abstract nouns ("fruit," "soda," "something hydrating") to specific objects — the LLM must know that "fruit" maps to "apple" (the only fruit in the repertoire) and that "hydrating" maps to "water bottle." The 14 percentage point gap between SayCan and the baselines is smaller than for embodiment but still meaningful, suggesting that the affordance functions help disambiguate cases where the LLM's semantic mapping is uncertain.

• NL Verbs: SayCan achieves 100% planning vs. No VF at 87% and Generative at 93%. The gap is narrowest here, indicating that for instructions where the action type is explicit ("throw away," "recycle," "restock"), the LLM's semantic knowledge is sufficient to select appropriate skills without much help from affordances.

• Single Primitives: SayCan achieves 100% planning vs. No VF at 73% and Generative at 87%. The poor performance of No VF on single primitives (e.g., "How would you hold the snickers?") is notable — it suggests that without affordances, the LLM sometimes selects a semantically appropriate but incorrect skill (e.g., "go to the snickers" instead of "pick up the snickers" if the object is not visible), and the affordance function corrects this by preferring skills that are physically feasible from the current state.

Comparison Against End-to-End Baselines — The Necessity of Task Decomposition

The BC NL baseline (feeding the full high-level instruction directly into the language-conditioned behavioral cloning policy) achieves 0% execution success rate across all instruction families, including single primitives (Table 2). This is a critical negative result: end-to-end language-conditioned policies, even when trained on a large dataset of teleoperated demonstrations (68,000 episodes over 11 months), cannot handle instructions outside their training distribution. The BC policies were trained on atomic commands like "pick up the coke can," and when presented with "How would you pick up the coke can," the model has no mechanism to decompose the query into the underlying atomic skill — a framing effect that end-to-end policies are brittle to. The BC USE baseline (projecting the instruction to the nearest known command via USE embedding) achieves 60% on single primitives but 0% on all other families, confirming that embedding similarity cannot bridge the gap between atomic training commands and multi-step or abstract instructions. These results establish that SayCan's LLM-based decomposition is not merely beneficial but necessary — without it, the robot's existing skill repertoire is inaccessible for any instruction more complex than a direct match to a training command.

Performance by Instruction Family — Where SayCan Succeeds and Where It Struggles

Breaking down PaLM-SayCan's performance by instruction family (Table 2) reveals substantial variation:

• NL Single Primitive: 100% planning, 100% execution. SayCan perfectly handles single-skill commands like "How would you pick up the coke can," "How would you let go of the coke can," and "How would you navigate to the table."

• NL Verbs: 100% planning, 93% execution. SayCan reliably maps abstract verbs ("restock," "recycle," "compost," "deliver") to appropriate skill sequences. The 7% execution gap is attributable to policy failures on individual skills, not planning errors.

• Structured Language: 93% planning, 87% execution. These instructions explicitly spell out the solution (e.g., "How would you pick up the apple and move it to the trash"), which should make planning trivial. The 93% planning rate (100% at the execution level in the mock kitchen) and 7% failure rate at the planning level suggests occasional LLM errors even when the task is fully specified — likely from prompt-engineering artifacts or object confusion in edge cases.

• Crowd-Sourced: 87% planning, 87% execution. SayCan handles unstructured requests (e.g., "I opened a pepsi earlier. How would you bring me an open can?") with performance on par with structured instruction families, demonstrating robustness to natural language variation.

• NL Nouns: 67% planning, 47% execution. This is SayCan's weakest category among non-long-horizon tasks. The large gap between planning (67%) and execution (47%) — 20 percentage points — indicates that even when SayCan selects the correct plan, the underlying skills often fail to execute successfully. The planning failures (33% rate) reflect the challenge of mapping abstract nouns to specific objects: the LLM must infer that "fruit" means "apple," "something to clean the kitchen with" means "sponge," and "something with caffeine" means "coke can" or "pepsi can." When the LLM makes incorrect semantic mappings, the plan is wrong regardless of affordance quality. The paper notes that this is "due to the number of nouns possible (15 objects and 5 locations) versus number of verbs (6)" — the semantic search space for nouns is larger and more ambiguous.

• Embodiment: 64% planning, 55% execution. These tasks test whether SayCan understands the robot's current state (e.g., starting with an object already in hand, being at the destination already) and adjusts plans accordingly. The 64% planning rate — well below the 84% aggregate — indicates that state-dependent reasoning is a weakness. The paper reports that failures in this category "generally [are] a result of affordance function misclassification" — the value functions incorrectly estimate whether a skill is feasible from the current state, leading to plans that skip necessary steps or attempt impossible ones.

• Long-Horizon: 73% planning, 47% execution. These are the most challenging tasks (10+ steps). The 73% planning rate shows that SayCan can decompose extended instructions reasonably well, but the 47% execution rate is the lowest of any family — the gap of 26 percentage points between planning and execution is the largest across all families. The paper identifies early termination by the LLM as the primary failure mode: "most failures were a result of early termination by the LLM (e.g., bringing one object but not the second)." When asked to "bring me an apple, a coke, and water bottle," the system sometimes stops after delivering one item, indicating that the LLM's scoring of "done" vs. "continue" is imperfect for multi-object instructions. The execution failures compound this: even with a correct plan, chaining 10+ skills means a multiplicative probability of success, and individual skill failures accumulate.

Error Attribution — LLM vs. Affordance Failures

The paper provides a coarse error breakdown: 65% of the errors were LLM failures and 35% were affordance failures (Section 5.1). This attribution is aggregated across all instruction families and not broken down further, making it difficult to assess whether different families have different error profiles. The high proportion of LLM failures (nearly two-thirds) suggests that improving the language model — through scale, better prompt engineering, or techniques like chain-of-thought — has more room to improve overall performance than improving the affordance functions. However, this aggregate masks the family-specific patterns: embodiment failures are primarily affordance-driven (as noted above), while long-horizon failures are primarily LLM-driven (early termination). A per-family error breakdown would have been more informative for directing future research efforts.

LLM Ablation — Language Model Quality Directly Affects Robotic Performance

The comparison between PaLM-SayCan (540B parameters) and FLAN-SayCan (137B parameters) on the full robotic system (Table 3) shows that PaLM achieves 84% planning success vs. FLAN's 70% and 74% execution success vs. FLAN's 61%. This is a reduction in errors by roughly half for planning (16% error rate for PaLM vs. 30% for FLAN) and roughly one-third for execution (26% vs. 39%). The gap is present across all instruction families, with the largest absolute differences in families where SayCan is weakest:

• Long-Horizon: PaLM 73% planning, 47% execution vs. FLAN 47% planning, 33% execution — a 26 percentage point planning gap.
• NL Nouns: PaLM 67% planning, 47% execution vs. FLAN 60% planning, 53% execution — a smaller 7 percentage point planning gap but comparable execution due to skill failures dominating.
• NL Single: PaLM 100% planning, 100% execution vs. FLAN 67% planning, 67% execution — a 33 percentage point planning gap that is notably large for simple single-skill tasks, suggesting FLAN's instruction-following fine-tuning does not compensate for its smaller size on what should be the easiest category.

The paper also reports a LLM-only generative ablation (Appendix Table 6) comparing PaLM 540B, PaLM 62B, PaLM 8B, and FLAN 137B on planning with generative output and USE projection. The results show monotonic improvement with scale — PaLM 8B achieves 38% total planning, PaLM 62B achieves 72%, PaLM 540B achieves 74% — with the gap between 62B and 540B being surprisingly small (2 percentage points). FLAN achieves 43%, which is better than PaLM 8B but substantially worse than PaLM 62B and 540B. The paper notes that "the difference between the 62B and 540B model is small" in this setup but references chain-of-thought prompting results suggesting "this difference may be more pronounced on more challenging problems." The 540B model's advantage may be partially masked by the USE projection step, which discards some of the plan quality differences by mapping everything to the nearest in-repertoire skill.

Environment Transfer — Real Kitchen vs. Mock Kitchen

In the real kitchen (a different physical environment from the mock kitchen where skills were trained), PaLM-SayCan achieves 81% planning success and 60% execution success (Table 2, "Kitchen" columns). The planning degradation is minimal (84% → 81%, a 3 percentage point drop), confirming that the LLM's semantic reasoning is environment-independent and that the affordance functions generalize reasonably well to a new visual setting. The execution degradation is substantial (74% → 60%, a 14 percentage point drop), attributable to the skill policies and value functions experiencing domain shift — different lighting, camera angles, background clutter, and object placement distributions between the mock kitchen (training environment) and the real kitchen (deployment environment). The paper does not break down execution failures by skill type or attempt to isolate which specific skills degrade most in the new environment, but the overall pattern suggests that the modular architecture isolates the domain shift problem to the vision-based components rather than propagating it to the planning layer.

Ablation Studies and Robustness Checks

Number of prompt engineering examples: In a language-based simulator (Appendix D.3, Table 5), increasing the number of examples in the prompt from 0 to 17 increases planning success from 10% to 88% when termination is required. With 0 examples but no termination requirement, success is 52%, indicating that the pre-trained LLM already encodes some task decomposition knowledge but cannot reliably produce the specific "numbered steps ending with 'done'" format without demonstration. The jump from 10% (0 examples) to 64% (1 example) shows that a single example provides most of the format-learning benefit, while subsequent examples refine content knowledge — the further increase to 82% (4 examples) and 88% (17 examples) reflects improved task understanding rather than format adherence.

LLM size scaling (generative evaluation only, no affordances): Across 101 instructions (Appendix Table 6), PaLM 8B achieves 38% total planning, PaLM 62B achieves 72%, PaLM 540B achieves 74%. The gap between 62B and 540B is only 2 percentage points in aggregate, but this evaluation uses generative output with USE projection, which may mask differences — projecting generated plans to the nearest skill via embedding similarity can correct some LLM errors (e.g., if the LLM says "pick up the cola" when the skill is "pick up the coke can," the embedding might still map correctly). The paper notes that on long-horizon tasks specifically, PaLM 540B (60%) underperforms PaLM 62B (73%) in this evaluation, which is counterintuitive and likely reflects noise or an idiosyncrasy of the projection mechanism rather than genuine reversal of the scaling trend.

Drawer manipulation skills integration (Appendix E.3, Table 7): Adding drawer manipulation skills required adding only the skill descriptions to the candidate set, providing accompanying value functions, and adding 3 example prompts showing drawer-related plans. Over 21 drawer-related queries, planning success was 100% but execution success was only 33%, with failures primarily due to the manipulation policy failing to open the drawer wide enough to place objects or failing to close it completely. No loss in performance was observed for other instruction families, confirming that SayCan's modular architecture supports adding new skills without degrading existing capabilities.

Chain-of-thought prompting for negations and reasoning (Section 5.2, Table 4): Modifying the prompt to include an "Explanation:" line before the plan (generated first in generative mode, then included in scoring context) enables SayCan to handle tasks requiring reasoning that the base system fails on. Examples include "bring me a fruit-flavored drink without caffeine" (the model reasons: "The user has asked for a drink that is fruit-flavored and does not have caffeine, I will bring the lime soda") — successfully handling negation — and "the snack is too small, bring me something more filling" (the model reasons about relative properties of snacks). No quantitative accuracy on these reasoning tasks is reported; the results are presented as qualitative case studies demonstrating capability extension via prompt modification alone.

Multilingual queries (Appendix E.5, Table 8): Translating English instructions to Chinese, French, and Spanish produces "almost no performance drop in planning success rate." For example, "bring me a can of coke" and its Chinese translation both achieve 1.0 (100%) plan success; "throw away the coke can" and its translations all achieve 1.0. One failure is noted: the French translation of "I spilled my coke, can you bring me something to help clean" achieves 0.0 planning success, while English, Chinese, and Spanish translations all achieve 1.0 — likely an artifact of the French translation's specific phrasing rather than a systematic multilingual failure. This ablation demonstrates that the LLM's multilingual training data provides zero-shot language transfer with no changes to the English-only skill descriptions or policies.

Critical Assessment

Claim 1: "SayCan nearly doubles performance over non-grounded baselines."

This claim requires careful interpretation of what "performance" means. The aggregate planning success comparison (Table 2) shows SayCan at 84% vs. the No VF baseline at 67% and Generative at 74%. This is a substantial improvement — 17 percentage points over No VF, 10 percentage points over Generative — but calling it "nearly doubling" only makes sense if one frames it in terms of error rate: SayCan's 16% planning error rate is roughly half of No VF's 33% error rate and less than two-thirds of Generative's 26% error rate. The "doubling" framing is most accurate for the embodiment family, where SayCan achieves 64% vs. No VF at 18% — a 3.6× improvement — and for the specific head-to-head where PaLM-SayCan (84% planning) is compared to FLAN-SayCan (70% planning), where error rate drops from 30% to 16% (roughly halved). The claim is qualitatively supported by the data but the exact magnitude depends on which baseline and which metric (planning vs. execution, aggregate vs. per-family) is chosen.

What was not tested: The "Generative" baseline uses USE embedding projection to map generated text to skills, which can mask LLM generation errors — a generated plan that says "pick up the cola" might correctly project to "pick up the coke can" if the embedding is close enough. A stricter generative baseline that requires exact string matching to skill descriptions would likely perform worse and make SayCan's advantage larger. The paper does not report how often the Generative baseline produces malformed or unparseable output, only that the projection handles it.

Claim 2: "A robot's performance can be improved simply by enhancing the underlying language model."

This claim is strongly supported by the PaLM vs. FLAN comparison (Table 3: 84% vs. 70% planning, 74% vs. 61% execution) but with an important caveat: this is demonstrated for exactly two models, one substantially larger than the other (540B vs. 137B parameters), and the comparison conflates model scale, training data, and architecture. The PaLM 62B vs. 540B comparison in Appendix Table 6 shows near-identical generative performance (72% vs. 74%), which could be interpreted as diminishing returns from scale — or could be an artifact of the USE projection step masking differences. The claim that "improvement in language models translates to a similar improvement in robotics" would be more robustly supported if the evaluation included models at multiple points along a scaling curve and showed monotonic improvement in execution success (not just generative planning) across scales. The current evidence shows that going from a smaller, fine-tuned model (FLAN 137B) to a larger, more broadly trained model (PaLM 540B) improves performance, but does not isolate whether this is due to scale, training data diversity, architectural differences, or fine-tuning strategy.

What would strengthen this claim: A scaling curve showing robotic execution success as a function of LLM size for models from the same family (e.g., PaLM 8B, 62B, 540B all evaluated with full SayCan, not just generative evaluation). The paper only reports full SayCan results for PaLM 540B and FLAN 137B; the PaLM 8B and 62B are only evaluated in generative mode (Appendix Table 6), which does not test the full system. This is a significant gap — without execution results across scales, the claim that "improvement in language models translates to a similar improvement in robotics" is supported for one comparison point, not a trend.

Claim 3: "SayCan handles temporally-extended, abstract instructions in a zero-shot fashion."

The evidence for zero-shot generalization is in the architecture design (frozen LLM, no task-specific fine-tuning) and the diverse instruction families (crowd-sourced, abstract nouns, abstract verbs). The 84% planning success rate across 101 instructions with no task-specific training supports the zero-shot claim. However, the "zero-shot" nature has a boundary: the prompt contains 17 carefully constructed examples that span many of the task types encountered at evaluation time (multi-object delivery, throwing away items, bringing items to the user, etc.), which can be viewed as providing explicit demonstrations of the expected plan structures. The paper is transparent about this — the prompt engineering ablation (Table 5) shows that 0 examples yields only 10% planning success — but it means "zero-shot" refers to no fine-tuning or gradient updates, not to no task-specific examples. The 17 examples in the prompt serve a role analogous to a few-shot training set, albeit one that is processed entirely through the LLM's in-context learning rather than parameter updates.

What was not tested: The paper does not evaluate whether the prompt examples, which are fixed and include specific objects (coke cans, apples, sponges), bias the system toward those objects or plan structures. A systematic test of whether the system can generalize to completely novel object-referring expressions or task structures not represented in the prompt examples would strengthen the zero-shot claim.

Claim 4: "The primary bottleneck is in the range and capabilities of the underlying skills."

The paper states this in the limitations section (Section 8) and supports it partially through the execution-vs-planning gap: execution success (74%) trails planning success (84%) by 10 percentage points, indicating that a correct plan does not guarantee correct execution. The long-horizon family shows the largest gap (73% planning vs. 47% execution, a 26 percentage point difference). The drawer manipulation case study (100% planning, 33% execution) provides a specific example where planning is near-perfect but physical execution fails. However, the error attribution (65% LLM failures, 35% affordance failures) actually suggests the opposite — that the LLM is the larger source of errors than the affordance functions (which directly reflect skill capabilities). The paper does not separate execution failures into "skill policy errors" vs. "affordance mispredictions," which are different types of bottlenecks. A skill can be physically capable of succeeding (high affordance at execution time) but still fail due to stochastic policy execution, while an affordance function can incorrectly predict low probability for a skill that would have succeeded. These failure modes have different implications for whether the bottleneck is skill capability or skill modeling.

What would strengthen this analysis: A breakdown of execution failures into: (a) failures where the plan was correct but individual skills failed to execute successfully, (b) failures where the plan was incorrect (LLM or affordance error), and (c) failures where the plan was correct but the state changed between planning and execution (e.g., an object was knocked over during navigation). The current error reporting conflates planning and execution failures in ways that make it difficult to precisely identify the system's weakest component.

Additional Weaknesses in the Experimental Design

Small evaluation set without statistical rigor. The 101 instructions are divided into 7 families, some with as few as 11 instructions (Embodiment). With no confidence intervals or repeated trials, differences of 10–20 percentage points between methods or families could be partially attributable to the specific instructions chosen. For the long-horizon family (15 instructions), SayCan's 73% planning rate means 11 out of 15 instructions were planned correctly — a difference of 1 instruction changes the rate by nearly 7 percentage points. The human rating protocol (3 raters, 2/3 agreement) adds robustness to individual rater variability but does not address test-set variability.

Absence of latency and real-time performance data. The paper provides no measurements of how long SayCan takes to plan each step or execute each instruction. For a system intended for real-world human-robot interaction, the time between instruction and action completion matters. The iterative LLM querying (one forward pass per skill per planning step) could introduce noticeable latency, especially with a 540B-parameter model. The paper's silence on this metric makes it impossible to assess whether the system's performance levels are achievable at interactive speeds.

Object location assumption limits generality claims. The paper assumes known object locations (navigation skills map directly to coordinates), which substantially simplifies the affordance computation. In a realistic deployment where objects must be visually detected and localized, the "go to" skills would have much less reliable affordances, and the planning system would need to handle cases where objects are not at expected locations. The paper acknowledges this limitation but does not test how performance degrades when object locations are uncertain.

No systematic test of the independence assumption. The core probabilistic factorization $p(c_i \mid i, s, \ell_\pi) \propto p(c_\pi \mid s, \ell_\pi) \cdot p(\ell_\pi \mid i)$ assumes conditional independence between semantic appropriateness and physical feasibility. No experiment tests this assumption directly — for example, by comparing the product to a jointly-trained model that could capture correlations, or by analyzing failure cases where both the LLM score and the affordance score are systematically wrong in the same direction. The 16% planning failure rate might be partially attributable to independence violations rather than component errors, but this cannot be assessed from the reported data.

The chain-of-thought and multilingual results are qualitative, not quantitative. The chain-of-thought case study (Table 4) reports 3 successful examples without quantifying success rate across a test set of reasoning-requiring instructions. The multilingual results (Table 8) report 12 instructions (3 English, 3 Chinese, 3 French, 3 Spanish) without stating whether these are the only ones tested or a representative sample. These case studies demonstrate capability extension but do not provide the quantitative rigor needed to assess how broadly the extensions work.

6. Limitations and Trade-offs

Limitation 1: Difficulty Estimation Requires a Full Inference Pass on the Entire Candidate Set

The assumption or constraint. SayCan's scoring interface requires the LLM to evaluate the log-probability of every candidate skill description at each decision step. The paper treats this as computationally unremarkable and reports no latency, throughput, or FLOP measurements. The only discussion of computational cost appears in Appendix D.3, where the authors describe a "language-based simulator" built specifically because "as SayCan as a whole requires affordances from a world embodiment, it is not straightforward to optimize this structure and tune parameters quickly" — acknowledging that direct experimentation on the physical system is expensive, but never quantifying the inference cost of the LLM scoring itself.

The consequence. For a robot with hundreds of skills (the paper defines 551 skills across 7 families and 17 objects, though only a subset are used in experiments), scoring every skill at every decision step means that each planning step requires hundreds of forward passes through a 540B-parameter model like PaLM. While the paper notes that scoring can be parallelized across candidates (since each skill's continuation string is independent), this parallelization requires either batch inference infrastructure or sequential execution on a single accelerator — and even with batching, a 540B-parameter model's forward pass latency is nontrivial for interactive human-robot interaction. For long-horizon tasks requiring 10+ planning steps, the cumulative inference time could make the system impractical for real-time deployment. Furthermore, the scoring cost scales linearly with both the number of planning steps and the number of candidate skills, meaning that as the robot's skill repertoire grows (more objects, more action types, more locations), the per-decision latency grows proportionally — a scaling behavior that the paper does not discuss or measure.

What evidence exists in the paper. No latency, throughput, or computational cost measurements are reported anywhere in the paper or appendices. The hardware specifications for LLM inference are not disclosed. The number of skills used in the 101-instruction evaluation is not explicitly stated, but can be inferred from the skill families: roughly 15 pick skills (one per object), approximately 5–7 go-to skills (one per known location), plus place and terminate skills — on the order of 20–30 candidate skills per decision step. At this scale, the scoring cost is manageable, but the paper provides no evidence about how this cost scales or what the actual wall-clock time is for a typical SayCan planning episode. The open-source tabletop implementation (Section 6) uses GPT-3 as the LLM, but similarly reports no latency data.

Mitigation status. Not addressed. The paper acknowledges neither the computational cost of LLM scoring nor its scaling behavior as a limitation. Potential mitigations — hierarchical scoring (first select action type, then object), using a smaller LLM for scoring and a larger one only for generative reasoning, caching LLM scores for skills whose semantic appropriateness is unlikely to change between steps — are not discussed. The modular architecture means the LLM could in principle be replaced with a smaller, faster model for scoring, but the ablation results (Appendix Table 6) show that smaller models (PaLM 8B: 38% planning) perform substantially worse on the generative evaluation, and no scoring-mode results for smaller models are reported.

Limitation 2: Hard Problems — The System Cannot Handle Instructions Requiring Novel Reasoning Beyond the LLM's Knowledge

The assumption or constraint. SayCan treats the LLM as a frozen knowledge source and queries it for skill probabilities via prompt engineering. This means the system can only select plans that the LLM's pre-trained knowledge, combined with the prompt examples, supports. The paper explicitly acknowledges that the system inherits the LLM's limitations:

"First, we expect this method to inherit the limitations and biases of LLMs, including the dependence on the training data." (Section 8)

The paper also reports specific failure modes: struggles with negation (e.g., "bring me a snack that isn't an apple") and ambiguous references (e.g., "asking for drinks with caffeine") are noted as "a known issue inherited from underlying language models" (Section 5.1). The chain-of-thought case study (Section 5.2, Table 4) demonstrates a partial mitigation — adding an "Explanation:" step to the prompt enables solving some negation and reasoning tasks — but this is presented as a qualitative demonstration with no quantitative success rate reported.

The consequence. For instructions that require the LLM to reason about novel properties, relationships, or constraints not well-represented in its training data, SayCan will produce incorrect plans. The long-horizon instruction family (Table 2) shows a 73% planning success rate — the lowest of any non-embodiment family — with the paper explicitly attributing most failures to "early termination by the LLM (e.g., bringing one object but not the second)." This failure mode reveals a deeper issue: the LLM's scoring interface treats each candidate skill independently at each step, but knowing whether to continue or terminate a multi-object instruction requires reasoning about set completion ("have I satisfied all parts of the instruction?"), which the current factored architecture does not explicitly support. The chain-of-thought extension partially addresses this by having the LLM generate an explanation before scoring, but this doubles the LLM inference cost (generate, then score) and still relies on the LLM's ability to reason correctly about set completion — a capability that is unreliable, as the 73% long-horizon planning rate shows.

The paper does not report per-object or per-constraint accuracy breakdowns that would reveal which specific reasoning types most often fail. The 101-instruction test set includes instructions requiring negation ("something that is not a fruit from the close counter"), comparative reasoning ("something more filling"), exclusion ("two different sodas"), and causal reasoning ("recover from a workout" → healthy items), but success rates for these specific reasoning subtypes are not disaggregated from the family-level aggregates.

What evidence exists in the paper. The planning success rates per instruction family (Table 2) provide indirect evidence: NL Nouns (67%) and Embodiment (64%) are the weakest non-long-horizon families, and these are precisely the families most dependent on semantic reasoning (mapping abstract nouns to objects, understanding state-dependent constraints). The explicit discussion of negation failures (Section 5.1) and the chain-of-thought case study (Table 4) demonstrate both the existence of the limitation and a partial mitigation. The paper does not report which specific instructions within each family failed, preventing analysis of whether failures cluster on particular reasoning types.

Mitigation status. Partially addressed via the chain-of-thought prompting extension (Section 5.2), which modifies the prompt to include an "Explanation:" field that the LLM generates before scoring skills. This enables the system to handle negations and reasoning tasks that the base system cannot, but the extension is only demonstrated qualitatively (3 successful examples in Table 4) with no quantitative evaluation on a test set of reasoning-requiring instructions. The paper frames chain-of-thought as a case study demonstrating capability extension, not as a fully evaluated mitigation. The fundamental limitation — that the system is bounded by the LLM's reasoning capabilities and cannot acquire new reasoning skills from interaction — remains.

Limitation 3: Object Location Knowledge Is Assumed, Not Learned

The assumption or constraint. SayCan assumes that object locations are known. The paper states this explicitly:

"Since the focus of this work is mainly on planning, we assume the location of objects are known. Thus any navigation skill maps to the coordinate of the object with a classical planning-based navigation stack." (Appendix D.2)

This means the go-to affordances are purely distance-based: $p_{\text{affordance}}^{\text{goto}} = \text{clamp}((d_{\text{max}} - d) / (d_{\text{max}} - d_{\text{min}}), 0, 1)$, where $d$ is the Euclidean distance from the robot to the known object coordinates. The robot does not need to visually search for objects, and the affordance does not incorporate perceptual uncertainty about whether the object is actually at the assumed location.

The consequence. In any realistic deployment where object locations are not instrumented a priori — a home, an unfamiliar office, a dynamic environment where humans move objects — the go-to affordances would be meaningless. The system would need to search for objects, detect them visually, and estimate their pose, all of which introduce substantial uncertainty that the current affordance formulation does not capture. A distance-based affordance that reports high probability for "go to the apple" because the robot is close to where the apple was last seen would be dangerously miscalibrated if the apple has been moved.

This assumption also limits the claimed generality of the approach. The paper presents SayCan as a method for grounding LLMs in physical affordances, but a core component of physical grounding — knowing where objects are — is provided by external instrumentation rather than learned from sensor data. The pick affordances (learned RL value functions) do incorporate visual perception for determining whether an object is reachable from the current state, but the navigation component that brings the robot to the right general location relies entirely on known coordinates. This creates an asymmetry in the grounding: the manipulation side is visually grounded, while the navigation side is not.

What evidence exists in the paper. The assumption is stated clearly in Appendix D.2 but is not discussed in the main paper's limitations section (Section 8). The evaluation environments (mock kitchen and real kitchen) both have fixed, known object locations, so all reported results depend on this assumption. The environment transfer results (Table 2: 84% planning in mock kitchen vs. 81% in real kitchen) show that planning degrades only slightly when the physical environment changes — but this test changes the visual appearance and layout, not the object-location knowledge, so the assumption's robustness to location uncertainty is not tested. No experiment tests SayCan in an environment where object locations are unknown or must be visually detected.

Mitigation status. Not addressed, though the paper gestures toward it indirectly. The open-source tabletop implementation (Section 6) replaces the learned value functions with a ViLD object detector as the affordance source, which does not assume known locations — the detector finds objects in the camera image. However, this is a simplified tabletop domain without navigation, so the object-location problem is fundamentally different. The paper does not discuss how the known-location assumption could be relaxed in the full mobile manipulation setting, nor does it suggest that integrating visual object search into the affordance functions is a priority for future work. The follow-up work by Huang et al. (2022, cited as "Inner Monologue") builds on SayCan to incorporate environment feedback including scene descriptors, which could address location uncertainty, but this is not evaluated in the current paper.

Limitation 4: The System Cannot Recover from Skill Execution Failures

The assumption or constraint. SayCan's planning loop (Algorithm 1) selects a skill, executes it, observes the new state, and replans. However, the replanning only accounts for successful execution — the affordance of a successfully completed skill drops (the skill is no longer needed, so its score falls), and the LLM's history is updated to reflect that the step was completed. If a skill fails — the robot drops the object, fails to grasp, or navigates to the wrong location — the system has no mechanism to detect the failure and retry. The paper acknowledges this explicitly:

"At the current stage, the system is not easily able to react to situations where individual skills fail despite reporting a high value, though this could potentially be addressed by appropriate prompting of the language model for a correction." (Section 8)

The consequence. The gap between planning success (84%) and execution success (74%) in the mock kitchen — and the larger gap in the real kitchen (81% planning vs. 60% execution) — indicates that execution failures are common. In the current system, an execution failure is unrecoverable: the LLM believes the skill was completed (because it was selected and executed) and plans the next step accordingly, producing a plan that may be nonsensical given the true (unobserved) state. For example, if the robot attempts to pick up a coke can but the grasp fails, the LLM will next plan to "bring it to you" under the false belief that the can is in the gripper, and the robot will navigate to the user with an empty hand. The instruction will be scored as an execution failure by human raters (correctly), but the failure mode is a compound of undetected skill failure and subsequent plan invalidation — the system has no opportunity to recover.

This limitation is particularly severe for long-horizon tasks, where the probability of at least one skill failure accumulates with plan length. If each skill has a 95% success rate, a 10-step plan has roughly a 40% chance of at least one failure — and every failure is unrecoverable. The long-horizon execution success rate of 47% (Table 2) reflects both planning errors and unrecovered skill failures, but the paper does not decompose this rate into "correct plan, skill failure" vs. "incorrect plan" vs. "correct plan, skill failure causes downstream plan invalidation."

What evidence exists in the paper. The planning-to-execution gap provides indirect evidence: in the mock kitchen, 84% of plans are correct but only 74% execute successfully, meaning 10% of instructions fail despite correct plans. In the real kitchen, 81% of plans are correct but only 60% execute successfully — a 21 percentage point gap, with skill failures and domain shift as the dominant causes. The paper identifies "manipulation failures" as the primary cause of the drawer case study's 33% execution rate (Table 7), explicitly noting that "the robot fails to open the drawer wide enough to put objects in it, or fails to completely close the drawer." The system does not retry the drawer opening; it proceeds as if it succeeded, and the subsequent put-in-drawer step fails because the drawer is not actually open. The failure analysis notes that "65% of the errors were LLM failures and 35% were affordance failures" (Section 5.1), but this attribution only covers planning errors — it does not address execution failures that occur despite correct plans.

Mitigation status. The paper gestures toward a solution — "appropriate prompting of the language model for a correction" — and cites the concurrent work "Inner Monologue" (Huang et al., 2022) which builds on SayCan to incorporate environment feedback through natural language. However, the current paper provides no mechanism for failure detection (e.g., success classifiers, visual change detection, force sensing), no mechanism for communicating failure to the LLM (e.g., appending "the previous step failed" to the history), and no evaluation of whether the LLM could generate appropriate recovery plans if it knew about failures. The termination affordance (0.1) provides a weak safety valve — if all other skills have near-zero affordances (because the robot is in an unexpected state after a failure), the system will eventually terminate — but this is a graceful degradation, not a recovery mechanism. The limitation means that the 74% execution success rate is an upper bound on what the current architecture can achieve without adding failure detection and recovery capabilities.

Limitation 5: Prompt Engineering as a Hidden Task-Specific Training Signal

The assumption or constraint. SayCan uses a fixed prompt containing 17 examples of human-robot dialog (Appendix D.3, Listing 1). The paper treats these examples as a form of zero-shot or few-shot conditioning rather than as task-specific engineering. However, the prompt engineering ablation (Appendix D.3, Table 5) shows that the system's performance is heavily dependent on these examples: with 0 examples, planning success is 10% (with termination) or 52% (without termination requirement); with 4 examples, it jumps to 82%; with 17 examples, it reaches 88%. The paper notes specific design choices that mattered: explicit numbering, line breaks between steps, careful object naming, dialog framing, and avoiding over-representation of specific objects in examples.

The consequence. The prompt serves as a de facto training set — it specifies the plan format, the available skills and objects, the expected level of abstraction, and typical plan structures. The 17 examples were iteratively refined using a "language-based simulator" specifically built for prompt optimization (Appendix D.3), which means the prompt was tuned against a proxy for the evaluation tasks. This is not fundamentally different from training a small model on a handful of examples, except that the "training" happens through in-context learning rather than parameter updates. The practical consequence is that deploying SayCan in a new domain (different robot, different skills, different objects, different typical task structures) would require a similar prompt engineering effort — constructing examples, testing in simulation, iterating on format and content — which may be as labor-intensive as collecting a small training dataset. The paper's headline claim of "zero-shot" instruction following (Section 7, in the related work discussion) should be understood as zero additional parameter updates, not zero task-specific engineering.

The specificity of the prompt also raises questions about generalization. The examples use specific objects (coke can, apple, sponge, mountain dew, jalapeno chips) that overlap with the evaluation objects. The evaluation instructions use exactly these objects — "bring me a coke," "throw away the apple," "find a sponge" — meaning the prompt examples may be providing the LLM with explicit object-to-skill mappings that it would not infer from its pre-training alone. If the evaluation had used entirely non-overlapping objects (e.g., "bring me a tennis ball" or "throw away the newspaper"), the LLM's ability to generalize would depend on whether its pre-training contained sufficient common-sense knowledge about those objects in kitchen or robot contexts — a property that the paper does not evaluate.

What evidence exists in the paper. The prompt engineering ablation (Appendix D.3, Table 5) directly demonstrates the dependence on example count and quality. The paper also reports (Appendix D.3) that "examples which overly include objects used in the actual planning tend to bias results to those objects," confirming that the prompt examples influence scoring in ways that could harm generalization to novel objects. The 101-instruction evaluation does not include instructions with objects outside the prompt examples' vocabulary, so this bias is not measured. The paper reports a total of 17 examples were used, and the iterative refinement process that produced them is described in Appendix D.3, but the process itself (how many iterations, what failure modes were discovered and corrected, what alternative formats were tested and rejected) is not detailed.

Mitigation status. Not addressed as a limitation, though the paper is transparent about the prompt's content (Listing 1) and the ablation results (Table 5). The paper does not discuss the transferability of the prompt to new domains, the cost of prompt engineering for a new robot platform, or the sensitivity of results to prompt variations beyond example count. The fact that the chain-of-thought variant required a different prompt structure (adding "Explanation:" lines) further demonstrates that prompt design is non-trivial and capability-specific. Future work on automated prompt optimization or meta-learning of prompt formats could mitigate this, but the current system requires manual, domain-specific prompt construction.

Limitation 6: Single Environment, Single Robot, Single Task Domain

The assumption or constraint. All experiments are conducted in office kitchen environments with a single mobile manipulator platform (Everyday Robots) performing tasks involving 15 known objects and 5 known locations. The 101 evaluation instructions all fall within the domain of kitchen fetch-and-deliver, cleanup, and restocking tasks. The paper does not test SayCan on a different robot platform, in a different environment type (e.g., warehouse, hospital, home), or on a different category of tasks (e.g., assembly, inspection, social interaction). The paper states this scope explicitly in Section 4:

"We propose 551 skills that span seven skill families and 17 objects, which include picking, placing and rearranging objects, opening and closing drawers, navigating to various locations, and placing objects in a specific configurations. In this study we utilize the skills that are most amenable to more complex behaviors via composition and planning as well as those that have high performance at the current stage of data collection."

The consequence. The generality of the SayCan architecture — the claim that it provides a general-purpose interface between language models and robotic affordances — is supported only within this narrow domain. Several design choices that work well for kitchen manipulation might not transfer. The object-location assumption (Limitation 3) is more plausible in a known office kitchen than in a dynamic home. The skill repertoire structure (pick, go-to, place, terminate) maps cleanly to fetch-and-deliver tasks but might not capture the action space for assembly (insert, align, screw), inspection (scan, measure, record), or social tasks (gesture, follow, hand-over). The LLM's pre-training likely contains substantially more knowledge about kitchen tasks than about specialized industrial or medical tasks, so the semantic reasoning capability demonstrated here may not extend to domains where the LLM has less relevant training data.

The evaluation's small scale (101 instructions, 500 total human ratings across 3 raters per instruction) means that per-family success rates have wide confidence intervals — for the 11-instruction Embodiment family, a single instruction changes the success rate by 9 percentage points. The paper does not provide statistical error bounds, making it difficult to assess whether the observed differences between methods and families are reliable.

What evidence exists in the paper. The environment transfer test (mock kitchen → real kitchen) provides the only evidence about domain generalization, and it tests only visual domain shift (different lighting, background, camera placement) within the same task category. The planning rate drops only slightly (84% → 81%), suggesting the LLM component is robust to visual environment changes, while the execution rate drops substantially (74% → 60%), indicating that the learned skill policies and value functions are sensitive to visual distribution shift. The open-source tabletop implementation (Section 6) demonstrates the architecture in a different domain (colored blocks and bowls on a tabletop with a UR5 robot), but uses a different affordance source (ViLD object detector) and a different LLM (GPT-3), and reports no quantitative results — it serves as a proof-of-concept for architectural generality but not as experimental evidence.

The paper does not discuss how the choice of objects (15 common kitchen items) and locations (5 semantically meaningful areas) affects the LLM's performance. It is plausible that PaLM has more pre-training knowledge about coke cans and apples than about specialized tools or domain-specific objects, and that the LLM's semantic reasoning capability is correlated with the frequency of relevant concepts in its training data — but this hypothesis is untestable without experiments in different domains.

Mitigation status. Not addressed. The limitations section (Section 8) does not mention domain or platform generalization as a limitation, focusing instead on the LLM's biases, skill repertoire breadth, and failure recovery. The paper's contribution is presented as a general architecture, but the experimental validation is entirely within a single domain. Future work replicating SayCan on different robots, in different environments, and with different task categories is necessary to determine whether the architecture's claimed generality holds in practice. The open-source tabletop implementation provides a starting point for such replication, but the lack of quantitative results means it does not serve as validation.

7. Implications and Future Directions

How This Work Changes the Landscape

SayCan represents a conceptual reframing rather than a paradigm shift — it does not invent new methods for language modeling, reinforcement learning, or robot control, but it demonstrates that the interface between these components is itself a first-class design problem whose solution unlocks capabilities that no individual component achieves alone. The paper's core reframing is to treat the connection between language models and robotic skills as a stateless probabilistic product — $p(c_\pi \mid s, \ell_\pi) \cdot p(\ell_\pi \mid i)$ — which requires no joint training, no shared representations, and no parameter updates to either component. Before SayCan, the dominant mental models for connecting language to robot behavior implicitly assumed that tight coupling was necessary: either the language model needed to see sensor data (multi-modal grounding), or the policy needed to be trained on language embeddings end-to-end (representation transfer), or the plan needed to be generated as free text and post-processed (prompt engineering). SayCan shows that a much looser coupling — two independent probability distributions multiplied together at decision time — is sufficient for zero-shot generalization across 101 real-world instructions spanning 7 families of linguistic and task variation.

This reframing has immediate consequences for how the field allocates research effort. It makes improving language models and improving robotic skills into independent investment tracks that compound rather than requiring coordination. The paper's most striking empirical result in this direction is Table 3: swapping PaLM 540B for FLAN 137B reduces planning errors by roughly half (16% error rate vs. 30%) with zero changes to the robot — no retraining, no recalibration, no new data collection. This means that NLP advances in model scale, training data diversity, prompting techniques, and reasoning capabilities translate directly to robotic system improvements as long as the architecture supports modular replacement. Conversely, improvements to the robot's skill policies — better grasping, more reliable navigation, new manipulation capabilities — automatically raise the ceiling on what SayCan can accomplish because the LLM can immediately incorporate new skill descriptions into its planning without any retraining. The drawer manipulation case study (Section 5.2: 100% planning on new tasks by adding 3 prompt examples and new skill options) demonstrates this in the other direction.

The paper also reconciles a tension that had been brewing implicitly between two camps: those who argued that language models contain sufficient world knowledge for robotic planning (the prompt-engineering-only approach of concurrent work like Huang et al., 2022) and those who argued that physical grounding requires joint training on sensorimotor data (the end-to-end language-conditioned policy approach). SayCan shows that both camps were partially right and partially wrong. The language model does contain remarkably useful procedural knowledge — the LLM knows that cleaning a spill involves finding a sponge, that "recover from a workout" suggests water and fruit rather than soda and chips, that you must pick up an object before you can bring it to someone. The paper's prompt engineering ablation shows that even with 0 examples, the LLM achieves 52% planning success when not required to terminate, confirming substantial latent task knowledge. But the LLM's knowledge is incomplete without physical grounding — the No VF baseline achieves only 67% planning overall and a catastrophic 18% on embodiment tasks where state awareness matters. The end-to-end alternative is also shown to be insufficient: BC NL achieves 0% across the board because policies trained on atomic commands cannot decompose abstract instructions. SayCan's contribution is not to declare either side correct but to show that the right factorization makes both approaches complementary — the LLM provides the semantic reasoning that end-to-end policies lack, and the value functions provide the physical grounding that ungrounded LLMs lack, and the multiplicative interface combines them with no joint training burden.

Perhaps the deepest implication is that SayCan establishes value functions as the canonical grounding interface between language and robotics. The paper shows that value functions provide exactly the right abstraction: a single scalar per skill that collapses complex visuomotor feasibility into "can this be done from here?" without requiring the LLM to understand images, depth maps, robot kinematics, or grasp planning. This is a specific and consequential design choice. Object detectors would provide only presence/absence information, not action feasibility given the robot's configuration. Success classifiers would provide only discriminative signal, not the sequential value propagation that TD learning enables. Explicit world models would require hand-specified preconditions and effects that do not scale to open-vocabulary objects. Value functions, trained at scale via RL with sparse rewards, automatically integrate perception, motor control, and task dynamics into a single decision-relevant probability — and as the paper demonstrates, they can be calibrated into well-behaved affordance scores with simple min-max normalization. This positions learned value functions, not perception modules or symbolic planners, as the natural complement to semantic reasoning in embodied AI systems.

However, the paper also narrows the scope of what we should expect language models to contribute to robotics. The 73% planning rate on long-horizon tasks and the explicit attribution of 65% of errors to LLM failures makes clear that language models are not reliable planners, even for tasks well within their pre-training distribution. The LLM's role in SayCan is not to generate complete plans autonomously — it scores candidate next steps, and the environment (through affordances) and the iterative replanning loop provide correction. This is a more modest but more realistic vision for LLMs in robotics than the "LLM as autonomous task planner" framing that was emerging in concurrent work. SayCan suggests that the right architecture is a tight loop between semantic proposal and physical verification, not a one-shot plan generation followed by execution.

Follow-Up Research This Work Enables

Closed-loop failure recovery through language-based environment feedback. The paper explicitly identifies the inability to recover from skill execution failures as a key limitation, and the concurrent work "Inner Monologue" (Huang et al., 2022, cited in Section 5.2) builds directly on SayCan to address this. A concrete follow-up experiment would instrument SayCan with per-skill success detectors (trained as binary classifiers on the same teleoperated demonstration data used for BC training), then modify the planning loop to append a natural language status message to the LLM's context after each execution attempt — e.g., "The robot tried to pick up the coke can but the grasp failed" or "The robot successfully picked up the coke can." The key measurement would be execution success rate on the 101-instruction benchmark with failure recovery enabled vs. the current 74% baseline, with a specific focus on whether long-horizon tasks (currently 47% execution) benefit disproportionately since they have the most accumulated failure risk. A strong result would show that the 47% long-horizon execution rate rises substantially when the system can retry failed skills rather than proceeding blindly. A negative result — failure recovery helps minimally because the LLM does not know how to generate appropriate recovery plans — would reveal a fundamental limitation of the frozen-LLM approach and suggest that recovery planning requires fine-tuning on interaction data.

Scaling laws for LLM-driven robotic planning. The paper provides exactly one head-to-head comparison of full SayCan systems with different LLMs (PaLM 540B vs. FLAN 137B, Table 3), and the PaLM 8B and 62B models are evaluated only in generative mode without affordances (Appendix Table 6). A natural follow-up would evaluate full SayCan (with affordances, scoring interface, and physical execution) across the PaLM scaling curve — 8B, 62B, 540B — on the 101-instruction benchmark. The key questions: Does execution success scale monotonically with LLM size when affordances are included? Does the gap between the 62B and 540B models remain small as in the generative evaluation, or do affordances amplify the difference (because a more accurate LLM probability better complements the affordance signal)? Does scaling behavior differ by instruction family — do abstract noun tasks benefit more from scale than single primitives? A finding that scaling saturates at moderate model sizes would suggest that prompt engineering and reasoning techniques matter more than raw scale for this task distribution, while a finding of continued improvement would strengthen the paper's claim that NLP progress directly benefits robotics and would inform resource allocation decisions about which LLM to deploy in production robotic systems.

Replacing prompt engineering with learned skill selection policies. The prompt engineering ablation (Appendix Table 5) shows that SayCan's performance is heavily dependent on the 17 hand-crafted examples — planning success drops from 88% to 10% without examples (with termination required). This dependence means that deploying SayCan in a new domain requires a manual prompt engineering cycle that the paper describes as nontrivial (iterative refinement using a language-based simulator, careful attention to object naming, numbering, line breaks, and dialog framing). A concrete follow-up would train a lightweight skill selection policy that takes the instruction embedding and history as input and outputs a distribution over next skills — essentially distilling the LLM's scoring behavior into a small, domain-specific model that can be trained from the LLM's own scored outputs on a set of training instructions. The experiment would compare this distilled policy's planning success against the original LLM scoring on the 101-instruction benchmark, with held-out instruction families to test generalization. If the distilled policy matches LLM performance, deployment cost drops dramatically (replacing 540B-parameter forward passes with a small neural network) and the prompt engineering bottleneck is eliminated for new domains. If the distilled policy underperforms, the experiment would reveal whether the LLM's contribution is primarily few-shot pattern matching (which distillation should capture) or deeper semantic reasoning (which distillation might miss), clarifying the LLM's specific role in the system.

Testing the independence assumption with joint models. SayCan's core factorization $p(c_\pi \mid s, \ell_\pi) \cdot p(\ell_\pi \mid i)$ assumes conditional independence between physical feasibility and semantic appropriateness. No experiment in the paper tests this assumption directly. A diagnostic follow-up would train a joint model that predicts the combined score $p(c_i \mid i, s, \ell_\pi)$ directly — for example, a small multi-modal network that takes the LLM's text embedding of the instruction, the USE embedding of the skill description, and the visual state representation as input, and outputs a single scalar for each skill. This model could be trained on the SayCan execution traces (which skill was selected, whether the overall instruction succeeded) using the same data that already exists from the 101-instruction evaluation. The key comparison would be planning success rate of the joint model vs. the factored product on the same benchmark. If the joint model matches or exceeds the factored product, the independence assumption is harmless (or even suboptimal — there are correlations the factored model misses). If the joint model substantially underperforms, it suggests the factored approach provides beneficial regularization by preventing the model from learning spurious correlations between language and visual features. This experiment would clarify whether the factorization is merely a convenient engineering choice or a principled inductive bias, with implications for whether future systems should maintain the modular separation or move toward tighter integration.

Domain transfer: replicating SayCan on a different robot, environment, and task family. The paper's central architectural claim — that the LLM + value function interface is general — is supported only in office kitchen environments with an Everyday Robots mobile manipulator performing fetch-and-deliver tasks. A strong test of generality would replicate SayCan on a substantially different platform (e.g., a stationary industrial arm for assembly tasks, a quadcopter for inspection, or a humanoid for household chores) with a different LLM (to test whether the architecture is model-agnostic) and a different skill training pipeline (to test whether value functions from non-MT-Opt RL methods work equally well as the grounding interface). The replication should use the same 101-instruction structure adapted to the new domain (single primitives, abstract nouns, abstract verbs, embodiment, crowd-sourced, long-horizon) to enable direct comparison. The critical measurements would be: (1) Does SayCan achieve non-trivial planning success in the new domain with no architecture changes? (2) Does the planning-vs-execution gap have the same structure (execution trailing planning, embodiment and long-horizon being hardest)? (3) Does the LLM's semantic knowledge transfer to the new task vocabulary, or does it require domain-specific prompt engineering that is as labor-intensive as training a task-specific planner? A positive result would validate the architecture's generality and make SayCan a standard baseline for language-conditioned robotics across domains. A negative result — SayCan works only for kitchen manipulation — would appropriately scope the contribution and redirect research toward domain-specific grounding mechanisms.

Stress-testing the scoring interface with large skill repertoires. The paper's experiments use approximately 20–30 candidate skills (15 pick, ~5 go-to, 1 place, 1 terminate). The scoring interface requires evaluating the LLM on every candidate at every decision step, which scales linearly with skill count. A stress-test would expand the repertoire to hundreds of skills (e.g., all 551 skills defined in the paper, or a combinatorial expansion where "pick up the [object]" is instantiated for 100+ objects) and measure: (1) planning latency as a function of skill count, with and without batching of LLM queries; (2) planning success rate — does the larger candidate space increase confusion errors where the LLM assigns high probability to semantically similar but incorrect skills?; (3) whether hierarchical scoring (first score action types: "pick", "go to", "place"; then score objects/ locations within the selected action type) maintains accuracy while reducing computation. This experiment would determine the practical scalability limits of the scoring approach and whether the architecture needs fundamental modification (e.g., learned candidate pruning, retrieval-based skill selection) to handle the skill repertoires that real-world deployment would require. The paper's silence on latency and computational cost makes this a high-priority follow-up for anyone considering production deployment.

Practical Applications and Downstream Use Cases

Voice-controlled mobile manipulators in semi-structured environments. The most direct application of SayCan is the one demonstrated: a mobile robot in an office kitchen that non-expert users can command with natural language. The 84% planning success and 74% execution success in the mock kitchen, combined with the demonstration that crowd-sourced instructions (87% planning, 87% execution) perform on par with structured ones, suggests the system is already at a useful level for environments where the cost of occasional failure is low — tidying a break room, fetching items for office workers, assisting in a hospital pantry. The key enabler is that SayCan requires no per-user or per-instruction training: a new employee can say "I spilled my coffee, can you bring me something to clean it up?" and the robot can respond without the facility manager having programmed a "spill cleanup" routine. The 60% execution rate in the real kitchen indicates that substantial engineering work remains to make the low-level skills robust to environment variation, but the planning side (81% in the real kitchen) degrades minimally, suggesting that the language understanding and task decomposition generalize well across visual domains. A practical deployment would pair SayCan's planning with continuous policy improvement (the evaluation-driven policy graduation process described in Appendix C.3, which raised per-skill success rates over 11 months from near-zero to the levels reported) and potentially with the closed-loop recovery from the Inner Monologue extension.

Data generation for training end-to-end language-conditioned policies. The paper's BC NL baseline — feeding high-level instructions directly to the language-conditioned policy — achieves 0% success, but this is because the policy was never trained on high-level instructions. SayCan provides a mechanism for automatically generating training data for such policies: run SayCan on thousands of instructions (synthesized by varying objects, locations, and action types), record the successful execution traces (plan + execution video + final state), and use the resulting paired data of (high-level instruction, successful execution trajectory) to train an end-to-end policy that maps instructions directly to action sequences. This is essentially a distillation approach: SayCan's modular system acts as a "teacher" that produces supervised data for a more efficient "student" policy that does not require LLM queries at inference time. The 84% planning success rate means a substantial fraction of instruction attempts would produce valid training data, and the failed attempts (16% planning errors, plus execution failures) can be filtered out automatically since SayCan's execution traces include success/failure labels. This application is attractive because it amortizes SayCan's inference cost — the expensive LLM queries happen once during data generation, and the deployed policy is a standard neural network that runs at control frequency with no language model overhead.

Benchmark for embodied language understanding. The 101-instruction evaluation set, with its deliberate variation across 7 instruction families (single primitives, abstract nouns, abstract verbs, structured language, embodiment, crowd-sourced, long-horizon), provides a structured test of whether a robotic system genuinely understands language or merely pattern-matches. The paper's breakdown of results by family reveals where different approaches fail: end-to-end policies (BC NL, BC USE) fail on everything beyond single primitives because they cannot decompose instructions; ungrounded LLM approaches (No VF, Generative) fail disproportionately on embodiment tasks because they lack state awareness; SayCan fails most on long-horizon tasks because of early termination by the LLM. This diagnostic structure makes the benchmark useful beyond this paper — any new approach to language-conditioned robotics can be evaluated against the same families to reveal its specific failure modes. The paper's open-source tabletop implementation (Section 6) provides a lower-cost entry point for researchers who lack access to a mobile manipulator, and the instruction family taxonomy is domain-agnostic enough to be adapted to other task categories (e.g., assembly, navigation, social interaction) while preserving the same analytical structure. A practical contribution would be to release the full 101-instruction set with the human rating protocol and the per-family breakdown as a standardized evaluation suite for the field, analogous to how benchmarks like ALFRED and BEHAVIOR standardized evaluation for simulated instruction following.

When to Prefer This Method

The paper positions SayCan against two specific alternatives — end-to-end language-conditioned policies (the BC NL and BC USE baselines, representing the dominant paradigm in language-conditioned imitation learning and RL at the time) and ungrounded LLM planning (the No VF and Generative baselines, representing concurrent approaches to using LLMs for embodied task planning). The tradeoffs are explicit and supported by experimental comparisons in Table 2.

Prefer SayCan over end-to-end language-conditioned policies when:

• The instructions are abstract, temporally extended, or require task decomposition (the BC baselines achieve 0% on all non-single-primitive families, while SayCan achieves 67–100% planning), because the LLM provides the decomposition capability that end-to-end policies trained on atomic commands fundamentally lack.
• The instruction vocabulary includes synonyms, paraphrases, abstract categories, or multilingual queries that go beyond the policy's training distribution (BC USE achieves 0% on all multi-step instructions because embedding similarity cannot bridge the gap between atomic training commands and novel phrasings; SayCan achieves 87% on crowd-sourced instructions and identical planning rates in Chinese, French, and Spanish via the LLM's pre-trained language understanding).
• Modular upgrade paths are valued — either the LLM or the skill repertoire can be improved independently, and the combined system benefits (demonstrated by the PaLM→FLAN swap improving planning from 70% to 84% with no robot changes, and by the drawer skill addition requiring only 3 prompt examples with no degradation to existing task performance).
• Zero-shot deployment on novel instruction types is required with no task-specific training data (SayCan's frozen LLM + pre-trained value functions require no gradient updates for new instructions; end-to-end policies require collecting demonstration data for each new instruction type).

Prefer end-to-end language-conditioned policies over SayCan when:

• The task domain is narrow and the instruction distribution is known and stable (all instructions map to a small set of atomic commands), making the LLM's semantic generality unnecessary overhead.
• Inference latency is critical and the cost of LLM queries (hundreds of forward passes through a 540B-parameter model per planning step) is prohibitive — a trained end-to-end policy can run at control frequency on modest hardware.
• The robot's skill repertoire is small enough that direct language conditioning without decomposition is sufficient (the BC NL baseline's 0% failure is partly because the training data only included atomic commands — if the policy had been trained on multi-step instruction demonstrations collected via teleoperation, it might perform comparably to SayCan on in-distribution tasks).

Prefer SayCan over ungrounded LLM planning when:

• The robot operates in a physical environment where state awareness matters — the embodiment instruction family shows SayCan at 64% planning vs. 18% for No VF, a 46 percentage point gap that isolates the affordance functions' contribution.
• Safety or reliability concerns make physically impossible plans unacceptable — the No VF baseline will happily select "pick up the sponge" when the sponge is not visible or reachable, while SayCan's affordance multiplication suppresses infeasible skills (the product goes to zero when either term is near zero).

Prefer ungrounded LLM planning over SayCan when:

• No value functions or success detectors are available for the robot's skills, and training them would require substantial RL data collection — the Generative baseline achieves 74% planning with no physical grounding at all, which may be an acceptable starting point for rapid prototyping or simulated environments where execution failures have no real-world cost.
• The LLM's semantic knowledge is the primary bottleneck, and the affordance functions add computational cost without addressing the dominant failure mode (65% of SayCan's errors are LLM failures — in domains where the LLM is the weak link, investing in better prompting or chain-of-thought reasoning may yield higher returns than adding affordances).

The paper does not articulate tradeoffs between SayCan and classical task-and-motion planning (TAMP) approaches, nor between SayCan and methods that fine-tune the LLM on interaction data (which were nascent at the time). These comparisons would require additional experiments that the paper does not provide, so forced tradeoff matrices with these alternatives would be speculative rather than paper-grounded.