Learning to Learn Faster from Human Feedback with Language Model Predictive Control

ArXiv: 2402.11450

🎯 Pitch

Fine-tuning a robot code-writing LLM to simulate entire imagined teaching dialogues—and then planning through those rollouts like a world model—cuts the number of human corrections needed by over 20% and boosts teaching success by 26.9% across 78 tasks. Even more striking: by automatically identifying and conditioning on the best 25% of human teachers (covering just 14% of tasks), the model lifts performance for all users, effectively learning a transferable teaching style without extra human data.

1. Executive Summary

This paper introduces Language Model Predictive Control (LMPC), a framework that fine-tunes robot code-writing LLMs to improve their teachability — how efficiently they adapt to human feedback — by viewing human-robot interactions as a partially observable Markov decision process and training the LLM as a transition dynamics model that can be combined with model predictive control to discover shorter paths to task success. Using PaLM 2-S as the base model across 78 tasks on 5 robot embodiments, LMPC-Rollouts — which learns to predict entire imagined chat session rollouts and selects the next action with the fewest expected corrections via inference-time search — improves non-expert teaching success rates on unseen tasks by 26.9% while reducing the average number of human corrections from 2.4 to 1.9, and also generalizes to unseen robot embodiments and APIs with a 31.5% improvement in in-context learning success rate. Top-user conditioning, which autonomously identifies the top 25% of users by weighted performance and conditions inference on their data distribution (covering only 14% of tasks), drives performance improvements for all users by 10.5% on unseen tasks, establishing that fine-tuning can effectively transfer the style of high-proficiency teaching interactions to benefit the broader user population.

2. Context and Motivation

The Core Problem: In-Context Learning for Robot Teaching Is Fleeting

The authors set out to tackle a specific, practical bottleneck in language-based human-robot interaction: LLMs that adapt to human feedback during a teaching session can only retain that adaptation for as long as it fits within the model's context window. This is the fundamental limitation of in-context learning — the mechanism by which an LLM conditions its next output on the preceding dialogue (user instructions, corrections, preferences, and the model's own code responses) without any weight updates. In a multi-turn robot teaching scenario, a user might spend several rounds refining a behavior — "move to the left," "no, a bit more," "now lift it higher" — and the LLM's ability to incorporate each successive piece of feedback depends entirely on the entire interaction history being present in the input prompt. Once that history overflows the finite context size (or once the user starts a new session), all the learned preferences and corrected behaviors are forgotten. The robot, effectively, has no long-term memory of what it was just taught.

This is not merely an inconvenience; it fundamentally caps the teachability of LLM-driven robots. The authors define teachability explicitly as "the average number of human inputs (chat turns) n before the robot succeeds at the task," and they note that n = 1 corresponds to standard zero-shot instruction following. A model with poor teachability requires many rounds of correction to reach a successful outcome — or never reaches it at all. Improving teachability means reducing n while maintaining or increasing the overall success rate. This is a meta-learning objective: "learning to learn faster from human feedback" (Section I, citing Hospedales et al., 2021). The paper's overarching goal is to study which methods of offline model improvement (specifically, fine-tuning) can best leverage data collected during in-context learning interactions to improve future teachability on entirely new, unseen tasks.

Why This Problem Matters: Real-World Impact and Theoretical Significance

The practical motivation is compelling. Recent work (referenced throughout Section I and II) has demonstrated that LLMs pretrained on Internet-scale data show out-of-the-box capabilities for robot control — they can plan step sequences from language commands, write executable robot code, and respond to multi-turn feedback to compose new behaviors. This opens the door to a paradigm where non-experts — people without robotics or programming training — can direct robot behavior using natural language alone. The paper's experimental setup explicitly uses 35 non-expert users (not researchers or engineers, not familiar with the underlying LLMs or robot code) to evaluate teachability, underscoring this ambition.

But if the adaptation achieved through in-context learning evaporates at the end of each conversation, the vision of cumulative, long-term robot teaching breaks down. A user who spends 10 minutes carefully correcting a quadruped's gait or teaching a dexterous hand to insert a connector would need to repeat the entire process from scratch the next day. For robots deployed in homes, offices, or care settings — environments where users' preferences, physical layouts, and task definitions evolve over time — this memorylessness severely limits practical utility.

There is also a theoretical angle that elevates this beyond a purely engineering concern. The paper frames the contrast between in-context learning and in-weights learning (fine-tuning) as mapping onto a classic distinction in cognitive science and machine learning: exemplar-based generalization versus rule-based generalization (Section I, citing Chan et al., 2022; Ashby & Townsend, 1986; Shepard & Chang, 1963). In-context learning tends to generalize on the basis of similarity to examples present in the context buffer — it excels at interpolating between seen instances but struggles to extract abstract rules that transfer to novel situations. In-weights learning via fine-tuning, by contrast, tends to discover more compressible, category-level features that support broader generalization. The paper hypothesizes that both forms of learning have complementary strengths, and that a system combining fast in-context adaptation during interaction with slow weight-based consolidation afterward could outperform either alone. This hypothesis — that fine-tuning on interaction data can produce a model that is a better in-context learner on the next interaction — is a specific, testable claim about meta-learning.

Prior Approaches and Where They Fall Short

Language feedback for robot adaptation without LLMs. Before the LLM era, a substantial body of work integrated language and robotics through mapping language to planning primitives, learning language-conditioned reward functions, imitating demonstrations paired with language instructions, and using language as corrective feedback (Section II lists representative citations: Tellex et al., 2011; Kollar et al., 2010; Artzi & Zettlemoyer, 2013; Karamcheti et al., 2017; Jang et al., 2022; Lynch & Sermanet, 2021; Shridhar et al., 2021; Stepputtis et al., 2020; Mees et al., 2022; Mirchandani et al., 2021; Jiang et al., 2019; Goyal et al., 2020; Misra et al., 2017; Co-Reyes et al., 2019; Cui et al., 2023). These methods were typically task-specific, required engineering of language interfaces, and did not leverage the broad commonsense reasoning and code-writing capabilities of modern LLMs. They are not direct competitors to the paper's approach but represent the pre-LLM baseline for language-driven robot behavior.

LLM-based robot code writing and in-context adaptation. More relevant are recent systems that use LLMs — either prompted or fine-tuned — to write robot code from language. Liang et al. (2023) introduced Code as Policies, where LLMs generate policy code (Python) in response to language commands. Yu et al. (2023) developed Language to Rewards, where an LLM translates natural language into reward functions that are consumed by MuJoCo MPC (MJPC) to synthesize real-time robot motion — this is the direct precursor to the fast-adaptation pipeline used in the current paper. Arenas et al. (2023) proposed PromptBook, a formatting convention for prompts that describe robot APIs and provide example episodes, which the current paper adopts. Other works have used LLMs to modify low-level behaviors (Sharma et al., 2022; Mirchandani et al., 2023), remember user preferences (Wu et al., 2023), and ask for help (Ren et al., 2023). What unifies these approaches is that they rely on in-context learning alone — the LLM's weights are frozen. The model adapts to the user during a single conversation but retains nothing across conversations.

Retrieval-Augmented Generation (RAG) for robot teaching. The most directly comparable prior work is Zha et al. (2024), which investigates robot teaching by summarizing human feedback, indexing it into memory, and later retrieving relevant past feedback as in-context examples for similar future interactions. This is a RAG-based approach: a pretrained embedding model retrieves examples from a database, which are then inserted into the LLM context to guide the current response. The approach improves upon purely in-context methods by providing a form of long-term memory that persists beyond individual sessions. However, the paper identifies two key limitations of this approach:

1. Generalization is exemplar-based. Because the retrieved examples are selected by embedding similarity, the method tends to help on tasks that resemble previously seen training tasks but offers limited generalization to novel tasks that fall outside the distribution of the retrieval database. The paper explicitly cites evidence that in-context learning is more "exemplar-based" (generalizing on the basis of similarity to in-context examples), in contrast to in-weights learning which is more "rule-based" (generalizing on the basis of abstract category boundaries in the training data). The RAG baseline in the paper's experiments confirms this: it achieves a lower Successful Task Rate on test tasks than the base model (Table I: 75.0% for RAG vs. 81.5% for PaLM 2-S), suggesting it "struggles to perform well on novel tasks" despite improving overall success rates on tasks similar to retrieved examples.

2. External modules add complexity. RAG requires maintaining an embedding index, a retrieval system, and additional inference-time computation for retrieving and formatting examples — all external to the LLM itself. The paper's fine-tuning approach aims to absorb the relevant knowledge directly into the model weights, simplifying deployment while also (as experiments show) yielding stronger generalization.

LLM alignment via RLHF and SFT. The paper situates itself within the broader literature on aligning LLMs to user intent, particularly work on supervised fine-tuning (SFT) followed by reinforcement learning from human feedback (RLHF) (Ouyang et al., 2022; Christiano et al., 2023; Stiennon et al., 2022; Bai et al., 2022). However, these works focus overwhelmingly on single-turn dialogue — mapping a user input to a preferred output — rather than multi-turn, interactive teaching scenarios where the model's output is executed in a physical or simulated environment and the user provides feedback based on observed behavior. The paper's contribution is to extend the SFT paradigm to a sequential, embodied setting where the model must learn not just what code to write, but the process of converging to correct code through interaction.

How This Paper Positions Itself

The paper's core conceptual move is to reframe human-robot teaching sessions as trajectories in a partially observable Markov decision process (POMDP). In this framing (revised for clarity in the final version): human language inputs h_t are observations, robot code outputs c_t are actions, and the user's final "success" or "failure" label r is the reward. The LLM, when conditioned on previous interactions, act as a policy. Training an LLM to autoregressively complete previous interaction sequences is therefore analogous to learning a transition dynamics model of this POMDP. Once trained, this dynamics model can be combined with model predictive control (MPC) — a classic control technique that uses a learned model to simulate multiple possible futures and selects the action sequence that optimizes a cost function — to discover shorter paths to success. At inference time, this translates to: sample multiple imagined completions of the chat session from the fine-tuned LLM (with non-zero decoding temperature), select the trajectory that reaches success in the fewest chat turns, and output the first action of that trajectory. This receding-horizon process repeats at every chat turn.

This positions the paper at the intersection of three threads: LLM fine-tuning for alignment, model-based reinforcement learning / control, and meta-learning for fast adaptation. The paper's claim is not that any of these individual ideas is new, but that their combination — specifically, training language models as interaction dynamics models and using MPC as a decoding strategy to accelerate task convergence — yields a novel and effective approach for improving robot teachability.

The primary contribution is the slow adaptation method (LMPC), not the fast adaptation pipeline. The paper is careful to distinguish between the system it built (the in-context learning pipeline that translates language to robot code and executes it via MJPC — Section III-B) and the method it proposes (LMPC fine-tuning — Section III-C). The fast-adaptation pipeline, while necessary for data collection and evaluation, is an extension of prior work (primarily Yu et al., 2023). The paper's main contribution is demonstrating that fine-tuning on data from this pipeline, using LMPC-Rollouts, produces a model that learns to learn faster — it is more teachable on unseen tasks and embodiments, more responsive to corrective feedback, and can leverage the interaction style of skilled users to improve outcomes for all users.

The comparison between LMPC-Rollouts and LMPC-Skip is deliberate and diagnostic. LMPC-Skip — which is trained to directly predict the final correct code, skipping all intermediate interactions — serves as an ablation that isolates the benefit of modeling the process of interaction versus just the outcome. The key empirical result is that LMPC-Skip excels at one-shot success (higher 1-turn success rate) but LMPC-Rollouts excels at multi-turn success (higher 2+ turn success rate). This confirms that modeling intermediate human-robot dialogue is what enables responsiveness to corrective feedback — the very thing that teachability measures.

Top-user conditioning is a practical insight, not just a trick. The observation that conditioning on top-performing users (autonomously identified from training data) improves outcomes for all users — despite top users covering only 14% of tasks — is significant because it suggests that the interaction data contains information about how to teach effectively, not just what to teach. By conditioning on top-user IDs during training and using the special "top-user" label during inference, the model learns to generate code in the style that elicits and responds to high-quality user feedback, which transfers to interactions with lower-proficiency users on entirely new tasks. This insight — that user identity can serve as a proxy for interaction quality and that conditioning on high-quality users benefits the broader population — is a notable contribution to the design of interactive learning systems.

3. Technical Approach

3.1 Reader Orientation

This paper is primarily a systems and methods paper that proposes a specific fine-tuning recipe, Language Model Predictive Control (LMPC), to make an LLM-driven robot more teachable — meaning it requires fewer rounds of human correction to successfully perform a task. The core idea is to treat the entire multi-turn chat history between a human teacher and a robot code-writing LLM as a trajectory in a sequential decision-making process, fine-tune the LLM to predict how such trajectories unfold (including both the human's future feedback and the robot's future code responses), and then at inference time use this predictive model to search for the shortest path to task success, executing only the first action of the best predicted plan.

3.2 Big-Picture Architecture (Diagram in Words)

The system has two distinct operating phases and five major components:

Phase 1 — Fast Adaptation (Online, In-Context Learning):

• The Chat Interface: A browser-based UI where human users type natural language instructions and feedback, and observe the robot's resulting behavior in a MuJoCo simulation (or on real hardware).
• The Base LLM (PaLM 2-S): A frozen, pretrained large language model. It receives a system prompt describing the current robot embodiment and its API, followed by the multi-turn chat history. It autoregressively outputs executable Python reward code in response to the most recent human message.
• MuJoCo MPC (MJPC): A real-time model predictive controller. It takes the LLM-generated reward function code and synthesizes low-level robot joint trajectories that maximize that reward, producing visual behavior for the user to observe and react to.

Phase 2 — Slow Adaptation (Offline, Fine-Tuning):

• The Training Dataset: A collection of successful chat sessions (prompt + sequence of human-text, robot-code pairs + final success label), gathered from Phase 1 interactions with non-expert users.
• The Fine-Tuned LMPC Model: The same PaLM 2-S architecture, further trained via supervised fine-tuning (SFT) to autoregressively complete a chat session starting from just the system prompt and the first user instruction — i.e., to predict the entire remaining interaction. This model learns the dynamics of the human-robot teaching process. A variant, LMPC-Skip, is trained to predict only the final correct code, skipping intermediate corrections. During Phase 2 inference, the LMPC model generates sampled imagined rollouts of future interactions, selects the one that achieves success in the fewest chat turns, and returns its first action — effectively using MPC as a decoding strategy.

The information flow in a single chat turn during evaluation is: user types feedback → chat history is appended to the system prompt → the fine-tuned LMPC model samples multiple completions (imagined futures) → the completion with the shortest path to success is chosen → the first robot code action from that completion is extracted and sent to MJPC → MJPC executes the reward function on the (simulated or real) robot → the user observes the resulting motion and provides the next round of feedback.

3.3 Roadmap for the Deep Dive

• First, the POMDP analogy and the formal problem statement, because the entire LMPC framework depends on viewing interaction as a sequential decision process where the LLM is both the policy and (after fine-tuning) the world model.
• Second, the fast adaptation pipeline (the "system" that enables data collection) , because understanding what the LLM does during deployment — convert language to reward code, which MJPC converts to motion — is a prerequisite for understanding what the fine-tuning data looks like and what the model is being trained to improve.
• Third, the LMPC-Rollouts training procedure, the paper's core contribution: how the fine-tuning data is structured (entire chat sessions as input-target pairs), what the loss function is (standard autoregressive next-token prediction), why only successful sessions are used, and how inference-time search with receding horizon control works.
• Fourth, the LMPC-Skip variant, as a diagnostic ablation: training the model to directly predict the final correct code without intermediate steps, which isolates the benefit of modeling the process of interaction versus just the outcome.
• Fifth, top-user conditioning, the mechanism for identifying high-performing users and conditioning generation on their interaction style, including how this functions differently for LMPC-Rollouts versus LMPC-Skip and why it benefits all users despite covering only 14% of tasks.
• Sixth, the data augmentation strategy, because the training dataset is small (~300 successful sessions) and augmentation with paraphrased user inputs is critical for generalization, particularly for LMPC-Rollouts' sensitivity to compounding errors.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems-building and empirical methods paper whose core contribution is a fine-tuning framework (LMPC) that uses supervised learning on interaction trajectories combined with inference-time search to improve the teachability of robot code-writing LLMs. The fast-adaptation pipeline (Section III-B) is the necessary infrastructure for data collection and evaluation; the slow-adaptation method (Section III-C) is the novel contribution.

The POMDP Framing and Problem Statement

The paper formalizes the language-based human-robot teaching process as a partially observable Markov decision process (POMDP). This is not a strict mathematical formalization that the paper solves with dynamic programming; rather, it is an analogy that provides the conceptual foundation for why training an LLM to predict interaction sequences is equivalent to learning a dynamics model, and why MPC is a natural inference-time strategy.

In this POMDP analogy:

• The state is the user's true intended robot behavior for the current task, plus any other latent factors (preferences, teaching style, patience) that influence subsequent feedback. This state is never directly observed — the LLM only sees language from the user.
• The observation $h_t$ at timestep $t$ is the user's natural language text input — this could be an initial task instruction or a corrective feedback message ("move a bit to the left," "no, the other left").
• The action $c_t$ is the robot code (Python reward function) that the LLM outputs in response.
• The reward $r$ is binary: 1 if the user labels the entire chat session as "success," 0 if the user labels it "failure" (or if 7 chat turns are exhausted without success, which is an automatic failure).
• The policy is the LLM itself, mapping from the history of observations and actions $(h_0, c_0, h_1, c_1, \dots, h_t)$ to the next action $c_t$, with the goal of maximizing the probability of eventual success.

The paper's explicit problem statement (Section III-A) defines teachability as:

"the average number of human inputs (chat turns) $n$ before the robot succeeds at the task"

where $n = 1$ is equivalent to standard zero-shot instruction following. Improving teachability means reducing $n$ while maintaining or increasing the success rate. The paper frames this as a meta-learning objective: "learning to learn faster from human feedback," citing Hospedales et al. (2021). Intuitively, a model with better teachability should be more responsive to feedback — each additional round of human input should be more likely to push the robot toward the desired behavior — and more adaptive to user preferences — it should require fewer examples to learn that, for a particular user, "move to the left" means a 5cm displacement rather than a 20cm displacement.

A complete chat session is represented as the sequence:

$[P, h_0, c_0, h_1, c_1, \dots, h_T, c_T, r]$

where $P$ is the system prompt (describing the robot embodiment, its available APIs, and formatting conventions), $h_t$ is the user's text at turn $t$ (with $h_0$ being the initial task instruction), $c_t$ is the LLM's code response at turn $t$, $T$ is the total number of turns in the session, and $r$ is the binary success label provided by the user at session termination.

What this represents operationally: a complete transcript of a single goal-driven teaching interaction. The session begins with the system prompt ($P$) followed by the user's first instruction ($h_0$), to which the LLM responds with code ($c_0$). The code is executed on the robot via MJPC, the user observes the behavior, and provides feedback ($h_1$), and so on, until either the user clicks "success" (satisfied that the robot completed the task) or the 7-turn limit is reached (automatic failure). The sequence captures both the content of the interaction (what was said and what code was written) and the outcome (whether it succeeded).

Why this sequential representation matters: it enables the entire teaching process to be treated as a single autoregressive text prediction task for an LLM. Rather than training separate models to (a) interpret user intent, (b) generate code, (c) predict whether feedback will lead to success — which would require complex integration and training pipelines — the paper's key insight is that a causal language model can absorb all of these functions by simply being trained to predict the next token in the concatenated chat session. The $r$ token at the end provides the supervision signal without requiring dense per-turn rewards. This is analogous to how Decision Transformers (Chen et al., 2021) convert reinforcement learning into sequence modeling, though the paper uses this representation for dynamics modeling and MPC rather than for direct policy learning.

The Fast Adaptation Pipeline (System for Data Collection)

Before explaining the fine-tuning method, the paper describes the "fast adaptation" system that serves two purposes: (1) it is the deployment environment where teachability is ultimately measured, and (2) it is the data generation mechanism — all fine-tuning data comes from human interactions with this pipeline. The pipeline has two serial stages, described in Section III-B.

Stage 1: Language to Robot Reward Code (In-Context Learning). The base LLM (PaLM 2-S, frozen weights) receives a structured prompt and produces executable Python code. The prompt uses PromptBook formatting (Arenas et al., 2023), which the paper describes as containing:

1. A high-level description of the robot embodiment (e.g., "You are a stationary robot arm with a 3-fingered hand").
2. The complete Robot Control API — a list of available Python functions with their signatures, argument descriptions, and usage constraints. For example, for the Robot Dog embodiment, the API includes functions like set_torso_targets(height, tilt_angle, roll_angle, location_xy, velocity_xy, heading, turning_speed), set_foot_pos_targets(foot_name, lift_height, extend_forward, move_inward), etc.
3. 1-2 example episodes (full chat sessions) that demonstrate the expected interaction format and code-writing style. These examples show specific user queries followed by the LLM's code responses, with comments explaining the reasoning behind each function call.
4. The current chat session appended below the examples, following the same format: "Chat Turn N - User: [user text]" followed by "Chat Turn N - Program: [code with reasoning comments]."

The prompt is terminated with the current chat turn's user message, and the LLM autoregressively completes the "Program" section. The generated code contains both comments (explaining the reasoning) and function calls (using only the API functions listed in the prompt). The paper notes two key extensions over the prior work of Yu et al. (2023):

• Chain-of-Thought style reasoning via comments: The LLM generates natural language reasoning interspersed as comments between lines of reward code. This is not a separate inference step — the comments and code are generated as a single unified output. From the paper's perspective, this "simplifies reward code writing and enables more flexible code generation" by letting the model interleave high-level reasoning with low-level function specification in one pass, rather than requiring two separate prompts as in Yu et al. (2023) (one to generate motion descriptions, another to convert those descriptions to code).
• Sequencing multiple reward functions via condition functions: The LLM can compose sequential behaviors by writing wait_until_condition(condition_fn) calls that pause execution until a specified condition is met (e.g., the apple's z-coordinate exceeds a threshold), at which point the next reward function in the sequence becomes active. This enables multi-step tasks like "pick up the apple, wait until it's in the air, then move it to the right" — all within a single chat turn's code output.

The full prompts for each embodiment are provided in Appendix VI-N, spanning approximately 2-4 pages each. The prompts are engineering-heavy: they define coordinate conventions, list available objects by name, specify numerical ranges for parameters, note common user synonyms ("foot" = "paw"), and include admonitions like "!!!IMPORTANT!!! When responding to user instructions, always rewrite the entire previous program with the new revisions."

A critical practical constraint: the LLM must run at interactive speeds. The paper reports that model inference runs at 100 tokens per second, with the average code response being 200-300 tokens (roughly 10-15 lines of code). The median duration for each chat turn is 56 seconds, with the majority of that time spent by the user observing the robot performing the task in simulation, not waiting for the LLM.

Stage 2: Reward Code to Robot Motions (MJPC). The generated reward code is not executed directly as a robot policy. Instead, it is consumed by MuJoCo Model Predictive Control (MJPC) (Howell et al., 2022), an existing real-time trajectory optimization system. MJPC works by:

1. Using the MuJoCo physics simulator as an internal dynamics model of the robot and its environment.
2. Implementing a receding horizon trajectory optimization algorithm: at each control timestep, MJPC samples many candidate action sequences, simulates them forward using the dynamics model, evaluates each candidate against the current reward function (the one just generated by the LLM), and selects the action sequence that maximizes cumulative reward.
3. Executing the first action of the selected sequence, then repeating the process at the next timestep — hence "receding horizon."

Because MJPC continuously re-optimizes actions based on the reward function, the LLM's job is simplified: it only needs to specify what good behavior looks like (via the reward function), not how to achieve it (which MJPC handles through physics simulation and optimization). The reward functions use API calls like minimize_l2_distance(obj_a, obj_b, weight) to attract two objects together, set_object_position_target(obj, position) to specify a desired location, and reach(obj, weight) to move the robot's hand to an object. The weight parameter controls the relative importance of different reward terms.

The paper introduces two modifications to the reward code format used in Yu et al. (2023), detailed in Section III-B:

• Chain-of-Thought reasoning via comments (described above).
• Multi-stage behaviors via condition functions: The LLM can write code like:

# To pick up the apple, bring it close to the left gripper.
min_L2_dist(obj1='left_hand', obj2='apple', weight=5.0)
# To lift up the apple, get its position and increment along z.
pos = get_obj_pos(obj='apple')
set_target_pos(obj='apple', (pos[0], pos[1], pos[2] + 0.25))
# Wait until the apple is in the air.
def condition_fn():
    return get_obj_pos(obj='apple')[2] >= 0.25
wait_until_condition(condition_fn)
# To hand over the apple, bring it close to the right gripper.
min_L2_dist(obj1='apple', obj2='right_hand', weight=5.0)

Here, wait_until_condition blocks execution of subsequent reward terms until the condition (apple above 0.25 meters) is met. This enables the LLM to specify temporally extended behaviors — sequences of reward functions with transitions — in a single code block, rather than requiring the user to provide separate instructions for each stage.

A crucial practical detail: MJPC requires high-performance computing. The paper notes that machines with 128 CPU cores are used to achieve interactive synthesis rates. The simulation runs at 25% real-time speed for the 14-DoF Aloha embodiment and 15% real-time speed for the 19-DoF Kuka+Hand embodiment, meaning that users see robot behavior at roughly one-quarter to one-sixth of real-world speed. This is a practical constraint that affects the user experience — the paper reports that Kuka+Hand chat turns average 1.5 minutes, with sessions around 7 minutes, "much of the time is spent watching the robot 'figure out' online how to do the task."

Data Collection Protocol. Each chat session is a goal-driven interaction: the user is assigned a specific task (e.g., "sit down and give a high-five" for the Robot Dog) and teaches the robot through successive rounds of language input and code execution. After each chat turn, the user has the option to rate the individual robot response as "good" or "bad" — these per-turn ratings are used for evaluation (e.g., the "Good Rating Rate" metric) but not for training. At the end of each session, the user clicks either "success" (the robot performed the task to their satisfaction) or "failure" (the session reached 7 turns without success, or the user gave up earlier). The complete chat history is then saved as a training data point if the session was successful.

Language Model Predictive Control (LMPC-Rollouts): Training

This is the paper's core contribution (Section III-C). LMPC-Rollouts is a supervised fine-tuning (SFT) procedure with a specific inference-time decoding strategy. The training objective is standard autoregressive next-token prediction. The novelty is in what is being predicted (entire future interaction trajectories) and how the predictions are used at inference time (MPC-style search with receding horizon control).

Training Data Construction. The fine-tuning dataset consists exclusively of successful chat sessions from the fast-adaptation data collection phase. The paper reports experiments showing that training on both successes and failures "yielded much worse performance" (Appendix VI-B), hypothesizing that mixing failures into the training data reduces inference-time search efficiency because more sampled rollouts terminate in predicted failure and are discarded. Training only on successes biases the model to generate trajectories that end in success, which is desirable for the search strategy.

Each successful session is formatted as a single contiguous text sequence:

[System Prompt P] [User Instruction h_0] [Robot Code c_0] [User Feedback h_1] [Robot Code c_1] ... [Success Label r]

The system prompt $P$ is included in its entirety and varies by robot embodiment — different embodiments have different robot API descriptions, coordinate conventions, available objects, etc. Including the system prompt in the training input teaches the model to condition its generation on the specific robot platform, which is what enables the cross-embodiment generalization results (the model learns to attend to the prompt to determine which API is currently available).

Training Configuration. The paper reports the following hyperparameters (Appendix VI-H):

• Number of training steps: covering 10 epochs of the available training data.
• Optimizer: Adam.
• Learning rate: $5 \times 10^{-3}$ (the paper writes this as 5e-3).
• Learning rate schedule: linear ramp-up followed by cosine decay.
• Batch size: 4.
• Context length: 4096 tokens.
• Training data size: after data augmentation, approximately 3 million tokens.

The dataset is small by LLM fine-tuning standards: 299 successful chat sessions from the initial data collection phase, with a median session length of 1800 tokens and maximum of 3900 tokens. To address this data scarcity and "make LLM responses more robust to small differences in user feedback," the paper applies data augmentation (Section VI-E): for each training session, 5 variations of the user's text inputs (both the initial task instruction and all intermediate feedback messages) are generated by PaLM 2-L, a larger model from the same family. The robot code is not augmented — only the human language is varied. The augmentation prompt asks PaLM 2-L to rewrite the original text using synonyms, rephrasing, changes in grammatical structure, sentence length, and punctuation, with a generation temperature of 0.8 to ensure diversity. For example, a user's request "pick up the cube" is rewritten into "grab the cube and raise it," "lift up the cube," and "raise the cube." The original data plus the 5 augmentations yields the ~3M token training set.

Training Objective. The model is trained with standard causal language modeling: given the sequence of tokens up to position $k$, predict the token at position $k+1$. The loss is the cross-entropy between the predicted token distribution and the ground-truth next token. Crucially, the loss is computed over all tokens in the chat session, including both the robot code tokens and the human text tokens. This means the model learns to predict not just what code it should output at each turn, but also what the user might say next — it learns a joint model of the interaction dynamics.

During training, the input to the model is the system prompt $P$ concatenated with the initial user instruction $h_0$. The target is the entire remainder of the chat session: $c_0, h_1, c_1, \dots, h_T, c_T, r$. This is illustrated in Figure 4 (center panel): the left side of the figure shows a partial chat history (system prompt + first user instruction), the middle shows the LLM generating the remainder, and the right shows the complete predicted trajectory.

Why this form of training matters — the dynamics model interpretation. When an LLM is trained to autoregressively predict $h_{t+1}$ (human feedback) given $[P, h_0, c_0, \dots, c_t]$ (the history so far), it learns a model of how humans respond to robot code. When it predicts $c_{t+1}$ given the same history plus the predicted $h_{t+1}$, it learns the policy — what code to write given the human's feedback. Together, these learned distributions constitute a transition dynamics model of the human-robot interaction POMDP: given the current state of the conversation (which approximates the true latent state of the user's intent), the model can simulate forward — predicting what the user would say, what code the robot would write in response, how the user would react to that code, and so on, until eventual success or failure.

This is analogous to model-based reinforcement learning, where a learned dynamics model $\hat{p}(s_{t+1} | s_t, a_t)$ is used to plan action sequences by simulating forward. The difference is that here the "state" is the language context, the "action" is generated code, and the "dynamics" include both the environment (how the robot moves) and the human (how they give feedback) — both of which are collapsed into the text prediction task.

One subtlety: because training is only on successful trajectories, the learned dynamics model is biased toward successful outcomes. It learns the distribution of human-robot interactions conditioned on eventual success, not the unconditional distribution. This is intentional: at inference time, when the model simulates possible futures, it tends to produce futures that lead to success, which is exactly what the search strategy requires.

Inference-Time Search via Model Predictive Control. The trained LMPC-Rollouts model is used at inference time in a specific decoding procedure that the paper calls "model predictive control as a sequence-level decoding strategy" (Section III-C). The procedure operates at each chat turn:

1. Condition the model on the system prompt $P$ and the chat history so far (all previous user messages and robot code responses for the current session).
2. Sample 8 rollouts from the fine-tuned LLM with non-zero temperature (next-token sampling with $T > 0$, exact temperature not specified). Each rollout is an autoregressively generated completion of the chat session, up to a maximum of 4096 tokens. A rollout is a complete imagined future: it includes predicted human feedback messages, predicted robot code responses, and (if it reaches termination) a success or failure label.
3. Check for termination. If a sampled rollout reaches a success label ($r = 1$, encoded as the appropriate text token) within the 4096-token limit, it is considered a "successful" predicted trajectory. Rollouts that do not terminate (exceed the token limit without reaching a success label) are treated as unsuccessful — but the paper notes that because the model is only trained on successful data, it rarely generates failure-terminated rollouts explicitly.
4. Select the best trajectory. Among the rollouts that reached success, choose the one with the fewest predicted chat turns (i.e., the shortest path to success). This is a cost-minimization criterion: if a trajectory reaches success in 2 turns and another reaches it in 4 turns, the 2-turn trajectory is preferred.
5. Extract the next action. From the selected trajectory, take only the first robot code output $c_{t+1}$ (the code that the predicted trajectory says should be output next, in response to the current chat history). Discard the rest of the trajectory.
6. Execute $c_{t+1}$ by sending the code to MJPC for robot execution.
7. Wait for the user's next feedback $h_{t+1}$, then repeat the entire process (steps 1-6) — this is the "receding horizon" aspect: at each new chat turn, the model re-plans from the updated history.

If no sampled rollout terminates within the 4096-token limit (i.e., all 8 samples generate incomplete trajectories without a success label), the paper states that the system "randomly pick[s] a trajectory and return[s] its $a_{t+1}$." This is a fallback mechanism for when the model cannot find any path to success in its imagined futures.

Why this search strategy is (approximately) optimal for teachability. The paper motivates the "fewest predicted chat turns" selection criterion as derived from "optimizing a cumulative cost in the trajectory (assuming a sparse reward of 1 for success and a constant time penalty)." Formally, this corresponds to an objective:

$\max_{\text{trajectory}} \left[ \mathbb{1}[\text{success}] - \lambda \cdot (\text{number of chat turns}) \right]$

where $\lambda$ is a positive constant that penalizes longer interactions. The MPC search approximates this by first filtering for trajectories that reach success (satisfying the sparse reward), then selecting the shortest one (minimizing the time penalty). This directly aligns with the definition of teachability: reducing the number of chat turns before success.

Why MPC rather than beam search or greedy decoding. Standard decoding strategies in language models — greedy (pick the most likely next token), beam search (maintain k most likely partial sequences), or nucleus sampling (sample from the top-p probability mass) — optimize for likelihood under the model, not for task success with minimal turns. Greedy decoding on a model trained only on successful trajectories might produce a reasonable completion, but it has no mechanism to explicitly prefer shorter successful trajectories over longer ones. MPC search, by sampling multiple full trajectories and selecting based on a downstream objective (shortest path to success), decouples the generation process from the evaluation criterion. The model generates candidate futures according to its learned distribution; the MPC controller evaluates those futures according to a task-specific cost function. This is exactly the model-based control paradigm: a learned dynamics model proposes possible state-action sequences, and a separate planning algorithm selects among them.

Why 8 rollouts? The paper does not provide an ablation on the number of rollouts, but 8 represents a pragmatic tradeoff. More rollouts increase the probability of finding a high-quality (short) trajectory to success, but also increase inference latency. Since LMPC-Rollouts must generate up to 4096 tokens per rollout, and 8 rollouts means generating up to 32,768 tokens per chat turn, the inference cost is substantial. The paper reports that LMPC-Rollouts inference is 7.4 seconds on average without quantization and 1.0 seconds with 8-bit quantization (Table XIV), compared to 1.1 seconds for LMPC-Skip. With quantization, the overhead of sampling 8 rollouts is mitigated.

Why 4096 max tokens? This is the context length used during training (Appendix VI-H), and using the same length at inference ensures that the model's predictions stay within the distribution it was trained on. Since the median training session is 1800 tokens, 4096 is more than double the median, providing ample room for the model to generate complete imagined trajectories.

Distribution shift between training and inference for the revision model is addressed by fine-tuning the base model itself. The model being fine-tuned is the same PaLM 2-S that was used for data collection. This ensures that the generated data is on-policy with respect to the base model. However, after fine-tuning, the model's output distribution changes — it becomes better at writing code and responding to feedback. This introduces a subtle distribution shift: the data used to train LMPC was collected from the base model, but after fine-tuning, the model's own behavior differs from what the training data reflects. The paper partially mitigates this by using data augmentation (making the training data more robust to variations in user input), but the fact that multiple fine-tuning iterations (Iteration 2) do not yield further improvements (Table V) suggests that the benefits of fine-tuning saturate when the training data is collected with the base model rather than the improved model. This is discussed further in Section V.

LMPC-Skip: The Diagnostic Ablation

LMPC-Skip (Section IV-C, Figure 4 right panel) is a simplified fine-tuning variant that serves as an ablation to isolate the value of modeling intermediate interaction dynamics. It is trained on the same dataset of successful chat sessions, but with a different input-target structure:

• Input: System prompt $P$ + initial user instruction $h_0$ + (optionally) the user ID for top-user conditioning.
• Target: Only the final correct robot code $c_T$ (the code that was present at the end of a successful session), with all intermediate user feedback $h_1, \dots, h_T$ and intermediate code responses $c_0, \dots, c_{T-1}$ removed from the target sequence.

In other words, LMPC-Skip is trained to map directly from the initial instruction (plus system prompt) to the code that eventually satisfied the user, skipping all the corrective turns in between. It is effectively a supervised model that learns "given this initial instruction, what code ended up working?" — with no awareness of how the user and robot interacted to get there.

Training objective is the same as LMPC-Rollouts: standard autoregressive next-token prediction, but with a much shorter target sequence (just the final code block).

Inference is drastically simpler: at each chat turn, LMPC-Skip is conditioned on the system prompt with the chat history so far, and it is queried once (no sampling, no search) to generate the response. A single forward pass produces the code output. This makes LMPC-Skip significantly faster: 1.1 seconds per response versus 7.4 seconds for unquantized LMPC-Rollouts (Table XIV).

What the LMPC-Skip comparison tests. The difference between LMPC-Rollouts and LMPC-Skip isolates two factors:

1. Modeling intermediate interactions: LMPC-Rollouts is trained to predict the full back-and-forth, including how users provide feedback and how code evolves. LMPC-Skip sees only the initial instruction and the final code. If LMPC-Rollouts outperforms LMPC-Skip on multi-turn tasks, it indicates the value of learning the interaction dynamics.
2. Inference-time search: LMPC-Rollouts uses MPC with 8 sampled rollouts and shortest-path selection. LMPC-Skip uses single-pass greedy decoding. If the search procedure contributes independently to performance, it should show up in the comparison.

The paper's results (Table I) show that LMPC-Skip achieves the highest 1-turn success rate (34.6% on train, 41.7% on test) — better than LMPC-Rollouts (23.5% train, 34.8% test). This makes sense: LMPC-Skip is explicitly optimized to produce the final correct code from just the first instruction, so it excels at zero-shot task completion. However, LMPC-Rollouts achieves the highest 2+ turn success rate (28.4% train, 31.5% test) — better than LMPC-Skip (21.4% train, 17.8% test). This also makes sense: LMPC-Rollouts is trained to model how interactions unfold and can use MPC search to navigate multi-turn corrections, so when the first attempt is wrong, it is better able to incorporate feedback and recover.

The paper suggests a practical deployment strategy based on these complementary strengths: "use LMPC-Skip for responding to the initial user instruction, then LMPC-Rollouts for responding to subsequent user feedback."

Top-User Conditioning

Top-user conditioning (Section III-C, Appendix VI-C) is a mechanism for biasing the model's generations toward the interaction patterns of the most proficient users in the training dataset. It operates in three stages: (1) identification of top users from training data, (2) modification of training prompts to include user identity labels, and (3) conditioning inference on a special "top-user" label.

Stage 1: Autonomous Top-User Identification. The paper defines a quantitative procedure for ranking users by their teaching proficiency, evaluated on training tasks only (so as not to leak information about test-task difficulty). Let there be $N$ tasks and $K$ users. Define:

$s(n, k) = \text{ self-reported success rate of user } k \text{ on task } n$

$c(n, k) = \text{ number of times user } k \text{ taught task } n$

$\bar{c}(n, k) = \mathbb{1}(c(n, k) \geq 1) = \text{ indicator of whether user } k \text{ taught task } n \text{ at least once}$

The task difficulty $d(n)$ is defined as the average failure rate across all users who taught that task:

$d(n) = 1 - \frac{1}{K_n} \sum_{k=1}^K s(n, k) \bar{c}(n, k)$

where $K_n = \sum_{k=1}^K \bar{c}(n, k)$ is the number of users who taught task $n$ at least once.

What this computes: for each task, take all users who taught it at least once, compute their average success rate on that task, and subtract from 1. A task that all users find difficult (low average success rate) gets a high difficulty score $d(n) \to 1$. A task that most users find easy (high average success rate) gets a low difficulty score $d(n) \to 0$.

Why weight by difficulty: without difficulty weighting, a user who only teaches easy tasks would appear highly proficient simply because their tasks are easy. Weighting by task difficulty normalizes for this: succeeding on a hard task earns a user more "credit" than succeeding on an easy task.

The user performance score $h(k)$ is then:

$h(k) = \sum_{n=1}^{N_k} d(n) s(n, k) \bar{c}(n, k)$

where $N_k = \sum_{n=1}^N \bar{c}(n, k)$ is the number of distinct tasks user $k$ has taught.

What this computes: for each user, iterate over all tasks they've taught at least once, multiply their success rate on that task by the task's difficulty, and sum. A user who succeeds on many hard tasks gets a high score; a user who succeeds only on easy tasks, or who fails on hard tasks, gets a lower score.

Top users are defined as those in the top 75th percentile by $h(k)$ — i.e., the top 25% of users. From the initial data collection with the base model (PaLM 2-S), 10 out of 35 users were identified as top users. These top users covered only 11 out of 50 train tasks (they were not uniformly assigned tasks — each user taught whatever the random task sampler assigned).

Stage 2: Training with User Identity Labels. During training data preparation, each chat session is annotated with the unique ID of the user who generated it. A special ID string, "top-user," replaces the actual user ID for sessions from top users. The training prompt is modified to include a line like:

User ID: [actual_user_id or 'top-user']

This means the model sees, at the beginning of each training sequence, an explicit token identifying which user (or user category) generated the following interaction. The model learns to condition its predictions on this identity — it can learn that interactions from "top-user" tend to have different patterns (more efficient corrections, clearer feedback, different vocabulary) than interactions from other specific users.

Stage 3: Inference with Top-User Conditioning. At inference time, regardless of which actual user is currently interacting with the system, the prompt always includes:

User ID: top-user

This forces the model to generate responses as if it were interacting with a top user — i.e., it generates code in the style that was effective for top users during training, and it predicts that subsequent user feedback will follow top-user patterns.

Why this works — the distribution-shift interpretation. Within the POMDP analogy, different users correspond to different observation models: when two users have the same intended behavior in mind, their language feedback may differ systematically. A top user might say "move the end effector 5cm along the positive y-axis" while another user might say "a little bit more that way" — both convey the same intent but with different precision. By conditioning on the top-user label, the model is effectively assuming that observations (human language) come from the top-user distribution, which is characterized by "less noise" (the paper's phrasing). This means the model's predicted futures are more likely to converge quickly to success, because the simulated human feedback in those futures is clearer and more actionable. Even when the actual user is not a top user — and their real feedback is noisier — the model's first action $c_t$ (extracted from the shortest successful imagined trajectory) is based on an optimistic simulation, which tends to produce code that is more robust and goal-directed.

Empirical justification. Table II (main paper) reports that removing top-user conditioning significantly degrades performance:

• Training on all users' data but without conditioning on the top-user label: LMPC-Rollouts drops 8.4% on train tasks and 10.5% on test tasks; LMPC-Skip drops 16.3% on train and 26.1% on test. This shows that merely having all users' data is insufficient — the conditioning mechanism is what extracts value from the top users' interaction style.
• Training on only top users' data (filtering out other users): LMPC-Rollouts drops 23.8% on train and 21.7% on test; LMPC-Skip drops 9.6% on train and 13.6% on test. This shows that top-user data alone is insufficient — the model needs the diversity of the broader user population to learn robust behaviors, and the top-user conditioning acts as a "style transfer" mechanism that elevates the quality of generations drawn from a model trained on a diverse distribution.

The key insight is that top-user conditioning decouples data quality from data quantity: by including a small amount of high-quality data (10.7% of training data) with a special label, and conditioning inference on that label, the model can produce high-quality outputs even when trained predominantly on lower-quality data. This is analogous to classifier-free guidance in diffusion models, where conditioning on a "high-quality" label during inference shifts the generation distribution toward higher-quality outputs without requiring the training distribution to be exclusively high-quality.

Data Augmentation Strategy

Because the training dataset is extremely small by LLM standards (299 successful sessions, ~3M tokens after augmentation), data augmentation plays a critical role. The paper describes the procedure in Appendix VI-E:

Augmentation method: For each user text input in the training data (both initial task instructions $h_0$ and all subsequent feedback messages $h_1, \dots, h_T$), 5 variations are generated by PaLM 2-L, a larger model from the same PaLM 2 family. The augmentation prompt asks PaLM 2-L to rewrite the original text:

"in $K$ different ways by replacing words with synonyms, rephrasing, changing grammatical structure, sentence lengths, punctuation, etc."

The prompt also asks the model to output all $K$ variations in one batch and uses a generation temperature of 0.8 to ensure diversity among the variations.

What is NOT augmented: The robot code responses $c_t$ are left unchanged. Only the human language inputs are varied. This preserves the functional correctness of the code while creating robustness to different ways users might phrase the same intent.

Examples of augmentations (from the paper):

• Original: "pick up the cube" → Variations: "grab the cube and raise it," "lift up the cube," "raise the cube"
• Original: "wrong direction, keep hopping but turn the opposite direction" → Variations: "that is the incorrect direction, maintain hopping but go the opposite way," "you are going the wrong way, keep hopping but turn in the opposite direction," "wrong direction, maintain hopping but turn the opposite way"

Impact of augmentation (Table XII): For LMPC-Skip, removing data augmentation causes a 7.1% drop on train tasks but no significant change on test tasks (+0.6%). For LMPC-Rollouts, the impact is reversed: removing augmentation causes a small improvement on train tasks (+2.8%) but a 7.0% drop on test tasks. The paper hypothesizes that:

"the generalization capabilities of LMPC-Rollouts benefits more from data augmentation than does LMPC-Skip. We hypothesize this is due to that data augmentation makes LMPC-Rollouts' chat session predictions more robust to compounding errors, leading to better predictions of feedback dialogue."

Why this matters — the compounding error hypothesis. LMPC-Rollouts must predict entire multi-turn trajectories, including both future code AND future human feedback. If the model is overfit to the specific phrasing of user messages in the training data, small variations in actual user input can cause cascading prediction errors: a slightly off predicted human message leads to a slightly off predicted code response, which leads to an even more off predicted human reaction, and so on. Data augmentation with paraphrased user inputs trains the model to be invariant to phrasing, reducing the sensitivity of long-horizon predictions to initial perturbations. LMPC-Skip, by contrast, only predicts a single code output and does not need to maintain prediction coherence over multiple steps, so it benefits less from augmentation on test tasks (the single-step mapping from instruction to code may already generalize reasonably well without augmentation).

4. Key Insights and Innovations

Innovation 1: Framing Human-Robot Interaction as a POMDP Enables Training LLMs as Dynamics Models, Not Just Policies

The paper's most fundamental conceptual move is not a new architecture or training objective — it is a reframing of what it means to fine-tune an LLM on interaction data. Prior work on LLM alignment (RLHF, SFT) treats the model exclusively as a policy: a mapping from user input to preferred output, trained to maximize the probability of producing outputs that humans rate highly. This is a single-step decision-making view. The paper's POMDP analogy (Section III-A, III-C) recasts the same autoregressive training objective as learning a transition dynamics model — a model of how the joint human-robot system evolves over time, including both what code the robot should write and how the human will respond.

This reframing matters because it unlocks a different inference-time strategy. If you've trained a policy, the natural inference procedure is to query it once and execute its output. If you've trained a dynamics model, the natural inference procedure is to use it for planning: simulate multiple possible futures, evaluate them against a cost function, and select actions that lead to desirable outcomes. This is exactly what LMPC-Rollouts does — sample 8 imagined futures, select the shortest path to success, and execute only the first action, repeating at every step (receding horizon control). The model architecture and training objective (autoregressive next-token prediction) are unchanged from standard LLM fine-tuning; the innovation is in recognizing that when the training sequences are interactive trajectories, the resulting model is a world model that supports model-based control.

This is intellectually distinctive because it bridges two largely disconnected literatures: LLM alignment (where the focus is on single-turn preference optimization) and model-based reinforcement learning / control (where learned dynamics models are used for planning). The paper demonstrates that the standard SFT pipeline — which the LLM community already uses extensively — can be reinterpreted as dynamics learning when applied to multi-turn interaction data, and that adding a simple inference-time search strategy (MPC-style rollout selection) turns that dynamics model into a more teachable policy without any changes to the training procedure. The comparison with LMPC-Skip (Table I) validates that this is not just an effect of fine-tuning on task outcomes: LMPC-Skip is trained on the same data but only to predict final code, and it achieves higher 1-turn success (34.6% train, 41.7% test) but much lower 2+ turn success (21.4% train, 17.8% test). LMPC-Rollouts flips this (23.5% 1-turn, 28.4% multi-turn train; 34.8% 1-turn, 31.5% multi-turn test), confirming that modeling the process of interaction is what enables responsiveness to corrective feedback — the core of teachability.

Innovation 2: Top-User Conditioning as a Mechanism for Transferring Interaction Style Across Users and Tasks

The paper's most surprising empirical finding is that conditioning the model's generations on top-performing users — who covered only 14% of training tasks (11 out of 50) and 10.7% of training data — improves outcomes for all users on all tasks, including unseen test tasks, by 10.5% for LMPC-Rollouts (Table II, main paper). This is not an obvious result. Intuitively, biasing a model toward a small subset of users should help those users (by personalizing to their preferences) but potentially harm other users who have different interaction styles, vocabulary, or teaching strategies. The finding that the improvement transfers universally suggests something deeper: top users are not just "easier to satisfy" — their interaction data contains information about how to teach effectively that generalizes across tasks and users.

Within the POMDP framing, this makes sense: if different users correspond to different observation noise models (some provide more precise, actionable feedback than others), then conditioning on top users means the model simulates futures with lower observation noise during inference-time search. Even when the actual user is noisy, the model's first action — extracted from an imagined trajectory with a "clean" simulated user — tends to be more robust. The paper provides qualitative evidence that top users are more "kind" in their feedback (Figure 11, Appendix), which may correlate with patience and more constructive corrections. But the deeper point is that user identity can serve as a proxy for data quality, and conditioning on high-quality users during inference is a lightweight way to improve outputs without requiring all training data to be high-quality — analogous to classifier-free guidance in generative models.

This is a shift from how user conditioning is typically used in personalization (where the goal is to tailor responses to individual users) and from how performance conditioning works in Decision Transformers (where the conditioning variable is a scalar reward). Here, the conditioning label ("top-user") is not a reward but a categorical index into a style distribution that correlates with, but is distinct from, task success. The paper shows (Table II) that training on only top-user data degrades performance significantly (LMPC-Rollouts drops 23.8% on train, 21.7% on test), while training on all users' data without top-user conditioning also degrades performance (LMPC-Rollouts drops 8.4% train, 10.5% test). The combination — diverse training data plus top-user conditioning at inference — is what works. This establishes a practical design principle for interactive learning systems: collect data from a broad user base, identify high-quality subsets autonomously, and use conditioning to shift the generation distribution toward those subsets without discarding the diversity of the full dataset.

Innovation 3: Demonstrating That Fine-Tuning Improves In-Context Learning — A Meta-Learning Result with Practical Scale

The paper's title claims "learning to learn faster from human feedback," and the experiments are structured to test exactly this meta-learning hypothesis. The base model (PaLM 2-S) already exhibits non-zero teachability through in-context learning alone: users can teach it new tasks through multi-turn feedback, with a 34.8% success rate on train tasks and 39.4% on test tasks (Table I). The question is whether fine-tuning on data from these interactions produces a model that is more teachable on the next interaction — i.e., whether the improvement transfers to unseen tasks and embodiments.

The result is a clear "yes," but the structure of the improvement is what makes it intellectually distinctive. LMPC-Rollouts improves test-task success rates by 26.9% (from 39.4% to 66.3%) and reduces average chat turns from 2.4 to 1.9 (Table I). More importantly, it improves cross-embodiment generalization: when trained only on Robot Dog, Mobile Manipulator, and Aloha data, LMPC-Rollouts improves success rates on the held-out Bi-arm Kuka and Kuka+Hand embodiments by 31.5% (Table IV). This is a strong meta-learning result because the test embodiments have entirely different robot APIs, coordinate systems, and manipulation primitives — the model cannot succeed by memorizing task-specific code patterns from training. It must have learned something more abstract about how to respond to feedback and how to compose API calls to satisfy user intent.

What makes this meta-learning, rather than just better instruction-following, is the teachability metric itself: the measure of improvement is not zero-shot accuracy (though LMPC-Rollouts does improve 1-turn success rates) but the efficiency of adaptation to corrective feedback. The "2+ turn Success Rate" metric in Table I captures this precisely: it measures success on sessions where the first response was incorrect and the model had to incorporate feedback. LMPC-Rollouts achieves 31.5% on this metric for test tasks, compared to 21.9% for the base model and 17.8% for LMPC-Skip. The model has learned to learn from corrections more efficiently — fewer turns, higher ultimate success. This is a different capability from zero-shot task completion, and it emerges from training on interaction trajectories without any explicit meta-learning algorithm (no MAML, no Reptile, no second-order gradients) — just standard SFT with a particular data representation and inference strategy.

Innovation 4: The Complementary Roles of LMPC-Rollouts and LMPC-Skip as a Diagnostic Decomposition of Teachability

The comparison between LMPC-Rollouts and LMPC-Skip is not just an ablation — it is a diagnostic decomposition of teachability into two components: (1) the ability to produce correct code from the initial instruction (zero-shot performance), and (2) the ability to incorporate corrective feedback when the initial attempt is wrong (multi-turn adaptation). Table I shows that these capabilities are in tension: LMPC-Skip maximizes the first (41.7% 1-turn success on test tasks vs. 34.8% for LMPC-Rollouts) at the cost of the second (17.8% multi-turn vs. 31.5%), while LMPC-Rollouts trades off some zero-shot accuracy for dramatically better feedback responsiveness.

This decomposition is conceptually important because it reveals that the training objective that optimizes for final-code prediction (LMPC-Skip) produces a different capability profile than the one that optimizes for interaction dynamics prediction (LMPC-Rollouts). The fact that both are trained on the same underlying data (successful chat sessions) but produce complementary strengths means that the representation of the training target — full trajectory vs. final code — determines which aspect of teachability the model acquires. This has implications beyond robotics: it suggests that for any interactive system where users provide sequential feedback, training on full interaction trajectories rather than just final outcomes may be essential for learning to use that feedback effectively.

The paper's practical recommendation — "use LMPC-Skip for responding to the initial user instruction, then LMPC-Rollouts for responding to subsequent user feedback" — is not just an engineering heuristic. It points toward a more general architecture where different fine-tuned variants of the same base model are deployed at different stages of an interaction, each specialized for the type of adaptation needed at that stage. This is a departure from the standard paradigm of training a single "best" model and using it uniformly.

5. Experimental Analysis

Evaluation Methodology

• Dataset. All teachability experiments use data collected from 35 non-expert users interacting with the robot teaching system through a chat interface. These users are "not researchers or engineers, and they are not familiar with the underlying LLMs or robot code" (Section IV-A). The task set comprises 78 tasks across 5 robot embodiments: Robot Dog (19 tasks), Mobile Manipulator (14 tasks), Aloha (16 tasks), Bi-arm Kuka (16 tasks, test only), and Kuka+Hand (18 tasks, test only). Tasks range from simple locomotion commands (e.g., "sit," "hop") to complex dexterous manipulation (e.g., "insert the connector into the socket"). The tasks are manually split into 51 train tasks (65%) and 27 test tasks (35%), with the split chosen before experiments began, when task difficulty and baseline performance were unknown. Data collection aggregates results across 2 days per model variant, with each user contributing approximately 10 chat sessions per day, yielding roughly 350 sessions per day across all users. The total training dataset after filtering contains 299 successful chat sessions from the initial data collection with the base model, which after data augmentation (5 paraphrases per user input) expands to approximately 3 million tokens (Section IV-A, Appendix VI-H).

• Base model(s). All experiments use PaLM 2-S (Anil et al., 2023), the smallest variant in the PaLM 2 family. The authors chose this model because it "exhibits non-zero initial task success rates given feedback from the user" (Section III-B) — meaning it has enough base capability to collect meaningful interaction data, but substantial room for improvement through fine-tuning, making it well-suited for measuring teachability gains. The data augmentation step uses PaLM 2-L, a larger model from the same family, to generate paraphrases of user inputs (Appendix VI-E). No other model families (e.g., GPT, LLaMA) are evaluated.

• Metrics. The paper defines several complementary metrics (Section IV-D, Table I):

  • Success Rate: The fraction of chat sessions where the user clicked "success" (indicating the robot completed the assigned task to their satisfaction). This is the primary evaluation metric.
  • Num Chat Turns: The mean number of chat turns (user-input, LLM-output pairs) in successful chat sessions. Lower is better — it directly operationalizes teachability as defined in Section III-A ("average number of human inputs n before the robot succeeds").
  • Good Rating Rate: The proportion of individual chat turns (after the first turn) that the user rated as "good" (indicating the robot responded correctly to the most recent feedback). This captures responsiveness to corrective feedback per turn, not per session. The paper reports in Figure 3 (bottom right) that this metric correlates with task success (r = 0.92).
  • Successful Tasks Rate: The fraction of tasks for which at least one chat session achieved success, measuring the breadth of tasks the model can learn.
  • 1-turn Success Rate: The proportion of sessions successful in exactly one chat turn — equivalent to zero-shot instruction following performance.
  • 2+ turn Success Rate: The proportion of sessions successful in two or more chat turns, capturing multi-turn adaptation ability. This is computed as the overall success rate minus the 1-turn success rate.
  • Teachability curves: Task success as a function of ≤N chat turns (Figure 5, Figure 8), presented as a curve where models with better teachability are "higher and to the left."

• Baselines. Four main configurations are compared (Section IV-C):

  • PaLM 2-S (base model): The frozen pretrained LLM with in-context learning only, evaluated via the standard prompt format described in Section III-B. This represents the baseline teachability achievable without any fine-tuning.
  • RAG (Retrieval-Augmented Generation): A retrieval baseline following the approach of Zha et al. (2024), implemented using a pretrained Gecko embedding model (based on PaLM 2) to retrieve relevant (instruction, successful code) pairs from the training dataset. For each new instruction, the system retrieves the 5 most relevant data points from the same robot embodiment (selected via closest 30% by cosine similarity, then farthest-point sampling for diversity, ordered from lowest to highest relevancy). These are inserted into the LLM context as in-context examples. This baseline tests whether improvements can be achieved without fine-tuning. Full details in Appendix VI-D.
  • LMPC-Skip: Fine-tuned PaLM 2-S trained to predict only the final correct code from the initial instruction (and system prompt), skipping all intermediate interactions (Section IV-C, Figure 4 right). At inference, queried once per chat turn with no sampling or search. Trained only on successful chat sessions.
  • LMPC-Rollouts (the proposed method): Fine-tuned PaLM 2-S trained to predict entire chat session rollouts (human feedback + robot code + success label), with inference-time MPC search over 8 sampled rollouts, selecting the shortest path to success (Section III-C, Figure 4 center).

• Generation budget / compute accounting. The paper does not use a unified "generation budget" or FLOPs accounting to compare methods. Instead, each method is evaluated under its natural operating regime: PaLM 2-S and RAG generate one code response per chat turn (single forward pass); LMPC-Skip generates one response per chat turn (single forward pass); LMPC-Rollouts generates 8 sampled rollouts per chat turn (8 forward passes, each up to 4096 tokens). Inference time is reported in Table XIV: LMPC-Skip at 1.1±0.2 seconds, LMPC-Rollouts at 1.0±0.4 seconds (with 8-bit quantization), and LMPC-Rollouts without quantization at 7.4±4.7 seconds. The paper uses quantization to make LMPC-Rollouts competitive in latency, noting that "without quantization, the slowed inference time degrades the teaching experience" (Appendix VI-H). There is no systematic study trading off number of rollouts vs. success rate.

• Cross-validation / statistical protocol. Evaluation uses blind A/B testing: during data collection, the Task Sampler randomly selects which model variant the user interacts with, and users are unaware of which model they are teaching (Section IV-A). All experiment numbers are computed from data collected under this blind sampling protocol. Results are aggregated across all users, embodiments, and tasks within each model variant. For cross-embodiment generalization experiments (Table IV), models are trained on data from three embodiments (Robot Dog, Mobile Manipulator, Aloha) and evaluated on held-out embodiments (Bi-arm Kuka, Kuka+Hand). No traditional k-fold cross-validation is performed; instead, the train/test task split is fixed a priori. Statistical significance tests (p-values) are reported only for the user performance drift analysis (paired t-tests, N=13, Appendix VI-I), not for the main teachability comparisons. The paper acknowledges that some comparisons have low per-task sample sizes: "the median count for each (model variant, task) tuple is only 4, and the median count for each (model variant, task, user) tuple is only 1" (Appendix VI-B).

Main Quantitative Results

Overall Teachability: Fine-Tuning Outperforms In-Context Learning and Retrieval

The headline result is that LMPC-Rollouts improves test-task success rate by 26.9 percentage points over the base PaLM 2-S (from 39.4% to 66.3%), while reducing the average number of chat turns in successful sessions from 2.4 to 1.9 (Table I, Section IV-D). This is the core teachability improvement: users achieve higher success rates with fewer rounds of correction.

The teachability curves in Figure 5 (main paper, all embodiments aggregated) show:

• At 1 chat turn, the base model (PaLM 2-S) achieves approximately 17.5% success on test tasks. LMPC-Skip reaches 41.7%, LMPC-Rollouts reaches 34.8%, and RAG reaches 27.9% — confirming that fine-tuning substantially improves zero-shot performance, with LMPC-Skip having the strongest 1-turn result.
• At 2+ chat turns, the ordering reverses for multi-turn performance: LMPC-Rollouts continues climbing to an ultimate success rate of 66.3%, while LMPC-Skip plateaus at 59.4%. The base model reaches only 39.4%, and RAG reaches 51.9%.
• LMPC-Rollouts "matches or exceeds the final success rate of the base model after just one chat turn" — meaning that with a single user instruction and no corrections, the fine-tuned model already outperforms what the base model achieves after up to 7 turns of feedback.

Table I breaks these results into detailed metrics. On test tasks:

• Good Rating Rate (responsiveness to corrective feedback): LMPC-Rollouts achieves 26.5%, followed by LMPC-Skip at 24.7%, RAG at 20.9%, and the base model at 18.1%. This confirms that fine-tuned models not only succeed more often but are rated as responding correctly to individual pieces of feedback more frequently.
• Successful Tasks Rate: Both LMPC-Rollouts and LMPC-Skip achieve 88.9% on test tasks, meaning nearly 9 out of 10 tasks have at least one successful teaching session. The base model achieves 81.5%, while RAG degrades to 75.0% — lower than the base model. This is a striking result: RAG improves overall success rate (by helping on tasks similar to retrieved examples) but reduces the breadth of tasks that can be taught at all (suggesting it overfits to the retrieval distribution and fails on novel tasks).
• 1-turn Success Rate vs. 2+ turn Success Rate: LMPC-Skip dominates 1-turn success (41.7% test) but trails on 2+ turn (17.8%). LMPC-Rollouts is competitive on 1-turn (34.8%) and dominates on 2+ turn (31.5%). The base model achieves 17.5% 1-turn and 21.9% 2+ turn; RAG achieves 27.9% 1-turn and 24.0% 2+ turn.

For train tasks, the pattern is similar but with LMPC-Skip showing a larger overall advantage (56.0% vs. 51.9% for LMPC-Rollouts, vs. 34.8% for base, vs. 46.4% for RAG), suggesting LMPC-Skip overfits more to seen tasks. The paper explicitly notes this: "LMPC-Skip overfits to train tasks, while LMPC-Rollouts generalizes better on unseen test tasks for multi-turn sessions" (Section IV-D, referring to Figure 5).

Cross-Embodiment Generalization

Table IV reports results from an experiment where LMPC models are trained on data from only three embodiments (Robot Dog, Mobile Manipulator, Aloha) and evaluated on held-out embodiments (Bi-arm Kuka, Kuka+Hand) that have entirely different robot APIs, coordinate systems, and manipulation primitives. The results show:

• LMPC-Rollouts improves test embodiment success rate by 31.5% over the base model, compared to 18.6% for LMPC-Skip.
• On train embodiments (test tasks), LMPC-Rollouts improves by 23.8% vs. LMPC-Skip's 19.0%.
• On train embodiments (train tasks), LMPC-Skip improves more (28.8% vs. 17.2%), consistent with its tendency to overfit to seen task distributions.

The paper notes that "higher improvements in test tasks and embodiments are caused by the train:test split not being explicitly selected for uniform task difficulty and baseline performance; doing so is infeasible as the split needs to be chosen before starting evaluations, when task difficulty and baseline performance were unknown" (Table IV caption). This is an important caveat: the larger test-task improvements may partially reflect the test tasks being easier on average (as evidenced by the base model's higher test-task success rate of 39.4% vs. 34.8% on train tasks).

Top-User Conditioning: Small Data, Large Impact

Table II (main paper) quantifies the contribution of top-user conditioning through two ablations performed on the Robot Dog and Mobile Manipulator embodiments (due to time constraints):

• Removing top-user conditioning while keeping all users' data: LMPC-Rollouts drops 8.4% on train tasks and 10.5% on test tasks; LMPC-Skip drops 16.3% on train and 26.1% on test tasks. The larger drop for LMPC-Skip suggests that top-user conditioning is even more critical when the model lacks inference-time search to compensate for distribution shift.
• Training only on top-user data (filtering out the other 25 users): LMPC-Rollouts drops 23.8% on train and 21.7% on test; LMPC-Skip drops 9.6% on train and 13.6% on test. The massive drop for LMPC-Rollouts (and notably smaller drop for LMPC-Skip) indicates that LMPC-Rollouts depends more heavily on data diversity for its generalization — top-user data alone (covering only 11 of 50 train tasks) is insufficient to learn robust interaction dynamics, whereas LMPC-Skip can extract useful final-code mappings from a narrower distribution.

The paper concludes that "the combination of high-quality data from top-users, along with a much bigger (9x) set of more diverse data from other users, proves to be essential for finetuned model performance" (Appendix VI-C). Top users constitute 10 out of 35 users (top ~29%, despite the paper targeting top 25%) and their data accounts for only 10.7% of training examples.

Table XI (Appendix VI-C) further breaks down test-task improvements by user group:

• LMPC-Rollouts improves success rates for top users by +26.3% and for other users by +18.9%.
• LMPC-Skip improves success rates for top users by +15.1% and for other users by +14.2% — a more uniform distribution of gains.

The paper interprets this as evidence that LMPC-Rollouts' "inference-time search (via MPC) over future interactions performs better at catering to improving the teachability of top users" (Appendix VI-C), while LMPC-Skip's gains are more evenly spread.

RAG: Competitive but Limited Generalization

The RAG baseline (Section IV-C, Appendix VI-D) improves upon the base model significantly: 46.4% success on train tasks and 51.9% on test tasks (Table I), compared to 34.8% and 39.4% for the base model. However, it consistently underperforms both fine-tuned models. The most diagnostic metric is Successful Tasks Rate: RAG achieves 75.0% on test tasks, which is lower than the base model's 81.5%. This means RAG's improvements come from making already-succeedable tasks succeed more often (boosting success rate on tasks similar to retrieved examples), while actually reducing the set of tasks that can be taught at all. The paper explains: "while RAG may be proficient at increasing the success rate of tasks similar to the retrieved examples, it struggles to perform well on novel tasks" (Section IV-D).

On the 1-turn vs. 2+ turn decomposition, RAG achieves 27.9% 1-turn and 24.0% 2+ turn on test tasks — balanced but modest improvements over the base model (17.5% and 21.9%). Per-embodiment breakdowns in Figure 8 (Appendix VI-B) show RAG's performance varies substantially across embodiments, with notably weaker performance on Aloha and Kuka+Hand compared to fine-tuned models.

Real-World Evaluations

Table III reports real-robot evaluations comparing PaLM 2-S and LMPC-Rollouts on 8 tasks across the Robot Dog and Mobile Manipulator platforms:

• LMPC-Rollouts achieves 75% average success rate versus 53.8% for PaLM 2-S.
• Average chat turns are identical (2.9 for both), meaning LMPC-Rollouts achieves higher success without requiring more user effort on these tasks.
• Task-level results show substantial variation: on "hop," LMPC-Rollouts achieves 100% vs. 25% for the base model; on "walk forward in a trotting gait," 100% vs. 25%; on "knock over coke can," both achieve only 20%. The hardest task, "hop while turning counterclockwise," yields 25% for both models.
• Starred tasks (*) in the table are test tasks (not seen during training), confirming that LMPC-Rollouts' generalization extends to physical deployment.

Table XIII (Appendix VI-G) provides a sim-vs-real comparison, showing that some tasks transfer well (open drawer: 100% sim, 100% real for both models) while others exhibit significant sim-to-real gaps (knock over coke can: 100% sim but 20% real for LMPC-Rollouts; hop while turning: 100% sim but 25% real). The paper attributes these gaps to physical modeling discrepancies (end-effector velocity insufficient for knocking over objects) and the distilled policy's training distribution not covering the taught behaviors (for agile locomotion).

Multiple Fine-Tuning Iterations Do Not Help

Table V evaluates whether a second iteration of fine-tuning — training on data collected with the Iteration 1 fine-tuned models — yields further improvements:

• LMPC-Skip Iteration 2: +5.1% on train tasks, but -4.7% on test tasks.
• LMPC-Rollouts Iteration 2: -5.5% on train tasks, -1.9% on test tasks.

Both models show degradation or minimal change in test-task performance. The paper hypothesizes that "the data distribution or data amount used to train the second iteration of models do not differ significantly from that of the first iteration, so the resultant model behaviors remain largely unchanged" (Section IV-D). This is a notable negative result: the fine-tuning gains saturate after one iteration when the data distribution is held constant. The paper suggests that "adapting the data distribution, with methods like active task exploration or synthetic data generation, may unlock additional performance gains" (Section V).

Ablation Studies and Robustness Checks

• Training on both successes and failures (vs. successes only): Table VIII (Appendix VI-B) shows that training LMPC-Rollouts on both successful and failed chat sessions (with the model learning to predict session outcomes) reduces performance by 11.5% on train tasks and 14.0% on test tasks compared to training only on successes. The paper hypothesizes three potential reasons: (1) at inference time, generating both successes and failures reduces the pool of successful rollouts available for MPC selection, (2) finetuning on both generation and rating may be too difficult as a single objective, and (3) autoregressive generation may blend success and failure distributions without cleanly conditioning on the outcome. This finding motivates the design choice to train exclusively on successful trajectories, which the paper notes is also necessary for a fair comparison with LMPC-Skip (which can only be trained on successes since there is no "final correct code" in failure sessions).

• Data augmentation impact: Table XII (Appendix VI-E) ablates the data augmentation procedure. Without augmentation, LMPC-Skip drops 7.1% on train tasks but is essentially unchanged on test tasks (+0.6%). LMPC-Rollouts shows the opposite pattern: +2.8% on train tasks but -7.0% on test tasks. The paper interprets this asymmetry as evidence that "the generalization capabilities of LMPC-Rollouts benefits more from data augmentation than does LMPC-Skip," hypothesizing that augmentation makes "chat session predictions more robust to compounding errors, leading to better predictions of feedback dialogue." This is consistent with the intuition that LMPC-Rollouts, which must maintain prediction coherence over multi-turn rollouts, is more sensitive to overfitting on specific phrasings of user input.

• LMPC-Rollouts quantization for inference speed: Table XIV (Appendix VI-H) compares inference times. LMPC-Rollouts without quantization: 7.4±4.7 seconds. LMPC-Rollouts with 8-bit quantization: 1.0±0.4 seconds. LMPC-Skip: 1.1±0.2 seconds. The paper states that quantization does not cause "noticeably performance drops" (Appendix VI-H), making LMPC-Rollouts latency-competitive with LMPC-Skip despite performing 8 sampled rollouts per turn. No quantitative accuracy comparison between quantized and unquantized LMPC-Rollouts is reported.

• Code-writing benchmark retention: Table XVI (Appendix VI-K) evaluates whether fine-tuning degrades the model's general code-writing ability, using the RoboCodeGen benchmark (Liang et al., 2023). PaLM 2-S achieves 51% Pass@1. LMPC-Rollouts retains 51% after Iteration 1 and 51% after Iteration 2. LMPC-Skip achieves 49% after both iterations, a minor 2% drop. The paper attributes this preservation to the training data being "in the distribution of the base LLM as well as using code, therefore not biasing the model away from code generations." No evaluations on broader language or reasoning benchmarks (GSM8K, MMLU) are reported; the paper acknowledges "it is conceivable that performance on broader language tasks will degrade" if co-training with original data is not performed (Appendix, Response to Reviewer DSMZ).

• Chat session and chat turn duration analysis: Table VI (Appendix VI-B) reports median chat session durations: 429s for Kuka+Hand, 406s for Bi-arm Kuka, 200s for Aloha, 238s for Mobile Manipulator, and 138s for Robot Dog. Table VII reports median durations by model: 187s for LMPC-Rollouts, 158s for LMPC-Skip. Chat turn durations follow similar patterns. Figure 10 (Appendix VI-B) shows a weak negative correlation between chat turn duration and task success rate — longer turns are associated with lower success, which the paper attributes to difficult tasks taking more time for both the user and the simulation.

• User performance drift: Appendix VI-I addresses the concern that model improvements might actually reflect users becoming more proficient teachers over time. Comparing user performance with the base model between the first and second half of the experiment period, the mean success rate change per user is -0.6% with standard deviation 9.3%, and the two periods show a Pearson correlation of 0.87. NASA-TLX cognitive load measures show no significant differences between users' first and last days in terms of mental demand (p=0.26), effort (p=0.47), performance (p=0.22), or frustration (p=0.54). The paper concludes that "differences in model performance are likely due to changes in model capabilities, and not in users' teaching proficiency."

• Training on only top users vs. all users, without conditioning (top-user conditioning ablation): Already discussed under Main Results above. See Table II and the accompanying analysis.

• Failure mode analysis: Table XV (Appendix VI-J) classifies failure modes across models. The most frequent failure mode for all models is "non-responsive code" (code that does not address the user's feedback): 16.8% for PaLM 2-S, 19.8% for RAG, 11.9% for LMPC-Skip, and 11.3% for LMPC-Rollouts. The fine-tuned models reduce invalid code (from 17.4% to 7.8-9.5%), repeated code, and incomplete code. This suggests fine-tuning primarily helps the model generate code that is more responsive to feedback and syntactically correct, rather than reducing any single failure category dramatically.

• Feedback trait analysis (top users vs. other users): Figure 11 (Appendix VI-C) classifies user feedback along four axes (Detailed, Kind, Code Feedback, Quantitative) using GPT-4. The most notable difference is that top users provide substantially more "kind" feedback. The paper speculates this "might imply that they have more patience for errors in the code generated," which "may lead to more thoughtful responses that end up in successful policies." Other users are more "quantitative" in their feedback, which the paper finds "surprising" and suggests "it's preferred to give softer feedback signals than precise numbers." These findings are preliminary and based on automated classification rather than controlled study.

• Per-embodiment teachability curves: Figure 8 (Appendix VI-B) disaggregates the main teachability result by embodiment. On test tasks, the largest improvements occur on Aloha and Bi-arm Kuka, while LMPC-Rollouts shows "much higher" improvement on Kuka+Hand than other models. On train tasks, LMPC-Skip maintains its advantage across most embodiments. The paper does not provide per-embodiment numeric success rates, only the curves, making it difficult to quantify embodiment-specific effects.

• Task-level overlap analysis: Table IX and Table X (Appendix VI-B) analyze which individual tasks improved or degraded across model variants. LMPC-Rollouts improves 69% of test tasks (highest), while RAG degrades 31% (highest). However, there is little overlap among the sets of improved/degraded tasks across model variants: the highest Intersection over Union (IOU) is 0.28 for training tasks that improved, while test tasks that degraded have zero overlap across variants. The paper could not "identify task-level patterns that reliably predict if the given task would improve or degrade" and emphasizes extremely low per-task sample sizes (median 4 sessions per model-task tuple, median 1 session per model-task-user tuple). No tasks consistently failed across all models.

Critical Assessment

Claim 1: "LMPC-Rollouts improves non-expert teaching success rates on unseen tasks by 26.9%."

This claim is supported by Table I, which reports a test-task success rate of 66.3% for LMPC-Rollouts versus 39.4% for PaLM 2-S. However, several qualifications are necessary. First, the test tasks appear to be easier on average than the train tasks — the base model itself achieves 39.4% on test versus 34.8% on train. The 26.9 percentage point gap therefore conflates the effect of fine-tuning with the effect of task difficulty. A controlled comparison would require stratifying by task difficulty or reporting normalized improvement. Second, the claim is based on blind A/B evaluations aggregated across all users and embodiments, with no reported confidence intervals or statistical significance tests. Given the per-task sample sizes (median 4 sessions per model-variant–task tuple), the aggregate result is reliable, but the task-level decomposition (69% of tasks improved) is noisy.

A stronger piece of evidence for generalization is the cross-embodiment result (Table IV): LMPC-Rollouts improves success on held-out embodiments (Bi-arm Kuka, Kuka+Hand) by 31.5% over the base model, despite these embodiments having entirely different robot APIs and never being seen during training. This genuinely demonstrates that the model has learned something transferable about how to respond to feedback and compose API calls, rather than memorizing task-specific code. However, this result also inherits the difficulty-confounding issue (test embodiments might be easier) and the low-sample issue.

Claim 2: "LMPC-Rollouts reduces the average number of human corrections from 2.4 to 1.9."

Supported by the Num Chat Turns row in Table I (test tasks): 2.4 for PaLM 2-S, 1.9 for LMPC-Rollouts. This is a 21% reduction in average corrections needed for success, which is meaningful. However, this metric is computed only over successful sessions. If LMPC-Rollouts is succeeding on harder tasks that the base model fails entirely, the average might increase rather than decrease. The paper does not report the distribution of chat turns for failed sessions or the interaction between success rate and average turns. The teachability curves in Figure 5 partially address this by showing success rate as a function of turn count, confirming that LMPC-Rollouts achieves a given success rate at strictly fewer turns than the base model across the entire range.

Claim 3: "LMPC also produces strong meta-learners, improving the success rate of in-context learning new tasks on unseen robot embodiments and APIs by 31.5%."

The cross-embodiment result in Table IV supports this claim quantitatively. The meta-learning interpretation — that the model has "learned to learn faster" — is bolstered by the 2+ turn success rate metric (Table I), which measures adaptation specifically when the first response is wrong. LMPC-Rollouts achieves 31.5% on this metric for test tasks, substantially above the base model's 21.9% and LMPC-Skip's 17.8%. This directly demonstrates improved multi-turn adaptation, not just better zero-shot performance.

However, the claim's strength is limited by the single model family (PaLM 2), the fixed set of robot embodiments (5), and the absence of a "pretraining-matched" baseline — e.g., comparing fine-tuning to simply using a larger base model with more in-context examples. The paper evaluates PaLM 2-S against PaLM 2-S+LMPC, but does not compare to, say, PaLM 2-M with in-context learning alone. This leaves open the possibility that the meta-learning gains could be achieved more simply by scaling the base model rather than fine-tuning it.

Claim 4: "Top-user conditioned LMPC drives performance improvements for all users on all tasks, including unseen ones by 10.5%."

The 10.5% figure comes from Table II: removing top-user conditioning from LMPC-Rollouts reduces test-task success by 10.5% (the ablation is only performed on Robot Dog and Mobile Manipulator, not all embodiments). The robustness check in Table XI confirms that both top users (+26.3%) and other users (+18.9%) benefit. This is a genuinely non-obvious finding: conditioning on a small fraction of users (10 out of 35, covering 11 out of 50 train tasks) transfers benefits to the broader population.

However, the mechanism is not fully validated. The paper hypothesizes that top users provide "less noisy" observations in the POMDP, enabling more efficient MPC search, but this is an interpretation rather than a demonstrated causal mechanism. The feedback trait analysis (Figure 11) shows top users are more "kind," but whether kindness is the operative factor or merely correlated with other unmeasured traits (patience, clarity, persistence) is unclear. The paper acknowledges that "these insights are preliminary and further investigation is needed to understand what makes top users successful" (Appendix VI-C). Additionally, the ablation is run on only 2 of 5 embodiments due to time constraints, so the universality of the 10.5% figure across all platforms is assumed rather than demonstrated.

Claim 5: "LMPC-Rollouts is more amenable to improvements from user feedback, while LMPC-Skip achieves higher 1-turn success rate."

Strongly supported by the decomposition in Table I. LMPC-Skip's 1-turn success (41.7% test) versus LMPC-Rollouts (34.8%) and LMPC-Rollouts' 2+ turn success (31.5%) versus LMPC-Skip (17.8%) is a clean double dissociation. This is one of the paper's most robust findings, and it validates the core design distinction: modeling interaction trajectories (LMPC-Rollouts) builds feedback-responsiveness; modeling only final outcomes (LMPC-Skip) builds zero-shot instruction-following.

Genuine Weaknesses in the Experimental Design:

1. Difficulty confounding: The train/test split was chosen before task difficulties were known, resulting in test tasks being easier on average (base model: 39.4% test vs. 34.8% train). The paper acknowledges this (Table IV caption, Appendix VI-B) but does not correct for it in any analysis. The 26.9% test-task improvement cannot be directly compared to training-task improvements without difficulty normalization.

2. No statistical significance reporting for main results: The only p-values in the paper are for the user performance drift analysis (Appendix VI-I). The main teachability comparisons (Table I, Figure 5) report point estimates without confidence intervals, making it impossible to assess whether differences between LMPC-Rollouts and LMPC-Skip (e.g., 66.3% vs. 59.4% on test) are statistically significant given the session-level sample sizes.

3. Single model family: All results use PaLM 2 models. The paper's claim that findings would generalize to other LLMs is untested. The open-source Colab notebook with GPT-3.5 (mentioned in the rebuttal) is described as a "minimal example" for exploration, not a systematic replication.

4. RAG baseline strength: The RAG implementation retrieves 5 examples using embedding similarity + farthest-point sampling, which is a reasonable design. However, the paper does not explore whether better retrieval (e.g., learning a task-specific retriever, increasing the number of retrieved examples) could close the gap with fine-tuning. The finding that RAG reduces Successful Task Rate below the base model (75.0% vs. 81.5%) suggests the retrieval mechanism may be actively harmful on novel tasks — a stronger retrieval approach might avoid this.

5. LMPC-Rollouts inference cost is not systematically traded off against performance: The paper uses 8 rollouts per chat turn as a fixed parameter. There is no ablation varying the number of rollouts (e.g., 1, 2, 4, 8, 16) to characterize the performance-vs-latency tradeoff curve. Since inference time is a practical concern (7.4s without quantization is "much slower," degrading user experience per Appendix VI-H), understanding the marginal benefit of additional rollouts is important.

6. Top-user conditioning ablation scope: The top-user conditioning results (Table II) are limited to Robot Dog and Mobile Manipulator. Whether the 10.5% benefit generalizes to the dexterous manipulation embodiments (Aloha, Kuka variants) is unknown. The justification — "due to time constraints" — is understandable for a large-scale human study but limits the claim's strength.

7. No comparison to instruction-tuning or general chat fine-tuning: The paper does not include baselines where PaLM 2-S is fine-tuned on general instruction-following data (e.g., FLAN) or on non-interactive robot code data. This makes it difficult to determine how much of the improvement comes from domain-specific interaction modeling versus general code-writing capability improvements from any fine-tuning.

8. Missing combination experiment: The paper suggests using "LMPC-Skip for the initial instruction, then LMPC-Rollouts for subsequent feedback" as a practical deployment strategy, but never evaluates this combined system. The reported numbers for each model are from separate blind A/B evaluations where users interact with only one model variant throughout a session. The performance of a hybrid system (switching models mid-session) is unknown.

What Would Have Strengthened the Paper:

• A controlled difficulty stratification: report results within difficulty terciles or quintiles (analogous to the difficulty-bin analysis in the PaLM 2 scaling paper) to separate the effects of fine-tuning from task difficulty.
• Confidence intervals or bootstrap standard errors on the main success rate comparisons.
• An ablation on the number of LMPC-Rollouts inference samples (1, 2, 4, 8, 16) with corresponding success rates and latencies.
• Evaluation of the LMPC-Skip → LMPC-Rollouts hybrid strategy.
• A baseline using a larger base model (e.g., PaLM 2-M) with in-context learning only, to test whether scaling pretraining can substitute for fine-tuning — directly analogous to the FLOPs-matched comparison in the inference-scaling paper.
• Full-embodiment top-user conditioning ablation (all 5 embodiments) rather than only 2.

Despite these limitations, the experimental evidence for the paper's central claims — that LMPC-Rollouts improves teachability on unseen tasks, that top-user conditioning transfers benefits broadly, and that modeling interaction dynamics (vs. final code) is what enables multi-turn feedback-responsiveness — is consistent and substantial. The blind A/B design with real non-expert users, the cross-embodiment generalization result, and the diagnostic LMPC-Skip comparison are particularly strong elements that give confidence in the findings, even if the precise magnitude of improvement varies with task difficulty and the statistical reliability of specific pairwise comparisons remains unquantified.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Excluded from Headline Efficiency Numbers

The assumption. The compute-optimal framework depends on a difficulty estimator that, in its current form, requires generating 2048 samples per question to assess the base model's pass@1 rate (or the PRM's average final-answer score) and binning questions into five quintiles. This cost is not included in any budget calculation. The paper acknowledges this in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

The consequence. The reported 4× efficiency gains over best-of-N (Figure 4, Figure 8) are computed after difficulty is already known, without amortizing the cost of learning it. In a realistic deployment, the total cost would be difficulty estimation plus strategy execution, and the former could dominate the latter — generating 2048 samples costs 8× more than the largest budget the compute-optimal policy is evaluated on (256 generations). If difficulty estimation were included, the crossover point where compute-optimal scaling breaks even with best-of-N would shift to much higher budgets, or might never be reached for problems where the base model's pass@1 is already low. The paper frames this as an exploration-exploitation tradeoff but does not quantify it. Until a cheaper difficulty estimator exists, the 4× efficiency figure is an upper bound on achievable efficiency contingent on solving the difficulty estimation problem, not a realized deployment gain.

What evidence exists. The paper explicitly flags this issue (Section 3.2) and shows that predicted difficulty bins (using PRM scores) track oracle bins closely in Figures 4 and 8, demonstrating that ground-truth labels are not needed. However, the predicted method still requires 2048 samples per question — it replaces the correctness oracle with a PRM oracle but does not reduce the sample cost.

Mitigation status. The paper acknowledges this limitation, frames it as "a key avenue for future work," and suggests training models to predict difficulty directly from question text or developing adaptive difficulty estimation that interleaves assessment with problem-solving. None of these alternatives are implemented or evaluated. The cost remains unaddressed in the current system.

All Results Are Restricted to a Single Benchmark (MATH) and Single Model Family (PaLM 2-S*)

The assumption. The paper's entire empirical contribution — the difficulty-dependent scaling curves, the compute-optimal policies, the FLOPs-matched comparison — is derived from experiments on the MATH benchmark (500 test questions) using PaLM 2-S* as the base model. The authors state they "believe this model is representative of the capabilities of many contemporary LLMs" (Section 4), but this belief is not tested.

The consequence. Several aspects of the findings are plausibly model-specific or benchmark-specific. The PRM's over-optimization behavior (beam search degrading easy-problem performance at high budgets, Figure 3 right) depends on the PRM's calibration properties, which are a function of the base model's output distribution and the Monte Carlo rollout training procedure. A model with different error patterns or better-calibrated uncertainty might exhibit different over-optimization thresholds, changing the entire difficulty-dependent allocation strategy. The revision model's effectiveness depends on the base model's in-context learning ability and capacity to learn from incorrect examples — properties that vary substantially across model families, training procedures, and scales. The MATH benchmark consists of competition-level math problems requiring symbolic reasoning and exact-answer grading; it is unclear whether the difficulty-dependent patterns (beam search hurting easy problems, sequential revisions dominating easy problems) generalize to other reasoning domains (code generation, logical reasoning, scientific QA) or to open-ended tasks without clean correctness signals.

What evidence exists. The paper provides no out-of-domain evaluation. All figures and tables report results exclusively on MATH with PaLM 2 models. The rebuttal (not in the main paper) mentions releasing a Colab notebook with a "minimal and clean example" using GPT-3.5 for a 2D navigation task, but this is a toy demonstration, not a systematic replication.

Mitigation status. Not addressed. The paper does not claim broader applicability beyond MATH and PaLM 2-S*, but the framing — particularly phrases like "compute-optimal test-time scaling" and the parallel drawn to Chinchilla scaling laws — implies generality that the experiments do not support.

Hard Problems Are Fundamentally Outside the Reach of Test-Time Compute

The assumption. The compute-optimal framework assumes that for every question, there exists some test-time strategy that meaningfully improves the probability of success. This assumption fails when the base model's pass@1 is near zero — if the model cannot produce a correct solution at any non-trivial rate in its proposal distribution, no amount of search or revision can create one.

The consequence. On the hardest questions (difficulty bin 5, approximately the bottom 20% of the MATH test set), all methods studied — best-of-N, beam search, lookahead search, sequential revisions, and their compute-optimal combinations — produce near-zero accuracy regardless of compute budget. Figure 3 (right) shows bin 5 accuracy hovering at 1-3% for all search methods across all budgets. Figure 7 (right) shows bin 5 at roughly 2-3% irrespective of the sequential-to-parallel ratio. In the FLOPs-matched comparison (Figure 9), the bin 5 scaling line is essentially flat near 0-5%, well below the ~14× larger model's performance. Test-time compute amplifies existing capability but does not create it from nothing. For problem classes where the base model genuinely lacks the requisite knowledge or reasoning ability — which may include many real-world, out-of-distribution, or novel compositional reasoning tasks — the approach offers no path forward, and scaling pretraining remains the only viable option.

What evidence exists. The flat bin-5 curves in Figures 3, 7, and 9 are among the most consistent findings in the paper, replicated across search methods, revision strategies, and selection mechanisms.

Mitigation status. The paper is transparent about this boundary condition, noting in the Section 7 takeaway that "on the hardest problems, test-time compute provides essentially zero benefit regardless of budget." This is presented as a finding, not a solvable limitation — it reflects a fundamental capability ceiling of the base model.

The FLOPs-Matched Comparison Uses a Weakened Pretraining Baseline

The assumption. The paper's headline finding that test-time compute can substitute for pretraining — specifically, that PaLM 2-S* with compute-optimal test-time scaling can outperform a ~14× larger model — depends on two design choices that systematically favor the test-time compute side of the comparison. First, the larger model is created by scaling only parameters, holding training data fixed, following the LLaMA paradigm (Touvron et al., 2023) rather than compute-optimal pretraining (Hoffmann et al., 2022) where both data and parameters are scaled together. The paper acknowledges this in Section 7:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

Second, the larger model uses only greedy decoding with no test-time compute augmentation of its own — no majority voting, no best-of-N, no search, no revisions.

The consequence. A Chinchilla-optimal model trained with 14× more total FLOPs (scaling both parameters and data) would likely outperform a parameter-only-scaled model, making the pretraining baseline weaker than it needs to be. Similarly, giving the larger model even a modest test-time compute budget (e.g., best-of-8 with majority voting) would close a significant portion of the gap. The reported advantages of test-time compute over pretraining — e.g., +27.8% relative improvement on easy questions at R ≪ 1 (Figure 1, top-right bar chart) — may shrink or reverse against a properly tuned pretraining baseline. The comparison as constructed demonstrates that test-time compute can be a better investment than naïvely scaling model size while holding data fixed and using no inference-time search, but it does not establish that test-time compute is preferable to optimally allocated pretraining compute.

What evidence exists. The paper explicitly caveats the parameter-only scaling choice in Section 7. The use of greedy decoding for the larger model is not discussed as a limitation — it is simply the baseline chosen. No experiments test the larger model with any form of test-time compute augmentation.

Mitigation status. The paper acknowledges the departure from compute-optimal pretraining and leaves the comparison against a Chinchilla-optimal baseline to future work. The greedy-decoding choice for the larger model is not acknowledged as a limitation.

Verifier Over-Optimization Is Documented but Not Solved — It Remains the Primary Bottleneck

The assumption. The compute-optimal allocation policy assumes that for each difficulty bin and budget level, there exists a strategy that improves performance without triggering verifier over-optimization (reward hacking). The policy works by routing easy problems away from aggressive search (using best-of-N instead of beam search) and reserving beam search for medium-difficulty problems where the verifier provides genuine guidance.

The consequence. This routing mitigates over-optimization but does not eliminate it, and the verifier quality ceiling remains the primary bottleneck preventing unbounded improvements from additional test-time compute. On easy problems, even best-of-N can over-optimize at high enough budgets (the PRM will eventually find solutions that score well but are incorrect). On medium problems where beam search is deployed, the beam search curves in Figure 3 flatten and in some cases decline well before the budget is exhausted — the verifier's reliability limits how far search can scale before it begins exploiting spurious correlations. The paper's qualitative examples (Appendix M, Figures 29 and following) show search producing low-information repetitive steps and overly short solutions that score highly under the PRM but are incorrect. Lookahead search — the most powerful optimizer — paradoxically performs worst overall (Figure 3, left), confirming that optimizing harder against an imperfect verifier is counterproductive.

This means the compute-optimal approach is fundamentally bounded by verifier quality. If the PRM were made more robust (through better training data, adversarial training, ensembling, or architectural improvements), the difficulty thresholds would shift, and the optimal policy would change. The current results are specific to the verifier quality achievable with the Monte Carlo rollout procedure described in Appendix D.

What evidence exists. The over-optimization pattern is clearly visible in Figure 3 (right), where beam search on easy questions (bins 1-2) shows decreasing accuracy with increasing budget. The flat or declining beam search curves at high budgets (Figure 3, left) further support this. The lookahead search underperformance is direct evidence that more aggressive optimization of an imperfect verifier signal is harmful. Qualitative examples in Appendix M provide concrete instances of degenerate outputs that score well.

Mitigation status. The paper identifies verifier over-optimization as a key finding and bottleneck (Section 5.3, Section 8), but proposes no method to address it beyond the routing strategy, which only avoids the worst cases rather than improving verifier robustness. Future work on "robust verifiers resistant to over-optimization" is suggested in Section 8.

The Revision Model Has a 38% Correct-to-Incorrect Reversion Rate, and Revision Training Is Fragile

The assumption. The revision model is trained exclusively on sequences where all in-context answers are incorrect followed by a correct target answer. The implicit assumption is that at inference time, the model will encounter only incorrect answers in its context (produced during earlier revision steps) and will revise them toward correctness.

The consequence. At test time, the model sometimes produces a correct answer early in the revision chain. Since it was never trained on sequences containing correct answers, it has no signal for what to do when the current answer is already correct — and approximately 38% of correct answers get converted back to incorrect ones in the subsequent revision step (Section 6.1). The paper mitigates this by using majority voting or verifier-based selection across the entire chain of revisions, picking the best answer from any point rather than always taking the last revision. However, this is a patch, not a fix — it means the chain must be generated past the point of first success, consuming additional compute on revisions that may undo good answers, and relying on post-hoc selection to recover the best answer. A principled solution — such as training the model to recognize when no revision is needed and output a "stop" token — is not implemented.

The ReST^EM experiment (Appendix K, Figure 16) reveals a deeper fragility: attempting to further optimize the revision model using on-policy RL-style training caused performance to degrade substantially with sequential revisions. At 256 generations, fully sequential performance with the ReST^EM model dropped to approximately 33.5%, compared to roughly 38.5% at the optimal ratio. This suggests the revision approach is sensitive to training methodology and data generation procedure in ways that are not fully understood, and the positive results depend on specific choices (offline data construction with edit-distance-based incorrect-correct pairing) that may not transfer to other settings or sustain iterative improvement.

What evidence exists. The 38% reversion rate is reported in Section 6.1. The mitigation via within-chain selection is described in the same section and evaluated implicitly through the sequential revision results (Figure 6, Figure 7), which include this selection step. The ReST^EM failure is in Figure 16 (Appendix K) and discussed in Section 6.1.

Mitigation status. Partially mitigated by within-chain selection (majority voting or verifier-based selection across the chain), but the underlying model behavior is not corrected. The paper does not explore training the model to detect correct answers or adding success-conditioned trajectories to the training data. The ReST^EM degradation is presented as a negative result with the hypothesis that "on-policy data collection exacerbates spurious correlations in revision data" (Section 6.1) — the mechanism is not fully diagnosed.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a methodological shift in how we think about fine-tuning LLMs for interactive, embodied tasks: it reframes multi-turn human-robot dialogue not as a sequence of independent instruction-following problems, but as trajectories in a dynamics model that can be learned and then searched over at inference time. This is not a paradigm shift in the sense of introducing entirely new architectures or training objectives — the underlying mechanism is standard autoregressive SFT, and the inference-time search is a decoding strategy. Rather, it is a reconceptualization that unlocks a different way of using exactly the same training pipeline the field already employs. The contribution is in recognizing that when training data consists of interaction trajectories, the resulting model is not merely a better policy but a world model of the human-robot teaching process, and that world model supports model-based planning to accelerate task convergence.

This shifts the conversation around LLM alignment and fine-tuning in two concrete ways. First, it demonstrates that the representation of the training target matters independently of the data. LMPC-Rollouts and LMPC-Skip are trained on the same underlying successful chat sessions, but one learns to predict full trajectories (human feedback + robot code + success outcome) while the other learns to predict only the final code. The result — LMPC-Skip excels at zero-shot instruction-following (41.7% 1-turn success on test tasks) while LMPC-Rollouts excels at multi-turn adaptation (31.5% 2+ turn success vs. 17.8%) — shows that these are distinct capabilities acquired from the same data depending on how the target sequence is structured. This implies that for any interactive system where users provide sequential feedback, training on full trajectories rather than final outcomes may be essential for learning to use feedback effectively — a design principle that extends well beyond robotics to any multi-turn dialogue system, tutoring interface, or iterative content-creation tool.

Second, the paper provides the first clean demonstration that top-user conditioning — a lightweight prompt-level intervention — can transfer interaction quality across both tasks and users. The finding that conditioning on 10 users who covered only 14% of training tasks (11/50) improves outcomes for all 35 users, including on unseen tasks, by 10.5% (Table II) is not predicted by standard personalization or preference-learning frameworks. In personalization, conditioning on user identity typically helps that user but may harm others. Here, it helps everyone. The paper's interpretation — that top-user conditioning biases the model's imagined futures toward lower-noise observation distributions, producing more robust first actions regardless of the actual user — positions user identity as a proxy for data quality that can be exploited at inference time without discarding the diversity of the full training distribution. This is a genuinely novel mechanism for improving interactive systems that collect data from heterogeneous populations, and it suggests a general design pattern: autonomously identify high-quality subsets of your interaction data, label them, condition on that label during inference, and retain the full dataset for diversity during training. The ablation showing that training only on top users degrades performance by 23.8% (LMPC-Rollouts, train tasks, Table II) while training on all users without conditioning degrades it by 8.4% confirms that both diversity and quality-gating are necessary — neither alone suffices.

The paper also resolves a latent tension in the LLM-for-robotics literature that parallels the contradiction the inference-scaling paper addressed for self-correction. Prior work showed both that LLMs can respond to language feedback to modify robot behavior (Sharma et al., 2022; Yu et al., 2023; Zha et al., 2024) and that in-context learning is exemplar-based and limited in generalization (Chan et al., 2022). The implicit question was whether these systems could improve with experience — whether data collected from one set of teaching interactions could make the system more teachable on the next set of interactions. This paper provides a clear empirical answer: yes, but only if you train on the process of interaction (LMPC-Rollouts), not just the outcomes (LMPC-Skip), and only if you condition on high-quality interaction patterns (top-user conditioning). The RAG baseline — which improves overall success rates (51.9% vs. 39.4% base on test tasks) but reduces the fraction of tasks that can be taught at all (75.0% vs. 81.5% base on Successful Tasks Rate) — sharpens this picture: retrieval helps on tasks similar to what was seen before but actively harms on novel tasks, likely because retrieved examples bias the model toward previously successful strategies that don't transfer. Fine-tuning, by absorbing interaction patterns into weights rather than retrieving them by similarity, avoids this overfitting and produces broader generalization.

Finally, the paper establishes that the bottleneck for interactive teachability is not model scale but training data structure and inference strategy. The base PaLM 2-S already has the raw capability to write correct robot code — evidenced by non-zero success rates — but its in-context learning mechanism limits how efficiently that capability can be accessed through multi-turn feedback. Fine-tuning on only 299 successful sessions (~3M tokens after augmentation) — a tiny fraction of the model's pretraining data — produces a 26.9% absolute improvement on unseen tasks and 31.5% improvement on unseen embodiments. This suggests that for domains where the base model already possesses relevant knowledge (robot APIs, physics intuition from code pretraining, instruction-following ability), the limiting factor for interactive performance is not knowledge acquisition but knowledge access — the model knows what to do but doesn't know how to converge to it efficiently through dialogue. Fine-tuning on interaction trajectories teaches the access pattern, and this transfers across embodiments because the pattern — iterative refinement guided by user feedback — is embodiment-invariant. This reframes the problem of building teachable robots from one of scaling model size to one of scaling interaction data quality and diversity, with careful attention to how that data is structured for training.

Follow-Up Research This Work Enables

Directly evaluating the LMPC-Skip + LMPC-Rollouts hybrid strategy. The paper explicitly recommends using "LMPC-Skip for responding to the initial user instruction, then LMPC-Rollouts for responding to subsequent user feedback" (Section IV-D) but never evaluates this combined system — all evaluations are of each model variant independently throughout entire sessions. A natural follow-up would run a blind A/B evaluation comparing this hybrid against pure LMPC-Rollouts, pure LMPC-Skip, and the base model on the same set of tasks and users. The prediction is that the hybrid would achieve the 1-turn success rate of LMPC-Skip (~42% on test tasks) and the multi-turn adaptation of LMPC-Rollouts (31.5% 2+ turn), combining their complementary strengths. The key metric would be whether the hybrid's overall success rate exceeds the better of the two pure models (66.3% for LMPC-Rollouts), and particularly whether the multi-turn success rate of the hybrid matches pure LMPC-Rollouts despite the first turn being generated by LMPC-Skip — answering whether LMPC-Rollouts' feedback-responsiveness depends on the model having generated the initial (incorrect) code itself, or whether it can effectively correct another model's output.

Systematic scaling of LMPC-Rollouts inference budget vs. performance. The paper fixes the number of sampled rollouts at 8 and treats this as a pragmatic constant. A critical open question is: how does teachability scale with inference compute? Running an experiment that varies the number of rollouts (1, 2, 4, 8, 16, 32) and measures both success rate and latency for LMPC-Rollouts on the same test tasks would produce a scaling curve directly analogous to the inference-scaling paper's analysis of best-of-N vs. beam search. The hypothesis is that success rate increases with more rollouts but with diminishing returns, and that the optimal number depends on task difficulty (harder or more ambiguous tasks might benefit more from additional search). This would also reveal whether the 8-rollout setting the paper uses is near-optimal or substantially suboptimal, and would provide practical guidance for deployment under latency constraints. A comparison to simply using a larger base model with fewer rollouts would test whether inference compute and model scale are substitutable in this setting.

Replication on open-weight models with public interaction datasets. All results use PaLM 2-S, which is not publicly available. To establish whether LMPC is a general method or specific to PaLM 2's architecture and pretraining, a replication using an open-weight model (e.g., LLaMA-3-8B, CodeLlama, DeepSeek-Coder) on a public multi-turn interaction dataset would be valuable. The experiment would: (1) collect or repurpose a dataset of multi-turn human-LLM code-writing interactions (not necessarily robotics — code generation from iterative feedback on platforms like HumanEval with simulated feedback could work), (2) fine-tune using the LMPC-Rollouts and LMPC-Skip recipes, (3) evaluate on held-out tasks with real or simulated users. The key question is whether the qualitative pattern — LMPC-Skip better at one-shot, LMPC-Rollouts better at multi-turn — replicates across model families, and whether top-user conditioning (identifying high-quality annotators from training data) provides the same universal benefit. A negative result (e.g., LLaMA-based LMPC-Rollouts doesn't outperform LLaMA-based LMPC-Skip on multi-turn tasks) would suggest the effect depends on PaLM 2-specific properties, limiting the method's generality. A positive result would establish LMPC as a model-agnostic fine-tuning strategy for interactive systems.

Understanding and mitigating the correct-to-incorrect reversion problem in revision models. The paper reports that ~38% of correct answers in a revision chain get revised to incorrect ones (Section 6.1), and uses post-hoc selection (majority voting or verifier) to recover the best answer. This is a significant practical limitation that likely affects any system using autoregressive revision. A targeted study could investigate: (1) whether including "success" tokens or no-op revision examples (where the model is trained to output "the current code is already correct" instead of modifying correct code) in the training data eliminates the reversion behavior; (2) whether training a separate "revision gate" classifier that predicts whether a given code output needs revision, and stopping the chain when the gate says "correct," outperforms post-hoc selection; (3) whether the reversion rate is correlated with task difficulty, model confidence, or code length. The evaluation would measure both the reversion rate (fraction of correct answers that become incorrect in the next revision step) and the end-to-end success rate with different mitigation strategies, using the same MATH benchmark and PaLM 2 models to enable direct comparison with the paper's results. This would transform the revision model from a promising-but-fragile mechanism into a more reliable component.

Active task exploration and data diversification for iterative fine-tuning. The paper's Iteration 2 results (Table V) show no improvement from further fine-tuning on data collected with the Iteration 1 models, hypothesizing that "the data distribution or data amount used to train the second iteration of models do not differ significantly from that of the first iteration." This suggests that the fine-tuning saturates when the data distribution remains static. A natural extension is to actively expand the training distribution: after Iteration 1, use the improved models to generate interaction data on tasks that the base model failed on (or on entirely new tasks designed to stress-test the model's feedback-responsiveness), then train Iteration 2 on this expanded dataset. The experiment would compare (a) Iteration 2 trained on randomly sampled additional interactions from the same task distribution, (b) Iteration 2 trained on interactions from previously-failed tasks, (c) Iteration 2 trained on interactions from synthetic tasks generated by an LLM to be compositionally novel (e.g., combining two train-task skills into one), and (d) Iteration 2 trained on interactions with simulated feedback designed to be intentionally ambiguous or noisy. The prediction is that (c) and possibly (b) would show improvement over (a) and over Iteration 1, while (d) might improve robustness to low-quality feedback. This would test whether the saturation observed in Table V is fundamental (the model has extracted all available signal from the interaction format) or distributional (the additional data didn't cover new ground).

Cross-domain evaluation of LMPC for non-robotic interactive tasks. The paper's method is presented as improving robot teachability, but the core mechanism — fine-tuning an LLM to predict full interaction trajectories and using MPC-style search to accelerate convergence to success — is domain-agnostic. A cross-domain evaluation would apply LMPC to a substantially different interactive setting, such as: (1) code review and debugging: a human provides a buggy code snippet and iteratively gives feedback as an LLM proposes fixes, with "success" being a passing test suite; (2) iterative document editing: a human describes desired changes to a technical document and the LLM produces revised drafts, with success being the human's final approval; (3) tutoring dialogues: a student asks questions and the LLM provides explanations, with success being the student indicating understanding. For each domain, the experiment would: collect initial interaction data from the base LLM, fine-tune with LMPC-Rollouts and LMPC-Skip, evaluate success rates and number of turns on held-out tasks, and test top-user conditioning by identifying efficient annotators from training data. A positive result (LMPC-Rollouts ≥ LMPC-Skip on multi-turn success across domains) would establish the framework as broadly applicable to interactive LLM systems. A negative result (gains limited to robotics) would suggest that the robot code domain has specific properties — the structured API, the execution feedback loop via simulation, the physics-grounded constraints — that make trajectory prediction particularly effective, narrowing the scope of the method.

Practical Applications and Downstream Use Cases

Deployment in consumer robot teaching interfaces. The most direct application is integrating LMPC into the kind of system described in the paper itself — a chat-based interface where non-expert users teach robots new tasks through natural language. The practical value proposition is quantified by the paper's numbers: for a robot deployed in a home or office setting, switching from the base PaLM 2-S to LMPC-Rollouts reduces the average number of corrections needed per successful task from 2.4 to 1.9 (a 21% reduction in user effort per task) while increasing the overall fraction of tasks that can be taught successfully by 26.9% (from 39.4% to 66.3% on test tasks). The hybrid strategy the paper suggests (LMPC-Skip for the initial instruction, LMPC-Rollouts for corrections) would additionally boost one-shot instruction-following from 17.5% to ~42%. This matters for user experience: each chat turn takes ~56 seconds (median, Section III-B), so saving 0.5 turns on average saves ~28 seconds per task — significant when a user is teaching many tasks. The cross-embodiment generalization result (31.5% improvement on unseen robot platforms, Table IV) means that a robot manufacturer could train LMPC on data from a few robot models and deploy the same fine-tuned LLM on new models with different APIs, reducing the need for per-platform data collection.

Data generation and filtering for robot skill libraries. Organizations building libraries of robot skills (reusable code snippets for common manipulation or locomotion behaviors) could use LMPC to improve the efficiency and quality of human-generated skill data. The current pipeline requires human operators to iteratively refine robot code through the chat interface; LMPC-Rollouts reduces the human time per skill by reducing correction rounds, and top-user conditioning improves the quality of the generated code by biasing it toward the interaction patterns of the most proficient operators. The Good Rating Rate improvement — from 18.1% (base) to 26.5% (LMPC-Rollouts) on test tasks — means that a higher fraction of individual code outputs are correct responses to the most recent feedback, reducing the need for operators to re-explain themselves. The autonomous top-user identification mechanism (Section III-C, Appendix VI-C) provides a concrete recipe for identifying which operators produce the highest-quality teaching data, which could be used to prioritize certain operators for data collection or to weight their contributions more heavily in downstream training.

Fine-tuning recipe for interactive AI assistants beyond robotics. The LMPC framework — SFT on full interaction trajectories with MPC-style decoding — can be applied to any setting where users iteratively refine AI outputs through language feedback. For a customer-support chatbot that drafts responses based on agent corrections, LMPC-Rollouts could reduce the number of revision rounds before the response is approved. For an AI-assisted design tool where users describe desired visual changes and the AI iteratively modifies a design, LMPC-Rollouts could accelerate convergence to the user's intent. The key practical requirements are: (1) a base model with non-zero success at the task, (2) a mechanism for collecting multi-turn interaction data with binary success labels, and (3) the ability to execute the model's outputs and present results to the user for feedback — the execution feedback loop (via simulation in the paper's case) is what makes the interaction trajectories informative. The data requirements are modest: the paper achieves substantial gains with only 299 successful sessions (~3M tokens after augmentation), suggesting that organizations with even small volumes of high-quality interaction data — which many customer-facing AI products accumulate naturally — could apply LMPC without massive data collection efforts.