Never Stop Learning: The Effectiveness of Fine-Tuning in Robotic Reinforcement Learning

ArXiv: 2004.10190

🎯 Pitch

A vision-based grasping policy pre-trained on 580,000 real grasps can adapt to drastic environmental changes—such as a 10 cm gripper offset, harsh lighting, or transparent bottles—using less than 0.2% of the original training data, and surprisingly, continually fine-tuning this policy across a sequence of challenges incurs almost no cumulative performance penalty.

---

1. Executive Summary

This paper studies how to efficiently adapt vision-based robotic manipulation policies to new variations—including changes in background, lighting, object appearance, and robot morphology—by fine-tuning via off-policy reinforcement learning, using less than 0.2% of the data necessary to learn the task from scratch. The experiments evaluate a grasping policy pre-trained with QT-Opt on 580,000 real grasp attempts across five challenge tasks (checkerboard backing, harsh lighting, extended gripper, offset gripper, transparent bottles), demonstrating that simple offline fine-tuning recovers substantial performance—from 50% to 90% on checkerboard backing, from 32% to 63% on harsh lighting, and from 43% to 98% on a 10 cm gripper offset—with as few as 100–800 exploration grasps. The paper further establishes that this fine-tuning approach supports continual learning across a sequence of adaptations with minimal performance penalty compared to single-step fine-tuning—only 4–7% degradation across three tasks while actually improving by 8% on transparent bottles—while showing that pre-training via RL is essential, as training from scratch or adapting from supervised ImageNet features both fail with such small amounts of data.

2. Context and Motivation

The Core Problem: Robots Deployed as Static Policies Don't Adapt

The fundamental gap this paper addresses is the disconnect between the vision of robot learning—systems that learn from their mistakes and continuously adapt—and the reality of how reinforcement learning for robotics is actually practiced and evaluated. The paper articulates this tension directly in Section I:

"Despite this potential, most of the robot learning systems today are deployed as a fixed policy and they are not being adapted after their deployment."

This matters because the standard paradigm in robotic reinforcement learning treats each new task as an independent learning problem: initialize a policy from scratch, collect entirely new data in a stationary environment, train until convergence, evaluate the frozen policy, and stop. There is no mechanism for the robot to say "the lighting changed, let me adjust" or "these transparent bottles are new, I need to learn how to grasp them." The robot's learned behavior is frozen at deployment time, rendering it incapable of handling the kind of persistent environmental change that is inevitable in any real-world setting.

This gap is significant for both practical and conceptual reasons:

• Physical deployment reality: A robot operating in a home, warehouse, hospital, or factory will encounter changing lighting throughout the day, new objects introduced into its workspace, hardware wear and modifications (e.g., replacing or extending a gripper), and varying backgrounds. A static policy that was only trained under one set of conditions will inevitably fail when these conditions shift—as the paper empirically demonstrates in Table I, where base policy performance drops by 36–55% under environmental changes alone.
• Economic inefficiency of re-training from scratch: If each new condition requires collecting hundreds of thousands of grasp attempts from scratch (as the base policy required—580,000 offline grasps plus 28,000 online grasps, per Section III-A), the cost of maintaining a robot fleet in dynamic environments becomes prohibitive. The paper's finding that adaptation can succeed with less than 0.2% of the original data (800 vs. ~608,000 grasps) represents a potential order-of-magnitude change in the economics of robot deployment.
• The continual learning promise unfulfilled: The sub-field of lifelong learning (Thrun, 1998, cited as [65]) has long envisioned agents that accumulate knowledge over their lifetime, but this vision remains largely unrealized in physical robotics. As the paper notes, most lifelong learning research focuses on supervised classification tasks, not on sensorimotor skills where the agent must both perceive and act in a changing world.

The Static Paradigm Traps RL in Its Worst Sample-Efficiency Regime

Section I makes a critical economic argument: the standard "train from scratch for each task" approach forces reinforcement learning to operate in what the authors call its "worst-case regime for sample efficiency." When you start from a randomly initialized policy with no prior knowledge, every new skill must be acquired through exploration that begins essentially at random—the cost to acquire a new task is "dominated by sample efficiency of the learning algorithm and the complexity of the task, as reflected in the cost of acquiring diverse task data starting from naïve (e.g. random) exploration."

This framing is crucial. The paper draws an explicit parallel to how other fields of machine learning escaped this trap:

"Most machine learning models successfully deployed in the real world, such as those used for computer vision and natural language processing (NLP) do not live in this regime. For instance, the predominant method of acquiring a new computer vision task is to start learning the new task with a pre-trained model for a related task, acquired from a pre-collected data set, and fine-tune that model to achieve the new task."

This is the paper's central motivation: robotics should adopt the fine-tuning paradigm that transformed computer vision and NLP. In those fields, the sample complexity of learning a new task is dominated by task novelty—how different the target is from the pre-training task—rather than by task complexity—how hard the task would be to learn from nothing. The paper argues that this shift in the sample-efficiency regime is what enabled the explosion of practical applications in CV and NLP, and that robotics has not yet made this transition.

Where Prior Approaches Fall Short

The paper positions itself against a landscape of related work, identifying specific limitations in each:

1. Transfer from ImageNet Features Is Insufficient

A number of prior works used pre-trained ImageNet convolutional features as a starting point for robotic policy learning (Levine et al., 2016 [36]; Finn et al., 2016 [13]; Gupta et al., 2018 [17]; Pinto and Gupta, 2016 [49]). The logic is appealing: ImageNet provides powerful visual representations, and since robotic tasks involve visual perception, starting with those representations should help. The paper directly tests this hypothesis and finds it fails dramatically in the low-data regime:

"training from scratch or adapting from supervised ImageNet features are both unsuccessful with such small amounts of data" (Abstract)

In the real-robot experiments (Table II, "ImageNet" column), an ImageNet-initialized ResNet50 architecture with 800 exploration grasps—the maximum data budget in the study—achieves at best 47% on the offset gripper task (only a 4% improvement over the untuned base policy) and 0% on checkerboard backing, harsh lighting, and the baseline grasping task. The paper's explanation for this failure is revealing:

"We first attempted to fine-tune these ImageNet-based policies while holding the ImageNet feature layers constant, but this procedure failed to achieve any non-zero success rate. This suggests that, unlike adapting computer vision networks to new visual tasks, adapting end-to-end robot learning to new sensorimotor tasks may require changing the features used to represent the problem, and not just the post-processing of said features." (Section VI-C)

This is a critical insight: robotic skills require hand-eye coordination—the tight coupling between perception and action—and frozen visual features from a task-agnostic supervised training objective don't encode the kind of perceptual distinctions that matter for manipulation. The paper also notes that even when all layers are allowed to update during ImageNet-based fine-tuning, the performance remains poor, suggesting that the initial representation itself—what features the network learns to compute from the start—shapes whether fine-tuning with scarce data can succeed.

2. Simulation-to-Real Transfer Requires Significant Engineering

Another prominent approach to sample efficiency is training in simulation and transferring to reality, using techniques like domain randomization (Tobin et al., 2017 [66]; Peng et al., 2018 [48]; OpenAI et al., 2019 [45]), progressive networks (Rusu et al., 2016 [55]), or learned simulation parameter matching (Rastogi et al., year not specified, [52]; Jeong et al., 2019 [28]). The paper acknowledges these approaches but argues they "either require significant engineering effort to construct an appropriate simulation or significant supervision" (Section II). Building an accurate simulator for the checkerboard background, harsh halogen lighting, or transparent bottles would itself be a substantial modeling challenge—the very kind of bespoke per-task engineering that the paper wants to avoid.

3. Meta-Learning Assumes the Distribution of Tasks Is Known in Advance

A substantial body of work applies meta-learning to robotics, aiming to learn representations or initialization points that can be quickly adapted to new dynamics (Nagabandi et al., 2018 [41]; Alet et al., 2018 [2]; Nagabandi et al., 2018 [42]) or new objects (Finn et al., 2017 [14]; James et al., 2018 [27]; Yu et al., 2018 [69]; Bonardi et al., 2019 [3]). The paper distinguishes its approach from meta-learning on a crucial point:

"We consider adaptation to a broad class of changes including dynamics, object classes, and visual observations, including conditions that shift substantially from the training conditions, and do not require the full set of conditions to be represented during the initial training phase." (Section II)

The emphasis on "conditions that shift substantially from the training conditions" and "do not require the full set of conditions to be represented during the initial training phase" is key. Meta-learning typically requires training across a distribution of tasks that are representative of the adaptation targets. But in a real deployment, a robot might encounter something entirely unforeseen—a gripper physically extended by 1 cm, or a new kind of transparent object—that was never part of any training distribution. The paper's fine-tuning approach makes no such assumption: it adapts to changes that were completely absent during pre-training.

4. Goal-Conditioned and Multi-Task RL Share Data but Don't Adapt Features

Goal-conditioned RL (Agrawal et al., 2016 [1]; Nair et al., 2018 [44]; Pathak et al., 2018 [47]; Pong et al., 2019 [50]) and multi-task learning (Ruder, 2017 [54]) train policies that can handle multiple goals within a shared representation space. The paper notes these methods "share data and representations across multiple goals and objects" (Section II), but they still operate within a fixed task distribution defined during training. They don't provide a mechanism for post-deployment adaptation to genuinely novel conditions that shift the visual or dynamics distribution. A goal-conditioned policy trained on opaque objects won't suddenly know how to handle transparent ones—the representation wasn't built to accommodate that kind of visual shift.

5. Model-Based RL Adaptation Focuses on Predictive Models, Not End-to-End Policies

Recent work in model-based RL has explored adaptation through fine-tuning predictive models (Dasari et al., 2019 [8]), online search over pre-learned model or policy repertoires (Cully et al., 2015 [6]; Chatzilygeroudis et al., 2018 [5]; Merel et al., 2019 [38]), or learning simulator parameters from real data (Rastogi et al. [52]; Jeong et al., 2019 [28]). The paper distinguishes its contribution by demonstrating that "fine-tuning is successful with a model-free RL approach" (Section II)—that is, without learning an explicit dynamics model, which adds complexity and potential modeling error. Directly fine-tuning a model-free Q-function avoids the intermediate modeling step entirely.

6. The QT-Opt Pre-Training System Provides an Unusual Starting Point

The paper builds on the QT-Opt system (Kalashnikov et al., 2018 [29]), which pre-trains a grasping Q-function on 580,000 real grasp attempts across 1,000 diverse objects, followed by 28,000 online grasp attempts. This is an unusually strong pre-training baseline—most prior work on robotic adaptation does not start from a policy that already achieves 86% success on a challenging test set of unseen objects. This strong starting point is both a strength and a constraint: it allows the paper to study adaptation in a regime where the base policy is highly competent but brittle to specific distribution shifts, which mirrors the practical scenario of deploying a well-trained system and needing to handle edge cases. However, it also means the experimental conclusions may be tied to the specific properties of QT-Opt pre-training—the paper doesn't explore whether deep pre-training vs. shallower pre-training changes adaptation behavior, for example.

How This Paper Positions Itself

The paper's positioning is distinctive in several ways:

It prioritizes empirical demonstration over algorithmic novelty. Section I explicitly states:

"Instead of focusing on the robot's performance in the environment in which it was trained, we purposefully modify the robot and its environment, characteristic of the persistent change of the real world, and investigate its ability to adapt. Likewise, rather than proposing a new adaptation algorithm, with new complexity and caveats, we show how to successfully adapt robotic policies to substantial changes, using only the most basic components of existing off-policy reinforcement learning algorithms."

This is a deliberate methodological choice. Rather than adding algorithmic complexity—new loss functions, new architectures, new meta-learning objectives—the paper asks: given that we already have off-policy RL algorithms that can learn from past experience, can we simply continue training when conditions change? The answer, as established by the experiments, is a strong "yes" across a range of challenging distribution shifts. The contribution is not a new algorithm but a careful empirical demonstration that an existing, simple approach is surprisingly effective at a problem that the field has assumed requires more sophisticated solutions.

The paper frames this as a "first" in an important sense:

"To our knowledge, this work is the first to demonstrate that simple fine-tuning of off-policy reinforcement learning can successfully adapt to substantial task, robot, and environment variations which were not present in the original training distribution (i.e. off-distribution)."

The emphasis on "off-distribution" adaptation is the key qualification. Prior work had shown transfer between related tasks or within a task distribution, but not to conditions that shift the visual or dynamics distribution in ways the policy was never trained to handle.

It frames fine-tuning through a conceptual decomposition. The paper provides a clear four-step framework for fine-tuning in Appendix A: (1) pre-training, (2) exploration, (3) initialization, and (4) adaptation. This decomposition serves multiple purposes: it makes explicit the design choices that any fine-tuning method must address, it allows systematic ablation of those choices (e.g., "should we re-initialize the head or keep all parameters?"), and it connects the robotic RL fine-tuning problem to the well-established fine-tuning paradigms in computer vision and NLP. The authors also draw an explicit distinction between offline fine-tuning (where all adaptation uses a fixed collected dataset, with no further robot interaction during training) and online fine-tuning (where the robot continues to collect data during adaptation, and the updated policy is re-deployed for further exploration). The bulk of the paper's large-scale experiments use offline fine-tuning, motivated both by practicality (not needing to keep a robot in the loop during training) and by the finding in Appendix B-C that online fine-tuning with selective re-initialization suffers from exploration collapse.

It explicitly targets continual learning as the end goal. While the main experiments demonstrate single-step fine-tuning, the paper's ambition is broader. Section V describes a "continual learning experiment" where a single lineage of policies is fine-tuned across a sequence of five challenge tasks. The framing in the introduction is explicit about this aspiration:

"We believe this simple adaptation scheme provides a promising solution for creating a lifelong learning robotic agent, and show this potential using a simple continual learning experiment."

This positions fine-tuning not just as a tool for one-off adaptation, but as a mechanism for continuous improvement over a robot's operational lifetime. The paper's finding that continual fine-tuning imposes only a small performance penalty (4–7% at most) compared to single-step fine-tuning, and in one case actually improves (transparent bottles: +8%), is critical evidence for this claim.

Why the Specific Challenge Tasks Were Chosen

The paper's selection of five challenge tasks is itself a contribution to the problem framing. Rather than testing adaptation on arbitrary variations, the authors systematically probe distinct categories of distribution shift:

• Visual appearance of the scene (background, lighting): The checkerboard backing and harsh lighting challenge the policy's ability to handle low-level visual changes that don't affect the underlying grasping physics but corrupt the perceptual signal.
• Robot kinematics and embodiment (gripper extension, lateral offset): These tasks change the robot's physical relationship to the workspace. The 1 cm gripper extension changes the kinematics (lengthening the gripper distally) while also lowering the robot's relative pose by 1 cm. The 10 cm lateral offset shifts the entire gripper substantially—the paper notes that during pre-training "this policy experienced absolutely no variation in robot morphology" (Section III-B). This probes the policy's ability to recompute the hand-eye coordination mapping.
• Object properties (transparent bottles): This tests adaptation to fundamentally different visual material properties—transparency creates ambiguous depth cues and makes object boundaries difficult to distinguish, which is why the base policy "often grasps where two bottles are adjacent, i.e., as though it cannot differentiate which parts of the scene are inside vs outside a bottle" (Section III-B). This probes the policy's ability to learn new perceptual categories.

The paper notes that each of these was chosen because milder versions did not challenge the base policy. Conventional variations in workspace surface color, standard-intensity household lighting changes, and lateral offsets up to 5 cm all had no effect—the base policy was robust to these. The paper deliberately selected the threshold at which robustness breaks down, because that's the regime where adaptation is necessary. This thresholding approach is methodologically important: it demonstrates that the base policy isn't simply fragile—it generalizes well within its training distribution—and that the challenge tasks represent genuine distribution shifts that require learning, not just within-distribution variation that a more robust policy would handle.

3. Technical Approach

3.1 Reader Orientation

This is primarily an empirical methods paper whose core idea is that a robot can adapt to substantial environmental and morphological changes by simply continuing to train its pre-existing reinforcement learning policy on new data from the changed conditions—a process the paper calls "fine-tuning"—rather than requiring specialized adaptation algorithms or re-training from scratch. The system being studied is a vision-based robotic grasping pipeline that takes raw RGB images and outputs gripper motor commands; the paper's central claim is that this pipeline, when pre-trained on a general grasping task with off-policy RL, can be adapted to handle specific failure modes (adversarial backgrounds, harsh lighting, transparent objects, modified gripper geometry) using fewer than 800 new grasp attempts—less than 0.2% of the original training data.

3.2 Big-Picture Architecture (Diagram in Words)

The system has four major stages that execute in sequence:

1. Pre-training (offline → online QT-Opt): A Q-function neural network is trained using the QT-Opt off-policy RL algorithm, first on a static dataset of 580,000 real grasp attempts collected from a diverse object corpus, then refined through 28,000 online grasp attempts where the robot collects new data using its current policy and the Q-function is updated accordingly. This produces the "base policy"—a vision-based closed-loop grasping controller that takes a monocular RGB image and outputs Q-value estimates over candidate motor actions.

2. Exploration on the target task: The base policy is deployed on a modified version of the robot or environment (the "Challenge Task") to collect a modest dataset of grasp attempts—25, 50, 100, 200, 400, or 800 attempts. These attempts include both successful and failed grasps, providing signal about what the base policy handles well and where it fails under the new conditions.

3. Offline fine-tuning (parameter initialization + continued RL training): The QT-Opt algorithm is re-initialized with all parameters from the pre-trained Q-function. Training continues using a combined replay buffer containing both the original 580,000+ base task experiences and the newly collected target task experiences, sampled with equal probability. Training proceeds for 500,000 gradient steps at a reduced learning rate (25% of the pre-training rate), using the same Bellman error objective as pre-training. No additional robot interaction occurs during this phase.

4. Evaluation on the target task: The fine-tuned policy is deployed on the robot under the modified conditions and evaluated over 50 or more grasp attempts on unseen test objects (the same challenging subset of six objects used throughout). Success rate is the primary metric.

Information flows linearly: the original pre-training dataset → pre-trained Q-function → target task exploration dataset → combined replay buffer → continued Bellman updates → fine-tuned Q-function → deployed policy.

3.3 Roadmap for the Deep Dive

• First, the QT-Opt pre-training process (Section III-A and Kalashnikov et al., 2018 [29]), because the entire adaptation approach depends on having a strong off-policy pre-trained Q-function that can be resumed—understanding what the base policy knows and how it was trained is prerequisite to understanding what fine-tuning can add.

• Second, the five Challenge Tasks and the threshold-based method for selecting them (Section III-B), because these define the specific distribution shifts the system must adapt to and establish the performance baselines that fine-tuning must improve upon.

• Third, the fine-tuning procedure itself (Section IV-A and Appendix A)—the four-step conceptual framework, the specific choices made at each step (full parameter initialization, equal-probability data mixing, reduced learning rate, fixed gradient step budget), and the rationale for using offline rather than online fine-tuning.

• Fourth, the comparison methods (training from scratch, ImageNet initialization) and what they were designed to test (Section IV-B and Appendix B-C), because understanding the ablations is essential to interpreting the claim that "pre-training via RL is essential."

• Fifth, the continual learning experiment architecture (Section V), because it extends single-step fine-tuning to a sequential chain of adaptations and introduces the question of whether fine-tuning for Task N+1 degrades performance from Task N.

• Finally, the practical engineering choices that made the large-scale real-robot study feasible—including dataset partitioning, evaluation protocols, and the offline stopping-point problem (Section VI).

3.4 Detailed, Sentence-Based Technical Breakdown

QT-Opt Pre-Training: The Foundation for Adaptation

The paper does not propose a new pre-training method; it inherits the QT-Opt system from Kalashnikov et al. (2018) [29] and uses it as an off-the-shelf base policy. Understanding QT-Opt's mechanics is essential because every aspect of the fine-tuning procedure—the choice of Bellman error objective, the sampling strategy from combined replay buffers, the architecture of the Q-function, and the hyperparameter regime—is inherited directly from this pre-training pipeline.

What QT-Opt learns. QT-Opt trains a deep Q-function that maps from visual observations and candidate motor actions to scalar Q-values estimating the expected future return (probability of successful grasp, in this binary-reward setting). At deployment time, the policy selects the action with the highest Q-value from a set of sampled candidates. This is a continuous-action formulation: rather than discretizing the action space, the Q-function takes an action vector as input and the policy optimizes over actions by sampling candidates and evaluating them through the Q-network.

Two-stage training protocol. Section III-A describes the pre-training as occurring in two stages:

1. Offline stage: The Q-function is trained using only previously-collected data—580,000 real grasp attempts gathered over a corpus of 1,000 visually and physically diverse objects. These grasp attempts were collected using a variety of policies (not just the final policy), producing a dataset that covers diverse grasping strategies, successes, and failures. Training is purely offline: no new data is collected during this stage.

2. Online stage: After offline convergence, training continues with 28,000 additional grasp attempts collected by the robot using the current Q-function as its policy. After each batch of data collection, the Q-function is updated using both the new data and the offline dataset, and the updated Q-function is re-deployed to collect the next batch. This stage refines the policy by focusing data collection on the distribution of states and actions the current policy actually visits, correcting for the distribution shift between the offline data's behavior policy and the learned policy.

The authors clarify their terminology: they refer to this as "online" rather than "on-policy," because "the policy is still updated by the off-policy reinforcement learning algorithm"—that is, the Q-function update uses experience replay from a buffer containing data from many previous policies, not just the most recent one.

Architecture of the Q-function. The Q-function network processes visual input through a convolutional neural network (CNN) that extracts spatial features from the monocular RGB image. The robot's action (a vector encoding the desired gripper pose and grasp parameters) is injected into the network at an intermediate layer—the paper describes this as action features being added "in the middle of the CNN" (Section IV-B when contrasting with the ResNet50 ImageNet architecture where actions are concatenated "at the end of the CNN"). The network then produces a scalar Q-value through fully-connected layers.

The key design property inherited by fine-tuning is that perception and action processing are tightly interleaved within the architecture. The action features influence intermediate visual feature computation, which means the network can learn hand-eye coordination where visual features are computed differently depending on what action is being considered. This is why, as the paper argues in Section VI-C, freezing visual features (as in the failed ImageNet constant-features experiment) breaks the system: the visual features are not separable from the action context.

QT-Opt's Bellman error objective. The core training signal for Q-learning is the Bellman error—the difference between the Q-function's current estimate of a state-action pair's value and a target estimate that incorporates the reward actually received plus the discounted value of the best action at the next state. For the grasping task, the reward is binary: 1 for a successful grasp (the object is lifted above a threshold height) and 0 otherwise. Since grasping is an episodic task with a single step (the robot attempts a grasp and immediately observes success or failure), the discount factor is essentially zero in practice—there are no multi-step returns to propagate.

The QT-Opt algorithm extends standard Q-learning with several important mechanisms that carry over to fine-tuning:

• Cross-entropy method (CEM) for action selection: At deployment time, the policy selects actions by sampling a set of candidate actions from a Gaussian distribution, evaluating them through the Q-function, refitting the sampling distribution to the highest-scoring candidates, and repeating. This allows optimization over a continuous action space without requiring a separately learned policy network.

• Bellman backups with target networks: QT-Opt maintains a separate target Q-network whose parameters are periodically copied from the main Q-network. The Bellman target is computed using this target network, which stabilizes training by breaking the feedback loop where the same network being updated also provides the update target.

• Clipped double Q-learning: To reduce overestimation bias, QT-Opt trains two Q-functions and uses the minimum of their predictions when computing the Bellman target. This conservative estimate prevents the policy from exploiting over-optimistic Q-values for actions that haven't been sufficiently explored.

What the base policy can and cannot do. After pre-training, the base policy achieves 96% accuracy on a set of previously-unseen test objects and 86% on the challenging subset of six objects used for the fine-tuning experiments (Section III-A). This high baseline performance is important for interpreting the fine-tuning results: the base policy is competent but brittle. It succeeds on a wide range of objects and conditions within its training distribution, but it fails on specific distribution shifts because it has never encountered the visual patterns (checkerboard edges in place of object edges), physical relationships (transparency disrupting depth perception), or kinematic mappings (offset gripper changing the visual field) that those shifts introduce.

This brittleness is a consequence of what the Q-function learned during pre-training—it learned to exploit reliable correlations between visual features and grasp success that hold in the training distribution but break under the challenge conditions. For example, in the base environment with a featureless grey background, any strong edge in the image is almost certainly part of a graspable object. The checkerboard backing introduces strong edges that are not objects, and the base policy "often fools the robot into grasping at checkerboard edges rather than objects" (Section III-B) because its learned heuristic—"edges are graspable"—was perfectly valid during training but catastrophically wrong under the shift.

This sets up the fine-tuning problem: the Q-function doesn't need to be rebuilt from scratch—it needs to be corrected where its learned heuristics break. The structural knowledge (how to approach objects, how to close the gripper, how to verify a grasp) is still valid; only the perceptual mapping from specific visual patterns to grasp affordances needs updating for each challenge.

The Five Challenge Tasks: Threshold-Based Selection of Distribution Shifts

The paper's selection of challenge tasks follows a principled methodology: for each category of potential distribution shift, the authors tested increasingly severe variations until they found one that significantly degraded base policy performance. Milder versions that the base policy handled robustly were discarded. This ensures that each challenge task represents a genuine distribution shift requiring new learning, not just a within-distribution variation that a more robust base policy might have handled.

Background (Checkerboard Backing). The workspace surface was modified by gluing a 1-inch black-and-white checkerboard pattern to the bin bottom. The paper reports that "conventional variations in the workspace surface, such as uniform changes in color or specularity, have no effect on the base policy's performance" (Section III-B). The checkerboard was chosen specifically because it introduces structured edges that the base policy misinterprets as object boundaries. The resulting performance drop is from 86% to 50%, a 36-percentage-point degradation. This task probes the policy's ability to disambiguate task-relevant edges (object boundaries) from task-irrelevant edges (background patterns)—a visual discrimination problem that requires learning which specific visual features in the workspace predict grasp success.

Lighting Conditions (Harsh Lighting). A high-intensity halogen light source was placed parallel to the workspace, creating a bright spot in the camera view and intense light-dark contrasts. The paper notes that "mild perturbations in lighting conditions (i.e., those which can be created by standard-intensity household lights) have no effect on the base policy's performance" (Section III-B). The halogen light was chosen because it produces visual artifacts (glare, overexposure, robot reflections visible in the bin) that corrupt the visual features the policy normally relies on. Performance drops from 86% to 32%, a 55-percentage-point degradation—the largest drop of any challenge. This task probes the policy's ability to extract grasp-relevant visual information under severe photometric corruption.

Gripper Shape (Extend Gripper 1 cm). The parallel gripper was extended by 1 cm and significantly narrowed in the process, changing the robot's kinematics by lengthening the gripper distally and lowering its relative pose with respect to the workspace. This is a physical modification to the agent's embodiment, not just a visual change. Performance drops from 86% to 76%, a more modest 10-percentage-point degradation. This task probes the policy's ability to recalibrate the hand-eye coordination mapping when the physical relationship between commanded actions and visual feedback changes—the gripper tip now extends further than the policy expects, so grasp positions that were previously correct may now cause the gripper to collide with the workspace or miss the object.

Robot Morphology (Offset Gripper 10 cm). The gripper was translated laterally by 10 cm—approximately a full gripper or arm link width. The paper emphasizes that "during training this policy experienced absolutely no variation in robot morphology" (Section III-B), and that offsets up to 5 cm had no impact on performance. The 10 cm offset drops performance from 86% to 47%, a 39-percentage-point degradation. This task probes adaptation to large-scale morphological change, where the entire visual field of the robot shifts relative to its end-effector, requiring a fundamental recomputation of the visuomotor mapping rather than just a local adjustment. The paper notes that this shift changes "the location of the robot in its own visual field drastically" (Section VI-C, discussing Figure 7).

Unseen Objects (Transparent Bottles). Completely transparent plastic beverage bottles were introduced that were not present in the pre-training object corpus. The paper reports robustness to many novel objects, as long as they "have significant opaque components"—including labeled drink bottles grasped with 98% success despite never appearing during training. However, transparency causes failure: the base policy "often grasps where two bottles are adjacent, i.e., as though it cannot differentiate which parts of the scene are inside vs outside a bottle" (Section III-B). Performance drops from 86% to 49%, a 37-percentage-point degradation. This task probes the policy's ability to learn new perceptual object categories—specifically, learning that transparent surfaces indicate object boundaries that must be navigated around, not grasped through.

Why these five categories? Together, they span four distinct axes of distribution shift: visual appearance (background, lighting), robot embodiment (gripper shape, morphology offset), and object properties (transparency). This breadth is deliberate: the paper aims to show that fine-tuning works across categorically different types of change, not just one type. If fine-tuning only worked for lighting changes but not for morphological changes, the conclusion would be much weaker. The inclusion of both perceptual and physical shifts—and the finding that fine-tuning handles both—is central to the paper's claim of generality.

The Fine-Tuning Procedure: Design Choices and Their Rationale

The fine-tuning method is deliberately minimal. Section IV-A describes it as "a very simple fine-tuning procedure" and the paper emphasizes that it uses "only the most basic components of existing off-policy reinforcement learning algorithms" (Section I). This minimalism is a feature, not a bug: the paper's argument is precisely that no special adaptation machinery is needed—standard continued training with appropriate data mixing suffices.

The procedure can be decomposed using the four-step conceptual framework from Appendix A:

Step 1: Pre-training (already completed). The base Q-function was trained with QT-Opt as described above. This is treated as a fixed starting point; the paper does not experiment with alternative pre-training methods or amounts.

Step 2: Exploration. The base policy is deployed on the target challenge task to collect exploration data. Critically, the exploration policy is the pre-trained policy itself—no exploration noise, no epsilon-greedy randomization, no separate exploration strategy. The paper states "in our study, and what we believe to be most representative of a real-world continual learning scenario, we always use the pre-trained policy for exploration" (Appendix A, Step 2 description). This is an important design choice: it means the exploration data is concentrated on states and actions that the pre-trained policy actually visits, which are precisely the states where it fails under the new conditions. Random exploration would produce many states and actions that are irrelevant to the task (random motor commands rarely produce informative grasps), wasting the scarce data budget.

The consequence is that the exploration dataset is on-distribution for the base policy but off-distribution for the target task's optimal policy. The data contains examples of the base policy trying grasps that work in the original environment but fail under the new conditions—exactly the kind of negative examples needed to update the Q-function's faulty heuristics.

Step 3: Initialization. The Q-function network is initialized with all parameters from the pre-trained base policy. The paper explicitly rejects selective initialization (e.g., re-randomizing the final layers while keeping early CNN layers frozen):

"The standard implementation of this step is to start with the entire pre-trained network. Some techniques may choose to use only a subset of the pre-trained network (e.g., truncating the last few layers of a CNN)." (Appendix A, Step 3 description)

The negative results from Appendix B, Figure 8 demonstrate why full initialization is necessary: in a simulated grasping experiment comparing full initialization to "add a new head" (re-randomizing the last layers), the full initialization policy reaches base-policy performance after 5,000 exploration grasps (~12% of the base training data), while the new-head policy barely reaches 30% after the same amount of data. The paper explains this gap as a consequence of online exploration requirements:

"online fine-tuning requires to maintain a policy that can competently explore the target task at all times, any method which compromises the performance of such a policy–even temporarily–has a high risk of failing as a sample-efficient fine-tuning technique" (Appendix B)

Even though the main experiments use offline fine-tuning (no exploration-collection loop during training), the principle still applies: re-randomizing any parameters would discard knowledge that the pre-training encoded, forcing the network to re-learn that knowledge from the tiny target dataset—exactly the sample-inefficient regime the paper aims to escape.

Step 4: Adaptation (the training update). The fine-tuning update uses the identical QT-Opt algorithm as pre-training, with three specific modifications:

1. Combined replay buffer: Training examples are sampled from both the base task dataset (580,000+ offline grasps plus 28,000 online grasps) and the target task dataset (25–800 grasps), with equal probability of sampling from each. The paper explored different data mixing ratios in simulation (Appendix C, Figure 9): higher fractions of target task data allow the policy to achieve higher performance faster, but the equal-probability choice for the real-robot experiments ensures that the base task knowledge is not forgotten while the policy adapts to the new condition. This sampling strategy is implemented by maintaining separate data buffers and randomly selecting which buffer to sample from for each minibatch.

2. Reduced learning rate: The learning rate is set to $10^{-4}$, which the paper states is 25% of the learning rate used for pre-training. This prevents the policy from rapidly overfitting to the small target dataset and overwriting the pre-trained parameters. A full learning rate would cause the Bellman error gradient on the tiny target dataset to dominate the update, quickly erasing the pre-training knowledge.

3. Fixed training budget: Fine-tuning proceeds for 500,000 gradient steps regardless of the target dataset size or task. The paper selected this number based on offline validation experiments (Section VI-B, Figure 6), which showed that performance on the offset gripper task with 400 exploration grasps peaks around 500,000 steps and then "precipitously drops and never recovers, dropping below even the initial performance of the base policy from which it was trained, as the initialization is being overwritten by overfitting to the target samples." The 500,000-step choice is a compromise—"not necessarily optimal for all of our tasks and datasets" (Section VI-B)—but provides a consistent experimental protocol.

Objective function for fine-tuning. The training objective during fine-tuning is exactly the same Bellman error minimization used during pre-training—there is no auxiliary loss, no regularization term, no distillation objective. The Q-function is updated to minimize:

$\mathcal{L}_{\text{Bellman}} = \mathbb{E}_{(s, a, r, s') \sim \mathcal{D}} \left[ \left( Q_\theta(s, a) - \left( r + \gamma \min_{i=1,2} Q_{\bar{\theta}_i}(s', a') \right) \right)^2 \right]$

where $\mathcal{D}$ is the combined replay buffer sampling equally from base and target data, $s$ is the visual observation (RGB image), $a$ is the gripper motor command, $r \in \{0, 1\}$ is the binary grasp success reward, $s'$ is the next observation (irrelevant for single-step grasping with $\gamma \approx 0$), $Q_\theta$ is the main Q-function with parameters $\theta$, $Q_{\bar{\theta}_1}$ and $Q_{\bar{\theta}_2}$ are the two target Q-functions with periodically copied parameters, and $a'$ is the action selected by the CEM optimizer using the target Q-functions.

What it computes: For each sampled transition, the current Q-value estimate $Q_\theta(s, a)$ is compared against a target computed as the reward plus the minimum of the two target networks' predictions for the optimal next action. Since the grasping task has $\gamma \approx 0$ and a binary reward, the target essentially reduces to the reward itself—the network is learning to predict whether a grasp will succeed, with the clipped double-Q minimum providing a conservative estimate. The squared error penalizes deviations between the predicted Q-value and this target. The expectation is over the minibatch, producing a scalar loss per gradient step.

Why this form: QT-Opt's Bellman error minimization is inherited without modification because it is the same learning problem—estimate grasp success from visual observations and actions—just under a shifted data distribution. Adding regularization terms would be unnecessary given the small number of gradient steps relative to pre-training, and the equal-probability data mixing from the combined replay buffer already serves as an implicit regularizer by ensuring the network continues to see examples where its original heuristics are correct. The clipped double-Q target is retained because overestimation bias would be even more severe on the small target dataset, where the Q-function might overfit to optimistic predictions for underexplored actions.

Why offline rather than online fine-tuning. The paper explicitly frames the choice of offline fine-tuning as motivated by practicality:

"We also believe that offline fine-tuning is more practical than online fine-tuning, due to the inherent complexity of placing a robot in the loop of a reinforcement learning algorithm" (Appendix C)

Online fine-tuning would require robot interaction during training: deploy the current partially-fine-tuned policy, collect more data, update, and repeat. This creates several practical challenges: (1) the robot must be available and operational for the entire training duration, (2) the updated policy must maintain sufficient exploration capability to collect informative new data at each iteration, (3) the risk of policy degradation (as in Figure 6) would translate directly to wasted robot time and potential hardware damage. Offline fine-tuning avoids all of these by decoupling data collection (a one-time batch of 25–800 grasps) from training (500,000 gradient steps on a GPU).

The downside, which the paper identifies explicitly in Section VI-B, is the offline stopping-point problem: without online evaluation, there is no way to determine when the policy has peaked and overfitting has begun. The paper treats this as an open problem and points to off-policy evaluation methods (Irpan et al., 2019 [26]) as a potential solution.

Comparison Methods: What "Training from Scratch" and "ImageNet Initialization" Test

The paper includes two comparison methods that serve as ablations to isolate the contribution of RL-based pre-training to fine-tuning success. Both comparisons use the same 800-exploration-grasp dataset as the most data-rich fine-tuning experiments and the same 500,000 gradient step budget.

Training from Scratch (random initialization). The Q-function network is initialized with random parameters. All other aspects of the pipeline are identical: the same QT-Opt training algorithm, the same 800-grasp dataset, the same number of gradient steps. The purpose is straightforward: if fine-tuning succeeds while training from scratch fails, the pre-trained parameters are essential—the target task cannot be learned from the small dataset alone.

The results in Table II confirm this dramatically: training from scratch achieves 0% success on checkerboard backing, extend gripper, offset gripper, and the baseline grasping task. It achieves 4% on harsh lighting and 27% on transparent bottles—the only task where random initialization makes any headway, and even then at less than half the base policy's pre-adaptation performance (49%). This establishes that the 800-grasp dataset, by itself, contains insufficient information to learn grasping from nothing. The pre-trained parameters provide an initialization point in parameter space that is close enough to a good solution on the target task that 500,000 gradient steps can reach it; random initialization is too far away.

ImageNet Initialization (ResNet50 features). The Q-function architecture is modified by replacing the CNN trunk with a ResNet50 architecture (He et al., 2016 [21]) initialized with weights from supervised ImageNet classification training. The action features are concatenated at the end of the CNN ("rather than the adding them in middle of the CNN, as in the original architecture," Section IV-B). The remaining fully-connected layers are randomly initialized. During fine-tuning, all parameters are updated, including the ResNet50 layers.

This comparison tests a specific hypothesis: that the adaptation success comes from having a good visual representation, and that ImageNet features might provide an equivalent or better starting point since they are trained on a much larger and more diverse dataset of natural images. The results refute this hypothesis: the best ImageNet-based policy achieves 47% on the offset gripper task (only 4 percentage points above the base policy's pre-adaptation performance of 43%), and achieves 0% on half the tasks. Notably, the paper first attempted to fine-tune while freezing the ImageNet feature layers:

"We first attempted to fine-tune these ImageNet-based policies while holding the ImageNet feature layers constant, but this procedure failed to achieve any non-zero success rate" (Section VI-C)

This failure is diagnostic. If the problem were merely that the robot needs to recognize objects (a standard computer vision task), frozen ImageNet features should suffice—the policy would just need to learn to map recognized object locations to grasp actions. The fact that frozen features fail completely means the adaptation requires changing how visual features are computed, not just re-weighting existing features. The hand-eye coordination mapping—where early visual features are computed differently depending on the action being considered—cannot be achieved with frozen features. The paper's interpretation:

"This suggests that, unlike adapting computer vision networks to new visual tasks, adapting end-to-end robot learning to new sensorimotor tasks may require changing the features used to represent the problem, and not just the post-processing of said features." (Section VI-C)

This is a core argument of the paper: robotic skills require integrated perception-action learning that supervised visual pre-training cannot provide. The ImageNet features encode what objects look like, not how to interact with them, and the distinction is critical in the low-data regime.

Dataset Construction and Evaluation Protocol

The experimental design for the large-scale evaluation (Section IV-B) involves careful dataset partitioning and evaluation protocols to enable systematic comparison across tasks, data sizes, and methods.

Dataset collection and partitioning. For each of the 5 challenge tasks plus the baseline grasping task (6 conditions total), the authors collected 800 grasp attempts. Each of these 6 datasets was then partitioned into 6 tiers by number of exploration grasps: 25, 50, 100, 200, 400, and 800. This yields 36 individual datasets—6 conditions × 6 data sizes. For each dataset, a separate fine-tuned policy was trained using the offline procedure (500,000 gradient steps, combined replay buffer, reduced learning rate). This combinatorial design allows the paper to answer two questions simultaneously: (1) does fine-tuning improve over the base policy? (the across-data-size trend), and (2) how much data is needed? (the across-data-size comparison).

Evaluation protocol. Each fine-tuned policy was evaluated by deploying it on the real robot under the corresponding challenge condition and executing 50 or more grasp attempts. The bin contents were shuffled between trials using "a randomly-generated sequence of sweeping movements with the end-effector" (Section IV-B, Step d) to reduce evaluation variance—without shuffling, the bin state after a failed or successful grasp influences the next trial, introducing temporal correlation that inflates the effective sample size.

Scale of the experiment. The paper reports that the full experiment "required more than 15,000 grasp attempts and 14 days of real robot time, and was conducted over approximately one month" (Section IV-B). The 15,000 grasp attempts estimate can be reconstructed: 48 policies (36 fine-tuned + 6 scratch comparisons + 6 ImageNet comparisons) × 50+ evaluation grasps each ≈ 2,400 evaluation grasps, plus the initial 4,800 data collection grasps (6 conditions × 800 grasps), plus the continual learning experiment and any pilot runs—consistent with the reported total.

The Continual Learning Experiment Architecture

Section V extends the single-step fine-tuning procedure to a sequential chain of adaptations, testing whether fine-tuning can serve as a building block for continual (lifelong) learning. The architecture is straightforward: after fine-tuning on one challenge task, the resulting adapted policy—not the original base policy—becomes the initialization for fine-tuning on the next challenge task.

Sequence ordering. The fixed sequence is: Harsh Lighting → Transparent Bottles → Checkerboard Backing → Extend Gripper 1 cm → Offset Gripper 10 cm. The paper does not explain why this specific order was chosen, nor does it experiment with alternative orderings. Each transition uses the maximum data budget (800 exploration grasps for the new task).

What this experiment tests. The naive concern with continual fine-tuning is catastrophic forgetting: when fine-tuning on Task N+1, the policy might overwrite the adaptations it made for Task N, losing performance on the earlier task. The continual learning experiment partially addresses this concern by providing an upper bound on forgetting: since the evaluation at the end of the chain measures performance on the final task (Offset Gripper 10 cm), it measures whether the intermediate fine-tuning steps degraded the policy's ability to learn the final task compared to single-step fine-tuning from the original base policy. If catastrophic forgetting were severe, the policy lineage would accumulate parameter changes that make it increasingly difficult to adapt to new tasks, resulting in lower final-task performance for the continual chain compared to the single-step variant.

Results interpretation. The paper reports that continual fine-tuning performance on the final three tasks (Checkerboard, Extend Gripper, Offset Gripper) is 4–7 percentage points below the single-step versions—small differences that are "within the margin-of-error of our evaluation procedure" (Section V). The Transparent Bottles task actually improves by 8 points in the continual setting. This suggests that catastrophic forgetting is not a major issue in this experimental setup—the adaptations for previous tasks do not significantly impair the ability to adapt to new tasks, at least over a chain of five tasks. However, the paper does not evaluate performance on the earlier tasks after subsequent fine-tuning steps (i.e., does the policy still grasp under harsh lighting after being fine-tuned on transparent bottles, checkerboard, etc.?). This leaves open the question of whether the earlier-task performance is preserved or forgotten, though the paper frames this as future work (Section VII).

The implicit assumption. The continual learning experiment relies on the assumption that the parameters learned for one challenge task are at least partially compatible with learning the next task—that there is a region of parameter space where performance on all tasks is reasonable, and that sequential fine-tuning approximately navigates along this region. The fact that the performance penalty is small supports this assumption, but the paper does not characterize the geometry of the parameter space or provide theoretical justification.

The Offline Stopping-Point Problem: A Practical Limitation

Section VI-B identifies a critical limitation of offline fine-tuning: without a mechanism to detect when performance has peaked, the choice of when to stop training must be made heuristically, and the consequences of stopping too late are severe.

The overfitting phenomenon (Figure 6). For the offset gripper task with 400 exploration grasps, the real-robot evaluation performance follows a characteristic trajectory: it rises quickly from the base policy's 43% to around 80–90% by 500,000 gradient steps, then "precipitously drops and never recovers, dropping below even the initial performance of the base policy." The drop is not gradual—it is catastrophic regression to worse-than-initial performance. The mechanism is overfitting: with only 400 examples of the target task and 500,000 gradient steps, the Q-function eventually memorizes the specific grasps in the dataset rather than learning generalizable features, overwriting the pre-trained parameters that provided robustness. The paper frames this as the initialization being "overwritten by overfitting to the target samples" (Section VI-B).

Why this matters for continual learning. In a continual learning setting, the problem compounds: if each fine-tuning step pushes the policy past the point of overfitting, the initialization for the next step is a degraded policy rather than a peak-performance policy. This would amplify the performance penalty across the chain. The paper's continual learning experiment likely avoided this because it used a fixed 500,000-step budget that was chosen to be just before the overfitting cliff, but a robot operating autonomously without prior knowledge of when to stop would not have this information.

Proposed solution. The paper points to off-policy evaluation methods (Irpan et al., 2019 [26]) as a potential solution that would "allow us to continuously monitor progress of the online fine-tuning process without costly real-robot evaluations" (Section VII). Off-policy evaluation techniques estimate a policy's performance using only logged data, without deploying the policy. If an off-policy estimate could reliably detect the peak and signal that training should stop, the overfitting problem would be solved without requiring robot interaction during training.

Parameter Change Analysis: What Fine-Tuning Actually Modifies

Figure 7 provides a window into the internal mechanism of adaptation by visualizing the cosine distance between the parameters of the pre-trained and fine-tuned networks for each challenge task. The analysis reveals a non-obvious pattern: even purely morphological changes induce substantial updates to the visual processing layers.

For the visual challenge tasks (Checkerboard Backing, Harsh Lighting), large parameter changes in the convolutional layers are expected—the policy needs to learn to interpret new visual patterns. More surprising is the finding for the Offset Gripper 10 cm task, which changes only the robot's morphology, not the visual environment. Yet Figure 7 shows that this morphological shift induces "substantial changes to the network's image-processing parameters (e.g. layers conv2–conv7)" (Section VI-C).

The paper's interpretation is that this reflects the integrated nature of visuomotor control:

"Offsetting the gripper not only changes robot morphology, it changes the location of the robot in its own visual field drastically. In order to perform effective visual servoing with a new morphology, both the image and action-processing parts of the network must be updated." (Section VI-C)

This explains why ImageNet initialization fails: the visual features needed for manipulation are not just object recognition features—they include the spatial relationship between the robot's own body and the objects it manipulates. When the gripper is offset by 10 cm, the hand-eye coordination mapping is completely disrupted: the same visual observation now corresponds to a different physical relationship, and the policy must learn to reinterpret visual features in terms of the new kinematic configuration. This requires changing how the visual features themselves are computed, not just how they are mapped to actions.

Normalization of parameter distances. The bar heights in Figure 7 are "normalized by the magnitude of parameter changes induced in the Q-function network by fine-tuning the baseline grasping task" (Figure 7 caption). This normalization is important: it controls for the fact that some amount of parameter change occurs simply from continued training on any data (including the baseline grasping task), and isolates the additional change attributable to each specific challenge. Without this normalization, it would be unclear whether observed parameter changes reflect adaptation to the challenge or just the effect of additional training.

Summary of Design Choices and Their Justifications

• Full parameter initialization (no selective re-initialization): Prevents destroying pre-trained knowledge that is still relevant to the target task. The simulation experiment in Appendix B shows that re-randomizing even the final layers severely degrades sample efficiency. The empirical finding that even visual perception layers need updating (Figure 7) further justifies why freezing any part of the network would be counterproductive.

• Equal-probability data mixing from base and target datasets: Provides implicit regularization by ensuring the network continues to see examples where its original heuristics are correct, preventing catastrophic forgetting. The simulation ablations (Appendix C) suggest that higher target-data ratios would accelerate adaptation, but the equal-probability choice is a conservative default that avoids premature specialisation.

• Reduced learning rate (25% of pre-training rate): Prevents the small target dataset from dominating the gradient updates and overwriting pre-trained parameters. This is a standard transfer learning practice from computer vision and NLP (Howard and Ruder, 2018 [23, 24], Section II and Appendix A).

• Offline fine-tuning with a fixed gradient step budget: Maximizes practicality by decoupling data collection from training, allowing the computationally expensive gradient updates to occur without robot involvement. The fixed budget eliminates the need for online evaluation during training, at the cost of the offline stopping-point problem (Section VI-B).

• Pre-trained policy as the exploration policy (no exploration noise): Concentrates the limited exploration budget on states and actions the policy actually visits—precisely where it fails under the new conditions. Random exploration would waste data on irrelevant regions of the state-action space.

• QT-Opt retention without modification: The paper deliberately avoids changing the RL algorithm itself, isolating the contribution of pre-training to adaptation performance. If a modified algorithm were necessary, the paper's claim that "simple continued training works" would be weaker.

• Combinatorial dataset construction (6 tasks × 6 data sizes): Enables systematic analysis of both the effect of data quantity and the effect of task type on fine-tuning performance. Without varying data size, the paper could not make its central claims about sample efficiency (<0.2% of original data).

4. Key Insights and Innovations

Innovation 1: Reframing Robotic Adaptation as a Data-Regime Problem, Not an Algorithm Problem

The paper's most distinctive intellectual move is not proposing a new adaptation algorithm but rather diagnosing why adaptation has been hard and showing that the difficulty is primarily a data-regime problem, not an algorithm problem. The dominant assumption in the field—implicit in the proliferation of meta-learning, sim-to-real, and domain adaptation methods—was that adapting robotic skills to new conditions requires specialized machinery: learned initializations that are "close" to task-specific solutions, explicit domain randomization, or structured representations that factor out environmental variation. The paper challenges this assumption head-on by demonstrating that none of this machinery is necessary when you start from a sufficiently strong pre-trained policy and use the most basic continued training procedure available.

The diagnostic insight is subtle and worth unpacking. Prior work on robotic adaptation almost universally operated in a regime where the pre-training task and target task were learned under comparable data budgets—a policy trained on a few hundred demonstrations adapted to a new task with a few dozen more. In that regime, the pre-trained policy has relatively little encoded knowledge, and specialized algorithms are genuinely needed to extract and transfer what limited structure exists. But the QT-Opt pre-training pipeline flips this: the base policy is trained on 580,000 grasps across 1,000 objects, encoding an enormous amount of structural knowledge about grasping. In this data-rich pre-training regime, the pre-trained parameters already contain most of what's needed—they're in a region of parameter space where a good target-task solution is reachable via a modest number of gradient steps on a tiny dataset. The "algorithm" for adaptation is just continued Bellman error minimization; the enabling condition is the depth of pre-training, not the cleverness of the adaptation mechanism.

This reframing parallels a well-known phenomenon from supervised deep learning. In the early 2010s, substantial research went into designing specialized transfer learning architectures and objectives. By the mid-2010s, the field had converged on a simpler insight: with a sufficiently large pre-training dataset (ImageNet) and a sufficiently deep network, simple fine-tuning with a modest learning rate works remarkably well across a wide range of tasks. The specialized methods weren't wrong—they were solving a problem that disappeared when pre-training scale crossed a threshold. The present paper argues, through empirical demonstration, that robotic RL is crossing an analogous threshold: when pre-training reaches the scale of hundreds of thousands of real-world interactions, the adaptation problem simplifies dramatically.

The evidence for this reframing is the systematic failure of the scratch-training and ImageNet-initialization baselines (Table II): training from scratch on the 800-grasp datasets achieves 0% on four of six tasks, and ImageNet initialization achieves at best 47% on one task while scoring 0% on three others. These aren't just worse—they're catastrophically worse, demonstrating that the target-task data alone contains essentially no learnable signal for grasping without the pre-trained initialization. The pre-trained Q-function isn't providing a mild head start; it's providing the only pathway to a successful policy in the low-data regime. This turns the conventional framing on its head: rather than asking "what algorithm can extract useful knowledge from pre-training?," the paper suggests the right question is "at what pre-training scale does the adaptation algorithm become irrelevant?"

The significance of this reframing extends beyond the specific QT-Opt system. It suggests that the robotics community's focus on developing ever-more-sophisticated adaptation algorithms may be misallocated relative to the bottleneck: scaling up pre-training. If the limiting factor is the depth of pre-trained knowledge rather than the cleverness of the adaptation procedure, then research effort is better spent on building large-scale, diverse pre-training datasets and the infrastructure to collect them, rather than on incrementally improving meta-learning objectives. This is a fundamentally different research prioritization than what the field had been pursuing, and the paper's evidence—while restricted to one pre-training pipeline and one task family—provides the most compelling real-robot demonstration to date that the threshold where "simple fine-tuning suffices" is achievable with current methods.

Innovation 2: Demonstrating That RL Pre-Training Is Qualitatively Different from Supervised Visual Pre-Training for Sensorimotor Adaptation

The paper's comparison between QT-Opt pre-training and ImageNet initialization is not just a baseline—it's a diagnostic experiment that reveals a fundamental property of visuomotor skill representations. The finding that ImageNet features, even when fine-tuned end-to-end rather than frozen, produce policies that achieve at best 47% success (and 0% on half the tasks) with the same 800-grasp data budget establishes that supervised visual pre-training and reinforcement learning pre-training encode fundamentally different kinds of knowledge, and that this difference is decisive for sample-efficient sensorimotor adaptation.

The dominant assumption in much of robot learning has been that perception and action can be partially decoupled: a good visual representation (from ImageNet, from self-supervised learning, from object detectors) provides a foundation on which action selection can be built. Under this assumption, adapting to a new visual condition (checkerboard background, harsh lighting, transparent objects) should primarily require updating the action-selection layers while the visual backbone either remains fixed or requires only minor tuning. The paper systematically falsifies this assumption. The frozen-ImageNet-features variant "failed to achieve any non-zero success rate" (Section VI-C), and even the fine-tuned variant dramatically underperforms. This means the perceptual changes needed for adaptation—learning that checkerboard edges are not object boundaries, that glare-corrupted pixels don't indicate grasp affordances, that transparent surfaces indicate object boundaries—require restructuring the visual feature hierarchy itself, not just re-weighting existing features.

The deeper insight is that grasping, as learned by QT-Opt, is not a two-stage process of "perceive then act." The Q-function architecture injects action information into the middle of the CNN, creating a tight coupling where visual features are computed conditioned on the action being evaluated. This means the network learns visual features that are specific to the grasping task—features that encode "graspability" rather than "objectness." ImageNet features encode object categories, boundaries, and textures that are useful for classification but not necessarily for manipulation. A checkerboard pattern looks like a checkerboard to an ImageNet-trained network regardless of whether grasping there would succeed; a QT-Opt-trained network has learned that edges in the workspace are grasp affordances because during pre-training, they always were. The adaptation required for the checkerboard task is not learning to recognize checkerboards—it's learning that specific visual patterns that previously predicted grasp success no longer do, and that requires rewriting the feature detectors themselves.

Figure 7 provides mechanistic evidence for this claim: even the purely morphological "Offset Gripper 10 cm" task induces substantial changes in early-to-mid convolutional layers (conv2–conv7), demonstrating that visual feature computation must be restructured when the robot's embodiment changes, even though the visual environment is identical. This is because the offset changes the spatial relationship between the robot's own body (visible in the camera image) and the workspace, requiring the network to learn a new hand-eye coordination mapping that necessarily changes how visual information is processed.

The theoretical significance is that sensorimotor skills are not perception plus action—they are integrated perception-for-action representations that supervised visual pre-training cannot provide. This has direct implications for research prioritization: efforts to improve robotic learning through better visual representations (from larger supervised datasets, from self-supervised learning on internet video, from multi-modal models) may hit a fundamental ceiling because the representations needed for manipulation are inherently task-specific. The paper's finding suggests that interaction-based pre-training—where the robot learns vision and action jointly through trial and error—produces a qualitatively different and more adaptable representation than perception-only pre-training, regardless of the scale of the perception-only data.

Innovation 3: Establishing the Performance-Degradation Threshold as a Systematic Method for Defining Adaptation Benchmarks

The paper's method for constructing Challenge Tasks—testing increasingly severe variations until finding the threshold where base policy performance degrades meaningfully—is a methodological contribution to how adaptation research should be evaluated. It replaces an ad-hoc approach to task selection with a principled procedure that ensures the benchmark tasks genuinely require learning rather than just reflecting within-distribution variance.

The standard practice in transfer learning and domain adaptation research is to select target tasks based on convenience or convention: a different dataset, a different simulator, a different robot platform. The problem with this approach is that it provides no guarantee that the target task is a genuine distribution shift rather than just a difficult instance of the base task distribution. A base policy might fail on a particular object not because it requires adaptation but because the base training didn't include enough similar objects—a data coverage problem, not a distribution shift problem. Conversely, a base policy might succeed on a seemingly different task because the learned features are robust to that particular variation, giving a false negative for adaptation methods. Without systematically characterizing the robustness boundary of the base policy, it's impossible to know whether an adaptation experiment is measuring the right thing.

The paper's thresholding approach addresses this by explicitly finding the point of failure for each category of variation. For background changes: uniform color and specularity changes had no effect; a checkerboard pattern caused a 36-percentage-point drop. For lighting: standard household lights had no effect; a high-intensity halogen source caused a 55-point drop. For morphology: lateral offsets up to 5 cm had no effect; a 10 cm offset caused a 39-point drop. For objects: novel opaque objects were grasped with up to 98% success; transparent bottles caused a 37-point drop. In each case, the paper identifies a regime where the base policy is robust (demonstrating that it generalizes well within its training distribution) and a regime just beyond that boundary where it fails (demonstrating a genuine distribution shift). The Challenge Tasks are defined at the failure boundary, ensuring that successful adaptation represents new learning rather than within-distribution generalization.

This methodology is significant because it provides a definition of adaptation difficulty calibrated to the specific pre-trained policy. A different base policy—trained on different data, using a different algorithm, with different inductive biases—would have different robustness boundaries and therefore different appropriate Challenge Tasks. This calibration is essential for fair comparison: two adaptation methods evaluated on the same Challenge Tasks might show different performance not because one method is better at adaptation but because the tasks were easier for one base policy's robustness boundary. The paper's approach makes this calibration explicit, allowing readers to assess whether performance improvements represent genuine adaptation or just within-distribution robustness.

The methodology also has implications for how the field should think about "robustness" versus "adaptation." The fact that the base policy was robust to substantial within-distribution variation (novel opaque objects, 5 cm offsets, uniform color changes) but brittle to specific distribution shifts (checkerboard edges, glare, transparency) suggests that robustness and adaptability are distinct capabilities that require different evaluation protocols. A policy can be highly robust without being adaptable (it handles within-distribution variation well but cannot learn when the distribution shifts), or adaptable without being robust (it can learn quickly from new data but requires that data for even minor variations). The thresholding methodology disentangles these by explicitly measuring both: what variations the policy handles without adaptation (robustness), and what variations require new learning (adaptability).

Innovation 4: Demonstrating That Continual Fine-Tuning Does Not Incur Significant Forgetting—and That This Is Surprising

The continual learning experiment in Section V is not just an extension of the single-step fine-tuning results—it's a finding that contradicts a widely-held assumption about neural network adaptation. The default expectation in the continual learning literature is that sequentially fine-tuning on new tasks causes catastrophic forgetting of previously learned capabilities, and that specialized methods (elastic weight consolidation, progressive networks, experience replay) are necessary to prevent this. The paper shows that, at least for this specific setting, sequential fine-tuning without any forgetting mitigation imposes only a 4–7% performance penalty on the final task compared to single-step fine-tuning, and in one case (transparent bottles) actually provides an 8% improvement.

The significance of this finding lies in what it implies about the geometry of the parameter space around the pre-trained solution. If fine-tuning on Task A moved the parameters to a narrow, specialized region that was incompatible with Task B, then initializing from the Task-A-adapted policy for Task B adaptation would produce substantially worse final performance than initializing from the original base policy—the Task A adaptation would have "used up" parameter capacity that Task B needs. The fact that this doesn't happen—that the performance penalty is "within the margin-of-error of our evaluation procedure" (Section V)—suggests that the adaptations for different challenge tasks occupy partially overlapping or compatible regions of parameter space, and that the pre-trained initialization provides a sufficiently broad foundation that multiple task-specific adaptations can coexist without destructive interference.

This has a provocative implication: the common framing of catastrophic forgetting as a fundamental barrier to continual learning may be task-specific and scale-dependent. When the base pre-training is sufficiently broad (580,000 grasps across 1,000 objects) and the target adaptations are modest in scope (learning to handle one specific distribution shift at a time), the pre-trained parameters may provide enough capacity and structure that sequential adaptations don't conflict. Catastrophic forgetting might primarily be a problem when (a) the base training is narrow, (b) the target tasks require mutually incompatible representations, or (c) the adaptation is extensive enough to substantially overwrite the base knowledge. The paper's experimental setup—QT-Opt pre-training, five diverse but related target tasks, modest adaptation data budgets—happens to avoid all three conditions, and the empirical result that forgetting is negligible suggests these conditions may be more common in practical robotics than the continual learning literature assumes.

The paper is appropriately cautious about this interpretation. It acknowledges that the experiment only evaluates performance on the final task in the chain, not on earlier tasks after subsequent fine-tuning steps, and that longer chains might reveal accumulation of forgetting. But the result is nonetheless significant as an existence proof: continual fine-tuning without forgetting mitigation can work, at least over modest time horizons, when the pre-training foundation is strong enough. This shifts the burden of proof: rather than assuming catastrophic forgetting is universal and designing methods to combat it, the field should characterize when forgetting occurs and whether pre-training scale can push that threshold beyond practical deployment horizons.

Innovation 5: Identifying the Offline Stopping-Point Problem as the Critical Bottleneck for Deployment

While the paper's primary empirical results are positive (fine-tuning works, continual learning works), its identification of the offline stopping-point problem in Section VI-B is arguably the most practically important negative finding. The paper shows that with offline fine-tuning, performance follows a characteristic trajectory—rapid improvement followed by catastrophic collapse to worse-than-initial performance—and that without an online evaluation signal, there is no reliable way to know when to stop training. This is not a minor implementation detail; it's a fundamental barrier to deploying offline fine-tuning in autonomous systems.

The significance of this problem becomes clear when considering the deployment scenario the paper envisions. A robot operating "on the job" encounters a new condition (say, harsh lighting after sunset), collects a batch of experience under that condition, and needs to adapt. If it trains too little, it leaves performance on the table. If it trains too much—even slightly beyond the peak—it overfits to the collected batch and its performance collapses below the pre-adaptation baseline, potentially causing failures that damage the robot or its environment. Without a way to detect the peak, the robot is choosing between two failure modes: under-adaptation and catastrophic regression. The 500,000-step budget used in the paper was chosen through offline experimentation (Figure 6) and would not be available to a robot encountering a novel condition for which no validation experiments have been run.

This problem connects to a broader challenge in offline reinforcement learning that the field had not adequately addressed at the time of this work: overfitting in Q-learning with small datasets is qualitatively different from overfitting in supervised learning. In supervised learning, overfitting manifests as a growing gap between training and validation performance, and early stopping on a held-out validation set reliably prevents it. In offline Q-learning, the equivalent would be off-policy evaluation—estimating policy performance from logged data without deployment—but as the paper notes, this is an unsolved problem (Irpan et al., 2019, cited as [26]). The Q-function can achieve low Bellman error while representing a terrible policy, because Bellman error measures temporal-difference consistency, not policy quality. The precipitous drop in Figure 6 likely occurs because the Q-function begins to memorize the specific transitions in the small target dataset, producing highly accurate value estimates for states and actions in the dataset while losing the ability to generalize to nearby states that the policy will actually visit at deployment time.

The paper's framing of this as an open problem, coupled with the dramatic visualization in Figure 6, effectively sets an agenda for follow-up work: offline fine-tuning needs off-policy evaluation as a stopping criterion. Without it, the method is powerful in the laboratory (where validation experiments can determine the stopping point) but unreliable in autonomous deployment. The paper points to this as a key direction for future work (Section VII), and the clarity with which it demonstrates the failure mode makes it a compelling call to action. This is an example of a negative result that advances the field by identifying exactly what must be solved to transition from demonstration to deployment.

5. Experimental Analysis

Evaluation Methodology

• Dataset. All experiments use the real-world grasping setup with the QT-Opt pre-training corpus described in Section III-A: 580,000 offline grasp attempts across ~1,000 visually and physically diverse objects, followed by 28,000 online grasp attempts on the same object corpus. The base policy achieves 96% accuracy on a general set of previously-unseen test objects and 86% on the challenging subset of six test objects used for the main fine-tuning experiments (Section III-A). For each of the five Challenge Tasks plus the base grasping task, a target task dataset of 800 exploration grasps is collected and partitioned into six tiers (25, 50, 100, 200, 400, 800 grasps), yielding 36 individual adaptation datasets (Section IV-B, Step a).

• Base model. The base policy is a Q-function neural network trained with the QT-Opt algorithm (Kalashnikov et al., 2018 [29]), taking monocular RGB images as input and producing Q-value estimates over candidate motor actions for closed-loop grasping. The architecture processes visual input through a CNN with action features injected at an intermediate layer and produces a scalar Q-value through fully-connected layers (Section IV-B, "ImageNet" comparison description and Section VI-C, Figure 7). The policy selects actions at deployment time via the cross-entropy method (CEM) optimizer sampling candidate actions and selecting the one with maximum Q-value. This model is chosen because it represents a state-of-the-art vision-based grasping system trained at scale (608,000 total grasp attempts) on a diverse object corpus, providing a strong foundation for studying adaptation.

• Metrics. The primary metric is grasp success rate (%), measured as the fraction of grasp attempts where the robot successfully lifts an object above a threshold height (binary reward as described in Section III-A). Each policy is evaluated by deploying it on the real 7-DoF Kuka arm and executing 50 or more grasp attempts under the target challenge conditions. The bin contents are shuffled between trials using "a randomly-generated sequence of sweeping movements with the end-effector" to reduce temporal correlation between consecutive trials (Section IV-B, Step d). All success rates in Table II are reported from evaluations of N ≥ 50 grasp attempts.

• Baselines.

  • Base policy (pre-adaptation): The pre-trained QT-Opt policy before any fine-tuning on the target challenge task, evaluated under the challenge conditions (Table I, column "Base Policy ∆"). This measures how much performance degrades due to the distribution shift.
  • Training from scratch ("Scratch"): A Q-function with randomly-initialized parameters, trained using the same 800-grasp target task dataset and the same 500,000 gradient step budget as the fine-tuning experiments. This measures whether the target task can be learned from the small dataset alone without pre-training (Section IV-B, Step c).
  • ImageNet initialization ("ImageNet"): A modified Q-function architecture where the CNN trunk is replaced with a ResNet50 architecture (He et al., 2016 [21]) initialized with weights from supervised ImageNet classification training. Action features are concatenated at the end of the CNN rather than injected in the middle. All parameters, including the ResNet50 layers, are updated during the 500,000 gradient steps. This measures whether supervised visual pre-training provides equivalent adaptation benefits to RL pre-training (Section IV-B, Step c). An additional variant that froze the ImageNet feature layers was also tested but "failed to achieve any non-zero success rate" (Section VI-C).
  • Base grasping task fine-tuning: Fine-tuning on additional data from the original grasping task (not a challenge variation), serving as a control for the effect of continued training itself (Table II, "Baseline Grasping Task" row).

• Generation budget / compute accounting. The unit of data is "exploration grasps"—a single grasp attempt consisting of the robot approaching an object, closing the gripper, and lifting, producing a binary success/failure outcome. The data budget is varied across six tiers: 25, 50, 100, 200, 400, and 800 grasp attempts per challenge task. The base policy was trained on approximately 608,000 total grasps (580,000 offline + 28,000 online), making the 800-grasp maximum budget less than 0.2% of the original data. Training compute is measured in gradient steps: all fine-tuning and comparison experiments use 500,000 gradient steps with a learning rate of $10^{-4}$ (25% of the pre-training learning rate). No additional robot interaction occurs during the offline training phase.

• Cross-validation / statistical protocol. There is no cross-validation in the standard sense—each policy is evaluated once on 50 or more grasp attempts on the target challenge task. To reduce evaluation variance, bin contents are shuffled between trials. The paper acknowledges that the fixed 500,000 gradient step budget "was not necessarily optimal for all of our tasks and datasets" (Section VI-B) and was chosen based on offline experiments with the offset gripper task at 400 exploration grasps (Figure 6). The small deltas between continual and single-step fine-tuning (4–7%) are explicitly characterized as "within the margin-of-error of our evaluation procedure" (Section V), though no formal confidence intervals are reported.

Main Quantitative Results

Base Policy Robustness and Challenge Task Definition

Before evaluating fine-tuning, the paper first establishes that the base policy's robustness breaks down at specific thresholds for each category of distribution shift (Section III-B, Table I). These measurements serve as the pre-adaptation baselines against which fine-tuning improvements are measured:

The largest degradations occur for the visual corruption tasks (harsh lighting: −55%; checkerboard: −36%) and the morphological shift (offset gripper: −39%), while the gripper extension produces a more modest drop (−10%). Transparent bottles produce a −37% degradation, consistent with the visual challenge category. These baseline measurements establish that each Challenge Task represents a genuine distribution shift requiring adaptation—milder versions of each variation (uniform color changes, standard household lights, offsets up to 5 cm) had no effect on base policy performance (Section III-B, per-task descriptions).

Fine-Tuning Performance Across Tasks and Data Budgets

Table II presents the full matrix of results: 5 challenge tasks × 6 data budgets (25, 50, 100, 200, 400, 800 grasps) × 3 methods (fine-tuning from pre-trained, training from scratch with 800 grasps, ImageNet initialization with 800 grasps), plus the baseline grasping task control.

Headline results at the maximum data budget (800 grasps):

• Checkerboard Backing: Fine-tuning reaches 90% (+40 percentage points over the base policy's 50%), essentially recovering performance to the original 86% baseline grasping level and exceeding it. The best-performing intermediate budget is 400 grasps, which also achieves 90%. Training from scratch achieves 0%; ImageNet initialization achieves 0%.
• Harsh Lighting: Fine-tuning reaches 63% (+31 points over the base policy's 32%). This represents substantial recovery but notably does not reach the original base policy performance, indicating that the harsh lighting condition remains challenging even with adaptation. Training from scratch: 4%; ImageNet initialization: 2%.
• Extend Gripper 1 cm: Fine-tuning reaches 93% (+18 points over the base policy's 76%), surpassing the original base policy performance. Notably, 25 exploration grasps already produce substantial improvement (93% at 25 grasps in one measurement, though performance at 25 and 50 grasps is unstable—see discussion below). Training from scratch: 0%; ImageNet initialization: 14%.
• Offset Gripper 10 cm: Fine-tuning reaches 98% (+55 points over the base policy's 47%), the absolute highest success rate of any experiment and a larger absolute improvement than the base policy's original 86%. Training from scratch: 37%; ImageNet initialization: 47%.
• Transparent Bottles: Fine-tuning reaches 66% (+17 points over the base policy's 49%). This is the smallest improvement among the five tasks, reflecting the difficulty of the transparency challenge. Training from scratch: 27%; ImageNet initialization: 20%.
• Baseline Grasping Task (control): Fine-tuning reaches 98% (+12 points over the base policy's already-strong 86%), demonstrating that continued training on additional in-distribution data further improves an already-competent policy. Training from scratch: 0% (since the Q-function is randomly initialized and 800 grasps are insufficient to learn grasping from nothing for a randomly initialized network); ImageNet initialization: 12%.

Data efficiency across tasks (Figure 5):

Figure 5 plots fine-tuning success rate against exploration grasps for three selected tasks: Checkerboard Backing, Offset Gripper 10 cm, and Transparent Bottles. The trends reveal several patterns:

• Offset Gripper 10 cm shows the most dramatic low-data improvement: 73% at 25 grasps (+30 points from 43%), reaching 91% at 400 grasps and plateauing at 98% by 800 grasps. The improvement from 25 to 100 grasps is essentially flat (73% → 50% → 60% at 25, 50, 100 grasps respectively, based on Table II), reflecting the instability noted by the authors.
• Checkerboard Backing shows strong but unstable low-data performance: 67% at 25 grasps, dropping to 48% at 50 grasps, then recovering to 71% at 100 grasps, and reaching 90% by 400 grasps. The non-monotonic pattern at low data budgets illustrates the variance issue discussed in Section VI-A: "these performance improvements in the very low-data regime (e.g. ≤200 grasp attempts) are also unstable."
• Transparent Bottles shows the most gradual improvement and the lowest ceiling: 46% at 25 grasps (actually below the base policy's 49%—an instance of fine-tuning potentially degrading performance in the very low-data regime), rising to 65–66% at 100–800 grasps. The small absolute improvement (+17 points at best) despite 800 grasps indicates that transparency is fundamentally harder to adapt to than visual corruption or morphological changes.

Comparison to baselines at equivalent data (800 grasps):

• Training from scratch fails almost uniformly: Only harsh lighting (4%) and transparent bottles (27%) achieve non-zero success. The 0% scores on checkerboard backing, extend gripper, offset gripper, and baseline grasping confirm that 800 grasps are fundamentally insufficient to learn grasping from random initialization. The 27% on transparent bottles is the highest scratch-training result, likely because the transparent bottles' visual properties make the task hard regardless of initialization, and the scratch-trained policy achieves a modest level of performance through random grasping strategies that occasionally succeed.
• ImageNet initialization fails with one partial exception: The best ImageNet result is 47% on offset gripper 10 cm (only 4 points above the base policy's pre-adaptation 43%), with 0% on checkerboard, harsh lighting, and baseline grasping. The offset gripper result is notable as the only case where ImageNet initialization meaningfully exceeds the base policy's pre-adaptation performance, but at 47% it is less than half the 98% achieved by fine-tuning at the same data budget. The paper notes that the frozen-features ImageNet variant achieved 0% across all tasks (Section VI-C), demonstrating that the visual features must be updated during adaptation, not just re-weighted.

Data quantity trends (across the 25–800 grasp range, Table II):

The relationship between exploration data and performance is broadly positive but non-linear:

• For Extend Gripper 1 cm, performance is already high (93%) at 25 grasps, suggesting the required adaptation is minor and a handful of examples suffice—consistent with the modest 10-percentage-point base policy degradation.
• For Offset Gripper 10 cm, performance rises quickly to 73% at 25 grasps, plateaus around 50–60% for 50–200 grasps, then jumps to 91% at 400 grasps and 98% at 800 grasps. This suggests two phases: rapid correction of gross errors with a few examples, then slower refinement requiring more data.
• For Checkerboard Backing, performance is unstable below 400 grasps (oscillating between 48% and 71%) but reaches a stable 90% at 400–800 grasps.
• For Harsh Lighting, performance shows high variance at low budgets (23%, 16%, 52% at 25, 50, 100 grasps) and reaches 63% at 800 grasps, the lowest ceiling among all tasks. This suggests that some aspects of the harsh lighting corruption cannot be fully overcome with 800 grasps of fine-tuning.
• For Transparent Bottles, performance hovers in the 43–66% range without a clear monotonic trend, indicating that transparency adaptation requires more than 800 examples to reliably improve.

Continual Learning Results

Table III presents the results of sequentially fine-tuning across five challenge tasks, compared to base policy performance and single-step fine-tuning at the 800-grasp budget:

The first task in the sequence (Harsh Lighting) is identical to the single-step experiment by construction. For the remaining tasks, the continual fine-tuning performance is within 4–7 percentage points of the single-step variants (or better, in the case of Transparent Bottles at +8%). The authors characterize these deltas as "within the margin-of-error of our evaluation procedure" (Section V), concluding that "continual fine-tuning does not impose a drastic performance penalty compared to single-step fine-tuning." The Transparent Bottles result (+8% over single-step) is notable: the intermediate fine-tuning on Harsh Lighting appears to have actually improved the policy's ability to subsequently adapt to transparent objects, suggesting positive transfer between the visual corruption challenge and the transparency challenge.

Critically, this experiment only evaluates the final task in the chain—it does not measure whether fine-tuning on Task N+1 degrades performance on Task N. The paper acknowledges this limitation: "measuring the how continual fine-tuning updates for new tasks affects the performance of previously-seen tasks" is listed as future work (Section VII).

Offline Overfitting and the Stopping-Point Problem

Figure 6 presents an experiment specifically designed to characterize the overfitting phenomenon during offline fine-tuning. For the Offset Gripper 10 cm task with 400 exploration grasps, the real-robot success rate is measured at different numbers of gradient steps:

• At $10^6$ steps, performance is approximately 85% (read from Figure 6 blue curve).
• At $3 \times 10^5$ steps, performance reaches approximately 88%.
• At $5 \times 10^5$ steps (the chosen budget for the main experiments), performance peaks at approximately 90–92%.
• At $7 \times 10^5$ steps, performance has dropped to approximately 65%.
• At $3 \times 10^6$ steps, performance collapses to approximately 20%, well below the base policy's 43% (yellow dotted line) and even below the scratch-training policy's performance with 800 grasps (green dotted line, approximately 37%).

The curve shows a characteristic shape: rapid improvement from the base policy baseline, a peak at around 500,000 steps, then a "precipitous" drop that "never recovers, dropping below even the initial performance of the base policy from which it was trained" (Section VI-B). The paper attributes this to "the initialization being overwritten by overfitting to the target samples." The fixed 500,000-step budget for the main experiments was chosen based on this curve and similar experiments, but the paper explicitly acknowledges that this choice was "not necessarily optimal for all of our tasks and datasets" (Section VI-B), and that the variance in the main results (Table II) partly reflects suboptimal stopping points for some task/data-size combinations.

Ablation Studies and Robustness Checks

Full parameter initialization vs. selective re-initialization (adding a new head): In a simulated grasping experiment (Appendix B, Figure 8), the paper compares online fine-tuning with full parameter initialization against fine-tuning where the final layers (the "head") are re-initialized randomly. The base task is grasping opaque blocks (base policy achieves 98% trained from scratch on 43,000 grasps) and the target task is grasping semi-transparent blocks. Both policies start at approximately 15% success. The full-initialization policy reaches base-policy performance (98%) after approximately 5,000 exploration grasps, corresponding to approximately 12% of the base training data. The new-head policy barely reaches 30% after the same 5,000 grasps. This demonstrates that selective re-initialization is "unsuitable for sample-efficient fine-tuning" because it destroys knowledge that the pre-training encoded and forces re-learning from limited data. The paper's interpretation emphasizes that online fine-tuning compounds this problem: the degraded exploration policy resulting from partial re-initialization fails to collect informative data, creating a feedback loop of poor exploration → poor data → poor updates (Appendix B).

Data mixing ratio (base task vs. target task data): Appendix C, Figure 9 studies the effect of varying the fraction of target task data in the combined replay buffer during online fine-tuning in simulation. Target data fractions of 1%, 24%, 44%, 71%, and 99% were tested. The results show a "predictable relationship with sample efficiency: higher shares of target task data allow the fine-tuning policy to achieve higher performance faster." The 99% target data curve reaches approximately 95% success fastest, while the 1% curve barely improves above the base policy. For the real-robot experiments, the paper chose equal-probability sampling (50% base, 50% target), which represents a conservative choice that prioritizes retention of base-task knowledge over maximum adaptation speed. The equal-probability choice was not compared against other ratios in the real-robot setting.

Frozen ImageNet features vs. fine-tuned ImageNet features: The paper tested two ImageNet-based variants. The fine-tuned variant (all ResNet50 parameters updated during training) produced the results in Table II: best performance of 47% on offset gripper, 0% on half the tasks. The frozen-features variant (ImageNet layers held constant) "failed to achieve any non-zero success rate" across all tasks (Section VI-C). This ablation demonstrates that even when starting from powerful supervised visual features, the task of learning visuomotor control from limited data requires updating the visual feature hierarchy itself—not just re-weighting existing features through later layers. The paper interprets this as evidence that "adapting end-to-end robot learning to new sensorimotor tasks may require changing the features used to represent the problem, and not just the post-processing of said features" (Section VI-C).

Parameter change analysis across challenge tasks: Figure 7 visualizes the cosine distance between pre-trained and fine-tuned parameters for each layer of the Q-function network, broken out by challenge task. The bar heights are normalized by the magnitude of parameter changes induced by fine-tuning on the baseline grasping task (i.e., continued training on in-distribution data). Key findings from this analysis (Section VI-C):

• Visually-dominated challenges (Checkerboard Backing, Harsh Lighting) induce large changes in the convolutional layers, as expected.
• The purely morphological Offset Gripper 10 cm task induces "substantial changes to the network's image-processing parameters (e.g. layers conv2–conv7)," demonstrating that morphological shifts require restructuring visual feature computation due to the changed hand-eye coordination mapping.
• The Transparent Bottles task induces relatively smaller parameter changes compared to the visual corruption tasks, consistent with its smaller performance improvement (+17 points vs. +40 for checkerboard).

Online vs. offline fine-tuning: While not tested as a direct A/B comparison in the real-robot experiments, the paper provides simulation results (Appendix B-C) and qualitative arguments for preferring offline fine-tuning. The simulation experiments demonstrate that online fine-tuning's requirement to maintain competent exploration at all times creates a brittleness: any design choice that temporarily degrades the policy (like selective re-initialization) risks a performance collapse from which the policy cannot recover with limited data. The paper also argues that offline fine-tuning is "more practical" because it decouples data collection from training, eliminating the need for continuous robot access during the gradient update phase (Appendix C).

Critical Assessment

Claim: "Fine-tuning via off-policy reinforcement learning leads to substantial performance gains using less than 0.2% of the data necessary to learn the task from scratch."

What the experiments demonstrate. Table II shows that fine-tuning with 800 exploration grasps (0.13% of the 608,000 total pre-training grasps) achieves performance improvements of +17 to +55 percentage points across the five challenge tasks, reaching success rates of 63–98%. The 0.2% figure is validated: 800 / 608,000 ≈ 0.13%. However, the claim that this is "less than 0.2% of the data necessary to learn the task from scratch" is supported by the scratch-training baselines only indirectly—the scratch-trained policies achieve 0% on four of six tasks, meaning the amount of data "necessary to learn from scratch" is unknown (it could be 2,000 grasps, or 10,000, or 50,000). The 4% and 27% scratch-training results on harsh lighting and transparent bottles suggest those tasks require less data from scratch for modest performance, but the comparison is not tight enough to quantify "how much data from scratch would achieve equivalent performance."

What remains untested. The claim implicitly assumes that the 800-grasp fine-tuned policy and an equivalent-performance scratch-trained policy would differ only in data quantity, but the paper does not establish the data-scaling curve for training from scratch on these specific tasks. The 27% scratch result on transparent bottles suggests that for some tasks, modest performance is achievable from scratch with 800 grasps, and the claimed 0.2% ratio might be substantially different if measured at a performance-matched rather than data-matched comparison point. Additionally, the claim does not account for the fact that the fine-tuning procedure uses 500,000 gradient steps on a combined replay buffer including all base data—the total data seen during fine-tuning is actually ~608,800 transitions, not 800.

Claim: "Pre-training via RL is essential: training from scratch or adapting from supervised ImageNet features are both unsuccessful with such small amounts of data."

What the experiments demonstrate. This claim is robustly supported for the specific data budget (800 grasps) and the specific tasks tested. Scratch training achieves 0% on four of six tasks. ImageNet initialization achieves 0% on three of six tasks and a maximum of 47% on offset gripper (only 4 points above the base policy's pre-adaptation performance, and less than half the 98% achieved by fine-tuning). The frozen-ImageNet-features variant achieves 0% across all tasks. These results clearly establish that neither random initialization nor supervised visual pre-training provides an adequate starting point for learning grasping from 800 examples.

Caveats and missing comparisons. The ImageNet comparison uses a modified architecture (ResNet50 with action concatenated at the end of the CNN rather than injected in the middle), which conflates the effect of the pre-training data (ImageNet classification) with the effect of the architecture (action features processed after visual features rather than interleaved). A fairer comparison would have used the identical QT-Opt architecture initialized with ImageNet-trained convolutional filters of equivalent depth, or pre-trained the identical architecture on a supervised task. The paper's claim that RL pre-training is "essential" could be more precisely stated as "QT-Opt pre-training with interleaved perception-action processing is essential"—the architectural choice and the training objective are confounded. Additionally, no comparison is made to other forms of pre-training that might encode sensorimotor knowledge without RL, such as learning from human demonstration videos, self-supervised interaction, or predictive models. The claim is about RL specifically, but the experiments only rule out ImageNet; other pre-training modalities remain untested.

Claim: "Simple fine-tuning can adapt to substantial task, robot, and environment variations which were not present in the original training distribution (i.e., off-distribution)."

What the experiments demonstrate. The five Challenge Tasks span four distinct categories of distribution shift: visual appearance (checkerboard, lighting), robot embodiment (gripper extension, lateral offset), and object properties (transparency). The base policy experienced "absolutely no variation in robot morphology" during pre-training (Section III-B, offset gripper description), and transparent bottles were "not present in the training set" (Section III-B). The large performance drops in Table I (32–50% on visually-challenged tasks, 47% on morphology, 49% on transparent objects) confirm that these are genuine distribution shifts, and the fine-tuning improvements in Table II (achieving 63–98%) demonstrate successful adaptation. The claim of "off-distribution" adaptation is further supported by the thresholding methodology: milder variations within each category (uniform color changes, standard lights, offsets up to 5 cm, novel opaque objects) did not degrade performance, meaning the Challenge Tasks are specifically at and beyond the base policy's robustness boundary.

Caveats. The claim covers "substantial" variations, but the paper does not characterize the magnitude of the distribution shift in any quantitative metric (e.g., KL divergence between base and target observation distributions, embedding distance between base and target object representations). The subjective assessment of "substantial" is supported by the performance drops but not by an independent measure of distribution shift magnitude. Furthermore, the five tasks, while diverse, are all variations of the same underlying skill (grasping objects from a bin). Whether the findings generalize to adapting between fundamentally different skills (e.g., from grasping to pushing, or from bin-picking to assembly) is not tested. The claim is specific to within-skill adaptation, though the paper's language ("substantial task, robot, and environment variations") implies broader applicability.

Claim: "Continual fine-tuning imposes very little performance penalty compared to single-step fine-tuning."

What the experiments demonstrate. Table III shows that sequential fine-tuning across five tasks achieves final-task performance within 4–7 percentage points of single-step fine-tuning, with the transparent bottles intermediate result exceeding single-step by 8 points. The 91% final accuracy on offset gripper in the continual chain is only 7 points below the 98% single-step result, both far above the 43% base policy baseline. This is strong evidence that, in this specific setup, sequential adaptation does not catastrophically degrade the ability to learn new tasks.

Critical missing evaluation: backward transfer. The experiment only measures forward transfer—how well the policy learns later tasks after earlier adaptations—but does not measure backward transfer—whether the policy retains performance on earlier tasks after later adaptations. This is a fundamental gap: the central challenge of continual learning is catastrophic forgetting of old tasks, not degraded ability to learn new ones. The paper acknowledges this explicitly ("measuring how continual fine-tuning updates for new tasks affects the performance of previously-seen tasks" is future work, Section VII), but the claim that continual fine-tuning "imposes very little performance penalty" is misleading without this measurement. A policy that achieves 91% on offset gripper but has forgotten how to grasp under harsh lighting (dropping from 63% back to 32%) would not represent successful continual learning. The paper's conclusion that "this method can perform continual adaptation, and may serve as the basis for a continual end-to-end robot learning method" (Section V) is stated more cautiously than the claim, but the forward-transfer-only evaluation means the suitability for continual learning is only partially demonstrated.

Additional concerns about the continual learning experiment. The sequence of five tasks is fixed, and no experiments test alternative orderings. It is possible that the specific ordering (Harsh Lighting → Transparent Bottles → Checkerboard → Extend Gripper → Offset Gripper) is particularly favorable, and that other orderings would reveal more significant interference. The experiment also uses 800 grasps per task (the maximum budget), which may mask gradual accumulation of interference that would be visible at lower per-task budgets. The paper does not report the computational cost of maintaining and sampling from the combined replay buffer across five fine-tuning stages—whether the replay buffer accumulates data from all previous tasks or is reset between stages is not specified.

General methodological strengths and weaknesses

Strengths:

• Real-robot scale: 15,000+ grasp attempts over 14 robot-days is a substantial real-world evaluation by the standards of robot learning research.
• Systematic data budget variation: The combinatorial design (6 data sizes × 5 tasks) provides a much richer picture than a single-data-size comparison would, revealing non-monotonicities and instability in the low-data regime.
• Thresholding methodology for task selection: Calibrating Challenge Tasks to the base policy's robustness boundary is a principled approach that should be adopted more widely.
• Transparency about failures and limitations: The paper explicitly discusses the offline stopping-point problem (Figure 6), the instability of low-data fine-tuning (Section VI-A), and the negative result with frozen ImageNet features (Section VI-C). It also reports the failure of the ReST-like revision model in the prior sections.

Weaknesses:

• Single model family, single task domain: All results are for grasping with QT-Opt. The paper does not test whether the findings hold for other RL algorithms (e.g., policy gradient methods), other robot morphologies (e.g., multi-fingered hands, mobile manipulators), or other task types (e.g., pushing, insertion, assembly). The claim that the approach works across "a broad range of possible variations" (Section IV-B) is supported only for grasping variations.
• No formal statistical treatment: Evaluation uses 50+ grasp attempts per policy, but no confidence intervals, error bars, or significance tests are reported. The claim that 4–7% differences are "within the margin-of-error" is asserted without quantification.
• Confounded comparison with ImageNet baseline: Architecture differences between QT-Opt (action injected mid-CNN) and ImageNet baseline (action concatenated after ResNet50) prevent clean isolation of the pre-training data effect from the architectural effect.
• Fixed hyperparameter choices: The learning rate (25% of pre-training), gradient step budget (500,000), and data mixing ratio (50/50) are fixed across all tasks and data sizes. The paper acknowledges these are suboptimal for some conditions, but does not explore how sensitive results are to these choices. If fine-tuning performance varies substantially with these hyperparameters, the reported results may understate or overstate what simple fine-tuning can achieve with better tuning.
• No evaluation of forgetting in continual learning: As discussed above, this is the most significant gap in the experimental design relative to the paper's stated goals.
• Difficulty estimation cost unaccounted for in related sections: While not directly applicable to this experimental section, the paper does not address how a robot deployed "on the job" would determine which Challenge Task it is experiencing, in order to know when to trigger fine-tuning. The experiments assume the distribution shift is known and a labeled dataset of the new condition is collected.

Missing experiments that would strengthen the paper

• Backward transfer evaluation in continual learning: Measure performance on each earlier task after each subsequent fine-tuning step, to quantify catastrophic forgetting.
• Alternative task orderings in continual learning: Test whether the sequence matters.
• Architecture-matched ImageNet comparison: Re-run the ImageNet experiment with the identical QT-Opt architecture (action injection mid-CNN) but with ImageNet-initialized convolutional layers.
• Data scaling curve for training from scratch: Train policies from scratch with progressively larger datasets (800, 2,000, 5,000, 10,000, 50,000 grasps) on each Challenge Task to establish how much data from-scratch training actually requires to match fine-tuning performance, providing a direct measurement of the "200× data reduction" claim.
• Online vs. offline fine-tuning direct comparison: A real-robot head-to-head comparison of online and offline fine-tuning at matched data budgets would strengthen the paper's stated preference for offline methods.
• Sensitivity to pre-training scale: Fine-tune from intermediate pre-training checkpoints (e.g., after 100,000 grasps, after 300,000 grasps, after the full 608,000) to characterize how fine-tuning efficacy depends on pre-training quantity, which directly tests the paper's implicit claim that pre-training scale is the key enabler.
• Cross-task generalization of fine-tuned policies: Test whether fine-tuning on checkerboard backing improves robustness to harsh lighting (or vice versa), to characterize whether fine-tuning produces narrow, task-specific corrections or broader robustness improvements.

6. Limitations and Trade-offs

6.1 No Measurement of Catastrophic Forgetting in Continual Learning

The assumption or constraint. The continual learning experiment in Section V assumes that sequential fine-tuning is viable because performance on the final task does not substantially degrade compared to single-step fine-tuning. However, the experiment only measures forward transfer—whether intermediate adaptations impair the ability to learn subsequent tasks. It never evaluates backward transfer: whether fine-tuning on Task N+1 destroys performance on Task N, which is the classical catastrophic forgetting problem that defines the continual learning challenge. The paper explicitly acknowledges this gap:

"We would also like to further assess our method's suitability for continual adaptation... measuring how continual fine-tuning updates for new tasks affects the performance of previously-seen tasks." (Section VII)

The consequence. A robot deployed with this method might successfully adapt to a new condition (say, transparent bottles after harsh lighting) but silently lose its adaptation to earlier conditions (the harsh lighting capability degrades back toward the base policy's 32%). The continual learning experiment in Table III shows that the policy achieves 91% on offset gripper at the end of the chain, but provides no information about whether it still achieves 63% under harsh lighting, 74% on transparent bottles, 86% on checkerboard backing, or 88% with the extended gripper. If catastrophic forgetting is occurring, the policy is not actually "continually learning"—it is sequentially specializing, with each new adaptation partially overwriting previous ones. The paper's claim that "this simple adaptation scheme provides a promising solution for creating a lifelong learning robotic agent" (Section I) is unsupported without this measurement.

What evidence exists in the paper. The only evidence that forgetting might not be catastrophic is indirect: the small (4–7%) performance penalty on forward transfer suggests the parameter space accommodates multiple adaptations without destructive interference. But this is weak evidence—a policy could easily maintain the structural knowledge needed to learn new tasks while simultaneously overwriting the task-specific corrections for old tasks. The parameter change analysis in Figure 7 shows that different challenge tasks induce changes across overlapping parameter groups (all tasks affect conv2–conv7, for example), which makes interference plausible. The paper provides no measurement of whether the weight changes for Task N+1 undo the weight changes that produced the improved performance on Task N.

Mitigation status. Not addressed experimentally. Acknowledged as future work in Section VII. The continual learning experiment as designed tests only one half of the continual learning problem (accommodation of new tasks) while leaving the other half (retention of old tasks) completely unmeasured. A practitioner deploying this method in a sequential adaptation setting would need to implement their own backward-transfer evaluation or risk silent capability regression on earlier conditions.

---

6.2 The Offline Stopping-Point Problem Prevents Autonomous Deployment

The assumption or constraint. The paper's offline fine-tuning procedure assumes that the number of gradient steps can be determined in advance. Section IV-A specifies 500,000 gradient steps for all experiments, and Section VI-B reveals that this number was chosen based on offline validation experiments (Figure 6) that measured real-robot performance at multiple checkpoints during fine-tuning on a specific task (Offset Gripper 10 cm with 400 exploration grasps). The paper is transparent that this choice is not universally optimal:

"The point at which overfitting begins is a function of the initialized model, target dataset, learning algorithm, and many other factors, and is not necessarily stable or easily predictable." (Section VI-B)

The consequence. An autonomous robot encountering a novel condition cannot run the equivalent of Figure 6 to determine when to stop training—doing so would require repeatedly deploying intermediate checkpoints to the real robot and measuring success, which defeats the purpose of offline fine-tuning (avoiding robot-in-the-loop training). Without a stopping criterion, the robot faces a binary risk: stop too early and leave performance on the table, or stop too late and suffer catastrophic regression. Figure 6 shows that continuing past the peak causes performance to collapse to 20%, well below the base policy's 43% and even below a scratch-trained policy. In a deployment scenario, this regression could cause the robot to fail at its primary task—potentially damaging objects, the environment, or itself—with no warning signal that the policy has degraded. The fixed 500,000-step budget used in the paper was determined through costly real-robot experimentation that would need to be repeated for every new condition, new data budget, and new task type.

What evidence exists in the paper. Figure 6 provides direct evidence of the failure mode. For the Offset Gripper 10 cm task with 400 exploration grasps, performance peaks around 90–92% at 500,000 steps, drops to approximately 65% by 700,000 steps, and collapses to approximately 20% by 3 million steps—below the base policy and scratch-training baselines. The paper reports that this shape (rapid improvement, peak, precipitous drop) is characteristic, and that the optimal stopping point is not generalizable: the fixed 500,000-step budget "was not necessarily optimal for all of our tasks and datasets" (Section VI-B), and the variance in Table II results partly reflects this suboptimality.

Mitigation status. The paper identifies off-policy evaluation (Irpan et al., 2019 [26]) as a potential solution but does not implement or test it. The proposal is that an off-policy estimator could monitor the policy's predicted performance during training and signal when improvement stops, enabling early stopping without real-robot evaluation. However, the paper does not demonstrate that existing off-policy evaluation methods are reliable enough for this purpose—off-policy evaluation in the low-data regime where overfitting is most severe is itself an unsolved problem. Until a reliable stopping criterion is developed, offline fine-tuning remains a laboratory technique requiring human-supervised validation, not an autonomous adaptation capability.

---

6.3 Difficulty Estimation Cost Is Not Accounted for in the Adaptation Framework

The assumption or constraint. The fine-tuning procedure assumes that the robot knows when to trigger adaptation and what condition it needs to adapt to—that it can detect a distribution shift, identify that performance has degraded, and collect a labeled dataset of the new condition. In the paper's experiments, these are provided by the experimenters: the Challenge Tasks are deliberately constructed with major environmental changes (checkerboard glued down, halogen light positioned, transparent bottles introduced, gripper physically modified), and the exploration dataset is collected under these known modified conditions. In a deployed setting, the robot would need to autonomously distinguish between transient failures (a difficult object configuration that the base policy happens to fail on) and persistent distribution shifts that require adaptation, and then decide whether to allocate its limited onboard compute to fine-tuning.

The paper acknowledges the related difficulty estimation problem in the context of the prior work sections, but does not address it for its own method. The thresholding approach for selecting Challenge Tasks (Section III-B) demonstrates that the experimenters systematically determined the failure boundary—but this required repeatedly testing the base policy under progressively severe variations, which itself consumes substantial robot time and is not a procedure the robot could perform autonomously.

The consequence. A robot using this method in practice would need a separate change detection and adaptation triggering mechanism that is entirely unaddressed by the paper. Without such a mechanism, the robot faces two failure modes: (1) it fails to adapt when it should, continuing to use a degraded policy under persistently changed conditions, or (2) it attempts to adapt when it shouldn't, fine-tuning on a small dataset of failures caused by random bad luck rather than genuine distribution shift, and overfitting to noise. The second failure mode is particularly dangerous in combination with the stopping-point problem (Section VI-B): fine-tuning on a dataset that doesn't represent a genuine new condition could produce a policy that overfits to the specific failure cases and performs worse than the original base policy.

What evidence exists in the paper. The paper provides no experiments on autonomous shift detection or adaptation triggering. The exploration dataset collection procedure (Section IV-B, Step a) is entirely human-directed: experimenters set up the modified condition, run the base policy for 800 grasps, and label each grasp as success or failure using the automated reward signal (object lift height). The 0.2%-of-original-data efficiency claim accounts only for the exploration data collected after the shift has been identified and the adaptation decision has been made—it does not account for the cost of detecting that adaptation is needed or verifying that the condition has genuinely changed.

Mitigation status. Not addressed. The paper does not propose a mechanism for autonomous shift detection, does not estimate how many grasp attempts would be needed to reliably distinguish a persistent distribution shift from sampling noise, and does not include any detection-related costs in its data efficiency calculations. A practitioner deploying this method would need to implement their own monitoring infrastructure—likely involving statistical tests on the robot's recent success rate, which themselves consume grasp attempts that should be counted against the adaptation budget.

---

6.4 The ImageNet Baseline Architecture Is Not Fairly Compared

The assumption or constraint. Section IV-B's ImageNet comparison is used as the primary evidence for the paper's central claim that "pre-training via RL is essential" (Abstract) and that supervised visual pre-training cannot substitute for RL-based pre-training. However, the ImageNet baseline uses a different network architecture than the QT-Opt base policy: the CNN trunk is replaced with a ResNet50 architecture, and—critically—the action features are concatenated at the end of the CNN rather than injected at an intermediate layer, as in the original QT-Opt architecture. The paper describes this modification directly:

"We initialize the remaining fully-connected layers with random parameters, and concatenate the action input features at the end of the CNN (rather than the adding them in middle of the CNN, as in the original architecture)." (Section IV-B)

The consequence. This design confounds two variables: (1) the source of pre-training (RL vs. supervised ImageNet classification) and (2) the architecture's capacity for integrated perception-action processing. The paper's own Figure 7 analysis demonstrates that fine-tuning induces changes in early and mid-level convolutional layers even for purely morphological shifts, and the paper argues (Section VI-C) that this reflects the need for hand-eye coordination—visual features computed differently depending on the action being considered. The QT-Opt architecture enables this through mid-network action injection; the ResNet50 architecture with end-of-network action concatenation processes all visual features independently of the action, then attempts to combine them afterward. If action-conditioned visual processing is essential for sample-efficient visuomotor learning—as the parameter change analysis suggests—then the ImageNet baseline's failure may be attributable to the architectural choice rather than the pre-training data source.

A fair comparison would have tested the identical QT-Opt architecture initialized with ImageNet-trained convolutional filters, or equivalently, tested the ResNet50-mid-injection architecture. The paper can therefore claim that "QT-Opt pre-training + QT-Opt architecture outperforms ImageNet pre-training + ResNet50-end-concatenation architecture," but cannot cleanly attribute this to the pre-training objective alone.

What evidence exists in the paper. The paper reports that a frozen-ImageNet-features variant (where the ResNet50 ImageNet layers were held constant during training) "failed to achieve any non-zero success rate" across all tasks (Section VI-C). This demonstrates that pure ImageNet features without any adaptation are insufficient. However, the fine-tuned ImageNet variant (where all ResNet50 parameters were updated) also achieves poor results (0% on three tasks, max 47% on offset gripper), and this result is the one used to support the "RL pre-training is essential" claim. The paper's interpretation is that "adapting end-to-end robot learning to new sensorimotor tasks may require changing the features used to represent the problem, and not just the post-processing of said features" (Section VI-C). This is a claim about the necessity of feature adaptation, not about the necessity of RL pre-training specifically—supervised pre-training might be sufficient if combined with an architecture that permits the same degree of feature restructuring, but this hypothesis is not tested.

Mitigation status. Not addressed. The paper does not acknowledge the architectural confound, does not run an architecture-matched ImageNet baseline, and does not discuss alternative explanations for the ImageNet baseline's failure. A practitioner evaluating whether to invest in RL pre-training vs. supervised pre-training would need to run this controlled comparison themselves.

---

6.5 Results Are Specific to a Single Task, Robot, and Pre-Training Regime

The assumption or constraint. All experiments use the identical underlying skill—grasping objects from a bin—on a single robot platform (7-DoF Kuka arm with a parallel gripper) with a single RL algorithm (QT-Opt) and a single pre-training scale (580,000 offline + 28,000 online grasps). The paper claims that its findings demonstrate adaptation "to a broad range of possible variations" (Section IV-B) and frame the approach as "a promising solution for creating a lifelong learning robotic agent" (Section I), implying generality beyond the specific experimental configuration.

The consequence. Several aspects of the findings may not transfer to other settings:

• The pre-training scale threshold is unknown. The paper's core implicit argument is that deep RL pre-training at sufficient scale makes simple fine-tuning effective. But it provides no evidence about what "sufficient scale" means—would fine-tuning work after 50,000 grasps? 100,000? 300,000? Without a pre-training scaling curve, a practitioner cannot estimate how much base training is needed before this approach becomes viable for their domain.

• Grasping may be unusually amenable to this approach. Grasping is an episodic, single-step task with binary success feedback—the Q-function needs only to predict a single value per state-action pair, with no multi-step credit assignment. Tasks requiring extended temporal reasoning (e.g., assembly sequences, multi-step tool use, navigation) would face more complex Bellman backups where errors compound across timesteps, and the small target datasets used here (25–800 episodes) may be insufficient for adapting temporally-extended behaviors.

• QT-Opt's specific properties may matter. QT-Opt uses clipped double Q-learning to reduce overestimation bias, a CEM optimizer for continuous action selection, and an architecture that injects action features mid-CNN. Other off-policy algorithms (SAC, TD3, DDPG) or policy gradient methods (PPO, TRPO) might exhibit different fine-tuning dynamics—particularly with respect to the overfitting phenomenon in Figure 6, where the Bellman error objective may be more or less susceptible to memorization depending on the specific algorithm.

• The 86% base policy success rate provides a favorable starting point. The challenge tasks are constructed such that the base policy is competent but brittle—it achieves 32–76% pre-adaptation performance, providing a strong initialization that is close to a good target-task solution. If the base policy were substantially worse (e.g., 20% pre-adaptation), the target dataset might contain too few successful examples for the Q-function to learn from, since QT-Opt relies on Bellman backups from successful outcomes to propagate value information.

What evidence exists in the paper. The paper provides no experiments varying the pre-training scale, the task family, the robot platform, or the RL algorithm. The related work (Section II) cites a range of robotic RL tasks (locomotion, pushing, peg insertion, throwing) but the paper's own experiments are exclusively grasping. The simulation experiments in Appendix B and C use a grasping variant (opaque blocks to semi-transparent blocks), maintaining the same task structure. The paper does not discuss which properties of the experimental setup are likely to generalize and which are specific.

Mitigation status. Not addressed experimentally. The paper's claims use broad language ("a broad range of possible variations," "a promising solution for creating a lifelong learning robotic agent") that implies generality, but the evidence is restricted to the specific configuration tested. A practitioner in a different domain (e.g., mobile manipulation, dexterous in-hand manipulation, assembly) would need to replicate the core findings—including the threshold for pre-training sufficiency and the offline stopping-point behavior—before adopting this approach.

---

6.6 The Method Offers No Path Forward for Tasks Entirely Outside the Base Policy's Capability

The assumption or constraint. The fine-tuning approach works by correcting specific failure modes in an otherwise-competent policy—the base policy already achieves non-trivial performance on all challenge tasks (32–76% pre-adaptation), and fine-tuning improves this to 63–98%. The paper does not test cases where the base policy's performance is near zero on a new task or condition, because the thresholding methodology (Section III-B) explicitly selected challenge variants where the base policy's performance degrades substantially but not to zero. The method thus assumes that the base policy provides a "foothold" of competence on the target condition—some non-zero success rate that provides positive examples in the exploration dataset from which the Q-function can learn.

The paper acknowledges this boundary implicitly in its framing of the challenge tasks:

"Once we find a modification that is sufficiently-severe to compromise the base policy's performance in each category, we use it to define a 'Challenge Task'" (Section III-B)

The emphasis is on finding modifications that "compromise" performance (reduce it from 86% to 32–76%), not modifications that eliminate it entirely. What happens with modifications that are so severe that the base policy achieves 0% is untested.

The consequence. If a robot encounters a condition where its base policy completely fails—achieving 0% success and thus providing no positive reward signal in the exploration dataset—offline fine-tuning has no mechanism to improve. The Q-learning objective relies on Bellman backups from successful outcomes; with zero successes in the target dataset, all transitions have reward 0, and the Bellman target reduces to zero for all state-action pairs. The fine-tuning update would simply drive all Q-values toward zero, destroying the pre-trained Q-function's knowledge without replacing it with anything useful. This is a fundamentally different failure mode from the overfitting shown in Figure 6—it is not a problem of training too long, but of the target dataset containing insufficient signal for any amount of training to help.

This limitation is particularly relevant for the paper's "lifelong learning" framing. In a genuine lifelong deployment, a robot might encounter conditions that are not just "challenging for the base policy" but "impossible for the base policy"—for instance, a gripper modification so extreme that every grasp attempt fails, or an object type so different that the base policy never even makes contact. The current method provides no mechanism for acquiring competence from zero in these cases; it can only refine existing competence.

What evidence exists in the paper. The paper provides no experiments where base policy performance is at or near 0% on the target condition. The lowest pre-adaptation performance is 32% (Harsh Lighting). The "training from scratch" baseline in Table II demonstrates that 800 grasps are insufficient to learn grasping from zero—policies achieve 0% on four of six tasks—but this is a different question from whether fine-tuning from a pre-trained policy could bootstrap from 0% target-task success. The paper's model of adaptation is one of correction (fixing specific flawed heuristics) rather than acquisition (learning entirely new capabilities), and it does not test the boundary where correction fails because there are no correct examples to learn from.

Mitigation status. Not addressed. The paper does not discuss this limitation, does not characterize the minimum base-policy performance needed for fine-tuning to succeed, and does not propose mechanisms for handling cases where the base policy is completely ineffective on the new condition. A practitioner deploying this method would need to separately assess whether new conditions degrade the base policy to non-zero levels before triggering fine-tuning, and would need a fallback strategy (e.g., re-training from scratch with human supervision, or reverting to teleoperation) for cases where the degradation is total.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the conversation around robotic adaptation from algorithm design to pre-training scale as the key enabler. The dominant assumption in the robot learning community—implicit in the proliferation of meta-learning, sim-to-real transfer, and domain adaptation methods—had been that adapting sensorimotor skills to new conditions requires specialized machinery: learned initializations close to task-specific solutions, explicit domain randomization, or structured representations that factor out environmental variation. This paper challenges that assumption through a simple but powerful demonstration: none of that machinery is necessary when you start from a sufficiently strong pre-trained policy and use the most basic continued training procedure available.

The magnitude of this shift should be understood precisely. This is not a paradigm shift in the Kuhnian sense—the paper does not propose a new theory of learning or a new algorithm. Rather, it is a reframing of the problem's bottleneck. The paper argues, through systematic empirical evidence, that the difficulty of robotic adaptation is primarily a data-regime problem, not an algorithm problem. When pre-training reaches the scale of hundreds of thousands of real-world interactions (580,000 offline grasps + 28,000 online grasps in this work), the pre-trained parameters already encode most of what's needed for a range of related tasks. The "algorithm" for adaptation reduces to continued Bellman error minimization on a tiny target dataset (as few as 25–800 grasps, representing less than 0.2% of the pre-training data). The enabling condition is the depth of pre-training, not the cleverness of the adaptation mechanism.

This reframing resolves a latent tension in the field between two lines of research that had been pursued largely independently. On one side, the meta-learning and few-shot adaptation community focused on algorithms that could extract and transfer structure from limited pre-training data—implicitly operating in a regime where pre-training and adaptation data budgets were comparable. On the other side, the large-scale robot learning community (QT-Opt, grasping from 50k tries, large-scale imitation learning) focused on scaling up data collection for a single task, with adaptation treated as a secondary concern. This paper demonstrates that these two lines of research converge at scale: when pre-training is large enough, the adaptation algorithm becomes almost irrelevant, and the meta-learning problem simplifies dramatically. This is directly analogous to what happened in computer vision and NLP, where the proliferation of sophisticated transfer learning architectures in the early-to-mid 2010s gave way to the simpler insight that large-scale pre-training plus modest fine-tuning dominates across tasks.

The consequence is a significant reweighting of research priorities:

• Directions that become more attractive: Scaling up pre-training datasets for robotic skills—investing in the infrastructure to collect hundreds of thousands of diverse real-world interactions across varied objects, environments, and conditions—becomes the highest-leverage activity. Improving the quality and diversity of base policies, rather than the cleverness of adaptation algorithms, is the most direct path to enabling sample-efficient adaptation. Research on off-policy evaluation for early stopping (Section VI-B, Figure 6) becomes critical, because the offline stopping-point problem is the primary barrier to deploying this approach autonomously. Work on autonomous distribution-shift detection and adaptation triggering becomes essential, since the paper assumes human experimenters identify when fine-tuning is needed and what data to collect.

• Directions that become less urgent: Incremental improvements to meta-learning objectives (e.g., MAML variants, learned loss functions for adaptation) lose relative importance if the primary bottleneck is pre-training scale rather than algorithmic efficiency. The paper's results suggest that the marginal benefit of a better adaptation algorithm shrinks as pre-training grows, while the marginal benefit of more pre-training data remains large. Similarly, complex sim-to-real pipelines that require per-task simulator construction may be less necessary if real-world pre-training at scale can provide a foundation that adapts to real-world variations with minimal data. The paper's finding that ImageNet initialization fails dramatically (0% on half the tasks, Section VI-C) also suggests that efforts to improve robotic learning through better static visual representations—from larger supervised datasets, self-supervised learning on internet video, or multi-modal models—may hit a fundamental ceiling because the representations needed for manipulation are inherently action-conditioned and interaction-based. This does not mean perception-only pre-training is useless, but it suggests that the returns to scaling it may diminish relative to scaling interaction-based pre-training.

The paper's reconciliation of its findings with prior negative results on adaptation is also significant. Prior work that found adaptation difficult or ineffective (e.g., the ineffectiveness of frozen ImageNet features for robotic fine-tuning, the challenges of few-shot imitation, the brittleness of policies to visual changes) was likely operating in a pre-training-sparse regime where the base policy had insufficient encoded knowledge for simple fine-tuning to succeed. The paper's positive results emerge at a pre-training scale that was uncommon in prior work, suggesting that many earlier negative findings may reflect insufficient pre-training rather than fundamental limitations of fine-tuning as an adaptation mechanism. This provides a unifying explanation for conflicting results in the literature and a clear prescription: before concluding that adaptation requires algorithmic innovation, first check whether pre-training was done at sufficient scale.

---

Follow-Up Research This Work Enables

Characterizing the pre-training scale threshold for effective fine-tuning. The paper's central implicit claim is that deep RL pre-training at sufficient scale makes simple fine-tuning effective, but it provides no evidence about what "sufficient" means. A critical follow-up would train a panel of base policies at logarithmically-spaced pre-training data budgets (e.g., 10k, 30k, 100k, 300k, 600k grasps from the same QT-Opt pipeline) and measure fine-tuning performance on each challenge task from each pre-training checkpoint. This would produce a pre-training scaling curve for adaptation efficacy, directly analogous to the model-size or data-size scaling curves that drove progress in language modeling. The key question: is there a threshold where fine-tuning performance jumps (suggesting a phase change in the learned representations), or is the improvement smooth (suggesting gradual accumulation of reusable knowledge)? The answer determines whether practitioners should invest in reaching a specific pre-training volume or can expect gradual returns from incremental scaling. The paper's existing setup—580,000 grasps, five challenge tasks, six data budgets per task—provides a template that could be extended by varying only the pre-training checkpoint, keeping all other variables (algorithm, architecture, challenge tasks, evaluation protocol) fixed.

Measuring backward transfer in the continual learning chain. The paper's continual learning experiment (Section V, Table III) leaves its most important question unanswered: does fine-tuning on Task N+1 destroy performance on Task N? A direct follow-up would replicate the five-task continual learning chain and, after each fine-tuning step, evaluate the policy on all previously-seen tasks—not just the final task in the chain. This would produce a 5×5 matrix of task performance (rows: training stage, columns: evaluation task) that reveals both forward transfer (does training on Task A help Task B?) and backward transfer (does training on Task B hurt Task A?). The paper's existing infrastructure—five challenge tasks with established evaluation protocols (50+ grasps each, bin shuffling between trials)—makes this experiment immediately feasible without new hardware or data collection. The key measurement is whether earlier-task performance regresses to the base policy baseline (e.g., harsh lighting dropping from 63% back to 32% after subsequent fine-tuning stages) or remains stable. If regression is severe, the paper's claim that this method can serve "as a building block for future continual learning methods" (Section V) requires qualification; if regression is minimal, the result would be a significant finding that challenges prevailing assumptions about catastrophic forgetting in neural network adaptation. The experiment would also benefit from testing alternative task orderings—the fixed sequence in the paper (Harsh Lighting → Transparent Bottles → Checkerboard → Extend Gripper → Offset Gripper) may be particularly benign, and other orderings might reveal interference that the current ordering masks.

Architecture-controlled comparison of RL pre-training vs. supervised visual pre-training. The paper's claim that "pre-training via RL is essential" (Abstract) is supported by a comparison that confounds pre-training objective (RL vs. ImageNet classification) with network architecture (QT-Opt mid-CNN action injection vs. ResNet50 end-CNN action concatenation). A rigorous follow-up would compare three pre-training conditions on the identical QT-Opt architecture: (1) RL pre-training as in the paper, (2) supervised ImageNet pre-training of the convolutional layers followed by random initialization of the action-processing layers, and (3) self-supervised pre-training (e.g., SimCLR, MoCo) of the convolutional layers using the robot's own camera images collected during the base policy's data collection phase but without action labels. All three would be fine-tuned on the same 800-grasp challenge task datasets using the identical protocol. This would isolate the effect of the pre-training objective from the architectural confound and answer a specific question: is the key ingredient interaction (learning features that predict action outcomes), or could any representation learning on the robot's visual stream—even without action labels—provide comparable benefits? The paper's Figure 7 shows that even morphological shifts induce changes in early convolutional layers, suggesting that action-conditioned feature learning may be the critical factor; this experiment would test that hypothesis directly.

Online vs. offline fine-tuning head-to-head on real robots. The paper advocates for offline fine-tuning based on simulation experiments (Appendix B-C) and practical arguments about decoupling data collection from training, but never compares online and offline fine-tuning at matched data budgets in the real-robot setting. A direct comparison would use the same challenge tasks and initial data budgets (e.g., 100, 400, 800 exploration grasps), but for the online condition, split the data budget across multiple rounds of data collection and training—for example, 800 grasps could be collected as 8 rounds of 100 grasps each, with the policy updated between rounds and redeployed for the next batch. The offline condition would use all 800 grasps at once, as in the paper. Both would use the same total gradient step budget (500,000 × number of rounds for online, 500,000 for offline) to ensure fair compute comparison. The key measurement is whether online fine-tuning's ability to collect on-policy data from improved policies outweighs the exploration collapse risk identified in Appendix B. The paper's simulation results suggest online fine-tuning can fail catastrophically with poor initialization choices, but whether it outperforms offline fine-tuning with the full-initialization approach used in the main experiments is unknown. This experiment directly addresses the practical deployment question: should a robot fine-tune as it collects data (online), or batch-collect data first and train afterward (offline)?

Extending the adaptation benchmark to non-grasping manipulation skills. All experiments in the paper are on grasping from a bin—an episodic, single-step task with binary success feedback and no temporal credit assignment. Whether the findings generalize to tasks requiring extended temporal reasoning is unknown. A natural extension would replicate the pre-training and fine-tuning protocol for a multi-step manipulation task—for example, pick-and-place with obstacle avoidance, peg insertion with multiple contact phases, or tool use requiring a sequence of actions. The pre-training would follow the same QT-Opt offline→online pipeline but with a task-appropriate reward function (e.g., sparse reward for task completion, dense reward for progress). Challenge tasks would be constructed using the same thresholding methodology: for pick-and-place, challenges might include transparent obstacles, background visual clutter, gripper offset, or novel object shapes that the base policy was not trained on. The key question is whether the 0.2% data efficiency holds when the Q-function must learn multi-step credit assignment from limited data, or whether temporal credit assignment fundamentally requires more target-task data because errors in value estimation compound across timesteps. The paper's observation that the QT-Opt grasping reward is essentially single-step (γ ≈ 0, Section III-A explanatory text) means the adaptation problem studied here is unusually simple from a credit assignment perspective; testing on multi-step tasks would reveal whether this simplicity is essential to the method's success.

Detecting distribution shifts and triggering adaptation autonomously. The paper's fine-tuning procedure assumes human experimenters identify when a distribution shift has occurred and what condition to adapt to. In a genuine "lifelong learning" deployment, the robot must detect shifts autonomously. A concrete follow-up would develop and evaluate a Change Detection Module that monitors the robot's recent success rate and triggers fine-tuning when it deviates significantly from the expected base-policy performance, accounting for the natural variance due to object arrangement, grasp difficulty, and sampling noise. The module would need to balance two error types: false positives (triggering fine-tuning due to a random run of failures, wasting compute and risking overfitting to noise) and false negatives (failing to detect a genuine persistent shift). This could be framed as a sequential hypothesis testing problem: after each grasp attempt, update a Bayesian posterior over whether the success probability has changed, and trigger adaptation when the posterior exceeds a threshold. The experiment would use the paper's existing data—the base policy's success rate on the baseline task (86%) and the challenge tasks (32–76%) provide the null and alternative distributions—and would measure how many grasp attempts are needed to reliably detect each challenge task's shift at various false-positive rate targets. This directly addresses the paper's unaccounted difficulty estimation cost: the detection delay (grasps consumed before triggering) becomes an additional cost that must be amortized against the fine-tuning data budget when computing total adaptation efficiency.

---

Practical Applications and Downstream Use Cases

Warehouse and logistics robots adapting to inventory changes. A robot arm deployed in a fulfillment center for order picking encounters new product types, packaging materials, and bin configurations as inventory rotates. The base policy can be pre-trained on the facility's standard product corpus (analogous to the paper's 1,000-object pre-training set), then fine-tuned on small batches of experience with new products as they enter circulation. The paper's finding that transparent bottles—a challenging perceptual category absent from pre-training—can be grasped at 66% success after only 100 exploration grasps (approximately 1 hour of robot time, Section IV-B) translates directly: when a new product with unusual visual properties (transparent packaging, reflective surfaces, irregular shapes) enters inventory, the robot can adapt during a single overnight data collection and training cycle rather than requiring a manual re-training campaign. The 0.2% data budget means the adaptation cost is negligible compared to the cost of the pre-training data already invested.

Domestic robots handling changing home environments. A home robot performing manipulation tasks (clearing dishes, organizing clutter, fetching objects) encounters continuously changing conditions: lighting varies by time of day and season, new objects appear (purchases, gifts, items moved by residents), and the robot may receive hardware modifications (cleaning attachments, gripper replacements). The paper's demonstration that fine-tuning handles all four categories of change—lighting (63% from 32%), novel objects (66% from 49%), background changes (90% from 50%), and morphology modifications (98% from 47%)—suggests a single pre-trained base policy can serve as the foundation for a robot that continuously adapts throughout its deployment lifetime. The continual learning result (Table III) is particularly relevant: a home robot could adapt to winter lighting, then to new holiday decorations, then to a replacement gripper, with only a 4–7% performance penalty per transition. The practical deployment model would involve periodic "adaptation nights" where the robot collects experience under current conditions (e.g., during low-activity hours with modified lighting) and fine-tunes offline, waking up with an updated policy.

Flexible manufacturing cells with reconfigurable tooling. In small-batch manufacturing, robot workcells are frequently reconfigured—grippers, fixtures, and sensors are swapped between production runs. The conventional approach of re-teaching or re-programming robots for each configuration is a major bottleneck. The paper's demonstration on the Extend Gripper 1 cm and Offset Gripper 10 cm tasks (recovering to 93% and 98% respectively, from 76% and 47%) shows that fine-tuning can adapt to physical modifications that change the robot's kinematics and visual field. A manufacturing robot pre-trained on a standard grasping and insertion task could be fine-tuned after each tooling change with a small batch of trial cycles collected during the first hour of the new production run. The 98% success rate on offset gripper—the highest in the study—is particularly striking because the base policy "experienced absolutely no variation in robot morphology" during pre-training (Section III-B), so this is not just refining an existing capability but learning a fundamentally new visuomotor mapping. This suggests that even substantial hardware modifications (the 10 cm offset is approximately a full link width) are within scope for fine-tuning-based adaptation.

Agricultural robots adapting to seasonal and environmental variation. Crop-picking robots face extreme seasonal variation in lighting, background (soil, mulch, leaves changing throughout the growing season), and object appearance (fruit ripeness, size, and occlusion patterns varying week to week). The paper's Harsh Lighting challenge—a high-intensity halogen source producing glare and intense light-dark contrasts—is directly analogous to the lighting variation between dawn, noon, and dusk operations, or between sunny and overcast conditions. The recovery from 32% to 63% under harsh lighting (Table II, 800 grasps) demonstrates that a pre-trained policy can partially adapt to severe photometric corruption, though the lower ceiling compared to other tasks (+31 points vs. +40 to +55 for other challenges) suggests lighting adaptation may require more data or additional sensor modalities. A practical deployment might combine fine-tuning with a lightweight lighting normalization pre-processing stage, using the fine-tuning to handle the residual variation that normalization cannot eliminate. The 800-grasp budget corresponds to less than a day of operation at typical picking speeds, making weekly or even daily adaptation cycles feasible.