GR-RL: Going Dexterous and Precise for Long-Horizon Robotic Manipulation

🎯 Pitch

Human demonstrations, long assumed optimal for robot learning, actually inject harmful noise into policies tasked with millimeter-precision skills like threading shoelaces. GR-RL fixes this by filtering out suboptimal demo fragments and then using online reinforcement learning to realign the policy with its own deployment behavior, enabling a robot to autonomously lace a shoe with 83.3% success.

1. Executive Summary

GR-RL introduces a multi-stage reinforcement-augmented training pipeline that transforms a generalist vision-language-action (VLA) policy into a specialist for long-horizon dexterous manipulation—specifically, autonomously lacing a shoe by threading shoelaces through multiple eyelets. The framework operates on ByteMini-v2 bimanual robots and layers three mechanisms: data filtering with a learned task progress evaluator (using offline RL with distributional critics trained on sparse rewards to identify and discard suboptimal demonstration transitions where progress drops), morphological symmetry augmentation (mirroring observations, actions, and language instructions across the bimanual axis), and online reinforcement learning (steering the action diffusion model's denoising process via a latent-space noise predictor to close the deployment mismatch from temporal ensembling and receding horizon control). Starting from a GR-3 baseline at 45.7% success, filtered behavior cloning raises performance to 61.6%, symmetry augmentation pushes it to 72.7%, and online steering RL achieves a final 83.3% success rate, establishing that reinforcement-augmented specialization over noisy human demonstrations can produce millimeter-precision, long-horizon manipulation—but only when offline filtering prevents the policy from memorizing suboptimal fragments and online alignment corrects the training-inference distribution shift.

2. Context and Motivation

The Core Problem: Generalist VLA Policies Are Not Reliable Specialists

The fundamental tension this paper addresses is the gap between generality and reliability in robot foundation models. Vision-language-action (VLA) policies trained on large, diverse datasets of internet data and human demonstrations have demonstrated impressive generalization across objects, environments, and semantic concepts. Systems like RT-2 [10], Octo [57], π0.5 [25], GR-3 [12], and others can handle a surprisingly broad range of manipulation tasks given free-form language instructions. However, as the paper states directly in the introduction:

"being general is not equivalent to being reliable, and current VLA policies still fall short in two fundamental aspects for real-world deployment: (1) Dexterity with precision – millimeter-level control over deformable objects remains unsolved. (2) Long-horizon robustness – errors accumulate over steps, and it gets worse when coupled with high-precision dexterous manipulation."

This is not merely a "better performance" goal — it represents a qualitative gap in capability. The paper's motivating task of shoe lacing crystallizes this: the robot must handle deformable objects (both shoelace and shoe), achieve millimeter-level insertion precision (threading the lace tip through an eyelet), and maintain robustness over a long sequence of interdependent actions where any early mistake compounds into downstream failure. Generalist VLAs, trained primarily through imitation learning on broad datasets, are not optimized for this regime.

Why This Gap Matters

Real-world deployment demands. For robot learning to transition from laboratory demonstrations to practical deployment, policies must handle tasks that require precision, dexterity, and extended temporal reasoning. Assembly, surgical assistance, electronic repair, and caregiving all share the characteristics that make shoe lacing hard: soft-body interaction, fine motor control, and multi-step dependencies. A generalist policy that can identify a shoe in an image but cannot thread its laces is not a useful deployed system.

The suboptimal demonstration problem. The paper makes a claim that directly challenges an assumption embedded in most VLA training pipelines:

"Assuming the optimality of human demonstrations is core to existing VLA policies. However, we claim that in highly dexterous and precise manipulation tasks, human demonstrations are noisy and suboptimal."

This matters because the dominant paradigm for training robot policies — behavior cloning on human teleoperated demonstrations — treats every action in every demonstration as equally worth imitating. The paper argues, and provides evidence through its task progress visualizations (Figure 3), that this is fundamentally wrong for precision tasks. Human operators hesitate, retry, make mistakes, and exhibit inconsistent behaviors when attempting millimeter-accurate manipulation. A policy trained naively on these trajectories will learn to reproduce the hesitation, the retries, and the errors — not just the successful threading maneuvers. The explicit framing of this problem as one of demonstration quality rather than model architecture or dataset scale represents a significant conceptual shift.

The deployment mismatch. Beyond data quality, the paper identifies a second, subtler problem: the actions a policy sees during training are not the actions that get executed during deployment. When deploying chunking-based policies (which predict a sequence of future actions), practitioners routinely apply post-processing for smooth execution — temporal ensembling (averaging overlapping predictions across time steps), receding horizon control (executing only the first few actions and re-planning), and trajectory optimization with jerk and continuity constraints. These system-level optimizations are "necessary for the smooth execution of a learning-based policy" but "inevitably [cause] the mismatch between model training and inference." In precision manipulation, where millimeter errors matter, this mismatch is not a negligible detail — it is potentially the difference between threading the eyelet and missing it entirely.

Where Prior Approaches Fall Short

The paper organizes the limitations of prior work along several axes:

1. Pure imitation learning on human demonstrations. The standard approach — collect teleoperated demonstrations and apply behavior cloning — achieves only 45.7% success on shoe lacing with GR-3 (Figure 5, left). This low baseline from a state-of-the-art generalist VLA illustrates the ceiling imposed by suboptimal data. Behavior cloning treats all transitions equally and cannot distinguish between the teleoperator's fluent threading motion and their hesitant fumbling before a retry. The policy learns a multi-modal action distribution that averages over contradictory behaviors, producing neither the fluent success trajectory nor the recovery behavior — just a degraded blend.

The paper notes that naive extensions to behavior cloning "will result in sub-optimal and limited skills in shoelacing" [42], citing Luo and Demiris's earlier work on bi-manual robot shoe lacing which used cloning with first and third-person interfaces and achieved limited results.

2. Classic motion planning approaches. The paper references prior work on bi-manual shoe lacing using predefined action primitives and designed patterns [39–41]. These methods rely on explicit modeling of the shoelace state, known shoe geometry, and hand-crafted motion sequences. The key limitation is clear: "generalization to unseen configurations, recovering from failures, and other dexterous skills remain an open question." If the shoe is in a slightly different orientation, the shoelace is crossed differently, or an intermediate step fails, the pre-programmed sequence cannot adapt. These methods achieve task completion under controlled conditions but cannot handle the variability of real-world deployment.

3. Real-world RL without offline data curation. A significant body of prior work has explored using reinforcement learning to improve manipulation policies beyond imitation learning [2, 26, 28, 31, 43–45, 53, 54]. However, applying RL directly to long-horizon precision tasks presents a chicken-and-egg problem: exploration requires the policy to occasionally succeed at the millimeter-precision task to receive the sparse reward signal, but without a reasonably competent starting policy, success is vanishingly unlikely. The paper's insight is that offline filtering creates a stronger base policy that reduces the exploration search space, making online RL feasible. Prior work that skipped this step would face the same exploration challenge.

The paper also identifies limitations in simulation-to-real transfer approaches. Some recent work applies on-policy RL to VLAs in simulation [13, 35, 38, 51, 63], but "transferring their success to real-world scenarios remains difficult because real-world interactions are sample inefficient and noisy." World model approaches [18–22, 46, 62, 67] attempt to address this by training RL policies against learned simulators, but "introduce further issues given inaccurate visual predictions" — particularly problematic for millimeter-precision tasks where small visual prediction errors translate to physical insertion failures.

4. Prior data filtering approaches. The idea of filtering demonstration data is not new, but prior methods typically relied on heuristic criteria or human annotation to identify suboptimal segments. The paper's position is that human labeling of suboptimal fragments "is non-trivial and might introduce even more subjective and noisy human priors." At millimeter precision, a human annotator watching a video may not reliably distinguish between a threading motion that just barely succeeds and one that just barely fails — the visual difference can be sub-pixel. An automated, reward-driven approach avoids this subjectivity.

5. The specific prior work gap: no learning-based shoe lacing. The paper makes a strong claim about novelty: "to the best of our knowledge, GR-RL is the first learning-based model that can thread the shoelace through multiple eyelets." Prior shoe lacing systems (all cited references are from Luo and Demiris, 2023-2025) are motion-planning-based with predefined primitives, not learned from data. The paper positions GR-RL as breaking new ground by using learning to handle the variability, recovery, and generalization that classic approaches cannot.

How This Paper Positions Itself

Building on a generalist foundation to create a specialist. GR-RL does not propose training a policy from scratch. It starts from GR-3 [12], a large-scale VLA policy co-trained on internet data, robot trajectories, and human demonstrations. This is important positioning: the paper is not arguing against generalist foundation models but rather proposing a pipeline to specialize them for challenging downstream tasks. The relationship is complementary — generalist training provides broad visual understanding and manipulation priors; the GR-RL pipeline adds precision, robustness, and recovery behaviors.

Reinforcement-augmented, not reinforcement-replacing. The paper explicitly frames its approach as "reinforcement-augmented" rather than pure RL. Offline behavior cloning remains the core training mechanism; RL serves two specific, targeted roles: (1) providing a task progress signal for data filtering (offline RL with sparse rewards), and (2) correcting the deployment mismatch through closed-loop interaction (online RL with latent-space steering). This modular use of RL is a pragmatic compromise that avoids the exploration challenges of end-to-end RL on long-horizon tasks while still capturing RL's key benefit — the ability to optimize for task success rather than demonstration imitation.

Reconciliation with prior conflicting findings. The paper's observation that human demonstrations are suboptimal for precision tasks provides a unifying explanation for a subtle tension in the literature. Behavior cloning has been remarkably successful for many manipulation tasks (pushing, picking, placing), where suboptimal demonstrations still roughly cover the desired behavior distribution. But for precision insertion, the gap between a hesitant, error-prone demonstration and optimal behavior is much larger and more consequential. The paper's framing explains why the same imitation learning recipe that works for broad manipulation fails specifically at the dexterity-precision frontier.

Connection to concurrent work. The paper explicitly references π∗0.6 [24], a concurrent work that also uses distributional critics for real-world RL on high-precision manipulation. The key differentiating claim is about the filtered behavior cloning approach: "similar to π∗0.6, we both adopt distributional critics that learn the progress of the task. However, instead of performing advantage-conditioned denoising, we directly perform filtered behavior cloning and also observe a strong performance boost. Given the stronger base offline policy, it helps reduce the search space during online exploration." This is a specific architectural claim: that filtering data and training a clean behavior cloning policy is more effective than conditioning action generation on advantage estimates, because it produces a more focused initial policy that makes online RL's exploration problem easier.

The data-centric philosophy. Underlying the entire pipeline is a thesis that data quality matters more than model architecture for precision manipulation. The three stages — filtering, augmentation, online alignment — are all fundamentally about improving the data the policy learns from rather than designing a more expressive model. Filtering removes bad data, augmentation creates useful synthetic data, and online RL generates on-policy data that matches deployment conditions. This data-centric framing distinguishes GR-RL from approaches that focus on architectural innovations (larger models, better action representations, more sophisticated attention mechanisms) while leaving data quality as an unexamined assumption.

3. Technical Approach

3.1 Reader Orientation

GR-RL is a training pipeline, not a new model architecture — it takes an existing generalist VLA policy (GR-3) and progressively refines it through three stages: filtering out bad demonstration data, augmenting with synthetic mirrored trajectories, and fine-tuning through real-world trial-and-error. The core problem it solves is that human demonstrations for millimeter-precision manipulation tasks contain hesitation, errors, and retries that a naive behavior-cloned policy will dutifully learn to reproduce, yielding suboptimal performance. The "shape" of the solution is a data-centric specialization pipeline: first use offline RL to automatically identify which moments in a demonstration represent genuine progress toward the goal, discard everything else, train a clean policy on the filtered data, then let that policy interact with the real world to fix the remaining gap between what it learned in training and what actually gets executed at deployment.

3.2 Big-Picture Architecture (Diagram in Words)

The system has six major components connected in a sequential pipeline:

Human demonstration dataset — bimanual teleoperated trajectories of the shoe lacing task, containing both successful completions and annotated retry points. This is the raw input material that is suboptimal.
Distributional critic $Q_\phi$ (task progress evaluator) — trained once via offline TD3+BC on the demonstration data augmented with hindsight failure trajectories. Given a sparse binary reward at episode end, it learns to predict, at every timestep, a distribution over expected future return — which the paper interprets and uses as a task progress score $\rho_t$ . Transitions where this score drops are filtered out.
Filtered dataset — the subset of demonstration transitions where the task progress score $\rho_t$ does not exhibit a drop greater than threshold $\delta$ across the action chunk window. This is the cleaned training data.
VLA policy $\pi_\theta$ (GR-RL base model) — a Mixture-of-Transformer architecture sharing Qwen2.5-VL-3B-Instruct as the VLM backbone, with an action diffusion transformer (DiT) head trained via flow matching. Initially cloned from the filtered dataset, then augmented with symmetry data.
Noise predictor $\pi_{\theta'}$ (online RL module) — a lightweight (51.5M parameter) transformer added after the frozen VLM backbone that predicts the initial noise $\epsilon_t$ fed to the action DiT, steering the denoising process toward higher-return actions.
Online deployment loop — the ByteMini-v2 bimanual robot executes the steered policy in closed loop, collecting on-policy trajectories that populate replay buffers. Two critics (one in action space $Q_\phi$ , one distilled into noise space $Q_{\phi'}$ ) are updated alongside the noise predictor using off-policy RL.

Information flows: raw demonstrations → offline RL critic training → task progress filtering → behavior cloning on filtered data → symmetry augmentation → offline checkpoint as warm-start → online loop (execute steered policy, collect trajectory, store in buffer, sample batch, update critic and noise predictor, repeat). The VLM backbone is frozen during online RL — only the noise predictor and critic heads are updated.

3.3 Roadmap for the Deep Dive

First, the model architecture (Section 3.4.1): the Mixture-of-Transformer design, how the VLM backbone connects to the action diffusion head, and the critic's Q-chunking with distributional RL. Understanding the architecture is prerequisite because all three training stages modify different parts of it.
Second, the offline RL critic training and data filtering (Section 3.4.2): how the sparse reward is defined, why hindsight failure trajectories are created, how TD3+BC trains the distributional critic, and the precise filtering rule that converts Q-value predictions into a cleaned dataset. This is the most conceptually novel stage and needs careful treatment.
Third, morphological symmetry augmentation (Section 3.4.3): the specific transformations applied to images, proprioception, actions, and language to create mirrored training examples from bimanual symmetry.
Fourth, online steering RL (Section 3.4.4): the latent-space noise predictor, the two-critic distillation scheme (action-space to noise-space), the RL objectives with the KL-style divergence penalty, the buffer management strategy, and why this approach is chosen over naive action-space exploration.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and methods paper whose core idea is that reinforcement learning can be applied in three targeted ways — offline filtering, symmetry augmentation, and online latent-space steering — to specialize a generalist VLA policy into a precision manipulation specialist, without requiring architectural changes or end-to-end RL from scratch.

3.4.1 Model Architecture: Mixture-of-Transformer VLA with Distributional Critic

The policy $\pi_\theta$ . The policy is a Mixture-of-Transformer (MoT) architecture with approximately 5 billion total parameters, following the design of GR-3. It takes three inputs: a language instruction $l$ (free-form text such as "thread the shoelace through the eyelet on the left"), an observation $o_t$ (specifically three RGB camera views — head camera, left wrist camera, right wrist camera), and the robot's proprioceptive state $s_t$ (joint positions, gripper states, mobile base pose). The output is an action chunk $a_t = a_{t:t+k}$ , a sequence of $k$ future actions predicted simultaneously.

The architecture splits into two major sub-components connected in series:

VLM backbone (Qwen2.5-VL-3B-Instruct). This is the vision-language understanding component. It processes the three RGB images and the text instruction through a transformer that has been pretrained on web-scale vision-language data. The paper follows an optimization from GR-3: rather than using the full output of every transformer layer, it uses only the KV cache from the latter half of the VLM layers. The forward pass through the early VLM layers still computes activations for the later layers to attend to, but only the key-value representations from the deeper layers are passed to the action head. This is a compute-performance tradeoff — the early layers extract visual features (edges, textures, object boundaries), while the later layers produce semantic representations (what object is where, what the instruction means), and only the latter are needed for action generation.

Action Diffusion Transformer (DiT) head. The final hidden states from the VLM backbone (the KV cache representations) are concatenated with the proprioceptive state $s_t$ and passed to a diffusion transformer that generates action chunks. The generation process uses flow matching — a specific formulation of diffusion models where the forward noising process follows a straight-line path from data to noise (rather than the stochastic differential equation used in DDPM-style diffusion). The paper cites Lipman et al. [34] and Liu [36] for flow matching and builds on the action generation approach used in $\pi_0$ [8].

In flow matching, the model learns to predict a velocity field $v(\cdot)$ that transports samples from a standard Gaussian distribution $\mathcal{N}(0, 1)$ to the action data distribution. During inference, an initial noise sample $\epsilon \sim \mathcal{N}(0, 1)$ is iteratively denoised by following the learned velocity field through an ODE solver. The output is an action chunk $a_t$ of length $k$ (the paper does not specify $k$ explicitly, but it is implied to be the standard chunk length used in GR-3 and related systems, typically 10–25 steps).

The policy is trained with a standard behavior cloning objective: maximize the likelihood of the demonstration actions under the flow-matching generative model. Given a dataset of $(l, o_t, s_t, a_t^{\text{demo}})$ tuples, the flow matching loss trains the DiT to predict the velocity that transforms noise into $a_t^{\text{demo}}$ .

The critic $Q_\phi$ . The critic is a separate, causally-masked transformer that evaluates the quality of actions. It takes the same observation inputs $(o_t, l, s_t)$ as the policy, plus a candidate action chunk $a_t$ , and outputs a prediction of the expected future return. The paper adopts Q-chunking: instead of predicting a single scalar Q-value, the critic predicts a chunk of Q-values — one value for each action in the action chunk $a_t = a_{t:t+k}$ . This is motivated by prior work [29, 31, 52] that found evaluating sequences of actions rather than individual actions improves temporal credit assignment.

The critic uses distributional reinforcement learning. Rather than regressing to a scalar Q-value (which represents the expected cumulative reward), the distributional critic predicts a discrete probability distribution over possible returns. The paper sets the supports of this distribution to be bounded between $[0, 1]$ — meaning the critic's output is a categorical distribution over $N$ bins (the exact $N$ is not specified, but typical distributional RL uses 51 or 101 bins) uniformly spaced between 0 and 1. The critic is trained with a cross-entropy loss against a target distribution constructed from the Bellman backup:

The distributional Bellman target projects the next-state distribution forward by one timestep (scaling by the discount factor $\gamma$ , shifting by the immediate reward $r_t$ ), and the critic's predicted distribution is trained to match this target via cross-entropy:

$L(Q_\phi) = \text{cross\_entropy}\left(Q_\phi(o_t, l, s_t, a_t), \mathcal{T}Q_{\bar{\phi}}(o_{t+1}, l, s_{t+1}, \pi(o_{t+1}, l, s_{t+1}))\right)$

where $Q_\phi$ is the current critic, $Q_{\bar{\phi}}$ is a target network (an exponentially moving average of the critic's parameters for stability), and $\mathcal{T}$ is the distributional Bellman operator that applies the discount and reward to the next-step distribution.

What this equation computes in operational terms: for each action chunk in the training data, the critic produces a predicted distribution over possible cumulative returns. The target distribution is constructed by taking the critic's own (target network) prediction at the next timestep, scaling it by $\gamma$ , shifting it by the immediate reward $r_t$ , and projecting onto the fixed bin support. The cross-entropy between these two distributions is the training loss — minimized when the predicted distribution matches the Bellman target. This produces a critic that, at any timestep, outputs a probability distribution over how much cumulative reward the policy can expect from that point onward.

Why this form (distributional + bounded): The paper argues that bounded distributional critics give "much stronger robustness than the non-distributional ones" under sparse reward settings — and this is the central justification for the design choice. Three mechanisms explain this:

Boundedness prevents value explosion. In long-horizon tasks with sparse binary rewards, a regression-based critic receives almost no reward signal for the vast majority of timesteps. The temporal difference learning signal propagates backward from the sparse terminal reward through many steps of bootstrapping. With unbounded regression (standard MSE on scalar Q-values), early-timestep predictions can explode to arbitrarily large values because there is no upper-bound constraint — the critic can overestimate without penalty. By setting the distribution support to $[0, 1]$ , the distributional critic cannot predict values outside this range, providing a structural prior that prevents over-estimation.
Distributional modeling captures uncertainty. When the reward signal is weak (most of the trajectory has zero reward), the target distribution is relatively flat — reflecting high uncertainty about whether the task will succeed. A scalar critic would collapse this to a point estimate (e.g., "0.37"), losing the uncertainty information. The distributional critic preserves it. The paper's Figure 7 (discussed in later sections) shows this empirically: the non-distributional critic exhibits "severe over-estimation, especially in earlier parts of trajectories where the reward supervision signal is weak," while the distributional critic produces well-calibrated values.
The $[0, 1]$ bound doubles as a progress indicator. Because the support is bounded between 0 and 1, and because $\gamma^t \in [0, 1]$ for any finite $t$ with $\gamma < 1$ , the expected value of the distribution (its mean $\rho_t$ ) naturally falls in $[0, 1]$ and can be interpreted as a probability of eventual task success from that timestep. This interpretation — whether correct in a strict probabilistic sense or simply a useful heuristic — is what enables the filtering mechanism in the next section.

A design choice that matters: critic input includes actions. The critic $Q_\phi(o_t, l, s_t, a_t)$ evaluates specific action chunks, not just state-action pairs in the typical RL sense. This means the critic can score arbitrary proposed action chunks, making it usable both for filtering (scoring demonstration actions) and for online RL (scoring policy-generated actions for improvement).

3.4.2 Stage 1: Offline RL Critic Training and Data Filtering

This is the most conceptually novel stage of the pipeline. The goal is to automatically identify and remove suboptimal segments from human demonstration data without human annotation. The mechanism has three sub-stages: (a) augmenting the demonstration dataset with hindsight failure trajectories, (b) training a distributional critic via offline RL on this augmented dataset, and (c) using the critic's value predictions to filter transitions.

Sub-stage A: Hindsight failure trajectory creation.

The raw demonstration dataset consists of teleoperated trajectories of the shoe lacing task. The paper states that "most of the collected trajectories end in success" — the teleoperators generally complete the task. However, within these successful trajectories, operators mark retry keyframes: timesteps where they made a mistake and restarted a sub-task (e.g., the shoelace missed the eyelet, or the gripper dropped the lace). These are annotated during data collection.

Given a successful trajectory $\tau_{0:T}$ of length $T$ , and $M$ retry keyframes at timesteps $m_0, m_1, ..., m_{M-1}$ , the paper creates $M$ failed trajectories by truncating the original trajectory at each retry point: $\tau_{0:m_i}$ for $i = 0, ..., M-1$ . These truncated trajectories represent what would have happened if the robot had stopped at the retry point — they are failures because the task was not completed. Additionally, the original full successful trajectory $\tau_{0:T}$ is retained.

This procedure is a form of hindsight experience replay (cited from Andrychowicz et al. [1]) but applied at the trajectory level rather than the goal level. The key insight: the retry keyframes, which were annotated as "mistakes" by the teleoperator, naturally partition the trajectory into segments. By treating each pre-retry segment as a failed episode, the dataset gains failed examples that are structurally similar to the policy's own potential failures — because they come from real human errors at the same task. This is crucial for the critic to learn to distinguish progress from regress.

Sub-stage B: Sparse reward definition and offline RL training.

The paper defines a sparse binary reward that is non-zero only at the very end of an episode:

$r(o_t, l, s_t, a_t) = \begin{cases} \gamma^{T-t} \mathbb{1}(\tau), & t > T - k \\ 0, & t \leq T - k \end{cases}$

where:

$\mathbb{1}(\tau)$ is 1 if trajectory $\tau$ is successful (the shoelace was threaded through the correct eyelet and put down), and 0 otherwise.
$T$ is the length of the trajectory.
$k$ is the action chunk prediction horizon.
$\gamma$ is the discount factor (not explicitly stated, but standard values in RL are 0.95–0.99).

What this reward function computes in operational terms: for the last $k$ timesteps of a trajectory (i.e., the final action chunk window), the reward equals $\gamma^{T-t}$ if the trajectory was successful, and 0 otherwise. For all earlier timesteps (the vast majority), the reward is exactly 0. The $\gamma^{T-t}$ term discounts the terminal reward back to the timestep where it is assigned — a timestep closer to the end receives a larger value (less discounting) than one further from the end.

Why this form: standard sparse reward RL for manipulation typically gives $r = 1$ at the terminal step of a successful episode only. This paper's variant spreads the terminal reward across the last $k$ timesteps, each receiving a differently discounted fraction. The reason is tied to the Q-chunking architecture: the critic predicts a chunk of Q-values of length $k$ , and to train these predictions meaningfully, the critic needs reward signal across the entire chunk. If reward were only at $t = T$ , the last $k$ Q-values in the chunk would have to propagate reward through bootstrapping with zero immediate reward, making training harder. Spreading the reward across the final $k$ steps provides direct supervision to each element of the Q-chunk.

Training algorithm: TD3+BC. The paper uses TD3+BC [17], an offline RL algorithm that adds a behavior cloning regularization term to the standard TD3 (Twin Delayed Deep Deterministic Policy Gradient) algorithm. TD3 itself is an actor-critic algorithm for continuous action spaces that uses twin critics (two Q-networks, taking the minimum of their predictions to reduce overestimation), delayed policy updates (updating the policy less frequently than the critics), and target policy smoothing (adding noise to target actions). The "+BC" addition is crucial for offline RL: it adds a term to the policy loss that penalizes the policy for deviating from the actions in the dataset:

$L(\pi) = -\lambda Q(s, \pi(s)) + \mathbb{E}_{(s,a) \sim \mathcal{D}} \|a - \pi(s)\|^2$

The first term encourages the policy to maximize Q-values (standard RL); the second term pulls the policy toward the dataset actions (behavior cloning regularization). The hyperparameter $\lambda$ controls the tradeoff. This is essential for offline RL because the Q-function is only reliable for actions near the dataset distribution — extrapolating to unseen actions leads to arbitrary overestimation. The BC term keeps the policy in-distribution.

Training data and setup details. The critic is trained on the full augmented dataset: the original successful trajectories plus the hindsight failure trajectories. The paper states that the objective uses distributional RL with the cross-entropy formulation described above. Training hyperparameters are not exhaustively listed in the paper, but the use of TD3+BC implies standard hyperparameters from that work (typically: learning rate $3 \times 10^{-4}$ , batch size 256, discount factor 0.99, target network update rate 0.005, policy delay of 2 critic updates per policy update, Gaussian smoothing noise standard deviation 0.2 clipped to $[-0.5, 0.5]$ ).

Sub-stage C: Filtering transitions by task progress.

After training, the critic $Q_\phi$ can evaluate any transition $(o_t, l, s_t, a_t)$ and produce a categorical distribution over expected returns. The paper defines the task progress $\rho_t$ as the mean of this distribution:

$\rho_t := \text{mean}(Q_\phi(o_t, l, s_t, a_t))$

What this computes in operational terms: for a given observation-action pair, the critic predicts a discrete distribution over possible returns. Taking the mean of this distribution gives a scalar between 0 and 1 that represents the expected cumulative discounted reward from this point — which, given the sparse binary reward structure, maps approximately to the probability of eventual task success times the discount factor effect. A $\rho_t$ near 1 means the critic is highly confident the task will succeed from this point; a $\rho_t$ near 0 means the critic believes failure is likely.

The filtering rule. A transition at timestep $t$ is marked as suboptimal if there exists a drop in the progress sequence $\rho_t, \rho_{t+1}, ..., \rho_{t+k}$ greater than a threshold $\delta$ :

$\exists i \in \{0, ..., k-1\} : \rho_{t+i} - \rho_{t+i+1} > \delta$

What this rule detects: a significant decrease in predicted task success probability within the action chunk window. The intuition, grounded in Figure 3's examples, is that when a teleoperator makes a mistake — dropping the shoelace, missing the eyelet, performing an imprecise handover — the predicted probability of task success drops sharply at that moment. Transitions before and including such a drop are filtered out. The threshold $\delta$ controls sensitivity; the paper does not specify the exact value, but it is tuned to capture the sudden drops visible in Figure 3 without filtering out the normal small fluctuations that occur during smooth progress.

Why filter at the transition level rather than segment level: the filtering operates per-timestep, removing individual $(o_t, l, s_t, a_t)$ tuples from the dataset, not whole trajectory segments. If a trajectory has a momentary error (a drop) followed by a recovery and eventual success, only the transitions around the error are removed — the recovery and post-recovery segments are retained because their progress values are high again. This is visible in Figure 3(b) where the progress value drops during failed handover attempts and then rises again when the shoelace is regrasped successfully. A segment-level filter would discard the entire trajectory, losing valuable recovery behavior data.

Result of filtering: the filtered dataset is used to train the policy $\pi_\theta$ with standard behavior cloning (flow matching objective) — no RL is involved in policy training at this stage. The result is a policy that has not seen the teleoperator's fumbling, hesitation, or failed insertion attempts during training, and therefore does not learn to reproduce them.

Why this works when regression-based progress fails. The paper explicitly compares the distributional critic's progress predictions against a regression baseline (Figure 3) and a non-distributional critic baseline (Figure 7). The regression baseline, trained to predict $t/T$ directly from successful trajectories only, "tends to overly smooth the progress prediction" and "is less sensitive to subtle failure" — it cannot detect the millimeter-level difference between a successful threading and a just-barely-failed one because it never saw failures. The non-distributional critic suffers from severe over-estimation on early timesteps because the binary sparse reward provides weak learning signal, and the unbounded regression allows Q-values to explode. The distributional critic with bounded support and hindsight failure data avoids both problems: it sees failures, so it learns the visual signature of mistakes; and its bounded output prevents over-estimation.

3.4.3 Stage 2: Morphological Symmetry Augmentation

After filtering, the policy is trained with behavior cloning on the cleaned dataset, but with an additional data augmentation step applied during training. The augmentation exploits the morphological symmetry of the bimanual robot setup: the left and right arms are mechanically identical (7-DoF each, identical end-effectors, mirrored kinematics), and the task can be performed on either side.

Image augmentation. For the three RGB camera views (head, left wrist, right wrist):

All images are flipped horizontally (left-right mirror).
The left wrist camera image is swapped with the right wrist camera image after flipping. This ensures that the "left wrist" view after augmentation shows what was originally the right wrist view (now flipped), and vice versa — maintaining the semantic that left/right cameras correspond to left/right wrists.

Proprioception and action augmentation. The robot's proprioceptive state $s_t$ and the action chunk $a_t$ contain joint positions, velocities, and end-effector poses in the robot's coordinate frames. The augmentation procedure:

Transforms all proprioceptive and action data into the world frame (a fixed global coordinate system).
Applies mirror symmetry in the world frame: the y-axis is negated (left becomes right and vice versa), while x and z axes remain unchanged (assuming the mirror plane is the robot's sagittal plane — the vertical plane dividing left from right).
Transforms the mirrored world-frame poses back into local wrist frames for each arm. Because the arms are now on opposite sides after mirroring, the "left arm" local frame after augmentation corresponds to what was the right arm before augmentation.

In operational terms: if the original trajectory shows the left arm moving to coordinates $(x, y, z)$ in the world frame, the augmented trajectory shows the right arm moving to $(x, -y, z)$ in the world frame, with all joint angles appropriately mirrored. The result is a trajectory that is kinematically valid (the mirrored robot configuration is physically realizable) and represents the same manipulation but performed by the opposite hand.

Language augmentation. The text instruction is modified to reflect the spatial mirroring. The paper gives a concrete example: "the hole on the left" becomes "the hole on the right". This is essential because the policy conditions on language — if the images, actions, and proprioception are all mirrored but the language still says "left," the model receives conflicting signals about which eyelet to target.

Effectiveness and scope. The paper reports that symmetry augmentation increases success rate from 61.6% (filtered BC only) to 72.7% (filtered BC + augmentation) — a gain of 11.1 percentage points (Figure 5, left). The stage-by-stage breakdown in Figure 6 shows that augmentation improves performance "in all stages, although with a smaller magnitude" than data filtering or online RL for any individual stage. This makes sense: symmetry augmentation does not teach new behaviors or fix specific errors; it effectively doubles the dataset size (every trajectory has a mirrored counterpart) and encourages the policy to learn left-right invariant features, which improves generalization to shoes presented on either side or in different orientations.

Why this method over alternatives: domain randomization (adding noise to observations and actions) is a common augmentation strategy but would be counterproductive for millimeter-precision tasks — perturbing actions by even small amounts could change a successful threading into a failed one. Symmetry augmentation, by contrast, produces kinematically valid augmented trajectories that are guaranteed to be physically realizable (because the robot is symmetric) and semantically correct (because the task can be solved from either side). The augmentation is "simple yet effective" precisely because it respects the physical constraints of the problem.

3.4.4 Stage 3: Online Steering RL for Deployment Alignment

The filtered-and-augmented behavior cloning policy achieves 72.7% success, but there remains a gap to reliable deployment. The paper identifies the root cause as a mismatch between training and deployment actions: during training, the policy learns to predict raw action chunks that match the demonstration data. During deployment, these predicted chunks are post-processed through temporal ensembling (weighted averaging of overlapping predictions from consecutive timesteps), receding horizon control (executing only the first few actions and re-planning), and trajectory optimization (imposing jerk and continuity constraints). These optimizations are essential for smooth physical execution but mean the actions actually executed by the robot differ from what the policy predicted. In millimeter-precision manipulation, this discrepancy matters.

Why naive exploration fails. The paper argues that performing online RL by adding noise directly to action outputs (e.g., Gaussian noise on joint positions or end-effector poses) is ineffective: "since the task requires millimeter precision to complete, adding noise to wrist poses or joint positions hardly leads to success." Random perturbations in action space will almost certainly push the robot away from the narrow manifold of successful trajectories — the shoelace tip will miss the eyelet, the gripper will drop the lace, or the handover will fail. Without occasional successes, the sparse reward provides no learning signal, and the RL update collapses.

The solution: latent-space noise steering. Instead of perturbing actions directly, the paper adds a noise predictor $\pi_{\theta'}$ — a small transformer with 51.5 million parameters — that operates before the action diffusion model. The noise predictor takes the VLM backbone's output (the KV cache representations) as input and outputs a perturbation to the initial noise $\epsilon_t$ that seeds the flow-matching denoising process.

Recall how the flow-matching action generation works during standard inference:

Sample initial noise $\epsilon \sim \mathcal{N}(0, 1)$ (a Gaussian vector of the same dimensionality as the action chunk).
Iteratively denoise through the learned velocity field $v(\cdot)$ conditioned on the VLM representations, producing the action chunk $a_t$ .
The denoising process is deterministic given the initial noise and the conditioning — different initial noise samples produce different action chunks (this is how the policy generates diverse behaviors), but for a fixed noise, the output is fixed.

The noise predictor $\pi_{\theta'}$ intercepts step 1: instead of sampling $\epsilon \sim \mathcal{N}(0, 1)$ , the system uses $\epsilon_t \sim \pi_{\theta'}(o_t, l, s_t)$ . The noise predictor outputs a learned perturbation to the standard Gaussian noise, and this perturbation steers the denoising process toward action chunks that the critic $Q_\phi$ predicts will yield higher returns.

The online RL objective for the noise predictor:

$L(\pi_{\theta'}) = \mathbb{E}_{(o_t, l, s_t) \sim \mathcal{D}} \left[-Q_{\phi'}(o_t, l, s_t, \epsilon_t) + c \max\left(\frac{1}{2}\|\epsilon_t\|^2 - \beta, 0\right)\right], \quad \epsilon_t \sim \pi_{\theta'}(o_t, l, s_t)$

where:

$Q_{\phi'}$ is a critic that has been distilled into noise space — it predicts Q-values given the noise $\epsilon_t$ rather than the action $a_t$ (explained below).
$c$ is a weighting coefficient for the divergence penalty (hyperparameter).
$\beta$ is a threshold on the $\ell_2$ norm of the predicted noise.
$\mathcal{D}$ is the replay buffer (mixture of off-policy and on-policy data).

What this loss computes in operational terms: the first term $-Q_{\phi'}(o_t, l, s_t, \epsilon_t)$ is the standard RL objective — maximize the expected Q-value, which means finding noise vectors that lead to high-return action chunks. The second term is a penalty on divergence from the standard normal distribution: $\frac{1}{2}\|\epsilon_t\|^2$ is the negative log-likelihood of $\epsilon_t$ under $\mathcal{N}(0, 1)$ (up to a constant). When this norm exceeds the threshold $\beta$ , the penalty activates. When it's below $\beta$ , the penalty is zero (the $\max(\cdot, 0)$ ensures this).

Why this penalty form: the policy's action diffusion model was trained on noise samples from $\mathcal{N}(0, 1)$ — this is the distribution it expects. If the noise predictor outputs vectors far from this distribution (very large norms or unusual directions), the resulting action chunks will be out-of-distribution for the diffusion model, producing arbitrary or physically invalid actions. The penalty acts as a soft constraint: explore within a bounded region around the standard Gaussian, where the diffusion model's denoising is reliable. The $\beta$ threshold provides a "trust region" — small deviations are permitted (and necessary for exploration), but large deviations are penalized. This is functionally similar to KL-constrained policy optimization (used in PPO and related algorithms) but applied to the latent noise space rather than action space.

Critic distillation into noise space. Computing $Q_{\phi'}(o_t, l, s_t, \epsilon_t)$ requires a critic that takes noise as input rather than actions. The naive approach — backpropagating through the entire action diffusion model from actions to noise — would be computationally prohibitive (the denoising process involves many iterative steps through the DiT). Instead, the paper distills a separate critic $Q_{\phi'}$ in noise space:

$L(Q_{\phi'}) = \text{cross\_entropy}\left(Q_{\phi'}(o_t, l, s_t, \epsilon_t), Q_\phi(o_t, l, s_t, \pi_\theta(o_t, l, s_t \mid \epsilon_t))\right)$

where:

The input noise $\epsilon_t$ is sampled from a mixture distribution: with probability 0.5, from the standard normal $\mathcal{N}(0, 1)$ ; with probability 0.5, from the noise predictor $\pi_{\theta'}(o_t, l, s_t)$ .
$Q_\phi$ is the action-space critic (trained with TD3 as described in Stage 1, and further fine-tuned online).
$\pi_\theta(o_t, l, s_t \mid \epsilon_t)$ is the action chunk produced by running the diffusion model's denoising process starting from noise $\epsilon_t$ .

What this computes: the noise-space critic $Q_{\phi'}$ is trained to predict the same Q-value distribution as the action-space critic $Q_\phi$ . For a given noise vector $\epsilon_t$ , the system runs the full denoising process to get the corresponding action $\pi_\theta(\cdot|\epsilon_t)$ , evaluates it with the action-space critic $Q_\phi$ , and uses that as the target for $Q_{\phi'}$ . The 50/50 sampling from $\mathcal{N}(0,1)$ and the noise predictor ensures good coverage: the standard Gaussian samples ensure $Q_{\phi'}$ learns the Q-values for the regions the diffusion model was trained on (the original behavior distribution), while the noise predictor samples ensure $Q_{\phi'}$ learns Q-values for the regions the policy is actually exploring during online RL.

Operational summary of the distillation: $Q_{\phi'}$ is a lightweight approximator that maps $(\text{observation}, \text{noise}) \to \text{Q-value distribution}$ , bypassing the expensive denoising step during policy optimization. The noise predictor's loss uses $Q_{\phi'}$ directly (no denoising needed in the inner loop), making online RL computationally feasible. The distillation loss ensures $Q_{\phi'}$ stays synchronized with $Q_\phi$ as both are updated online.

Buffer management for sample-efficient adaptation. The paper maintains two replay buffers:

Off-policy buffer: contains trajectories from multiple sources. Before online RL begins, it is populated with online rollouts of the offline-trained checkpoint (a form of warm-start RL [66]). During online training, when on-policy data becomes "stale" (generated by checkpoints more than two versions old), it is moved to the off-policy buffer. This buffer enables the algorithm to reuse past experience and maintain stability — the off-policy data provides coverage of the behavior distribution the critic was originally trained on.
On-policy buffer: contains trajectories generated only by the two most recent checkpoints. This ensures that the most recent data reflects the current policy's behavior distribution, which is critical for the critic to accurately evaluate the policy's current actions.

Batch sampling strategy: each training batch is sampled evenly from the off-policy and on-policy buffers. This balances two competing objectives: (a) on-policy data provides accurate Q-value estimates for the current policy's actions (low bias), while (b) off-policy data provides diverse coverage of the state-action space and prevents catastrophic forgetting (low variance). The 50/50 split is a common heuristic in off-policy RL for robotics [43, 44].

Online training loop. The procedure, as described in the paper, operates as follows:

Warm-up: before any online training, the offline checkpoint (Filtered BC + Augmentation, achieving 72.7% success) is deployed to collect 673 trajectories on the real robot. These trajectories populate the off-policy buffer. The VLM backbone of $\pi_\theta$ is then frozen — it will not be updated during online RL.
Critic warm-up: the action-space critic $Q_\phi$ and the noise-space critic $Q_{\phi'}$ are fine-tuned on the 673 rollout trajectories (with the VLM backbone frozen) to align their predictions to the actual deployment distribution. This handles any initial distribution shift between demonstration data (which the critic was originally trained on) and the policy's rollout data.
Online collection loop: the policy runs with the noise predictor active, generating 12 new episodes on the real robot. Each episode produces a trajectory with sparse reward (1 if successful, 0 otherwise).
Training step: after every 12 new episodes, the system performs 50 optimization steps. Each step:
- Samples a batch from the 50/50 mix of off-policy and on-policy buffers.
- Updates the action-space critic $Q_\phi$ using distributional TD3 (standard Bellman backup with twin critics, target networks, and delayed updates — though the paper doesn't specify if twin critics are used, TD3 implies it).
- Updates the noise-space critic $Q_{\phi'}$ using the distillation loss (Equation 4).
- Updates the noise predictor $\pi_{\theta'}$ using the RL objective with divergence penalty (Equation 3).
- The VLM backbone and the action DiT remain frozen — only $Q_\phi$ , $Q_{\phi'}$ , and $\pi_{\theta'}$ (51.5M parameters total) are updated.
Buffer management: the 12 new episodes are added to the on-policy buffer. When the on-policy buffer has data from more than 2 checkpoints, the oldest data is moved to the off-policy buffer.
Repeat from step 3 until convergence or budget exhaustion. The paper evaluates the checkpoint at 500 online training steps, which achieves the final 83.3% success rate.

Important design choice: no teleoperated data in the online buffer. The paper explicitly states: "we intentionally choose not to mix teleoperated trajectories into the buffer to prevent the policy from training on mismatched dynamics." During online RL, the policy's actions go through temporal ensembling and trajectory optimization before execution — the executed actions are not the raw predicted actions. If teleoperated trajectories (which have no such post-processing) were in the buffer, the critic would receive mixed signals: some trajectories reflect the deployment dynamics, others reflect raw prediction dynamics. This would degrade the critic's accuracy for the very purpose it's needed — evaluating the quality of the policy's actual deployed actions.

Why this approach over end-to-end RL: the entire online RL stage updates only 51.5M parameters (the noise predictor and critics) out of the total 5B, with the VLM backbone and action head frozen. This is a form of parameter-efficient fine-tuning that achieves several practical benefits:

Computational efficiency: backpropagating through the full 5B model for each RL update would be prohibitively expensive for real-time robot learning.
Stability: the frozen backbone ensures the policy does not catastrophically forget its general visual understanding and manipulation priors during online RL — it only adjusts how it explores in the latent noise space.
Sample efficiency: with fewer trainable parameters, the RL updates converge faster with limited real-world interaction data.

The paper's Figure 5 (right) shows the online training curve: a moving average of the success rate over a window of 24 episodes. Initially, the success rate decreases — this is the expected dip from distribution shift when the noise predictor starts exploring. After this adaptation phase, the success rate rapidly recovers and surpasses the offline performance, eventually exceeding 90% (the final evaluated checkpoint at step 500 achieves 83.3% — the gap between the moving average peak and the evaluated checkpoint suggests some variance in the process, consistent with the paper's acknowledged "behavior-drifting problem" during online RL).

3.4.5 Inference-Time Deployment and Trajectory Optimization

During final evaluation (when computing the reported success rates), the system operates with the trained noise predictor active (not reverting to standard Gaussian noise), meaning the online RL policy is what gets evaluated — not the original offline checkpoint. Additionally, the paper mentions a trajectory optimization module applied at inference: "we incorporate a trajectory optimization module that imposes constraints on jerk and temporal continuity to refine the predicted action chunks." Jerk is the third derivative of position with respect to time — minimizing jerk produces smooth, natural-looking motions. Temporal continuity constraints ensure that consecutive action chunks connect smoothly (no sudden jumps between the end of one chunk and the start of the next, even with receding horizon control). This module is applied on top of the already-steered policy output, further refining the actions before execution. The paper does not provide implementation details of this module (it may be a standard quadratic program or spline-based smoother), but its presence is notable: even after online RL alignment, some post-processing remains necessary for physically smooth execution.

3.4.6 Summary of Design Choices and Their Justifications

Distributional critic with bounded $[0, 1]$ support over regression: prevents value over-estimation in sparse reward settings, captures uncertainty through full distribution rather than point estimate, and provides a naturally interpretable progress score.
Hindsight failure trajectory creation from retry keyframes over manual suboptimality labeling: automatic, leverages existing teleoperator annotations made during data collection, and produces failed trajectories that are distributionally similar to the policy's potential failure modes.
Transition-level filtering by progress drop over segment-level filtering: retains recovery behaviors and post-recovery segments that are valuable for training, while precisely excising only the suboptimal moments.
Filtered behavior cloning over advantage-conditioned denoising (as in concurrent work $\pi^* 0.6$ ): produces a cleaner, more focused base policy that reduces the exploration search space for online RL.
Symmetry augmentation over domain randomization: produces kinematically valid, semantically correct augmented trajectories that respect the physical constraints of millimeter-precision manipulation, unlike random perturbations.
Latent noise-space steering over action-space exploration: avoids the exploration problem inherent to millimeter-precision tasks (random action perturbations almost never succeed), and constrains exploration to the manifold of physically valid actions through the diffusion model's denoising process.
Divergence penalty with $\beta$ threshold over unconstrained noise optimization: prevents the noise predictor from producing noise vectors far from the training distribution, which would cause the diffusion model to generate out-of-distribution, physically invalid actions.
Critic distillation into noise space over backpropagating through the diffusion model: computationally feasible for online RL (avoids dozens of denoising steps per gradient computation), and the 50/50 sampling strategy ensures good coverage of both the original behavior distribution and the exploration region.
50/50 off-policy/on-policy buffer split and exclusion of teleoperated data from online buffer: balances stability (off-policy diversity) with accuracy (on-policy relevance), and prevents the critic from being confused by trajectories that don't reflect deployment dynamics (teleoperated data has no temporal ensembling or trajectory optimization).
Frozen VLM backbone during online RL (51.5M trainable parameters out of 5B): parameter-efficient, computationally feasible, prevents catastrophic forgetting of general visual understanding, and is sufficient because the noise predictor only needs to learn which latent perturbations produce better actions — not change the fundamental action generation process.

4. Key Insights and Innovations

Innovation 1: Rewriting the Role of RL — From Policy Optimizer to Data Quality Auditor

The most intellectually distinctive move in GR-RL is not that it uses reinforcement learning to improve a manipulation policy (this is well-established), but rather what it uses RL for in the offline stage: not to train a policy, but to audit the quality of human demonstrations. This inverts the conventional relationship between demonstrations and RL. In most robot learning pipelines, demonstrations are treated as ground truth that RL refines or builds upon. GR-RL instead treats demonstrations as suspect evidence that must be cross-examined before a policy is allowed to learn from them.

Prior work on learning from suboptimal demonstrations typically falls into two camps. The first camp uses inverse RL or preference learning to infer a reward function that explains the demonstrations, then optimizes against it — implicitly assuming the demonstrations reveal what's desirable through their distribution, even if individually noisy. The second camp uses RL to improve beyond demonstrations through exploration, treating behavior cloning as a warm-start that RL will eventually surpass. GR-RL does neither of these things. Its offline critic is trained purely to produce a progress score, and this score is used to delete data — not to weight it, not to relabel it, but to permanently excise transitions from the training set. The critic is a filter, not a teacher.

What makes this conceptually novel:

It treats demonstration quality as a latent variable that can be recovered from sparse outcome signals. The paper never asks a human to label which moments are good and which are bad. Instead, it exploits a structural fact about the data collection process: teleoperators press a "retry" button when they make a mistake, creating retry keyframes that partition trajectories. By truncating trajectories at these keyframes to create hindsight failures and training a distributional critic on both successes and failures, the system learns to recognize the visual and kinematic signature of progress toward the goal without anyone specifying what that signature looks like. The critic's Q-value distribution captures whether an observation-action pair is on a path that probabilistically leads to success — and this signal emerges purely from the temporal structure of sparse rewards propagated through distributional Bellman backups.

This is fundamentally different from prior data filtering approaches in robot learning. Standard filtering methods use heuristic criteria (e.g., minimum episode length, velocity thresholds, end-effector position bounds) or require human annotators to label suboptimal segments. Heuristics fail at millimeter precision because the difference between a near-miss and a success may be invisible to simple metrics. Human annotation fails because at this precision level, a human watching a video cannot reliably judge whether a threading motion would have succeeded — the margin is sub-pixel. The RL-based progress evaluator solves both problems: it is automated (no annotation cost) and it has access to outcome information (did this trajectory eventually succeed or fail) that a human watching a single transition lacks.

The distributional critic design is not an implementation detail — it is a conceptual requirement. The paper devotes Figure 3 and Figure 7 to showing that only a bounded distributional critic produces progress estimates sensitive enough to detect millimeter-level mistakes while simultaneously avoiding the over-estimation that plagues regression-based critics in sparse reward settings. A non-distributional critic trained with the same TD3+BC algorithm fails (Figure 7): its Q-values explode on early timesteps because the sparse binary reward provides almost no learning signal for the vast majority of transitions, and the unbounded regression happily extrapolates to arbitrarily large values. The distributional critic's structural constraint — that values must live in [0, 1] — is not a regularization trick; it is a hard prior that prevents a known failure mode of temporal difference learning on long-horizon sparse-reward problems. This is an insight about what critic architecture enables reliable progress estimation, not simply an architecture choice that happened to work.

Evidence anchoring. The progression from GR-3 baseline (45.7%) to filtered BC (61.6%) in Figure 5 (left) is a 15.9 percentage-point gain attributable to filtering alone — larger than the gains from symmetry augmentation (11.1 points) and comparable in magnitude to the online RL gain (10.6 points over the augmented baseline). The qualitative evidence is equally important: Figure 3 shows the progress score dropping sharply at moments the teleoperator fumbled (dropped shoelace, missed eyelet, imprecise handover), and rising during successful threading and recovery behaviors. These drops are precisely what the filtering rule captures, and they map onto intuitive notions of task progress that a human observer would identify — but that the system learned without any explicit "this is a mistake" labels.

Distinguishing fundamental from incremental. This is a fundamental reframing of how to handle suboptimal demonstrations, not an incremental improvement to filtering heuristics. The core idea — that a critic trained with sparse outcome signals can serve as an automated data quality auditor — is general beyond shoe lacing. Any manipulation task where human demonstrators occasionally make recognizable mistakes (pressing a retry button, verbalizing frustration, or simply having their errors identifiable through outcome-based hindsight labeling) could benefit from this approach. The conceptual machinery (hindsight failure creation + distributional critic + progress-drop filtering) is task-agnostic.

Innovation 2: Diagnosing and Treating the Deployment Mismatch as a First-Class Problem

GR-RL identifies a problem that most robot learning papers either ignore or handle through ad-hoc engineering: the actions a policy predicts during training are not the same actions that get executed during deployment, and for millimeter-precision tasks, this discrepancy is not a minor detail — it is a primary failure source. The paper gives this problem a name and a principled treatment rather than sweeping it under the "system engineering" rug.

This matters because the standard assumption in behavior cloning is that if the policy accurately predicts the training actions, it will succeed at deployment. Temporal ensembling, receding horizon control, and trajectory smoothing are treated as post-hoc improvements that make execution smoother without changing the fundamental relationship between predicted and executed actions. GR-RL argues that this assumption breaks down at high precision: the very optimizations that make physical execution possible introduce a distribution shift between what the policy was trained to predict and what the robot actually does. The policy learns to output action chunks that, when post-processed, may no longer achieve the intended effect.

Prior work has addressed related issues through different lenses. Some systems avoid the problem entirely by not using chunking or post-processing at deployment (predicting single actions at high frequency), but this sacrifices the temporal consistency benefits that make action chunking attractive for smooth manipulation. Others attempt to make training match deployment by applying the same post-processing to training actions, but this creates a moving target — the smoothened training actions no longer match what the demonstrator actually did, potentially removing useful high-frequency corrections. The paper's approach — align through interaction, not through simulation — is distinct: rather than trying to make training match deployment in the offline stage, it acknowledges the mismatch and uses online RL to teach the policy what actions, when post-processed, actually succeed.

The latent-space noise steering solution is notable for what it doesn't do. It does not modify the action diffusion model (the DiT is frozen). It does not modify the VLM backbone (frozen). It does not explore in action space (the paper argues this is hopeless for millimeter-precision tasks). Instead, it adds a lightweight module that learns to perturb the random seed of the diffusion process toward seeds that produce high-return action chunks. This is conceptually elegant in a specific way: it separates the generation of valid actions (handled by the frozen diffusion model, which was trained on filtered demonstration data and thus produces actions that look like plausible manipulation) from the selection of which valid actions to execute (handled by the noise predictor, which is trained with RL to favor seeds that lead to success). The diffusion model ensures the actions are physically plausible; the noise predictor ensures they are effective under deployment conditions.

The divergence penalty (beta-threshold on noise norm) is the mechanism that makes this separation work. Without it, the noise predictor could output noise vectors far from the standard normal distribution that the diffusion model was trained on, producing actions that are out-of-distribution and potentially nonsensical. With it, the noise predictor operates within a trust region where the diffusion model's denoising is reliable. This is effectively a KL-constrained policy optimization but applied in a learned latent space (the diffusion model's noise space) rather than the action space — and it's this choice of which latent space to constrain that is the conceptual contribution, not the constraint mechanism itself (which is standard in RL).

Evidence anchoring. Figure 5 (right) shows the online training curve, with the characteristic initial dip (distribution shift as the noise predictor begins exploring) followed by recovery and eventual improvement beyond the offline baseline. The moving average success rate eventually exceeds 90%, while the offline baseline was at 72.7% — the online RL stage closes more than half of the remaining failure gap. Figure 6 provides stage-by-stage evidence: the "Thread into the eyelet" stage is where the online RL improvement is most dramatic, consistent with the argument that the deployment mismatch primarily affects precision-critical moments (threading through a millimeter-scale eyelet) rather than coarse manipulation (picking up the shoelace).

Distinguishing fundamental from incremental. The diagnosis — that deployment post-processing creates a distribution shift that matters — is a conceptual contribution with implications beyond this paper. Any robot learning system that uses action chunking, temporal ensembling, or trajectory optimization at deployment should consider whether the training-deployment mismatch is within acceptable tolerances for its precision requirements. The paper provides a concrete methodology (online RL with latent-space steering) for closing this gap when it matters, but the awareness of the problem as a first-class concern is arguably more important than the specific solution.

Innovation 3: A Data-Centric Recipe That Reframes Specialization as Curation + Alignment

GR-RL does not introduce a new model architecture, a new RL algorithm, or a new representation. Its model is a GR-3 variant; its offline RL uses TD3+BC (from Fujimoto and Gu, 2021); its online RL adapts the latent-space steering method from Wagenmaker et al. (2025); its symmetry augmentation is a straightforward mirroring operation. What makes the paper significant is not any individual component but the composition and sequencing of these components into a coherent data-centric specialization recipe, and the empirical demonstration that this recipe — applied to a generalist VLA — produces capabilities (autonomous shoe lacing) that neither the generalist policy nor any prior learning-based system could achieve.

This constitutes an insight because it suggests a different mental model for how to build capable robot specialists. The dominant narrative in robot learning has been that scaling up models and datasets will eventually unlock precision and dexterity — train a large enough VLA on enough diverse data, and millimeter-precision insertion will emerge. GR-RL implicitly argues against this: even GR-3, a state-of-the-art generalist VLA trained on internet-scale vision-language data and large robot demonstration datasets, achieves only 45.7% on shoe lacing — less than a coin flip. The paper's data-centric pipeline more than doubles this to 83.3% using the same base model architecture and the same human demonstration dataset, simply by cleaning, augmenting, and aligning the data.

The progression in Figure 5 (left) tells a story about where the gains come from. Filtering: +15.9 points (removing bad data). Augmentation: +11.1 points (creating synthetic data that respects physical symmetries). Online alignment: +10.6 points (generating on-policy data that matches deployment conditions). The total gain is 37.6 percentage points — the majority of which comes from improving data quality and coverage, not from architectural changes or more expressive models. This is a data-centric success story that parallels trends in other domains (e.g., the recognition in NLP that data quality often matters more than model size for specialized tasks) but has been less systematically demonstrated in robot manipulation at this precision level.

The specific ordering of stages matters conceptually. Filtering comes first — you cannot augment bad data, and you cannot align a policy that has memorized suboptimal behaviors. Augmentation comes second — it expands the filtered dataset's coverage without introducing new noise, leveraging the structural prior of bimanual symmetry. Online RL comes last — and critically, it operates on a policy that is already strong (72.7%), which makes exploration feasible because the policy succeeds often enough to provide a meaningful reward signal. If online RL were applied to the unfiltered GR-3 baseline (45.7%), the exploration problem would be much harder because successes would be too sparse for the noise predictor to learn from. The pipeline is stage-gated: each stage creates the conditions for the next stage to succeed.

Comparison to concurrent work sharpens the contribution. The paper explicitly contrasts with π*0.6 [24], which also uses distributional critics for real-world RL on high-precision manipulation but takes a different approach: advantage-conditioned denoising rather than filtered behavior cloning. The paper's claim is that filtered BC produces a stronger initial policy, which reduces the search space for online RL — an argument about the interaction between offline data curation and online learning efficiency. This is an insight about pipeline design, not about any individual algorithmic choice.

Evidence anchoring. Beyond the aggregate success rates, Figure 6 provides a stage-by-stage decomposition that reveals where each pipeline component helps. Data filtering and online RL "largely reduce the failure during threading" — the precision-critical stage where the deployment mismatch most matters. Augmentation improves all stages with smaller magnitude, consistent with its role as a generalization enhancer rather than a precision fix. This decomposition validates the paper's claim that different stages address different failure modes: filtering removes imitation of demonstrated errors, augmentation improves robustness to spatial variation, and online RL aligns predicted actions with executed outcomes at the precision bottleneck.

Distinguishing fundamental from incremental. As a single recipe, this is incremental — the individual components exist in prior work. As a demonstration that generalist VLAs can be specialized through data-centric pipelines to achieve qualitatively new capabilities, it is more significant. The paper provides a template that other practitioners can follow: start with a generalist VLA, audit your demonstration data with an outcome-conditioned progress evaluator, filter aggressively, augment with physically grounded transformations, and close the deployment gap with sample-efficient online RL. Whether this template transfers to other precision-manipulation tasks (suturing, soldering, assembly) is an open question, but the paper provides a concrete starting point and the evidence that the approach can work at a challenging frontier task.

Innovation 4: The Distributional Critic as a Progress Detector — A Diagnostic Tool, Not Just a Value Function

GR-RL's use of a distributional critic to detect suboptimal transitions represents a conceptual shift in what value functions are for in robot learning. In standard RL, the value function is a means to an end: it guides policy improvement by estimating which actions lead to higher returns. In GR-RL's offline stage, the value function plays a fundamentally different role: it is a diagnostic instrument that reveals the internal structure of demonstration trajectories — where progress is being made, where mistakes occur, and which transitions genuinely contribute to task completion.

This is novel because it extracts information from the value function that is orthogonal to policy optimization. The progress score ρ_t is not used to train the policy through advantage weighting or Q-filtered BC (both standard techniques in offline RL). It is used purely for trajectory forensics — to examine each demonstration and ask: "was this transition actually moving the task forward, or was it part of a fumble that the demonstrator later recovered from?" The value function becomes a data analysis tool rather than a training signal.

The specific property that makes this work — the critic's ability to detect drops in progress — rests on a subtle aspect of how distributional RL interacts with sparse rewards in long-horizon tasks. In a successful trajectory, the distributional critic's predicted mean should generally increase (or at least not sharply decrease) as the task progresses, because the remaining distance to the terminal reward is shrinking. A sharp drop indicates that the critic believes, based on its training on both successful and failed trajectories, that the current observation-action pair resembles moments that preceded failure in the training data. The critic has learned the visual precursors of failure — the slightly off-center grasp, the misaligned approach vector, the unstable handover — without anyone labeling these precursors. This is an emergent capability of combining hindsight failure data with distributional value learning, and it is what gives the filtering its precision (as shown qualitatively in Figure 3).

The comparison with the regression-based progress predictor (Figure 3) is the key evidence that this is a genuine finding, not a lucky engineering choice. The regression baseline — trained to predict t/T on successful trajectories — fails at detecting subtle failures because it never saw failures during training. It learns a smooth temporal interpolation that is insensitive to the millimeter-level differences between success and failure. The non-distributional RL critic (Figure 7) fails because its unbounded values over-estimate early-timestep returns in the absence of reward signal. Only the combination of (a) distributional value learning with bounded support and (b) training on both successes and hindsight failures produces a progress signal that is simultaneously sensitive to mistakes and well-calibrated across the trajectory.

Distinguishing fundamental from incremental. This is an incremental technical contribution with conceptual implications. The use of value functions for data filtering is not entirely new — prior work has used Q-values to relabel or weight transitions in offline RL datasets. But the paper's specific argument — that a distributional critic trained on a specific construction of hindsight failure data can serve as an automated, high-precision data quality auditor for demonstration trajectories — and its empirical demonstration that this auditing catches errors invisible to both temporal regression and non-distributional critics — makes a case that distributional critics are uniquely suited to this diagnostic role in sparse-reward, long-horizon, precision-manipulation settings. This finding, if it generalizes to other tasks, has implications for how the field approaches demonstration data curation: rather than relying on human annotation or heuristic filters, one could systematically construct hindsight failure data (where possible, via retry annotations or outcome-based truncation) and use distributional critics as automated quality auditors.

5. Experimental Analysis

Evaluation Methodology

Dataset. The evaluation uses human teleoperated demonstrations of the shoe lacing task collected on ByteMini-v2 bimanual robots. The paper does not report the exact number of demonstration trajectories, but specifies that the offline model's rollout collection for critic warm-up comprises 673 trajectories, and that the online RL phase operates over hundreds of episodes (the training curve in Figure 5, right, extends to 175 episodes of online collection). Retry keyframes are annotated during data collection, enabling hindsight failure trajectory creation. The demonstrations are collected by experienced teleoperators and "most of the collected trajectories end in success" (Section 3.1). No other datasets or benchmarks are used for evaluation — this is a single-task, real-world robot evaluation.
Base model(s). The starting point is GR-3 (Cheang et al., 2025), a large-scale VLA policy co-trained on internet-scale vision-language data, robot trajectories, and human demonstrations, using Qwen2.5-VL-3B-Instruct as the vision-language backbone with a flow-matching action diffusion head. The total model size is approximately 5 billion parameters. The paper uses GR-3 as the base model because it represents a state-of-the-art generalist VLA, and the goal is specifically to demonstrate that a generalist policy can be specialized into a reliable dexterous manipulation expert through the GR-RL pipeline. The GR-3 baseline achieves 45.7% success on shoe lacing with standard behavior cloning on all teleoperated demonstrations (Figure 5, left).
Metrics. The primary metric is success rate — the fraction of evaluation trials in which the robot successfully threads the shoelace through the correct eyelet and puts it down on the table. The paper also reports stage-by-stage success rates (Figure 6), decomposing the task into intermediate milestones: picking up the correct shoelace, threading into the correct eyelet, handing over to the other gripper, and pulling the shoelace tight. Each stage's success is conditional on the previous stage being completed — the height of each hatched area in Figure 6 represents the drop in success rate from the previous stage to the current one. During online RL, a binary success signal per episode is recorded and smoothed with a moving average over a window of 24 episodes to produce the training curve (Figure 5, right). The paper does not report confidence intervals, standard deviations, or statistical significance tests for any success rates.
Baselines. The paper evaluates five configurations that correspond to successive stages of the pipeline:
- GR-3: The base VLA policy trained with behavior cloning on all human teleoperated data without filtering, augmentation, or online RL. This is the "standard approach" baseline that represents the state of generalist VLA performance on this task.
- Filtered BC: Behavior cloning on the filtered dataset only, after applying the distributional critic-based progress filter (Section 3.1). No symmetry augmentation, no online RL.
- Filtered BC + Aug: Behavior cloning on the filtered dataset with morphological symmetry augmentation applied during training (Section 3.2). No online RL.
- GR-RL: The full pipeline — filtered BC with augmentation, followed by online steering RL (Section 3.3). This is the final system evaluated at the checkpoint after 500 online training steps.
- For the progress evaluator analysis (Figures 3, 7), additional baselines include a regression-based progress predictor (trained to predict t/T directly from successful trajectories) and a non-distributional critic (trained with the same TD3+BC algorithm but using scalar MSE regression instead of categorical cross-entropy).
Generation budget / compute accounting. The paper does not report test-time compute budgets in terms of FLOPs or inference time. Since this is a real-robot evaluation rather than a pure model comparison, the relevant resource constraints are: (1) the quantity of human teleoperated demonstration data (fixed for all methods), (2) the number of online robot interaction episodes during RL (673 rollout trajectories for critic warm-up, then 12 new episodes per training cycle, with 50 optimization steps per cycle), and (3) the total online training steps (500 steps used for final evaluation). The number of evaluation trials per reported success rate is not explicitly stated, though Figure 5 (right) shows evaluation occurring over at least 175 episodes of online collection. For model comparisons, all policies use the same 5B-parameter architecture — the only difference during online RL is the 51.5M-parameter noise predictor added to GR-RL.
Cross-validation / statistical protocol. The paper does not report any cross-validation, train/validation/test splits, or statistical significance testing. The online RL evaluation appears to use the full set of collected episodes for both training and evaluation (since Figure 5, right, plots the binary success signal for all online episodes and the smoothed success rate over the same data). This is standard practice in real-robot RL where data is scarce and expensive, but it means the reported success rate of 83.3% is measured on episodes that overlap with the training distribution, and no held-out test set of novel shoe configurations or initial conditions is separately evaluated. The paper does not describe any protocol for ensuring that evaluation episodes are distinct from training episodes or for measuring generalization to unseen configurations, shoe types, or environmental conditions.

Main Quantitative Results

Overall Pipeline Performance (Figures 5 and 6)

The central result is the progressive improvement in success rate across the three pipeline stages, culminating in 83.3% for GR-RL:

GR-3 baseline: 45.7% success. The generalist VLA, trained with standard behavior cloning on raw teleoperated demonstrations, fails on more than half of evaluation trials. Figure 6 shows that the largest failure mode is during the threading stage (success drops from 90.6% at pick-up to 46.4% after threading — a 44.2 percentage point drop, represented by the hatched area for that stage). The handover and pull-out stages incur additional losses (final success: 45.7%).
Filtered BC: 61.6% success — a gain of 15.9 percentage points over GR-3. This improvement is attributed entirely to removing suboptimal transitions via the learned task progress filter. Figure 6 reveals that the filtering substantially reduces failure during threading: the success rate after threading rises to 63.8% (a 29.0 point drop from pick-up, compared to 44.2 points for GR-3). Performance at the pick-up stage also improves slightly (92.8% vs. 90.6%).
Filtered BC + Aug: 72.7% success — an additional gain of 11.1 percentage points from symmetry augmentation. Figure 6 shows that augmentation provides smaller but broad improvements across all stages: pick-up rises to 96.2%, threading success rises to 71.0% (a 25.2 point drop from pick-up), and subsequent stages also improve incrementally. The paper notes that augmentation "improves the model performance in all stages, although with a smaller magnitude" than filtering or online RL.
GR-RL (full pipeline): 83.3% success — an additional gain of 10.6 percentage points from online steering RL. Figure 6 shows that online RL provides its largest benefit at the threading stage (success rises to 83.3%, only a 6.3 point drop from the previous stage, compared to 25.2 points for the offline augmented model). The final pull-out stage sees the most modest improvement (83.3% from 72.7%, a 2.3 percentage point effect). Overall, the pipeline produces a total improvement of 37.6 percentage points over the GR-3 baseline.

The online training curve (Figure 5, right) shows the dynamics: an initial dip in the moving average success rate as the noise predictor begins exploring and the policy experiences distribution shift, followed by recovery and eventual growth beyond the offline baseline to over 90% on the smoothed curve. The final evaluated checkpoint at 500 training steps achieves 83.3% — the paper does not explain the gap between the smoothed performance peak (over 90%) and the evaluated checkpoint's performance.

Stage-by-Stage Failure Analysis (Figure 6)

The stacked bar visualization in Figure 6 reveals where each pipeline component reduces failure:

Pick-up stage: Success rates are high across all models (90.6%–97.9%). This stage is relatively easy — the robot needs to locate and grasp the correct shoelace end, which does not require millimeter precision. The small improvements from filtering, augmentation, and online RL are consistent with the observation that this stage is not the primary bottleneck.
Threading stage: This is the critical bottleneck and where the pipeline has its largest effect. The drop from pick-up to threading success is 44.2 percentage points for GR-3, 29.0 points for filtered BC, 25.2 points for filtered BC + aug, and only 6.3 points for GR-RL. The paper states: "Data filtering and online RL can largely reduce the failure during threading." This is consistent with the paper's argument that filtering removes the teleoperator's fumbling and missed-threading demonstrations, and that online RL aligns the policy's actions with the deployment dynamics specifically at the precision-critical insertion moment.
Handover stage: The bimanual handover (passing the shoelace from one gripper to the other after threading) shows moderate improvements across pipeline stages. The drop from threading to handover success is 12.3 points for GR-3 and narrows to 6.3 points for GR-RL. The paper does not provide a specific explanation for why handover improves, but it is plausible that better threading execution leads to more stable shoelace positioning for the handover.
Pull-out stage: The final stage (pulling the shoelace tight) shows the smallest improvements. The drop from handover to final success is approximately 4–6 points across all models, suggesting this stage is not the primary failure mode for any variant.

Progress Evaluator Comparisons (Figures 3 and 7)

Distributional critic vs. regression-based predictor (Figure 3). Figure 3 presents three qualitative examples of progress predictions over time for both the distributional critic (labeled "GR-RL") and the regression baseline. In all three cases, the regression-based predictor produces smoother, less responsive progress estimates:

Figure 3(a): When the shoelace misses the eyelet, the distributional critic's progress value drops sharply, while the regression predictor continues a gradual increase. During the successful threading that follows, the distributional critic rises sharply, while the regression predictor rises more gradually and with a delay.
Figure 3(b): During multiple failed handover attempts, the distributional critic shows distinct drops corresponding to each failure, while the regression predictor remains relatively flat.
Figure 3(c): When the robot intentionally puts down the shoelace to adjust its grasping pose (a temporarily regressive action that enables long-term progress), the distributional critic shows a value jump upward (recognizing the positive long-term effect), while the regression predictor is "almost flat during the adjustment." This is the most conceptually interesting example because it demonstrates that the distributional critic captures delayed-gratification actions — actions that look like moving away from the goal locally but are necessary for eventual success.

Distributional vs. non-distributional critic (Figure 7). Figure 7 compares progress predictions from the distributional critic (GR-RL) against a non-distributional critic trained with the same TD3+BC algorithm but using standard scalar regression. Both critics are evaluated on a high-quality successful trajectory. The paper reports:

"Due to the long horizon and binary sparse reward in our setting, the non-distributional critic suffers from severe over-estimation, especially in earlier parts of trajectories where the reward supervision signal is weak. The value prediction of our distributional critic falls in a predefined range, thus converging to a reasonable scale more robustly and demonstrating better alignment with the true temporal order."

The key observable in Figure 7 is that the non-distributional critic's predicted values drift substantially above 1.0 (reaching approximately 2.0 at some points on the y-axis), which is an over-estimation artifact — the true expected return from any point in a successful trajectory with a sparse binary reward of 1 cannot exceed 1 (the maximum possible return). The distributional critic, constrained to the [0, 1] support, produces values that increase monotonically and stay within physically meaningful bounds.

Qualitative Behavior Analysis (Figure 8)

Figure 8 presents seven qualitative examples of robust behaviors exhibited by GR-RL:

Figure 8(a): Threads shoelace for a shoe with a different color — demonstrating generalization beyond the training shoe appearance.
Figure 8(b): Regrasps the shoelace when it drops — demonstrating recovery from a common failure mode.
Figure 8(c): Re-attempts when the shoelace is not threaded precisely through the eyelet — demonstrating retry behavior after a near-miss.
Figure 8(d): Adjusts the grasp pose intentionally on the surface of the shoe — initially grasping far from the tip, the model drops the shoelace on the deformable shoe surface and regrasps closer to the tip before threading. This demonstrates deliberate environmental interaction to improve task conditions.
Figure 8(e): Reorients the shoe before threading — pulling the shoe from the left side to straighten it. This demonstrates spatial reasoning about the task setup.
Figure 8(f): Adjusts the positions of both the shoe and the shoelace before threading — pulling the shoe from the far side of the table, then repositioning the shoelace.
Figure 8(g): Pulls out the correct shoelace end from underneath another when the laces are crossed — demonstrating the ability to identify and manipulate the target shoelace even in visually ambiguous configurations.

These examples are qualitative by nature (still frames from execution videos) and are presented without quantitative frequency estimates. The paper does not report how often these recovery behaviors succeed vs. fail, or what fraction of the remaining 16.7% of failure cases correspond to situations where these recovery strategies were attempted but unsuccessful.

Ablation Studies and Robustness Checks

Progress evaluator type (distributional critic vs. regression baseline vs. non-distributional critic): Figure 3 compares the distributional critic's progress predictions against a regression-based temporal predictor, showing that the distributional critic detects subtle failures and long-term positive effects that the regression baseline misses. Figure 7 compares the distributional critic against a non-distributional critic trained with the same TD3+BC algorithm, showing that the non-distributional critic suffers from severe over-estimation in early trajectory segments while the distributional critic produces well-calibrated values within [0, 1]. The paper does not report the downstream effect on filtering quality or final policy success rate for policies trained with data filtered by the regression or non-distributional critics. This is a notable missing ablation — it would directly quantify the practical importance of the distributional design choice for the end-task performance.

Benefit of data filtering (GR-3 vs. Filtered BC): Figure 5 shows a 15.9 percentage point improvement from filtering alone (45.7% to 61.6%). This is not strictly an ablation (it's a stage comparison), but it serves as an existence proof that filtering matters. The paper does not ablate the filtering threshold δ — the sensitivity of the filtering rule to this hyperparameter is unknown. It also does not compare filtering using progress drops against alternative filtering criteria (e.g., velocity-based heuristics, human annotation of suboptimal segments, or filtering based on trajectory length or smoothness).

Benefit of symmetry augmentation (Filtered BC vs. Filtered BC + Aug): Figure 5 shows an 11.1 percentage point improvement from augmentation (61.6% to 72.7%). The paper does not ablate individual components of the augmentation (e.g., image flipping only, proprioception mirroring only, language flipping only) or compare against alternative augmentation strategies (e.g., color jittering, random cropping, action noise). The specific claim that morphological symmetry augmentation is effective is supported, but whether simpler augmentations would achieve similar gains is not tested.

Benefit of online steering RL (Filtered BC + Aug vs. GR-RL): Figure 5 shows a 10.6 percentage point improvement from online RL (72.7% to 83.3%). The paper does not ablate key design choices: (1) the effect of the divergence penalty (training without the max(1/2 ||ε_t||² - β, 0) term to see if unconstrained noise exploration degrades or improves performance), (2) the effect of the 50/50 off-policy/on-policy buffer sampling (vs. pure on-policy or pure off-policy), (3) the effect of excluding teleoperated data from the online buffer (vs. including it), (4) the effect of the noise-space critic distillation with 50/50 sampling from N(0,1) and the noise predictor (vs. only sampling from the noise predictor, or only from N(0,1)), and (5) the necessity of the two-critic architecture vs. a single critic. The paper also does not report an ablation where online RL is performed directly in action space (without the latent noise predictor) to quantify the claimed exploration difficulty — this would be a strong validation of the central argument for latent-space exploration.

Frozen VLM backbone during online RL: The paper states that only 51.5M parameters are updated during online RL (the noise predictor and critics), with the full 5B VLA backbone frozen. There is no ablation comparing frozen backbone vs. full fine-tuning or partial unfreezing. The paper argues this is for stability and computational efficiency but provides no empirical evidence that unfreezing would hurt.

Offline critic warm-up with rollout data: The paper collects 673 trajectories from the offline checkpoint to warm up the critic before online RL. There is no ablation on the necessity or quantity of this warm-up data — whether fewer trajectories would suffice, or whether skipping warm-up entirely would degrade performance.

Online RL data quantity and training steps: Figure 5 (right) shows the success rate trajectory over 175 episodes of online collection with 50 optimization steps per 12 episodes. The paper evaluates at 500 training steps (corresponding to approximately 120 episodes, or roughly 10 cycles of 12 episodes each). There is no systematic study of how performance varies with the number of online training steps, no report of training beyond 500 steps, and no discussion of whether performance plateaus, degrades, or continues improving with more online data.

Window size for moving average: The online training curve uses a window of 24 episodes for smoothing. There is no sensitivity analysis for this window size and how it affects the perceived training dynamics.

Trajectory optimization module at inference: The paper mentions incorporating "a trajectory optimization module that imposes constraints on jerk and temporal continuity to refine the predicted action chunks" during evaluation, but provides no ablation measuring its contribution to success rate. It is unclear whether the reported 83.3% would be achievable without this module, or whether the module contributes significantly to the online RL gain (by smoothing the policy's outputs and reducing the training-deployment mismatch further).

Critical Assessment

The experiments demonstrate conclusively that the multi-stage GR-RL pipeline substantially improves shoe lacing success rate compared to standard behavior cloning (45.7% → 83.3%). However, what precisely the experiments demonstrate about each individual claim requires careful parsing.

Does the filtering mechanism genuinely identify suboptimal transitions, or is it simply removing parts of trajectories that happen to look different? The qualitative evidence in Figure 3 is compelling — the progress evaluator drops at moments that human observers would identify as mistakes (shoelace dropped, missed eyelet, failed handover). However, the paper never evaluates whether the filtered dataset is actually cleaner by any independent criterion beyond the fact that training on it improves success rate. The improved success rate could be due to the filtering removing genuinely suboptimal transitions, or it could be because the filtering removes difficult transitions (where the task is harder for incidental reasons like poor lighting, awkward shoe positioning, or partial occlusion) and the policy simply learns on an easier subset of the data. The paper cannot distinguish between "cleaning noise" and "removing difficulty" because it has no ground-truth annotation of which transitions are suboptimal. The regression-vs-distributional comparison in Figure 3 partially addresses this by showing that the distributional critic is more sensitive to failures, but the downstream effect on filtering quality (what transitions each filter would keep and what the resulting policy performance would be) is not reported.

Does online RL genuinely align the training-deployment mismatch, or is it simply providing additional on-policy data for a standard RL improvement step? The paper argues that the improvement from online RL comes from closing the gap between predicted actions and post-processed executed actions. But the online RL stage uses a noise predictor that is trained to maximize Q-values — this is a standard RL improvement objective that would produce gains even if there were no deployment mismatch, simply because the critic identifies better actions than the behavior-cloned policy. The paper provides no ablation that isolates the "alignment" benefit from the "optimization" benefit. One could imagine an experiment: compare online RL with post-processed action execution (the actual system) against online RL with raw action execution (no temporal ensembling or trajectory optimization, removing the mismatch). If the gap between these two conditions closes during online RL, that would support the alignment claim. Without such an ablation, the improvement from online RL could equally be attributed to standard policy improvement through RL.

Does the pipeline genuinely produce generalization, or is the 83.3% measured on a narrow distribution? The paper evaluates on the same shoe lacing task that the demonstrations were collected for, and the online RL training and evaluation distributions appear to overlap (the paper uses the same set of rolled-out episodes for both training and evaluation, as evidenced by Figure 5 right plotting the binary success signals and smoothed rate over all online episodes). There is no separate test set of held-out shoe configurations, table positions, lighting conditions, or shoe types. Figure 8 qualitatively shows the policy handling shoes of different colors and recovering from various failure modes, but there is no quantitative generalization evaluation. The paper's claim to be "the first learning-based policy capable of lacing up shoes" is supported by the existence of an 83.3% success rate on shoe lacing trials, but the reliability of this capability across the range of conditions it would encounter in deployment is not quantified.

The stage-by-stage analysis (Figure 6) suggests the primary bottleneck is threading, and the pipeline's gains are concentrated there. This is internally consistent with the paper's narrative — threading is where millimeter precision matters most, and it's where filtering (removing demonstrations of missed threading) and online RL (aligning actions for precise insertion) should help most. But this pattern could also arise if the policy overfits to the specific threading conditions seen during training and the 83.3% reflects memorization rather than generalizable threading skill. The paper cannot rule this out without generalization testing.

The missing ablations weaken several specific claims. The claim that latent-space exploration is necessary (vs. action-space exploration) is argued from first principles but not tested — no comparison of online RL with action-noise exploration is reported. The claim that the divergence penalty is important for constraining exploration is not ablated. The claim that distributional critics are essential for progress evaluation is supported by the comparison with non-distributional critics (Figure 7) but the downstream effect on filtering quality is not reported — does a policy trained on data filtered by a non-distributional critic perform worse? By how much? Similarly, the regression-based temporal predictor (Figure 3) is shown to produce smoother progress estimates, but whether filtering based on those estimates would produce a worse policy is not tested.

The real-robot evaluation imposes inherent limitations that the paper does not fully acknowledge. The sample sizes are not reported (how many trials per reported success rate?). There are no error bars, confidence intervals, or statistical tests. The online RL training curve (Figure 5, right) shows substantial variance in per-episode success (dots are scattered across 0 and 1), and the smoothed curve's peak (~90%) differs from the final evaluated checkpoint's performance (83.3%), suggesting sensitivity to evaluation timing and checkpoint selection. Without variance estimates, a 37.6 percentage point total improvement is clearly meaningful, but sub-comparisons (e.g., is the 11.1 point gain from augmentation statistically distinguishable from the 10.6 point gain from online RL?) cannot be assessed.

The single-task nature of the evaluation limits claims about the framework's generality. The paper presents GR-RL as a framework for specializing generalist VLAs — but it only demonstrates specialization to one task. Whether the same pipeline would improve performance on other precision-manipulation tasks (suturing, soldering, assembly, button-sewing) is not tested. Whether the pipeline transfers to a different robot platform or a different base VLA is not tested. The paper's concluding claim that GR-RL "provides a step toward enabling generalist robot foundation models to specialize into reliable real-world experts" is aspirational — the experiments demonstrate this step for exactly one (task, robot, base model) combination.

The comparison to concurrent work (π*0.6) is conceptual, not empirical. The paper claims a specific advantage over concurrent work (that filtered BC produces a stronger base policy than advantage-conditioned denoising), but this claim is not tested — the two approaches are not compared on the same task with the same base model. This is not a weakness per se (concurrent work cannot always be experimentally compared), but the reader should understand this as a hypothesis, not a demonstrated finding.

A genuinely missing experiment: end-to-end comparison of the full pipeline against alternative data-cleaning approaches. The paper's core methodological contribution is the progress-evaluator-based filtering. An informative experiment would compare: (a) GR-RL with the distributional critic filter, (b) the same pipeline but with filtering performed by a human annotator watching videos and marking suboptimal segments, and (c) the same pipeline with no filtering (standard BC on all data). This would isolate whether the automated filter matches or exceeds human annotation quality, which is the practical claim the paper makes. Similarly, a comparison against oracle filtering (using ground-truth knowledge of which transitions led to eventual success, if available from the hindsight trajectory structure) would establish an upper bound on what filtering can achieve.

Summary of evidence strength. The experiments provide strong evidence that the three-stage pipeline improves success rate on shoe lacing by a large margin over standard behavior cloning. They provide moderate evidence that data filtering based on a learned progress evaluator contributes meaningfully to this improvement (through the stage-wise comparison in Figure 5). They provide weak evidence for the specific claim that online RL addresses the deployment mismatch specifically (rather than providing a standard RL improvement). And they provide no evidence for generalization of the method to other tasks, robots, or base models.

6. Limitations and Trade-offs

6.1 Single Task, Single Robot, Single Base Model — The Framework's Generality is Untested

The assumption or constraint. The paper presents GR-RL as a general framework for turning "a generalist vision-language-action (VLA) policy into a highly capable specialist for long-horizon dexterous manipulation" (Abstract). However, every experiment in the paper uses exactly one task (shoe lacing), one robot platform (ByteMini-v2), and one base VLA (GR-3 with Qwen2.5-VL-3B-Instruct backbone). The paper does not evaluate GR-RL on any other dexterous manipulation task, nor does it test the pipeline with a different base VLA (e.g., Octo, π0.5, OpenVLA), nor does it deploy on a different robot hardware platform. The conclusion acknowledges this implicitly by framing the work as "a small step towards capable real-world specialist robot policies" (Section 7), but the claim that the pipeline constitutes a general specialization recipe rests entirely on a single demonstration.

The consequence. A practitioner attempting to apply GR-RL to their own precision-manipulation task faces several unknowns: (1) Task structure dependence — shoe lacing has a natural decomposition into stages (pick, thread, handover, pull) with recognizable retry points that can be annotated during data collection. Tasks without this clean structure (e.g., deformable object folding, continuous insertion without discrete sub-goals) may not admit the same hindsight failure construction or progress-evaluator filtering. If teleoperators cannot identify clear "retry moments," the entire Stage 1 filtering mechanism loses its training signal. (2) Robot morphology dependence — the symmetry augmentation exploits the specific bimanual morphology of ByteMini-v2. A single-arm robot, a robot with asymmetric arms, or a mobile manipulator without wrist cameras would require a different augmentation strategy. (3) Base model dependence — GR-3's particular architecture (flow-matching action diffusion head, Qwen2.5-VL backbone, KV-cache sharing) is assumed throughout the pipeline. The online steering RL stage (Section 3.3) depends specifically on the existence of a diffusion denoising process with a latent noise space — it cannot be applied to, e.g., a policy that directly regresses action tokens or uses a different generative formulation. Whether the pipeline transfers or whether each component must be re-engineered per base model is unknown.

What evidence exists in the paper. None. The paper contains no ablation or evaluation on a second task, a second robot, or a second base VLA. The qualitative generalization examples in Figure 8 show different shoe colors and initial configurations, but these are all within the shoe lacing task distribution — not a different manipulation task. The paper also provides no theoretical argument for why the pipeline should be task- or model-agnostic.

Mitigation status. The paper does not attempt to mitigate this limitation. The authors position the result as a first demonstration rather than a fully validated framework, stating "We hope GR-RL provides a step toward enabling generalist robot foundation models to specialize into reliable real-world experts" (Abstract). This is candid but leaves the generality claim as an untested hypothesis. A practitioner should view GR-RL as a proven recipe for shoe lacing on ByteMini-v2 with GR-3, not a validated framework for arbitrary precision-manipulation specialization.

6.2 Difficulty Estimation Cost Is Not Accounted For — The Data Filtering Requires Full Offline RL Training

The assumption or constraint. The data filtering stage (Section 3.1) requires training a distributional critic via TD3+BC on the full demonstration dataset augmented with hindsight failure trajectories, then evaluating this critic on every transition to compute progress scores ρ_t and applying the filtering rule. This process is computationally expensive: it involves training a separate 5B-parameter model (the critic shares the architecture scale of the policy), and this training is essentially a prerequisite cost paid before any policy improvement is realized. The paper does not report the compute budget, wall-clock time, or GPU-hours required for this critic training step, nor does it factor this cost into any efficiency comparison. The headline number — 83.3% success — is achieved after offline critic training, offline BC training, symmetry augmentation, 673 rollout trajectories for warm-up, and hundreds of online RL episodes. The cost of the critic training alone could be comparable to or exceed the cost of training the baseline GR-3 policy.

The consequence. For a practitioner deciding whether to adopt GR-RL, the question is not simply "does 83.3% beat 45.7%?" but rather "does the total resource investment (compute, engineering time, robot time) justify the improvement?" The paper provides no accounting that would enable this calculation. Specifically: (1) The offline critic training adds a new model training step that does not exist in the standard behavior cloning pipeline — this is pure overhead if the filtering does not produce sufficient gains. (2) The filtering step itself is computationally cheap once the critic is trained (forward passes over the dataset), but the critic training is not. (3) The paper does not compare against simpler, cheaper filtering heuristics that might achieve some fraction of the gain at much lower cost — e.g., removing the last K timesteps before each annotated retry point, filtering based on end-effector velocity or jerk thresholds, or simply having a human watch the videos and mark suboptimal segments. A practitioner needs to know whether the distributional critic filter's sophistication is worth its cost relative to simpler alternatives.

What evidence exists in the paper. The paper reports no compute budgets for any stage. The 673 rollout trajectories collected for critic warm-up provide one concrete data point (Section 3.3), but this is only the warm-up — the initial offline critic training on the full augmented demonstration dataset is not quantified. The paper also does not ablate cheaper filtering alternatives against the distributional critic filter. The regression-based progress predictor (Figure 3) and non-distributional critic (Figure 7) are compared qualitatively but never used to actually filter data and train a downstream policy — so the performance gap between "smart filtering" and "cheap filtering" is unknown.

Mitigation status. The paper does not address this limitation, nor does it flag it as a concern for practitioners. The authors treat the offline critic training as a one-time cost whose magnitude is outside the scope of the paper. Section 7 notes behavior drifting during online RL as a limitation but does not mention the unaccounted offline costs. The practical implication is that the headline 37.6 percentage-point improvement comes with an unquantified upfront investment, and a deployment team would need to independently assess whether the pipeline is cost-effective for their setting.

6.3 Hard Problems Remain Unsolved — 16.7% of Trials Still Fail, and the Failure Modes Are Uncharacterized

The assumption or constraint. GR-RL achieves 83.3% success on shoe lacing, a substantial improvement over the 45.7% baseline. However, this means approximately 1 in 6 evaluation trials still fails, and the paper provides almost no analysis of what those failures look like, whether they are systematic (e.g., always on shoes with particular properties) or random, and whether the remaining errors are fundamentally different from those the pipeline already addressed. The stage-by-stage breakdown in Figure 6 shows the cumulative attrition through task stages, but this represents the average over all models — it does not isolate the failure modes specific to GR-RL, nor does it characterize the remaining failure cases qualitatively. The paper states in Section 7 that "one of the major issues of our current pipeline is the behavior-drifting problem. Given a sparse and noisy reward, our policy behavior could be unstable during online RL," but this identifies a mechanism (instability) rather than characterizing what the policy actually does when it fails.

The consequence. The 83.3% number, while impressive for a first demonstration, is not high enough for reliable autonomous deployment in many practical settings. If shoe lacing is one step in a larger assembly or caregiving pipeline, a 1-in-6 failure rate means the overall system reliability degrades multiplicatively. A practitioner needs to know: (1) Are the remaining failures concentrated in a particular sub-stage (e.g., 15 of the 16.7 points lost at threading)? If so, targeted improvements might close most of the gap. (2) Are failures correlated with environmental factors (shoe color, lighting, initial shoelace configuration, table position)? If so, the evaluation distribution may not match deployment conditions. (3) Do the failures produce unsafe or damaging behaviors (e.g., the robot tangling the shoelace, applying excessive force to the shoe, colliding with itself)? The paper provides no information to answer these questions.

More specifically, the "behavior-drifting problem" acknowledged in the Limitations (Section 7) suggests that the policy's behavior can change in undesirable ways during online RL due to the sparse reward signal. This implies the 83.3% checkpoint may not be stable — further online training might degrade performance if the policy drifts away from successful behaviors faster than the RL signal can correct it. The paper does not report training beyond 500 steps, so whether performance plateaus, oscillates, or degrades with more training is unknown. A practitioner running online RL in their own setting would need to monitor for this drift and decide when to stop training — the paper provides no guidance on how to detect or prevent it.

What evidence exists in the paper. Figure 6 provides stage-by-stage success rates, showing that for GR-RL, the largest remaining drop occurs at the threading stage (from 83.3% after threading to the same 83.3% final — note that the threading and final numbers are identical for GR-RL in Figure 6, meaning all failures occur during threading or earlier; the post-threading stages show no additional attrition). This suggests the 16.7% remaining failures are concentrated at the precision-critical threading moment and in pre-threading stages (pick-up success is 97.9%, so 2.1% fail there, and the remaining ~14.6% fail during threading). The paper provides no further characterization. The online training curve (Figure 5, right) shows that the smoothed success rate peaks above 90% before the evaluated checkpoint, indicating that (a) the policy achieved higher success rates at some points during training, and (b) the checkpoint selection matters — evaluating at a different step could yield different results. The binary success dots show sustained scatter between 0 and 1 throughout training, confirming that failures persist even as the average improves.

Mitigation status. The paper partially acknowledges the instability (behavior drifting) in Section 7 and suggests it "is possibly due to the limited capacity of the lightweight noise predictor, or the challenging credit assignment issue in the large latent action space," but provides no experimental investigation of these hypotheses and no proposed solution. The failure mode characterization is essentially absent — a significant gap for a paper claiming to produce a reliable specialist.

6.4 No Held-Out Generalization Evaluation — The 83.3% May Reflect Memorization, Not Robust Manipulation Skill

The assumption or constraint. The paper reports a single success rate (83.3%) evaluated on what appears to be the same distribution of shoe configurations, environmental conditions, and initial states used during training and online RL. There is no held-out test set of novel shoe types, unseen initial shoelace configurations, different table heights, varied lighting conditions, or new background clutter. The online RL evaluation (Figure 5, right) plots the binary success signal and smoothed success rate over all online episodes — implying that the episodes used for evaluation are the same ones used for training, or at minimum drawn from the same distribution without a held-out split. The paper does not describe any protocol for measuring generalization, and the qualitative examples in Figure 8, while showing different shoe colors and recovery behaviors, are cherry-picked success cases — they do not provide a quantitative generalization estimate.

The consequence. A practitioner deploying GR-RL for shoe lacing in a real-world setting (e.g., a shoe factory, an assistive robotics scenario) would encounter shoes that differ from the training data in multiple ways: different brands with different eyelet shapes and materials, different shoelace thicknesses and stiffnesses, different initial lacing states (partially laced, fully unlaced, tangled), different lighting, different table surfaces. The paper provides no evidence that the 83.3% success rate would hold under any of these distribution shifts. More critically, the online RL stage — which contributes 10.6 percentage points of the total gain — fine-tunes the noise predictor specifically on the distribution of states visited during online collection. If this distribution is narrow (e.g., one or two shoe types used for all online episodes), the noise predictor's improvements may not transfer to even slightly different initial conditions. The policy could have learned to thread these specific shoes reliably while remaining brittle to variation — a classic failure mode in robot learning when evaluation is not separated from training.

The paper's qualitative claim is that GR-RL demonstrates "reliable real-world" manipulation. But reliability implies consistent performance across the range of conditions the system will encounter in deployment. Without a held-out generalization test, the reported 83.3% is best understood as a training-distribution performance estimate, not a deployment-reliability guarantee.

What evidence exists in the paper. Figure 8(a) claims the policy can "thread shoelace for a shoe with a different color," but this is a single qualitative example — not a quantitative evaluation across multiple shoe types. No other generalization dimension is tested. The paper does not report: (1) the number of distinct shoes used during data collection, offline training, online RL, or evaluation; (2) whether the evaluation shoes were seen during training; (3) the range of initial shoelace configurations (e.g., how many different crossing patterns, how many different positions of the lace ends relative to the eyelets); (4) whether the lighting, background, and table position were varied systematically or remained constant. The absence of any held-out evaluation protocol means the paper cannot distinguish between "the policy learned to lace shoes" and "the policy learned to lace these particular shoes in this particular setup."

Mitigation status. The paper does not acknowledge this as a limitation. The Abstract claims GR-RL is a "step toward enabling generalist robot foundation models to specialize into reliable real-world experts" without qualifying that the demonstrated reliability has not been tested beyond the training distribution. A practitioner should treat the 83.3% figure as an in-distribution performance estimate and expect that performance on genuinely novel shoes or configurations will be lower — potentially substantially lower — depending on how much variation was present in the (unreported) training distribution.

6.5 Latency and Online Interaction Burden — Wall-Clock Constraints Are Not Discussed

The assumption or constraint. The GR-RL pipeline makes a fundamental tradeoff between computation and performance that manifests as real wall-clock time during deployment and training. Three specific sources of latency exist: (1) Online RL data collection — the system requires 673 rollout trajectories for critic warm-up (Section 3.3), then hundreds more episodes during online training, each involving physical robot execution of the full shoe lacing task (which takes tens of seconds per attempt). The total robot-hours required for online RL is substantial and is not reported. (2) Inference-time cost — the GR-RL policy during deployment uses both the noise predictor (which requires an additional forward pass) and the trajectory optimization module (which "imposes constraints on jerk and temporal continuity to refine the predicted action chunks," Section 5). These add latency to each control step, and for millimeter-precision manipulation where control frequency matters, this overhead could affect performance. (3) No parallelization for online interactions — unlike offline training, the online RL stage is inherently sequential: the robot must physically execute an episode, collect the outcome, update the model, and repeat. This cannot be parallelized across multiple robots without a multi-robot data collection setup, which the paper does not describe or evaluate.

The consequence. For a practitioner, the relevant metric for deploying GR-RL is not just the final 83.3% but the total resource investment to get there. The online RL stage consumed at minimum: 673 warm-up episodes + 120 episodes during the first 10 training cycles (12 episodes per cycle, 10 cycles to reach 500 steps) + additional episodes beyond the evaluated checkpoint if training continued. At, say, 30–60 seconds per shoe lacing attempt (a reasonable estimate for a long-horizon bimanual task), this represents 7–13+ hours of continuous robot operation, not counting resets between episodes, hardware failures, or human supervision. This is feasible for a research lab but may be prohibitive for a production deployment where robot time is the primary constraint.

The inference-time latency from the noise predictor and trajectory optimization module is likely modest relative to the VLM backbone (51.5M parameters vs. 5B), but the paper provides no timing measurements. For a control loop running at, say, 10–20 Hz (typical for visuomotor policies), every additional millisecond of inference latency reduces the effective control bandwidth. In millimeter-precision threading, where the robot's approach vector must be accurate to within the eyelet diameter (likely 3–5 mm), reduced control frequency could degrade precision — the same latency constraints the online RL was meant to address.

What evidence exists in the paper. None. The paper provides no timing data for: (a) inference latency of the base GR-3 policy vs. GR-RL with noise predictor vs. GR-RL with noise predictor and trajectory optimization; (b) wall-clock time per shoe lacing episode during data collection, evaluation, and online RL; (c) total robot-hours consumed by the online RL stage; (d) training time (GPU-hours) for the offline critic, the filtered BC policy, and the online RL updates. The paper also does not address the engineering effort required to maintain a reliable online RL loop on physical hardware (reset mechanisms, safety monitoring, human oversight), which is a substantial hidden cost in real-world RL deployments.

Mitigation status. The paper does not discuss latency or online interaction burden, treating the online RL stage as a one-time training cost that is outside the scope of the evaluation. However, for a paper that frames itself as enabling "reliable real-world experts" (Abstract), the practical cost of producing that expert is directly relevant to adoption. A practitioner would need to independently estimate whether the 37.6 percentage-point improvement justifies the unquantified robot-hours, compute-hours, and engineering effort required by the GR-RL pipeline compared to alternatives (e.g., collecting more demonstrations, using a larger base model, or accepting lower success rates).

6.6 The Behavior Drifting Problem During Online RL Remains Unresolved

The assumption or constraint. The paper explicitly identifies behavior drifting as a "major issue" in the Limitations (Section 7):

"Given a sparse and noisy reward, our policy behavior could be unstable during online RL. This is possibly due to the limited capacity of the lightweight noise predictor, or the challenging credit assignment issue in the large latent action space."

This limitation is structural to the online RL design. The noise predictor π_θ' (51.5M parameters) must learn to perturb the latent noise fed to a frozen 5B-parameter diffusion model in a way that improves long-horizon task success, using only a binary sparse reward at episode end. The credit assignment problem — determining which of the many noise perturbations across the trajectory contributed to the eventual success or failure — is extremely challenging. The sparse reward provides no intermediate feedback, so the noise predictor must rely on the critic's value estimates, which themselves are bootstrapped and subject to error. If the critic overestimates the value of a particular type of perturbation (analogous to the over-estimation documented in Figure 7 for the non-distributional critic), the noise predictor may drift toward producing those perturbations even though they do not genuinely improve success.

The consequence. The behavior drifting problem means the online RL stage is not a monotonic improvement process — performance can degrade as well as improve. The training curve in Figure 5 (right) shows this: the moving average success rate dips at the beginning (the acknowledged "distribution shift from offline to online RL"), recovers and grows to over 90%, but the final evaluated checkpoint at 500 steps achieves only 83.3%. The gap between the moving average peak and the evaluated checkpoint — approximately 7+ percentage points — suggests that performance varied substantially during training, and that the specific checkpoint selected for evaluation was not the peak-performing one. A practitioner running the pipeline would need to decide when to stop training, but the paper provides no criterion for doing so (early stopping based on a validation metric, a maximum training budget, or a stability heuristic). Without such a criterion, the practitioner risks either stopping too early (leaving performance on the table) or too late (suffering from behavior drift that degrades the policy).

More fundamentally, the drifting problem means the online RL stage may not be stable in a production setting where the robot operates continuously and the policy is periodically updated. If a deployed system continues online RL to adapt to new shoes or conditions, the behavior drift could cause the policy to temporarily or permanently degrade on the original task distribution. The paper provides no analysis of whether the drifted behavior is recoverable (e.g., by reverting to an earlier checkpoint) or whether it represents a permanent loss of capability.

What evidence exists in the paper. The training curve (Figure 5, right) is the primary evidence. The smoothed success rate (window size 24) shows a clear non-monotonic trajectory: initial decline, rapid improvement, peak above 90%, and then the evaluated checkpoint at 83.3%. The paper does not show training beyond 175 episodes or 500 optimization steps, so it is unknown whether performance would continue to oscillate, plateau, or diverge further. The binary success dots show substantial variance throughout — successes and failures are interleaved even late in training — consistent with a stochastic process where the policy sometimes drifts into lower-success regimes and sometimes recovers. The paper acknowledges the problem explicitly in Section 7 but does not quantify how often the policy drifts, how much performance degrades during drift episodes, or what triggers recovery.

Mitigation status. The paper identifies the problem but offers only speculative hypotheses for its cause ("possibly due to the limited capacity of the lightweight noise predictor, or the challenging credit assignment issue"), with no experimental investigation and no proposed solutions. It flags the issue for "future study" (Section 7). A practitioner adopting GR-RL should expect to encounter behavior instability during online RL and should budget for: (a) monitoring the training curve to detect drift, (b) checkpointing frequently to enable rollback, (c) potentially implementing additional stabilization mechanisms (e.g., more conservative KL penalties, smaller update steps, or trust-region constraints beyond the existing divergence penalty) not described in the paper, and (d) accepting that the final performance may be sensitive to the stopping point and may not be reproducible across training runs.

7. Implications and Future Directions

How This Work Changes the Landscape

GR-RL does not introduce a new model architecture, a new RL algorithm, or a new representation. It introduces a new mental model for how to build capable robot specialists from generalist foundation policies — and that mental model is data-centric rather than model-centric. This represents a meaningful shift in emphasis, not a paradigm revolution, but the shift has concrete consequences for what research directions the community should prioritize.

The core reframing: specialization is curation + alignment, not architecture search. Before GR-RL, the dominant approaches to improving robot manipulation performance fell into three buckets: scale up the model and training data (the VLA scaling hypothesis), design better action representations or policy architectures, or apply RL to improve beyond demonstrations. GR-RL argues—through its results, not just its rhetoric—that for precision manipulation at the frontier of what VLAs can do, the quality and alignment of the training data matters more than any of these factors. The same base model (GR-3, 5B parameters), the same demonstration dataset, and the same action representation produce 45.7% success with naive behavior cloning and 83.3% when the data is cleaned, augmented, and aligned — a 1.82× improvement without changing the model. This is not an argument against scaling or architecture research, but it is a strong argument that data quality work deserves at least as much attention as model design, and perhaps more at the precision frontier.

This aligns with trends in other domains (NLP's recognition that data curation often dominates model size for specialized tasks, computer vision's use of automated data cleaning pipelines), but the robot learning community has been slower to internalize this because robot data is expensive to collect and the intuition that "every demonstration is precious" militates against aggressive filtering. GR-RL provides a concrete counterexample: throwing away data (the filtered transitions) improved performance by 15.9 percentage points. This reverses the default assumption from "keep everything" to "audit everything, keep only what genuinely contributes to task progress."

Reconciling prior contradictions about RL for manipulation. The paper's staged approach helps explain a tension that has existed in real-world robot RL: some works report large gains from online RL fine-tuning of behavior-cloned policies [43, 44, 53], while others find that online RL is unstable or provides marginal benefits. GR-RL suggests that the difference may be attributable to the quality of the base policy before RL begins. If behavior cloning is performed on noisy, suboptimal demonstrations (the standard approach), the resulting policy has learned to reproduce hesitation, retries, and errors — and online RL must both unlearn these behaviors and learn better ones, a much harder exploration problem. By filtering first and applying online RL only to a clean base policy (72.7%), GR-RL achieves rapid, stable improvement with only 51.5M trainable parameters. This implies that offline data curation is not merely helpful for online RL — it may be a prerequisite for making online RL feasible on long-horizon precision tasks. The finding also provides a lens for re-evaluating prior negative results: did those systems fail because RL is fundamentally hard for manipulation, or because they started from policies that had memorized suboptimal behaviors?

Verifier over-optimization analogue in the manipulation domain. A subtle but important connection: the machine learning community has extensively studied how reward models in RLHF can be over-optimized, leading to policies that score highly under the learned reward but perform poorly on the true task. GR-RL documents a structurally similar phenomenon in robot manipulation, but with a twist: the problem is not over-optimization of a learned reward model during online RL (the sparse binary reward used for online RL is ground-truth), but rather the memorization of suboptimal behaviors during imitation learning. The offline filtering stage can be understood as a defense against a form of "over-imitation" — just as a policy can over-optimize a learned reward, a behavior-cloned policy can over-imitate a noisy demonstration distribution, reproducing errors that happen to be present in the training data. The filtering mechanism acts as an automated "reward-informed data auditor" that prevents this, analogous to how KL penalties or early stopping prevent reward over-optimization in RLHF. This connection suggests that insights from the reward-model robustness literature (ensembling, calibration, constrained optimization) might transfer to the problem of demonstration quality auditing.

Research directions that become more attractive, and those that become less so. The paper's results make data-centric specialization pipelines a more attractive research investment: automated demonstration quality assessment, task-progress-aware data curation, physically-grounded augmentation for manipulation, and sample-efficient parameter-efficient RL for closing sim-to-real or train-to-deploy gaps. It makes architecture-only approaches to precision manipulation (e.g., larger VLAs, better action tokenizers, more expressive diffusion models) relatively less attractive as standalone solutions — the paper shows that even a state-of-the-art 5B VLA achieves less than 50% success without data curation, implying that architectural scaling alone is unlikely to cross the precision barrier for dexterous tasks. It also makes simulation-only RL for dexterous manipulation slightly less attractive by demonstrating that real-world online RL is feasible and effective when properly staged, though the high robot-hour cost (Section 6.5) means that sim-to-real approaches remain essential for tasks where real-world interaction is even more expensive than shoe lacing.

The difficulty estimation parallel. For readers familiar with the compute-optimal test-time scaling literature (where models estimate problem difficulty and allocate computation accordingly), GR-RL's task progress evaluator serves an analogous function: it estimates demonstration quality and allocates training data accordingly (keeping good transitions, discarding bad ones). Both involve using a learned model to audit data rather than generate it, and both find that adaptive, difficulty-or-quality-aware allocation substantially outperforms uniform treatment. This is an underexplored paradigm — using learned critics not as policy teachers but as data diagnosticians — and GR-RL demonstrates it can work at the frontier of physical manipulation.

Follow-Up Research This Work Enables

1. Quantifying the downstream effect of filtering quality on policy performance. The paper shows that distributional critics produce better progress estimates than regression-based predictors (Figure 3) and non-distributional critics (Figure 7), but it never closes the loop: a policy trained on data filtered by the regression predictor or the non-distributional critic would reveal how much the filtering quality actually matters for end-task performance. A direct experiment: train three filtered BC policies, each using data filtered by (a) the distributional critic, (b) the non-distributional critic, and (c) the regression-based temporal predictor, all with the same filtering threshold δ. Measure the success rate of each on shoe lacing. This would convert the qualitative evidence in Figures 3 and 7 into a quantitative claim about the practical value of distributional critics for data filtering. A negative result — if all three filters produce similar policy performance — would suggest that the filtering rule (detecting drops above threshold δ) is robust to the specific progress estimator, and that simpler, cheaper estimators suffice. Either outcome is informative.

2. Ablating the necessity of latent-space exploration vs. action-space exploration. The paper's central argument for the online steering RL design is that action-space exploration is hopeless for millimeter-precision tasks because random action perturbations almost never produce success. This is argued from first principles but never tested empirically. A direct experiment: run the identical online RL pipeline (same buffer management, same 50/50 sampling, same divergence penalty, same number of training steps) but with the noise predictor replaced by an action perturbation module that adds Gaussian noise directly to the diffusion model's output actions, constrained by a similar KL-style penalty to keep actions near the offline distribution. Compare the learning curve (success rate vs. online episodes) and final performance against the latent-space approach. If action-space exploration fails to improve or degrades performance, the paper's core design claim is validated and the community gains clear guidance for when latent-space methods are necessary. If action-space exploration works comparably well, the conceptual justification for the more complex latent-space approach weakens substantially, and practitioners can use a simpler implementation.

3. Stress-testing the pipeline on a second precision-manipulation task with different structure. The paper evaluates on exactly one task (shoe lacing), which has a natural linear structure (pick → thread → handover → pull) with cleanly identifiable retry points that enable hindsight failure creation. A critical follow-up would apply the identical GR-RL pipeline to a second long-horizon dexterous manipulation task with different structural properties — for example, button sewing (requires piercing fabric at precise locations, handling a deformable thread, tying a knot), cable insertion (USB or Ethernet plug insertion requiring sub-millimeter alignment and force sensing), or suturing on a phantom (medical training task with deformable tissue, curved needle manipulation, and knot-tying). Each task tests different aspects of the pipeline: button sewing tests whether the symmetry augmentation transfers (buttons can be on either side, but the sewing motion is not strictly symmetric), cable insertion tests whether the progress evaluator can detect insertion failures from visual and proprioceptive signals alone (without the deformable object tracking that makes shoelace progress visible), and suturing tests the full pipeline under domain shift from industrial to medical manipulation. The key measurement: does the staged pipeline produce gains of comparable magnitude relative to the behavior cloning baseline across these tasks, or does the effectiveness of each stage depend strongly on task structure?

4. Replacing the retry-keyframe requirement with automated suboptimality detection. The offline filtering stage depends on teleoperators annotating retry keyframes during data collection — pressing a button when they make a mistake, which enables hindsight failure trajectory creation. This is a reasonable requirement for research data collection but limits the pipeline's applicability to existing demonstration datasets (which lack retry annotations) and to teleoperation interfaces that support real-time annotation. A valuable extension: automatically detect suboptimal segments without retry annotations. One approach: instead of truncating at annotated retry keyframes, use the distributional critic itself in an iterative procedure — train an initial critic on the full successful trajectories only, use it to score transitions, identify drops that exceed some threshold as candidate suboptimal segments, create hindsight failures by truncating at those drops, retrain the critic on the augmented dataset, and iterate. This bootstrapping approach would eliminate the annotation requirement entirely. The experiment would compare the automated approach against the annotated-retry approach on shoe lacing, measuring both filtering quality (do the automatically identified drops correspond to human-annotated retry points?) and downstream policy performance. A positive result would make the pipeline applicable to any demonstration dataset, dramatically expanding its practical reach.

5. Characterizing the behavior drifting problem and testing stabilization mechanisms. The paper identifies behavior drifting during online RL as "one of the major issues" (Section 7) but provides only speculative hypotheses for its cause and no experimental investigation. A systematic study would: (a) Run online RL for substantially more steps (e.g., 2,000–5,000) to characterize the long-term training dynamics — does performance oscillate around a mean, drift monotonically downward, or plateau? (b) Track diagnostic metrics during training, including the average noise predictor output norm (to detect divergence from the trust region), the KL divergence between the noise predictor's output distribution and N(0,1), and the variance of the critic's Q-value estimates (to detect overestimation). (c) Test stabilization mechanisms: a stronger divergence penalty (larger c or smaller β in Equation 3), periodic resetting of the noise predictor to N(0,1) output, ensembling multiple noise predictors, or adding a behavior cloning regularization term to the noise predictor loss (analogous to TD3+BC's approach for offline RL). The measurement would be the stability of the success rate over extended training, quantified as the variance of the moving average and the maximum drawdown from peak performance. This would provide practical guidance for practitioners deploying online RL and scientific insight into why latent-space policy optimization becomes unstable.

6. Quantifying the generalization envelope — what distribution shift can GR-RL tolerate? The paper provides no held-out generalization evaluation, making the 83.3% success rate an in-distribution estimate of unknown scope. A systematic generalization study would evaluate GR-RL (the final 83.3% checkpoint) on held-out conditions across multiple axes: (a) Shoe variation: 5–10 distinct shoe types varying in color, size, material stiffness, and eyelet shape/number, none seen during training or online RL. (b) Shoelace variation: different lace thicknesses, materials (cotton, synthetic, round, flat), and tip types (aglet presence/absence). (c) Initial configuration variation: different crossing patterns of the laces, different positions of the lace ends (tucked under the shoe, hanging off the table, tangled), different shoe orientations and table positions. (d) Environmental variation: different lighting conditions (bright, dim, directional), different background clutter, different table surface textures. For each axis, measure the success rate and compare against the in-distribution 83.3%. This would produce a quantitative "generalization map" that tells practitioners what variation the policy can handle robustly and what variation requires further data collection or adaptation. It would also reveal whether the pipeline's gains are concentrated on in-distribution performance (suggesting overfitting) or extend to meaningful generalization (suggesting the policy learned a robust manipulation skill). A negative result — substantial degradation on even modest distribution shift — would indicate that the online RL stage overfit to the specific shoes and conditions used during interaction, and that the pipeline needs an explicit generalization stage (e.g., domain randomization during online RL, or multi-shoe training).

Practical Applications and Downstream Use Cases

1. Industrial shoe manufacturing quality assurance and rework. In shoe factories, lacing is often performed manually or with specialized jigs because existing automation (motion-planning-based systems with predefined patterns [39–41]) cannot handle the variability of shoe styles, materials, and initial lace states. GR-RL's 83.3% success rate on a learning-based system — with demonstrated recovery behaviors (regrasping after drops, re-attempting after missed threading, adjusting grasp pose) — suggests a path toward flexible automation for small-batch or high-variability shoe production lines. The specific benefit: a single GR-RL-specialized policy could potentially handle multiple shoe models without reprogramming, reducing the changeover time that makes traditional automation uneconomical for short production runs. The 83.3% success rate is not yet high enough for fully autonomous deployment without human oversight, but a workstation where the robot performs lacing and a human operator intervenes on the remaining 16.7% of cases would already reduce manual labor by 5× relative to fully manual lacing. The key deployment consideration is generalization to novel shoe types (Section 6.4) — a factory considering GR-RL would need to evaluate on their specific product line, as the paper's generalization envelope is uncharacterized.

2. Assistive robotics for activities of daily living. Shoe lacing is a canonical activity of daily living (ADL) that is challenging for individuals with limited fine motor control, such as those with Parkinson's disease, arthritis, or spinal cord injuries. A wheelchair-mounted or home-assistant bimanual robot equipped with GR-RL could provide independence in self-dressing tasks. The paper's ByteMini-v2 platform is a mobile manipulator (wheeled base with torso lift), making it physically suitable for approaching a seated person and manipulating shoes at foot level. The 83.3% success rate means approximately 5 out of 6 attempts succeed autonomously; for the remaining 1 in 6, the robot could request human assistance or re-attempt with a different strategy. The recovery behaviors documented in Figure 8 (regrasping dropped laces, re-attempting after missed threading, adjusting shoe orientation) are particularly relevant for assistive settings where the initial shoe position and lace configuration may be unpredictable. The primary deployment barrier is safety — the robot operates in close proximity to a person's feet, and the paper provides no safety analysis (force limits, collision detection, emergency stop behavior). A safety-audited version with compliant control and proximity-based speed reduction would be required before any real-world assistive deployment.

3. Data generation for self-improving robot foundation models. The paper's pipeline can be viewed as a high-quality data generation engine: the final GR-RL policy (83.3% success) produces successful shoe lacing trajectories that are substantially cleaner than the original human demonstrations (which contained hesitations, retries, and errors). These trajectories could be fed back into the generalist VLA training pipeline — essentially, using GR-RL to generate expert demonstrations for shoe lacing (and potentially other precision tasks) that improve the next generation of generalist policies. The paper itself suggests this in the Limitations: "distilling the improved policy into the base VLA could be a potential direction for obtaining both capable and general manipulation policies" (Section 7). The practical workflow: use GR-RL to generate 1,000+ successful shoe lacing trajectories across diverse shoe types and conditions, filter these to keep only smooth, fluent executions, and add them to the training dataset for the next version of GR-3 or another generalist VLA. The benefit is that the specialist knowledge (millimeter-precision threading, recovery behaviors, bimanual coordination) gets distilled back into the generalist, potentially improving the generalist's performance on shoe lacing without requiring the full GR-RL pipeline at deployment time. The 83.3% success rate means that approximately 5 of every 6 generation attempts produce usable data, making this a cost-effective data augmentation strategy at scale.

4. Parameter-efficient online adaptation for deployed robot fleets. The online RL stage of GR-RL updates only 51.5M parameters (the noise predictor and critics) while leaving the 5B VLA backbone frozen. This is a highly efficient adaptation mechanism: a fleet of deployed robots could share a common base VLA (trained centrally) and each adapt its noise predictor to its specific operating conditions (shoe types, lighting, table setup, wear and tear on the robot's joints) using a modest number of online interactions. The architecture supports this naturally because the noise predictor is small and operates in a latent space that is shared across the fleet (the VLM backbone's output representation). A practical system: a central server distributes the frozen VLA backbone plus an initial noise predictor; each robot collects 50–100 episodes in its deployment environment, fine-tunes its own noise predictor using the GR-RL online RL procedure, and achieves locally optimized performance. The paper's training curve (Figure 5, right) suggests that 100–150 online episodes may be sufficient for substantial improvement (the moving average exceeds 80% within roughly 75 episodes). This fleet-learning architecture would address a key challenge in robot deployment — the distribution shift between the training facility and each deployment site — without requiring expensive per-site data collection or full model retraining.

When to Prefer This Method

The paper positions GR-RL against two alternatives: (1) standard behavior cloning on raw human demonstrations (represented by the GR-3 baseline at 45.7%), and (2) classic motion-planning approaches with predefined action primitives [39–41]. The paper also contrasts with concurrent work (π*0.6) that uses advantage-conditioned denoising instead of filtered behavior cloning. However, since this contrast is conceptual rather than empirical, a formal decision matrix is not supported by the paper's evidence. Instead, the paper articulates specific conditions under which GR-RL's approach is valuable, and conditions under which simpler alternatives may suffice. The following guidance is grounded in the paper's explicit claims and results.

Prefer the GR-RL specialization pipeline over standard behavior cloning when:

The task requires millimeter-level precision and involves deformable objects where human demonstrations are likely to contain hesitations, retries, and suboptimal fragments (the paper demonstrates this for shoe lacing; tasks with similar precision requirements — threading, insertion, suturing — likely share this property).
The base generalist VLA achieves non-trivial but unreliable performance (GR-3's 45.7% on shoe lacing) — high enough that some correct behaviors are present in the demonstrations, but low enough that significant improvement is possible through data curation.
Teleoperators can annotate retry keyframes during data collection, enabling hindsight failure trajectory creation for the offline critic training.
The robot platform has exploitable morphological symmetries (bimanual, symmetric arms with wrist cameras) that enable physically-grounded data augmentation.
A modest number of online robot interactions (hundreds of episodes) is feasible — the pipeline uses 673 warm-up trajectories plus ~120 episodes for the first 500 online RL steps.
The deployment conditions are similar enough to training conditions that online RL can close the remaining gap without requiring extensive generalization to novel environments (the paper's generalization envelope is uncharacterized beyond qualitative examples).

Prefer standard behavior cloning (with or without simple filtering heuristics) when:

The task does not require millimeter precision (e.g., pick-and-place of rigid objects, pushing, coarse manipulation), and human demonstrations are likely to be near-optimal throughout.
The base VLA already achieves high success rates without filtering (e.g., >80%), leaving limited room for improvement from data curation.
Online robot interaction is prohibitively expensive or infeasible (e.g., safety-critical tasks, extremely slow execution, remote deployment without human oversight).
Retry keyframe annotation is impossible (e.g., existing demonstration datasets without annotations, teleoperation interfaces without real-time marking capability).
Task success is not well-captured by a sparse binary reward, making the critic training and progress evaluation unreliable (e.g., tasks with continuous quality metrics rather than binary success/failure, tasks where "progress" is not monotonic).

Prefer offline-only filtering and augmentation (without online RL) when:

The filtered-and-augmented behavior cloning policy already achieves sufficient success rates for the deployment requirements (72.7% on shoe lacing may be adequate for some applications, especially if human oversight is available for the remaining failures).
Online robot interaction is feasible but the behavior drifting problem documented in the paper is unacceptable for the deployment setting (e.g., continuous autonomous operation where policy degradation cannot be tolerated).
The deployment mismatch (temporal ensembling, trajectory optimization) is minimal — i.e., the actions executed at deployment closely match the actions predicted during training, making online alignment unnecessary. The paper does not provide a method for diagnosing whether this mismatch is significant without running online RL, so this condition requires engineering judgment.

🎯 Pitch

1. Executive Summary

2. Context and Motivation

The Core Problem: Generalist VLA Policies Are Not Reliable Specialists

Why This Gap Matters

Where Prior Approaches Fall Short

How This Paper Positions Itself

3. Technical Approach

3.1 Reader Orientation

3.2 Big-Picture Architecture (Diagram in Words)

3.3 Roadmap for the Deep Dive

3.4 Detailed, Sentence-Based Technical Breakdown

3.4.1 Model Architecture: Mixture-of-Transformer VLA with Distributional Critic

3.4.2 Stage 1: Offline RL Critic Training and Data Filtering

3.4.3 Stage 2: Morphological Symmetry Augmentation

3.4.4 Stage 3: Online Steering RL for Deployment Alignment

3.4.5 Inference-Time Deployment and Trajectory Optimization

3.4.6 Summary of Design Choices and Their Justifications

4. Key Insights and Innovations

Innovation 1: Rewriting the Role of RL — From Policy Optimizer to Data Quality Auditor

Innovation 2: Diagnosing and Treating the Deployment Mismatch as a First-Class Problem

Innovation 3: A Data-Centric Recipe That Reframes Specialization as Curation + Alignment

Innovation 4: The Distributional Critic as a Progress Detector — A Diagnostic Tool, Not Just a Value Function

5. Experimental Analysis

Evaluation Methodology

Main Quantitative Results

Overall Pipeline Performance (Figures 5 and 6)

Stage-by-Stage Failure Analysis (Figure 6)

Progress Evaluator Comparisons (Figures 3 and 7)

Qualitative Behavior Analysis (Figure 8)

Ablation Studies and Robustness Checks

Critical Assessment

6. Limitations and Trade-offs

6.1 Single Task, Single Robot, Single Base Model — The Framework's Generality is Untested

6.2 Difficulty Estimation Cost Is Not Accounted For — The Data Filtering Requires Full Offline RL Training

6.3 Hard Problems Remain Unsolved — 16.7% of Trials Still Fail, and the Failure Modes Are Uncharacterized

6.4 No Held-Out Generalization Evaluation — The 83.3% May Reflect Memorization, Not Robust Manipulation Skill

6.5 Latency and Online Interaction Burden — Wall-Clock Constraints Are Not Discussed

6.6 The Behavior Drifting Problem During Online RL Remains Unresolved

7. Implications and Future Directions

How This Work Changes the Landscape

Follow-Up Research This Work Enables

Practical Applications and Downstream Use Cases

When to Prefer This Method

Adjacent in domain

Stabilizing Reinforcement Learning with LLMs: Formulation and Practices

ARM-Thinker: Reinforcing Multimodal Generative Reward Models with Agentic Tool Use and Visual Reasoning