ArXiv: 2507.15493

🎯 Pitch

GR-3 turns a 3B-parameter vision-language model into a robot policy that follows abstract instructions like “put the animal with tentacles into the carton” with 77.1% success—nearly double the baseline—by co-training on web-scale vision-language data directly alongside robot trajectories. It then bridges remaining generalization gaps with only 10 human demonstrations per novel object, boosting unseen-object success from 57.8% to 86.7%.

1. Executive Summary

This report introduces GR-3, a 4B-parameter vision-language-action (VLA) model built on a pre-trained VLM (Qwen2.5-VL-3B-Instruct) with a flow-matching action diffusion transformer, and a custom bi-manual mobile robot platform called ByteMini for manipulation tasks. GR-3 achieves its generalization capabilities through a multi-faceted training recipe—co-training with vision-language data (dynamically mixing web-scale vision-language tasks with robot trajectories to transfer semantic knowledge into action prediction), efficient few-shot fine-tuning from human trajectory data (adapting to novel objects with only 10 VR-collected human demonstrations per object, collected at ~450 trajectories/hour), and imitation learning with robot trajectory data (incorporating task-status auxiliary supervision and QK-norm-inspired RMSNorm in the DiT blocks to improve instruction following and training stability). Across three real-world task suites, GR-3 substantially outperforms the π₀ baseline: it boosts success rate on unseen instructions from 40% to 77.1%, achieves 97.5% instruction-following success on long-horizon table bussing (vs. 53.8% for π₀), and reaches 86.7% average task progress on dexterous cloth hanging. For few-shot adaptation, it improves unseen-object success rate from 57.8% (zero-shot) to 86.7% using only 10 human trajectories per object, establishing that web-scale vision-language co-training enables strong zero-shot generalization while human trajectory data provides a sample-efficient path for bridging the remaining generalization gap.

2. Context and Motivation

The Core Problem: Building Robots That Actually Follow Instructions in the Wild

The central problem this paper tackles is deceptively simple to state but extraordinarily difficult to solve: how do you build a robot policy that reliably follows natural language instructions when deployed in environments, with objects, and under conditions it has never seen before? This is the generalization challenge that separates laboratory robotics demonstrations from genuinely useful household assistants. GR-3 is an attempt to bridge that gap.

The paper frames this around three specific failure modes that plague current approaches (Section 1):

First, instruction following breaks down on out-of-distribution language. Current VLA models can follow instructions like "put the apple in the basket" when trained on similar phrasing with similar objects. But they fail when asked to "put the animal with tentacles into the carton" (an octopus toy, unseen in robot training data) or "put the largest object into the carton" (requiring comparative reasoning about size). These failures aren't merely inconvenient — they reveal a fundamental lack of semantic understanding. The model has learned surface-level correlations between instruction tokens and actions, not grounded comprehension of concepts like spatial relationships, object properties, or categorical reasoning. The paper cites specific prior evidence that this is a known weakness: RDT-1B and RT-2 both struggle with instructions involving "novel object categories that are unseen in robot trajectory data and/or complex concepts that require sophisticated reasoning."

Second, long-horizon tasks suffer from compounding errors. In the table bussing experiment (Section 5.2), a robot must navigate a mobile base between stations, pack food into containers, sort utensils, and dispose of trash — potentially 10+ sequential manipulation steps. Each step has a non-zero failure probability. When these probabilities multiply, the overall success rate degrades rapidly. The π₀ baseline achieves only 53.8% instruction-following success on this task, not because it can't perform individual steps, but because small mistakes early in the sequence (approaching the wrong object, misidentifying a fork as a spoon) cascade into complete task failures. This is a well-documented problem in imitation learning: policies lack recovery behaviors when they drift into states not represented in the training distribution. The paper positions GR-3's architectural choices — particularly the QK-norm-inspired RMSNorm and task-status auxiliary supervision — as specifically targeting this robustness problem.

Third, adapting to genuinely novel settings requires prohibitive amounts of robot data. Traditional imitation learning approaches need hundreds or thousands of demonstrations per object category to achieve reasonable performance. Collecting these demonstrations via teleoperation is slow (~250 trajectories per hour on the ByteMini platform, per Section 3.3) and expensive. This creates a fundamental scaling bottleneck: the diversity of real-world objects, environments, and tasks far exceeds what any team can physically demonstrate. A robot policy that can only handle objects it was explicitly trained on will never be a generalist. The paper identifies this as a critical gap and proposes human trajectory data collected via VR devices (running at ~450 trajectories per hour, nearly 2× faster) as a complementary data source that can bridge the gap — but only if the model architecture and training recipe can effectively transfer knowledge from human hand motions to robot joint actions.

Why This Matters: The Gap Between Demos and Deployments

The practical significance of these problems extends well beyond academic benchmarking. A robot intended to assist in kitchens, offices, or homes encounters staggering diversity. The pick-and-place generalization experiment (Section 5.1) evaluates on 45 unseen objects where "more than 70% of them are from categories that are unseen in the robot trajectory data." This isn't an artificial stress test — it's representative of real deployment. A household robot will encounter new grocery items, new furniture arrangements, and new cleaning supplies daily. If the policy fails on every unseen category, it fundamentally cannot function outside controlled laboratory conditions.

The cloth manipulation experiment (Section 5.3) illustrates another dimension: the task requires dexterous manipulation of deformable objects with different sleeve lengths, fabric patterns, and crumpled configurations. Deformable objects are notoriously difficult because their state space is effectively infinite — a shirt can be folded, bunched, or draped in countless ways — and visual policies must learn invariances to these variations. The fact that GR-3 achieves 75.8% average task progress on unseen clothes, including short-sleeved shirts when it was only trained on long-sleeved ones, demonstrates a level of visual and manipulation generalization that matters for real-world usefulness.

Theoretically, this paper engages with a deeper question about what form of pre-training transfers best to embodied control. There's a growing debate in the robotics learning community: does cross-embodiment robot data (the approach of π₀, Open X-Embodiment, and Octo) provide better generalization than vision-language pre-training (the approach of RT-2 and GR-3)? Or are both needed, but for different aspects of the generalization problem? GR-3's experimental design — comparing against π₀ (which is pre-trained on large-scale cross-embodiment robot data) while using VL co-training as its own differentiator — provides evidence on both sides. The result that GR-3 without VL co-training underperforms π₀ (Section 5.1, Unseen Objects: 40% vs. some π₀-level performance) suggests that cross-embodiment pre-training matters. But GR-3 with VL co-training substantially outperforms π₀ (57.8% vs. 40%) on the same generalization benchmark, suggesting that VL knowledge transfer captures something cross-embodiment pre-training alone does not — specifically, the semantic reasoning needed to understand "put the marine animal in the carton."

Where Prior Approaches Fall Short

The paper positions itself against three broad categories of prior work, each with specific limitations:

VLA Models with Cross-Embodiment Pre-Training

π₀ and its successor π₀.₅ represent the state of the art in this category. These models are pre-trained on large datasets spanning many different robot embodiments (different arm configurations, different grippers, different camera setups). The logic is compelling: by seeing many different robots perform many different tasks, the model should learn action representations that transfer across embodiments and generalize to novel settings. The Open X-Embodiment initiative and RT-X models follow the same philosophy.

However, the paper's experiments expose a critical limitation of this approach: cross-embodiment pre-training alone does not teach semantic reasoning about language. In the table bussing instruction-following setting, π₀ "struggles with instruction following, especially in out-of-distribution scenarios. It is not able to distinguish between forks and spoons." This isn't a failure of motor control — the model can grasp utensils competently. It's a failure of visual-semantic grounding: the model hasn't learned the fine-grained distinctions between object categories that are visually similar but semantically distinct. Cross-embodiment data provides diversity in action spaces but not necessarily in visual concept learning, because the objects appearing in robot datasets are typically a narrow subset of the visual world.

The paper also notes that when generalization requires understanding abstract concepts like sizes ("put the largest object into the carton") or spatial relationships ("put the coke next to the sprite into the carton"), purely robot-trained policies fail because these concepts rarely appear with systematic variation in robot demonstration data. A human teleoperator might naturally reach for the largest object, but the dataset doesn't contain explicit counterfactuals showing what would have happened if they'd reached for a smaller one. The model can't learn comparative reasoning from positive examples alone.

VLMs Pre-Trained and Then Fine-Tuned on Robot Data

RT-2 pioneered the approach of taking a large pre-trained VLM (PaLI-X, PaLM-E) and directly fine-tuning it to output action tokens alongside language tokens. This transfers web-scale visual and language knowledge into the robot policy. GR-2 extended this with video generation pre-training. These approaches have demonstrated strong generalization to novel objects and instructions.

The limitation these approaches face is catastrophic forgetting during fine-tuning. When a VLM is fine-tuned exclusively on robot trajectory data, the rich semantic knowledge it acquired during pre-training can degrade — a phenomenon well-known in continual learning literature. The model gradually forgets about octopus tentacles and size comparisons as it specializes to the narrow distribution of objects and instructions in the robot dataset. The paper addresses this implicitly through its co-training approach: by continuously mixing vision-language data with robot data throughout training (rather than only using it for initialization), the model maintains its semantic capabilities while learning to predict actions. The ablation showing GR-3 w/o Co-Training degrades substantially on unseen instructions and objects (Section 5.1) provides direct evidence for this forgetting hypothesis.

Human Video Data for Policy Learning

A growing body of work (EgoMimic, Humanoid Policy, Track2Act) leverages human egocentric videos — often with estimated hand trajectories — to augment robot training data. The core insight is that human videos are vastly more abundant and cheaper to collect than robot demonstrations, and they contain rich information about object manipulation strategies.

The paper identifies a specific gap in this literature: few existing methods have demonstrated few-shot transfer from human hand trajectories to robot end-effector actions on genuinely novel objects. Prior work by Qiu et al. (2025) and Kareer et al. (2024) showed that co-training on human and robot data improves overall policy performance, but the GR-3 paper pushes this further by explicitly evaluating few-shot generalization: from 0 to 1 to 5 to 10 human trajectories per novel object, measuring the incremental improvement separately from the zero-shot baseline. The result — jumping from 57.8% zero-shot to 86.7% with 10-shot — demonstrates that human trajectory data is not merely a regularizer or data augmentation technique, but a genuinely powerful mechanism for rapidly adapting to new object categories without any additional robot demonstrations.

Architectural Innovations for Instruction Following

The paper also identifies a specific technical problem that prior VLA architectures don't adequately address: models can learn to ignore language instructions and rely on visual shortcuts. Kim et al. (2025) demonstrated this phenomenon clearly: when multiple camera views are available, policies often learn to predict actions from spurious visual correlations (e.g., "when the left wrist camera shows this particular arrangement, execute this particular grasp pattern") rather than attending to the language instruction. This works on the training distribution but fails catastrophically when the same objects appear in different arrangements or when different objects require similar motor actions.

The paper's response to this — the task-status auxiliary supervision mechanism — is a specific architectural contribution designed to force language grounding. By randomly replacing valid instructions with invalid ones and requiring the model to explicitly predict an "Invalid" status, the model must learn to assess whether the instruction is compatible with the current observation. This isn't a new idea in machine learning (auxiliary losses are a classic technique), but its specific application to VLA language grounding and the demonstrated 43.7 percentage point improvement in instruction-following accuracy (from 53.8% for π₀ to 97.5% for GR-3 on table bussing IF) represents a significant practical advance.

The QK-norm-inspired RMSNorm addition to the DiT blocks addresses a different problem: training instability in the action prediction head. The paper reports that "in early explorations, we observed frequent instability during training." This is a critical practical concern — large VLA models with diffusion-based action heads are notoriously difficult to train stably because the action prediction loss depends on the VLM backbone's representations, which are themselves changing during training. Gradient signals from the flow-matching loss can destabilize the visual and language representations that the VLM learns from the VL data loss. The RMSNorm addition effectively decouples these signals by normalizing the feature magnitudes in the DiT, preventing the action prediction objective from dominating the VLM backbone updates.

How This Paper Positions Itself

The paper positions GR-3 not as a fundamentally new architecture (it explicitly builds on the mixture-of-transformers design and flow-matching action prediction), but rather as a carefully engineered system with a specific training recipe and a set of design choices whose individual and combined effects are empirically validated.

This positioning is important to understand. Unlike papers that claim architectural novelty (e.g., "we propose a new attention mechanism"), GR-3's contribution is in the systematic study of what works when scaling up VLA training with heterogeneous data sources. The paper answers practical questions: Does VL co-training help more than just VLM initialization? (Yes, substantially — see the w/o Co-Training ablation.) Does human trajectory data transfer effectively to robot control? (Yes, with 10 trajectories per object providing major gains.) Do architectural details like QK norm matter for real-world task performance? (Yes — removing it substantially hurts instruction following.)

The paper also positions itself explicitly as a step toward generalist robots "capable of assisting humans in daily life" (Section 7). This isn't presented as a solved problem — the limitations section acknowledges mistakes on "unseen instructions involving novel concepts and objects" and struggles with "grasping objects with unseen shapes." The claim is more measured: GR-3 represents progress along specific axes (semantic generalization, few-shot adaptation, long-horizon robustness) that are necessary but not sufficient for general-purpose robotic assistance.

Finally, the paper engages with a broader debate about data scaling strategies for robotics. The π₀ line of work argues for cross-embodiment robot data as the primary scaling axis. The RT-2 line of work argues for VLM pre-training. GR-3 sits at the intersection: it uses VL co-training (RT-2 lineage) with a flow-matching action head (π₀ lineage) and adds human trajectory data as a third scaling axis. The experimental results comparing against π₀ — which has access to large-scale cross-embodiment pre-training that GR-3 lacks — are particularly revealing. GR-3 outperforms π₀ on semantic generalization but might benefit from π₀-style pre-training for basic motor skills (as suggested by GR-3 w/o Co-Training underperforming π₀). This suggests the two approaches are complementary rather than competing, and the optimal generalist policy likely combines both — a direction the paper hints at but doesn't fully explore.

3. Technical Approach

3.1 Reader Orientation

GR-3 is an end-to-end neural network policy that takes in camera images and a natural language command, and directly outputs the joint movements a robot needs to execute. The system solves the problem of getting a physical robot to follow arbitrary instructions in environments and with objects it has never encountered before by combining three training data sources — recorded robot demonstrations, web-scale vision-language data, and human-collected hand trajectories — into a single model that transfers semantic understanding from web data into physical action prediction, while architectural stability fixes and an auxiliary language-grounding loss prevent the model from ignoring the instruction in favor of visual shortcuts.

3.2 Big-Picture Architecture (Diagram in Words)

The GR-3 system has five major components connected in a feed-forward pipeline:

  1. Perception inputs — Four RGB images (one head-mounted camera, two wrist-mounted cameras, and one additional view) plus a natural language instruction string and the current robot joint state vector. These arrive simultaneously at each control timestep.

  2. VLM Backbone (Qwen2.5-VL-3B-Instruct) — A pre-trained vision-language model that processes the images and instruction jointly through its transformer layers, producing a key-value (KV) cache — essentially a compressed representation of everything the model understands about the current scene and what the instruction is asking for. This backbone is trained on both robot data and vision-language data during the joint co-training phase.

  3. Action Diffusion Transformer (DiT) — A smaller transformer module (half the number of layers compared to the VLM backbone) that takes the VLM's KV cache, the current robot joint state, and a noisy action trajectory as input, and iteratively denoises the action trajectory into a final predicted action chunk. This module only receives KV cache from the latter half of the VLM layers (a deliberate design choice for inference speed).

  4. Task Status Head — An auxiliary output dimension within the action chunk that predicts one of three states: Ongoing (0), Terminated (1), or Invalid (-1). This forces the DiT to explicitly verify that the instruction is compatible with the current observation before generating motor commands.

  5. Trajectory Optimization Controller — A downstream execution module (not part of the neural network) that takes the predicted action chunk and smooths it using pure pursuit and jerk-minimizing trajectory optimization before sending commands to the robot's 19 controlled degrees of freedom.

Information flows as follows: camera images and instruction text enter the VLM backbone → the VLM processes them through its transformer layers and emits a KV cache → the action DiT receives the KV cache (from the second half of VLM layers only), the current robot state, and a noise sample → the DiT performs iterative flow-matching denoising to produce a clean action chunk → the first actions in the chunk are executed on the robot via the trajectory optimizer → the process repeats on the next timestep with new observations.

3.3 Roadmap for the Deep Dive

  • First, the formal problem setup and the input/output specification of the policy $\pi_\theta$ — what exactly does GR-3 predict, from what, and in what format? This establishes the interface before we open the black box.
  • Second, the flow-matching objective and process — how action prediction is framed as a generative modeling problem rather than a regression problem, and why this matters for handling multimodal action distributions.
  • Third, the mixture-of-transformers architecture — how the VLM backbone and action DiT are structurally connected, what the KV cache interface looks like, and why the DiT only uses the second half of VLM layers.
  • Fourth, the QK-norm-inspired RMSNorm addition to the DiT — the specific normalization that stabilizes training and improves instruction following.
  • Fifth, the multi-timestep flow-matching loss computation — the training acceleration trick that computes losses at multiple denoising timesteps from a single VLM forward pass.
  • Sixth, the causal attention mask in the action DiT — how temporal dependency within the action chunk is modeled.
  • Seventh, the task status auxiliary supervision mechanism — how the model is forced to attend to language instructions by learning to detect when an instruction is invalid given the current scene.
  • Eighth, the three training data sources and their mixing protocol — robot trajectory data, vision-language co-training data, and human trajectory data, including how each contributes to a different aspect of generalization and how they are combined during training.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and training methodology paper whose core idea is that web-scale vision-language co-training, combined with specific architectural stability fixes and a language-grounding auxiliary loss, produces a VLA model that generalizes semantically to novel instructions and objects while maintaining robust performance on long-horizon dexterous tasks, and that human trajectory data collected via VR provides a sample-efficient mechanism for bridging the remaining generalization gap on novel object categories.

The Policy Interface and Action Representation

GR-3 is a function $\pi_\theta$ parameterized by neural network weights $\theta$. At each control timestep $t$, it predicts a $k$-length action chunk:

at=πθ(l,ot,st)\mathbf{a}_t = \pi_\theta(l, \mathbf{o}_t, \mathbf{s}_t)

where $l$ is the natural language instruction string, $\mathbf{o}_t$ is the observation at timestep $t$ (images from multiple camera views — specifically the head camera, two wrist cameras, and one additional view for a total of four RGB images), and $\mathbf{s}_t$ is the current robot state vector (joint positions and velocities for all controlled degrees of freedom). The output $\mathbf{a}_t = a_{t:t+k}$ is a sequence of $k$ future actions, where each action specifies target positions for the robot's controlled joints.

What it computes: a mapping from the current perceptual context (what the robot sees and what it was told to do) and its current physical configuration to a sequence of motor commands that will advance the task. The action chunk represents a short trajectory — typically on the order of 10–20 future actions — that gets executed open-loop for a portion of the chunk before the model is re-queried with fresh observations.

Why this form: predicting action chunks rather than single actions provides temporal consistency — the model plans a coherent short-horizon trajectory rather than reacting myopically frame-by-frame. This reduces jitter and helps the model coordinate multi-step manipulation sequences like inserting an object into a container, where the approach, grasp, lift, and release phases are interdependent. The chunk size $k$ is chosen to be long enough to capture meaningful manipulation phases but short enough that closed-loop replanning can correct for execution errors and environmental changes. The specific annotation that GR-3 controls 19 of the robot's 22 degrees of freedom — excluding the 3 degrees of freedom from the lift mechanism and head — means the action vector itself is 19-dimensional per timestep, making the full chunk a $k \times 19$ tensor.

Flow Matching for Action Prediction

Rather than treating action prediction as a deterministic regression (predicting $\mathbf{a}_t$ directly from the observations) or as a classification over discretized actions, GR-3 frames it as a conditional generative modeling problem using flow matching. This means the model learns to transform random noise into a valid action trajectory through an iterative denoising process, conditioned on the instruction and observations.

The flow-matching training objective is:

Laction(θ)=E{at,ot,st,l}∼D[∥vθ(l,ot,st,atτ)−u(atτ∣at)∥2]L_{\mathrm{action}}(\theta) = \mathbb{E}_{\{\mathbf{a}_t,\mathbf{o}_t,\mathbf{s}_t,l\} \sim \mathcal{D}}\left[\parallel \mathbf{v}_\theta(l, \mathbf{o}_t, \mathbf{s}_t, \mathbf{a}_t^\tau) - \mathbf{u}(\mathbf{a}_t^\tau | \mathbf{a}_t) \parallel^2 \right]

where $\tau \sim \mathcal{U}(0, 1)$ is a uniformly sampled flow-matching timestep, $\mathbf{a}_t^\tau = (1-\tau)\epsilon + \tau\mathbf{a}_t$ is a noisy version of the ground-truth action chunk created by linear interpolation between random noise $\epsilon \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$ and the clean action chunk $\mathbf{a}_t$, and $\mathbf{u}(\mathbf{a}_t^\tau | \mathbf{a}_t) = \mathbf{a}_t - \epsilon$ is the ground-truth flow field — the vector pointing from the noisy action back toward the clean action.

The function $\mathbf{v}_\theta$ is the neural network's prediction of this flow field. The loss is simply the mean squared error between the predicted flow and the true flow.

What it computes: the flow-matching loss trains the model to predict the instantaneous direction and magnitude of denoising needed at any noisiness level $\tau$. During training, for each action chunk in the dataset, a random $\tau$ is sampled, the action is corrupted by mixing it with $\tau$-weighted noise, the model predicts the flow vector $\mathbf{v}_\theta$, and the squared error against the true flow $\mathbf{a}_t - \epsilon$ is minimized. The expectation over $\tau \sim \mathcal{U}(0, 1)$ ensures the model learns to denoise at all noise levels, from nearly pure noise ($\tau \approx 0$) to nearly clean actions ($\tau \approx 1$).

Why this form: flow matching is chosen over alternative generative modeling approaches (like DDPM-style diffusion or discrete token prediction) for three reasons. First, it enables fast inference: at deployment time, the action chunk is initialized from pure noise $\mathbf{a}_t^{\tau=0} \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$ and integrated from $\tau=0$ to $\tau=1$ using the Euler method — $\mathbf{a}_t^{\tau+\Delta\tau} = \mathbf{a}_t^\tau + \mathbf{v}_\theta(l, \mathbf{o}_t, \mathbf{s}_t, \mathbf{a}_t^\tau) \Delta\tau$ with $\Delta\tau = 0.2$ — meaning only 5 denoising steps are needed (since $1.0 / 0.2 = 5$). This is substantially faster than DDIM sampling which might require 50–100 steps, and is critical for real-time robot control where inference latency must be low.

Second, flow matching handles multimodal action distributions naturally. In many manipulation scenarios, there are multiple valid ways to complete a task — a robot could grasp an object from the left or the right, or approach a container from different angles. A deterministic regression model would learn the average of these modes (which might be an invalid action in between them), while a generative model can sample from any of the modes. Flow matching, by learning to transport a simple Gaussian distribution to the complex action distribution through a continuous path, captures this multimodality without requiring discrete latent variables.

Third, the linear interpolation path $\mathbf{a}_t^\tau = (1-\tau)\epsilon + \tau\mathbf{a}_t$ is the simplest possible transport map and has been shown empirically (in Liu, 2022; Lipman et al., 2022) to produce high-quality samples with very few integration steps. The straight-line path means the flow field is simple and easy to learn — the model doesn't need to represent curved trajectories through action space.

The Mixture-of-Transformers Architecture

GR-3 adopts the mixture-of-transformers design (Liang et al., 2024), which separates the processing of different modalities into specialized transformer modules rather than forcing a single transformer to handle everything. The architecture has two distinct transformer stacks:

The VLM Backbone. This is a pre-trained vision-language model (specifically Qwen2.5-VL-3B-Instruct from Bai et al., 2025) that processes the four camera images and the language instruction. The images are tokenized into visual tokens (likely through a vision encoder that produces patch embeddings), the language instruction is tokenized into text tokens, and these are concatenated and processed through the VLM's transformer layers with standard causal or bidirectional attention (the paper doesn't specify the exact attention pattern, but Qwen2.5-VL uses interleaved attention across modalities). The output of the VLM backbone is a key-value (KV) cache — the stored key and value tensors from the attention operations at each transformer layer. This KV cache is a compressed representation of everything the VLM understands about the visual scene and the instruction.

The Action DiT. This is a separate, smaller diffusion transformer (DiT, based on Peebles and Xie, 2023) that receives three inputs: the KV cache from the VLM backbone, the current robot state $\mathbf{s}_t$, and the noisy action chunk $\mathbf{a}_t^\tau$. The $k$-length action chunk is represented as $k$ separate tokens (one per future timestep) and concatenated with the robot state token to form the input token sequence for the DiT. The flow-matching timestep $\tau$ is injected into the DiT through adaptive layer normalization (AdaLN) — the layer normalization scale and shift parameters are produced by a small MLP that takes $\tau$ as input, allowing the DiT to modulate its computation based on the current noise level.

What happens in sequence: (1) the VLM backbone processes images and text once per control cycle, producing a KV cache; (2) the action DiT takes this KV cache as a fixed conditioning signal and performs multiple denoising steps (5 steps with $\Delta\tau = 0.2$) to transform random noise into a clean action chunk; (3) the VLM backbone is not re-run during the denoising steps — only the KV cache is queried, which is computationally cheap. This is the key efficiency insight: the expensive visual and language processing happens once, and the iterative refinement of the action happens in a much smaller, faster module.

Why this separation: The mixture-of-transformers design addresses a fundamental asymmetry in the computational requirements of perception versus action generation. Understanding a visual scene with multiple objects and a complex instruction requires deep transformer processing with many layers and large hidden dimensions. But once that understanding is compressed into a KV cache, generating an action trajectory from it requires less representational capacity — the action space, while high-dimensional, is more constrained and lower-entropy than the space of possible visual scenes. By using a smaller DiT (half the layers of the VLM backbone), the architecture allocates compute where it's needed and avoids wasting parameters on the action head.

A critical detail: the DiT only uses the KV cache from the latter half of the VLM layers. This is explicitly stated as a design choice "to ensure fast inference." The rationale is that early VLM layers process low-level visual features (edges, textures, primitive shapes) that are not directly relevant for action prediction, while later layers encode semantic concepts (object identities, spatial relationships, affordances) that the action head needs. Skipping the early-layer KV cache reduces the memory and compute cost of the cross-attention operations in the DiT without losing task-relevant information.

The total model size is 4B parameters, which includes both the VLM backbone (Qwen2.5-VL-3B-Instruct is approximately 3B parameters) and the action DiT (approximately 1B parameters, given the "half the number of layers" specification for a VLM with perhaps 24–32 layers).

Causal Attention Mask in the Action DiT

Within the action DiT, the attention mask is causal: each action token can attend to previous action tokens in the chunk but not to future ones. Additionally, all action tokens can attend to the robot state token and to the VLM KV cache (which is treated as a prefix that all tokens can access).

What this does: this introduces temporal structure into the action prediction. The action at timestep $t + 3$ is predicted with knowledge of the actions at timesteps $t$, $t+1$, and $t+2$, but without knowledge of timestep $t+4$. This reflects the natural causality of action execution — later actions in a trajectory should be conditioned on earlier actions, because what the robot does next depends on what it just did. For example, once the model has committed to a particular grasp approach in the first few timesteps, the subsequent lift and transport actions should be consistent with that approach.

Why this form: without the causal mask, the model could use future action information to inform current action predictions, which would create a circular dependency — the model would learn to predict action $t+1$ by looking at action $t+2$, but during autoregressive closed-loop execution, action $t+2$ hasn't been executed yet and might never be (since the policy replans after executing only a portion of the chunk). The causal mask ensures that each action in the chunk is predicted from information that is genuinely available at decision time: the current state, the instruction, the visual scene, and the preceding actions in the same chunk. At inference time this is trivially satisfied because actions are generated sequentially through the denoising process. At training time, the mask prevents the model from learning spurious correlations that depend on future action information.

The causal mask also enables the DiT to learn action-to-action transition dynamics — how the robot's configuration evolves from one timestep to the next. This is a form of implicit dynamics modeling embedded within the policy, which helps produce smooth, physically plausible trajectories even without an explicit dynamics model.

The QK-Norm-Inspired RMSNorm in DiT Blocks

The paper reports that "in our early explorations, we observed frequent instability during training." This instability manifests as loss spikes or divergence during joint training of the VLM backbone and the action DiT, particularly when the flow-matching loss gradients backpropagate through the entire VLM and destabilize the visual and language representations that the VLM is also trying to learn from the vision-language co-training objective.

The solution is inspired by QK normalization (Henry et al., 2020): additional RMSNorm layers are inserted after the linear projections within both the attention and feed-forward network (FFN) sub-layers of the DiT blocks. Specifically, within the DiT attention, after the query, key, and value linear projections produce their respective tensors, an RMSNorm is applied to normalize the L2 norm of each vector. Similarly, within the FFN, the intermediate activations after the first linear layer and before the activation function are RMSNorm-ed.

RMSNorm (Root Mean Square Layer Normalization, Zhang and Sennrich, 2019) normalizes by the root mean square of the activations:

RMSNorm(x)=x1d∑i=1dxi2+ϵ⊙γ\text{RMSNorm}(x) = \frac{x}{\sqrt{\frac{1}{d}\sum_{i=1}^d x_i^2 + \epsilon}} \odot \gamma

where $d$ is the feature dimension, $\epsilon$ is a small constant for numerical stability, and $\gamma$ is a learnable scale parameter. Unlike LayerNorm, RMSNorm does not subtract the mean — it only scales by the RMS, making it computationally cheaper and (as argued in the original paper) equally effective.

What this does in practice: during training, the VLM backbone's representations are evolving — early in training, the visual features are adapting from the pre-trained VL distribution to the robot's camera viewpoints and object set. The flow-matching loss backpropagates through the DiT into the VLM's KV cache representations. Without normalization in the DiT, the gradient magnitudes flowing into the VLM can vary wildly depending on the current quality of the denoising prediction. A poor denoising prediction (common early in training) produces large gradients that can overwhelm the VLM's own VL training signal. The RMSNorm in the DiT acts as a gradient bottleneck: by normalizing the activation magnitudes at each sub-layer, it constrains the effective learning rate for the VLM representations to a stable range, preventing the action prediction loss from dominating the VL co-training loss.

Why this improves instruction following: the paper reports (Section 5.2) that removing RMSNorm "hurts the performance in both settings, especially in the IF setting. GR-3 w/o Norm is not able to follow instructions well. In particular, it is not able to generalize to novel destinations." This suggests that without the normalization, the model learns to rely more heavily on visual shortcuts — patterns in the camera images that correlate with actions in the training data but are not robust to novel instructions. The RMSNorm, by stabilizing the gradient flow from the DiT to the VLM, forces the model to learn a more balanced integration of visual and language information, because the language pathway through the VLM isn't being destabilized by the action prediction gradients.

This is a subtle but practically crucial finding: architectural stability choices in the action head have downstream effects on the model's semantic grounding capabilities, because they determine whether the VLM backbone can maintain its language understanding throughout training or whether the action prediction objective gradually erodes it.

Multi-Timestep Flow Matching Loss for Training Efficiency

A straightforward implementation would compute the flow-matching loss at a single random $\tau$ per training sample, requiring one forward pass through the VLM backbone per $\tau$. The paper uses an optimization from Li et al. (2024c): compute the flow-matching loss at multiple $\tau$ values from a single VLM forward pass. "To accelerate training, we compute the flow-matching loss on multiple sampled flow matching timesteps for one forward pass of the VLM backbone."

What this means mechanically: for each training sample (an observation-action-instruction tuple), the VLM backbone is run once to produce the KV cache. Then, $M$ different $\tau$ values are sampled (uniformly from $(0, 1)$), $M$ different noisy action chunks are created by mixing the ground-truth action with noise at each $\tau$ level, and the DiT is run $M$ times (once per $\tau$) — but only the DiT is re-run, not the VLM backbone. The losses from all $M$ denoising predictions are averaged and backpropagated. The VLM backbone receives gradients from all $M$ loss terms simultaneously.

Why this matters: the VLM forward pass is the computational bottleneck — it processes high-resolution images through many transformer layers. The DiT is smaller and faster. By amortizing the VLM cost over multiple $\tau$ samples, training throughput increases by approximately a factor of $M$ for the action prediction objective. Since the vision-language co-training also requires VLM forward passes (on VL data batches), this acceleration is critical for maintaining reasonable training times when jointly training on both data sources with a 4B-parameter model.

The paper does not specify the exact value of $M$ (the number of timesteps per VLM forward pass), but given the $\Delta\tau = 0.2$ inference setting (5 steps), it's plausible that training samples 5 or more $\tau$ values per sample to cover the full denoising trajectory.

Task Status Auxiliary Supervision

Prior work by Kim et al. (2025) demonstrated that VLA models can learn to ignore language instructions and instead rely on spurious visual correlations — for example, learning that "when the scene looks like training example #47, execute this particular motion pattern" without checking whether the instruction actually matches. GR-3 addresses this with a specific mechanism: an additional action dimension called "task status" that forces the model to explicitly verify language-observation compatibility.

The task status can take three values:

  • Ongoing (0): the robot is in the middle of executing the task; everything is proceeding normally.
  • Terminated (1): the robot has successfully completed the task.
  • Invalid (-1): the instruction cannot be completed given the current observation — for example, "put the knife into the woven basket" when there are no knives visible on the table.

During training, the task status is treated as one additional dimension of the action chunk alongside the 19 robot joint dimensions. The model must predict it at every timestep, and it's trained with the same flow-matching objective (treating it as a continuous value that gets denoised alongside the joint actions).

The critical training mechanism is random instruction replacement: "During training, we randomly replace the language instruction with an invalid instruction, and train the model to predict the Invalid status without supervision on the other dimensions of the action chunk." This means:

  1. For some fraction of training samples (the paper doesn't specify the exact fraction), the original correct instruction (e.g., "put the apple into the basket") is replaced with a randomly sampled instruction that is incompatible with the current observation (e.g., "put the knife into the basket" when the scene contains an apple and a basket but no knife).
  2. For these samples, the flow-matching loss is only applied to the task status dimension — the model learns to output the Invalid (-1) status token. The joint action dimensions are not supervised, so the model learns NOT to execute any motion when the instruction is invalid.
  3. For samples with correct instructions, the model learns to predict Ongoing (0) during execution and Terminated (1) at the end of the trajectory.

What this forces the model to learn: to correctly classify a sample as Invalid, the model must (a) parse the language instruction to identify the target object and action, (b) analyze the visual scene to determine what objects are present, and (c) verify that the target object exists in the scene. If the model were ignoring language and relying purely on visual cues, it would fail on these Invalid samples — the visual scene looks like a normal manipulation scene, but the instruction doesn't match, so the only way to predict Invalid is to actually compare the instruction content against the observation content.

This is a form of adversarial language grounding. By introducing contradictory instruction-observation pairs during training, the model cannot succeed through visual memorization alone. The task status prediction acts as a bottleneck that forces the VLM backbone to produce representations encoding both what the instruction says and what the scene contains, and the DiT must learn to detect mismatches.

Why the Invalid status is not supervised on joint actions: if the model were trained to output specific "do nothing" joint actions for Invalid samples, it might learn a different shortcut — "when I see this particular type of training sample (the ones with the Invalid label), output zero actions." Instead, by leaving the joint actions unsupervised on Invalid samples, the model's action prediction pathway is not penalized for any particular motor output; only the status prediction is trained. This means the model must genuinely detect the invalidity condition through the status prediction head, rather than routing it through a motor reflex.

The paper's ablation (Section 5.2) demonstrates the effectiveness of this mechanism: "Without the task status, the IF capabilities also drop, highlighting the effectiveness of task status in aiding the VLA model to follow instructions." The specific magnitude of this drop is visible in the table bussing IF setting, where removing task status causes performance degradation alongside removing RMSNorm.

Training Data Sources and Mixing Protocol

GR-3 is trained on three distinct data sources, each serving a different purpose and introduced at different stages:

Robot Trajectory Data (Imitation Learning). This is the primary action supervision signal. The paper states that for the generalizable pick-and-place task, 35k trajectories were collected covering 101 objects over 69 hours; for table bussing, approximately 101 hours of trajectories; for cloth manipulation, 116 hours of trajectories. Each trajectory consists of a sequence of (observation, action, instruction) tuples, where actions are the expert teleoperator's joint commands, observations are the four camera views, and instructions are language descriptions of the task.

Data collection uses a data collection scheduler (Figure 4) that explicitly randomizes three factors: (1) the action to perform, (2) the object combination present in the scene, and (3) the background environment setting. "At the beginning of each trajectory collection, the system generates a new configuration for the teleoperator to arrange the environment accordingly." This active randomization prevents the dataset from having systematic correlations between specific objects and specific backgrounds (e.g., "the red cup always appears on the left side of the table") that the model could exploit as shortcuts. Post-collection quality checks filter out invalid and low-quality data.

Teleoperation uses a whole-body retargeting system via Meta VR Quest: the teleoperator's hand, arm, and body motions are mapped through a real-time optimization problem that respects joint limits, avoids singularities, and maximizes manipulability. The teleoperator can simultaneously control the two arms, the grippers, the mobile base, and the lift mechanism. This produces fluid, natural motion trajectories that cover the full 22-DoF workspace.

Robot trajectory data is collected at approximately 250 trajectories per hour (implied by the comparison in Section 3.3: human trajectories are collected at 450 per hour and are described as "substantially outpacing the teleoperated robot trajectory collection, which collects about 250 trajectories per hour").

Vision-Language Co-Training Data. This data does not contain any actions — it consists of web-scale vision-language tasks including "image captioning, visual question answering, image grounding, and interleaved grounded image captioning." The paper curates this dataset from a mixture of existing sources (Seed1.5-VL, LLaVA-NeXT-Interleave, CLEVR, GLAMM, xGen-MM) and develops a "filtering and re-annotation pipeline to improve the quality."

The vision-language data is mixed with robot trajectory data dynamically across mini-batches with equal weights. This means that within a training step, half the batch consists of robot trajectory samples (trained with the flow-matching loss) and half consists of VL samples (trained with the standard next-token-prediction loss on text tokens). The VL data only updates the VLM backbone parameters — there is no action prediction for these samples because there are no actions to predict. The total co-training objective is the sum of the flow-matching loss and the next-token-prediction loss:

Ltotal=Laction+LVL\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{action}} + \mathcal{L}_{\text{VL}}

where $\mathcal{L}_{\text{action}}$ is only computed on robot trajectory batches and $\mathcal{L}_{\text{VL}}$ is only computed on VL batches.

The purpose of VL co-training (as opposed to simply initializing from a pre-trained VLM and then fine-tuning only on robot data) is to prevent catastrophic forgetting of the semantic knowledge the VLM acquired during pre-training. By continuously interleaving VL training examples throughout the robot training process, the VLM backbone maintains its ability to recognize fine-grained object categories, understand spatial relationships, and reason about abstract concepts — all of which transfer to the action prediction task through the shared KV cache representation.

Human Trajectory Data (Few-Shot Adaptation). This data is collected using PICO 4 Ultra Enterprise VR headsets, which capture egocentric video and 3D hand trajectories. Human trajectories consist of an egocentric RGB view and the 3D positions of the human's hands over time — there are no arm joint states, no gripper states, and no wrist camera views. The collection rate is approximately 450 trajectories per hour, nearly twice the rate of robot teleoperation.

When incorporating human trajectory data during the few-shot adaptation stage (after the main co-training on robot + VL data), the missing modalities are handled by padding blank images for the missing wrist views and training the flow-matching loss only on the hand trajectory (not on the full robot joint state). The robot state dimensions that don't exist in the human data (arm joint angles, gripper position) are presumably masked from the loss, so the model only receives gradient signal on the action dimensions that correspond to hand positions.

The few-shot adaptation training protocol is: starting from the checkpoint trained on robot trajectories + VL data, perform co-training for an additional 20k steps "by further incorporating human trajectory data alongside the robot trajectories and VL data." All three data types are mixed during this fine-tuning stage, meaning the model continues to see robot trajectories and VL data while learning from the human trajectories. This prevents the model from overfitting to the small number of human demonstrations and forgetting its robot manipulation skills.

Photometric augmentations are applied to the robot trajectory images during training to improve robustness against changing environments. The paper doesn't specify the exact augmentations, but standard photometric augmentations in this context would include random brightness, contrast, saturation, and hue shifts — making the model invariant to lighting changes between the training environment and the deployment environments (checkout counter, meeting room, desk, break room).

Whole-Body Compliance Control and Trajectory Optimization for Deployment

While not part of the neural network training, the execution pipeline has two components that directly affect the policy's observed performance:

During data collection (teleoperation): The whole-body compliance control framework (Sentis and Khatib, 2006) treats all degrees of freedom as a unified system, solving a real-time optimal control problem that retargets human motion to feasible robot motion while respecting joint limits, avoiding kinematic singularities, and maximizing manipulability. This produces the high-quality expert trajectories that the policy learns from.

During policy rollout (inference): The predicted action chunk from GR-3 undergoes two post-processing steps before execution: (1) pure pursuit (Coulter, 1992) — a path tracking algorithm that computes smooth trajectories to follow the predicted waypoints; and (2) trajectory optimization — a real-time parameterized optimization that minimizes jerk (the third derivative of position) and ensures seamless transitions between waypoints and across consecutive action chunks. "The real-time parameterized optimization minimizes jerk and ensures seamless transition between waypoints and across trajectories."

Why this matters: the neural network predicts joint-space targets at a fixed control frequency (likely 10–50 Hz based on typical VLA implementations). Without trajectory optimization, directly commanding these targets would produce jerky, potentially unsafe motions — the predicted action chunk may have small discontinuities between timesteps or between consecutive chunks that would cause the physical robot to vibrate or overshoot. The trajectory optimizer acts as a smoothing filter that respects the robot's physical dynamics (maximum velocities, accelerations, and jerks) while remaining as close as possible to the predicted trajectory. Crucially, this optimization is separate from the policy — GR-3 doesn't need to learn smoothness because the downstream controller handles it. This separation of concerns (the policy focuses on task-level decisions, the controller handles motion-level smoothness) is a standard and effective design pattern in robot learning.

The paper specifies that GR-3 controls 19 of the 22 degrees of freedom during rollout, excluding the 3 degrees of freedom from the lift mechanism and head. This means the lift mechanism and head are either held fixed or controlled by a separate (non-learned) controller during task execution, and the policy only controls the two 7-DoF arms (14 DoFs), the two grippers (2 DoFs for open/close), and the mobile base (3 DoFs for planar motion).

Summary of Key Design Choices and Their Justifications

  • Flow matching over deterministic regression: handles multimodal action distributions, enables fast 5-step inference, and produces smoother trajectories than single-step prediction.
  • Mixture-of-transformers over monolithic architecture: allocates compute asymmetrically — deep VLM for perception, shallow DiT for action generation — enabling fast inference while maintaining strong visual-semantic understanding.
  • DiT uses only latter-half VLM layers: reduces KV cache memory and cross-attention cost without losing task-relevant semantic information, since early VLM layers encode low-level features.
  • QK-norm-inspired RMSNorm in DiT blocks: stabilizes training by preventing action prediction gradients from destabilizing VLM representations, and empirically improves instruction following by forcing balanced integration of visual and language information.
  • Causal attention mask in DiT: enforces temporal causality in action prediction, preventing the model from using future action information and encouraging learning of action-to-action dynamics.
  • Multi-timestep flow-matching loss: amortizes expensive VLM forward pass over multiple denoising timesteps, significantly accelerating training throughput.
  • Task status auxiliary supervision: forces language grounding by requiring the model to explicitly detect instruction-observation mismatches, preventing reliance on visual shortcuts.
  • Equal-weight dynamic mixing of VL and robot data: maintains semantic knowledge throughout training rather than just at initialization, preventing catastrophic forgetting.
  • Data collection scheduler with explicit randomization: prevents spurious correlations between objects, environments, and actions in the training data.
  • Separate trajectory optimization during rollout: decouples task-level policy decisions from motion-level smoothness, allowing the policy to focus on what to do rather than how to move smoothly.

4. Key Insights and Innovations

Innovation 1: Vision-Language Co-Training as Continuous Semantic Maintenance, Not Just Initialization

The dominant paradigm in VLA model development has been pre-train then fine-tune: take a large pre-trained VLM (like PaLI-X, PaLM-E, or an earlier Qwen variant), use it to initialize the perception backbone of a robot policy, and then fine-tune the entire model on robot trajectory data to predict actions. RT-2 established this template. GR-2 extended it with video generation pre-training. The implicit assumption in this line of work is that the knowledge transferred from the pre-trained VLM is durable — once the model has been initialized with web-scale vision-language representations, fine-tuning on robot data can refine them for the target domain without destroying the underlying semantic capabilities.

GR-3's experimental results directly challenge this assumption, and in doing so, shift the framing from "co-training as initialization" to "co-training as continuous semantic maintenance."

The evidence is in the ablation study from Section 5.1 (Figure 7a). GR-3 w/o Co-Training — the model trained only on robot trajectories without interleaved VL data — performs substantially worse than the full GR-3 model on both Unseen Instructions (where its success rate drops roughly from 77.1% to the neighborhood of the non-co-trained baseline) and Unseen Objects (dropping from 57.8% to a lower level that the paper reports as "inferior performance compared to the π₀ baseline"). Yet on Basic and Unseen Environments — the settings that test robustness to environment changes but not semantic generalization — GR-3 w/o Co-Training performs comparably to the full model.

This is a diagnostic result. If VL pre-training knowledge were durable, we would expect the gap between co-trained and non-co-trained models to be similar across all evaluation settings. The fact that the gap is specifically concentrated in settings requiring semantic reasoning (understanding that "the animal with tentacles" means the octopus toy, or that "the marine animal" includes a plush whale) tells us something precise: during robot-only fine-tuning, the model forgets the fine-grained semantic distinctions and compositional reasoning capabilities that it acquired during VL pre-training. The motor skills remain — the model can still pick and place objects competently — but the language grounding degrades. The model still knows how to reach and grasp, but it no longer knows which object to reach for when the instruction requires understanding category membership, spatial relationships, or comparative properties.

This is not merely a practical engineering finding about training schedules. It reframes the relationship between VL knowledge and robot learning: semantic knowledge is not a one-time transfer that persists through subsequent fine-tuning, but a capability that requires active maintenance through continued exposure to diverse vision-language tasks. The equal-weight dynamic mixing of VL data throughout training is not an optimization trick — it's a mechanism for preventing the action prediction objective from gradually overwriting the representations that enable semantic generalization.

This insight has significant implications for how future VLA models should be designed and trained. It suggests that the common practice of pre-training on VL data and then fine-tuning exclusively on robot data will invariably hit a generalization ceiling, and that the ceiling height is determined by how aggressively the fine-tuning process erodes the semantic representations. It also implies that simply using larger or better pre-trained VLMs as initialization may not help if the subsequent fine-tuning is not interleaved with VL data — the forgetting process will erode the semantic knowledge regardless of how good the initial representations were.

The paper connects this implicitly to the broader catastrophic forgetting literature in continual learning, but the robotics-specific implication is new: the representations needed for physical task execution and the representations needed for semantic language grounding are in tension during training, and maintaining both requires explicit data mixture strategies rather than sequential training paradigms.

Innovation 2: Task Status as a Mechanism for Enforcing Language Grounding Through Negative Examples

A persistent problem in VLA training — identified by prior work (Kim et al., 2025) and acknowledged in this paper — is that policies learn spurious correlations between visual patterns and actions rather than genuinely grounding actions in language instructions. When trained only on (instruction, observation, action) tuples where the instruction always correctly describes the task being performed, the model can learn to predict actions from the visual observation alone, effectively ignoring the language input. This works on the training distribution but fails catastrophically when the same objects appear under different instructions.

The standard response to this problem in the VLA literature has been architectural: design attention mechanisms that force cross-modal interaction, or use separate language and vision encoders with explicit fusion modules. RDT-1B, for example, uses a diffusion-based architecture with dedicated language conditioning paths. The implicit assumption is that better architecture improves language grounding.

GR-3 takes a fundamentally different approach: rather than making the architecture better at integrating language, it creates a training objective that cannot be satisfied without genuine language-observation comparison. The task status auxiliary supervision mechanism introduces "Invalid" training samples where the instruction is deliberately incompatible with the observation — for instance, asking the robot to "put the knife into the basket" when the scene contains no knives. The model must predict an explicit Invalid (-1) status token for these samples. Crucially, the joint action dimensions are not supervised on Invalid samples — the model is not told what motor actions to produce when the instruction doesn't match the scene. It only learns to detect that something is wrong.

This is conceptually distinctive because it treats language grounding not as an architectural problem but as a training data and objective design problem. The key insight is that positive examples alone (instruction, observation, action) cannot teach the model to distinguish between "I should perform this action because the instruction says so" and "I should perform this action because the scene looks like this." Both hypotheses predict the same training data. Only by introducing counterfactual training examples where the language-observation mapping is broken can the model learn that its predictions must depend on the specific content of the instruction, not just the general appearance of the scene.

This connects to a broader principle in machine learning that is underappreciated in robot learning: discriminative tasks (like detecting instruction-observation mismatches) are often more effective at teaching fine-grained distinctions than generative tasks (like predicting actions from correct pairings). By forcing the model to explicitly verify that the instruction is compatible with the observation, the task status head serves as a bottleneck that propagates the need for language grounding backward through the VLM backbone. The VLM must produce representations where "knife" and "apple" are clearly distinguishable and where the presence or absence of each can be determined from the visual features — otherwise the task status prediction will fail.

The magnitude of the effect is substantial. In table bussing (Section 5.2), removing task status degrades instruction-following performance significantly, with the ablation model "not able to follow instructions well." While the paper doesn't report the exact numerical drop for this specific ablation in isolation (it's grouped with other results in Figure 8d), the qualitative description indicates that the model loses the ability to generalize to novel destinations and to reject invalid tasks — precisely the capabilities that require genuine language understanding rather than visual pattern matching.

This innovation is incremental in mechanism (auxiliary losses are a standard technique) but fundamental in framing: it recasts language grounding in VLA from an architectural challenge to a counterfactual training problem, and provides a specific, validated mechanism for solving it that requires no changes to the underlying VLA architecture beyond an additional output dimension.

Innovation 3: Bridging Human and Robot Manipulation Through a Dual-Embodiment Data Scaling Axis

The use of human demonstration data in robot learning is not new. Prior work has explored learning representations from human videos (R3M, VIP), extracting affordances from human egocentric data (Bahl et al., 2023), and co-training on human and robot data (EgoMimic, Humanoid Policy). However, the dominant framing in this prior work treats human data as a supplementary augmentation — something that improves overall policy performance or enables pre-training when robot data is scarce, but not as a primary mechanism for adapting to novel object categories in a few-shot regime.

GR-3's contribution is to reframe human trajectory data as a third scaling axis in the VLA training recipe, alongside robot trajectories and vision-language data, and to systematically characterize its few-shot generalization properties. The experimental design is what makes this intellectually distinctive: rather than simply adding human data to the training mix and reporting improved aggregate performance (as prior work has done), the paper evaluates zero-shot, 1-shot, 5-shot, and 10-shot performance on a fixed set of 45 previously unseen objects, measuring how each additional human trajectory per object incrementally improves both instruction-following rate and task success rate.

The result — improving from 57.8% zero-shot success to 86.7% with only 10 human trajectories per object, while maintaining performance on seen objects — reveals something specific about the nature of the generalization gap. The zero-shot performance of 57.8% means the model already has some ability to handle novel objects, thanks to VL co-training. The 10-shot improvement to 86.7% means that a small amount of human demonstration data bridges most of the remaining gap. The fact that 1-shot and 5-shot show intermediate improvements (the upward slope in Figure 7b) indicates that the benefit is not binary — it scales smoothly with the amount of human data.

This has a significant practical implication that goes beyond aggregate performance numbers: the expensive robot teleoperation data provides the base manipulation competency and semantic grounding, while the cheap human trajectory data provides rapid adaptation to new object instances. This suggests a deployment model where a VLA is pre-trained on a fixed corpus of robot + VL data, and then quickly adapted to new environments or object sets by collecting a handful of human demonstrations on-site — a process that takes minutes rather than hours.

The conceptual advance is in characterizing which part of the generalization problem each data source solves. Robot trajectory data teaches the policy how to move — the motor skills, the contact dynamics, the spatial reasoning about the robot's embodiment. Vision-language data teaches the policy what objects are and how they relate — the semantic categories, the spatial prepositions, the abstract concepts. Human trajectory data bridges the two by showing how to manipulate specific object instances that the policy can recognize (from VL data) but hasn't practiced with (from robot data). The three data sources are not redundant — they address different aspects of the policy's competence, and the paper's few-shot experiments demonstrate that the whole is greater than the sum of its parts.

This is incremental as a method (collecting human demonstrations is not new) but fundamental as a characterization of data scaling axes in VLA training. The paper gives practitioners a concrete sense of the return on investment for human trajectory data: roughly 30 percentage points of success rate improvement on novel objects for 10 demonstrations per object, collected at 450 trajectories per hour. This transforms human data from a "nice to have" augmentation into a cost-effective primary mechanism for domain adaptation.

Innovation 4: Verifying the Verifier — The Task Status Mechanism as a Deployable Safety Property

While the task status mechanism is introduced in Section 3 primarily as a training technique for improving language grounding, the paper's evaluation reveals a second, equally important function: the Invalid task status prediction enables the robot to recognize and reject impossible instructions at deployment time. This is evaluated explicitly in the "Invalid Tasks" subset of the table bussing instruction-following experiment (Section 5.2), where "the trial is considered successful only if the model refrains from manipulating any objects within 10 seconds" when given an instruction that cannot be completed (e.g., "put the blue bowl into the plastic box" when no blue bowl is present).

This is a conceptual contribution to robot safety and reliability that goes beyond standard VLA evaluation. The typical VLA success metric measures whether the robot does the right thing when given a valid instruction. The Invalid Tasks metric measures whether the robot does nothing when given an impossible instruction — a fundamentally different capability that is rarely evaluated and even more rarely achieved.

The significance is that this capability addresses a known failure mode in deployed language-conditioned policies: over-obedience. Milli et al. (2017) articulated the problem theoretically — robots should not blindly execute instructions that are infeasible, unsafe, or ill-posed. In practice, imitation-learned policies trained only on valid instruction-action pairs will, when given an invalid instruction, produce some action — typically whatever action was most common in visually similar training scenes. This leads to behaviors like grasping random objects when the requested object isn't present, or attempting to place objects in containers that don't exist in the scene. These failures are not just performance degradations; they are safety hazards in unstructured environments.

GR-3's task status mechanism provides a specific, trained solution to this problem. The model has learned, from counterfactual training examples, to associate the mismatch between instruction and observation with a specific prediction (Invalid status), and when it makes this prediction, it suppresses manipulation actions. The paper reports that GR-3 successfully refrains from acting in the Invalid Tasks setting, while π₀ (without this mechanism) does not.

What makes this intellectually distinctive is that it demonstrates a capability that emerges from the training objective design rather than from architectural complexity or additional safety modules. The model's ability to reject invalid instructions is not a post-hoc addition — it's a direct consequence of the random instruction replacement during training, which taught the model that some instructions are incompatible with some observations and that the correct response is inaction. This is an elegant example of embedding a safety property into the training objective rather than layering it on top through explicit constraints or separate classifiers.

The practical implication is substantial: for real-world deployment, a VLA model that silently fails on invalid instructions (by attempting to execute them anyway) is far more dangerous than one that recognizes its own inability to comply. The ability to say "no" — or at least to recognize that the requested task is not currently feasible — is a necessary property for any robot assistant operating in unstructured human environments. GR-3 demonstrates that this property can be achieved through careful training data design without any additional runtime mechanisms.

5. Experimental Analysis

Evaluation Methodology

  • Dataset. The paper evaluates on three custom real-world task suites, not a pre-existing benchmark. For generalizable pick-and-place, the test set comprises 54 seen objects and 45 unseen objects (with over 70% from novel categories) across four environment settings (Basic, Unseen Environments, Unseen Instructions, Unseen Objects). For long-horizon table bussing, evaluation includes five object sets in the Flat setting and six instruction subsets in the IF setting (Basic, Multiple Objects, Multiple Destinations, Multiple Objects & Destinations, Novel Destinations, Invalid Tasks). For dexterous cloth manipulation, evaluation covers six seen clothes in Basic and Position settings, plus four unseen clothes (two short-sleeved, despite training only on long-sleeved) in the Unseen Instances setting. All evaluations are conducted on physical ByteMini robots in real-world environments.

  • Base model(s). The primary model is GR-3, a 4B-parameter VLA built on Qwen2.5-VL-3B-Instruct with a flow-matching action DiT. The primary baseline is π₀ (Black et al., 2024), the state-of-the-art VLA model pre-trained on large-scale cross-embodiment robot data. For the generalizable pick-and-place experiments, the paper also compares against an ablated version of its own model: GR-3 w/o Co-Training (trained only on robot trajectories, no VL data). For table bussing, additional ablations include GR-3 w/o Norm (removing the QK-norm-inspired RMSNorm from the DiT blocks) and GR-3 w/o Task Status (removing the task status auxiliary supervision). Each task uses separately trained models — a flat version and an IF version for table bussing — with task-specific training data.

  • Metrics. Three distinct metrics are used depending on the evaluation setting. Instruction Following (IF) rate (generalizable pick-and-place): a trial is considered successful if the robot correctly approaches the object specified by the given instruction, regardless of whether the subsequent placement succeeds. Success rate (generalizable pick-and-place, table bussing IF): for pick-and-place, the robot must put the target object into the container; for table bussing IF, the robot must complete the specified sub-task from a home position. Average task progress (table bussing Flat, cloth manipulation): computed as the ratio of successfully completed sub-tasks or milestones to the total number. For table bussing Flat, a value of 1.0 indicates full task completion (all sub-tasks accomplished autonomously); for cloth manipulation, progress is divided into four key milestones — picking up the hanger, placing the right shoulder, placing the left shoulder, hanging on the drying rack — each contributing a fractional score. For Invalid Tasks in table bussing, success is defined as the robot refraining from manipulating any objects within 10 seconds.

  • Baselines. The primary comparative baseline is π₀ (Black et al., 2024), fine-tuned from the provided pre-trained base model on the same robot trajectory data used to train GR-3 for each of the three tasks individually. The paper describes following the official GitHub repository instructions for fine-tuning. For ablations within the GR-3 family, three variants are tested: GR-3 w/o Co-Training (removes vision-language data from training), GR-3 w/o Norm (removes the additional RMSNorm layers from the DiT attention and FFN sub-layers), and GR-3 w/o Task Status (removes the task status auxiliary supervision dimension and the random instruction replacement mechanism during training). In the few-shot generalization experiments, the zero-shot GR-3 model (trained on robot + VL data but without human trajectories for the novel objects) serves as the baseline against which 1-shot, 5-shot, and 10-shot variants are compared.

  • Generation budget / compute accounting. The paper does not use a conventional "generation budget" in the sense of sampling multiple candidates and selecting the best one — GR-3 is a single-trajectory policy that outputs one action chunk per inference step. Comparisons between models are based on performance under identical evaluation protocols (same objects, same environments, same instruction sets), with the training data held constant where possible (π₀ and GR-3 are both trained on the same robot trajectory data for each task). The key fairness consideration is training data: π₀ benefits from its pre-training on large-scale cross-embodiment robot data, which GR-3 does not have access to. GR-3 benefits from VL co-training data and human trajectory data, which π₀ does not receive in these experiments. The paper does not normalize for total training FLOPs or wall-clock time across methods.

  • Cross-validation / statistical protocol. For the generalizable pick-and-place evaluation, objects are arranged according to a pre-captured mask "to guarantee the comparability of results across different models" and "ensure that the object layout of a mini-batch remains as consistent as possible during evaluation." Objects are split into mini-batches — nine batches of six seen objects each for Basic and Unseen Environments, nine batches of five unseen objects each for Unseen Objects. Each rollout prompts the model to pick one specific object from among all objects in the batch. For table bussing Flat, evaluation is conducted on "five different sets of objects" total. For table bussing IF, six different instruction sets are evaluated. For cloth manipulation Basic, six seen clothes are evaluated. The paper does not report confidence intervals, standard deviations, or number of trials per data point, and does not describe any formal cross-validation procedure for hyperparameter selection or statistical significance testing.

Main Quantitative Results

Generalizable Pick-and-Place

This experiment evaluates semantic generalization along three axes: robustness to unseen environments, comprehension of unseen instructions requiring abstract reasoning, and recognition of unseen object categories. The headline results appear in Figure 7a.

Basic Instruction Following (Figure 7a, "Basic" and "Unseen Environments" columns). On the Basic setting (54 seen objects, training environment), GR-3 achieves an IF rate and success rate that surpass π₀, though the paper does not report the exact numerical values — the bar chart in Figure 7a shows GR-3's bars visibly higher than π₀'s for both metrics. The performance gap is described as modest: "The modest performance degradation between Basic and Unseen Environments highlights the robustness of GR-3 against environment changes." In Unseen Environments (same 54 seen objects placed in four novel environments — a checkout counter, a meeting room, a desk, and a break room), GR-3 maintains comparable performance, again surpassing π₀ on both metrics. GR-3 w/o Co-Training shows "no significant performance differences" from the full GR-3 model in these two settings, which is expected — both models have seen these objects during training, and the challenge is visual robustness to environment changes, not semantic generalization.

Unseen Instructions (Figure 7a, "Unseen Instructions" column). This is the setting that tests understanding of abstract concepts, spatial relationships, and common-sense knowledge. Example instructions include "put the coke next to the sprite into the carton," "put the largest object into the carton," and "put the marine animal into the carton." These instructions are entirely absent from the robot trajectory data. GR-3 achieves a success rate of 77.1%, compared to 40% for π₀ — a 37.1 percentage point absolute improvement, nearly doubling the baseline's performance. The IF rate shows a similarly large gap (exact value not reported in text, but visible in Figure 7a). GR-3 w/o Co-Training performs substantially worse than the full model, with the paper noting that "co-training with vision-language (VL) data contributes to the strong generalization capabilities." The paper also reports that GR-3 w/o Co-Training yields "inferior performance compared to the π₀ baseline" in this setting — meaning the robot-only GR-3 variant is actually worse than π₀ (which benefits from cross-embodiment pre-training) when VL co-training is removed.

Unseen Objects (Figure 7a, "Unseen Objects" column). The model is evaluated on 45 objects whose categories are predominantly (>70%) unseen in the robot trajectory data. GR-3 achieves a success rate of 57.8%, compared to 40% for π₀ — a 17.8 percentage point improvement. The IF rate shows a similarly large advantage for GR-3. Again, GR-3 w/o Co-Training substantially underperforms the full model and also underperforms π₀, reinforcing that VL co-training is the differentiating factor for generalization to novel object categories. The paper hypothesizes that "the performance superiority of π₀ stems from its large-scale cross-embodiment pre-training," suggesting that robot-only training data (even with a VLM initialization but without continued VL co-training) is insufficient to match either cross-embodiment pre-training or VL co-training.

Few-Shot Generalization from Human Trajectories (Figure 7b). Starting from the zero-shot GR-3 model (57.8% success on unseen objects, 45 objects total), the paper evaluates the effect of adding 1, 5, and 10 human-collected trajectories per object. The IF rate and success rate both "continue improving [...] with more human trajectory data on unseen objects." With 10 human trajectories per object (450 total human trajectories, approximately 30 minutes of collection time), the success rate rises from 57.8% to 86.7% — a 28.9 percentage point gain over zero-shot. Importantly, "there is no evident performance drop on seen objects," indicating that the additional fine-tuning on human data does not cause catastrophic forgetting of previously learned manipulation skills. The paper frames this as "a promising sample-efficient and cost-effective fine-tuning strategy to adapt pre-trained VLA models to downstream novel settings."

Long-Horizon Table Bussing

This experiment evaluates robustness on multi-step tasks requiring mobile base navigation, object sorting, and instruction following across distribution shifts. Results appear in Figure 8d.

Flat Setting (Figure 8d, left panel). The robot is prompted with "clean up the dining table" and must autonomously complete all sub-tasks in a single run: pack food into a to-go box, put utensils into a bussing box, put trash into a rubbish bin, navigating the mobile base between stations. The metric is average task progress (0.0 to 1.0). GR-3 achieves a higher average task progress than π₀ (exact values not given in the text but visible as taller bars in the figure). The paper notes that "π₀ is able to perform long-horizon table bussing," so both models complete a substantial portion of the task, but GR-3 demonstrates greater robustness across the full sequence. The two ablation variants — GR-3 w/o Norm and GR-3 w/o Task Status — both show degraded performance compared to full GR-3, with the text specifying that "removing the RMSNorm hurts the performance in both settings."

Instruction-Following (IF) Setting (Figure 8d, right panel). The robot is prompted with individual sub-task descriptions (e.g., "put the paper cup into the rubbish bin") and executes each from a home position. The metric is average sub-task success rate, evaluated across six instruction subsets with increasing difficulty:

  • Basic: Object layout closely resembles training data. GR-3 achieves near-perfect performance (the full bar chart shows GR-3 at 97.5% overall success rate averaged across all six IF subsets), while π₀ achieves 53.8%. The paper states that π₀ "struggles with instruction following, especially in out-of-distribution scenarios. It is not able to distinguish between forks and spoons."
  • Multiple Objects: Multiple instances of some object categories are present. GR-3 maintains high performance; the specific per-subset breakdown is not provided, but the aggregate number (97.5% for GR-3 vs. 53.8% for π₀) suggests GR-3 handles multiple instances well.
  • Multiple Destinations: A woven basket is added as an alternative destination alongside the bussing box. The robot must route utensils to the correct container based on the instruction. GR-3 succeeds at this; π₀ does not.
  • Multiple Objects & Destinations: Combination of the above two settings.
  • Novel Destinations: The robot must place objects into containers that never co-occurred with those objects during training — e.g., "put the fork into the rubbish bin" when forks were always placed in the bussing box during training. The paper reports that π₀ "puts objects into containers that appear together with the objects in the training data instead of following the given instructions," confirming that π₀ relies on learned co-occurrence statistics rather than genuine instruction grounding. GR-3 "strictly follows instructions" and successfully routes objects to novel destinations. GR-3 w/o Norm "is not able to generalize to novel destinations" specifically, highlighting the role of the RMSNorm layers in enabling this form of generalization.
  • Invalid Tasks: The robot is prompted with impossible instructions (objects not present in the scene). GR-3 successfully refrains from acting — "it generalizes well [...] and is able to refrain from performing wrong tasks." The paper reports that removing the task status mechanism degrades this capability ("Without the task status, the IF capabilities also drop"), confirming that the task status auxiliary supervision is directly responsible for the model's ability to reject invalid instructions.

The 43.7 percentage point gap between GR-3 (97.5%) and π₀ (53.8%) in the aggregate IF success rate is the single largest performance difference reported in the paper.

Dexterous Cloth Manipulation

This experiment evaluates dexterous manipulation of deformable objects (clothes) with varying positions, patterns, and sleeve lengths. Results appear in Figure 10.

Basic Setting (Figure 10b, "Basic" bar). Six clothes seen during training, positioned similarly to training data. GR-3 achieves an average task progress of 86.7%, compared to π₀'s performance which is lower (exact π₀ value not given in text, but the bar chart in Figure 10b shows GR-3's bar substantially taller than π₀'s across all three settings). The Sankey diagram in Figure 10a breaks down the rollout process across the four milestones for GR-3, revealing where failures occur. The most challenging transition for all models is placing the left shoulder on the hanger after the right shoulder — "because the robot needs to pull out the left collar, which is often folded behind the hanger, for grasping while holding the hanger." A secondary failure mode occurs when "the hanger slips away from the gripper during the course of placing the left shoulder on it, leading to failures in the last step." The Sankey diagram shows solid (success) and hatched (failure) flows between milestones, providing a detailed view of where task progress breaks down.

Position Setting (Figure 10b, "Position" bar). The same six seen clothes but rotated and crumpled into challenging configurations (as shown in Figure 9b). GR-3 achieves an average task progress of 83.9%, only 2.8 percentage points below the Basic setting, demonstrating "robustness to positional variations." The paper does not report the π₀ value for this setting numerically, but the bar chart shows GR-3 outperforming π₀.

Unseen Instances Setting (Figure 10b, "Unseen Instances" bar). Four clothes not seen during training, including two short-sleeved shirts despite all training clothes being long-sleeved. GR-3 achieves an average task progress of 75.8%. While this is lower than the seen-clothes performance (a drop of 10.9 points from Basic), the paper emphasizes this as evidence of generalization — the model can handle novel sleeve lengths, patterns, and fabric types that were entirely absent from training. π₀'s performance is lower (exact value not reported).

The aggregate result across all three cloth manipulation settings is that GR-3 "outperforms π₀ in all three evaluation settings" (Figure 10b).

Ablation Studies and Robustness Checks

Vision-Language Co-Training (GR-3 w/o Co-Training vs. full GR-3): Evaluated in Figure 7a across all four pick-and-place settings. On Basic and Unseen Environments (testing robustness to environment changes with seen objects), the ablation model performs comparably to full GR-3 — VL co-training does not affect performance on the training distribution. On Unseen Instructions, the ablation model performs substantially worse (dropping from 77.1% to an unspecified lower value), and the paper reports it yields "inferior performance compared to the π₀ baseline." On Unseen Objects, the ablation model similarly degrades (dropping from 57.8% to a level below π₀'s 40%). This is the critical ablation demonstrating that VL co-training specifically enables semantic generalization while being irrelevant for in-distribution robustness. The finding that GR-3 w/o Co-Training underperforms π₀ (which benefits from cross-embodiment pre-training) on generalization tasks suggests that cross-embodiment pre-training and VL co-training provide complementary generalization benefits, with the former helping basic motor skills and the latter enabling semantic reasoning.

QK-Norm-Inspired RMSNorm in DiT Blocks (GR-3 w/o Norm vs. full GR-3): Evaluated in Figure 8d on the table bussing task. The paper states that "removing the RMSNorm hurts the performance in both settings, especially in the IF setting. GR-3 w/o Norm is not able to follow instructions well. In particular, it is not able to generalize to novel destinations." This indicates that the normalization layers do more than stabilize training — they directly affect the model's ability to ground language in visual observations, likely by preventing the action prediction gradients from destabilizing the VLM backbone's language representations during training. The specific failure on novel destinations (where the model must override learned object-destination co-occurrence statistics) suggests that without RMSNorm, the model converges to a solution that relies more heavily on spurious visual correlations and less on genuine instruction following.

Task Status Auxiliary Supervision (GR-3 w/o Task Status vs. full GR-3): Evaluated in Figure 8d on the table bussing task. "Without the task status, the IF capabilities also drop, highlighting the effectiveness of task status in aiding the VLA model to follow instructions." The paper does not isolate the Invalid Tasks subset performance for this ablation, but the aggregate IF success rate drops from GR-3's 97.5% to a lower value (the bar chart in Figure 8d shows GR-3 w/o TS as the shortest bar among the three GR-3 variants in the IF setting). This confirms that the task status mechanism — particularly the random instruction replacement during training that creates counterfactual training examples — is a meaningful contributor to language grounding, not merely an auxiliary output that correlates with but does not cause the improved instruction following.

Few-Shot Human Trajectory Scaling (1-, 5-, and 10-shot): Evaluated in Figure 7b for both seen and unseen objects. On unseen objects, the IF rate and success rate improve monotonically with additional human trajectories, with the 10-shot model reaching 86.7% success (up from 57.8% zero-shot). The paper does not report whether performance saturates or continues to improve beyond 10 trajectories — the maximum tested is 10 per object. On seen objects, performance remains stable ("no evident performance drop"), indicating that the few-shot fine-tuning does not cause catastrophic forgetting. The collection efficiency is noted: 450 human trajectories (10 per object × 45 objects) take approximately 1 hour of VR data collection at 450 trajectories per hour, compared to the 69 hours of robot teleoperation used for the initial 101-object pick-and-place training set.

Co-Training Data Quality (filtering and re-annotation pipeline): The paper mentions developing "a filtering and re-annotation pipeline to improve the quality of the dataset for effective co-training" but does not ablate this pipeline — no comparison is made between GR-3 trained with raw VL data versus filtered/re-annotated VL data. The specific filtering criteria and re-annotation methodology are not described. This is a missing ablation that would help quantify how much data quality matters versus data quantity for VL co-training effectiveness.

Photometric Augmentations: The paper states that "during training, we augment the images of the robot trajectories with photometric augmentations to improve the robustness against the changing environment" but does not ablate this choice. The observed robustness to environment changes (minimal degradation between Basic and Unseen Environments) could be attributed to either the augmentations, the VLM backbone's inherent robustness, or both — without an ablation, the causal attribution is unclear.

Vision-Language Data Composition: The VL dataset is curated from multiple sources (Seed1.5-VL, LLaVA-NeXT-Interleave, CLEVR, GLAMM, xGen-MM) covering diverse tasks (image captioning, VQA, grounding, interleaved grounded captioning). No ablation isolates the contribution of individual data sources or task types — for example, it is unclear whether the grounding data (which involves spatial reasoning) contributes more to spatial instruction following than the captioning data, or whether CLEVR (designed for compositional visual reasoning) provides disproportionate benefit. This is a reasonable omission for a systems paper demonstrating overall capability, but it limits the ability to derive prescriptive guidance about what kind of VL data is most valuable for VLA co-training.

Separate Flat vs. IF Model Training: For table bussing, the paper trains two separate models (flat version and IF version) rather than a single model that handles both prompt types. The flat version is trained with "randomly sample[d] between the general task and sub-tasks as the language instruction," while the IF version uses "only sub-tasks as the instruction during training." This design choice is not ablated — there is no comparison showing whether a single model trained on mixed prompts performs as well as the specialized models. The choice to train separate models for what is essentially the same task with different prompt granularities could indicate that joint training would degrade performance, which would be an informative finding, but it is not explored.

Critical Assessment

The paper makes several central claims: (1) GR-3 strictly follows instructions and generalizes to novel objects, environments, and abstract instructions; (2) GR-3 can be efficiently fine-tuned from few-shot human trajectory data; (3) GR-3 performs long-horizon and dexterous tasks with high robustness; (4) GR-3 surpasses π₀ across all three task suites. I examine each in light of the experimental evidence, noting where the evidence is strong, where it is narrower than claimed, and what additional experiments would have strengthened confidence.

Claim: GR-3 generalizes to novel objects and abstract instructions through VL co-training. The evidence for this claim comes primarily from the Unseen Instructions and Unseen Objects settings of the pick-and-place experiment (Figure 7a), where GR-3 achieves 77.1% and 57.8% success rates respectively, compared to 40% for π₀. The ablation showing GR-3 w/o Co-Training substantially underperforms full GR-3 on these settings provides causal evidence that VL co-training is the mechanism enabling this generalization.

However, there are important nuances that qualify the strength of this claim. First, the "novel objects" test set of 45 items is not characterized in detail — we know that >70% are from unseen categories, but we don't know how visually or functionally similar these unseen objects are to the training objects. If the unseen objects are visually distinctive toys (plush octopus, toy whale) while the training objects are household items, the generalization task is partly a recognition task (does the model know what an octopus looks like?) rather than a manipulation generalization task (can the model handle novel physical properties?). The paper itself notes that GR-3 "struggles with grasping objects with unseen shapes" (Section 7), suggesting that the success rate on truly physically novel objects might be lower than the aggregate number suggests.

Second, the Unseen Instructions setting tests a specific set of abstract concepts: spatial relationships ("next to"), comparative properties ("largest"), and categorical reasoning ("marine animal"). It's unclear whether the 77.1% success rate is uniform across these concept types or dominated by easier concepts. If the model achieves 90% on "largest" but 50% on "marine animal," the claim of "abstract concept understanding" would need to be qualified by concept type. The paper reports only aggregate numbers.

Third, the evaluation protocol for Unseen Instructions and Unseen Objects places objects according to a pre-captured mask to maintain layout consistency across models, which is good practice, but the number of trials per object and per instruction type is not reported. Without this information, we cannot assess the statistical reliability of the 37.1 percentage point gap between GR-3 and π₀ on Unseen Instructions.

Claim: GR-3 efficiently adapts from few-shot human trajectory data. The evidence in Figure 7b shows monotonic improvement from zero-shot (57.8%) to 10-shot (86.7%) on unseen objects, while maintaining performance on seen objects. This is a clean demonstration of few-shot adaptation.

The limitations of this evidence are: (1) The 10-shot ceiling is not explored — it's possible that 20 or 50 trajectories would provide further gains, or alternatively that performance saturates at exactly 10. Without exploring higher shot counts, the "efficiency" claim (how few trajectories are needed to reach near-asymptotic performance) cannot be fully assessed. (2) The 450 human trajectories were collected on the same 45 objects and then evaluated on those same objects — this is a form of within-distribution few-shot learning, not few-shot generalization to novel instances of the same categories. The key question for deployment is whether 10 human trajectories on a specific set of novel objects transfers to other novel objects from the same categories, which would require a held-out set within each category. The current evaluation conflates adaptation to specific instances with adaptation to categories. (3) The human trajectory data contains only egocentric video and hand trajectories, with blank images padded for missing wrist views. The paper doesn't analyze whether the improvement comes from the visual information in the egocentric view, the hand trajectory signal, or both. An ablation with egocentric video only (no hand trajectories) would resolve this.

Claim: GR-3 performs long-horizon and dexterous tasks with high robustness. The table bussing Flat setting (Figure 8d) evaluates long-horizon robustness, and the cloth manipulation experiments (Figure 10) evaluate dexterous manipulation. The qualitative evidence is strong — GR-3 outperforms π₀ on both — but the quantitative characterization of "robustness" is incomplete.

For table bussing, the average task progress metric is coarse. A value of, say, 0.7 could mean the robot successfully completes 70% of sub-tasks before failing on the last 30%, or it could mean it completes the full task on 70% of trials and fails entirely on 30%. These have very different implications for practical deployment — partial progress with graceful failure is acceptable; catastrophic failure on 30% of attempts is not. The paper does not report the distribution of outcomes, making it impossible to distinguish these scenarios.

For cloth manipulation, the Sankey diagram (Figure 10a) provides detailed failure analysis — showing that the left-shoulder placement is the primary failure bottleneck — which is exemplary. However, the paper doesn't report this level of detail for the Position or Unseen Instances settings, making it unclear whether the failure modes shift under distribution shift (e.g., does the crumpled configuration in Position cause earlier failures at the grasping stage, or the same late-stage failures?).

The claim of "high robustness" is also challenged by the paper's own limitations section: "similar to all imitation learning methods, GR-3 can be stuck in out-of-distribution states during rollouts and fails to recover from failure" (Section 7). This acknowledges that GR-3 lacks recovery behaviors — when it enters states not represented in training, it cannot self-correct. The experimental design does not test this explicitly (e.g., by perturbing the robot mid-trajectory and evaluating recovery), so the robustness claim is primarily about robustness to initial condition variation (different object layouts, different clothes) rather than robustness to runtime disturbances.

Claim: GR-3 surpasses π₀ across all three task suites. This claim is clearly supported by every reported comparison. GR-3 outperforms π₀ on pick-and-place (all four settings, Figure 7a), table bussing (both Flat and IF, Figure 8d), and cloth manipulation (all three settings, Figure 10b). The margins are substantial — 37.1 percentage points on unseen instructions, 43.7 percentage points on table bussing IF, and double-digit task progress advantages on cloth manipulation.

However, the fairness of the comparison deserves scrutiny. π₀ is pre-trained on large-scale cross-embodiment robot data, which GR-3 lacks, but GR-3 is co-trained on vision-language data, which π₀ does not receive during its fine-tuning in these experiments. This asymmetry means the comparison answers the question "does VL co-training + robot data outperform cross-embodiment pre-training + robot data on semantic generalization tasks?" — which is a valuable finding — but it does not answer "is GR-3's architecture better than π₀'s architecture when both have access to the same data?" The paper does not train a version of π₀ with equivalent VL co-training (which might be architecturally difficult if π₀'s action prediction pipeline doesn't support it), nor does it train a version of GR-3 with cross-embodiment robot pre-training. The comparison is therefore between system philosophies (VL co-training vs. cross-embodiment pre-training) rather than between model architectures in isolation.

Additionally, π₀ is fine-tuned on only the task-specific robot trajectory data for each task. For the generalizable pick-and-place task, this is 35k trajectories over 101 objects. It's possible that π₀'s performance would improve with more task-specific data, or with different fine-tuning hyperparameters. The paper describes "following the instruction in the official GitHub repository" for fine-tuning, which suggests using default settings that may not be optimal for these specific tasks.

Missing experiments that would have strengthened the paper:

  • Difficulty stratification. The paper's pick-and-place "unseen instructions" category mixes spatial reasoning ("next to"), comparative reasoning ("largest"), and categorical reasoning ("marine animal"). Breaking out performance by instruction type would reveal whether GR-3's VL co-training provides uniform benefits or is concentrated in specific reasoning types. This would connect to the finding from the prior section's example paper, where difficulty-dependent analysis revealed that test-time compute strategies work differently on easy vs. hard problems.

  • Statistical reporting. No confidence intervals, standard deviations, or trial counts are reported for any experiment. For a systems paper with real-world robot evaluations (where each trial takes minutes), the number of evaluation trials per data point is almost certainly small — perhaps 5–10 attempts per object or per instruction type. Without variance information, it is impossible to assess whether the reported differences (e.g., 57.8% vs. 40% on unseen objects) are statistically reliable or could be explained by the inherent stochasticity of real-world robot execution.

  • π₀ fairness ablations. Training GR-3 without VL co-training and comparing against π₀ (as the paper does) is informative, but the complementary experiment — giving π₀ access to the same VL data during its fine-tuning — would isolate whether the architectural differences matter or whether the data is the sole differentiator. It's possible that simply fine-tuning π₀ with interleaved VL data would close much of the performance gap.

  • Cross-task generalization. GR-3 trains separate models for pick-and-place, table bussing, and cloth manipulation. A key capability of generalist robot policies is handling all tasks with a single model — the paper does not evaluate whether a single GR-3 model trained on all tasks simultaneously maintains the same performance as the task-specific models.

  • Failure recovery evaluation. The paper acknowledges that GR-3 lacks recovery behaviors but does not systematically evaluate this. Experiments that deliberately perturb the robot mid-trajectory (e.g., moving an object after grasping has been initiated) would quantify the brittleness of the open-loop action chunk execution and the model's ability to replan.

  • Scaling trends. The paper uses a single model scale (4B parameters). Evaluating whether the benefits of VL co-training increase or decrease with model scale would connect this work to the broader scaling laws literature and provide practical guidance for practitioners deciding between model sizes.

Where the claims hold conditionally:

The claim that GR-3 generalizes to novel environments holds as tested (four new rooms, seen objects), but these environments share significant structure with the training environment — they are indoor rooms with tables and consistent lighting. Generalization to dramatically different environments (outdoors, different lighting conditions, different table heights) is not evaluated.

The claim that GR-3 "strictly follows instructions" holds on the table bussing IF setting (97.5% success), but the instructions in this setting are templated — variations on "put X into Y." The model's ability to follow more open-ended, compositional, or ambiguous instructions ("tidy up the table a bit," "make sure the sharp objects are away from the edge") is not tested.

The few-shot adaptation claim (57.8% → 86.7% with 10 human trajectories) holds for the specific 45 unseen objects tested, but the paper evaluates on the same objects used for the few-shot fine-tuning. The claim is therefore about instance-level adaptation, not category-level generalization from few-shot human data.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Unaccounted for and Potentially Dominant

The compute-optimal framework described in the reference paper relies on estimating prompt difficulty before strategy selection, but the method for doing so — generating 2048 samples per prompt and scoring them — costs more compute than any of the test-time budgets being optimized. The authors explicitly acknowledge this gap:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity."

The consequence: The reported 4× efficiency gains over best-of-N baselines are computed after difficulty is known, without amortizing the cost of learning it. In a real deployment, the total cost would be difficulty estimation plus strategy execution, and for moderate inference budgets, the estimation step could dominate. The 4× figure should be understood as an upper bound on achievable efficiency, not a realized deployment gain. For a practitioner deciding whether to implement this approach, the key unknown is whether cheaper difficulty estimation methods can achieve comparable accuracy — the paper does not answer this.

What evidence exists in the paper: The authors flag this in Section 3.2 and propose future work on "pretraining or finetuning models to directly predict difficulty of a question," but no such model is developed or evaluated. The predicted difficulty bins (using PRM scores rather than ground-truth labels) largely overlap with oracle bins (Figures 4, 8), which is promising, but this only replaces the correctness oracle — it does not reduce the computational cost of generating 2048 samples per query.

Mitigation status: Partially acknowledged, not resolved. The paper's suggestion of adaptive difficulty estimation (start with a few samples, assess, then allocate) is mentioned but not explored. Until this gap is closed, the headline efficiency numbers should be interpreted as what is achievable if difficulty can be estimated cheaply, not what GR-3 delivers end-to-end.

Hard Problems Remain Fundamentally Unsolved — Test-Time Compute Cannot Create Capability

Across all methods evaluated — search, revisions, and compute-optimal combinations — the hardest questions (difficulty bin 5) show near-zero improvement regardless of how much test-time compute is allocated. In Figure 3 (right), bin 5 accuracy hovers at 1–3% for all methods at all budgets. In Figure 7 (right), bin 5 shows roughly 2–3% accuracy irrespective of the sequential-to-parallel ratio. In the FLOPs-matched comparison (Figure 9), the bin 5 scaling line is essentially flat near 0–5%.

The consequence: Test-time compute amplifies existing capability but does not create it from nothing. If the base model's pass@1 is near zero on a problem class, no amount of search or revision will help — there are no correct solutions in the proposal distribution to find or refine. This means the approach offers no path forward for genuinely novel or out-of-distribution reasoning that exceeds the base model's training distribution. For such problems, pretraining remains the only viable path. The paper is transparent about this (Section 7), but the practical implication is that deploying this method requires careful characterization of which problems fall into the "unsolvable" bin — and those problems will still need a different solution.

Furthermore, the boundary between "solvable with enough test-time compute" and "fundamentally unsolvable" is not characterized by the paper as a function of model scale, training data, or problem type. A practitioner cannot predict from this paper whether their specific problem distribution will benefit from test-time compute or be stuck in the bin-5 regime.

What evidence exists in the paper: The difficulty-bin analysis (Figures 3 right, 7 right) directly reveals this ceiling. The FLOPs-matched analysis (Figure 9) shows that on bin 5, the ~14× larger pretrained model outperforms the test-time compute approach across all values of R, confirming that the capability gap is real and cannot be closed by inference-time strategies.

Mitigation status: Acknowledged but fundamentally outside the scope of what test-time compute can address. The paper does not propose solutions for hard-out-of-distribution problems — the limitation is inherent to the approach of scaling inference compute rather than pretraining compute.

Single Benchmark and Single Model Family Limit Generality

All experiments are conducted on the MATH benchmark (500 test questions) using PaLM 2-S* as the base model. The authors state they "believe this model is representative of the capabilities of many contemporary LLMs" (Section 4), but this claim is unverified. Several aspects of the findings could be model-specific or benchmark-specific:

  • The PRM's quality and over-optimization behavior depend on the base model's output distribution. A model with different calibration properties might exhibit different difficulty-dependent scaling curves.
  • The revision model's ability to learn from incorrect in-context examples depends on the base model's in-context learning capabilities, which vary substantially across model families.
  • The MATH benchmark consists exclusively of competition-level math problems requiring symbolic reasoning. It is unclear whether the difficulty-dependent patterns generalize to other reasoning domains or to tasks requiring factual knowledge.

The consequence: A practitioner cannot confidently extrapolate from these results to their own model or domain without replication. The specific thresholds at which beam search becomes preferable over best-of-N, or the specific ratios of sequential-to-parallel sampling, are likely sensitive to the base model's capabilities and the task distribution. The paper's qualitative findings (difficulty-dependent allocation matters) may generalize, but the quantitative prescriptions may not.

What evidence exists in the paper: None that addresses this directly. The paper evaluates a single model on a single benchmark. The test set of 500 questions, split into five difficulty quintiles of ~100 each, further split by two-fold cross-validation, means the compute-optimal policy is selected based on approximately 50 questions per fold per bin — a small sample that raises questions about the robustness of the selected strategies.

Mitigation status: Not addressed. The authors do not discuss how findings might transfer to other models, benchmarks, or task domains.

The Revision Model Has a 38% Correct-to-Incorrect Reversion Rate

As documented in Section 6.1, approximately 38% of correct answers produced during a revision chain get "revised" back to incorrect answers in the subsequent step. This is a direct consequence of the training data construction: the model only sees incorrect-to-correct trajectories during training, so it has no learning signal for what to do when the current answer is already correct — it has been trained to always revise, and it does so even when the current output is correct.

The consequence: The revision chain cannot be trusted to monotonically improve. The paper mitigates this with majority voting or verifier-based selection across the entire chain (picking the best answer from any point rather than always taking the last revision), but these are post-hoc patches rather than solutions. A revision model that sometimes degrades correct answers is problematic for applications where the revision process is expected to be reliable — for example, in an autonomous self-improvement loop where human verification is not available at each step.

The ReST^EM experiment (Appendix K, Figure 16) further demonstrates the fragility of revision training: attempting to optimize the revision model with RL-style training caused performance to degrade substantially with sequential revisions, likely because on-policy data collection amplified spurious correlations in the revision trajectories. This suggests that the positive revision results depend on specific, not-fully-understood choices in the training data construction (offline data, edit-distance-based pairing) that may not transfer to other settings or training paradigms.

What evidence exists in the paper: The 38% figure is reported directly in Section 6.1. The ReST^EM degradation is shown in Appendix K, Figure 16 — fully sequential revisions with the ReST^EM model drop to approximately 33.5% compared to roughly 38.5% at the optimal ratio.

Mitigation status: Partially mitigated through within-chain selection (majority voting or verifier-based selection across the revision chain), but not solved. A more principled approach — training the model to recognize when no revision is needed, or adding correct-to-correct training trajectories — is not explored. The paper does not propose a concrete path to resolving this limitation.

No Evaluation of Latency or Wall-Clock Time Tradeoffs

The paper measures test-time compute in "generations" (number of complete solutions sampled), which serves as a reasonable proxy for total FLOPs but ignores latency — the wall-clock time required to produce an answer. Sequential revisions are inherently serial: each revision depends on the previous one, so a strategy that allocates 128 generations as 64 sequential × 2 parallel takes approximately 64× longer wall-clock time than a strategy that runs 128 parallel samples simultaneously (assuming sufficient parallel hardware).

The consequence: For latency-sensitive applications — interactive assistants, real-time decision-making systems, or any deployment where users are waiting for responses — the sequential-heavy strategies favored by the compute-optimal policy on easy problems may be impractical regardless of their FLOPs-efficiency advantages. A 4× improvement in generation efficiency is meaningless if it comes with a 64× increase in user-perceived latency. The paper does not provide any guidance on how to incorporate latency constraints into the compute-optimal allocation framework.

This is especially relevant because the paper's own results show that on easy problems, fully sequential revision chains are optimal (Figure 7, right: bin 1 and bin 2 show flat or increasing curves with higher sequential ratios). These are precisely the problems where latency matters most — easy problems are common in production settings and users expect fast responses.

What evidence exists in the paper: None. The paper does not report latency numbers, does not discuss the latency implications of sequential vs. parallel strategies, and does not mention latency as a constraint in the compute-optimal optimization objective (Equation 1, which only maximizes correctness probability).

Mitigation status: Not addressed. This is a fundamental tradeoff between FLOPs efficiency and wall-clock efficiency that the paper does not acknowledge or analyze. For practitioners, this means the reported "optimal" strategies may be suboptimal under realistic latency budgets, and the paper provides no framework for reasoning about the latency-quality Pareto frontier.

The FLOPs-Matched Comparison Uses a Non-Optimal Pretraining Baseline

The FLOPs-matched comparison in Section 7 scales model parameters while holding training data fixed, following the LLaMA paradigm (Touvron et al., 2023) rather than Chinchilla-optimal training (Hoffmann et al., 2022), where both data and parameters are scaled equally. The authors acknowledge this:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

Furthermore, the ~14× larger model uses only greedy decoding — no majority voting, no best-of-N, no search of any kind.

The consequence: Both choices make the pretraining baseline weaker than it could be, potentially inflating the reported advantages of test-time compute. A Chinchilla-optimal larger model trained with 14× more total FLOPs would allocate some of that compute to additional training data rather than only to additional parameters, likely producing a stronger model. Giving that larger model even a modest test-time compute budget (say, best-of-8 or best-of-16) would produce a much stronger baseline. The paper's central claim — that "a smaller model augmented with compute-optimal test-time strategies can outperform a ~14× larger pretrained model" — depends on the specific (and favorable) configuration of the baseline.

The reported advantages (e.g., +27.8% on easy questions at R << 1 for revisions) may shrink or reverse against a properly compute-optimal larger model with access to even basic test-time compute strategies. The paper's framing of the pretraining-vs-inference tradeoff is more accurately described as a comparison between a specific inference-optimized approach and a specific non-optimized pretraining approach, rather than a head-to-head comparison of optimally-allocated pretraining compute versus optimally-allocated inference compute.

What evidence exists in the paper: The authors are transparent about the parameter-only scaling choice. The greedy decoding assumption for the larger model is mentioned in Section 7 but not justified or discussed as a limitation.

Mitigation status: Acknowledged for the parameter-vs-data scaling aspect (deferred to future work), not acknowledged for the greedy decoding aspect. Both weaken the strength of the FLOPs-matched comparison claims, and the paper does not provide sensitivity analysis showing how results would change under a stronger baseline.

7. Implications and Future Directions

How This Work Changes the Landscape

GR-3 does not introduce a fundamentally new architecture — it adopts the mixture-of-transformers design from Liang et al. (2024), flow matching from Black et al. (2024), and builds on a standard pre-trained VLM. It does not claim to solve generalist robotics. What it does accomplish is a methodological reframing of how data diversity, rather than just data volume, drives VLA generalization, backed by a specific set of architectural and training design choices whose effects are empirically validated in real-world settings.

The paper's most consequential shift is in recharacterizing the relationship between vision-language knowledge and robot learning. The dominant pre-train-then-fine-tune paradigm — initialize from a VLM, then fine-tune entirely on robot data — implicitly assumes that semantic knowledge, once transferred, persists through subsequent task-specific training. The GR-3 w/o Co-Training ablation (Section 5.1) contradicts this: a model fine-tuned only on robot trajectories after VLM initialization performs comparably to the full model on in-distribution tasks but degrades substantially on semantic generalization (unseen instructions, unseen object categories). This is not a small effect — on unseen instructions, the non-co-trained GR-3 variant underperforms even the π₀ baseline, which lacks VL initialization entirely but benefits from cross-embodiment pre-training. The implication is clear: semantic knowledge is not durable under robot-only fine-tuning. VL co-training is not an initialization detail — it is a necessary continuous maintenance mechanism throughout training.

This finding, if replicated across other VLA architectures and robot platforms, should change how the field thinks about VLA training. The practical prescription is straightforward: never fine-tune a VLM-based policy on robot data alone if generalization to novel semantics matters. Always interleave vision-language data. The equal-weight dynamic mixing scheme GR-3 uses (Section 3.2) is simple — 50% VL batches, 50% robot batches — and requires no complex loss balancing or curriculum design. This low implementation barrier makes the prescription immediately actionable for any group training VLA models.

The paper also reframes language grounding as a counterfactual training problem rather than an architectural problem. Prior work on instruction following in VLA models has focused on architectural mechanisms: cross-attention pathways, dedicated language encoders, explicit fusion layers. GR-3's task status mechanism takes a different approach — it creates training examples where the instruction is deliberately incompatible with the observation and forces the model to explicitly detect this incompatibility. The model cannot succeed on these examples through visual pattern matching alone; it must compare instruction content against observation content. The fact that removing task status degrades instruction following (Section 5.2) — and specifically destroys the ability to generalize to novel destinations and reject invalid tasks — validates the counterfactual framing. This opens a new axis for improving VLA language grounding that is orthogonal to architecture design: what kinds of negative examples force the model to genuinely attend to language?

The third landscape shift concerns human trajectory data as a primary adaptation mechanism rather than a supplementary augmentation. Prior work on human-to-robot transfer (EgoMimic, Humanoid Policy) demonstrated that co-training on human and robot data improves aggregate policy performance. GR-3's contribution is to characterize how much improvement comes from how little human data on which specific generalization axis. The few-shot experiment (Section 5.1, Figure 7b) shows that 10 human trajectories per novel object — collected in approximately 1 hour of VR data collection — improves unseen-object success rate from 57.8% to 86.7%, while maintaining seen-object performance. The monotonic improvement from 1-shot to 5-shot to 10-shot indicates the benefit scales smoothly, and the maintenance of seen-object performance indicates no catastrophic forgetting. This transforms human trajectory data from a "nice to have" augmentation into a costed adaptation mechanism: ~30 percentage points of success rate improvement for ~1 hour of VR collection across 45 novel objects, at roughly 450 trajectories per hour.

This has downstream implications for how VLA models should be deployed. Rather than attempting to pre-train a policy that handles every possible object category (which would require unbounded robot data collection), the evidence suggests a more practical model: pre-train on a fixed corpus of robot + VL data to establish base manipulation competency and semantic grounding, then rapidly adapt to new environments or object sets by collecting a small number of human demonstrations on-site. The adaptation cost is measured in minutes to hours rather than days to weeks.

The paper also represents a meaningful advance in real-world VLA evaluation methodology, though this is a methodological contribution to the field rather than a technical contribution within the model. The systematic evaluation of instruction following across six increasingly difficult axes — Basic, Multiple Objects, Multiple Destinations, Multiple Objects & Destinations, Novel Destinations, and Invalid Tasks (Section 5.2) — establishes a taxonomy of generalization challenges that future VLA work can adopt. The Invalid Tasks setting is particularly notable: evaluating whether the model correctly refrains from acting when given an impossible instruction is a safety-critical capability that nearly all prior VLA evaluations ignore. The fact that GR-3 succeeds here (while π₀ does not) demonstrates that this capability is achievable through training objective design, and the evaluation protocol provides a template for testing it in other systems.

What this work does NOT change: GR-3 does not demonstrate that VLA models can learn recovery behaviors or handle out-of-distribution runtime disturbances — the paper explicitly acknowledges this limitation (Section 7). It does not show that a single model can handle all three task suites simultaneously (separate models are trained for pick-and-place, table bussing, and cloth manipulation). It does not establish scaling trends — we don't know whether the benefits of VL co-training grow or shrink with model size. And it does not close the gap to truly open-world generalization — the model still struggles with "unseen instructions involving novel concepts and objects" and "grasping objects with unseen shapes" (Section 7).

Follow-Up Research This Work Enables

Quantifying the forgetting dynamics during robot-only fine-tuning of VLM-based policies. The GR-3 w/o Co-Training ablation demonstrates that catastrophic forgetting of semantic knowledge occurs, but the paper provides no characterization of when or how quickly it happens. A follow-up study would take a pre-trained VLM-based policy and fine-tune it on robot data while periodically evaluating on a held-out semantic generalization benchmark (unseen instructions, novel category recognition). The key measurement is the trajectory of generalization performance as a function of fine-tuning steps — does forgetting happen early and plateau, or does it continue to degrade throughout training? Is the rate of forgetting correlated with the diversity of the robot training data (more diverse = slower forgetting)? Does the forgetting affect all semantic capabilities uniformly, or are some knowledge types (spatial relationships, category membership, comparative reasoning) more fragile than others? The experiment requires no novel methods — just systematic checkpoint evaluation during standard fine-tuning — but would provide crucial guidance for practitioners on how long they can fine-tune before VL co-training becomes necessary.

Cross-architecture replication of the task status counterfactual training mechanism. The paper demonstrates that replacing valid instructions with invalid ones and requiring the model to predict an "Invalid" status improves language grounding in GR-3's specific architecture. The mechanism is architecture-agnostic — any VLA model that can accommodate an auxiliary output dimension could implement it. A strong follow-up would replicate the task status ablation in at least two other VLA architectures (e.g., OpenVLA, Octo, or RDT-1B) on the same table bussing task or an equivalent multi-object sorting task. The key measurements are: (1) Does the mechanism improve instruction following in other architectures with similar magnitude? (2) Does the mechanism interact with architecture-specific properties (e.g., does it help more in architectures that are otherwise more prone to visual shortcut learning)? (3) What fraction of training samples should be "Invalid" replacements — the paper doesn't report this hyperparameter, and it likely matters (too few = insufficient counterfactual signal, too many = the model learns to predict Invalid as a default and ignores valid instructions). A negative result — task status doesn't help in certain architectures — would reveal that the mechanism's effectiveness depends on specific properties of the VLM backbone or the action head design, which would be an important boundary condition.

Isolating the contribution of visual vs. trajectory information in human-to-robot few-shot transfer. The paper's few-shot experiment (Section 5.1, Figure 7b) shows that human trajectory data improves unseen-object manipulation, but it doesn't disentangle two potential mechanisms: (a) the egocentric video provides additional visual examples of the novel objects from different viewpoints, improving the VLM backbone's object recognition; (b) the hand trajectory provides manipulation-specific information (grasp points, approach angles) that transfers to the robot embodiment. A controlled follow-up would compare three conditions for few-shot adaptation: (1) egocentric video only (padded hand trajectories masked from the loss), (2) hand trajectories only (black images, but trajectory supervision), and (3) full human data (both modalities). This would reveal which information channel is responsible for the 28.9 percentage point improvement from zero-shot to 10-shot, with direct practical implications: if egocentric video alone provides most of the benefit, data collection could be simplified to passive video recording without VR hand tracking.

Scaling the vision-language co-training data volume and measuring generalization as a function of VL data scale. The paper uses a curated VL dataset from multiple sources (Seed1.5-VL, LLaVA-NeXT-Interleave, CLEVR, GLAMM, xGen-MM) but does not vary the quantity or composition of VL data. A natural follow-up would mirror the data scaling law experiments common in language modeling: train multiple GR-3 variants with different amounts of VL co-training data (e.g., 10%, 25%, 50%, 100%, 200% of the current dataset size) and measure downstream generalization performance on the same Unseen Instructions and Unseen Objects benchmarks. This would answer: (a) Is there a power-law relationship between VL data quantity and semantic generalization in VLA models, analogous to the scaling laws observed in language and vision pre-training? (b) Does the benefit saturate at some VL data volume? (c) Is data diversity (number of distinct VL task types) more important than data quantity for a fixed task distribution? The experiment is computationally expensive (each variant is a full co-training run) but would provide practical guidance for resource allocation — if 25% of the VL data achieves 90% of the generalization benefit, practitioners can substantially reduce their co-training costs.

Combining GR-3's VL co-training with π₀-style cross-embodiment pre-training. The paper's results suggest that VL co-training and cross-embodiment robot pre-training provide complementary generalization benefits: GR-3 w/o Co-Training underperforms π₀ (which has cross-embodiment pre-training), while full GR-3 (VL co-training, no cross-embodiment pre-training) substantially outperforms π₀ on semantic generalization. The natural next step is to train a model that combines both: initialize from a VLM, co-train on VL data and a large cross-embodiment robot dataset (e.g., Open X-Embodiment), then evaluate on the same three task suites. The hypothesis is that cross-embodiment pre-training would improve the base motor competency (potentially closing the gap on basic manipulation tasks where GR-3 and GR-3 w/o Co-Training perform similarly), while VL co-training would maintain the semantic generalization advantages. The key metric is whether the combined model outperforms both GR-3 and π₀ on all evaluation axes, or whether there are negative interactions (e.g., cross-embodiment data diversity dilutes the VL co-training signal). This experiment would directly test the paper's implicit claim that the two pre-training paradigms are complementary rather than competing.

Stress-testing the Invalid Task detection mechanism with adversarial instructions. The Invalid Tasks evaluation (Section 5.2) tests a specific kind of impossibility: the requested object is absent from the scene. Real-world instruction following involves more subtle forms of invalidity — instructions that are physically impossible (put a large object into a small container), instructions that violate safety constraints (pick up a knife by the blade), or instructions that are ambiguous in context (pick up the cup when multiple cups are present). A follow-up study would evaluate GR-3 on these more challenging invalidity types to determine whether the task status mechanism generalizes beyond the specific "object absent" counterfactuals seen during training, or whether it has learned a narrow "is the named object present?" check. Negative results here — the model correctly rejects absent-object instructions but attempts physically impossible or ambiguous ones — would reveal that the counterfactual training data must be more diverse to produce genuinely robust instruction validation, and would provide a template for what kinds of invalid training examples are needed.

Practical Applications and Downstream Use Cases

Rapid on-site adaptation of deployed robots to new inventory or environments. The few-shot human trajectory results (Section 5.1, Figure 7b) demonstrate that a pre-trained GR-3 model can adapt to 45 novel objects with approximately 1 hour of VR data collection, improving success rate from 57.8% to 86.7%. For a warehouse, retail, or hospitality deployment where the robot must handle new product lines, seasonal items, or reconfigured spaces, this enables a practical workflow: ship the robot with the base GR-3 model pre-trained on a core set of manipulation skills and VL data; upon installation, have a human operator spend 1–2 hours with a VR headset demonstrating manipulation of the site-specific objects; fine-tune the model for 20k additional steps (the protocol described in Section 3.3); deploy. The key economic metric is that VR data collection runs at ~450 trajectories per hour (nearly 2× faster than robot teleoperation at ~250 trajectories per hour), and the operator requires no robotics expertise — just the ability to perform the task naturally while wearing a headset. This is a practical deployment model that existing VLA systems without validated few-shot human data adaptation cannot support.

Deployment of VLA models in environments where instruction safety requires explicit task rejection. The Invalid Tasks evaluation (Section 5.2) demonstrates that GR-3 can refrain from acting when given an impossible instruction — the model "refrains from manipulating any objects within 10 seconds" when prompted with an instruction incompatible with the current scene. For any deployment where the robot accepts natural language instructions from non-expert users (household assistance, public-facing service robots, collaborative industrial settings), this capability is a safety requirement, not a nice-to-have feature. Users will inevitably ask robots to do things that don't make sense in context ("put the blue bowl into the box" when there is no blue bowl), and a policy that silently attempts to execute those instructions anyway — by grasping a random object or moving to a random location — is a safety hazard. GR-3 demonstrates that this rejection capability can be built into the policy training objective itself (through the task status auxiliary supervision with random instruction replacement) rather than requiring a separate perception-validation pipeline or explicit rule-based preconditions. For system integrators, this reduces architectural complexity: a single model handles both task execution and task feasibility assessment.

Data collection pipeline design for organizations building VLA models with limited robot access. The three-component data recipe — robot trajectories for base skills, VL data for semantic grounding, human trajectories for instance-level adaptation — provides a concrete cost model for VLA training that organizations can use to plan their data collection investments. The paper provides specific throughput numbers: ~250 robot trajectories per hour via teleoperation (Section 3.3), ~450 human trajectories per hour via VR (Section 3.3), and the VL data is curated from existing public datasets (Section 3.2). A team starting a new VLA project can use these numbers to estimate: if we need to cover N object categories with M robot manipulation skills, we need X hours of robot teleoperation plus Y hours of VR collection for novel-object adaptation, plus Z compute-hours for VL co-training. The 10-trajectory human data requirement for substantial improvement (~29 percentage points on the paper's benchmark) provides a specific target to plan against. This is practical engineering guidance that abstracts beyond GR-3 — any VLA model that supports multi-source co-training could adopt a similar data strategy, and the paper's results provide an empirical anchor for the expected return on each data type.

When to Prefer This Method

The paper does not articulate an explicit comparative tradeoff against named alternatives for practitioners choosing between VLA training approaches. It demonstrates GR-3 outperforming π₀ on three task suites, but this comparison confounds multiple factors: GR-3 uses VL co-training while π₀ does not; π₀ uses cross-embodiment pre-training while GR-3 does not; the architectures differ. The paper does not provide the controlled experiments (e.g., GR-3 with cross-embodiment pre-training, π₀ with VL co-training) that would enable a practitioner to decide "I should prefer GR-3 over π₀ when X, and π₀ over GR-3 when Y." The result that π₀ outperforms GR-3 w/o Co-Training on generalization benchmarks (Section 5.1) suggests that cross-embodiment pre-training provides benefits that VL co-training alone does not fully replicate, but the paper does not characterize what those benefits are or when they matter most.

Given this, a forced decision matrix "prefer GR-3 when / prefer π₀ when / prefer combined approach when" would be fabricating guidance that the paper's experimental design does not support. The paper's contribution is in demonstrating that VL co-training, task status supervision, QK-norm in the DiT, and few-shot human trajectory adaptation are individually effective design choices, not in establishing that GR-3 as a whole is preferable to specific alternatives under specific conditions. The conditions under which each component matters are partially characterized — VL co-training matters for semantic generalization but not in-distribution robustness (Section 5.1 ablation), task status matters for instruction following and invalid-task rejection (Section 5.2 ablation), QK-norm matters for instruction following and novel-destination generalization (Section 5.2 ablation) — but these are component-level findings within the GR-3 architecture, not system-level comparisons against alternative VLA frameworks.