GATS: Gather-Attend-Scatter

🎯 Pitch

Frozen pretrained models can be stitched into multimodal systems without fine-tuning simply by steering their internal activations—the paper introduces GATS to do this and shows that disabling this steering causes a catastrophic performance collapse from 89% to 30% on a robotics task, proving that activation reprogramming, not just cross-modal pooling, is the essential ingredient.

1. Executive Summary

This paper introduces GATS (Gather-Attend-Scatter), a novel architectural module for combining multiple pretrained foundation models — both frozen and trainable — into larger multimodal networks without fine-tuning the original components, thereby preventing catastrophic forgetting of pretrained knowledge. The approach is demonstrated across games (Atari Pong), robotics (Language-Table, YCB), and multimodal generation (image captioning and text-to-image) using frozen Chinchilla language models, Phenaki video models, and pretrained ViT vision transformers. GATS operates through three mechanisms — gathering activations from all component models with per-modality local context windows, attending over them in a shared projected space via transformer layers, and scattering the combined representations back to steer each model's forward pass via gated residual connections — enabling frozen vision models to be reprogrammed for downstream tasks without parameter updates. On Language-Table, the GATS-based image agent achieves 89.0% success rate while keeping the 2.7B-parameter vision model frozen, and ablations reveal that disabling vision steering causes a catastrophic drop from 89.0% to 30.4%, establishing that the steering mechanism is the critical enabler of effective frozen-model integration rather than a minor auxiliary component.

2. Context and Motivation

The Core Problem: Building Multimodal Systems Without Destroying Pretrained Knowledge

The fundamental problem this paper addresses is deceptively practical: how do you combine multiple specialized pretrained models into a single multimodal system without ruining what each model already knows? The AI community has invested enormous resources into training large-scale foundation models — Chinchilla for language (Hoffmann et al., 2022), Flamingo for vision-language (Alayrac et al., 2022), Phenaki for video generation (Villegas et al., 2023), Parti for text-to-image (Yu et al., 2022) — and each represents a concentrated repository of knowledge acquired from massive, diverse datasets. The natural impulse when building a multimodal system (say, a robot that understands language instructions and processes visual input) is to take these models and wire them together somehow. But how?

The naive approaches all have fatal flaws. One option is to fine-tune the pretrained models on your multimodal task data. This works in the sense that you get a single unified model, but you risk catastrophic forgetting — the model overwrites its hard-won pretrained knowledge with whatever patterns exist in your (typically much smaller and narrower) multimodal dataset. A language model that once understood the nuance of "carefully place the fragile object on the shelf" might, after fine-tuning on a limited robotics dataset, collapse that understanding to a coarse distribution over a handful of action tokens. The paper puts this concern front and center:

"In contrast to traditional fine-tuning, GATS allows for the original component models to remain frozen, avoiding the risk of them losing important knowledge acquired during the pretraining phase." (Section 1)

Another option is to train from scratch on multimodal data — the monolithic approach exemplified by Gato (Reed et al., 2022), which trains a single transformer to process text, images, and actions as a flat sequence of tokens. This avoids the forgetting problem but introduces its own pathologies. Monolithic models force all modalities to be processed sequentially — the vision tokens must wait for the language tokens to finish, and vice versa — creating a serial bottleneck that is fundamentally incompatible with the asynchronous, parallel nature of multimodal perception. In a robotics setting, visual frames arrive at 5–30 Hz while language instructions arrive once per episode, yet a monolithic transformer forces them into the same processing pipeline at the same rate.

"One major drawback of monolithic architectures like Gato is their sequential token processing, forcing modalities to wait for each other. The monolithic models also allocate equal compute to each token, making leveraging truly large-scale models (e.g., for language) impractical." (Section 2.5)

This equal-compute problem is particularly acute. A 1.3B-parameter language model might be perfectly sized for understanding instructions, but the same 1.3B parameters applied uniformly to every video frame token would be computationally prohibitive at real-time video rates. Monolithic architectures force a one-size-fits-all compute budget, preventing you from pairing a massive frozen language model with a lightweight action policy that needs to run at control frequency.

A third approach is to use cross-attention adapters, most notably the Flamingo architecture (Alayrac et al., 2022). Flamingo interleaves pretrained language model layers with trainable cross-attention layers that inject visual features, conditioning text generation on images while keeping both the vision encoder and language model frozen. This works well for vision-to-language tasks — image captioning, visual question answering — but it is fundamentally asymmetric. The cross-attention connection is a one-way street: vision informs language, but language does not inform vision. In a robotics setting, where visual processing benefits from knowing which object is the target of the current instruction ("lift the apple" activates different visual features than "lift the banana"), this one-way conditioning is insufficient. The paper explicitly shows that when GATS steering is disabled — reducing the architecture to something resembling a Flamingo-style cross-attention model — performance on Language-Table collapses from 89.0% to 30.4% (Table 1, "No vision steering" ablation). This dramatic drop is the single most important empirical finding in the paper: unidirectional conditioning is not enough for interactive multimodal tasks.

Why This Problem Matters: The Multimodal Integration Gap

The gap this paper addresses sits at the intersection of several converging trends in AI, each of which amplifies its practical importance:

First, the foundation model ecosystem is fragmenting into specialists. We no longer have just "large language models" — we have language models, vision models, video models, audio models, and code models, each pretrained on different data distributions with different architectures, tokenizers, and output spaces. The paper's list of pretrained components (Section 3.2) — Chinchilla (language), Phenaki (video), Parti tokenizer (image), ViT (vision transformer) — reflects the reality that no single model excels at everything. The practical question is not "should we use pretrained models?" but "how do we make them work together?"

Second, robotics and embodied AI demand asynchronous multimodal processing. In the robotic agent described in Section 2.5, the language model processes a single instruction at episode start, while the video model processes frames at environment step frequency (5+ Hz), and the action model runs at the same rate to produce motor commands. These are fundamentally different timescales, and an effective architecture must respect them rather than forcing synchronization. GATS enables this by gathering only the most recent activations from each modality (using per-modality local context windows $N_m$ ) while delegating long-term processing to the component models themselves. This is not a minor implementation detail — it is the architectural property that makes real-time robotic control with large frozen models feasible.

Third, the cost of pretraining is enormous and growing. Training a 1.3B Chinchilla model is a multi-million-dollar endeavor; training a 7B ViT similarly so. The ability to reuse these investments across many downstream tasks — without additional fine-tuning that might degrade the original capabilities — has direct economic implications. The paper's image-based Language-Table agent achieves 89.0% success while training only 495M GATS and action parameters (Table 1), compared to the 3.1B total parameters in the system. The frozen pretrained models (accounting for ~2.6B parameters) contribute knowledge without contributing to training cost.

Where Existing Approaches Fall Short

The paper identifies specific, concrete limitations of prior work that GATS is designed to address:

Flamingo-style cross-attention is unidirectional and modality-specific. The Flamingo approach (Alayrac et al., 2022) pioneered the idea of injecting visual information into frozen language models via interleaved trainable layers. But this architecture is hard-coded for one specific modality pairing (vision → language) and one direction of influence. Adding a third modality (e.g., actions) requires engineering new cross-attention pathways. Adding bidirectional influence (language → vision) requires a fundamentally different design. The paper positions GATS as a generalization: a symmetrical connector where "no modality has to be treated in any special way" (Section 2.6). The same GATS layer that conditions vision on language can also condition language on vision, actions on both, or vision on actions — there is no architectural asymmetry baked into the design.

Adapter methods like LoRA (Hu et al., 2022) address parameter efficiency, not multimodal fusion. LoRA adds low-rank adaptation matrices to pretrained layers, enabling fine-tuning with far fewer trainable parameters. But LoRA modifies a single model's forward pass; it provides no mechanism for fusing activations across multiple independently trained models operating on different modalities. The paper explicitly distinguishes GATS from LoRA:

"In contrast to LoRA, GATS layers merge several different model layer outputs with an attention operation. Our focus is the ability to use multiple modalities rather than parameter-efficient adaptation." (Section 6)

This is a critical distinction. Parameter-efficient fine-tuning solves the problem of "how do I adapt one model without updating all its weights?" GATS solves the fundamentally different problem of "how do I make multiple frozen models talk to each other?"

Prior multimodal robotics approaches either fine-tune or use frozen feature extractors as fixed pipelines. RT-2 (Brohan et al., 2023) fine-tunes a pretrained vision-language model to directly output action tokens — a single-model approach that risks forgetting. CLIP-based approaches (Gadre et al., 2022; Khandelwal et al., 2022; Shridhar et al., 2021) use frozen CLIP vision encoders as fixed feature extractors, but the features are extracted once and treated as immutable observations — there is no mechanism for the action policy's needs to feed back and influence visual processing. The paper's steering mechanism (Section 2.2) provides exactly this feedback: by modifying the activations of the frozen vision model mid-forward-pass, GATS enables the action-relevant visual features to be dynamically reprogrammed based on context from other modalities, without updating the model's parameters.

No existing method provides a general-purpose blueprint for connecting arbitrary pretrained models. The paper's framing in Section 2.6 emphasizes that GATS is "agnostic to the specific details of the neural networks being combined." You provide a set of pretrained models, specify which are frozen and which are trainable, define per-modality context lengths, and GATS handles the rest. The same architecture that controls a robot (Section 4) also generates images from text and captions from images (Section 5) — only the component models and training objectives change.

How This Paper Positions Itself

The paper frames GATS not as a new model architecture per se, but as a general-purpose connectivity module — a "universal connector for pretrained models" (Section 2.6). This framing is important because it distinguishes GATS from research that proposes a specific multimodal model for a specific task. GATS is a design pattern, a way of thinking about multimodal architecture construction, rather than a point solution.

The intellectual lineage is explicitly traced to Alayrac et al. (2022)'s insight that "a given network's behavior can be reprogrammed by modifying its activations" (Section 1). GATS extends this insight from a single vision-to-language connection to a fully connected graph of modality interactions. The key architectural innovation is the gather-attend-scatter sequence:

Gather: each GATS layer collects the most recent $N_m$ activations from each modality $m$ , where $N_m$ is a per-modality budget. This creates a local sliding window into each model's processing that respects the different timescales at which modalities operate.
Attend: the gathered activations are projected into a shared dimensionality $d$ via per-modality learned projections $p_m$ , then processed together by a standard transformer self-attention block. This is where cross-modal information fusion happens — the projected vision token can attend to projected language tokens, regardless of their original dimensionalities.
Scatter: the attention outputs are projected back to each modality's native dimensionality via learned projections $r_m$ , then added back to the original activations through a gated residual connection $g_m(z_i) \cdot r_m(z_i)$ . The gating function $g_m$ (a scalar in $[0,1]$ ) learns how much to steer each activation — providing a learned mechanism to modulate the influence strength, which is absent in simpler concatenation or cross-attention approaches.

The paper demonstrates that this mechanism is not a minor architectural tweak but the enabling factor for frozen-model integration. The ablation where vision steering is disabled (Table 1) — reducing GATS to an asymmetric cross-attention from frozen vision features to the action model — causes a 58.6 percentage point drop in Language-Table success rate (89.0 → 30.4). This is not a marginal improvement; it is the difference between a functional agent and one that essentially fails.

The paper also positions itself within a broader vision of modular AI construction. The experiments span three very different domains — games, tabletop robotics, and multimodal generation — to demonstrate that GATS is not a robotics-specific hack or a vision-language-specific trick. Section 5.3 describes a particularly elegant demonstration of modularity: the GATS parameters from a pretrained vision-only model can be discarded and replaced with a new, larger GATS module trained from scratch for a bimodal vision-language task, and the new module "rapidly matches ViT's image generation performance" while simultaneously acquiring captioning ability (Figure 11). This suggests that GATS modules can serve as a kind of "multimodal interface" that can be swapped, upgraded, or extended without touching the underlying pretrained models — a vision of composable AI that stands in stark contrast to the prevailing monolithic paradigm.

3. Technical Approach

This is primarily a systems and architecture paper whose core idea is that a single, learnable, symmetrical module — the GATS (Gather-Attend-Scatter) layer — can serve as a universal bridge between multiple independently pretrained neural networks, enabling them to be combined into larger multimodal systems without modifying their original weights. The module operates by collecting recent activations from every attached model, projecting them into a shared representational space for cross-modal attention, and then injecting the fused information back into each model's forward pass through learned, gated residual connections, effectively "reprogramming" frozen models to attend to and incorporate information from other modalities.

3.1 Reader Orientation

The system being built is a modular multimodal architecture — think of a robot brain composed of off-the-shelf components (a frozen language model, a frozen vision model, and a trainable action policy) that communicate through a lightweight intermediary module rather than being merged or fine-tuned into a single monolithic network. The problem it solves is the multimodal integration problem: how to let independently trained models with different architectures, dimensionalities, and processing rates share information bidirectionally without catastrophic forgetting or serial bottlenecks. The "shape" of the solution is an interleaved bridge — GATS layers are inserted between the existing layers of all component models, gathering activations from each, attending over everything together, and scattering context-aware signals back to steer the forward pass of each model, all while keeping the pretrained weights frozen and respecting the natural asynchronous rates of different modalities.

3.2 Big-Picture Architecture (Diagram in Words)

The GATS-based multimodal system consists of four major components:

Component Models ( $M$ pretrained networks): These are the foundation models being connected — a language model (e.g., Chinchilla 1.3B), a vision model (e.g., Phenaki video model or ViT image model), and an action model (typically trained from scratch for the specific robotic task). Each processes tokens of its native modality independently. Some are frozen (language, vision); some may be trainable (action). Each has its own layer count $L_1, L_2, \ldots, L_M$ , hidden dimensionality, and processing rate.
GATS Module ( $K$ GATS layers): A stack of $K$ transformer layers (typically much smaller than the component models — e.g., 12–18 layers with width 512) that are interleaved between the layers of all component models according to a proportional spacing formula. Each GATS layer gathers recent activations from every component model, projects them into a shared dimensionality $d$ , applies self-attention, and scatters updated representations back.
Per-Modality Projections ( $p_m$ , $r_m$ , $g_m$ ): For each modality $m$ , three learned functions: a projection $p_m$ that maps from the modality's native dimensionality to the common GATS width $d$ , a reconstruction $r_m$ that maps back from $d$ to the native dimensionality, and a gate $g_m$ that outputs a scalar in $[0,1]$ controlling how strongly the GATS-modified activation replaces the original. In all experiments, these are simple linear transformations (with layer norm on $g_m$ ). There are $2M$ of these functions total.
Action Head (agent experiments only): A small MLP that takes the final-layer outputs of the action model's transformer and produces logits over discretized actions. Trained with standard cross-entropy loss against behavioral cloning targets.

Information flow: A prompt arrives in potentially multiple modalities at different rates. For each forward pass of a component model, at each GATS interleaving point, the GATS layer takes the most recent $N_m$ activations from that model (where $N_m$ is a per-modality context window), combines them with the most recent $N_{m'}$ activations from all other models, projects everything to dimension $d$ , applies self-attention, and routes the updated representations back to the originating model via the gated residual. The component model then continues its forward pass using the GATS-steered activations instead of its original ones. This process repeats at each of the $K$ interleaving points, enabling continuous cross-modal conditioning throughout the depth of every model.

3.3 Roadmap for the Deep Dive

First, the GATS Layer in isolation (Section 3.4.1): We will walk through the three core operations — gather, attend, scatter — at the level of a single GATS layer processing a single embedding. This establishes the mathematical machinery (projections, attention, gated residuals) and the key design decisions (per-modality context windows, shared projection space, learned gating). Understanding a single GATS layer is prerequisite to understanding how they are interleaved.
Second, the interleaving mechanism (Section 3.4.2): We will explain how $K$ GATS layers are distributed across the layers of $M$ component models with different depths using the proportional spacing formula. We will cover what "steering" means operationally — which activations get modified and where those modifications occur — and the design choice between symmetric and asymmetric steering.
Third, the hyperparameter space (Section 3.4.3): We will enumerate all configurable parameters of a GATS module and provide the specific values used in different experimental settings (vision pretraining, agent experiments, bimodal generation), including projected embedding sizes, context lengths, transformer dimensions, and training configurations.
Fourth, two worked examples as architectural blueprints (Sections 3.4.4 and 3.4.5): We will reconstruct the two architectures described in the paper — cross-attention via GATS (Section 2.4) and the GATS-based robotic agent (Section 2.5) — to show exactly how the abstract GATS components are instantiated for specific use cases. These concrete walkthroughs ground the general formalism in specific layer counts, context lengths, and modality configurations.
Fifth, the training methodology (Section 3.4.6): We will cover how GATS parameters are trained — what objectives are used in different settings, which models are frozen vs. trainable, and the specific training hyperparameters (learning rates, batch sizes, step counts) for each experiment. We will also cover classifier-free guidance as an inference-time technique that complements GATS training.
Finally, the modular substitution property (Section 3.4.7): We will explain the technique described in Section 5.3 — discarding a pretrained GATS module and training a new one from scratch to create a bimodal model from unimodal components — and why this property matters for composable AI.

3.4 Detailed, Sentence-Based Technical Breakdown

3.4.1 The GATS Layer: Gather, Attend, Scatter

A single GATS layer is architecturally similar to a standard transformer layer — it contains self-attention followed by a feedforward network, with residual connections and layer normalization — but its context comes from multiple heterogeneous sources rather than a single homogeneous sequence. The layer's defining characteristic is how it constructs its input (the gather step) and distributes its output (the scatter step). What happens in between — the attention and feedforward — is a conventional transformer block operating on projected representations.

Assumptions about the input. The paper assumes that at any point in time, the GATS layer receives a sequence of embeddings $(x_1, x_2, \ldots, x_t)$ . Each embedding $x_i$ originates from exactly one of $M$ modalities, and there exists a modality assignment function $m(x_i) \in \{1, 2, \ldots, M\}$ . Critically, embeddings from different modalities may have different dimensionalities — $x_i$ from a language model might be 2048-dimensional, while $x_j$ from a video model might be 4096-dimensional — but embeddings from the same modality always share the same size. This heterogeneity is the fundamental challenge that the gather-attend-scatter sequence addresses.

The Gather step — constructing the local context window $G$ . Unlike a standard transformer that attends over the most recent $N$ tokens regardless of their origin, GATS imposes a separate budget $N_m$ on each modality $m$ . The total context size $N$ is partitioned across modalities:

$N = N_1 + N_2 + \cdots + N_M$

where $N_m$ is the maximum number of embeddings retained from modality $m$ in the GATS attention window.

The gather step selects a subsequence $G \subseteq (x_1, \ldots, x_t)$ satisfying two conditions:

Per-modality capacity constraint: For each modality $m$ , the number of elements in $G$ belonging to modality $m$ does not exceed $N_m$ : $|\{x_i \in G : m(x_i) = m\}| \leq N_m$ .
Recency priority: The most recent embeddings from each modality are always included. Formally, if an embedding $x_i$ of modality $m$ is included in $G$ , then all more recent embeddings $x_j$ (with $j > i$ ) of the same modality $m$ must also be included. This is the "sliding window" property — as new embeddings arrive, older ones are evicted from $G$ when the budget $N_m$ is exceeded.

Only the selected subsequence $G$ participates in subsequent attention computation. Embeddings not in $G$ (i.e., $x_i \notin G$ ) are ignored by that GATS layer — they pass through unmodified.

What the gather step computes: a fixed-capacity, modality-balanced sliding window over the complete history of activations from all component models. Given the ordered sequence of all embeddings and the per-modality budgets, the gather operation outputs a subset $G$ that respects both the capacity limits and recency constraints. This subset becomes the input to the attention step.

Why this form: The per-modality partitioning is essential because modalities operate at fundamentally different rates. A video model might produce 100 frames of activations during a single episode, while the language model produces activations only once at episode start. A standard transformer with a fixed context length $N = 512$ would either waste capacity retaining stale language embeddings or lose recent video embeddings. The partitioned budget ensures each modality gets a guaranteed "voice" in the cross-modal attention regardless of its update frequency. The recency priority reflects the assumption that more recent information is more relevant for current processing — an inductive bias that is appropriate for sequential decision-making and streaming perception tasks.

The projection step — mapping to a common space. Since embeddings $x_i \in G$ have heterogeneous dimensionalities, they cannot be directly fed into a standard dot-product attention mechanism. GATS defines, for each modality $m$ , a learned projection function $p_m$ that maps from the modality's native dimensionality to a common projected size $d$ :

$x_i \leftarrow p_{m(x_i)}(x_i)$

where $p_m$ is a simple learned linear transformation (a matrix multiplication plus bias). All projected embeddings now have dimension $d$ , which is a hyperparameter typically much smaller than the native dimensionalities of the pretrained models (e.g., $d = 512$ when connecting models with hidden sizes of 2048–4096). This dimensionality bottleneck serves two purposes: it forces the model to learn compact cross-modal representations, and it limits the computational cost of the subsequent self-attention operation.

What the projection computes: a dimensionality reduction (or occasionally expansion) of each gathered embedding from its native size to the common GATS width $d$ . Given an embedding $x_i$ from modality $m$ , the projection outputs a $d$ -dimensional vector that represents that modality's information in a format consumable by the shared attention mechanism.

Why this form: A linear projection is parameter-efficient (requiring only $(\text{native\_dim} \times d + d)$ parameters per modality) and preserves the linear structure of the original representations, making it compatible with the residual connections used throughout transformer architectures. Alternative approaches — such as padding all embeddings to the maximum dimensionality or using zero-padded concatenation — would either waste compute or lose information. The linear projection compresses while preserving the ability of the downstream attention to learn which dimensions of the source modality are relevant for cross-modal interaction.

The Attend step — cross-modal self-attention. Once $G$ has been projected to a uniform dimension $d$ , the attend step applies a standard transformer layer. For each projected embedding $x_i \in G$ :

$x_i \leftarrow \text{FFW}(\text{Attention}(x_i, G))$

where $\text{Attention}(x_i, G)$ is standard multi-head self-attention over the full set $G$ with query $x_i$ , and $\text{FFW}$ is a position-wise feedforward network. The paper notes that standard transformer components — layer norms, residual connections, and positional encodings — are applied as usual and are "folded inside $\text{Attention}$ and $\text{FFW}$ functions" in the notation. In practice, the full computation is:

$x_i \leftarrow \text{LayerNorm}\left(x_i + \text{FFW}\left(\text{LayerNorm}\left(x_i + \text{MultiHeadAttention}(x_i, G, G)\right)\right)\right)$

What the attend step computes: a context-aware update to each embedding in $G$ that incorporates information from all other embeddings in $G$ , regardless of their originating modality. The attention mechanism computes pairwise similarity scores between every pair of embeddings in $G$ , uses these to create weighted combinations, and passes the results through the feedforward network. The output for each embedding $x_i$ is a $d$ -dimensional vector that blends its original content with relevant information from other modalities.

Why this form: Self-attention over the full gathered set $G$ is what enables cross-modal fusion. Unlike approaches that use separate encoder-decoder cross-attention pathways for each modality pair (which would scale as $O(M^2)$ ), a single shared self-attention operation over all projected embeddings scales as $O(|G|^2)$ independently of $M$ . The per-modality segmentation is handled entirely by the gathering budget $N_m$ ; the attention mechanism itself treats all embeddings uniformly once projected. This symmetry — "no modality has to be treated in any special way" (Section 2.6) — is the architectural property that makes GATS a universal connector.

The Scatter step — projecting back with gated residuals. After attention and feedforward processing, each embedding $z_i$ (the output of the FFW block) must be returned to its originating modality's native dimensionality and reintegrated into the component model's forward pass. Two operations handle this:

Reprojection: A learned per-modality linear transformation $r_m$ maps from the GATS dimension $d$ back to the modality's native size: $r_m(z_i)$ .
Gated residual: A learned gating function $g_m$ outputs a scalar in $[0,1]$ that controls how much of the GATS-modified representation to blend with the original:

$x_i \leftarrow x_i + g_{m(x_i)}(z_i) \cdot r_{m(x_i)}(z_i)$

where $g_m$ is a linear transformation followed by a layer norm (which constrains the output to a reasonable range; the paper doesn't specify the exact activation that enforces $[0,1]$ , but sigmoid is the standard choice for gating mechanisms in this context). The original $x_i$ is the unprojected native-dimensional embedding from before the gather step — the residual connection operates in the native space, not the projected space.

What the scatter step computes: a modified version of each original embedding that incorporates cross-modal context. The computation takes the GATS-processed projected embedding $z_i$ , re-expands it to native dimensionality via $r_m$ , scales it by the learned gate value $g_m(z_i)$ , and adds it to the original unprojected embedding. The gate modulates how much cross-modal influence is injected at each embedding position.

Why this form: The gated residual is the mechanism that implements selective steering. If $g_m(z_i) \approx 0$ for a particular embedding, the GATS output is effectively ignored and the component model continues processing as if GATS weren't present — this is the behavior for modalities that are "not steered." If $g_m(z_i) \approx 1$ , the full GATS-processed signal replaces the original. Learned intermediate values allow the model to decide, on a per-embedding basis, how much cross-modal information is useful. This is a significant improvement over unconditional concatenation or fixed-weight averaging, and it is what enables GATS to be applied to frozen models without disrupting their internal representations when cross-modal information is irrelevant.

For embeddings not in the gathered set $G$ (i.e., $x_i \notin G$ ), no computation occurs — they pass through unaltered. This is what enables the per-modality context windowing to work: only the most recent $N_m$ embeddings per modality incur the cost of GATS processing; older embeddings remain in the component model's own context for long-range dependency handling.

3.4.2 Interleaving GATS with Component Models

The GATS module consists of $K$ GATS layers. These layers are not applied as a single block at the beginning or end of the component models; rather, they are interleaved — distributed throughout the depth of every component model — so that cross-modal conditioning occurs at multiple levels of abstraction. The paper refers to this as GATS "lying between" layers of the component transformers.

The proportional spacing formula. Each component model $m$ has $L_m$ transformer layers (numbered $0$ through $L_m - 1$ ). The $k$ -th GATS layer (for $k = 1, \ldots, K$ ) is interleaved such that it receives as input the output from layer $l_{k,m}$ of component model $m$ , where:

$l_{k,m} = \min\left(\max\left(1, \left\lfloor k \cdot \frac{L_m}{K} \right\rfloor\right), L_m - 1\right)$

where $\lfloor \cdot \rfloor$ denotes the floor function (rounding down to the nearest integer).

What this formula computes: the index of the component model layer whose output feeds into the $k$ -th GATS layer. The expression $k \cdot L_m / K$ computes the proportional position of the $k$ -th GATS layer within the depth of model $m$ . For example, if a component model has $L_m = 24$ layers and GATS has $K = 12$ layers, then GATS layer $k = 6$ (the midpoint) would be positioned after component model layer $l_{6,m} = \lfloor 6 \cdot 24 / 12 \rfloor = 12$ .

Why this formula: The proportional interleaving ensures that GATS interactions are distributed across the full depth range of each component model, rather than being concentrated at one end or the other. Early GATS layers condition low-level features on cross-modal context (e.g., early visual features are steered toward task-relevant regions), while later GATS layers condition high-level representations (e.g., semantic visual representations are enriched with language understanding). The min and max clamp to $[1, L_m - 1]$ ensure that (a) the first GATS layer always has at least one transformer layer's worth of processing to condition on, and (b) the last GATS layer still leaves at least one transformer layer after it for final processing before the output. This prevents GATS from intercepting raw input embeddings (which might not have useful structure) or directly modifying final output logits (which should be the component model's responsibility).

Concrete example. Consider the setup from Figure 5: a language model with $L_{\text{lang}} = 6$ layers, a video model with $L_{\text{video}} = 4$ layers, an action model with $L_{\text{action}} = 2$ layers, and $K = 1$ GATS layer. The formula gives:

For the language model: $l_{1,\text{lang}} = \lfloor 1 \cdot 6 / 1 \rfloor = 6$ , clamped to $\min(\max(1, 6), 5) = 5$ , so GATS sits between layers 5 and 6.
For the video model: $l_{1,\text{video}} = \lfloor 1 \cdot 4 / 1 \rfloor = 4$ , clamped to $3$ , so GATS sits between layers 3 and 4.
For the action model: $l_{1,\text{action}} = \lfloor 1 \cdot 2 / 1 \rfloor = 2$ , clamped to $1$ , so GATS sits between layers 1 and 2.

Thus the single GATS layer receives language activations from layer 5, video activations from layer 3, and action activations from layer 1, allowing cross-modal interaction at these respective depths.

The steering mechanism. An embedding that has been modified by GATS (via the scatter step) replaces the original activation that would have fed into the next layer of the component model. If the component model $m$ is designated as steered (i.e., $m \in S$ where $S \subseteq \{1, \ldots, M\}$ ), the next layer of that component model receives the GATS-modified activation. If $m \notin S$ (not steered), the original unmodified activation is used instead, but the embeddings from $m$ are still gathered and attended over — their information can influence other modalities even if they themselves are not modified.

Why steering is optional: The paper allows manual specification of the steering set $S$ rather than learning it from data. This design choice reflects a practical consideration: in some deployments, you may want a large frozen language model to provide information to other modalities (its activations are gathered and attended to) but not to receive conditioning from them (its activations are not modified). This could be important if modifying the language model's internal representations introduced instability or if the language model is serving as a fixed semantic backbone rather than an adaptive component. The paper acknowledges that steering could "potentially be learned from data" (Section 2.2), which would turn $S$ into a trainable or gated decision rather than a hyperparameter.

Computational implications of interleaving. Because GATS layers are inserted between existing layers, the forward pass of each component model is no longer a simple sequential application of its own layers. When model $m$ reaches layer $l_{k,m}$ , instead of immediately feeding to layer $l_{k,m} + 1$ , the activations are routed to the $k$ -th GATS layer, which gathers from all models, processes, and scatters back before model $m$ continues. This means the forward passes of different component models must be synchronized at GATS interleaving points. However, between interleaving points, models can process independently — and if models have different numbers of layers, they will naturally process at different rates, reaching GATS rendezvous points at different wall-clock times.

3.4.3 Hyperparameters and Configurations

The paper defines the hyperparameter space of a GATS module and provides specific values used across different experimental settings. The full hyperparameter set for a GATS module is:

Number of component models $M$ : The number of distinct pretrained models being connected (e.g., $M = 2$ for vision-language, $M = 3$ for language-vision-action).
Local context lengths $N_1, N_2, \ldots, N_M$ : Per-modality budgets for the gather step, controlling how many recent embeddings from each modality participate in cross-modal attention.
Projected embedding size $d$ : The dimensionality of the shared representation space into which all modalities are projected before attention.
Steering subset $S \subseteq \{1, 2, \ldots, M\}$ : Which component models receive GATS-modified activations (i.e., are steered).
Transformer hyperparameters for each GATS layer: Number of attention heads, layer width (which is $d$ ), MLP hidden size (typically $6 \times d$ ), and any standard transformer architectural choices (activation function, normalization placement).
Number of GATS layers $K$ : The depth of the GATS module, controlling how many interleaving points exist.

The projection and gating functions. For every modality $m$ , three learned functions are defined:

$p_m$ : a linear transformation projecting from the modality's native embedding size to $d$ .
$r_m$ : a linear transformation projecting from $d$ back to the native size.
$g_m$ : a linear transformation outputting a scalar, followed by a layer normalization (which the paper describes as producing a value in $[0,1]$ , implying a sigmoid activation after the layer norm).

All three depend only on $d$ and the modality's native embedding size, requiring no additional hyperparameters. This keeps the interface minimal: to add a new modality, you only need to specify its native dimensionality, and the projection/gating functions are automatically instantiated with the correct input/output sizes.

Hyperparameter values across experimental settings. The supplementary materials provide detailed tables, which we summarize here:

ViT Vision Pretraining (Table 3, "Vision pretraining" column):

GATS transformer blocks: 12
Attention heads: 8
Layer width ( $d$ ): 512
MLP hidden size: 3072
Total GATS parameters: 124M

Bimodal Model Finetuning (Table 3, "Bimodal finetuning" column):

GATS transformer blocks: 12
Attention heads: 16
Layer width ( $d$ ): 2048
MLP hidden size: 12288
Total GATS parameters: 988M

The significant increase in GATS capacity for the bimodal model (from 124M to 988M parameters) reflects the added complexity of joint image-and-text generation compared to image generation alone. The projected embedding size quadruples (512 → 2048), enabling the GATS module to represent richer cross-modal interactions.

Image-Based Agent GATS (Table 5, "Image" column):

GATS transformer blocks: 18
Attention heads: 8
Layer width ( $d$ ): 512
MLP hidden size: 3072

Video-Based Agent GATS (Table 5, "Video" column):

GATS transformer blocks: 12
Attention heads: 8
Layer width ( $d$ ): 512
MLP hidden size: 3072

Action Module Hyperparameters (Table 4):

Image-based: 24 transformer blocks, 32 attention heads, width 1024, MLP hidden 6144, no time-space factorization
Video-based: 24 transformer blocks, 4 attention heads, width 512, MLP hidden 3072, with time-space factorization (odd layers attend over space, even layers attend over time)

The time-space factorization in the video action module is a memory-saving technique: rather than applying full self-attention over all tokens from all frames (which would scale quadratically), the model alternates between spatial attention (within each frame) and temporal attention (across frames for each spatial position). This factorization uses attention masks such that even-indexed layers only attend over tokens within the same frame, and odd-indexed layers only attend over tokens at the same spatial location across different time steps.

The choice to share hyperparameters across GATS layers. The paper states that "in our experiments all layers share the same hyperparameters but in theory all of them can be different" (Section 2.3). Sharing reduces the hyperparameter search space and likely works because cross-modal integration needs are relatively consistent across depths. Allowing layer-specific hyperparameters could enable early GATS layers to focus on low-level fusion (using larger $d$ ) while later layers focus on high-level fusion (using smaller $d$ ), but this complexity is not explored.

3.4.4 Worked Example: Cross-Attention via GATS (Section 2.4)

The paper presents a deliberately simplified configuration to demonstrate that GATS can reproduce the behavior of a Flamingo-style vision-to-text cross-attention model, establishing GATS as a strict generalization of that approach.

Setup. Two component models: a vision model (modality 1) and a language model (modality 2), so $M = 2$ . The vision model processes a single image to produce $V$ visual feature embeddings. The language model processes text tokens autoregressively. Only the language model is steered: $S = \{2\}$ . The GATS context lengths are set to $N_1 = V$ (always retain all vision features) and $N_2 = 1$ (only retain the most recent language token).

How this works step-by-step. When the most recent language token is processed by the language model's first layer, its activation is routed to the first GATS layer (interleaved between some early layers of the language model). The gather step selects all $V$ vision embeddings (since $N_1 = V$ ) plus the one most recent language token (since $N_2 = 1$ ). The attend step computes self-attention over these $V + 1$ embeddings. The scatter step updates the language token's activation (since language is steered) and leaves vision activations unmodified (since vision is not steered). The updated language token then continues through the next language model layer.

At the next GATS interleaving point, the same process repeats: the full set of vision features (still $V$ embeddings) and the newest language token pass through attention, and the language token is updated. This is functionally equivalent to cross-attention from text to vision features — the language token can query against all visual features at every GATS layer, exactly as in the Flamingo architecture.

The key design choices that enable this. The projected embedding size $d$ is set to match the language model's native embedding size, which means $p_2$ and $r_2$ (the language projections) can both be identity functions — the language token is never projected away from its native space. Only $p_1$ (vision projection) and $r_2$ (language reprojection) are non-trivial. The gating function $g_2$ controls how strongly the vision-conditioned representation influences the language token.

Additionally, "extra text embeddings representing the image position within the text can be added" to further resemble Flamingo's architecture, which uses special <image> tokens to mark where visual information should be attended to. The paper mentions this but does not elaborate on the exact mechanism.

Why this example matters. It demonstrates that GATS is not imposing a new, restrictive architectural paradigm but rather subsumes existing successful architectures as special cases of a more general framework. A practitioner who is comfortable with Flamingo-style vision-language models can adopt GATS without losing the ability to reproduce that architecture, while gaining the ability to extend it to bidirectional conditioning, additional modalities, or asynchronous processing streams — all of which are impossible in the original Flamingo design.

3.4.5 Worked Example: GATS-Based Robotic Agent (Section 2.5)

This is the primary architecture used in the agent experiments (Section 4) and represents the full generality of GATS. Three modalities are connected: language (instructions), video (camera frames), and action (proprioception and motor commands).

Modality processing rates. A critical design feature is that the three modalities operate at different rates:

Language: Processed once at episode start when the instruction is given. The language model's activations are computed once and cached. The paper notes: "These activations can be cached for frequent instructions, enabling the use of a larger language model without sacrificing inference speed" (Section 2.5). This caching is a practical deployment optimization — the language model's entire forward pass (which might involve 1.3B parameters) runs once and its outputs are stored for reuse across potentially hundreds of environment steps.
Video: Processed at every environment time step (one frame per step). The video model maintains its own long context of previous frames, so each new frame is processed alongside tokens from prior time steps.
Action: Processed at every environment time step, producing discretized action tokens. The action model similarly maintains context across time steps, conditioning current action predictions on the history of proprioception observations and past actions.

GATS context design. For each environment step, the GATS module gathers:

The cached language activations (which persist across all time steps, always available in the gather window since $N_{\text{lang}}$ is set to cover the full instruction encoding).
The most recent video activations (from the current frame, based on $N_{\text{video}}$ ).
The most recent action activations (from the current step, based on $N_{\text{action}}$ ).

The paper emphasizes that GATS "only attends to recent activations, delegating long-term processing to the individual models, resulting in negligible computational overhead" (Figure 5 caption). This delegation is the key insight: the video model's own transformer layers handle long-range temporal dependencies (how objects have moved over many frames), while GATS handles immediate cross-modal coordination (which object from the current frame matches the language instruction).

Steering configuration. Both the video model and the action model are steered ( $S$ includes both vision and action). The language model is not steered — its frozen processing remains unmodified. This asymmetric steering reflects an inductive bias: language understanding should be stable and independent of the specific visual scene, while visual processing and action selection should be context-dependent and grounded in the instruction. The language model provides a fixed semantic "backbone" that conditions the other modalities without itself being influenced by them.

Why the language model is processed once. This design choice is motivated by inference efficiency. The language model (e.g., Chinchilla 1.3B) is by far the largest component. Running it at every environment step would make real-time control impossible. By processing the instruction once, caching the activations, and using them as a persistent conditioning signal (always present in the GATS gather window), the system decouples language processing rate from control rate. The paper explicitly connects this to a critique of monolithic architectures, which force all modalities into the same processing rhythm: "GATS-based architectures overcome this problem, as tokens from each modality can be processed by the corresponding unimodal models simultaneously" (Section 2.5).

Extending to multiple camera views (YCB adaptation). The YCB environment provides two camera views (front and back of the basket). The GATS-based agent handles this by feeding both camera views to the same vision model and exposing their activations to GATS as separate modalities. In the GATS framework, a "modality" is defined by its embedding size and processing characteristics — two camera views processed by the same vision encoder are treated as two separate modalities even though they share the same model architecture. This means the gather step allocates separate context windows $N_{\text{cam1}}$ and $N_{\text{cam2}}$ for each view, and the attention step allows embeddings from both views to interact with language and action embeddings.

The paper contrasts this with a cross-attention approach, where adding a second camera view would require adding new cross-attention pathways that operate independently. In GATS, both camera views participate in the same self-attention operation, allowing them to jointly inform (and be informed by) language and action processing — "GATS empowers each modality (including both camera views which are seen as separate modalities) with its own 'voice' thanks to the symmetrical nature of the steering mechanism, allowing all modalities to dynamically share and refine information for a richer scene understanding" (Section 4.2).

Constructing the action input. The paper uses a fixed number of trainable input embeddings for each time step rather than feeding proprioception values directly. For image-based agents, 8 trainable embeddings are used per time step; for video-based agents, 16 are used. Past actions are not fed as input. This means the action model receives only these learned embeddings (which are steered by GATS based on language and vision context) and must learn to represent the relevant state implicitly rather than receiving explicit proprioceptive readings. This design choice simplifies the interface between the action model and the rest of the system — the action model only needs to handle embeddings of a fixed size, while GATS handles the variable-sized inputs from other modalities.

3.4.6 Training Methodology

GATS parameters are the only components trained in most experiments (along with the action model in agent settings). The pretrained foundation models — Chinchilla, Phenaki, ViT — remain frozen throughout. This section details the training objectives, configurations, and a complementary inference-time technique (classifier-free guidance).

Agent training objectives. For the Atari Pong, Language-Table, and YCB experiments, the agent is trained using behavioral cloning — standard supervised learning on pre-collected demonstration data. The action head outputs logits over discretized actions, and training minimizes cross-entropy loss against the demonstrated action:

$\mathcal{L}_{\text{BC}} = -\sum_{t} \log \pi_\theta(a_t | o_t, I)$

where $\pi_\theta$ is the GATS-based policy parameterized by trainable GATS and action model parameters $\theta$ , $a_t$ is the demonstrated action at time $t$ , $o_t$ is the observation (video frame + proprioception), and $I$ is the language instruction. The language and vision models are not updated — gradients flow through the GATS layers and into the action model, but stop at the frozen model boundaries.

Training hyperparameters for agent experiments. From the supplementary materials:

Learning rate: $1 \times 10^{-4}$ with a linear warmup over 750 steps
Image-based agents: trained for 100,000 steps with batch size 1024
Video-based agents (Language-Table): trained for 250,000 steps with batch size 512
Video-based agents (YCB): trained with batch size 256 (reduced due to memory demands from two camera streams)
Optimizer: Adam (Kingma and Ba, 2015) — specific $\beta$ values are not reported

The longer training for video-based agents (250k vs. 100k steps) reflects the increased complexity of processing temporal video features compared to static image features.

Classifier-free guidance for agent actions. At inference time, the agent uses classifier-free guidance (Ho and Salimans, 2022), adapted from diffusion models to the behavioral cloning setting. The policy is computed as the composition of a conditional and unconditional prediction:

$\pi(x, c) = \text{Softmax}\left(l(x, c) + \lambda(l(x, c) - l(x))\right)$

where $l(x, c)$ is the logit vector (pre-softmax scores) of the predicted action given observation $x$ and language conditioning $c$ , $l(x)$ is the logit vector without language conditioning (i.e., with the text input masked), and $\lambda \geq 0$ is the guidance strength. In all experiments, $\lambda = 0.5$ .

What this computes: The guidance term $l(x, c) - l(x)$ represents the component of the prediction that is specifically attributable to the language instruction — it subtracts out the "default" behavior the model would produce without instructions. When $\lambda > 0$ , this language-specific component is amplified, pushing the action distribution toward tokens that the language instruction makes more likely compared to the instruction-free baseline. The Softmax then converts the adjusted logits to a probability distribution.

Why this form: The subtraction $l(x, c) - l(x)$ can be understood as extracting the conditional mutual information between the instruction and the action — it captures what the instruction adds beyond what the observation alone implies. By amplifying this component, the guidance sharpens the model's focus on instruction-following behavior. The paper reports that "enabling the classifier-free guidance consistently improves the performance of our agents" (Table 1, with improvements of up to 5.8 percentage points on Language-Table). During training, the text input is randomly masked with probability 0.02, which simultaneously trains the unconditional policy $l(x)$ via the same behavioral cloning objective applied when the mask is active.

Bimodal vision-language training (Section 5.1). The 9.3B bimodal model is trained on image-text data with two alternating passes per batch:

Language-then-vision pass: Language tokens are processed first, followed by vision tokens. The training objective is MaskGIT loss (Chang et al., 2022) — a masked token prediction objective where some fraction of image tokens are randomly masked, and the model must predict them given the unmasked image tokens and the text conditioning. This trains text-to-image generation.
Vision-then-language pass: Vision tokens are processed first, followed by language tokens. The training objective is standard autoregressive next-token prediction cross-entropy loss on the language tokens, conditioned on the preceding vision tokens. This trains image captioning.

Why two passes with the same GATS layers: The symmetrical nature of GATS — where the same self-attention mechanism operates regardless of which modality "came first" — means the identical GATS parameters can be used for both conditional image generation and conditional text generation. The model learns a shared cross-modal representation that works bidirectionally. This is impossible with asymmetric architectures like Flamingo, which hard-code the direction of conditioning.

ViT pretraining (Section 5.2). The 2.7B and 8.5B parameter ViT models are pretrained with only the MaskGIT objective on image-text data. The language model (Chinchilla 1.3B) is frozen, while the vision transformer and GATS module are trained from scratch. Training configuration: batch size 2048, learning rate $1 \times 10^{-5}$ , 1 million steps. The GATS module used during ViT pretraining is the smaller 124M-parameter variant (Table 3, "Vision pretraining").

Why pretrain a ViT with GATS rather than as a standalone model: The ViT is being trained for eventual use in GATS-based multimodal systems. By training it with GATS from the start — where GATS conditions the vision transformer on frozen language model features — the resulting ViT learns representations that are inherently language-aligned. This means when the ViT is later frozen and used in an agent, its internal features already encode concepts that are semantically meaningful and language-groundable, making the subsequent GATS steering more effective.

3.4.7 Modular Substitution: Discarding and Replacing GATS Parameters

Section 5.3 describes a property of GATS-based models that the paper positions as a significant practical benefit: the ability to discard the GATS parameters from a pretrained unimodal model and substitute a new, larger GATS module trained from scratch for a new multimodal task.

The substitution procedure. The ViT model pretrained as described in Section 5.2 consists of three components: a frozen Chinchilla 1.3B language model, a trainable ViT vision model (7.0B parameters in the 8.5B total variant), and a 124M-parameter GATS module that conditions the ViT on language. To create the 9.3B bimodal model (Section 5.1), the 124M GATS parameters are discarded entirely. A new GATS module with 988M parameters is trained from scratch, while both the language model and the ViT vision model remain frozen with their pretrained weights.

Why this works. The ViT's vision parameters (7.0B) were trained to produce useful visual representations when any GATS module provides language conditioning. The specific 124M GATS that was co-trained with the ViT learned one particular mapping from language features to visual features. But the vision transformer itself learned general-purpose visually-semantic representations that are not tightly coupled to that specific mapping. When a new, larger GATS module is substituted, it learns a different mapping — one that simultaneously supports MaskGIT image generation and autoregressive caption generation — while reusing the same frozen vision representations.

Evidence of effectiveness. Figure 11 shows that the bimodal model's MaskGIT loss drops below 5.45 after only 50,000 training steps, while the original ViT (trained from scratch with no pretrained vision component) required 750,000 steps to reach the same loss. This represents a 15× speedup in reaching equivalent image generation performance. The paper describes this as the model "rapidly matching ViT's image generation performance."

What this enables. The substitution property means that GATS modules can be treated as interchangeable interfaces between pretrained components. A single frozen vision model could be paired with different GATS modules for different downstream tasks — one GATS for robotic control, another for captioning, a third for visual question answering — without ever retraining or fine-tuning the vision model itself. This is a step toward the "composable AI" vision where foundation models become reusable infrastructure and task-specific adaptation is confined to lightweight connector modules.

The architectural requirement for substitution. For this to work, the GATS module and the component models must have clean, standardized interfaces. The interface is defined by: (1) the layer at which GATS interleaves with each component model, (2) the native dimensionality of each modality's embeddings, and (3) the projected embedding size $d$ . As long as a new GATS module respects these interface specifications, it can be swapped in without any modification to the frozen models. This is analogous to how USB interfaces enable peripheral swaps — the connector standard is what matters, not the internal hardware.

4. Key Insights and Innovations

Innovation 1: Symmetrical Steering as a Generalization of Unidirectional Cross-Attention

The paper's most conceptually distinctive move is reframing multimodal integration from a directed wiring problem into a symmetrical communication problem. Prior to GATS, the dominant paradigm for connecting frozen pretrained models — exemplified by Flamingo (Alayrac et al., 2022) — was to insert trainable cross-attention layers that inject information from one modality into another in a fixed, hard-coded direction. Flamingo connects vision → language; the visual encoder sends features to the language model, but the language model never sends information back to vision. This unidirectionality is baked into the architecture: you add vision-to-text cross-attention layers at specific positions in the language model, and that's the only pathway. If you later want text → vision conditioning, or vision → action conditioning, or action → vision conditioning, each new pathway requires designing, implementing, and training a separate cross-attention mechanism.

GATS replaces this with a fully connected graph where every modality can influence every other modality through a single shared mechanism. The architectural move that enables this is the gather-attend-scatter sequence operating on all modalities symmetrically — the same self-attention operation over projected embeddings from every source, followed by per-modality gated injections back into each model's forward pass. There is no architectural distinction between a "source" modality and a "target" modality. The gating function $g_m$ learns per-modality, per-position influence strengths, so the model can effectively learn to be asymmetric (vision strongly influences action, language weakly influences vision) without that asymmetry being hard-coded into the connectivity pattern.

This matters beyond implementation convenience. It means the same GATS architecture that does vision-to-text conditioning (Section 2.4) also does text-to-vision conditioning, language-conditioned action selection, and vision-informed language understanding — all with identical mechanisms and code. When the paper adapts the YCB agent to handle two camera views, the change is described as "trivial" (Section 4.2): you simply feed both camera streams to the same vision model and expose their activations as separate GATS modalities. There is no new cross-attention pathway to design, no new parameters to allocate, no architectural asymmetry to resolve. The symmetrical design makes adding or removing modalities an operational decision rather than an architectural one.

The paper demonstrates that Flamingo-style cross-attention is a special case of GATS with specific hyperparameter settings ( $M = 2$ , $S = \{2\}$ , $N_1 = V$ , $N_2 = 1$ ) rather than a fundamentally different architecture class (Section 2.4). This is intellectually significant because it unifies what previously appeared to be distinct architectural families under a single framework, paralleling how the original Transformer paper unified sequence-to-sequence models that previously required separate encoder-decoder architectures for different modality pairings.

Innovation 2: The Steering Mechanism as Activation Reprogramming Rather Than Feature Extraction

A second conceptual innovation is the paper's framing and empirical demonstration that frozen models can be meaningfully reprogrammed by modifying their intermediate activations, without changing their weights. This is a fundamentally different approach to leveraging pretrained models than the dominant paradigm of "feature extraction + downstream head."

In the standard feature extraction approach (used by CLIP-based robotics systems such as Gadre et al., 2022; Khandelwal et al., 2022; Shridhar et al., 2021), you run the frozen vision model, take its final-layer (or penultimate-layer) embeddings, and feed those frozen features into a trainable policy. The vision model is a fixed function: given an image, it produces a representation, and that representation is immutable regardless of what the policy needs. There is no feedback — the policy's requirements cannot influence visual processing.

The Language-Table ablation in Table 1 provides the paper's most compelling evidence for why this matters. When vision steering is disabled — reducing GATS to something approximating the feature extraction paradigm, where GATS still gathers and attends over vision embeddings but does not modify them — the success rate drops catastrophically from 89.0% to 30.4%. This 58.6 percentage point gap is not a marginal improvement; it is the difference between a functional agent and one that essentially fails at the task. The fact that this drop occurs even though GATS still has access to the vision features (they are gathered and attended over) and the action model is identical in both conditions isolates the steering mechanism as the critical factor: being able to modify the vision model's internal representations mid-forward-pass is what makes frozen-model integration work.

What's novel here is not the idea that activations can be modified — adapter layers and prefix tuning do this — but the demonstration that modifying activations of a frozen, independently pretrained model can radically alter its effective behavior for a downstream task without any parameter updates to the model itself. The paper characterizes this as "leveraging a property of neural networks shown by Alayrac et al. (2022) that a given network's behavior can be reprogrammed by modifying its activations" (Section 1), and extends it from a single vision-to-language connection to a general multimodal framework.

The Phenaki finetuning experiment (Table 1, "Video finetuned" rows) reinforces this insight from a different angle. When the Phenaki video model is finetuned on Language-Table data (without action labels, using a future-frame prediction objective) and then frozen, the non-steered agent improves dramatically (24.2 → 74.8), nearly matching the steered agent using the original pretrained Phenaki (76.8). But critically, the steered agent using the un-finetuned Phenaki still slightly outperforms the non-steered agent using the finetuned Phenaki. This means GATS steering can achieve comparable or better domain adaptation than explicit finetuning, without the risk of catastrophic forgetting that finetuning entails. The steering mechanism is acting as a learned runtime adapter that reprograms the frozen model's forward pass to produce task-relevant features, while the model's weights — and all the knowledge they encode — remain untouched.

Innovation 3: Per-Modality Local Context Windows as a Solution to Asynchronous Multimodal Processing

A subtler but practically crucial innovation is the paper's treatment of time as a first-class design constraint in multimodal architectures. Prior multimodal systems — particularly monolithic transformers like Gato (Reed et al., 2022) — impose a single processing rate on all modalities. If the transformer processes tokens at 5 Hz, then vision, language, and action tokens all arrive at exactly that rate, interleaved in a flat sequence. This is architecturally simple but computationally wasteful and conceptually wrong: language instructions arrive once per episode, while video frames arrive continuously, and forcing them into the same temporal granularity means either wasting compute on redundant language processing or losing temporal resolution on vision.

GATS's gather step — with separate context budgets $N_1, N_2, \ldots, N_M$ — decouples the processing rates of different modalities. The paper formalizes this as a sliding window that retains the most recent $N_m$ embeddings from each modality $m$ , where $N_m$ is chosen based on the modality's update frequency and the relevant temporal horizon for cross-modal interaction. This is not merely an implementation trick; it is an architectural acknowledgment that modalities operate on different timescales, and that cross-modal integration should respect rather than erase those differences.

The significance becomes clear in the robotic agent design (Section 2.5), where the language model is run once at episode start and its activations are cached, while the video and action models run at every environment step. The cached language embeddings remain in the GATS gather window indefinitely (since $N_{\text{lang}}$ can be set to cover them), providing persistent conditioning across potentially hundreds of time steps without recomputation. This is what makes it possible to use a 1.3B-parameter language model in a real-time control loop: the largest model runs once, and the per-step overhead is limited to the much smaller video model, action model, and lightweight GATS attention over recent windows.

The paper explicitly contrasts this with monolithic architectures, noting that "GATS-based architectures overcome this problem, as tokens from each modality can be processed by the corresponding unimodal models simultaneously" (Section 2.5). The word "simultaneously" is key — video and action tokens are processed in parallel by their respective models, with GATS providing cross-modal conditioning at synchronization points, rather than being forced into a single sequential pipeline. This is a fundamental architectural property, not a performance optimization, and it reflects a design philosophy that treats modalities as cooperating independent processes rather than serialized token streams.

Innovation 4: GATS as a Discardable Interface — Modular Substitution for Composable AI

Section 5.3 describes an architectural property that, while not a core algorithmic contribution, represents a significant conceptual move toward composable AI systems: the ability to discard a GATS module trained for one task and substitute a new GATS module trained for a different task, while reusing the same frozen foundation models. The paper demonstrates this concretely: the 124M-parameter GATS module trained alongside the 7.0B ViT for image generation is thrown away, replaced with a 988M-parameter GATS module trained from scratch for joint image-and-text generation, while the frozen Chinchilla language model and ViT vision model remain unchanged. The new module recovers image generation performance 15× faster than training from scratch (50k steps vs. 750k steps to reach equivalent MaskGIT loss, Figure 11) while simultaneously acquiring captioning ability.

The innovation here is not the technical mechanism — it's just retraining a new module — but the architectural philosophy it enables. In conventional fine-tuning, adapting a pretrained model to a new task modifies the model weights, entangling task-specific adaptations with general capabilities. If you want to use the same model for multiple tasks, you either maintain separate fine-tuned copies (wasteful) or compromise on per-task performance. In the GATS paradigm, the pretrained models are permanent, stable infrastructure — like an operating system — while GATS modules are lightweight, task-specific interfaces that can be developed independently, swapped at will, and even upgraded without touching the underlying models.

This is a step toward what the paper's framing implies but doesn't quite state: a future where foundation models are deployed once as shared services, and downstream applications interact with them through standardized GATS interfaces that handle all multimodal routing, projection, and conditioning. The paper shows that this works for vision-to-language, language-to-vision, and vision-language-to-action; the natural extension is to arbitrary modality combinations with the same underlying models. The symmetrical design of GATS — where no modality is privileged — is what makes this vision coherent: you don't need to redesign the interface when you add audio, depth sensing, or tactile feedback; you just add new projection functions and context windows.

This contribution is incremental rather than fundamental — the paper demonstrates feasibility for one specific substitution (vision-only GATS → bimodal GATS) and on one model family — but it establishes a design pattern that, if validated across more diverse settings, could significantly change how multimodal AI systems are built and deployed.

5. Experimental Analysis

Evaluation Methodology

Dataset. The agent experiments span three environments: (1) Atari Pong from the Arcade Learning Environment (Bellemare et al., 2013), trained on 20,000 episodes sampled from a Muesli RL training run with average reward 17.89 (Reed et al., 2022); (2) Language-Table (Lynch et al., 2022), an open-source simulated tabletop manipulation benchmark using a mix of 442,226 real episodes and 181,020 simulated episodes collected at 5 Hz, with batches re-weighted to 75% simulated data; (3) YCB (built on the MuJoCo-based MoMa library), using a Panda Franka Emika arm with a Robotiq 2F-85 gripper, trained on 34,325 episodes (94% successful) generated via online RL with MPO (Abdolmaleki et al., 2018), featuring 12 YCB objects split into training objects (used as both distractors and targets) and test objects (used only as targets during evaluation, shown in Figure 6). The bimodal vision-language experiments use acquired image-text data (not a named public benchmark) with the Parti tokenizer operating at 256×256 resolution.
Base model(s). The paper draws from several pretrained foundation model families: Chinchilla (Hoffmann et al., 2022) for language, primarily the 1.3B-parameter variant; Phenaki (Villegas et al., 2023), a 1.1B-parameter causal video model that processes 11 input frames of 128×128 resolution through a C-ViViT encoder producing 16×16 discrete token frames; ViT vision transformers (Dosovitskiy et al., 2021) pretrained by the authors at two scales (2.7B and 8.5B total parameters, both including a frozen 1.3B Chinchilla language model conditioned via GATS); and the Parti (Yu et al., 2022) image tokenizer. The choice of Chinchilla 1.3B is not explicitly justified beyond it being "representative" and the models being "diverse"; the Phenaki and ViT models are chosen to compare video-based vs. image-based perception. For the bimodal model, the 7.0B ViT and 1.3B Chinchilla are combined into a 9.3B-parameter system where both foundation models remain frozen.
Metrics. Agent experiments report success rate as the primary metric. For Atari Pong, this is the game score (the agent "consistently achieving perfect scores" according to Section 4.3). For Language-Table, success rate is averaged across 500 evaluation episodes (100 per each of five canonical task families; Lynch et al., 2022), reported as a percentage. For YCB, no quantitative metric is reported — only qualitative examples of successful lifting behavior (Figure 9). For the bimodal model, training progress is measured by MaskGIT loss (Chang et al., 2022) for image generation and cross-entropy loss for text generation (Figure 11); image generation quality and captioning accuracy are assessed qualitatively through human inspection of examples (Figures 10 and 12).
Baselines. Several ablated versions of GATS serve as baselines rather than external methods:
- No vision steering (Table 1): GATS still gathers and attends over vision features, but the scatter step does not modify vision model activations. The paper describes this as resembling "a traditional cross-attention model" analogous to Flamingo (Alayrac et al., 2022). This is the primary baseline for quantifying the contribution of the steering mechanism.
- Video finetuned (no steering) (Table 1): The Phenaki video model is finetuned on Language-Table data (without action labels, using a future-frame prediction objective with causal masking) before being frozen and used without steering. This isolates the effect of domain adaptation through finetuning vs. steering.
- Video finetuned (with steering) (Table 1): The finetuned Phenaki is used with GATS steering, testing whether finetuning and steering are complementary or redundant.
- No classifier-free guidance (Table 1): Inference without the guidance term ( $\lambda = 0$ in the classifier-free guidance formula), isolating the contribution of guidance at test time.
- Greedy decoding for the ViT's image generation serves as a comparison point for the bimodal model's convergence speed (Figure 11, where the original ViT's MaskGIT loss curve is compared to the bimodal model's).
There are no comparisons to prior published methods on Language-Table or YCB — the paper does not report RT-2 (Brohan et al., 2023), Gato (Reed et al., 2022), or other multimodal agent results on these environments. The only external reference point is the Visual Language Model (VLM) success rates mentioned in the original Language-Table paper (Lynch et al., 2022), but those numbers are not reproduced or directly compared in Table 1.
Generation budget / compute accounting. The paper does not use a standardized compute budget metric across experiments. Instead, it reports trainable parameter counts as a proxy for training cost (Table 1): the image-based agent trains 495M parameters while keeping 2.6B frozen (out of 3.1B total); the video-based agent trains 182M parameters while keeping 1.1B frozen (out of 1.3B total); the bimodal model trains 988M GATS parameters while keeping 8.3B frozen (out of 9.3B total). For agent experiments, training steps (100k for image-based, 250k for video-based on Language-Table) and batch sizes (1024 for image-based, 512 for video-based, 256 for YCB video-based) are reported in the supplementary materials. There is no FLOPs analysis, no inference latency measurement, and no wall-clock time comparison between configurations. The paper argues qualitatively that GATS has "negligible computational overhead" (Figure 5 caption) because local context windows are short and projected embedding sizes are small, but this claim is never quantified.
Cross-validation / statistical protocol. No cross-validation, statistical significance testing, or confidence intervals are reported. Language-Table results are averaged over 500 episodes (100 per task family), and the paper implicitly treats this as the evaluation metric. There is no mention of multiple training runs, seed averaging, or error bars on any reported results. For the bimodal model, learning curves are shown as single runs (Figure 11). This is a significant gap — the dramatic ablation differences (89.0% vs. 30.4% for steering vs. no steering) are large enough that they likely survive variance, but smaller differences (e.g., 76.8% vs. 74.8% for steered pretrained Phenaki vs. non-steered finetuned Phenaki in Table 1) could plausibly fall within run-to-run noise, and the paper provides no tools to assess this.

Main Quantitative Results

Agent Experiments: Atari Pong

The Atari Pong experiment serves as a minimal viability test. The image-based GATS agent (using only vision and action modalities, with language tokens masked out) "quickly reaches human-expert performance, consistently achieving perfect scores" (Section 4.3). No numerical reward is reported beyond the statement that the fully trained Muesli agent's average reward is 17.89 (Section 4.1) — the GATS agent's actual score is not given. Three gameplay snapshots in Figure 7 show the agent winning two games without losing a point. This result primarily establishes that the GATS architecture does not catastrophically fail on a simple vision-only control task before moving to more complex multimodal settings.

Agent Experiments: Language-Table

Table 1 contains the paper's core quantitative results. The headline numbers:

Image-based GATS agent: 89.0% success rate (3.1B total parameters, 495M trainable). This uses the 2.7B ViT image model with frozen Chinchilla 1.3B language conditioning, plus a trainable action model and GATS module. The agent operates on 4-frame visual context with top-down camera views at an unspecified resolution (resized to 256×256 for the Parti tokenizer).
Video-based GATS agent: 76.8% success rate (1.3B total parameters, 182M trainable). This uses the 1.1B Phenaki video model with 5-frame visual context at 128×128 resolution.
Video finetuned (with steering): 75.6%. The finetuned Phenaki with steering enabled performs slightly worse than the un-finetuned Phenaki with steering (76.8%). The difference is small (1.2 percentage points), and without error bars or multiple runs, it is unclear whether this is a meaningful degradation or noise. The paper interprets this as showing that finetuning "does not achieve any improvement" over steering the original pretrained model.
No classifier-free guidance ablations: Image-based drops from 89.0% to 83.2% (−5.8 points); video-based drops from 76.8% to 73.6% (−3.2 points); video finetuned drops from 75.6% to 75.0% (−0.6 points). The guidance provides consistent but variable-magnitude improvements, with the largest gain on the image-based agent.

The paper states these results "should not be interpreted as suggesting a general superiority of image-based models over video-based models" (Section 4.4), correctly noting confounds: different pretraining datasets, model sizes, and input resolutions. The image-based agent has 2.4× more total parameters and processes higher-resolution images (256×256 vs. 128×128), making direct comparison uninformative about the image-vs-video question.

The steering ablation. This is the most important result in the paper:

Image-based agent without vision steering: 30.4% (vs. 89.0% with steering). A difference of 58.6 percentage points.
Video-based agent without vision steering: 24.2% (vs. 76.8% with steering). A difference of 52.6 percentage points.
Video finetuned without vision steering: 74.8% (vs. 75.6% with steering). A difference of only 0.8 percentage points.

The interpretation hinges on the finetuned-video result. When the Phenaki model is frozen without any domain adaptation, disabling steering causes performance to collapse (24.2%). When the Phenaki model is first finetuned on Language-Table frames, disabling steering still achieves 74.8% — nearly matching the steered pretrained model (76.8%). This demonstrates that steering and finetuning serve overlapping functions: both adapt the frozen model's representations to the downstream task. Steering does this at inference time without modifying weights; finetuning does it by updating weights. The fact that the non-steered finetuned model (74.8%) slightly underperforms the steered pretrained model (76.8%) is the paper's key evidence that steering can substitute for finetuning.

However, the steered finetuned model (75.6%) does not outperform the steered pretrained model (76.8%), suggesting that finetuning does not add value on top of steering for this specific configuration. The paper interprets this as GATS "eliminating the need for additional finetuning" — but this conclusion is drawn from a single model (Phenaki) on a single task, and the 1.2-point difference could easily be noise.

Agent Experiments: YCB

The YCB results are purely qualitative. Figure 9 shows the video-based GATS agent successfully lifting a lemon, a screwdriver, and a tennis ball — the first two are training objects, and the tennis ball is a test object (as shown in Figure 6's training/test split). The paper states: "Good results of our agent highlight the efficacy in this complex, multi-camera environment" (Section 4.5). No success rate, reward, or any quantitative metric is reported. The YCB section primarily demonstrates the ease of adapting GATS to multiple camera views (by treating each view as a separate modality) rather than providing rigorous performance evidence.

Bimodal Vision-Language Model

The 9.3B bimodal model (1.3B frozen Chinchilla + 7.0B frozen ViT + 988M trainable GATS) is evaluated through:

Image generation quality: Figure 10 shows five text-to-image generations for manually entered prompts. The images "adhere closely to the given text prompts" by qualitative assessment. The same images are then fed back as visual prompts (with cleared context) to generate captions, which the paper presents alongside the original prompts. This is a qualitative demonstration of bidirectional capability, not a quantitative evaluation. No metrics like FID, CLIP score, or human preference ratings are reported.
Captioning quality: Figure 12 shows five captions generated for public-domain Wikimedia Commons images (verified not to be in the training data). The captions are described as "accurately reflect[ing] the visual elements present in the images," but again no quantitative metric is reported (no BLEU, ROUGE, CIDEr, or SPICE scores).
Convergence speed: Figure 11 (top) shows that the bimodal model's MaskGIT loss drops below 5.45 after approximately 50,000 training steps, while the original ViT (trained from scratch without a frozen vision backbone) required approximately 750,000 steps to reach the same loss. This represents a ~15× speedup in reaching equivalent image generation performance by reusing the frozen ViT's weights. Figure 11 (bottom) shows the captioning cross-entropy loss continuing to decrease over training, confirming that the model simultaneously improves on text generation while maintaining image generation performance. The paper describes the MaskGIT loss as eventually "dipping below 5.45" for the bimodal model — the exact convergence value for the ViT baseline is not stated numerically but is visible in the figure.

Ablation Studies and Robustness Checks

Vision steering disabled (Table 1, "No vision steering"): This is the primary ablation and is discussed extensively above. The 58.6-point drop for the image-based agent and 52.6-point drop for the non-finetuned video-based agent establish that GATS steering — not just gathering and attending to features — is the mechanism responsible for frozen-model integration. The near-complete recovery for the finetuned video model (74.8% without steering vs. 75.6% with steering) suggests a nuanced interpretation: steering matters when there is distribution shift between pretraining and deployment, but its marginal value diminishes when models are already adapted to the target domain.

Classifier-free guidance disabled (Table 1, "No classifier-free guidance"): Removing guidance at inference time consistently reduces performance across all configurations. The magnitude varies: image-based agents lose 5.8 points, video-based lose 3.2 points, and video finetuned lose only 0.6 points. The paper does not ablate the guidance strength $\lambda$ — all experiments use $\lambda = 0.5$ without exploring other values — so it is unclear whether 0.5 is near-optimal or if the sensitivity to this hyperparameter varies across modalities.

Model finetuning vs. steering (Table 1, "Video finetuned" rows): The combination of finetuning and steering conditions creates a 2×2 design: {pretrained, finetuned} × {steering, no steering}. The results show:

Pretrained + no steering: 24.2% (baseline failure case)
Pretrained + steering: 76.8% (large steering benefit)
Finetuned + no steering: 74.8% (finetuning recovers most of the gap)
Finetuned + steering: 75.6% (no additional benefit)

This suggests that finetuning and steering are substitutes, not complements, for this specific configuration. However, the experiment does not exhaustively explore finetuning quality — the Phenaki was finetuned with a specific objective (future frame prediction) and a specific amount of data (the Language-Table dataset without action labels). Different finetuning strategies or data quantities might yield different complementarity patterns.

Number of trainable parameters (Table 1, "Unfrozen params" column): The image-based agent trains 495M parameters while the video-based agent trains 182M. The paper does not ablate parameter count (e.g., by reducing the image-based agent's action model or GATS module to match the video-based agent's budget) to determine whether the 12.2-point performance gap (89.0% vs. 76.8%) is due to the vision modality, the model architecture, or simply the 2.7× difference in trainable parameters.

Bimodal model: GATS parameter substitution (Section 5.3, Figure 11): The substitution experiment — discarding the 124M GATS from ViT pretraining and training a new 988M GATS for the bimodal task — demonstrates that GATS modules are not tightly coupled to their original training objective and can be swapped. The 15× convergence speedup (50k steps vs. 750k steps to equivalent MaskGIT loss) provides quantitative evidence that the frozen ViT's representations transfer. However, the paper does not ablate whether a smaller bimodal GATS (e.g., 124M parameters) would have sufficed, or whether the 988M size is necessary for the bimodal task.

Time-space factorization in the video action module (Table 4): The video-based agent uses a time-space factorization in its action model (odd layers attend over space, even layers attend over time). No ablation comparing factorized vs. full attention is reported, so the contribution of this design choice to performance or memory efficiency is unknown. The paper states it is used "to conserve device memory" — suggesting it was a practical necessity rather than a performance optimization — but the memory savings are not quantified.

Number of trainable input embeddings for the action model (Section 4.2): The image-based agent uses 8 trainable embeddings per time step; the video-based agent uses 16. The paper does not ablate this choice or explain the rationale for the difference. It is possible that video processing benefits from more learned state tokens, or that this number was simply carried over from separate development workflows without systematic tuning.

Negative result: Finetuning does not improve performance when steering is already enabled (Table 1): The steered finetuned video agent (75.6%) does not outperform the steered pretrained video agent (76.8%). This is a genuinely informative negative result — it suggests that if GATS steering is working well, additional domain-specific finetuning of the frozen model may be redundant or even slightly harmful. However, this conclusion is based on a single model/task combination with a narrow finetuning strategy.

Critical Assessment

Claim 1: GATS "enables seamless combination of pretrained foundation models" and achieves "state-of-the-art performance"

What the experiments actually show. The Language-Table image-based agent achieves 89.0% success. Is this state-of-the-art? The original Language-Table paper (Lynch et al., 2022) reports baseline success rates for various methods (including a "LSTM + CLIP" baseline and behavioral cloning with different architectures), but the GATS paper does not reproduce or directly compare against these numbers. The term "state-of-the-art" appears in Section 7 ("achieve state-of-the-art performance") but the paper never establishes what the state of the art actually is on Language-Table — no prior published results are cited in Table 1, no comparative baselines are run, and no leaderboard position is claimed. The 89.0% number could be excellent or merely competitive; the paper provides no frame of reference.

What's missing. A direct comparison against at least one prior method on Language-Table — such as the behavioral cloning baselines from Lynch et al. (2022) or a CLIP-based agent (Gadre et al., 2022) — would situate the 89.0% result. Without this, the claim of "state-of-the-art" is unsubstantiated. The same applies to YCB, where no quantitative metric is even reported.

Verdict: The paper demonstrates that GATS-based agents can achieve high success rates on Language-Table, but the claim of state-of-the-art performance is not supported by the reported experiments. The paper would need at minimum a comparison against published results on the same benchmark.

Claim 2: Steering is the critical mechanism enabling frozen-model integration

What the experiments actually show. The steering ablation (Table 1) provides strong evidence for this claim on Language-Table: disabling vision steering causes success rates to collapse for non-finetuned models (89.0% → 30.4% for image-based; 76.8% → 24.2% for video-based). The gap is large enough that statistical testing is almost certainly unnecessary. The finetuned-video recovery (24.2% → 74.8% when finetuning is added) provides the complementary control: it confirms that the collapse is specifically due to distribution shift (which finetuning can also address), not to some other architectural flaw in the non-steered configuration.

What's missing. The steering ablation is performed only on Language-Table with Phenaki and ViT. There is no steering ablation for Atari Pong (where language is masked out anyway, so steering might operate differently), for YCB, or for the bimodal vision-language model. The paper does not investigate which layers of the frozen model benefit most from steering — disabling steering at early vs. late GATS interleaving points would reveal whether the reprogramming effect is primarily low-level or high-level. There is also no analysis of what the steering mechanism actually changes in the vision model's representations — no probing experiments, no attention map visualizations, no feature similarity analyses. The claim that steering "reprograms" the frozen model is plausible given the performance gap, but we have no window into what that reprogramming consists of.

Verdict: The claim is well-supported for Language-Table with the tested model configurations, but the paper does not characterize how steering works or demonstrate that the finding generalizes beyond this specific task and model combination. The single-task, single-model nature of the ablation limits the strength of the conclusion.

Claim 3: GATS generalizes across diverse domains (games, robotics, multimodal generation)

What the experiments actually show. The paper demonstrates GATS in three settings: Atari Pong (vision-only control), Language-Table and YCB (language-vision-action), and bimodal vision-language generation. The diversity of settings is genuine — these are different modalities, different tasks, and different training objectives. The same core GATS mechanism (gather-attend-scatter with per-modality projections) is reused across all settings, supporting the claim of architectural generality.

What's missing. However, the evaluation depth varies dramatically across settings:

Atari Pong: No numerical result reported beyond "perfect scores." This is a minimal demonstration, not a rigorous evaluation. Pong is a solved problem for many methods.
Language-Table: Strong quantitative results with informative ablations. This is the paper's empirical center of gravity.
YCB: Purely qualitative (a few example images in Figure 9). No success rate, no comparison to baselines, no ablation. The YCB "results" demonstrate that GATS can be applied to a multi-camera manipulation task, not that it performs well.
Bimodal model: Qualitative image generation and captioning examples, plus convergence curves. No quantitative evaluation of generation quality or captioning accuracy against standard metrics or baselines.

The paper claims to demonstrate "the utility and versatility of GATS" (Section 1), but the evidence for "utility" is concentrated in one environment (Language-Table), while the other settings provide evidence for "versatility" (it can be applied) without demonstrating "utility" (it works well compared to alternatives). This is a real asymmetry in the empirical support.

Verdict: The claim of versatility (GATS can be applied to diverse settings) is supported. The claim of utility (GATS works well in those settings) is supported for Language-Table, weakly supported for Atari (no comparison point), and unsupported for YCB and the bimodal model (no quantitative baselines).

Claim 4: GATS has "lightweight inference overhead" and "negligible computational overhead"

What the experiments actually show. The paper provides only a qualitative argument for this claim: GATS projected embeddings are small ( $d = 512$ in most configurations), context windows are short (the specific $N_m$ values are not stated, but the paper emphasizes they are "short" and "relatively small"), and the number of GATS layers ( $K = 12$ or $18$ ) is smaller than the component models' layer counts. The supplementary materials provide parameter counts for the GATS modules — 124M for ViT pretraining, 988M for the bimodal model, and unspecified-but-small counts for agents (the action model dominates the trainable parameters in Table 1) — but no FLOPs analysis, no latency measurements, and no comparison of wall-clock time between GATS-based and alternative architectures.

The paper argues that "the only added overhead is caused by running GATS layers" (Section 2.6) and that GATS "only attends to recent activations, delegating long-term processing to the individual models, resulting in negligible computational overhead" (Figure 5 caption). But even a "lightweight" attention operation over multiple modalities repeated at 12–18 interleaving points could add non-trivial latency, especially in a real-time control loop. The paper provides no evidence for the "negligible" characterization.

What's missing. A basic FLOPs comparison between GATS-based agents and alternatives (monolithic transformer, Flamingo-style cross-attention) at equivalent parameter counts. Latency measurements (milliseconds per environment step) for the video-based agent, particularly comparing cached vs. uncached language model forward passes. An ablation of GATS depth ( $K$ ) showing how performance and inference cost trade off.

Verdict: The claim of lightweight overhead is asserted but not demonstrated. This is a significant gap given that inference efficiency is one of the paper's stated motivations for GATS (avoiding the serial bottleneck of monolithic architectures). Without quantitative latency or FLOPs data, the efficiency argument remains a design claim rather than an empirical finding.

Claim 5: GATS enables modular substitution — discarding and replacing GATS modules while reusing frozen models

What the experiments actually show. The experiment in Section 5.3 demonstrates one specific substitution: discarding a 124M GATS module (trained for vision-only image generation) and training a new 988M GATS module (for bimodal vision-language generation), while keeping the 7.0B ViT and 1.3B Chinchilla frozen. The new module converges ~15× faster than training from scratch (Figure 11). This is a compelling demonstration that the frozen ViT's representations transfer and that a new GATS module can learn to interface with them for a different task.

What's missing. The experiment tests exactly one substitution on one model family. It does not test whether the original 124M GATS could have been reused or fine-tuned for the bimodal task (which would be a more direct test of modularity), or whether the 988M GATS trained for bimodal generation could be subsequently replaced with yet another GATS for a third task. The substitution is unidirectional (from simpler to more complex) — we don't know whether a bimodal GATS can be swapped for a vision-only GATS, or whether GATS trained with one frozen vision model (e.g., ViT-2.7B) can be transferred to another (e.g., ViT-8.5B). The paper frames this as a general property ("GATS parameters coupled with the pretrained modules can be substituted with a new single GATS model trained from scratch," Section 5.3), but the evidence is a single demonstration.

Verdict: The modular substitution claim is demonstrated for one specific case and is genuinely interesting, but the paper overstates its generality. A broader set of substitution experiments — different tasks, different model scales, different numbers of modalities — would be needed to establish this as a general property rather than a one-off demonstration.

Overall Assessment of Experimental Rigor

The paper's empirical strengths are clear: the Language-Table results are substantial, the steering ablation is decisive, and the diversity of application domains demonstrates architectural flexibility. The paper also deserves credit for transparently reporting both positive results (steering benefits) and negative results (finetuning doesn't add value when steering works; ReST-EM mentioned in Section 6 as harming revision performance) where they occur.

The weaknesses are equally clear: (1) no quantitative metrics for YCB or the bimodal model's generation quality; (2) no external baselines or prior-work comparisons anywhere in the paper; (3) no statistical reporting (error bars, significance tests, multiple runs); (4) no ablation of key hyperparameters (guidance strength $\lambda$ , GATS depth $K$ , projected embedding size $d$ , per-modality context lengths $N_m$ ); (5) no characterization of inference cost despite efficiency being a stated motivation; and (6) heavy reliance on a single benchmark (Language-Table) for all quantitative agent results. The Atari and YCB experiments function more as existence proofs than rigorous evaluations, and the bimodal model section would be significantly strengthened by standard generation quality metrics.

The paper's empirical contribution is best characterized as a strong proof of concept with one well-validated application (Language-Table) rather than a comprehensive evaluation across the claimed scope. The steering mechanism's importance is convincingly demonstrated for one task; its generality, efficiency, and modularity remain more asserted than proven.

6. Limitations and Trade-offs

6.1 The Steering Mechanism Is Characterized Only Through a Binary Ablation — We Don't Know What It Actually Changes

The assumption or constraint. The paper's central claim is that GATS steering "reprograms" frozen pretrained models by modifying their intermediate activations, enabling them to serve downstream tasks without weight updates. The paper demonstrates this via a single ablation: disabling all vision steering causes a catastrophic performance drop (Table 1, 89.0% → 30.4% for the image-based agent). The authors interpret this gap as evidence that steering is the critical enabling mechanism for frozen-model integration.

However, the paper provides no characterization of what steering actually does to the frozen model's representations. There are no probing experiments, no attention map visualizations, no representational similarity analyses comparing steered vs. unsteered activations, and no layer-by-layer analysis of where and how steering modifies the vision model's internal computation. The paper asserts that steering works by "leveraging a property of neural networks shown by Alayrac et al. (2022) that a given network's behavior can be reprogrammed by modifying its activations" (Section 1), but the nature of this reprogramming — whether it suppresses irrelevant features, amplifies task-relevant ones, injects entirely new information, or something else — remains opaque.

The consequence. This opacity has several practical implications. First, it makes it difficult to predict when steering will be effective for a new task or a new pretrained model. The paper shows it works for Phenaki and ViT on Language-Table, but a practitioner considering a different vision model (e.g., a CLIP encoder, a DINOv2 backbone) or a different task (e.g., navigation rather than manipulation) has no principled way to assess whether steering will succeed or what hyperparameters to tune. Second, the lack of mechanistic understanding means there is no guidance on where to interleave GATS layers for maximum effect. The proportional spacing formula (Equation 4) is a simple heuristic; perhaps steering early layers is more important than steering late ones, or vice versa, but the paper provides no evidence either way. Third, if steering fails on a new domain, a practitioner has no diagnostic tools — no way to inspect whether the failure is due to insufficient steering strength, projection capacity, or fundamentally incompatible pretrained representations.

What evidence exists in the paper. The Table 1 ablation establishes that steering matters without providing how it matters. The finetuned-video experiment provides one mechanistic hint: finetuning the Phenaki model on Language-Table frames (via future-frame prediction) restores near-complete performance even without steering (74.8% vs. 76.8%), suggesting that steering and finetuning serve overlapping functions in adapting representations to the target distribution. But this is a behavioral observation, not a mechanistic one — we still don't know whether steering modifies the same features that finetuning modifies, or whether it achieves adaptation through a different mechanism entirely.

Mitigation status. The paper does not discuss this limitation or propose future work to characterize the steering mechanism. The authors treat the ablation result as sufficient evidence that steering works, without acknowledging the interpretability gap. Addressing this would require experiments like: (1) probing the vision model's representations at different layers for task-relevant information (object identity, spatial location, grasp affordances) with and without steering; (2) visualizing attention patterns in the GATS self-attention to see which modalities attend to which others; (3) measuring the magnitude of steering modifications (the gate values $g_m(z_i)$ ) across different layers and input conditions to understand when steering is active vs. dormant.

6.2 No Quantitative Performance Metrics for YCB Robotics or the Bimodal Generation Model — The "Versatility" Claim Rests on a Single Benchmark

The assumption or constraint. The paper frames GATS as a general-purpose tool demonstrated "across games, robotics, and multimodal input-output systems" (Section 1). However, the empirical evidence for versatility is radically uneven across these domains:

Atari Pong: No numerical result is reported beyond the qualitative statement that the agent "quickly reaches human-expert performance, consistently achieving perfect scores" (Section 4.3). The baseline Muesli agent's reward (17.89, Section 4.1) is given, but the GATS agent's actual score is never stated.
Language-Table: Strong quantitative results with informative ablations (Table 1). This is the only setting where rigorous evaluation occurs.
YCB: Purely qualitative — five images in Figure 9 showing successful lifts of three objects (lemon, screwdriver, tennis ball). No success rate, no reward, no failure analysis, no comparison to prior methods. The paper states "Good results of our agent highlight the efficacy in this complex, multi-camera environment" (Section 4.5) without defining or measuring "good."
Bimodal model: Qualitative image generation examples (Figure 10) and captioning examples (Figure 12) with convergence curves (Figure 11). No FID, CLIP score, BLEU, ROUGE, or any standard generation metric is reported. No comparison to prior text-to-image or image captioning models.

The consequence. A practitioner evaluating whether to adopt GATS for a robotics application would find precisely one quantitative result — Language-Table — with no evidence that the performance generalizes to other tasks, embodiments, or data regimes. The YCB environment is described as "a more challenging domain that demands manipulation of objects given multiple camera viewpoints" and "heightened precision and coordination" (Section 4.1), but the paper provides no way to assess whether GATS succeeds or fails at this challenge relative to alternatives. If the YCB success rate were, say, 40%, the paper's framing would be misleading; if it were 90%, it would be a significant contribution. Without numbers, neither case can be made.

For the bimodal model, the absence of standard generation metrics is particularly problematic because Figure 10 and 12 are curated examples. A model could produce five impressive images while having a 90% failure rate on a random sample — the paper provides no way to distinguish this scenario from genuinely reliable generation. The convergence curve in Figure 11 shows that training is proceeding (loss is decreasing), but loss curves do not validate output quality.

What evidence exists in the paper. The Language-Table results (Section 4.4, Table 1) are the only quantitative evidence for agent performance. The Atari, YCB, and bimodal sections provide existence proofs that GATS can be applied in these domains, not evidence that it performs well.

Mitigation status. The paper does not acknowledge the uneven evaluation depth as a limitation. The term "state-of-the-art" appears in Section 7 without specifying which results support this claim or providing comparative baselines. To address this, the paper would need at minimum: (1) reported success rates on YCB with comparison to a baseline method; (2) FID, CLIP score, or human preference ratings for the bimodal model's image generation against a comparable text-to-image model; (3) quantitative captioning metrics (BLEU, CIDEr, SPICE) against a standard image captioning baseline.

6.3 Inference Overhead Is Asserted as "Negligible" but Never Measured — The Efficiency Advantage Over Monolithic Architectures Is a Design Claim, Not an Empirical Finding

The assumption or constraint. One of the paper's stated motivations for GATS is that monolithic architectures like Gato suffer from "sequential token processing, forcing modalities to wait for each other" and "equal compute allocation to each token" (Section 2.5). GATS is positioned as the solution: "tokens from each modality can be processed by the corresponding unimodal models simultaneously" and the overhead is "negligible" because "GATS only attends to recent activations, delegating long-term processing to the individual models" (Figure 5 caption). The paper further claims that "GATS projected embeddings $d$ are small and GATS local context lengths $N_1, N_2, \ldots, N_M$ are short" (Section 2.6), making the added computation lightweight.

These claims are never quantified. The paper reports no FLOPs analysis, no latency measurements, no memory profiling, and no wall-clock time comparisons between GATS-based architectures and alternatives (monolithic transformers, Flamingo-style cross-attention, or simple feature extraction pipelines). The trainable parameter counts in Table 1 (495M for image-based, 182M for video-based) quantify training cost but not inference cost, since the frozen models' parameters still participate in the forward pass and contribute to latency.

The consequence. Without quantitative efficiency data, a practitioner cannot assess whether GATS is appropriate for their latency budget. Consider the video-based agent operating at 5 Hz in Language-Table. If each environment step requires: (1) a forward pass through the 1.1B-parameter Phenaki video model, (2) a forward pass through the action model (likely 100M+ parameters based on Table 4), (3) 12 GATS layers gathering from all modalities, projecting to $d=512$ , computing self-attention over the gathered window, and scattering back — this could easily exceed the 200ms per-step budget for 5 Hz control, especially on embedded hardware. The cached language model activations help (the Chinchilla 1.3B runs only once), but the per-step cost of the video model plus GATS layers might still be substantial.

Moreover, the GATS module itself, while described as "lightweight," involves cross-modal self-attention over the gathered set $G$ . If $G$ contains, say, 30–50 embeddings across all modalities (a plausible estimate given multiple camera views, action history, and language), and this attention is repeated at 12–18 interleaving points throughout the component models, the cumulative cost is not obviously negligible. The 988M-parameter GATS in the bimodal model (Table 3) is larger than many entire language models — calling its 12 transformer layers with width 2048 and 16 attention heads "lightweight" requires context that the paper does not provide.

What evidence exists in the paper. The supplementary materials (Tables 3–5) provide architecture specifications (transformer blocks, attention heads, layer widths) for the GATS modules, from which approximate FLOPs could be estimated by a motivated reader. But the paper itself performs no such calculation and provides no latency measurements for any configuration. The claim of simultaneous processing is also not validated — the forward passes of different component models are synchronized at GATS interleaving points (Section 3.4.2), which may serialize computation in practice even if the models are logically independent.

Mitigation status. The paper does not acknowledge the absence of efficiency measurements as a limitation. The discussion in Section 2.6 ("Lightweight inference overhead") treats the efficiency argument as settled by design rather than requiring empirical validation. To credibly claim "negligible" overhead, the paper would need to report: (1) FLOPs per inference step for each agent configuration; (2) wall-clock latency per environment step on standard hardware; (3) comparison to a monolithic baseline of equivalent total parameter count; (4) an ablation of GATS depth $K$ showing the performance-latency tradeoff curve.

6.4 All Quantitative Agent Results Come from a Single Benchmark and a Single Model Family — Transferability Across Tasks, Embodiments, and Pretrained Models Is Unverified

The assumption or constraint. The paper's strongest quantitative evidence — the Language-Table results with the steering ablation — comes from one simulated tabletop manipulation environment using one family of pretrained models (Chinchilla for language, Phenaki or a custom ViT for vision). The Atari Pong experiment is a minimal test with no numerical results. The YCB experiment provides qualitative demonstrations only.

The paper explicitly positions GATS as "agnostic to the specific details of the neural networks being combined" (Section 2.6) and capable of integrating "any deep neural network" (Section 1). This is a claim of architectural generality — the GATS mechanism should work with any pretrained transformer regardless of its architecture, training data, or output modality. But the experiments test this generality claim weakly: two vision architectures (Phenaki video model, custom ViT), one language architecture (Chinchilla), and one embodiment class (tabletop manipulation with a single arm).

The consequence. Several important generalization questions are left entirely open:

Task diversity: Language-Table tests instruction-following tabletop manipulation. Would GATS-based agents work for navigation (requiring spatial reasoning over large environments), dexterous manipulation (requiring fine-grained force control), or multi-agent coordination (requiring modeling other agents' intentions)? The paper provides no evidence.
Embodiment transfer: All robotic experiments use simulated arms (a 2D constrained arm in Language-Table, a Panda Franka Emika in YCB). Would the same GATS architecture work for a mobile manipulator, a drone, or a humanoid? The per-modality context window design assumes consistent processing rates across embodiments, but a drone with 30 Hz visual input and 100 Hz control would stress the GATS gather mechanism differently than a 5 Hz tabletop arm.
Pretrained model diversity: The experiments use two large-scale pretrained vision models (Phenaki 1.1B, ViT 2.7B) and one language model family (Chinchilla 1.3B). What if a practitioner wants to use a different language model (e.g., LLaMA, PaLM) with a different architecture, tokenizer, or pretraining objective? What if the vision model is a CLIP encoder, a DINOv2 backbone, or a video model pretrained on egocentric data rather than third-person video? The paper's claim of agnosticism is architectural, but the empirical validation is narrow.
Failure modes: The paper reports no systematic failure analysis on any benchmark. We don't know whether GATS agents fail due to perception errors (missing the target object), language grounding errors (misunderstanding the instruction), action execution errors (poor motor control), or cross-modal coordination failures (GATS failing to route the right information at the right time). Without failure mode analysis on even the well-studied Language-Table benchmark, it's impossible to diagnose where GATS's strengths and weaknesses lie.

What evidence exists in the paper. The diversity of application domains (games, tabletop robotics, object manipulation, image generation, captioning) demonstrates that GATS can be applied in diverse settings, but the depth of evaluation in each setting varies from substantial (Language-Table) to negligible (Atari, YCB). There is no experiment that systematically varies the pretrained model, the task, or the embodiment while holding other factors constant.

Mitigation status. The paper does not discuss generalization scope as a limitation. The authors present the domain diversity as evidence of versatility, without acknowledging that the evaluation depth is insufficient to support strong claims about transferability. A more cautious framing would present Language-Table as the primary validation and the other settings as preliminary demonstrations. To address this limitation, future work would need: (1) evaluation on at least one additional robotics benchmark with quantitative success metrics; (2) experiments swapping the pretrained vision or language model for alternatives (e.g., CLIP instead of ViT) to test model-agnosticism; (3) systematic failure analysis on Language-Table to characterize where steering succeeds and fails.

6.5 Classifier-Free Guidance and Steering Are Confounded — We Don't Know Which Mechanism Drives the Performance Gains

The assumption or constraint. The paper reports that classifier-free guidance (Section 3.1, Equation 5) "consistently improves the performance of our agents" (Table 1), with gains ranging from 0.6 to 5.8 percentage points depending on the configuration. However, the paper applies guidance at inference time to all reported results (except the "No classifier-free guidance" ablation rows in Table 1), and the guidance strength $\lambda = 0.5$ is held constant across all experiments without any sweep or tuning study. The guidance mechanism amplifies the difference between the conditional policy $l(x, c)$ (with language instruction) and the unconditional policy $l(x)$ (with masked text), effectively sharpening the model's focus on instruction-following behavior.

The problem is that guidance and steering are applied jointly in the main results, but their individual contributions and potential interactions are not disentangled. The 89.0% image-based agent result includes both steering and guidance. The 30.4% "no vision steering" result also includes guidance (since only vision steering is disabled, not the guidance mechanism). We know that removing guidance from the full agent reduces performance (89.0% → 83.2%), and we know that removing steering reduces performance catastrophically (89.0% → 30.4%). But we don't know how much of the steering benefit is actually attributable to steering itself versus an interaction between steering and guidance.

The consequence. This confounding has several implications for interpreting the paper's main claims:

Overattribution to steering: The 58.6-point drop when steering is disabled might partially reflect the fact that the non-steered architecture cannot effectively leverage the guidance mechanism, not that steering per se is responsible for the entire 58.6 points. If the non-steered architecture produces features that are poorly aligned with the instruction, the guidance term $l(x, c) - l(x)$ would be near zero (since the model cannot distinguish conditional from unconditional behavior), making guidance ineffective regardless of $\lambda$ . In this case, steering would be important primarily because it enables guidance to work, not because it directly improves the policy.
Uncertainty about the optimal $\lambda$ : The paper uses $\lambda = 0.5$ throughout without justifying this choice or reporting sweeps. If the optimal $\lambda$ varies across configurations (e.g., requiring stronger guidance for the image-based agent than the video-based agent), the reported performance gaps between configurations might partly reflect suboptimal guidance tuning rather than genuine architectural differences.
Guidance as a confound in the finetuning comparison: The steered pretrained Phenaki (76.8%) slightly outperforms the non-steered finetuned Phenaki (74.8%). But both numbers include guidance. If guidance interacts differently with steered vs. finetuned representations, this 2.0-point gap might reflect a guidance artifact rather than a steering advantage.

What evidence exists in the paper. Table 1 reports guidance-off ablations for three configurations: image-based agent drops 5.8 points, video-based agent drops 3.2 points, and video finetuned drops only 0.6 points. The variable magnitude confirms that guidance's benefit depends on the configuration, but the paper does not explore why the finetuned model benefits less from guidance or whether tuning $\lambda$ per configuration would change the relative ordering.

Mitigation status. The paper does not discuss the confounding between steering and guidance. The separate "No classifier-free guidance" rows in Table 1 partially address this by showing that the main results hold even without guidance (83.2% for image-based, 73.6% for video-based, both substantially above the no-steering baselines), but they don't address the interaction question. To disentangle these mechanisms, the paper would need: (1) a factorial experiment crossing {steering, no steering} × {guidance, no guidance} × {multiple $\lambda$ values} to measure main effects and interactions; (2) an analysis of the guidance term $l(x, c) - l(x)$ magnitude and variance across steered and non-steered conditions to test whether steering enables larger or more reliable guidance signals.

6.6 Difficulty Estimation Is Free in This Setting — The Paper Provides No Path to Practical Deployment Where Modality Alignment Quality Varies

The assumption or constraint. The GATS mechanism assumes that the pretrained models being connected produce well-structured, semantically meaningful activations that can be productively combined through cross-modal attention. This assumption holds for the models tested — Chinchilla 1.3B was trained on a large, high-quality text corpus; Phenaki was trained on diverse video data; the custom ViT was explicitly trained with language conditioning via GATS. In all cases, the pretrained models have strong, well-aligned internal representations that make cross-modal steering feasible.

But this is not guaranteed for arbitrary pretrained models. A video model trained only on autonomous driving data might have representations that are poorly aligned with the concepts needed for tabletop manipulation (where the relevant features are object shapes, colors, and spatial relationships rather than lane markings and traffic signs). A language model fine-tuned on code might have degraded representations for natural language instructions. The paper's implicit assumption is that the "quality" of cross-modal interaction is determined entirely by the GATS module's capacity to learn useful projections and attention patterns, not by the underlying compatibility of the pretrained representations. However, if two pretrained models represent similar concepts in incommensurable ways (e.g., a CLIP vision encoder's notion of "red cube" is orthogonal to a Chinchilla language model's notion of "red cube"), no amount of GATS training can bridge that gap — the projections $p_m$ are linear and therefore limited in their capacity to align fundamentally different representational geometries.

The consequence. A practitioner attempting to connect pretrained models from different research groups, trained on different data distributions with different objectives, may find that GATS steering provides limited or no benefit — not because GATS is architecturally flawed, but because the pretrained models' representations are too misaligned for linear projection to reconcile. The paper provides no diagnostic for assessing this compatibility before committing to a GATS-based architecture, and no guidance on minimum requirements for pretrained model quality (e.g., "your vision model should have been trained with at least some language supervision" or "your language model's embedding dimension should be within 2× of your vision model's").

This is particularly relevant for the modular substitution vision outlined in Section 5.3. If a frozen ViT trained with Chinchilla conditioning is "compatible" with a new GATS module, but a frozen ViT trained with a different language model is not, then the composability promise is conditional on pretraining alignment rather than being a universal property of the GATS interface.

What evidence exists in the paper. The paper inadvertently provides evidence for this limitation through the Phenaki finetuning experiment (Table 1). The pretrained Phenaki model, when used without steering, achieves only 24.2% on Language-Table — dramatically worse than the 76.8% with steering. This suggests that the pretrained Phenaki representations are substantially misaligned with the Language-Table task distribution (the paper notes that Phenaki was "pretrained with 11 input frames" and "a temporal causal mask [was applied] for efficient execution" in the agent, acknowledging distribution shift; Section 4.4). Steering bridges this gap, but the gap exists in the first place. If the representations were even more misaligned — say, from a video model trained only on nature documentaries — steering might not be sufficient. The paper provides no characterization of how much misalignment GATS can compensate for.

The finetuned Phenaki result (74.8% without steering) further reinforces this: finetuning aligns the representations to the target domain, after which steering provides minimal additional benefit. This suggests that GATS's value is specifically in bridging moderate distribution shift, not in enabling arbitrary cross-model integration.

Mitigation status. The paper does not discuss representation compatibility as a limitation or provide diagnostics for assessing it. The authors present the linear projections $p_m$ as sufficient for bridging arbitrary modality gaps without discussing their representational capacity limits. Future work could address this by: (1) measuring representational similarity (e.g., CKA, SVCCA) between pretrained models' activations before GATS training and correlating it with downstream steering effectiveness; (2) testing GATS with deliberately misaligned pretrained models (e.g., a CLIP vision encoder vs. a Chinchilla language model with no shared pretraining) to establish the limits of linear projection; (3) exploring nonlinear projections $p_m$ (e.g., small MLPs) to increase the capacity for aligning disparate representational geometries.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a genuinely new architectural primitive — the Gather-Attend-Scatter module — that reframes multimodal integration from a directed wiring problem into a symmetrical communication problem. The magnitude of this shift is best understood not as a paradigm overthrow (the core components — transformers, residual connections, attention — are all standard) but as a conceptual reframing with immediate practical consequences for how multimodal systems are built. Before GATS, the dominant approach for connecting pretrained models was to design specific cross-attention pathways for each modality pair (Flamingo: vision → language; various robotics systems: vision → action; CLIP-based agents: frozen vision features → policy). Each new pathway required architectural engineering, hyperparameter tuning, and verification that the connection didn't destabilize training. GATS replaces this with a single, reusable module that makes no architectural distinction between source and target modalities — the same self-attention operation handles vision → language, language → vision, action → language, or any combination thereof, with the direction of influence controlled by learned gating functions rather than hard-coded connectivity.

The practical consequence is a substantial reduction in the engineering complexity of building multimodal systems. The YCB adaptation (Section 4.2) — where adding a second camera view is described as "trivial" — illustrates the operational shift. In a cross-attention paradigm, adding a new camera would require designing a new cross-attention pathway, allocating new parameters, and verifying training stability. In GATS, it requires adding a new per-modality projection function $p_m$ and a new context budget $N_m$ , with the rest of the architecture unchanged. This is not merely implementation convenience; it changes what kinds of multimodal experiments are feasible. A researcher who wants to test whether adding audio, depth sensing, or tactile feedback improves robotic manipulation can do so without redesigning the model architecture each time.

The paper also reconciles a tension that has been present but unarticulated in the multimodal learning literature: the conflict between parameter reuse (keeping pretrained models frozen to preserve knowledge) and effective cross-modal conditioning (which typically requires modifying those models' internals). The dominant solutions have been polar — either accept frozen feature extraction with limited cross-modal interaction (CLIP-based robotics systems), or fine-tune and risk forgetting (RT-2). GATS demonstrates a third path: modify the activations of frozen models via gated residual injections, achieving strong cross-modal conditioning without weight updates. The Language-Table steering ablation (89.0% → 30.4% when vision steering is disabled, Table 1) provides the clearest evidence to date that activation reprogramming — not just feature extraction — is the critical mechanism for making frozen models work in interactive multimodal settings. This finding shifts attention away from the false dichotomy of "freeze or fine-tune" toward a more nuanced question: what is the minimal intervention in a frozen model's computation that enables effective downstream adaptation?

Perhaps the most forward-looking contribution is the demonstration that GATS modules can be discarded and replaced while reusing the same frozen foundation models (Section 5.3, Figure 11). The 15× convergence speedup when substituting a new GATS module for a bimodal task — reusing the frozen 7.0B ViT and 1.3B Chinchilla — is an existence proof for a composable AI workflow where foundation models are permanent infrastructure and task-specific adaptation is confined to lightweight, swappable connector modules. This is not yet a practical reality (the experiment tests a single substitution on a single model family), but it establishes a design pattern that, if validated more broadly, could significantly change how the field thinks about model deployment. Instead of fine-tuning and deploying separate copies of large models for each downstream task, a single frozen instance could serve many tasks through different GATS interfaces.

The research directions that become more attractive after this work include: (1) standardizing the "GATS interface" so that pretrained models from different sources can be plugged together without custom engineering; (2) developing methods for automatically learning which modalities should steer which others, rather than manually specifying the steering set $S$ ; (3) characterizing exactly what steering does to frozen model representations (probing, visualization, causal intervention studies); and (4) scaling GATS to larger numbers of modalities and more diverse pretrained models.

The research directions that become less attractive are those that assume multimodal integration requires either (a) fine-tuning all components, with the attendant forgetting risk and deployment cost, or (b) designing custom cross-attention pathways for each new modality pair. GATS provides a simpler default that, while not yet proven universally superior, is architecturally more general and experimentally easier to extend. A researcher starting a new multimodal project today should, at minimum, consider whether the directed-wiring complexity of cross-attention is justified over the symmetrical GATS alternative.

Follow-Up Research This Work Enables

1. Probing what GATS steering actually changes in frozen models — and where. The paper establishes that steering matters (the 58.6-point drop in Table 1) but provides no window into how it works. A strong follow-up would systematically characterize the effect of steering on the frozen vision model's internal representations across different layers and task conditions. Concrete experiments: (1) Train linear probes on the Phenaki or ViT model's intermediate activations to predict task-relevant features (object identity, spatial location, grasp affordances) at each layer, comparing steered vs. unsteered conditions. This would reveal whether steering primarily amplifies existing task-relevant features, suppresses distractors, or injects entirely new information. (2) Measure the magnitude of steering modifications — the gate values $g_m(z_i)$ from Equation 3 — as a function of layer depth, input complexity, and task difficulty. Are early visual layers steered more than late ones, or vice versa? Does steering become more aggressive when the scene is cluttered with distractors? (3) Ablate steering at individual GATS interleaving points (keeping some GATS layers active and others dormant) to identify which layers of the vision model benefit most from cross-modal conditioning. The proportional spacing formula (Equation 4) is a heuristic — the data might reveal that steering early layers (low-level features) or late layers (semantic features) dominates. This work would transform steering from a black-box phenomenon into a principled design tool.

2. Testing the limits of GATS's model-agnosticism with deliberately misaligned pretrained models. The paper demonstrates GATS with models that share substantial implicit alignment — Chinchilla and Phenaki were both trained on large-scale web data; the custom ViT was explicitly co-trained with Chinchilla conditioning. What happens when the pretrained models are less compatible? A stress-test experiment would pair (a) a CLIP vision encoder (trained with contrastive language-image alignment) with a Chinchilla language model (trained with autoregressive next-token prediction), and (b) the same Chinchilla with a DINOv2 vision backbone (trained with self-supervised learning, no language alignment at all). The hypothesis is that GATS performance degrades as representational alignment decreases, and the experiment would measure how much degradation and whether increasing GATS capacity (larger $d$ , deeper $K$ , nonlinear projections) can compensate. If linear projections are insufficient for bridging substantially different representational geometries, this would motivate research into learned nonlinear projection functions within the GATS framework — a natural extension that preserves the symmetrical architecture while increasing cross-modal alignment capacity. The Phenaki finetuning result (Table 1) already hints at this limitation: the pretrained Phenaki without steering achieves only 24.2%, suggesting substantial distribution shift that steering compensates for. Testing with even more misaligned models would establish the boundary conditions for GATS's claimed agnosticism.

3. Learning the steering set S from data rather than specifying it manually. The current GATS design requires the practitioner to specify which modalities are steered ( $S \subseteq \{1, \ldots, M\}$ ) as a hyperparameter (Section 2.2). The paper notes this "could potentially be learned from data" but does not explore it. A natural extension would replace the manual $S$ with a learned per-modality steering probability or continuous steering strength parameter that is optimized during training. Concrete approach: instead of a binary steer/don't-steer decision, parameterize a learnable scalar $\alpha_m \in [0,1]$ for each modality that multiplies the gating function output in Equation 3: $x_i \leftarrow x_i + \alpha_m \cdot g_m(z_i) \cdot r_m(z_i)$ . Train $\alpha_m$ jointly with the GATS parameters using the same downstream objective. The hypothesis is that $\alpha_m$ would converge to near-zero for modalities where cross-modal conditioning is irrelevant or harmful (e.g., a frozen language model that should remain stable) and near-one for modalities that benefit from reprogramming. This would eliminate a manual design choice and make GATS more truly "automatic" — you connect the models, specify the training objective, and the architecture learns who should influence whom. A particularly interesting variant would make $\alpha_m$ input-dependent, allowing a modality to be steered on some examples but not others (e.g., the vision model is reprogrammed when the scene is cluttered but left alone when it's simple).

4. Systematic comparison of GATS vs. fine-tuning for preserving pretrained capabilities on held-out tasks. The paper argues that GATS avoids "the risk of losing important knowledge acquired during the pretraining phase" (Section 1) but never empirically demonstrates preserved knowledge. A critical follow-up would evaluate the frozen models before and after being used in a GATS-based agent on their original pretraining tasks. For the Phenaki video model, measure video generation quality (FVD, IS) on a held-out video dataset after the model has been used in a Language-Table agent for 250k training steps. For the Chinchilla 1.3B, measure perplexity on a held-out text corpus. The prediction is that GATS-steered models retain full performance on their original tasks (since weights are frozen), while a fine-tuned baseline would show degradation. However, there is a subtle concern the paper does not address: if GATS steering modifies activations strongly enough to "reprogram" model behavior for the downstream task, could these modifications propagate through the residual stream and affect the model's behavior even on in-distribution inputs? A rigorous test would run frozen models with GATS steering disabled on their original tasks after GATS training, confirming that the weights are truly untouched and no indirect adaptation occurred. This experiment would directly validate the paper's core value proposition: that GATS provides strong downstream adaptation without sacrificing any pretrained capability.

5. Scaling GATS to more modalities, larger models, and real-time hardware — quantifying the actual efficiency boundary. The paper claims "negligible computational overhead" (Section 2.6) without measurement. A crucial engineering follow-up would implement a GATS-based agent on physical robot hardware and measure: (1) end-to-end inference latency per control step as a function of the number of modalities, GATS depth $K$ , and projected embedding size $d$ ; (2) memory consumption during deployment, including the cost of caching language model activations; (3) a head-to-head latency comparison against a monolithic transformer and a Flamingo-style cross-attention model with matched parameter counts. The experiment would sweep $K = \{4, 8, 12, 18, 24\}$ and $d = \{256, 512, 1024, 2048\}$ on a fixed robotic task (Language-Table) to produce a Pareto frontier of performance vs. latency. This would either validate the "lightweight" claim with concrete numbers or reveal where GATS becomes too expensive for real-time control — information essential for practitioners deciding whether to adopt the architecture. The video-based agent's time-space factorization (Table 4) suggests memory was already a concern during development; quantifying these costs would provide the deployment guidance the current paper lacks.

6. Combining GATS with parameter-efficient fine-tuning for a full spectrum of adaptation strategies. The paper positions GATS steering as an alternative to fine-tuning, but the two approaches address different aspects of the adaptation problem: steering provides cross-modal conditioning, while fine-tuning (even parameter-efficient methods like LoRA) adapts a model's internal computation to a new domain. The finetuned Phenaki experiment (Table 1) tests the extreme case where the model is fully finetuned on domain data, but a more practical combination would use LoRA to provide mild domain adaptation while GATS handles cross-modal steering. A factorial experiment crossing {GATS steering, no steering} × {LoRA on vision model, frozen vision model} × {LoRA on language model, frozen language model} would characterize when each mechanism is most valuable. The prediction is that LoRA helps most when the pretrained model's domain gap is large (e.g., Phenaki trained on internet video applied to tabletop robotics), while GATS steering helps most when cross-modal grounding is critical (e.g., identifying which of several objects matches an instruction), and the combination outperforms either alone. This would refine the paper's "steering vs. fine-tuning" framing into a more nuanced "steering and fine-tuning solve different problems and can be composed."

Practical Applications and Downstream Use Cases

1. Rapid prototyping of multimodal robotic agents with off-the-shelf pretrained models. The most immediately actionable use case is for robotics research groups that want to build language-conditioned manipulation agents without maintaining separate fine-tuned copies of large vision and language models. Using GATS, a team can take a pretrained vision model (e.g., a publicly available ViT or video model), a pretrained language model (e.g., any open-source LLM), and a small custom action model, connect them through a GATS module, and train only the GATS and action parameters (495M trainable out of 3.1B total for the image-based agent, Table 1). The paper shows this achieves 89.0% success on Language-Table while keeping both foundation models frozen, meaning the same instances can be shared across multiple projects without risk of cross-contamination from task-specific fine-tuning. The per-modality context window design (Section 2.1) — where the language model runs once and caches its activations — means the largest model in the system (1.3B parameters) contributes no per-step latency, making this feasible on hardware that could not run the language model at control frequency.

2. Multi-camera robotic perception without architectural redesign. The YCB adaptation (Section 4.2) demonstrates a specific practical benefit: adding camera views to a GATS-based system requires no new cross-attention pathways, no additional architectural engineering, and no changes to the underlying vision model. A robotic workcell that starts with one camera and later adds a wrist-mounted camera for fine manipulation can simply register the new stream as an additional GATS modality with its own context budget $N_m$ and projection function $p_m$ . The self-attention mechanism automatically allows the new view to influence and be influenced by existing modalities. This is in contrast to cross-attention architectures where each new camera view requires designing a new cross-attention module and retraining potentially the entire system. For industrial robotics deployments where camera configurations evolve over time, this architectural flexibility translates directly to reduced engineering cost and faster iteration cycles.

3. Composable multimodal deployment where a single frozen model instance serves multiple applications. The GATS substitution experiment (Section 5.3) — discarding a 124M GATS module and training a new 988M one while reusing the same frozen 7.0B ViT and 1.3B Chinchilla — suggests a deployment model where large foundation models are hosted as shared infrastructure and thin GATS interfaces provide task-specific routing. A single frozen Chinchilla 1.3B instance could simultaneously serve: (a) a robotic manipulation agent (via a GATS module connecting it to a vision model and action head), (b) an image captioning service (via a different GATS module connecting it to a ViT), and (c) a visual question answering system (via yet another GATS module), all without loading multiple copies of the language model into memory. The paper shows this works for two tasks (image generation and captioning) using the same frozen models; extending this to 5-10 simultaneous tasks with independent GATS modules would validate the composable deployment model. The memory savings are potentially substantial: instead of loading 5 fine-tuned copies of a 1.3B model (requiring ~6.5B parameters in GPU memory), a single frozen instance plus 5 GATS modules (potentially ~100-500M parameters each) would require substantially less total memory.

4. Self-improving multimodal systems where the connector module is updated without touching foundation model weights. An emerging paradigm in AI is iterative self-improvement: use a deployed model to collect data, train an improved version on that data, and redeploy. With conventional fine-tuning, each iteration produces a new copy of the full model, making the improvement cycle expensive and risking degradation of capabilities not represented in the self-collected data. With GATS, only the connector module needs updating — the foundation models remain frozen, preserving their full breadth of pretrained knowledge. The paper's results with the bimodal model (Section 5.1) show that a new GATS module can be trained to high performance (converging to MaskGIT loss of 5.45 after 50k steps, Figure 11) while the frozen models remain untouched. In a production setting, this means continuous improvement loops could run with ~10% of the parameter updates (988M GATS parameters out of 9.3B total for the bimodal model) compared to full-model fine-tuning, with zero risk of degrading the foundation models' general capabilities through distributional drift in the self-collected data.