Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Models

ArXiv: 2411.04996

🎯 Pitch

A multi-modal transformer can match the image quality of a dense baseline using just 55.8% of the FLOPs—and speech quality with 37.2%—simply by giving each modality its own feed-forward networks, attention projections, and layer norms. A 760M-parameter MoT model even outperforms a 1.4B-parameter dense model on image generation and captioning metrics. The key insight is that modality-specific processing catches all the efficiency gains while global self-attention preserves cross-modal learning.

---

1. Executive Summary

This paper introduces Mixture-of-Transformers (MoT), a sparse multi-modal transformer architecture that decouples all non-embedding parameters—feed-forward networks, attention projection matrices, and layer normalization—by modality while retaining global self-attention across the full interleaved input sequence (e.g., text tokens use a dedicated FFN and attention projections, image tokens use separate ones, but all tokens attend to each other). Evaluated across three multi-modal pretraining settings on models up to 7B parameters—autoregressive text-and-image generation (Chameleon), three-modality generation adding speech (Chameleon+Speech), and multi-objective training with diffusion-based images (Transfusion)—MoT matches the dense baseline's image performance using only 55.8% of the FLOPs in the Chameleon 7B setting and reaches comparable speech performance with only 37.2% of the FLOPs, while a 760M MoT model outperforms a 1.4B dense baseline across CLIP score, FID score, and CIDEr score for image generation and captioning. The paper establishes that modality-aware, rule-based parameter partitioning outperforms learned routing (Mixture-of-Experts) and yields compounding efficiency gains as GPU count scales, particularly for non-text modalities where the dense baseline's uniform processing is most wasteful.

2. Context and Motivation

The Core Problem: Multi-Modal Foundation Models Are Computationally Prohibitively Expensive

The fundamental challenge this paper addresses is straightforward to state but difficult to solve: training a single foundation model to jointly process text, images, and speech requires dramatically more compute than training a text-only language model of comparable capability, and this compute is spent inefficiently because different modalities have fundamentally different processing requirements that a uniform dense transformer cannot exploit.

To understand the scale of this problem, consider the numbers the paper provides: Chameleon (Chameleon Team, 2024)—an early-fusion model that tokenizes both text and images into discrete tokens and applies a single autoregressive objective across the combined sequence—required 9.2 trillion training tokens (including image tokens) to match the text performance of LLaMA2, which was trained on only 2 trillion text tokens. That is a more than 4× increase in training data, and the associated compute cost scales accordingly. The underlying reason, which the paper demonstrates empirically through PCA analysis of internal representations (Figure 2b-e, Appendix Figure 23), is that different modalities occupy distinct, well-separated regions of the feature space in a dense transformer—text tokens cluster together, image tokens cluster together, speech tokens cluster together—despite the architecture processing all inputs as uniform discrete tokens with no modality-specific priors. This clustering suggests that a single set of parameters is being forced to serve as a compromise representation for fundamentally different statistical distributions, which is inherently inefficient.

The problem is not merely academic. Multi-modal models capable of both understanding and generating across text, images, and speech are central to the next generation of AI applications—content creation, cross-modal translation, conversational agents that can see and hear, assistive technologies—but the computational barrier to training them means that only organizations with massive compute budgets can participate. Making multi-modal pretraining more efficient is therefore both a practical concern (reducing the cost and environmental impact of training) and a democratization concern (enabling smaller research groups to contribute to this frontier).

Why Prior Solutions Fall Short

The paper identifies three categories of prior approaches, each with significant limitations that prevent them from fully addressing the efficiency challenge.

Late-fusion approaches are computationally light but capability-limited. Early multi-modal LLMs (Alayrac et al., 2022; Liu et al., 2023; Laurençon et al., 2023) used separate encoders for different modalities—for example, a vision encoder processing images and a language model processing text, with cross-attention or learned adapter layers connecting them. These models are efficient because each modality gets its own specialized encoder, but they are fundamentally understanding-only: they cannot generate images, only text conditioned on images. The paper's goal is multi-modal generation, not just understanding, so late-fusion approaches are excluded from the solution space by design.

Learned sparsity (Mixture of Experts) introduces training instability and routing challenges. MoE architectures (Shazeer et al., 2017; Fedus et al., 2022; Jiang et al., 2024) offer a general-purpose sparsity mechanism: each transformer layer contains multiple expert feed-forward networks, and a learned router decides which expert(s) to activate for each input token. This allows the model to scale parameter count without proportionally scaling FLOPs, since only a subset of experts is active per token.

However, MoE introduces several well-documented pathologies that the paper explicitly enumerates as motivation for developing an alternative:

1. Load imbalance: The learned router can collapse to always selecting a small subset of experts, leaving others unused. This requires auxiliary load-balancing losses that complicate the training objective and may trade off model quality for balance. In multi-modal settings, this problem is amplified because modalities have different frequencies in the training data—speech tokens, for instance, appear less frequently than text tokens, potentially starving some experts.

2. Bi-level optimization instability: The router and the experts are co-trained from scratch, meaning both are simultaneously under-trained in the early stages. The router makes poor assignment decisions, and the experts receive unstable gradients from those assignments, creating a feedback loop that can cause training divergence, particularly at large scales.

3. Inference-mode mismatch: The paper uses Expert Choice (EC) routing (Zhou et al., 2022) for its MoE baselines, which has each expert select its top-k input tokens (rather than each token choosing its top-k experts). This ensures perfect load balance during training but violates causality during autoregressive generation—experts can select future tokens that shouldn't be visible yet. The paper acknowledges this as a confounding factor: MoE-4x's validation performance may be overestimated because the router has access to future tokens (information leakage), or underestimated because EC routing behaves differently on out-of-distribution validation data. Either way, the routing mechanism itself is a source of brittleness.

4. Diminishing returns in multi-modal settings: The paper's own experiments show that MoE-4x exhibits "diminishing returns as model size increased. While it showed some speedup in image modality at smaller scales, this advantage diminished at the 7B scale" (Section 3.2.3). At 7B in the Chameleon setting, MoE-4x actually loses its advantage over the dense model for the image modality (Figure 6q-r). This suggests that the learned routing approach does not scale well for multi-modal data.

Modality-aware sparsity exists but is applied too narrowly. Prior work has recognized that different modalities benefit from different parameters. VLMO (Bao et al., 2022b), MoMA (Lin et al., 2024), and related approaches apply modality-aware sparsity only to the feed-forward network (FFN) layers of the transformer, while keeping attention projection matrices and layer normalization shared across modalities. CogVLM (Wang et al., 2023) uses a separate visual expert but is limited to generating text outputs only (not images). Playground v3 (Liu et al., 2024b) combines a frozen text LLM with a trainable image transformer but is built on a pre-trained LLM rather than being trainable from scratch.

These approaches leave significant efficiency on the table because they don't address three critical components:

• Attention projection matrices (W_Q, W_K, W_V, W_O): These determine how tokens are mapped into query, key, value, and output spaces for self-attention. If different modalities have different feature distributions (as Figure 2 demonstrates), then using the same projection matrices for all modalities forces a compromise representation that is suboptimal for each individually.

• Layer normalization: While seemingly minor, LayerNorm parameters encode modality-specific scaling and shifting statistics. The paper's ablation (Section 3.5, Figure 14) shows that untying LayerNorms provides negligible additional benefit beyond untying FFN and attention, but this finding is only established empirically—prior work never explored it systematically.

• Training from scratch on all modalities: Both CogVLM and Playground v3 build on pre-trained text LLMs, which means the text parameters are already converged before multi-modal training begins. This is a fundamentally different regime from training all modalities jointly from scratch, where the optimization dynamics are more complex and the potential for modality conflict is higher.

The Central Insight: Modality-Specific Processing Should Permeate the Entire Architecture

The paper's motivating hypothesis, which it supports with empirical evidence of modality clustering (Figure 2, Appendix Figure 23), is that a dense transformer processing interleaved multi-modal tokens is forced to use the same parameters to model fundamentally different data distributions, and this inefficiency is the root cause of the inflated training costs. The natural remedy is to give each modality its own processing parameters while retaining a shared mechanism for cross-modal interaction.

This is not an arbitrary choice—it builds on a specific empirical observation that the paper documents in detail. Figure 23 shows PCA visualizations of the internal activations of a Chameleon+Speech 7B dense model at four different training checkpoints (4%, 24%, 50%, 100%) and four different layers (1, 5, 17, 32). At every checkpoint and every layer, text, speech, and image tokens form distinct, well-separated clusters. This separation emerges early in training (visible even at 4%) and persists throughout. The paper's interpretation is that these are "inherent differences in modality processing" that a uniform architecture cannot efficiently capture.

Furthermore, the paper notes that "these modalities often exhibit conflicting training dynamics in a dense transformer model (Figure 15), complicating optimization and increasing computational load." The leave-one-out analysis (Section 4) provides direct evidence for this: when two modalities are forced to share a transformer tower (e.g., text and speech combined in one tower while image gets its own), both modalities suffer compared to the fully separated MoT configuration. This suggests that modalities compete for parameter capacity when sharing a tower, and separating them eliminates this conflict.

How MoT Positions Itself Relative to Existing Work

The paper positions MoT as a generalization and unification of two existing ideas: modality-aware sparsity (from VLMO, MoMA, CogVLM) and rule-based routing (from modality-specific expert assignment). Specifically, MoT differs from prior modality-aware sparse models in four key dimensions that the paper emphasizes:

1. Scope of decoupling: MoT applies modality-specific parameters to all non-embedding components—FFN, attention projections (W_Q, W_K, W_V, W_O), and layer normalization—rather than only the FFN. The ablation in Section 3.5 quantifies the benefit of this broader decoupling: starting from a dense model, untying only the FFN provides substantial gains (especially for images), and further untying the attention Q, K, V matrices yields an additional ~33% FLOPs saving for the image modality and ~10% for text on the Obelisc held-out set.

2. Training from scratch: Unlike CogVLM (which builds on a pre-trained LLM) and Playground v3 (which freezes the text backbone), MoT is designed to be trained entirely from scratch with all modalities jointly. This is a harder setting but demonstrates that the architectural benefit is fundamental rather than an artifact of pre-training.

3. Deterministic, rule-based routing: MoT's modality assignment is based on the known modality of each token—no learned router, no load-balancing loss, no inference-mode mismatch. This directly addresses the instability and routing pathologies that MoE architectures suffer from. The paper provides explicit evidence that this matters: MoE-4x consistently underperforms MoT in non-text modalities (image, speech), and at the 7B scale in the Chameleon setting, MoE-4x actually shows no advantage over the dense baseline for images while MoT requires only 34.8% of the training steps.

4. Complementarity with MoE: MoT and MoE are not mutually exclusive. The paper demonstrates this through the hybrid "MoT + Text MoE-4x" configuration (Section 5), where the text tower of MoT uses MoE-4x layers while the image tower remains standard MoT. This hybrid outperforms both pure MoT and pure MoE-4x on text while preserving MoT's image advantages. This positions MoT as a framework within which other sparsity mechanisms (including learned routing) can be selectively deployed.

The paper also explicitly connects to the Transfusion framework (Zhou et al., 2024), where text uses autoregressive objectives and images use diffusion-based objectives. In this setting, a key additional motivation emerges: the dense Transfusion model already gains efficiency from separating training objectives by modality. MoT further separates parameters by modality, and the paper hypothesizes (and shows) that the combination yields compounding benefits—particularly for the image diffusion task, which is computationally heavy and benefits disproportionately from having dedicated parameters.

The Practical Stakes: Why Efficiency Matters Beyond Academic Interest

The paper grounds its motivation in practical deployment considerations. Training a 7B Chameleon model from scratch requires hundreds of GPUs for weeks—the paper's own experiments use 384 A100 GPUs for the 7B configuration. A 44.2% reduction in required training FLOPs (the Chameleon 7B result) translates to hundreds of thousands of GPU-hours saved, which directly reduces cost and carbon footprint. The wall-clock time results (Section 6.2.2, Figure 19) show that MoT matches the dense model's image quality in 47.2% of the wall-clock training time—meaning a research team could run nearly twice as many experiments in the same time budget.

Moreover, the paper demonstrates through its horizontal scaling analysis (Section 6.2.1, Figure 18) that MoT's efficiency gains increase with GPU count—as you scale from 16 to 256 GPUs (which is the direction that large-scale training is moving), the relative advantage of MoT grows. For image validation loss, the percentage of training steps MoT needs to match the dense model drops from 42.1% (16 GPUs) to 21.6% (256 GPUs). This suggests that MoT's architecture is particularly well-suited to the large-scale distributed training regimes that are becoming standard for foundation model development, making the contribution timely and practically significant.

3. Technical Approach

This is an architecture design paper whose core idea is that decoupling all non-embedding transformer parameters by modality—using deterministic, rule-based routing instead of learned routing—yields a sparse architecture that trains multi-modal foundation models substantially faster than dense baselines, without the training instability and inference-mode mismatch problems that plague Mixture-of-Experts approaches.

3.1 Reader Orientation

MoT is a drop-in replacement architecture for a standard dense transformer that processes interleaved sequences of tokens from different modalities (text, images, speech). You can think of it as splitting a single monolithic transformer into separate modality-specific sub-transformers that share only the embedding layer and the global self-attention mechanism, while keeping everything else—feed-forward networks, attention projection matrices, and layer normalization—dedicated to each modality.

The problem it solves is that dense transformers waste computation by forcing the same parameters to handle text tokens (discrete, highly compressible, low-dimensional semantics), image tokens (spatial, high-frequency, large sequences), and speech tokens (temporal, redundant, different time scales) uniformly. MoT addresses this by giving each modality its own processing "tower" within a single model, activated deterministically based on token modality, so that the computational cost per token stays identical to the dense baseline but learning is more efficient because parameters don't compete across modalities. The result is that MoT reaches the same performance as a dense model in ~45-56% of the training FLOPs for two-modality settings and as little as ~23% of the training steps for speech when added as a third modality.

3.2 Big-Picture Architecture (Diagram in Words)

The MoT architecture has four major components:

1. Shared Embedding Layer — Converts discrete text tokens (via a text vocabulary embedding), discrete image tokens (via a pre-trained VQ-VAE tokenizer embedding), and discrete speech tokens (via a pre-trained speech tokenizer embedding) into a common $d$-dimensional vector space. This is the only truly shared component — all modalities start from the same vector representation space.

2. Modality Indexing and Token Grouping Logic — At each transformer layer, before any computation happens, tokens are grouped by their modality (text, image, speech) into separate batches. This is a pure indexing operation — no learned routing, no auxiliary losses, just a deterministic mask based on which token belongs to which modality. The grouping ensures that modality-specific parameters are applied only to the correct tokens.

3. Per-Modality Transformer Towers — For each modality $m \in \{\text{text}, \text{image}, \text{speech}\}$, MoT maintains separate instances of (a) attention query/key/value/output projection matrices ($W^m_Q, W^m_K, W^m_V, W^m_O$), (b) feed-forward network parameters ($\text{FFN}^m$, the standard SwiGLU MLP), and (c) layer normalization parameters ($\text{LayerNorm}^m_{\text{attn}}, \text{LayerNorm}^m_{\text{ffn}}$). These parameters are applied independently to the tokens of each modality, so text tokens go through the text-specific QKV projections, text-specific FFN, etc., while image tokens go through image-specific ones.

4. Global Self-Attention Mechanism — After modality-specific projections produce queries, keys, and values for all tokens, these are reassembled into their original interleaved sequence order, and a single joint attention operation is performed across the entire sequence regardless of modality. This is the mechanism that enables cross-modal interactions: an image token can attend to a text token, a speech token can attend to an image token, etc. The attention computation itself is standard scaled dot-product attention, identical to a dense transformer.

Information flows through a single MoT layer as follows: (1) The input sequence of interleaved multi-modal tokens enters the layer. (2) The modality indexing logic identifies which tokens belong to which modality and logically groups them. (3) Each group undergoes modality-specific QKV projections and FFN processing. (4) The processed token representations are reassembled into the original sequence order. (5) Global self-attention is computed across the full sequence, using the modality-specific projections. (6) Residual connections and modality-specific layer normalization are applied. (7) The output proceeds to the next MoT layer, where the process repeats.

The critical design property is that every step adds exactly the same number of FLOPs per token as the dense baseline. There is no extra computation from routing, no overhead from a learned gating network, no top-k selection — just a deterministic dispatch based on token modality. The sparsity comes from the fact that parameters for one modality don't participate in processing tokens of another modality, but because each token still activates exactly one set of parameters (its modality's), the total computation is identical.

3.3 Roadmap for the Deep Dive

• First, the core mathematical formulation — I'll walk through the formal specification of a MoT transformer layer (Equations 2 and 3, Algorithm 1), showing exactly how the decoupled parameters interact with the global self-attention mechanism and how this differs from a standard dense layer. This is the architectural heart of the paper.

• Second, the training and evaluation configurations — I'll detail the hyperparameters used across all three experimental settings (Chameleon, Chameleon+Speech, Transfusion), including model scales (37M to 7B), hidden dimensions, layer counts, batch sizes, GPU counts, sequence lengths, and total training tokens. Understanding these is essential for interpreting the results because the paper's claims are anchored to specific compute budgets.

• Third, the Mixture-of-Experts baseline implementation — The paper compares against MoE not as an afterthought but as a central point of contrast. I'll explain the Expert Choice routing used for MoE baselines, its causal violation issue, and how this creates both favorable and unfavorable biases in the MoE results relative to MoT.

• Fourth, the design choices and their justifications — This section unifies the architectural decisions: why decouple all non-embedding parameters rather than just FFNs? Why use global self-attention rather than cross-attention? Why deterministic routing rather than learned routing? What does the ablation study (Section 3.5) reveal about the relative importance of untying FFN, attention, and LayerNorm?

• Fifth, the hybrid MoT+MoE architecture — I'll explain the proof-of-concept combination where the text tower of MoT uses MoE-4x FFN layers, while the image tower remains standard MoT, and why this demonstrates complementarity rather than competition between the two sparsity paradigms.

3.4 Detailed, Sentence-Based Technical Breakdown

Core Mathematical Formulation: The MoT Transformer Layer

A standard dense transformer layer, as defined in Equation 1 of the paper, processes an input sequence $x = (x_1, \ldots, x_n)$ uniformly:

$a = \text{Attn}(x, \theta_{\text{attn}})$ $h = x + \text{LayerNorm}_{\text{attn}}(a)$ $\text{output} = h + \text{LayerNorm}_{\text{ffn}}(\text{FFN}(h, \theta_{\text{ffn}}))$

Here, $\theta_{\text{attn}}$ represents the shared attention parameters (Q, K, V, O projection matrices), $\theta_{\text{ffn}}$ represents the shared feed-forward network parameters, and LayerNorm parameters are also shared across all tokens regardless of modality. Every token — whether it represents a word, an image patch, or a speech frame — undergoes exactly the same transformation with exactly the same parameters.

MoT modifies this by introducing modality-specificity at every non-embedding parameter level, as expressed in Equation 2:

$a = \text{GlobalAttn}(x, \{\theta^m_{\text{attn}}\}_{m \in \{\text{text}, \text{image}, \text{speech}\}})$ $h_i = x_i + \text{LayerNorm}^{m_i}_{\text{attn}}(a_i)$ $\text{output}_i = h_i + \text{LayerNorm}^{m_i}_{\text{ffn}}(\text{FFN}(h_i, \theta^{m_i}_{\text{ffn}}))$

where $m_i \in \{\text{text}, \text{image}, \text{speech}\}$ is the modality of token $i$, $\theta^m_{\text{attn}}$ is the set of attention parameters {$W^m_Q, W^m_K, W^m_V, W^m_O$} specific to modality $m$, $\theta^{m_i}_{\text{ffn}}$ is the FFN parameters for that modality, and $\text{LayerNorm}^{m_i}_{\text{attn}}$ and $\text{LayerNorm}^{m_i}_{\text{ffn}}$ are the modality-specific layer normalization parameters.

What this equation computes: For each token $x_i$ in the input sequence, the layer applies the attention projection matrices associated with that token's modality ($m_i$) to compute its query, key, and value vectors, then performs global attention over the full sequence, then applies the modality-specific output projection, residual connection, and layer normalization, then applies the modality-specific FFN with another residual connection and layer normalization. The result is a transformed representation $output_i$ that has been processed by parameters specialized for its modality while having had the opportunity to attend to tokens of all modalities.

Why this form: The alternative would be a dense model (Equation 1) where all parameters are shared, forcing text tokens and image tokens to use the same QKV projections despite having fundamentally different statistical properties (as demonstrated by the PCA analysis in Figure 2 showing modality clustering in feature space). Another alternative would be modality-specific cross-attention (common in late-fusion models), but that would separate modalities into different processing streams that interact only through bottleneck attention, limiting cross-modal interaction depth. MoT's form preserves the full connectivity of global self-attention while giving each modality dedicated processing parameters, so cross-modal interactions happen at every layer but don't force parameters to compromise between modalities. The residual connections remain standard (additive, identity-mapped), meaning the only change from a dense transformer is the substitution of modality-conditional parameters for shared parameters.

The Global Self-Attention Mechanism with Modality-Specific Projections

The $\text{GlobalAttn}$ function in Equation 2 is defined formally in Equation 3:

$\text{GlobalAttn}(x, \{\theta^m_{\text{attn}}\}) = \left(\text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right) V\right) W^{m_i}_O$

where the modality-specific projections are applied per-token:

$Q_i = x_i W^{m_i}_Q, \quad K_i = x_i W^{m_i}_K, \quad V_i = x_i W^{m_i}_V$

Here, $d_k$ is the dimensionality of the key vectors (equal to the hidden dimension divided by the number of attention heads), $W^{m_i}_Q, W^{m_i}_K, W^{m_i}_V \in \mathbb{R}^{d \times d_k}$ are the modality-specific query, key, and value projection matrices (per attention head), and $W^{m_i}_O \in \mathbb{R}^{d_k \times d}$ is the modality-specific output projection matrix. The full Q, K, V matrices are formed by concatenating the per-token projections across the entire sequence:

$Q = [Q_1; Q_2; \ldots; Q_n], \quad K = [K_1; K_2; \ldots; K_n], \quad V = [V_1; V_2; \ldots; V_n]$

What this computes: Each token in the sequence produces its query, key, and value vectors using the projection matrices belonging to its modality. A text token uses $W^{\text{text}}_Q$ to produce its query, while an image token uses $W^{\text{image}}_Q$. These differently-projected vectors are then concatenated into the full Q, K, V matrices, and standard dot-product attention is computed across the entire sequence. The attention output for position $i$ is then multiplied by the output projection matrix $W^{m_i}_O$ corresponding to that token's modality.

Why this form: This design preserves two critical properties. First, the attention computation $\frac{QK^T}{\sqrt{d_k}}$ operates in a shared vector space — the dot products between tokens of different modalities are computed even though those tokens were projected by different matrices. This means that a text token's query vector (produced by $W^{\text{text}}_Q$) can be compared against an image token's key vector (produced by $W^{\text{image}}_K$), enabling cross-modal attention. The fact that the projections are modality-specific allows them to learn how to best represent each modality's tokens for attention interaction — for example, learning that image tokens should project their keys into a space where text queries about visual concepts produce high similarity scores. Second, the output projection $W^{m_i}_O$ is also modality-specific, meaning the attention output is transformed back into a representation optimized for that modality's subsequent processing (FFN, next layer). This end-to-end modality-specificity means the Q, K, V, and O projections can all specialize.

The paper explicitly contrasts this with cross-attention architectures (Alayrac et al., 2022; Aiello et al., 2023), noting that global self-attention "normalizes attention weights across tokens of different modalities while reducing the number of layers in the architecture." In a cross-attention design, text and image would be processed by separate tower layers that interact only at designated cross-attention points, effectively halving the depth of information exchange. In MoT, every layer is a cross-modal interaction point, which is computationally equivalent to the dense baseline but benefits from modality-specific projections.

Algorithmic Implementation: Step-by-Step MoT Layer Computation

Algorithm 1 in the paper provides the computational walkthrough of a single MoT layer, which translates the mathematical formulation into implementable operations. I'll walk through it in detail because it reveals the practical mechanics that the equations abstract.

Lines 3-5: Token Grouping by Modality. For each modality $m \in \{\text{text}, \text{image}, \text{speech}\}$, the algorithm identifies the set of indices $I_m = \{i : m_i = m\}$ (the positions in the sequence where tokens of that modality appear) and extracts the corresponding token representations $X_m = \{x_i : i \in I_m\}$. This is a pure indexing operation — no computation, no routing probabilities, no auxiliary losses. The modality of each token is known from the data preprocessing (it's a property of the input, not something the model needs to infer).

Line 6: Modality-Specific Attention Projections. For each modality $m$, the grouped tokens $X_m$ are projected using that modality's query, key, and value matrices:

$Q_m = W^m_Q X_m, \quad K_m = W^m_K X_m, \quad V_m = W^m_V X_m$

Since $X_m$ is a matrix of shape $|I_m| \times d$ (number of tokens of modality $m$ by hidden dimension), and $W^m_Q, W^m_K, W^m_V$ are each $d \times d_k$ (per-head), this operation is a standard matrix multiplication — identical to what happens in a dense transformer, but applied only to the subset of tokens of each modality.

Lines 8-9: Sequence Reassembly and Global Attention. The per-modality queries, keys, and values are reassembled into the original sequence order:

$Q = \bigcup_{m \in M} Q_m, \quad K = \bigcup_{m \in M} K_m, \quad V = \bigcup_{m \in M} V_m$

Then global self-attention is computed:

$A = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right) V$

This is exactly the standard scaled dot-product attention — every token can attend to every other token, regardless of modality. The cost of this attention is $O(n^2 d_k)$, exactly the same as the dense baseline for the same sequence length $n$.

Lines 11-15: Modality-Specific Output Projection and FFN. For each modality $m$, the attention outputs for tokens of that modality are extracted ($A_{I_m}$), transformed by the modality-specific output projection $W^m_O$, then passed through residual connection and modality-specific layer normalization:

$O_m = W^m_O A_{I_m}$ $H_m = X_m + \text{LayerNorm}^m_{\text{attn}}(O_m)$

Then the modality-specific FFN is applied:

$F_m = \text{FFN}_m(H_m)$ $Y_m = H_m + \text{LayerNorm}^m_{\text{ffn}}(F_m)$

Lines 16: Return. The outputs for all modalities are returned, ready to be reassembled into the interleaved sequence for the next layer.

What this algorithm computes, operationally: For each MoT layer, the computation first separates tokens by modality, processes each group through its dedicated projection matrices, recombines them for full cross-modal attention, then separates them again for dedicated output processing and FFN transformation. This "group-process-recombine" cycle happens at every layer.

Why this particular algorithmic structure: The grouping (lines 3-5) and reassembly (lines 8-9) operations are what enable the architectural innovation — by physically separating tokens by modality for their dedicated computations, the model can use entirely different parameter sets without any learned routing mechanism. The grouping/reassembly is the implementation of the "deterministic, rule-based routing" that the paper contrasts with MoE's learned routing. The computational overhead of grouping and reassembly is minimal (pure indexing, no floating-point operations), which is why MoT can match the dense baseline's FLOPs exactly.

Training and Evaluation Configurations Across Experimental Settings

The paper evaluates MoT across three distinct multi-modal pretraining settings, each with its own data, objectives, hyperparameters, and model scales. I'll detail each separately because the configurations differ substantially, and understanding them is essential for interpreting the results.

Chameleon Setting (Section 3.2.1): Autoregressive Text and Image Generation. This setting follows the Chameleon (Chameleon Team, 2024) architecture: both text and images are tokenized into discrete tokens and trained with a standard autoregressive next-token prediction objective. Images are tokenized using a pre-trained VQ-VAE (Gafni et al., 2022) that converts each image into 1,024 discrete tokens. The training data comprises "roughly equal amount of text and image tokens" from Chameleon's mixed-modal corpus. Validation is performed on held-out sets from four datasets: Obelisc (Laurençon et al., 2023), MS-COCO (Lin et al., 2014), Flickr30k (Plummer et al., 2015), and Shutterstock. For MS-COCO and Flickr30k, the paper specifically uses the Karpathy test splits and reports text-to-image and image-to-text conditional perplexity.

Model architectures span five scales, as specified in Table 1. At 37M parameters: hidden dimension 256, 4 layers, 8 attention heads. At 94M: hidden dimension 512, 8 layers, 8 heads. At 443M: hidden dimension 1024, 24 layers, 16 heads. At 1.5B: hidden dimension 2048, 24 layers, 16 heads. At 7B: hidden dimension 4096, 32 layers, 32 heads. All models use a sequence length of 4096 tokens. Training steps are 160,000 for the three smaller scales (37M, 94M, 443M) and 120,000 for the two larger scales (1.5B, 7B). Batch sizes and GPU counts scale accordingly: 32 GPUs with batch size 12 per GPU for 37M (1.57M tokens per batch), up to 384 GPUs with batch size 2 per GPU for 7B (3.15M tokens per batch). Total training tokens range from 0.168 trillion (94M model at 160k steps × 1.05M tokens/batch) to 0.377 trillion (7B model at 120k steps × 3.15M tokens/batch). Most configurations process approximately 0.252 trillion tokens.

Chameleon+Speech Setting (Section 3.3.1): Adding Speech as Third Modality. This extends the previous setting by incorporating discrete speech tokens as a third modality, trained autoregressively alongside text and images. Speech is tokenized using "an in-house tokenizer, a variant of DinoSR (Liu et al., 2024a), which extracts semantic tokens with a vocabulary size of 500. Each token represents 40ms of audio content (25Hz)." Table 2 details the speech training data, which combines four speech-only datasets (People's Speech: 16,404 hours, 1.2B tokens; Voxpopuli English: 23,166 hours, 1.6B tokens; LibriLight: 55,308 hours, 4B tokens; Multilingual LibriSpeech English: 44,585 hours, 3.2B speech tokens + 0.5B text tokens) and one speech+text dataset (Spotify: 57,290 hours, 4.2B speech tokens + 0.7B text tokens). The three-modality training dataset combines this speech data with the Chameleon text-and-image dataset at a sampling ratio of 1:6 (speech:Chameleon).

Table 3 specifies the architectures for this setting: four scales (443M, 880M, 1.5B, 7B) with hidden dimensions from 1024 to 4096, layers from 24 to 32, and heads from 16 to 32. The 880M configuration (hidden dimension 1536, 24 layers, 24 heads) is specific to this setting and used in the ablation experiments in Section 3.5. Training steps and token counts match the Chameleon setting (160k steps for 443M, 120k for 880M and above). Speech validation uses the LibriLight (LL60K) and People's Speech (PPL30K) held-out sets.

Transfusion Setting (Section 3.4.1): Autoregressive Text + Diffusion Images. This setting follows Transfusion (Zhou et al., 2024): text uses standard autoregressive language modeling loss, while images use diffusion-based training objectives. Images are represented as continuous latent patches using a Variational Autoencoder (VAE) (Kingma & Welling, 2022) that performs 8×8 spatial downsampling, with each image represented as 256 continuous tokens (as opposed to 1,024 discrete tokens in Chameleon). Text data comes from the Llama 2 corpus (2 trillion tokens), and images from 380 million licensed Shutterstock images with captions, center-cropped and resized to 256×256 pixels. Multimodal examples are formatted with special beginning-of-image (BOI) and end-of-image (EOI) tokens enclosing the image sequence. In most experiments, 0.5 trillion tokens (or patches) are sampled from the two modalities at a 1:1 ratio.

The transformer architecture in Transfusion differs from Chameleon in a key aspect: rather than using standard causal attention throughout, it uses a hybrid attention mechanism. Causal attention is applied across the full sequence to preserve the autoregressive property (each token can only attend to past tokens), but bidirectional attention is used within each image, meaning image patches can attend to all other patches within the same image while only attending to preceding tokens or image patches outside their own image. This is a design inherited from Transfusion, not a MoT-specific innovation.

The combined loss function (Equation 4) is:

$L_{\text{Transfusion}} = L_{\text{LM}} + \lambda \cdot L_{\text{DDPM}}$

where $\lambda$ is a balancing coefficient set to 5 following preliminary experiments. $L_{\text{LM}}$ is the standard autoregressive cross-entropy loss on text tokens, and $L_{\text{DDPM}}$ is the diffusion denoising loss:

$L_{\text{DDPM}} = \mathbb{E}_{x_0, \epsilon, t}\left[\|\epsilon - \epsilon_\theta(x_t, t, c)\|^2\right]$

Here, $x_0$ is the clean image latent, $\epsilon \sim \mathcal{N}(0, I)$ is Gaussian noise, $t$ is the diffusion timestep, $x_t = \sqrt{\bar{\alpha}_t} x_0 + \sqrt{1 - \bar{\alpha}_t} \epsilon$ is the noised latent at timestep $t$, $\bar{\alpha}_t = \prod_{s=1}^t \alpha_s$ is the cumulative noise schedule (cosine scheduler from Nichol & Dhariwal, 2021), $c$ is the conditioning context (text), and $\epsilon_\theta$ is the neural network predicting the added noise. During inference, 250 diffusion steps are used for sampling.

What this loss computes: The language modeling loss $L_{\text{LM}}$ encourages the model to predict the next token correctly (standard autoregressive training). The diffusion loss $L_{\text{DDPM}}$ encourages the model to predict the noise that was added to the image latent, which is equivalent to learning to reverse the diffusion process — given a noised image latent, predict the noise so it can be removed. The $\lambda$ coefficient balances these two objectives so neither dominates training.

Why this form: The Transfusion design separates not just the parameters (as MoT does) but also the training objectives by modality. The paper's hypothesis is that MoT's parameter decoupling compounds with Transfusion's objective decoupling to yield even greater efficiency gains — and indeed, the results (Section 3.4.2) show particularly strong acceleration for the image diffusion task, where the diffusion objective is computationally heavy and benefits disproportionately from having dedicated image parameters.

Table 4 specifies Transfusion architectures: four scales (163M, 760M, 1.4B, 7B). Hidden dimensions: 768 (163M), 1536 (760M), 2048 (1.4B), 4096 (7B). Layers: 16 (163M), 24 (760M, 1.4B), 32 (7B). Attention heads: 12 (163M), 24 (760M), 16 (1.4B), 32 (7B). All models use sequence length 4096, batch size 2 per GPU with 2.10M tokens per batch (except 163M and 760M which use 4 per GPU, 128 GPUs, also 2.10M tokens/batch). Training runs for 250,000 steps, processing 0.524 trillion tokens. U-Net patch encoding parameters add a fixed 0.27B parameters across all configurations. Optimization uses AdamW ($\beta_1 = 0.9, \beta_2 = 0.95, \epsilon = 1\times 10^{-8}$), learning rate $3\times 10^{-4}$ with 4000 warmup steps and cosine decay to $1.5\times 10^{-5}$, weight decay 0.1, and gradient clipping norm 1.0.

Evaluation in Transfusion uses three types of metrics: (1) text-to-text perplexity on 20M held-out tokens from Wikipedia and C4 (Raffel et al., 2020); (2) text-to-image metrics including diffusion validation loss on held-out Conceptual 12M (CC12M; Changpinyo et al., 2021), zero-shot FID (Heusel et al., 2017) on 30,000 randomly selected MS-COCO validation prompts, and CLIP score (Radford et al., 2021) for text-image alignment; (3) image captioning via CIDEr score (Vedantam et al., 2015) on the Karpathy test split of MS-COCO. Unless otherwise noted, classifier-free guidance (Ho & Salimans, 2022) is not used for evaluation to simplify comparison — guidance requires per-model hyperparameter tuning. For 7B MoT and dense models, classifier-free guidance of 5 is used when generating example images, and at guidance level 1.6, the 7B MoT achieves COCO-30k FID of 8.14 (compared to 9.22 for a dense model trained on 1T tokens with richer data).

Mixture-of-Experts Baseline Implementation

The paper uses Mixture-of-Experts (MoE) as a primary comparative baseline, specifically a 4-expert configuration (MoE-4x), with additional experiments using 2-expert (MoE-2x) and 3-expert (MoE-3x) variants at smaller scales. The implementation choice is significant and requires careful explanation because it introduces known biases that affect how the results should be interpreted.

Expert Choice (EC) routing (Zhou et al., 2022). The paper uses Expert Choice routing rather than the more common Token Choice routing. In Token Choice routing, each token selects its top-k experts based on routing weights. In Expert Choice, each expert selects its top-k tokens from the input. Specifically, for a layer with $E$ experts and capacity factor $C$, each expert independently computes routing scores for all tokens, selects the top $k = C \times n / E$ tokens (where $n$ is sequence length), and processes only those tokens. This guarantees perfect load balance by construction — every expert processes exactly the same number of tokens — which eliminates the need for auxiliary load-balancing losses that complicate MoE training.

What this computes, operationally: For each MoE layer, the router computes a score for each (token, expert) pair. Each expert $e$ then selects the $k$ tokens with the highest scores for that expert and processes them through its FFN. Tokens not selected by any expert pass through a residual connection unchanged (or are handled by a small shared FFN in some implementations). The output for each token is the sum (or weighted sum) of the outputs from the experts that selected it.

The causal violation problem at inference. Expert Choice routing, by having each expert select tokens based on the full sequence (including future tokens), violates the causal dependency required for autoregressive generation. During training, this is not a problem because the entire sequence is available — the model can look at all tokens when deciding which expert processes which position. But during autoregressive inference, the model only has access to past tokens; it cannot know which expert should process a future token before that token is even generated. The paper acknowledges this explicitly: "However, EC cannot be directly applied to auto-regressive generation, as it violates the causal dependency between tokens in a sequence, where each token is generated based solely on the previous ones."

How the paper handles this. For evaluation, the paper uses Expert Choice routing exactly as in training, evaluating only on validation perplexity (not generation quality for MoE models). This creates two confounding factors that the paper transparently acknowledges:

1. Possible overestimation of MoE-4x performance: Because Expert Choice during evaluation can access future tokens (information leakage), the validation loss may be artificially lowered relative to a causally valid inference setting. The router can use information about upcoming tokens to make better expert assignments.

2. Possible underestimation on out-of-distribution data: Expert Choice routing is sensitive to data distribution — if the validation data (e.g., speech datasets LL60K and PPL30K) differs from the training distribution, the expert assignment based on training statistics may be suboptimal, leading to worse-than-expected validation performance.

Why this matters for interpreting the MoT vs. MoE comparison: The paper consistently finds that MoE-4x underperforms MoT in non-text modalities (image, speech) and that this gap grows with model scale. At the 7B Chameleon scale, MoE-4x shows "diminishing returns... with advantages disappearing at 7B" for images (Figure 6q-r). In the speech setting, MoE-4x shows a particularly striking pattern: it outperforms the dense baseline on speech training loss (Figure 9e, 9m, 9u) but underperforms on speech validation loss (Figure 9g, 9h, 9o, 9p, 9w, 9x). The paper attributes this to data distribution shift and potential overfitting due to MoE-4x's larger parameter count. The key point is that even if MoE-4x's validation performance is overestimated (due to causal violation), it still underperforms MoT, making the MoT advantage robust.

Design Choices and Their Justifications

Why decouple all non-embedding parameters rather than only FFNs? This is the most fundamental design choice in MoT, and the ablation study in Section 3.5 provides direct evidence for it. Using a 880M parameter configuration in the Chameleon setting (Table 5: hidden dimension 1536, 24 layers, 24 heads, 128 GPUs, 2.10M tokens/batch, 120k steps, 0.252 trillion tokens), the paper compares three progressively more decoupled architectures:

1. Dense baseline: No modality-specific parameters — all tokens share everything.

2. FFN-only untying (as in MoMA, Lin et al., 2024): Each modality gets its own FFN parameters, but attention QKV projections and LayerNorms remain shared. The paper reports that this alone "significantly improves model performance, with substantial gains on the image modality" (Figure 14).

3. FFN + Attention QKV untying: In addition to separate FFNs, each modality gets its own $W_Q, W_K, W_V$ matrices (but not $W_O$ or LayerNorms). The paper reports that this yields additional FLOPs savings of "approximately 33.3%... for the image modality and 10%... for the text modality compared to only performing untying in the feedforward module" on the Obelisc held-out set.

4. Full MoT (FFN + Attention + LayerNorms): Adding LayerNorm untying on top of FFN and attention untying has "a negligible impact on evaluation performance."

What these results mean for the design: The FFN is the most important component to untie — it accounts for the largest fraction of FLOPs (the SwiGLU FFN has $3d^2$ parameters per layer, compared to $4d^2$ for all attention projections combined), and it serves as the primary "memory" component in transformers, storing learned patterns. Having separate FFNs for each modality means the model can maintain modality-specific knowledge without interference. Untying the attention QKV matrices provides additional but smaller gains — these matrices determine how tokens are projected into the query, key, and value spaces, so having modality-specific projections allows each modality to optimize how it represents itself for cross-modal attention. Untying $W_O$ and LayerNorms provides negligible additional benefit, suggesting that once the major computation components (FFN, QKV) are separated, the remaining shared parameters don't create a significant bottleneck.

Why global self-attention rather than cross-attention? The paper explicitly addresses this in a footnote to Equation 2: "Comparing to works that utilize cross-attention to fuse information from different modalities (Alayrac et al., 2022; Aiello et al., 2023), our formulation using global self-attention normalizes attention weights across tokens of different modalities while reducing the number of layers in the architecture." The key points are:

• Normalized attention weights: Global self-attention uses a single softmax over the entire sequence, meaning the attention distribution naturally balances within-modality and cross-modality attention. Cross-attention architectures typically have separate attention operations for self-attention (within modality) and cross-attention (between modalities), requiring separate softmax normalizations that can create imbalances.

• Reduced layer count: Cross-attention architectures often double the number of attention operations per layer (one self-attention, one cross-attention), effectively halving the depth of processing for a given parameter budget. MoT's global self-attention maintains the same number of attention operations as the dense baseline.

• Early-fusion compatibility: Global self-attention is the natural design for early-fusion models like Chameleon, where all modalities are tokenized and interleaved into a single sequence. Cross-attention typically assumes separate encoders for different modalities, which is a late-fusion design incompatible with MoT's goal of end-to-end training from scratch.

Why deterministic, rule-based routing rather than learned routing? The paper provides both empirical and theoretical justifications:

• Training stability: Learned routing introduces a bi-level optimization problem — the router and the experts are co-trained, and errors in one propagate to the other. The paper notes that "both experts and routers are under-trained in the early stages," creating instability. MoT's deterministic routing eliminates this entirely — the modality of each token is known from preprocessing, so there is no learned component to fail.

• No load-balancing issues: MoE architectures require auxiliary losses to prevent the router from collapsing to a small subset of experts. In multi-modal settings, this is amplified because modalities have different frequencies — speech tokens might appear less frequently, causing speech-token experts to be starved unless explicitly balanced. MoT's per-modality towers are always used whenever tokens of that modality appear, so there is no imbalance to manage.

• No inference-mode mismatch: The Expert Choice causal violation problem (Section 3.2.1) doesn't apply to MoT because modality assignment doesn't depend on sequence context at all — it's purely deterministic based on the token's type. This means MoT during inference is exactly identical to MoT during training, with no information leakage.

• Empirical evidence: The paper shows (throughout Sections 3.2-3.4) that MoE-4x consistently underperforms MoT on non-text modalities, and the gap grows with scale. At 7B in Chameleon, MoE-4x's image modality advantage over the dense model disappears entirely, while MoT retains a substantial speedup (34.8% of training steps to match dense image loss; Figure 6r). In speech, MoE-4x shows training loss improvement but validation loss degradation, suggesting overfitting or routing instability.

Why share the embedding layer? The embedding layer is the only shared component in MoT. The paper doesn't explicitly justify this choice, but it follows from the early-fusion design: all modalities need to be mapped into a common vector space for the global self-attention to compute meaningful cross-modal attention scores. If the embeddings were also modality-specific, there would be no shared representational space, and the attention dot products between tokens of different modalities would be uninterpretable (vectors from different spaces). The shared embedding imposes a minimal parameter cost (vocabulary sizes are small compared to transformer layer parameters) while enabling the core cross-modal interaction mechanism.

The Hybrid MoT+MoE Architecture

Section 5 of the paper presents a proof-of-concept combining MoT and MoE, which is not the main contribution but demonstrates an important property: MoT is not mutually exclusive with learned sparsity — it's a framework within which MoE can be selectively deployed.

Configuration for Chameleon (Section 5.1): Using a 443M configuration in the Chameleon+Speech setting, the paper replaces the FFN of the text transformer tower with a 4-expert MoE layer (MoE-4x), while the image and speech towers remain standard MoT (dense FFNs). This means text tokens are processed through a learned routing mechanism that selects among 4 experts per FFN layer, while image and speech tokens continue to be processed by their dedicated, non-expert FFNs. The result (Figure 16) shows that this hybrid "MoT + Text MoE-4x" outperforms pure MoT on text training loss (requiring 43.1% of dense model's steps to match text loss, compared to 61.0% for pure MoT), while preserving MoT's image modality advantages (26.3% vs. 27.1% of dense steps for image loss). The averaged training loss across modalities is matched in 35.6% of dense steps (vs. 43.5% for pure MoT and 63.0% for pure MoE-4x).

Configuration for Transfusion (Section 5.2): Using a 760M configuration, the same hybrid approach is applied: the text FFN becomes MoE-4x, while the image diffusion tower remains standard MoT. Results (Figure 17) show accelerated text training loss reduction (50.4% of dense steps vs. 97.0% for pure MoT) with preserved image advantages (17.1% vs. 17.6% of dense steps for image loss). On text validation (C4, Wikipedia), the hybrid achieves the best performance, while image quality metrics (CLIP, FID) remain comparable to pure MoT.

What this demonstrates: MoT's modular architecture — where each modality has its own transformer tower — makes it straightforward to apply different sparsity mechanisms to different modalities. Text, which benefits from the increased capacity of MoE (as shown by MoE-4x's text training loss improvements), can use learned routing, while image and speech, where MoE underperforms MoT (due to routing instability, distribution shift, or overfitting), can use the deterministic per-modality towers. This is not a "one is better than the other" finding but rather evidence that MoT's framework enables heterogeneous architectures optimized per modality, which could be a direction for future multi-modal foundation models.

Parameter Scaling Properties: Why MoT Has a Lower Parameter-to-FLOPs Ratio Than MoE

Section 6.1 provides a theoretical analysis of a property that matters for distributed training throughput: the Parameter-to-FLOPs (PpF) ratio. Lower PpF means that for a given computational cost, the model has fewer parameters relative to how much work it does, which reduces communication overhead in distributed training (where parameters must be synchronized across GPUs).

The analysis considers a transformer layer with hidden dimension $D$, SwiGLU FFN (which has $3D^2$ parameters per FFN — two projection matrices of size $D \times D_{\text{ffn}}$ where $D_{\text{ffn}} \approx \frac{8}{3}D$ for SwiGLU, plus gating), and attention projections ($4D^2$ parameters for $W_Q, W_K, W_V, W_O$ combined, each $D \times D$).

For MoE with $E$ experts: each expert adds a full FFN, so a MoE layer has $E \times |\text{FFN}| = 3E D^2$ parameters in the FFN, plus a router of size $E \times D$ (negligible for large $D$). The additional parameters relative to a dense layer are $(E-1) \times 3D^2$. These extra parameters increase the PpF ratio.

For MoT with $K$ modalities: each modality has its own FFN ($3D^2$) and attention projections ($4D^2$), for a total of $7D^2$ parameters per modality. The additional parameters relative to a dense layer (which has one set of FFN + attention = $7D^2$) are $(K-1) \times 7D^2$.

Since typical $E$ (number of experts) can be "a few to even hundreds" (the paper cites Dai et al., 2024; Muennighoff et al., 2024; DeepSeek-AI et al., 2025), while $K$ is typically small (2-4 modalities), MoT generally achieves a lower PpF ratio than MoE — it has fewer additional parameters for a given FLOP budget. This matters in large-scale cloud training where "compute capacity has increased significantly faster than network bandwidth" (Luo et al., 2018; 2024), making communication-bound operations a bottleneck. The paper's horizontal scaling analysis (Figure 18) confirms that MoT's advantages grow with GPU count, consistent with the hypothesis that lower PpF reduces communication overhead at scale.

Why this matters for the technical approach: MoT's efficiency gains come from two sources — reduced FLOPs-to-performance (faster convergence per compute unit) and reduced communication overhead (lower PpF). The first source is the primary focus of the paper (training step matching, validation loss comparisons). The second source is a secondary benefit that translates the FLOPs savings into wall-clock speedups, as documented in Figure 19. The PpF analysis explains why MoT's wall-clock advantages are larger than its raw FLOP advantages would suggest — the architecture is not just more sample-efficient but also more hardware-efficient.

System-Level Implementation Considerations

Section 6.1 discusses implementation overheads of both MoE and MoT, providing practical context for the architectural comparison.

MoE overheads: "MoEs suffer from overheads due to the additional operations of performing Top-K selection, indexing tokens, and scattering and adding expert outputs. These operations are sequentially dependent on each other, making it challenging to hide the resulting latency." The key phrase is "sequentially dependent" — the router must compute scores before experts can be assigned, experts must process before outputs can be gathered, and the scheduling of these operations creates bubbles in GPU utilization.

MoT overheads: MoT's overheads come from two sources: "First, the CPU-GPU synchronization required for grouping tokens by modality for element-wise projections and reassembling them for attention results in significant overhead, mostly attributed to frequent GPU-CPU synchronization due to masking for specific modalities. Second, the sequential processing of modalities can also lead to underutilization of GPU resources and imbalance, particularly when tokens of different modalities are unevenly distributed across local batches and GPUs."

However, the paper notes that MoT's overheads "can be minimized via diligent engineering": caching sequence indices for each modality reduces indexing costs (these indices don't change during processing — the modality of each token is fixed), and specialized operations like Grouped GEMMs (Nvidia's grouped general matrix multiply APIs) or Megablock-style block sparse matrix multiplication (Gale et al., 2022) can perform imbalanced projections across modalities in a single operation. The paper reports that "we did not observe these overheads on the critical path in our training setup," indicating that with standard engineering practices (PyTorch 2 Compiler, Fully Sharded Data Parallel), these overheads are manageable at the scales studied.

4. Key Insights and Innovations

Innovation 1: Modality-Aware Sparsity as a Deterministic Alternative to Learned Routing — Not an Approximation, But a Superior Design Choice

The dominant approach to model sparsity in large-scale transformers has been Mixture-of-Experts (MoE), where a learned router dynamically assigns tokens to expert feed-forward networks. The field has invested heavily in making this learned routing work — developing auxiliary load-balancing losses (Fedus et al., 2022), expert choice routing to guarantee balance (Zhou et al., 2022), inference-time routing corrections (Zhong et al., 2024), and various initialization and stabilization tricks. The implicit assumption has been that learned routing is necessary for sparsity — that the model needs to discover which parameters specialize to which inputs, because we can't know a priori what the right partitioning should be.

This paper upends that assumption by demonstrating that in multi-modal settings, deterministic routing by modality is not just simpler, but strictly better. The modality of each token is known from preprocessing — text, image, speech — and assigning modality-specific transformer towers based on this known attribute eliminates every pathology that plagues MoE: no load imbalance, no bi-level optimization instability, no inference-mode mismatch, no auxiliary losses. The paper doesn't just claim this theoretically; it provides systematic evidence across three experimental settings (Chameleon, Chameleon+Speech, Transfusion) and seven model scales that MoE-4x consistently underperforms MoT on non-text modalities, with the gap widening as scale increases (Figure 6q-r shows MoE-4x's image advantage disappearing entirely at 7B, while MoT maintains a 3× speedup).

What makes this a fundamental rather than incremental contribution is the conceptual reframing it enables. MoE treats sparsity as a learning problem — the routing network must discover the right assignment. MoT treats sparsity as an architectural design problem — we know that text, images, and speech have different statistical properties (as the PCA visualizations in Figure 2 and Appendix Figure 23 empirically confirm), so we should just give them separate parameters. This shifts the question from "how do we make the router learn the right partitioning?" to "what is the right partitioning in the first place?" The answer — at least for early-fusion multi-modal models — turns out to be trivially simple: partition by modality.

The significance of this finding extends beyond the specific performance numbers. It suggests that the field's obsession with learned routing may have been misallocated for multi-modal settings. The paper's hybrid experiment (Section 5, Figure 16) — where MoE-4x is used only in the text tower while image and speech towers remain standard MoT — shows that learned routing can still help within a modality, but the primary efficiency gain comes from across-modality partitioning. This is a diagnostic insight that reframes how to think about sparsity in heterogeneous data settings: separate the known sources of heterogeneity first (modality), then apply learned sparsity within homogeneous subsets if needed.

Innovation 2: The "Everything Should Be Decoupled" Principle — Evidence That Modality-Specificity Should Permeate the Full Transformer Stack, Not Just the FFN

Prior modality-aware sparse architectures — VLMO (Bao et al., 2022b), MoMA (Lin et al., 2024), CogVLM (Wang et al., 2023) — applied modality-specific parameters only to the feed-forward network (FFN) layers. The rationale was intuitive: the FFN is the largest component by parameter count and serves as the primary "memory" of the transformer, so separating it by modality should capture most of the benefit. The attention projections and layer normalization were left shared, presumably because they handle a more "generic" computation (forming queries, keys, and values for attention) that shouldn't need modality-specificity.

MoT challenges this limited scope not through argument but through empirical ablation (Section 3.5, Figure 14). The paper shows that untying the FFN alone provides substantial gains — particularly for the image modality — which validates prior work. But it then shows that further untying the attention Q, K, V projection matrices yields an additional ~33% FLOPs savings for images and ~10% for text on the Obelisc held-out set, compared to FFN-only untying. This is a non-trivial additional gain. The implication is that how a modality projects its tokens into query, key, and value spaces for attention is not a generic computation — it benefits from modality-specific optimization. Text tokens and image tokens, even when embedded in a shared space, "want" to be represented differently for the purpose of computing attention scores against tokens of other modalities.

Why is this conceptually important? Because it establishes a principle of maximal decoupling: in a multi-modal transformer, any parameter that processes modality-specific information should be modality-specific. The only genuinely shared component should be the embedding layer (which maps all modalities into a common space for cross-modal attention) and the global attention mechanism itself (which computes dot products across modalities). Everything else — how tokens project their queries, how they compute keys, how they transform attention outputs, how they apply feed-forward transformations, and even how they normalize — is more efficient when specialized.

The paper is appropriately nuanced about where this principle hits diminishing returns: LayerNorm untying provides "negligible impact" beyond FFN and attention untying. This is itself informative — it tells us that the normalization statistics are sufficiently similar across modalities (when computed after modality-specific processing) that shared LayerNorms don't create a bottleneck. The principle is therefore not dogmatic but empirical: decouple until the gains disappear.

This contribution is incremental relative to prior modality-sparse work (it extends the scope of an existing idea), but it is fundamental in its implications for architecture design: it establishes that attention projection specialization matters, which prior work either assumed away or never tested. Future multi-modal architectures should default to modality-specific attention projections, not treat shared projections as the baseline.

Innovation 3: The Diagnosis of Modality Competition — Evidence That Modalities Conflict When Sharing Parameters, and That This Conflict Is the Root Cause of Multi-Modal Training Inefficiency

The paper goes beyond demonstrating that MoT is faster — it provides diagnostic evidence for why the dense baseline is slow. Through the Leave-One-Out (LOO) analysis in Section 4 (Figure 15), the paper shows what happens when two modalities are forced to share a transformer tower while the third gets its own. The results reveal a pattern of modality competition that is non-obvious and has implications beyond the MoT architecture.

When text and speech share a tower (LOO-image configuration), both modalities suffer — text training loss degrades relative to full MoT, and speech degrades relative to full MoT. When image and speech share (LOO-text), only speech degrades while image largely maintains MoT's gains. When text and image share (LOO-speech), both lose MoT's benefits. This is not a symmetric effect — the competition is non-reciprocal, with some modality pairings being more damaging than others. Speech appears to be the most "fragile" modality, suffering when paired with anything else. Image is robust when paired only with speech, but degrades when paired with text.

The paper characterizes this as "non-reciprocal modality competition effects" — a diagnostic concept that had not been articulated before. The implication is that the inefficiency of dense multi-modal training is not just about "wasted" parameters (using the same FFN for text and images), but about active interference: the gradients from one modality pull shared parameters in directions that harm performance on the other modality. By separating modalities into their own towers, MoT eliminates this gradient interference, allowing each modality to optimize independently while still benefiting from cross-modal attention.

This insight connects to and explains a phenomenon the paper notes in passing: Chameleon required 9.2 trillion tokens to match LLaMA2's text performance on 2 trillion tokens, representing a more than 4× inflation in data requirements. The modality competition diagnosed here provides a mechanistic hypothesis for that inflation — the text parameters in the dense Chameleon are constantly being pulled by image gradients, requiring more data to converge to the same text optimum that a text-only model would reach much faster. The Transfusion setting (where text and images have separate objectives) partially mitigates this by decoupling the loss functions, which may explain why text performance improves in Transfusion even without direct changes to text training — the gradient interference is reduced.

This is a fundamental diagnostic contribution rather than an incremental finding, because it recharacterizes the multi-modal training problem. The challenge is not just "more data is needed" but "modalities compete for parameter capacity in ways that are pairing-specific." This opens a research direction: understanding and predicting which modality pairings are most competitive, and designing architectures or training schedules that minimize this interference.

Innovation 4: The Transfusion-MoT Synergy — Parameter Decoupling and Objective Decoupling Are Multiplicative, Not Redundant

The Transfusion setting (Section 3.4) is where the paper makes its most subtle but perhaps most forward-looking contribution. Transfusion (Zhou et al., 2024) already introduces a form of modality-specificity at the objective level: text uses autoregressive language modeling loss, images use diffusion-based denoising loss. This separation of objectives was shown to improve text performance relative to Chameleon, even without changing the architecture. MoT introduces modality-specificity at the parameter level: separate transformer towers for each modality. The natural question is whether these two forms of modality-specificity are redundant — if the objectives are already separated, does parameter separation still help?

The paper's results show they are multiplicative, not redundant. In the Transfusion 7B setting (Figure 10b-c), MoT achieves the dense model's image training loss in ~30% of the steps — a 3.3× speedup — which is larger than the image modality speedup in the Chameleon 7B setting (34.8% of steps, or ~2.9×). The 760M MoT model, using half the FLOPs of the 1.4B dense baseline, actually outperforms the larger model on image quality (CLIP: 0.214 vs 0.206; FID: 21.145 vs 24.688; CIDEr: 0.320 vs 0.286; Figure 11). These gains are substantial and go beyond what either objective decoupling (Transfusion) or parameter decoupling (MoT) achieves alone.

However, an interesting asymmetry emerges: MoT's text performance gains are "marginal to none" in Transfusion, whereas they were substantial in Chameleon (54.6-66.2% of steps to match dense text loss). The paper hypothesizes that this is because Transfusion's objective decoupling already provides much of the benefit for text — the text parameters are less "distracted" by image gradients when the image objective is separate, so the additional gain from parameter decoupling is smaller. In contrast, the image diffusion task is computationally heavy, and having dedicated image parameters for the diffusion denoising network yields outsized benefits.

This finding has conceptual significance because it establishes that architectural sparsity (MoT) and objective sparsity (Transfusion) are complementary design dimensions that can be independently optimized. Future work can explore more sophisticated objective separations — different learning rates, different optimizers, different regularization — on top of parameter separation, with the expectation that the gains will compound. It also suggests a general principle: the more heterogeneous the computational requirements across modalities (as diffusion is heavier than autoregressive language modeling), the larger the benefit from parameter decoupling.

This contribution is incremental in mechanism (it's applying MoT to a new setting) but fundamental in its implication for architecture design: it establishes that modality-specificity should operate at multiple levels of the training stack — objectives, parameters, potentially even optimization hyperparameters — and that these levels are synergistic.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper evaluates across three distinct settings, each with different datasets. In the Chameleon setting (autoregressive text+image), the training data comprises roughly equal parts text and image tokens from Chameleon Team (2024)'s mixed-modal corpus; validation uses held-out sets from Obelisc (Laurençon et al., 2023), MS-COCO (Lin et al., 2014; Karpathy test split), Flickr30k (Plummer et al., 2015; Karpathy test split), and Shutterstock. In the Chameleon+Speech setting, speech data from SpiRit-LM (Nguyen et al., 2024) — combining People's Speech (16,404 hours), Voxpopuli English (23,166 hours), LibriLight (55,308 hours), Multilingual LibriSpeech English (44,585 hours), and Spotify (57,290 hours) — is mixed with the Chameleon text+image data at a 1:6 sampling ratio; speech validation uses LibriLight (LL60K) and People's Speech (PPL30K) held-out sets. In the Transfusion setting, text comes from the Llama 2 corpus (2 trillion tokens) and images from 380 million licensed Shutterstock images with captions, with 0.5 trillion tokens sampled at a 1:1 ratio; text evaluation uses 20M held-out tokens from Wikipedia and C4 (Raffel et al., 2020), image evaluation uses diffusion validation loss on held-out Conceptual 12M (CC12M; Changpinyo et al., 2021) and generation metrics (FID, CLIP, CIDEr) on MS-COCO 30K prompts.

• Base model(s). All experiments use transformer models trained from scratch — no pre-trained weights are used. The paper trains thirteen models total across three settings, spanning scales from 37M to 7B parameters. For the Chameleon setting: 37M, 94M, 443M, 1.5B, and 7B (Table 1). For Chameleon+Speech: 443M, 880M, 1.5B, and 7B (Table 3). For Transfusion: 163M, 760M, 1.4B, and 7B (Table 4). The paper argues that PaLM-style models at these scales are "representative of the capabilities of many contemporary LLMs" and that training from scratch provides the cleanest evaluation of architectural differences.

• Metrics. The paper employs modality-specific metrics across settings. Training and validation loss (cross-entropy for autoregressive modalities, diffusion denoising loss for Transfusion images) is the primary metric for measuring training efficiency, reported separately per modality using step-matching analysis (the number of training steps MoT requires to reach the dense baseline's final loss). In the Transfusion setting, additional generation quality metrics are reported: zero-shot FID (Fréchet Inception Distance; Heusel et al., 2017) on 30,000 MS-COCO validation prompts for image photorealism (lower is better); CLIP score (Radford et al., 2021) for text-image alignment (higher is better); and CIDEr score (Vedantam et al., 2015) for image captioning quality (higher is better). For speech, validation loss is reported on LL60K and PPL30K held-out sets. Text validation loss is reported on Obelisc, COCO, Flickr, and SSTK (Chameleon) or C4 and Wikipedia (Transfusion). For Chameleon MS-COCO and Flickr30k, text-to-image and image-to-text conditional perplexity is used.

• Baselines. Three baselines are compared throughout. Dense transformer: the standard architecture with all parameters shared across modalities, trained with identical FLOPs to MoT. MoE-4x: a Mixture-of-Experts model with 4 experts per FFN layer using Expert Choice routing (Zhou et al., 2022), which has each expert select its top-k tokens to guarantee load balance (Section 3.2.1). At smaller scales in Chameleon, additional MoE-2x and MoE-3x variants are compared. MoMA-style FFN-only untying (Lin et al., 2024): used as an ablation baseline in Section 3.5, where only FFN parameters are modality-specific while attention projections and LayerNorms remain shared. The paper explicitly flags that the MoE-4x evaluation uses Expert Choice routing during validation, which violates autoregressive causality and may overestimate MoE performance (information leakage from future tokens) or underestimate it on out-of-distribution validation data — this is acknowledged as a confounding factor.

• Generation budget / compute accounting. All comparisons across architectures are FLOPs-controlled: MoT, dense, and MoE models at a given scale have identical training and inference FLOPs per token. For MoE, the "activated parameters" (parameters actually used per token) are reported, which are matched to the dense and MoT parameter counts for fair comparison. Training steps and batch sizes are kept consistent across architectures at each scale, so equal steps means equal total FLOPs. In the step-matching analysis (e.g., Figure 5b), the paper plots the training steps required by MoT to reach the same loss values as the dense baseline, and reports the ratio as a percentage. Wall-clock time measurements (Section 6.2.2) use normalized GPU training time on 256 GPUs (AWS p4de.24xlarge instances with NVIDIA A100 GPUs), measuring the fraction of the dense model's training time needed to match its performance. For the FLOPs-matched comparison across model sizes (e.g., 760M MoT vs. 1.4B dense), the models operate at the same FLOPs budget — the smaller MoT model uses half the FLOPs of the larger dense model.

• Cross-validation / statistical protocol. The paper does not use cross-validation in the traditional sense. Strategy selection (compute-optimal allocation per modality) is not performed — MoT is a deterministic architecture, so there is no hyperparameter to select per difficulty bin or prompt. The evaluation protocol is to train all models from scratch with identical data, hyperparameters, and FLOPs, then compare their loss curves and final metrics directly. For the step-matching analysis, curves are plotted at the granularity of training steps, and the ratios are computed by finding the step at which the MoT curve crosses the dense model's final loss value. For horizontal scaling analysis (Figure 18), global batch size and total training tokens are scaled proportionally with GPU count while keeping training steps constant, enabling comparison of how MoT's relative advantage changes with compute scale. The paper does not report confidence intervals, statistical significance tests, or multiple training runs with different random seeds for any of its results — all curves appear to be from single training runs, which is a limitation discussed in the Critical Assessment.

Main Quantitative Results

Performance in the Chameleon Setting: Autoregressive Text and Image Generation

Headline result at 7B scale. In the Chameleon 7B setting, MoT matches the dense baseline's final training loss at 120,000 steps in only 60,000 steps, requiring 45.5% of the dense model's training steps for equivalent pre-training performance (Figure 5a-b). This translates to a 44.2% reduction in required training FLOPs. For the image modality specifically, MoT requires only 34.8% of the dense model's training steps to match final image training loss (Figure 5c-d). For text, MoT requires 55.8% of the dense model's training steps (Figure 5e-f). These gains are not uniform: MoT achieves its largest advantage in the image modality, where the dense model's uniform processing is most wasteful.

Validation loss confirmation. MoT at the 55.8% training checkpoint (67,000 steps) achieves validation losses comparable to or lower than the dense model's final validation loss across all datasets tested (Figure 5g-n). On Obelisc image validation loss, MoT (55.8% checkpoint) reaches approximately 3.95 vs. the dense model's final ~3.95. On COCO image validation, MoT reaches ~5.92 vs. dense final ~5.92. On Flickr image validation, MoT reaches ~6.08 vs. dense final ~6.10. On SSTK image validation, MoT reaches ~5.25 vs. dense final ~5.28. For text validation (Figure 5k-n), MoT matches or beats the dense baseline on Obelics (~2.40 vs. ~2.40), COCO (~2.86 vs. ~2.86), Flickr (~3.05 vs. ~3.05), and SSTK (~1.82 vs. ~1.82). The consistency across four diverse held-out datasets (Obelisc, MS-COCO, Flickr30k, Shutterstock) strengthens the claim that MoT's training efficiency translates to genuine generalization improvements, not merely faster overfitting to the training distribution.

Comparison with MoE-4x at 7B. MoE-4x shows limited improvement in the image modality at 7B scale, with its advantage over the dense baseline "diminishing... with advantages disappearing at 7B" (Section 3.2.3). Figure 6q shows that MoE-4x's image training loss curve tracks closely to the dense baseline — it requires approximately 101.2% of the dense model's steps to match image loss (Figure 6r, s = 1.012), meaning it provides essentially no speedup. In contrast, MoT requires only 34.8% (s = 0.348). For text modality, both MoT and MoE-4x outperform the dense baseline, with MoT at 55.8% of steps and MoE-4x at 63.5% (Figure 6t). This asymmetry — MoE works for text but fails for images at scale — is a key empirical finding that the paper uses to argue that learned routing is not a general solution for multi-modal sparsity.

Performance across model scales (37M to 7B). Figure 6 shows step-matching results across five scales. For the image modality, MoT consistently delivers substantial speedups: at 37M (Figure 6a-b), MoT matches dense image loss at 26.0% of steps vs. MoE-4x at 49.6%; at 94M (Figure 6e-f), MoT at 19.9% vs. MoE-4x at 35.8%; at 443M (Figure 6i-j), MoT at 27.1% vs. MoE-4x at 49.0%; at 1.5B (Figure 6m-n), MoT at 35.0% vs. MoE-4x at 74.5%; at 7B (Figure 6q-r), MoT at 34.8% vs. MoE-4x at 101.2% (no advantage). The trend is clear: MoT maintains roughly 3-5× image modality speedup across all scales, while MoE-4x's advantage monotonically degrades from ~2× at 37M to none at 7B.

For the text modality (Figure 6d,h,l,p,t), MoT and MoE-4x are more comparable, with both showing 1.5-2× speedups over the dense baseline. At 37M: MoT 54.6% vs. MoE-4x 59.1%. At 94M: MoT 53.0% vs. MoE-4x 53.1%. At 443M: MoT 61.0% vs. MoE-4x 73.1%. At 1.5B: MoT 66.2% vs. MoE-4x 71.3%. At 7B: MoT 55.8% vs. MoE-4x 63.5%. The paper notes that MoE-4x shows "comparable or slightly better" text performance at some scales, but the advantage over MoT is small and inconsistent. Validation loss results (Appendix Figure 24) confirm these training loss trends across all scales and both modalities.

Why the image advantage is so large. The paper does not explicitly quantify the FLOP distribution between modalities, but the image advantage is mechanistically interpretable: images are tokenized into 1,024 discrete tokens (vs. variable-length text), and with roughly equal amounts of text and image tokens in training, the image modality represents a substantial fraction of total FLOPs. The dense transformer is forced to use the same FFN and attention projections to model text's discrete, categorical structure and images' spatial, continuous structure — a severe representational mismatch that MoT resolves by giving each modality dedicated processing parameters. The fact that the speedup for images (~3×) is larger than for text (~1.8×) suggests that the image modality is where the dense model's parameter sharing is most harmful.

Performance with Speech as Third Modality (Chameleon+Speech)

Headline result at 7B scale. When speech is added as a third modality, MoT achieves the dense model's speech training loss in only 22.9% of the training steps (Figure 8a-b). This is the largest speedup reported in the paper — MoT trains speech nearly 4.4× faster than the dense baseline. For speech validation loss, MoT reaches baseline performance in 37.2% of the FLOPs on PPL30K (Figure 8f, s = 0.372) and 31.3% on LL60K (Figure 8d, s = 0.313). On image and text modalities (which are also present in this setting), MoT at the 55.8% training checkpoint (carried over from the Chameleon 7B analysis) achieves validation losses comparable to the dense baseline's final loss (Figure 8g-n), confirming that adding speech doesn't degrade MoT's efficiency on the other modalities.

MoE-4x's speech instability. A striking pattern emerges in the speech modality: MoE-4x outperforms the dense baseline on training loss (Figure 8a) but underperforms on validation loss for both LL60K (Figure 8c, MoE-4x loss ~1.24 vs. dense ~1.05 at 120k steps) and PPL30K (Figure 8e, MoE-4x loss ~1.43 vs. dense ~1.40 at 120k steps). This is the classic signature of overfitting or distribution shift sensitivity: the learned router performs well on in-distribution data but fails to generalize to held-out speech datasets. The paper attributes this to the Expert Choice routing's sensitivity to data distribution shifts and MoE-4x's larger raw parameter count making it "prone to overfitting, hence contributing to its underperformance in speech validation loss, especially given the smaller amount of unique speech tokens in the combined dataset." On PPL30K, MoE-4x requires 84.0% of dense steps to match validation loss (Figure 8f), meaning its speedup is modest, and on LL60K it requires 97.9% — essentially no advantage. MoT, in contrast, shows consistent acceleration on both training and validation speech metrics.

Scalability across model sizes for speech (443M, 880M, 1.5B). Figure 9 shows speech modality results across smaller scales, and the speedup is remarkably consistent: MoT matches the dense model's speech training loss in 15.1% of steps at 443M (Figure 9f), 19.4% at 880M (Figure 9n), and 19.9% at 1.5B (Figure 9v). The speech validation loss patterns (Figure 9g-h, o-p, w-x) confirm that MoT consistently achieves lower validation loss than both dense and MoE-4x across all scales. MoE-4x's validation loss for speech is consistently worse than the dense baseline at 443M and 880M (Figure 9g-h, o-p), and only matches it at 1.5B (Figure 9w-x). This reinforces the finding that deterministic modality partitioning is particularly important for speech — a modality with distinct temporal characteristics and a smaller unique token count in the combined dataset — where learned routing fails to generalize.

Image and text consistency. At all three smaller scales, MoT maintains its image and text modality advantages (Figure 9a-d, i-l, q-t). Image training loss speedups: 26.2% at 443M, 31.1% at 880M, 33.6% at 1.5B. Text training loss speedups: 55.7% at 443M, 57.2% at 880M, 59.5% at 1.5B. These are consistent with the Chameleon-only results, showing that the addition of speech does not interfere with MoT's efficiency on the original two modalities. Appendix Figure 25 provides validation loss confirmation for image and text across all Chameleon+Speech scales.

Performance in the Transfusion Setting: Multi-Objective Training

Headline result at 7B scale. In the Transfusion 7B setting, MoT accelerates image modality pre-training substantially, matching the dense model's image training loss in approximately 30% of the training steps (Figure 10b-c). On image validation loss (Figure 10d-e), MoT requires 37.4% of dense steps to match (s = 0.374). Generation quality metrics at 7B show: CLIP score of 0.222 vs. 0.226 for dense (Figure 10f, MoT slightly lower); FID score of 18.862 vs. 19.502 for dense (Figure 10g, MoT better); CIDEr score for image captioning of 0.406 vs. 0.315 for dense (Figure 10h, MoT substantially better). The CIDEr improvement of +9.1 points (+28.9% relative) is particularly notable — it suggests that modality-specific parameters are especially beneficial for cross-modal tasks like image captioning, where the image representations produced by the dedicated image tower may be more informative for text generation. For reference, a dense Transfusion model trained on 1T tokens with richer data achieves COCO-30k FID of 9.22 at guidance level 1.6, while the 7B MoT achieves 8.14 at the same guidance level — despite being trained on only 0.5T tokens — indicating that MoT's efficiency advantage translates to competitive or superior generation quality even against models trained on more data.

Text performance in Transfusion. The paper reports that MoT's text performance improvement in Transfusion was "marginal to none" (Section 3.4.2), in contrast to the Chameleon setting where MoT showed 1.5-2× text speedups. Appendix Figure 26 confirms this: MoT requires 92.6% of dense steps to match text training loss at 1.4B (Figure 26-12), 97.0% at 760M (Figure 26-7), and 98.0% at 163M (Figure 26-2). Text validation loss on C4 and Wikipedia is essentially identical between MoT and dense across scales (Figure 26-3,4,8,9,13,14). The paper hypothesizes that Transfusion's objective decoupling (separate autoregressive loss for text, diffusion loss for images) already provides much of the benefit that MoT's parameter decoupling would otherwise provide — text parameters are less "distracted" by image gradients when the loss functions are separate. This is a hypothesis, not a proven mechanism, but it's consistent with the observation that the text speedup in Transfusion is minimal while the image speedup remains large.

760M MoT vs. 1.4B dense: smaller model outperforms larger model. At 760M parameters, MoT (using half the training/inference FLOPs of the 1.4B dense baseline) outperforms the larger model across all image metrics (Figure 11). Training loss: 760M MoT achieves the 1.4B dense baseline's image training loss in 25.9% of the steps (Figure 11b, s = 0.259), meaning it's substantially more sample-efficient even at half the size. Generation quality: CLIP score 0.214 vs. 0.206 (Figure 11c, +3.9% relative); FID score 21.145 vs. 24.688 (Figure 11d, -14.4% relative); CIDEr score 0.320 vs. 0.286 (Figure 11e, +11.9% relative). For context, a 163M MoT model (roughly 1/9th the size) achieves image training loss comparable to the 1.4B dense (Figure 11a-b, s = 0.883), though it lags in evaluation metrics (CLIP 0.195 vs. 0.206, FID 26.920 vs. 24.688, CIDEr 0.232 vs. 0.286). The 760M MoT represents a sweet spot: half the FLOPs but better quality on all metrics.

Scalability across model sizes in Transfusion (163M, 760M, 1.4B). Figure 12 provides comprehensive per-scale results. For image modality: training loss speedups are consistent — 163M MoT at 25.4% of dense steps, 760M at 17.6%, 1.4B at 17.5% (Figure 12-2,11,20). Image validation loss speedups mirror this: 28.6%, 24.7%, 23.1% respectively (Figure 12-4,13,22). FID scores show dramatic improvements: at 163M, MoT achieves 21.586 vs. 27.428 for dense (Figure 12-6, -21.3%); at 760M, 15.749 vs. 25.576 (Figure 12-15, -38.4%); at 1.4B, 15.850 vs. 19.318 (Figure 12-24, -18.0%). CLIP scores also consistently favor MoT: 0.195 vs. 0.185 (163M), 0.214 vs. 0.196 (760M), 0.217 vs. 0.206 (1.4B). For text modality: MoT matches dense on C4 and Wikipedia validation loss (Figure 12-7,8,16,17,25,26) but consistently improves captioning CIDEr: 0.232 vs. 0.142 (163M, +63.4%), 0.320 vs. 0.251 (760M, +27.5%), 0.335 vs. 0.286 (1.4B, +17.1%). The CIDEr improvement is largest at the smallest scale and shrinks proportionally as scale increases, suggesting that smaller dense models particularly struggle with cross-modal tasks and benefit disproportionately from modality-specific parameters.

MoE-4x in Transfusion. MoE-4x shows unstable behavior: it achieves lower text training loss than dense (Figure 26-2, s = 0.268 at 163M; Figure 26-7, s = 0.495 at 760M) but worse generalization on text validation (C4 and Wikipedia losses are higher than both dense and MoT at all scales; Figure 12-7,8,16,17,25,26). For image modality, MoE-4x shows some training loss improvement (Figure 12-2, s = 0.691 at 163M; Figure 12-11, s = 0.595 at 760M) but its validation loss matching is worse than MoT (Figure 12-4, s = 0.914 vs. 0.286 at 163M; Figure 12-13, s = 0.948 vs. 0.247 at 760M). The paper attributes this to "the fact that Transfusion processes discrete text tokens and continuous image tokens, which complicates router generalization during inference" — a fundamental limitation of learned routing when modalities have different representational formats (discrete vs. continuous), not just different statistical distributions.

System-Level Results: Wall-Clock Time and Horizontal Scaling

Wall-clock time advantage (Figure 19). On 256 A100 GPUs (AWS p4de.24xlarge), the 7B MoT in the Chameleon setting matches the dense model's image training loss in 47.2% of the wall-clock training time (Figure 19b, s = 0.472), and continues to improve beyond that point. For text, MoT requires 75.6% of the dense model's training time (Figure 19d, s = 0.756). In contrast, MoE-4x shows a 1.7× slowdown in the image modality compared to dense (Figure 19b, s = 1.707) and no speed advantage for text (s = 1.070). Validation loss results (Figure 19e-h) are consistent: MoT matches dense image validation loss in 58.4% of wall-clock time and text validation loss in 74.9% of wall-clock time. The MoE-4x slowdown is attributed to the communication overhead of expert routing (the higher PpF ratio analyzed in Section 6.1) and the sequential dependencies in MoE operations that create GPU utilization bubbles. These wall-clock results are crucial because they demonstrate that MoT's FLOPs savings translate to real time savings in a realistic distributed training environment — the paper's reduction in training steps (e.g., 45.5% for Chameleon 7B) is not offset by per-step overheads.

Horizontal scaling analysis (Figure 18). In the Chameleon 443M setting, as GPU count scales from 16 to 256 (with global batch size and total training tokens scaled proportionally), MoT's relative advantage grows. For image validation loss on Obelisc: the percentage of training steps MoT needs to match the dense baseline decreases from 42.1% at 16 GPUs to 21.6% at 256 GPUs (Figure 18b,f,j,n). For text validation loss: from 75.7% at 16 GPUs to 50.9% at 256 GPUs (Figure 18d,h,l,p). This is a non-obvious and practically important finding: it suggests that MoT's lower Parameter-to-FLOPs (PpF) ratio (analyzed in Section 6.1) reduces communication overhead, and this benefit becomes more pronounced as distributed training scales to more GPUs where communication bandwidth is the bottleneck. MoE-4x also shows some improvement with GPU count (image: 84.6% → 48.8%; text: 88.9% → 53.1%), but consistently lags behind MoT at every GPU count. The paper notes this analysis was "conducted under specific AWS infrastructure conditions" and that "further investigation is needed to generalize these findings across different hardware configurations."

Fine-tuning Results (Transfusion 7B)

After fine-tuning on an internal visually appealing dataset and on image editing tasks (Section 3.4.4, Figure 13, Appendix B Figures 20-22), the 7B Transfusion MoT model demonstrates capabilities including text rendering ("GO BIG OR GO MOT" on a blackboard), detailed hand generation, fictional image generation (chrome-plated duck arguing with a turtle), photorealistic generation (corgi wearing a wizard hat), and instruction-based image editing (changing a stop sign to say "GO"). The paper reports that "after fine-tuning, MoT demonstrates better quality and higher faithfulness compared to the fine-tuned dense models (see Appendix B)." Figure 20 shows prompts where MoT outperforms dense (lychee-inspired spherical chair, anthropomorphic cheeseburger, chrome-plated duck), Figure 21 shows prompts where both are comparable (cow-head person in tuxedo, translucent pig, Rembrandt painting of a raccoon), and Figure 22 shows prompts where both struggle (avocado in therapy, crocodile made of water, graffiti hamster, espresso machine from human souls). The paper notes these models are trained on only 0.5 trillion tokens, significantly less than SOTA image generation models, and that "text faithfulness can greatly improve with extended training."

Ablation Studies and Robustness Checks

Modality untying in different transformer components (Section 3.5, Figure 14): Using a 880M Chameleon model (Table 5), four architectures are compared on Obelisc and Shutterstock validation losses: (1) dense baseline; (2) FFN-only untying; (3) FFN + attention QKV untying; (4) full MoT (FFN + attention + LayerNorm). FFN-only untying provides substantial gains, particularly for images — the paper reports this as the primary source of benefit, validating Lin et al. (2024)'s MoMA approach. Adding attention QKV untying provides an additional ~33.3% FLOPs saving for image modality and ~10% for text on the Obelisc held-out set relative to FFN-only untying. Adding LayerNorm untying on top has "negligible impact on evaluation performance." The paper notes that "the FLOPs savings from adding attention untying to feedforward untying are smaller than those from adding feedforward untying to the dense model," attributing this to the FFN accounting for a larger proportion of FLOPs given the context length (4096) and the FFN's role as a memory component where modality-specific parameters are particularly effective. The key non-obvious finding is that attention projection untying matters — prior work assumed it didn't — and that the benefit is larger for images (~33%) than text (~10%), suggesting that how modalities project into query/key/value space is more critical for spatially structured tokens than for discrete text tokens.

Leave-One-Out (LOO) modality separation analysis (Section 4, Figure 15): Using 443M models in the Chameleon+Speech setting, the paper compares the full three-tower MoT against three "two-tower" variants where two modalities share a tower and one gets its own (LOO-image: text+speech combined; LOO-text: image+speech combined; LOO-speech: text+image combined) and the dense baseline (all three in one tower). The results (Figure 15f-n) show: (1) combining modalities consistently degrades performance — every LOO variant performs worse in its combined modalities than full MoT; (2) the impact is non-reciprocal — LOO-speech (text+image combined) preserves some benefits for image (Figure 15j, LOO-speech loss close to MoT) but loses benefits for text (Figure 15g, LOO-speech loss close to dense); LOO-text (image+speech combined) preserves MoT's text gains but causes speech degradation; (3) isolation is beneficial — LOO-text achieves the lowest text loss (even better than full MoT, Figure 15g), LOO-image achieves the lowest image loss, and LOO-speech achieves the lowest speech loss, suggesting that complete isolation can sometimes outperform mixed processing for a specific modality. The practical takeaway is that separating modalities into dedicated towers eliminates gradient interference, with speech being the most fragile and benefiting most from isolation. This constitutes the primary mechanistic evidence for the paper's claim that "conflicting training dynamics in a dense transformer model (Figure 15) complicating optimization" — the LOO analysis directly demonstrates that combining modalities in shared towers degrades performance, and the degradation patterns reveal which modality pairings are most competitive.

Comparison with MoE-2x and MoE-3x at smaller scales (Figure 6): At 37M, 94M, and 443M scales in Chameleon, the paper compares MoT against MoE with 2, 3, and 4 experts. The trend is consistent: MoT outperforms all MoE variants on image modality across all scales. At 37M (Figure 6a-b): MoT at 26.0% of dense steps vs. MoE-2x at 79.4%, MoE-3x at 34.2%, MoE-4x at 49.6%. At 94M (Figure 6e-f): MoT at 19.9% vs. MoE-2x at 45.7%, MoE-3x at 44.9%, MoE-4x at 35.8%. At 443M (Figure 6i-j): MoT at 27.1% vs. MoE-2x at 63.5%, MoE-3x at 58.2%, MoE-4x at 49.0%. For text modality, MoE variants sometimes outperform MoT (e.g., at 37M MoE-3x at 79.5% vs. MoT at 54.6%; at 94M MoE-4x at 53.1% vs. MoT at 53.0%), but the differences are small and inconsistent. The takeaway is that varying the number of experts doesn't change the fundamental pattern: MoE works reasonably well for text but fails to match MoT's image modality speedups, and the gap between MoT and MoE grows with scale.

Hybrid MoT+MoE-4x architecture (Section 5, Figures 16, 17): As a proof-of-concept, the paper replaces the text FFN in MoT with MoE-4x experts while keeping the image tower as standard MoT. In the Chameleon 373M setting (Figure 16): "MoT + Text MoE-4x" requires 43.1% of dense steps for text training loss (vs. 61.0% for pure MoT and 73.1% for pure MoE-4x), while preserving image loss speedup (26.3% vs. 27.1% for pure MoT). Validation results show the hybrid achieves best text performance while maintaining comparable image performance. In the Transfusion 760M setting (Figure 17): the hybrid requires 50.4% of dense steps for text training loss (vs. 97.0% for pure MoT and 49.5% for pure MoE-4x), with image training loss at 17.1% (vs. 17.6% for pure MoT). Text validation on C4 and Wikipedia shows the hybrid achieving the best performance, and image generation metrics (CLIP, FID) remain comparable to pure MoT. The key finding is complementarity: MoT and MoE address different bottlenecks, and combining them yields additive benefits for text without sacrificing image quality. This is not a main result but establishes that MoT is a modular framework within which learned sparsity can be selectively deployed per modality.

Fine-tuning robustness (Section 3.4.4, Appendix B): After fine-tuning both 7B MoT and dense Transfusion models, MoT maintains its quality advantage. The paper reports that "the performance gain of MoT over the dense baseline is well maintained after fine-tuning" and provides qualitative examples in Figures 13 and 20-22. For quantitative evidence, Appendix Figure 20 shows example prompts where MoT fine-tuned model generates clearly better images than the dense fine-tuned model (e.g., "A photo of a person with the head of a cow, wearing a tuxedo and black bowtie"). This demonstrates that MoT's pre-training efficiency gains are not erased by fine-tuning — the better representations learned during pre-training transfer to downstream image quality. However, the paper notes that "text faithfulness can greatly improve with extended training," suggesting that the 0.5T token budget is not saturated and further scaling could widen the gap.

Multiple data distribution validation (Figures 5, 8, 12, 24, 25): Throughout all settings, validation losses are reported on multiple held-out datasets rather than a single benchmark. In Chameleon: Obelisc, MS-COCO, Flickr30k, and Shutterstock (4 datasets, Figure 5g-n). In Chameleon+Speech: Obelisc, MS-COCO, Flickr30k, SSTK plus LL60K and PPL30K for speech (6 datasets, Figure 8c-n). In Transfusion: C4, Wikipedia, CC12M, and MS-COCO for generation metrics (Figure 10e-h, Figure 12). The consistency of MoT's advantage across all these distributions — without any dataset-specific tuning — strengthens the claim that the efficiency gains are genuine and not an artifact of overfitting to a particular validation set.

Critical Assessment

The experiments in this paper are unusually comprehensive for an architecture paper: thirteen models trained from scratch across three settings, multiple scales per setting, systematic comparisons against multiple baselines, validation on multiple held-out datasets, wall-clock time measurements, horizontal scaling analysis, and qualitative fine-tuning results. However, several important claims warrant closer scrutiny to determine whether the experiments actually demonstrate what the paper asserts.

Claim 1: MoT matches dense performance with substantially fewer FLOPs (55.8% in Chameleon 7B, 37.2% in speech, etc.). Supported with qualification. The step-matching methodology (e.g., Figure 5b) provides clear evidence that MoT reaches equivalent training loss faster than the dense baseline. The validation loss results (Figure 5g-n) confirm that this training efficiency translates to held-out performance, not just faster overfitting. However, the step-matching ratios are computed by finding the point where the MoT curve crosses the dense model's final loss value — this measures how quickly MoT reaches the dense model's endpoint, but does not tell us whether MoT would saturate at a higher or lower asymptote if both were trained for longer. The curves in Figure 5a show MoT continuing to improve at 120k steps, while the dense model appears to be approaching a plateau. If MoT's asymptotic performance is higher, the step-matching ratio would underestimate the true efficiency gain (since MoT doesn't just reach the dense model's performance faster — it may eventually surpass it). Conversely, if MoT plateaus earlier, the ratio would overestimate. The paper does not train any model to saturation, so the asymptotic behavior is unknown. This is a significant uncertainty: the 55.8% figure could be 40% or 70% depending on where the dense model's loss curve would eventually land. The validation loss results showing MoT at 55.8% checkpoint matching dense final validation loss partially addresses this (since validation loss is less prone to overfitting), but the concern remains for the step-matching methodology applied to training loss.

Claim 2: MoT consistently outperforms MoE-4x, especially on non-text modalities, and MoE's advantages diminish with scale. Supported with a major caveat. The empirical evidence for MoE-4x's underperformance is strong (Figures 5-9, 12), but the Expert Choice routing used for MoE evaluation introduces a known confound: the router can access future tokens during validation, which may overestimate MoE's performance (Section 3.2.1). If EC routing is overestimating MoE-4x's validation performance and MoE-4x still underperforms MoT, then the gap is even larger than reported — this makes the claim conservative. However, the paper does not implement a causally valid inference routing for MoE-4x (e.g., using token choice routing or a post-hoc routing predictor), which means we cannot know the true inference performance of MoE-4x. The paper's choice to "evaluate all models using the same EC routing as during training, focusing exclusively on validation perplexity" is pragmatic, but it means the MoE-4x vs. MoT comparison is not on fully equal footing — MoE-4x has an unfair advantage (future token access) and still loses. The paper acknowledges this transparently, but readers should understand that the reported MoE-4x performance is likely an upper bound, making MoT's advantage conservative but also making the exact magnitude of the advantage uncertain.

A subtler concern: MoE-4x has more total parameters than MoT (since it adds E-1 expert FFNs per layer), even though the activated parameters per token are matched. The paper argues that this should benefit MoE-4x (more capacity), making MoT's outperformance more impressive. But in practice, more total parameters can also mean more communication overhead in distributed training (higher PpF), which could hurt wall-clock performance independently of model quality. The paper separates these effects (training steps are FLOPs-controlled; wall-clock time is measured separately in Figure 19), but for the training loss comparisons, the additional parameters could be a confound if they affect optimization dynamics (e.g., more parameters = more capacity but harder to optimize with the same learning rate). The paper does not tune MoE-4x hyperparameters separately — it uses the same training recipe as dense and MoT — so suboptimal MoE-4x tuning cannot be ruled out as a contributing factor to its underperformance.

Claim 3: A 760M MoT model outperforms a 1.4B dense baseline on image metrics. Supported, with a sample size caveat. Figure 11 shows 760M MoT outperforming 1.4B dense on CLIP (0.214 vs. 0.206), FID (21.145 vs. 24.688), and CIDEr (0.320 vs. 0.286). These are meaningful gaps — the FID improvement is substantial (3.5 points). However, the comparison is based on a single training run per model. Without multiple seeds, we cannot assess whether the gap is larger than training variance. FID is known to be sensitive to the specific set of generated images — 30,000 prompts is a reasonable sample size, but the variance of FID estimates across different generation runs is not reported. The CIDEr improvement (0.320 vs. 0.286, +11.9%) is notable, but CIDEr is computed on the Karpathy test split of MS-COCO, which is a fixed set of ground-truth captions, so the metric variance comes from the model's generation stochasticity, not the evaluation set. The paper does not report multiple generation runs or confidence intervals for any of these metrics.

Claim 4: MoT's wall-clock advantages are substantial (47.2% of dense time for image quality). Supported but environment-specific. Figure 19 provides clear evidence that MoT achieves dense-matching image quality in less wall-clock time on 256 A100 GPUs. However, the paper explicitly notes that these results were "obtained using a specific AWS setup" and that "we expect the relative performance of MoE, MoT, and dense models to vary across different clusters." The wall-clock results depend on the communication topology, GPU interconnect bandwidth, and implementation efficiency of the modality grouping/reassembly operations. The paper reports that "we did not observe these overheads on the critical path in our training setup," but this may not generalize to other hardware configurations — clusters with slower CPU-GPU synchronization (since MoT requires grouping tokens by modality, which can involve CPU-GPU transfers) or different GPU topologies could see larger overheads. The paper's claim that MoT's overheads "can be minimized via diligent engineering" is forward-looking; the current results demonstrate that it's possible to achieve these speedups in one specific environment, not that it's trivial or universal.

Claim 5: MoT's benefits increase with GPU count (horizontal scaling). Provocative but preliminary. Figure 18 shows that MoT's relative advantage grows from 16 to 256 GPUs (image validation loss matching drops from 42.1% to 21.6% of dense steps). This is a compelling trend, but it's based on a single model scale (443M) and a single setting (Chameleon). The paper frames this as evidence that MoT's lower PpF ratio reduces communication overhead at scale, which is mechanistically plausible. However, the experiment confounds GPU count with total batch size and total training tokens — all three are scaled proportionally. It could be that the larger batch sizes (made possible by more GPUs) are what benefit MoT, rather than the increased GPU count per se. An ablation that keeps batch size constant while varying GPU count (using gradient accumulation) would be needed to isolate the scaling variable, but this is not performed. Additionally, the highest GPU count tested (256) is modest by modern standards — whether the trend continues at 512, 1024, or more GPUs (where communication bottlenecks become even more severe) is an open question.

What experiments would have strengthened the paper:

• Multiple training runs with different random seeds to establish variance for the step-matching ratios, validation losses, and generation metrics. All reported results are from single training runs, making it impossible to assess whether observed differences (e.g., CLIP 0.214 vs. 0.206) are statistically reliable.

• Training to convergence (or close to it) for at least one scale in one setting, to establish whether MoT's asymptotic performance differs from the dense baseline. The paper only trains to fixed step budgets and compares at those points. If MoT saturates earlier or later than dense, the step-matching methodology could be misleading.

• A causally valid MoE inference baseline — implementing token-choice routing or a trained routing predictor for MoE-4x inference, to enable a fair comparison that doesn't give MoE an unfair advantage (future token access) or disadvantage (EC routing on OOD data). This would make the MoT vs. MoE comparison fully rigorous.

• Scaling to larger models and token budgets. The largest model is 7B trained on 0.377T tokens (Chameleon) or 0.524T tokens (Transfusion). For context, state-of-the-art multi-modal models train on trillions of tokens. The paper's findings might not extrapolate — modality competition effects might change as models approach saturation, and MoE's routing might stabilize with more training. The paper acknowledges this implicitly by noting that the Transfusion model is "not yet saturated even at 2 trillion tokens" (Zhou et al., 2024).

• Direct measurement of gradient interference between modalities in the dense baseline, to provide mechanistic evidence for the "modality competition" hypothesis. The PCA visualizations (Figure 2, Appendix Figure 23) show that modalities occupy distinct regions of feature space, but this is correlational — it doesn't demonstrate that gradients from different modalities conflict. Gradient cosine similarity measurements between modality-specific losses would directly test the paper's central mechanistic claim.

• Training MoT with more than three modalities (e.g., video, code, music) to test the claim that the approach generalizes beyond text, images, and speech. The paper's framework (Section 2.2) is presented as a general architecture for any set of modalities, but only three are evaluated.

• Ablation of the shared embedding layer — what happens if embeddings are also modality-specific? The paper keeps embeddings shared without justification, and an ablation showing whether this matters would clarify the boundary of what should and shouldn't be decoupled.

Where the claims hold conditionally:

• MoT provides the largest speedups for image and speech modalities (2-5× training acceleration), moderate speedups for text in Chameleon (~1.5-2×), and marginal-to-no speedup for text in Transfusion (where objective decoupling already provides similar benefits). The claim "MoT reduces pretraining computational costs" is therefore modality-dependent — it's most impactful when the dense baseline is most inefficient (processing spatially or temporally structured data with the same parameters as discrete text).

• MoT outperforms MoE-4x consistently for image and speech, but only sometimes for text. For text in Chameleon, MoT and MoE-4x are comparable (Figure 6d,h,l,p,t). For text in Transfusion, MoE-4x achieves lower training loss but worse validation loss (a generalization failure). The claim that "MoT outperforms MoE" needs to be qualified by modality.

• The 760M MoT > 1.4B dense result (Figure 11) holds for image generation quality metrics (CLIP, FID, CIDEr) but not for text metrics, where MoT and dense are comparable. The claim that a smaller MoT model "outperforms a larger dense model" is true specifically for image-related tasks in the Transfusion setting.

• The horizontal scaling trend (Figure 18) is demonstrated for one model scale (443M) and one setting (Chameleon). Generalization to other scales, other settings, and larger GPU counts is plausible (given the PpF analysis) but not empirically established.

• All wall-clock results are hardware-specific (AWS p4de.24xlarge, NVIDIA A100 GPUs, 256 GPUs) and the paper explicitly cautions against overgeneralization. A reader deploying on a different cluster should treat the absolute numbers as illustrative.

Despite these qualifications, the paper's central empirical contribution — that deterministic modality-specific parameter partitioning provides substantial training efficiency gains over both dense and MoE baselines in multi-modal settings — is robustly supported across the range of experiments conducted. The consistency of the finding across three settings, multiple scales, multiple validation datasets, and both training and wall-clock metrics makes it unlikely to be an artifact. The main open questions concern extrapolation (to larger models, longer training, more modalities) and precise quantification (variance, asymptotics), which the paper appropriately flags as future work.

6. Limitations and Trade-offs

The Cost of Difficulty Estimation Is Not Included in the Headline Efficiency Gains

The assumption or constraint. The paper's central efficiency claims — MoT matches dense performance in 55.8% of the FLOPs for Chameleon 7B, 37.2% for speech, etc. — assume that the modality of each token is known at zero cost. However, in a true early-fusion setting where modalities are mixed within a single interleaved sequence, the model must know which token belongs to which modality in order to route it to the correct tower. The paper implicitly relies on this information being available from the data preprocessing pipeline, but never accounts for the cost of determining or storing modality labels, nor does it consider what happens when modality boundaries are ambiguous (e.g., tokens that could represent either text or structured data, or sequences where the modality of a generated token is not known in advance during autoregressive decoding).

The consequence. In practice, the modality of each token must be tracked throughout the data pipeline — stored alongside each token, passed through batching and shuffling, and available at every transformer layer for the grouping operation (Algorithm 1, lines 3-5). While this is straightforward for the training settings studied (where text, image, and speech tokens are discretized by separate tokenizers and clearly labeled), it introduces a dependency on accurate modality metadata that may not exist in all multi-modal settings. More importantly, during autoregressive generation, the model must decide which modality a newly generated token belongs to in order to route it through the correct tower for subsequent layers. The paper does not address how this routing decision is made during inference — is it based on a special "modality switch" token? Is the modality of generated tokens predetermined by the task? For the Chameleon setting, where images are generated as 1,024 discrete tokens following a beginning-of-image (BOI) token, the modality is known from context. But for more open-ended multi-modal generation scenarios where the model can freely intermix modalities, the routing mechanism would need to predict modality before processing each token — a chicken-and-egg problem that the paper does not discuss.

What evidence exists in the paper. None. The paper does not analyze the cost, latency, or accuracy of modality determination in any setting. The efficiency numbers (55.8%, 37.2%, etc.) are computed purely from training FLOPs, and the wall-clock measurements (Figure 19) assume modality labels are available instantaneously. Section 6.1 mentions that "caching sequence indices for each modality can substantially reduce indexing costs and needs to be done only once per iteration, as the modalities of tokens do not change," but this addresses the computational overhead of the grouping operation within a layer, not the data pipeline cost of providing modality labels for every token.

Mitigation status. The paper does not acknowledge this as a limitation. It treats modality labels as freely available metadata, which is reasonable for the studied settings (text and images come from separate tokenizers; speech from yet another) but is not guaranteed for all multi-modal architectures. For deployment, a system would need to either (a) rely on special delimiter tokens to indicate modality switches (as Transfusion does with BOI/EOI tokens), or (b) predict modality from token embeddings, which adds a routing step that undercuts MoT's claim of zero learned routing overhead.

---

All Results Are from a Single Model Family with Identical Training Hyperparameters Across Architectures

The assumption or constraint. Every experiment in the paper uses transformer models trained from scratch with the same hyperparameters across all architectures (dense, MoT, MoE-4x) at each scale and setting. The learning rate, batch size, optimizer settings, warmup schedule, weight decay, gradient clipping, and training data are held constant — the only thing that varies is the architecture. The paper explicitly states this in the Transfusion setup: "We randomly initialize all model parameters, and optimize them using AdamW (β1 =0.9, β2 =0.95, ε =1e-8) with a learning rate of 3e-4, warmed up for 4000 steps and decaying to 1.5e-5 using a cosine scheduler" (Section 3.4.1). Similar fixed recipes are used for Chameleon (Table 1) and Chameleon+Speech (Table 3). At no point does the paper sweep learning rates, batch sizes, or other hyperparameters for any architecture.

The consequence. This creates a potential confound: MoT's apparent efficiency advantage could be partially or fully attributable to the dense baseline and MoE-4x being trained with suboptimal hyperparameters. MoT's modality-specific towers have effectively independent optimization dynamics per modality — each modality's parameters receive gradients only from tokens of that modality, meaning the effective learning rate per parameter, the gradient statistics, and the optimal regularization could differ. The paper's use of a single hyperparameter set for all architectures means MoT may be closer to its optimal training regime by construction (since its parameters are specialized and less subject to gradient interference), while the dense model's shared parameters may require a different learning rate or regularization to handle the multi-modal gradient mix. This is the classic "no free lunch" problem in architecture comparisons: the dense model might perform better under a different hyperparameter configuration, but we can't know because no sweep was performed.

Similarly, MoE-4x is known to be sensitive to auxiliary loss coefficients (for load balancing), expert capacity factors, and routing initialization — but the paper freezes these at a single configuration. The Expert Choice routing used (Zhou et al., 2022) has its own hyperparameters (capacity factor, number of tokens per expert) that are not reported or swept. If MoE-4x underperformance is partly due to suboptimal routing configuration, the MoT vs. MoE comparison overstates MoT's relative advantage.

What evidence exists in the paper. The consistency of results across multiple scales (37M to 7B) and three settings provides some robustness — if the dense or MoE baselines were merely poorly tuned at one scale, we might expect the advantage to disappear at another scale. The fact that MoT's advantage persists (and even grows with scale for images; Figure 6) suggests the effect is not purely a hyperparameter artifact. However, the paper never reports learning rate sensitivity, and in the Chameleon 1.5B setting, MoT's text speedup (66.2% of dense steps; Figure 6p) is slightly worse than at 443M (61.0%; Figure 6l), suggesting some scale-dependent variation in the relative advantage that could be influenced by hyperparameters. The horizontal scaling analysis (Figure 18) shows MoT's advantage changing with batch size (which varies with GPU count), further suggesting that training configuration affects the measured efficiency.

Mitigation status. The paper does not acknowledge this as a limitation. No hyperparameter sweep is performed for any architecture in any setting. The practical consequence is that the reported speedup numbers (55.8%, 37.2%, etc.) should be interpreted as achievable acceleration under the specific training recipe used, not as an intrinsic property of the MoT vs. dense architectures. A practitioner tuning hyperparameters for their specific deployment might find the gap either smaller or larger than reported.

---

The 7B Scale Is the Largest Model Tested; Extrapolation to Larger Models and Longer Training Is Unsupported

The assumption or constraint. The paper's largest model is 7B parameters, trained on 0.377 trillion tokens (Chameleon), 0.377 trillion tokens (Chameleon+Speech), or 0.524 trillion tokens (Transfusion). For context, contemporary multi-modal foundation models are substantially larger and trained on far more data: the paper itself notes that "Chameleon is trained on 9.2 trillion training tokens" (Section 1) and that the Transfusion model is "not yet saturated even at 2 trillion tokens" (Section 3.4.4). All of the paper's scaling trends — MoT's growing advantage with scale for images, MoE-4x's diminishing returns, the horizontal scaling behavior — are established over a range of 37M to 7B parameters. The paper explicitly states: "With this setup, we focus on evaluating the relative performance of the proposed architecture and the baseline at various FLOPs budgets, rather than conducting a scaling law study" (Section 3.2.1, footnote).

The consequence. Several of the paper's most important claims depend on scaling behavior that may not extrapolate. The finding that MoE-4x's image advantage "diminishes" and "disappears" at 7B (Section 3.2.3, Figure 6q-r) is based on a trend from 37M to 7B. At 7B, MoE-4x requires 101.2% of dense steps to match image loss, having degraded from 49.6% at 37M. This is a clear downward trend, but it's possible that at 70B or 700B parameters, with proportionally more training data, MoE's learned routing stabilizes and recovers its advantage. Similarly, MoT's image speedup of ~3× at 7B is slightly smaller than the ~4-5× seen at 94M and 37M (Figure 6), suggesting the relative advantage might be eroding with scale — if this trend continues, MoT could provide progressively smaller gains at 70B and beyond.

The Transfusion setting is particularly sensitive to this limitation. The paper reports impressive results at 760M (outperforming 1.4B dense) and 7B (matching dense image quality in ~30% of steps), but the text modality shows "marginal to none" improvement at 7B. The paper hypothesizes that Transfusion's objective decoupling already provides most of the text benefit. If this hypothesis is correct, then at larger scales and longer training, MoT's text advantage in Transfusion might not just be marginal — it could be negative (MoT slightly worse than dense) if the modality towers don't benefit from the cross-modal gradient signal that the dense model receives. The paper provides no evidence either way.

What evidence exists in the paper. The scaling trends from 37M to 7B (Figure 6) are the paper's basis for claiming that MoT's advantages persist and MoE's disappear. The trend in image modality speedup for MoT is: 26.0% (37M) → 19.9% (94M) → 27.1% (443M) → 35.0% (1.5B) → 34.8% (7B). This is not monotonically improving — it fluctuates and then stabilizes around 30-35%. For text: 54.6% → 53.0% → 61.0% → 66.2% → 55.8%. Again, not a clear scaling trend. The paper does not fit any scaling law (power law or otherwise) to predict behavior at larger scales. The horizontal scaling analysis (Figure 18) extends to 256 GPUs with a 443M model, but this is a single data point on a completely different scaling axis (distributed training scale, not model scale).

Mitigation status. The paper acknowledges this implicitly through its statement about "not conducting a scaling law study," but does not discuss the specific risks of extrapolating the 7B trends to the 70B+ models that would be deployed in practice. The "Future Work" section (Section 8) mentions scaling up the Transfusion model with more data, but this is presented as an opportunity rather than as a necessary validation of the current results. A practitioner deciding whether to adopt MoT for a 70B+ multi-modal model has no direct empirical evidence from this paper to guide that decision.

---

MoT's Text Modality Gains Are Inconsistent and Context-Dependent, Limiting Its Applicability to Language-Heavy Multi-Modal Tasks

The assumption or constraint. The paper presents MoT as a general architecture for multi-modal pretraining that "significantly reduces pretraining computational costs" (Abstract). However, the efficiency gains are highly modality-asymmetric. In Chameleon, MoT reduces image training FLOPs by ~3× (34.8% of dense steps at 7B) but text FLOPs by only ~1.8× (55.8%). In Transfusion, the text advantage drops to essentially zero — MoT requires 97.0% of dense steps to match text training loss at 760M (Figure 26-7) and shows "marginal to none" improvement at 7B (Section 3.4.2). The paper hypothesizes that this is because Transfusion already decouples objectives by modality, but the consequence is that in the setting with the best image generation quality (Transfusion), MoT provides no text speedup at all. A practitioner training a multi-modal model where text generation quality is the primary concern (e.g., a visual question-answering system, a document understanding model) would see substantially less benefit from MoT than the headline numbers suggest.

The consequence. MoT's efficiency advantage is fundamentally tied to modalities where the dense baseline's uniform processing is most wasteful — images (spatial structure, large token sequences), speech (temporal redundancy, different time scales), and diffusion-based image generation (heavy compute). For text — which is typically the most important modality in multi-modal LLMs, often dominating evaluation benchmarks and downstream applications — MoT provides either modest gains (Chameleon) or none (Transfusion). This means that the overall training cost reduction for a multi-modal model depends heavily on the modality mix in the training data. If text tokens constitute 50% of the training data, and MoT provides no text speedup, the effective overall speedup is half the image speedup. The paper's "55.8% of FLOPs" figure for Chameleon 7B reflects roughly equal text and image tokens; if the token mix were 80% text / 20% images, the effective speedup would be substantially smaller.

Furthermore, the paper's finding that MoT text speedup disappears in Transfusion raises an uncomfortable question: is MoT's text advantage in Chameleon actually a result of fixing a specific pathology of that setting (gradient interference between autoregressive text and autoregressive image objectives), rather than a fundamental benefit of modality-specific parameters for text? If so, then in any multi-modal setting with well-designed objectives (separate losses, careful gradient management), MoT may provide no text benefit at all. The paper does not investigate this systematically.

What evidence exists in the paper. The text speedup numbers are reported transparently throughout: Chameleon 7B at 55.8% (Figure 6t), Chameleon text across scales at 54.6-66.2% (Figure 6d,h,l,p,t), Transfusion 7B at ~97% or worse (Figure 26). The paper acknowledges the Transfusion text result: "MoT shows little improvement on text across the scales" (Section 3.4.3) and "the text performance improvement of MoT in the Transfusion setting was less pronounced compared to the Chameleon setting" (Section 3.4.2). The LOO analysis (Figure 15) shows that isolating text in its own tower (LOO-text configuration) achieves the best text loss, suggesting that text does benefit from separation — but this is in the Chameleon+Speech setting, where text competes with two other modalities. In the two-modality Transfusion setting, that competition appears reduced by objective decoupling.

Mitigation status. The paper acknowledges the text limitation ("we acknowledge the need for further investigation into text performance and plan to explore additional modifications or hybrid strategies (e.g., integrating MoE elements selectively) in future work"; Section 3.4.2) and provides the hybrid MoT+MoE-4x experiment (Section 5) as a partial solution — using learned sparsity in the text tower to recover text speedups without sacrificing image efficiency. However, this hybrid adds back the complexity (learned routing, load balancing) that MoT was designed to avoid, somewhat undermining the simplicity argument for MoT. A practitioner who needs strong text performance would need to implement this hybrid, which is more complex than pure MoT and not validated at scale (only tested at 373M in Chameleon and 760M in Transfusion).

---

The Architecture Assumes Clear Modality Boundaries and Does Not Handle Mixed-Modality Tokens or Fine-Grained Cross-Modal Fusion

The assumption or constraint. MoT's fundamental design choice is to route each token to exactly one modality-specific tower based on a discrete modality label (text, image, speech). This assumes that (a) every token belongs unambiguously to exactly one modality, (b) modalities are sufficiently distinct that separate processing parameters are beneficial, and (c) the token-level modality assignment is sufficient for cross-modal interaction — i.e., cross-modal information flow only needs to happen through the global self-attention, not through shared processing within a token. These assumptions hold for the settings studied (discretely tokenized images, discretely tokenized speech, text), but they break down in several important scenarios that the paper does not address.

The consequence. Consider a multi-modal model that processes: (a) continuous image tokens (as in Transfusion) — these are not discrete, and the "boundary" between image patches and surrounding text tokens is a design choice, not an inherent property; (b) interleaved text and structured data (e.g., tables, code, JSON) — these share representational properties with both text and structured modalities, and forcing them into a single tower might lose the benefits of specialization; (c) fine-grained visual grounding — where a text token like "the red ball" should be processed with awareness of both linguistic structure and the visual features it refers to, potentially requiring a single token to activate both text and image processing; (d) emergent modalities — if the model learns to represent concepts that don't cleanly fit into "text," "image," or "speech" (e.g., a generated diagram that combines text and visual elements at the token level).

In all these scenarios, MoT's hard modality assignment forces each token into exactly one processing tower, preventing the kind of fine-grained, per-token fusion that might be optimal. The paper implicitly acknowledges this limitation by not testing any setting with ambiguous modality boundaries. Even within the Transfusion setting, the continuous image tokens are treated as a distinct "modality" separate from text, but it's unclear whether this binary separation is optimal — perhaps some image patches (e.g., ones containing text) would benefit from partial text-tower processing, or vice versa. MoT provides no mechanism for such soft or partial routing.

This limitation is particularly salient because the paper's own PCA analysis (Figure 2, Appendix Figure 23) shows that modality separation emerges during training — the feature space clusters by modality. MoT hardcodes this separation into the architecture from initialization, which might prevent the model from discovering useful cross-modal representations that lie between the modality clusters. If the optimal representation for some tokens blends text-like and image-like features, MoT's architecture precludes that.

What evidence exists in the paper. None directly. The LOO analysis (Section 4, Figure 15) shows that forcing modalities to share towers degrades performance, which supports the claim that separate towers are better than fully shared processing. But the LOO analysis only tests extreme cases — complete sharing vs. complete separation of whole modalities — not the intermediate case where a token might route partially to multiple towers. The paper does not ablate the effects of soft routing (e.g., a learned interpolation between modality towers), multi-tower per token (processing a token through text AND image towers and combining outputs), or dynamic modality assignment (allowing tokens to change modality assignment mid-network).

Mitigation status. The paper does not acknowledge this limitation. The architecture is presented as generally applicable to "any set of modalities" (Section 2.2), and the fixed modality assignment is treated as a feature (simplicity, no learned routing) rather than a constraint. The hybrid MoT+MoE experiment (Section 5) shows that the architecture can be extended with learned sparsity within a modality, but this doesn't address the fundamental issue of hard modality boundaries. Future work on "soft" modality routing — where each token's tower assignment is a learned, continuous weighting rather than a hard binary choice — could address this, but would reintroduce the routing complexity that MoT avoids.

---

MoT's Wall-Clock Advantages Are Hardware-Specific and the Reported Overheads May Not Generalize

The assumption or constraint. The paper's wall-clock time measurements (Section 6.2.2, Figure 19) and horizontal scaling analysis (Section 6.2.1, Figure 18) are obtained on AWS p4de.24xlarge instances with NVIDIA A100 GPUs, using Fully Sharded Data Parallel (FSDP) in full shard mode with PyTorch 2 Compiler. The paper explicitly states: "our results were obtained using a specific AWS setup, specified above. Therefore, we expect the relative performance of MoE, MoT, and dense models to vary across different clusters" (Section 6.2.2). The wall-clock results depend on the balance between computation time (FLOPs) and communication time (gradient synchronization, parameter sharing), which is hardware-specific. MoT has lower communication volume (lower PpF ratio; Section 6.1) but introduces modality grouping/reassembly operations that require CPU-GPU synchronization. Whether MoT is faster in wall-clock time depends on whether the communication savings outweigh the grouping overheads, and this tradeoff changes with GPU interconnect bandwidth, CPU-GPU transfer speed, and the specific implementation of the grouping logic.

The consequence. A practitioner deploying MoT on a different cluster — e.g., with NVIDIA H100 GPUs (different FLOPs/bandwidth ratio), with a different parallelism strategy (tensor parallelism, pipeline parallelism instead of pure FSDP), or with a different modality mix (highly imbalanced batch sizes across modalities) — may see different wall-clock speedups than reported, potentially including slowdowns relative to the dense baseline. The paper's own analysis identifies MoT's overheads: "First, the CPU-GPU synchronization required for grouping tokens by modality for element-wise projections and reassembling them for attention results in significant overhead, mostly attributed to frequent GPU-CPU synchronization due to masking for specific modalities. Second, the sequential processing of modalities can also lead to underutilization of GPU resources and imbalance, particularly when tokens of different modalities are unevenly distributed across local batches and GPUs" (Section 6.1). These are potential failure modes that could make MoT slower than dense in certain configurations, but the paper reports that "we did not observe these overheads on the critical path in our training setup" — a statement that applies only to the specific AWS environment tested.

The imbalance issue is particularly concerning for practical deployment. In a typical multi-modal training batch, the number of text tokens, image tokens, and speech tokens can vary dramatically (e.g., a batch might have 10,000 text tokens but only 500 speech tokens). MoT processes each modality sequentially in separate GEMM operations (Algorithm 1, lines 6 and 11-15), meaning the GPU will be underutilized during the speech processing (few tokens, small matrix multiply) and heavily utilized during text processing. This load imbalance across the sequential modality steps can create GPU idle time that offsets the communication savings. The paper acknowledges this as a potential issue but does not measure it for imbalanced batches.

What evidence exists in the paper. Figure 19 shows MoT achieving wall-clock speedups on 256 A100 GPUs, and Figure 18 shows that MoT's advantages grow with GPU count. The paper notes that MoT's overheads "can be minimized via diligent engineering" (Section 6.1), citing Grouped GEMMs and Megablock-style block sparse matrix multiplication as potential solutions, but none of these optimizations are implemented or evaluated in the paper. The reported wall-clock results are from a baseline implementation with PyTorch 2 Compiler, without specialized grouped matrix multiply kernels. The paper provides no ablation showing wall-clock time with and without these optimizations, making it impossible to assess how much of the reported efficiency is due to the architecture vs. the implementation quality.

Mitigation status. The paper is transparent about the hardware-specificity of the wall-clock results and explicitly warns against overgeneralization. However, it does not provide any guidance on how to predict wall-clock performance on different hardware — e.g., a simple analytical model based on FLOPs, communication volume, and GPU characteristics. The suggestion that specialized GEMM kernels can mitigate overheads is forward-looking but untested. A practitioner would need to profile MoT on their specific hardware with their specific modality distribution to determine whether the wall-clock advantages materialize. The paper's training step matching results (which are hardware-independent) provide a more reliable basis for decision-making, but the training step advantage only translates to wall-clock advantage if the per-step overhead of MoT relative to dense is small — and that overhead is hardware-specific.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper reframes a core design assumption in multi-modal architecture research: that learned sparsity through Mixture-of-Experts routing is the primary path to efficient multi-modal training. The systematic evidence that deterministic, rule-based routing by modality outperforms learned routing across all non-text modalities, with the gap widening at scale, is a genuinely disruptive finding. It doesn't just propose an incremental improvement to MoE — it demonstrates that the central technical challenge MoE was designed to solve (how to route tokens to specialized parameters) has a trivially simple answer in multi-modal settings: route by modality type, which is known from the data pipeline.

The magnitude of this shift is substantial but bounded. MoT doesn't make MoE obsolete — the hybrid experiments in Section 5 show that learned routing can still help within the text modality, and the paper explicitly positions MoT and MoE as complementary rather than competing. However, the finding that MoE-4x's image advantage degrades from ~2× at 37M to nothing at 7B (Figure 6r, s = 1.012), while MoT maintains a ~3× speedup, forces a reconsideration of where engineering effort should be directed. The field has invested heavily in solving MoE's routing pathologies — load balancing, training instability, inference-mode mismatch — under the implicit assumption that learned routing is necessary. This paper provides strong evidence that for the specific case of modality heterogeneity, architectural sparsity (MoT) is not just simpler but better, freeing researchers from having to solve MoE's problems in multi-modal contexts.

The paper also reconciles a tension in prior work that was hiding in plain sight. VLMO (Bao et al., 2022b), MoMA (Lin et al., 2024), and related approaches applied modality-specific sparsity only to FFN layers, leaving attention projections shared. Their results were promising but left unexplained gains on the table. MoT's ablation (Section 3.5, Figure 14) shows that the missing piece — attention projection untying — provides an additional ~33% FLOPs saving for images. This doesn't contradict prior work; it completes it, establishing that the principle of modality-specificity applies broadly across the transformer stack and that the FFN-only scope of earlier approaches was unnecessarily conservative.

A more subtle shift concerns the diagnosis of modality competition. The Leave-One-Out analysis (Section 4, Figure 15) provides mechanistic evidence that different modalities actively interfere with each other's gradients when sharing parameters — speech degrades when combined with image, text degrades when combined with speech, and the effects are non-reciprocal. This reframes the multi-modal training problem from "more data is needed" to "modalities compete for parameter capacity in predictable, pairing-specific ways." The consequence is that future multi-modal architectures should be designed with an explicit theory of which modalities can share parameters and which must be separated, rather than defaulting to either fully shared or fully separated extremes.

Research directions that become more attractive:

• Hardware-aware architecture design that treats modality separation as a systems optimization lever. MoT's lower PpF ratio (Section 6.1) and growing advantage with GPU count (Figure 18) suggest that the architecture is well-suited to distributed training regimes where communication bandwidth is the bottleneck. This opens a line of work on co-designing modality partitioning strategies with the specific communication topology of the training cluster — e.g., colocating modality towers that interact heavily on the same GPU while separating ones that don't.

• Gradient interference analysis for modality pairs. The LOO analysis establishes that some modality pairings are more competitive than others, but doesn't provide a predictive theory. Directly measuring gradient cosine similarity between modality-specific losses in a shared-parameter model, and correlating that with the performance degradation observed in LOO configurations, would turn the qualitative finding into a quantitative design tool — enabling architects to predict which modalities should share towers before training.

• Learned sparsity within, deterministic separation across. The hybrid MoT+MoE-4x results (Section 5, Figures 16-17) are preliminary but suggest a clean division of labor: separate modalities deterministically, then apply learned sparsity within each modality-specific tower. This is more principled than current MoE approaches that force a single router to handle all modality heterogeneity and all within-modality specialization simultaneously.

Research directions that become less attractive:

• Improving MoE routing mechanisms specifically for multi-modal data. If modality-based routing already solves the across-modality sparsity problem better than any learned router, then efforts to design better multi-modal MoE routers (better load balancing, better initialization, modality-aware routing) are addressing a problem that has a simpler architectural solution. The paper's evidence suggests these efforts would at best match MoT's performance (for text) or fall short (for images and speech).

• Fully shared architectures for early-fusion multi-modal models. The paper provides convergent evidence — PCA clustering (Figure 2), training loss speedups (Figures 5-12), the LOO analysis (Figure 15) — that full parameter sharing across modalities is fundamentally inefficient. Future multi-modal foundation models should default to some form of modality-specific parameter allocation, with full sharing requiring specific justification rather than being the baseline assumption.

Follow-Up Research This Work Enables

Extending MoT to 70B+ scale models with 2T+ tokens to test the extrapolation of efficiency gains. The paper's largest model is 7B trained on 0.377–0.524T tokens, while state-of-the-art multi-modal models train on trillions of tokens. The scaling trends for MoT's image modality speedup (26% → 20% → 27% → 35% → 35% from 37M to 7B; Figure 6) are not monotonic, and for text (55% → 53% → 61% → 66% → 56%) are similarly noisy. A scaling law study at 7B, 13B, 34B, and 70B with matched training data would establish whether the speedup asymptotes, degrades, or improves, and at what model size MoT's advantages saturate. This is critical for deployment decisions: if MoT's image speedup shrinks to ~1.5× at 70B, the architectural complexity may not be justified. Conversely, if the speedup grows (consistent with the horizontal scaling trend in Figure 18), MoT becomes increasingly attractive for frontier models. A strong follow-up would train MoT and dense models at 7B, 34B, and 70B on Chameleon data, fit power laws to the step-matching ratios, and predict the speedup at 400B parameters.

Measuring gradient interference between modalities to validate and quantify the modality competition hypothesis. The paper's central mechanistic claim — that modalities compete for shared parameters, and MoT fixes this — is supported only indirectly through LOO configuration loss comparisons and PCA visualizations. A direct test would: (1) Train a dense Chameleon model and, at regular intervals, compute the cosine similarity between the gradient of the text loss and the gradient of the image loss with respect to the shared FFN and attention parameters. (2) Determine whether higher gradient conflict (negative cosine similarity) correlates with slower convergence, and whether the conflict is concentrated in specific layers or specific parameter types. (3) Compare the gradient conflict in a dense model to the effective gradient alignment in a MoT model (where text and image parameters are separate, so conflict is zero by construction). This would transform the modality competition claim from an inference based on loss curves to a directly measured optimization phenomenon, and would reveal whether certain modality pairings (e.g., text+speech vs. text+image) have systematically different interference patterns that could inform architecture design.

Soft modality routing: testing whether continuous, learned modality-specificity outperforms hard assignment. MoT forces each token to be processed by exactly one modality tower. This is simple but may be suboptimal for borderline cases — e.g., image patches containing text, or tokens representing cross-modal concepts. A natural extension would be a "soft MoT" where each token computes a learned, continuous weighting over the modality towers (via a small learned router that outputs a softmax over K modalities), and the token's representation is a weighted combination of the outputs from all towers. The key question is whether soft routing recovers MoT's efficiency (by allowing the router to learn hard assignment when beneficial) while improving performance on ambiguous tokens. A strong experiment would compare hard MoT, soft MoT, and dense on a multi-modal benchmark with fine-grained cross-modal tasks (e.g., visual question answering where the answer depends on specific image regions and text reasoning). If soft routing matches hard MoT's training speed while improving downstream task performance, it would establish that the value of modality-specificity is in the capacity to specialize, not necessarily in the forced specialization, and would open a middle ground between deterministic routing and learned routing.

Systematic evaluation of MoT on modality-imbalanced training data. All experiments in the paper use roughly balanced modality mixtures (1:1 text:image in Chameleon, 1:6 speech-to-other in Chameleon+Speech, 1:1 text:image in Transfusion). In real deployments, modality distributions can be highly skewed — e.g., a model trained primarily on text with occasional images, or a speech-heavy model with rare text. In such settings, MoT's modality towers would receive vastly different amounts of gradient updates, potentially causing the under-sampled modality towers to undertrain while the over-sampled towers converge normally. A stress-test experiment would train MoT with text:image ratios of 99:1, 95:5, 80:20, 50:50, 20:80, 5:95, and 1:99, measuring per-modality convergence speed relative to a dense baseline. This would reveal whether MoT's efficiency gains persist under data imbalance, or whether the sparse modality towers require a minimum amount of training data to be effective (below which the dense model's parameter sharing provides a regularization benefit). The LOO analysis (Figure 15) hints that speech is fragile when combined with other modalities; this experiment would quantify fragility as a function of data frequency.

MoT for early-fusion video-language models. The paper evaluates text, image, and speech — three modalities with distinct tokenization and statistical properties. Video adds temporal structure at multiple timescales (frame-level, clip-level, scene-level) that neither text nor static images capture. A natural extension is to treat video as a fourth modality in the MoT framework with its own dedicated transformer tower, and test whether the modality competition avoidance observed for speech (Section 4) extends to video. The specific hypothesis: video tokens will benefit from specialized temporal processing parameters (e.g., FFN layers that learn spatiotemporal patterns) while global self-attention handles cross-modal grounding (e.g., attending from a video clip to a text description). A strong experiment would train a 4-modality MoT (text, image, speech, video) on a combined dataset, compare to a 4-modality dense baseline and a 4-expert MoE baseline, and measure whether the video modality shows speedups comparable to speech (~4×; Figure 8b) or more modest gains like text (~1.8×). The existing LOO analysis methodology (Section 4) would directly extend to diagnose which of the three original modalities competes most with video.

Cross-modal transfer learning: using MoT's modular architecture for efficient fine-tuning on new modalities. MoT's modality-specific towers are naturally modular — the text tower, image tower, and speech tower are independent parameter sets that only interact through global self-attention. This suggests a transfer learning scenario: pre-train a MoT model on text and images, then add a new modality (e.g., music, 3D point clouds, sensor data) by attaching a randomly initialized new modality tower while keeping the existing towers frozen. Because the frozen text and image towers already produce well-structured representations, the new modality tower should learn to project its tokens into the shared attention space more efficiently than training from scratch. A strong experiment would compare (a) training a dense model from scratch on text+image+new_modality, (b) training an MoT from scratch on all three, and (c) MoT transfer: pre-train MoT on text+image, freeze those towers, add and train only the new_modality tower. The key metric is the training data required for the new modality to reach target performance, and whether the transfer approach preserves performance on the original two modalities (which full training from scratch might degrade due to catastrophic forgetting or modality competition).

Practical Applications and Downstream Use Cases

Cost-efficient multi-modal pretraining for teams with limited compute budgets. The paper's central finding — MoT matches dense 7B Chameleon performance with 55.8% of the training FLOPs (Figure 5), and a 760M MoT outperforms a 1.4B dense baseline on image generation quality (Figure 11: CLIP 0.214 vs 0.206, FID 21.145 vs 24.688) — directly translates to reduced GPU-hour costs. For a team training a 7B multi-modal model from scratch on ~0.4T tokens (the paper's Chameleon 7B configuration), MoT would reduce the required GPU-hours by ~44%, representing hundreds of thousands of dollars in cloud compute savings at current A100/H100 pricing. For teams constrained to smaller GPU clusters, the 760M MoT result is even more impactful: it enables training a model with better-than-1.4B-dense image quality using half the GPUs and half the per-step time, making competitive multi-modal generation accessible to academic labs and startups.

On-device or edge deployment of multi-modal models via MoT's modular architecture. MoT's modality-specific towers enable a deployment optimization that dense models cannot easily support: loading only the modality towers needed for a specific inference request. A user asking a text-only question does not need the image or speech tower parameters in GPU memory. A user generating an image from a text prompt needs the text tower (for encoding the prompt) and the image tower (for generating output), but not the speech tower. This selective loading can reduce memory footprint and inference latency for single-modality or two-modality requests relative to a dense model that must load all parameters regardless of the request type. The paper's fine-tuning results (Section 3.4.4, Figures 13, 20-22) demonstrate that MoT's quality advantage persists after fine-tuning, meaning a deployed MoT model can serve text-only, image-only, and multi-modal requests from the same checkpoint while dynamically managing memory. While the paper does not benchmark inference memory or latency for selective tower loading, the architectural modularity makes this a straightforward systems optimization that production deployments can exploit.

Large-scale batch inference for multi-modal data processing pipelines. Organizations that process large volumes of multi-modal data — e.g., generating captions for millions of images, producing text descriptions for video archives, transcribing and summarizing audio content — can use MoT's efficiency gains to reduce inference costs proportionally. Since MoT matches dense quality with fewer training FLOPs, and the inference FLOPs per token are identical to dense (Section 3.1), the primary inference benefit comes from MoT's ability to reach higher quality with a smaller model. A pipeline using the 760M MoT Transfusion model instead of a 1.4B dense model (Figure 11) achieves better image captioning (CIDEr 0.320 vs 0.286) and better image generation (FID 21.145 vs 24.688) at half the inference FLOPs per token. For a pipeline processing 100 million images, this 2× reduction in per-image compute translates to tens of thousands of GPU-hours saved, with no quality compromise — and in fact, quality improvement.