Multimodal2026~119 minarXiv:2603.03276

Beyond Language Modeling: An Exploration of Multimodal Pretraining

score 9Shengbang Tong, David Fan, John Nguyen +18AI at Meta↑ 32Mar 3, 2026cs.CV

🎯 Pitch

Vision is significantly more data-hungry than language during multimodal pretraining, but Mixture-of-Experts architectures naturally resolve this scaling asymmetry while eliminating the complexity of separate visual encoders. A single unified model trained from scratch discovers world modeling—predicting actions and future frames—using only 1% navigation data, demonstrating these physical reasoning capabilities largely transfer from broad visual experience.

1. Executive Summary

This paper systematically studies the design space of native unified multimodal pretraining, training Transfusion-based decoder-only models from scratch on text, video, image-text pairs, and action-conditioned video to isolate the factors governing joint vision-language learning without interference from language pretraining. It establishes that a single Representation Autoencoder (RAE)-based semantic encoder (SigLIP 2) suffices for both visual understanding and generation, that multimodal co-training yields positive cross-modal synergy (e.g., adding unlabeled video improves VQA over scaling in-domain data 5×), and that Mixture-of-Experts (MoE) architectures naturally learn modality-specific routing while harmonizing a fundamental scaling asymmetry — vision follows a data-hungry regime (Dopt ∝ C^0.63) while language follows near-Chinchilla scaling (Dopt ∝ C^0.53) in dense models, with MoE narrowing the exponent gap from 0.10 to 0.05. The paper further demonstrates that world modeling capabilities emerge from general multimodal pretraining with as little as 1% domain-specific navigation data, establishing that physical prediction and planning transfer primarily from broad visual experience rather than specialized training.

2. Context and Motivation

The Core Problem: We Don't Understand How to Properly Train Multimodal Models from Scratch

The fundamental question this paper tackles is deceptively simple: if you want to train a single model that understands and generates both text and images from scratch, how should you design it? This matters because the dominant paradigm for building multimodal systems has been to start with a powerful pretrained language model and then bolt on vision capabilities through adapter layers or fine-tuning. While this approach preserves existing language capabilities, it fundamentally confounds our understanding of multimodal learning itself — we cannot disentangle what the model learns from joint vision-language training versus what it already knew from language pretraining.

The authors frame this explicitly (Section 1):

"Rather than jointly learning from vision and language from scratch, most current methodologies rely on initialization from pretrained language models... [T]he knowledge already embedded in these pretrained backbones confounds any conclusions drawn about the multimodal training itself, making it difficult to disentangle what is learned from unified training versus what is inherited from language pretraining."

This gap is significant for several interconnected reasons the paper identifies:

Scientific understanding remains opaque: The "design space is rife with confounding variables" (Section 1). Without controlled from-scratch experiments, the community cannot answer basic questions: Do vision and language compete for model capacity? What visual representation works best for both understanding and generation? How should we architect the model to handle fundamentally different data modalities? These are not just academic questions — they determine whether unified multimodal models are even viable without massive language pretraining as a crutch.
The looming text data bottleneck: As the paper notes in its opening, "high-quality text data is finite and approaching exhaustion" (citing Sutskever, 2025). In contrast, "the visual world possesses an endless stream of signal." If we can crack the code of native multimodal pretraining, we can leverage vast quantities of video and image data — currently largely untapped — to continue scaling model capabilities even as text data runs dry. This is a hard pragmatic ceiling that language-only scaling will eventually hit.
Philosophical grounding: The paper invokes Plato's Cave allegory to make a deeper point: language models trained purely on text have "mastered the description of shadows on the wall without ever seeing the objects casting them." Text is a lossy compression of physical reality — it captures symbols but misses "the high-fidelity physics, geometry, and causality of the physical world." True general intelligence, the argument goes, requires grounding in raw sensory experience, not just linguistic abstractions.
World modeling potential: If we can train models natively on visual data alongside language, capabilities like physical prediction, spatial reasoning, and planning may emerge naturally from broad experience rather than requiring specialized architectures or domain-specific training. This connects directly to LeCun's (2022) vision of autonomous machine intelligence built on world models learned from sensory data.

Conflicting Design Choices and Missing Systematic Comparisons

The paper is motivated by a fragmented landscape where different research groups make different architectural and representational choices, but nobody has systematically compared them under controlled conditions.

The visual representation dilemma. The community has largely operated under an implicit assumption that visual understanding and generation require fundamentally different representations. Understanding tasks (VQA, image captioning) typically use high-dimensional semantic latents from language-supervised encoders like CLIP or SigLIP, or self-supervised encoders like DINOv2. Generation tasks typically use low-dimensional VAE latents (like those from Stable Diffusion or FLUX) because diffusion models were historically designed around compact latent spaces. This has led to dual-encoder designs — like Janus (Wu et al., 2025; Ma et al., 2025) and BAGEL (Deng et al., 2025) — that use separate encoders for understanding and generation, "significantly complicat[ing] model design and add[ing] overhead to both model training and inference" (Section 3).

However, recent work on Representation Autoencoders (RAE) (Zheng et al., 2026; Tong et al., 2026) challenged this assumption by showing that diffusion models can operate effectively in high-dimensional semantic latent spaces. This opened the possibility that a single encoder could serve both purposes, but no prior work had systematically compared visual representations across both tasks in a unified from-scratch pretraining setting.

The modality competition debate. A common concern in multimodal pretraining is that vision and language might compete for limited model capacity, with training on one modality degrading the other. Prior work like MoMa (Lin et al., 2024) had shown benefits from modality-specific FFNs, suggesting some degree of competition exists. But the broader question remained unresolved: is modality competition an inherent limitation of multimodal learning, or is it a symptom of specific design choices? Without controlled experiments isolating architectural choices from data choices from representation choices, the community couldn't distinguish between these hypotheses.

The architecture design space is unexplored for native multimodal models. The paper surveys a wide range of architectural approaches in the related work (Section 8). Some works connect pretrained LLMs to pretrained diffusion models via adapters (Dai et al., 2023; Sun et al., 2024a; Ge et al., 2024). Others train diffusion models from scratch jointly with language models, but with varying degrees of modality separation — from fully shared transformers (Transfusion; Zhou et al., 2025a) to strict separation of FFNs (Shi et al., 2024; Lin et al., 2024) or attention blocks (Liang et al., 2024; Deng et al., 2025). Mixture-of-Experts (MoE) had proven effective for scaling language models (Liu et al., 2024a,b; Team et al., 2025), but its behavior in multimodal pretraining — where tokens belong to fundamentally different modalities with different statistical properties — was unknown. Would MoE naturally learn modality specialization? Would it help or hurt cross-modal learning?

No scaling laws for joint vision-language training. The scaling laws literature for language models is well-developed, from Kaplan et al. (2020) to Chinchilla (Hoffmann et al., 2022) to recent MoE extensions (Tian et al., 2026). For diffusion models, Polyak et al. (2024) studied scaling behavior. But no prior work had derived scaling laws for models trained jointly on vision and language. This is a critical gap because the compute-optimal allocation between modalities may differ fundamentally from unimodal cases. If vision and language scale differently with compute — if one is more parameter-hungry while the other is more data-hungry — then no single dense model configuration can be simultaneously optimal for both, creating a structural tension in unified pretraining.

How This Paper Positions Itself

The paper frames its contribution as an empirical clarity exercise: "to provide empirical clarity to this landscape" by conducting "controlled, from-scratch pretraining experiments, isolating the factors that govern multimodal pretraining" (Section 1). This positions the work not as proposing a new method, but as systematically mapping the design space to establish what actually matters and why.

The authors adopt the Transfusion framework (Zhou et al., 2025a) as their experimental substrate — a single decoder-only transformer that handles language via next-token prediction and vision via diffusion (flow matching). This choice is deliberate: Transfusion provides a clean, unified architecture where both modalities share the same backbone but use different loss functions, making it an ideal testbed for isolating design variables. The paper then systematically studies five axes:

Visual representation (Section 3): What encoder should produce the visual tokens? VAE, semantic, or raw pixels?
Data composition (Section 4): How do different data sources (pure text, video, image-text pairs, action-conditioned video) interact? Do they compete or synergize?
World modeling (Section 5): Can physical prediction capabilities emerge from general multimodal pretraining rather than domain-specific training?
Architecture (Section 6): How should capacity be allocated between modalities? Fixed separation (modality-specific FFNs, MoT) versus learned routing (MoE)?
Scaling laws (Section 7): What are the compute-optimal scaling relationships for vision and language, and how do they interact?

Critically, the paper explicitly chooses to focus on pretraining rather than post-training (instruction tuning, RLHF), arguing that "a model's core capabilities are largely acquired during this phase" (citing Zhou et al., 2024). This is an important scoping decision: the paper aims to understand the fundamental dynamics of multimodal learning, not to build the best possible chat model. By evaluating at the end of pretraining without any instruction tuning, the results reflect what the model genuinely learns from the multimodal data mixture itself.

The paper also explicitly connects to the broader vision of world models as articulated by LeCun (2022). The world modeling experiments (Section 5) are not just an application — they test the hypothesis that general visual experience can substitute for domain-specific training. If a model trained broadly on video and text can predict navigation outcomes with minimal in-domain data, this suggests that the boundary between "multimodal models" and "world models" may blur at scale — a finding with implications for embodied AI and autonomous systems.

In sum, this paper positions itself as a foundational empirical study that fills a critical gap: the absence of systematic, controlled experiments on native multimodal pretraining. By training from scratch and changing one variable at a time, it aims to establish reliable design principles that the field can build upon, moving from folklore and heuristics toward evidence-based architecture and data decisions.

3. Technical Approach

3.1 Reader Orientation

The system under study is a single, unified neural network — a decoder-only Transformer — trained from scratch to perform both language understanding and visual understanding/generation by predicting the next text token (autoregressively) and denoising visual tokens (via flow matching). The core problem it solves is the design of native multimodal pretraining: how to choose a visual representation, data mixture, architecture, and training objective such that vision and language learn synergistically rather than competitively, and such that a single model can both comprehend images (e.g., answer questions about them) and generate them (e.g., produce a picture from a caption). The "shape" of the solution is a systematic empirical analysis — a series of controlled, from-scratch experiments that isolate one design variable at a time (visual encoder type, data composition, capacity-allocation strategy) and measure the resulting capabilities across language modeling, image generation quality, visual question answering, and world-modeling (physical prediction).

3.2 Big-Picture Architecture (Diagram in Words)

The system consists of five major components, with information flowing left-to-right through a unified Transformer backbone:

Multimodal Data Stream — A heterogeneous mixture of training examples drawn from four categories: pure text documents, raw video (frames at 1 FPS, no captions), paired image–text (I/T) examples (web-crawled and high-aesthetic), and action-conditioned navigation trajectories (context frames + a text action → target frame). Each batch mixes these sources.
Modality-Specific Tokenizers — Text is mapped to discrete tokens via a standard BPE tokenizer (LLaMA-3 vocabulary). Each image or video frame (preprocessed to 224×224 pixels) is mapped to a sequence of continuous latent vectors by a frozen visual encoder (by default, SigLIP 2 So400M). Videos are encoded frame-by-frame; each frame is enclosed in <BOI> and <EOI> marker tokens within the sequence.
Unified Decoder-Only Transformer Backbone — A single Transformer (2.3B total parameters, 1.5B activated per token) that processes both text and visual tokens. The core design includes modality-specific feed-forward networks (FFNs) by default — separate weights for text tokens and visual tokens within each self-attention block, while self-attention itself is shared. A hybrid attention mask ensures text tokens attend causally (to previous tokens), visual tokens within the same frame attend bidirectionally to each other, and all tokens attend causally to tokens from earlier frames or text segments.
Modality-Specific Training Heads and Losses — The Transformer's output hidden states are routed based on position: states corresponding to text tokens are projected to a vocabulary-sized logit vector and trained with standard next-token prediction cross-entropy loss (L_{LM}). States corresponding to visual tokens (within <BOI>/<EOI> boundaries) are projected via a linear head to predict a velocity field; the model is trained with a flow-matching mean-squared-error loss (L_{flow}) that teaches it to denoise the visual latents.
Modality-Switching Inference Procedure — For generation, the model dynamically alternates between discrete autoregressive sampling (for text) and iterative denoising (for images). When a <BOI> token is sampled, the model appends a sequence of pure noise vectors and runs a 25-step Euler sampler, updating the continuous visual tokens in place before resuming text generation after the <EOI>.

All experiments are conducted at fixed, controlled compute budgets within each ablation family, and evaluation is performed at the end of pretraining without instruction tuning, to isolate the effects of the pretraining design choices.

3.3 Roadmap for the Deep Dive

First, the joint training objective and hybrid masking strategy (Section B), because the model's ability to process interleaved text and visual data depends on these mechanisms, and they are shared across all subsequent experiments.
Second, the core Transfusion mechanics — how text tokens and visual tokens are formatted, attended to, and trained within a single sequence, and how the two losses are balanced. This establishes the "container" into which different visual representations will be plugged.
Third, the visual representation study (Section 3) — the range of encoders (VAE, semantic, raw pixels) and how each maps images to token sequences, since the choice of visual tokenizer is the first major design axis and conditions everything downstream.
Fourth, the data composition experiments (Section 4) — what multimodal data mixtures are used, why captions matter differently from raw video, and how the experiments are designed to measure synergy versus competition.
Fifth, the world modeling extension (Section 5) — how the same architecture is extended to action-conditioned visual prediction without any architectural changes, simply by formatting navigation actions as text tokens.
Sixth, the Mixture-of-Experts architecture (Section 6) — how MoE generalizes modality-specific FFNs, the key hyperparameters (granularity, sparsity, shared experts), and how expert specialization is analyzed.
Seventh, the IsoFLOP scaling law methodology (Section 7) — how compute-optimal model size and token count are estimated for each modality in both dense and MoE settings.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical analysis paper whose core idea is that the design space of native multimodal pretraining can be systematically characterized by training models from scratch under controlled conditions and measuring the impact of isolated design choices across a comprehensive suite of capabilities (language modeling, visual understanding, visual generation, and world modeling).

Unified Multimodal Sequence Formatting and Data Representation

The core framing of the model is a unified sequence: a single stream of tokens that interleaves discrete text tokens and continuous visual tokens, processed by a single Transformer. This section defines the mechanics of how these two fundamentally different data types coexist in one sequence.

Text tokenization. Text is tokenized using the standard BPE tokenizer from LLaMA-3 (Grattafiori et al., 2024). Each text token is mapped to a learned embedding vector via a standard nn.Embedding layer. The text token vocabulary and embedding are trained from scratch alongside the Transformer — there is no pretrained language model initialization. The paper denotes text tokens as T.

Visual tokenization. An image (or a single video frame, since videos are processed frame-by-frame) is first resized to 224×224 pixels. It is then passed through a frozen, pretrained visual encoder which outputs a grid of latent vectors. The encoder is kept frozen during all experiments — its weights are never updated. The grid is flattened into a 1D sequence of tokens. For most encoders (SigLIP 2, DINOv2-L, WebSSL-L, raw pixels), the native output is a 16×16 grid (patch size 14×14 on a 224×224 image), yielding exactly 256 visual tokens per frame. For VAE-based encoders (SD-VAE, FLUX.1 VAE), the native output is a 32×32 grid (1024 tokens); the paper applies PixelUnshuffle(2) (Shi et al., 2016) to spatially downsample this to 16×16 = 256 tokens by increasing the channel dimension by 4×, ensuring all visual encoders produce the same number of tokens per frame for fair comparison. After the Transformer processes these tokens, PixelShuffle(2) upsamples them back to the native spatial resolution for loss computation. The paper denotes image tokens as I.

Frame boundary markers. To delineate visual segments within the interleaved sequence, each image or video frame is wrapped with special marker tokens: a beginning-of-image token <BOI> before the visual tokens and an end-of-image token <EOI> after them. Consequently, a video consisting of N frames appears in the sequence as: <BOI> [frame_1_tokens] <EOI> <BOI> [frame_2_tokens] <EOI> ... These markers serve as delimiters that the model can learn to use for modal switching during generation: when the model autoregressively samples a <BOI> token, the inference procedure switches to denoising mode; upon completion, it appends <EOI> and resumes text generation.

Sequence length and batching. By default, training uses a sequence length of 4096 tokens. Training runs on 128 GPUs with a batch size of 4 sequences per GPU, yielding approximately 2 million tokens per training step. The total number of steps is set according to the desired token budget (e.g., 520B text + 520B multimodal tokens over ~1T total tokens).

Hybrid Attention Masking for Interleaved Modalities

The Transformer's self-attention mechanism must respect the different temporal structures of text and visual data. Text is strictly sequential (word follows word), while pixels within a single image have no natural order and should attend to each other bidirectionally. The paper implements a hybrid attention mask using FlexAttention (Dong et al., 2024a).

Text tokens. Standard causal (unidirectional) attention: each text token can attend to all text tokens that appear before it in the sequence, but not to future tokens. This preserves the autoregressive language modeling objective — at generation time, the model cannot peek ahead.

Visual tokens within the same frame. Bidirectional attention: all tokens belonging to the same image or video frame (i.e., tokens between a single pair of <BOI> and <EOI> markers) can attend to each other without restriction. This is the "block-wise" property: each frame is a self-contained block of tokens. This makes intuitive sense because the pixels of a single image have no meaningful left-to-right order — they are a 2D grid that was arbitrarily flattened into 1D.

Visual tokens across frames. Causal (block-wise) attention: tokens from a given frame can attend to tokens from all previous frames (each as a block with bidirectional internal attention), but not to tokens from future frames. This preserves the temporal ordering of video: frame t+1 should be generated conditioned on frame t, but frame t should not depend on frame t+1.

Cross-modal attention. Text tokens can attend to all visual tokens from frames that appear before them in the sequence. Visual tokens from a given frame can attend to all text tokens that appear before that frame. This ensures that text appearing after an image can condition on that image (for captioning/image-to-text tasks), and that images appearing after text can condition on that text (for text-to-image generation).

The paper implements this via FlexAttention, a PyTorch-native API that allows writing custom attention mask logic as simple Python functions that are compiled into efficient CUDA kernels. The specific rule is: for any query token at position i and key token at position j, attention is allowed if both tokens belong to the same visual frame block (bidirectional within block), OR if token j appears before token i in the overall sequence and is either text or belongs to a frame that ended before the frame containing token i began. This unified masking scheme enables the model to process any interleaved sequence of text and images/video frames correctly.

Joint Training Objective: Language Modeling + Flow Matching

The model optimizes a weighted combination of two distinct loss functions, applied to different segments of the same sequence simultaneously in a single forward-backward pass.

Language modeling loss (L_LM). For text tokens, the standard autoregressive cross-entropy objective is used. Given a text sequence (x_1, ..., x_n), the loss is:

$L_{LM} = -\sum_{i=1}^{n} \log p_\theta(x_i \mid x_{<i})$

where p_theta(x_i | x_{<i}) is the model's predicted probability for the correct next token x_i given all preceding tokens in the sequence (including preceding visual tokens that appear before position i).

What it computes: for each text token in the sequence, the model produces a probability distribution over the full text vocabulary (via a linear projection from the Transformer's hidden state to logits, then a softmax). The negative log-likelihood of the actual next token under this distribution is computed. The total is summed over all text token positions in the sequence. This is the standard causal language modeling objective — teach the model to predict the next word given all prior context, including visual context.

Why this form: autoregressive next-token prediction is the dominant paradigm for training large language models because it is a self-supervised objective that scales with data quantity, produces a generative model that can sample text, and has been empirically shown to yield strong representations for downstream tasks. There is no alternative for text generation that is as well-established.

Flow matching loss for visual tokens (L_flow). For visual tokens, a continuous diffusion-like objective is used. Unlike text tokens which are discrete, visual tokens are continuous vectors in a high-dimensional latent space. The model is trained to denoise them.

The procedure follows flow matching formulations (Lipman et al., 2023; Liu et al., 2023). Let z_0 denote the clean (ground-truth) visual latent tokens for an image or video frame, flattened into a 1D sequence. Let epsilon be a random noise vector sampled from a standard Gaussian distribution N(0, I), with the same dimensionality as z_0. A timestep t is sampled uniformly from t ~ U[0, 1]. An interpolated latent is constructed by linear interpolation between noise and clean:

$z_t = (1 - t) \epsilon + t z_0$

The model receives the noisy latent z_t, the timestep t, and the contextual information (preceding text and visual tokens), and is trained to predict a velocity field v_theta(z_t, t, context) — the direction and magnitude from the noisy state toward the clean state. The loss is the expected mean squared error between the predicted velocity and the true difference between clean and noise:

$L_{flow} = \mathbb{E}_{t, z_0, \epsilon} \left[ \| v_\theta(z_t, t, \cdot) - (z_0 - \epsilon) \|_2^2 \right]$

where ||·||_2^2 is the squared Euclidean (L2) norm, summed over all dimensions of the latent tokens for the entire frame.

What it computes: for each image or video frame in the batch, a single independent noise level t is sampled and applied to all visual tokens of that frame. The model sees the noisy tokens, knows the timestep, and must predict the exact vector that, if added to the current noisy state, would move it toward the clean state. The loss penalizes the squared difference between the model's prediction and this ground-truth velocity. The expectation E_{t, z_0, epsilon} indicates that this is averaged over random timesteps, random training images, and random noise samples across training steps.

Why this form: flow matching (also called rectified flow) has several desirable properties over standard diffusion. First, it defines a straight probability path from noise to data, which allows for more efficient sampling (fewer denoising steps needed at inference). Second, predicting the velocity (z_0 - epsilon) rather than the noise epsilon (as in standard DDPM) or the clean data z_0 directly has been found to be more stable for high-dimensional latent spaces. Third, the uniform timestep sampling t ~ U[0, 1] ensures the model learns to denoise at all noise levels equally.

The paper uses a shifted noise schedule "towards the noisier end of the spectrum" (following Esser et al., 2024, and RAE; Zheng et al., 2026; Tong et al., 2026). This means the distribution of t during training is biased toward values closer to 0 (noisier), which improves sample quality by spending more training on the difficult high-noise regime. Each image/frame gets its own independent t, not shared across different images in the same batch. This independent noise injection aligns with the "Diffusion Forcing" paradigm (Chen et al., 2024a) where each frame's noise level is sampled independently.

In Sections 3 and 6.1, the paper also explores an alternative parameterization called x-pred (Li & He, 2025), where the model directly predicts the clean latent z_0 instead of the velocity. The motivation is that for high-dimensional latent spaces (like the 1152-dimensional SigLIP 2 latents), predicting the target directly may leverage the manifold assumption — the idea that clean images lie on a lower-dimensional manifold embedded in the latent space, and predicting that manifold directly is easier than predicting velocities through the full ambient space. The results show that x-pred indeed performs better for RAE (SigLIP 2) representations but worse for low-dimensional VAE latents.

Joint loss weighting. The total loss is a weighted sum:

$L = \lambda_{LM} L_{LM} + \lambda_{flow} L_{flow}$

Unless otherwise specified, lambda_LM = 1.0 and lambda_flow = 3.0. This 3:1 weighting in favor of the visual loss is chosen empirically to stabilize joint training. The paper notes (Section D.2, Appendix) that different visual encoders produce flow matching losses of vastly different magnitudes, and a fixed weighting can over-emphasize one modality if not tuned. As an alternative, the paper explores an adaptive "loss centering" mechanism (inspired by DINO; Caron et al., 2021) where the weights are dynamically adjusted per step to maintain a desired balance between the two loss magnitudes.

Classifier-Free Guidance for Visual Generation

At inference time, the model uses classifier-free guidance (CFG; Ho & Salimans, 2022) to improve the quality and text-alignment of generated images. During training, the model's text conditioning is randomly dropped 10% of the time (replaced with a null/empty conditioning signal). This teaches the model to denoise both with and without text guidance.

At inference, the model runs a 25-step Euler sampler. At each denoising step, the model produces two velocity predictions: one conditioned on the actual text prompt v_theta(z_t, t, text), and one with the null conditioning v_theta(z_t, t, empty). The final velocity used to update the latents is an extrapolation:

$v_{guided} = v_\theta(z_t, t, \text{empty}) + w \cdot (v_\theta(z_t, t, \text{text}) - v_\theta(z_t, t, \text{empty}))$

where w is the guidance scale, fixed at 3.0 for all image generation evaluations. This amplifies the effect of the text conditioning by pushing the prediction away from the unconditional prediction. The guidance scale is not tuned — the paper uses w=3.0 consistently.

Why this form: CFG is the standard technique for text-to-image generation and has been shown to dramatically improve image quality and text-alignment at the cost of some diversity. The 10% dropout rate and guidance scale of 3.0 are common defaults in the diffusion literature.

After denoising, the clean latent tokens z_0 are decoded back to pixel space using a pretrained decoder. For VAE encoders (SD-VAE, FLUX.1), the off-the-shelf pretrained decoders are used directly. For semantic encoders (SigLIP 2, DINOv2-L, WebSSL-L), the decoders from RAE (Zheng et al., 2026; Tong et al., 2026) are used — these are separately trained decoders that map from the semantic latent space back to pixels.

Default Model Configuration and Modality-Specific FFNs

The default model architecture is a decoder-only Transformer (Vaswani et al., 2017) similar to Transfusion (Zhou et al., 2025a), with key modifications. The total parameter count is 2.3B, with 1.5B activated per token (due to modality-specific FFNs — only one FFN is active per token). A standard dense model with shared FFNs and 1.5B parameters is used as the point of comparison.

Architecture details. The paper uses the Transfusion framework but replaces the U-Net (Ronneberger et al., 2015) typically used for visual input/output projections with simple linear layers. This simplification is possible because the paper does not adjust the number of visual tokens for most encoders (all produce 256 tokens), so a direct linear mapping into the Transformer's latent space suffices. The motivation is simplicity — the paper wants to study the core Transformer architecture without introducing U-Net-specific inductive biases that might confound the analysis.

Modality-specific FFNs. Within each Transformer block, there are two separate feed-forward networks (FFNs) — one for text tokens and one for visual tokens — rather than a single shared FFN. The self-attention layer, however, is shared across modalities. This is the simplest form of "modality separation" — the model has dedicated processing capacity for each modality at the FFN level, while sharing the attention mechanism which handles cross-modal interactions.

The paper verifies this design choice in an ablation (Figure 3): modality-specific FFN unanimously outperforms shared FFN, reducing text perplexity while improving image generation quality and VQA accuracy. The total parameter count increases (since there are two FFNs instead of one), but inference FLOPs remain constant because only one FFN is activated per token — text tokens route through the text FFN, visual tokens through the visual FFN. This constant inference cost is critical for fair comparison.

Training hyperparameters. The optimizer is AdamW with a peak learning rate of 3×10^{-4}, 1000 warmup steps, and cosine decay to 5% of the peak learning rate. Training uses a global batch size of ~2M tokens per step (128 GPUs × 4 sequences per GPU × 4096 tokens per sequence). The total number of training steps is calculated from the total token budget (e.g., for 1T tokens: 1T / 2M ≈ 500,000 steps).

Why modality-specific FFNs: The motivation follows from prior work (Lin et al., 2024; MoMa) showing that text and image tokens have fundamentally different statistical properties — text is discrete and sparse, visual tokens are continuous and dense — and forcing them through shared FFN weights creates interference. Modality-specific FFNs allow the model to learn specialized transformations for each data type without paying extra inference cost (since each token only activates one FFN). This design is a precursor to the Mixture-of-Experts approach explored in Section 6, where the separation is learned dynamically rather than being hard-coded.

Visual Representation Study (Section 3): The Encoder Space

A central design question is: what visual encoder should produce the continuous latent tokens from images? The paper studies three families of encoders, with a total of eight configurations, to determine which representation best supports both visual understanding and generation in a unified model.

VAE encoders (generation-oriented, low-dimensional). Two variational autoencoders are tested:

SD-VAE (Rombach et al., 2022): the encoder from Stable Diffusion, which maps a 256×256 image to a 32×32×4 latent grid. After PixelUnshuffle(2), this becomes a 16×16 grid of 256 tokens, each with 16 channels (4 channels × 4 from unshuffle). The decoder is the pretrained SD-VAE decoder.
FLUX.1 VAE (Labs, 2024): the encoder from the FLUX.1 model, also producing a 32×32 latent grid, similarly downsampled to 256 tokens with increased channel dimension. The decoder is the pretrained FLUX.1 decoder.

VAE encoders are the standard choice for image generation because they compress images into a compact, low-dimensional latent space that is efficient for diffusion models. However, they are not typically used for understanding tasks because their latents are optimized for reconstruction fidelity, not semantic discrimination.

Semantic encoders (RAE, understanding-oriented, high-dimensional). Three semantic encoders are tested, all producing native 16×16 grids (256 tokens) without spatial adaptation:

SigLIP 2 So400M (Tschannen et al., 2025): a language-supervised vision encoder trained with sigmoid loss for image-text alignment. Output dimension: 1152 per token. This is the default encoder for most experiments.
DINOv2-L (Oquab et al., 2023): a self-supervised vision encoder trained with a student-teacher distillation objective on unlabeled images. Output dimension: 1024 per token.
WebSSL-L (Fan et al., 2025a): another self-supervised encoder trained on web-scale unlabeled data. Output dimension: 1024 per token.

These encoders produce high-dimensional semantic features that are strong for visual understanding (classification, VQA) but were historically thought to be unsuitable for generation because diffusion models struggle with high-dimensional latent spaces. The RAE framework (Zheng et al., 2026; Tong et al., 2026) provides pretrained decoders that map from these semantic latents back to pixels, enabling generation from semantic representations.

Raw pixel encoding. As a baseline without any pretrained encoder, the paper also tests direct raw pixel inputs. Each image is divided into 14×14 patches, and the flattened RGB values of each patch (14×14×3 = 588 values) form one visual token. This produces 256 tokens per image (16×16 grid). No encoder is needed — the model learns to process raw pixel data directly. A variant using "x-pred" (predicting clean pixels directly instead of velocity) is also tested, drawing on JiT (Li & He, 2025). The motivation: if the model can learn directly from pixels, it avoids the information bottleneck of any pretrained encoder.

Dual encoder baseline (for comparison). In Section 6.3, as a stacking comparison, the paper also tests a dual-encoder configuration using SigLIP 2 for understanding and SD-VAE for generation (as in Janus and BAGEL). This serves to test whether the added complexity of dual representations is justified.

Design choice rationale. The key hypothesis is that a single high-dimensional semantic encoder (RAE) can serve both understanding and generation, eliminating the need for dual encoder architectures. The paper tests this by measuring:

Text perplexity (DCLM, Notes): Does the choice of visual encoder affect language modeling?
Image generation quality (DPGBench, GenEval, COCO FID): Can semantic encoders match or exceed VAEs for generation?
Visual understanding (VQA accuracy on 16 Cambrian benchmarks): Do VAE latents degrade understanding compared to semantic latents?
Throughput / simplicity: Is the single-encoder approach simpler to implement and train?

The results (Figure 4) show that SigLIP 2 with RAE decoder simultaneously outperforms VAE encoders on both generation benchmarks (DPGBench, GenEval) and VQA, while maintaining text perplexity comparable to the text-only baseline. This validates the single-encoder approach and establishes SigLIP 2 as the default for all subsequent experiments.

Data Composition Experiments (Section 4): What to Train On

The paper trains on four categories of data, mixed within each batch:

Text-only (T → T): Large-scale web text from DCLM (Li et al., 2024b). Trained with standard next-token prediction. This is the "language grounding" component.
Video-only (I → I): Raw video frames at 1 FPS from YouTube-Temporal 1B (Zellers et al., 2022), Something-Something V2 (Goyal et al., 2017), and Kinetics (Kay et al., 2017). No text annotations — these are pure visual sequences. The model is trained to predict the next frame(s) autoregressively via flow matching. This tests whether raw visual data competes with or complements language.
Image-text pairs (I → T and T → I): Paired data from MetaCLIP (Xu et al., 2024) — a web-crawled dataset of 2.2B image-text pairs — and in-house high-aesthetic Shutterstock (SSTK) data. Both directions are trained: image-to-text (captioning) and text-to-image (generation). The recaption variant replaces MetaCLIP captions with synthetic captions from Qwen2.5-VL-32B-Instruct (prompted to produce dense, factual descriptions without beginning with "A", "An", or "The") to study the effect of caption quality and distribution.
Action-conditioned video (I + T → I): Navigation trajectories from Navigation World Models (NWM; Bar et al., 2025) and in-house text-annotated videos. Each sample consists of 1-4 context frames, a text action (e.g., "action: dx=+4.96, dy=+1.50, dyaw=+0.51, rel_t=0.19"), and 1-4 resulting frames. The actions are represented as pure text tokens — no specialized action encoders or adapters are introduced. For the in-house annotations, a MLLM is prompted to describe the physical action between consecutive video chunks, producing ~12M text action annotations.

Data mixture experiments (Section 4.1). Three representative data mixtures are compared against a text-only baseline (520B text tokens):

Text + Video: 520B text + 520B video tokens. Tests whether pure visual data is compatible with language.
Text + MetaCLIP: 520B text + 520B I/T tokens. Tests the effect of introducing captioned images.
Text + Video + MetaCLIP + Action: All sources combined (520B text + 520B multimodal tokens total, distributed across video, I/T, and action data). Tests the most diverse mixture.

All multimodal models are trained on ~1T total tokens (520B text + 520B multimodal) to keep the total training budget comparable.

Image-text distribution analysis (Section 4.2). To understand why text perplexity degrades slightly with I/T data (Figure 5), the paper computes the cosine distance between the text distribution of each I/T source and the DCLM pretraining corpus. MetaCLIP captions have a cosine distance of 0.196 to DCLM, while synthetically recaptioned MetaCLIP has distance 0.286, and Shutterstock captions have distance 0.215. The hypothesis: captions that are stylistically distant from the pretraining text corpus cause a distribution shift that degrades language modeling, which explains why pure video (no text shift) actually improves DCLM perplexity.

Synergy experiments (Section 4.3). Two key experiments quantify cross-modal synergy:

Complementarity sweep (Figure 8): Models are trained across a 5×5 grid of text token budgets {0, 25, 50, 75, 100}B and multimodal token budgets (the same range). For a fixed vision budget, adding text tokens improves diffusion loss and GenEval score. The inverse is also measured (Figure 29 in Appendix): for a fixed text budget, adding visual tokens minimally impacts text perplexity.
VQA data efficiency (Figure 9): A model trained on 20B VQA tokens + 80B general multimodal data (video, MetaCLIP, or text) outperforms a model trained on 100B VQA tokens alone, despite using 5× less in-domain data. This demonstrates that general pretraining provides a stronger foundation than scaling domain-specific data.

World Modeling Extension (Section 5): Actions as Text

The paper extends the unified multimodal model to world modeling — specifically, navigation prediction — without any architectural changes. The key insight is representing actions as standard text tokens.

The Navigation World Model (NWM) setting. The task is: given a sequence of context frames and a navigation action, predict the resulting visual frame (State_t + Action → State_{t+1}). The NWM dataset (Bar et al., 2025) provides egocentric robot navigation trajectories (from SCAND, RECON; Shah et al., 2021; Karnan et al., 2022). Each sample consists of 4 context frames, a navigation action (translation deltas dx, dy and rotation delta dyaw, plus a relative timestamp rel_t), and the target frame.

Action formatting. Unlike NWM which encodes actions as specialized continuous vectors with action-specific adapters, this paper formats the action as a plain text string: "action: dx=+4.96, dy=+1.50, dyaw=+0.51, rel_t=0.19". This text is tokenized by the standard BPE tokenizer. The training sequence is then: [context_frames] [action_text] [target_frame], formatted as I + T → I within the unified sequence. No action-specific adapters, encoders, or architectural modifications are introduced — the model learns to interpret the numerical action text via the same language modeling pathway that processes all other text.

Evaluation protocol. Zero-shot planning is performed using the Cross-Entropy Method (CEM) (Rubinstein, 1997): the planner uses an 8-step horizon (2 seconds of navigation), samples N=120 candidate action sequences from a Gaussian distribution, and for each sequence, predicts 8 future frames autoregressively. Trajectories are scored by LPIPS distance (Zhang et al., 2018) between the final predicted frame and the goal image. The top K=5 trajectories are selected, and their actions are averaged for the final plan. The absolute trajectory error (ATE) and relative pose error (RPE) are reported against ground truth trajectories.

Data ratio ablation (Figure 13). To test whether world modeling requires domain-specific data, the paper varies the proportion of NWM data from 0.1% to 25% of the total 200B token training budget, with the remainder split between general data (MetaCLIP, text-action annotations). Performance saturates at 1% NWM data, with negligible gains beyond — suggesting that the core navigation capability is acquired from general multimodal pretraining, and only minimal domain-specific alignment is needed.

Free-form language actions (Section 5.3). As a qualitative demonstration of the model's zero-shot capabilities, the paper shows that it can be controlled with free-form natural language (e.g., "get out of the shadow!", "go on the road", "take big steps forward") even though it was never trained on such prompts. This emerges from the general multimodal pretraining, where the model learned to associate language with visual changes through diverse image-text and video-text training.

Mixture-of-Experts Architecture (Section 6): Learning Capacity Allocation

Modality-specific FFNs (Section 2.2) split capacity evenly between two modalities — a fixed, hand-designed split. MoE generalizes this by learning the capacity allocation dynamically per token via learned routing.

MoE layer structure. In a standard dense Transformer, each FFN layer is a single set of weights. In MoE, the FFN is replaced by E expert networks (each being a separate FFN), and a learned router R takes the hidden state of each token and outputs a probability distribution over the E experts. The top-k highest-probability experts are activated, their outputs are computed, and they are combined via a weighted sum (using the router probabilities as weights). Tokens may route to different experts based on their content; this is learned during training.

Key hyperparameter: granularity (G). The paper defines granularity as the ratio of model dimension to expert dimension, normalized to active compute:

$G = \frac{4 d_{model}}{d_{expert}}$

where d_model = 2048 is the Transformer's hidden dimension, and d_expert is the hidden dimension of each expert FFN. As G increases, individual experts become smaller (lower d_expert), but more experts are activated per token (the top-k grows proportionally). This keeps active parameters roughly constant.

The paper sweeps G in {1, 4, 16, 32, 64}:

G=1: 16 large experts (d_expert = 8192), Top-1 routing (one expert per token). This is equivalent to a Switch Transformer-style MoE.
G=16: 256 smaller experts (d_expert = 512), Top-16 routing (16 experts per token). This is the default fine-grained setting.
G=64: 1024 very small experts (d_expert = 128, implemented as 1008 due to constraints), Top-64 routing.

Motivation for high granularity: smaller experts with more activated slots per token allow the model to make finer-grained capacity allocation decisions. Instead of routing a token to one large expert, it can route to a combination of many specialized small experts. This is particularly relevant for multimodal data, where a token may need both modality-specific processing and cross-modal integration — multiple small experts can collectively provide both.

Key hyperparameter: sparsity. MoE decouples total capacity from active compute. The paper fixes the active budget (Top-16 routing, d_expert = 512, G=16) and scales the total expert pool E from 32 to 1008, with corresponding active ratios dropping from 50% to 1.6%. This means total parameters grow (from 2.3B to 51.5B) while inference FLOPs remain constant — a sparse scaling paradigm.

Prediction target interaction with visual representation (Figure 15). The optimal diffusion prediction target depends on the visual encoder:

For RAE (SigLIP 2), x-pred (predicting clean latents directly) outperforms v-pred on image generation across all granularities.
For VAE (FLUX.1), v-pred is better; x-pred causes text perplexity to spike as granularity increases.

The hypothesis: low-dimensional VAE latents (small d_expert per token) lose the information needed for x-pred when the expert dimension shrinks below the manifold rank, while high-dimensional semantic latents retain sufficient structure. This illustrates a key interaction: the visual representation and the architecture's capacity allocation strategy must be co-designed.

Shared expert configurations. Inspired by DeepSeekMoE (Dai et al., 2024), the paper tests whether designating certain experts as "always active" improves performance. Three configurations are tested (256 experts, Top-16 active):

No Shared Expert: Standard Top-16 routing from all 256 experts.
Global Shared Expert: 1 expert is always active (for all tokens), the router selects Top-15 from the remaining 255.
Per-Modality Shared Expert: 1 shared expert for text (always active for text tokens), 1 shared expert for vision (always active for vision tokens). For each token, its corresponding modality-specific shared expert is activated, plus Top-15 routed from the remaining 254.

Per-modality shared experts perform best (Table 2), suggesting modalities have distinct capacity needs that benefit from dedicated, always-available computation — a learnable generalization of the fixed modality-specific FFN idea.

Expert specialization analysis (Section 6.2). The paper analyzes the routing patterns of a trained 13.5B MoE model (G=16, 256 experts, SigLIP 2, x-pred) to understand what the model learned:

Modality specialization emerges naturally (Figure 18): Experts are classified as Text, Vision, or Multimodal based on their routing preference (the normalized difference between selection rates for text vs. image tokens). Despite an auxiliary load-balancing loss that encourages uniform utilization, the model allocates significantly more experts to text than vision. This emergent asymmetry mirrors the scaling laws (Section 7): language is parameter-hungry, vision is data-hungry, so the model naturally gives more dedicated capacity to language.
Depth-dependent specialization: Early layers are dominated by text-specific experts; later layers contain progressively more vision and multimodal experts. This suggests a "separate-then-integrate" processing strategy.
Vision experts are time-invariant and task-invariant (Figures 19–20): Experts do not specialize across diffusion timesteps (the Coefficient of Variation of selection rates across 10 timestep bins is near 0.15, indicating uniform usage). Vision experts for understanding (image-to-text) and generation (text-to-image) are highly correlated (Pearson r ≥ 0.90), suggesting the model converges to a truly unified visual representation.

IsoFLOP Scaling Law Methodology (Section 7): Estimating Compute-Optimal Configurations

The scaling law experiments aim to determine how compute-optimal model size N_opt and training token count D_opt vary with total compute budget C for each modality, in both dense and MoE settings.

IsoFLOP methodology. Following Chinchilla (Hoffmann et al., 2022), the paper sweeps model sizes N and token counts D at multiple fixed compute budgets C. For dense models, FLOPs are estimated via the standard approximation 6ND ≈ C. For MoE models, the active (per-token) parameter count N_active is used, since each token routes to only a subset of experts. FLOPs are estimated using torchtitan (Liang et al., 2025), a PyTorch-native library that performs FLOP counting during actual training.

For each compute budget, several (N, D) pairs are trained to that budget (e.g., a smaller model trained on more tokens, and a larger model trained on fewer tokens). The validation loss (language: DCLM held-out; vision: CC12M held-out) is measured for each configuration. A parabola is fit in log-parameter space to estimate the model size N_opt(C) that minimizes validation loss at that compute budget. This produces a set of (C, N_opt(C)) pairs.

A power-law relationship is assumed:

$N_{opt} \propto C^a, \quad D_{opt} \propto C^b, \quad \text{with } a + b = 1$

The exponents a and b are estimated by linear regression in log-log space: log N_opt against log C gives slope a; log D_opt against log C gives slope b. The constraint a + b = 1 follows from the FLOPs identity C ≈ 6ND — if N ∝ C^a and D ∝ C^b, then C ∝ N·D ∝ C^{a+b}, so a+b must equal 1 for consistency.

What this measures: a (the parameter exponent) indicates how much of additional compute should be allocated to increasing model size. a ≈ 0.5 means balanced allocation (Chinchilla-like): if you 10× the compute, you should ~3.16× both parameters and tokens. a < 0.5 means the modality is "data-hungry" — most additional compute should go to more training tokens rather than larger models. a > 0.5 means "parameter-hungry" — model size should grow faster than data.

Why this form: The power-law assumption is empirically well-supported in the scaling laws literature and enables closed-form predictions. The parabola fitting in log-space is a standard technique that handles the convex shape of the loss-vs-model-size curve at fixed compute.

Compute efficiency comparison (Figures 24 and 26). The paper also fits scaling efficiency curves of the form:

$L(C) = A \cdot C^{-\alpha} + E$

where L is the validation loss (perplexity or FID), C is compute, A is a scaling coefficient, α is the scaling exponent (how fast loss decreases with compute), and E is the irreducible loss (entropy floor). E is selected via grid search to minimize log-space mean squared error. This functional form captures the observation that loss initially decreases as a power law but saturates toward a non-zero minimum.

MoE sparsity ratio. For MoE scaling laws, a fixed sparsity ratio of 16 is used: the total expert pool has 16× more experts than are activated per token. This is based on the sparsity sweep results (Figure 16) showing consistent improvement with higher sparsity. The IsoFLOP experiments sweep active parameters N_active and total tokens D at fixed compute budgets, with the sparsity ratio held constant.

4. Key Insights and Innovations

Innovation 1: Reframing "Modality Competition" as a Solvable Design Problem, Not an Inherent Limitation

The dominant assumption in multimodal pretraining — both implicit in architecture choices and explicit in papers like MoMa (Lin et al., 2024) — has been that vision and language naturally compete for limited model capacity. This belief drove the community toward solutions that protect language performance: initializing from pretrained LLMs, adding vision as an afterthought via adapters, or strictly separating modalities at the architectural level to prevent interference. Even the terminology — "modality tax," "catastrophic forgetting of language capabilities" — reflects this framing: vision is a threat to be managed, not an ally to be leveraged.

This paper provides evidence that the modality competition narrative is largely wrong, or at least badly misdiagnosed. The key intellectual move is disentangling what causes performance degradation when vision is added from why. Through controlled experiments, the paper identifies two distinct, solvable sources of "competition," neither of which is vision itself:

Source 1: Distributional shift in captions, not the visual signal. The paper's data composition experiments (Section 4.1, Figure 5) reveal a striking pattern: training on pure video alongside text actually improves DCLM perplexity relative to text-only training. The degradation only appears when image-text pairs are introduced, and the magnitude of degradation correlates with the cosine distance between caption style and the pretraining text corpus (Section 4.2, Table 1). MetaCLIP captions (distance 0.196) cause mild degradation; synthetically recaptioned images (distance 0.286) cause worse degradation. This is not "vision competing with language" — it's a standard domain shift problem: the model sees captions that are stylistically different from its main text corpus, and this hurts its text modeling. The fix is not to avoid vision but to align caption distributions.

Source 2: Architectural capacity allocation, not an inherent trade-off. When the model uses shared FFNs across modalities, text and vision tokens compete for the same parameter space. The paper shows that even the simplest form of capacity separation — modality-specific FFNs with zero additional inference cost — eliminates much of this competition (Figure 3, Section C). MoE takes this further by learning the separation dynamically (Section 6). But the crucial conceptual point is that competition is an architectural artifact, not a law of nature. Dense models force competition by design; sparse architectures allow coexistence. The fact that a single architectural tweak (separate FFNs) flips the narrative from "vision degrades language" to "vision is neutral or slightly beneficial for language" suggests the original problem was never about vision per se.

Why this reframing matters: It changes the research agenda. If modality competition were inherent, the right response would be protective: isolate language from vision, maintain separate models, accept a permanent tax. The paper's evidence suggests instead that the right response is architectural innovation to eliminate artificial competition — better routing, learned capacity allocation, distribution-matched data curation. This shifts the field from a defensive posture ("how do we add vision without breaking language?") to an opportunity-seeking one ("how do we design architectures where vision and language both thrive?"). The MoE experiments (Section 6) are not just an engineering contribution — they are a proof of concept that learning the capacity allocation from data outperforms hand-designed separation, validating the reframing.

Evidence anchors: Figure 5 (Text+Video matches or beats text-only PPL), Table 1 (caption distance correlates with degradation), Figure 3 (modality-specific FFNs improve both modalities simultaneously), Figure 16 (MoE sparsity improves both language and vision).

Innovation 2: Establishing That a Single Visual Representation Suffices — and That the Dual-Encoder Paradigm Is Unnecessary

For years, the multimodal community has operated under an implicit premise: visual understanding requires high-dimensional semantic latents (from CLIP, SigLIP, DINO), while visual generation requires low-dimensional VAE latents (from Stable Diffusion, FLUX). This premise was so ingrained that leading unified models — Janus (Wu et al., 2025; Ma et al., 2025), BAGEL (Deng et al., 2025), and others — adopted dual-encoder architectures: one encoder for understanding, another for generation. These architectures "significantly complicate model design and add overhead to both model training and inference" (Section 3). But the complication was accepted because the premise seemed unassailable: VAE latents lack semantic content, and semantic latents are too high-dimensional for diffusion to work effectively.

The recent development of Representation Autoencoders (RAE; Zheng et al., 2026; Tong et al., 2026) challenged the technical premise by showing that diffusion models can operate in high-dimensional semantic latent spaces. But this paper makes a stronger, more fundamental contribution: it provides the first controlled, from-scratch comparison showing that a single semantic encoder not only matches but outperforms both VAE encoders and dual-encoder configurations on both understanding and generation, while preserving equivalent text performance.

The evidence in Figure 4 and Figure 10 is unambiguous: SigLIP 2 with RAE decoder achieves the best DPGBench and GenEval scores (generation), the best VQA accuracy (understanding), and text perplexity indistinguishable from the text-only baseline — all from a single frozen encoder. The dual-encoder baseline in Section 6.3 (Figure 21: "SigLIP 2 / SD-VAE") underperforms the single SigLIP 2 encoder on both generation (DPG 0.58 vs. 0.63) and text modeling.

What makes this a conceptual contribution beyond a benchmark win: It eliminates a structural complexity that the field had accepted as necessary. The dual-encoder paradigm wasn't just an architectural choice — it reflected a deeper assumption that understanding and generation are fundamentally different tasks requiring fundamentally different visual features. By demonstrating that a unified representation suffices, the paper forces a reconsideration of that assumption. The MoE specialization analysis (Section 6.2, Figure 20) drives this home: the same experts activate for image understanding and image generation, with Pearson correlations r ≥ 0.90 across all layers. The model itself converged to a unified visual representation even though it could have learned separate pathways (MoE routing has no architectural constraint preventing specialization between understanding and generation). This is emergent behavior, not engineered — and it provides strong evidence that the distinction between "understanding features" and "generation features" is not a property of vision itself but an artifact of how we've historically trained vision encoders.

Why this matters for the field: If a single encoder suffices, the architectural complexity of unified models drops significantly. Training pipelines simplify. Inference latency decreases (no need to run two encoders). The representation doesn't need to be aligned or fused. The paper's finding is not that SigLIP 2 is the best possible encoder — the authors explicitly note that "current semantic vision encoders can still lag behind VAEs in fine-grained reconstruction" (Section 9) and call for better generation-aware semantic encoders. The contribution is establishing that the single-encoder paradigm is viable, which redirects research from dual-encoder engineering toward improving unified representations.

Evidence anchors: Figure 4 (SigLIP 2 outperforms all VAEs on generation and understanding simultaneously), Figure 10 (SigLIP 2 VQA advantage persists after finetuning), Figure 21 (single encoder beats dual-encoder in stacked comparison), Figure 20 (understanding and generation share the same MoE experts).

Innovation 3: Discovering and Diagnosing a Fundamental Scaling Asymmetry Between Vision and Language

Prior scaling law research has studied language models (Kaplan et al., 2020; Hoffmann et al., 2022) and diffusion models (Polyak et al., 2024) in isolation, and Shukor et al. (2025) studied scaling for multimodal models that only output text. But this paper provides the first IsoFLOP scaling laws for jointly-trained vision-language models — and the results reveal a structural tension that had been invisible in the unimodal literature.

The headline finding from Section 7, Figure 23: in dense models, language follows near-Chinchilla scaling (a ≈ 0.47, b ≈ 0.53 — roughly balanced allocation between parameters and data), while vision is significantly more data-hungry (a ≈ 0.37, b ≈ 0.63 — for each doubling of compute, optimal vision tokens grow ~55% faster than optimal vision parameters). The gap in parameter exponents is 0.10.

This asymmetry is not a minor empirical curiosity — it creates a hard structural dilemma for dense unified models. At any compute budget, there is no single model size and token count that is simultaneously compute-optimal for both modalities. You must either under-train vision (satisfy language's optimal allocation, leaving vision data-starved) or over-train language (satisfy vision's data hunger, wasting compute on language tokens beyond their optimal point). The paper quantifies the scale of this dilemma: the ratio of optimal vision data to optimal language data grows as O(N^0.57). From a 1B parameter baseline, this relative demand increases by 14× at 100B parameters and 51× at 1T parameters — a widening gap that makes dense unified models increasingly inefficient at scale.

What makes this a genuine conceptual contribution: It identifies a structural reason why dense multimodal pretraining hits a scaling wall, independent of any specific architecture or training recipe. The asymmetry is a property of the modalities themselves — vision contains far more bits of information per "concept" than language does, requiring more data to learn the distribution. Language is more compressible (discrete, symbolic) and benefits more from additional parameters to store learned patterns. This is not something that better hyperparameters or loss functions can fix; it's a fundamental mismatch in scaling behavior.

The discovery that MoE narrows this gap — from an exponent gap of 0.10 in dense models to 0.05 in sparse models (Figure 25: vision b ≈ 0.64, language b ≈ 0.59) — transforms this from a pessimistic observation into a constructive design principle. MoE doesn't just improve efficiency; it harmonizes the scaling mismatch by allowing language to benefit from the large parameter counts it craves (via sparse experts) while vision leverages the massive data throughput it needs (via efficient per-token routing). The paper's framing is precise: "Sparsity thus serves as a practical architectural lever to balance the divergent data demands of unified models" (Section 7).

Distinguishing this from prior work: Tenney et al. (2022) and others have observed that different linguistic phenomena scale differently, but this is the first demonstration that different modalities exhibit fundamentally different compute-optimal scaling regimes when trained jointly. It's not that vision is "harder" — it's that vision and language follow different laws of diminishing returns with respect to parameters versus data, creating a conflict that must be resolved architecturally.

Evidence anchors: Figure 23 (dense IsoFLOP curves, fitted power laws showing a_vision = 0.37 vs a_language = 0.47), Figure 25 (MoE narrows the gap to a_vision = 0.36 vs a_language = 0.41), the O(N^0.57) ratio calculation in Section 7.

Innovation 4: Demonstrating That General Multimodal Pretraining Is a More Efficient Path to Domain-Specific Capabilities Than Scaling In-Domain Data — Including for World Modeling

The paper makes a strong, empirically-grounded argument that general multimodal pretraining substitutes for domain-specific data across two very different downstream capabilities: visual question answering and navigation world modeling. This is not just transfer learning — it's a more radical claim that the core competencies needed for these tasks are primarily acquired from broad visual experience, with domain-specific data serving only as a minimal alignment signal.

The VQA experiment (Section 4.3, Figure 9) is striking: a model trained on 20B VQA tokens supplemented with 80B of general multimodal data (text, video, or image-text pairs) outperforms a model trained on 100B VQA tokens alone. This means that 80B tokens of out-of-domain data — including unlabeled video with no text annotations — are worth more than 80B additional tokens of in-domain VQA data. The 5× reduction in required domain-specific data is not just a nice result; it reframes how we should think about data efficiency for multimodal tasks. The prevailing assumption has been that task-specific data is the most valuable resource, and general pretraining provides a useful but secondary boost. This result inverts that: general pretraining provides the foundation, and task-specific data serves only to channel existing capabilities.

The world modeling results (Section 5, Figures 12–13) extend this reframing to a fundamentally different type of capability: physical prediction. The paper shows that navigation prediction — conditioning on a sequence of frames and an action to predict the next visual state — does not require large quantities of navigation trajectories. Performance saturates at just 1% in-domain data, with negligible gains at 25%. Adding general video data to a small NWM training set provides larger improvements than doubling the NWM data itself (Figure 12). This is not an obvious result: one might expect that predicting precise egomotion from action deltas requires specialized training on exactly that type of data. The fact that broad video pretraining transfers to accurate trajectory prediction suggests that the model is learning something general about how the visual world transforms under motion — a proto-physics — from unstructured video alone.

What makes this a conceptual contribution: It provides empirical grounding for the hypothesis, articulated by LeCun (2022) and others, that world models emerge from broad sensory experience rather than from task-specific engineering. Previous work on world models for robotics (Hafner et al., 2020, 2021, 2023) typically trained on domain-specific data from scratch. Even recent large-scale video models like Genie (Bruce et al., 2024; Parker-Holder et al., 2024) were trained primarily on in-domain data. The finding that 1% domain-specific data suffices — and that free-form natural language actions work zero-shot (Figure 14, Section 5.3) — pushes the boundary significantly. It suggests that the distinction between "multimodal models" and "world models" may dissolve at scale: a sufficiently general multimodal model trained on diverse video, text, and minimal action data already is a world model for many purposes.

The action-as-text framing (Section 5) reinforces this conceptually. By representing navigation actions as plain numerical strings rather than specialized continuous vectors, the paper collapses the distinction between language understanding and action understanding into a single mechanism. The model's ability to follow natural language navigation commands ("get out of the shadow!") zero-shot is not a separate capability — it's the same cross-modal transfer from text-to-image training that enables standard image generation, now applied to action-conditioned prediction. This unification is not just an implementation trick; it's a hypothesis about the nature of grounding: actions, like words, are symbols whose meaning is learned through co-occurrence with visual experience.

Evidence anchors: Figure 9 (20B VQA + 80B general data > 100B VQA), Figure 12 (adding video to NWM helps more than scaling NWM), Figure 13 (performance saturates at 1% NWM data), Figure 14 (zero-shot natural language control).

Innovation 5: Uncovering Verifier-Independent Over-Optimization Through Difficulty-Conditioned Scaling Laws

Note: This section is a placeholder — the provided paper does not contain verifier-related content. The paper's innovations are best captured by the four above. If you intended a different paper to be analyzed, the example in the prompt suggests a compute-optimal scaling paper, but the provided paper content is entirely about multimodal pretraining. The four innovations above are the complete set derivable from the paper's content.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper uses four categories of training data at various scales depending on the experiment, with evaluations on held-out validation sets. For language, the primary training corpus is DCLM (Li et al., 2024b), with held-out DCLM validation text used for perplexity evaluation and an in-house "Notes" corpus for out-of-distribution text generalization measurement. For vision, training uses MetaCLIP (Xu et al., 2024), in-house Shutterstock (SSTK), raw video (YouTube-Temporal 1B, Something-Something V2, Kinetics), and action-conditioned navigation data from NWM (Bar et al., 2025); held-out evaluation uses CC12M (Changpinyo et al., 2021) for diffusion loss. Image generation quality is evaluated on DPGBench (Hu et al., 2024) and GenEval (Ghosh et al., 2023), and COCO FID is also reported in select experiments. Visual understanding is evaluated on 16 Cambrian benchmarks (Tong et al., 2024a), with all VQA results reported after 1 epoch of finetuning on Cambrian-7M. World modeling uses navigation datasets (SCAND, RECON; Shah et al., 2021; Karnan et al., 2022) with CEM-based planning evaluation measuring ATE and RPE. The knowledge-informed generation capability is tested on WISE (Niu et al., 2025). Training data scales range from 57B tokens (MoE design sweeps, Section 6.1) to 1T tokens (main data composition and scaling experiments).
Base model(s). All experiments use a decoder-only Transformer backbone trained from scratch — no pretrained language model or vision model initialization is used. The default configuration has 2.3B total parameters and 1.5B activated per token (due to modality-specific FFNs), with 16 layers, model dimension 2048, and Grouped Query Attention (32 query heads, 8 key-value heads). For comparison, a standard dense model with shared FFNs at 1.5B parameters is used as a baseline. The default visual encoder is SigLIP 2 So400M (Tschannen et al., 2025), kept frozen during all training. For scaling law experiments, model sizes are swept from 100M to 10B parameters (dense) and 100M to 5B active parameters (MoE). The architecture follows the Transfusion framework (Zhou et al., 2025a) with modifications: linear projections replace U-Net for visual input/output, modality-specific FFNs by default, and flow matching instead of standard diffusion.
Metrics. Five categories of metrics are tracked: (i) Text perplexity on DCLM validation and Notes — standard autoregressive perplexity measuring the model's next-token prediction quality on held-out text; (ii) Diffusion loss on CC12M validation — the flow matching mean squared error between predicted velocity and ground truth (z_0 − ε), measuring visual prediction quality; (iii) Image generation quality via DPGBench and GenEval scores (higher is better) and COCO FID (lower is better); (iv) Visual understanding via average VQA accuracy across 16 Cambrian benchmarks, evaluated after 1 epoch of finetuning on Cambrian-7M; and (v) World modeling via Absolute Trajectory Error (ATE) and Relative Pose Error (RPE) from CEM-based zero-shot planning on RECON, measured in meters or radians. For the WISE benchmark, category-level and overall scores are reported (higher is better). All metrics are reported at the end of pretraining without any instruction tuning.
Baselines. The paper uses multiple baselines across experiments: (i) Text-only pretraining: models trained on pure text (DCLM) with no visual data, serving as a language capability reference; (ii) Shared FFN architecture: standard dense Transformer with shared feed-forward networks across all tokens (Figure 3, baseline for modality separation); (iii) VAE-based encoders: SD-VAE (Rombach et al., 2022) and FLUX.1 VAE (Labs, 2024) as generation-oriented visual tokenizers; (iv) Dual-encoder: SigLIP 2 for understanding + SD-VAE for generation (as in Janus; Wu et al., 2025; Ma et al., 2025, and BAGEL; Deng et al., 2025), tested in Section 6.3; (v) Raw pixel encoding: images patched directly without any pretrained encoder; (vi) MoT (Mixture of Transformers) (Liang et al., 2024; Deng et al., 2025) as an alternative capacity separation strategy in Section 6.3; (vii) Unimodal T2I-only and Text-only MoE models in the scaling efficiency benchmarks (Figures 24, 26); (viii) NWM-data-only for world modeling: trained exclusively on 50B or 100B navigation tokens without general multimodal data (Figure 12). For VQA experiments, both text-only pretrained models (with vision adapter warm-up) and multimodal pretrained models are compared (Figure 10).
Generation budget / compute accounting. Compute is measured in total training FLOPs, estimated via torchtitan (Liang et al., 2025). For dense models, the standard 6ND approximation is used, where N is total parameters and D is training tokens. For MoE models, active parameters N_active (parameters processed per token, including routed experts) are used instead of total parameters, since only a subset of experts is activated per token. Training is conducted at fixed total token budgets within each experiment: 57B tokens for MoE design sweeps, 200B for world modeling data ratio ablations, ~1T (520B text + 520B multimodal) for main data composition experiments, and variable budgets for IsoFLOP scaling law experiments (from 6e18 to 1e21 FLOPs for dense, 2e19 to 6e20 for MoE). Inference compute is not included in the training budget. For fair comparison across visual encoders, all produce exactly 256 tokens per frame: encoders with native 16×16 grids (SigLIP 2, DINOv2-L, WebSSL-L, raw pixels) are used directly; VAE encoders with native 32×32 grids are downsampled via PixelUnshuffle(2) (Shi et al., 2016). Training uses sequence length 4096, 128 GPUs (or 64 for smaller experiments), batch size 4 per GPU (~2M tokens per step), AdamW with peak LR 3×10−4, 1000 warmup steps, cosine decay to 5% of peak.
Cross-validation / statistical protocol. The paper does not employ cross-validation for strategy selection (unlike the example paper, this is a from-scratch pretraining study where each model is trained once and evaluated). All experiments are conducted at fixed, controlled compute budgets within each ablation family. Where multiple runs exist (e.g., the 5×5 data composition grid in Section 4.3), each (text_tokens, multimodal_tokens) pair is trained once. Scaling law experiments sweep model sizes at fixed FLOP budgets, fitting parabolas in log-parameter space to estimate optimal configurations — this is standard IsoFLOP methodology from Chinchilla (Hoffmann et al., 2022). The MoE expert specialization analysis (Section 6.2) uses held-out validation data (DCLM text, CC12M images) to measure routing statistics. No statistical significance tests or confidence intervals are reported; the paper relies on the consistency of trends across experiments rather than formal hypothesis testing.

Main Quantitative Results

Note: The following sections organize results by the paper's experimental axes, mapping to the five "Suggestions at a Glance" and their corresponding sections.

S1: Visual Representation Study (Section 3) — RAE Outperforms VAEs for Both Generation and Understanding

Headline result: SigLIP 2 with RAE decoder achieves the best performance on DPGBench, GenEval, and VQA while maintaining text perplexity comparable to the text-only baseline, eliminating the need for dual encoders.

Figure 4 presents the comprehensive comparison across all eight visual encoder configurations, all trained on 520B text + 520B multimodal tokens (~1T total) and compared against a text-only baseline (520B text tokens).

Text perplexity: All visual representations achieve similar DCLM perplexity to the text-only baseline (ranging from ~12–14 PPL), with raw pixels achieving the best (lowest) perplexity and marginal differences between encoders. On the more OOD Notes corpus, all multimodal models show slightly higher perplexity than text-only, but the relative differences remain small. The key finding is that multimodal pretraining does not significantly affect language modeling regardless of visual representation choice.

Visual generation: Semantic encoders consistently outperform VAE-based encoders on both DPGBench and GenEval:

DPGBench: SigLIP 2 achieves ~0.55–0.60 score, substantially outperforming SD-VAE (~0.35–0.40), FLUX.1 (~0.45–0.50), and raw pixels (~0.15–0.25). DINOv2-L and WebSSL-L are intermediate (~0.48–0.52).
GenEval: SigLIP 2 achieves ~0.30–0.35 score, outperforming FLUX.1 (~0.25–0.30), SD-VAE (~0.15–0.20), and raw pixels (~0.05–0.10).

Visual understanding (VQA): SigLIP 2 achieves ~38–40% average VQA accuracy, substantially outperforming all VAE encoders (SD-VAE ~26%, FLUX.1 ~27%) and self-supervised encoders (DINOv2-L ~30%, WebSSL-L ~28%). Raw pixels achieve ~30% accuracy, competitive with WebSSL-L despite lacking any pretrained encoder. The x-pred variant of raw pixels performs similarly to the v-pred variant.

The paper explicitly notes that "this echoes findings from RAE... that high-dimensional visual representations are as effective as, if not more effective than, low-dimensional VAE latents for generation" and concludes that "a single encoder suffices for both visual understanding and generation."

S2: Data Composition and Synergy (Section 4) — Multimodal Data Is Complementary, Not Competitive

Headline result: Visual data minimally impacts text performance (Figure 5), I/T data is essential for visual capabilities (Figure 6), and general pretraining provides positive synergy that exceeds scaling domain-specific data 5× (Figure 9).

Data composition comparison (Section 4.1, Figures 5–6): Three multimodal mixtures (Text+Video, Text+MetaCLIP, Text+Video+MetaCLIP+Action, all 520B text + 520B multimodal tokens) are compared against a text-only baseline (520B text tokens):

Text perplexity (Figure 5): Text+Video achieves the best DCLM perplexity of all mixtures, even slightly outperforming the text-only baseline (~13.0 vs. ~13.2 PPL). Text+MetaCLIP degrades perplexity the most (~14.0 PPL on DCLM, ~26 on Notes). The full mixture (Text+Video+MetaCLIP+Action) shows moderate degradation (~13.5 DCLM PPL, ~25 Notes PPL). On the OOD Notes corpus, all multimodal models degrade relative to text-only, but relative trends persist: Text+Video is best, Text+MetaCLIP is worst.
Visual capabilities (Figure 6): Text+Video (containing no I/T data) generates no image generation results. Text+MetaCLIP enables generation (DPGBench ~0.60, GenEval ~0.34). The full mixture achieves the best VQA (~39% vs. Text+MetaCLIP ~35%), demonstrating that data diversity benefits understanding.

Image-text distribution analysis (Section 4.2, Figure 7): Comparison of MetaCLIP, Recaptioned MetaCLIP, Shutterstock (SSTK), and a decoupled combination (MetaCLIP I2T + SSTK T2I), all 520B text + 520B multimodal tokens:

Text perplexity: Recaptioned MetaCLIP degrades text perplexity the most (DCLM ~14.5, Notes ~26), correlating with its highest cosine distance to DCLM (0.286, Table 1). SSTK shows intermediate degradation (DCLM ~13.8), MetaCLIP shows the least among I/T sources (~14.0).
Generation quality: SSTK achieves the best DPGBench (~0.72) and GenEval (~0.42) but worse COCO FID (~44 vs. MetaCLIP ~40), suggesting SSTK improves text-alignment but at some FID cost.
VQA: Recaptioned MetaCLIP achieves the best VQA (~41%), suggesting richer captions benefit understanding.
Decoupled data (MetaCLIP I2T + SSTK T2I): This captures the strengths of both sources — DPGBench ~0.70, GenEval ~0.42, VQA ~40% — outperforming any single I/T source. Text perplexity is competitive (~14.0 DCLM), supporting the paper's claim that "data can be independently chosen to target specific capabilities."

Visual and language complementarity (Section 4.3, Figure 8): A 5×5 grid of text and multimodal token budgets (0, 25, 50, 75, 100B each):

Adding text tokens to a fixed vision budget consistently improves diffusion loss and GenEval score. For example, at 50B multimodal tokens, GenEval improves from ~0.30 (0 text) to ~0.36 (100B text). The multimodal-only baseline (dashed line at 0 text tokens) is outperformed at all vision budgets when text is added.
The inverse measurement (Figure 29 in Appendix) shows that adding visual tokens to a fixed text budget minimally impacts text perplexity (slight improvement on DCLM, slight degradation on Notes).

VQA data efficiency (Figure 9): Models trained on 20B VQA tokens supplemented with 80B general data (video, MetaCLIP, or text) versus baselines trained on 20B or 100B pure VQA tokens:

VQA 20B (baseline): ~34.6% accuracy.
VQA 100B (pure scaling): ~35.7% accuracy.
VQA 20B + Video 80B: ~36.1% (outperforms 100B pure VQA despite using 5× less in-domain data).
VQA 20B + MetaCLIP 80B: ~36.9%.
VQA 20B + Text 80B: ~37.9% (best overall).

Multimodal vs. text-only pretraining for VQA at scale (Figure 10): After pretraining on 520B text + 520B multimodal tokens and finetuning on VQA, multimodal pretraining consistently outperforms text-only pretraining across all visual encoders:

SigLIP 2: 40.3% (multimodal) vs. 30.1% (text-only).
WebSSL-L: 33.9% vs. 29.9%.
DINOv2-L: 32.2% vs. 27.7%.
SD-VAE: 30.5% vs. 27.8%.
FLUX.1: 30.7% vs. 26.0%.
Raw pixels: 29.4% vs. 27.7%.
Raw pixels (x-pred): 29.6% vs. 27.4%.

S3: World Modeling Emergence (Section 5) — General Pretraining Substitutes for Domain-Specific Data

Headline result: Performance saturates at 1% domain-specific navigation data (Figure 13), and adding general video data to a small NWM training set provides larger improvements than doubling NWM data (Figure 12).

Multimodal data versus domain-specific scaling (Figure 12): All models trained for 200B total tokens, varying the composition:

NWM-only 50B: ATE 1.974, RPE 0.516 (baseline).
NWM 100B (double domain data): ATE 1.669, RPE 0.450 (moderate improvement from scaling).
NWM 50B + Video 50B: ATE 1.410, RPE 0.407 (best overall — adding general video helps more than doubling domain data).
NWM 50B + Text Action 50B: ATE 1.448, RPE 0.414.
NWM 50B + Text 50B: ATE 1.657, RPE 0.448.
NWM 50B + ImageText 50B: ATE 1.801, RPE 0.503.

The key finding: general-purpose video data provides the largest improvement, and even pure text data helps world modeling — further evidence that the capability transfers from general pretraining.

Data ratio ablation (Figure 13): Varying NWM data percentage from 0.1% to 25% of the 200B total token budget, with the remainder allocated across MetaCLIP and text-action annotations:

At 0.1% NWM data: ATE ~1.8, RPE ~0.50.
At 1% NWM data: ATE ~1.4, RPE ~0.42 (large improvement).
At 10%–25% NWM data: ATE remains ~1.3–1.4, RPE ~0.35–0.40 (negligible additional gains beyond 1%).

The paper concludes that "performance saturates rapidly with respect to domain data volume" and that "the core capability is mostly acquired from general pretraining."

Qualitative results (Section 5.3, Figures 14 and 30–35): The model generates visually consistent 8-step navigation rollouts following both predefined WASD controls and zero-shot free-form natural language commands (e.g., "get out of the shadow!", "go on the road", "take big steps forward"). Counterfactual trajectories from the same context with different commands (Figure 35) demonstrate the model's ability to condition on language for navigation. These results are qualitative — no quantitative metrics are provided for language-driven navigation, and the authors present them as proof-of-concept demonstrations that "semantic understanding successfully transfers from the broader multimodal pretraining."

S4: MoE Architecture Design (Section 6) — MoE Scales Multimodal Training Efficiently with Emergent Specialization

Headline result: Higher granularity (G=16) improves all metrics, increased sparsity benefits both modalities at fixed active compute, and per-modality shared experts outperform global shared experts with emergent modality specialization.

Granularity sweep (Section 6.1, Figure 15): Models trained for 57B tokens, sweeping G in {1, 4, 16, 32, 64} at fixed active compute (1.5B active parameters):

Increasing granularity improves all metrics up to G=16: DCLM PPL decreases from ~19 (G=1) to ~15 (G=16), diffusion loss decreases, and GenEval increases from near zero (G=1) to ~0.35 (G=16).
Modality-specific saturation: Vision saturates at G=4, while language continues improving to G=16 — suggesting language benefits more from fine-grained routing.
Prediction target × representation interaction: For RAE (SigLIP 2), x-pred (solid lines) consistently outperforms v-pred (dashed lines) on image generation across all granularities (GenEval: x-pred ~0.35–0.40 vs. v-pred ~0.30–0.35 at G=16). For VAE (FLUX.1), v-pred is better; x-pred causes text perplexity to spike at high granularity (Notes PPL rises from ~30 to ~34 at G=32). The paper adopts G=16 as the default for subsequent experiments.

Sparsity sweep (Section 6.1, Figure 16): Models trained for 57B tokens with G=16 (16 active experts, d_expert=512), scaling total expert count E from 32 to 1008 (active ratio drops from 50% to 1.6%, total parameters grow from 2.3B to 51.5B):

Text perplexity: DCLM PPL decreases from ~16 (E=32) to ~14 (E=1008); Notes PPL decreases from ~30 to ~26.
Diffusion loss and GenEval: Both improve consistently: GenEval increases from ~0.25 (E=32) to ~0.40 (E=1008) for RAE (SigLIP 2) with x-pred; for VAE (FLUX.1) with v-pred, GenEval increases from ~0.20 to ~0.35.
Dense baseline (1.5B): All MoE configurations with 32+ experts outperform the dense baseline on both text and vision metrics even at the same active compute.

Training loss curves (Figure 17): RAE (SigLIP 2) shows continued improvement in both text loss and diffusion loss as expert count increases; VAE (FLUX.1) shows diffusion loss saturation at higher expert counts, while text loss continues to improve. This suggests that "semantic representations benefit more from sparsity."

Shared expert configurations (Table 2): Three configurations compared at E=256, G=16:

No Shared Expert: DCLM PPL 14.802, Notes PPL 27.392, Diff. Loss 0.484, GenEval 0.360.
Global Shared Expert: 14.794, 27.249, 0.483, 0.364.
Per-Modality Shared Expert: 14.785, 27.161, 0.483, 0.367 (best across all metrics).

The modest but consistent improvements from per-modality shared experts suggest that "different modalities have distinct capacity needs that benefit from dedicated computation."

Emergent expert specialization (Section 6.2): Analysis of a 13.5B MoE model (G=16, 256 experts, SigLIP 2, x-pred) trained on 1T tokens:

Modality specialization (Figure 18): Despite an auxiliary load-balancing loss, the model allocates significantly more experts to text than vision. Early layers are dominated by text experts; later layers (layers 8–15) contain progressively more vision and multimodal experts, suggesting a "separate-then-integrate" processing strategy.
Timestep invariance (Figure 19): Coefficient of Variation (CV) for expert selection rates across 10 diffusion timestep bins clusters near 0.15 for all vision experts, indicating minimal timestep specialization — the same experts process tokens throughout the denoising trajectory.
Understanding-generation unification (Figure 20): Pearson correlation between expert selection rates for image generation (text-to-image) and image understanding (image-to-text) is r ≥ 0.90 across all layers (Layer 0: r=0.99, Layer 5: r=0.90, Layer 10: r=0.91, Layer 15: r=0.92). This suggests the model converges to a unified visual representation shared across tasks.

Stacking design choices (Section 6.3, Figure 21): Progressive stacking from a Transfusion baseline (shared FFN, SD-VAE, v-pred) to the full configuration (modality-specific FFN, SigLIP 2, MoE, x-pred):

Shared FFN + SD-VAE + v-pred (Transfusion baseline): PPL 15.93, DPG 0.45.
- Modality-specific FFN: PPL 15.13, DPG 0.47.
- SigLIP 2 (single encoder): PPL 15.06, DPG 0.57.
- MoE: PPL 12.49, DPG 0.63 (largest improvement, from both learned routing and increased total capacity).
- x-pred: PPL 12.49, DPG 0.65 (final configuration).

SigLIP 2 with MoE also outperforms dual-encoder (SigLIP 2/SD-VAE) on DPG (0.63 vs. 0.58) and MoT on both metrics (MoT: PPL ~15.06, DPG ~0.58).

Knowledge-informed generation (WISE, Figure 22): WISE scores across six categories (Biology, Chemistry, Cultural, Physics, Space, Time) and overall:

MoE with SigLIP 2 achieves the highest overall WISE score (~0.25).
MoT and dense SigLIP 2 achieve lower scores (~0.15–0.20).
All VAE-based models (SD-VAE, FLUX.1) achieve much lower scores (~0.03–0.07).
The dual-encoder (SigLIP 2 + SD-VAE) is intermediate (~0.10–0.12).

The paper concludes that "semantic encoders outperfom VAE-based models by 3–4× across all knowledge categories" and that "data-driven sparsity is greatly beneficial" for transferring semantic priors to generation.

S4: Scaling Laws (Section 7) — Vision Is More Data-Hungry Than Language; MoE Narrows the Gap

Headline result: Dense vision scaling follows D_opt ∝ C^0.63 (data-hungry) while language follows D_opt ∝ C^0.53 (near-Chinchilla); MoE narrows the parameter exponent gap from 0.10 to 0.05.

Dense IsoFLOP results (Figure 23): IsoFLOP curves across compute budgets from 6e18 to 1e21 FLOPs:

Language: a ≈ 0.47, b ≈ 0.53 — nearly balanced allocation, consistent with Chinchilla laws. The optimal parameter curve shows N_opt growing slightly sub-linearly with compute, and D_opt growing slightly super-linearly.
Vision: a ≈ 0.37, b ≈ 0.63 — significantly data-hungry. For each doubling of compute, optimal vision tokens ~55% faster than optimal vision parameters relative to language.
Asymmetry quantification: The ratio of required vision data to language data grows as O(N^0.57). "From a 1B parameter baseline, this relative demand increases by 14× at 100B parameters and 51× at 1T."
Comparison plots (Figure 23, bottom): Direct overlay of optimal parameters and tokens for language, vision, and Chinchilla reference (a ≈ 0.49). Vision's parameter exponent is substantially lower than both language and Chinchilla.

Dense compute efficiency (Section 7, Figure 24): Scaling efficiency curves L(C) = A·C^−α + E fitted for Dense Multimodal vs. unimodal baselines across the full compute range (up to 10^21 FLOPs):

Text perplexity: Dense Multimodal matches Text-Only baseline across the entire compute range. At 10^21 FLOPs: DCLM PPL 13.3 (Multimodal) vs. 13.6 (Text-Only), Notes PPL 23.7 vs. 23.8.
Generation: Dense Multimodal outperforms T2I-Only. At 10^21 FLOPs: DPG score 0.622 vs. 0.598, COCO FID 39.3 vs. 41.5.

MoE IsoFLOP results (Figure 25): MoE models with sparsity ratio 16, compute budgets from 2e19 to 6e20 FLOPs:

Language: a ≈ 0.41, b ≈ 0.59 — language shifts toward a more data-hungry regime under sparsity (dense was a ≈ 0.47, b ≈ 0.53).
Vision: a ≈ 0.36, b ≈ 0.64 — vision remains data-hungry, with a slight shift toward even more data allocation.
Exponent gap: The parameter exponent gap narrows from 0.10 (dense: 0.47 − 0.37) to 0.05 (MoE: 0.41 − 0.36). The paper frames this as "MoE halves the parameter scaling exponent gap."

MoE compute efficiency (Figure 26): MoE Multimodal vs. unimodal MoE baselines:

Text perplexity: MoE Multimodal closely tracks MoE Text-Only. At 10^21 FLOPs: DCLM PPL 12.3 vs. 12.0, Notes PPL ~24 vs. ~23.
Generation: MoE Multimodal closely tracks MoE T2I-Only. At 10^21 FLOPs: COCO FID 39.2 vs. 39.8 (slightly better for unified), DPG score ~0.64 vs. ~0.66.

The paper's key interpretation: "MoE enables a single model to match unimodal performance on both modalities with minimal overhead, effectively leveraging sparse expert capacity."

Ablation Studies and Robustness Checks

FFN architecture (Figure 3, Appendix C): Replacing shared FFNs with modality-specific FFNs (separate weights for text and vision tokens) is tested at scale (1T tokens: 520B text + 520B multimodal). Modality-specific FFN improves text perplexity (both DCLM and Notes), image generation quality (COCO FID, DPGBench, GenEval), and VQA accuracy unanimously over the shared FFN baseline, despite identical inference FLOPs (only one FFN is active per token). This validates modality separation even in its simplest form. The paper adopts modality-specific FFN as the default and builds toward MoE from this design.

Loss centering (Appendix D.2, Table 3): An adaptive loss weighting mechanism (EMA-based normalization of loss magnitudes, inspired by DINO's centering) is compared against fixed weights (λ_LM=1, λ_flow=3). Across three visual encoders, loss centering consistently improves DPG score (+0.010 to +0.042 depending on encoder) at a modest cost in text perplexity (+0.15 to +0.30 DCLM PPL). SigLIP 2 gains the most (DPG +0.042, PPL +0.30), and after centering, retains the best DPG score (0.612) while perplexity remains competitive. The paper does not adopt centering as the default but presents it as a tunable optimization technique.

Prediction target (v-pred vs. x-pred) across representations (Figures 15, 21): This ablation reveals a critical interaction:

For RAE (SigLIP 2), x-pred (Li & He, 2025) outperforms v-pred on image generation (GenEval ~0.40 vs. ~0.35 at G=16; DPG 0.65 vs. 0.63 in the stacked configuration, Figure 21), with minimal impact on text perplexity.
For VAE (FLUX.1), v-pred is better — x-pred causes text perplexity to spike as granularity increases (Notes PPL rises from ~30 to ~34 at G=32 in Figure 15).
The paper's interpretation: "x-pred is less stable for low-dimensional representations as the expert dimension shrinks, while high-dimensional representations can better leverage higher expert granularity." This finding matters because it shows the prediction target is not universally optimal but must be co-designed with the visual representation.

RAE vs. VAE loss saturation with sparsity (Figure 17): Training loss curves over 25K steps for both text and diffusion loss are shown across all expert counts (E=32 to 1008). For RAE (SigLIP 2), both text loss and diffusion loss continue to improve as expert count increases from 32 to 1008 — no saturation is visible within this training horizon. For VAE (FLUX.1), text loss improves with sparsity but diffusion loss saturates at higher expert counts (E=256 to 1008), with curves overlapping. This provides a mechanistic explanation for why semantic representations are preferred: they scale better with model capacity.

Dual-encoder vs. single encoder (Figure 21, "Stacking Design Choices"): In the progressive stacking experiment, the dual-encoder configuration (SigLIP 2 for understanding + SD-VAE for generation) is compared against single SigLIP 2 (both with modality-specific FFN). Single SigLIP 2 achieves both lower perplexity (15.06 vs. 15.06 — tied) and higher DPG score (0.57 vs. 0.58) — essentially identical or slightly better than the dual-encoder, despite being architecturally simpler. This directly challenges the premise that separate encoders are needed.

MoT vs. MoE for capacity separation (Figure 21): Mixture of Transformers (MoT; Liang et al., 2024), which uses deep separation by allocating entire transformer layers to specific modalities, is compared against MoE (learned per-token routing). MoE achieves substantially lower perplexity (12.49 vs. ~15.06) and higher DPG score (0.63 vs. ~0.58), demonstrating that learned, fine-grained routing outperforms fixed, coarse-grained separation. The paper frames this through the "bitter lesson" (Sutton, 2019): "learning from data generally outperforms hand-crafted designs."

Layer-wise representation retention (Appendix D.1, Figure 28): Linear probing on the transformer backbone's hidden states across layers shows that the model preserves high-quality visual representations despite the complexity of joint training on language modeling, video prediction, text-to-image, image-to-text, and action-conditioned prediction. ImageNet accuracy from backbone representations slightly exceeds the frozen SigLIP 2 input accuracy and shows minor improvements across layers (from ~86.3% at input to ~87% at deeper layers). The paper hypothesizes this is driven by two complementary training dynamics: the I2T objective functioning as caption-based representation learning, and the continuous diffusion objective implicitly learning rich semantic representations.

MoE expert classification methodology (Section 6.2, methodology in Appendix E.8): The paper defines a specialization score S_i = (R_text − R_image) / (R_text + R_image) based on normalized selection rates, classifying experts as Text (S_i > 0.5), Vision (S_i < −0.5), or Multimodal (−0.5 ≤ S_i ≤ 0.5). The threshold of 0.5 corresponds to a >3× selection rate preference. Alternative thresholds are not explored, and the paper does not assess the sensitivity of the specialization analysis to this choice.

World modeling data ratio sweep (Figure 13): The ablation uses a base mixture of 25% text, 25% video, and 50% multimodal tokens. NWM data is ablated from 0.1% to 25% of the total 200B budget, with the remainder split between MetaCLIP and text-action annotations. This design keeps the total training budget and the core multimodal composition constant, varying only the domain-specific fraction.

Critical Assessment

Claim 1: "A single RAE-based encoder simplifies the architecture by excelling at both visual understanding and generation, while preserving text performance."

The evidence in Figures 4 and 10 provides strong support for this claim within the scope tested. SigLIP 2 with RAE decoder indeed achieves the best generation metrics (DPGBench, GenEval), the best VQA accuracy, and text perplexity equivalent to the text-only baseline. The dual-encoder ablation in Figure 21 confirms that the single encoder matches or exceeds the dual-encoder configuration. The MoE specialization analysis (Figure 20) further supports this by showing that understanding and generation activate the same experts, suggesting a genuinely unified representation.

However, several qualifications are important. First, the claim's scope is limited to the specific encoders tested — SigLIP 2, DINOv2-L, WebSSL-L, SD-VAE, FLUX.1 VAE. It does not establish that any semantic encoder will outperform any VAE; it establishes that the best semantic encoder (SigLIP 2) outperforms the tested VAEs. The paper acknowledges that "current semantic vision encoders can still lag behind VAEs in fine-grained reconstruction" (Section 9), citing Zhang et al. (2025b). The generation metrics used (DPGBench, GenEval) primarily measure text-alignment and object-level composition rather than fine-grained texture fidelity, where VAEs may retain advantages not captured by these benchmarks.

Second, all experiments use models at the 2.3B scale. Whether a single semantic encoder would maintain its advantage over dual-encoder approaches at much larger scales (10B+, 100B+) is not tested. The dual-encoder approach allocates separate capacity to understanding and generation; at small scales, this splitting may hurt more than at large scales where capacity constraints are looser.

Third, the "preserving text performance" claim is true for in-distribution text (DCLM) but shows consistent mild degradation on OOD text (Notes corpus) across all visual representations in Figures 4 and 5. This degradation is small but systematic, suggesting some universal cost to multimodal pretraining on text generalization.

Claim 2: "Train with multimodal data (e.g. video, image-text). Visual data does not degrade language modeling, and diverse pretraining yields synergy for downstream tasks such as world modeling and VQA."

The evidence partially supports this claim but with important nuances. The first part — "visual data does not degrade language modeling" — is supported for pure video data (Figure 5: Text+Video matches or slightly exceeds text-only DCLM PPL) but is not uniformly true. Image-text pairs do degrade text perplexity, and the paper itself attributes this to distributional shift in captions (Section 4.2). The statement "visual data does not degrade language modeling" is therefore too broad — it should be "visual data without distribution-shifting text does not degrade language modeling." The paper's more precise conclusion is that "vision itself is not the primary cause of modality competition."

The synergy claims are well-supported by the VQA data efficiency experiment (Figure 9) — supplementing 20B VQA tokens with 80B general data beats 100B pure VQA, a genuine 5× data efficiency gain. The world modeling synergy (Figure 12) is also convincing: adding general video provides larger improvements than doubling domain-specific data. However, the world modeling evaluation is limited to the RECON dataset with CEM planning — a single navigation benchmark. Whether the claimed "emergence" of world modeling transfers to other physical prediction tasks (object dynamics, long-horizon planning, multi-environment generalization) is not tested.

The paper's scaling law results (Section 7) provide an important counterpoint to the synergy narrative: while co-training is beneficial at fixed compute, the divergent scaling behavior of vision and language creates a structural tension at scale that MoE is needed to resolve. The synergy exists, but it may not scale indefinitely without architectural innovation.

Claim 3: "Unified multimodal pretraining unlocks world modeling. Represent actions as text without architectural changes; capabilities emerge via general training with minimal domain-specific data."

This claim is supported by the data ratio ablation showing saturation at 1% NWM data (Figure 13) and the multimodal data comparison showing general video outperforms domain-specific scaling (Figure 12). The qualitative zero-shot language control (Figures 14, 30–35) is an impressive demonstration that is not quantitatively evaluated — the paper provides no metric for language-guided navigation accuracy.

A significant limitation is that the world modeling experiments use the NWM protocol exactly (4 context frames, CEM-based planning, LPIPS scoring, 8-step horizon). While the paper emphasizes that no action-specific adapters or architectural changes were introduced, the evaluation protocol itself is highly domain-specific and borrowed from prior work. Whether the model would perform physical prediction in more open-ended settings (arbitrary video continuation, physics simulation, counterfactual reasoning) is not tested.

The claim that "capabilities emerge via general training" is supported by the low-data regime results but is correlational — the paper does not demonstrate that the model trained on 200B tokens with 1% NWM data performs better at zero-shot language control than a model trained only on NWM data (which was not tested with language actions). The "emergence" narrative, while plausible, is not subjected to a rigorous causal test isolating the contribution of general pretraining versus domain-specific alignment.

Claim 4: "Use MoE in unified models: it outperforms hand-crafted separation strategies and naturally learns modality-specific specialization from the data."

The evidence for MoE outperforming hand-crafted separation is clear in Figure 21: MoE achieves substantially lower perplexity and higher DPG than both dense and MoT baselines at the same training budget (though note that total parameter count is higher for MoE — 13.5B total vs. 1.5B active; the comparison is at fixed active compute, not fixed total parameters). The emergent specialization analysis (Figures 18–20) is thorough and well-quantified.

However, several limitations are important. First, the MoE design sweep was conducted at only 57B tokens — a relatively small scale compared to the 1T token experiments in other sections. Whether the optimal granularity (G=16), prediction target (x-pred for SigLIP 2), and shared expert configuration found at 57B tokens hold at 1T tokens is not verified. Scaling law experiments (Section 7) use MoE at higher token counts but do not re-optimize granularity or sparsity ratio at each scale.

Second, the MoE scaling laws (Figure 25) use a fixed sparsity ratio of 16. Whether this is optimal at all scales is not tested — the sparsity sweep (Figure 16) shows monotonic improvement up to E=1008, suggesting higher sparsity might be even better, but this sweeps total expert count at fixed active compute and training tokens, not at IsoFLOP conditions.

Third, the efficiency claims for MoE ("matches unimodal performance") in Figure 26 are narrowly true — MoE Multimodal tracks unimodal baselines closely — but the unimodal baselines themselves are also MoE models. The claim is that sparse unified models can match sparse unimodal models, not that unified models can match dense unimodal models at the same compute. Whether this would hold against a dense text-only model of equivalent total parameters is not tested.

Methodological strengths: The paper's systematic approach — controlling one variable at a time, fixing compute budgets within each ablation family, and evaluating at the end of pretraining without instruction tuning — provides a clean signal for understanding pretraining dynamics. The breadth of evaluation (text perplexity, image generation, VQA, world modeling, scaling laws) is comprehensive and allows cross-validation of findings across different capabilities.

Methodological weaknesses:

Single model family, single scale: All non-scaling experiments use the 2.3B (1.5B active) model architecture. The paper does not demonstrate that its findings — optimal visual representation, data synergy, MoE granularity — hold at larger scales (10B+). Given that the paper's own scaling laws show divergent behavior between modalities, design choices made at 2.3B may not be optimal at 100B.
Training budget inconsistencies: The MoE design sweep uses 57B tokens, the data composition experiments use 1T tokens, and the scaling law experiments span a wide range. The paper does not always verify that findings from the smaller-scale sweeps (e.g., optimal G=16) hold at the larger training budgets used in other sections.
Frozen visual encoder: Throughout all experiments, the visual encoder is frozen. This is a deliberate design choice to isolate the Transformer's learning dynamics, but it means the paper's conclusions about "a single encoder sufficing" do not address whether end-to-end training of the encoder would further improve performance or change the relative ranking of representations. The paper is explicitly studying the use of pretrained encoders, not their joint optimization.
No interleaved image-text data: The paper excludes interleaved data (images and text appearing together in natural document structure) from its training mixtures, acknowledging this in Appendix A as a limitation. Many real-world multimodal applications involve interleaved data, and the paper's findings about synergy and competition may not fully generalize to this setting.
VQA evaluation protocol: All VQA results are reported after 1 epoch of finetuning on Cambrian-7M. While this is a standard protocol (following Tong et al., 2024a), it means the reported accuracies reflect both pretraining quality and finetuning efficiency, not pure pretraining capability. The 1-epoch finetuning may interact differently with different visual encoders, potentially confounding the pretraining comparison.
World modeling evaluation scope: The CEM-based planning evaluation is limited to a single dataset (RECON) with a specific horizon (8 steps). The paper does not evaluate how world modeling scales with model size, token budget, or the percentage of general video data — the data ratio ablation (Figure 13) is the only systematic world modeling experiment. The qualitative language-control results (Figures 14, 30–35) are not quantitatively benchmarked, leaving open the question of how reliable or accurate language-driven navigation actually is.
WISE benchmark interpretation: The 3–4× improvement of semantic encoders over VAE encoders on WISE (Figure 22) reflects knowledge-informed generation — the ability to generate images that accurately depict factual concepts. However, WISE is a relatively new benchmark, and the paper does not discuss potential confounds (e.g., whether VAE-based models simply generate lower-resolution or noisier images that the WISE evaluation pipeline penalizes independently of factual accuracy).

Missing experiments that would strengthen the paper:

Scaling the optimal configuration: The paper identifies an optimal configuration (SigLIP 2 + MoE G=16 + x-pred + per-modality shared expert) but does not train this configuration to 1T tokens or beyond. The stacking experiment in Figure 21 uses this configuration but the training token count is not explicitly stated for that experiment (it appears to use the same 57B tokens as the MoE sweeps, given the model architecture and training details). Scaling this configuration to match the 1T token budgets used in the data composition experiments would validate that the design choices compound at scale.
End-to-end fine-tuning of the visual encoder: A natural extension would be to unfreeze the visual encoder during pretraining (or during a second phase) to measure whether joint optimization of the encoder and the Transformer backbone further closes the gap to dual-encoder approaches or improves generation quality.
Larger-scale MoE granularity sweeps: The granularity sweep at 57B tokens shows that language benefits from higher granularity than vision (G=16 vs. G=4). Sweeping modality-specific granularities (e.g., G=16 for language experts, G=4 for vision experts) could yield further improvements and would test whether the optimal capacity allocation is genuinely different between modalities.
Zero-shot evaluation without VQA finetuning: Reporting VQA accuracy before finetuning (purely from pretraining) would distinguish whether multimodal pretraining genuinely learns visual understanding capabilities or merely provides a better initialization for downstream finetuning. The current protocol cannot disambiguate these.
Multiple world modeling environments: Evaluating navigation world modeling on additional datasets (beyond RECON) and on different tasks (object manipulation, physical reasoning) would strengthen the generality of the "emergence" claim. The current single-dataset evaluation leaves open the possibility that RECON-specific cues (not general physical understanding) drive the results.
Training compute vs. inference compute trade-off for world modeling: The paper shows that 1% NWM data suffices, but this is evaluated at a fixed 200B total token budget. How this percentage scales with total budget — does a 1T token model need 0.1% NWM data or still 1%? — is not explored.

Overall assessment: The experiments collectively provide strong evidence for the paper's four headline claims, but each claim comes with boundary conditions that the paper partially acknowledges. The most robust findings are: (1) SigLIP 2 with RAE outperforms the tested VAEs at 2.3B scale; (2) multimodal co-training yields positive transfer for VQA and world modeling within the evaluated budgets; and (3) MoE with high granularity and sparsity improves unified multimodal training, with emergent modality specialization. The scaling law asymmetry between vision and language is a genuine empirical discovery with implications for architecture design, though it is derived from IsoFLOP fits at moderate scales and would benefit from validation at larger compute budgets. The weakest evidential link is the "emergence" framing for world modeling — the results are consistent with emergence but do not isolate general pretraining as the causal factor, and the evaluation is limited to a single benchmark.

6. Limitations and Trade-offs

The Single Model Scale and Single Benchmark Constraint

The assumption or constraint. All non-scaling experiments in this paper — the visual representation study, the data composition analysis, the world modeling investigation, and the MoE architecture design space — are conducted with a model at the 2.3B total parameter scale (1.5B active). The scaling law experiments sweep from 100M to 10B parameters (dense) and 100M to 5B active (MoE), but the design choices that inform the main conclusions (SigLIP 2 as optimal encoder, G=16 as optimal MoE granularity, x-pred for RAE, per-modality shared experts) were all optimized at the 2.3B scale on 57B–1T tokens. Furthermore, all experiments use a single model architecture family — the Transfusion-based decoder-only Transformer with flow matching — and all evaluations are conducted on a single task family: MATH-equivalent visual tasks (VQA on Cambrian benchmarks) and navigation world modeling (RECON). The paper does not replicate its findings on a different model architecture, a different base Transformer design, or a substantially different domain (e.g., video understanding, robotic manipulation, or code generation from visual specifications).

The consequence. The central risk is that the paper's headline design recommendations are scale-dependent or architecture-dependent in ways the experiments cannot detect. The paper's own scaling laws (Section 7) reveal that vision and language exhibit fundamentally different compute-optimal scaling behavior, with the ratio of required vision data to language data growing as O(N^0.57). This means that design choices made at 2.3B parameters — where the data imbalance is modest — may not hold at 100B or 1T parameters, where vision's data hunger relative to language has grown by 14× or 51× respectively. The optimal MoE granularity (G=16), the optimal visual representation (SigLIP 2 vs. alternatives), and the optimal data mixture ratios were all determined at a scale where the asymmetry is mild. A practitioner training a 100B-parameter unified model cannot confidently extrapolate these choices.

The single-benchmark limitation is equally concerning. The finding that semantic encoders outperform VAEs for generation is evaluated on DPGBench and GenEval, which primarily measure text-image alignment and object-level composition — not fine-grained texture fidelity, photorealism, or high-resolution detail where VAEs have historically excelled. The paper acknowledges this indirectly in Section 9: "current semantic vision encoders can still lag behind VAEs in fine-grained reconstruction (Zhang et al., 2025b)." A practitioner building a text-to-image system where photorealism matters more than semantic accuracy might find the VAE advantage on those dimensions — but the paper's benchmarks do not measure them. Similarly, the world modeling results are evaluated on a single navigation benchmark (RECON) with a single planning algorithm (CEM, 8-step horizon). Whether the "emergence" of world modeling from general pretraining transfers to longer horizons, different environments, or different physical prediction tasks (object dynamics, manipulation, multi-agent interaction) is entirely untested.

What evidence exists in the paper. The paper partially acknowledges the scale limitation in Section 9, where it notes that "current semantic vision encoders can still lag behind VAEs in fine-grained reconstruction" and calls for "developing better generation-aware semantic representations." The data composition experiments (Section 4.1, Figure 5) at 1T tokens do validate some findings at larger scale — specifically, that vision data minimally impacts text performance. But the MoE architecture design space was only explored at 57B tokens (Section 6.1), and the paper does not verify whether the optimal granularity, sparsity ratio, or prediction target found there holds at 1T tokens. The authors state explicitly in Appendix A that "while we explore many facets of multimodal Mixture-of-Experts (MoE) design, our scope is not exhaustive." The single-benchmark limitation for world modeling is partially visible in Figure 12, where only RECON ATE and RPE are reported, with no evaluation on other navigation datasets or physical prediction tasks.

Mitigation status. The paper's scaling law experiments (Section 7) partially address the scale concern by providing IsoFLOP analyses up to 10^21 FLOPs for dense and 6×10^20 for MoE models, demonstrating that the key findings (asymmetry between modalities, MoE narrowing the gap) hold across at least an order of magnitude in compute. However, the scaling laws were fit using the fixed design choices optimized at small scale — the paper does not re-optimize MoE granularity or visual representation at each IsoFLOP budget. So the scaling laws characterize how a particular configuration scales, not whether that configuration remains optimal at each scale. The paper does not propose a mitigation for the benchmark specificity, though the authors acknowledge in Section 9 that they "hope our findings will encourage the community to close this gap by developing better generation-aware semantic representations." A practitioner would need to validate the design recommendations at their target scale and on their target tasks before adopting them.

The Difficulty Estimation Cost Is Not Accounted for Anywhere

The assumption or constraint. The entire paper assumes that visual data — raw video, image-text pairs, action-conditioned trajectories — is freely available at scale and that the primary bottleneck is model architecture and training recipe design. This assumption is implicit throughout: the scaling laws in Section 7 optimize parameter counts and token counts at fixed FLOP budgets, but never account for the cost of acquiring, storing, filtering, or preprocessing the multimodal training data. The paper uses MetaCLIP (2.2B image-text pairs), in-house Shutterstock, YouTube-Temporal 1B, and NWM navigation data, all of which are proprietary or require substantial engineering infrastructure to assemble. The paper provides no analysis of how data quality, data diversity, or data acquisition costs scale with model size.

The consequence. A practitioner attempting to replicate or scale these results faces a hard, unacknowledged cost: the data engineering pipeline. Unlike text data, which can be crawled from the web with relatively standard pipelines, video data at scale requires massive storage (raw video is orders of magnitude larger than text), video decoding infrastructure, frame extraction at consistent FPS, and curation to ensure diversity and quality. The paper's finding that vision is "significantly more data-hungry than language" (Section 7, D_opt ∝ C^0.63 for vision vs. C^0.53 for language) means this data acquisition cost grows super-linearly with compute — a 10× increase in compute requires a ~4× increase in language tokens but a ~4.3× increase in vision tokens, and vision tokens are vastly more expensive per token to acquire, store, and process. The paper's scaling recommendations are therefore compute-optimal but may be wildly cost-inefficient when data acquisition is included in the total budget.

Furthermore, the paper's central finding about world modeling "emerging" from general pretraining (Section 5, Figures 12–13) depends critically on the quality and diversity of the general video data. If a practitioner's available video data is less diverse or lower quality than the YouTube-Temporal 1B and Kinetics data used in the paper, the "emergence" may fail to materialize — but the paper provides no characterization of what properties of the video data drive the world modeling transfer. The recaption experiments (Section 4.2, Figure 7) require access to a 32B-parameter VLM (Qwen2.5-VL-32B-Instruct) for caption generation — a substantial additional inference cost not accounted for in any training budget.

What evidence exists in the paper. The paper does not measure data acquisition, storage, or preprocessing costs anywhere. The scaling law experiments (Section 7, Figures 23–26) use training FLOPs as the sole cost metric. The data composition experiments (Section 4) compare token-equivalent mixtures without discussing the relative difficulty or cost of obtaining those tokens. The recaption pipeline (Appendix G.2) describes using vLLM with "8-way tensor parallelism per node" — an engineering detail that hints at the computational cost but provides no FLOP or dollar accounting. The paper explicitly excludes interleaved data from its training mixtures (Appendix A), which would further increase data engineering complexity.

Mitigation status. Not addressed. The paper frames its contribution as an empirical study of pretraining dynamics, not a deployment guide, and this is a reasonable scoping decision. However, the paper's suggestions — "train with multimodal data," "embrace diverse data," "unlock world modeling" — are presented as actionable recommendations without caveats about data acquisition costs. A practitioner reading the paper to inform infrastructure decisions would find no guidance on how to estimate or manage these costs. The "bitter lesson" framing in Section 6.3 ("learning from data generally outperforms hand-crafted designs") implicitly endorses scaling data as the solution, but the paper never grapples with the fact that visual data scaling is fundamentally more expensive and logistically complex than text data scaling.

Verifier Over-Optimization Is Not Characterized, Constraining Generation Scaling

The assumption or constraint. The paper trains image generation using flow matching with classifier-free guidance (CFG) at a fixed guidance scale of 3.0, evaluated on DPGBench and GenEval. However, unlike language model evaluation where perplexity on held-out text provides a well-calibrated measure of model quality, the relationship between diffusion loss (the training objective) and generation quality is mediated by the CFG scale, the number of sampling steps, and the decoder quality. The paper does not explore how generation metrics behave under more aggressive optimization — higher CFG scales, more sampling steps, or best-of-N selection — and does not characterize whether the model's generation quality can be improved post-hoc through test-time compute, or whether it suffers from a verifier over-optimization problem analogous to what has been documented in LLM reasoning tasks (where beam search against a reward model eventually exploits the verifier and degrades output quality).

The consequence. The paper's generation metrics (DPGBench, GenEval, COCO FID) are reported at a single inference configuration: 25 Euler steps, CFG=3.0, single sample. This means the reported numbers represent a lower bound on achievable generation quality — better results might be obtainable with more inference compute, different guidance scales, or best-of-N selection — but also that the scaling behavior of generation quality with test-time compute is unknown. If increasing inference compute causes generation quality to saturate or degrade (analogous to the verifier over-optimization documented in the example paper from the prompt), then the scaling law projections in Section 7 (which predict validation loss, not downstream generation quality) may overestimate real-world generation capability at scale.

More subtly, the paper's conclusion that "Semantic encoders consistently outperform VAE-based encoders on both visual understanding and generation" (Section 3) depends on the specific generation benchmarks used. DPGBench and GenEval evaluate text-image alignment and object composition. If VAE-based generation benefits more from test-time compute optimization (e.g., higher CFG scales), the relative ranking could shift. The paper's fixed evaluation protocol cannot detect such shifts.

What evidence exists in the paper. The paper provides no test-time compute scaling analysis for generation. The CFG scale of 3.0 is stated to be un-tuned: "We use a fixed CFG of 3.0 to run all image generation evaluations and do not tune this value" (Section 2.1). The number of sampling steps (25) is also fixed. There is no ablation varying inference-time hyperparameters. The paper does not report whether generation quality correlates with diffusion loss at the model scale — Figures 23 and 25 show validation loss scaling, but Figures 24 and 26 switch to downstream metrics (DPG, FID) without explicitly characterizing the relationship between loss and metric across the compute range. The training loss curves in Figure 17 show diffusion loss continuing to decrease for RAE models, but whether this translates to continued GenEval/FID improvements at larger compute budgets is not verified.

Mitigation status. Not addressed. The paper focuses entirely on pretraining compute scaling and does not study inference-time compute allocation. This is a deliberate scoping decision — the paper states it aims to understand pretraining dynamics, not inference optimization — but it leaves open a significant practical question: if a practitioner trains a model following the paper's recommendations and finds generation quality unsatisfactory, can they improve it by spending more test-time compute, or is the only path to train a larger model? The paper's scaling law projections implicitly assume that lower validation loss translates monotonically to better generation, but this assumption is not validated, and the diffusion literature has documented cases where this relationship breaks down (e.g., due to CFG scale interaction with model scale).

The Revision Model's 38% Correct-to-Incorrect Reversion Rate Has No Known Mitigation

Note: This limitation is from the example paper in the prompt, not the provided multimodal pretraining paper. The multimodal pretraining paper does not have a revision model. I will identify a structurally analogous limitation from the multimodal paper instead.

World Modeling Evaluation Is Qualitative and Limited to a Single Benchmark with a Fixed Planning Algorithm

The assumption or constraint. The world modeling experiments (Section 5) evaluate navigation prediction on a single dataset (RECON) using a single planning algorithm (Cross-Entropy Method with N=120 samples, 8-step horizon, LPIPS scoring, top-K=5 trajectory averaging). The qualitative demonstrations of free-form language control (Figures 14, 30–35) are presented without any quantitative metrics — no success rate, no distance error, no comparison to baseline methods. The paper claims that "physical prediction capabilities emerge primarily from general multimodal pretraining" (Section 5 opening paragraph), but the evidence for this claim is limited to: (1) the data ratio ablation on RECON with CEM (Figure 13), (2) the multimodal data comparison on RECON with CEM (Figure 12), and (3) qualitative rollouts with no metrics.

The consequence. A practitioner evaluating whether to adopt this approach for a world modeling application faces considerable uncertainty. The paper demonstrates that general pretraining helps on RECON with CEM planning, but does not establish: (a) whether the benefit transfers to other navigation environments (different scene types, different robot dynamics, different action spaces), (b) whether CEM planning is necessary or whether simpler planning methods (e.g., greedy action selection) would show the same benefit, (c) whether the quality of language-driven navigation (Figures 14, 30–35) is practically useful — the rollouts look visually coherent but may exhibit accumulating drift or physically implausible predictions that the qualitative examples do not reveal, and (d) how world modeling accuracy scales with model size, training tokens, or video data diversity — the data ratio ablation in Figure 13 is the only systematic experiment, and it uses a single model configuration at 200B tokens.

The "emergence" framing in particular requires stronger evidence than is provided. The fact that performance saturates at 1% NWM data (Figure 13) is consistent with the hypothesis that general pretraining provides the core capability, but alternative explanations are possible: perhaps RECON navigation is simply an easy task that requires minimal training data regardless of pretraining quality, or perhaps the CEM planning algorithm is robust to model errors and masks differences in underlying model quality. Without a control experiment showing that a model trained only on general data (0% NWM) performs near chance, or that a model without general pretraining requires substantially more NWM data to saturate, the causal role of general pretraining is not isolated.

What evidence exists in the paper. The paper reports RECON ATE and RPE in Figures 12 and 13, with lower numbers indicating better performance. The data ratio ablation (Figure 13) shows ATE dropping from ~1.8 at 0.1% NWM data to ~1.4 at 1%, then plateauing — but the 0.1% data point is still far from a zero-shot baseline. The text of Section 5 states that "the model reaches competitive performance with as little as 1% in-domain data," but "competitive" is relative — the absolute ATE values (~1.3–1.4) are not compared to any external baseline (e.g., a specialized NWM model, or a simple constant-velocity baseline). The qualitative examples (Section 5.3, Appendix F.2) are presented as "demonstrations" without quantitative evaluation. The paper does not report metrics for language-driven navigation, nor does it ablate whether the context frames alone (without action text) would produce similar rollouts.

Mitigation status. The paper partially acknowledges this limitation in Appendix A by listing "interleaved data" as a gap, but does not explicitly discuss the narrowness of the world modeling evaluation. The "Limitations and Future Work" section (Appendix A) states: "In the long run, we believe the capability to spontaneously generate visual latents will drive the convergence of multimodal models and world modeling, fostering 'System 2' behaviors that are grounded in the real world." This is aspirational — it acknowledges that the current work is an early step toward this convergence, not a demonstration that it has been achieved. However, the main text of Section 5 makes stronger claims ("World Modeling Emerges from Multimodal Pretraining," "world modeling transfers from general training") than the evidence fully supports. A practitioner should treat the world modeling results as proof-of-concept rather than as a validated capability ready for deployment.

Hard Problems — Tasks Requiring Fine-Grained Reconstruction or Out-of-Distribution Visual Understanding — Remain Unsolved

The assumption or constraint. The paper's experiments demonstrate that semantic encoders (specifically SigLIP 2) outperform VAE encoders on the evaluated benchmarks: DPGBench, GenEval (text-to-image alignment), and Cambrian VQA (visual question answering). However, these benchmarks do not comprehensively cover the space of visual capabilities. In particular, tasks requiring fine-grained spatial reconstruction — high-resolution image generation with detailed textures, faithful reproduction of small objects or text within images, pixel-level editing — are not evaluated. The paper acknowledges this in Section 9: "current semantic vision encoders can still lag behind VAEs in fine-grained reconstruction." Similarly, the VQA benchmarks in the Cambrian suite primarily test semantic understanding (object recognition, attribute binding, spatial reasoning at a coarse level) rather than low-level visual acuity (reading small text in images, counting objects in crowded scenes, detecting subtle visual differences).

The consequence. A practitioner selecting a visual representation based on this paper might choose SigLIP 2 over a VAE for a unified understanding-and-generation system, only to discover that the model cannot generate images with legible text, cannot faithfully reproduce fine textures, or fails at VQA tasks requiring precise visual discrimination. The paper's headline claim — "a single encoder suffices for both visual understanding and generation" — is true for the capabilities measured but may not hold for applications requiring high-fidelity generation or fine-grained visual understanding. The RAE decoder (which maps from SigLIP 2 latents back to pixels) is a separately trained component whose quality determines reconstruction fidelity, and the paper provides no characterization of its limitations beyond the layer-wise analysis in Appendix D.1 (Figure 28a), which shows that PSNR drops from 29.6 dB at layer 0 to 20.9 dB at the final SigLIP 2 layer — a substantial degradation in pixel-level information.

The scaling law results in Section 7 further suggest that this limitation may not resolve with scale alone. The paper finds that VAE-based models exhibit diffusion loss saturation as sparsity increases (Figure 17), but even RAE-based models — while they continue to improve — may hit reconstruction quality ceilings imposed by the frozen encoder's information bottleneck. The paper does not investigate whether fine-tuning the visual encoder during pretraining would recover the lost fine-grained information, or whether hybrid approaches (e.g., multi-layer features from the encoder) could bridge the gap.

What evidence exists in the paper. The layer-wise analysis (Appendix D.1, Figures 27 and 28a) provides direct evidence: "ImageNet accuracy rises steadily from 12.8% (layer 0) to 86.3% (layer 26), while PSNR drops from 29.6 dB to 20.9 dB. This reveals a fundamental trade-off in which deeper encoder layers progressively discard fine-grained spatial information in favor of increasingly abstract semantic features." The qualitative reconstruction results in Figure 27 visually demonstrate this trade-off: earlier layers produce sharper, more faithful reconstructions, while the final layer (used for all main experiments) shows blurrier, less detailed outputs. The paper notes that "training on features from multiple layers could in principle recover this fidelity, but at the cost of simplicity," and calls for "future vision encoders that achieve a better balance between semantic abstraction and spatial fidelity within a single representation."

The generation benchmarks themselves (DPGBench, GenEval) are described in Section 2.4 without detailed analysis of what visual capabilities they measure. A practitioner cannot determine from the paper whether the benchmarks stress-test fine-grained reconstruction or primarily evaluate semantic alignment.

Mitigation status. Partially addressed. The paper explicitly acknowledges the reconstruction fidelity limitation in Section 9 and Appendix D.1, and frames it as a call to action: "We hope our findings will encourage the community to close this gap by developing better generation-aware semantic representations." The paper's Figure 28a provides quantitative evidence of the trade-off, allowing practitioners to assess the magnitude of the information loss. However, the paper does not propose any mitigation within its own framework — no multi-layer feature approach, no encoder fine-tuning experiment, no hybrid VAE-semantic representation — leaving the gap as an open problem. The suggestion to use single semantic encoders (Suggestion 1) is presented without qualification in the main text, which could mislead practitioners who do not read the appendix analysis of reconstruction fidelity.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the conversation around multimodal pretraining from a defensive posture to an opportunity-seeking one. The dominant paradigm in the field has been to initialize from pretrained language models and carefully add vision capabilities through adapters or fine-tuning, implicitly treating vision as a threat to be managed — something that might "break" the hard-won language capabilities. This paper provides the first systematic, controlled evidence that vision and language are not inherently competitors for model capacity. The observed "modality competition" is an artifact of specific, solvable design choices: distributional shift in captions (not the visual signal itself) and architectural capacity allocation (dense models forcing competition that sparse models avoid).

This is not an incremental refinement of existing multimodal training recipes — it is a reframing of what the core problem actually is. The paper shows that the right architectural and data choices (modality-specific FFNs, MoE routing, pure video data) can make vision neutral or slightly beneficial for language modeling, rather than harmful. This directly challenges the premise underlying most current multimodal systems (LLaVA, Emu, Janus, BAGEL) that language must be protected from vision through pretrained initialization. If vision doesn't degrade language when properly architected, the entire pipeline of "pretrain language first, then bolt on vision" becomes an unnecessary complexity rather than a necessary safeguard. The paper's results suggest that native multimodal pretraining from scratch is not merely viable but potentially advantageous — enabling positive cross-modal synergy (Figure 9: general data beats 5× more in-domain data for VQA) and unlocking emergent capabilities like world modeling (Section 5) that language-pretrained-then-adapted models may not develop.

The second shift is architectural: from hand-designed modality separation to learned capacity allocation. The paper demonstrates that MoE with fine-grained routing (G=16) and high sparsity not only outperforms fixed separation strategies like modality-specific FFNs and Mixture of Transformers (Figure 21), but also naturally discovers modality specialization without human priors (Figures 18–20). The expert specialization analysis — showing text-dominated early layers, vision-dominated later layers, and unified understanding-generation experts — is emergent behavior learned from data, not engineered. This provides empirical validation for the "bitter lesson" (Sutton, 2019) in multimodal architecture design: learned routing outperforms hand-crafted separation. As the field moves toward larger unified models, this suggests that investing in better routing and sparsity mechanisms is more promising than designing increasingly sophisticated fixed separation schemes.

The third shift is methodological: establishing that from-scratch controlled experiments are necessary and feasible for understanding multimodal pretraining. Most prior work confounds pretraining dynamics with the capabilities inherited from pretrained LLMs. This paper demonstrates that training unified models from scratch at meaningful scale (1T tokens, 2.3B parameters) is practical and produces clear, interpretable signals about what drives multimodal learning. This opens the door for a new generation of controlled empirical studies that isolate specific variables — data composition, architecture, objective function — without the confounding influence of pretrained initialization. The paper's IsoFLOP scaling law methodology (Section 7), adapted from Chinchilla (Hoffmann et al., 2022) to the multimodal setting, provides a template for how such studies can quantify scaling behavior and reveal structural tensions (like the vision-language asymmetry) that would be invisible in single-scale experiments.

The paper also reconciles a tension in the literature around visual representations. Prior work was split: VAEs were assumed necessary for generation, semantic encoders for understanding, leading to dual-encoder architectures (Janus, BAGEL). Recent RAE work (Zheng et al., 2026; Tong et al., 2026) challenged the technical premise, showing diffusion can work in high-dimensional semantic spaces. This paper provides the controlled comparison that resolves the ambiguity: a single SigLIP 2 encoder with RAE decoder outperforms both VAE encoders and dual-encoder configurations on generation quality, visual understanding, and text preservation simultaneously (Figures 4, 10, 21). The dual-encoder paradigm — which adds significant engineering complexity — is shown to be unnecessary at the scales tested. This finding, combined with the MoE specialization analysis showing understanding and generation share the same experts (Figure 20), provides converging evidence that the distinction between "understanding features" and "generation features" is an artifact of historical encoder training paradigms, not a fundamental property of vision.

Which research directions become more attractive. The paper strongly argues for: (1) developing better generation-aware semantic encoders (since the main limitation of current semantic encoders is fine-grained reconstruction fidelity, not semantic capability); (2) exploring MoE architectures with learned routing as the default for multimodal models; (3) leveraging large-scale unlabeled video data as a strategic resource for scaling (since it's complementary to language and drives world modeling); (4) studying world modeling as an emergent property of broad multimodal pretraining rather than a separate research problem requiring specialized architectures; and (5) investigating interleaved multimodal data (the one major data type not studied here).

Which become less attractive. The paper's results cast doubt on: (1) the necessity of dual-encoder architectures for unified models (the added complexity is not justified by the evidence); (2) the assumption that pretrained LLM initialization is required to preserve language capabilities in multimodal training; (3) hard-coded modality separation strategies (MoT, fixed FFN splits) when learned routing (MoE) is available and performs better; and (4) the assumption that scaling domain-specific data is the most efficient path to downstream task performance (general pretraining provides stronger foundations for both VQA and world modeling).

Follow-Up Research This Work Enables

Jointly optimizing the visual encoder and the Transformer backbone during pretraining. The paper freezes the visual encoder throughout all experiments to isolate the Transformer's learning dynamics, but acknowledges that "current semantic vision encoders can still lag behind VAEs in fine-grained reconstruction" (Section 9). A natural extension is to unfreeze the encoder during a second phase of pretraining (or from the start) and measure whether end-to-end training: (a) improves generation fidelity by allowing the encoder to retain more fine-grained spatial information in its later layers, addressing the PSNR degradation documented in Figure 28a; (b) changes the relative ranking of encoder architectures (would a jointly-trained VAE catch up to SigLIP 2?); and (c) enables the model to develop entirely new visual features specialized for the joint understanding-generation task. A strong experiment would train three configurations — frozen SigLIP 2, fine-tuned SigLIP 2, and frozen FLUX.1 VAE — at matched compute budgets (accounting for encoder gradient computation) and evaluate on both standard generation benchmarks and a fine-grained reconstruction benchmark (e.g., image reconstruction PSNR/SSIM, text-in-image rendering accuracy). The hypothesis is that fine-tuning narrows but does not eliminate the semantic-VAE gap in reconstruction quality, since the architecture's inductive biases (patch size, attention patterns) constrain what can be recovered.

Characterizing test-time compute scaling for multimodal generation and its interaction with verifier over-optimization. The paper reports all generation results at a fixed inference configuration (25 Euler steps, CFG=3.0, single sample) and does not study how generation quality scales with inference-time compute. This is a critical gap because the paper's scaling law projections (Figures 24, 26) assume that lower diffusion loss translates to better downstream metrics, but the relationship may break down under aggressive guidance or best-of-N selection. A systematic study should measure: (a) how DPGBench, GenEval, and FID improve with increased sampling steps (10 to 100), CFG scale (1.0 to 7.0), and best-of-N selection (N=1 to 64) for models at different training compute budgets; (b) whether there is a "verifier over-optimization" regime where higher CFG scales degrade image quality or diversity (analogous to beam search degrading easy-problem accuracy in LLM reasoning), and whether this threshold shifts with model scale; (c) whether the optimal inference configuration found at one scale transfers to larger models or requires re-tuning. This would connect the paper's pretraining scaling laws to practical deployment, answering whether a smaller model with more inference compute can match a larger model with default inference settings — the multimodal analog of the "test-time compute vs. pretraining" tradeoff studied in the LLM scaling literature.

Stress-testing world modeling emergence across environments, horizons, and planning algorithms. The paper's world modeling results (Section 5) are promising but limited to a single navigation benchmark (RECON) with a single planning algorithm (CEM, 8-step horizon). To determine whether the "emergence from general pretraining" claim generalizes, a follow-up study should: (a) evaluate the same model on multiple navigation environments (e.g., SCAND for social navigation, Habitat for indoor navigation, a driving dataset for outdoor navigation) to test whether the learned physical prediction capability transfers across visual domains; (b) vary the planning horizon (4, 8, 16, 32 steps) to measure how prediction error accumulates and whether the data ratio saturation point (1% NWM data) shifts with horizon length; (c) compare CEM against simpler planning methods (greedy action selection, random shooting with a fixed action vocabulary) to determine whether the planning algorithm or the underlying world model is driving performance; (d) include a critical "general-only" baseline — a model trained on 200B tokens of general data with 0% NWM data — to isolate whether any navigation capability exists without domain-specific alignment, establishing a true zero-shot lower bound. The key question is whether the 1% saturation point is a property of the RECON task (which may be easy) or a general principle of knowledge transfer from broad visual experience.

Scaling the optimal configuration to 10B+ parameters with re-optimized design choices. The paper's design recommendations (SigLIP 2 + RAE, MoE G=16, x-pred, per-modality shared experts) were optimized at 2.3B parameters and 57B–1T tokens. However, the paper's own scaling laws show that the vision-language asymmetry grows with scale (O(N^0.57) ratio increase), and the optimal MoE granularity or sparsity ratio may shift at larger model sizes. A critical stress-test is to: (a) train the optimal configuration at 10B, 30B, and 100B active parameters on matched compute budgets, re-optimizing granularity, sparsity ratio, and prediction target at each scale; (b) measure whether the relative advantage of MoE over dense models grows or shrinks with scale (does the 0.05 exponent gap persist or narrow further?); (c) test whether the optimal data mixture ratio changes — if vision's data hunger grows relative to language, should the training mixture shift toward more visual tokens at larger scales? This experiment would validate whether the paper's design principles are scale-invariant or scale-dependent, providing essential guidance for practitioners planning large-scale unified model training.

Developing and benchmarking generation-aware visual encoders. The paper identifies a clear bottleneck: semantic encoders like SigLIP 2 discard fine-grained spatial information in later layers (PSNR drops from 29.6 dB to 20.9 dB, Figure 28a), which limits reconstruction fidelity in generation. The paper explicitly calls for "future vision encoders that achieve a better balance between semantic abstraction and spatial fidelity within a single representation" (Appendix D.1). A concrete research direction is to train a new encoder with a multi-task objective — combining contrastive language-image alignment (like SigLIP) with a reconstruction loss (like MAE or a VAE-style decoder) — and evaluate whether it: (a) matches SigLIP 2's VQA accuracy while improving PSNR in later layers; (b) improves generation metrics (FID, fine-grained text-in-image rendering) when used as the frozen encoder in the paper's Transfusion framework; and (c) enables better world modeling rollouts by preserving spatial details needed for accurate physical prediction. The paper's experimental framework (frozen encoder, Transfusion backbone, comprehensive evaluation suite) provides a ready-made testbed for benchmarking new encoder designs without confounding variables.

Investigating interleaved multimodal data as the missing training modality. The paper explicitly excludes interleaved image-text data (e.g., web pages, documents, textbooks where images and text appear together in natural sequence) and flags this as "a critical area for future exploration" (Appendix A). Interleaved data is qualitatively different from the paired image-text and separate video+text streams studied here — it requires the model to learn cross-modal references, grounded reasoning chains, and long-range dependencies where an image is referred to by text many tokens later. A strong follow-up would: (a) add interleaved data (e.g., from OBELICS, MMC4, or Wikipedia pages) to the training mixture at varying proportions, measuring whether it improves VQA over paired data alone (does the natural co-occurrence structure teach better grounding?); (b) evaluate on interleaved-specific benchmarks (e.g., multimodal reasoning tasks requiring referring to specific images in a sequence); and (c) test whether interleaved training provides additional world modeling benefits by teaching the model about temporal and causal relationships between text descriptions and visual events. The paper's hybrid attention masking (Section B) already supports interleaved sequences, so the infrastructure exists — the open question is whether interleaved data adds capability beyond what paired data and video provide.

Practical Applications and Downstream Use Cases

Cost-efficient training of multimodal models for VQA applications. The paper's VQA data efficiency result (Figure 9) has direct practical implications: a model trained on 20B VQA tokens supplemented with 80B general multimodal data (text, video, or image-text pairs) outperforms a model trained on 100B pure VQA tokens, despite using 5× less expensive, task-specific annotated data. For a team building a visual question answering system, this means the majority of the training budget should be spent on assembling diverse, general-purpose multimodal data (which can be crawled or synthetically generated at scale) rather than on expensive human-annotated VQA datasets. The 80B supplementary tokens can include unlabeled video (which costs only storage and decoding), web-crawled image-text pairs (available in billions via MetaCLIP-style pipelines), or even pure text — all of which are cheaper per token than task-specific VQA annotations. The paper provides a concrete recipe: pretrain on a mixture of 20% in-domain VQA data and 80% general multimodal data, then fine-tune on the target VQA dataset for 1 epoch. The 37.9% VQA accuracy achieved with Text supplementation (vs. 35.7% for 5× more VQA data) translates to reduced annotation costs and faster iteration cycles.

On-device or edge deployment of unified understanding-and-generation models with a single encoder. The paper demonstrates that a single SigLIP 2 encoder with RAE decoder suffices for both visual understanding and generation (Figures 4, 10, 21), eliminating the need for dual-encoder architectures (which run two separate vision encoders at inference time). For on-device deployment where memory and compute are tightly constrained, this is a significant practical advantage: running one encoder instead of two halves the vision encoding latency and memory footprint. At the 2.3B parameter scale studied, the single-encoder model achieves 40.3% VQA accuracy (Figure 10) and 0.65 DPG score (Figure 21) — performance that dual-encoder models at this scale would struggle to match while also being architecturally simpler. The frozen encoder also means the vision backbone (SigLIP 2) can be quantized or optimized independently of the Transformer, enabling separate deployment optimizations. For applications like mobile assistants that need to both answer questions about images and generate images, the single-encoder design reduces the engineering burden and the on-device resource requirements.

Leveraging large-scale unlabeled video as a strategic resource for model scaling. The paper's finding that pure video data is complementary to language modeling — Text+Video matches or slightly improves DCLM perplexity relative to text-only training (Figure 5) — has direct implications for organizations facing the text data bottleneck. As high-quality text data becomes increasingly scarce and expensive (the paper cites Sutskever, 2025 on this point), unlabeled video represents a vast, largely untapped reservoir of training signal. The paper shows that 520B tokens of video data can be incorporated without degrading language performance and with positive downstream benefits: improved world modeling (Figure 12), better VQA when combined with small amounts of in-domain data (Figure 9), and emergent zero-shot language-guided navigation (Figures 14, 30–35). For a team with access to large video archives (e.g., YouTube-scale data), the paper provides an existence proof and a training recipe: video frames at 1 FPS, block-wise causal masking, frame-wise flow matching, and modality-specific FFNs or MoE routing. The scaling law results (Section 7) further suggest that video data's value grows with scale — vision is data-hungry (D_opt ∝ C^0.63), so abundant video tokens become increasingly important as compute budgets increase.

Navigation and embodied AI systems that require minimal in-domain trajectory data. The world modeling results (Section 5) demonstrate that a model pretrained on general multimodal data (text, video, image-text pairs) with only 1% navigation-specific data achieves near-optimal trajectory prediction on RECON (ATE ~1.4, RPE ~0.42; Figure 13). For robotics and embodied AI teams, collecting high-quality navigation trajectories (which requires physical robot deployment or high-fidelity simulation) is often the most expensive and time-consuming part of the ML pipeline. The paper's finding suggests a radically more efficient approach: invest in broad multimodal pretraining on easily-available data (web video, text, image captions), then fine-tune on a tiny amount of domain-specific trajectories. The 1% data requirement means that instead of collecting 500 hours of navigation data, a team might need only 5 hours — a 100× reduction in the most expensive data type. The zero-shot language control capability (Figure 14) further suggests that once pretrained, the model can be instructed with natural language commands ("go to the tree," "get out of the shadow") without any language-annotated navigation data, enabling flexible human-robot interaction without per-command training examples.

When to Prefer This Method

The paper articulates a clear tradeoff between native multimodal pretraining from scratch (its studied approach) and the dominant alternative of initializing from a pretrained language model and adapting to vision. The decision hinges on several factors grounded in the paper's empirical findings:

Prefer native multimodal pretraining (from scratch) when:
- You have access to large quantities of diverse multimodal data — particularly unlabeled video and image-text pairs — and your text data is limited or you want to scale beyond the text data ceiling. The paper shows pure video is complementary to language (Figure 5) and general multimodal data provides 5× better VQA data efficiency than scaling in-domain data (Figure 9).
- Your target application requires emergent cross-modal capabilities that may not develop when vision is added post-hoc — specifically, world modeling (Section 5), zero-shot language-guided visual prediction, or unified visual understanding and generation from a single representation.
- You are willing to invest in architectural design to avoid modality competition — specifically, MoE with fine-grained routing (G=16), high sparsity, and per-modality shared experts. The paper shows these design choices are critical to recovering the full benefits of joint training (Figure 21: MoE achieves PPL 12.49 vs. 15.93 for the baseline without these choices).
- Your deployment benefits from architectural simplicity — a single visual encoder rather than dual encoders, reduced inference overhead, and no dependency on pretrained LLM weights that may have licensing restrictions or inference costs.
Prefer LLM-initialized multimodal adaptation when:
- Your primary constraint is preserving specific language capabilities — instruction following, safety alignment, reasoning behaviors — that you have already invested in through language-only post-training (RLHF, instruction tuning). The paper focuses on pretraining and does not study how post-training transfers, so the risk of losing these capabilities in from-scratch training is not characterized.
- You lack the compute budget to train a multimodal model from scratch to convergence. The paper's experiments use 1T tokens at 2.3B parameters, requiring substantial infrastructure. LLM-initialized approaches can leverage existing pretrained weights and require less multimodal training.
- Your application requires fine-grained visual reconstruction fidelity (high-resolution texture, text-in-image rendering, pixel-level editing) where current semantic encoders with RAE decoders lag behind VAEs. The paper acknowledges this limitation (Section 9, Appendix D.1) and does not claim to have solved it.
- You are operating at a scale where the vision-language scaling asymmetry (Section 7) becomes severe — the paper's scaling laws predict that the optimal ratio of vision to language data grows as O(N^0.57), making it increasingly difficult to satisfy vision's data hunger in a from-scratch joint training setup without MoE architectures to narrow the gap.

Dependency tree

Papers this one builds on, walking backwards through the same domain and shared topics.

Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuning2023· pretraining← LLM-judged similar
- An Empirical Study of Scaling Instruct-Tuned Large Multimodal Models2023· multimodal← LLM-judged similar
- Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing2023· pretraining← in-domain predecessor
- XLNet: Generalized Autoregressive Pretraining for Language Understanding2019· pretraining← similar topic
ArtLLM: Generating Articulated Assets via 3D LLM2026· multimodal← in-domain predecessor
- Cube: A Roblox View of 3D Intelligence2025· architecture← LLM-judged similar
- A Mixed Diet Makes DINO An Omnivorous Vision Encoder2026· multimodal← in-domain predecessor

What it influenced

Later papers in the same chain. Direction reversed.

From Narrow to Panoramic Vision: Attention-Guided Cold-Start Reshapes Multimodal Reasoning2026· multimodal→ in-domain predecessor
- MASQuant: Modality-Aware Smoothing Quantization for Multimodal Large Language Models2026· multimodal→ in-domain predecessor
- GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learning2025· multimodal→ similar topic
- Seeing, Listening, Remembering, and Reasoning: A Multimodal Agent with Long-Term Memory2025· multimodal→ similar topic
- InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency2025· multimodal→ similar topic
Apollo: An Exploration of Video Understanding in Large Multimodal Models2024· multimodal→ similar topic
- EgoEdit: Dataset, Real-Time Streaming Model, and Benchmark for Egocentric Video Editing2025· diffusion→ LLM-judged similar
- 4D-RGPT: Toward Region-level 4D Understanding via Perceptual Distillation2025· multimodal→ LLM-judged similar
- ARC-Hunyuan-Video-7B: Structured Video Comprehension of Real-World Shorts2025· training-methods→ LLM-judged similar
- The Script is All You Need: An Agentic Framework for Long-Horizon Dialogue-to-Cinematic Video Generation2026· agents→ LLM-judged similar
- Rethinking Video Generation Model for the Embodied World2026· multimodal→ LLM-judged similar
- OmniTransfer: All-in-one Framework for Spatio-temporal Video Transfer2026· vision→ LLM-judged similar
- Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Survey2024· multimodal→ in-domain predecessor
- Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation2024· multimodal→ similar topic
- InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactions2024· multimodal→ similar topic
- Omni-RGPT: Unifying Image and Video Region-level Understanding via Token Marks2025· multimodal→ similar topic
NextFlow: Unified Sequential Modeling Activates Multimodal Understanding and Generation2026· multimodal→ similar topic