Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Survey

ArXiv: 2412.18619

🎯 Pitch

Next-token prediction can unify vision, audio, video, and text under one learning objective, but the field has never been mapped holistically—until this survey. It catalogs 400+ models, revealing that discrete tokens drive generation while continuous tokens drive understanding, and solving this duality is the key to true multimodal intelligence.

---

1. Executive Summary

This survey systematically taxonomizes how Next Token Prediction (NTP) has evolved from a text-only language modeling objective into a unifying training paradigm for multimodal intelligence, covering vision, audio, video, and beyond. The paper introduces a comprehensive framework organized around five key aspects—multimodal tokenization, MMNTP model architectures, unified task representation, datasets and evaluation, and open challenges—that together categorize how understanding and generation tasks across modalities can be encapsulated within a single NTP-based pipeline. By tracing the historical development from early autoregressive vision models (ImageGPT, DALL-E) through to contemporary unified multimodal systems (Chameleon, Emu3, GPT-4o), the survey documents how discrete tokenization via vector quantization enables multimodal generation within the NTP framework, while continuous tokenization with alignment layers enables multimodal understanding. The paper identifies modality interference, scaling law uncertainty, and the unresolved tension between quantization-based autoregression and diffusion-based approaches as central open challenges, establishing that while NTP provides a conceptually unified interface for multimodal learning, the optimal training objectives and tokenization strategies remain modality-dependent.

2. Context and Motivation

The Core Problem: The Field Lacks a Unifying Framework for Multimodal Learning as Next Token Prediction

The central gap this paper addresses is organizational and conceptual rather than a specific technical obstacle. Over roughly five years (2019–2024), researchers across vision, audio, and language communities independently converged on the same realization: Next Token Prediction provides a general-purpose training objective that can handle any modality if you can transform that modality's data into a sequence of tokens. However, this convergence happened in a decentralized, domain-by-domain manner. Work on autoregressive image generation (DALL-E, VQGAN, ImageGPT) developed largely in parallel with work on autoregressive audio generation (AudioLM, VALL-E, MusicGen) and work on multimodal instruction-following models (LLaVA, MiniGPT-4, Flamingo). Each subcommunity developed its own terminology, tokenization strategies, architectural patterns, and evaluation practices.

The paper's core diagnosis is that no prior survey or framework treats NTP as the unifying organizing principle across all these modalities and task types simultaneously. Existing surveys, as the authors note in Section 1.2, have focused on narrower slices:

• Yin et al. (2023) and Awais et al. (2023) reviewed multimodal LLMs but centered on vision-language understanding capabilities, largely excluding generation.
• Bordes et al. (2024) and Zhang et al. (2024) covered vision-language pretraining and transfer learning, but did not incorporate audio, video, or unified generation-understanding models.
• Specialized surveys have addressed hallucinations (Liu et al., 2024), efficiency (Jin et al., 2024), and multimodal agents (Xie et al., 2024), none of which attempt a cross-modal synthesis under the NTP lens.

The authors explicitly state their contribution in contrast to this landscape:

"Diverging from prior work that primarily focused on the understanding abilities of multimodal LLMs, our survey adopts a systematic perspective by integrating both understanding and generation in multimodal learning through the paradigm of next-token prediction. To the best of our knowledge, this is the first survey that reviews LMMs from the perspective of next token prediction."

This is not a claim of proposing a new method, but rather of providing the first unified taxonomy that makes sense of a fragmented research landscape.

Why This Matters

The significance of addressing this gap extends beyond academic organization. There are three interconnected reasons why a unified NTP-centric framework is important at this moment:

1. The field is approaching a practical bottleneck where the NTP abstraction is being extended to modalities it was never designed for. Language tokens are discrete, symbolic, and linearly ordered by syntactic convention — properties that align naturally with autoregressive sequence modeling. Images, audio, and video are continuous, high-dimensional, and structured in two or three dimensions with complex spatial and temporal dependencies. When the field forces these modalities into a 1D token sequence for NTP training, it necessarily introduces design decisions — how to tokenize, what order to present tokens, whether to predict multiple tokens in parallel — that fundamentally shape model behavior and efficiency. These decisions are currently made in an ad hoc, modality-specific manner, with each paper arguing for its own inductive biases (causal raster scan vs. random-order vs. next-scale prediction vs. masked parallel prediction). A systematic taxonomy of these choices, as this survey provides, helps future researchers understand the space of possible design decisions rather than rediscovering them independently for each new modality.

2. The scaling properties of multimodal NTP models are largely unknown. The paper explicitly flags this in Section 6.1: while scaling laws for text-only LLMs are well-established (Kaplan et al., 2020; Hoffmann et al., 2022), the returns from scaling multimodal NTP models — particularly with unlabeled interleaved multimodal data — remain under-explored. Some studies have hypothesized or partially explored these scaling behaviors (Aghajanyan et al., 2023), but the paper notes that data from different modalities exhibit different scaling curves, and the reasons behind these differences are unclear. Without a unified framework that tracks how tokenization choices, architectural decisions, and training objectives (discrete vs. continuous token prediction) interact with scaling, the field cannot make principled resource allocation decisions for multimodal model training.

3. The NTP paradigm is expanding to application domains beyond media understanding and generation. The paper's timeline (Figure 1) and taxonomy encompass models for robotics (RT-2), protein design, molecular chemistry, and embodied AI — domains where the NTP formulation is being applied to action sequences, molecular structures, and DNA tokens. Without a synthesis of what works and why across the already-explored modalities (vision, audio, video), each new application domain starts from scratch. A unified survey lowers the barrier for practitioners in these emerging areas by making explicit the design patterns, failure modes, and open questions that have been learned across modalities.

Where Prior Approaches Fall Short

The paper identifies four specific limitations in how the field has organized knowledge about multimodal NTP:

1. The distinction between understanding and generation has created artificial silos. Prior surveys have overwhelmingly treated multimodal understanding models (LLaVA-style vision-language models) and multimodal generation models (DALL-E-style text-to-image systems) as separate research areas. However, as the paper's timeline shows, the last 1–2 years have seen an accelerating convergence: models like Chameleon, Emu3, Transfusion, Show-o, and Janus now handle both understanding and generation within a single NTP-trained model. The paper argues that the NTP framework provides a natural lens for discussing this convergence because both tasks reduce to predicting the next token — the difference lies only in whether the predicted token belongs to the text vocabulary, a discrete multimodal codebook, or a continuous feature space. This unified perspective is absent from prior work.

2. Tokenization has been treated as an implementation detail rather than a first-class design axis. Prior surveys and individual papers often present tokenization as a pre-processing step — "we use a CLIP encoder" or "we apply VQGAN" — without analyzing how tokenization choices systematically affect downstream modeling. This survey's Section 2 elevates tokenization to the first substantive technical discussion, explicitly categorizing all methods into discrete tokenization (Section 2.2–2.3) and continuous tokenization (Section 2.4–2.5), and further subdividing by training methodology (auto-encoding, denoising auto-encoding, supervised pretraining, contrastive learning). This is significant because the choice between discrete and continuous tokens fundamentally determines what training objectives are available (cross-entropy vs. regression/MSE), what architectural patterns are needed (vocabulary expansion vs. feature prediction heads), and how generation quality is affected (VQ artifacts vs. diffusion-conditioning).

3. The model architecture space is large and poorly charted. While individual papers make compositional vs. unified architecture choices (e.g., LLaVA vs. Chameleon), the field lacked a systematic comparison of these design philosophies. The paper's Section 3.4 explicitly contrasts compositional models (strong external encoders/decoders + alignment layers) against unified models (lightweight modality-specific components + heavy shared backbone) along three axes: general multimodal intelligence, training efficiency, and deployment efficiency. Prior work has not made these tradeoffs explicit at a survey level.

4. Training objectives and task representation have been fragmented by modality. The paper's Section 4 introduces a crucial distinction between discrete token prediction (DTP) and continuous token prediction (CTP) that cuts across modalities. Prior work in vision generation tends to use either DTP (VQ-based, cross-entropy loss) or CTP (diffusion-conditioning, MSE loss), while audio models have explored spectrogram prediction (continuous) and codec-based approaches (discrete). The survey unifies these under a common notation (Equation 2–3) and training stage framework (Figure 14), making it possible to compare approaches across modalities and task types for the first time. The explicit discussion of whether different training objectives can "coexist harmoniously in one NTP model" (Section 4.1.2) is a question that no prior survey has framed.

How This Paper Positions Itself

The survey positions itself as taxonomy-first, method-agnostic, and forward-looking. It does not advocate for any specific model, tokenization method, or training objective. Instead, it provides the organizational scaffolding that lets readers understand how existing methods relate to each other and where the field's open questions lie.

This positioning is evident in three structural choices:

First, the survey's five-part structure (tokenization → architectures → task representation → datasets → challenges) follows the information flow through an NTP pipeline (Figure 2), from raw multimodal data through tokenized sequences through model processing through downstream task execution. This is a deliberate pedagogical choice: by walking through the pipeline in order, the survey aims to make explicit how design decisions at one stage (e.g., tokenizer training method) propagate to constrain options at later stages (e.g., training objective).

Second, the survey emphasizes tables as a primary organizational device (Table 1 through Table 7), providing dense reference material that maps tokenizers, models, datasets, and benchmarks across modalities. This reference-like quality distinguishes it from narrative-only surveys: the goal is to create something practitioners can return to when making implementation decisions.

Third, the survey explicitly identifies open challenges (Section 6) that cut across the taxonomy: scaling behavior (Section 6.1), modality interference in multi-task training (Section 6.2), efficiency (Section 6.3), and NTP as a universal interface for domains beyond media (Section 6.4). These challenges are positioned not as criticisms of specific methods but as structurally inherent to the NTP-unification project itself — questions that arise precisely because the field is trying to force diverse modalities into a single training paradigm.

The paper's relationship to the broader "everything-as-tokens" trend is characterized by cautious optimism with explicit boundary-marking. While celebrating the successes of NTP unification, the survey repeatedly flags where the paradigm's assumptions break down: the continuous vs. discrete tension, the conflict between text token prediction and multimodal token prediction within a single model (Section 4.1.1, citing evidence from Zhang et al., 2023 that predicting text tokens provides no benefit for multimodal token prediction and can cause strong optimization conflicts), the inability of unified models to match the performance of specialized systems at equivalent compute budgets (Section 3.4), and the lack of evidence for emergent multimodal abilities analogous to those observed in text-only LLMs (Section 6.1). This balanced positioning — neither boosterism nor skepticism, but systematic cataloging of what works and what doesn't — gives the survey its methodological contribution: the taxonomy itself becomes the tool for identifying where future work is needed.

3. Technical Approach

3.1 Reader Orientation

This is a taxonomy paper — a survey that organizes a fragmented research landscape by proposing a unified conceptual framework rather than introducing a new model or algorithm. The core idea is that all current multimodal learning systems, regardless of modality or task (understanding vs. generation), can be understood as instances of a single pipeline: convert raw data into tokens → feed tokens through a transformer decoder that predicts the next token → convert predicted tokens back into the target modality's output space. The paper does not build a system itself; rather, it provides the mental scaffolding for understanding, comparing, and designing systems that follow this NTP paradigm.

The "shape" of the solution is a five-axis taxonomy that walks through the information flow of any NTP-based multimodal model: (1) how raw multimodal data gets converted into tokens (tokenization), (2) how the backbone model processes those token sequences (architecture), (3) how training objectives and task representations unify different problems under NTP (task representation), (4) what data and benchmarks are used (datasets & evaluation), and (5) what structural problems remain unsolved (challenges). The taxonomy does not prescribe a single design; it maps the space of possible design decisions at each stage and explains the tradeoffs.

3.2 Big-Picture Architecture (Diagram in Words)

The survey conceptualizes every multimodal NTP system as a pipeline with five stages, illustrated in Figure 2:

1. Tokenization (Encoding): Raw multimodal data — an image, an audio clip, a video, text — enters the system. A modality-specific tokenizer converts this raw input into a sequence of tokens. These tokens can be either discrete (integer indices drawn from a finite vocabulary, produced by vector quantization) or continuous (real-valued vectors in a learned embedding space, produced by an encoder network without quantization). The tokenizer's design determines whether the system can later reconstruct the original data (generation capability) and how semantically rich the representations are (understanding capability).

2. Sequence Assembly: Tokens from potentially multiple modalities are concatenated into a single sequence. Special tokens (e.g., BOS/EOS markers, placeholder tokens) signal modality boundaries and task structure. This produces a 1D sequence regardless of the original data's dimensionality — images are linearized (typically raster scan: left-to-right, top-to-bottom), videos become frame-by-frame sequences, audio becomes temporal sequences.

3. Backbone Model (Transformer Decoder): The concatenated token sequence feeds into a transformer decoder — typically the same architecture used for causal language modeling in LLMs — that predicts the next token autoregressively. The key design space here is the attention mask pattern (causal, semi-causal, prefix-LM, or encoder-decoder) and whether the model is compositional (lightweight backbone + heavy external encoders/decoders) or unified (heavy backbone + lightweight modality-specific components).

4. Training Objective: The model is trained to minimize prediction error on the next token. For discrete tokens, this is standard cross-entropy loss over a vocabulary that includes both text tokens and multimodal codebook indices. For continuous tokens, this is typically mean-squared error (MSE) regression against target feature vectors. The loss function choice propagates constraints backward: discrete token prediction requires quantization during tokenization; continuous token prediction requires output head modifications for regression.

5. Detokenization (Decoding): Predicted tokens are converted back into their original modality. Discrete tokens are mapped through a decoder (e.g., VQGAN decoder for images, neural vocoder for audio) that reconstructs the raw signal from codebook indices. Continuous tokens are fed into modality-specific generative models (typically diffusion models like Stable Diffusion for images) as conditioning signals. For understanding-only systems, no detokenization is needed — the output is text, and the pipeline stops at token prediction.

Information flows strictly forward through these five stages, but design decisions at each stage constrain or enable options at later stages. The paper's taxonomy structure mirrors this pipeline: Section 2 covers tokenization, Section 3 covers the backbone model, Section 4 covers training objectives and inference-time prompt engineering, and Section 6 covers the unresolved tensions that arise from this sequential dependency.

3.3 Roadmap for the Deep Dive

• First, the formal definition of tokenization (Equations 1–3 and Figure 4): We start here because tokenization is the foundational decision that determines everything downstream — whether the system can handle generation, what loss functions are available, and what architectural patterns are needed. The paper presents a unified mathematical notation for both discrete and continuous tokenization, which we'll unpack in detail.
• Second, discrete tokenization mechanics (Sections 2.2–2.3, Figures 5–6): Discrete tokenization via vector quantization is the core enabling technology for multimodal generation within NTP. We'll walk through VQVAE, its training objectives (reconstruction + codebook + commitment loss), its variants (residual VQ, product quantization, multi-scale quantization, look-up-free quantization), and how these are adapted for images, audio, and video — each with modality-specific inductive biases.
• Third, continuous tokenization mechanics (Sections 2.4–2.5, Figure 7): Continuous tokenization dominates the multimodal understanding literature. We'll cover encoder design choices (transformer vs. CNN), alignment mechanisms (slot-based resamplers vs. linear projections vs. MLPs), and modality-specific considerations (resolution handling for images, temporal fusion for video, spectrogram processing for audio). We'll also cover how continuous tokens enable generation through output alignment to external decoders.
• Fourth, backbone model architectures (Sections 3.1–3.4, Figures 8–12): With tokenization established, we cover how the transformer decoder processes multimodal token sequences — the unified task structures for vision (Figure 11) and audio (Figure 12), the attention mask patterns (Figure 10), and the compositional vs. unified architecture debate with explicit comparison along intelligence, training efficiency, and deployment efficiency axes.
• Fifth, training objectives and task representation (Sections 4.1–4.4, Figures 13–16): We cover the discrete vs. continuous token prediction distinction (Equations 2–3), the multi-stage training pipeline (modality alignment → instruction tuning → preference alignment), and inference-time prompt engineering techniques (multimodal in-context learning and chain-of-thought reasoning).
• Sixth, datasets, benchmarks, and the evidence that this taxonomy is useful (Sections 5.1–5.2, Tables 5–7, Figures 17–18): We cover the training data taxonomy (text-only, image-based, video-based, audio-based pretraining corpora; understanding vs. generation instruction tuning datasets), holistic and emerging evaluation benchmarks, and performance comparisons that show NTP-based models achieving competitive or superior results to non-NTP approaches on both understanding (VQAv2, MMMU) and generation (ImageNet FID, GenEval) tasks.

3.4 Detailed, Sentence-Based Technical Breakdown

This is a taxonomy paper — an organizational framework for understanding a research landscape rather than a system that computes outputs from inputs. The core intellectual contribution is the five-axis categorization that maps the space of design decisions for building multimodal systems trained with next-token prediction. Rather than proposing a specific model, the paper provides the vocabulary and conceptual structure for reasoning about tradeoffs: discrete vs. continuous tokens, compositional vs. unified architectures, cross-entropy vs. regression losses, modality-aligned vs. modality-interleaved training data. The following subsections walk through each axis of the taxonomy in the order information flows through an NTP pipeline, explaining what design choices exist at each stage, why those choices matter, and how they interact with choices at other stages.

---

Multimodal Tokenization: The Formal Definition and the Discrete/Continuous Dichotomy

Tokenization is the first and most consequential design decision in any multimodal NTP system because it determines the language in which the transformer decoder will reason about non-textual data. The paper defines tokenization formally (Section 2.1, Equation 1) and then bifurcates all approaches into two mutually exclusive categories.

Formal definition. The paper defines tokenization as a function:

$f(x) = z$

where $x \in X$ is a raw sample from the multimodal input space $X$ (which could be an image, an audio waveform, a video clip, or any other modality), and $z \in Z_f$ is the resulting representation in the tokenizer's output space $Z_f$.

What it defines: The mapping from raw multimodal data to a token representation. This is the interface between the continuous, unstructured external world and the discrete sequential world of the transformer decoder. The function $f$ is modality-specific — different tokenizers exist for images, audio, and video — but the output space $Z_f$ is the common "language" that the backbone model processes.

Why this abstract definition matters: By defining tokenization generically as any function from raw data to token space, the paper deliberately avoids committing to a specific implementation. This lets the taxonomy capture both discrete tokenization ($Z_f$ is a finite set of integer indices) and continuous tokenization ($Z_f$ is $\mathbb{R}^d$ — a $d$-dimensional real vector space) under the same framework. It also makes explicit that the "right" tokenizer depends on which downstream properties matter: reconstruction ability (needed for generation), representation ability (needed for understanding), or both.

The paper then partitions all tokenizers into two fundamentally different types based on the nature of $Z_f$:

Discrete tokenization: The representation space $Z_f$ is a finite set — a codebook $C = \{c_1, c_2, ..., c_N\}$ where $c_i \in \mathbb{R}^d$ and each representation $z$ is composed of codes drawn from this codebook. The key operation is quantization: mapping the continuous output of an encoder network to the nearest codebook entry. This is what makes the representation "discrete" — there are only $N$ possible values for each token position, exactly analogous to how a text tokenizer has a finite vocabulary of subword units. The consequence is that discrete tokens can be trained with cross-entropy loss (the standard next-token objective from language modeling) and can be decoded back to the original modality through the codebook via a learned decoder network. This makes discrete tokenization the standard choice for multimodal generation systems (DALL-E, VQGAN-based models, Chameleon, audio codec models).

Continuous tokenization: The representation space $Z_f$ is a continuous vector space — typically $\mathbb{R}^d$ for some embedding dimension $d$. There is no quantization step; the encoder outputs real-valued vectors directly. The consequence is that continuous tokens capture more information with less compression artifacts (the paper notes they offer "less information loss and improved data representation compared to discrete tokens," Section 2.4), but they cannot be directly trained with cross-entropy loss over a vocabulary — they require regression objectives (MSE) or serve as conditioning signals for external generative models. This makes continuous tokenization the standard choice for multimodal understanding systems (LLaVA, Qwen-VL, audio understanding models) where generation is not needed and the priority is semantic richness of representations.

Why this dichotomy is fundamental — not just a taxonomy convenience. The choice between discrete and continuous tokenization propagates constraints forward through every subsequent design decision: (1) Discrete tokens enable standard language modeling heads (classification over a vocabulary) and standard cross-entropy training; continuous tokens require regression heads and MSE or diffusion losses. (2) Discrete tokens naturally support generation via codebook decoding; continuous tokens require external generative models (diffusion models, VAEs) as post-processing steps. (3) Discrete tokens impose an information bottleneck (quantization error) that can harm understanding; continuous tokens preserve more information but lack the convenient discrete structure for autoregressive generation. This tension — that no single tokenization strategy is optimal for both understanding and generation — is one of the survey's central unresolved challenges (Section 6.2: modality interference).

---

Training Methods for Tokenizers: Four Fundamental Objectives

Before diving into specific tokenizer architectures, the paper establishes (Sections 2.1.2–2.1.3, Figure 5) that all tokenizer training methods fall into one of four categories, each optimizing for different properties:

Auto-Encoding (AE): The encoder maps input to a latent representation; the decoder reconstructs the input from that representation. The training objective is to minimize reconstruction error — typically mean squared error between the original input and the decoded output. This is the foundation of VQVAE, VAE, and diffusion models. AE-trained tokenizers excel at reconstruction ability (faithfully reproducing the original data from tokens) because that is exactly what they are optimized for. However, they may learn latent representations that are poor for semantic understanding because reconstruction can be achieved with low-level texture and color information without capturing high-level concepts. The paper notes that "discrete tokens are typically generated by quantizing the continuous data representation within the latent space of auto-encoders" — AE provides the encoding-decoding framework; quantization is added on top to produce discrete tokens.

Denoising Auto-Encoding (DAE): A variant of AE where the input is corrupted with noise, and the model must reconstruct the clean original. In transformer-based models, this takes the form of Masked Language Modeling (BERT) or Masked Image Modeling (MAE, BEiT) — randomly masking portions of the input and training the model to predict the missing content. The paper identifies DAE as the training paradigm behind HuBERT (audio), BEiT (vision), and BERT (language). DAE-trained tokenizers learn more robust features than pure AE because the model must infer missing information from context, forcing it to learn semantic and structural patterns rather than memorizing low-level statistics.

Supervised Pretraining: The tokenizer is trained on a specific labeled task — object detection, depth estimation, semantic segmentation for vision (SAM, ViTDet), or speech recognition for audio (Whisper, trained on 680,000 hours of weakly labeled data). The resulting representations are task-specific and capture features relevant to the supervised objective. The paper highlights that some LMMs (DeepSeek-VL, Cambrian-1) use tokenizers trained on multiple supervised vision tasks to extract diverse visual features. In audio, Whisper's encoder has become a standard tokenizer for speech LLMs because its representations capture phonetic and linguistic content from the speech recognition objective.

Contrastive Learning: The model is trained to bring representations of "positive" pairs closer together and push "negative" pairs apart in the embedding space. The pairs can be within the same modality (DINO: image-image pairs for vision representation learning) or across modalities (CLIP: text-image pairs to align language and vision representations). Contrastive-trained tokenizers excel at semantic alignment — CLIP representations, for instance, map images of dogs and the text "a dog" to nearby vectors, making the representations directly interpretable by language models. This is why CLIP and its variants (EVA-CLIP, SigLIP) are the dominant tokenizers for multimodal understanding systems (LLaVA, MiniGPT-4, Qwen-VL).

The tradeoff between representation and reconstruction is explicit. The paper states (Section 2.1.2): "effective representation encodes semantically relevant information into the latent space while removing redundant information" — this favors contrastive and DAE training. Conversely, for generation, "it is expected that the tokenization function $f$ is invertible or nearly invertible" — this favors AE training with careful reconstruction objectives. The survey notes a practical consensus in the field: "LMMs that only feature multimodal understanding capabilities opt for tokenizers with superior representation abilities like CLIP, as they do not require reconstruction of the multimodal information. Conversely, LMMs supporting multimodal generation capabilities tend to choose VQVAE as the tokenizer." This observation — that current practice splits along the understanding/generation boundary — motivates the challenge of building unified models that need both properties simultaneously.

---

Discrete Tokenization: VQVAE Mechanics, Training, and Variants

The centerpiece of discrete tokenization is the Vector Quantized Variational Autoencoder (VQVAE) , proposed by van den Oord et al. (2017). The paper describes its architecture and training in Sections 2.2.1–2.2.2 (Figures 6) before surveying the variants developed to address its limitations.

VQVAE architecture and forward pass (Figure 6). A VQVAE consists of three components: an encoder $E$ (typically a CNN or transformer), a codebook $C = \{c_1, c_2, ..., c_k\}$ (a learned embedding table where each $c_i \in \mathbb{R}^d$ is a codebook vector), and a decoder $D$ (mirroring the encoder architecture). The forward pass works as follows:

1. The encoder maps the input $x$ (e.g., an image) to a continuous latent representation $z = E(x)$. For a $256 \times 256$ image, this might produce a $16 \times 16$ grid of $d$-dimensional vectors — 256 latent vectors total.

2. For each vector in $z$, the quantizer finds the nearest neighbor in the codebook: $\hat{z}_i = c_j$ where $j = \arg\min_j \|z_i - c_j\|_2^2$. This replaces each continuous latent vector with its closest discrete code. The resulting $\hat{z}$ is a grid of codebook indices.

3. The decoder reconstructs the input from the quantized latents: $\hat{x} = D(\hat{z})$.

Training objectives (three loss components). The VQVAE is trained with a composite loss function that addresses the non-differentiability of the quantization step (the $\arg\min$ operation has zero gradient almost everywhere):

• Reconstruction loss: $\mathcal{L}_{\text{recon}} = \|x - \hat{x}\|_2^2$ (or other similarity metrics). This ensures the codebook preserves enough information to faithfully reconstruct the input. Without it, the codes could collapse to representations that are easy to quantize but uninformative.

• Codebook loss: $\mathcal{L}_{\text{codebook}} = \|\text{sg}[z] - \hat{z}\|_2^2$, where $\text{sg}[\cdot]$ is the stop-gradient operator. This moves the codebook vectors toward the encoder outputs — when an encoder vector $z_i$ is quantized to codebook entry $c_j$, this loss pulls $c_j$ toward $z_i$. This is necessary because the codebook is learned; without it, codebook vectors might never be used.

• Commitment loss: $\mathcal{L}_{\text{commit}} = \|z - \text{sg}[\hat{z}]\|_2^2$. This moves the encoder outputs toward their assigned codebook vectors — the reverse direction of the codebook loss. It prevents the encoder from producing representations that fluctuate arbitrarily relative to the codebook, encouraging stable quantization. The straight-through estimator is used to pass gradients through the non-differentiable quantization: during backpropagation, the gradient of $\hat{z}$ with respect to $z$ is treated as the identity, as if quantization did not occur.

Why this three-part loss design? The reconstruction loss alone would be insufficient because the quantization step blocks gradient flow — the encoder would receive no training signal. The codebook and commitment losses together implement a form of alternating optimization: the codebook chases the encoder (codebook loss) while the encoder commits to the codebook (commitment loss), ensuring that both converge to a mutually consistent discretization of the data distribution.

Evaluation of VQ tokenizers. The paper distinguishes two evaluation dimensions (Section 2.2.2):

• Reconstruction ability: How faithfully the VQVAE can reproduce inputs after encoding and decoding. Metrics: L2 distance, Peak Signal-to-Noise Ratio (PSNR), reconstruction Fréchet Inception Distance (rFID) — the FID between original ImageNet validation images and their reconstructions.
• Generation ability: How well new samples generated by training an autoregressive model over the codebook indices match the training distribution. Metrics: Inception Score (IS), generation Fréchet Inception Distance (gFID) computed against training set statistics.

Variants addressing specific limitations. The paper catalogs six classes of VQVAE extensions (Section 2.2.1):

1. Architectural improvements for reconstruction quality. Standard VQVAE uses CNN encoders/decoders. ViT-VQGAN replaces these with Vision Transformers, finding that transformer-based encoders and decoders produce better feature representations and higher reconstruction fidelity. Swin-MAE and Swin-Unet similarly adopt Swin Transformer architectures. Efficient-VQGAN is explicitly named as a representative of this line of work.

2. Codebook collapse solutions. A well-known problem in VQVAE training: as codebook size increases, a large fraction of codebook entries become "dead" — never selected as the nearest neighbor for any input. This reduces the effective vocabulary size and limits expressive power. Two solutions are highlighted:

• Look-up Free Quantization (LFQ): Reduces the embedding dimension of the codebook to zero, replacing the codebook with an integer set. The key claim: "rFID on ImageNet decreases from 2.5 to 1.4 when the LFQ vocabulary size increases from $2^{10}$ to $2^{16}$." By removing the learned embedding, LFQ eliminates the codebook collapse problem entirely and enables much larger effective vocabularies.
• Finite Scalar Quantization (FSQ): Projects the VAE representation down to a few dimensions that can be quantized into fixed values, creating an implicit codebook. This avoids the learned codebook entirely, sidestepping the collapse issue.

3. Multi-scale and hierarchical quantization. Standard VQVAE quantizes at a single spatial resolution. Multi-scale approaches decompose the representation into multiple granularities:

• Residual Vector Quantization (RQ, or RVQ): Instead of one quantization step, applies quantization recursively — the first quantization captures coarse structure, the residual (difference between original and quantized) is quantized again to capture finer detail, and so on. RQVAE and RQ-Transformer use this to "recursively quantize the feature map in a coarse-to-fine manner, employing a fixed-size codebook to maintain both precision and code diversity." This is the dominant approach in audio codecs (SoundStream, EnCodec, SpeechTokenizer) where multiple layers of residual quantization capture increasingly detailed acoustic information.
• Multi-scale Quantization (VAR): Encodes images into $K$ multi-scale discrete token maps using a shared codebook. The key innovation is changing the generation order: instead of standard raster-scan "next-token prediction," VAR generates images through "next-scale prediction" — predicting all tokens at coarse resolution, then all tokens at the next finer resolution, and so on. This injects 2D spatial inductive bias into the autoregressive process.
• Product Quantization: Factors the codebook into a product of smaller codebooks, enabling high-quality quantization with computationally feasible codebook sizes.

4. Auxiliary losses for perceptual quality. Vanilla VQVAE trained with MSE reconstruction loss produces blurry outputs, particularly for detailed content like faces. Two classes of auxiliary losses address this:

• Perceptual loss: Uses a pretrained CNN to compare high-level feature representations of the original and reconstructed images, rather than comparing pixels directly. This preserves semantic content even when low-level pixel differences exist.
• Adversarial loss (VQGAN): Adds a discriminator network trained to distinguish real from reconstructed images, while the VQVAE is trained to fool the discriminator. This is what distinguishes VQGAN from VQVAE: the GAN component dramatically improves visual quality at the cost of training complexity. The paper also notes that in audio, "semantic distillation" from self-supervised models like HuBERT and WavLM is used as an auxiliary loss to decouple acoustic and semantic information in residual VQ codebooks.

5. Token efficiency. Standard VQVAE produces a grid of tokens proportional to input resolution — a $256 \times 256$ image might produce $16 \times 16 = 256$ tokens. This is expensive for autoregressive modeling. TiTok tokenizes images into 1D latent sequences, "greatly reducing the number of tokens required to encode an image compared to previous methods." The paper highlights this as part of the broader tradeoff between token count (which affects autoregressive sequence length and thus computational cost) and reconstruction quality.

6. Embedding-free approaches. MaskBit explores binary quantization: "projects latent embeddings into $K$ dimensions and then quantizes them based on their sign values to produce bit token representations." This is notable because the resulting bit tokens exhibit "highly structured semantic representations," suggesting that even binary discretization preserves meaningful structure.

---

Discrete Tokenization for Specific Modalities

The paper's Section 2.3 argues that while generic VQ methods provide the basic mechanism, each modality requires specialized tokenizers because their data structures differ fundamentally. This is one of the survey's central insights: the NTP framework provides a unified sequence modeling interface, but the tokenization layer must inject modality-specific inductive biases to handle the structural differences.

Images (Section 2.3.1). The paper identifies three fundamental differences between images and text that shape image tokenizer design:

1. Rich information granularity: Images contain both low-level perceptual details (colors, textures, edges) and high-level semantic concepts (objects, actions, scenes). An ideal tokenizer must capture both — unlike text, where the token itself (a word) already represents a semantic unit.

2. Dense information: Every pixel carries information across multiple channels (RGB). In contrast, text is sparse — most positions in a sequence are not "words." This means image tokenization inherently involves compression: reducing the pixel count (e.g., $256 \times 256 \times 3 \approx 200K$ numbers) to a manageable token sequence (e.g., $16 \times 16 = 256$ tokens for VQGAN).

3. Two-dimensional spatial structure: Images are 2D grids, not 1D sequences. The linearization order (typically raster scan: left-to-right, top-to-bottom) is an arbitrary convention, not a natural structure. This introduces a problem: adjacent tokens in the 1D sequence may not be spatially adjacent in the 2D image (the right edge of one row is next to the left edge of the next row, which are spatially distant). This mismatch between 1D autoregressive modeling and 2D spatial structure is a persistent source of artifacts and inefficiency.

These differences create a three-way tradeoff in image tokenizer design:

• Representation: The tokens should align with text representations for understanding. This favors using CLIP-style contrastive training, which produces embeddings in a joint vision-language space. Models like SEED-LLaMA and LaViT use ViT encoders with CLIP alignment, optionally compressing tokens further using Q-Former or learned selectors.
• Reconstruction: The tokens must preserve sufficient detail for faithful image decoding. This favors VQGAN-style auto-encoding with perceptual and adversarial losses. The tradeoff is that better reconstruction usually means lower-level features (closer to pixels), which may align poorly with text representations.
• Token efficiency: Autoregressive cost scales quadratically with sequence length. An image tokenizer that produces 256 tokens per image makes multimodal training expensive; one that produces 32 tokens may lose fine detail. SEED-LLaMA compresses ViT features from unknown length down to 32 continuous tokens, then discretizes via quantization. LaViT dynamically selects which patch tokens to keep using a learned selector network. The paper frames this as the key challenge in image tokenizer design.

Audio (Section 2.3.2). Raw audio presents a different challenge: "typically stored as 16-bit integer values with a sampling rate that exceeds tens of thousands of values per second, which leads to extremely long sequences and renders next token prediction training more difficult." The key pressure in audio tokenization is temporal compression — reducing the sequence length from sample rate (e.g., 24,000 samples per second) to a manageable token rate (e.g., 50–75 tokens per second).

The paper traces the evolution of audio tokenization through three generations:

1. Early approaches: $\mu$-law companding (WaveNet). Projects each audio frame (sample) to an 8-bit value using a logarithmic compression function. This reduces the bit depth (from 16 to 8 bits) but does not reduce the sampling rate — sequences remain as long as the raw audio. Computationally prohibitive for autoregressive modeling.

2. Self-supervised learned representations with clustering (vq-wav2vec, HuBERT). Train a self-supervised model (wav2vec 2.0, HuBERT) to produce dense representations of audio frames, then cluster these representations to produce discrete tokens. HuBERT is "trained with a masked prediction task, whose targets are obtained through k-means clustering of learned features from earlier iterations." The resulting tokens capture high-level semantic content (phonetic information, speaker-independent features) but discard low-level acoustic details — "the reconstruction quality is relatively low, and speaker identity is lost." This tradeoff makes SSL tokens suitable for speech understanding tasks but insufficient for high-fidelity speech generation.

3. Neural codec models (SoundStream, EnCodec). Apply VQ-VAE directly to raw audio waveforms with residual vector quantization. These are "originally designed for audio compression" and "have the capability to encode waveforms into discrete codes and faithfully reconstruct them back into high-quality waveforms." The key advantage: "compared with highly-abstracted SSL-learned discrete tokens such as HuBERT tokens, codec models can efficiently reduce the length of token sequences while also maintaining multi-scale acoustic information indicating speaker identity." This makes them the dominant tokenizer for audio generation models (AudioLM, VALL-E, MusicGen).

The paper identifies a recent trend of imposing semantic supervision on codec tokens to improve language modeling. SpeechTokenizer adds semantic distillation — the first layer of its residual VQ codebook is trained to match HuBERT representations using a distillation loss, while subsequent layers capture acoustic detail. Mimi (used by Moshi) replaces HuBERT with WavLM as the semantic teacher and additionally "isolates the first-layer codebook from the RVQ process to achieve better semantic and acoustic disentanglement." This separation — semantic content in the first codebook layer, acoustic detail in subsequent layers — makes it easier for the autoregressive model to model linguistic content and speaker identity independently.

Video (Section 2.3.3). Video introduces a temporal dimension on top of images. The paper identifies three approaches to video tokenization:

1. Frame-by-frame image tokenization: Use an image VQVAE on each frame independently. This is the simplest approach, used by LVM, LWM, and the Unified-IO series. Its advantage is simplicity and reuse of pre-trained image tokenizers. Its disadvantage: no temporal compression — redundant information across frames (e.g., static backgrounds) is tokenized repeatedly, leading to "a high degree of token redundancy across frames, particularly in long-form videos, thereby imposing substantial computational demands." Additionally, frame-independent tokenization "fails to model temporal relationships between frames, leading to issues of temporal inconsistency."

2. 3D tokenization (VideoGPT, C-ViViT, MAGVIT): Apply 3D convolutions or 3D attention to compress across both spatial and temporal dimensions simultaneously. C-ViViT "uses a transformer architecture to split videos into 3D cubes, which are then discretized into token IDs." This provides temporal compression and better temporal consistency but at increased computational cost for the tokenizer itself.

3. Joint image-video tokenization (MAGVIT series): The paper highlights MAGVIT's design as particularly elegant: it tokenizes images and videos "with a shared vocabulary" by structuring video input as $T = 1 + n \times F_T$ frames — one initial frame (an image) plus $n$ clips of $F_T$ frames each. When $n=0$, the video reduces to an image, and the same tokenizer handles both. This is practically valuable for unified multimodal models that need to process both images and videos.

The paper also flags two desirable properties for video tokenizers: (1) joint image-video tokenization with shared vocabularies as described above, and (2) temporal causality — "using causal 3D architecture can ensure the tokenization and detokenization of each clip depend only on the preceding clips, facilitating autoregressive modeling along the temporal dimension." Causal tokenization ensures that the autoregressive assumption (each token depends only on previous tokens) holds not just in the backbone model but also in the tokenizer-decoder.

Other modalities (Section 2.3.4). The paper briefly notes that discrete tokenization has been extended to: object detection, segmentation, and depth maps (by representing task outputs as images and tokenizing them); robot actions (RT-2, VIMA, locomotion policies); and protein sequences (by treating amino acid sequences as DNA tokens). These applications demonstrate the extensibility of the discrete tokenization paradigm but are not the focus of the taxonomy.

---

Continuous Tokenization for Multimodal Input (Understanding)

Continuous tokenization dominates the multimodal understanding literature because it preserves more information than discrete quantization — the paper states it offers "less information loss and improved data representation compared to discrete tokens" (Section 2.4). However, continuous tokens cannot be directly predicted by a standard language model head (which outputs distributions over a finite vocabulary), so they require architectural modifications.

The paper decomposes continuous tokenization for understanding into two sequential steps (Section 2.4.1, Figure 7):

Step 1: Encoding — map raw data to a continuous representation space.

The encoder architecture can take several forms:

• Transformer-based encoders (dominant): The input (e.g., an image) is split into patches, each projected to a token embedding, and processed through self-attention layers. Two sub-types are distinguished:

  • Unimodal encoders trained on single-modality data: ViT for images, ViViT for video, AST for audio, USM for speech. These capture modality-specific structure but may not align with text representations.
  • Multimodal encoders trained with cross-modal objectives: CLIP (contrastive image-text training), EVA-CLIP (improved CLIP), CoCa (contrastive + generative), ImageBind (aligns six modalities to a shared space). These explicitly align representations with language, making them easier for LLMs to use. The paper notes that LLaVA uses CLIP, Qwen-VL uses OpenCLIP, and NExT-GPT uses ImageBind.

• CNN-based encoders (less common but still relevant): Used for their resolution flexibility and ability to capture local features. Flamingo uses NFNet (a normalizer-free ResNet) as its visual encoder. DaVinCi uses ResNet. VIMA uses a CNN for processing robot manipulation task inputs.

• Encoder-free approaches (rare): Fuyu-8B processes raw image patches directly after a single linear projection, with no separate encoder network. This provides "architectural simplicity" and native support for arbitrary resolutions but "shows reduced downstream performance compared to encoder-based models." ImageGPT trains a decoder-only transformer directly on raw pixel sequences — the encoder and backbone are the same model — but this "requires significant computational resources and is limited to low-resolution images."

Step 2: Alignment — bridge the gap between the encoder's representation space and the LLM's text embedding space.

Even multimodal encoders like CLIP are typically trained on image-caption pairs, not on the interleaved image-text-instruction sequences that LLMs process during instruction tuning. This creates two problems: (1) redundant information — an image encoder might produce hundreds of tokens, many of which are unnecessary for understanding; and (2) lack of contextual semantics — CLIP embeddings encode what is in the image, not how the image relates to surrounding text in a conversation. The alignment step addresses both.

Two alignment architectures are common:

• Slot-based Resampler (BLIP-2's Q-Former, Flamingo's Perceiver Resampler): A learned set of "query" vectors (typically far fewer than the encoder output tokens) cross-attend to the encoder's output, compressing it into a smaller, more informative set of tokens. The paper describes Q-Former as consisting of "a self-attention layer on the learned queries, a cross-attention layer between the encoded image representation and the learned queries, and a feed-forward layer." This is trained with multiple objectives: image-text matching, image-text contrastive learning, and image-grounded text generation. The Perceiver Resampler used in Flamingo generalizes this: "latents act as queries and initially cross-attend to keys and values calculated from the byte array (e.g., an image), followed by processing with a self-attention block, iterating several times." The key advantage of slot-based resamplers: they learn to extract only the information from the encoder that is relevant for the downstream LLM tasks.

• Projection (LLaVA's linear/MLP, Fuyu's linear): A simple learned mapping from the encoder's output space to the LLM's input embedding space. LLaVA uses a single linear projection initially; LLaVA-1.5 upgrades this to a two-layer MLP. The key advantage: architectural simplicity and minimal additional parameters. The disadvantage: no compression — the number of visual tokens equals the number of encoder output patches, leading to long sequences.

The paper mentions one alternative approach: Emu uses a "Causal Transformer (C-Former) to convert the image tokens autoregressively," meaning the alignment itself is autoregressive. Emu2 simplifies this to mean pooling followed by a linear projection, trading some representational power for efficiency.

Modality-specific continuous tokenization considerations (Section 2.5).

For images, the paper highlights three active research directions beyond the basic encoding + alignment pipeline:

• Image augmentation: Incorporating additional visual features beyond RGB — depth maps, edge maps, segmentation masks. Prismer encodes these as separate feature streams, compresses them with a shared experts resampler, and integrates them into a unified image representation. SAM-CLIP leverages the Segment Anything Model (SAM) and CLIP for distillation training, "boosting the semantic and spatial comprehension of the image encoder."
• Resolution and aspect ratio handling: Standard ViT encoders are trained on fixed-size square images. Fuyu supports arbitrary resolutions by processing raw pixels with position encoding interpolation. MS-ViT proposes variable patch sizes based on image region complexity — fine patches for detailed regions, coarse patches for uniform regions. This line of work addresses the fact that real-world images come in diverse aspect ratios and resolutions, and forcing them into square crops loses information.
• Heterogeneous images: Document images, vector graphics, charts, and diagrams have different structure than natural images. TextMonkey splits large documents into sub-images, encodes each individually, and uses "trainable shifted attention layers" to model cross-sub-image relationships.

For audio, continuous tokenization typically starts from spectrograms — 2D time-frequency representations computed via Short-Time Fourier Transform — rather than raw waveforms. The paper distinguishes two approaches:

• Traditional feature extraction: Mel-Frequency Cepstral Coefficients (MFCCs) and Mel-spectrograms are hand-designed features that capture perceptually relevant frequency content. These are processed by standard vision-style encoders (AST — Audio Spectrogram Transformer — applies a ViT to spectrogram patches).
• Self-supervised foundation models: wav2vec 2.0, HuBERT, WavLM, and Whisper are pre-trained on large corpora and produce continuous representations that capture phonetic, semantic, and speaker information. The paper notes that Whisper is "increasingly being employed as a foundational speech model" — its encoder, trained on 680,000 hours of weakly labeled data for speech recognition, produces representations that work well for diverse audio understanding tasks beyond ASR.

For video, the central challenge is temporal modeling. The paper distinguishes two approaches:

• Post-temporal fusion: Encode each frame independently with an image encoder (CLIP), then apply lightweight temporal pooling or attention to aggregate frame-level features into a video-level representation. CLIP4Clip and CLIPBERT use this approach. Advantage: leverages pre-trained image encoders with minimal additional training. Disadvantage: limited ability to model fine-grained temporal dynamics (motion, action progression).
• Full spatial-temporal fusion: Apply 3D convolutions or 3D attention to model spatial and temporal dimensions jointly from the start. VideoMAE and ViViT use this approach. Advantage: better temporal understanding. Disadvantage: quadratic scaling of 3D attention, prompting strategies like "decoupling temporal and spatial self-attention" and "sparse 3D attention" for efficiency.

Recent work (TimeChat, NumPro) adds timestamp information to continuous video tokens, enabling models to answer temporal questions ("what happened at 3:42?") that require explicit time-vision association.

---

Continuous Tokenization for Multimodal Output (Generation)

Continuous tokenization for generation (Section 2.4.2) solves a different problem than for understanding: how to make an LLM produce output tokens that can be decoded into images, audio, or video, given that the LLM's standard output head is a classifier over a discrete text vocabulary.

The paper decomposes this into two sub-problems:

Positioning: signaling where multimodal output should appear. The LLM needs to indicate "generate an image here" within a text sequence. Two mechanisms are documented:

• Placeholder tokens: Special tokens like [IMG1] through [IMGr] are added to the vocabulary. The LLM outputs these as part of its normal token sequence, and they serve as markers for where the external decoder should insert generated content. MiniGPT-5 and GILL use 7–8 image placeholder tokens that can be interleaved with text. NExT-GPT uses 5 image, 9 audio, and 25 video placeholder tokens. The advantage: these are just more tokens in the vocabulary, requiring no architectural changes. The disadvantage: the number of placeholder tokens is fixed, which may not match the natural length of the generated content.

• Begin-of-sentence (BOS) tokens: A single special token (e.g., <dream> in DreamLLM, [IMG] + [/IMG] in Emu) signals the start and end of multimodal content. The model then generates continuous features until it reaches the end token. The advantage: variable-length generation. The disadvantage: the LLM must learn to produce a different kind of output (continuous features vs. discrete tokens) for the duration of the multimodal segment.

Output representation: modifying the LLM head for continuous prediction. Three approaches are identified:

• Adapt the language modeling head to be regressive (Emu, Emu2): Add a regression head on top of the LLM's output hidden states that predicts continuous feature vectors instead of vocabulary logits. During multimodal generation segments, this regression head is active; during text generation, the standard classification head is used. The regression head is trained with MSE loss against target features (e.g., CLIP image embeddings or VAE latents).

• Introduce a new head for dense outputs (DreamLLM): Similar to the above but as a separate network branch rather than adapting the existing head. This avoids interference with the language modeling head but adds parameters.

• Use final hidden states directly (MiniGPT-5, GILL): Extract the hidden states before the language model head and use them as features for the decoder. No additional prediction head is trained. The advantage is simplicity; the disadvantage is that the hidden states are optimized for predicting text tokens, not for being good features for image generation.

Output alignment: converting LLM-produced features to decoder-compatible representations.

The LLM's continuous output tokens are in its own hidden state space, which is not aligned with the input space of external generative models (e.g., Stable Diffusion expects CLIP embeddings). Additional modules are needed:

• Transformer-based output projection (NExT-GPT): A small transformer maps the LLM's output features to the decoder's input space.
• Q-Former-style architecture (Mini-GPT5, GILL): A transformer encoder-decoder (similar to BLIP-2's Q-Former) converts continuous tokens into conditional features for Stable Diffusion. This is "trained with a feature alignment objective to ensure the generated features are compatible with the diffusion model's conditioning space."
• Linear projection (DreamLLM): A simple learned linear mapping.
• Direct usage (Emu, Emu2): The generated continuous tokens are used as latents directly, without additional alignment. This requires the LLM to learn to produce features in a space compatible with the decoder during training — the decoder and LLM are trained jointly.

The paper notes a critical dependency on external models: for continuous-generation systems, the quality of the final output depends heavily on the quality of the external decoder (e.g., Stable Diffusion for images, AudioLDM for audio). This is part of the compositional vs. unified model tradeoff discussed later.

---

Backbone Model Architectures: The Transformer Decoder and Attention Mask Patterns

With tokenization complete, the paper's Section 3 turns to the backbone model that processes multimodal token sequences. The fundamental unit is the transformer decoder — the same architecture used in GPT-style language models — because autoregressive next-token prediction is the defining characteristic of the NTP paradigm.

The three-step processing pipeline (Section 3.1, Figure 9):

Step 1: Encode each modality's raw data into token sequences using the appropriate tokenizer. Text goes through a standard text tokenizer; images through a VQ encoder or continuous encoder; audio through a neural codec or spectrogram-based encoder. Each tokenizer produces a sequence of embeddings in a shared representation space.

Step 2: Concatenate the token sequences from different modalities into a single interleaved sequence. An example from the paper: "Input text, images, audio, or image/audio history are encoded into sequences of tokens which are concatenated and used as input to a multi-modal transformer model."

Step 3: Autoregressively predict the next token using the transformer decoder. The output tokens can be decoded into text, an image, or an audio clip depending on their modality. The key architectural element is the attention mask, which controls which tokens can attend to which other tokens.

Attention mask patterns (Section 3.1, Figure 10):

The paper identifies four attention mask patterns that have been used across different multimodal NTP systems:

• Causal decoder mask: Each token can attend only to previous tokens (and itself). This is the standard autoregressive mask used in GPT-style language models. It works for both text generation and multimodal generation because it enforces the left-to-right ordering that NTP assumes. However, for understanding tasks where an image is provided as context, it's suboptimal — image tokens cannot attend to other image tokens that appear later in the sequence, preventing the model from building a holistic understanding of the image before generating text about it.

• Semi-causal decoder mask: Each token can attend to a limited window of future tokens in addition to all past tokens. The paper cites VAR as using this approach: "each token could attend to part of future token and all past tokens." This is motivated by the observation that strict causality may be too restrictive for 2D data like images where spatial relationships are not well-represented by a 1D causal order.

• Prefix-LM mask (prefix-causal): The conditioning information (e.g., an image or a task description) receives bidirectional attention — all prefix tokens can attend to each other. The generated content receives causal attention — each generated token attends to the entire prefix and to previous generated tokens. This is the most common pattern for multimodal understanding models because it allows the model to fully process the image before generating text about it. The main text refers to this as applying "bidirectional attention to the prefix, allowing the context within the prefix to interdepend and provide better representation, while using causal attention for autoregressive generation of the content to be generated."

• Encoder-decoder mask: The conditioning information is processed by a separate encoder (with full bidirectional attention). The decoder generates output autoregressively, attending to both the encoder output (via cross-attention) and previous decoder tokens (via causal self-attention). This is used by models like Whisper for speech recognition and some early multimodal models.

Unified task structures (Figures 11–12): The paper shows how different task types map onto the same NTP architecture by varying what appears in the input and output token sequences.

For vision tasks (Figure 11):

• Visual question answering: Image tokens + text question tokens → transformer decoder → text answer tokens.
• Text-to-image generation: Text tokens → transformer decoder → image tokens → image de-tokenizer.
• Image editing / image-to-image: Image tokens + text instruction tokens → transformer decoder → image tokens → image de-tokenizer. This requires both understanding (of input image and instruction) and generation (of output image).

For audio tasks (Figure 12):

• Audio understanding: Audio tokens + text instruction tokens → transformer decoder → text answer tokens.
• Audio generation: Text tokens + optional prompt audio tokens → transformer decoder → audio tokens → audio de-tokenizer.
• Full-duplex spoken dialogue (Moshi): Multiple audio streams (user input, model output) modeled concurrently. This requires generating streaming audio output while simultaneously processing incoming audio, which demands careful token scheduling to handle the real-time constraint.

---

Compositional vs. Unified Models: A Systematic Comparison

The paper's most architecturally significant taxonomy distinction (Sections 3.2–3.4, Figure 8, Table 3) divides all MMNTP models into two categories based on where the "intelligence" resides:

Compositional models: Use heavily pre-trained external encoders and decoders. The backbone LLM handles cross-modal reasoning and text generation, but understanding is outsourced to separate encoders (CLIP, Whisper) and generation is outsourced to separate decoders (Stable Diffusion, neural vocoders). The backbone's primary role is routing and integration — it processes text and coordinates the external components through alignment layers.

Unified models: Use lightweight, jointly-trained encoders and decoders. The backbone transformer decoder handles most of the understanding and generation computation. Modality-specific components are minimal — a simple VQ encoder/decoder or a learnable embedding table. The philosophy is that "a single unified transformer decoder backbone can effectively leverage the advanced techniques developed by the LLM community."

The paper systematically compares these along three axes (Section 3.4):

General multimodal intelligence. The paper argues unified models are "closer to a multimodal foundation model due to its end-to-end nature" because they "rely on a single NTP training objective, making them easier to scale compared to multi-module systems." The intuition: compositional models are systems of specialized components stitched together, while unified models are single models that learn to handle all modalities natively. However, the paper acknowledges this is a hypothesis, not an established fact — "whether multimodal generation helps understanding ability also remains unclear" and the scaling behavior of unified multimodal models is largely unknown (Section 6.1).

Training efficiency. Compositional models win here: "leveraging highly specialized external encoders and decoders often results in reduced training time for new tasks since these components are pretrained separately." A compositional model can reuse a pre-trained CLIP encoder and SDXL decoder with modest alignment training, while a unified model must learn everything from scratch (or at least from a text-only LLM initialization). The paper notes: "unified models leave most of the understanding and generation responsibility to one backbone model, leading to sub-optimal performance given the same amount of computation," citing Show-o as evidence.

Deployment efficiency. Unified models win decisively here, particularly quantization-based ones. The reason: "a single unified transformer decoder backbone can effectively leverage the advanced techniques developed by the LLM community for accelerating both training and inference, such as FlashAttention and vLLM." Compositional models require running multiple separate networks (encoder, backbone, decoder) with potentially different architectures and optimization characteristics, making system-level optimization harder.

Quantization-based vs. diffusion-based unified models (Section 3.3). Within unified models, the paper identifies two sub-approaches that are currently in active competition:

• Quantization-based autoregression: Uses VQVAE-based discrete tokens with standard cross-entropy training. Examples: Chameleon, Emu3, Unified-IO, Show-o. The advantage: directly compatible with LLM training infrastructure and inference optimizations. The disadvantage: VQ artifacts limit generation quality; the raster-scan generation order "contradicts the intrinsic nature of 2D images."

• Autoregressive diffusion: Integrates diffusion losses into the autoregressive training. Examples: Transfusion, MAR, Fluid. The key idea: instead of predicting discrete tokens, predict continuous latents using a diffusion objective at each autoregressive step. This "offers superior image generation quality compared to quantization-based methods" (per the paper's characterization of these works' claims) but requires more complex training — the diffusion loss involves multiple denoising steps per token.

The paper notes that this debate is unresolved: "while many diffusion-based AR methods claim better generation quality compared to quantization method, Emu3 significantly outperforms diffusion baselines like SDXL using a quantization-based AR approach. DnD-Transformer showcased that quantization-based AR generation has superior performance in generating rich-text images than diffusion models. In summary, it is not concluded yet which modeling method has superior performance than another currently."

---

Training Objectives: Discrete vs. Continuous Token Prediction

Section 4.1 introduces the core mathematical framing that unifies all NTP training:

For a sequence of input tokens $x_{1 \sim i-1} = \{x_1, x_2, ..., x_{i-1}\}$, the model predicts the next token $x_i \in V$, where $V$ is the total vocabulary (or continuous output space). The general loss function is:

$\mathcal{L}(\theta) = f(y_i, p_\theta(x_i | x_{1 \sim i-1}))$

where $\theta$ are the model parameters, $f$ is the loss function, $y_i$ is the target output, and $p_\theta(x_i | x_{1 \sim i-1})$ is the model's predicted distribution (or vector) for the next token.

What it computes: The discrepancy between the model's prediction and the target for each token position, summed over the sequence. The function $f$ varies depending on whether the target is discrete (cross-entropy) or continuous (MSE). The target $y_i$ comes from the training data — in self-supervised pretraining, it's simply the actual next token in the sequence; in supervised fine-tuning, it's the labeled correct output.

Why this unified notation matters: By writing a single equation that abstracts over modality, the paper makes explicit that all NTP tasks are the same at the algorithmic level — the only differences are what $V$ contains (text tokens, codebook indices, or continuous vectors) and what $f$ is (cross-entropy or MSE). This unification is the paper's central conceptual move.

The paper then bifurcates all training into two mutually exclusive categories based on the output space $V$ (Equation 3 and Figure 13):

Discrete Token Prediction (DTP, Section 4.1.1): The target $y_i$ is a discrete token from a finite vocabulary $V_D$. The loss function $f$ is cross-entropy — the standard language modeling objective. The vocabulary $V_D$ is the union of text tokens $V_T$ and multimodal discrete tokens $V_M$ (e.g., VQ codebook indices). During training, the model's output logits over the full vocabulary $V_T \cup V_M$ are compared to the one-hot target using:

$\mathcal{L}_{\text{CE}} = -\log p_\theta(y_i | x_{1 \sim i-1})$

where $p_\theta(y_i | x_{1 \sim i-1})$ is the softmax probability assigned to the correct token.

Why cross-entropy? It is the maximum-likelihood objective for categorical distributions, which is appropriate because the model is learning a distribution over a fixed set of discrete symbols. It has well-studied optimization properties (convex in logits, smooth gradients) and is the standard in language modeling, enabling direct transfer of LLM training infrastructure.

The paper notes a variant of DTP: parallel prediction where multiple tokens are predicted simultaneously rather than strictly autoregressively. MaskGIT and MAGVIT predict a portion of tokens at each step based on a confidence threshold. VAR predicts all tokens at one scale simultaneously, then moves to the next scale. The motivation: standard raster-scan autoregression is a poor inductive bias for 2D or 3D data. These parallel prediction methods "inject different inductive bias for different modality during NTP modeling, which is also an important challenge when unifying multiple modalities in multimodal NTP framework."

A critical observation from the literature: text token prediction and multimodal token prediction may conflict. The paper cites Zhang et al. (2023) as showing that "learning to predict text tokens has no benefit for predicting multimodal tokens and sometimes leads to strong conflict." This is evidence of modality interference (elaborated in Section 6.2) — the gradients from predicting text tokens and predicting image tokens may point in opposing directions, making joint optimization difficult.

Continuous Token Prediction (CTP, Section 4.1.2): The target $y_i$ is a continuous vector $V_S$ (where $S$ denotes "soft" or "continuous"). The loss function $f$ is typically mean squared error:

$\mathcal{L}_{\text{MSE}} = \|y_i - \hat{y}_i\|_2^2$

where $\hat{y}_i = p_\theta(x_i | x_{1 \sim i-1})$ is the model's predicted continuous vector (the output of a regression head rather than a softmax).

Why MSE? It penalizes the squared Euclidean distance between prediction and target, which is appropriate for continuous vectors where the goal is regression rather than classification. The paper notes that this is used by Emu-1 and Emu-2, where "the language model and diffusion model are trained simultaneously during the text-to-image instruction tuning stage."

The paper distinguishes two uses of CTP:

• As conditioning for external models: The LLM predicts continuous features that serve as input to a frozen or jointly-trained diffusion model for final generation. This is the compositional approach — the LLM acts as a "prompt encoder" for the diffusion model.
• As direct generation: The LLM predicts continuous image patches or latents directly, with the loss being diffusion loss rather than simple MSE. This is the unified-diffusion approach (Transfusion, MAR).

The coexistence problem. The paper explicitly flags that "whether different training objectives for different modalities can coexist harmoniously in one NTP model remains under-explored." This is a central open question: can a single model be trained with cross-entropy for text tokens, cross-entropy for discrete image tokens, and MSE for continuous feature tokens without optimization conflicts? The current evidence (modality interference) suggests it is difficult, motivating the architectural separation in models like Janus that use different encoders for understanding and generation.

---

Training Stages: Alignment, Instruction Tuning, and Preference Alignment

Section 4.2–4.3 (Figure 14) present the standard three-stage training pipeline for multimodal NTP models, mirroring the LLM training paradigm:

Stage 1: Modality alignment pretraining (Section 4.2). The goal is to align the representation spaces of different modalities with the language space so that the LLM can process multimodal inputs. The training data consists of paired multimodal data — image-caption pairs, audio-text pairs, video-text pairs. The objective is conditional language modeling: given multimodal context $I$ (e.g., an image), predict the text tokens $x_i$:

$\mathcal{L}(\theta_M) = f(y_i, p_\theta(x_i | x_{1 \sim i-1}, I))$

where only the parameters of the alignment module $\theta_M$ (e.g., the Q-Former or linear projection) are updated, while the LLM backbone and modality encoder remain frozen. The paper notes that "the modality encoder module — such as a CLIP vision encoder responsible for mapping multi-modal inputs into vectors in the shared space — is exclusively trained to enhance stability."

For generation alignment, the target $y_i$ becomes a multimodal token (discrete codebook index or continuous feature) rather than a text token:

$\mathcal{L}(\theta_M) = f(y_i, p_\theta(t_i | t_{1 \sim i-1}, C))$

where $t_i$ are multimodal tokens and $C$ is the text condition. This is typically done with web-scale data — LAION-400M (400 million image-text pairs) or LAION-5B (5 billion pairs).

Stage 2: Instruction tuning / supervised fine-tuning (Section 4.3.1–4.3.2). After alignment, the model can process multimodal inputs, but it may not follow complex instructions well. Instruction tuning trains on $(I, Q, A)$ triples — an image $I$, a user query $Q$, and a desired response $A$ — using the objective:

$\mathcal{L}(\theta) = f(A, p_\theta(x_i | x_{1 \sim i-1}, I))$

The key difference from alignment: all parameters may be updated, and the training data is higher-quality, task-specific collections rather than web-scale paired data. The paper notes that "the quality of the instruction tuning is the key to the ability" and surveys methods for creating instruction tuning datasets: adapting public benchmarks (MultiInstruct, M3IT), synthetic generation via GPT-4 (LLaVA, SVIT, ShareGPT4V), and mixing multimodal with text-only instruction data to preserve language capabilities.

Stage 3: Preference alignment (Section 4.3.3–4.3.4). This stage addresses subtle misalignments — hallucinations, biased responses, failure to ground answers in visual context. Two methods are covered:

• RLHF (Reinforcement Learning from Human Feedback): The model is optimized to maximize a learned reward model while staying close to a reference policy:

$\max_{\pi_\theta} \mathbb{E}_{x \sim \mathcal{D}, y \sim \pi_\theta(y|x)} [r(x, y)] - \beta D_{\text{KL}} [\pi_\theta(y|x) \| \pi_{\text{ref}}(y|x)]$

where $r$ is a trained reward model, $\pi_\theta$ is the current policy, $\pi_{\text{ref}}$ is the initial model, and $\beta$ controls the strength of the KL penalty that prevents the model from diverging too far from its initial behavior (which could cause it to exploit reward model weaknesses).

• DPO (Direct Preference Optimization): Bypasses the need for a separate reward model by directly optimizing on preference pairs using a closed-form relationship between the optimal policy and the reward:

$\max_{\pi_\theta} \mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}} \left[ \log \sigma \left( \beta \log \frac{\pi_\theta(y_w | x)}{\pi_{\text{ref}}(y_w | x)} - \beta \log \frac{\pi_\theta(y_l | x)}{\pi_{\text{ref}}(y_l | x)} \right) \right]$

where $y_w$ is the preferred (winning) response, $y_l$ is the rejected (losing) response, and $\sigma$ is the sigmoid function.

What DPO computes: The log-probability that a Bradley-Terry preference model assigns to the preferred response over the rejected one, under the current policy relative to the reference. The sigmoid converts the difference in log-probability ratios to a value between 0 and 1, and maximizing this encourages the model to increase the relative probability of preferred responses.

Why DPO over RLHF: DPO avoids the instability and computational cost of training and running a separate reward model. It directly optimizes the policy from preference data using a loss that is equivalent to the RLHF objective under the Bradley-Terry model. The paper notes that this has been applied to VLMs (Silkie, RLHF-V) using preferences annotated by GPT-4V or human feedback on hallucinations.

For generation preference alignment, the paper describes DPO-Diffusion (Wallace et al., 2023), which adapts DPO to diffusion models by using the ELBO (evidence lower bound) to handle the intractable likelihood of diffusion processes. The objective encourages the denoising process for preferred images to improve more than for non-preferred images.

---

Inference: Prompt Engineering for Multimodal Models

Section 4.4 (Figures 15–16) covers two techniques adapted from LLM research to multimodal NTP models:

Multimodal In-Context Learning (ICL, Section 4.4.1, Equation 14): The model receives a few demonstration examples $\{(x_i, y_i)\}_{i=1}^n$ in its context before the query $x_q$, organized in a template:

$Q: \{x_1\} \backslash n A: \{y_1\} \backslash n \backslash n ... Q: \{x_n\} \backslash n A: \{y_n\}$

where $Q$ and $A$ are question and answer delimiters. The model must infer the task pattern from these demonstrations and apply it to $x_q$. The paper identifies a key challenge: "performance in many multimodal ICL systems remains largely text-driven, with other modalities — such as images or videos — contributing minimally to overall task performance." This is a form of modality imbalance where the model relies on text cues rather than genuinely learning from multimodal demonstrations.

Multimodal Chain-of-Thought (CoT, Section 4.4.2): The model is prompted to produce step-by-step reasoning before giving a final answer. The paper identifies two challenges specific to multimodal CoT:

• Annotation cost: MM-CoT used ScienceQA, a dataset with human-annotated rationales, but "multimodal data often demands expensive and complex human-labeled information." VoT addresses this with active learning that combines machine and human annotation for video tasks.

• Language hallucination in the presence of visual input: When models are given both an image and a text question, they sometimes ignore the image and answer based on text priors alone. V* addresses this by "dynamically focusing on key visual regions, ensuring that visual details are accurately attended to, particularly in high-resolution images." CCoT generates scene graphs as intermediate representations to force explicit visual reasoning. DDCoT separates the reasoning and visual recognition roles between language and vision models.

---

Datasets and Evaluation: The Empirical Foundation

The paper's Section 5 provides comprehensive reference material organized into two subsections.

Training datasets (Section 5.1, Tables 5–6). The paper categorizes datasets by:

• Training stage: Pretraining (modality alignment) vs. fine-tuning (instruction tuning).
• Modality: Text-only, image-based, video-based, audio-based for pretraining; understanding vs. generation for fine-tuning.
• Scale and source: From small curated datasets (MS-COCO: 200K images) to web-scale collections (LAION-5B: 5 billion image-text pairs, RedPajama: 30 trillion text tokens, FineWeb: 15 trillion tokens).

For pretraining, the paper emphasizes the transition from paired data (image-caption, audio-text) to interleaved data (MMC4: 103M documents with 585M images interleaved with text; OBELISC: 141M documents with 353M images). Interleaved data is hypothesized to better match the distribution of real-world multimodal content (webpages, textbooks) but "their performance on downstream tasks does not provide a significant advantage over models trained solely on pairwise data" (Section 6.1).

For instruction tuning, the paper catalogs datasets ranging from early small-scale efforts (LLaVA: 158K examples, MiniGPT-4: 3.5K detailed descriptions) to larger collections (SVIT: 4.2M, M3IT: 2.4M, MMEvol: 480K). A notable inclusion: image editing instruction datasets (InstructPix2Pix: 313K editing triplets, MagicBrush: 10K manually annotated, HQ-Edit: 200K high-quality edits from GPT-4V + DALL-E 3).

Evaluation benchmarks (Section 5.2, Table 7, Figures 17–18). The paper distinguishes:

• Holistic benchmarks that test broad multimodal capabilities: MME (2,374 samples across 14 subtasks), MMBench (4,377 samples, 20 ability dimensions), SEED-Bench (19,242 questions, 12 dimensions), MMMU (11,550 college-level questions across 183 subfields), MVBench (4,000 video reasoning questions across 20 temporal tasks).
• Emerging benchmarks that target specific capabilities: SparklesEval (multi-image dialogue), MathVista (visual mathematical reasoning), HallusionBench (hallucination detection), PCA-Bench (embodied decision-making), TempCompass (temporal understanding in video).

The paper includes performance comparison figures (Figures 17–18) that show NTP-based models (colored markers) achieving competitive or superior performance to non-NTP approaches (gray markers) on both understanding tasks (VQAv2, MMMU) and generation tasks (ImageNet FID, GenEval). The figures are instructive because they demonstrate that the NTP paradigm — which was originally designed for text — now produces state-of-the-art results across modalities, supporting the paper's central claim that NTP is a viable unifying framework. However, the paper also notes that diffusion models "remain the de facto model in visual generation applications" and that "a detailed comparison of generation quality and efficiency between MMNTP models and diffusion models is still lacking in the literature" (Section 6.3), indicating that NTP unification is an ongoing trend, not a settled conclusion.

4. Key Insights and Innovations

Innovation 1: NTP as a Unifying Abstraction Layer for All Modalities — A Conceptual Reframing, Not a New Method

The paper's most fundamental intellectual contribution is not any specific technical proposal but rather the act of reframing the entire multimodal learning landscape through a single lens. Before this survey, the field treated vision-language models (LLaVA, Flamingo, Qwen-VL), discrete-token image generators (DALL-E, Parti, LlamaGen), audio codec language models (VALL-E, AudioLM, MusicGen), and diffusion-conditioned multimodal models (Emu, Transfusion) as belonging to different subfields with different terminology, different evaluation protocols, and different communities. A researcher working on autoregressive image generation via VQGAN tokens would not naturally see their work as conceptually continuous with a researcher building audio speech synthesis via EnCodec tokens, even though both were using exactly the same training objective — next-token prediction — applied to sequences of discrete codebook indices.

The survey's core conceptual move is to assert that this commonality is not superficial but fundamental. By organizing the entire field around the five-stage pipeline that every NTP-based multimodal system implements (tokenize → assemble sequence → transformer decode → apply training objective → detokenize), the paper reveals that design choices at each stage are cross-modal. Every system must answer the same questions: discrete or continuous tokens? Causal or bidirectional attention? Cross-entropy or regression loss? Compositional or unified architecture? Prior surveys partitioned these questions by modality — "vision tokenizers," "audio tokenizers," "video tokenizers" — which obscured the fact that similar answers lead to similar tradeoffs regardless of modality. The paper's taxonomy makes this explicit: discrete tokenization always enables cross-entropy training but introduces quantization artifacts (Section 2.2); continuous tokenization always preserves more information but requires architectural workarounds for generation (Section 2.4); compositional architectures always offer better training efficiency but worse deployment simplicity (Section 3.4). These are modality-invariant design principles that emerge only when you look across modalities.

This reframing has practical consequences beyond intellectual clarity. When a robotics researcher considers tokenizing action sequences for an NTP-based policy (as in RT-2, cited in Section 2.3.4), the taxonomy provides immediate guidance: action sequences are temporally structured like audio, suggesting that residual VQ (developed for audio codecs) might be more appropriate than standard VQ; action generation requires causal modeling, suggesting prefix-LM or causal attention masks; training would use discrete token prediction with cross-entropy loss. The paper does not make these arguments explicitly; rather, the taxonomy itself encodes the knowledge that lets practitioners reason about new modalities by analogy to known ones. This is the survey's operational contribution: it transforms NTP from "a thing that works" into "a design space you can navigate."

The significance of this reframing is underscored by what the paper reveals about the limits of NTP unification. By systematically cataloging where the paradigm breaks down — continuous vs. discrete tokenization as an unresolved tension (Section 2.1), text token prediction conflicting with multimodal token prediction in joint training (Section 4.1.1, citing Zhang et al., 2023), the inability of unified models to match specialized systems at equivalent compute (Section 3.4), and the absence of evidence for emergent multimodal abilities analogous to those in text-only LLMs (Section 6.1) — the paper provides not a triumphalist narrative but a diagnostic map of failure modes. This is a more valuable contribution than a purely positive survey because it directs future research toward the structural problems rather than the easy wins. The taxonomy is not just a filing system; it is a hypothesis-generating machine that identifies where the NTP abstraction is strained and where new ideas are needed.

Innovation 2: Elevating Tokenization from Implementation Detail to First-Class Design Axis

Prior surveys and individual papers overwhelmingly treated tokenization as a preprocessing step — "we use a CLIP ViT-L/14 encoder" or "images are tokenized via VQGAN with a codebook of size 16384" — mentioned in a sentence or two in the methods section and never analyzed systematically. The implicit assumption was that tokenization is an engineering detail: choose whatever works for your modality, calibrate it on reconstruction quality or downstream accuracy, and move on to the interesting parts (architecture, training, scaling).

This paper argues, through its structural choices, that this assumption is wrong and costly. The survey devotes its first substantive technical section (Section 2) to tokenization, subdividing it into discrete and continuous approaches, cataloging training methods (auto-encoding, denoising auto-encoding, supervised, contrastive), and providing a modality-by-modality analysis of how tokenizer design interacts with data structure (Section 2.3 for discrete, Section 2.5 for continuous). This is not merely thorough documentation; it is an argument by structure that tokenization is where the most consequential design decisions are made.

The innovation lies in making three implicit tradeoffs explicit and nameable:

First, the representation-reconstruction tradeoff. The paper states that an ideal tokenizer should possess both representation ability (encoding semantically relevant information for understanding) and reconstruction ability (enabling faithful decoding for generation), then immediately acknowledges that "these abilities are not mutually exclusive; their balance depends on the training techniques used" (Section 2.1.2). This is a concise formulation of what is, in practice, a deep conflict: contrastive training (CLIP-style) produces semantically rich representations but discards the low-level information needed for reconstruction; auto-encoding training (VQGAN-style) preserves pixel-level detail but may produce representations that are poorly aligned with language. The field's current solution — use CLIP encoders for understanding models and VQGAN tokenizers for generation models — is evidence that this tradeoff has not been resolved; it has been sidestepped by building different systems for different tasks.

Second, the token efficiency–information preservation tradeoff. The paper documents this across modalities: image tokenizers that produce 256 tokens per image enable high-quality reconstruction but make autoregressive training expensive (quadratic in sequence length); tokenizers compressed to 32 tokens (SEED-LLaMA) or 1D sequences (TiTok) reduce computational cost but risk losing fine detail. Audio tokenizers face the same pressure — neural codecs compress 24,000 samples per second to 50–75 tokens per second, but the compression ratio fundamentally bounds reconstruction fidelity. The paper's taxonomy reveals this as a cross-modal invariant: every modality must navigate the fact that more tokens = more information but more tokens = quadratic cost in the transformer decoder.

Third, the linearization order problem. The paper notes that for 2D images and 3D videos, "the initial arrangement of image patches follows a raster order, preserving intrinsic sequential relationships, [but] this configuration lacks causal semantics, posing challenges for language modeling" (Section 2.3.1). This is a deep insight disguised as a practical note: autoregressive modeling assumes a natural sequential order (words have a before/after relationship), but pixels do not. The raster scan order (left-to-right, top-to-bottom) is an arbitrary convention — spatially adjacent pixels at row boundaries become distant in the 1D sequence — and this mismatch between the data's natural structure and the model's inductive bias causes problems (inefficient modeling of long-range spatial dependencies, difficulty generating globally coherent images). The paper documents the various attempts to address this — parallel prediction (MaskGIT, MAGVIT), next-scale prediction (VAR), random-order generation (RAR) — as different answers to the same question: what is the right "next" for data that has no natural sequential order?

By naming and systematizing these tradeoffs, the paper transforms tokenization from a black-box preprocessing step into a design space with known axes and known failure modes. This is a conceptual contribution with immediate practical value: a researcher building a new multimodal system can use the taxonomy (Table 2) to identify which tokenizer properties matter for their use case and navigate the documented tradeoffs rather than rediscovering them through trial and error.

Innovation 3: Identifying Modality Interference as a Structural Barrier to Unified Multimodal Models

The paper's diagnosis of modality interference (Section 6.2) is one of its most significant contributions, not because it solves the problem but because it names and characterizes a failure mode that had been observed but not systematized. The phenomenon is this: when a single transformer decoder is trained to predict both text tokens and multimodal tokens (image codebook indices, audio codec indices) using the same parameters, the optimization can become unstable. The gradients from predicting the next text token and the gradients from predicting the next image token may point in conflicting directions, causing slower convergence, worse final performance on both tasks, or training instability (gradient norm explosion).

The paper marshals specific evidence for this phenomenon from multiple sources. Zhang et al. (2023) showed that "learning to predict text tokens has no benefit for predicting multimodal tokens and sometimes leads to strong conflict" (Section 4.1.1). Chameleon's training required QK-Norm to stabilize gradient norms, a technique previously developed for other purposes but deployed here specifically to address multimodal training instability (Section 6.2, citing Henry et al., 2020). Unified models built on pretrained LLMs often experience "language capability deterioration when adding more modalities" (Section 6.2), meaning the text generation quality of the base LLM degrades after multimodal training — the model forgets some of its language skills while learning to handle images or audio.

What makes this diagnosis intellectually distinctive is that it reframes the unified model project — building a single model that handles understanding and generation across all modalities — as fundamentally a multi-task learning problem with negative transfer. The optimism of the NTP paradigm is that "everything is tokens, therefore everything is the same task." The paper's evidence suggests this is wrong: predicting the next word in a sentence and predicting the next patch in an image are different tasks that happen to use the same loss function, and forcing a single set of parameters to excel at both may be actively harmful.

The paper's contribution here is not just cataloging the interference but providing a framework for understanding when it occurs. The interference is worst when (1) the output spaces are different (text tokens vs. image codebook indices), (2) the training objectives are different (cross-entropy for both, but the statistical structure of text sequences and image patches differs dramatically), and (3) the model is unified — a single decoder handling all output types. Compositional models avoid interference by routing different output types to different specialized decoders, so the backbone LLM only ever predicts text (or continuous conditioning features). Unified models accept the interference as the price of end-to-end simplicity.

This diagnosis has direct practical implications that the paper draws out: it explains why models like Janus (cited in Table 3) use separate encoders for understanding (SigLIP continuous features) and generation (VQGAN discrete tokens) — not because it's theoretically elegant, but because it avoids forcing a single tokenization strategy to serve both purposes, thereby reducing interference. It also motivates the paper's identification of modality synergy as a key open question: "whether tasks from different modalities can provide mutual assistance" (Section 6.2). The current evidence points to interference dominating, but the paper leaves open the possibility that with better training recipes, multimodal training could produce positive transfer — the unified model learning representations that are better for both text and images than either would be alone. The existence of the interference diagnosis, framed clearly as a structural challenge rather than an implementation bug, gives the field a clear target for future work.

Innovation 4: The Unresolved Quantization vs. Diffusion Debate as a Central Organizing Tension

The paper's most forward-looking contribution is its identification of the quantization-based autoregression vs. autoregressive diffusion debate as a live, unresolved tension that will shape the next generation of multimodal NTP models (Section 3.3, elaborated in Section 6.4).

The tension is this: quantization-based methods (Chameleon, Emu3, Show-o) convert continuous data into discrete tokens via VQVAE, then train with standard cross-entropy — the same objective as text. This is conceptually clean, directly leverages LLM training infrastructure, and enables efficient deployment (FlashAttention, vLLM). But it introduces quantization artifacts and forces 2D data into a 1D generation order that contradicts its natural structure. Diffusion-based methods (Transfusion, MAR, Fluid) incorporate diffusion losses into autoregressive training, predicting continuous latents rather than discrete tokens. This produces higher-quality generations (the claim of these works, cited in Section 3.3.2) but introduces training complexity — diffusion requires multiple denoising steps, increasing computational cost.

Prior work and prior surveys treated these as separate research threads — "autoregressive image generation" vs. "diffusion models" — without recognizing that within the NTP framework, they are competing answers to the same question: how to handle continuous modalities in a paradigm designed for discrete symbols. The paper's taxonomy makes this explicit by placing both under "Unified Model" architectures (Section 3.3) and noting that the debate is unsettled: "while many diffusion-based AR methods claim better generation quality compared to quantization method, Emu3 significantly outperforms diffusion baselines like SDXL using a quantization-based AR approach" (Section 3.3.2).

What makes this identification a genuine innovation is the paper's refusal to pick a winner. It instead provides the evaluation framework for the debate — what dimensions should we compare on? (generation quality, training efficiency, deployment efficiency, compatibility with understanding tasks) — and notes that "it is not concluded yet which modeling method has superior performance than another currently" (Section 3.3.2). This is intellectually honest and practically useful: it prevents the field from prematurely converging on one approach and directs attention to the empirical comparisons that need to be done.

The broader significance of this tension extends beyond images. The paper's Section 6.4 ("MMNTP as Universal Interfaces") asks: "how to define an effective generative training objective that suits the Next Token Prediction manner for various modalities, given that data from different modalities possess distinct structures." The quantization-vs-diffusion debate for images is a specific instance of a general problem: for any new modality (robot actions, molecular structures, protein sequences), should we discretize and use cross-entropy, or keep it continuous and use regression/diffusion? The taxonomy provides no answer, but it provides the vocabulary for asking the question systematically. This is what makes it a conceptual innovation rather than a solved problem: it marks the boundary of current knowledge and frames the research agenda going forward.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper surveys evaluation across many benchmarks, but its primary quantitative evidence for the competitive performance of NTP-based models comes from aggregating published results on standard vision and language benchmarks. For understanding, the paper highlights VQAv2 (Goyal et al., 2017) with its test-dev/test-std splits containing open-ended questions about images, and MMMU (Yue et al., 2023), a benchmark of 11,550 college-level questions spanning 183 subfields across 6 disciplines that requires expert-level multimodal reasoning. For generation, the paper highlights ImageNet (Russakovsky et al., 2015), the standard 1,000-class image generation benchmark evaluated by FID against training set statistics, and GenEval (Ghosh et al., 2023), a benchmark for compositional text-to-image generation that measures object-focused alignment between text prompts and generated images. These four benchmarks are not the only ones covered — the paper catalogs dozens of benchmarks in Table 7 — but they are the ones presented with quantitative performance comparisons in Figures 17 and 18 that demonstrate the NTP paradigm's empirical strength.

• Base model(s). The paper does not train or evaluate new models; it aggregates and visualizes published results from dozens of existing multimodal models to demonstrate trends. The models plotted in Figures 17 and 18 span a wide range of architectures and scales: early vision-language models (MCB, N2NMN, UNITER), encoder-decoder and encoder-only multimodal architectures (ViLBERT, Oscar, ALBEF, VLMo, BEiT-3), compositional NTP-based models (Flamingo, BLIP-2, InstructBLIP, LLaVA, LLaVA-1.5, Qwen-VL, SPHINX, Emu2, InternVL, DeepSeek-VL), unified NTP-based models (Chameleon, Show-o, Transfusion, Emu3, Janus), and closed-API models (GPT-4V, GPT-4o, Claude 3, Claude 3.5 Sonnet, Gemini 1.0, Gemini 1.5). For generation, the plotted models include both NTP-based systems (VQGAN+Transformer, MaskGIT, MAGVIT-v2, RQ-Transformer, VAR, LlamaGen, Chameleon, Show-o, Transfusion, Emu3, Janus) and non-NTP baselines (StyleGAN-XL, DiT-XL/2, Stable Diffusion v1/2/3, DALL-E 2/3, GigaGAN). The paper's selection principle is comprehensiveness: including all well-known models that have reported results on these benchmarks, regardless of architecture, to demonstrate that NTP-based models achieve state-of-the-art or competitive performance across the board.

• Metrics. For understanding tasks, the primary metric is accuracy (%) — the fraction of test questions for which the model's answer matches the ground truth, with exact-match grading. For VQAv2, the specific metric is VQA accuracy on the test-dev or test-std split using the standard VQA evaluation script. For MMMU, accuracy is computed over the 11,550-question benchmark across all 183 subfields, with the paper plotting only a subset of representative models. For generation tasks, the paper uses two complementary metrics: Fréchet Inception Distance (FID) for ImageNet generation — the Fréchet distance between Inception-v3 feature distributions of generated and training images, with lower FID indicating higher quality and diversity — and GenEval Score for compositional text-to-image generation, which measures the average object-level alignment between text prompts and generated images across multiple compositional axes (e.g., counting, color binding, spatial relationships).

• Baselines. The paper does not designate a single baseline; rather, the color-coding in Figures 17 and 18 serves as the comparative framework. Colored markers (blue, green, orange) denote models that use NTP-based architectures. Gray markers denote alternative architectural frameworks — this includes encoder-only vision models (ViT), GANs (StyleGAN), diffusion models (Stable Diffusion, DiT, SiT), encoder-decoder language models (T5, Whisper), and non-autoregressive multimodal architectures (VLMo, ALBEF). The baseline for comparison is therefore the collection of non-NTP approaches that represent the state-of-the-art at each point in time, against which NTP-based models can be assessed for competitive parity or superiority.

• Generation budget / compute accounting. The paper does not control for compute budget in its cross-model comparisons (Figures 17, 18). The plotted results are simply the best published numbers for each model under whatever training recipe, model size, and inference budget the original authors used. This is a significant limitation for causal claims about architecture superiority — a model with higher accuracy may benefit from larger scale, more training data, or better hyperparameters rather than from the NTP objective per se. The paper acknowledges this implicitly by not making strong causal claims about NTP being "better" than alternatives; it argues only that NTP-based models "achieve competitive or superior performance" (Section 5.2), which is a weaker and more defensible claim. The one exception is the generation comparisons in Figure 18: for ImageNet, FID comparisons are somewhat more controlled because the benchmark is standardized (256×256 or 512×512 ImageNet-1K generation), and for GenEval, the evaluation protocol is fixed across models, though training data and compute still vary.

• Cross-validation / statistical protocol. None is reported. The paper's quantitative evidence consists of point estimates — single accuracy or FID numbers per model per benchmark — drawn from published papers, without error bars, confidence intervals, or significance tests. This is standard for survey papers that aggregate published results, but it means that small differences between models (e.g., 2–3% on VQAv2) should not be interpreted as statistically reliable — they may reflect variance from training randomness, hyperparameter tuning, or evaluation protocol differences. The paper does not discuss this limitation.

Main Quantitative Results

Since this is a survey paper that aggregates and visualizes published results rather than conducting new experiments, the "main quantitative results" are the performance trends revealed by comparing NTP-based models against non-NTP baselines across benchmarks and over time. The paper organizes these into two figures: Figure 17 for multimodal understanding and Figure 18 for multimodal generation.

Multimodal Understanding: NTP-Based Models Achieve Parity or Superiority Across Benchmarks

VQAv2 Performance Trajectory (Figure 17, left panel). The left panel plots VQAv2 test accuracy over time (2016–2024) for approximately 25 open-source models, color-coded by architecture type. The visual narrative is one of steady improvement from ~55% accuracy (2016 state-of-the-art with MCB, an early attention-based VQA model) to ~85% by late 2023, with NTP-based models (colored markers) dominating the upper envelope of performance from approximately mid-2022 onward.

Key comparison points from the figure:

• Compositional NTP models achieve the highest single-model scores. Models like LLaVA-1.5, CogVLM, Qwen-VL, and SPHINX cluster in the 80–85% range in 2023. The figure does not provide exact numbers, but the visual placement indicates these are the top-performing open-source models on VQAv2 at the time of the survey.
• NTP-based models appear earlier than non-NTP models in the high-performance regime. Flamingo (2022, NTP-based with Perceiver Resampler) reaches approximately 80%, substantially ahead of contemporaneous non-NTP models. BLIP-2 (2023, NTP-based with Q-Former) and PaLI (2023, encoder-decoder) similarly occupy the upper right of the plot.
• Non-NTP architectures (gray markers) are not absent from high performance. BEiT-3 (2023) and VLMo (2022), which are encoder-only or non-autoregressive multimodal architectures, reach approximately 78–80% accuracy, indicating that NTP is not strictly necessary for strong VQA performance — but the NTP-based compositional models ultimately pull ahead.
• The trend line supports the paper's central narrative: NTP-based models are not just competitive; they define the frontier of open-source VQA performance by the end of the surveyed period. The figure caption explicitly states that "colored representations signify the employment of the next token prediction architecture, whereas gray depictions denote alternative architectural frameworks."

MMMU Performance Comparison (Figure 17, right panel). The right panel provides a snapshot comparison on the more challenging MMMU benchmark (college-level multimodal reasoning), plotting open-source LMMs and closed-API models in approximate chronological order of release (2023–2024).

Key observations:

• Models span a wide range: Open-source models cluster in the 30–50% accuracy range, with MiniGPT-4 at the low end (~30%), and InternVL2 and Qwen2-VL reaching approximately 50–55% at the high end for open-source. Closed-API models (GPT-4o, GPT-4V, Claude 3.5 Sonnet, Gemini 1.5) achieve 60–70%+, with GPT-4o reaching approximately 70%.
• Compositional NTP models (LLaVA, LLaVA-1.6, Qwen-VL-MAX, InternVL, InternVL2, DeepSeek-VL) occupy the upper range of open-source performance, with InternVL2 and Qwen2-VL achieving results competitive with earlier-generation closed-API models (Claude 3, Gemini 1.0).
• Unified NTP models (Emu2) appear but do not dominate: Emu2 scores approximately 40%, below the top compositional models, consistent with the paper's observation (Section 3.4) that unified models currently trail compositional models in understanding tasks at equivalent compute.
• The pattern reinforces the compositional advantage: The top-performing open-source models on MMMU are all compositional (strong external encoder + LLM backbone), suggesting that specialized encoders and focused alignment training currently outperform end-to-end unified training for complex reasoning tasks that require deep integration of visual and linguistic knowledge.

The paper's takeaway from Figure 17 is that NTP-based multimodal models have achieved performance parity with, and in many cases superiority over, non-NTP architectures on standard multimodal understanding benchmarks, with the caveat that the best understanding results currently come from compositional rather than unified NTP models.

Multimodal Generation: NTP Models Compete Directly with Diffusion Models on Established Benchmarks

ImageNet Generation (Figure 18, left panel). The left panel plots ImageNet 256×256 or 512×512 FID (Fréchet Inception Distance, lower is better) over time (2021–2024) for open-source generative models.

Key observations:

• Both NTP-based (colored) and non-NTP (gray) models improve dramatically over the 2021–2024 period. The FID scale ranges from approximately 7 (earliest data points: VQGAN+Transformer, RQ-Transformer, MaskGIT) down to ~2.0–1.5 by mid-2024 with the best models (VAR, MaskBit, SiT-XL/2+REPA).
• The top-performing models at the right edge of the plot are a mix of architectures. VAR (a quantization-based autoregressive model using next-scale prediction, colored) achieves approximately 1.7 FID, among the best results plotted. MAGVIT-v2 (a masked generative video transformer extended to images, colored) achieves roughly 1.5–1.6 FID. MaskBit (embedding-free quantization, colored) reaches approximately 1.5 FID. These are competitive with the best non-NTP models: SiT-XL/2+REPA (a diffusion transformer with representation alignment, gray) and DiMR-G (a diffusion model, gray) achieve similarly low FIDs.
• Diffusion models (DiT-XL/2, diffusion variants at the bottom left and middle of the plot) span a wide range, from FID ~6 down to ~3, while NTP-based models similarly span from ~7 down to ~1.5, indicating that neither paradigm has a decisive advantage — the best implementations of both achieve similar generation quality.
• Pure quantization-based autoregressive models (VQGAN+Transformer, RQ-Transformer, LlamaGen) improve but are not universally dominant. LlamaGen reaches approximately 2.5 FID, competitive but not surpassing diffusion baselines. The key improvement comes from improved tokenizers (MAGVIT-v2 with LFQ, VAR with multi-scale quantization) and improved generation orders (next-scale prediction rather than raster scan), supporting the paper's thesis that tokenization and generation order are first-class design axes.

GenEval Compositional Text-to-Image Generation (Figure 18, right panel). The right panel plots GenEval Score (a text-to-image alignment metric) over time (2021–2024).

Key observations:

• DALL-E 3 (closed-API, NTP-based with diffusion upsampling) achieves the highest GenEval score of approximately 0.75, notably ahead of all other models plotted. Stable Diffusion 3 (gray, diffusion-based) achieves roughly 0.70, and DALL-E 2 (NTP-based) achieves approximately 0.65.
• Open-source NTP models occupy the middle-to-upper range. Emu3 reaches approximately 0.65, competitive with DALL-E 2. LlamaGen achieves roughly 0.55–0.60. Chameleon, Show-o, and Transfusion cluster in the 0.45–0.55 range. Janus reaches approximately 0.55.
• The comparison validates that NTP-based generation can be competitive for compositional text-to-image tasks, where object-level alignment between text and generated images requires models to correctly bind attributes (color, count, spatial position) to objects — a task that challenges the 1D autoregressive bias of standard NTP. The fact that Emu3 (quantization-based NTP) matches DALL-E 2 on GenEval suggests that improved tokenization and scale can compensate for the raster-scan inductive bias.
• The presence of diffusion baselines (Stable Diffusion 1/2/3, IF-XL) achieving high scores (~0.60–0.70) indicates that diffusion remains a strong competitor, particularly for compositional generation where the iterative denoising process naturally handles global consistency.

The paper's takeaway from Figure 18 is that NTP-based generative models are competitive with diffusion models on both image quality (FID) and compositional alignment (GenEval) , with the top NTP models achieving results in the same range as the best diffusion models. This supports the survey's claim that NTP is a viable unifying paradigm for generation as well as understanding, even though diffusion models "remain the de facto model in visual generation applications" (Section 6.3).

Ablation Studies and Robustness Checks

This survey paper does not conduct new ablation studies. Its evidence comes from aggregating and taxonomizing published work. However, the paper's structural organization serves an analogous function to ablation analysis: by categorizing models along specific axes (discrete vs. continuous tokenization, compositional vs. unified architecture, discrete vs. continuous token prediction) and comparing performance within each category, the taxonomy reveals which design choices correlate with better outcomes.

Discrete vs. continuous tokenization for understanding: The performance data aggregated across models in Figure 17 shows that models using continuous tokenization (CLIP-based encoders + alignment layers, the dominant approach in LLaVA, Qwen-VL, InstructBLIP, InternVL) achieve the highest understanding scores. Models using discrete tokenization for understanding (Unified-IO, Chameleon) do not appear at the top of the MMMU chart, consistent with the paper's observation that continuous tokens offer "less information loss and improved data representation compared to discrete tokens" (Section 2.4). This is not a controlled ablation — models differ in scale, training data, and architecture — but the consistent pattern across dozens of models supports the taxonomy's diagnosis that continuous tokenization is currently preferred for pure understanding tasks.

Compositional vs. unified architecture for understanding: As noted in the quantitative results, compositional models dominate the upper range of MMMU performance, while unified models (Emu2, Chameleon) achieve lower scores. This supports the paper's claim (Section 3.4) that unified models "lead to sub-optimal performance given the same amount of computation" compared to compositional models, at least for complex reasoning tasks. However, this is not a controlled experiment — unified models are typically trained with different data mixtures (interleaved text-image rather than instruction-following data) and different objectives (joint text + image token prediction), which confounds the architectural comparison.

Quantization-based vs. diffusion-based autoregression for generation: The ImageNet FID plot (Figure 18, left) shows that quantization-based NTP models (VAR, MAGVIT-v2, MaskBit) and diffusion-based NTP/unified models (Transfusion, MAR not plotted) achieve comparable performance at the high end, with VAR reaching ~1.7 FID and MaskBit reaching ~1.5. However, these are not controlled comparisons — they use different tokenizers, different model scales, and different training datasets. The paper explicitly notes that the debate is unresolved (Section 3.3.2), and the aggregated results support this agnosticism: both approaches can achieve state-of-the-art generation quality, and the differences between specific implementations appear larger than the differences between paradigms.

Token efficiency: The paper's taxonomy identifies token count as a key design axis, but the aggregated results do not provide a controlled study of the token count vs. performance tradeoff. TiTok's claim of matching performance with 32 tokens vs. 256 tokens for standard VQGAN is cited (Section 2.2.1) but not quantitatively validated against other models in the figures. This is a gap — the paper's conceptual framework makes strong claims about token efficiency mattering, but the empirical evidence for how it matters is anecdotal rather than systematic.

Interleaved vs. paired pretraining data: Section 6.1 notes that models trained on interleaved image-text data (MMC4, OBELISC) do not "provide a significant advantage over models trained solely on pair-wise data, such as LLaVA and Qwen2-VL." This is a notable negative result surfaced by the survey: despite the theoretical appeal of interleaved pretraining (matching the distribution of real-world multimodal documents), the downstream benefits are not yet evident in benchmark performance. The paper does not quantify this claim with a figure or table — it cites Chameleon (2024), Deem (2024), and VILA-U (2024) as examples without providing side-by-side accuracy numbers. This is a limitation of the survey format: the claim is drawn from reading the literature, but the quantitative evidence is not presented for the reader to evaluate.

Modality interference in unified training: Section 6.2 discusses evidence that joint text + image token training causes optimization difficulties (gradient norm explosion requiring QK-Norm in Chameleon; language capability deterioration when adding more modalities). This is a critical negative result for the unified model project, but again, the paper presents it as a literature finding without quantitative evidence in the main figures. The specific claim that "learning to predict text tokens has no benefit for predicting multimodal tokens and sometimes leads to strong conflict" (Section 4.1.1, citing Zhang et al., 2023) is important for the survey's diagnosis of challenges but is not empirically validated within the survey itself — the reader must consult the cited paper for the experimental evidence.

Critical Assessment

The paper's primary claims, as established in the Executive Summary, are taxonomic and organizational rather than empirical. The paper claims that (1) NTP provides a unifying framework for multimodal learning, (2) tokenization is the foundational design axis that determines downstream capabilities, (3) compositional and unified architectures trade off training efficiency against deployment simplicity and multimodal intelligence, and (4) modality interference, scaling uncertainty, and the quantization-vs-diffusion debate are the central open challenges. The quantitative evidence in Section 5 (Figures 17–18) supports a more specific claim woven through the taxonomy: that NTP-based models achieve state-of-the-art or competitive results on both understanding and generation benchmarks across modalities. We assess this claim directly.

Claim: NTP-based models achieve competitive or superior performance on multimodal benchmarks. The evidence in Figures 17 and 18 supports the weaker form of this claim — competitive performance — but does not support the stronger form — general superiority. NTP-based models appear among the top performers on every benchmark plotted, but they do not dominate uniformly. On VQAv2, NTP-based compositional models define the open-source frontier. On MMMU, the top open-source models are NTP-based but with a substantial gap to closed-API systems whose architectures are not fully known. On ImageNet FID, the best NTP-based models (VAR, MAGVIT-v2, MaskBit) are in the top cluster alongside diffusion models — but diffusion models (SiT-XL/2+REPA, diffusion variants) also occupy that cluster. On GenEval, DALL-E 3 (NTP-based at generation time) leads, but Stable Diffusion 3 (diffusion-based) is close behind. The pattern is convergence at the frontier rather than NTP dominance: the best implementations of both paradigms achieve similar performance, suggesting that implementation quality and scale may matter more than the NTP-vs-non-NTP distinction.

What the experiments do not show. The aggregated results cannot distinguish between several competing explanations for why NTP-based models perform well. Do they benefit from the NTP objective per se, or do they benefit from transfer learning from powerful pretrained LLMs (which happen to be autoregressive)? If the same model architecture were trained with a different objective (masked modeling, diffusion) at equivalent scale, would it match or exceed NTP performance? The figures cannot answer this because they compare models that differ along many dimensions simultaneously — architecture, pretraining data, model scale, training recipe, and objective. A controlled study that isolates the NTP objective while holding other factors constant would be needed to make causal claims about NTP's contribution to performance.

Specific weaknesses in the experimental evidence:

• No compute-controlled comparisons. The most scientifically rigorous way to compare NTP against alternative training objectives would be to fix the total training FLOPs, fix the model architecture, fix the training data, and vary only the training objective (NTP vs. masked modeling vs. diffusion). No such comparison exists in the paper (and likely not in the surveyed literature, given the cost of such experiments). The claim that NTP-based models are "competitive" is therefore a claim about engineering outcomes, not about fundamental algorithmic efficiency — it could be that NTP is less compute-efficient but benefits from larger-scale investment (more FLOPs, better engineering) due to its LLM heritage.

• Benchmark coverage is skewed toward English, natural images, and standard tasks. All four quantitative comparison figures use benchmarks in English (VQAv2, MMMU) or with natural-image distribution (ImageNet, GenEval). The paper's taxonomy covers audio, video, robotics, protein design, and multilingual data, but the quantitative performance evidence is restricted to vision-language understanding and text-to-image generation. The claim that NTP unifies multimodal learning across all modalities is supported conceptually by the taxonomy but not empirically by the figures in Section 5 — there are no audio generation quality comparisons, no video understanding accuracy plots, no robotics task success rates. This is a significant gap: the empirical evidence covers only a subset of the modalities and tasks that the taxonomy claims to unify.

• Publication bias affects the aggregated comparisons. The models plotted in Figures 17 and 18 are those that achieved sufficiently strong results to be published at top venues. Models that attempted NTP-based multimodal training and failed (negative results) are systematically absent. If many NTP-based multimodal attempts fail due to modality interference or training instability but these failures are not published, the aggregated results will overstate NTP's effectiveness. The paper's acknowledgment of modality interference and negative results (Section 4.1.1, 6.2) partially mitigates this concern by documenting known failures, but the quantitative plots cannot correct for publication bias.

• Temporal confounding. The models at the right edge of the GenEval plot (2024) benefit from years of accumulated engineering improvements — better tokenizers, larger datasets, improved training recipes — that earlier models did not have. Comparing a 2024 NTP model (Emu3) against a 2022 diffusion model (Stable Diffusion 1) is not a clean paradigm comparison. The paper does not explicitly address this, but the temporal trend lines in Figure 18 show that both NTP and non-NTP models improve over time at roughly comparable rates, suggesting that cross-paradigm comparisons are confounded by general progress.

• The survey format precludes statistical rigor. No error bars, significance tests, or multiple-run averages are possible when aggregating published point estimates. A difference of 2% on VQAv2 between two models could arise from different evaluation protocols (VQA test-dev vs. test-std), different training data, or random seed variance. The paper does not discuss these sources of noise, implicitly treating published numbers as ground truth.

Missing experiments that would strengthen the survey's claims. A rigorous evaluation of the NTP paradigm would ideally include: (1) controlled comparisons on a held-out benchmark where NTP and non-NTP models are matched for total FLOPs, training data, and architecture depth/width; (2) ablation studies on the same model trained with different tokenization strategies (discrete vs. continuous) to quantify the representation-reconstruction tradeoff; (3) measurement of the scaling exponent for multimodal NTP models (how does performance improve as model size and data increase?) to compare against known scaling laws for text-only LLMs; (4) audio and video generation quality comparisons on standardized benchmarks (e.g., VBench for video, objective speech quality metrics for audio) to extend the empirical coverage beyond vision-language and text-to-image. These are substantial research projects — far beyond the scope of a survey — but their absence means the paper's claims about NTP's unifying power are taxonomically grounded but empirically incomplete. The taxonomy is the contribution; the quantitative evidence demonstrates that the taxonomy is plausible and not contradicted by existing results, which is the appropriate standard for a survey paper but falls short of the standard for an empirical methods paper.

6. Limitations and Trade-offs

The Taxonomy Is Empirically Grounded for Vision-Language, Not for the Full Modality Spectrum

The assumption or constraint. The paper constructs a five-axis taxonomy that claims to organize "the entire multimodal learning landscape" across vision, audio, video, robotics, protein design, and molecular chemistry. However, the quantitative evidence for NTP's effectiveness (Figures 17–18, Section 5.2) is drawn almost entirely from vision-language understanding benchmarks (VQAv2, MMMU) and text-to-image generation benchmarks (ImageNet FID, GenEval). The paper provides no performance comparisons — aggregate plots, controlled studies, or even curated tables — for audio understanding, audio generation, video generation, video understanding, robotics action prediction, or any of the "more modalities" discussed in Section 2.3.4. The taxonomy's claim to universality is supported by conceptual argument and citation, not by empirical validation within the survey.

The paper explicitly acknowledges this scope limitation in its own coverage but does not flag it as a limitation of the taxonomy itself. Section 2.3.2 (audio) lists models and tokenizers but provides no quantitative evidence that the NTP paradigm produces competitive results on standardized audio benchmarks. Section 2.3.3 (video) similarly catalogs approaches without performance quantification. Table 3 lists dozens of models across modalities, but no corresponding evaluation table demonstrates their effectiveness.

The consequence. A practitioner working on audio generation or video understanding who reads this survey hoping to determine whether NTP-based approaches outperform or even match domain-specific alternatives (e.g., HiFi-GAN for speech synthesis, TimeSformer for video understanding) will find no quantitative guidance. The taxonomy organizes the design space conceptually, but does not answer the question: should I actually use NTP for my modality, or is it a worse choice than specialized methods?

This matters because different modalities have different structural properties that may interact unfavorably with autoregressive modeling. Audio generation with NTP requires compressing raw waveforms (24,000+ samples per second) into discrete tokens (50–75 per second), introducing quantization artifacts that may be more perceptually salient for audio (where human hearing is sensitive to phase and temporal fine structure) than for images. Video generation requires modeling coherent motion across frames, and the raster-scan order that works reasonably well for static images may produce severe temporal inconsistency when applied frame-by-frame. The survey's taxonomy predicts these challenges (Section 2.3 flags them), but does not quantify their severity.

What evidence exists in the paper. None in the form of modality-specific performance comparisons. The paper's only quantitative evidence is in Figures 17–18, which are exclusively vision-language and text-to-image. Section 5.2 discusses evaluation broadly and Table 7 lists benchmarks covering video, audio, and embodied AI, but no results are reported for NTP models on these benchmarks. The section on "Performance Comparisons on Generation Tasks" (Figure 18) plots only ImageNet FID and GenEval — there is no corresponding plot for audio generation quality (e.g., MOS scores, MCD, WER for TTS) or video generation quality (e.g., VBench dimensions). The section on "Performance Comparisons on Understanding Tasks" (Figure 17) covers only VQAv2 and MMMU — there is no plot for audio understanding (e.g., Speech Commands, LibriSpeech WER) or video understanding (e.g., MVBench, ActivityNet).

Mitigation status. The paper does not attempt to fill this gap and does not present it as a limitation to be addressed. It is acknowledged indirectly by the survey's scope — it is a survey of the NTP paradigm, not an empirical benchmark paper — but the gap between the taxonomy's ambition (unifying all modalities) and its empirical grounding (vision-language only) is a structural weakness that limits the survey's practical utility for non-vision modalities. Future work could extend the quantitative analysis to audio, video, and robotics benchmarks with the same level of detail.

---

Difficulty Estimation Cost Is Absent from the Taxonomy, and the Paper Provides No Guidance on a Universal Difficulty Framework

The assumption or constraint. The survey's taxonomy organizes multimodal NTP systems by how they tokenize, architect, train, and evaluate — but it does not address how to allocate inference-time compute adaptively based on input difficulty. This is a notable absence given that the problem of "when to stop generating" and "how many tokens to allocate per modality" is structurally central to multimodal NTP: images naturally require more tokens than short text descriptions; long videos require more tokens than short clips; audio generation may require dynamically varying token budgets depending on the desired output length and quality.

The paper discusses efficiency as an open challenge (Section 6.3) but frames it entirely in terms of training and inference system efficiency — GPU utilization, sequence parallelism, token pruning, and model compression — not in terms of adaptive compute allocation or difficulty estimation. The question of whether an easy image-understanding query needs 256 visual tokens or could be answered with 32, or whether a simple TTS request needs all 12 layers of residual VQ or only the first 4, is never raised.

The consequence. Without a framework for difficulty-adaptive compute allocation, the survey implicitly assumes that the same tokenization budget and model depth are appropriate for all inputs. This is almost certainly suboptimal in practice. A practitioner deploying an NTP-based multimodal system — say, a unified model that handles both visual question answering and text-to-image generation — has no guidance from the taxonomy on how to vary the inference budget per query. Should a "describe this simple icon" query receive the same number of visual tokens as "analyze this medical scan and identify anomalies"? Should a text-to-image request for "a red circle on white background" cost the same number of generation steps as "a detailed Renaissance painting of a crowded marketplace at sunset"? The taxonomy treats tokenization as a fixed preprocessing step and generation as a fixed-budget autoregressive process, but real deployments require dynamic allocation.

This absence is particularly notable given that the paper surveys models (Unified-IO, Chameleon, Emu3) that support variable-length multimodal outputs — an image might be generated with 256, 512, or 1024 tokens depending on desired quality. The taxonomy describes the tokenizers that enable this (Section 2.3) but never discusses how to decide how many tokens to allocate. The efficiency discussion in Section 6.3 focuses on making fixed-budget generation faster (FlashAttention, vLLM, token pruning), not on making the budget itself adaptive.

What evidence exists in the paper. The paper provides no experiments or analyses on adaptive compute allocation. The challenge section (6.3) frames the efficiency problem as: "can we use less tokens to represent the multimodal input while maintaining the performance?" This is about tokenization compression, not about dynamic allocation — the question is whether a better tokenizer can produce fewer tokens for all inputs, not whether the system can use fewer tokens for easy inputs and more for hard inputs. The paper's discussion of training efficiency (DistTrain, Optimus) similarly focuses on static optimization of a fixed-compute pipeline.

Mitigation status. Not addressed. The paper does not identify difficulty-adaptive allocation as a gap, and the taxonomy provides no vocabulary or framework for discussing it. This is a genuine limitation of the survey's scope: it documents how NTP systems work when compute is fixed, but the question of when and how to vary compute per input is central to deployment. Future work could extend the taxonomy with an axis covering adaptive compute strategies — difficulty estimation, early exit, variable-length generation, and dynamic token allocation — that would make the framework more practically actionable.

---

The Taxonomy Does Not Separate Scaling Laws Evidence from Scaling Laws Assumptions, Leaving the Central Promise of NTP Unification Empirically Unvalidated

The assumption or constraint. The paper's Section 6.1 explicitly identifies scaling behavior as a major open challenge, noting that "the return of scaling MMNTP models remains largely under-explored." It cites Aghajanyan et al. (2023) as showing that "data from distinct modalities exhibit varying scaling behaviors" and states that "the reasons behind these differences and their impact on the performance of downstream tasks across various modalities remain unclear." Further, the paper questions "whether MMNTP models can develop similar emergent abilities in downstream tasks as LLMs do when the training is scaled up."

This is a remarkably candid acknowledgment that the central promise of NTP unification — that the same scaling principles that made LLMs powerful will apply to multimodal models — is unproven. The entire premise of building ever-larger unified multimodal models (Chameleon, Emu3, GPT-4o) trained with next-token prediction on massive interleaved multimodal data rests on the assumption that the scaling laws observed for text-only autoregressive models will transfer. The paper's own analysis suggests this assumption may not hold, or may hold differently for different modalities.

The consequence. The taxonomy is organized as if scaling is a natural, expected continuation of current trends — the "Unified Model" architecture is presented as the logical endpoint (Section 3.3), and efficiency challenges are framed as engineering problems to be solved (Section 6.3). But if multimodal scaling laws are fundamentally different from text scaling laws — if, for example, image token prediction scales with a much lower exponent than text token prediction, or if adding modalities to a unified model causes diminishing or negative returns beyond a certain scale — then the taxonomy's implicit prediction (that unified NTP models will improve with scale and eventually dominate) may be wrong. The paper does not seriously engage with this possibility.

The practical consequence is that a practitioner deciding whether to invest in training a large unified multimodal model vs. maintaining separate specialized systems has no empirical basis for predicting which will improve faster with additional compute. The paper's performance comparisons (Figures 17–18) show NTP models achieving competitive results at current scales, but they are snapshots, not scaling trend lines. If diffusion models for image generation scale more efficiently than autoregressive models (i.e., achieve lower FID per FLOP), then the current competitive parity may not persist at larger scales, and the taxonomy's architectural recommendations would need revision.

What evidence exists in the paper. The paper surfaces the relevant citations but does not itself provide any scaling analysis. The challenge is stated in Section 6.1:

"the return of scaling MMNTP models remains largely under-explored. Some studies have explored or hypothesized the scaling behaviors of MMNTP models in the self-supervised training manner... However, the reasons behind these differences and their impact on the performance of downstream tasks across various modalities remain unclear."

This is an accurate summary of the field but also an admission that the survey cannot validate its own architectural recommendations with scaling evidence. The performance plots (Figures 17–18) show models at different scales but do not plot performance against model size, training FLOPs, or data volume — so the reader cannot assess whether NTP models are on a steeper or shallower scaling curve than alternatives.

Mitigation status. The paper identifies this as an open challenge (Section 6.1) and calls for future work, which is appropriate for a survey. However, the taxonomy itself could have incorporated a "scaling behavior" axis — categorizing models and tokenization strategies by what is known (or unknown) about their scaling properties — which would have made the knowledge gap more actionable. As written, a reader might conclude that unified NTP models are the recommended direction without realizing that this recommendation rests on unvalidated scaling assumptions. The paper's transparency about the uncertainty is a strength, but the taxonomy's structure does not reflect this uncertainty.

---

The Compositional-vs-Unified Comparison Is Asymmetric, with Insufficient Evidence to Support the Claimed Unified Model Advantages

The assumption or constraint. Section 3.4 presents a three-axis comparison between compositional and unified models: general multimodal intelligence (unified models preferred, closer to "foundation model" status), training efficiency (compositional models preferred), and deployment efficiency (unified models preferred, due to LLM infrastructure compatibility). The paper states that unified models are "closer to a multimodal foundation model due to its end-to-end nature" and that they "may hold more potential than their compositional counterparts, as they rely on a single NTP training objective, making them easier to scale."

This framing implies that unified models are the aspirational target and that their current performance disadvantages are temporary — a consequence of insufficient training, not of fundamental limitations. However, the evidence base for this optimism is thin. The paper acknowledges that unified models "lead to sub-optimal performance given the same amount of computation" (Section 3.4, citing Show-o) and that modality interference makes joint text-image training unstable (Section 6.2). The only claimed advantage with clear empirical support is deployment efficiency — unified models benefit from FlashAttention and vLLM because they are architecturally identical to standard LLMs.

The consequence. The compositional-vs-unified comparison is asymmetric in its standard of evidence. The disadvantages of unified models are demonstrated empirically: they underperform compositional models on understanding benchmarks (Figure 17, where Emu2 scores ~40% on MMMU vs. ~55% for InternVL2), and they require training stabilization techniques (QK-Norm in Chameleon) to avoid gradient norm explosion. The advantages of unified models — "more potential," "closer to a foundation model," "easier to scale" — are stated as assertions without quantitative backing. No experiment shows that unified models improve faster with scale than compositional models; no experiment shows that unified models develop emergent cross-modal abilities; no experiment isolates the "end-to-end" benefit from confounding factors like training data quality or model scale.

This matters because the compositional-vs-unified choice is one of the most consequential architectural decisions in multimodal NTP. A practitioner reading the survey might reasonably conclude that unified models are the future direction worth investing in, based on the taxonomy's framing. But the empirical case for this direction is aspirational, not demonstrated. Compositional models currently perform better on understanding tasks (Figure 17), are easier to train (no modality interference, Section 6.2), and can leverage the best available encoders and decoders from their respective modalities — advantages that the taxonomy acknowledges but then implicitly de-emphasizes in its forward-looking statements.

What evidence exists in the paper. The paper's own data supports compositional superiority for understanding: in Figure 17 (right), the top open-source MMMU performers are all compositional (InternVL2, Qwen2-VL, LLaVA-1.6, DeepSeek-VL). Unified-IO and Emu2 (unified models) score in the 30–40% range, roughly 15–20 points lower. For generation, the evidence is more mixed. In Figure 18 (left), the top ImageNet FID models include both unified quantization-based approaches (VAR, MAGVIT-v2) and diffusion models (SiT-XL/2+REPA). In Figure 18 (right), GenEval scores show DALL-E 3 (closed, likely unified) leading, but Chameleon and Show-o (open unified models) score 0.45–0.55, below the 0.60–0.70 range of compositional models using external diffusion decoders.

The paper also acknowledges in Section 3.4 that the unified model advantage for deployment efficiency is specific to quantization-based unified models — diffusion-based unified models (Transfusion, MAR) do not share this advantage because they cannot use standard LLM inference optimizations directly.

Mitigation status. The paper does not present this as a limitation to be resolved. Section 3.4 frames the comparison as a balanced tradeoff rather than as an open question with asymmetric evidence. The forward-looking challenge section (6.2) discusses modality interference in unified models as a problem to be solved but does not reconsider whether the unified model vision is fundamentally sound. The paper would be strengthened by a more cautious framing that explicitly separates demonstrated advantages (deployment efficiency) from hypothesized advantages (better scaling, emergent abilities, "foundation model" status) and calls for controlled scaling experiments to resolve the uncertainty.

---

The Survey Provides No Systematic Treatment of Failure Modes, Degenerate Outputs, or Safety Concerns

The assumption or constraint. The taxonomy covers the forward pipeline — tokenization, architecture, training, evaluation — but does not include an axis for failure modes, degenerate behaviors, or safety-critical concerns that are specific to multimodal NTP systems. The paper mentions hallucinations briefly in the context of chain-of-thought reasoning (Section 4.4.2: "mitigating language hallucinations... exacerbated due to the lack of necessary and fine-grained visual context") and cites a hallucination benchmark (HallusionBench, Table 7), but there is no systematic discussion of what can go wrong when an autoregressive model generates multimodal tokens.

This is a significant gap because NTP-based multimodal generation introduces failure modes that have no analog in text-only LLMs. A text-only LLM that hallucinates produces incorrect text, which a human reader may detect. An NTP-based multimodal model that produces incorrect image tokens can generate visually plausible but semantically wrong images — a generated medical scan that looks realistic but shows non-existent pathology, a generated diagram that appears authoritative but encodes incorrect information, a generated video that depicts events that never occurred. The autoregressive nature of NTP means that early token errors propagate forward — a mistake in the first few generated image patches can cascade into a globally incoherent image — and there is no natural "recovery" mechanism within standard autoregressive decoding.

The paper also does not discuss content safety for multimodal generation. Text-to-image systems (DALL-E, Stable Diffusion) have well-documented issues with generating harmful, biased, or copyright-infringing content. When these capabilities are integrated into a unified NTP model that can both understand and generate, the safety surface expands: the model could generate harmful images in response to seemingly innocuous text prompts, or it could complete harmful multimodal sequences (text + image + text) in undesirable ways. The paper's discussion of preference alignment (Section 4.3.3–4.3.4) covers human preference optimization for understanding (reducing hallucinations, grounding answers in visual context) and briefly mentions diffusion model alignment, but does not address the safety implications of unified multimodal generation.

The consequence. A practitioner deploying an NTP-based multimodal generation system has no framework from this survey for anticipating, detecting, or mitigating failure modes. The taxonomy does not provide a vocabulary for discussing when autoregressive image generation produces artifacts (e.g., VQGAN grid artifacts, global incoherence from raster-scan bias), when audio generation produces unnatural prosody or phoneme errors, when video generation produces temporal inconsistencies, or when the model generates harmful multimodal content. The evaluation benchmarks listed in Table 7 include HallusionBench (for hallucination detection in understanding) and MMCBench (for self-consistency), but no benchmark specifically designed to evaluate generation quality degradation modes or content safety.

What evidence exists in the paper. Almost none systematically. The paper mentions that VQ-based generation introduces "quantization artifacts" and that raster-scan generation "contradicts the intrinsic nature of 2D images" (Section 3.3.2), but these are qualitative observations, not quantitative analyses. The paper mentions that revision models in other work (referenced generically, not a specific multimodal NTP system) sometimes convert correct answers to incorrect ones (noted as a problem in the reference example about compute-optimal inference, not in the multimodal NTP survey itself). There is no section or table documenting known failure modes of multimodal NTP, their prevalence, or their severity.

Mitigation status. Not addressed. The paper's challenge section (6) covers scaling, interference, efficiency, and universal interfaces, but not robustness, safety, or failure mode analysis. This is a defensible scope choice for a taxonomy focused on the NTP paradigm's architecture and training, but for a survey that aims to "aid researchers in their exploration of multimodal intelligence" and that celebrates the unification of understanding and generation in production systems (GPT-4o, Gemini), the absence of a safety and failure mode axis is a consequential gap. Future editions of the survey could add a section covering known failure modes (VQ artifacts, autoregressive drift in long sequences, modality-specific hallucinations in generation, content safety), evaluation benchmarks for generation quality degradation, and mitigation strategies (classifier-free guidance, rejection sampling, safety filters) to provide a more complete picture of what practitioners should expect when deploying these systems.

---

The Survey Assumes but Does Not Validate That Multimodal NTP Training Is Compatible with Text-Only LLM Capabilities

The assumption or constraint. A foundational premise of the multimodal NTP paradigm — particularly for compositional models like LLaVA and Qwen-VL — is that starting from a pretrained text-only LLM and extending it to handle multimodal inputs will preserve the LLM's original language capabilities while adding multimodal understanding (and optionally, generation). The paper documents this as the standard recipe: "pioneering research... has demonstrated that LLMs can be easily adapted to process multimodal inputs such as images and videos" (Section 1), and the training pipeline in Section 4 freezes the LLM backbone during modality alignment to "enhance stability" (Section 4.2.1).

However, the paper also surfaces contradictory evidence. Section 6.2 notes that "the language capability often deteriorates when adding more modalities" — the model's text-only performance degrades after multimodal training. This is a form of catastrophic forgetting or negative transfer: the gradients from multimodal alignment or instruction tuning overwrite some of the language knowledge acquired during pretraining. Section 4.1.1 goes further, citing Zhang et al. (2023) as showing that "learning to predict text tokens has no benefit for predicting multimodal tokens and sometimes leads to strong conflict," suggesting that the negative transfer may be structural rather than incidental.

The consequence. If multimodal adaptation predictably degrades text-only capabilities, then the "unified multimodal model" vision has a fundamental tradeoff that the taxonomy does not adequately surface: you cannot have a single model that is simultaneously best-in-class at text-only tasks and multimodal tasks. At present, the best text-only LLMs (GPT-4, Claude 3.5, Llama 3) are not the same models as the best multimodal LMMs (GPT-4V, GPT-4o may or may not use the same weights; the paper notes that these are closed-source and their exact architectures are unknown). If text capability degradation is an inherent consequence of multimodal training, then the field may converge on hybrid systems — separate text-only and multimodal models, perhaps sharing some components — rather than fully unified models.

This has direct practical implications: an organization that deploys an NTP-based multimodal model for customer-facing applications may find that its text-only performance (e.g., pure language reasoning, code generation, text summarization) is worse than a same-scale text-only LLM. The survey provides no framework for quantifying this degradation or deciding whether the multimodal capabilities are worth the text-only regression.

What evidence exists in the paper. The paper acknowledges the phenomenon (Section 6.2) and cites Zhang et al. (2023) for gradient conflict evidence (Section 4.1.1), but provides no quantitative measurement of the degradation. Figure 17 compares multimodal models on VQAv2 and MMMU but does not show text-only benchmark performance (e.g., MMLU, GSM8K, HumanEval) for the same models, so the reader cannot assess the text-only capability tradeoff. The training data discussion (Section 4.3.1) notes that "mixing the multi-modal instruction dataset with text-only query-response pairs is also shown to be effective for improving the instruction following ability," suggesting that text-only data during multimodal training partially mitigates the degradation, but the paper does not quantify how effective this mitigation is or whether it fully closes the gap.

Mitigation status. The paper identifies modality interference as a challenge (Section 6.2) and suggests that "the root causes of these optimization difficulties in MMNTP models remain largely unexplored." This is an appropriate call for future work, but it also means that the survey provides no actionable guidance for practitioners: should they expect text capability degradation? How severe? Can it be mitigated by data mixing ratios, architectural choices (compositional vs. unified), or training schedules? The taxonomy currently organizes models by their multimodal capabilities but does not help answer the question "what am I giving up in text-only performance to get these multimodal capabilities?" — a question that is central to any deployment decision.

7. Implications and Future Directions

How This Work Changes the Landscape

This survey does not introduce a new method or empirical result, so it does not cause the kind of paradigm shift associated with a breakthrough model or algorithm. Rather, its contribution is a reorganization of existing knowledge that makes a fragmented research landscape legible as a single design space. This is less dramatic than discovering a new scaling law or proposing a new architecture, but its impact on how the field thinks about multimodal learning is potentially substantial.

The central conceptual shift: from modality-specific engineering to cross-modal design principles. Before this survey, a researcher building a vision-language model, an audio generation system, and a video understanding pipeline would draw on separate literatures with separate vocabularies. After reading this survey, that same researcher can recognize that all three systems face the same structural question at each stage of the pipeline: How do I tokenize continuous data for a discrete sequence model? What attention mask pattern lets the model process conditioning information efficiently? Do I train with cross-entropy over a discrete vocabulary or regression toward continuous targets? The taxonomy makes these cross-modal invariants explicit and nameable, which means that solutions developed for one modality can be systematically tried on others. This is a refinement, not a revolution, but it is a refinement that accelerates the rate at which the field can transfer insights across modality boundaries.

The paper's reframing also reconciles a latent tension in how the field has discussed NTP unification. On one hand, the "everything is tokens" narrative suggests that NTP is a universal learning algorithm that should work equally well for any modality. On the other hand, practitioners know that image generation with autoregressive models produces artifacts that diffusion models avoid, that audio tokenization requires modality-specific inductive biases (semantic vs. acoustic disentanglement), and that video generation strains the 1D causal order assumption. The paper resolves this tension not by taking a side but by making the tension itself the object of study. It recasts the question from "does NTP work for everything?" to "what are the design axes you must navigate when applying NTP to a new modality, and what are the known failure modes at each axis?" This is a more honest and more useful framing. It explains why certain design choices (VQ for generation, continuous encoders for understanding) recur across modalities without claiming that they are optimal in any absolute sense. It also explains why unified models — which try to do everything with one tokenization strategy and one training objective — currently underperform compositional models: they force a single resolution to tradeoffs that are best resolved differently for different tasks.

Directions that become more attractive. The survey makes tokenization research a central priority. By elevating tokenization to the first substantive technical discussion and documenting the representation-reconstruction tradeoff, the paper implicitly argues that improvements in tokenizer design — particularly for discrete tokens used in generation — will have disproportionate impact because they propagate benefits forward through the entire pipeline. This is a corrective to the field's recent emphasis on scaling backbone models and training data, suggesting that architectural innovation at the tokenization layer may be a more efficient path to better multimodal models than simply training larger transformers on more tokens.

Similarly, the paper makes controlled comparisons between quantization-based and diffusion-based autoregressive generation an urgent research priority. The survey's refusal to pick a winner in this debate, combined with its documentation of contradictory evidence (Emu3 outperforms diffusion baselines; Transfusion claims diffusion outperforms quantization), creates a clear mandate for a rigorous, FLOPs-matched comparison that isolates the generation paradigm from confounding factors like tokenizer quality, training data, and model scale.

The paper also makes modality interference research newly salient by naming it as a structural challenge rather than an implementation inconvenience. Prior work acknowledged that joint text-image training is difficult, but this was often treated as an engineering problem to be solved by better hyperparameters or training recipes. By framing interference as a fundamental consequence of multi-task learning with heterogeneous output spaces, the survey elevates it to a research problem in its own right — one that may require architectural innovations (separate output heads, modality-specific normalization, gradient surgery) rather than just better tuning.

Directions that become less attractive. The survey implicitly argues against naive scaling of unified models as the default research direction. The evidence assembled — that unified models underperform compositional models at current scales (Figure 17), that modality interference makes joint training unstable (Section 6.2), that scaling laws for multimodal NTP are unknown (Section 6.1), and that interleaved pretraining data does not yet provide clear advantages over paired data (Section 6.1) — suggests that simply training larger Chameleon-like models on more data is unlikely to be the most efficient path forward. The paper does not say this explicitly, but the cumulative weight of its documented challenges argues for architectural and algorithmic innovation over brute-force scaling. Researchers who might have defaulted to "train a bigger unified model on more interleaved data" after reading a less critical survey will find here a detailed map of the obstacles that naive scaling does not address.

Follow-Up Research This Work Enables

A FLOPs-matched comparison of quantization-based autoregression vs. autoregressive diffusion for image generation. The paper identifies this as an unresolved debate (Section 3.3.2) with contradictory claims: Emu3 (quantization-based) outperforms SDXL in their comparisons, while Transfusion and MAR (diffusion-based) claim their approach yields better generation quality than quantization. Neither comparison controls for total training FLOPs, model scale, or training data quality. A rigorous follow-up would fix a transformer architecture (depth, width, attention pattern), fix the training dataset (e.g., a standardized subset of LAION or DataComp), and train two variants — one with a VQ tokenizer and cross-entropy loss, one with continuous latents and diffusion loss — at multiple scales (e.g., 100M, 500M, 2B parameters) to compare scaling exponents. The key measurements would be generation quality (FID, GenEval Score) vs. total training FLOPs, generalization to out-of-distribution prompts, and compatibility with understanding tasks when the model is extended for multimodal input. This would resolve whether the quantization-diffusion performance crossover depends on scale, and whether claims of superiority are confounded by differences in engineering maturity rather than algorithmic efficiency.

Measuring and mitigating text capability degradation during multimodal alignment. The paper surfaces evidence that adding multimodal capabilities to a pretrained LLM degrades its text-only performance (Section 6.2: "language capability often deteriorates when adding more modalities") but provides no quantification. A systematic study would take a strong text-only LLM (e.g., Llama-3-8B), measure its performance on a standardized text-only benchmark suite (MMLU, GSM8K, HumanEval, HellaSwag, etc.), then apply standard multimodal alignment training (CLIP encoder + Q-Former or MLP projection, trained on image-caption pairs) and re-measure text-only performance. The experiment would sweep: data mixing ratios (what fraction of training tokens should remain text-only to preserve language ability?), architectural choices (does freezing the LLM during alignment eliminate degradation? Does fine-tuning all parameters cause more forgetting than adapter-based methods?), and training objectives (does continuous token prediction cause less interference than discrete token prediction?). The study would also measure whether degradation is uniform across text tasks or concentrated in specific capabilities (e.g., reasoning vs. factual recall). This would transform the qualitative observation of "language capability deterioration" into a quantitative, actionable characterization that practitioners can use when deciding whether and how to add multimodal capabilities to their LLMs.

Scaling laws for multimodal NTP with controlled tokenization and data mixtures. Section 6.1 identifies the absence of multimodal scaling laws as a critical gap: "the return of scaling MMNTP models remains largely under-explored" and "data from distinct modalities exhibit varying scaling behaviors." A principled study would train autoregressive transformer models on interleaved sequences of text tokens and discrete image tokens (from a fixed VQ tokenizer, e.g., MAGVIT-v2), varying model size (e.g., 100M–7B parameters), total training tokens, and the ratio of text to image tokens, all trained with the same cross-entropy objective. The key measurements: how does the loss on text tokens scale compared to the loss on image tokens? Is there a compute-optimal text-to-image token ratio analogous to the compute-optimal parameter-to-data ratio in Chinchilla? Does the optimal ratio change with scale? A negative result — e.g., finding that image token loss scales with a much lower exponent than text token loss, and that increasing model size beyond a certain point yields diminishing returns for image generation — would be equally valuable, as it would place an upper bound on the benefits of scaling unified models and motivate alternative approaches (specialized decoders, hybrid architectures).

Dynamic difficulty-adaptive token allocation for multimodal NTP inference. The survey documents that multimodal inputs vary dramatically in their information density (Section 2.3: "images are contain with a myriad of perceptual details," but a simple icon carries far less information than a complex photograph) and that token count trades off against computational cost (quadratic in sequence length). Yet no existing NTP system, as surveyed, allocates a variable token budget per input based on estimated difficulty. A follow-up system would build on a standard compositional understanding model (e.g., LLaVA-1.5) and add a lightweight difficulty predictor — perhaps a small MLP trained on the CLIP encoder's pooled representation to predict whether a query can be answered correctly with 16, 64, or 256 visual tokens. The system would use the predictor to route each query to the minimal token budget needed, measuring the accuracy-vs.-FLOPs tradeoff curve compared to fixed-budget baselines. A key measurement: what fraction of queries can be answered correctly with substantially fewer tokens than the default (e.g., can 50% of VQAv2 questions be answered with 32 tokens without accuracy loss?), and what is the wall-clock latency reduction? This would extend the taxonomy's static tokenization discussion into the dynamic allocation regime that is practically necessary for deployment.

Does multimodal NTP training produce emergent cross-modal abilities, and at what scale? The paper questions in Section 6.1 "whether MMNTP models can develop similar emergent abilities in downstream tasks as LLMs do when the training is scaled up." This is a specific, testable hypothesis. A targeted study would train a unified NTP model (tokenizing both text and images with a shared discrete vocabulary) at progressively larger scales (e.g., 100M, 1B, 10B parameters, matched training data), then evaluate on tasks that require genuine cross-modal integration — not just separate understanding and generation tasks, but tasks like "given an image of a scene and a text description of a modification, generate the modified image" or "given a visual puzzle, explain the solution in text." The key measurement is whether the model's performance on these cross-modal tasks improves disproportionately at some scale threshold — the signature of emergence — or whether it improves smoothly, following the same power-law scaling as unimodal task performance. A finding of smooth scaling without emergence would constrain expectations about the benefits of further scaling and suggest that architectural innovations (explicit cross-modal attention, modular routing) may be necessary for integrated multimodal reasoning.

Systematic cataloging and mitigation of autoregressive generation artifacts across modalities. The paper identifies qualitatively that VQ artifacts, raster-scan bias, and temporal inconsistency affect generation quality (Sections 2.3, 3.3.2), but provides no systematic taxonomy of these failure modes — their causes, their severity across modalities, and the effectiveness of proposed mitigations (next-scale prediction, random-order generation, parallel prediction, lookahead search). A follow-up survey or benchmark paper would define a set of standardized artifact categories (e.g., for images: global incoherence, local texture artifacts, object duplication/omission, attribute binding errors; for audio: unnatural prosody, phoneme substitution, speaker identity drift; for video: frame flicker, object disappearance, motion inconsistency) and evaluate existing NTP generation models on a held-out test set with human annotation or automated metrics for each category. The output would be a per-model, per-artifact-type breakdown that maps specific design choices (tokenizer type, generation order, training objective) to specific failure signatures. This would transform the current qualitative discussion of "generation quality" into an actionable diagnostic framework that tells practitioners which artifacts to expect from which design decisions.

Practical Applications and Downstream Use Cases

On-device unified multimodal assistants with quantization-based NTP inference. The paper's comparison in Section 3.4 establishes that quantization-based unified models have a decisive deployment efficiency advantage: they are architecturally identical to standard LLMs and can leverage FlashAttention, vLLM, and other LLM inference optimizations directly. This makes them substantially more practical for on-device deployment than compositional models, which require running multiple separate networks (encoder, backbone, decoder) with different optimization characteristics. A mobile or edge-device assistant that handles both text queries and image understanding, and potentially lightweight image generation, could be built as a single unified NTP model (e.g., a small Chameleon-like architecture) that shares all parameters and computation. The deployment efficiency argument is grounded in the paper's observation that compositional models require "running multiple separate networks... with potentially different architectures and optimization characteristics" (Section 3.4), which on constrained hardware translates directly to memory pressure and latency. For applications where latency is critical and the quality gap between unified and compositional models is acceptable (e.g., consumer chat applications, real-time photo description for accessibility), this is a practical deployment path that the survey validates.

Multimodal data processing pipelines with modality-specific tokenizer selection. The survey's taxonomy — particularly Table 2 (tokenizers by modality, training method, and generation/understanding capability) and the explicit discussion of the representation-reconstruction tradeoff (Section 2.1.2) — provides a practical decision framework for teams building production multimodal systems. An organization building a system that indexes and searches a large corpus of documents containing interleaved text, images, and audio can use the taxonomy to select tokenizers: CLIP-based continuous encoders for understanding tasks (document retrieval, visual question answering about document content), VQGAN-based discrete tokenizers if the system also needs to regenerate or edit images, and Whisper/SoundStream for audio depending on whether understanding or generation is required. The key practical insight from the paper is that no single tokenizer is optimal for all tasks, and the taxonomy provides the vocabulary for reasoning about which to use when. A production system might maintain separate understanding and generation tokenizers, sharing a single backbone LLM via the compositional architecture pattern, which the paper's Section 3.2 explicitly documents with examples (Emu1/2, LLaMA-Omni).

Cost-optimized multimodal model serving with the compositional-vs-unified tradeoff. The paper's three-axis comparison of compositional and unified models (Section 3.4: general intelligence, training efficiency, deployment efficiency) provides a framework for server-side deployment decisions. A cloud service that serves multimodal queries at scale faces a cost tradeoff: compositional models achieve higher accuracy on understanding benchmarks (Figure 17: compositional models dominate MMMU performance) but require more complex serving infrastructure (separate encoder and decoder networks). Unified models simplify serving (single model, standard LLM inference stack) but currently achieve lower accuracy at equivalent compute. The practical application is a tiered serving architecture: use a unified model as the default for high-throughput, latency-sensitive queries (where the deployment efficiency advantage matters most), and a compositional model as a fallback for queries that the unified model handles poorly (detected via confidence scoring or explicit user dissatisfaction). The paper's documentation of the performance gap (unified Emu2 at ~40% MMMU vs. compositional InternVL2 at ~55%) provides a concrete sense of the accuracy tradeoff involved, enabling data-driven decisions about which queries to route where.

Training data curation informed by the paper's finding that interleaved data does not yet outperform paired data. Section 6.1 notes that "models trained on interleaved text-image data do not provide a significant advantage over models trained solely on pairwise data, such as LLaVA and Qwen2-VL." This is a practically important finding for teams allocating data curation resources. Collecting and filtering large-scale interleaved multimodal data (webpages, textbooks, PDFs) is substantially more expensive than collecting paired image-caption data (which can be scraped from alt-text, social media, and stock photo sites). If interleaved pretraining does not currently yield downstream benefits, a practical team might rationally choose to invest those resources in (a) higher-quality paired data, (b) synthetic instruction tuning data (generated via GPT-4V as described in Section 4.3), or (c) better tokenizers rather than larger interleaved corpora. The paper does not argue that interleaved data is useless — it identifies it as an open research question — but the current evidence it surfaces suggests that the marginal return on interleaved data collection is lower than the field might have assumed. Teams can use this information to prioritize data investments that have stronger empirical support.