Architecture2024~129 minarXiv:2412.08635

Multimodal Latent Language Modeling with Next-Token Diffusion

score 6Yutao Sun, Hangbo Bao, Wenhui Wang +5Microsoft Research↑ 49Dec 11, 2024cs.CL

🎯 Pitch

LatentLM achieves state-of-the-art text-to-speech using 10× fewer decoding steps than discrete-token models by directly generating continuous latent vectors autoregressively. It also unifies image generation and understanding within a single causal Transformer, outperforming diffusion-only and discrete-token baselines when scaling up training tokens.

1. Executive Summary

This paper introduces Latent Language Modeling (LatentLM), a unified framework that seamlessly handles both discrete data—such as text and code—and continuous data—such as images, audio, and speech—using causal Transformers. The core technical contribution is next-token diffusion, which autoregressively generates continuous latent vectors one by one, where each vector is produced by a lightweight diffusion head conditioned on the Transformer's hidden state (operationalized by replacing the standard softmax classification head with a denoising process at continuous-token positions). To make latent representations suitable for autoregressive generation, the authors also propose σ-VAE, a variational autoencoder variant that enforces a fixed variance in the latent space to prevent variance collapse and increase robustness to exposure bias during sequential decoding (operationalized by sampling a scalar variance per example from a zero-mean Gaussian rather than learning per-channel variances). Across three modalities, LatentLM demonstrates substantial gains: on ImageNet class-conditional generation, it outperforms Diffusion Transformers in scaling behavior; as a multimodal LLM backbone, it achieves favorable text-to-image FID and image-to-text perplexity compared to both Transfusion and vector-quantized models when training tokens are scaled up; and in text-to-speech synthesis, it surpasses the state-of-the-art VALL-E 2 in speaker similarity and robustness while requiring 10× fewer autoregressive decoding steps (operating at a frame rate of 7.5–15 versus 75 for neural codec baselines), establishing that continuous latent autoregression can match or exceed discrete-token approaches while achieving substantially higher compression ratios only when the latent variance is explicitly controlled for the autoregressive regime.

2. Context and Motivation

The Core Problem: Multimodal Models Need a Unified Generative Framework

The fundamental problem this paper addresses is architectural fragmentation in multimodal generative modeling. Most deployed systems that handle both discrete data (text, code) and continuous data (images, audio, video, robot actions) are pipeline-based: they compose independent modules—an automatic speech recognition system feeds text into a language model, which feeds text into a text-to-image model, and so on. The authors identify two specific pathologies of this approach (Section 1):

End-to-end optimization is impossible. Each module is trained independently, so errors cascade and cannot be corrected across module boundaries during training. If the speech recognizer produces a slightly wrong transcription, the downstream language model has no way to signal that error back to the ASR module.
Information loss between modules. Pipeline components typically communicate via text prompts—a severely lossy bottleneck. When an image understanding module reduces a complex visual scene to a text description for a downstream reasoner, the compression discards information that might be critical for the task (spatial relationships, visual nuance, precise quantities).

The paper's ambition is to build a single model that can natively perceive and generate any combination of modalities—text, images, speech, video, robot actions—under one training objective, one architecture, and one inference procedure. This is not merely an engineering convenience; it enables fundamentally new capabilities like multimodal in-context learning, self-reflection on generated outputs without re-encoding, and joint reasoning across modalities that share internal representations.

Why This Matters: Beyond the Engineering Argument

The significance of unified multimodal modeling extends beyond integration convenience. The authors argue (Introduction) that unification enables three classes of capabilities that are difficult or impossible with pipelines:

Multimodal reasoning with latent state tracking. A unified model can track intermediate reasoning states as latent vectors, enabling, for example, step-by-step planning where the model plots a trajectory on an input map image without ever converting to text. The latent vectors serve as a shared "working memory" that can flow between modalities.
Self-reflection and iterative refinement. Because the same model both generates and understands continuous data, it can inspect its own generated images or speech, detect errors, and revise them—all internally, without re-encoding through external modules. This closes the loop that pipeline systems leave open.
Cross-modal knowledge transfer. When speech and text are represented with similar tokenization granularity (enabled by the high compression ratios of continuous latent vectors—approaching 1:1 with BPE tokenization at 6400× compression, as reported in Table 5), knowledge learned from text corpora can transfer more naturally to speech tasks, and vice versa.

These capabilities are not hypothetical aspirations in the paper—they are directly enabled by the proposed architecture and motivate the design choices.

Three Prior Approaches, Three Fundamental Limitations

The paper identifies three main research strands that have attempted to handle continuous and discrete data in a single model, and argues each makes a significant compromise (Section 1):

Approach 1: Vector Quantization—"Treat Everything as Discrete Tokens"

Methods in this strand (VQ-VAE-based models such as DALL-E [RPG+21], VALL-E [WCW+23], Chameleon [Tea24], and LlamaGen [SJC+24]) quantize continuous inputs into discrete codes using a learned codebook, then process these codes with standard autoregressive Transformers alongside text tokens. The continuous data is reconstructed by a VQ-VAE decoder conditioned on the discrete codes.

Where this falls short (Section 1, Table 1, Table 5):

Lossy tokenization creates a restrictive bottleneck. Quantization is inherently lossy—the discrete codebook forces the model to choose the nearest neighbor, discarding fine-grained information. This manifests as degraded reconstruction quality. In speech, the paper shows (Table 5) that even the best discrete tokenizers (e.g., DAC at 75 frame rate) achieve worse reconstruction metrics (higher Mel Distance, lower PESQ) than continuous σ-VAE at comparable or higher compression ratios.
Low compression ratios force long sequences. Discrete tokenizers for high-fidelity data require many tokens per second of audio or per image patch. VALL-E 2 operates at a frame rate of 75 (75 autoregressive steps per second of speech). In images, LlamaGen-XXL (Table 1) compresses a 256×256 image into 256 discrete tokens. Long sequences quadratically increase the cost of Transformer attention, limiting scalability to high-resolution or long-duration data.
Saturation in scaling. Figure 8a shows that the VQ-MLLM (vector-quantized image tokenizer) baseline saturates as training tokens increase—its FID curve flattens earlier than LatentLM or Transfusion. This suggests the quantization bottleneck fundamentally limits how much the model can benefit from additional training data.

The continuous representations in LatentLM address all three: the σ-VAE is near-lossless (Table 5 shows reconstruction quality competitive with low-compression-ratio discrete codecs even at 6400× compression), the compression ratio is 10–80× higher, and Figure 8a shows FID continues improving with scaled training tokens.

Approach 2: Diffusion Unification—"Treat Everything as Continuous"

Methods in this strand (UniDiffuser [BNX+23b], CoDi [TYZ+23]) unify modalities by treating all data—including discrete text—as continuous and applying diffusion-based generative modeling.

Where this falls short:

Compromises discrete data modeling. Diffusion models are designed for continuous data and are fundamentally mismatched to discrete sequences. The paper does not provide extensive discrete-data evaluations against these methods, but the conceptual argument is clear: applying continuous noise to discrete tokens (words, code tokens) and learning to denoise them is a poor model of the categorical, next-token structure that makes language modeling effective. The Transfusion baseline (Table 3) demonstrates this indirectly—when Transfusion adds noise to images during training to perform diffusion, it degrades image-to-text understanding (VQAv2: 35.36 vs. LatentLM's 38.72), because the noisy image representations interfere with the model's ability to condition text generation on visual input.
Bidirectional attention restricts variable-length applications. Image-level diffusion models like DiT use bidirectional (non-causal) attention because denoising at pixel $i$ can depend on pixels both before and after. This makes them unsuitable for autoregressive, variable-length generation—a fundamental requirement for dialogue, streaming speech synthesis, and interactive applications. LatentLM's causal architecture naturally supports these.

Approach 3: Transfusion—"Share Weights but Not Objectives"

Transfusion [ZYB+24] is the most directly comparable prior work. It shares Transformer weights between text and image modalities but uses different objectives and attention patterns: next-token prediction with causal masking for text, and sequence-level diffusion with bidirectional attention for images.

Where this falls short (Section 3.2.2, Table 3, Figure 8):

Training-inference mismatch for images. During training, Transfusion adds noise to image tokens and learns to denoise them bidirectionally. During inference for image understanding tasks, the model receives clean images—creating a distribution shift that degrades performance. Table 3 shows this quantitatively: LatentLM achieves 38.72 on VQAv2 versus Transfusion's 35.36, and 54.5 CIDEr on MS-COCO captioning versus Transfusion's 43.4. The gap is attributed to LatentLM's consistent treatment of images during training and inference (no noise added at training time).
Bidirectional diffusion restricts context usage. Because Transfusion's image diffusion is non-causal, it cannot naturally incorporate variable-length context or perform streaming generation. The paper notes this but does not provide a direct empirical ablation—the limitation is architectural rather than performance-bound.
Different objectives create optimization conflicts. The model must simultaneously optimize a denoising loss (for image diffusion) and a cross-entropy loss (for text prediction), with different weightings and different noise schedules. While Transfusion shows this can work, the paper argues that having a single, consistent objective (autoregressive next-token prediction, with diffusion only as the decoding mechanism for continuous tokens) simplifies training and enables better knowledge sharing across modalities. Table 3's language modeling perplexity results (2.73 for LatentLM vs. 2.74 for Transfusion) show a small but consistent advantage.

How LatentLM Positions Itself

LatentLM is positioned as resolving all three compromises simultaneously (Section 1, Section 2):

Against VQ approaches: Use continuous latent vectors instead of discrete codes, maintaining high-fidelity reconstruction at much higher compression ratios (Table 5: σ-VAE at 6400× compression achieves Mel Distance 0.852 vs. Encodec at 10× compression achieving 0.823—better quality with 640× less data).
Against diffusion unification: Keep the language modeling paradigm (causal Transformer, next-token prediction) for both discrete and continuous data, using diffusion only as a decoding head for continuous tokens, not as the overarching training framework. This preserves the well-established effectiveness of autoregressive language modeling for discrete data while extending it naturally to continuous data.
Against Transfusion: Unify not just the model weights but also the objective (autoregressive next-token prediction/diffusion) and the attention pattern (causal everywhere). The paper's name—Latent Language Modeling—emphasizes this: "language modeling" signals the causal, next-token paradigm, and "latent" signals that continuous data is represented in a compressed latent space rather than as discrete tokens.

The architecture diagram (Figure 2) makes this positioning concrete: a single causal Transformer backbone processes all modalities, with the only difference being the head—a softmax classification head for discrete tokens and a diffusion head for continuous vectors. The training objective is a simple weighted sum of standard LM loss and diffusion loss, both computed token-by-token in a single forward pass.

The Variance Collapse Problem: A Previously Unrecognized Barrier

A key motivation that the paper identifies—and that prior work largely overlooked—is variance collapse in VAEs as a barrier to autoregressive generation (Section 2.3). The standard VAE objective (Equation 4) includes a KL divergence term that encourages the latent distribution to match a unit Gaussian prior. In practice, this often leads to posterior collapse where some latent dimensions have near-zero variance—the encoder learns to ignore them because the decoder can reconstruct well enough from the remaining dimensions, and the KL penalty discourages using them.

For non-autoregressive generation (e.g., latent diffusion models like LDM, where all latent vectors are generated simultaneously), this variance collapse is benign or even beneficial—less variance means more deterministic encoding, which can improve reconstruction. But for autoregressive generation, where each latent vector is sampled sequentially conditioned on previous vectors, low variance in the latent space is catastrophic. The reason (explained in Section 2.3 and empirically validated in Figure 6) is exposure bias: during inference, the model conditions on its own possibly-noisy predictions. If the latent space has small variance, small prediction errors can push the generated vector outside the distribution the VAE decoder expects, causing cascading quality degradation. Larger latent variance makes the decoder more robust to these errors because it's trained to handle a wider spread of inputs.

Figure 6 provides the empirical evidence: the "stars" (VAE tokenizers tuned for image-level diffusion models, which have small variance) produce dramatically worse FID scores under LatentLM compared to σ-VAE tokenizers with larger, explicitly controlled variance. Under DiT (non-autoregressive), the variance choice is largely irrelevant—all tokenizers perform similarly. But under LatentLM's autoregressive decoding, larger variance monotonically improves FID. This is the paper's key insight for why prior VAE tokenizers cannot simply be dropped into autoregressive models, and it motivates the design of σ-VAE (Equation 5) that enforces a per-example scalar variance $\sigma \sim \mathcal{N}(0, C_\sigma)$ .

The Broader Landscape: A Missing Scaling Law for Inference

The paper also implicitly positions itself in the context of the scaling laws literature, though it does not cite Hoffmann et al. (2022) directly. The image generation scaling experiments (Figure 4, Section 3.1.2) show that LatentLM's FID improves more favorably with model size than DiT—a finding analogous to how Chinchilla showed that data scaling matters as much as parameter scaling. Here, the "scaling law" is architectural: causal autoregressive generation with continuous latent vectors scales better than bidirectional diffusion because:

It enables key-value caching during autoregressive inference (Figure 7 shows 2.47–2.84× throughput improvements), making larger models practically deployable.
It avoids the computational cost of multiple full-model forward passes per image (DiT requires 20–50 denoising steps, each a full Transformer forward pass; LatentLM requires only one Transformer forward pass plus lightweight diffusion head steps).

The multimodal LLM results (Figure 8) extend this: as training tokens scale from ~50B to ~200B, LatentLM's text-to-image FID and image-to-text perplexity improve more consistently than baselines, suggesting the unified objective scales better with data than approaches that use different training regimes for different modalities.

In summary, the paper addresses a clear architectural gap—no existing approach successfully unifies continuous and discrete data generation in a single causal Transformer with a single objective—and identifies a previously underappreciated technical barrier (VAE variance collapse under autoregressive decoding) that explains why prior attempts at continuous latent autoregression have been limited. The motivation is both practical (deployment efficiency, simpler training) and capability-driven (enabling new forms of multimodal reasoning that pipelines cannot support).

3. Technical Approach

3.1 Reader Orientation

This paper develops LatentLM, a single causal Transformer that can generate both text (discrete tokens) and continuous data (images, audio, speech) using the same autoregressive "predict the next token" paradigm. The core problem it solves is architectural fragmentation: previous systems required separate models, objectives, and attention patterns for different modalities, but LatentLM uses one consistent autoregressive framework where the only difference is at the output—a softmax classification head for discrete tokens versus a lightweight diffusion head for continuous latent vectors.

3.2 Big-Picture Architecture (Diagram in Words)

The system has four major components connected in a pipeline:

σ-VAE Tokenizer — compresses raw continuous data (images, audio, speech) into compact latent vectors. It encodes input into a mean vector $\mu$ , injects a fixed-variance Gaussian noise $\sigma \cdot \epsilon$ to produce the latent vector $z = \mu + \sigma \cdot \epsilon$ , and decodes $z$ back to the original data space. The key innovation is that $\sigma$ is a scalar (same for all channels) sampled per example from $\mathcal{N}(0, C_\sigma)$ , preventing variance collapse.
Causal Transformer Backbone — a standard autoregressive Transformer with causal masking that processes sequences of interleaved discrete and continuous tokens. It takes as input a packed sequence $X_0 = [x_1, ..., x_N] \in \mathbb{R}^{N \times d}$ (where discrete tokens are looked up from an embedding table and continuous tokens are the σ-VAE latent vectors) and produces contextualized hidden states $[h_1, ..., h_N]$ at the output.
Dual-Decoding Heads — attached to each Transformer output state. For discrete token positions, a softmax classifier $\text{softmax}(h_i W_v)$ predicts the next token from a fixed vocabulary. For continuous vector positions, a lightweight diffusion head $\epsilon_\theta(x_i^t, t, h_i)$ iteratively denoises a random vector into the target latent vector, conditioned on the Transformer state $h_i$ .
σ-VAE Decoder — converts the generated latent vectors back into raw continuous data (images, audio waveforms). This is only used at the final output stage, not during the autoregressive generation loop.

Information flows as follows: raw continuous data → σ-VAE encoder → latent vectors → packed into sequence with discrete embeddings → causal Transformer processes all tokens autoregressively → at continuous positions, diffusion head generates each vector conditioned on hidden state → σ-VAE decoder reconstructs raw data. For discrete positions, the softmax head directly samples the next token.

3.3 Roadmap for the Deep Dive

First, the core generative mechanism: next-token diffusion (Section 2.1) — how diffusion is used as a decoding head rather than a full-generation framework, and why this matters for autoregressive modeling.
Second, the training and inference procedures (Section 2.2) — the unified objective, how loss is computed per-token, and the autoregressive decoding loop.
Third, σ-VAE and the variance collapse problem (Section 2.3) — the mathematical motivation, the design of per-example scalar variance, and why it is essential for autoregressive generation.
Fourth, the diffusion head architecture — the lightweight residual network with AdaLN-Zero conditioning, and why it needs to be efficient.
Fifth, the image generation instantiation (Section 3.1) — full configuration, scaling experiments, and how the system is instantiated for class-conditional generation.
Sixth, the multimodal LLM instantiation (Section 3.2) — how discrete and continuous generation are combined in a single sequence, training data mixture, and evaluation.
Seventh, the text-to-speech instantiation (Section 3.3) — the streaming-capable convolutional tokenizer, compression ratio analysis, and ablation studies.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an architectural innovation paper whose core idea is that continuous data can be autoregressively generated token-by-token using a causal Transformer, provided (a) the data is compressed into continuous latent vectors via a VAE with controlled variance, and (b) each continuous vector is decoded using a lightweight diffusion head rather than a vector-quantized classification head.

The Core Generative Mechanism: Next-Token Diffusion (Section 2.1)

The defining technical contribution of LatentLM is next-token diffusion: treating diffusion not as a full-sequence generative framework (as in image-level diffusion models like DiT) but as a per-token decoding mechanism within an otherwise standard autoregressive Transformer.

What makes this different from prior diffusion models. In a standard diffusion model like DiT [PX23], the entire image is generated simultaneously: the model takes a full grid of noisy latent vectors, performs multiple denoising steps (typically 20–50), and outputs all vectors at once. The Transformer processes the entire sequence with bidirectional attention, seeing all positions simultaneously. In next-token diffusion, each continuous vector is generated one at a time in autoregressive order: the Transformer processes the history (previous tokens) once, produces a single hidden state $h_i$ , and then the diffusion head uses $h_i$ as a conditioning signal to iteratively denoise the next continuous vector. The Transformer backbone is computed once per token during inference (reusing the key-value cache), while only the lightweight diffusion head runs multiple denoising steps per token.

This design is what enables the extreme efficiency gains reported in Figure 7: because the Transformer (which contains the vast majority of parameters) runs once per token rather than once per denoising step, the total FLOPs for generating an image are dramatically lower than for image-level diffusion models.

The mathematical formulation (DDPM variant). The paper uses denoising diffusion probabilistic models (DDPM) [HJA20] or flow matching [LCBH+22] as the design choice, with DDPM described in detail.

The forward process (noise addition) transforms a clean latent vector $x_i^0 = z_i$ (the σ-VAE encoding of a continuous data patch) into progressively noisier versions $x_i^t$ over $T$ steps:

$q(x_i^t|x_i^{t-1}) = \mathcal{N}(x_i^t; \sqrt{1 - \beta_t} \cdot x_i^{t-1}, \beta_t I)$

where $\beta_t \in (0, 1)$ is a per-step noise variance from a predefined schedule (the paper uses a cosine schedule in the image generation experiments, following [SH22]), $I$ is the identity covariance matrix, and $\mathcal{N}(\cdot; \mu, \Sigma)$ denotes a Gaussian distribution with mean $\mu$ and covariance $\Sigma$ .

What it computes: at each step $t$ , the current vector $x_i^{t-1}$ is scaled by $\sqrt{1-\beta_t}$ (slightly shrinking it toward zero) and then independent Gaussian noise with variance $\beta_t$ is added to each dimension. After $T$ steps, the vector approximates pure noise $\mathcal{N}(0, I)$ .

Why this form: the Markov chain structure with small-per-step noise allows the model to learn a simple denoising step (predicting the noise added in one step) rather than having to jump directly from pure noise to data. The schedule controls the tradeoff: small $\beta_t$ early in the process preserves structure, while larger $\beta_t$ later adds more destructive noise.

A crucial property for training efficiency is that we can sample $x_i^t$ directly from $x_i^0$ without iterating through all intermediate steps:

$x_i^t = \sqrt{\bar{\alpha}_t} \cdot x_i^0 + \sqrt{1 - \bar{\alpha}_t} \cdot \epsilon$

where $\bar{\alpha}_t = \prod_{i=1}^t (1 - \beta_i)$ is the cumulative product of survival probabilities (the fraction of original signal remaining after $t$ steps), and $\epsilon \sim \mathcal{N}(0, I)$ is a random Gaussian noise vector drawn independently for each training sample.

What it computes: the noisy vector at step $t$ is a simple interpolation between the clean vector $x_i^0$ and pure noise $\epsilon$ , with mixing coefficient $\sqrt{\bar{\alpha}_t}$ for the signal and $\sqrt{1-\bar{\alpha}_t}$ for the noise. When $t = 0$ , $\bar{\alpha}_0 = 1$ and we get the clean vector. When $t = T$ , $\bar{\alpha}_T \approx 0$ and we get pure noise.

Why this reparameterization matters: it allows the training loss to be computed by sampling a single timestep $t$ and noise vector $\epsilon$ per training example, constructing $x_i^t$ directly via the closed form, and asking the model to predict $\epsilon$ . Without this, training would require simulating the full $T$ -step Markov chain per example, which is computationally prohibitive.

The reverse process (denoising) is parameterized by a model $\epsilon_\theta(x_i^t, t, h_i)$ that predicts the noise $\epsilon$ that was added to produce $x_i^t$ , conditioned on the Transformer hidden state $h_i$ and the timestep $t$ :

$p_\theta(x_i^{t-1}|x_i^t, h_i) = \mathcal{N}(x_i^{t-1}; \frac{1}{\sqrt{1-\beta_t}}(x_i^t - \frac{\beta_t}{\sqrt{1-\bar{\alpha}_t}}\epsilon_\theta(x_i^t, t, h_i)), \tilde{\beta}_t I)$

where $\tilde{\beta}_t$ is a variance term derived from the noise schedule.

What it computes: given a noisy vector $x_i^t$ , the model predicts the noise component $\epsilon_\theta$ , subtracts a scaled version of this prediction from $x_i^t$ to recover an estimate of $x_i^{t-1}$ , and then adds a small amount of Gaussian noise (with variance $\tilde{\beta}_t$ ) to prevent the process from becoming deterministic.

The training loss is the mean squared error between the predicted noise and the actual noise:

$\mathcal{L}_{\text{Diff}}(x_i, h_i) = \mathbb{E}_{x_i, t, \epsilon} \left[ \| \epsilon - \epsilon_\theta(x_i^t, t, h_i) \|^2 \right]$

where $x_i$ is the ground-truth latent vector (from the σ-VAE encoder), $t \sim \text{Uniform}(1, T)$ is a randomly sampled timestep, $\epsilon \sim \mathcal{N}(0, I)$ is the sampled noise, and $x_i^t = \sqrt{\bar{\alpha}_t} \cdot x_i + \sqrt{1 - \bar{\alpha}_t} \cdot \epsilon$ is the constructed noisy vector.

What it computes: for each continuous token in the training sequence, the model receives the Transformer hidden state $h_i$ (which encodes all previous tokens in the sequence) and a noisy version of the target vector $x_i^t$ , and must predict the noise $\epsilon$ that was added. The loss is the squared L2 distance between the predicted and actual noise vectors. This is averaged over all continuous tokens and all sampled timesteps.

Why this form: predicting the noise (rather than directly predicting $x_i^0$ or $x_i^{t-1}$ ) is empirically more effective for diffusion models, as established in [HJA20]. The reason is that the noise prediction target has constant variance across timesteps (always $\mathcal{N}(0, I)$ ), whereas the signal prediction target $x_i^0$ has very different scales at different noise levels. Additionally, the L2 loss is the natural choice for Gaussian noise because it corresponds to maximum likelihood estimation under the assumption of isotropic Gaussian prediction error.

Inference procedure. During autoregressive generation, when the model needs to produce a continuous token at position $i$ :

The Transformer processes the history $x_{<i}$ and produces hidden state $h_i$ .
A vector of pure Gaussian noise $x_i^T \sim \mathcal{N}(0, I)$ is sampled as the starting point.
For $t = T, T-1, ..., 1$ : the diffusion head predicts $\epsilon_\theta(x_i^t, t, h_i)$ , and this prediction is used to compute $x_i^{t-1}$ according to the reverse process equation.
The final $x_i^0$ is the generated latent vector for position $i$ .
This vector becomes part of the history for generating position $i+1$ .

The paper uses DPM-Solver [LZB+22a, LZB+22b] to accelerate this process, reducing the number of denoising steps from the training-time $T = 1000$ (DDPM) to as few as 3–20 steps during inference. DPM-Solver is a fast ODE solver that exploits the semi-linear structure of the diffusion reverse process, enabling large step sizes without significant quality degradation.

Flow matching alternative. The paper mentions that flow matching [LCBH+22] can also be used as an alternative to DDPM. In flow matching, the forward process is a deterministic linear interpolation between data and noise (rather than stochastic diffusion), and the model learns to predict the velocity field that transports noise to data. The inference then integrates this ODE. The paper does not provide detailed comparisons between DDPM and flow matching, leaving this as an implementation flexibility rather than a core design choice.

The Diffusion Head Architecture

The diffusion head $\epsilon_\theta(\cdot)$ in Equation (3) is deliberately designed to be lightweight — a small neural network that processes one noisy vector at a time, conditioned on the Transformer hidden state. This is critical for efficiency because during inference, only this head runs multiple times per continuous token, while the heavy Transformer backbone runs once.

The architecture (Section 2.1, "Head Architecture") is a residual network incorporating:

Pre-RMSNorm [ZS19]: layer normalization applied before each sub-layer (feedforward network), rather than after. The paper uses RMSNorm throughout the entire model (both Transformer backbone and diffusion head), which normalizes by the root mean square of activations without learning a bias term.
Feedforward networks: the head contains multiple (typically 3–6) feedforward layers. Each layer applies a linear transformation, a non-linear activation (SwiGLU [Sha20, RZL17] is used in the Transformer backbone; the head likely uses similar activations), and another linear transformation.
AdaLN-Zero conditioning [PX23]: the head is conditioned on both the diffusion timestep $t$ and the Transformer hidden state $h_i$ through adaptive layer normalization. Specifically, the conditioning signals are used to predict scale and shift parameters for the layer normalization operations within the head, and the final residual connection is initialized to zero (hence "Zero") to ensure that at initialization, the head outputs zero — meaning the model initially predicts zero noise, which is the correct prediction when $t = 0$ (no noise added).

Why AdaLN-Zero: it provides a clean inductive bias for diffusion. At the start of training, when the model has no useful conditioning information, the zero-initialized output means the head predicts no noise regardless of the input, which is the correct behavior at $t=0$ . As training progresses, the head learns to modulate its predictions based on $t$ (more noise prediction at higher $t$ ) and $h_i$ (conditioning on the context). The alternative — standard concatenation-based conditioning — would require the model to learn from scratch that the output should depend on the conditioning signal, which is a harder optimization problem.

Efficiency rationale. The paper states (Section 2.2) that the diffusion head "is usually lightweight" and that "reusing the computation of the Transformer backbone improves training efficiency while introducing minimal overhead." In the image generation experiments (Table 1, configuration details in Appendix A), the LatentLM-L model has a Transformer with hidden size 1024, 32 layers, and FFN dimension 2730, plus a diffusion head with 6 layers. In the multimodal LLM experiments (Appendix D), the Transformer has 24 layers with hidden size 2048, and the diffusion head has 6 layers. In the TTS experiments (Appendix E), the Transformer has 24 layers with hidden size 1024, and the diffusion head has only 3 layers.

The key computational insight: during inference, the Transformer runs once per generated token (whether discrete or continuous), while the diffusion head runs $T_{\text{inf}}$ times per continuous token (where $T_{\text{inf}}$ is the number of inference denoising steps, typically 3–20). If the head is small relative to the backbone (e.g., 6 head layers vs. 32 backbone layers in LatentLM-L), the total FLOPs per continuous token are roughly $\text{FLOPs}_{\text{backbone}} + T_{\text{inf}} \times \text{FLOPs}_{\text{head}}$ , which is dominated by the backbone for reasonable $T_{\text{inf}}$ . This contrasts with image-level diffusion models like DiT, which require $T_{\text{inf}} \times \text{FLOPs}_{\text{backbone}}$ total FLOPs because the entire Transformer runs at each denoising step.

Model Training and Inference Procedures (Section 2.2)

Training objective. The model is trained end-to-end with a single, unified objective that combines the standard language modeling loss for discrete tokens and the diffusion loss for continuous tokens:

$\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{LM}} + \alpha \cdot \mathcal{L}_{\text{Diff}}$

where $\mathcal{L}_{\text{LM}}$ is the cross-entropy loss for discrete token prediction, $\mathcal{L}_{\text{Diff}}$ is the diffusion loss from Equation (3), and $\alpha$ is a hyperparameter controlling the relative weight of the continuous generation objective.

The discrete token loss is:

$\mathcal{L}_{\text{LM}} = -\sum_{x,i} \log P_d(x_i | x_{<i})$

where $x_i$ is a discrete token, $x_{<i}$ are all previous tokens in the sequence, and $P_d(x_i | x_{<i}) = \text{softmax}(h_i W_v)$ is the predicted probability from the softmax head (Equation 1). The sum is over all discrete token positions in all training sequences.

What it computes: the standard autoregressive language modeling objective — the model must predict each discrete token given all previous tokens, and the loss is the negative log-likelihood of the correct token. This is computed token-by-token with causal masking ensuring no information leakage from future tokens.

Why a weighted sum: the $\alpha$ hyperparameter balances the scale of the two losses. Diffusion losses are typically much smaller in magnitude than cross-entropy losses because they operate in a continuous vector space with L2 distance, while cross-entropy operates on probability distributions. The paper does not report the specific $\alpha$ value used — this is a missing detail, and the value likely depends on the modality and model scale.

Training efficiency technique. The paper uses a practical optimization: for each continuous token in a training batch, the model samples multiple diffusion timesteps (typically four, as stated in Section 2.2) and computes the diffusion loss at each timestep in a single forward pass. This works because the Transformer hidden state $h_i$ (which is the most expensive computation) is the same regardless of the timestep — only the diffusion head's computation varies with $t$ . By feeding the same $h_i$ through the lightweight diffusion head multiple times with different $t$ values, the model gets more diffusion training signal per Transformer forward pass, improving training efficiency.

Special tokens for head switching. The paper uses special tokens to indicate transitions between the softmax head and the diffusion head. <BOD> (Beginning of Diffusion) signals that the next token should be generated using the diffusion head, and <EOD> (End of Diffusion) signals a switch back to the language modeling head. This allows the model to handle sequences with interleaved discrete and continuous data (e.g., "Describe this image: <BOD> [image latent vectors] <EOD> What is shown in the image?").

Inference: autoregressive decoding loop. The generation process follows the standard autoregressive paradigm:

The model begins with a prompt sequence (which may contain both discrete and continuous tokens).
At each step, the Transformer processes the entire history so far (using key-value caching to avoid recomputing previous steps) and produces the hidden state for the next position.
If the next token is discrete (indicated by the absence of <BOD>), the softmax head computes $P_d(x_i | x_{<i})$ and a sampling algorithm (greedy, top-p, or temperature sampling) selects the token.
If the next token is continuous (indicated by <BOD>), the diffusion head runs the iterative denoising process conditioned on $h_i$ to produce the latent vector, which is then appended to the history.
The process continues until an end-of-sequence token is generated or a maximum length is reached.
After all latent vectors are generated, the σ-VAE decoder reconstructs the raw continuous data from the latent vectors.

Key efficiency property. The Transformer backbone is computed in a single pass per token during inference. The key-value cache from all previous positions is reused, so generating token $i$ requires only O(1) new Transformer computation (the attention and FFN for position $i$ ), not O( $i$ ). This is a fundamental advantage over image-level diffusion models, which require O( $N \cdot T_{\text{inf}}$ ) Transformer computations for an image with $N$ patches (one full forward pass per denoising step, no caching possible because the denoising is non-causal).

Latent Vector Representation: σ-VAE (Section 2.3)

The σ-VAE tokenizer is the component that enables continuous data to be represented as a sequence of latent vectors suitable for autoregressive generation. Its design is motivated by a specific failure mode of standard VAEs in autoregressive settings.

The variance collapse problem. Standard VAEs are trained with the objective:

$\text{maximize } \mathbb{E}_{q_\phi(z|x)}[\log p_\psi(x|z)] - D_{\text{KL}}[q_\phi(z|x) \| p(z)]$

where $q_\phi(z|x)$ is the encoder (parameterized by $\phi$ ), $p_\psi(x|z)$ is the decoder (parameterized by $\psi$ ), and $p(z) = \mathcal{N}(0, I)$ is the prior distribution. The encoder outputs parameters of a Gaussian distribution — typically a mean $\mu$ and a per-channel variance $\sigma^2$ — and the latent vector is sampled as $z = \mu + \sigma \odot \epsilon$ where $\epsilon \sim \mathcal{N}(0, I)$ .

The KL divergence term $D_{\text{KL}}[q_\phi(z|x) \| p(z)]$ encourages the encoder distribution to match the unit Gaussian prior. In practice, this often leads to posterior collapse: the encoder learns to set $\sigma \approx 0$ for many (or all) channels, effectively turning the VAE into a deterministic autoencoder. This happens because the reconstruction loss can be optimized more easily without the randomness from the latent sampling, and the KL penalty discourages using the latent dimensions.

For image-level diffusion models (which use non-autoregressive generation), this variance collapse is benign: the latent vectors are all generated simultaneously by the diffusion process, and less variance simply means more deterministic encoding. But for autoregressive generation, low latent variance is catastrophic due to exposure bias: during inference, the model conditions on its own previously generated latent vectors, which contain prediction errors. If the latent space has very small variance, these small prediction errors can push the generated vector outside the narrow distribution that the VAE decoder expects, causing severe reconstruction artifacts. With larger latent variance, the decoder is trained to handle a wider range of inputs, making it more robust to these errors.

The σ-VAE solution. The paper proposes a simple modification: instead of learning per-channel variances, enforce a fixed scalar variance $\sigma$ for all channels, sampled per example:

$\mu = \text{Encoder}_\phi(x)$

$z = \mu + \sigma \odot \epsilon, \quad \text{where } \epsilon \sim \mathcal{N}(0, 1), \quad \sigma \sim \mathcal{N}(0, C_\sigma)$

$\hat{x} = \text{Decoder}_\psi(z)$

where $C_\sigma$ is a hyperparameter controlling the scale of variance, $\sigma$ is a scalar (same for all channels of the latent vector, but different for each training example), $\epsilon \sim \mathcal{N}(0, 1)$ is element-wise independent Gaussian noise, and $\odot$ denotes element-wise multiplication.

What it computes: the encoder produces a mean vector $\mu$ (deterministic given the input). A scalar variance $\sigma$ is sampled from a zero-mean Gaussian with standard deviation $C_\sigma$ , independently for each training example. This scalar $\sigma$ is multiplied element-wise with a random standard Gaussian vector $\epsilon$ to produce the perturbation $\sigma \cdot \epsilon$ , which is added to $\mu$ to produce the latent vector $z$ . The decoder then reconstructs the original data from $z$ .

Why this form: by using a single scalar $\sigma$ for all channels rather than per-channel variances, the model is forced to have uniform variance across the latent space — there is no way for the encoder to collapse individual channels. The variance is controlled entirely by the hyperparameter $C_\sigma$ , making it easy to tune. During training, $\sigma$ varies per example (some examples get more noise, some less), which trains the decoder to be robust to a range of latent scales. During inference, the autoregressive model generates latent vectors with approximately the same variance distribution, so the decoder generalizes well.

The training objective simplifies to:

$\text{minimize } \|\hat{x} - x\|^2_2 + \beta \cdot \|\mu\|^2_2$

where the first term is the reconstruction error (L2 distance between input and reconstruction), and the second term is an L2 penalty on the mean vector $\mu$ (pushing it toward zero, which aligns with the Gaussian prior). The hyperparameter $\beta$ controls the tradeoff between reconstruction fidelity and prior adherence (analogous to $\beta$ -VAE [HMP+16]). This objective follows from the standard VAE ELBO (Equation 4) under the σ-VAE's fixed-variance assumption.

Why no KL divergence term: in the standard VAE, the KL term encourages the per-channel variances to approach 1 and the means to approach 0. In σ-VAE, the variance is fixed by design (not learned), so there is no variance-related KL term to compute — only the L2 penalty on $\mu$ remains to encourage the mean to stay near zero. This makes the objective simpler and avoids the posterior collapse instability.

Connection to exposure bias robustness. The paper validates the σ-VAE design through a controlled experiment in Figure 6. Tokenizers with different fixed variances $\sigma$ are trained and evaluated under both DiT (non-autoregressive diffusion) and LatentLM (autoregressive). Under DiT, all tokenizers perform similarly — the variance choice is largely irrelevant because there is no exposure bias. Under LatentLM, larger variance monotonically improves FID when classifier-free guidance is not used (CFG=1.0). The "stars" in Figure 6 represent tokenizers tuned for latent diffusion models (which have naturally small variance because they prioritize reconstruction fidelity) — these perform dramatically worse under LatentLM. This empirically confirms the paper's central claim: variance collapse is a real barrier to autoregressive generation with latent vectors, and σ-VAE directly addresses it.

Implementation for different modalities. The σ-VAE architecture varies by modality:

Images (Section 3.1.3): The encoder is initialized from a base-size BEiT-3 [WBD+23] checkpoint with 12 Transformer layers. The decoder is randomly initialized with 12 Transformer layers. Total tokenizer parameters: 172M. The image patch size is 16 (a 256×256 image produces 256 patches, each mapped to a latent vector). The tokenizer is trained on ImageNet for 200 epochs with perceptual loss [ZIE+18, JAFF16] and GAN loss [IZZE17] in addition to the reconstruction loss, following standard practice from [RBL+22, ERO21]. The optimizer is AdamW with β = (0.0, 0.99) and learning rate 3e-4, weight decay 0.01, layer-wise learning rate decay of 0.65 on the encoder.
Speech (Section 3.3.1): A convolutional architecture with streaming capability (causal 1D convolutions, no Transformers). The encoder has multiple stages with hierarchical downsampling: for compression ratios of 1600, 3200, and 6400, the downsampling factors are [2, 4, 5, 5, 8], [4, 4, 5, 5, 8], and [4, 5, 5, 8, 8] respectively. Each stage contains ConvNeXt blocks [LMW+22] with 1D causal convolutions. The channel count doubles at each downsampling stage, starting from 32 and increasing to 1024. The encoder has approximately 120M parameters. The decoder mirrors the encoder architecture. The discriminator uses multi-period discriminator [KKB20] and complex STFT discriminator from DAC [KSL+23].

Compression ratio analysis. Table 5 provides a comprehensive comparison of σ-VAE against prior speech tokenizers. The compression ratio is defined as the audio sample rate divided by (number of quantizers × frame rate). For σ-VAE with latent dimension 32, at a frame rate of 15 (15 latent vectors per second of speech), the compression ratio is 1600× (24,000 Hz sample rate / (1 quantizer × 15 frame rate)). At a frame rate of 3.75, the compression ratio is 6400×. These are 10–80× higher than neural codec tokenizers like Encodec (40×) and DAC (160×), while achieving better or comparable reconstruction quality (Mel Distance 0.813–0.852 vs. 0.823–0.987 for low-compression codecs).

Why higher compression is possible with continuous representations. Discrete tokenizers (vector quantization) achieve low compression ratios because they must encode fine-grained audio details into discrete codebook indices — each index can only represent $\log_2(K)$ bits of information for a codebook of size $K$ . Achieving high fidelity requires either many codebook indices per time step (multiple quantizers, the $N_q$ column in Table 5) or a high frame rate. Continuous vectors, in contrast, can encode much more information per vector because each dimension is a 32-bit floating-point number — a latent vector of dimension 32 can represent $32 \times 32 = 1024$ bits of information, vastly more than a single discrete codebook index. This is why σ-VAE achieves competitive reconstruction at 6400× compression while discrete codecs require 10–40× compression.

Image Generation Instantiation (Section 3.1)

The image generation experiments serve as the primary benchmark for LatentLM's core generative capabilities and scaling properties.

Model configuration (LatentLM-L, Table 1):

Transformer hidden size: 1024
Number of Transformer layers: 32
Feedforward network intermediate dimension: 2730
Number of attention heads: 16 (implied by hidden size / 64 standard)
Diffusion head: 6 layers of feedforward networks with AdaLN-Zero conditioning
Total parameters: 479M
Training steps: 250,000
Batch size: 2048 (resulting in approximately 400 epochs over the 1.28M ImageNet training images)
Optimizer: AdamW with β = (0.9, 0.98)
Learning rate: cosine schedule with maximum 5e-4, 100 warmup steps
Weight decay: 0.1
Number of diffusion timesteps during training: 1000 (DDPM), with 4 timesteps sampled per forward pass
Number of inference denoising steps: 20 (DPM-Solver)
Classifier-free guidance scale: 1.65 (for final model), 1.75 (for scaling experiments)
Cosine beta schedule with v-prediction [SH22]

What v-prediction means: instead of predicting the noise $\epsilon$ directly, the model predicts $v = \sqrt{\bar{\alpha}_t} \cdot \epsilon - \sqrt{1-\bar{\alpha}_t} \cdot x_0$ , which is a velocity-like quantity. This is then converted back to $\epsilon$ or $x_0$ during inference. v-prediction was introduced in [SH22] for progressive distillation and has been found to be more stable than $\epsilon$ -prediction for certain noise schedules.

Baseline alignment for fair comparison. The paper explicitly states (Table 1 caption, Section 3.1.1) that model configurations are aligned with previous work. For example, the FFN dimension of 2730 follows DiT-XL/2's configuration (where FFN = 8/3 × hidden size). The number of training epochs (400) matches U-ViT-H/2 and DiT-XL/2.

Scaling experiments (Section 3.1.2). Four model sizes are trained for 75,000 steps (approximately 120 epochs) with the same LatentLM architecture but varying dimensions:

Size	Parameters	Hidden Dim.	Layers	Heads	Learning Rate
Medium	455M	1024	24	16	8e-4
Large	1.03B	1536	24	12	3e-4
XL	1.82B	2048	24	16	2e-4
3B	3.68B	2560	32	20	1.6e-4

The DiT baseline is augmented with RMSNorm and SwiGLU for consistency, and its FFN dimension is set to 8/3 × hidden size to match parameter counts with LatentLM (which uses 4 × hidden size for FFN but has additional parameters in the diffusion head).

Resolution scaling (Table 2). A 1.82B model trained at 384×384 resolution for 100,000 steps achieves FID 2.51 versus 3.19 at 256×256. The higher resolution produces longer latent sequences (more patches), which provides richer detail but also increases computation proportionally.

Inference efficiency analysis (Figure 7). This is a critical practical result:

Figure 7a: at batch size 128, LatentLM achieves 2.47× higher throughput than DiT for the 3.68B model. The gap increases with model size because DiT's cost scales as $T_{\text{inf}} \times \text{FLOPs}_{\text{backbone}}$ (the full Transformer runs at each of $T_{\text{inf}} = 20$ denoising steps), while LatentLM's cost scales as $\text{FLOPs}_{\text{backbone}} + T_{\text{inf}} \times \text{FLOPs}_{\text{head}}$ (the Transformer runs once, only the lightweight head runs $T_{\text{inf}}$ times).
Figure 7b: at 1.82B parameters, as batch size increases from 8 to 256, LatentLM's throughput scales more favorably than DiT's. With group-query attention (GQA) [ALTdJ+23], LatentLM achieves 2.84× throughput improvement at batch size 256. GQA reduces the key-value cache memory and the attention computation by sharing key-value heads across query heads, which is particularly beneficial for autoregressive models that maintain a large KV cache.
Appendix C (Figure 11) extends this analysis to 1.03B, 3.68B, 9.35B, and 17.96B models, consistently showing 2–3× throughput advantages for LatentLM. The KV cache reuse is the key enabler: once the Transformer generates the hidden states for all positions, only the new position requires full attention computation; previous positions' hidden states are cached.

Effect of tokenizer variance (Figure 6, Section 3.1.3). The controlled experiment with fixed-variance σ-VAE tokenizers reveals:

Under DiT (image-level diffusion), FID is approximately flat across variance values — the choice of tokenizer variance is not critical for non-autoregressive generation. This is because DiT generates all latent vectors simultaneously, so there is no accumulation of autoregressive errors.
Under LatentLM without classifier-free guidance (CFG=1.0), FID improves monotonically as tokenizer variance increases. This directly validates the central hypothesis: larger latent variance makes the decoder more robust to the exposure bias inherent in autoregressive generation.
The "stars" (tokenizers tuned for latent diffusion models, typically with naturally small variance because they optimize for reconstruction fidelity without considering autoregressive robustness) perform dramatically worse under LatentLM — FID around 40–50 versus 20–25 for larger-variance tokenizers.
With classifier-free guidance (CFG > 1.0), the relationship is more complex — there appears to be an optimal variance range, as guidance interacts with the variance. CFG scales the conditioning signal, which affects the latent vectors' distribution and thus their compatibility with the decoder.

The practical recommendation: retrain a σ-VAE tokenizer with explicitly controlled variance for LatentLM, rather than reusing tokenizers designed for image-level diffusion models.

Multimodal LLM Instantiation (Section 3.2)

This section demonstrates how LatentLM unifies text understanding, text generation, image understanding, and image generation in a single model.

Training data mixture (Section 3.2.1). Three data types are mixed in a 2:1:1 ratio:

Text-only data (2 parts): Common Crawl, RefinedWeb [PMH+23], and StarCoder [LAZ+23] — standard web text and code corpora.
Image-text pairs (1 part): LAION-2B [SBV+22], LAION-400M [SVB+21], COYO-700M [BPK+22], Conceptual Captions [SDGS18, CSDS21] — images with associated captions.
Interleaved image-text data (1 part): Web documents from Common Crawl that contain both text and images in natural sequence, filtered following [HDW+23, PWD+23].

The 2:1:1 ratio means that for every four training tokens, approximately two come from text-only data, one from captioned images, and one from interleaved documents. This ensures the model maintains strong language capabilities while learning to handle visual data.

Model configuration:

Transformer: 1.3B parameters total
Hidden size: 2048
Number of layers: 24
Feedforward dimension: 6144 (3× hidden size)
Number of attention heads: 16
Vocabulary size: 100,288
Text tokenizer: tiktoken-cl100k_base
Training sequence length: 4096
Batch size: 4M tokens (the number of sequences per batch depends on sequence length)
Diffusion head: 6 layers
Optimizer: AdamW with β = (0.9, 0.98)
Learning rate: 3e-4 with cosine schedule, 500 warmup steps
Weight decay: 0.1

Training procedure. The model is trained for 50,000 steps (200B tokens) for the main comparison, with scaling experiments extending token counts further (Figure 8). The loss is the weighted sum of LM loss and diffusion loss (Equation $\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{LM}} + \alpha \cdot \mathcal{L}_{\text{Diff}}$ ), computed per-token on the packed sequences. For interleaved data, the special tokens <BOD> and <EOD> mark transitions between text generation (softmax head) and image generation (diffusion head).

Baseline configurations:

VQ-MLLM (Vector-Quantized Multimodal LLM): Uses the VQ-VAE tokenizer from LlamaGen [SJC+24] to convert images into discrete code sequences. These codes are treated as tokens from an extended vocabulary and generated with the standard softmax head (no diffusion). This represents the "treat everything as discrete tokens" approach.
Transfusion [ZYB+24]: Shares Transformer weights but uses different objectives — next-token prediction for text (causal attention), and sequence-level diffusion for images (bidirectional attention, the full image is denoised simultaneously using the Transformer backbone). A 6-layer ViT is used as the image head to align parameter counts.

Training-inference consistency. A key design choice that distinguishes LatentLM from Transfusion: during training, LatentLM processes clean image latent vectors (from the σ-VAE encoder) in the autoregressive sequence. The diffusion head is trained to generate these vectors from noise conditioned on the Transformer state, but the Transformer itself never sees noised images during training. In contrast, Transfusion adds noise to image tokens during training (to learn the denoising process) and uses bidirectional attention for this denoising. This means Transfusion's image representations during training (noised) do not match those during inference (clean), which the paper argues degrades image understanding performance.

Evaluation metrics:

Language modeling: validation perplexity on held-out text — lower is better, measures how well the model predicts the next text token.
Text-to-image generation: FID [HRU+17] on MS-COCO [LMB+14b] — lower is better, measures the distributional distance between generated and real images. CLIP score [RKH+21] — higher is better, measures the semantic similarity between the generated image and the text prompt.
Image-to-text generation: CIDEr score [VLZP15] on MS-COCO image captioning — higher is better, measures the quality of generated captions. VQAv2 accuracy [GKSS+17] — higher is better, measures the model's ability to answer questions about images.
Scaling curves (Figure 8): FID and validation perplexity as a function of training tokens (from ~25B to ~200B) — shows whether the model continues to improve with more data.

Key architectural detail for multimodal sequences. When a training document contains interleaved text and images, the sequence is constructed as: [text tokens], <BOD>, [image latent vectors], <EOD>, [more text tokens]. The model predicts text tokens using the softmax head and image latent vectors using the diffusion head, all within the same autoregressive loop. The causal masking ensures that each token can attend to all previous tokens (whether text or image), enabling cross-modal conditioning — the text generated after an image can depend on the visual content, and the image generated after text can depend on the textual description.

Text-to-Speech Instantiation (Section 3.3)

This section demonstrates LatentLM's applicability to high-fidelity audio generation with extreme efficiency.

Speech σ-VAE tokenizer architecture (Section 3.3.1). Unlike the image tokenizer (which uses Transformers), the speech tokenizer uses a purely convolutional architecture:

Encoder: Multiple stages in a hierarchical structure. Each stage contains ConvNeXt blocks [LMW+22] with 1D causal convolutions (replacing the original 2D convolutions). Downsampling is applied between stages with specific factors depending on the target compression ratio. Example for 1600× compression: downsampling factors [2, 4, 5, 5, 8] are applied sequentially, reducing the temporal resolution by 2×4×5×5×8 = 1600×. The number of channels doubles at each stage: starting from 32 and increasing to 64, 128, 256, 512, and finally 1024.
Quantizer: Continuous VAE quantizer (the σ-VAE mechanism). No codebook — the latent vectors are continuous. The latent dimension can be 16, 32, or 64 (Table 6).
Decoder: Mirror of the encoder, with upsampling replacing downsampling.
Discriminator (for adversarial training): Multi-period discriminator [KKB20] and complex STFT discriminator from DAC [KSL+23]. These provide adversarial loss signals that improve the perceptual quality of the reconstructed audio.
Total encoder parameters: ~120M

Training data for the tokenizer. The σ-VAE tokenizer is trained on a diverse corpus including speech (DNS Challenge 4 clean subset, Common Voice v7), general audio (FSD50K, AudioSet), and music (MUSDB, Jamendo). All data are resampled to 24kHz monophonic. This diverse training ensures the tokenizer generalizes across audio domains, not just clean speech.

LatentLM configuration for TTS:

Transformer hidden size: 1024
Number of layers: 24
Number of attention heads: 16
Feedforward dimension: 4096
Diffusion head: 3 layers of feedforward networks
Optimizer: AdamW with β = (0.9, 0.98)
Learning rate: 7.5e-4 with cosine schedule, 10,000 warmup steps
Batch size: 5M tokens
Training steps: 100,000
Weight decay: 0.01

Training data for the TTS model. The Libriheavy corpus [KYY+24], a labeled version of LibriLight [KRZ+20] containing 50,000 hours of speech from approximately 7,000 speakers (English audiobooks from LibriVox).

Frame rate and compression ratio. The frame rate is the number of autoregressive steps needed to generate 1 second of speech. For σ-VAE:

Latent dimension 32, compression 1600× → frame rate 15 (24,000 Hz / 1600 = 15 latent vectors per second)
Latent dimension 64, compression 3200× → frame rate 7.5
Latent dimension 128, compression 6400× → frame rate 3.75

Compare to neural codec baselines: VALL-E 2 [CLZ+24] uses a frame rate of 75 (and also requires a non-autoregressive model for coarse-to-fine generation), and Voicebox [LVS+23] uses a frame rate of 100. LatentLM at frame rate 7.5 is 10× fewer autoregressive steps than VALL-E 2, and at frame rate 3.75 it is 20× fewer. This directly translates to faster inference and lower computational cost.

Zero-shot TTS evaluation (Table 4). Two settings:

Reference utterance as prompt: A short speech sample from the target speaker is provided as conditioning, and the model generates speech for a new text in that speaker's voice.
3-second prefix as prompt: The first 3 seconds of a target utterance are provided, and the model continues the speech (speech continuation).

Metrics: Speaker Similarity (SIM, using WavLM-TDNN), Word Error Rate for content (WER-C using Conformer-Transducer), and Word Error Rate for intelligibility (WER-H using HuBERT-Large). Higher SIM is better; lower WER is better.

At frame rate 15, LatentLM achieves SIM 0.697 (ref utterance) and 0.571 (3s prefix), surpassing VALL-E 2 (0.643, 0.504) and Voicebox (0.662, 0.593). WER-C is 1.2–1.4 versus 1.5–1.6 for baselines. At frame rate 7.5, LatentLM achieves SIM 0.656 (ref utterance), still competitive with VALL-E 2 while using 10× fewer autoregressive steps.

Classifier-free guidance for TTS (Figure 10a). CFG scale of 4 is optimal, with significant gains over no guidance (CFG=1). At CFG=4, SIM peaks and WER drops. Higher CFG scales (8–16) cause degradation, likely because the unconditional and conditional distributions diverge too far.

Inference sampling steps for diffusion head (Figure 10b). The number of denoising steps for the diffusion head (per continuous token) is swept from 1 to 20. With CFG=4:

3 steps: competitive results
5 steps: strong performance with fast inference
10–20 steps: marginal further improvement

The paper recommends 5 diffusion steps per token for fast inference. Combined with the frame rate of 15, generating 1 second of speech requires 15 autoregressive Transformer steps × 5 diffusion head steps = 75 total refinement steps, but the heavy Transformer computation is done only 15 times (the 5 diffusion steps run the lightweight head only).

Ablation: compression ratio vs. latent dimension (Table 6, Section 3.3.6). Key finding: increasing the latent dimension compensates for higher compression. At the same compression ratio of 1600×, increasing the latent dimension from 16 to 32 improves tokenizer reconstruction SIM from 0.700 to 0.870, and TTS SIM from 0.545 to 0.661. At latent dimension 32 and compression 640×, TTS SIM is 0.655 — similar to the 1600×/32-dim configuration. This suggests the information capacity per latent vector (controlled by dimension) matters more than the frame rate for quality, as long as the dimension is sufficient.

Why this matters for efficiency: the frame rate directly determines the number of autoregressive steps, which is the main bottleneck for Transformer inference (since each step requires a full forward pass through the Transformer, even with KV caching). The diffusion head steps are an additional cost but much smaller. A higher latent dimension with a lower frame rate (e.g., dimension 128, frame rate 3.75) maximizes efficiency while maintaining quality, because it reduces the number of expensive Transformer forward passes.

Comparison with prior continuous speech generation (MELLE). MELLE [MZL+24] also uses continuous representations for speech but operates at frame rate 62 — LatentLM achieves better performance at frame rates of 7.5–15 (4–8× fewer steps). The paper attributes this to σ-VAE's variance control, though no direct ablation against MELLE's tokenizer is provided.

Summary of Design Choices and Their Justifications

Next-token diffusion over full-sequence diffusion: enables autoregressive generation with KV caching, reducing inference FLOPs by 2–3× compared to image-level diffusion (Figure 7). Also enables variable-length generation and streaming, which are impossible with bidirectional diffusion.
Diffusion as a head over diffusion as a framework: keeps the Transformer backbone consistent with language modeling, simplifying multimodal training and enabling knowledge sharing across modalities (Table 3 shows better language modeling perplexity than Transfusion).
σ-VAE with fixed scalar variance over standard VAE: prevents variance collapse, which is catastrophic for autoregressive generation (Figure 6). The scalar variance is a simple, controllable mechanism that can be tuned per application (image vs. speech).
Multiple timesteps per forward pass (typically 4): improves training efficiency because the expensive Transformer computation is reused for multiple diffusion loss evaluations, getting more signal per batch.
DPM-Solver for inference (3–20 steps vs. 1000 training steps): dramatically reduces inference time without significant quality loss, by exploiting the semi-linear ODE structure of the diffusion reverse process.
Causal attention everywhere (no bidirectional attention): keeps training and inference consistent (unlike Transfusion, which uses bidirectional attention for image training but produces clean images during inference). Also enables KV caching for efficient autoregressive generation.
Per-example scalar variance $\sigma \sim \mathcal{N}(0, C_\sigma)$ over per-channel learned variances: forces uniform variance across the latent space (no channel can collapse), is trivially tunable via $C_\sigma$ , and trains the decoder to handle a range of input scales (improving robustness to autoregressive errors).
Continous over discrete latent representations: achieves 10–80× higher compression ratios for speech (6400× vs. 80× for neural codecs, Table 5) while maintaining or improving reconstruction quality. Avoids the quantization bottleneck that limits scaling (VQ-MLLM saturates in Figure 8a).
Single unified objective ( $\mathcal{L}_{\text{LM}} + \alpha\mathcal{L}_{\text{Diff}}$ ) over separate training regimes: simplifies implementation (reuses LLM training infrastructure), avoids optimization conflicts between different objectives, and enables direct knowledge transfer between modalities.

4. Key Insights and Innovations

Innovation 1: Autoregressive Generation of Continuous Data Through Per-Token Diffusion Decoding

The paper's foundational conceptual move is recognizing that diffusion need not be a full-sequence generative framework—it can be demoted to a per-token decoding mechanism within an otherwise standard autoregressive Transformer. This is a fundamental reframing of the relationship between autoregressive models and diffusion models, which the field has treated as competing paradigms for entirely different generative problems.

What the field did before. Prior work organized around two mutually exclusive camps. The first camp (DiT [PX23], LDM [RBL+22], U-ViT [BNX+23a]) treated diffusion as a sequence-level process: the entire image is a single monolithic generation target, denoised simultaneously with bidirectional attention over all positions. This makes diffusion fundamentally incompatible with autoregressive decoding, key-value caching, and variable-length generation. The second camp (LlamaGen [SJC+24], VQGAN [ERO21], VALL-E 2 [CLZ+24]) committed fully to the autoregressive paradigm but forced continuous data through a quantization bottleneck to make it look like discrete tokens—accepting the information loss, low compression ratios, and saturation effects that come with discretization. The implicit assumption across both camps was that you must choose: either generate everything at once with diffusion (efficient for continuous data, poor for discrete) or generate everything token-by-token with next-token prediction (efficient for discrete, requires quantization for continuous). Transfusion [ZYB+24] attempted to bridge this gap by sharing weights between the two paradigms, but it kept the objectives and attention patterns separate—a détente, not a unification.

What makes LatentLM's framing distinctive. The paper's key insight is that these are not competing paradigms because they operate at different levels of the generation hierarchy. The autoregressive Transformer handles sequence-level structure—what comes next, in what order, conditioned on what history—while diffusion handles token-level refinement—transforming a random vector into a specific continuous representation at a single position. By making diffusion a lightweight head that conditions on a frozen Transformer hidden state, LatentLM preserves everything that makes autoregressive models effective (causal attention, KV caching, variable-length generation, streaming) while gaining everything that makes diffusion effective for continuous data (high-fidelity generation without quantization, controllable variance). The per-token diffusion head is to continuous data what the softmax classification head is to discrete data: an output mechanism that maps a contextualized hidden state to a data point, with the iterative nature of diffusion being the continuous analog of the discrete sampling operation.

This is a fundamental architectural shift, not an incremental refinement. It changes the taxonomy of generative models from a binary choice (autoregressive vs. diffusion) to a compositional one (autoregressive sequence modeling + per-token decoding mechanism), where the decoding mechanism can be softmax (for discrete), diffusion (for continuous), or potentially other operators. The paper's own framing—"latent language modeling"—reflects this conceptual ambition: it is language modeling because the causal, next-token structure is preserved, and it is latent because continuous data enters through a compressed representation space.

Evidence for why this matters beyond a single metric. The distinction between per-token and full-sequence diffusion is not cosmetic—it produces qualitatively different scaling behavior. Figure 7 shows that as model size increases from 1B to 3.8B parameters, DiT's throughput drops steeply (because the full Transformer must run at each of 20 denoising steps), while LatentLM's throughput remains high (because the Transformer runs once, and only the lightweight head iterates). At 3.8B parameters with batch size 128, the difference is 2.47×. This gap would only widen with more denoising steps or larger models. More fundamentally, the ability to reuse KV caches enables generation of variable-length continuous sequences—streaming speech synthesis, video frame generation, interleaved image-text dialogue—that are architecturally impossible for bidirectional diffusion. The TTS results (Table 4) concretely demonstrate this: generating speech at a frame rate of 7.5 (7.5 autoregressive steps per second) with 5 diffusion head steps per token would be impossible if each diffusion step required a full Transformer forward pass—the computational cost would be 10× higher, matching what VALL-E 2 and Voicebox actually require.

Innovation 2: Variance Collapse as the Hidden Barrier to Continuous Autoregressive Generation

The paper identifies, names, and systematically addresses a failure mode that was largely invisible to the field: VAE variance collapse makes autoregressive generation with latent vectors fail, even when the same VAE works perfectly for non-autoregressive diffusion. This is a diagnostic contribution—the kind of insight that explains why prior attempts at continuous autoregression underperformed and what must change to fix it.

What the field assumed. The standard practice in latent diffusion models is to train a VAE (or VQ-VAE) tokenizer that optimizes for reconstruction fidelity, often with a small KL weight or no KL term at all—effectively an autoencoder. These tokenizers naturally have low latent variance because determinism helps reconstruction. The dominant assumption was that a good tokenizer for latent diffusion would also be good for other downstream uses of the latent space. GIVT [TEM23], which directly predicts VAE latent vectors with Gaussian mixture models, is the closest prior attempt at continuous autoregression, but it did not diagnose variance collapse as a specific failure mode—instead, it attributed difficulties to the general challenge of modeling high-dimensional continuous distributions. The field lacked a clear explanation for why continuous autoregression seemed harder than it should be.

What makes σ-VAE conceptually distinctive. The paper's diagnosis is precise and mechanistic: autoregressive generation introduces exposure bias—the model conditions on its own (potentially erroneous) predictions. In a low-variance latent space, even small prediction errors produce latent vectors that fall outside the narrow distribution the decoder was trained on, causing cascading degradation. In a high-variance latent space, the decoder has been trained to handle a wider range of inputs, so it is robust to these errors. The innovation is not the σ-VAE architecture itself (fixed scalar variance is a simple design choice), but the recognition of variance as the critical control variable for continuous autoregressive generation and the controlled experiment that proves it.

Figure 6 provides one of the paper's most elegant results: the same set of tokenizers, when used with DiT (non-autoregressive), produce essentially flat FID across variance levels—variance doesn't matter. When used with LatentLM (autoregressive), FID improves monotonically with variance when classifier-free guidance is off (CFG=1.0). The "stars" (tokenizers from standard latent diffusion models) cluster at the high-FID, low-variance regime under LatentLM but are unremarkable under DiT. This is a clean dissociation: the tokenizers aren't "bad"—they're bad for autoregressive generation specifically, and the mechanism is variance.

This is an incremental architectural change (scalar variance instead of per-channel variance) with fundamental diagnostic implications. It tells the field: when building tokenizers for autoregressive models, do not reuse tokenizers designed for diffusion models. Do not treat variance as a nuisance to be minimized. Instead, treat variance as a hyperparameter that controls robustness to autoregressive errors. The paper's recommendation to train σ-VAE from scratch with explicit variance control, rather than adapting existing tokenizers, follows directly from this diagnosis.

Significance beyond LatentLM. The variance collapse diagnosis applies to any system that uses VAEs for autoregressive generation of continuous latents—including future approaches that might replace next-token diffusion with other decoding mechanisms. It is a general principle: for autoregressive generation, the latent space must be explicitly regularized to maintain sufficient variance. This insight did not exist in the literature before this paper in a form that connected VAE design choices to autoregressive failure modes with empirical validation.

Innovation 3: Continuous Representations as a Path to Scaling Autoregressive Generation Beyond the Quantization Bottleneck

The paper provides the first systematic evidence that continuous latent representations fundamentally scale better than discrete quantization for autoregressive generation of high-fidelity continuous data. This is not merely a performance advantage at one scale—it is a reframing of the scaling ceiling for multimodal autoregressive models.

What the field assumed. The dominant approach to autoregressive generation of continuous data—from DALL-E [RPG+21] to VALL-E [WCW+23] to Chameleon [Tea24] to LlamaGen [SJC+24]—has been vector quantization: compress continuous data into discrete code sequences, then use standard next-token prediction. The implicit assumption was that the autoregressive Transformer architecture requires discrete tokens (since the softmax head outputs a categorical distribution), so quantization is a necessary evil. The costs were acknowledged (lossy compression, long sequences from low compression ratios) but treated as unavoidable tradeoffs.

What makes LatentLM's evidence distinctive. The paper provides converging evidence across three modalities that continuous representations escape the quantization bottleneck:

Image generation (Table 1): LatentLM-L (479M parameters, causal, continuous) achieves FID 2.24, outperforming LlamaGen-XXL (1.4B parameters, causal, discrete, FID 2.34) and nearly matching LlamaGen-XL (775M, FID 2.62). The continuous model achieves better quality with 2–3× fewer parameters.
Multimodal LLM scaling (Figure 8a): VQ-MLLM's text-to-image FID saturates as training tokens increase—the curve flattens while LatentLM and Transfusion continue improving. This is direct evidence of a quantization-imposed ceiling: adding more training data stops helping because the discrete code representation has discarded information that the model needs to improve generation quality further. Continuous representations have no such ceiling because the latent vectors preserve fine-grained information.
Speech compression (Table 5): σ-VAE at 6400× compression (continuous) achieves Mel Distance 0.852, comparable to Encodec at 10× compression (discrete, Mel Distance 0.823) and DAC at 10× compression (Mel Distance 0.355, but note this is with 32 quantizers). The compression ratio is 640× higher with competitive reconstruction quality. In TTS (Table 4), this translates to 10× fewer autoregressive steps than VALL-E 2 while achieving better speaker similarity (0.697 vs. 0.643 with reference utterance).

Why this is a fundamental insight about scaling. The quantization bottleneck is not a constant penalty—it worsens at scale. As models and datasets grow, the information discarded by quantization becomes the limiting factor. You cannot train your way past it with more compute or more data, because the missing information is gone at the tokenizer stage. Continuous representations, by preserving information, allow generation quality to continue improving with model and data scale. The LLM scaling curves (Figure 8) and the image generation scaling curves (Figure 4) both support this: LatentLM's performance improves more steeply and for longer with scale than discrete-token baselines.

This reframes the choice between discrete and continuous representations from a matter of convenience (discrete tokens work with existing LLM infrastructure) to a matter of scaling ceilings (discrete tokens impose a hard limit that continuous representations do not). The paper explicitly notes that at 6400× compression, the speech token rate (3.75 frames per second) is already comparable to BPE text tokenization—approaching a 1:1 ratio between speech duration and sequence length. At that point, the efficiency argument for discrete tokens collapses entirely: continuous representations achieve similar sequence lengths with much higher fidelity.

Innovation 4: Unifying Objectives, Not Just Architectures, as the Key to Multimodal Transfer

The paper makes a subtle but consequential distinction between sharing weights (which Transfusion does) and sharing the learning objective (which only LatentLM does). This reframes the multimodal unification problem as one of training signal compatibility, not just parameter efficiency.

What Transfusion attempted. Transfusion [ZYB+24] was the state-of-the-art approach to unified multimodal modeling before LatentLM. It shares Transformer weights across text and image modalities, which is a significant achievement—the same attention matrices and feedforward networks process both types of data. However, it uses different training objectives: next-token prediction with causal masking for text, and sequence-level diffusion with bidirectional attention for images. The model must learn two different tasks under two different attention patterns, and the training signals from these tasks can conflict.

What LatentLM's evidence reveals. The paper's empirical results (Table 3) reveal that objective unification matters more than the prior literature suggested:

Language modeling perplexity: LatentLM (2.73) vs. Transfusion (2.74) vs. VQ-MLLM (2.79). The differences are small, suggesting that sharing backbone parameters already captures most of the cross-modal benefit for text prediction.
Text-to-image FID: LatentLM (14.54) vs. Transfusion (16.10) vs. VQ-MLLM (16.92). A larger gap favoring objective unification.
Image-to-text understanding: LatentLM achieves 54.5 CIDEr (image captioning) and 38.72 VQAv2 accuracy vs. Transfusion's 43.4 and 35.36. These are substantial gaps—15–25% relative improvement on understanding tasks.

The asymmetry is revealing: image understanding benefits more from objective unification than image generation does, because Transfusion's training procedure actively harms understanding. During Transfusion training, images are deliberately noised to learn the diffusion denoising process—but during inference for understanding tasks, images are presented clean. This training-inference mismatch degrades the model's ability to extract information from clean images. LatentLM avoids this entirely: images are always processed clean through the Transformer backbone, with diffusion used only as a generation mechanism (decoding from hidden states), never as a training-time corruption of inputs.

The conceptual reframing. This shifts the multimodal unification problem from "can we share parameters?" (which Transfusion answered affirmatively) to "can we design a training procedure where the signals from different modalities are mutually reinforcing rather than conflicting?" LatentLM's answer is to make the autoregressive next-token paradigm universal, with diffusion as a transparent drop-in replacement for the softmax head at continuous positions. The Transformer sees the same thing during training and inference, regardless of modality. The loss function is the same shape (prediction error) regardless of whether the target is a word, an image patch, or a speech frame. This consistency is what enables the improved knowledge transfer: the model learns representations that are useful for predicting the next token, whether that token is discrete or continuous, and these representations transfer across modalities because they're learned under the same objective.

This is a conceptual refinement of the unification problem more than a technical breakthrough—replacing a bidirectional denoising objective with a causal prediction objective is architecturally straightforward once the next-token diffusion head exists. But the paper's demonstration that this refinement produces meaningful gains on understanding tasks (while matching or exceeding on generation) makes the case that objective unification is worth the architectural commitment, even when weight-sharing alone has already been achieved.

Evidence for mutual reinforcement. The fact that LatentLM achieves both better language modeling perplexity and better image understanding than Transfusion (Table 3) suggests the unified objective does not just avoid interference—it actively enables positive transfer. The model learns visual representations that are shaped by the same predictive objective as its linguistic representations, making them more directly useful for tasks that require mapping between modalities (captioning, VQA). This is consistent with the paper's broader ambition of enabling "multimodal-native reasoning" (Section 4), where the model can seamlessly move between modalities without retranslating through text.

Innovation 5: Inference-Time Efficiency as a First-Class Architectural Property

While efficiency is often treated as an engineering afterthought in generative model papers, LatentLM makes a compelling case that the architectural decision to make diffusion per-token rather than per-sequence is fundamentally an efficiency argument with scaling implications, not just an implementation detail.

The architectural source of the efficiency gain. The key computation in any Transformer-based generative model is the forward pass through the self-attention and feedforward layers. In image-level diffusion (DiT), this computation is performed N_steps times for the entire image, where N_steps is typically 20–50. Each step processes all image patches with bidirectional attention, and there is no caching across steps because the noise level changes. In LatentLM, the Transformer forward pass is performed once per image patch (or per speech frame), and the key-value cache allows each new patch to be computed by attending to previously generated patches without recomputing them. The diffusion head's denoising steps (typically 3–20, depending on the DPM-Solver configuration) only involve the lightweight head, not the full backbone.

The consequence, quantified in Figure 7, is that LatentLM achieves 2.47–2.84× higher inference throughput than DiT at equivalent model sizes and batch sizes, with the gap widening for larger models and larger batches. This is not a small optimization—it is the difference between a 3.8B parameter model being deployable at interactive latencies or not.

Why this matters at scale. The efficiency argument compounds as models grow. Appendix C (Figure 11) extends the throughput analysis to 9.35B and 17.96B models, where the advantages persist or grow. The KV cache—which is the enabling mechanism for LatentLM's efficiency—scales linearly with sequence length in memory but sub-linearly in computation (because cached states are read, not recomputed). In contrast, full-sequence diffusion scales linearly in computation with both sequence length and number of denoising steps, with no caching possible.

The connection to the paper's larger vision. The efficiency gains are not just about deployment cost—they enable capabilities that the paper envisions in Section 4 (Conclusion and Future Work). Streaming speech synthesis at a frame rate of 7.5 with real-time latency would be impossible if each frame required 20 full Transformer forward passes. Long-video generation "in an interleaved way" (generating scripts and video frames conditionally) would be computationally infeasible with full-sequence diffusion because the Transformer would need to be recomputed for every denoising step of every frame. Interactive multimodal dialogue, where the model generates text and images in response to user input with latency measured in hundreds of milliseconds, requires the per-token efficiency that KV caching enables.

This reframes the diffusion-vs-autoregressive debate from one about generation quality to one about inference-time scalability. The paper demonstrates that for the class of models where sequence generation happens autoregressively (which includes all LLM applications), per-token diffusion is strictly more efficient per FLOP than full-sequence diffusion, even if both achieve the same generation quality. The architectural choice is efficiency-driven, not quality-driven—and at the scales where these models are deployed, efficiency is a capability constraint, not an optimization target.

A cautionary note. The paper does not provide a FLOPs-matched quality comparison between LatentLM and DiT. That is, if DiT were given the same total inference FLOPs as LatentLM (by reducing the number of denoising steps), would the quality gap narrow or disappear? The throughput comparison (Figure 7) shows LatentLM is faster at equivalent model sizes—but if DiT could achieve similar quality with fewer denoising steps through distillation or better solvers, the efficiency advantage might shrink. The paper's scaling experiments (Figure 4) show LatentLM outperforming DiT at equivalent model sizes under their respective standard inference budgets, but these are not FLOPs-matched at inference time. This is a missing baseline that future work should address—though in practice, the architectural advantages of KV caching (streaming, variable-length generation) remain regardless of pure FLOPs parity.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper uses ImageNet [DDS+09] at 256×256 and 384×384 resolutions for class-conditional image generation (1.28M training images, 50K validation images used for FID computation); MS-COCO [LMB+14b] for text-to-image generation and image captioning evaluation; VQAv2 [GKSS+17] for visual question answering; LibriSpeech test-clean for zero-shot text-to-speech synthesis evaluation; and LibriTTS test-other for speech tokenizer reconstruction quality assessment. For TTS training, the Libriheavy corpus [KYY+24] is used (50,000 hours of labeled speech from ~7,000 speakers). The text training corpus for multimodal LLMs includes Common Crawl, RefinedWeb [PMH+23], and StarCoder [LAZ+23]. Image-text paired data comes from LAION-2B [SBV+22], LAION-400M [SVB+21], COYO-700M [BPK+22], and Conceptual Captions [SDGS18, CSDS21]. Interleaved image-text documents are filtered from Common Crawl following [HDW+23, PWD+23].
Base model(s). For image generation, the primary model is LatentLM-L with 479M parameters (Transformer hidden size 1024, 32 layers, FFN dimension 2730, 16 attention heads, diffusion head with 6 layers). For scaling experiments, four model sizes are used: 455M, 1.03B, 1.82B, and 3.68B parameters (detailed in Table 7, Appendix A). For multimodal LLM experiments, a 1.3B-parameter causal Transformer is used (hidden size 2048, 24 layers, FFN 6144, 16 heads, vocabulary 100,288, 6-layer diffusion head). For text-to-speech synthesis, the Transformer backbone has ~350M parameters (hidden size 1024, 24 layers, FFN 4096, 16 heads, 3-layer diffusion head). The papers also trains a ~120M-parameter σ-VAE speech tokenizer with a convolutional architecture. The image σ-VAE tokenizer uses 172M parameters (12-layer Transformer encoder initialized from BEiT-3 [WBD+23], 12-layer randomly initialized decoder). The choice of PaLM-scale models across experiments is motivated by the paper's goal of demonstrating competitive performance with established baselines at comparable parameter counts—the 479M LatentLM-L is deliberately aligned with DiT-XL/2 (675M) and MAR-L (479M) configurations for fair comparison (Table 1).
Metrics. For image generation: Fréchet Inception Distance (FID) [HRU+17] on 50K samples (lower is better, measures distributional distance between generated and real images) and Inception Score (IS) [SGZ+16] (higher is better, measures both image quality and diversity). For text-to-image generation: FID on MS-COCO and CLIP score [RKH+21] (measures semantic similarity between generated image and text prompt). For image-to-text: CIDEr score [VLZP15] on MS-COCO captions (higher is better, measures caption quality through consensus with reference captions) and VQAv2 accuracy [GKSS+17] (higher is better, measures visual question answering performance). For language modeling: validation perplexity (PPL) on held-out text, where lower values indicate better next-token prediction. For text-to-speech: Speaker Similarity (SIM) measured via WavLM-TDNN [CWC+22] (higher is better, cosine similarity between speaker embeddings of generated and reference speech); Word Error Rate using Conformer-Transducer (WER-C) [GQC+20] and HuBERT-Large (WER-H) [HBT+21] (lower is better, measures content preservation and intelligibility). For speech tokenizer reconstruction: Mel Distance (lower is better, log Mel spectrogram distance), PESQ-WB [RBHH01] (higher is better, perceptual speech quality), STOI [THHJ10] (higher is better, speech intelligibility through short-time correlation), VISQOL [CLS+20] (higher is better, perceptual quality via spectral similarity), and UTMOS [SXN+22] (higher is better, reference-free mean opinion score). For inference efficiency: throughput measured in images/second on a single H100 GPU.
Baselines. Image generation baselines include: non-causal-masking models—LDM-4 [RBL+22] (400M, latent diffusion), DiT-XL/2 [PX23] (675M, diffusion transformer), U-ViT-H/2 [BNX+23a] (501M, ViT-based diffusion), MaskGIT [CZJ+22] (227M, masked generative image transformer), and MAR-L [LTL+24] (479M, masked autoregressive model); causal-discrete models—VQGAN [ERO21] (1.4B, vector-quantized autoregressive), ViT-VQGAN [YLK+21] (1.7B), LlamaGen-XL and LlamaGen-XXL [SJC+24] (775M and 1.4B, Llama-based autoregressive with VQ tokenizer); and causal-continuous models—GIVT-Causal-L+A [TEM23] (1.67B, Gaussian mixture autoregressive). For multimodal LLMs: VQ-MLLM (vector-quantized multimodal LLM using VQ-VAE tokenizer from LlamaGen [SJC+24] with discrete image codes) and Transfusion [ZYB+24] (shared Transformer weights with causal text prediction and bidirectional image diffusion, using a 6-layer ViT image head). For TTS: VALL-E 2 [CLZ+24] (neural codec language model operating at frame rate 75 with additional non-autoregressive model), Voicebox [LVS+23] (flow-matching based speech generation at frame rate 100), and MELLE [MZL+24] (autoregressive continuous speech generation at frame rate 62). Speech tokenizer baselines include Encodec [DCSA22] at multiple bitrates, DAC [KSL+23] at multiple bitrates, DAClow [SD24] at 160× and 80× compression, Mimi [DMO+24] at 240× and 480× compression, and WavTokenizer [JJC+24] at 320× and 600× compression.
Generation budget / compute accounting. For image generation, the compute budget is measured in number of function evaluations during inference: DiT uses 250 DDPM sampling steps by default (reduced to 20 in the paper's DiT reimplementation with DPM-Solver for fair comparison with LatentLM's 20 DPM-Solver steps), while each LatentLM continuous token requires a single Transformer forward pass plus T_inf diffusion head steps (20 for image generation, swept from 3–20 for TTS). For throughput comparisons (Figure 7), the measurement unit is images/second at a given batch size and model size on identical H100 hardware. For multimodal LLMs, the training budget is measured in number of training tokens (200B for the main comparison, with scaling up to ~200B in Figure 8). For TTS, the generation budget is measured in autoregressive steps per second of speech (frame rate): VALL-E 2 uses 75, LatentLM uses 7.5–15. Unlike some prior work that reports FLOPs-matched comparisons, this paper does not match total inference FLOPs between LatentLM and baselines—the throughput comparison (Figure 7) measures wall-clock throughput rather than performing a controlled FLOPs budget allocation experiment.
Cross-validation / statistical protocol. The paper does not employ cross-validation for its main results. Model selection appears to be done on the standard ImageNet validation set (50K images for FID computation) and MS-COCO validation set. For TTS, evaluation is on the standard LibriSpeech test-clean split. There is no mention of multiple training runs with different random seeds for error bars or statistical significance testing. The multimodal LLM scaling curves (Figure 8) show results at different training token counts from a single training run each, without confidence intervals. The controlled tokenizer variance experiment (Figure 6) sweeps individual hyperparameter values (variance σ) with single training runs at each setting. This absence of replicate experiments and confidence intervals is a notable limitation—particularly for the multimodal LLM results where the 200B-token training runs are computationally expensive and variance across seeds could meaningfully affect the comparative rankings in Table 3, given the small absolute differences in some metrics (e.g., language modeling PPL: 2.73 vs. 2.74 vs. 2.79).

Main Quantitative Results

Image Generation: Class-Conditional Generation on ImageNet

The headline result in Table 1 is that LatentLM-L achieves FID 2.24 and IS 253.8 on 256×256 ImageNet using 479M parameters and 400 training epochs. This is competitive with the best causal-masking models and approaches the performance of non-causal diffusion models.

Comparison with non-causal-masking models. DiT-XL/2 (675M parameters) achieves FID 2.27, U-ViT-H/2 (501M) achieves FID 2.29, and MAR-L (479M) achieves FID 1.78. LatentLM-L at FID 2.24 sits between DiT and U-ViT, and behind MAR-L. However, the paper argues (Section 3.1.1) that non-causal models "typically require iterative forward computation during inference"—meaning their inference FLOPs are higher because they cannot reuse key-value caches. MAR-L, while achieving the best FID at 1.78, uses a bidirectional Transformer for masked autoregressive modeling, which the paper notes "renders MAR unable to reuse key-value caches for multiple forward passes." The implication is that LatentLM achieves comparable quality with better inference efficiency, though the paper does not provide a direct FLOPs-matched or latency-matched comparison with MAR.

Comparison with causal-discrete models. LatentLM-L substantially outperforms all causal-discrete baselines while using fewer parameters: VQGAN (1.4B, FID 5.20), ViT-VQGAN (1.7B, FID 3.04), LlamaGen-XL (775M, FID 2.62), LlamaGen-XXL (1.4B, FID 2.34). LatentLM-L (479M) achieves FID 2.24, which is 0.10 better than LlamaGen-XXL with ~3× fewer parameters. This is the strongest empirical evidence for the paper's claim that continuous representations avoid the quantization bottleneck. The trend is consistent: continuous > discrete for causal autoregressive generation at comparable or smaller model sizes.

Comparison with causal-continuous models. GIVT-Causal-L+A (1.67B parameters, 500 epochs) achieves FID 2.59. LatentLM-L (479M, 400 epochs) achieves FID 2.24—outperforming GIVT by 0.35 FID with 3.5× fewer parameters and fewer training epochs. The paper attributes this to diffusion heads offering "more powerful modeling expressivity" than GIVT's Gaussian mixture models for predicting continuous vectors. This comparison is the most direct evidence that next-token diffusion is a more effective decoding mechanism for continuous latent vectors than direct density estimation with mixture models.

Scaling behavior with model size (Figure 4). The scaling curves for LatentLM and DiT across four model sizes (455M, 1.03B, 1.82B, 3.68B parameters) show that LatentLM consistently achieves lower FID than DiT at every model size. At 455M, LatentLM achieves ~7.5 FID vs. DiT's ~9.0 (estimated from Figure 4). At 3.68B, LatentLM achieves ~3.5 FID vs. DiT's ~5.0. The gap widens with model size—LatentLM's curve is steeper—indicating better scaling properties. This is trained for only 75,000 steps (~120 epochs), so these are not converged models, but the relative comparison is informative for scaling trends. The paper attributes this to the parameter efficiency of the autoregressive formulation: in DiT, the AdaLN conditioning parameters in each Transformer layer scale with model size and are used only for time-step conditioning, while in LatentLM, the equivalent parameters in the diffusion head are separate from the backbone and don't grow with the backbone's size.

Scaling to higher resolution (Table 2). A 1.82B LatentLM trained at 384×384 resolution for 100,000 steps achieves FID 2.51 with classifier-free guidance, compared to FID 3.19 at 256×256 with the same model configuration. The improvement is attributed to "richer details and additional information captured in the tokenizer with higher resolutions." The sequence length increases (more latent vectors for more patches), which "scales the decoding computation up"—the autoregressive generation takes proportionally more steps.

Inference throughput (Figure 7). At 3.68B parameters with batch size 128, LatentLM achieves 2.47× higher throughput than DiT. At 1.82B parameters with batch size 256 and group-query attention (GQA), the throughput advantage is 2.84×. The throughput of LatentLM increases with batch size (better GPU utilization from KV cache reuse), while DiT's throughput remains flat (no caching possible). Appendix C (Figure 11) extends this to 9.35B and 17.96B models, confirming the advantage persists at larger scales. These measurements use 20 diffusion inference steps for both models on a single H100 GPU.

Tokenizer variance effects (Figure 6). This controlled experiment trains σ-VAE tokenizers with different fixed variances σ, then trains both DiT and LatentLM on top of each tokenizer. The key pattern: under DiT (non-autoregressive), FID is approximately flat across the variance range (all tokenizers achieve similar FID regardless of σ). Under LatentLM with CFG=1.0 (no classifier-free guidance), FID monotonically improves as tokenizer variance increases—the curve drops from approximately FID 50 at very low variance to approximately FID 20 at the highest variance values tested. The "stars" in Figure 6 represent tokenizers tuned for previous latent diffusion models (which naturally have small variance because they prioritize reconstruction determinism). These achieve dramatically worse FID under LatentLM (FID ~40–50) compared to purpose-trained σ-VAE tokenizers with larger variance (FID ~20–25). Under settings with CFG > 1.0, the relationship becomes more complex—there appears to be an optimal variance range rather than monotonic improvement, suggesting that guidance interacts with latent variance. The experiment validates the paper's central claim that variance collapse is a genuine barrier to autoregressive generation and that σ-VAE's explicit variance control addresses it. A critical detail: the measurement uses "20 diffusion inference steps" and evaluates on the standard 50K ImageNet validation set.

Multimodal Large Language Models: Unified Understanding and Generation

The headline result in Table 3 is that a 1.3B LatentLM trained on 200B tokens (2:1:1 text:text-image:interleaved mixture) achieves language modeling PPL of 2.73, text-to-image FID of 14.54 on MS-COCO, and VQAv2 accuracy of 38.72. Compared to Transfusion and VQ-MLLM at the same parameter count and training data scale, LatentLM achieves the best or tied-best performance on every metric.

Language modeling (Table 3, PPL column). LatentLM achieves PPL 2.73 versus Transfusion's 2.74 and VQ-MLLM's 2.79. The differences are small—0.01 between LatentLM and Transfusion, 0.06 between LatentLM and VQ-MLLM—but the ranking (LatentLM < Transfusion < VQ-MLLM) is consistent with the paper's claim that unified autoregressive objectives enable better knowledge sharing across modalities. The paper attributes this to "the similarity between next-token prediction and next-token diffusion" benefiting unified modeling, but the small magnitude of the difference (less than 0.4% relative improvement over Transfusion) suggests that backbone weight sharing is the dominant factor for text quality, with objective unification providing a marginal additional benefit.

Text-to-image generation (Table 3, FID and CLIP columns). LatentLM achieves FID 14.54 and CLIP score 28.75 on MS-COCO. Transfusion achieves FID 16.10 and CLIP 28.66. VQ-MLLM achieves FID 16.92 and CLIP 29.33. LatentLM's FID is 1.56 better than Transfusion and 2.38 better than VQ-MLLM—a meaningful margin (roughly 10% relative improvement). The CLIP scores are close, with VQ-MLLM surprisingly achieving the highest CLIP (29.33 vs. LatentLM's 28.75). This is interesting because CLIP score measures semantic alignment with the text prompt, while FID measures distributional image quality. The fact that VQ-MLLM has the best CLIP score but worst FID suggests that discrete tokenization may produce images that match the text prompt semantically but are lower quality in terms of overall visual fidelity. LatentLM's pattern (best FID, competitive CLIP) suggests better visual quality without sacrificing semantic alignment.

Scaling curves for text-to-image (Figure 8a). As training tokens increase from approximately 25B to 200B, all three models improve in FID. However, VQ-MLLM's curve visibly saturates—the improvement from 100B to 200B tokens is minimal, with FID flattening around 17. Transfusion continues to improve but at a slower rate than LatentLM. LatentLM's curve is the steepest and shows no signs of saturation at 200B tokens—continuing to decrease from approximately FID 16 at 100B tokens to approximately FID 14.5 at 200B tokens. This is presented as evidence that continuous representations scale better with data and that quantization imposes a ceiling on generation quality improvements from additional training. The paper does not report whether the same saturation would occur for VQ-MLLM with a larger codebook or different compression ratio—it's unclear whether this is fundamental to any discrete representation or specific to the LlamaGen VQ-VAE tokenizer used.

Image-to-text generation (Table 3, MS-COCO CIDEr and VQAv2). LatentLM achieves CIDEr 54.5 on MS-COCO captioning, compared to Transfusion's 43.4 and VQ-MLLM's 37.4. This is a 25.6% relative improvement over Transfusion and a 45.7% improvement over VQ-MLLM—the largest gap in any metric. On VQAv2, LatentLM achieves 38.72 accuracy, compared to Transfusion's 35.36 and VQ-MLLM's 30.19. These understanding results are where LatentLM most decisively outperforms both baselines. The paper attributes this to training-inference consistency: LatentLM never adds noise to images at training time (the Transformer sees clean VAE latents and the diffusion head learns to generate them), while Transfusion adds noise to images during denoising training. This means Transfusion's image representations during training are corrupted by noise that isn't present during understanding tasks at inference, actively harming visual understanding. The paper does not provide an ablation where Transfusion is evaluated with its image encoder only (without the denoising objective) to isolate this effect, but the magnitude of the gap makes this explanation plausible.

Scaling curves for image-to-text (Figure 8b). The validation perplexity on image-to-text data decreases (improves) for all models as training tokens increase. LatentLM's curve is consistently below Transfusion's by a small but widening margin, and substantially below VQ-MLLM's throughout. The gap between LatentLM and VQ-MLLM is large (~0.5 PPL at 200B tokens) and suggests that the information loss from quantization harms the model's ability to extract textual descriptions from images. The persistent gap between LatentLM and Transfusion (~0.1 PPL at 200B tokens) is consistent but small, suggesting the noise-addition training in Transfusion has a measurable but modest effect on understanding quality at this scale.

Qualitative examples (Figure 9). The paper shows four text-to-image generation examples: "A majestic mountain range covered in snow," "A city street illuminated by lights," "A crystal lake surrounded by autumn trees," and "A small house in a wooden at sunset." These are generated by the 1.3B LatentLM model. The images show plausible compositions and lighting, but the paper provides no comparison images from baselines for qualitative assessment, and no human evaluation metrics (e.g., preference rates) to complement the automatic metrics.

Text-to-Speech Synthesis

The headline result in Table 4 is that LatentLM at frame rate 15 achieves speaker similarity (SIM) of 0.697 and WER-C of 1.2 for zero-shot TTS with reference utterances, outperforming VALL-E 2 (SIM 0.643, WER-C 1.5), Voicebox (SIM 0.662), and MELLE (SIM 0.625, WER-C 1.5). At frame rate 7.5, LatentLM achieves SIM 0.656, still matching VALL-E 2 while requiring 10× fewer autoregressive steps (7.5 vs. 75).

Reference utterance setting (Table 4, left columns). LatentLM at frame rate 15 achieves the best results across all metrics: SIM 0.697 (vs. ground truth 0.779), WER-C 1.2 (vs. ground truth 1.6), WER-H 1.8 (vs. ground truth 2.2). The 0.697 SIM is notably higher than VALL-E 2's 0.643 (+0.054) and Voicebox's 0.662 (+0.035). At frame rate 7.5, SIM drops to 0.656—still above VALL-E 2 (0.643) and Voicebox (0.662 is slightly higher). At frame rate 3.75, SIM drops further to 0.598 and WER-C rises to 1.7—performance degrades but remains competitive with some baselines (MELLE: SIM 0.625).

3-second prefix as prompt (Table 4, right columns). This is a more challenging setting where only 3 seconds of speech are provided as conditioning. LatentLM at frame rate 15 achieves SIM 0.571 and WER-C 1.4, substantially outperforming VALL-E 2 (SIM 0.504, WER-C 1.6) and MELLE (SIM 0.508). At frame rate 7.5, SIM is 0.532 with WER-C 1.6—still competitive. At frame rate 3.75, performance degrades significantly (WER-C 3.1, SIM 0.467). The gap between the 3-second prefix and reference utterance settings is smaller for LatentLM than for baselines (SIM drops by 0.126 for LatentLM at frame rate 15 vs. 0.139 for VALL-E 2 vs. 0.069 for Voicebox), suggesting LatentLM handles limited speaker context relatively well.

Speech tokenizer reconstruction quality (Table 5). σ-VAE with latent dimension 64 at compression ratio 3200× (frame rate 7.5) achieves Mel Distance 0.798, PESQ 2.756, STOI 0.929, VISQOL 4.289, and UTMOS 3.505. This is comparable to or better than discrete codecs at dramatically lower compression ratios. For reference:

Encodec at 10× compression: Mel Distance 0.823, PESQ 3.591, STOI 0.962 (σ-VAE achieves similar or slightly worse metrics at 320× higher compression)
DAC at 10× compression: Mel Distance 0.355, PESQ 4.424 (this uses 32 quantizers—σ-VAE with a single continuous channel achieves worse but competitive reconstruction)
DAClow at 160× compression: Mel Distance 0.916, PESQ 2.269 (σ-VAE at 3200× compression achieves better reconstruction despite 20× higher compression)
Mimi at 480× compression: Mel Distance 1.458, PESQ 1.568 (σ-VAE at 3200× achieves much better quality at 6.7× higher compression)

The pattern is clear: continuous σ-VAE achieves competitive or superior reconstruction quality at compression ratios 10–80× higher than discrete codecs. At 6400× compression (σ-VAE128, frame rate 3.75), Mel Distance is 0.852, PESQ 2.533, STOI 0.916—comparable to discrete codecs at 40–160× compression. The paper notes that at 6400× compression, "the resulting sequence length when used in a language model is already comparable to BPE tokenization, approaching a 1:1 ratio."

CFG scale ablation (Figure 10a). For zero-shot TTS with reference utterances, classifier-free guidance scales from 1 to 16 are evaluated (with 5 diffusion head sampling steps). CFG=4 yields the best performance: SIM peaks at approximately 0.69, and WER-C reaches its minimum at approximately 1.2. Without CFG (scale=1), performance is substantially worse (SIM ~0.55, WER-C ~2.8). At CFG=8, SIM drops slightly to ~0.67. At CFG=16, both SIM and WER-C degrade. This follows the standard CFG pattern: moderate guidance improves fidelity to the conditioning signal, but excessive guidance causes over-saturation and artifacts.

Inference sampling steps ablation (Figure 10b). With CFG=4, the number of diffusion head sampling steps per continuous token is swept from 1 to 20. 3 steps already achieve competitive results (SIM ~0.68, WER-C ~1.3). 5 steps provide the sweet spot (SIM ~0.69, WER-C ~1.2) with good efficiency. Performance plateaus from 5 to 20 steps with only marginal improvements. This demonstrates that the DPM-Solver acceleration is effective—the TTS system achieves strong performance with only 3–5 denoising steps per frame, making the total computation dominated by the 15 autoregressive Transformer steps per second of speech rather than the diffusion head's iterations.

Compression ratio vs. latent dimension ablation (Table 6). At compression 640× (frame rate 37.5, latent dimension 16), reconstruction SIM is 0.866 and TTS SIM is 0.655. At compression 1600× (frame rate 15) with the same latent dimension 16, reconstruction SIM drops to 0.700 and TTS SIM drops to 0.545. However, at compression 1600× with latent dimension 32, reconstruction SIM recovers to 0.870 and TTS SIM to 0.661—better than the 640× configuration despite 2.5× higher compression. This reveals that the information capacity per latent vector (controlled by dimension) matters more than the frame rate for quality. The practical implication: by increasing the latent dimension from 16 to 32, the model can use a lower frame rate (faster generation) while maintaining or improving quality. At compression 6400× with latent dimension 128 (Table 5), reconstruction remains strong (Mel Distance 0.852), suggesting even more aggressive compression is viable with sufficient latent capacity.

Ablation Studies and Robustness Checks

σ-VAE tokenizer variance (Figure 6, Section 3.1.3): Training LatentLM with tokenizers of different fixed variances shows that larger variance monotonically improves FID under no classifier-free guidance (CFG=1.0). Tokenizers tuned for latent diffusion models (small variance) produce FID ~40–50 versus FID ~20–25 for explicitly high-variance σ-VAE tokenizers. Under DiT, variance choice is irrelevant—all tokenizers achieve similar FID. This directly validates the core claim that variance collapse harms autoregressive generation specifically. Under CFG > 1.0, the relationship becomes non-monotonic with an optimal variance range, suggesting an interaction between guidance scale and latent robustness that warrants further investigation.

DiT vs. LatentLM scaling with model size (Figure 4): Four model sizes (455M, 1.03B, 1.82B, 3.68B) trained for 75K steps each. LatentLM achieves consistently lower FID at every size, with the gap widening at larger scales. At 3.68B: LatentLM FID ~3.5 vs. DiT FID ~5.0. This demonstrates that the autoregressive formulation scales more favorably than full-sequence diffusion in terms of parameter efficiency. Both models use 20 DPM-Solver inference steps.

Inference throughput with model size and batch size (Figure 7, Figure 11 in Appendix C): LatentLM achieves 2.47–2.84× higher throughput than DiT at equivalent model sizes, with the gap widening for larger batch sizes and larger models. Group-query attention (GQA) further improves LatentLM's throughput. The KV cache enables LatentLM's throughput to scale with batch size, while DiT's throughput remains flat. At 9.35B parameters, LatentLM still maintains a 2–3× advantage.

Multimodal LLM training token scaling (Figure 8): As training tokens increase from ~25B to ~200B, LatentLM's text-to-image FID and image-to-text perplexity improve more consistently than Transfusion and VQ-MLLM. VQ-MLLM's FID saturates, and Transfusion's improves but slower than LatentLM's. The validation perplexity curves (Figure 8b) show LatentLM below Transfusion by a small but persistent margin, and well below VQ-MLLM.

σ-VAE compression ratio and latent dimension for TTS (Table 6): Increasing latent dimension from 16 to 32 at 1600× compression improves SIM from 0.545 to 0.661 and WER-C from 1.6 to 1.5, surpassing the 640×/16-dim configuration (SIM 0.655, WER-C 1.4—the latter slightly better). This shows that latent capacity can compensate for reduced temporal resolution—a critical finding for deployment efficiency because lower frame rates mean fewer expensive autoregressive Transformer steps.

CFG scale for TTS (Figure 10a): CFG=4 is optimal. CFG=1 (no guidance) severely degrades performance (SIM ~0.55 vs. ~0.69). CFG scales ≥8 cause gradual degradation, consistent with standard CFG behavior where excessive guidance amplifies artifacts. The paper does not report whether the optimal CFG scale is sensitive to frame rate or latent dimension—a relevant ablation given that different latent configurations produce different distribution statistics.

Diffusion head inference sampling steps for TTS (Figure 10b): 3 steps achieve competitive results; 5 steps provide the sweet spot of quality and efficiency; >10 steps yield marginal returns. This is important because each continuous token requires T_inf head forward passes—keeping T_inf low is essential for the overall efficiency advantage over full-sequence diffusion. The paper does not explore whether fewer training diffusion steps (with distillation) could reduce the gap between training (1000 steps) and inference (3–5 steps), potentially improving quality further.

Speech tokenizer reconstruction quality across compression ratios (Table 5): σ-VAE at 6400× compression achieves Mel Distance 0.852, comparable to discrete codecs at 10–40× compression. Increasing compression from 1600× to 6400× causes only modest degradation (Mel Distance: 0.813 → 0.798 → 0.852 for latent dimensions 32, 64, 128 respectively). This robustness to extreme compression is a strong validation of continuous representations for high-fidelity data.

Classifier-free guidance scale for image generation (Section 3.1.1): The main LatentLM-L model uses CFG=1.65, while scaling experiments use CFG=1.75. The choice is stated but not ablated. The interaction between CFG and tokenizer variance (visible in Figure 6, where the monotonic improvement with variance only holds at CFG=1.0) suggests that further tuning could improve results, but this is not explored systematically.

Critical Assessment

The experimental results broadly support the paper's central claims, but several important caveats and missing comparisons temper the strength of the conclusions.

Claim: "LatentLM surpasses Diffusion Transformers in both performance and scalability." The evidence for this claim comes from the scaling curves in Figure 4, where LatentLM achieves lower FID than DiT at every model size tested (455M–3.68B). However, these models are trained for only 75,000 steps (~120 epochs)—well short of convergence (the full LatentLM-L model in Table 1 is trained for 250,000 steps / ~400 epochs). Scaling behavior at partial convergence may not reflect scaling behavior at full convergence. It is possible that DiT would catch up or close the gap with sufficient training, particularly since diffusion models often benefit from extended training. A convergence-matched comparison (e.g., training both to saturation at each model size) would strengthen the scaling claim substantially. Additionally, the claim of "surpassing" at the full training budget is not directly tested: the Table 1 results show LatentLM-L (FID 2.24) essentially matching DiT-XL/2 (FID 2.27), which is a more modest claim of parity rather than surpassing. The 2.47× throughput advantage (Figure 7) is a separate axis—LatentLM achieves similar quality with much faster inference, which is a genuine practical advantage but different from "surpassing in performance."

Claim: "LatentLM provides a general-purpose interface that unifies multimodal generation and understanding." The multimodal LLM experiments (Section 3.2, Table 3) demonstrate that a single LatentLM model can simultaneously perform text generation (perplexity 2.73), image generation (FID 14.54), image captioning (CIDEr 54.5), and visual question answering (VQAv2 38.72). This is genuine unification—one model, one architecture, one training procedure. However, the paper does not demonstrate the interleaved generation capability that the architecture theoretically supports. The experiments evaluate each task separately (text-to-image, image-to-text, text-only), not in combination. The claim of a "general-purpose interface" implies that the model can, for example, generate an image and then answer questions about it, or follow multimodal instructions like "describe this image and then generate a similar one in a different style." No such demonstrations or evaluations are provided, nor are metrics like multimodal dialogue performance. The theoretical capability exists, but the experimental validation is limited to isolated task evaluations. The Transfusion comparison suffers from an asymmetry: Transfusion's understanding degradation from noise-addition training is plausible but not directly ablated. An experiment where Transfusion is trained without the denoising objective (or with a frozen image encoder) and evaluated on understanding tasks would isolate whether the noise-addition or some other factor causes the gap. Without this, the causal attribution to training-inference mismatch remains suggestive rather than definitive.

Claim: "LatentLM outperforms Transfusion and vector quantized models in the setting of scaling up training tokens." Figure 8 provides clear evidence that LatentLM scales more favorably than VQ-MLLM (whose FID saturates) and modestly better than Transfusion (whose FID improves but slower). However, the "outperforms" claim at the 200B token scale relies on the specific configuration choices for baselines. VQ-MLLM uses the LlamaGen VQ-VAE tokenizer—different VQ-VAE configurations (larger codebook, different compression ratio, different training recipe) might scale differently. The saturation of VQ-MLLM might be specific to that tokenizer's information capacity rather than fundamental to all discrete representations. A systematic sweep of VQ-VAE configurations would be needed to claim that continuous representations fundamentally scale better, rather than that one specific VQ-VAE configuration was suboptimal. Similarly, Transfusion's gap to LatentLM may be partly due to the 6-layer ViT image head configuration—different head architectures or capacities might change the scaling behavior. The paper's ablation space for baselines is narrow, which makes the comparative claims about scaling somewhat fragile.

Claim: "LatentLM outperforms VALL-E 2 in speaker similarity and robustness, while requiring 10× fewer decoding steps." This is unambiguously supported by Table 4. At frame rate 7.5, LatentLM achieves SIM 0.656 (vs. VALL-E 2's 0.643) with 10× fewer autoregressive steps—both better quality and dramatically better efficiency. At frame rate 15, the SIM margin is even larger (0.697 vs. 0.643). The robustness evidence comes from the smaller degradation in the 3-second prefix setting. However, the paper does not report inference latency or total FLOPs for TTS—the "10× fewer decoding steps" refers only to autoregressive Transformer forward passes, not the total computation including diffusion head steps. With T_inf = 5 diffusion head steps per frame, the total number of refinement operations is 7.5 × 5 = 37.5 per second of speech for the frame rate 7.5 model. The relative efficiency advantage depends on the computational cost ratio between a Transformer forward pass and a diffusion head forward pass. If the diffusion head is 10% the cost of the backbone (plausible given 3 vs. 24 layers), the total cost per second would be approximately 7.5 × (1.0 + 5 × 0.1) = 11.25 backbone-equivalent operations. VALL-E 2 requires 75 + NAR_steps operations. The paper does not quantify the NAR cost, making a complete efficiency comparison incomplete. Additionally, the evaluation is on the LibriSpeech test-clean set—a relatively clean, read-speech corpus. Robustness to diverse speakers, accents, background noise, and expressive speech is untested, limiting the "robustness" claim to a narrow operational definition (SIM and WER on clean data).

Missing experiments that would strengthen the paper:

FLOPs-matched inference quality comparison. The throughput comparison (Figure 7) shows LatentLM is faster, but what if DiT is given equivalent inference FLOPs by reducing denoising steps (e.g., 250 → 50 → 20 → 5)? How does the FID vs. FLOPs curve compare? This is the standard way to evaluate efficiency in generative models and is notably absent.
Interleaved generation evaluation. The architecture supports generating text → image → text sequences. Demonstrating this capability with metrics (e.g., multimodal dialogue tasks, image editing via natural language followed by regeneration) would substantiate the "general-purpose interface" claim.
Multiple training runs with error bars. All results are single-run. For the multimodal LLM comparison where differences are small (PPL 2.73 vs. 2.74), variance across random seeds could change the ranking. Error bars or confidence intervals are especially important for the scaling curves.
Human evaluation for image and speech generation. FID, CLIP score, SIM, and WER are all automatic metrics. Human preference studies would provide a more robust quality assessment, particularly for detecting artifacts that automatic metrics miss.
Ablation of the Transfusion noise-addition mechanism. Training Transfusion without adding noise to images during training (or with a separate frozen image encoder for understanding tasks) would isolate whether the noise-addition training is genuinely the cause of Transfusion's understanding gap, or whether other architectural differences (like the separate ViT image head) matter more.
VQ-VAE configuration sweep for multimodal LLMs. Testing VQ-MLLM with different codebook sizes, compression ratios, and training recipes would establish whether the observed saturation is fundamental to discrete representations or specific to the LlamaGen tokenizer configuration.
Zero-shot evaluation beyond LibriSpeech. Evaluating TTS on out-of-domain speakers, noisy conditions, expressive speech, or cross-lingual settings would test the claimed robustness more thoroughly.
Full convergence training for scaling comparison. Training both LatentLM and DiT to convergence at each model size (not just 120 epochs) would provide a more definitive scaling law comparison.

Conditional nature of the claims. The paper's central efficiency advantage—that per-token diffusion is faster than full-sequence diffusion—holds unconditionally for autoregressive generation because it is an architectural property (KV caching vs. no caching). However, the quality parity and scaling advantages are conditional: they hold on ImageNet at the specific training budgets tested, on MS-COCO at the 200B-token scale with the specific model configurations used, and on LibriSpeech test-clean with the specific σ-VAE configurations. Whether these advantages persist on other datasets, at larger scales, with different hyperparameters, or with better-tuned baselines is an open question. The paper's claim that continuous representations "scale better" than discrete ones (Figure 8a) is demonstrated with one VQ-VAE configuration; the claim that LatentLM "outperforms DiT in scaling" (Figure 4) is demonstrated with partially trained models. These are suggestive rather than conclusive, and the paper would benefit from more cautious framing of these scaling claims.

6. Limitations and Trade-offs

The Cost of Difficulty Estimation Is Not Factored Into Efficiency Claims

The assumption or constraint. LatentLM's autoregressive generation relies on the σ-VAE tokenizer having sufficient latent variance to withstand exposure bias — small prediction errors during sequential decoding that accumulate and push generated vectors outside the distribution the decoder expects. The paper identifies variance as the critical control variable (Section 2.3, Figure 6) and proposes σ-VAE with a tunable scalar variance C_σ. However, the paper provides no method for automatically selecting the optimal variance for a given application, model size, or training budget. The variance is set as a fixed hyperparameter σ ∼ N(0, C_σ) where C_σ is chosen manually. The experiments treat variance as a dial to be swept (Figure 6 sweeps fixed σ values; Table 6 sweeps latent dimension and compression ratio combinations), but at deployment time, a practitioner must commit to a specific variance without knowing whether it is optimal for their data distribution or model scale.

The paper does not explicitly acknowledge this as a limitation — rather, it presents the tunability of σ-VAE variance as a feature ("the variance is easily controllable," Section 3.1.3). But tunability that requires expensive empirical sweeping is a practical burden, not a solved problem. In the image generation experiments, the variance sweep in Figure 6 required training multiple σ-VAE tokenizers (each taking 200 epochs on ImageNet, per Section 3.1.3) and then training full LatentLM models on top of each one. The computational cost of this sweep is substantial and is not included in any reported training budget.

The consequence. A practitioner deploying LatentLM to a new domain (e.g., medical images, music, video) cannot know a priori what tokenizer variance will work well. Too little variance, and the model suffers from exposure bias — autoregressive errors cascade, and generated quality degrades (the "stars" in Figure 6 show FID of ~40-50 for low-variance tokenizers versus ~20-25 for well-tuned ones, a 2× degradation). Too much variance, and reconstruction fidelity may suffer because the decoder must handle an unnecessarily wide input distribution. Under classifier-free guidance (CFG > 1.0 in Figure 6), the relationship becomes non-monotonic — there is an optimal variance range, and exceeding it degrades performance. This means variance tuning is not a simple "more is better" problem when CFG is used (which it is, in all the paper's best reported results — CFG=1.65 for ImageNet, CFG=4 for TTS).

Furthermore, the optimal variance likely interacts with other design choices that the paper does not explore: model size (larger models may be more robust to exposure bias and thus tolerate lower variance), training data scale (more data may allow the decoder to handle higher variance), compression ratio (higher compression means fewer autoregressive steps, reducing accumulated exposure bias, potentially reducing the need for high variance), and the specific modality (speech may have different variance requirements than images). These interactions are not characterized, leaving practitioners to treat variance as a hyperparameter to be tuned for each new setting, with no guidance beyond "don't use tokenizers designed for diffusion models" (Section 3.1.3).

What evidence exists in the paper. Figure 6 provides the primary evidence: the same tokenizer achieves dramatically different FID under LatentLM depending on its variance (ranging from ~50 to ~20 FID), while under DiT the choice is irrelevant. This establishes that the problem exists and that σ-VAE addresses it, but does not establish how to choose C_σ without sweeping. The TTS ablation (Table 6) shows that increasing latent dimension from 16 to 32 at 1600× compression improves reconstruction SIM from 0.700 to 0.870 and TTS SIM from 0.545 to 0.661 — demonstrating that latent capacity and variance interact, but without isolating variance as a separate variable. The paper does not report any experiment where C_σ is varied for the TTS tokenizer or the multimodal LLM tokenizer, leaving uncertainty about whether the image-domain findings generalize.

Mitigation status. The paper partially acknowledges the issue implicitly through its recommendation to "re-trained σ-VAE as tokenizers for LatentLM, rather than directly using previous ones" (Section 3.1.3). However, this is advice to not reuse existing tokenizers, not guidance on how to design new ones. The paper provides no automated method for selecting variance, no characterization of how variance requirements scale with model size or modality, and no lightweight proxy task that could predict whether a given tokenizer's variance is sufficient without full model training. The future work section (Section 4) does not mention variance selection as an open problem. This remains a practical barrier to adoption that the paper does not address.

Hard Problems Remain Unsolved — The Architecture Cannot Compensate for Fundamental Capability Gaps

The assumption or constraint. LatentLM unifies the generation mechanism for discrete and continuous data, but it does not change the fundamental relationship between model capacity, training data, and problem difficulty. The architecture assumes that the base model's representations are sufficient to support generation — the autoregressive paradigm amplifies existing capabilities through consistent training and efficient inference, but cannot create capabilities that the training data and model scale do not provide.

This is most visible in the multimodal LLM experiments (Section 3.2, Table 3). The 1.3B LatentLM model achieves VQAv2 accuracy of 38.72 — a substantial improvement over Transfusion (35.36) and VQ-MLLM (30.19), but still far below what much larger vision-language models achieve. The paper acknowledges its scale limitations implicitly by noting the training budget (200B tokens, 1.3B parameters), but does not characterize the capability ceiling of this scale. There is no experiment that shows whether these gaps would close with larger models or more data, or whether they reflect architectural limitations that scale cannot fix.

The consequence. The paper's architecture excels at what the model already knows how to do: generate images of categories it has seen, produce speech in voices it has learned, answer questions about visual content within its training distribution. But it provides no mechanism for systematic generalization beyond the training distribution. For a multimodal LLM deployed in practice, this means the model may produce fluent, well-formatted outputs that are factually wrong or visually implausible when asked about novel compositions or rare entities. The unified architecture does not address the fundamental challenge of compositional generalization — it makes the generation process more efficient and consistent, but does not change what the model can represent.

For image generation specifically, the evaluation is on class-conditional ImageNet — a setting where the model only needs to generate within 1000 known categories. The paper does not evaluate open-ended text-to-image generation with LatentLM as a standalone model (the MS-COCO FID results in Table 3 come from the multimodal LLM, not a dedicated text-to-image model). It is unclear whether the architectural advantages over DiT (better scaling, faster inference) would persist for open-ended generation where the conditioning signal is a free-form text description rather than a class label. The causal autoregressive generation might struggle with global coherence for complex scenes in ways that full-sequence diffusion (which can refine all patches jointly) does not — but this is untested.

For speech, the evaluation is on LibriSpeech test-clean (read audiobook speech from known speakers in clean conditions). The paper does not evaluate on accented speech, emotional speech, noisy conditions, conversational speech, or cross-lingual transfer. The "robustness" claimed in the abstract refers only to WER and SIM metrics on this clean test set, not to distributional robustness.

What evidence exists in the paper. The multimodal LLM results (Table 3) show VQAv2 at 38.72 and MS-COCO CIDEr at 54.5 — these are respectable but not state-of-the-art numbers for 1.3B-parameter models, and the paper does not compare against larger discrete-token multimodal models (e.g., Chameleon [Tea24] at 7B+ parameters) that might achieve better absolute performance despite the architectural disadvantages. The ImageNet results (Table 1) show FID 2.24 — competitive but behind MAR-L (FID 1.78), which uses a non-causal architecture. The TTS results (Table 4) are evaluated only on LibriSpeech test-clean. The paper does not include evaluations on out-of-distribution or compositional generalization tasks for any modality.

Mitigation status. The paper does not address this limitation. The future work section (Section 4) discusses scaling to video, world modeling, embodied AI, and cross-modal transfer — all of which depend on the same fundamental capability question. It does not propose experiments to characterize the scaling trajectory toward compositional generalization or the relationship between model scale and the types of errors that persist. A practitioner evaluating LatentLM for a production system would need to independently assess whether the 1.3B scale is sufficient for their use case, with no guidance from the paper on what scales of model and data would be needed.

The Multimodal LLM Training Data Mixture Is Complex and Its Sensitivity Is Uncharacterized

The assumption or constraint. The multimodal LLM experiments (Section 3.2) use a carefully constructed training data mixture: text-only data, image-text pairs, and interleaved image-text documents in a 2:1:1 ratio. This ratio is stated as a fixed configuration with no ablation. The text data sources (Common Crawl, RefinedWeb, StarCoder) and image-text sources (LAION-2B, LAION-400M, COYO-700M, Conceptual Captions) are also specified but not ablated. The paper implicitly assumes that this mixture generalizes — that the 2:1:1 ratio is a reasonable default that would work for other modalities, other model sizes, or other application domains.

The consequence. The performance of multimodal LLMs is known to be sensitive to the data mixture ratio. Too much text data, and the model may underinvest in visual representations. Too much image-text data, and text generation quality may degrade (the model "forgets" language). Interleaved data is included specifically to enable the multimodal reasoning capabilities that the architecture theoretically supports, but the paper never demonstrates these capabilities — the evaluation is on isolated tasks (text-to-image, image-to-text, text-only). A practitioner replicating this work would need to either copy the exact data mixture (which depends on access to the same datasets and filtering pipelines) or independently tune the ratio — an expensive undertaking given that each configuration requires training a model to 200B tokens to evaluate.

The undisclosed hyperparameters compound this problem. The paper mentions the loss weight α for the diffusion objective (L_total = L_LM + α · L_Diff) in Section 2.2, but never reports its value for the multimodal LLM experiments. This hyperparameter controls the relative importance of continuous generation quality versus discrete text quality — getting it wrong could mean the model generates blurry images or nonsensical text. Without this value, reproducing the results requires guessing or sweeping.

Additionally, the special tokens <BOD> and <EOD> that switch between the softmax head and the diffusion head (Section 2.2) are mentioned but their exact format, how they are inserted into training sequences, and how they interact with the tokenizer are not specified. For interleaved data, the construction of training sequences from web documents (e.g., how many images per document, how they are interleaved with text, whether truncation is applied) is not described in sufficient detail for reproduction.

What evidence exists in the paper. Table 3 reports the main results, and Figure 8 shows scaling curves over training tokens — but these are all at the fixed 2:1:1 ratio. There is no experiment that varies the ratio (e.g., 1:1:1, 3:1:1, 1:2:1) and measures the effect on language perplexity, image generation FID, and image understanding metrics. There is no ablation that removes interleaved data entirely and measures whether it contributes to the understanding improvements versus the generation improvements. The sensitivity of the results to α is not characterized. The paper's appendices (Appendix D) provide hyperparameters but do not include α, <BOD>/<EOD> formatting, or data construction details at the level needed for reproduction.

Mitigation status. The paper does not acknowledge this as a limitation. The data mixture is presented as a fixed experimental configuration, not as a design choice that requires validation. The lack of data mixture ablations is a significant gap because it means a practitioner cannot determine whether LatentLM's advantages over Transfusion and VQ-MLLM (Table 3) are due to the architecture, the data mixture, or an interaction between them. If Transfusion or VQ-MLLM were trained with a different data mixture (optimized for their respective architectures), the comparative results might change. This is not a hypothetical concern — different architectures often have different optimal data mixtures, and the paper's choice to fix the mixture for all baselines may advantage LatentLM if its architecture happens to be more compatible with the 2:1:1 ratio.

Throughput Advantages Do Not Imply Latency Advantages — the Serial Autoregressive Bottleneck Remains

The assumption or constraint. The paper reports inference throughput (images per second) in Figure 7 and Appendix C, demonstrating 2.47-2.84× improvements over DiT. Throughput measures how many samples the model can process in a given time when running a large batch — it is the relevant metric for offline, batch generation (e.g., generating a dataset of images). However, the paper does not report latency — the time to generate a single sample. Latency is the relevant metric for interactive applications (dialogue, real-time speech synthesis, image editing), where a user is waiting for a single output and cannot amortize computation across a batch.

The autoregressive architecture has an inherent latency disadvantage compared to full-sequence diffusion for generation tasks where the output has many tokens. Each continuous token requires one Transformer forward pass (which can reuse the KV cache, reducing per-token computation) plus T_inf diffusion head forward passes. The total generation time for an image with N latent patches is approximately N × (T_backbone_per_token + T_inf × T_head), where T_backbone_per_token is the time for one autoregressive step with KV cache. For speech with F frames per second, the latency to generate S seconds is F × S × (T_backbone_per_token + T_inf × T_head). In both cases, latency grows linearly with sequence length, and the autoregressive nature means tokens must be generated sequentially — there is no parallelism across sequence positions during generation.

In contrast, full-sequence diffusion (DiT) generates all image patches simultaneously: latency is T_diff_steps × T_full_backbone, which does not grow with the number of patches (though T_full_backbone itself grows quadratically with sequence length due to self-attention). For short sequences (low-resolution images, short speech clips), DiT's latency might be lower than LatentLM's because the Transformer processes all positions in parallel. For long sequences, the O(N²) attention cost of bidirectional processing eventually dominates, but the crossover point is not characterized.

The consequence. For interactive applications, a practitioner evaluating LatentLM would need to know the latency per generated output — not just the throughput at batch size 128 or 256. The paper's TTS results (Table 4) are the closest to a latency-sensitive application, but the paper reports only the number of autoregressive steps (frame rate), not the wall-clock time to generate 1 second of speech. At frame rate 15 with T_inf = 5 diffusion head steps, the total operations per second of speech are 15 × T_backbone + 75 × T_head. At frame rate 7.5, it's 7.5 × T_backbone + 37.5 × T_head. Without knowing the ratio of T_backbone to T_head, we cannot compute whether real-time generation is feasible (latency < 1 second for 1 second of speech). The paper's claim of "10× fewer decoding steps" refers only to autoregressive steps, not to total latency.

For image generation, the 256×256 ImageNet setting uses 256 latent vectors (16×16 grid of patches). Generating these requires 256 sequential autoregressive steps. At 20 diffusion head steps per token, the latency might be competitive with DiT's 20 full-model forward passes if each autoregressive step (with KV cache) is much faster than a full bidirectional forward pass — but this is an empirical question the paper does not answer. At 384×384 resolution (Table 2), the sequence length increases to 576 patches, which would increase latency by 2.25× in the autoregressive case. DiT's latency also increases (larger attention matrices), but the relationship between these two scaling behaviors is not measured.

What evidence exists in the paper. Figure 7 reports throughput (images/second) at batch sizes 8-256. Figure 7b shows that LatentLM's throughput improves with batch size (from ~1 image/sec at batch 8 to ~3 images/sec at batch 256 for the 1.82B model), which is exactly the pattern expected when per-sample latency is bottlenecked by sequential autoregressive steps: batching helps throughput by parallelizing across samples, but per-sample latency remains high. DiT's throughput is flatter because its per-sample computation is dominated by the fixed number of full Transformer forward passes. The paper does not report latency for a single sample at batch size 1 — the relevant metric for interactive use. The TTS experiments (Section 3.3) do not report real-time factor at all.

Mitigation status. The paper does not acknowledge the latency-throughput distinction. The efficiency claims are framed in terms of throughput, with no discussion of latency implications. The future work section (Section 4) mentions "scaling up to video and world models" where sequence lengths would be orders of magnitude longer than the settings tested — without addressing whether the serial autoregressive bottleneck would make such generation impractically slow. Techniques to reduce this bottleneck exist (speculative decoding, parallel decoding, non-autoregressive sequence generation) but are not discussed or evaluated.

No Demonstration of Interleaved Multimodal Generation or the "General-Purpose Interface"

The assumption or constraint. The paper's central architectural claim is that LatentLM "provides a general-purpose interface that unifies multimodal generation and understanding" (abstract, Section 1, Figure 2). The architecture supports this: the causal Transformer can process sequences of interleaved discrete and continuous tokens, with special tokens (<BOD>, <EOD>) switching between the softmax head and the diffusion head. This theoretically enables the model to, for example, read a text prompt, generate an image, and then answer questions about the generated image — all within a single autoregressive forward pass without external modules.

However, the paper never evaluates interleaved generation. Every experiment evaluates a single modality transition in isolation: class → image (Section 3.1), text → image or image → text or text → text (Section 3.2), or text + speech prompt → speech (Section 3.3). There is no experiment where the model generates text, then an image, and then more text conditioned on that image. There is no demonstration of multimodal dialogue (e.g., "Generate an image of a cat. Now make it a dog. What color is the animal in the image?"). There is no evaluation of image editing via natural language followed by regeneration. The "general-purpose interface" is an architectural capability, not an empirically demonstrated one.

The consequence. A practitioner drawn to LatentLM by the promise of unified multimodal generation would find that the paper provides no evidence the model can actually perform the unification tasks that pipeline-based approaches cannot. The special tokens <BOD> and <EOD> are mentioned (Section 2.2) but their exact behavior during training and inference is not described. For instance, it is unclear how the model handles the transition from generating an image back to generating text — does it need to "understand" the generated image to produce coherent follow-up text? If the generated image is of poor quality (as most generated images have some artifacts), does the text generation degrade? What happens if the model is asked to generate multiple images in sequence? The paper provides no answers.

This is particularly relevant for the paper's claimed advantages over Transfusion. Transfusion's bidirectional image diffusion means it cannot naturally handle interleaved sequences — the image is generated in a separate denoising process, and re-integrating it into an autoregressive text stream requires additional mechanisms. LatentLM's causal architecture should handle this naturally, making interleaved generation a key differentiator. But without evaluation, the advantage is theoretical rather than demonstrated.

The future work section (Section 4) implicitly acknowledges this gap by listing capabilities that are currently aspirational: "self-reflection can automatically correct produced images, which requires the multimodal language model to understand the generated image without encoding it again," "multimodal-native reasoning enables the model to track the search states via latent vectors, for example, step-by-step plotting the planned trajectory on the image of input map." These are described as future directions, not as capabilities of the current model.

What evidence exists in the paper. None. The paper does not include a single example of interleaved generation. All evaluations are single-step: given input of one modality, produce output of one modality. The multimodal LLM training data includes interleaved documents (Section 3.2.1), which should teach the model the format, but whether this training translates to generation capability is unmeasured. The special tokens are mentioned but their representation in the tokenizer, their effect on the attention mask, and their handling during loss computation are not specified.

Mitigation status. The paper does not treat this as an evaluated capability. It is described as a theoretical property of the architecture, with demonstrations deferred to future work. This is a significant gap because the "general-purpose interface" is the paper's headline positioning (title, abstract, Figure 1, Figure 2) — it is what distinguishes LatentLM from task-specific models — but the evaluation does not match this positioning. A paper making this claim should include at minimum a qualitative demonstration and ideally a quantitative benchmark (e.g., a multimodal dialogue task, an interleaved image-text generation task, or an instruction-following task involving multiple modalities). Without this, the architecture is a promising foundation for future work, but the claim of a "general-purpose interface" is not supported by the experimental evidence.

Single-Model-Family Evaluation Limits the Generality of the Scaling Claims

The assumption or constraint. All experiments in the paper use a single model architecture family: decoder-only causal Transformers with pre-RMSNorm, SwiGLU activations, and specific hyperparameter configurations derived from LLaMA [TLI+23] (Section 2). The image generation experiments use various model sizes (455M to 3.68B), but all are variants of this same architecture. The multimodal LLM uses a 1.3B variant. The TTS model uses a 350M variant. At no point does the paper test whether LatentLM's advantages over baselines transfer to different backbone architectures — different normalization schemes, different attention mechanisms, different activation functions, encoder-decoder architectures, or non-Transformer backbones (e.g., state-space models).

The scaling claims in particular depend on the specific architectural choices. Figure 4 shows that LatentLM scales better than DiT with model size, but this is demonstrated with one architecture (LLaMA-style decoder-only Transformer) versus another (DiT's adaLN-based bidirectional Transformer). The paper's attribution of the scaling advantage to the autoregressive formulation (per-token diffusion with KV caching) rather than to specific architectural details is plausible but not isolated. DiT uses AdaLN conditioning in every Transformer layer to incorporate timestep information, which adds parameters that scale with model size. LatentLM's diffusion head is separate from the backbone, so timestep conditioning parameters do not grow with the backbone. This is an architectural difference orthogonal to the autoregressive-vs-diffusion distinction, and the paper does not ablate it.

The consequence. A practitioner using a different model architecture — for example, a non-Transformer sequence model, or a Transformer with different normalization/activation choices — cannot assume that LatentLM's scaling advantages will transfer. The efficiency advantages from KV caching are specific to causal Transformers and would not apply to architectures that do not use key-value attention. The quality advantages from objective unification are demonstrated only against DiT (for image generation) and Transfusion (for multimodal LLMs), both of which share significant architectural DNA with LatentLM (Transformer backbone, similar normalization, similar training recipes). Against more architecturally diverse baselines (e.g., convolutional diffusion models, GAN-based generation for images, or recurrent architectures for speech), the advantages might not hold or might interact differently.

The multimodal LLM comparison (Section 3.2) is particularly architecture-bound: both LatentLM and Transfusion use a LLaMA-style Transformer backbone, and both use a similar training setup. The key difference is the objective (unified autoregressive vs. split autoregressive/diffusion) and the attention pattern (causal vs. bidirectional for images). The paper attributes the performance differences to these factors, but without testing against architecturally different baselines, alternative explanations (e.g., hyperparameter interactions, optimizer sensitivity, data mixture compatibility) cannot be ruled out.

What evidence exists in the paper. The paper compares against a range of prior work with different architectures (Table 1 includes convolutional models, ViT-based models, masked generative models), but these are prior published results with different training recipes, different data, and different hyperparameters — not controlled comparisons. The controlled comparisons (Figure 4 for DiT vs. LatentLM scaling, Table 3 and Figure 8 for Transfusion and VQ-MLLM vs. LatentLM) use architectures that are aligned to make the comparison as fair as possible (DiT is augmented with RMSNorm and SwiGLU to match LatentLM; Transfusion uses a ViT head to match parameter counts). This alignment is good experimental practice for isolating the objective and attention pattern differences, but it means the results are demonstrated only for this aligned architectural regime. The paper does not test, for example, whether a DiT model with the timestep conditioning moved to a separate head (like LatentLM's diffusion head) would close the scaling gap, or whether a U-Net-based diffusion model with the same latent space would show the same relative performance.

Mitigation status. The paper does not acknowledge architecture-dependence as a limitation. The LLaMA-style backbone is presented as a natural choice ("we adopt pre-RMSNorm and SwiGLU as improvements after LLaMA," Section 2), and the scaling claims are presented as properties of the autoregressive-vs-diffusion distinction, not of the specific backbone. However, the absence of architectural ablations means this attribution is not empirically validated. A practitioner considering a different backbone (e.g., a Mamba state-space model for longer sequences, or a mixture-of-experts Transformer for larger capacity) would need to independently validate whether LatentLM's advantages persist.

7. Implications and Future Directions

How This Work Changes the Landscape

LatentLM causes a methodological reframing of the multimodal unification problem, shifting the question from "can we share model weights across modalities?" (which Transfusion [ZYB+24] answered affirmatively) to "can we share the learning objective and attention pattern so that training signals from different modalities are mutually reinforcing?" This is not a paradigm shift in the sense of overturning established theory — the individual components (autoregressive Transformers, diffusion models, VAEs) are all well-established — but it is a reorganization of the generative model taxonomy that carries substantial practical consequences.

The key conceptual move is demoting diffusion from a full-sequence generative framework to a per-token decoding mechanism. Before this work, the field treated autoregressive models and diffusion models as competing paradigms for different data types — you used autoregressive for discrete tokens and diffusion for continuous arrays, and unifying them meant either quantizing continuous data (making it look discrete) or training separate objectives with shared weights (the Transfusion approach). LatentLM shows that these are not competing paradigms but composable ones operating at different levels of the generation hierarchy: the Transformer handles sequence-level structure (what comes next, in what order), while diffusion handles token-level refinement (transforming a random vector into a specific continuous representation at one position). This is analogous to how a language model's softmax head is not a "competing paradigm" to the Transformer backbone — it is a decoding mechanism that transforms the backbone's hidden state into a probability distribution. Diffusion becomes the continuous analog of softmax: an output mechanism that maps a contextualized hidden state to a data point, with iterative refinement being the continuous equivalent of discrete sampling.

The reframing resolves two tensions in the prior literature that appeared contradictory. First, why do discrete tokenizers saturate while continuous ones do not? The paper provides mechanistic evidence (Figure 8a): VQ-MLLM's FID flattens as training tokens increase because the quantization bottleneck discards information that the autoregressive model needs to improve generation quality further. Continuous representations preserve this information, allowing quality to continue scaling. This is not a quality difference at one scale — it is a scaling ceiling that only becomes visible when you push to larger training budgets. Second, why does autoregressive generation with VAE latents sometimes fail catastrophically while non-autoregressive generation with the same latents works fine? The paper identifies variance collapse as the mechanism (Figure 6): exposure bias from sequential decoding pushes samples outside the narrow distribution the decoder was trained on. The same latents work under DiT because there is no sequential conditioning — all vectors are denoised simultaneously with no error accumulation. This explains why prior attempts at continuous autoregression (e.g., GIVT [TEM23]) underperformed expectations: they used tokenizers optimized for non-autoregressive settings without recognizing variance as a control variable.

The paper also reshapes the pretraining-inference compute tradeoff in a way that is latent in the results but not explicitly framed by the authors. In the conventional diffusion paradigm (DiT), scaling inference requires more FLOPs because more denoising steps mean more full-model forward passes — the quality-compute curve is steep and linear. In LatentLM, inference compute scales primarily with sequence length (number of autoregressive steps), not with denoising steps (which only affect the lightweight head). This means the dominant factor for generation cost is the compression ratio of the tokenizer — how many latent vectors per second of audio or per image. If you can push the compression ratio from 15 vectors/second (TTS frame rate 15) to 3.75 vectors/second (frame rate 3.75), you get a 4× inference speedup with minimal quality loss (Table 4: SIM drops from 0.697 to 0.598, which is substantial but still competitive). This shifts the inference optimization problem from "how many denoising steps can we eliminate?" (the progressive distillation agenda in diffusion models) to "how much can we compress the tokenizer without losing fidelity?" — a different engineering target that the paper's σ-VAE makes newly tractable through explicit variance control.

Certain research directions become more attractive after this paper:

Continuous tokenizer design for autoregressive models is now clearly on the critical path. The paper shows that variance control is the key variable, making σ-VAE-like architectures (with fixed scalar variance) a natural starting point. Research on tokenizer training objectives that explicitly optimize for autoregressive robustness (not just reconstruction fidelity) is newly motivated.
Scaling laws for multimodal autoregressive models become empirically tractable. The paper's scaling curves (Figures 4, 8) show that LatentLM scales favorably with both model size and training tokens, but only at the specific configurations tested. A systematic study analogous to Chinchilla — varying model size, training tokens, tokenizer compression ratio, and training data mixture jointly — would be enabled by the unified architecture (no need to tune separate objectives for each modality).
Real-time multimodal generation becomes a nearer-term target. The TTS results (Table 4: frame rate 7.5 with 5 diffusion head steps achieving competitive quality) suggest that real-time speech generation is feasible. The inference throughput advantages (Figure 7: 2.47-2.84× over DiT) make deployment of larger multimodal models on consumer hardware more plausible.

Conversely, some directions become less attractive:

Vector quantization for autoregressive multimodal models now carries a demonstrated scaling ceiling (Figure 8a). While VQ approaches may remain useful for specific applications (e.g., where discrete codes enable external tool use or interpretable intermediate representations), the paper makes a strong case that they are not the optimal path for scaling generation quality.
Full-sequence diffusion for modalities that are naturally sequential (speech, video, interleaved documents) faces a new efficiency challenge. If per-token diffusion achieves comparable quality with 2-3× better throughput and enables streaming generation, the architectural advantages of bidirectional denoising (global coherence, non-autoregressive generation) need to be weighed against these practical benefits. The paper does not provide a FLOPs-matched quality comparison, but the throughput gap (Figure 7) is large enough to motivate such studies.

Reconciling prior contradictions. The paper's most elegant reconciliation is between the success of diffusion models for continuous data and the success of autoregressive models for discrete data. Prior work treated these as evidence that different modalities need different generative paradigms. LatentLM shows that both success stories are consistent with a single underlying principle: the autoregressive paradigm is optimal for modeling sequence-level dependencies, while diffusion is optimal for modeling token-level continuous distributions. The apparent conflict arose because previous work tried to use diffusion for sequence-level modeling (DiT) or avoid diffusion entirely by quantizing (VQ-VAE), rather than composing the two at their respective appropriate levels. This reconciliation is the paper's most durable conceptual contribution: it provides a template for thinking about generative architectures as hierarchical compositions, where the sequence model handles order and conditioning, and per-token decoders handle local data generation.

Follow-Up Research This Work Enables

FLOPs-matched inference quality comparison between per-token and full-sequence diffusion. The paper demonstrates throughput advantages (Figure 7: 2.47-2.84× faster than DiT) but never matches inference compute budgets. A strong follow-up would train a single σ-VAE tokenizer and then compare LatentLM and DiT under matched inference FLOPs: for each model size, measure the FID-vs-FLOPs curve by varying denoising steps (for DiT) and both autoregressive steps and diffusion head steps (for LatentLM). The key question: at equivalent inference FLOPs, does LatentLM still produce better images, or does the quality advantage stem from using more total compute per sample? This experiment would disentangle the architectural efficiency (KV caching, parameter separation) from the compute budget and establish whether per-token diffusion is fundamentally more compute-efficient or simply trades latency for throughput. The paper already provides the necessary infrastructure — the σ-VAE tokenizer, the aligned DiT baseline (augmented with RMSNorm and SwiGLU), and the DPM-Solver configuration — making this a straightforward controlled experiment.

Interleaved multimodal generation evaluation. The architecture theoretically supports generating text → image → text sequences by switching between softmax and diffusion heads using <BOD> and <EOD> tokens, but this capability is never evaluated. A strong follow-up would design a benchmark for multimodal dialogue or instruction-following where the model must generate interleaved sequences. A concrete proposal: take an existing visual dialogue dataset, reformat it as interleaved generation tasks (e.g., "Describe this image: [IMG]. What color is the object on the left? [TEXT]. Generate an image of that object in a different color: [IMG]. Is it now blue?"), and measure both the quality of generated images (FID, CLIP score) and the coherence of generated text conditioned on generated images. This would stress-test whether the unified objective genuinely enables cross-modal reasoning (the model must understand its own generated images to produce coherent follow-up text) or whether the performance degrades when conditioning on model-generated rather than ground-truth images. The paper's training data already includes interleaved documents (Section 3.2.1), so the model has been exposed to the format — the question is whether this exposure translates to generation capability.

Systematic characterization of the variance-robustness tradeoff. The paper identifies variance as a critical control variable (Figure 6) but only sweeps it for image generation with one model configuration. A strong follow-up would map the relationship between tokenizer variance, autoregressive model robustness, and reconstruction fidelity across multiple modalities and model scales. The experiment: train σ-VAE tokenizers at multiple variance values for images, speech, and (new) video, then train LatentLM models at multiple sizes on top of each, measuring both generation quality (FID, SIM, etc.) and robustness metrics (e.g., FID after injecting controlled errors into the latent vectors to simulate exposure bias). The expected finding: larger models tolerate lower variance (they produce more accurate latent vectors, so exposure bias is smaller), suggesting that the optimal variance is a function of model capacity. This would produce a practical guideline for setting C_σ without expensive sweeping, and would connect the variance-collapse phenomenon to broader questions about model calibration and uncertainty in autoregressive generation.

Combining next-token diffusion with non-autoregressive decoding for latency reduction. The serial autoregressive bottleneck means latency grows linearly with sequence length — a problem for long sequences (high-resolution images, long videos, extended speech). A strong follow-up would explore whether some subset of continuous tokens can be generated in parallel using a non-autoregressive variant of the diffusion head. One concrete approach: train the model to generate k continuous tokens simultaneously by using k parallel diffusion heads, each conditioned on the same shared prefix but denoised independently. Then evaluate the quality-latency tradeoff as k increases. This would determine whether the autoregressive bottleneck can be partially relieved without the architectural complexity of full-sequence diffusion. The paper's modular design (the diffusion head is separate from the backbone) makes this experiment architecturally straightforward — only the head would need to be replicated.

Extension to video generation with temporal σ-VAE compression. The paper briefly mentions video generation as future work (Section 4) but provides no results. A strong follow-up would extend σ-VAE to video by adding temporal downsampling (e.g., 3D convolutions or temporal attention) to compress along both spatial and temporal dimensions, then train a LatentLM to autoregressively generate video frames. The key research question: does the autoregressive paradigm maintain temporal consistency better than full-sequence diffusion for video, or does the causal constraint (frames can only attend backward in time) create drift that degrades long-term coherence? The experiment would compare LatentLM against a video diffusion baseline (e.g., a DiT variant with temporal attention) at matched model size and training budget, measuring FVD (Fréchet Video Distance) and human preference for temporal consistency. The paper's efficiency advantages (KV caching, per-token diffusion) are most impactful for long sequences, making video a natural stress test.

Stress-testing the scaling ceiling of discrete vs. continuous representations. The paper claims VQ-MLLM saturates because of the quantization bottleneck (Figure 8a), but tests only one VQ-VAE configuration (LlamaGen's tokenizer). A strong follow-up — and potentially a negative result that would refine the paper's claims — would sweep VQ-VAE configurations (codebook sizes from 1024 to 65536, compression ratios from 4× to 256× per dimension, with and without entropy coding) and train both VQ-MLLM and LatentLM at matched parameter counts on the same data mixture. The question: is there a VQ configuration that matches LatentLM's scaling curve, or is saturation fundamental to any discrete bottleneck? If a sufficiently large codebook (e.g., 64K entries with only 4× compression) closes the gap, then the paper's claim about continuous representations scaling better is really a claim about information capacity per token — and discrete tokens can achieve similar capacity with enough codebook size, at the cost of a larger softmax head. If no configuration closes the gap, the result strengthens the paper's architectural argument. Either outcome is scientifically valuable: the former would provide a practical guideline for when VQ is still viable, and the latter would establish a fundamental limitation of discretization.

Practical Applications and Downstream Use Cases

On-device speech synthesis with memory-constrained hardware. The combination of high compression ratios (3.75-15 frames per second, Table 4) and lightweight diffusion heads (3-5 denoising steps per frame, Figure 10b) makes LatentLM viable for on-device text-to-speech where both memory and compute are limited. At frame rate 7.5 with 5 head steps, generating 1 second of speech requires 7.5 Transformer forward passes plus 37.5 lightweight head forward passes. For the ~350M parameter TTS model (Appendix E), the backbone is roughly 350M parameters and the diffusion head is 3 feedforward layers — small enough to fit on a mobile GPU. The 10× reduction in autoregressive steps compared to VALL-E 2 (75 vs. 7.5) directly translates to lower latency and lower energy consumption, which are the binding constraints for on-device deployment. The LibriSpeech results (SIM 0.656, WER-C 1.2 at frame rate 7.5) demonstrate quality sufficient for voice assistant applications. A practical deployment would need to add streaming support (the convolutional σ-VAE encoder already supports this, Section 3.3.1) and optimize the Transformer for mobile inference (quantization, pruning), but the paper provides the architectural foundation.

Cost-efficient batch image generation for synthetic data pipelines. The 2.47-2.84× throughput advantage over DiT (Figure 7, at model sizes 1.8B-3.7B) directly translates to cost savings in batch generation scenarios: generating synthetic training data for downstream vision models, creating assets for game development, or producing design variations in industrial settings. At 3.68B parameters with batch size 128, LatentLM produces 2.47× more images per GPU-hour than a comparably sized DiT model. For a pipeline generating 10 million images, this is the difference between 1,000 GPU-hours and 2,470 GPU-hours. The ImageNet results (FID 2.24 at 256×256, FID 2.51 at 384×384, Tables 1 and 2) demonstrate quality competitive with the best diffusion models, meaning the efficiency gain does not come at the cost of visual fidelity. LatentLM also benefits from group-query attention (GQA, Figure 7b) which reduces memory usage — relevant when running multiple generation workers on a single GPU. The practical deployment would use the pretrained LatentLM-L model (479M parameters, weights released at https://aka.ms/next-token-diffusion) with classifier-free guidance at scale 1.65, directly applicable to class-conditional generation. For text-to-image, the multimodal LLM (1.3B parameters, Section 3.2) provides the foundation, though the paper's FID 14.54 on MS-COCO suggests room for improvement before production deployment.

Unified multimodal assistants with shared internal representations. The 1.3B multimodal LLM (Section 3.2) demonstrates simultaneous competence in text generation (PPL 2.73), image generation (FID 14.54), and image understanding (VQAv2 38.72, CIDEr 54.5) from a single model with a single training procedure. For a practitioner building a multimodal assistant (e.g., a customer support bot that can both read product images and generate diagrams in response), this eliminates the need to compose separate text, vision, and generation models with text-based interfaces between them. The practical benefit is not just reduced integration complexity but shared representations: the model's understanding of visual concepts (learned through captioning and VQA) directly informs its image generation, and vice versa. The paper's evidence for this comes from the comparison with Transfusion (Table 3): LatentLM achieves both better image generation (FID 14.54 vs. 16.10) and better image understanding (VQAv2 38.72 vs. 35.36), suggesting the unified objective enables positive transfer between modalities rather than interference. The practical deployment would need instruction-tuning on multimodal dialogue data (the current model is pretrained only), scaling to larger model sizes (1.3B is below the threshold for robust instruction following), and safety filtering for generated images, but the pretrained checkpoint provides the foundation.

Streaming speech-to-speech translation with low latency. While the paper demonstrates text-to-speech (Section 3.3), the architecture naturally extends to speech-to-speech tasks by chaining the σ-VAE encoder (which processes input speech into latent vectors) with the causal Transformer (which autoregressively generates output speech latents) and the σ-VAE decoder. The streaming capability of the convolutional tokenizer (Section 3.3.1: causal 1D convolutions, no future context needed) and the autoregressive Transformer (which generates output frame-by-frame using KV caching) together enable streaming translation where output speech begins before the input utterance finishes. At frame rate 7.5 with 5 diffusion head steps, the per-frame computation is T_backbone + 5 × T_head. If T_backbone ≈ 10 ms (feasible for a 350M parameter model on a modern GPU) and T_head ≈ 1 ms, the total per-frame latency is ~15 ms — well below the 133 ms frame interval (1 / 7.5), meaning the system can generate speech faster than real-time. This enables applications like live interpretation where total system latency (input processing + generation + output) must stay below ~500 ms to feel interactive. The paper does not provide speech-to-speech results, so this application requires extending the training to paired speech data, but the architectural components are all demonstrated.

When to Prefer This Method

The paper explicitly positions LatentLM against three alternative approaches to multimodal unification: vector quantization (VQ-VAE-based models), full-sequence diffusion (DiT-style models), and hybrid objectives (Transfusion). Based on the empirical evidence:

Prefer LatentLM over vector quantization when:

Generation quality at scale matters more than compatibility with existing discrete-token infrastructure. The evidence: Figure 8a shows VQ-MLLM's FID saturates with training tokens, while LatentLM continues improving. Table 5 shows σ-VAE achieves competitive reconstruction at 10-80× higher compression ratios than neural codecs.
Inference efficiency is critical. The TTS frame rate of 7.5-15 (Table 4) versus VALL-E 2's 75 means 5-10× fewer autoregressive steps, directly reducing latency and compute cost.
The data modality benefits from high-fidelity reconstruction (speech, music, medical imaging) where quantization artifacts are perceptually harmful.

Prefer LatentLM over full-sequence diffusion (DiT-style models) when:

Inference throughput matters (batch generation). The evidence: Figure 7 shows 2.47-2.84× higher throughput at equivalent model sizes.
Variable-length or streaming generation is required (dialogue, speech, video). Causal attention enables this; bidirectional diffusion does not.
Training-inference consistency is important for understanding tasks. The evidence: Table 3 shows LatentLM achieves 38.72 VQAv2 versus Transfusion's 35.36, attributed to the absence of noise-addition during training.
Larger models are planned. Figure 4 shows LatentLM's scaling curve is steeper than DiT's at the model sizes tested (455M-3.68B).

Prefer vector quantization or hybrid approaches when:

Latency for short sequences is the binding constraint (batch size 1, interactive applications with small outputs). DiT's parallel denoising may have lower latency for small images, though the paper does not provide direct latency measurements.
The task requires discrete intermediate representations (e.g., interpretable reasoning steps, external tool use via codebook indices, compression for transmission). Continuous vectors are opaque.
Existing infrastructure demands discrete tokens and retooling is impractical. The paper's method requires a custom training pipeline with diffusion heads and special tokens.
The application benefits from a mature ecosystem. VQ-based models have extensive tooling (tokenizers, inference libraries, fine-tuning recipes); LatentLM is a research system at the time of publication.

Prefer hybrid objectives (Transfusion-style) only if:

Bidirectional image generation is specifically required for global coherence reasons and the understanding gap (Table 3: VQAv2 35.36 vs. 38.72) is acceptable. The paper provides no evidence that bidirectional denoising produces more globally coherent images than causal autoregression — this is an untested hypothesis that would need validation for specific applications.

Dependency tree

Papers this one builds on, walking backwards through the same domain and shared topics.

FLEX ATTENTION: A PROGRAMMING MODEL FOR GENERATING OPTIMIZED ATTENTION KERNELS2024· architecture← in-domain predecessor
- KV Shifting Attention Enhances Language Modeling2024· architecture← in-domain predecessor
- Attention Residuals2026· architecture← similar topic
- Mixture-of-Depths Attention2026· architecture← similar topic
- Tensor Product Attention Is All You Need2025· architecture← similar topic
- MiniMax-01: Scaling Foundation Models with Lightning Attention2025· architecture← similar topic
- MoBA: Mixture of Block Attention for Long-Context LLMs2025· architecture← similar topic
- Multi-Token Attention2025· architecture← similar topic
- Log-Linear Attention2025· architecture← similar topic
- InfLLM-V2: Dense-Sparse Switchable Attention for Seamless Short-to-Long Adaptation2025· architecture← similar topic
- MoGA: Mixture-of-Groups Attention for End-to-End Long Video Generation2025· architecture← similar topic
- Every Attention Matters: An Efficient Hybrid Architecture for Long-Context Reasoning2025· architecture← similar topic
- Reducing Transformer Key-Value Cache Size with Cross-Layer Attention2024· architecture← similar topic
- Fast Conformer with Linearly Scalable Attention for Efficient Speech Recognition2023· architecture← similar topic
- DeepETA: How Uber Predicts Arrival Times Using Deep Learning2020· architecture← similar topic
FSVideo: Fast Speed Video Diffusion Model in a Highly-Compressed Latent Space2026· architecture← similar topic
- TurboDiffusion: Accelerating Video Diffusion Models by 100-200 Times2025· inference-optimization← LLM-judged similar
- Hybrid Linear Attention Done Right: Efficient Distillation and Effective Architectures for Extremely Long Contexts2026· architecture← in-domain predecessor
- Dynamic Large Concept Models: Latent Reasoning in an Adaptive Semantic Space2025· architecture← similar topic

What it influenced

Later papers in the same chain. Direction reversed.

MMaDA: Multimodal Large Diffusion Language Models2025· multimodal→ LLM-judged similar
- MMaDA-Parallel: Multimodal Large Diffusion Language Models for Thinking-Aware Editing and Generation2025· diffusion→ LLM-judged similar
- Harmony: Harmonizing Audio and Video Generation through Cross-Task Synergy2025· multimodal→ LLM-judged similar
- Seedance 1.5 pro: A Native Audio-Visual Joint Generation Foundation Model2025· multimodal→ LLM-judged similar
- DreamOmni3: Scribble-based Editing and Generation2025· vision→ LLM-judged similar
- DreamID-Omni: Unified Framework for Controllable Human-Centric Audio-Video Generation2026· multimodal→ LLM-judged similar
- GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learning2025· multimodal→ in-domain predecessor
Byte Latent Transformer: Patches Scale Better Than Tokens2024· architecture→ in-domain predecessor
- Bolmo: Byteifying the Next Generation of Language Models2025· llm-systems→ LLM-judged similar
- Fourier Position Embedding: Enhancing Attention's Periodic Extension for Length Generalization2024· architecture→ in-domain predecessor
FSVideo: Fast Speed Video Diffusion Model in a Highly-Compressed Latent Space2026· architecture→ similar topic
- HySparse: A Hybrid Sparse Attention Architecture with Oracle Token Selection and KV Cache Sharing2026· architecture→ in-domain predecessor
- SANA-Video: Efficient Video Generation with Block Linear Diffusion Transformer2025· architecture→ similar topic
- Dynamic Large Concept Models: Latent Reasoning in an Adaptive Semantic Space2025· architecture→ similar topic