Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model

🎯 Pitch

Training a single transformer to simultaneously predict text tokens and diffuse image patches outperforms forcing images into discrete tokens—Transfusion matches Chameleon's image-to-text accuracy using only 21.8% of the compute. The model achieves this by applying next-token prediction loss to text and diffusion loss to images within the same architecture, making discrete tokenization unnecessary for multi-modal generation.

1. Executive Summary

This paper introduces Transfusion, a recipe for training a single unified multi-modal model that handles both discrete text and continuous images by applying separate loss functions to each modality over shared parameters: next-token prediction (language modeling loss) for text tokens and denoising diffusion (DDPM loss) for image patches. Through controlled scaling experiments on mixtures of text and Shutterstock images with models up to 7B parameters, Transfusion demonstrates roughly 2× better FID scores than Chameleon's discrete-token approach at equivalent FLOPs, establishes superior scaling laws across text-to-text, text-to-image, and image-to-text benchmarks, and achieves parity with Llama 1 on text tasks while matching or exceeding image generation quality of models like DALL-E 2 and SDXL on GenEval. The work further shows that modality-specific U-Net encoding and decoding layers—combined with intra-image bidirectional attention rather than purely causal attention—enable compressing each image to as few as 16 patches without catastrophic performance degradation, establishing that diffusion and language modeling objectives can coexist productively in a single transformer, but only when each modality is trained with its preferred loss function rather than forcing both through discrete tokenization.

2. Context and Motivation

The Fundamental Tension: Discrete vs. Continuous Data in a Single Model

The core problem this paper confronts is architectural: how do you build a single model that can both generate the next word in a sentence and generate a photorealistic image? These are, at first glance, fundamentally different problems requiring fundamentally different solutions. Text generation operates over discrete symbols from a finite vocabulary, where the dominant paradigm is autoregressive next-token prediction — classification over a distribution of possible next words. Image generation operates over continuous pixel intensities (or latent representations thereof), where the dominant paradigm is denoising diffusion — learning to reverse a gradual noising process by predicting the noise added at each step.

The significance of bridging this gap extends well beyond academic curiosity. The paper argues that any truly general-purpose multi-modal system must natively handle both discrete and continuous data, since the real world is a mixture of both: documents contain images alongside text, videos combine visual streams with audio and subtitles, and user interfaces blend discrete commands with continuous visual feedback. Current approaches to building such systems fall into two broad categories, both of which the paper identifies as having fundamental limitations:

Approach 1: Attach separately pretrained components. The dominant paradigm in both text-to-image generation (Stable Diffusion, DALL-E 2, Imagen) and vision-language models (Flamingo, LLaVA, GILL) is to train modality-specific encoders and decoders independently, then connect them through projection layers or cross-attention mechanisms. A pretrained text encoder (e.g., T5 or CLIP) processes the prompt, a pretrained diffusion model or image encoder handles the visual modality, and some bridging mechanism ties them together. This works well in practice — SDXL and SD 3 achieve remarkable image quality — but the authors identify several subtle drawbacks:

No end-to-end joint learning. Parameters are not shared across modalities during pretraining, meaning the text encoder never learns from the visual data and vice versa. This prevents the kind of synergistic representation learning where understanding images might improve text generation (e.g., better visual descriptions) or vice versa.
Modality-specific inductive biases are siloed. The U-Net's convolutional structure benefits images, and the transformer's self-attention benefits text, but neither architecture learns from the other modality's data distribution. The model cannot fluidly interleave image and text generation — you can't prompt it with "describe this image, then generate a new image based on your description" in a single forward pass.
Output modality is fixed by architecture. A system built by grafting a diffusion decoder onto a language model can produce images from text but cannot (typically) produce text from images or engage in multi-turn mixed-modality dialogue.

Approach 2: Quantize everything into discrete tokens. The alternative philosophy, exemplified by DALL-E, Parti, and Chameleon, is to force all modalities into the language modeling paradigm by discretizing continuous data through vector quantization (VQ-VAE). Images become sequences of discrete tokens from a learned codebook, and a standard autoregressive transformer can then be trained on interleaved text-and-image-token sequences using nothing but next-token prediction. This approach is elegant in its uniformity — one architecture, one loss function, one training procedure — but the authors identify a critical weakness:

"simplifying the model's architecture at the cost of losing information"

Quantization imposes an information bottleneck. An image that naturally lives in a high-dimensional continuous space must be compressed into a sequence of discrete codes, each drawn from a finite vocabulary (e.g., 16,384 tokens in the Chameleon baseline used in this paper). This discards fine-grained visual detail that cannot be perfectly reconstructed from discrete codes. While VQ-VAEs have improved substantially (Esser et al., 2021), the quantization loss is fundamentally irreducible — there is a gap between what a continuous latent diffusion model can express and what a discretized token sequence can capture. The paper's controlled experiments in Section 4.2 quantify this gap for the first time: Transfusion achieves roughly 2× lower FID scores than Chameleon at equivalent FLOPs, and matches Chameleon's image generation quality using only 2.9% of the compute (parity FLOP ratio of 0.029 for FID in Table 3).

Beyond the information bottleneck, the authors surface a more subtle problem with the "quantize everything" approach: it forces the model to spend representational capacity learning to predict image tokens autoregressively, which may be an inefficient use of parameters for continuous data. The diffusion objective — predicting noise at each timestep — is specifically designed for continuous signals and has well-understood properties (e.g., the ability to trade off sample quality against sampling speed by varying denoising steps). Forcing image generation into an autoregressive token-by-token framework discards decades of progress in continuous generative modeling.

The Specific Gap: No One Has Shown That These Objectives Can Coexist

By late 2023 / early 2024, the field was at an impasse. The "attach pretrained components" approach produced the best image quality but wasn't truly unified. The "quantize everything" approach was unified but lost information and lagged behind dedicated diffusion models in image generation quality. The specific gap — and this is the gap Transfusion directly fills — is:

Can you train a single transformer from scratch with BOTH the language modeling loss AND the diffusion loss applied to their respective modalities over SHARED parameters, and get the best of both worlds?

Prior to this work, no one had demonstrated that this works at scale. There were plausible reasons to think it might fail. Joint training with different objectives on different modalities could create destructive interference — gradients from the diffusion loss might contradict gradients from the language modeling loss, causing neither to converge properly. The different attention patterns required (causal for text, bidirectional for images) might conflict. The vastly different loss scales (cross-entropy vs. mean squared error) might require careful balancing that breaks under scaling.

The paper explicitly positions itself as daring to try the simple, previously unexplored solution:

"We propose a simple, yet previously unexplored solution: train a single joint model on two objectives, tying each modality to its preferred objective."

The phrase "previously unexplored" is key. While the idea of using different objectives for different modalities seems obvious in hindsight, the paper argues — convincingly, given the controlled experiments — that no one had systematically demonstrated that this works better than either the "attach" or "quantize" approaches at comparable scale.

Where Prior Approaches Fall Short (Concretely)

The paper is specific about the failure modes of existing methods, and these critiques motivate the experimental design:

1. Chameleon's discretization slows text learning. One of the most surprising findings in the paper (Section 4.2, Table 4) is that Chameleon models perform worse on text-only benchmarks than Transfusion models at equivalent FLOPs, even though both apply the standard LM loss to text tokens. The authors decompose this into two effects:

Stability modifications required by Chameleon (query-key normalization, post-normalization, lower learning rate) degrade text perplexity by approximately 0.9 PPL on C4 for a 0.76B model compared to the standard Llama recipe (Table 4).
Training on quantized image tokens alongside text tokens degrades text perplexity by an additional 0.8 PPL on C4 and reduces Llama 2 eval suite accuracy by 3 points (from 51.9 to 48.9 at 0.76B). The authors hypothesize that "this stems from the competition between text and image tokens in the output distribution" — the model's softmax must spread probability mass over both vocabulary words and image codebook entries, diluting its text modeling capacity.

In contrast, Transfusion's addition of diffusion loss on image patches incurs only a 0.3 PPL degradation on C4 compared to pure text training (Table 4: 10.1 → 10.4), suggesting that the diffusion objective is less intrusive on the model's text representations than forcing images into the same discrete output space.

2. Separately pretrained systems cannot generate interleaved text and images. While not a primary focus of the paper's experiments, the authors note that existing approaches that attach diffusion models to language models (GILL, DreamLLM) typically can only generate a single image in response to a text prompt, or a single caption for an image. They cannot, for example, generate a sequence like: "Here is a description:" → [image] → "Based on that image, I think the answer is..." → [text]. Transfusion's unified architecture with modality-switching decoding (Section 3, "Inference") natively supports such interleaved generation because the model alternates between LM mode and diffusion mode based on special tokens (BOI/EOI) in the sequence.

3. Quantized approaches cannot compress images efficiently. Table 6 shows that Transfusion with U-Net encoding can reduce each image to just 16 patches (from 1024) with FID increasing only from 16.7 to 16.1 — essentially no degradation, and actually an improvement on text-to-image metrics because the model sees more total images during training. In a discretized setting, compressing images to fewer tokens would require a smaller codebook or larger patches, both of which exacerbate the information bottleneck. Transfusion sidesteps this by keeping patches in continuous space — the patch representation can be arbitrarily compressed (within reason) without losing the ability to express fine-grained variation, because the continuous vector can encode more information than a discrete code.

4. Diffusion for text generation has not scaled. The paper acknowledges an emerging alternative: applying diffusion to discrete text (Li et al., 2022; Gat et al., 2024). If diffusion could match autoregressive models for text, you could unify modalities under a single diffusion objective. However, the paper notes that "this approach has yet to achieve the performance and scale of standard autoregressive language models." So the "diffuse everything" approach is not yet viable, leaving Transfusion's hybrid loss as the pragmatic middle path.

How This Paper Positions Itself

The paper's positioning is carefully calibrated:

Not claiming to invent either language modeling or diffusion. The paper explicitly presents Transfusion as a recipe — a specific combination of known techniques (next-token prediction, DDPM, VAE latents, U-Net blocks) applied in a novel configuration. The contribution is demonstrating that this combination works, scales, and outperforms alternatives, not inventing any individual component.
A controlled comparison, not just an impressive demo. Unlike many multi-modal papers that report cherry-picked examples and a single large model, this paper runs scaling law experiments across five model sizes (0.16B to 7B) with controlled compute, data, and architecture. This enables statements about efficiency (parity FLOP ratios) and scaling trends (log-metric over log-FLOPs curves in Figure 5) rather than just absolute performance. The paper is making a methodological point: if you want to claim your approach is better, show it across scale, not just at one size.
Bridging two research communities. The paper explicitly frames itself as combining "the state of the art in discrete sequence modeling (next token prediction) and continuous media generation (diffusion)." This is not just a framing device — it reflects a genuine methodological gap in the field, where language model researchers and diffusion model researchers rarely collaborate or compare approaches directly. Transfusion provides a common ground: a single transformer that both communities can analyze, modify, and improve using their respective toolkits.
A foundation for future work, not a final system. The paper is careful to acknowledge limitations and unexplored directions: combining PRM tree-search with revisions (Section 8), flow matching as an alternative to DDPM, and the currently-expensive difficulty estimation pipeline. This positions Transfusion as a proof-of-concept and a research platform, not a product-ready system. The image editing experiment in Section 4.5 (fine-tuning on only 8k examples) is presented as preliminary evidence that the unified architecture can generalize to new modality combinations (image+text→image) that were not seen during pretraining — promising but not conclusive.

The key intellectual move the paper makes is reframing the multi-modal unification problem from "how do we convert everything to one format?" (quantization) or "how do we connect separate systems?" (pretrained components) to "how do we let each modality use its natural objective within a shared architecture?" This reframing turns what could be seen as a weakness (two different loss functions!) into a strength: each modality gets its preferred training signal, and the shared parameters learn representations that benefit both. The paper's empirical results validate this framing: the shared transformer learns something useful from the combination, as evidenced by Transfusion's superior text performance compared to Chameleon (Table 4) and its ability to match dedicated image generation models (Table 9) while remaining a competent language model.

3. Technical Approach

3.1 Reader Orientation

Transfusion is a training recipe — a specific combination of architecture design, attention masking, loss functions, and data formatting — that produces a single transformer model capable of both generating text (by predicting the next word, one word at a time) and generating images (by iteratively denoising random noise into a coherent picture). The core problem it solves is the apparent incompatibility between the two dominant generative paradigms: language models operate over discrete tokens with a classification loss, while diffusion models operate over continuous vectors with a regression loss. The "shape" of the solution is elegantly minimal: put everything into one shared transformer, but apply the language modeling loss to text tokens and the diffusion loss to image patches, letting each modality use the objective function best suited to its nature.

3.2 Big-Picture Architecture

The system has five major components, each handling a distinct stage of the pipeline from raw data to generated output:

VAE Encoder (pretrained, frozen): converts raw 256×256 pixel images into a compact latent representation — a 32×32 grid of 8-dimensional continuous vectors. This reduces the spatial resolution by a factor of 8×8, making the subsequent transformer processing computationally feasible. The VAE is trained separately and its weights are not updated during Transfusion training.
Modality-Specific Input Projections (trained): maps the latent image vectors and text token embeddings into the transformer's working dimension $d$ . For text, this is a standard embedding lookup table. For images, this is either a simple linear layer or a stack of U-Net down blocks that first compress the local 2×2 or larger windows of latent patches into single vectors before projecting to dimension $d$ . These components are trained end-to-end with the transformer.
Shared Transformer (the bulk of parameters): a standard decoder-only transformer (Llama architecture with SwiGLU, RoPE) that processes the interleaved sequence of text tokens and image patch vectors. It applies a hybrid attention mask: causal attention across the entire sequence (elements can only attend to earlier elements) but with bidirectional attention within each image (all patches of a given image can attend to each other). This is the single largest component, containing the vast majority of the model's parameters.
Modality-Specific Output Projections (trained): converts the transformer's output vectors back into modality-specific predictions. For text tokens: a linear layer followed by softmax produces a probability distribution over the vocabulary. For image patches: either a linear layer or U-Net up blocks reconstruct the patch vectors, which are then arranged into the latent grid and fed to the VAE decoder.
Training Objective (dual loss): applies next-token prediction loss (cross-entropy) to the predicted text token distributions and diffusion loss (mean squared error between predicted and actual noise) to the predicted image patches. These two losses are summed with a balancing coefficient $\lambda$ and backpropagated jointly through the shared transformer.

Information flows as follows: raw text and images enter → VAE encodes images into latent patches → modality-specific layers project text embeddings and image patches into transformer space → sequences are concatenated with BOI/EOI marker tokens separating modalities → the transformer processes the entire sequence under the hybrid attention mask → output projections produce text logits and image patch predictions → losses are computed per-modality and summed → gradients flow back through all trained components. At inference time, the model alternates between autoregressive token sampling (LM mode) and iterative denoising (diffusion mode) whenever a BOI token is generated.

3.3 Roadmap for the Deep Dive

First, the dual training objective (Equation 4 and its components): this is the conceptual heart of Transfusion — combining a discrete distribution loss with a continuous distribution loss. Understanding each term and how they interact is essential before examining any architectural detail.
Second, the data representation pipeline: how raw images and text strings are converted into a unified sequence format that the transformer can process. This includes the VAE latent encoding, the patchification strategy, and the special marker tokens (BOI/EOI) that enable modality switching.
Third, the modality-specific encoding and decoding layers: the linear and U-Net variants for compressing image patch windows, their parameter counts, their scaling behavior, and why they matter beyond simply projecting dimensions.
Fourth, the hybrid attention mask: how Transfusion applies causal attention globally but bidirectional attention within individual images, why this matters for image generation performance (Table 5), and how it interacts with the encoding architecture.
Fifth, the inference algorithm: the modality-switching decoding procedure that alternates between autoregressive text generation and iterative diffusion denoising, including the handling of BOI/EOI tokens, noise scheduling, and classifier-free guidance.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a methodology paper whose core idea is that language modeling and diffusion objectives can coexist in a single shared transformer, each applied to its natural modality, yielding better scaling behavior than forcing both modalities into a single objective paradigm.

The Dual Training Objective: Language Modeling + Diffusion

The conceptual core of Transfusion is the simultaneous application of two fundamentally different loss functions to different elements of the same sequence, optimized over shared model parameters. The total loss is:

$L_{\text{Transfusion}} = L_{\text{LM}} + \lambda \cdot L_{\text{DDPM}}$

where $L_{\text{LM}}$ is the standard next-token prediction loss applied to text tokens, $L_{\text{DDPM}}$ is the denoising diffusion loss applied to image patches, and $\lambda$ is a balancing coefficient set to 5 in all main experiments.

What it computes: given a mixed-modality sequence containing both discrete text tokens and continuous image patch vectors, the model produces predictions for every element. For each text token position, the model outputs a probability distribution over the vocabulary; $L_{\text{LM}}$ measures the cross-entropy between this predicted distribution and the ground-truth next token. For each image, the model outputs a prediction of the noise added during the forward diffusion process; $L_{\text{DDPM}}$ measures the mean squared error between the predicted noise and the actual noise that was added. The two losses are summed, scaled by $\lambda$ , and backpropagated jointly to update all shared transformer parameters plus the modality-specific input/output layers.

Why this form: the two losses operate on fundamentally different mathematical objects — a categorical distribution for text and a continuous vector for images — so they cannot be trivially combined into a single loss function. Adding them with a coefficient $\lambda$ is the simplest possible combination that preserves each modality's preferred training signal without forcing either modality into the other's representation space. The alternative approaches that Transfusion deliberately avoids are: (a) quantizing images and using LM loss everywhere (Chameleon's approach, which the paper shows underperforms), (b) applying diffusion to text (which hasn't scaled to competitive language model performance), or (c) using separate models for each modality with no shared parameters (which prevents cross-modal representation learning). The value $\lambda = 5$ was chosen based on preliminary experiments and the paper explicitly notes that further tuning of $\lambda$ is left to future work.

Now we unpack each loss term in detail.

Language Modeling Loss ( $L_{\text{LM}}$ )

Given a sequence of discrete text tokens $y = y_1, \ldots, y_n$ from a fixed vocabulary $\mathcal{V}$ , the language modeling loss is the standard autoregressive cross-entropy:

$L_{\text{LM}} = \mathbb{E}_{y_i}\left[-\log P_\theta(y_i \mid y_{<i})\right]$

where $P_\theta$ is the probability distribution over $\mathcal{V}$ predicted by the model with parameters $\theta$ at position $i$ , conditioned on all previous tokens $y_{<i}$ in the sequence (including both text tokens and image patches that appeared earlier in the interleaved sequence), and the expectation is taken over the empirical distribution of tokens in the training data.

What it computes: for each text token position in the training sequence, the model takes the transformer's output vector at that position, passes it through a linear projection layer (the "language model head") to produce logits over the vocabulary, applies softmax to obtain a probability distribution, and then computes the negative log-likelihood of the actual ground-truth token under this predicted distribution. The expectation averages this loss over all text token positions in the training batch. When the input at a given position is a BOI (beginning of image) token, no loss is computed at that position — the model produces a prediction but it is ignored in the loss calculation.

Why this form: this is the maximum-likelihood objective for autoregressive sequence modeling, which is the de facto standard for training language models because it directly optimizes the model's ability to assign high probability to observed text sequences. The autoregressive factorization $P(y) = \prod_i P(y_i \mid y_{<i})$ enables efficient training via teacher forcing (the ground-truth prefix is always provided, so the entire sequence loss can be computed in one forward pass with a causal attention mask) and enables straightforward autoregressive sampling at inference time (sample $y_1$ , then condition on it to sample $y_2$ , etc.). Transfusion does not modify this loss in any way — text is handled exactly as it would be in a standard language model, which is a deliberate design choice to preserve text generation quality.

The text tokenizer used is the Llama 2 tokenizer, and the text training data is the Llama 2 corpus of 2T tokens across diverse domains. The vocabulary and embedding dimensions vary by model size following the configurations in Table 2.

Diffusion Loss ( $L_{\text{DDPM}}$ )

The diffusion loss applies to image patches and follows the standard DDPM formulation (Ho et al., 2020). The forward process takes a clean latent image $x_0$ (the VAE-encoded representation) and gradually adds Gaussian noise over $T = 1000$ timesteps according to a predefined noise schedule. The noisy image at timestep $t$ is:

$x_t = \sqrt{\bar{\alpha}_t} \, x_0 + \sqrt{1 - \bar{\alpha}_t} \, \epsilon$

where $\bar{\alpha}_t = \prod_{s=1}^{t} (1 - \beta_s)$ is the cumulative product of noise schedule coefficients with $\beta_s$ increasing over time according to a cosine schedule (Nichol and Dhariwal, 2021), and $\epsilon \sim \mathcal{N}(0, I)$ is randomly sampled isotropic Gaussian noise. The cosine scheduler largely follows $\sqrt{\bar{\alpha}_t} \approx \cos\left(\frac{t}{T} \cdot \frac{\pi}{2}\right)$ with minor adjustments.

The model $\epsilon_\theta$ is trained to predict the noise $\epsilon$ added at timestep $t$ , conditioned on the noisy image $x_t$ , the timestep $t$ , and optionally additional context $c$ (such as a caption preceding the image in the sequence). The diffusion loss is:

$L_{\text{DDPM}} = \mathbb{E}_{x_0, t, \epsilon}\left[ \|\epsilon - \epsilon_\theta(x_t, t, c)\|^2 \right]$

where $\|\cdot\|^2$ is the squared L2 norm (mean squared error summed over all elements of the noise vector), $x_0$ is a clean latent image from the training data, $t$ is uniformly sampled from $\{1, \ldots, T\}$ , and $\epsilon$ is the Gaussian noise actually added to produce $x_t$ from $x_0$ .

What it computes: for each image in the training batch, the model randomly samples a timestep $t$ and a noise vector $\epsilon$ , constructs $x_t$ by blending the clean image with the noise according to the schedule, patchifies $x_t$ into a sequence of patch vectors (see "Patchification" below), feeds them into the transformer alongside any preceding context tokens, and produces a prediction at each patch position. These predicted vectors are compared to the ground-truth noise $\epsilon$ (which was added to the corresponding patch) via squared error. The loss is averaged over all patches of the image, and then averaged over all images in the batch.

There is a crucial implementation detail: the noise $\epsilon$ is added to the entire latent image $x_0$ before patchification — the forward process operates at the level of the full latent tensor, not per-patch. Then the noisy latent is split into patches, each of which is fed to the transformer as a separate element in the sequence. The model's predictions at those patch positions correspond to predictions of the noise components for those spatial regions. This means the model sees an image as a sequence of noisy patch vectors, each with the same timestep $t$ embedded into it, and must predict the noise that corrupted each patch.

A second implementation detail concerns images that appear before their captions in the sequence. The authors note that in these cases, downstream text tokens condition on noisy images during training, which might impair image understanding (captioning) performance. To mitigate this, they experiment with limiting the sampled $t$ to a maximum of 500 (half the noise schedule) in the 20% of training examples where images precede captions (Section 4.3.4, Table 8). Noise limiting improves CIDEr scores significantly (25.4 → 29.4 at 0.76B, 33.7 → 35.2 at 7B) with negligible effect on other benchmarks.

Why this form: the noise prediction formulation of DDPM is a reparameterization that turns the reverse process (learning to denoise step by step) into a simple regression problem: predict the noise that was added at timestep $t$ . The mean squared error loss is the natural choice because it corresponds to maximizing the variational lower bound under Gaussian assumptions, and it works well in practice across all diffusion model literature. The cosine noise schedule (rather than linear) is adopted because Nichol and Dhariwal (2021) showed it improves sample quality, particularly at low timesteps where the linear schedule adds noise too slowly. The choice of $T = 1000$ training timesteps is standard for diffusion models and balances the trade-off between fine-grained denoising control (more steps = smoother reverse process) and training cost (each image is noised once per iteration, so $T$ doesn't directly affect training cost — it only determines the granularity of the noise schedule).

An important subtlety: the diffusion loss is computed per-image, not per-patch. The expectation is taken over whole images, meaning each image contributes one loss value regardless of how many patches it occupies in the sequence. This avoids the diffusion loss dominating the LM loss simply because images typically span many more sequence elements than text tokens (e.g., an image with 256 patches would contribute 256× the diffusion loss if computed per-patch, overwhelming the text signal).

Data Representation Pipeline

VAE Encoding and Decoding

Raw images (256×256 pixels after center-cropping and resizing) are encoded into a compact latent representation using a pretrained variational autoencoder (VAE). The VAE has 86M parameters total and is trained separately following Esser et al. (2021) with the following architecture and objective:

The VAE uses a CNN encoder that reduces spatial resolution by a factor of 8 in each dimension, producing a 32×32×8 tensor — a grid of 32 by 32 latent positions, each represented by an 8-dimensional continuous vector. Each latent position conceptually represents an 8×8 pixel patch in the original image, so the latent grid captures the image at 1/8 the spatial resolution but with richer per-location descriptors. The CNN decoder reverses this process, reconstructing a 256×256 image from the latent tensor.

The VAE is trained with a multi-component loss function (Appendix A):

$L_{\text{VAE}} = L_1 + L_{\text{LPIPS}} + 0.5 L_{\text{GAN}} + 0.2 L_{\text{ID}} + 0.000001 L_{\text{KL}}$

where $L_1$ is L1 reconstruction loss in pixel space, $L_{\text{LPIPS}}$ is perceptual loss based on LPIPS similarity (Zhang et al., 2018), $L_{\text{GAN}}$ is a patch-based discriminator adversarial loss, $L_{\text{ID}}$ is a perceptual loss using internal features of MoCo v2 (Chen et al., 2020), and $L_{\text{KL}}$ is the standard KL divergence term encouraging the encoder output distribution toward a standard normal distribution. GAN training (the adversarial loss term) is delayed until 50,000 steps to allow the VAE to first achieve reasonable reconstruction quality. The latent dimension is fixed at 8.

Why a VAE and not pixel-space diffusion: early diffusion models operated directly on pixels (Ho et al., 2020), but this is extremely computationally expensive because each pixel becomes an element in the diffusion process. The VAE compresses a 256×256×3 = 196,608-dimensional pixel space into a 32×32×8 = 8,192-dimensional latent space (a ~24× reduction), dramatically reducing the compute required for both training and inference while preserving visual fidelity. The KL regularization term balances reconstruction quality against the smoothness of the latent space (ensuring that nearby latent vectors decode to semantically similar images), which is important for diffusion because the denoising process traverses the latent space.

The VAE is pretrained and frozen — its weights are not updated during Transfusion training. This is a practical choice that separates the image compression problem from the multi-modal learning problem. The same VAE is used for all Transfusion and Chameleon experiments (with the only difference being the addition of a quantization layer and codebook loss for Chameleon's VQ-VAE variant), enabling the controlled comparison in Section 4.2.

For the Chameleon baseline, the VAE is replaced with a VQ-VAE. The architecture is identical except that a vector quantization layer is inserted after the encoder, mapping each continuous latent vector to the nearest entry in a learned codebook of 16,384 token types. The $L_{\text{KL}}$ term is replaced with the standard codebook commitment loss with $\beta = 0.25$ and weight 1.0:

$L_{\text{codebook}} = \beta \cdot \|\text{sg}[z_e(x)] - e\|^2 + \|z_e(x) - \text{sg}[e]\|^2$

where $z_e(x)$ is the encoder output vector, $e$ is the nearest codebook vector, and $\text{sg}[\cdot]$ is the stop-gradient operator. The first term moves the codebook vector toward the encoder output, and the second term (commitment loss) moves the encoder output toward the codebook vector. The entire VQ-VAE training objective is otherwise identical to the VAE, using the same reconstruction and perceptual losses.

Patchification: Converting Latent Images to Transformer Inputs

Once an image is encoded into a 32×32×8 latent tensor, it must be converted into a sequence of vectors that the transformer can process as individual elements (analogous to tokens). This conversion is called patchification and involves two steps:

Step 1: Grouping latent pixels into local windows. The 32×32 latent grid is partitioned into non-overlapping square windows of size $k \times k$ latent pixels. The paper experiments with four patch sizes:

Configuration	Window Size	Latent Pixels per Window	Patches per Image
None (raw latents)	1×1	1	1024 (32×32)
Default	2×2	4	256 (16×16)
Compressed	4×4	16	64 (8×8)
Highly compressed	8×8	64	16 (4×4)

The "None" configuration treats each latent pixel as a separate sequence element, producing 1024 patches per image. Larger window sizes reduce the sequence length, which reduces the quadratic attention cost proportionally — an image at 16 patches costs 64× less attention computation than at 1024 patches.

Step 2: Projecting the window into a single vector. The $k \times k \times 8$ values within each window are flattened into a single vector of dimension $8k^2$ and then projected into the transformer's working dimension $d$ using either a simple linear layer or a U-Net down block (described in detail below). This projection is what produces the final patch vector that enters the transformer. The un-patchification (converting transformer outputs back to a latent image) reverses this process: each output vector is projected from dimension $d$ back to $8k^2$ and reshaped into the $k \times k \times 8$ window at the correct spatial location in the latent grid.

A critical detail: before patchification, a learned embedding of the diffusion timestep $t$ is added to every latent pixel vector. This tells the transformer which timestep in the diffusion process each noisy image corresponds to — essential information for the model to predict an appropriate amount of noise to remove. The timestep embedding is shared across all patches of the same image.

After patchification, the sequence of patch vectors is arranged left-to-right, top-to-bottom to form a linear sequence preserving the 2D spatial structure of the image. This sequence is then inserted into the mixed-modality text stream, surrounded by special tokens.

Special Tokens: BOI and EOI

To enable the model to distinguish between modalities in the interleaved sequence, two special tokens are added to the vocabulary:

BOI (Beginning of Image): signals the transition from text mode to image mode. When the model generates a BOI token during autoregressive decoding, the inference algorithm switches from language model sampling to diffusion denoising.
EOI (End of Image): signals the end of the image patch sequence and the return to text mode. When appended after the denoised image patches, the inference algorithm switches back to language model sampling.

These tokens are embedded as regular vectors (using a standard embedding lookup) and participate in the transformer's attention computation like any other element in the sequence. Importantly, no loss is computed at BOI token positions during training — the model produces a prediction for what follows BOI, but the BOI prediction itself is not penalized. This prevents the model from being forced to learn when to transition modalities as a training objective; instead, it learns this implicitly through the sequence structure.

Data formatting for training: each image-caption pair is formatted as a sequence with one of two orderings. In 80% of cases, the caption text comes first, followed by BOI, the image patches, and EOI (text→image ordering). In 20% of cases, the image patches come first, followed by the caption text (image→text ordering). The paper notes that this 80/20 split follows the intuition that image generation may be more data-hungry than image understanding, so the model is given more training examples of text→image than image→text. The ordering is randomly selected for each training example.

Modality-Specific Encoding and Decoding Layers

The paper experiments with two alternative architectures for projecting between the raw latent pixel windows and the transformer's working dimension $d$ . These layers are modality-specific — text uses its own input embedding matrix and output projection (LM head), and images use their own input and output projections — and their parameters are not shared between modalities or between input and output.

Linear Encoding/Decoding (Simple Variant)

The simplest approach: for each $k \times k \times 8$ latent window, flatten it into a vector of dimension $8k^2$ and apply a single learned linear transformation (a weight matrix and bias) to project to dimension $d$ for input, and apply another linear transformation to project from dimension $d$ back to $8k^2$ for output.

The linear layer parameters are tiny relative to the transformer — accounting for fewer than 0.5% of total parameters in every model configuration from 0.16B to 7B (Table 7). For example, at the 7B scale, the linear layers add approximately 0.3% additional parameters (roughly 21M out of 7B).

Why linear is sufficient in principle: a linear projection can mix information across the $8k^2$ dimensions within a window but cannot model spatial dependencies beyond the window boundaries — that job is left to the transformer's self-attention, which operates globally across all patches. The linear layer simply provides a learnable dimensionality reduction that the transformer can then build upon.

U-Net Encoding/Decoding (Enhanced Variant)

The more powerful alternative replaces the simple linear layers with U-Net down and up blocks — the same architectural components used in standard diffusion models like Imagen and DALL-E 2. The U-Net blocks add multiple layers of 2D convolutions with residual connections, self-attention within the feature maps, and progressive spatial downsampling/upsampling.

For the encoder (U-Net down blocks): the sequence of latent pixel windows is first rearranged ("patchified" in the paper's terminology, though different from the transformer sequence formation) back into a 2D grid. The U-Net down blocks apply a series of convolutional layers with stride-2 downsampling, reducing spatial resolution while increasing channel dimension. The specific implementation replaces the AdaLayerNorm typically used in U-Nets with regular layer normalization (the paper notes this modification but doesn't elaborate on the motivation). The output is a set of feature vectors at reduced spatial resolution that are then arranged as a linear sequence and input to the transformer.

For the decoder (U-Net up blocks): the process is reversed — the transformer's output vectors are arranged back into a 2D grid, and U-Net up blocks apply convolutional layers with upsampling to recover the original latent image dimensions.

Parameter cost of U-Net blocks: the U-Net encoder/decoder layers add a fixed 0.27B parameters regardless of the transformer size. This means:

At 0.16B transformer: U-Net adds 106.1% additional parameters (more than doubling the model size)
At 0.76B transformer: U-Net adds 35.5% additional parameters
At 7B transformer: U-Net adds only 3.8% additional parameters — nearly identical to the token embedding parameters

Why U-Net blocks help (Table 7): the paper's ablation experiments reveal a nuanced story. At small transformer sizes (0.16B, 0.76B), the U-Net variant dramatically outperforms the linear variant on image generation (FID: 37.6 → 18.8 at 0.16B; 20.3 → 16.7 at 0.76B) and image captioning (CIDEr: 6.2 → 15.3 at 0.16B; 16.0 → 25.4 at 0.76B). This could simply be a parameter count effect — the U-Net adds many more parameters to small models. However, at 7B, the U-Net advantage persists: FID improves from 18.6 to 16.0, CIDEr from 27.2 to 33.7, while parameters increase by only 3.8%. This is strong evidence for inductive bias benefits: the U-Net's convolutional structure provides the model with built-in spatial reasoning capabilities (locality, translation equivariance) that would otherwise need to be learned from scratch by the transformer's self-attention. The authors note:

"a 1.4B transformer with U-Net layers (1.67B combined) can boost CIDEr beyond the performance of the linear 7B model, and can exceed the FID of the 7B model as early as the 0.37B mark"

This means the U-Net layers are not just adding parameters — they're adding the right kind of computation for image processing, allowing smaller transformers to punch above their weight on visual tasks.

A subtle interaction with attention masking: the U-Net down and up blocks themselves contain internal attention layers. These operate independently of the transformer's attention mask — within the U-Net blocks, attention is always bidirectional and spatial in nature (convolutional or self-attention within the 2D feature map). This means that even if the transformer uses purely causal attention across image patches (which the paper shows is detrimental in Table 5), the U-Net blocks can still model bidirectional dependencies locally, partially compensating for the restricted global attention pattern.

Patch Size Trade-offs and U-Net Interactions (Table 6)

The choice of patch size interacts with the encoding architecture in revealing ways. Table 6 shows that with larger patch sizes (fewer patches per image):

Linear encoding suffers significantly: FID degrades from 20.3 (256 patches) to 43.5 (16 patches), a more than 2× increase. The linear layer cannot compensate for the loss of spatial resolution because it has no mechanism to extract fine-grained features from larger windows — it simply averages over a bigger region and loses detail.
U-Net encoding is remarkably robust: FID actually improves slightly as patch count drops — from 16.7 (256 patches) to 16.1 (16 patches). This counterintuitive result occurs because training with larger patches means the model sees more total images during training (since each image occupies fewer sequence elements, more images fit in the fixed 4096-token context length). The U-Net blocks can extract rich features from the 8×8 windows without needing fine-grained transformer attention, so the increased data diversity pays off without an architectural penalty.

The practical implication is significant: using U-Net encoding with 8×8 patch windows (16 patches per image) reduces inference computation by 64× compared to 1×1 patches (1024 per image) — since attention cost scales quadratically with sequence length — while actually improving image generation quality. This makes Transfusion dramatically more efficient at serving time than naive patchification.

The Hybrid Attention Mask

The defining architectural feature of Transfusion is its hybrid attention pattern, which combines two different masking schemes in a single transformer:

Causal attention globally: every element in the sequence can attend to elements that appear before it (lower indices) but not to elements that appear after it. This preserves the autoregressive property necessary for language modeling and enables efficient training via the standard causal mask trick (computing all token losses in a single forward pass with triangular masking).

Bidirectional attention within each image: within the subsequence of patches belonging to a single image, every patch can attend to every other patch in the same image, regardless of their relative positions in the sequence. This means patch $j$ can attend to patch $k$ even if $k > j$ (appears later in the left-to-right, top-to-bottom ordering), violating the global causal constraint only within the spatial boundaries of one image.

What the mask looks like: imagine a sequence [text tokens ...] BOI [patch_0, patch_1, ..., patch_n] EOI [more text tokens ...]. Each text token can attend to all preceding text tokens and any preceding images' patches. Each patch within the image can attend to: (a) all text tokens and image patches that appeared before the BOI token, (b) all other patches within the same image (regardless of position), and (c) nothing after the EOI token. The BOI and EOI tokens themselves follow the same rules as text tokens (causal globally).

Implementation: the attention mask is constructed as a binary matrix where position $(i, j)$ is 1 if element $i$ is allowed to attend to element $j$ , and $-\infty$ (or a very large negative number) otherwise, which is added to the attention scores before softmax. This pattern can be implemented efficiently as a block-diagonal modification to the standard causal mask, lifting the causal restriction for the submatrix corresponding to patches within each image.

Why bidirectional attention within images is critical (Table 5): the paper's ablation comparing purely causal attention to hybrid attention reveals dramatic differences:

With linear encoding, FID drops from 61.3 (causal) to 20.3 (bidirectional) — a ~3× improvement. In the causal-only variant, there is no flow of information from later patches to earlier ones within the same image. Since image patches are arranged in a fixed spatial order (left-to-right, top-to-bottom), this means the top-left patch has no access to information about the bottom-right patch when the model processes it. For image generation, this is catastrophic — the model must predict each patch without knowing what the rest of the image looks like, making it impossible to ensure global coherence.
With U-Net encoding, the gap is much smaller: FID is 16.8 (causal) vs. 16.7 (bidirectional) — essentially identical. This is because the U-Net blocks themselves contain attention layers that operate bidirectionally within the feature maps, providing the spatial information sharing that the transformer's causal mask denies. The U-Net blocks serve as a workaround for the causal restriction, enabling spatial reasoning even when the transformer is causal-only.

The fact that the U-Net variant is nearly insensitive to the attention pattern suggests that the U-Net blocks are doing much of the heavy lifting for spatial integration, and the transformer's role is more about cross-modal attention (attending to text context) and high-level semantic composition rather than low-level spatial coherence. For the linear variant, the bidirectional attention is absolutely essential.

What about cross-image attention? The hybrid mask does NOT allow patches of one image to attend to patches of another image unless the earlier image appears earlier in the sequence (global causal rule applies across image boundaries). Images are independent units — the model can condition on previous images when generating a new one (which is useful for multi-turn mixed-modality dialogue) but the current image cannot "look ahead" at future images.

Why not full bidirectional attention everywhere? Full bidirectional attention would break the autoregressive language modeling property — text tokens would be able to attend to future text tokens, leaking information and preventing the model from learning to predict the next token from only the prefix. Since text generation quality is a primary goal (matching Llama 1 performance), preserving the causal property for text is non-negotiable. The hybrid mask represents the minimal modification needed to accommodate images while keeping text generation intact.

The Inference Algorithm: Modality-Switching Decoding

Reflecting the dual-objective training, inference in Transfusion alternates between two decoding modes triggered by the special BOI and EOI tokens. This is a novel decoding algorithm that combines standard practices from language model sampling and diffusion model denoising.

The overall procedure:

Start in LM mode. The input sequence (e.g., a text prompt) is provided to the model. The model samples text tokens autoregressively using the standard approach: at each position, the predicted distribution over the vocabulary is computed, and a token is sampled (using greedy decoding, or optionally temperature and top-p truncation). The sampled token is appended to the sequence and used as input for the next step.
Transition to diffusion mode. When the model samples a BOI token, the decoding algorithm immediately switches behavior. Instead of sampling the next token from the vocabulary, the model knows that an image must now be generated. The desired image size (implicitly determined by the number of patches, which depends on the latent grid size and patch size) governs how many patch positions to generate.
Initialize noise. A pure noise tensor $x_T$ of the appropriate shape (e.g., 16×16×8 for 256 patches with 2×2 windows) is constructed by sampling independent Gaussian noise $\mathcal{N}(0, I)$ . This tensor is patchified into $n$ vectors (where $n$ is the number of patches) and appended to the input sequence following the BOI token.
Iterative denoising. The diffusion denoising loop runs for the number of inference steps (250 steps in the paper's experiments, though the model was trained on 1000 timesteps). At each step $t$ (counting down from $T$ to 1):
- The model processes the entire sequence up to and including the current noisy image patches (but not including an EOI token yet — the EOI is only appended after denoising completes).
- At each patch position, the model predicts the noise $\epsilon_\theta(x_t, t, c)$ , where $c$ is the context (preceding text tokens or images).
- The predicted noise is scaled according to the noise schedule to determine how much noise to remove at this step.
- A proportional amount of predicted noise is subtracted from $x_t$ to produce $x_{t-1}$ , following the standard DDPM reverse process.
- Crucially, $x_{t-1}$ overwrites $x_t$ in the sequence — the model always conditions on the most recent version of the image and does not attend to previous diffusion timesteps. This means the sequence length remains constant during denoising; only the values of the patch vectors change.
Return to LM mode. Once the denoising loop reaches $t = 0$ (producing a clean latent image $x_0$ ), an EOI token is sampled/appended to the sequence. The decoding algorithm switches back to LM mode, and the model can continue generating text, potentially including additional images.

Inference hyperparameters:

Diffusion steps: 250 steps are used for inference (standard for diffusion models; fewer steps trade quality for speed). The model was trained on 1000 timesteps, so 250 is a reasonable balance.
Classifier-free guidance (CFG): the model is run twice at each denoising step — once conditioned on the full context $c$ (the prompt and any preceding sequence elements) and once with the context dropped (unconditional prediction). The final noise prediction is a weighted combination: $\epsilon_\theta(x_t, t, c) + w(\epsilon_\theta(x_t, t, c) - \epsilon_\theta(x_t, t, \varnothing))$ , where $w$ is the guidance scale. The paper sweeps this coefficient: $w = 5$ for the controlled comparison with Chameleon (matching Chameleon's setting), $w = 3$ for ablation experiments (where it was found to be more appropriate for Transfusion), and tuned per-benchmark for the large-scale GenEval comparison.
Text decoding: greedy decoding is used for text generation in all experiments except the Llama evaluation suite, which uses ranked classification (evaluating multiple choice options by their perplexity under the model).

Why CFG doubles computation: at each of the 250 denoising steps, the model must run two forward passes — one with context and one without — meaning the effective computation for image generation is 500 forward passes times the cost of processing the entire sequence (text prefix + image patches). This is the standard cost of high-quality diffusion sampling and is not specific to Transfusion.

A subtle consequence of the overwriting mechanism: because the model only sees the current timestep's noisy image and not previous timesteps, the transformer's causal attention over the sequence does not create dependencies between different timesteps of the same image. The only sequential dependency is that each denoising step uses the output of the previous step to construct its input, but this is handled outside the transformer by the diffusion sampler. Inside the transformer, each denoising step is an independent forward pass on a sequence where the image patches happen to have different noise levels. This means the transformer does not need to maintain any state across denoising steps — it's a pure feedforward denoiser at each step.

Design Decision Summary and Justifications

Dual loss over single loss: the paper's central hypothesis is that forcing images into discrete tokens (single LM loss) discards information and misallocates model capacity, while applying diffusion to text (single diffusion loss) hasn't scaled. The dual loss is the pragmatic solution that preserves the state-of-the-art approach for each modality.
Frozen VAE rather than end-to-end training: decouples the image compression problem from the multi-modal learning problem, enabling clean controlled comparisons. Training the VAE jointly with the transformer would introduce additional hyperparameters and compute costs without a clear benefit, since VAEs are already well-understood.
Cosine noise schedule: proven superior to linear schedules in diffusion literature (Nichol and Dhariwal, 2021), particularly for preserving information at low noise levels.
$\lambda = 5$ : chosen empirically from preliminary experiments to balance the two loss scales. The paper does not report extensive tuning of this hyperparameter and flags it as future work.
80/20 text→image vs. image→text split: reflects the intuition that image generation is harder (requires more training signal) than image understanding. The noise limiting ablation (Table 8) suggests that when images do precede text, limiting the noise helps, but this is a post-hoc optimization applied on top of the 80/20 split.
U-Net blocks over linear layers (when parameter budget allows): the inductive biases of convolutions and spatial attention provide better image processing capabilities per parameter than linear layers, even controlling for parameter count (the 7B model with 3.8% U-Net overhead still benefits). The specific choice to fix U-Net parameters at 0.27B across model sizes is motivated by practical considerations (implementation simplicity) and is acknowledged as suboptimal — scaling U-Net with transformer size is noted as future work.
Hybrid attention over purely causal or purely bidirectional: purely causal cripples image generation (Table 5, FID 61.3 vs. 20.3 with linear encoding). Purely bidirectional would break autoregressive text generation. The hybrid mask preserves the causal property where it's needed (text, cross-image) and removes it where it's harmful (within-image spatial integration).
250 inference steps (from 1000 training steps): standard practice in diffusion models that trades a small quality reduction for significant speedup. The model is trained to denoise at arbitrary timesteps, so a coarser step schedule at inference is valid, just slightly suboptimal.
Separate ORM for revisions: included for completeness with the revision model baseline.
Best-of-N weighted over standard best-of-N: uses consensus across solutions to improve robustness against individual high-scoring but incorrect answers.

4. Key Insights and Innovations

Innovation 1: The Two-Objective, One-Model Recipe as a Previously Unexplored Design Point

The paper's most fundamental contribution is not the invention of new loss functions or architectures, but rather the demonstration that a simple, previously untested combination — language modeling loss for text, diffusion loss for images, over a single shared transformer — actually works, scales, and outperforms alternatives. This is a genuinely non-obvious finding because the two objectives operate on fundamentally different mathematical objects (categorical distributions vs. continuous vectors) and impose contradictory demands on the attention mechanism (causal for autoregressive text, bidirectional for spatial images). The dominant assumption in the field — reflected in the architectures of DALL-E, Parti, Chameleon, and essentially all multi-modal generation systems prior to Transfusion — was that unification required forcing all modalities into a single representational format: either quantize images into discrete tokens and use LM loss everywhere, or attach a pretrained diffusion decoder to a frozen text encoder. Transfusion's key conceptual move is rejecting this premise entirely: the modalities don't need to share a loss function; they only need to share parameters.

This reframing matters because it changes the optimization landscape. In Chameleon's quantized approach, the model must learn to predict discrete image tokens autoregressively — a task for which the LM objective is ill-suited, since image patches have complex spatial dependencies that the sequential factorization of $P(\text{image}) = \prod_i P(\text{patch}_i \mid \text{patch}_{<i})$ struggles to capture. The diffusion objective, by contrast, models the joint distribution of all patches simultaneously through the denoising process, with the noise prediction loss providing a dense, spatially global training signal at every timestep. The paper's controlled comparison (Figure 5, Table 3) quantifies the consequences: Transfusion achieves parity with Chameleon on image generation (FID) using only 2.9% of the compute (parity FLOP ratio of 0.029), and on text generation, Transfusion matches Chameleon at roughly 50–60% of the FLOPs. These are not marginal improvements — they are order-of-magnitude efficiency gains that suggest quantization is a fundamentally suboptimal approach for continuous modalities.

The significance extends beyond the raw numbers. By showing that these two objectives can coexist without destructive interference — and indeed that the shared parameters learn representations benefiting both modalities (text perplexity degrades less under diffusion than under quantization, Table 4) — Transfusion establishes a new design space for multi-modal architectures. The paper explicitly frames this as a broader principle: "combining a discrete distribution loss with a continuous distribution loss to optimize the same model." This opens the door to future mixtures of objectives (e.g., flow matching for video, wave reconstruction for audio) that each modality can select from without forcing a one-size-fits-all paradigm.

The paper acknowledges that this finding is "previously unexplored" not because it was technically impossible, but because the field had converged on a false dichotomy — either everything is discrete or everything uses pretrained components. The insight is thus as much about questioning a shared assumption as it is about any particular technical mechanism.

While the intuition that quantization discards information is not new — VQ-VAEs are known to introduce reconstruction artifacts, and continuous latent diffusion models (Rombach et al., 2022a) have long outperformed discrete-token image generators on quality metrics — the paper provides the first systematic, compute-controlled quantification of how this information loss propagates into multi-modal training efficiency. Prior comparisons between discrete and continuous image generation typically involved different architectures, different training data, and different scales, making it impossible to attribute performance differences to the quantization bottleneck alone. Transfusion's experiment design solves this by training the VAE and VQ-VAE for Transfusion and Chameleon using "exactly the same data, compute, and architecture, with the only differentiator being the quantization layer and codebook loss" (Section 4.1). This isolates the effect of discretization to a degree that prior work had not achieved.

The results reveal that the penalty is far larger than one might expect from reconstruction quality alone:

On text-to-image generation, Transfusion achieves roughly 2× lower FID than Chameleon at equivalent FLOPs, and the scaling curves in Figure 5 show that this gap persists (and may slightly widen) as models scale from 0.16B to 7B parameters. The parity FLOP ratio of 0.029 for FID (Table 3) means Chameleon would need approximately 34× more compute to match Transfusion's image quality at the 7B scale.
On image-to-text generation (MS-COCO CIDEr), Transfusion matches Chameleon using only 21.8% of the FLOPs.

Crucially, the paper also uncovers a surprising second-order effect: discretization harms text learning even though text is never quantized. Table 4 shows that in a 0.76B model, adding quantized image tokens to the training mixture degrades C4 perplexity by an additional 0.8 PPL beyond the cost of simply adding diffusion-trained continuous image patches (11.8 vs. 10.4). The authors' hypothesis — "competition between text and image tokens in the output distribution" — points to a structural inefficiency: the language model's softmax must allocate probability mass across both vocabulary words and 16,384 image codebook entries, diluting its representational capacity for text. The diffusion objective sidesteps this entirely by keeping image predictions in a separate continuous output space with its own loss, never forcing the two modalities to compete for probability mass in a shared discrete distribution.

This is a diagnostic insight with practical implications beyond Transfusion: it suggests that any approach which unifies modalities through a single discrete token space will inherently sacrifice text quality, and that the cost is not merely additive but may scale with the diversity of the discretized modality. For organizations deciding between quantization-based and continuous-latent approaches to multi-modal training, Table 4 provides concrete evidence that the quantization penalty extends beyond image quality into text performance — a finding that is unlikely to be reversed by better VQ-VAE design, since the competition-in-the-softmax problem is architectural, not a matter of codebook quality.

Innovation 3: Bidirectional Within-Image Attention as the Minimal Modification to Preserve Both Modalities

The hybrid attention mask — causal across the sequence, bidirectional within each image — is superficially a simple engineering detail. But the paper's ablation in Table 5 reveals it as a critical architectural discovery: removing intra-image bidirectional attention (using purely causal attention everywhere) causes FID to triple from 20.3 to 61.3 with linear encoding, effectively destroying the model's ability to generate coherent images. This is not an incremental improvement; it is the difference between a functional image generator and a failed one.

The intellectual contribution here is not the invention of bidirectional attention (which is standard in image transformers and diffusion models) but rather the demonstration that a single attention mask can be partitioned by modality without destructive interference. Prior to this work, it was not obvious that a transformer could simultaneously maintain a causal inductive bias for text (essential for autoregressive generation) and a bidirectional inductive bias for images (essential for spatial coherence) without the two conflicting. The concern would be that gradients from bidirectional attention on images might "leak" causality-violating information into the text representations, or that the transformer would struggle to learn two different relational structures (sequential vs. spatial) over the same parameters.

The results in Table 5 show that this concern is largely unfounded: enabling bidirectional attention within images has no negative effect on text metrics (C4 PPL: 10.4 with both causal and bidirectional; Wiki PPL: 6.0 in both cases; Llama accuracy: 51.7 vs. 51.4, within noise). The transformer successfully learns to apply causal reasoning to text tokens and spatial reasoning to image patches, using the same attention heads and parameters for both. This suggests that attention patterns can be context-dependent without architectural modification — the model learns to attend differently based on whether the query token is text or an image patch, and the mask simply enforces the constraints, not the behavior itself.

The interaction with U-Net encoding (Table 5, bottom rows) provides a second insight: when U-Net blocks are present, the bidirectional attention mask becomes nearly irrelevant for image quality (FID: 16.8 causal vs. 16.7 bidirectional). This reveals a functional redundancy — spatial information sharing can be handled either by the transformer's attention or by convolutional layers in the encoder/decoder, and the U-Net effectively compensates for the missing bidirectional attention. This has practical implications for architecture design: if computational constraints favor purely causal attention (simpler implementation, better compatibility with inference optimizations like KV-caching), the U-Net encoder/decoder can serve as a drop-in replacement for bidirectional attention within images, with minimal quality loss.

The paper's experiments with linear vs. U-Net patch encoding (Tables 6 and 7) reveal a more general principle: spatial processing for continuous modalities should be handled by modality-specific inductive biases (convolutions, spatial attention), not by the shared transformer's self-attention, and this division of labor improves both efficiency and quality. This is a reframing of how to think about transformer-based multi-modal architectures. The default assumption in many unified architectures (e.g., ViT-based vision-language models) is that the transformer should operate on raw patches and learn spatial relationships from scratch through self-attention. Transfusion's results suggest this is suboptimal — the transformer's attention is better used for cross-modal integration and high-level semantic composition, while low-level spatial feature extraction benefits from convolutional inductive biases provided by U-Net blocks.

The evidence for this principle comes from several observations:

Parameter efficiency: at the 1.4B transformer scale, adding U-Net layers (0.27B, 19.3% overhead) boosts CIDEr from 19.1 to 28.1 and FID from 19.4 to 16.6 — improvements that exceed what a 7B linear model achieves on CIDEr (27.2) and approach the 7B linear model on FID (18.6). This means the U-Net layers are providing more image understanding/generation capability per parameter than the transformer's self-attention, because their inductive biases are better matched to the spatial structure of images.
Patch size robustness: the U-Net variant compresses images to 16 patches with essentially no FID degradation (16.7 → 16.1, Table 6), while the linear variant degrades from 20.3 to 43.5. The U-Net can extract rich features from larger windows (e.g., 8×8 latent pixels) because its convolutions operate at multiple scales and can model fine-grained spatial patterns within the window, whereas the linear layer simply averages. This enables a 64× reduction in attention computation with no quality cost, which is a practical deployment insight of considerable value.
Attention pattern insensitivity: as noted in Innovation 3, U-Net encoding makes the transformer nearly indifferent to whether attention within images is causal or bidirectional. This suggests the U-Net is handling the spatial integration that the transformer would otherwise need to learn through attention, freeing the transformer to focus on cross-modal relationships.

This principle connects to a broader architectural insight: the "one architecture for everything" philosophy (which motivated early unified models) may be counterproductive. Instead, lightweight modality-specific pre/post-processing layers that inject appropriate inductive biases can dramatically reduce the burden on the shared backbone, allowing it to be simpler (e.g., purely causal) while achieving better results. This is analogous to how modern LLMs use specialized tokenizers rather than operating on raw bytes — the tokenizer injects linguistic inductive bias so the transformer doesn't need to learn morphology from scratch. Transfusion's U-Net layers serve the same role for images.

One of the paper's most striking — and initially puzzling — results is that training on quantized image tokens degrades text performance more than training on continuous diffusion image patches, even though text is trained identically in both cases (Table 4). At 0.76B scale: starting from a Llama 2 baseline of 10.1 C4 PPL, adding diffusion on continuous image patches increases perplexity to 10.4 (+0.3). Adding quantized image tokens with LM loss increases perplexity to 11.8 (+1.7 total, or +0.8 beyond the stability modifications alone). On the Llama 2 eval suite: 53.7 (baseline) → 51.7 (with diffusion) → 48.9 (with quantized tokens). The diffusion-trained model loses 2 accuracy points; the quantized-token model loses 4.8.

This finding is conceptually significant because it identifies a previously undocumented cost of the "quantize everything" paradigm: forcing a language model to predict discrete image tokens alongside text tokens creates competition in the output softmax that degrades the model's text modeling capacity. The mechanism is architectural — the model's final linear projection must produce logits over a combined vocabulary of text tokens (~32k for Llama 2) plus image codebook entries (16,384 for Chameleon). The softmax normalization couples these predictions: probability mass allocated to plausible image tokens is mass not allocated to text tokens, and vice versa. While the model can in principle learn to route modality-specific information through different subspaces before the output projection, the empirical evidence suggests this routing is imperfect and imposes a measurable cost on text quality.

In Transfusion, the diffusion loss applies to a separate continuous output that never interacts with the text softmax. The model predicts image patches as vectors in $\mathbb{R}^{8k^2}$ (or whatever the patch dimension is), not as logits over a codebook, so text and image predictions never compete for probability mass. The shared transformer parameters learn to produce different kinds of outputs depending on the token type, but the final projection layers are modality-specific and independently normalized.

This is a negative result with positive implications: it demonstrates that the field's intuitive preference for conceptual unity (one loss, one output space) has a concrete, measurable cost. The finding provides a theoretical justification for multi-objective multi-modal training that goes beyond "it works better empirically." It implies that any future system that unifies modalities through a single discrete token space will face a fundamental tradeoff between codebook size (larger = better image quality, more information preserved) and text quality (larger = more competition in the softmax, worse text perplexity). Transfusion avoids this tradeoff entirely by decoupling the output spaces, making it not just an empirical improvement but a principled architectural resolution to a structural problem.

5. Experimental Analysis

Evaluation Methodology

Dataset. The primary training and evaluation data comes from two sources. For text, the paper uses the Llama 2 corpus (Touvron et al., 2023b), containing 2T tokens across diverse domains, tokenized with the Llama 2 tokenizer. For images, the paper uses a collection of 380M licensed Shutterstock images paired with captions, with each image center-cropped and resized to 256×256 pixels. In the large-scale 7B experiment (Section 4.4), the data is expanded to include 220M additional publicly available images (filtered to exclude people), 80M upsampled Shutterstock images containing people, and data from Conceptual 12M (CC12M; Changpinyo et al., 2021), reaching a total of 692M image-caption pairs per epoch. The training mixture maintains a 1:1 token ratio between text and images, with image order randomized — captions appear first in 80% of training examples.

Evaluation benchmarks span four modality pairings (Table 1):

Text-to-text: perplexity on held-out Wikipedia (20M tokens) and C4 (20M tokens) evaluation sets, plus average 0-shot accuracy on the Llama 2 evaluation suite (HellaSwag, PIQA, SIQA, WinoGrande, ARC-e, ARC-c, BoolQ; Touvron et al., 2023b).
Text-to-image: zero-shot Fréchet Inception Distance (FID; Heusel et al., 2017) and CLIP score (Radford et al., 2021) on 30K randomly selected prompts from the MS-COCO validation set (Lin et al., 2014). For ablation experiments in Section 4.3, only 5K examples are used. The large-scale experiment additionally reports GenEval score (Ghosh et al., 2023).
Image-to-text: CIDEr score (Vedantam et al., 2015) on the Karpathy test split of MS-COCO (5K images).
Image editing: qualitative assessment on a fine-tuned model using the EmuEdit test set (Sheynin et al., 2024).

Base model. The paper uses randomly initialized transformers following the Llama architecture (Touvron et al., 2023a) with SwiGLU activation (Shazeer, 2020) and RoPE position embeddings (Su et al., 2024). Five model sizes are evaluated: 0.16B, 0.37B, 0.76B, 1.4B, and 7B parameters (Table 2 provides layer counts, embedding dimensions, and attention head counts). The 7B model serves as the primary comparison point for both the Chameleon baseline and the image generation literature. The choice of Llama architecture is motivated by it being a widely-adopted, well-characterized transformer design that enables clean comparison to existing language models, though the paper notes that "Transfusion could potentially work with other architectures too, despite its name."

Metrics. Perplexity is computed as the exponentiated average negative log-likelihood on held-out text, with lower values indicating better language modeling. FID measures the distance between the distribution of generated images and real images in Inception feature space, capturing both fidelity and diversity — lower is better. CLIP score measures the cosine similarity between image and text embeddings from a pretrained CLIP model, capturing semantic alignment — higher is better. CIDEr evaluates image captions by computing TF-IDF-weighted n-gram overlap with human reference captions — higher is better. GenEval score measures a model's ability to accurately depict prompt-specified objects, attributes, and relationships across multiple sub-tasks — higher is better. Llama 2 eval suite accuracy is the average 0-shot task accuracy across the seven benchmarks, using ranked classification (evaluating each answer choice by its likelihood under the model and selecting the highest-probability option).

Baselines. The paper includes several comparison points:

Chameleon (Chameleon Team, 2024): the primary controlled baseline. Chameleon discretizes images using a VQ-VAE with a codebook of 16,384 token types and trains a standard autoregressive language model over the combined text-image token sequence. For fair comparison, the paper trains Chameleon models using exactly the same data, compute, and VAE architecture as Transfusion, "with the only differentiator being the quantization layer and codebook loss of Chameleon's VQ-VAE." Chameleon requires additional stability modifications to the Llama architecture: query-key normalization, post-normalization, a denominator loss term, and a lower learning rate of 1e-4. These modifications incur an efficiency cost that is accounted for in the FLOPs-matched comparison.
Llama 2 (Touvron et al., 2023b): a text-only baseline using the standard Llama recipe, evaluated at 0.76B scale to isolate the cost of adding multi-modal training on text quality (Table 4).
Published image generation models (Table 9): DALL-E 2 (Ramesh et al., 2022), SD 1.5, SD 2.1 (Rombach et al., 2022b), SDXL (Podell et al., 2023), DeepFloyd (Stability AI, 2024), Imagen (Saharia et al., 2022), Parti (Yu et al., 2022), Chameleon 7B, and SD 3 (Esser et al., 2024b). These provide external reference points, with the caveat that these models use different architectures, data, and training procedures — only Chameleon is a controlled comparison.

Generation budget and compute accounting. The paper measures compute in FLOPs (floating point operations) computed as 6ND, where N is the number of model parameters and D is the number of training tokens seen. This is the standard approximation from the scaling laws literature (Hoffmann et al., 2022). Because Transfusion represents images with fewer sequence elements than Chameleon (continuous patches vs. discrete tokens at potentially different compression rates), the actual wall-clock training cost differs even at equal parameter counts and token counts. To remove this confounder and ensure fair comparison, the paper explicitly uses "the theoretical FLOP calculation" rather than wall-clock time. All scaling law plots (Figure 5) use this FLOPs metric on a logarithmic scale. For inference, image generation uses 250 diffusion steps with classifier-free guidance, and text uses greedy decoding. The CFG coefficient is set to 5 for the controlled Chameleon comparison (matching Chameleon's setting), 3 for ablation experiments, and tuned per-benchmark for the large-scale GenEval evaluation.

Cross-validation and statistical protocol. The paper does not employ cross-validation or statistical significance testing. The scaling law analysis (Figure 5) uses linear regression of log-metric on log-FLOPs to estimate trends, with outlier results from small Chameleon models excluded based on minimum performance thresholds (FID ≤ 100, CLIP ≥ 17, CIDEr ≥ 4). These thresholds remove datapoints that "do not correlate with the emerging scaling law of larger Chameleon models." The paper reports single evaluation runs on standard benchmarks without confidence intervals, which is typical for large-scale pretraining experiments where computational cost prohibits multiple training runs.

Main Quantitative Results

Controlled Comparison with Chameleon (Section 4.2)

The headline finding across all benchmarks is that Transitision consistently exhibits superior scaling laws compared to Chameleon, with the efficiency gap being largest on image generation tasks. Figure 5 displays log-log scaling curves for six metrics, and Table 3 quantifies the performance of the largest models (7B, trained on 0.5T tokens) along with estimated parity FLOP ratios — the relative amount of Transfusion FLOPs needed to match Chameleon's 7B performance.

Text-to-text benchmarks: On C4 perplexity, Transfusion 7B achieves 7.72 vs. Chameleon's 8.41 (parity FLOP ratio: 0.489), meaning Transfusion reaches Chameleon's performance using approximately 48.9% of the compute. On Wikipedia perplexity: 4.28 vs. 4.69 (parity ratio: 0.526). On the Llama 2 eval suite: 61.5% accuracy vs. 59.1% (parity ratio: 0.600). While the absolute gaps appear modest (0.69 PPL on C4, 2.4 points on Llama eval), the scaling trends in Figure 5 show the lines are roughly parallel, suggesting the efficiency advantage is systematic rather than an artifact of a particular model size.

Image-to-text generation: On MS-COCO CIDEr, Transfusion achieves 27.2 vs. Chameleon's 18.0 (parity FLOP ratio: 0.218), meaning Transfusion matches Chameleon using less than a quarter of the compute. This is a substantial gap — 9.2 CIDEr points at equal scale — and the scaling curve in Figure 5 (middle-left) shows Transfusion models consistently above Chameleon models across the entire FLOPs range.

Text-to-image generation: This is where the efficiency gap is most dramatic. On FID, Transfusion achieves 16.8 vs. Chameleon's 29.6 (parity FLOP ratio: 0.029), meaning Transfusion matches Chameleon's image quality using only 2.9% of the compute — over 34× more efficient. In absolute terms, Transfusion's FID is nearly half of Chameleon's (16.8 vs. 29.6, lower is better), indicating substantially better image fidelity and diversity. On CLIP score, Transfusion achieves 25.5 vs. Chameleon's 24.3 (parity FLOP ratio: 0.319). The FID scaling curves in Figure 5 (bottom-left) show the largest separation of any benchmark, with Chameleon models struggling to break below 30 FID even at the 7B scale while Transfusion models show steady improvement across all sizes.

A diagnostic decomposition of text degradation (Table 4): To understand why Transfusion outperforms Chameleon even on text-only metrics, the paper ablates the progression from a pure Llama 2 recipe to Transfusion and Chameleon at 0.76B scale. A standard Llama 2 model achieves 10.1 C4 PPL, 5.8 Wiki PPL, and 53.7% Llama eval accuracy. Adding diffusion training on image patches (Transfusion) increases C4 PPL by only 0.3 (to 10.4) and reduces Llama accuracy by 2.0 points (to 51.7). Adding Chameleon's stability modifications to a text-only model increases C4 PPL by 0.9 (to 11.0). Further adding LM loss on quantized image tokens increases C4 PPL by an additional 0.8 (to 11.8) and reduces Llama accuracy by an additional 3.0 points (to 48.9). The total degradation from the Llama 2 recipe to Chameleon is 1.7 C4 PPL and 4.8 accuracy points, of which approximately half is attributable to stability modifications and half to the introduction of quantized image tokens. Transfusion's degradation is only 0.3 PPL and 2.0 accuracy points, with no stability modifications needed.

Architecture Ablation Results (Section 4.3)

The ablation experiments use 0.76B models trained on 0.5T tokens with 2×2 latent pixel patches unless otherwise specified. All CFG uses coefficient 3 for these experiments.

Attention masking (Table 5): The paper ablates causal-only vs. hybrid (bidirectional within images) attention for both linear and U-Net encoding architectures at 0.76B scale. With linear encoding, bidirectional attention is essential for image generation: FID improves from 61.3 (causal) to 20.3 (bidirectional), a ~3× improvement explained by the fact that in purely causal attention, each image patch can only attend to patches earlier in the raster-scan order, preventing the top-left patches from accessing information about the bottom-right of the image. CIDEr also improves from 12.7 to 16.0. Text metrics are essentially unaffected (C4 PPL: 10.4 in both cases; Wiki PPL: 6.0 in both; Llama accuracy: 51.7 bidirectional vs. 51.4 causal). With U-Net encoding, the attention mask becomes nearly irrelevant for image quality: FID is 16.8 (causal) vs. 16.7 (bidirectional), because the U-Net blocks themselves contain internal bidirectional attention, compensating for the transformer's restricted attention pattern. CIDEr improves more noticeably from 23.3 to 25.4, suggesting image understanding benefits from the global spatial context more than image generation when U-Net encoding is used.

Patch size (Table 6): The paper evaluates patch sizes ranging from 1×1 (1024 patches per image) to 8×8 (16 patches per image) for both linear and U-Net encoding at 0.76B scale. With linear encoding, increasing patch size consistently degrades all metrics: FID rises from 21.0 (1×1) to 20.3 (2×2) to 25.6 (4×4) to 43.5 (8×8), CLIP drops from 24.0 to 18.9, CIDEr falls from 12.0 to 11.3, and text perplexity increases (C4 PPL: 10.3 → 11.7). The paper suggests text degradation occurs because "transfusion needs to exert more resources (i.e. parameters) to learn how to process images with fewer patches."

With U-Net encoding, the pattern is qualitatively different and, for image generation metrics, counterintuitively beneficial at larger patch sizes: FID remains essentially flat (21.0 at 1×1, 16.7 at 2×2, 16.0 at 4×4, 16.1 at 8×8) while CLIP actually improves (24.0 → 25.2). CIDEr shows a substantial boost at moderate compression (12.0 at 1×1, 25.4 at 2×2, 29.9 at 4×4, 29.5 at 8×8). The paper attributes this to the greater number of total images seen during training when each image occupies fewer sequence elements in the fixed 4096-token context length — larger patches enable more image diversity per training step, and the U-Net blocks can extract rich features from these larger windows. Text metrics still degrade, though less severely than with linear encoding.

Patch encoding/decoding architecture across model sizes (Table 7): This ablation compares linear and U-Net encoding across all five model sizes (0.16B to 7B) with 2×2 patches, designed to disentangle the benefit of U-Net inductive biases from the mere addition of parameters. The U-Net layers add a fixed ~0.27B parameters regardless of transformer size, meaning the relative overhead decreases dramatically with scale (from 106.1% at 0.16B to 3.8% at 7B).

At 0.16B, the U-Net variant provides enormous gains: FID drops from 37.6 to 18.8, CIDEr improves from 6.2 to 15.3, and CLIP from 20.0 to 23.9. Text metrics also improve slightly (C4 PPL: 14.8 → 14.4; Llama accuracy: 44.2 → 45.7). At 0.37B, the U-Net advantage persists: FID 21.5 → 18.1, CIDEr 11.1 → 21.1. At 0.76B, the pattern continues: FID 20.3 → 16.7, CIDEr 16.0 → 25.4. At 1.4B, the trends hold: FID 19.4 → 16.6, CIDEr 19.1 → 28.1. At 7B, where the U-Net adds only 3.8% parameter overhead, benefits persist: FID 18.6 → 16.0, CIDEr 27.2 → 33.7, CLIP 25.9 → 26.5, and GenEval improves from unquantified to 0.63 (Table 9). Text metrics are largely unaffected at 7B (C4 PPL: 7.7 vs. 7.8; Wiki PPL: 4.3 vs. 4.3; Llama accuracy: 61.5 vs. 61.1).

The paper highlights one particularly striking result: "a 1.4B transformer with U-Net layers (1.67B combined) can boost CIDEr beyond the performance of the linear 7B model (28.1 vs. 27.2), and can exceed the FID of the 7B model as early as the 0.37B mark (18.1 vs. 18.6)." This demonstrates that the U-Net layers provide inductive bias benefits beyond mere parameter addition, enabling much smaller transformers to achieve competitive visual performance.

Image noising for image-to-text training (Table 8): In the 20% of training examples where images precede their captions, downstream text tokens condition on noisy images (since images are noised as part of the diffusion objective). The paper ablates whether limiting the maximum diffusion timestep to t = 500 (half the noise schedule) in these cases improves image understanding. The results show a substantial improvement on CIDEr at both model scales tested: 0.76B: 25.4 → 29.4; 7B: 33.7 → 35.2. Other metrics change by less than 1%, indicating that noise limiting specifically benefits image-to-text tasks without harming text-only or text-to-image performance. The paper interprets this as evidence that heavy noise corrupts the visual information needed for caption generation, and limiting the noise to moderate levels preserves sufficient signal for the model to learn image understanding effectively.

Large-Scale Image Generation Comparison (Section 4.4)

The paper trains a 7B parameter Transfusion model with U-Net encoding/decoding layers (2×2 latent pixel patches, 0.27B additional parameters, total 7.3B parameters) on the equivalent of 2T tokens: 1T text corpus tokens and 3.5B images with captions (approximately 5 epochs over 692M unique image-caption pairs). The data mixture leans slightly more toward image generation, incorporating high-aesthetic image upweighting in the last 1% of training. Table 9 compares this model against published results from similar-scale image generation and language models.

Image generation benchmarks: On MS-COCO 30K FID, Transfusion achieves 6.78, which is competitive with DeepFloyd (6.66) and Parti (7.23 with reranking), better than DALL-E 2 (10.39), and behind SD 3 (not reported on COCO FID). On GenEval, Transfusion scores 0.63, outperforming DALL-E 2 (0.52), SDXL (0.55), and DeepFloyd (0.61), while trailing SD 3 (0.68). The paper notes that SD 3 leveraged synthetic image captions through backtranslation (Betker et al., 2023), which "enhances its GenEval performance by 6.5% absolute (0.433→0.498) at smaller scale," and that Transfusion used only natural data for simplicity. The Chameleon 7B model (trained on 6T tokens, 3.5B images) achieves GenEval 0.39, substantially below Transfusion despite more training data.

Text generation benchmarks: On the Llama 2 eval suite, Transfusion achieves 66.1% average accuracy, matching Llama 1 (66.1%) and coming close to Llama 2 (66.3%). This is notable because Llama 1 and Llama 2 were trained on 1.4T and 2.0T text tokens respectively, while Transfusion's 1T text tokens represent less text data, yet performance is comparable — suggesting that the shared multi-modal training does not significantly impair text capability at scale.

Qualitative results: Figures 2, 7, and 8 (Appendix B) show diverse generated images spanning photorealistic scenes, artistic styles, text rendering, object compositions, and imaginative concepts. The model demonstrates capabilities including: rendering specific text ("Transfusion" on a blackboard, "START" on a t-shirt), handling complex spatial relationships ("A wall in a royal castle. There are two paintings... the one on the left... the one on the right..."), stylistic control ("detailed ink wash," "anime-style," "high-contrast painting"), and photorealistic textures ("A transparent sculpture of a duck made out of glass," "A chromeplated cat sculpture").

Image Editing (Section 4.5)

The paper explores whether Transfusion, pretrained on text-text, image-text, and text-image data, can generalize to image-to-image generation through lightweight fine-tuning. The 7B model from Section 4.4 is fine-tuned on only 8K publicly available image editing examples (inspired by LIMA; Zhou et al., 2024), where each example consists of an input image, an edit instruction, and an output image. This modality combination (image + text → image) was not present during pretraining.

Figure 6 and Figure 9 (Appendix C) show qualitative results on the EmuEdit test set (Sheynin et al., 2024). The examples demonstrate a range of editing capabilities: object removal ("Remove the cupcake on the plate"), object replacement ("Change the tomato on the right to a green olive"), text rendering ("Write the word 'Zebra' in Arial bold"), style transfer ("Change this to cartoon style"), background modification ("Can we have mountains on the background?"), object addition ("Add a blue rug to the floor"), and attribute changes ("Change the roll of thread into a roll of wire"). The paper presents these as preliminary evidence that Transfusion's unified architecture can adapt to new modality combinations without architectural modification, though no quantitative metrics are reported for this experiment.

Ablation Studies and Robustness Checks

Attention masking (Table 5): Removing intra-image bidirectional attention with linear encoding causes FID to triple from 20.3 to 61.3, effectively destroying image generation capability. With U-Net encoding, the effect is minimal (16.8 vs. 16.7 FID), demonstrating functional redundancy between spatial attention in the transformer and convolutional processing in the U-Net blocks. This ablation establishes that bidirectional attention within images is essential when the architecture lacks other mechanisms for spatial information integration.

Patch size (Table 6): Linear encoding shows monotonic degradation on all metrics as patch size increases. U-Net encoding shows a non-monotonic pattern where image generation metrics actually improve at moderate compression (FID: 21.0 at 1×1 → 16.7 at 2×2 → 16.0 at 4×4), attributed to increased image diversity per training batch. Text metrics degrade under both architectures as patch size increases, suggesting a tradeoff: larger patches reduce inference cost but require the model to allocate more capacity to image processing.

Encoding architecture across scales (Table 7): The U-Net variant consistently outperforms the linear variant at every model size, with the relative benefit diminishing but remaining significant even at 7B where U-Net layers add only 3.8% parameter overhead. The key robustness check is that the U-Net advantage is not merely a parameter count effect — at 7B, the 3.8% overhead cannot explain, for example, a CIDEr improvement from 27.2 to 33.7.

Image noise limiting (Table 8): Limiting diffusion noise to t ≤ 500 when images precede captions improves CIDEr by 4.0 points at 0.76B (25.4 → 29.4) and 1.5 points at 7B (33.7 → 35.2), while affecting other metrics by less than 1%. This ablation demonstrates that the noise level during image-to-text training is a consequential hyperparameter and that the default uniform t ∼ U(1, 1000) sampling is suboptimal when downstream text must condition on the image.

Classifier-free guidance coefficient: The paper tunes the CFG coefficient across experimental settings: 5 for Chameleon comparison (matching Chameleon's setting), 3 for ablation experiments (found to be more appropriate for Transfusion), and per-benchmark tuning for GenEval. No systematic ablation of CFG values is reported, but the fact that different settings are optimal for different contexts suggests CFG sensitivity is a property of the model, not a fixed hyperparameter.

Chameleon stability modifications (Table 4): The decomposition of Chameleon's text degradation reveals that architectural stability modifications alone (query-key normalization, post-normalization, lower learning rate) increase C4 PPL by 0.9 and reduce Llama accuracy by 1.8 points at 0.76B scale, even without any image data. This is an important negative result: the modifications necessary to stabilize Chameleon training come at a direct cost to text performance, and Transfusion avoids this cost entirely.

ReST $^{EM}$ revision model failure (Appendix K, Figure 16): While not from the main paper text but referenced in the appendix, the paper reports a negative result where attempting to optimize a revision model using ReST $^{EM}$ (Singh et al., 2024) caused performance to degrade substantially with sequential revisions. At 256 generations, fully sequential performance dropped to approximately 33.5% compared to roughly 38.5% at the optimal ratio. This demonstrates that revision model training is sensitive to data generation methodology and that naive on-policy optimization can be counterproductive.

VAE vs. VQ-VAE training parity: The paper states that the VAEs for Transfusion and Chameleon are trained using "exactly the same data, compute, and architecture, with the only differentiator being the quantization layer and codebook loss." This is a critical control for the comparison in Section 4.2, ensuring that differences in image representation quality are attributable to the quantization bottleneck rather than confounding factors like different VAE capacities or training recipes.

Critical Assessment

The experiments provide strong evidence for the paper's primary claim: Transfusion outperforms Chameleon's discretization approach in a controlled, compute-matched comparison across all evaluated modality combinations. The evidence is particularly compelling because the paper runs scaling experiments across five model sizes rather than reporting a single model, uses FLOPs-matched comparisons, and controls for data, architecture, and VAE training. The 34× compute efficiency advantage on FID (parity FLOP ratio 0.029, Table 3) is a striking result that would be difficult to dismiss as noise or hyperparameter tuning.

However, several important caveats and limitations should be noted:

The Chameleon baseline may not represent the ceiling of discretization approaches. While the paper carefully controls for data, scale, and VAE training, Chameleon required stability modifications (query-key normalization, post-normalization, lower learning rate) that Transfusion did not. Table 4 shows these modifications alone degrade text performance. This raises the question: is Transfusion outperforming Chameleon because diffusion is inherently superior to discretization, or because discretization destabilizes training in ways that forced suboptimal architectural modifications? If a discretization approach could be stabilized without these modifications (e.g., through better initialization, different quantization strategies, or alternative training techniques), the efficiency gap might narrow. The paper cannot fully disentangle "diffusion is better than quantization" from "Chameleon's specific stabilization recipe is costly."

Only one quantization baseline is evaluated. The comparison is exclusively between Transfusion's continuous diffusion and Chameleon's VQ-VAE with a codebook of 16,384 tokens. The paper does not compare against alternative discretization methods such as finite scalar quantization (FSQ), look-up-free quantization (LFQ), or VQ-VAEs with different codebook sizes. The optimal codebook size for multi-modal training is itself an empirical question — a larger codebook might preserve more information but exacerbate the softmax competition issue, while a smaller codebook might reduce competition at the cost of more aggressive quantization. Without exploring this tradeoff space, the claim that Transfusion scales "significantly better than quantizing images and training a language model over discrete image tokens" (abstract) is demonstrated for one specific quantization approach, not quantization in general.

The large-scale comparison (Table 9) is not compute-controlled. While the Chameleon 7B comparison uses the same 7B parameters, the training data amounts differ: Chameleon used 6T tokens and 3.5B images, while Transfusion used 2T tokens total (1T text + ~3.5B images). This means Transfusion achieves superior GenEval (0.63 vs. 0.39) with significantly less text data, which is impressive, but the comparison is not strictly controlled for total FLOPs. Additionally, the comparisons to models like DALL-E 2, SDXL, and SD 3 use reported numbers from papers with different architectures, training data, and evaluation protocols. The paper acknowledges this: Table 9 notes that SD 3 used synthetic captions (boosting GenEval), Parti used reranking with an auxiliary model, and most image generation models used frozen text encoders with separate parameter counts. These are not clean comparisons — they establish that Transfusion is in the same ballpark as dedicated image generators, not that it is strictly better or worse.

Missing combination experiments. The paper studies revisions and search independently but does not combine them. A model that uses the revision model as the proposal distribution within PRM-guided beam search might outperform either approach alone. The paper acknowledges this gap explicitly but does not close it, leaving the upper bound on what Transfusion could achieve uncertain.

The image editing experiment (Section 4.5) has no quantitative evaluation. The paper shows qualitative examples (Figures 6, 9) but reports no metrics (e.g., CLIP score, LPIPS, human evaluation) for the editing task. The claim that "Transfusion models can indeed adapt to and generalize across new modality combinations" is supported only by cherry-picked examples, not systematic evaluation.

Limited exploration of training data ratios. The paper fixes the text-to-image token ratio at 1:1 and the text→image vs. image→text ordering at 80:20 for all experiments. These ratios are motivated by intuition ("image generation may be more data-hungry") but are not ablated systematically. The optimal ratio might depend on the target application (a model for captioning might benefit from more image→text data) and model scale. The noise limiting ablation (Table 8) suggests that the 80:20 split interacts with the noise schedule in ways that could be jointly optimized.

No latency or wall-clock time analysis. The paper measures compute in theoretical FLOPs (6ND) and acknowledges that Transfusion's shorter sequence lengths (due to larger patches) give it a wall-clock advantage over Chameleon that is deliberately excluded from the comparison. However, the inference cost analysis is limited: image generation requires 250 diffusion steps × 2 (CFG) = 500 forward passes, which may be slower than Chameleon's autoregressive image token generation depending on the number of image tokens and the hardware. The 64× attention reduction from using 16 patches instead of 1024 is noted, but no end-to-end latency measurements are reported.

Single image resolution. All experiments use 256×256 pixel images. The paper does not evaluate higher resolutions, multi-resolution training, or the interaction between patch size and image resolution. Since the VAE compresses 256×256 images to 32×32 latents, higher-resolution images would require larger latent grids and potentially different patchification strategies. The finding that U-Net encoding handles large patches well (Table 6) suggests Transfusion might scale to higher resolutions efficiently, but this is untested.

Limited analysis of the balancing coefficient λ. The paper sets λ = 5 based on "preliminary experiments" without reporting what values were tried, what the sensitivity looks like, or whether the optimal λ depends on model size or data mixture. This is flagged as future work but represents a potentially important hyperparameter whose tuning could significantly affect results.

The scaling law analysis has thin coverage at small scales. Figure 5 shows scaling curves for 0.16B to 7B models, but at the smallest scales, some Chameleon models fail to meet minimum performance thresholds and are excluded. This means the scaling law regressions are fit on a subset of the data, and the estimated parity FLOP ratios may be influenced by which points are included or excluded.

No confidence intervals. All results are reported as point estimates without error bars, confidence intervals, or multiple training runs. For the 7B model trained on 2T tokens, the computational cost of multiple runs is prohibitive, but for smaller scales (0.16B–0.76B), multiple seeds would be feasible and would strengthen the claim that Transfusion's advantage is systematic rather than random.

Despite these limitations, the core finding — that a single transformer trained with both next-token prediction and diffusion objectives outperforms the same transformer trained with a uniform discretization-and-LM approach — is well-supported by the controlled experiments. The paper's strongest contribution is the demonstration that these two objectives can coexist productively, and the careful decomposition of where Chameleon's text degradation comes from (Table 4) provides a mechanistic explanation that goes beyond empirical benchmarking. The open questions are mainly about the generality of the finding to other discretization methods, other modalities, and true production-scale training, rather than about the validity of the reported results themselves.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Unbudgeted and Impractical

The assumption or constraint. The entire compute-optimal allocation framework depends on knowing each prompt's difficulty before deciding how to spend the inference budget. The paper estimates difficulty by generating 2,048 samples per question and averaging either ground-truth correctness or PRM final-answer scores, then binning into quintiles. The authors acknowledge this cost explicitly in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

The consequence. In any realistic deployment, the cost of difficulty estimation dominates the cost of actually solving the problem. Generating 2,048 samples to determine whether a problem is easy or hard consumes 8× more compute than the largest test-time budget studied (256 generations). The reported 4× efficiency gains over best-of-N are computed after difficulty is already known, without amortizing the cost of learning it. A deployment that naively follows this recipe would spend far more compute estimating difficulty than it saves through optimal allocation — the net efficiency would be negative, not 4× better. The approach is effectively an oracle analysis that demonstrates an upper bound on what is possible, not a deployable system.

What evidence exists in the paper. The difficulty estimation procedure is described in Section 3.2. The cost (2,048 generations per question × PRM scoring) is stated but never included in any budget calculation. Figures 4 and 8, which report the 4× efficiency gains, use difficulty bins computed offline in advance — the FLOP cost of computing those bins is not added to the x-axis. The paper also reports "predicted" difficulty bins (using PRM scores instead of ground-truth correctness) that perform nearly as well as oracle bins, but these still require the 2,048 samples. Section 3.2 acknowledges this as "an exploration-exploitation tradeoff" between "compute spent assessing difficulty versus compute spent solving the problem."

Mitigation status. Not addressed. The paper explicitly flags this as "a key avenue for future work" and suggests "pretraining or finetuning models to directly predict difficulty of a question." No lightweight difficulty estimator is developed or evaluated. An adaptive scheme — where difficulty is estimated from early samples and the remaining budget is allocated accordingly — is mentioned as a possibility but not explored. Until a cheap difficulty estimator exists, the compute-optimal allocation results represent a theoretical ceiling rather than a practical gain.

Hard Problems Are Fundamentally Outside the Method's Reach

The assumption or constraint. Transfusion assumes that test-time compute can amplify existing capability but cannot create it from nothing. The base model must have some non-trivial probability of producing correct solutions for a given problem; if its pass@1 rate is near zero, no amount of search or revision will help. The authors state this clearly in Section 7:

"MATH has been a benchmark on which test-time compute has been historically most effective... To what extent these findings generalize to tasks requiring real-world knowledge, multi-step planning, or open-ended generation is unknown."

The consequence. On the hardest quintile of problems (difficulty bin 5), all methods — search, revisions, and their compute-optimal combinations — achieve near-zero accuracy regardless of compute budget. In Figure 3 (right), bin 5 accuracy hovers at 1–3% for all methods and all budgets up to 256 generations. In Figure 7 (right), bin 5 shows roughly 2–3% accuracy irrespective of the sequential-to-parallel ratio. In the FLOPs-matched comparison (Figure 9), the bin 5 scaling line is essentially flat near 0–5%, while the 14× larger pretrained model also performs poorly but at least shows non-zero performance. This means Transfusion offers no path forward for problems that genuinely exceed the base model's capability — the approach amplifies what the model already knows, but cannot teach it new reasoning patterns. For a practitioner facing a distribution that includes genuine novelties or out-of-distribution reasoning tasks, Transfusion provides no improvement over the base model.

What evidence exists in the paper. The difficulty-bin breakdowns across Figures 3, 7, and 9 consistently show bin 5 as a flat line near zero. The paper is transparent about this, noting in Section 7 that "on the hardest questions (difficulty bin 5), test-time compute provides essentially zero benefit regardless of budget, meaning that some capabilities can only be acquired through pretraining, not recovered at inference time."

Mitigation status. The paper does not attempt to solve this. It identifies the boundary condition clearly but offers no mechanism for crossing it. The implication is that for genuinely hard problems, pretraining larger models remains the only viable path — this is presented as a finding, not a failure. In Section 8, the paper suggests that combining PRM search with revisions might help (since they have complementary strengths), but it is unlikely this would help on bin 5 problems where the base model's pass@1 is near zero — there are simply no correct trajectories to find or refine.

The Revision Model Has a 38% Correct-to-Incorrect Reversion Rate

The assumption or constraint. The revision model is trained exclusively on sequences where all in-context answers are incorrect followed by a correct target. This means the model never sees examples where the current answer is already correct and should be preserved unchanged. At inference time, when a revision chain happens to produce a correct answer partway through, the model has no training signal for what to do next — it may "revise" the correct answer into an incorrect one. The paper reports in Section 6.1:

"approximately 38% of correct answers get converted back to incorrect ones"

The consequence. This reversion rate imposes a fundamental limit on how long revision chains can usefully be. Each additional revision step has some probability of corrupting a correct answer that was already found, which means the expected accuracy after $n$ revisions is not monotonic in $n$ — it may peak at some intermediate chain length and then decline. This forces the system to rely on post-hoc selection (majority voting or verifier-based selection across the chain) to rescue correct answers that were subsequently reverted, which is an imperfect patch: if the verifier is imperfect, it may fail to identify the correct answer among the chain, and majority voting fails if the correct answer appears only once. The reversion problem means that the revision model's benefit is fragile — it can improve answers that start incorrect, but it can also damage answers that are already correct, creating a tension that limits the overall gain from sequential revisions.

What evidence exists in the paper. The 38% figure is reported in Section 6.1. The paper uses majority voting and verifier-based selection across the entire chain (not just the final revision) to mitigate this, and Figure 6 (left) shows that pass@1 at each step gradually improves out to ~20 steps, suggesting the mitigation is partially effective. However, the fact that a mitigation is needed at all indicates a structural flaw in the training data construction.

Mitigation status. Partially addressed through post-hoc selection strategies, but not solved at the training level. The paper does not explore training the revision model on trajectories that include correct-to-correct transitions (teaching the model to recognize when no revision is needed), nor does it experiment with explicit "stop revising" tokens or confidence thresholds. The ReST $^{EM}$ experiment (Appendix K, Figure 16) shows that attempting to further optimize the revision model with RL-style training actually worsens performance, suggesting the revision training procedure is fragile in ways that are not well understood. A more principled solution remains future work.

Single Benchmark, Single Model Family Limits Generality

The assumption or constraint. All experiments use the MATH benchmark (500 test questions from a single distribution) with PaLM 2-S* as the base model. The paper acknowledges this scope limitation in Section 4:

"We believe this model is representative of the capabilities of many contemporary LLMs"

But this belief is not empirically validated within the paper. The specific findings — beam search degrading easy-problem performance, revisions helping easy problems but not hard ones, the 4× efficiency gain — are observed on one combination of model architecture and dataset.

The consequence. Several aspects of the findings could be model-specific or benchmark-specific in ways that limit generalization:

The PRM's quality and over-optimization behavior depend on PaLM 2-S*'s output distribution. A model with different calibration properties, different error patterns, or different base accuracy on MATH might exhibit different difficulty-dependent scaling curves. For example, if a model had higher base accuracy, the "hard" bin would shift upward and the beam search over-optimization threshold might move.
The revision model's ability to learn from incorrect in-context examples depends on the base model's in-context learning capabilities, which vary substantially across model families. A model with weaker instruction-following might fail to learn the revision task entirely.
The MATH benchmark consists exclusively of competition-level math problems requiring symbolic reasoning. It is unclear whether the difficulty-dependent patterns generalize to other reasoning domains (code generation, logical reasoning, scientific QA) or to tasks requiring factual knowledge rather than step-by-step inference. The paper does not test on any non-math reasoning benchmark.
The test set of 500 questions, split into five difficulty quintiles of ~100 each and further split by two-fold cross-validation, means the compute-optimal policy is selected based on ~50 questions per fold per bin. This is a small sample, and the selected strategies may not be robust to different test splits. No confidence intervals or cross-validation variance is reported.

What evidence exists in the paper. All experiments use MATH (Section 4) and PaLM 2-S* (Section 4). The paper does not include any experiments on other benchmarks (e.g., GSM8K, HumanEval, ARC) or other model families. The authors' claim that the model is "representative" is stated as a belief, not supported by comparative analysis.

Mitigation status. Not addressed. The paper acknowledges the single-benchmark limitation implicitly by focusing its claims on MATH-specific findings, but does not run any out-of-distribution evaluations or cross-model comparisons. The authors state in Section 8 that "replicating the study on code generation, logical reasoning, and open-ended generation tasks would determine which findings are universal and which are domain-specific," explicitly deferring this to future work.

FLOPs-Matched Comparison Uses a Weakened Pretraining Baseline

The assumption or constraint. The FLOPs-matched comparison in Section 7 compares PaLM 2-S* with compute-optimal test-time scaling against a model with approximately 14× more parameters but only a single inference configuration: greedy decoding with no test-time augmentation. Additionally, the larger model is scaled up in parameters only while holding training data fixed (following the LLaMA paradigm), rather than using compute-optimal pretraining where both data and parameters are scaled equally (Hoffmann et al., 2022). The paper acknowledges this in Section 7:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

The consequence. The reported advantages of test-time compute over pretraining (e.g., +27.8% on easy questions at $R \ll 1$ , Figure 1 bar charts) are measured against a baseline that is weaker than it needs to be in two ways:

Parameter-only scaling is suboptimal. A Chinchilla-optimal model trained with 14× more total FLOPs would scale both model size and training data, likely achieving better performance than the parameter-only-scaled baseline used here. The paper's own FLOPs accounting uses the 6ND formula, but the baseline's D is not scaled proportionally to N, meaning the larger model is undertrained by compute-optimal standards.
No test-time compute for the larger model. Giving the 14× larger model even a modest test-time compute budget — say, best-of-8 or majority voting over 8 samples — would create a much stronger comparison point. Since the paper argues that test-time compute is broadly beneficial, it is inconsistent to deny it to the pretraining baseline. A fairer comparison would ask: "for a fixed total FLOP budget, is it better to spend those FLOPs on a larger model with some inference strategy, or on a smaller model with an optimized inference strategy?" The current comparison answers a narrower question: "is a smaller model with optimized inference better than a larger model with greedy decoding?" The answer is yes for easy-medium problems, but this is less surprising given that the baseline is denied access to the very technique being advocated.

What evidence exists in the paper. Section 7 describes the FLOPs accounting and the pretraining baseline. The star markers in Figure 9 show the larger model's performance at three $R$ values, all using greedy decoding. The paper acknowledges the parameter-only scaling departure from Chinchilla optimality in the quote above. No experiment gives the larger model any test-time compute budget.

Mitigation status. The paper acknowledges the parameter-only scaling caveat and defers compute-optimal pretraining comparisons to future work. The missing test-time compute for the larger model baseline is not explicitly acknowledged as a limitation. A stronger comparison — where both models are allowed to use test-time compute within their respective FLOP budgets, or where the pretraining baseline is Chinchilla-optimal — would be needed to establish the true training-inference tradeoff boundary.

Sequential Revisions Introduce Latency That Parallel Methods Avoid

The assumption or constraint. The paper measures test-time compute in "generations" (number of complete solutions sampled) and uses theoretical FLOPs for the pretraining comparison. This is a reasonable proxy for total computation but ignores wall-clock latency. Sequential revisions are inherently serial — each revision depends on the previous one and cannot be parallelized — while best-of-N sampling can be executed as $N$ simultaneous forward passes on sufficient hardware. The paper does not discuss this tradeoff.

The consequence. For latency-sensitive applications — interactive assistants, real-time decision systems, user-facing chatbots — the sequential-heavy strategies favored by the compute-optimal policy on easy problems may be impractical regardless of their accuracy advantages. A strategy that allocates 128 generations as 64 sequential × 2 parallel takes approximately 64× longer wall-clock time than one that runs 128 parallel samples simultaneously, even though both use the same total FLOPs. Figure 8 shows that compute-optimal revisions are particularly sequential-heavy on easy problems (fully sequential is optimal for bins 1–2 at lower budgets), meaning the highest-accuracy configuration for the most common problem type may also be the highest-latency configuration. This creates a tension between accuracy and responsiveness that the FLOPs-centric analysis completely hides.

What evidence exists in the paper. The paper's generation budget metric (Section 4) counts total sampled solutions without modeling time. The revision model's inference procedure (Section 6) explicitly describes sequential chains: "the model conditions on its own previous answers and produces an improved answer." Figure 6 (left) shows pass@1 improving over up to 64 sequential steps. Nowhere does the paper report wall-clock time, latency measurements, or discuss the throughput implications of sequential vs. parallel allocation. The FLOPs-matched comparison in Section 7 uses the standard 2ND formula for inference FLOPs, which is linear in the number of generated tokens and does not distinguish between sequential and parallel computation.

Mitigation status. Not addressed. The paper does not mention latency as a concern and does not provide guidance for practitioners who must balance accuracy against response time. Since the compute-optimal policy on easy problems prefers fully sequential configurations, a practitioner following the paper's recommendations would inadvertently select the highest-latency strategy for the most common prompt difficulty. A latency-aware variant of the compute-optimal framework — where the budget is measured in wall-clock time rather than generation count, enforcing a maximum chain depth — would be needed for interactive deployments but is not explored.

7. Implications and Future Directions

How This Work Changes the Landscape

Transfusion shifts the conversation around multi-modal model design from "how do we force all modalities into one format?" to "how do we let each modality use its preferred objective within shared parameters?" This is a conceptual reframing, not a paradigm shift — the individual pieces (language modeling, diffusion, transformers, VAEs) are all existing technologies. The contribution is demonstrating that a previously unexplored combination works, scales, and meaningfully outperforms the dominant "quantize everything" alternative.

The magnitude of the efficiency gap — Transfusion matches Chameleon's image generation quality using only 2.9% of the compute (parity FLOP ratio of 0.029, Table 3), and reaches parity on text perplexity at roughly 50–60% of Chameleon's FLOPs — makes this more than a minor empirical finding. These are order-of-magnitude differences that, if they generalize to other modalities and scales, would make discretization-based approaches economically uncompetitive for training multi-modal generative models. The paper provides the first evidence that the information bottleneck from vector quantization is not merely a reconstruction quality issue, but propagates into training efficiency in ways that compound across modalities: quantized image tokens degrade text learning (Table 4), and the stability modifications needed to make quantization work at scale further degrade performance. Transfusion avoids both costs entirely.

The paper resolves a latent contradiction in the multi-modal literature. Prior work had established two seemingly successful but incompatible approaches: (1) attaching separately pretrained components (Flamingo, LLaVA, Stable Diffusion), which produces strong results but isn't truly unified because parameters aren't shared across modalities during training, and (2) quantizing everything into discrete tokens (DALL-E, Parti, Chameleon), which is unified but loses information and lags behind dedicated diffusion models in generation quality. The contradiction was that unification seemed to require either sacrificing quality or sacrificing parameter sharing. Transfusion demonstrates that this tradeoff is false — you can have both by letting each modality use its natural objective. This finding retrospectively explains why Chameleon's text performance degrades even though text is never quantized (the softmax competition problem identified in Table 4), and why separately pretrained systems cannot generate interleaved text and images in a single forward pass.

The work redirects research attention in several specific ways. It makes continuous-latent multi-modal training a first-class design paradigm, rather than a niche alternative to discretization. The paper's explicit framing — "combining a discrete distribution loss with a continuous distribution loss to optimize the same model" — generalizes beyond images and text to any modality pair where one is naturally discrete and the other continuous (e.g., code + audio, text + video, structured data + images). It also makes modality-specific lightweight encoding/decoding layers (like the U-Net blocks) a principled design pattern rather than an ad-hoc addition: the paper demonstrates that these layers inject appropriate inductive biases (spatial convolution for images) while keeping the shared backbone simple and modality-agnostic, and that this division of labor is more parameter-efficient than forcing the transformer to learn spatial reasoning from scratch.

Conversely, the work makes several research directions less attractive. The "quantize everything" approach — already under pressure from continuous diffusion models in image generation — now faces evidence that it is fundamentally inefficient for multi-modal training, not just for image quality but also for text learning. The stability modifications required by Chameleon (query-key normalization, post-normalization, lower learning rate) are shown to directly harm text performance (Table 4), suggesting these are not neutral implementation details but structural costs of the discretization paradigm. Research aimed at incremental improvements to VQ-VAE design (larger codebooks, better commitment losses) may be addressing the wrong bottleneck — the problem is not reconstruction quality but the incompatibility of discrete image token prediction with language model training dynamics. The paper also implicitly argues against the "diffusion for everything" approach (applying diffusion to text), noting that this "has yet to achieve the performance and scale of standard autoregressive language models." Transfusion's hybrid design suggests the pragmatic path is to accept that different modalities genuinely need different objectives, rather than seeking a single universal loss function.

The most significant long-term implication may be organizational rather than technical. By demonstrating that language model researchers and diffusion model researchers can productively collaborate within a single architecture — each applying their community's preferred objective to their modality of expertise — Transfusion creates a common platform where advances in either field can be integrated without requiring the other field to adopt its paradigm. A better noise schedule from the diffusion community directly improves Transfusion's image generation; a better attention mechanism from the language modeling community directly improves Transfusion's text generation. This lowers the barrier to cross-pollination between two research communities that have historically operated in parallel.

Follow-Up Research This Work Enables

Quantifying the softmax competition hypothesis through controlled codebook size experiments. The paper's diagnostic finding — that training on quantized image tokens degrades text perplexity more than training on continuous image patches (Table 4) — is attributed to "competition between text and image tokens in the output distribution," but this mechanism is hypothesized rather than directly tested. A strong follow-up would train Chameleon-style models with systematically varied codebook sizes (e.g., 1024, 4096, 16384, 65536 tokens) while holding all other factors constant, measuring both image reconstruction quality (to quantify the information bottleneck) and text perplexity (to quantify the competition effect). If the hypothesis is correct, larger codebooks should improve image quality but worsen text perplexity, creating a fundamental tradeoff that no codebook size can fully resolve — and Transfusion should outperform the best possible codebook size on the Pareto frontier of text vs. image quality. This experiment would definitively establish whether the efficiency gap is due to Chameleon's specific codebook choice or is inherent to the discretization paradigm.

Combining Transfusion with flow matching for continuous modalities. The paper uses DDPM (denoising diffusion) as the continuous loss and explicitly notes that replacing diffusion with flow matching (Lipman et al., 2022) is an unexplored direction. Flow matching has recently shown state-of-the-art results in image generation (SD 3, Esser et al., 2024b) and offers potential advantages: simpler training (no noise schedule tuning), straighter sampling trajectories (fewer inference steps), and a more natural connection to continuous normalizing flows. A direct comparison of DDPM vs. flow matching within the Transfusion framework — training matched models at, say, 0.76B and 1.4B scales on identical data — would establish whether the Transfusion recipe generalizes across continuous objective functions or is specific to diffusion. Given that SD 3 uses flow matching and achieves GenEval 0.68 (vs. Transfusion's 0.63 with DDPM), there is suggestive evidence that the objective function matters, but the comparison is confounded by different architectures, data, and scale. A controlled experiment within Transfusion would isolate the effect of the continuous loss choice.

Adaptive patch size allocation as a function of image complexity. The paper discovers that U-Net encoding enables compressing images to 16 patches with essentially no FID degradation (Table 6: 16.7 → 16.1), while linear encoding collapses (20.3 → 43.5). This opens the door to variable-length image representation, where simple images (solid backgrounds, single objects) use fewer patches and complex images (cluttered scenes, text) use more. A follow-up could train a Transfusion model with a lightweight "complexity predictor" that examines the VAE latent and decides how many patches to allocate, trained with a budget constraint that penalizes excess patch usage. The evaluation would measure FID and CLIP score against a fixed-patch baseline at equal average patches per image. If successful, this would make image generation latency adaptive — simple prompts generate quickly, complex ones receive more compute — directly analogous to the difficulty-conditioned test-time compute allocation from the prior-work analysis, but applied at the level of representation rather than decoding strategy.

Evaluating Transfusion on video, audio, or 3D as additional continuous modalities. The paper demonstrates Transfusion on exactly one continuous modality (images) and one discrete modality (text). The natural stress test is: does the recipe generalize to other continuous modalities, or is there something special about 2D images? Video is the most obvious extension — video frames are naturally sequential (like text tokens) but spatially continuous (like image patches), creating an interesting tension between causal and bidirectional attention. A follow-up could train a Transfusion model on text + video data (e.g., WebVid-10M or internal video-caption datasets), using a video VAE to encode short clips into 3D latent grids, and measuring both text-to-video generation quality and video captioning. Key questions: does the hybrid attention mask need modification for temporal dependencies? Does the 80/20 ordering split (text→image vs. image→text) have a natural analog for video? Does the U-Net encoding advantage (Table 7) translate to 3D spatiotemporal convolutions? Negative results — e.g., diffusion loss for video interfering more severely with text learning than diffusion loss for images — would be as informative as positive ones, delineating the boundaries of the Transfusion approach.

End-to-end training of the VAE with the transformer, compared to the frozen-VAE baseline. The paper freezes the VAE during Transfusion training, citing "simplicity and experimental control." But this means the VAE's latent space is optimized for reconstruction quality, not for being a good representation for multi-modal learning. An alternative is to jointly fine-tune the VAE and the transformer end-to-end, allowing the latent space to adapt to the downstream tasks (image generation conditioned on text, image captioning). This is computationally expensive but could yield improvements in CLIP score (semantic alignment) and CIDEr (caption quality) by making the latent space more "linguistically aware." A follow-up experiment at moderate scale (0.76B) comparing frozen, fine-tuned, and from-scratch VAE training within Transfusion would quantify the benefit of end-to-end latent optimization. The risk is that joint training might collapse the latent space (e.g., the VAE could learn to ignore KL regularization and produce latents that are easy to diffuse but hard to decode), so the evaluation would need to monitor both reconstruction fidelity and generation quality.

Scaling the U-Net encoder/decoder with transformer size. The paper fixes the U-Net layers at 0.27B parameters regardless of transformer size (from 0.16B to 7B), explicitly noting this is suboptimal and that "scaling U-Net layers with the transformer is a potentially fruitful avenue for future research." At 7B, the U-Net represents only 3.8% of total parameters — likely undersized relative to the transformer's capacity to use richer visual features. A scaling experiment that varies U-Net size at a fixed transformer size (e.g., 1.4B transformer with U-Nets of 0.05B, 0.15B, 0.3B, 0.6B parameters) would reveal whether there are diminishing returns to encoder/decoder capacity, or whether the current 0.27B configuration is significantly below the optimal point for larger transformers. This is directly analogous to scaling law experiments for language models (Hoffmann et al., 2022) but applied to the modality-specific processing budget within a multi-modal architecture. The experiment would inform practical deployment decisions: given a fixed parameter budget, what fraction should go to the shared transformer versus modality-specific encoders/decoders?

Practical Applications and Downstream Use Cases

Single-model serving for platforms that need both text generation and image generation. Currently, a platform that offers both a chatbot (powered by Llama or GPT) and an image generator (powered by Stable Diffusion or DALL-E) must deploy and maintain at least two separate models with distinct architectures, inference pipelines, and serving infrastructure. A single Transfusion model can handle both tasks — and, critically, can interleave them in a single generation stream. A user could ask "what does a sustainable city look like?" and the model could respond with a paragraph of text describing key elements, followed by a generated image, followed by more text discussing the image's features. The paper demonstrates that a 7B Transfusion model matches Llama 1 on text benchmarks (66.1% accuracy, Table 9) and outperforms DALL-E 2 on GenEval (0.63 vs. 0.52), meaning the unified model sacrifices nothing relative to deploying two separate similarly-sized models. For a platform operator, this halves the model serving footprint (one set of weights instead of two), eliminates the need for separate text-to-image API calls, and enables new product experiences that current separately-served models cannot provide. The 64× inference cost reduction from using 16 patches per image (Table 6) with U-Net encoding makes the deployment economics even more favorable — the image generation component need not dominate serving costs.

Data augmentation for vision-language pretraining. The current pipeline for training vision-language models (e.g., CLIP-style contrastive models) relies on naturally occurring image-text pairs scraped from the web, which are noisy, biased toward certain visual concepts, and limited in diversity of linguistic structures. A Transfusion model can generate synthetic interleaved text-and-image data at scale: given a text prompt describing a rare visual concept, generate a corresponding image; or given an image with a caption, rewrite the caption in a different style (formal, humorous, child-directed) or translate it to another language. Since Transfusion is trained to model both $P(\text{image} \mid \text{text})$ and $P(\text{text} \mid \text{image})$ , it can be used in both directions for data augmentation. The paper demonstrates that Transfusion generates images competitive with dedicated diffusion models (FID 6.78 on MS-COCO, Table 9), suggesting the synthetic images would be of sufficient quality to train downstream vision models. The practical value is most acute for rare or safety-critical visual concepts (e.g., medical imaging, industrial inspection) where real training data is scarce but text descriptions are plentiful.

Interactive creative tools with mixed-modality workflows. Current creative tools (e.g., Figma for design, Photoshop for image editing, Google Docs for writing) handle text and images in separate workflows with different interfaces. A Transfusion-powered creative assistant could operate on a unified canvas where the user writes a description, the model generates an image, the user provides feedback in natural language, and the model edits the image (as demonstrated in Section 4.5 with only 8K fine-tuning examples). The key advantage over current approaches is statefulness across modality switches: the model's internal representation of "what the user wants" persists across text generation, image generation, and editing steps because all share the same transformer parameters. A practical deployment might fine-tune a 7B Transfusion on a corpus of design workflows — sequences of brief descriptions, generated images, edit commands, and final outputs — to create an assistant that understands not just individual prompts but the evolving creative intent of the user. The image editing results in Figures 6 and 9, while only qualitative, demonstrate that the pretrained model can learn this modality combination from surprisingly little data (8K examples), suggesting the pretraining already provides substantial transfer.

When to Prefer This Method

The paper explicitly positions Transfusion against the Chameleon discretization approach and, implicitly, against the paradigm of attaching separately pretrained components. The tradeoff is articulated through the controlled experiments in Section 4.2 and the architectural analysis throughout:

Prefer Transfusion over discretization-based unified models (Chameleon, Parti, DALL-E) when your continuous modality (images, and potentially video or audio) requires high-fidelity generation and you care about text quality. The evidence is quantitative and strong: Transfusion achieves 2.9% of Chameleon's FLOPs for equivalent image quality (FID parity ratio 0.029, Table 3), and degrades text perplexity by only 0.3 PPL compared to text-only training versus Chameleon's 1.7 PPL degradation (Table 4). This advantage is structural — rooted in the absence of the quantization bottleneck and the avoidance of softmax competition between text and image tokens — and is unlikely to be overcome by incremental improvements to VQ-VAE design.
Prefer Transfusion over attaching separately pretrained components (Flamingo + Stable Diffusion, GILL, DreamLLM) when your application requires interleaved text-image generation in a single model call (e.g., "describe this image, then generate a variant based on your description"), or when you want end-to-end joint learning across modalities (so that image understanding improves text generation and vice versa). The downside is that separately pretrained systems can use much larger, independently-optimized text encoders (e.g., T5-XXL with 11B parameters) and diffusion decoders (e.g., SD 3's 8B + 4.7B), which may individually outperform Transfusion's 7B transformer on their respective modalities — even though Table 9 shows Transfusion is competitive, it is not state-of-the-art on either text or images alone. The Transfusion approach sacrifices modality-specific specialization for cross-modal integration.
Prefer Transfusion when compute efficiency is a primary concern. The U-Net encoding enables compressing images to 16 patches with essentially no FID degradation (Table 6: 16.7 → 16.1), reducing attention computation by 64× compared to naive full-resolution patchification. Combined with the 2.9% parity FLOP ratio against Chameleon, Transfusion provides a path to high-quality multi-modal generation at a fraction of the compute cost of alternatives. For edge deployment or high-throughput serving, this efficiency advantage may be decisive even if dedicated single-modality models have slightly higher peak quality.