Towards Scalable Pre-training of Visual Tokenizers for Generation

🎯 Pitch

Simply scaling up visual tokenizers with reconstruction loss can actually hurt generation quality—a tenfold compute increase makes image generation worse. This paper shows that jointly pre-training the tokenizer with perception and semantic objectives unlocks a new scaling law, where more compute, data, and model size reliably translate into better generative models, reaching a 1.11 gFID on ImageNet.

1. Executive Summary

This paper proposes Visual Tokenizer Pre-training (VTP), a unified pre-training framework for the visual tokenizer in latent diffusion models that jointly optimizes image-text contrastive, self-supervised, and reconstruction losses to encode high-level semantics rather than low-level pixel detail alone. Studying ViT-based autoencoders evaluated via downstream DiT training on ImageNet 256×256 class-conditional generation and LAION text-to-image generation, the work demonstrates that conventional reconstruction-only tokenizers exhibit a pre-training scaling problem—better reconstruction accuracy fails to improve and can even degrade generation quality (gFID rising from 55.04 to 58.56 as training FLOPs increase 10×)—while perception-oriented pre-training unlocks a new scaling law where generative performance improves consistently with tokenizer compute, parameters, and data, achieving a 65.8% FID improvement for a 10× compute increase and enabling 4× efficiency gains over saturated baselines. The final VTP-L model reaches 1.11 gFID with guidance on ImageNet alongside 0.36 rFID, 78.2% zero-shot accuracy, and 85.7% linear probing accuracy, establishing that semantic understanding is the key driver of generation and that representation-learning objectives must be integrated into tokenizer pre-training to achieve scalable generative performance.

2. Context and Motivation

The Core Problem: Reconstruction Quality and Generation Quality Are Decoupled

The fundamental problem this paper addresses is a paradox at the heart of latent diffusion model (LDM) design. LDMs, which currently dominate state-of-the-art image and video generation, operate in a two-stage pipeline: first, a visual tokenizer (typically a VAE) compresses images into a lower-dimensional latent space; second, a generative model (e.g., a diffusion transformer) learns to generate novel images by denoising random noise in that latent space (Rombach et al., 2022). This design decouples the heavy lifting of pixel-space modeling from the generative process, enabling training on compressed representations that are computationally tractable.

The standard approach to training the first-stage tokenizer is to optimize it for reconstruction fidelity: given an image, can the tokenizer encode it into latents and then decode those latents back into a faithful reproduction of the original? Metrics like PSNR, SSIM, and rFID (reconstruction Fréchet Inception Distance) measure how well the decoded image matches the input. Intuitively, one might expect that a tokenizer that reconstructs images more accurately would provide a better foundation for generation—after all, if the latent space preserves detailed image information, the diffusion model should have an easier time learning to produce realistic outputs.

This intuition is wrong. The paper demonstrates empirically that "better pixel-level reconstruction accuracy does not lead to higher-quality generation" (Section 1). In fact, the relationship can be inverse: as reconstruction quality improves with increased training compute, generation quality can actually degrade. Figure 4 shows this starkly: the rFID of a reconstruction-only tokenizer improves from 2.07 to 0.51 as training FLOPs scale by 10×—a substantial gain in reconstruction fidelity—while the downstream gFID (generation quality metric) worsens from 55.04 to 58.56 over the same compute range. More compute spent on the tokenizer produces a worse generative model.

This is not a minor inefficiency. It represents a foundational misalignment between the pre-training objective of visual tokenizers and their ultimate purpose in generative models. The phenomenon has been observed implicitly in prior work—Esser et al. (2024) and Yao et al. (2025) both document a trade-off between reconstruction and generation—but this paper is the first to characterize it explicitly as a scaling problem and to offer a systematic solution.

Why This Problem Matters: The Economic and Practical Stakes

The decoupling between reconstruction and generation has severe practical consequences for anyone training large-scale generative models.

Wasted compute at scale. Training visual tokenizers is typically a fraction of the total cost of building an LDM, but it is not negligible—especially as models push toward higher resolutions, larger datasets, and more aggressive compression ratios. If reconstruction-optimized training yields diminishing or negative returns for generation, then every FLOP spent beyond the initial saturation point is wasted. The paper frames this as a "pre-training scaling problem" (Section 1), arguing that the field lacks a principled understanding of how to scale tokenizer pre-training to improve generative performance. This gap is analogous to the pre-Chinchilla era in language model scaling, where researchers scaled model size without scaling data optimally, unknowingly leaving performance on the table.

Deployment constraints favor better tokenizers. In the two-stage LDM paradigm, the tokenizer is a shared upstream component: a single pre-trained tokenizer serves all downstream generative models (class-conditional, text-to-image, video, etc.). Improving the tokenizer improves every downstream model trained on its latents, without requiring any modification to the diffusion model architecture or training recipe. This makes tokenizer quality an exceptionally high-leverage research investment. If a better tokenizer can cut the diffusion model's training time in half or improve final FID by several points—both effects demonstrated in this paper—the downstream savings multiply across every generative model that uses that tokenizer.

The semantic bottleneck in generation. Generative models are evaluated not just on pixel-level realism but on semantic coherence, compositional correctness, and text-alignment (for text-to-image tasks). A tokenizer that prioritizes low-level information (textures, edges, fine details) at the expense of high-level structure may reconstruct images faithfully while encoding them in a way that is difficult for the diffusion model to generate from scratch. The diffusion model must learn to produce latent codes that decode into globally coherent images with correct object relations and semantic attributes. If the latent space is organized around low-level pixel statistics rather than semantic content, the diffusion model faces a much harder learning problem. This paper's core hypothesis is that "an effective latent space for generation should efficiently encode the core visual semantics" (Section 1), and that ignoring this semantic requirement is why reconstruction-only training hits a ceiling.

Prior Approaches and Where They Fall Short

The paper surveys existing strategies for improving visual tokenizers and categorizes their limitations.

Reconstruction-Only Training (the Standard Baseline)

The dominant paradigm for tokenizer pre-training is to use a combination of pixel-space reconstruction losses (L1, L2, perceptual loss, GAN loss) to train the encoder-decoder pair (Kingma and Welling, 2013; Esser et al., 2021; Rombach et al., 2022). The VAE framework adds a KL regularization term that encourages the latent distribution to be approximately Gaussian—convenient for the diffusion model to sample from—but this regularization is a weak semantic constraint. It encourages smoothness and coverage but does not explicitly enforce that semantically similar images map to nearby latent codes or that the latent space has a meaningful global structure.

The paper's Figure 4 demonstrates the systematic failure of this approach when scaled: across a 10× increase in training FLOPs, gFID monotonically degrades from 55.04 to 58.56. The curve in Figure 4 (right) shows that generation performance saturates early and then actually worsens, while reconstruction rFID continues improving. This is the "pre-training scaling problem" that motivates the entire work.

Why does this happen? The paper offers a mechanistic hypothesis (though it does not prove it causally): the reconstruction objective "biases the latent space toward low-level information and, as training scales up, increasingly drives it away from the structured latent space we ultimately desire" (Section 1). In other words, the tokenizer learns to allocate its limited latent capacity primarily to textures, edges, and pixel-level details that improve reconstruction metrics, at the expense of the semantic organization that makes generation tractable. The latent space becomes an efficient compression codec but a poor representation for learning to generate novel images.

Methods That Distill from Pre-Trained Representation Models

A class of prior work attempts to improve tokenizer quality by aligning the latent space with features from powerful pre-trained vision models. The intuition is straightforward: DINOv2, CLIP, and similar models produce semantically rich representations that capture object identity, spatial relationships, and high-level scene structure. If the tokenizer's latent space can be encouraged to match these representations, it should inherit their semantic properties.

Several specific approaches fall into this category:

VA-VAE (Yao et al., 2025): Aligns the VAE's latent space with features from visual foundation models to alleviate the reconstruction-generation trade-off. This work is a direct precursor to VTP, sharing some authors. It demonstrated that representation alignment helps but does not explore scaling behavior—it shows improvements at fixed compute scales.
REPA-E (Leng et al., 2025): Optimizes the VAE's feature space during DiT training (not during tokenizer pre-training) by leveraging supervision from a pre-trained foundation model. This is an in-the-loop rather than upstream approach.
RAE (Zheng et al., 2025): Leverages DINOv2 features and trains a separate pixel decoder for reconstruction. The encoder produces DINOv2-aligned latents, and a decoder is trained to reconstruct images from them. This is a representation-first approach.
MAETok (Chen et al., 2025a): Enhances latent representations by incorporating DINOv2 features into a masked image modeling (MIM) pre-training objective for the tokenizer.
ImageFolder (Li et al., 2024): Decouples semantic and pixel-level feature spaces to improve autoregressive image generation.

Critical limitation: dependence on frozen teacher models. The paper identifies a fundamental problem with distillation-based approaches: they are "inherently limited by existing foundational models, often leading to a low performance ceiling or substantial reconstruction loss" (Section 2.2). The pre-trained representation model (e.g., DINOv2) is frozen and provides a fixed target. The tokenizer can approach but never exceed its teacher's representational quality. Moreover, the teacher was trained for discriminative or self-supervised tasks, not for generation—there is no reason to believe its feature space is optimal for guiding a generative model. Finally, aggressive alignment to a pre-trained representation can hurt reconstruction fidelity, since the teacher's features discard precisely the low-level details (textures, edges) that reconstruction requires.

The paper provides concrete evidence for this ceiling effect. Table 3 shows that RAE, a distillation-based method, degrades in generation performance as the tokenizer scales from Small to Large (gFID worsens from 3.50 to 6.09), while VTP consistently improves (gFID improves from 5.46 to 2.81). The distillation approach hits a ceiling because the teacher's fixed representation capacity becomes the bottleneck.

Methods That Inject Specific Semantic Objectives

A second family of approaches enriches the latent space with targeted semantic objectives, but these are domain-specific and do not address general-purpose tokenizer pre-training:

VideoJAM (Chefer et al.): Concatenates optical flow to the latent space to improve motion generation in video models. This is specific to video and motion—it does not address the general image generation case.
ReDi (Kouzelis et al., 2025): Leverages pre-trained features for generation by incorporating them into the latent space. Again, a specific mechanism rather than a general pre-training strategy.
I-DeTok (Yang et al., 2025): Improves the latent space for both autoregressive and diffusion models via a joint pre-training strategy using MIM and noise injection, but does not explore broader semantic objectives like contrastive learning.

Critical limitation: narrow scope, no scaling analysis. These works improve generation under specific configurations but are not designed as general pre-training frameworks. Critically, "the scalability of the proposed methods generally remains unverified" (Section 2.2). None of these prior approaches have been shown to produce consistent improvements as compute, parameters, or data scale up—the core requirement for a truly effective pre-training paradigm.

Unified Understanding-and-Generation Tokenizers

A recent line of work attempts to build tokenizers that simultaneously support visual understanding tasks (classification, retrieval) and generation, rather than treating them as separate capabilities. Examples include VILA-U (Wu et al., 2024) and UniTok (Ma et al., 2025). These methods produce latents that can be used both for discriminative evaluation (zero-shot classification, linear probing) and for training generative models.

Critical limitation: performance ceiling. The paper's Table 2 shows that VTP significantly outperforms these unified tokenizers. VILA-U achieves 1.80 rFID and 73.3% zero-shot accuracy; UniTok achieves 0.41 rFID and 70.8% zero-shot. VTP reaches 0.36 rFID and 78.2% zero-shot accuracy simultaneously—better on both metrics. More importantly, prior unified tokenizers have not been shown to scale effectively with increased pre-training compute. They represent a fixed point in the design space but not a scalable recipe.

Self-Supervised and Contrastive Pre-Training in Vision (Unrelated to Generation)

The paper draws on well-established pre-training paradigms from representation learning—DINOv2 (Oquab et al., 2023) for self-supervised learning, CLIP (Radford et al., 2021) for image-text contrastive learning, MAE (He et al., 2022) for masked image modeling—but these methods were developed for discriminative visual understanding tasks, not for building tokenizers for generative models. DINOv2 produces powerful features for classification, segmentation, and retrieval; CLIP produces aligned image-text representations for zero-shot recognition. Neither was designed with the downstream goal of training a latent diffusion model.

The gap this paper fills. Prior to VTP, no work had systematically combined these representation-learning objectives with reconstruction in a tokenizer pre-training framework and studied how the resulting latent space scales for generative tasks. The paper explicitly notes this gap: "within the explicitly decoupled, two-stage framework of LDMs—comprising a visual tokenizer followed by a generative model—how to pretrain the first-stage tokenizer to enhance second-stage generative performance has not been systematically explored" (Section 2.1).

How This Paper Positions Itself

VTP is positioned not as an incremental improvement over any single prior method but as a paradigm shift in how visual tokenizers should be pre-trained. The paper makes this positioning explicit through several conceptual moves:

Reframing the tokenizer as a representation learner, not just a compressor. The central conceptual move is to view the visual tokenizer through the lens of representation learning (Section 2.1, Figure 1). Rather than asking "how can we compress images more efficiently?", VTP asks "how can we learn a latent space that represents visual information in a way that is useful for generation?" This reframing explains why reconstruction-only training fails (it optimizes for compression, not representation) and points toward the solution (integrate objectives that build semantic understanding into the latent space).

Introducing the "pre-training scaling problem" as a named challenge. By explicitly naming and characterizing the scaling failure of reconstruction-only training, the paper creates a clear benchmark for what counts as a solution. The goal is not just to outperform a baseline at a fixed compute budget, but to demonstrate that increasing the tokenizer's pre-training compute, parameters, and data reliably improves downstream generation—a scaling law.

Positioning VTP as the first scalable tokenizer pre-training framework. The core empirical claim is that VTP is "the first visual tokenizer to demonstrate scaling properties" (Section 1). Figure 7 shows that VTP's generation performance (gFID) improves monotonically as encoder size increases from Small to Large, as decoder layers increase, and as training data scales from 100K to 100M samples. In contrast, the autoencoder baseline (AE) is flat or slightly degrading across all three scaling axes. This is the evidence that VTP solves the pre-training scaling problem.

Using fixed downstream DiT training as a controlled evaluation protocol. A critical methodological choice that distinguishes this paper: all generation evaluations use an identical DiT training configuration—same architecture (LightningDiT-B), same number of epochs (80 for fast comparisons), same ImageNet dataset. The only variable is which pre-trained tokenizer's latents the DiT is trained on. This isolates the tokenizer's contribution cleanly. The paper emphasizes this repeatedly: "without modifying DiT training specs and FLOPs, solely scaling VTP pre-training consistently achieves gains" (Section 1). This rigor is essential because it prevents confounding from downstream training differences.

Offering a unified framework rather than a single trick. VTP is explicitly designed as a framework into which new representation-learning objectives can be plugged. The paper demonstrates this by showing that both CLIP and SSL objectives independently improve generation when combined with reconstruction, that combining them (CLIP+SSL+AE) yields the best results, and that the framework is "general" enough that "new and emerging representation learning techniques can be seamlessly integrated to establish even better performance bounds" (Section 4.3, Observation 4). This is a platform play, not a one-off method.

Reconciling understanding and generation under one tokenizer. Rather than treating understanding and generation as competing objectives (the reconstruction-generation trade-off), VTP demonstrates that semantic understanding drives generation quality. Figure 2 shows a strong positive correlation between comprehension (linear probing accuracy) and generation (gFID) during tokenizer pre-training. Table 2 shows VTP achieving state-of-the-art on both fronts simultaneously—1.11 gFID for generation alongside 78.2% zero-shot and 85.7% linear probing for understanding. This unification is important practically because it means a single pre-trained tokenizer can serve both generative and discriminative downstream tasks.

The Intellectual Lineage and the Missing Piece

To understand what VTP contributes, it helps to see where it fits in the intellectual progression:

VAEs for LDMs (Rombach et al., 2022): Established the two-stage paradigm but used reconstruction-only training with KL regularization. No semantic objectives.
Reconstruction-generation trade-off documented (Yao et al., 2025; Esser et al., 2024): Showed that better reconstruction does not guarantee better generation. Identified the problem but did not propose a scalable solution.
Distillation from pre-trained models (RAE, VA-VAE, REPA-E): Attempted to inject semantic quality by matching frozen teacher representations. Hit performance ceilings and showed negative scaling for some methods at larger sizes.
VTP (this paper): Integrates representation-learning objectives directly into tokenizer pre-training—not by distilling from a frozen teacher but by jointly optimizing the tokenizer on the representation tasks themselves. This eliminates the teacher bottleneck. The tokenizer's latent space is the representation being learned, so it can scale with model size and data without being limited by a fixed external model.

The crucial distinction: distillation methods (RAE, VA-VAE) ask "can we make the tokenizer's latents match DINOv2 features?"—tying performance to DINOv2's ceiling. VTP asks "can we train the tokenizer to solve DINOv2-style tasks (self-distillation, MIM) and CLIP-style tasks (image-text contrastive) simultaneously with reconstruction?"—so the tokenizer learns its own semantic representations that can scale with its own capacity and data. This is the conceptual breakthrough that unblocks the scaling behavior.

3. Technical Approach

3.1 Reader Orientation

VTP is a single Vision Transformer (ViT) autoencoder that is jointly trained on three objectives—reconstruction, self-supervised learning, and image-text contrastive learning—to produce a latent space that simultaneously captures pixel-level detail and high-level visual semantics. The problem it solves is the pre-training scaling failure of conventional visual tokenizers: reconstruction-only training produces latent spaces that get worse for generation as they get better at reconstruction, creating a ceiling that VTP breaks by making semantic understanding a first-class training target rather than an afterthought.

3.2 Big-Picture Architecture (Diagram in Words)

The VTP system has five major components:

ViT Encoder — a Vision Transformer that ingests an image $I \in \mathbb{R}^{3 \times H \times W}$ and compresses it into a $d$ -dimensional latent space at $H/16 \times W/16$ spatial resolution. This is the bottleneck representation that the downstream diffusion model will learn to generate. The encoder's intermediate features are accessed by multiple loss heads for different objectives.
EMA Teacher — an exponential moving average copy of the encoder that provides stable pseudo-labels for self-distillation and masked image modeling, following the DINOv2 recipe. It is updated via momentum, not gradient descent, ensuring consistent targets.
Text Encoder — a 12-layer transformer with hidden dimension 768 that encodes image captions $T$ into text features for the contrastive loss. It is trained jointly with the visual tokenizer to align the latent space with language.
Pixel Decoder — a decoder composed of N ViT blocks (typically 4 for fast experiments, up to 24 for final scaling) that lifts the compressed latent codes back to a feature space suitable for pixel reconstruction. A final pixel-shuffle layer converts features back to pixel space $I' \in \mathbb{R}^{3 \times H \times W}$ . The decoder is trained jointly with the encoder during pre-training, then optionally fine-tuned with a GAN objective in a second stage while the encoder remains frozen.
Loss Heads — separate prediction heads or computation paths for the three loss families: a reconstruction head (L1 + perceptual loss on decoded pixels), self-supervised heads (masked patch prediction and class token self-distillation), and a contrastive head (cosine similarity maximization between aligned image-text pairs).

Information flow during pre-training: An image enters the system → the ViT encoder processes it into latent codes and produces intermediate features (class tokens, patch tokens) → the EMA teacher processes augmented views in parallel for SSL targets → the text encoder processes the caption for contrastive targets → the pixel decoder reconstructs the image from the latent codes → four loss computations run simultaneously: L_rec compares reconstruction to input, L_mim and L_dino compare student predictions to teacher targets, L_clip maximizes similarity between visual and textual class tokens → gradients from all losses backpropagate through the encoder, decoder, and text encoder. The encoder is the shared backbone that must learn representations serving all objectives simultaneously.

3.3 Roadmap for the Deep Dive

First, the ViT autoencoder architecture and why ViT is chosen over CNNs, including the specific encoder-decoder asymmetry and the latent bottleneck configuration that enables efficient downstream generation.
Second, the reconstruction loss and the two-stage training strategy (L1+perceptual first, GAN fine-tuning second), since reconstruction is the foundation that all other objectives build upon and the source of the scaling problem VTP solves.
Third, the self-supervised learning component (MIM + self-distillation from DINOv2), because this is the mechanism that injects spatial-semantic perception into the latent space without requiring any labels.
Fourth, the contrastive learning component (CLIP-style image-text alignment), because this adds cross-modal semantic grounding that proves especially important for text-to-image generation.
Fifth, the overall multi-task objective and the batch sampling strategy, because these are the engineering decisions that make joint training feasible despite divergent batch size requirements (16K for CLIP vs. 2K for reconstruction).
Sixth, the downstream evaluation protocol (fixed DiT training, 80-epoch benchmark, linear probing on bottleneck features), because this is the controlled experimental framework that isolates the tokenizer's contribution and enables the scaling analysis.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical scaling paper whose core technical idea is that jointly training a visual tokenizer on representation-learning objectives alongside reconstruction transforms the pre-training scaling behavior from flat or negative (reconstruction-only) to positive and monotonic (VTP). The contribution is not a novel architecture or a novel loss function per se—all individual components are drawn from prior work (ViT from Dosovitskiy, DINOv2 from Oquab et al., CLIP from Radford et al., L1+perceptual+GAN from Esser et al.)—but rather the integration of these components into a unified tokenizer pre-training framework and the empirical demonstration that this integration unlocks scaling properties that no prior approach exhibits.

Vision Transformer Autoencoder Architecture

The visual tokenizer uses a fully Vision Transformer architecture for both the encoder and decoder, departing from the convolutional VAEs that dominate LDM implementations (Rombach et al., 2022; Esser et al., 2021). The paper argues for this choice on both practical and conceptual grounds: "Leveraging its flexibility in learning visual representations" (Section 3.1), the ViT architecture naturally accommodates the multiple representation-learning objectives that VTP integrates, whereas CNNs are more specialized for spatial processing.

The encoder is a standard ViT that tokenizes an input image $I \in \mathbb{R}^{3 \times H \times W}$ by dividing it into non-overlapping patches of size $16 \times 16$ , projecting each patch to a $d$ -dimensional embedding, prepending a learnable [CLS] token, adding positional embeddings, and processing the sequence through $L$ transformer blocks. The output is a set of patch tokens and a [CLS] token that serve as the basis for all downstream objectives. The paper employs QKNorm (Henry et al., 2020)—applying LayerNorm to queries and keys before the attention computation—to "enhance training stability" (Section 4.1), which is particularly important when training with multiple loss terms that can have conflicting gradient directions.

The bottleneck design is a critical architectural choice. Following the convention in (Ma et al., 2025), the latent space has $f = 16$ (a 16× spatial compression—each latent token corresponds to a $16 \times 16$ pixel patch) and $d = 64$ or $d = 256$ (the feature dimension per latent token). This is notably different from the typical LDM VAE which uses $f = 8$ and $d = 4$ . The paper's choice of $f = 16$ and $d = 64$ produces a latent space of shape $\mathbb{R}^{64 \times 16 \times 16}$ for a 256×256 image, which is more compact spatially but richer per-token than the LDM default ( $\mathbb{R}^{4 \times 32 \times 32}$ ). The paper argues this configuration helps "balance semantic comprehension with reconstruction quality" (Section 4.1), with the higher per-token dimension enabling richer semantic encoding while the larger patch size provides the compression rate needed for efficient diffusion model training.

The paper runs an ablation on this choice by testing $d = 256$ alongside $d = 64$ . Results in Figure 5 compare both settings across CLIP+AE and SSL+AE variants, showing that the scaling properties (correlated improvement in generation and understanding) hold for both bottleneck dimensions. This is important: it demonstrates that the scaling behavior is a property of the multi-objective training paradigm, not an artifact of a specific bottleneck configuration.

The decoder is architecturally asymmetric with respect to the encoder—a deliberate design choice for efficiency. The decoder consists of $N$ ViT blocks (the paper experiments with $N = 4$ for rapid experimentation and $N = 24$ for full-scale models) followed by a pixel-shuffle layer that converts the transformer features back to pixel-space RGB values. The decoder's ViT blocks operate on the concatenation of: (1) the latent tokens from the bottleneck, after they are lifted back to the encoder's feature dimension, and (2) learned positional embeddings that localize each token spatially. The pixel-shuffle layer rearranges feature channels into spatial resolution, a standard technique for upsampling in super-resolution and generation architectures.

The asymmetry between encoder and decoder serves an important purpose: the encoder must be powerful enough to learn rich representations (the paper scales it up to ViT-Large with ~300M parameters), while the decoder can be lighter because its job—pixel reconstruction from semantically structured latents—is easier than the encoder's job—compressing high-dimensional images into a structured latent space while simultaneously learning visual semantics. The paper validates that decoder scaling matters too: Figure 7(c) shows that increasing the decoder from 4 to 24 ViT-L layers improves gFID from 26.12 to 24.08 for VTP, while the autoencoder baseline sees essentially no improvement (57.87 → 58.32, statistically flat). This asymmetry is a practical optimization: invest parameters in the encoder where representation learning happens, and tune the decoder for reconstruction quality.

Why ViT over CNN? Table 1 provides the architectural comparison that motivates the ViT choice. A ViT-L autoencoder achieves 31.28 rPSNR (reconstruction quality) and 53.51 gFID (downstream generation quality) on ImageNet 256×256, compared to 30.63 rPSNR and 59.53 gFID for the convolutional LDM architecture. The ViT uses more parameters (607.2M vs. 70.3M) but fewer FLOPs (311.1G vs. 389.4G), since the LDM CNN uses a high-resolution intermediate feature map while the ViT operates at constant resolution. The key finding is not that ViT is categorically better (the reconstruction PSNR is similar), but that it is "a comparable alternative to CNNs, enabling simpler pre-training scaling experiments" (Table 1 caption). The ViT architecture's uniformity—same building block at every layer, no downsampling/upsampling layers, no architectural changes when scaling depth or width—makes it much easier to systematically vary model size and study scaling behavior, which is exactly what this paper aims to do.

Reconstruction Loss and Two-Stage Training Strategy

The reconstruction objective is the foundation of the VTP framework—it ensures the latent space preserves sufficient pixel-level detail that images can be faithfully decoded back to pixel space. This is non-negotiable for a generative tokenizer: if the decoder cannot reconstruct images from latents, then no generative model trained in latent space will produce sharp outputs.

However, the paper faces a practical obstacle. The GAN loss (Esser et al., 2021) that is standard for improving the perceptual quality of VAE reconstructions is "poorly compatible with the ViT architecture, which causes large gradient norms and low training stability" (Section 3.2). This is a known issue: ViT training dynamics differ from CNN training dynamics, and the adversarial training that works smoothly with CNN decoders can destabilize ViT training due to differences in gradient propagation through self-attention layers.

Two-stage training strategy. To resolve this, the paper adopts a simple but effective two-stage approach:

Stage 1 (pre-training): All parameters—encoder, decoder, text encoder, EMA teacher—are jointly optimized using only the L1 loss and a perceptual loss (LPIPS, from Zhang et al., 2018) between the input image $I$ and the reconstructed image $I'$ . No adversarial loss is used. The reconstruction loss is:

$L_{rec} = L_1 + L_{perceptual}$

where $L_1 = \lVert I - I' \rVert_1$ is the mean absolute pixel error and $L_{perceptual}$ is the LPIPS distance computed by passing both $I$ and $I'$ through a pre-trained deep network (typically VGG or AlexNet) and comparing their internal feature activations at multiple layers.

What it computes: $L_1$ penalizes per-pixel deviations directly—if the reconstructed image has a different RGB value at any location, it incurs a cost proportional to the absolute difference. This encourages global luminance and color accuracy. $L_{perceptual}$ compares the images in a learned feature space that is sensitive to structural and textural similarity—two images can have identical pixel values at most locations but different perceptual quality (e.g., one is blurry), and LPIPS captures this difference better than L1 alone. The sum of these two terms produces a reconstruction loss that balances pixel-level fidelity with perceptual quality.

Why this form: L1 loss alone (without perceptual loss) produces blurry reconstructions because the model hedges against uncertainty by averaging possible pixel values—the minimum expected L1 error for an uncertain pixel is the median, not a sharp high-frequency prediction. Perceptual loss pushes the model to produce sharper outputs by matching the feature statistics that pre-trained networks use to recognize objects and textures. The combination is standard in image restoration and generation. The absence of GAN loss in Stage 1 is the key design decision: it makes training stable with ViT architectures, enabling the joint optimization with SSL and contrastive objectives that would be impossible with adversarial training instability.

Stage 2 (decoder fine-tuning): After pre-training is complete, the encoder is frozen and only the pixel decoder is fine-tuned with a GAN objective—a patch-based discriminator and an adversarial loss $L_{GAN}$ in addition to $L_1$ and $L_{perceptual}$ . This stage "improves fidelity" (Section 3.2) without affecting the latent space structure that was learned during pre-training, since the encoder is frozen. The GAN loss encourages the decoder to produce images with realistic high-frequency details that fool the discriminator, sharpening textures and edges beyond what perceptual loss alone can achieve.

The two-stage design separates concerns: Stage 1 learns a semantically structured latent space with stable training; Stage 2 polishes reconstruction quality without destabilizing the representation learning. The paper does not provide ablation data on the two-stage approach versus end-to-end GAN training, but reports that "ViT-B achieves a reconstruction PSNR of 30.72 and gFID of 58.40" (Table 1) with this approach, competitive with CNNs.

Self-Supervised Learning Component (MIM + Self-Distillation)

The self-supervised learning component is the mechanism that injects spatial-semantic perception into the latent space without requiring any labels. VTP adopts the DINOv2 framework (Oquab et al., 2023), which itself combines two complementary SSL techniques: masked image modeling (MIM, from He et al., 2022) and self-distillation (from Caron et al., 2021, the original DINO).

Why both MIM and self-distillation? The paper follows the DINOv2 finding that these two objectives provide complementary signals. MIM trains the model to predict masked patches from visible context, which builds understanding of local spatial relationships and low-level visual structure—the model learns that a patch containing a dog's ear should be predictable from surrounding patches showing fur and head shape. Self-distillation trains the model to produce consistent global representations across different augmented views of the same image—the model learns that a close-up crop of a dog and a wide shot of the same dog should map to similar [CLS] token representations. Together, they produce features that capture both local texture (MIM) and global semantic identity (self-distillation), exactly the combination that VTP aims to encode in the latent space.

Data augmentation for SSL. For a given image $I$ , the paper applies data augmentation to produce two types of views:

Global crops $I_{global}$ : large crops covering a significant portion of the image (typically >50% area), capturing overall scene structure.
Local crops $I_{local}$ : smaller crops covering a smaller portion (typically <50% area), capturing fine-grained details.

These views serve different roles in the two SSL sub-components. The global crops carry the semantic identity information that self-distillation aligns; the local crops force the model to recognize objects from partial views, building invariance to occlusion and framing.

Masked Image Modeling (MIM). The MIM objective follows the iBOT protocol (Zhou et al., 2021), which extends masked autoencoding to work with the teacher-student architecture of DINO. The procedure is:

$I_{global}$ is patch-embedded and fed directly to the EMA teacher—no masking applied to the teacher input. The teacher processes the unmasked image and produces patch-level features for every position, serving as the ground-truth target for the masked positions.
A masked version of $I_{global}$ is created by randomly replacing a fraction of input patches (the paper does not specify the exact masking ratio, but DINOv2 and iBOT typically use 50-75%) with a learnable [MASK] token. This masked input is fed to the student (the main ViT encoder).
The student's output at the masked positions is compared to the teacher's output at the same positions using a prediction head—a small MLP that maps the student's features to match the teacher's feature dimension.
The MIM loss $L_{mim}$ is computed only on the masked patches, typically as the L2 distance or cosine distance between student predictions and teacher targets at those positions.

What $L_{mim}$ computes: For each masked patch position, the student must predict what the teacher "sees" at that position based on the surrounding visible context. The loss penalizes deviations between the student's inferred representation and the teacher's directly-computed representation. This forces the encoder to learn the spatial structure of images—how patches relate to each other—because it cannot simply memorize position-specific features; it must infer missing content from context.

Why this works for generation: MIM teaches the encoder that similar visual patterns (textures, object parts, edges) in different spatial contexts should produce similar latents, because the model must predict them from variable surrounding context. This builds a latent space that is locally consistent and spatially coherent—exactly the property that makes it easier for a diffusion model to learn to generate novel arrangements of these patterns.

Self-Distillation (DINO). The self-distillation objective follows the original DINO protocol (Caron et al., 2021) and its DINOv2 refinement:

Both $I_{global}$ and $I_{local}$ are passed to the student (the main ViT encoder). $I_{global}$ only is passed to the EMA teacher.
The [CLS] tokens from the student's processing of both views, and from the teacher's processing of the global view, are projected through separate prediction heads to produce probability distributions over a set of learned prototypes (in DINOv2, these are typically 65,536-dimensional vectors).
The cross-entropy loss $L_{dino}$ encourages the student's predictions (from both global and local views) to match the teacher's prediction (from the global view only). The teacher's prediction is sharpened with a temperature parameter to produce a "pseudo-label" that serves as the target.
The teacher is never trained with gradient descent. Instead, its parameters $\theta_{teacher}$ are updated as an exponential moving average of the student's parameters $\theta_{student}$ :

$\theta_{teacher} \leftarrow \lambda \theta_{teacher} + (1 - \lambda) \theta_{student}$

where $\lambda$ is typically 0.996 or higher during training, following a cosine schedule from 0.996 to 1.0.

What $L_{dino}$ computes: The student must produce consistent [CLS] token representations across different crops of the same image—a global crop and a local crop should map to the same prototype distribution. This is a form of instance-level contrastive learning without negative examples: the positive pair is (global view, local view) of the same image, and the loss encourages their representations to be similar. The EMA teacher provides a stable, slowly-evolving target that prevents the representation from collapsing to a trivial solution (all images map to the same prototype).

Why this works for generation: Self-distillation pushes the latent space to organize around object-level identity rather than pixel-level appearance. A crop of a dog's face and a full-body shot of the same dog should produce similar [CLS] representations despite having almost no pixels in common. This semantic grouping is precisely what reconstruction-only training lacks: the reconstruction objective doesn't care whether two views of the same object map to similar latents, only whether each can be decoded independently. By adding self-distillation, VTP ensures that the latent space respects semantic similarity—a property that the diffusion model can exploit to generate coherent objects rather than pixel-pattern collages.

Combined SSL loss. The overall self-supervised learning loss is the sum:

$L_{ssl} = L_{mim} + L_{dino}$

where both components are computed on the same batch of images with the same teacher-student architecture, sharing the encoder backbone. The paper follows "the established practices of DINOv2 and OpenCLIP" (Section 4.1) for the detailed implementation, including the specific masking ratios, crop scales, teacher momentum schedule, prototype count, and prediction head architectures.

What $L_{ssl}$ contributes to the VTP framework: It enforces two properties that reconstruction alone cannot guarantee. First, local spatial coherence: patches that are visually similar or contextually related should produce similar latent features (from MIM). Second, global semantic invariance: different views of the same object or scene should map to nearby points in latent space (from DINO). Together, these properties structure the latent space in a way that is "generation-friendly"—the diffusion model's denoising process can smoothly interpolate between semantically meaningful regions of the latent space rather than navigating a space organized by pixel statistics.

Contrastive Learning Component (Image-Text Alignment)

The contrastive learning component adds cross-modal semantic grounding to the latent space. Unlike SSL, which operates purely within the visual domain, CLIP-style contrastive learning aligns the visual latent space with a text encoder, creating a joint embedding space where images and their captions map to similar representations. This is particularly important for text-to-image generation, where the diffusion model must condition on text prompts and generate images that match the textual description.

The CLIP objective. Given a batch of $B$ image-text pairs $(I_i, T_i)_{i=1}^B$ , the procedure is:

Each image $I_i$ is encoded by the ViT encoder to produce a visual representation. The paper uses the [CLS] token from the encoder's output, projected through a linear layer to match the text embedding dimension, yielding $v_i \in \mathbb{R}^{d_{clip}}$ .
Each caption $T_i$ is tokenized and encoded by the text encoder (a 12-layer transformer with hidden dimension 768) to produce a text representation $t_i \in \mathbb{R}^{d_{clip}}$ , typically by taking the final hidden state of an [EOS] token or the embedding of the full sequence pooled via attention.
The cosine similarity between every image-text pair in the batch is computed: $s_{ij} = \frac{v_i^\top t_j}{\lVert v_i \rVert \lVert t_j \rVert}$ , forming a $B \times B$ similarity matrix.
The contrastive loss $L_{clip}$ maximizes the similarity of the $B$ diagonal entries (correct pairings) while minimizing the similarity of the $B^2 - B$ off-diagonal entries (incorrect pairings). This is typically implemented as a symmetric cross-entropy loss over the similarity matrix, where the rows and columns are treated as classification problems: row $i$ must classify which column $j$ contains the matching text, and column $j$ must classify which row $i$ contains the matching image.

What $L_{clip}$ computes: For each image, the model must pick its correct caption out of the entire batch based on cosine similarity. For each caption, the model must pick its correct image out of the entire batch. The loss is high when similar images and texts are far apart in the joint embedding space, or when dissimilar pairs are close. It is low when the correct pairings have substantially higher similarity than all incorrect pairings.

Why this form: The batch-wise contrastive formulation is crucial for scalability—it provides $B$ positive pairs and $B^2 - B$ negative pairs per batch, making the loss computationally efficient and statistically powerful. Alternative formulations like triplet loss or pairwise ranking would require explicit negative mining and scale poorly to large datasets. The symmetric formulation (image→text and text→image) ensures the embedding space is well-structured in both directions, which matters for text-to-image generation where the text serves as the query and the image is the target.

What $L_{clip}$ contributes to the VTP framework: It injects linguistic semantics into the visual latent space. After training, the [CLS] token representation of an image encodes not just what objects are present but what language would describe them—"a golden retriever playing in a park" and "a dog outdoors" map to different regions of the space even if they are visually similar, because the captions differ. This linguistic grounding is directly useful for text-to-image generation: the diffusion model conditions on text embeddings and must produce latents that decode into images matching the text. If the visual latent space is already aligned with the text embedding space (because they were trained jointly with contrastive loss), the diffusion model's conditioning task becomes easier—it is learning to map from the text embedding to the aligned visual latent, rather than learning a cross-modal mapping from scratch.

The paper provides direct evidence for this in Section 5: "the tokenizer trained with the CLIP loss exhibits a significant advantage in text rendering for text-to-image generation, substantially outperforming both AE and SSL+AE tokenizers" (Figure 8b caption). The CLIP objective specifically helps with generating readable text in images—a task that requires tight alignment between linguistic concepts (the words to render) and visual output (the pixel arrangement of those words).

The Overall Multi-Task Objective and Batch Sampling Strategy

The VTP training framework combines the three loss families into a single weighted objective:

$L_{total} = \lambda_{rec} L_{rec} + \lambda_{ssl} L_{ssl} + \lambda_{clip} L_{clip}$

where $\lambda_{rec} = 0.1$ , and $\lambda_{ssl}, \lambda_{clip} \in \{0, 1\}$ depending on which objectives are active for a given experiment.

What this computes: The total loss is a scalar sum of the three component losses, each weighted by its coefficient. The model's parameters (encoder, decoder, text encoder, prediction heads) are updated via gradient descent to minimize this sum across batches of training data. The loss is computed jointly on each batch—all three objectives contribute gradients simultaneously to the shared encoder backbone.

Why these weights: The paper provides limited theoretical justification for the specific weight choices, but the empirical pattern is clear: "a smaller reconstruction weight contributes to improved generative performance" (Section 4.1). Setting $\lambda_{rec} = 0.1$ while $\lambda_{ssl} = \lambda_{clip} = 1.0$ means the representation-learning objectives dominate the total loss magnitude by roughly an order of magnitude. This is a deliberate choice: the paper's central hypothesis is that semantic understanding drives generation, so the training signal should prioritize representation learning over pixel-level reconstruction. If $\lambda_{rec}$ were too large, the optimization would gravitate toward the easier-to-optimize low-level features that improve reconstruction at the expense of semantic structure, recreating the pre-training scaling problem.

The paper does not perform an extensive ablation over $\lambda$ values, which is a limitation—it's possible that other weight ratios would produce even better scaling behavior. However, the fixed choice of $0.1:1:1$ is sufficient to demonstrate the qualitative shift from flat/negative scaling (reconstruction-only) to positive scaling (VTP).

The batch sampling strategy is a critical but often overlooked engineering detail that makes multi-objective joint training feasible. The problem: CLIP contrastive learning requires extremely large batches to provide sufficient negative examples—"contrastive learning frameworks like CLIP demand extremely large batches (e.g., 16k or 32k)" (Section 3.6)—while self-supervised learning and reconstruction objectives are "typically effective with orders of much smaller batches (e.g., 4k)" (Section 3.6). This divergence arises because:

CLIP's contrastive loss operates on the $B \times B$ similarity matrix, where $B$ is the batch size. The quality of the learned embedding space depends heavily on having enough negative examples per positive pair. With a batch of 16K, each image sees 15,999 negative texts; with a batch of 2K, it sees only 1,999. Larger batches produce more discriminative features.
Reconstruction and SSL losses operate per-sample or per-patch and do not benefit from large batches in the same way. Reconstruction loss is computed pixel-by-pixel for each image independently; MIM loss is computed per-masked-patch; DINO loss uses only the samples within the batch for the teacher's prototype assignments but the prototype set itself provides a form of negative partitioning that reduces batch size dependence.

Simply using a batch size of 16K for all objectives would waste computation on objectives that don't need it, while potentially destabilizing training due to large-batch optimization challenges. Using a batch size of 2K for all objectives would starve the CLIP loss of negative examples.

The solution: batch sampling with different sizes per objective. The paper implements the following scheme:

A base batch of $B$ image-text pairs is loaded, where $B = 16,000$ (the CLIP batch size, $B_{clip}$ ).
All $B$ pairs are used for the CLIP contrastive loss—the full 16K batch provides the negative examples that CLIP needs.
For self-supervised learning, a random subset of size $B_{ssl} = 4,000$ is sampled from the $B$ pairs. These 4K images undergo the SSL pipeline (multi-crop augmentation, masking, teacher processing, etc.).
For reconstruction, a random subset of size $B_{rec} = 2,000$ is sampled from the $B$ pairs. These 2K images go through the encode-decode pipeline and generate reconstruction losses.

What this achieves: The shared encoder processes different subsets of images for different losses within the same training step. Every image in the base batch of 16K contributes to at least the CLIP loss; only the sampled subsets contribute to SSL and reconstruction gradients. This means the encoder receives gradient signals from all three objectives but with different effective batch sizes per objective, matching each objective's statistical requirements without wasting computation.

The sampling is described as "random sampled from $B$ " (Section 3.6), meaning the subsets are independent for each training iteration—a given image may contribute to reconstruction in one step and not the next. This stochasticity is not problematic because each objective sees enough total samples over the course of training to converge; the batch-level sampling just controls per-step compute and gradient variance.

Why this matters for scaling: The batch sampling strategy is what enables VTP to scale to large pre-training datasets (277M images) with practical hardware requirements. Without it, the entire batch of 16K would need to go through the full SSL pipeline (multi-crop augmentation expands each image into multiple views, increasing memory), and the full reconstruction pipeline (decoder forward pass). By decoupling the per-objective batch sizes, VTP can use compute-efficient implementations of CLIP at large batch sizes while keeping the per-step memory footprint of SSL and reconstruction manageable. This is not a theoretical contribution but a necessary engineering decision that makes the scaling experiments feasible.

Downstream Evaluation Protocol: Controlled DiT Training

The paper's evaluation methodology is as important to the technical contribution as the pre-training framework itself, because the central claim—VTP unlocks scaling properties—requires a controlled experimental setup that isolates the tokenizer's contribution. The protocol is:

Fixed downstream model. All generation evaluations use the LightningDiT-B architecture (Yao et al., 2025), a diffusion transformer that is a strong but standard baseline. Critically, the DiT training configuration is held constant across all tokenizer comparisons: same architecture, same optimizer, same learning rate schedule, same number of training epochs (80 for the main comparisons), same ImageNet dataset at 256×256 resolution. The paper explicitly states that evaluations are conducted "under a consistent protocol" and "without modifying DiT training specs and FLOPs" (Section 1, Section 4.1). The only variable is the tokenizer used to encode images into latents.

What this controls for: In the standard LDM pipeline, the tokenizer and the generative model are trained in separate stages with relatively little interaction during training (the tokenizer is frozen during DiT training). This means the DiT's training dynamics—convergence speed, final performance, sensitivity to hyperparameters—are directly influenced by the properties of the latent space it is trained on. By keeping the DiT architecture and training recipe fixed, any differences in generation quality (gFID) can be attributed to differences in the latent space produced by the tokenizer. This is a stronger claim than "our tokenizer works well with a carefully tuned DiT"—it says "for any given DiT configuration, VTP latents produce better results than autoencoder latents."

Epoch budget. The primary comparisons use 80 DiT training epochs, which is intentionally short relative to the typical 400-1400 epochs used to train state-of-the-art ImageNet diffusion models. This choice serves two purposes. First, it makes the scaling experiments computationally feasible—running hundreds of DiT training runs to evaluate different tokenizers at different pre-training budgets would be prohibitively expensive at full training length. Second, it provides a more sensitive measure of tokenizer quality: at 80 epochs, the DiT is far from convergence, so differences in how quickly it learns from different latent spaces are amplified. A tokenizer that enables faster convergence will show large improvements at 80 epochs that might shrink (relatively) at 1400 epochs where all models approach their asymptotic performance.

The paper validates this choice by also reporting results at longer training durations: Table 2 shows that VTP-L achieves 1.85 gFID at 600 epochs (unguided) and 1.11 gFID at 600 epochs with guidance, demonstrating that the 80-epoch benefits translate to final performance. The fast convergence is itself a contribution: "VTP attains 2.60 and 2.03 gFID without guidance in only 80 epochs, demonstrating remarkably fast convergence" (Section 6).

Metrics: rFID and gFID. The paper uses two related but distinct metrics:

rFID (reconstruction FID): The Fréchet Inception Distance between original images and their reconstructions from the tokenizer. Lower rFID means the encoder-decoder pipeline preserves image distribution statistics more faithfully. This measures reconstruction quality in a distributional sense, not just per-pixel accuracy.
gFID (generation FID): The FID between real ImageNet validation images and images generated by the downstream DiT. Lower gFID means the generative model produces more realistic and diverse images. This is the primary metric of interest.

The paper also reports rPSNR (peak signal-to-noise ratio of reconstructions) for architectural comparisons in Table 1, but shifts to rFID for the main scaling experiments since rFID is more correlated with perceptual quality.

Linear probing for understanding. A key innovation in the evaluation is measuring the semantic quality of the latent space through linear probing: training a linear classifier on top of the frozen bottleneck features to predict ImageNet classes. Crucially, the paper states: "We do not employ the feature enhancements common in the DINO series, which can substantially increase linear probing scores by leveraging multi-layer features. Instead, we probe only the reduced-dimensionality features from the bottleneck, thereby directly evaluating the inherent properties of the latent features" (Section 4.1).

This is methodologically important. Standard DINOv2 evaluation concatenates features from multiple layers, which boosts linear probing accuracy but doesn't reflect the quality of any single layer's representation. VTP probes only the actual latent codes that the diffusion model will use—the $d = 64$ or $d = 256$ compressed representation at the bottleneck. This means the linear probing accuracy is a direct measure of how semantically structured the generation latents are, not how much semantic information the encoder captures in total. If the bottleneck latents achieve 85.7% linear probing accuracy, it means the compressed representation that the DiT denoises contains enough semantic signal to classify ImageNet classes—a strong indicator that the latent space is organized semantically.

Downstream DiT architecture: LightningDiT. The paper uses LightningDiT (Yao et al., 2025) as the generative model, which is itself a work from overlapping authors that optimizes the standard DiT architecture for faster convergence. LightningDiT modifies the DiT design with improved position encoding, normalization, and conditioning. The choice of LightningDiT over vanilla DiT is pragmatic: it enables faster experimentation (80-epoch comparisons are feasible because LightningDiT converges faster) and represents a strong baseline. The paper acknowledges this by stating they "follow LightningDiT as a strong baseline" (Section 4.1).

Tokenization for DiT training. During DiT training, images are encoded once by the frozen tokenizer to produce latent codes, which are then stored or generated on-the-fly as training targets for the diffusion model. The DiT learns to denoise random noise into these latent codes. At inference time, the DiT generates latent codes from noise (and optionally class labels), which are then decoded by the frozen pixel decoder into images. The tokenizer is never updated during DiT training—this is the standard two-stage LDM paradigm.

The fixed evaluation protocol is what makes the scaling analysis in Figures 5, 6, and 7 possible and credible. Each point on those curves represents a DiT trained for exactly 80 epochs with exactly the same hyperparameters, differing only in which checkpoint of which pre-trained tokenizer produced the latents. The monotonic improvement of VTP's gFID with pre-training compute is therefore a property of the latent space, not of downstream hyperparameter tuning.

Why This Integration Unlocks Scaling: The Mechanistic Hypothesis

The paper does not provide a formal proof or causal mediation analysis of why VTP scales where reconstruction-only training does not, but it offers a consistent mechanistic hypothesis across multiple sections:

Reconstruction-only training overfits the latent space to pixel statistics. As training compute increases, the tokenizer gets better at allocating its limited latent dimensions to encode pixel-level information—textures, edges, fine-grained patterns—because these are what the reconstruction loss rewards. The latent space becomes an efficient compression codec. However, this allocation crowds out semantic structure: two images of different dogs in different poses with different backgrounds may map to very different latent codes because their pixel statistics differ, even though they share the semantic concept "dog." The diffusion model must then learn to generate latents that decode into dogs while navigating a space where "dogness" is not geometrically concentrated. This is harder than navigating a space where dogs of all kinds cluster together.

VTP prevents this over-specialization by forcing the latent space to serve multiple masters. The SSL loss (especially DINO) pushes the [CLS] token representations of different views of the same object to be similar—this directly penalizes the "two dogs map to very different latents" scenario. The CLIP loss pushes the visual [CLS] token to align with language, creating a cross-modal semantic axis. The reconstruction loss maintains pixel fidelity. The encoder must find a latent space that satisfies all three simultaneously—and the paper's results show that this multi-objective pressure produces a space that is both semantically structured (scaling understanding) and generation-friendly (scaling generation).

The scaling behavior emerges because larger models and more data improve representation quality. This is the standard scaling story from representation learning (DINOv2, CLIP): bigger ViT encoders trained on more diverse data learn better visual features. VTP inherits this property. The difference is that VTP channels these improved representations into the latent space that serves generation, rather than keeping them as side outputs or discarding them. As the encoder scales up (ViT-S → ViT-B → ViT-L), its capacity to learn semantically structured representations increases, and the multi-objective training ensures this increased capacity is used to structure the latent space rather than solely to improve reconstruction. As training data scales (100K → 1M → 10M → 100M), the encoder sees more diverse visual concepts and learns more general representations, which again benefit the latent space structure.

The autoencoder baseline cannot benefit from scaling because it lacks the semantic pressure. A larger autoencoder trained only on reconstruction will simply become a better pixel compressor—its additional capacity is allocated to encoding ever-finer pixel details. The rFID improves (from 2.07 to 0.51 in Figure 4) because the larger model can preserve textures and edges more faithfully. But the latent space structure does not become more semantic; if anything, it becomes less structured for generation because the overfitting to reconstruction increases the variance of latent codes for semantically similar images (the model encodes pixel-level differences that are irrelevant for generation). Hence gFID stays flat or degrades.

This hypothesis explains all three scaling dimensions:

Compute scaling (Figure 5): More training iterations with multi-objective loss → better convergence of the joint representation → improved generation. Reconstruction-only → overfitting to pixel reconstruction → degraded generation.
Parameter scaling (Figure 7b): Larger encoder with multi-objective loss → greater capacity for semantic structure → improved generation. Larger autoencoder → greater capacity for pixel compression → flat generation.
Data scaling (Figure 7a): More diverse data with multi-objective loss → more general semantic representations → improved generation. More diverse data with reconstruction-only → diminishing returns on pixel compression → flat generation.

The paper does not provide direct evidence for the causal mechanism (e.g., measuring the "semantic concentration" of the latent space directly, or showing that interpolating between latents produces semantically smooth transitions), but the consistent pattern across all three scaling dimensions and the strong correlation between understanding (linear probing) and generation (gFID) in Figures 2, 5, and 6 support the claim that semantic understanding is the mediating variable.

4. Key Insights and Innovations

Innovation 1: Diagnosing and Naming the "Pre-Training Scaling Problem" as a First-Class Research Challenge

The most conceptually important move in this paper is not proposing a new loss function or architecture, but identifying and explicitly naming a failure mode that the field had observed implicitly but never characterized as a systematic scaling pathology. The paper calls this the "pre-training scaling problem": the observation that scaling up compute, parameters, and data for reconstruction-only visual tokenizer training does not improve downstream generation—it actively degrades it (gFID worsens from 55.04 to 58.56 as training FLOPs increase 10× in Figure 4), even as reconstruction quality improves substantially (rFID drops from 2.07 to 0.51).

This is significant as a diagnostic contribution, not a methodological one. Prior work had documented the reconstruction-generation trade-off (Yao et al., 2025; Esser et al., 2024) but treated it as a fixed tension to be managed through careful hyperparameter balancing or architectural choices—essentially, finding the sweet spot on a static Pareto frontier. The field's implicit assumption was that more tokenizer pre-training compute would eventually help generation if you could just avoid overfitting, and that the observed trade-off was a training-dynamics issue rather than a fundamental misalignment of objectives.

VTP reframes this entirely. By plotting gFID against training FLOPs for a reconstruction-only tokenizer and showing the curve is not flat but monotonically degrading (Figure 4, right panel), the paper demonstrates that pouring more compute into the standard recipe is not merely inefficient—it is actively harmful. This transforms the problem from "how do we optimize the reconstruction-generation trade-off?" (an optimization framing where you balance two competing objectives) to "how do we make tokenizer pre-training scale at all?" (a scaling framing where the very notion that more compute helps is called into question). The conceptual parallel is to the pre-Chinchilla era in language model training, where researchers scaled model size without scaling data appropriately, unknowingly operating on a suboptimal scaling curve. VTP's diagnostic move is to show that the field has been on the wrong scaling curve entirely for visual tokenizer pre-training.

Why this is fundamental rather than incremental: The paper doesn't just propose a better method—it identifies that the default approach exhibits a qualitative pathology (negative scaling) that no amount of hyperparameter tuning within that paradigm can fix. This is the difference between "our method gets 2 points better FID" and "the standard approach gets worse as you scale it; here's why, and here's a different paradigm that gets better." The naming of the problem creates a clear benchmark for what counts as a solution: any candidate tokenizer pre-training method must now demonstrate that its generation performance improves, not just saturates, as pre-training resources increase. This raises the bar for the field from showing fixed-budget improvements to demonstrating scaling properties.

Evidence for this claim is centralized in Figure 4 (reconstruction-only scaling) and Figure 7 (VTP scaling all three axes), which together establish the contrast between the old and new scaling behaviors. Figure 5 further demonstrates that the scaling property is shared across different representation-learning objectives (CLIP+AE and SSL+AE both show positive generation scaling, while AE-only shows negative scaling), supporting the diagnosis that reconstruction-only training is the root cause, not a peculiarity of any specific architecture or dataset.

Innovation 2: Recasting the Visual Tokenizer as a Representation Learner Rather Than a Compression Engine

The paper's second conceptual contribution is a fundamental reframing of what a visual tokenizer is for. In the standard LDM paradigm inherited from Rombach et al. (2022), the tokenizer is understood as a compression module: its job is to take high-dimensional pixel data and produce a lower-dimensional latent code from which the original can be approximately recovered. The VAE framework formalizes this as learning a probabilistic mapping where the latent distribution is regularized to be smooth (via the KL term) to facilitate diffusion model training, but the core performance metric is reconstruction fidelity. Under this view, a "better" tokenizer is one that compresses more efficiently—higher PSNR at a given compression ratio, or higher compression ratio at a given PSNR.

VTP rejects this framing and replaces it with a representation-learning framing. The central claim is that "an effective latent space for generation should efficiently encode the core visual semantics" (Section 1). This shifts the evaluation criterion from "how well can you recover the input pixels?" to "how useful is the latent representation for downstream tasks?"—specifically, for the diffusion model's task of generating novel images that are both realistic and semantically coherent. The compression function is still necessary (you need reconstruction to produce pixel outputs), but it is no longer sufficient, and optimizing it in isolation is actively counterproductive.

This reframing matters because it opens up an entirely different design space. If the tokenizer is a compression engine, the natural approach is to improve the encoder-decoder architecture and training signal: better perceptual losses, adversarial training, maybe learned quantization. These are all improvements within the compression paradigm. If the tokenizer is a representation learner, the natural approach is to incorporate objectives from representation learning: self-supervised pre-training, contrastive learning, multi-modal alignment. These are fundamentally different signals that do not aim to improve pixel-level fidelity but rather to structure the latent space around semantic concepts.

What makes this distinctive: The paper doesn't just propose adding a representation-learning loss on top of reconstruction—it argues that the tokenizer's pre-training should be reconceptualized around representation learning, with reconstruction serving as a necessary but subordinate constraint. The weighting scheme (λ_rec = 0.1 while λ_ssl = λ_clip = 1.0) encodes this priority directly in the optimization. The finding that "a smaller reconstruction weight contributes to improved generative performance" (Section 4.1) is evidence that the old paradigm's emphasis on reconstruction was not just suboptimal but actively harmful—down-weighting reconstruction loss improves generation, confirming that pixel-level fidelity and semantic structure are in tension, and that the semantic structure should dominate.

Contrast with prior work: Prior approaches that attempted to inject semantic quality into tokenizers (VA-VAE, REPA-E, RAE) operated within a distillation paradigm: take a separately trained representation model (DINOv2, CLIP), freeze it, and align the tokenizer's features to match it. This is a "bolt-on" approach—the tokenizer is still fundamentally a compression engine, but it's being regularized to not stray too far from a semantic teacher. VTP's reframing is more radical: the tokenizer is the representation learner, jointly trained on the same objectives that produced the teacher models in the first place. The teacher is not an external oracle but an EMA of the student itself. This eliminates the teacher bottleneck that Table 3 demonstrates for RAE (performance degrades as the tokenizer scales up, because the frozen teacher's capacity becomes the ceiling), and it means the latent space can scale with model capacity—the encoder learns its own increasingly rich representations rather than being limited by a pre-computed target.

Evidence for the power of this reframing is distributed across the paper's scaling experiments. Figure 5 shows that both CLIP+AE and SSL+AE exhibit correlated improvement in understanding (linear probing) and generation (gFID) as pre-training compute scales, while AE-only improves reconstruction but sees understanding and generation stagnate. Figure 2 visualizes this directly: understanding and generation quality are positively correlated during tokenizer pre-training. The paper's claim is not just that representation learning helps generation (that would be a performance finding), but that it redefines the scaling behavior from negative/neutral to positive—a qualitative change in the nature of tokenizer pre-training, not a quantitative improvement.

Innovation 3: Establishing Semantic Understanding as the Causal Driver of Generation Quality

The paper goes beyond demonstrating correlation between understanding and generation to make a stronger claim: semantic understanding is the key causal mechanism by which tokenizer pre-training improves generation. This is not stated as a proven causal relationship (there's no intervention study or mediation analysis), but it is argued as the most parsimonious explanation for a consistent pattern of evidence.

The evidence structure is: (1) Reconstruction-only training improves reconstruction but degrades generation (Figure 4)—so reconstruction fidelity is not the driver. (2) Adding either CLIP or SSL to reconstruction training makes both understanding and generation improve together (Figure 5)—so representation learning is the common factor. (3) Understanding (linear probing) and generation (gFID) are positively correlated across training FLOPs and across different training recipes (Figure 2, Figure 5a/d, Figure 6a). (4) The best combination (CLIP+SSL+AE) achieves both the highest understanding (74.9% linear probing) and the best generation (27.8 gFID) under a fixed budget (Figure 6). (5) On text-to-image tasks, CLIP loss—which specifically adds linguistic semantic alignment—provides disproportionate benefits for text rendering (Figure 8b), a task that requires semantic understanding of the relationship between words and visual appearance.

What makes this an intellectual contribution rather than just an empirical observation: The paper is not merely reporting that understanding metrics and generation metrics happen to correlate. It is making a mechanistic claim about why reconstruction-only training fails and why multi-objective training succeeds. The hypothesis, stated explicitly in Section 1 and reiterated throughout, is that reconstruction biases the latent space toward low-level pixel statistics and drives it away from semantic structure as training scales, while representation-learning objectives counteract this drift by forcing the latent space to encode semantic relationships. The scaling behavior then follows: as compute, parameters, and data increase, the encoder's capacity to learn semantic representations increases, and this capacity is channeled into the latent space by the representation-learning losses, producing monotonic improvement in generation. Reconstruction-only training lacks this semantic gradient, so additional capacity is channeled into encoding pixel-level detail that is irrelevant or actively harmful for generation.

This causal framing has predictive consequences that the paper validates. If understanding drives generation, then: (a) any representation-learning objective that improves understanding should improve generation—tested and confirmed with both CLIP and SSL independently (Figure 5). (b) Combining multiple representation-learning objectives should give additive or synergistic benefits—tested and confirmed with CLIP+SSL+AE outperforming either alone (Figure 6). (c) The benefit should generalize to new domains where semantic understanding matters—tested and confirmed on text-to-image generation, where CLIP's cross-modal alignment provides targeted improvements (Figure 8). (d) Scaling understanding (through larger models or more data) should scale generation—tested and confirmed in Figure 7. These predictions follow from the causal hypothesis but would not be expected if the relationship were merely correlational or if some other hidden factor drove both understanding and generation.

Contrast with the field's prior assumption: The field's implicit causal model before VTP was that reconstruction quality drives generation quality, because better reconstruction means the latent space preserves more information about the original image, which should give the generative model more signal to work with. This model predicts that scaling tokenizer pre-training should improve both reconstruction and generation. The paper's Figure 4 falsifies this prediction: reconstruction improves but generation degrades. VTP's alternative causal model—understanding drives generation, and reconstruction is necessary but subordinate—explains the observed pattern and makes the successful predictions listed above. This is a theoretical advance in the sense of providing a better causal model of how tokenizer pre-training affects downstream generation, even if the individual components (CLIP, DINOv2, VAE) are drawn from prior work.

The strongest direct evidence for the causal claim is the targeted ablation in Figure 8b: adding CLIP loss to the VTP pre-training improves text rendering in text-to-image generation. Text rendering is a task where semantic understanding (knowing that the string "dog" should look like the word "dog" rendered in pixels) is directly and specifically useful—it's hard to argue that CLIP helps text rendering through some other mechanism like improved global image statistics. The specificity of this benefit supports the claim that semantic understanding, not some generic regularization effect, is the active ingredient.

5. Experimental Analysis

Evaluation Methodology

Dataset. The primary benchmark is ImageNet (Deng et al., 2009) at 256×256 resolution for class-conditional generation. Tokenizer pre-training uses an internally filtered subset of DataComp-1B with 277M image-text pairs (the paper explicitly notes this is "an internally filtered version," Section 4.1). For text-to-image experiments, training uses the LAION dataset (Schuhmann et al., 2021) with images resized to a short edge randomly sampled between 256 and 512. All reconstruction evaluations use the standard ImageNet validation set at 256 resolution.
Base model(s). The visual tokenizer uses a ViT encoder at three scales for scaling experiments—ViT-Small, ViT-Base, and ViT-Large (Dosovitskiy, 2020)—with a configurable ViT decoder (typically 4 layers for fast experiments, up to 24 for final models). The downstream generative model is LightningDiT (Yao et al., 2025), a diffusion transformer evaluated at two sizes: LightningDiT-B (for most experiments, following the baseline in Yao et al., 2025) and LightningDiT-XL (for text-to-image). The text encoder for contrastive learning is a 12-layer transformer with hidden dimension 768. The key architectural specification for the tokenizer is f16d64—16× spatial compression (patch size 16) with 64-dimensional bottleneck features—which the paper argues "balances semantic comprehension with reconstruction quality" (Section 4.1). An ablation dimension of d = 256 is also tested.
Metrics. The paper uses three families of metrics. For generation quality: gFID (generation Fréchet Inception Distance, reported as FID-10k) measures the distributional distance between generated and real ImageNet validation images—lower is better. For reconstruction quality: rFID (reconstruction FID) measures the distributional distance between original and reconstructed images—lower is better—and rPSNR (reconstruction Peak Signal-to-Noise Ratio) measures per-pixel fidelity in decibels—higher is better. For semantic understanding: linear probing accuracy (ImageNet Top-1 accuracy, %) is computed by training a linear classifier on frozen bottleneck features, explicitly "probing only the reduced-dimensionality features from the bottleneck" (Section 4.1) without the multi-layer feature concatenation common in DINO evaluation. For text-to-image: GenEval, T2I-CompBench, DPG-Bench, and TIIF-Long scores measure compositional generation, attribute binding, and prompt following quality (Figure 8a).
Baselines. The paper compares against several categories of prior work. Reconstruction-only tokenizers: a standard convolutional LDM VAE (Rombach et al., 2022) with f16d64 specification, and ViT-based autoencoders trained solely with reconstruction loss at various scales (AE-S, AE-B, AE-L). Perception-oriented tokenizers: VA-VAE (Yao et al., 2025) which uses representation alignment; REPA-E (Leng et al., 2025) which optimizes the VAE feature space during DiT training; RAE (Zheng et al., 2025) which leverages DINOv2 features with a separate pixel decoder. Unified understanding-generation tokenizers: VILA-U (Wu et al., 2024) and UniTok (Ma et al., 2025). Generative model baselines: standard DiT (Peebles and Xie, 2023), SiT (Ma et al., 2024), REPA (Yu et al., 2024), and DDT (Wang et al., 2025), all trained on ImageNet 256×256. Representation-learning baselines: SigLIP (Zhai et al., 2023), MAE (He et al., 2022), and DINOv2 (Oquab et al., 2023) provide reference points for linear probing and zero-shot accuracy.
Generation budget / compute accounting. The paper measures tokenizer pre-training scale in terms of total training FLOPs (GFLOPs) on the pre-training dataset, with the downstream DiT training FLOPs held strictly constant across all comparisons. This is a critical design choice: the paper explicitly states that evaluations are conducted "under a consistent protocol" where the DiT "training configuration and FLOPs" are "fixed" (Section 1, Section 4.1). Concretely, each point on the scaling curves represents a DiT-B trained for exactly 80 epochs on ImageNet 256×256 with identical hyperparameters, differing only in which tokenizer produced the latents. The compute accounting for tokenizer pre-training is based on total FLOPs consumed during the pre-training phase, varying from roughly 10^10 to 5 × 10^11 GFLOPs across scaling experiments. For data scaling, models are trained for a fixed 1.1 billion samples each on subsets of varying size. For parameter scaling, FLOPs vary with model size since all models train on the same dataset for the same number of samples.
Cross-validation / statistical protocol. The paper does not employ formal cross-validation on the generation evaluation (ImageNet has a fixed train/test split). For the scaling experiments, each tokenizer checkpoint is evaluated by training a DiT from scratch on ImageNet and reporting gFID on the standard validation set. The paper does not report error bars or confidence intervals on any metrics, nor does it perform multiple training runs with different random seeds to assess variance. The difficulty binning approach used in some scaling-law papers (cross-validating strategy selection on held-out folds) is not applicable here since VTP does not perform per-sample strategy selection. The linear probing accuracy is reported as a single number per tokenizer configuration, evaluated on the standard ImageNet validation protocol.

Main Quantitative Results

ViT as an Autoencoder Architecture

The paper first establishes that a Vision Transformer can serve as an effective autoencoder for latent diffusion, comparable to the convolutional architectures that dominate the field. Table 1 reports that ViT-L achieves 31.28 rPSNR and 53.51 gFID on ImageNet 256×256, compared to 30.63 rPSNR and 59.53 gFID for the standard convolutional LDM architecture (Rombach et al., 2022). The ViT uses substantially more parameters (607.2M vs. 70.3M) but fewer FLOPs (311.1G vs. 389.4G), which the paper attributes to the CNN operating at higher spatial resolution internally while the ViT maintains constant resolution. The ViT-B achieves 30.72 rPSNR and 58.40 gFID with 171.2M parameters and 87.7G FLOPs. These results position ViT as "a comparable alternative to CNNs, enabling simpler pre-training scaling experiments" (Table 1 caption)—the reconstruction quality is similar, and the uniform architecture facilitates systematic scaling studies.

Critically, Table 1 also reveals the reconstruction-generation trade-off that motivates the paper: ViT-L achieves the best reconstruction (31.28 rPSNR) but not the best generation (53.51 gFID vs. 58.40 for ViT-B). The ViT-B with worse reconstruction produces better downstream generation—a microcosm of the "pre-training scaling problem" that the paper diagnoses.

Reconstruction-Only Training Cannot Scale for Generation

Figure 4 provides the foundational negative result that defines the problem VTP solves. A ViT-L autoencoder trained solely with reconstruction objectives (L1 + perceptual loss, followed by GAN fine-tuning of the decoder) is tracked across increasing training FLOPs (from approximately 10^10 to 10^11 GFLOPs). The left panel shows reconstruction rFID improving substantially: 2.070 → 1.310 → 0.930 → 0.740 → 0.650 → 0.590 → 0.570 → 0.510 across eight measurement points at increasing compute. The right panel shows generation gFID degrading: 55.04 → 55.89 → 55.99 → 56.38 → 57.14 → 57.79 → 58.16 → 58.56 across the same compute range. This is the "scaling paradox" in its starkest form: a 10× increase in training FLOPs reduces rFID by 75% (from 2.07 to 0.51) while increasing gFID by 6% (from 55.04 to 58.56). The paper interprets this as evidence that the reconstruction objective "biases the latent space toward low-level information" (Section 1) and that scaling up reconstruction-only training "carries the risk of compromising generation performance" (Section 1).

This result is presented as a single run tracking one model's training trajectory, not as an average over multiple seeds. The paper does not report variance or seed sensitivity for this result, so the precise monotonicity of the degradation should be interpreted with caution—but the qualitative trend (reconstruction improves while generation worsens) is consistent with the broader scaling picture in Section 4.4-4.5.

Representation Learning Enables Positive Scaling: CLIP+AE and SSL+AE

Figure 5 shows the central scaling results that contrast reconstruction-only training with VTP's two single-objective variants. The figure is organized as a 2×3 grid: the top row uses bottleneck dimension d = 64, the bottom row uses d = 256; the three columns show generation-vs-understanding, generation-vs-reconstruction, and understanding-vs-reconstruction trade-offs respectively.

For d = 64 (top row, Figure 5a–c):

Figure 5a plots gFID (reversed axis, so upward is better) against linear probing accuracy. CLIP+AE and SSL+AE both show clear positive trajectories: as training FLOPs increase from 10^10 to 10^11, both understanding and generation improve together. CLIP+AE reaches approximately 34.1% linear probing and ~66.2 gFID at the lowest compute, improving to ~66.1% linear probing and ~34.8 gFID at the highest compute. SSL+AE reaches ~57.7% linear probing and ~55.0 gFID at the highest compute. In contrast, AE-only (gray dots) remains clustered at near-zero linear probing (~1–2%) and poor gFID (~60) regardless of compute budget—the points barely move, indicating stagnation rather than scaling.

Figure 5b plots gFID against rFID. AE-only shows an inverse relationship: as rFID improves (moving right on the reversed axis), gFID worsens (moving downward). This is the same pattern as Figure 4. In contrast, CLIP+AE and SSL+AE show that reconstruction and generation improve simultaneously—the points move upward and rightward together, indicating that adding representation learning objectives resolves the trade-off and enables complementary improvement in both metrics.

Figure 5c plots linear probing against rFID. AE-only achieves extremely low linear probing regardless of reconstruction quality. CLIP+AE and SSL+AE achieve substantial understanding while maintaining reconstruction quality that improves with compute.

For d = 256 (bottom row, Figure 5d–f): The patterns are qualitatively identical, confirming that the scaling behavior is not an artifact of the specific bottleneck dimension. At the highest compute, CLIP+AE (d = 256) achieves approximately 90.3 gFID (reversed) with ~78.0% linear probing, and SSL+AE achieves approximately 52.9 gFID with ~105.8 gFID (reversed). The absolute numbers differ (the d = 256 bottleneck generally achieves better generation at the same compute for CLIP, but SSL performs worse than at d = 64), but the qualitative scaling trend—representation learning enables concurrent improvement in understanding and generation, while reconstruction-only stagnates—is robust.

The paper draws four explicit observations from Figure 5 (Section 4.3):

(1) Feasibility of Hybrid Objectives: "Hybrid training combining representation learning with reconstruction is viable" (Section 4.3). Both understanding and reconstruction metrics exhibit "stable, simultaneous improvement" (Figure 5c, f) for CLIP+AE and SSL+AE, while AE-only maintains low understanding even as reconstruction improves.

(2) Negative Impact of Pure Reconstruction: "Solely relying on reconstruction proves counterproductive for downstream generation tasks" (Section 4.3). The negative yield in Figure 5b, e—where AE-only shows rFID improving but gFID degrading—directly supports this claim. The paper notes this occurs "as the computational budget increases" (Section 4.3), linking it to scaling specifically.

(3) Understanding as the Key Driver: The reversal of the negative trend when representation learning is added is "visible in Figure 5a, d and b, e" (Section 4.3). Joint optimization of reconstruction and representation learning "enables continuous, concurrent improvement in reconstruction, understanding, and generation as pre-training scales." The claim that understanding is the "key driving force necessary for effective generation" is supported by the fact that both CLIP and SSL—despite being qualitatively different objectives—produce the same qualitative shift in the scaling curves.

(4) Generality of Representation Learning: Both CLIP and SSL "consistently enhance generation performance" despite "significant differences in their training frameworks" (Section 4.3). The paper notes that their "scaling behaviors differ slightly" but both improve efficacy for downstream generation. This generalizability is important because it suggests VTP can incorporate future representation-learning advances.

Combining CLIP and SSL Achieves the Best Results

Figure 6 shows the scaling behavior of the full VTP framework (CLIP+SSL+AE) compared to the single-objective variants at f16d64. The figure plots three panels: generation-vs-understanding, generation-vs-reconstruction, and understanding-vs-reconstruction, with points from three compute levels (the CLIP+AE Best and SSL+AE Best points at their respective optimal compute budgets, and CLIP+SSL+AE at three compute levels).

Under the same computational budget, CLIP+SSL+AE achieves both the best generation (~27.8 gFID, reversed axis) and the best understanding (~74.9% linear probing), outperforming the optimal single-objective configurations. The CLIP+AE Best achieves ~34.1 gFID with ~81.3 gFID (reversed) and ~35% linear probing; the SSL+AE Best achieves ~57.7 gFID with ~55.0 gFID (reversed) and ~57% linear probing. The combination improves generation by roughly 19% over the best CLIP-only variant and roughly 50% over the best SSL-only variant (in raw gFID), while simultaneously achieving the highest linear probing accuracy.

The understanding-vs-reconstruction panel (Figure 6c) shows that CLIP+SSL+AE achieves substantially better understanding than either single-objective variant at comparable reconstruction quality. The paper interprets this as evidence that the multi-objective framework "enables the tokenizer to capture multi-scale features, enhancing both semantic alignment and spatial fidelity" (Section 4.3). This result is the basis for all subsequent data and parameter scaling experiments, which use the full CLIP+SSL+AE configuration.

Data Scaling: VTP Improves with More Training Data; Autoencoders Do Not

Figure 7a presents the data scaling experiment. Both VTP-ViT-Large and AE-ViT-Large are trained on four subsets of the DataComp-1B dataset—100K, 1M, 10M, and 100M samples—with each model trained for 1.1 billion total samples (meaning the smaller subsets are revisited many times). Downstream DiT training uses identical compute (80 epochs on ImageNet) for all tokenizer configurations.

The AE baseline shows essentially flat generation performance across all data scales: gFID of 58.37 at 100K samples, 58.39 at 1M, 57.23 at 10M, and 56.71 at 100M. The total improvement from 100K to 100M is only 1.66 gFID—a 2.9% relative improvement for a 1000× increase in training data. This confirms that reconstruction-only tokenizers do not benefit from data scaling.

In stark contrast, VTP shows dramatic improvement: gFID of 47.59 at 100K samples, 40.57 at 1M, 30.98 at 10M, and 27.45 at 100M. The total improvement from 100K to 100M is 20.14 gFID—a 42% relative improvement. Moreover, VTP already substantially outperforms the AE at 100K (47.59 vs. 58.37), and the gap widens dramatically as data scales. The paper emphasizes that "the downstream DiT training FLOPs remained strictly identical" (Section 4.5), isolating the data scaling benefit to the tokenizer pre-training phase.

The paper also reports a data scaling experiment for VTP-B in the same figure, achieving 30.98 gFID at 100M samples compared to 27.45 for VTP-L, consistent with the encoder scaling results.

Parameter Scaling: Larger Encoders and Decoders Improve VTP but Not Autoencoders

Figure 7b shows encoder parameter scaling. Three ViT encoder sizes (Small, Base, Large) are trained with both VTP (CLIP+SSL+AE) and AE-only configurations, with downstream gFID measured at fixed DiT training FLOPs.

For the AE configurations: AE-S achieves approximately 57.87 gFID, AE-B achieves approximately 57.52 gFID, and AE-L achieves approximately 57.42 gFID. The curve is nearly flat—a 15× increase in encoder parameters (from ~20M to ~300M) yields essentially zero improvement in downstream generation. The paper describes this as performance that "remains stagnant at about 57, regardless of the model capacity" (Section 4.4).

For VTP: VTP-S achieves approximately 31.28 gFID, VTP-B achieves approximately 27.83 gFID, and VTP-L achieves approximately 26.12 gFID. The improvement from Small to Large is roughly 5.16 gFID—a 16.5% relative improvement. The curve shows a clear downward slope, indicating that "VTP exhibits a clear scaling trend: its gFID improves steadily from 31.28 to 26.12 as the model size grows, forming a well-defined parameter scaling curve" (Section 4.4). This is the first evidence in the paper that VTP's scaling property extends beyond compute (training FLOPs) to model capacity.

Figure 7c shows decoder scaling. Using a VTP-L encoder, the paper compares a 4-layer ViT-L decoder against a 24-layer ViT-L decoder, with AE-L baselines at both decoder depths. For AE: the 4-layer decoder achieves 57.87 gFID and the 24-layer decoder achieves 58.32 gFID—essentially flat, with a slight degradation at the larger decoder size. For VTP: the 4-layer decoder achieves 26.12 gFID and the 24-layer decoder achieves 24.08 gFID—a clear improvement of 2.04 gFID (7.8% relative) from scaling the decoder. This demonstrates that VTP's scaling property extends to both encoder and decoder capacity independently, while autoencoders cannot exploit additional decoder parameters for generation improvement.

Text-to-Image Scaling: VTP Properties Generalize Beyond Class-Conditional Generation

Section 5 extends the evaluation to text-to-image (T2I) generation on LAION, using a DiT-XL architecture as the downstream generative model. Figure 8a shows four evaluation benchmarks (GenEval, T2I-CompBench, DPG-Bench, TIIF-Long) plotted against DiT training iterations (0 to 250K).

The key findings are: (1) VTP converges faster than the AE baseline. On GenEval, VTP-f16d24 with 0.5B pre-training samples reaches approximately 0.65 score at 250K iterations, substantially above the ViT-AE baseline at approximately 0.55. On T2I-CompBench, VTP reaches approximately 0.63 vs. 0.55 for AE. On DPG-Bench, VTP reaches approximately 82.5 vs. 75.0 for AE. This accelerated convergence mirrors the class-conditional results, demonstrating that semantic-aware tokenizers improve downstream training efficiency across tasks.

(2) Increased tokenizer pre-training compute improves downstream T2I generation. The VTP variant with 3.3B pre-training samples (larger compute) consistently outperforms the variant with 0.5B samples across all four benchmarks and at all iteration counts. On GenEval at 250K iterations, 3.3B-sample VTP achieves approximately 0.70 vs. 0.65 for 0.5B-sample VTP. This confirms that "VTP's scaling properties are not limited to class-conditional ImageNet generation but generalize to the more challenging T2I setting" (Section 5).

(3) CLIP loss is specifically beneficial for text rendering. Figure 8b ablates the effect of different semantic losses on T2I generation quality. The VTP variant trained with CLIP+SSL+AE (the full framework) outperforms both AE-only and SSL+AE variants, with a particularly pronounced advantage on text rendering tasks. The paper states that "the tokenizer trained with the CLIP loss exhibits a significant advantage in text rendering for text-to-image generation, substantially outperforming both AE and SSL+AE tokenizers" (Figure 8b caption). This is a targeted benefit: CLIP's image-text alignment directly helps the tokenizer encode linguistic concepts (like the visual appearance of specific words) into the latent space, which the diffusion model can then exploit.

Final Scaled Results and Comparison with Prior Work

Table 2 presents the comprehensive comparison against prior work on ImageNet 256×256 generation. The table is organized into three sections: perception/unified tokenizer baselines, convergence efficiency at 80 epochs, and long-period training results.

Convergence efficiency (80 epochs, no guidance): VTP-L (with a 675M-parameter DiT) achieves 2.62 gFID in 80 epochs without classifier-free guidance. This substantially outperforms VA-VAE at 4.29 gFID, RAE at 4.28 gFID, REPA-E at 3.46 gFID, and DDT at 6.62 gFID—all using the same or comparable DiT configurations. Scaling the DiT to 1.0B parameters with VTP-L achieves 2.03 gFID at 80 epochs, the best unguided result in this training-time regime. The paper emphasizes the exceptionally fast convergence: "VTP attains 2.60 and 2.03 gFID without guidance in only 80 epochs, demonstrating remarkably fast convergence" (Section 6).

Convergence efficiency (80 epochs, with guidance): With classifier-free guidance, VTP-L achieves 1.44 gFID (675M DiT), outperforming UniTok at 2.77 gFID (1.4B model), DDT at 1.52 gFID, and REPA-E at 1.67 gFID. RAE with an 835M-parameter DiT achieves 2.16 gFID at 80 epochs with guidance, substantially worse than VTP.

Long-period training (400–1400 epochs): With extended training, VTP-L (675M DiT, 600 epochs) achieves 1.85 gFID unguided and 1.11 gFID with guidance. This is the best reported result in the table: VA-VAE reaches 1.35 gFID with guidance at 800 epochs, REPA-E reaches 1.15 gFID at 800 epochs, RAE reaches 1.41 gFID at 800 epochs, and DDT reaches 1.26 gFID at 400 epochs. RAE* (an 839M-parameter variant trained for 800 epochs) achieves 1.13 gFID with guidance—competitive with VTP but requiring a larger DiT and 33% more training epochs (800 vs. 600). The standard DiT baseline (Peebles and Xie, 2023) at 1400 epochs achieves only 2.27 gFID with guidance, demonstrating the substantial headroom that better tokenizers provide.

Understanding metrics: VTP achieves 0.36 rFID, 78.2% zero-shot accuracy, and 85.7% linear probing accuracy on ImageNet. These understanding metrics surpass VILA-U (1.80 rFID, 73.3% zero-shot) and UniTok (0.41 rFID, 70.8% zero-shot), while simultaneously achieving better generation. The claim that VTP "pushes the boundaries of unified understanding and generation" (Section 6) is supported by this simultaneous improvement on both fronts.

Scalability comparison with RAE (Table 3): Under identical DiT training configurations (LightningDiT, 80 epochs, no guidance), RAE's generation performance degrades as the tokenizer scales: Small achieves 3.50 gFID, Base achieves 4.28, Large achieves 6.09. In contrast, VTP consistently improves: Small achieves 5.46 gFID, Base achieves 3.88, Large achieves 2.81. The paper interprets this as a demonstration that "VTP's downstream generation improves consistently as the tokenizer scales from S to L, whereas RAE's performance degrades at larger scales" (Section 6, Table 3 discussion). The RAE degradation is attributed to the fixed-capacity DINOv2 teacher becoming a bottleneck—the larger RAE encoder can better match the teacher's features, but since the teacher is fixed, the latent space cannot improve beyond the teacher's representational quality. VTP, by jointly learning representations rather than distilling them, avoids this ceiling.

Reconstruction comparison with RAE (Figure 9a): The paper provides visual examples of reconstructions from VTP and RAE, showing that "VTP always involves reconstruction during training, it preserves fine-grained details significantly better than RAE" (Section 6). This is the trade-off: RAE optimizes exclusively for semantic alignment with DINOv2, sacrificing reconstruction fidelity; VTP jointly optimizes both, maintaining reconstruction quality while achieving better generation.

Convergence visualization (Figure 9b): The convergence plot shows gFID against DiT training epochs for VA-VAE, RAE, and VTP-L. VTP-L achieves approximately 2.61 gFID at 80 epochs, substantially below VA-VAE at approximately 4.29 and RAE at approximately 4.28. The curves diverge early and the gap persists, indicating that VTP's latent space enables faster learning throughout training, not just better final performance.

Ablation Studies and Robustness Checks

Bottleneck dimension (d = 64 vs. d = 256): Figure 5 systematically compares the scaling behavior at two bottleneck dimensions. The qualitative patterns—representation learning enables positive scaling while reconstruction-only stagnates—hold for both configurations, confirming robustness to architectural hyperparameters. The d = 256 CLIP variant achieves substantially better generation than d = 64 CLIP at the same compute (approximately 90 vs. 66 gFID reversed), while SSL shows the opposite trend (approximately 53 vs. 55 gFID reversed). This interaction between bottleneck dimension and representation-learning objective is not explored further, but the robustness of the core scaling property is clear across both settings.

CLIP vs. SSL vs. CLIP+SSL: The comparison across Figures 5 and 6 constitutes a de facto ablation of which representation-learning objectives matter. CLIP alone enables positive scaling (Figure 5a, d). SSL alone enables positive scaling (Figure 5a, d). The combination (Figure 6) yields better results than either alone at comparable compute. This demonstrates that the objectives provide complementary benefits rather than being redundant—a finding that is not obvious a priori, since both could be seen as different routes to the same semantic-enrichment goal.

Reconstruction loss weight: The paper states in Section 4.1 that "we set λ_rec = 0.1, while λ_clip and λ_ssl are set to either 0 or 1" and notes that "a smaller reconstruction weight contributes to improved generative performance." This is presented as an empirical finding rather than an ablated result with multiple weight configurations. The paper does not provide a figure or table showing how generation performance varies as λ_rec is varied (e.g., 0.01 vs. 0.1 vs. 0.5 vs. 1.0), so the sensitivity to this hyperparameter is unclear. The choice appears to be based on preliminary experiments, but the lack of a systematic weight ablation is a limitation—it leaves open the question of whether even smaller reconstruction weights (or dynamically scheduled weights) would further improve generation.

GAN fine-tuning stage: The two-stage training procedure (L1+perceptual pre-training followed by GAN fine-tuning of the frozen decoder) is described in Section 3.2 and used throughout, but the paper does not ablate whether the GAN fine-tuning stage is necessary for the scaling benefits, or whether similar results could be achieved with end-to-end training that includes GAN loss from the start despite the reported instability. The GAN stage is applied to the decoder only, not the encoder, so it should not affect the latent space structure—but this assumption is not tested.

EMA teacher momentum schedule: The paper follows DINOv2's established practices (Section 4.1) but does not ablate the teacher momentum schedule or the MIM masking ratio. These are standard SSL hyperparameters known to affect representation quality, and the paper's choice to follow prior work's defaults means the robustness of VTP's scaling to SSL hyperparameters is assumed rather than demonstrated.

Decoder depth: Figure 7c shows that scaling the decoder from 4 to 24 layers improves VTP's generation (26.12 → 24.08 gFID) but not the autoencoder's (57.87 → 58.32 gFID). This is presented as a scaling result rather than an ablation, but it also serves to demonstrate that the decoder architecture matters and that VTP's scaling property extends to the decoder. The paper does not explore even larger decoders or alternative decoder architectures.

DiT size: Table 2 includes results for both 675M and 1.0B parameter DiT models trained on VTP latents, showing that the benefits persist when scaling the downstream model. At 80 epochs unguided: 675M DiT achieves 2.62 gFID, 1.0B DiT achieves 2.03 gFID. This demonstrates that VTP's benefit is not specific to a particular generative model scale, though only two sizes are tested.

Pre-training data filtering: The paper uses "an internally filtered version of DataComp-1B" (Section 4.1) without specifying the filtering criteria. This is a practical limitation for reproducibility—external researchers cannot replicate the exact training data distribution. The data scaling experiments (Figure 7a) show that VTP's advantage over AE grows with data scale, suggesting that data quality and diversity matter, but the sensitivity to specific filtering choices is unknown.

Negative result: Autoencoder data scaling is essentially flat. Figure 7a shows that training an autoencoder on 1000× more data (100K to 100M) improves gFID by only 2.9% (58.37 to 56.71). This is a striking negative result: it demonstrates that the pre-training scaling problem is not merely a function of insufficient data—even with abundant and diverse data, reconstruction-only training does not benefit from scale. This strengthens the paper's argument that the objective, not the data quantity, is the root cause.

Negative result: RAE degrades with scale (Table 3). RAE's gFID worsens from 3.50 (Small) to 6.09 (Large) under the same DiT training protocol, while VTP improves from 5.46 to 2.81. This directly supports the paper's claim that distillation-based approaches hit a performance ceiling and cannot scale, while VTP's joint representation-learning approach avoids this limitation.

Critical Assessment

Does the Evidence Support the Central Claim That VTP Enables Scaling Where Reconstruction-Only Training Fails?

The paper's central claim is that VTP "demonstrates a new scaling property for visual tokenizers: overcoming the early saturation of reconstruction-only training, downstream generation quality consistently improves as VTP pre-training scales in compute, parameters, and data" (Section 1, contribution summary).

Compute scaling (Figures 5 and 6): The evidence for compute scaling is clear and well-controlled. As tokenizer pre-training FLOPs increase from approximately 10^10 to 10^11, both CLIP+AE and SSL+AE variants show monotonic improvement in downstream gFID, while the AE-only variant shows degradation (Figures 4, 5a, 5b). The three-point scaling curve for CLIP+SSL+AE (Figure 6) also shows improvement. The key strength is the controlled evaluation protocol: all downstream DiT training is identical, isolating the tokenizer's contribution. The main limitation is the relatively narrow compute range (approximately one order of magnitude). The paper does not demonstrate whether the scaling trend continues for another 10× or 100× of compute, or whether it eventually saturates like the autoencoder—just at a higher performance level and a later point. The "scaling law" claim is supported for the tested range but extrapolation is not validated.

Parameter scaling (Figure 7b): Three encoder sizes (S, B, L) show clear improvement for VTP (31.28 → 27.83 → 26.12 gFID) and stagnation for AE (~57.87 → ~57.52 → ~57.42 gFID). The scaling curve is well-defined across roughly a 15× parameter range. However, the VTP improvement is diminishing: the jump from S to B (+3.45 gFID) is larger than B to L (+1.71 gFID), suggesting potential saturation. Testing a ViT-Huge or larger encoder would reveal whether the scaling trend continues or plateaus. The decoder scaling (Figure 7c) tests only two points (4 vs. 24 layers), which is insufficient to characterize a scaling law but sufficient to demonstrate that decoder capacity matters.

Data scaling (Figure 7a): Four data scales (100K, 1M, 10M, 100M) show dramatic improvement for VTP (47.59 → 40.57 → 30.98 → 27.45 gFID) and near-flat performance for AE (58.37 → 58.39 → 57.23 → 56.71). The data range is four orders of magnitude, which is a strong scaling test. The diminishing returns are visible: the jump from 10M to 100M (+3.53 gFID) is smaller than from 1M to 10M (+9.59 gFID), which is expected under typical scaling laws. A natural extrapolation question—whether 1B samples would yield further improvement—is left unanswered but the trend is consistent with continued (diminishing) gains.

A critical caveat on the "scaling law" framing: The paper uses the language of "scaling law" to describe VTP's behavior, but it does not fit a parametric function (e.g., power law) to the scaling data, does not report fitted exponents, and does not make quantitative predictions about performance at larger scales. This is a qualitative scaling law—"generation improves as compute/parameters/data increase"—rather than a quantitative one like Chinchilla or Kaplan. This doesn't invalidate the contribution, but it means the claims are more accurately described as "demonstrating positive scaling behavior" rather than "establishing a scaling law."

Does the Evidence Support the Claim That Semantic Understanding Is the Key Driver of Generation?

The paper claims that "understanding is a key driver of generation" (Section 1) and that semantic understanding is the mechanism by which VTP improves downstream generation. The evidence structure is:

Correlational evidence: Figures 2, 5a, 5d, and 6a all show positive correlation between linear probing accuracy (understanding) and gFID (generation) across different training configurations and compute scales. The correlation is strong and consistent—as understanding improves, generation improves.

Intervention-like evidence: Adding representation-learning objectives (CLIP, SSL, or both) changes the relationship between compute and generation from negative/neutral to positive (Figures 4 vs. 5). This is a between-method comparison that functions similarly to an intervention study: changing the training objective from reconstruction-only to reconstruction-plus-representation changes the scaling behavior. The fact that both CLIP and SSL independently produce this shift, despite being different learning paradigms, strengthens the causal interpretation—it suggests the common mechanism (semantic understanding) rather than a peculiarity of one objective is responsible.

Differential prediction evidence: The claim that understanding specifically drives generation makes a prediction that the CLIP objective (which provides cross-modal semantic alignment) should be particularly beneficial for tasks requiring cross-modal understanding. Figure 8b confirms this for text-to-image generation: CLIP loss specifically improves text rendering, a task that depends on the alignment between linguistic concepts and visual features. This is a stronger test of the causal claim than the general correlation, because it shows a specific benefit in a specific downstream task that directly requires the type of understanding the CLIP objective provides.

What's missing: The paper does not provide direct evidence that the latent space structure becomes more "semantic" under VTP training in a measurable way beyond linear probing. For example: measuring the semantic smoothness of the latent space (are interpolations between latents semantically coherent?), measuring nearest-neighbor consistency (do semantically similar images map to nearby latents?), or measuring the degree to which the latent space can be decomposed into semantic factors (object identity, pose, background separated along different dimensions). Linear probing measures whether semantic information is present, but not whether the geometry of the space is semantically organized in a way that specifically benefits diffusion model training. The causal claim would be strengthened by showing that semantic structure (not just semantic information) mediates the relationship between VTP pre-training and downstream generation quality.

Alternative explanations that are not ruled out: The multi-objective training might confer benefits through mechanisms other than semantic understanding. For instance, joint training with multiple objectives could act as a regularizer that prevents the latent space from over-specializing to any single objective—similar to how multi-task learning can improve generalization by preventing overfitting. The MIM objective might improve generation by teaching the model spatial prediction skills (filling in masked regions) that are directly relevant to the diffusion model's denoising task, independent of any semantic understanding. The CLIP objective might improve generation by aligning the latent distribution with a text-conditional prior that makes the diffusion model's conditioning task easier, even if the latent space doesn't capture semantic concepts per se. The paper does not disentangle these mechanisms—the "understanding drives generation" claim is the most parsimonious explanation given the evidence, but it is not the only possible explanation, and no experiment directly tests the hypothesized causal pathway.

Does the Evidence Support the Claim of State-of-the-Art Performance?

Generation performance (Table 2): VTP's 1.11 gFID with guidance on ImageNet 256×256 is the best reported number in the table. The margin over prior methods is meaningful: VA-VAE (1.35), REPA-E (1.15), RAE* (1.13), DDT (1.26). However, the comparison is not entirely like-for-like: VTP uses 600 DiT training epochs, while VA-VAE uses 800, REPA-E uses 800, and RAE* uses 800. VTP achieves better results with fewer epochs (25% fewer than VA-VAE/REPA-E), which strengthens the claim. But the DiT architectures differ across methods (LightningDiT for VTP/VA-VAE, custom configurations for RAE, REPA, etc.), and tokenizer pre-training compute budgets are not controlled or compared. The claim "surpassing all prior methods" is supported for the specific metrics reported, but the underlying experimental setups are heterogeneous.

Understanding performance: VTP's 78.2% zero-shot accuracy and 85.7% linear probing accuracy are strong, but not directly comparable to dedicated vision encoders like SigLIP (80.5% zero-shot, from Zhai et al., 2023) or DINOv2 (86.7% linear probing). The paper acknowledges that these baselines are provided "for reference" and that VTP's encoder is operating under a fundamentally different constraint—it must also serve reconstruction and operate at a 16× compression ratio with a 64-dimensional bottleneck. The fact that VTP approaches the performance of dedicated understanding models while simultaneously enabling state-of-the-art generation is the contribution, rather than surpassing them on understanding metrics alone.

Unified understanding and generation: The simultaneous achievement of 1.11 gFID (generation) and 0.36 rFID + 78.2% zero-shot (understanding) does surpass prior unified tokenizers VILA-U and UniTok, which achieve worse numbers on both fronts. This is the clean SOTA claim: under the specific definition of "unified tokenizer for understanding and generation," VTP achieves the best reported trade-off. The paper is careful to separate this claim from the broader generation-only or understanding-only SOTAs.

Genuine Weaknesses and Missing Experiments

No confidence intervals or seed analysis. All figures and tables report single-number metrics without error bars, standard deviations, or multi-seed averaging. The scaling curves could be sensitive to DiT training seed, which is known to introduce non-trivial variance in FID for diffusion models trained on ImageNet. Without multiple runs, it's impossible to assess whether the observed differences (e.g., 2.62 vs. 4.29 gFID at 80 epochs in Table 2) are statistically robust or within noise. This is particularly concerning for the smaller comparisons, like the three-point scaling curves in Figure 6, where a single anomalous DiT training run could distort the apparent scaling trend.

Narrow compute scaling range. The tokenizer compute scaling covers approximately one order of magnitude (10^10 to 10^11 GFLOPs). This is sufficient to demonstrate a qualitative difference from autoencoders (which degrade over this range) but insufficient to establish quantitative scaling-law parameters. A true scaling law analysis would span at least 2–3 orders of magnitude and fit functional forms. The paper uses "scaling law" language that implies more quantitative generality than the experiments directly support.

No end-to-end training comparison. VTP uses a two-stage process (tokenizer pre-training, then frozen tokenizer + DiT training). An alternative approach, exemplified by methods like REPA-E, jointly optimizes the VAE and DiT during training. The paper compares against REPA-E's published results but does not experiment with making VTP's tokenizer trainable during DiT training, which could potentially yield further gains. The frozen-tokenizer paradigm is a deliberate design choice (it's the standard LDM approach), but the absence of this comparison means the optimal point in the design space (fully decoupled vs. end-to-end) is not determined.

Limited text encoder and contrastive setup details. The text encoder is described as "a 12-layer transformer with a hidden dimension of 768" (Section 4.1), but the paper does not specify its initialization (random? pre-trained?), training data (all 277M captions? a filtered subset?), or contrastive loss details (temperature parameter, projection head architecture). These details matter for reproducibility and for understanding how much of the CLIP benefit comes from pre-trained text knowledge versus jointly-learned alignment.

No exploration of the interaction between DiT architecture and tokenizer properties. The paper uses LightningDiT-B/XL as a fixed downstream model, which is methodologically clean for isolating tokenizer contributions. However, it leaves open the question of whether different DiT architectures would interact differently with VTP latents—for instance, whether a DiT with different patch size, depth, or conditioning mechanism would benefit more or less from VTP's semantic structuring. The claim that VTP "generalizes" to other generation settings is supported by the LAION T2I experiments, but only with DiT-XL. Testing with other generative model families (e.g., SiT, U-ViT, autoregressive transformers in latent space) would broaden the evidence for generality.

Difficulty estimation and adaptive allocation are not explored. Unlike the compute-optimal scaling analysis in some prior work, VTP applies the same pre-training recipe regardless of image characteristics. The paper does not investigate whether certain types of images (e.g., highly textured vs. semantically complex) benefit differently from different loss weightings, or whether an adaptive tokenizer that adjusts its compression strategy per image could yield further gains.

The 277M DataComp-1B subset is not reproducible. The paper uses "an internally filtered version of DataComp-1B" without specifying the filtering methodology, data composition statistics, or distribution of caption quality. This makes exact reproduction impossible and introduces uncertainty about whether the scaling benefits depend on specific data properties (e.g., high-quality captions for CLIP, diverse visual concepts for SSL). The data scaling experiment (Figure 7a) partially mitigates this concern by showing benefits across different random subsets, but the base dataset's properties could still be a confound.

The reconstruction-generation paradox at extreme scales is not explored. Figure 4 shows that reconstruction-only training degrades generation as compute increases. Does VTP eventually face a similar tension at much larger scales? If reconstruction and semantic understanding inherently compete for latent capacity, then even VTP might eventually reach a point where further pre-training improves understanding at the expense of generation, or vice versa. The paper does not test whether VTP's positive scaling eventually saturates or reverses at substantially larger compute than the tested range. This is a natural next step that would determine whether VTP truly resolves the pre-training scaling problem or merely pushes the saturation point to a higher compute threshold.

No zero-shot or few-shot generation evaluation on diverse benchmarks. All generation evaluation beyond ImageNet and LAION uses standard metrics on standard benchmarks. The paper does not evaluate on out-of-distribution generation tasks, few-shot adaptation, or diverse real-world scenarios that would test the robustness of the learned latent space. The claim that VTP produces a "unified and rich representation" (Section 1) that should generalize broadly is not tested beyond in-distribution generation quality.

Summary of Evidential Support

The paper's three headline claims are:

VTP enables a new scaling property for visual tokenizers — Strongly supported within the tested ranges (1 OOM of compute, 15× parameters, 1000× data), with the caveat that the compute scaling range is narrow and the "scaling law" terminology implies more quantitative precision than the experiments provide. The contrast with autoencoder baselines is stark and consistent across all three scaling axes.
Understanding is the key driver of generation quality — Supported with strong correlational evidence and plausible mechanism but not causally proven. The correlation between linear probing and gFID is robust, and both CLIP and SSL independently shift scaling behavior from negative to positive. The differential benefit of CLIP for text rendering strengthens the causal interpretation. Missing: direct measurement of latent space semantic structure, disentanglement of understanding from alternative mechanisms (regularization, spatial prediction skill).
VTP achieves state-of-the-art unified understanding and generation — Supported for the specific definition of "unified tokenizer achieving both top-tier generation and strong understanding." Under identical DiT training, VTP-L achieves 1.11 gFID, which is the best reported number in Table 2, while simultaneously achieving understanding metrics that approach dedicated encoders. The comparison is fair but heterogeneous across methods in terms of DiT architecture and training epochs.

6. Limitations and Trade-offs

The Difficulty Estimation Cost for Tokenizer Selection Is Unaccounted For

The assumption or constraint. The paper demonstrates that VTP pre-training scales positively with compute, parameters, and data, but it does not address a related practical question: given multiple candidate tokenizer checkpoints trained at different scales, how does a practitioner decide which one to deploy? In the paper's experimental framework, this is implicitly solved by exhaustive evaluation—training a full DiT from scratch on each tokenizer checkpoint and selecting the one with the best gFID. This is extremely expensive. The paper does not propose or evaluate a lightweight proxy metric (e.g., linear probing accuracy on a small validation set, rFID on a subset of images) that could predict downstream generation quality without full DiT training. The authors do not acknowledge this gap explicitly, but it is a direct consequence of their evaluation methodology: every point on the scaling curves in Figures 5, 6, and 7 required training a DiT-B for 80 epochs on ImageNet. In a deployment setting, running dozens of 80-epoch DiT training runs to select a tokenizer is cost-prohibitive.

The consequence. The paper's headline result—that VTP scales positively—is demonstrated in a setting where the practitioner can afford to evaluate many tokenizer configurations. In practice, a team pre-training a tokenizer on a large dataset may not have the budget to run downstream DiT training for every candidate checkpoint. Without a reliable proxy metric, they cannot determine whether their tokenizer pre-training has reached diminishing returns or whether additional compute would yield further generation improvements. This introduces a risk: the practitioner might stop pre-training too early (leaving generation quality on the table) or too late (wasting compute for negligible gains). The paper's linear probing accuracy is correlated with generation quality (Figures 2, 5a, 6a), but the paper never validates whether linear probing can serve as a selection criterion—whether the tokenizer checkpoint with the best linear probing accuracy on a validation set reliably corresponds to the best downstream generation. The correlation may not be sufficiently tight to support checkpoint selection, especially at high performance levels where differences between checkpoints are small.

What evidence exists in the paper. Figures 5a and 6a show that linear probing and gFID are positively correlated across compute scales, which is suggestive but not sufficient. The paper does not report whether the rank-ordering of tokenizer checkpoints by linear probing matches the rank-ordering by downstream gFID. For example, in Figure 5a, the CLIP+AE and SSL+AE trajectories show different slopes—a given linear probing accuracy corresponds to different gFID values depending on the objective, meaning linear probing alone would not select the objectively best tokenizer for generation. The paper never evaluates a direct proxy-selection protocol (e.g., "train linear probe, pick best checkpoint, measure gFID, compare to oracle selection").

Mitigation status. Not addressed. The paper's evaluation framework assumes exhaustive downstream evaluation. Future work on lightweight tokenizer quality predictors—perhaps using the PRM-style scoring from companion literature, or training a small predictor of downstream gFID from tokenizer properties (bottleneck dimension, linear probing accuracy, rFID)—would close this gap. The paper's emphasis on understanding as the driver of generation (Section 4.3, Observation 3) gestures toward such a predictor, but no concrete proposal is developed or tested.

The Pre-Training Scale Is Limited to One Order of Magnitude in Compute; Quantitative Scaling Laws Are Not Established

The assumption or constraint. The paper makes claims about VTP "unlocking a new scaling law for generation" (Section 1, title) and demonstrating "a new scaling property" (Section 1, contribution summary). However, the tokenizer compute scaling experiments in Figures 5 and 6 span approximately one order of magnitude—from roughly 10^10 to 10^11 GFLOPs. The parameter scaling (Figure 7b) covers a ~15× range (ViT-S to ViT-L). The data scaling (Figure 7a) covers four orders of magnitude in dataset size, but all models are trained for a fixed 1.1 billion samples, so the effective compute varies less than the data range suggests (training on 100K samples for 1.1B iterations means ~11,000 epochs over the subset, introducing a different confound: extreme overfitting to a small set).

The paper does not fit a parametric function (e.g., a power law gFID ∝ C^(-α) relating gFID to compute C) to the scaling data, does not report fitted exponents, and does not make quantitative predictions about performance at larger scales. The "scaling law" language therefore describes a qualitative property (monotonic improvement) rather than a quantitative relationship. This is a meaningful distinction: a qualitative scaling property says "more compute helps"; a quantitative scaling law says "10× more compute improves gFID by approximately X points." The paper provides the former but not the latter.

The consequence. Without a fitted scaling law, practitioners cannot perform cost-benefit analysis. How much should they invest in tokenizer pre-training? If scaling from ViT-L to a hypothetical ViT-H (~4× parameters) yields a 0.5 gFID improvement or a 5.0 gFID improvement, the investment decision is entirely different. The paper's curves in Figure 7b already show diminishing returns: VTP-S to VTP-B improves by ~3.45 gFID, while VTP-B to VTP-L improves by ~1.71 gFID, suggesting the marginal benefit of additional parameters is declining. Without a functional form, it is unclear whether a ViT-H would continue this trend (another ~0.8 gFID improvement) or whether performance has effectively saturated at ViT-L scale, with the remaining gap to zero gFID being dominated by the DiT architecture and training procedure rather than tokenizer quality.

Relatedly, the one-order-of-magnitude compute range may not be sufficient to distinguish a true power-law scaling regime from a transient improvement phase. It is possible that VTP's gFID improves for the first ~10^11 GFLOPs but then saturates at a level determined by the fixed DiT architecture—that VTP, like the autoencoder, eventually hits a ceiling, just at higher compute and better absolute performance. The paper provides no evidence that VTP's scaling is fundamentally different from the autoencoder's in kind rather than in degree (i.e., that VTP avoids saturation rather than merely delaying it).

What evidence exists in the paper. Figures 4, 5, 6, and 7 all show data points within the limited ranges described. The paper acknowledges the compute scaling range implicitly by the x-axis limits on these figures. The diminishing returns in Figure 7b (encoder scaling) and Figure 7a (data scaling from 10M to 100M showing a smaller improvement than 1M to 10M) are visible but not discussed by the authors. The paper does not extrapolate or make claims about performance at larger scales—the "scaling law" framing is aspirational rather than predictive.

Mitigation status. Not addressed. The paper does not discuss the limitations of its scaling range or the distinction between qualitative scaling properties and quantitative scaling laws. Future work training VTP tokenizers at substantially larger compute budgets (10^12+ GFLOPs) with larger encoders (ViT-H, ViT-G) and fitting power-law or broken-power-law functions would be needed to establish whether VTP exhibits a true scaling law or a shifted saturation point. The paper's contribution would be on firmer ground if it explicitly characterized what it has demonstrated (positive scaling behavior over the tested range) versus what remains to be shown (a quantitative scaling law).

All Experiments Use a Single Model Family and a Single Downstream Architecture; Generality to Other Generative Paradigms Is Unverified

The assumption or constraint. Every experiment in the paper uses the same base architecture family: ViT encoders and decoders for the tokenizer, paired with LightningDiT for the downstream diffusion model. The ViT autoencoder architecture itself was validated as a "comparable alternative to CNNs" (Table 1, Section 4.2) for reconstruction-and-generation tasks, but all subsequent scaling experiments—the core of the paper's contribution—use exclusively ViT-based tokenizers. The paper does not test whether VTP's multi-objective pre-training paradigm transfers to convolutional VAE architectures (the dominant design in deployed LDMs like Stable Diffusion), to other generative model families (e.g., autoregressive transformers in latent space, masked generative models, flow matching models), or to other decoder architectures.

The paper states that its ViT autoencoder is designed "leveraging its flexibility in learning visual representations" (Section 3.1), implying that the ViT architecture is particularly well-suited to the integration of representation-learning objectives. CNNs, which lack the uniform sequence-processing structure that makes it natural to attach SSL and CLIP heads to intermediate features, might not support the same multi-objective training as cleanly. This raises the possibility that VTP's success is partly architectural—ViT's inductive biases for representation learning, combined with reconstruction—rather than purely a consequence of the multi-objective training formulation.

The consequence. A practitioner using a convolutional VAE (e.g., the SDXL VAE) cannot assume, based on this paper, that adding SSL and CLIP objectives to their tokenizer pre-training would produce similar scaling benefits. The architectural differences are substantial: CNNs process features hierarchically with spatial resolution changes, which complicates the attachment of global objectives like CLIP contrastive loss (which typically pools features to a single vector) and SSL objectives like DINO (which expects a [CLS] token). Adapting VTP to CNNs would require non-trivial engineering: deciding which layer's features to use for contrastive alignment, how to handle multi-crop augmentation with variable-resolution feature maps, and how to implement the EMA teacher for MIM on a CNN backbone. The paper provides no guidance on these design choices.

Similarly, the downstream generative model is always LightningDiT—a specific diffusion transformer architecture. Different generative paradigms (autoregressive, masked, flow-based) impose different requirements on the latent space. An autoregressive model, for instance, typically requires a discrete tokenizer (via VQ-VAE) rather than a continuous VAE, and the semantic structure that VTP builds might interact differently with autoregressive sequence modeling than with diffusion denoising. The paper's text-to-image experiments (Figure 8) use DiT-XL, extending the evaluation to text conditioning but not to a different generative architecture.

What evidence exists in the paper. The paper's Table 1 compares ViT-L against a convolutional LDM autoencoder for reconstruction-only training, showing comparable performance. But this comparison is for the baseline (reconstruction-only), not for VTP. There is no VTP-trained CNN tokenizer to compare against VTP-trained ViT tokenizers. The paper acknowledges the ViT choice as a deliberate architectural decision (Section 3.1) but does not discuss whether the multi-objective training paradigm is architecture-agnostic or ViT-specific.

The downstream model diversity is limited to LightningDiT at two sizes (B, XL) for class-conditional generation and DiT-XL for text-to-image. The paper does not evaluate VTP latents with, for example, a SiT (Ma et al., 2024), a U-ViT, a DiT with different patch sizes or conditioning mechanisms, or an autoregressive latent model. The paper's Table 2 compares against SiT, DiT, and other methods, but these are baselines using their own tokenizers, not VTP latents tested with alternative generative architectures.

Mitigation status. Not addressed. The paper treats the ViT-LightningDiT combination as the evaluation pipeline without discussing its specificity. The text-to-image results on LAION (Section 5) demonstrate that VTP's scaling property generalizes from class-conditional to text-conditional generation, which is a meaningful extension. But the architectural generality—to other tokenizer backbones and other generative model families—remains open. The paper's claim that VTP is a "unified visual tokenizer pre-training framework" (abstract) overstates the evidence for architectural generality.

The ~14× Larger Pretraining Baseline Is Not Compute-Optimal; the Case for Test-Time Compute Over Pretraining Is Overstated

The assumption or constraint. While this paper does not make a direct training-inference tradeoff claim in the style of a Chinchilla-style analysis, the conceptual framing of the "pre-training scaling problem" implicitly positions VTP as a solution that makes tokenizer pre-training compute translate to downstream generation gains—in contrast to reconstruction-only training, where "pouring extensive compute into visual tokenizer pre-training translates poorly to improved performance in generation" (Section 1). The paper demonstrates that VTP's generation performance improves as tokenizer pre-training compute scales, while the autoencoder baseline's generation performance stagnates or degrades.

However, the autoencoder baseline in the scaling experiments is a ViT architecture trained with L1 + perceptual loss reconstruction objectives followed by GAN fine-tuning. The paper does not establish that this is the best possible reconstruction-only tokenizer that could be built at each scale. The community has explored many tricks for improving VAE-based tokenizers beyond simple reconstruction losses: KL regularization tuning, perceptual loss variants, adversarial training at various strengths, latent dimension engineering, learned quantization, multi-scale architectures, latent-space data augmentation during DiT training, and more. The paper's autoencoder baseline, while reasonable, is a single point in a large design space. It is possible that a more carefully optimized reconstruction-only tokenizer would show better (or even positive) scaling behavior.

More specifically, the paper's autoencoder baseline uses λ_rec = 1.0 (pure reconstruction) while VTP uses λ_rec = 0.1 with λ_ssl = λ_clip = 1.0. The effective training signal differs not only in kind (reconstruction vs. reconstruction+SSL+CLIP) but in the relative weight of the reconstruction loss. The paper does not test whether simply reducing the reconstruction loss weight in a pure autoencoder would improve generation scaling—i.e., whether the benefit comes from adding semantic objectives or from de-emphasizing reconstruction. If a reconstruction-only tokenizer with λ_rec = 0.1 (and no other losses) showed improved generation scaling, the story would be more about loss weighting than about semantic understanding. This ablation is not performed.

The consequence. The paper's central narrative—"reconstruction-only training cannot scale for generation, VTP can"—relies on the autoencoder baseline being a fair representation of what reconstruction-only training can achieve. If a better reconstruction-only baseline exists (e.g., with different loss weights, different perceptual losses, different decoder architectures, or better GAN training stabilization), the contrast between VTP and reconstruction-only scaling might be less dramatic. The paper's qualitative conclusion—that semantic objectives help—would likely still hold, but the quantitative magnitude of the benefit (e.g., the 4× efficiency claim, the 65.8% FID improvement for 10× compute) might be overstated relative to a stronger baseline.

The paper's comparison with prior work partially mitigates this concern: VA-VAE, REPA-E, and RAE all show substantially worse generation than VTP at comparable DiT training budgets (Table 2). These are independently developed methods with their own design choices, not paper-internal baselines that could be deliberately weakened. However, none of these methods is a pure reconstruction-only tokenizer with an optimized training recipe—they all incorporate some form of representation alignment or distillation. The strongest evidence that reconstruction-only training inherently cannot scale comes from the paper's own internal scaling experiments (Figures 4, 7), which use a single training recipe.

What evidence exists in the paper. The autoencoder scaling experiments (Figures 4, 5, 7) all use the same training recipe described in Section 3.2 and Section 4.1. The paper reports that the ViT autoencoder with this recipe achieves 31.28 rPSNR and 53.51 gFID (Table 1), which is competitive with the convolutional LDM autoencoder (30.63 rPSNR, 59.53 gFID). This suggests the ViT autoencoder is not a strawman—it matches or exceeds the standard convolutional baseline. But this comparison is at one compute scale; it does not prove that the autoencoder's flat scaling curve is inherent to reconstruction-only training rather than specific to the chosen training recipe.

Mitigation status. Partially addressed through comparison with external baselines (VA-VAE, REPA-E, RAE) that show worse generation despite incorporating semantic objectives. The consistency of the autoencoder's negative scaling across compute (Figure 4), parameters (Figure 7b), and data (Figure 7a) makes it less likely that the result is an artifact of a specific hyperparameter choice—if the autoencoder's flat scaling were due to, say, suboptimal learning rate, one might expect different scaling behavior across data regimes. The paper does not discuss the possibility of a stronger reconstruction-only baseline or perform the λ_rec ablation that would distinguish loss-weighting effects from semantic-objective effects.

Hard Problems (Images That Are Inherently Difficult to Reconstruct or Require Rare Semantic Concepts) Are Not Analyzed Separately

The assumption or constraint. VTP is evaluated on ImageNet 256×256 class-conditional generation and LAION text-to-image generation, and performance is reported as aggregate metrics (gFID, linear probing accuracy) averaged over the entire validation set. The paper does not break down performance by image difficulty, semantic complexity, or reconstruction challenge. This matters because VTP's central hypothesis—that semantic understanding drives generation—implies that the benefit of VTP over reconstruction-only training should be largest for images where semantic structure is important for generation (e.g., images with multiple objects in complex spatial relationships, scenes requiring compositional understanding) and smallest for images where low-level texture and color statistics dominate (e.g., uniform textures, simple patterns, images where "what" is less important than "how it looks").

The paper's diagnostic experiments (Figures 2, 5, 6) show aggregate correlations between understanding and generation, but do not test whether this correlation holds at the per-image level or varies across image categories. It is possible that VTP improves generation primarily on semantically complex images and provides little benefit (or even a degradation, due to the reduced reconstruction weight) on simple or texture-dominated images. If so, the aggregate gFID improvement masks heterogeneity that matters for deployment—a practitioner might prefer a reconstruction-only tokenizer for a domain dominated by textures (e.g., material synthesis, satellite imagery) and VTP for a domain requiring compositionality (e.g., scene generation, text-to-image with complex prompts).

The consequence. The paper's claim that VTP is a general-purpose tokenizer pre-training framework implies that it improves generation across all image types. Without a difficulty-stratified analysis, this claim is unverified. A deployment where the image distribution skews toward the "easy" or "texture-heavy" end of the spectrum might see minimal benefit from VTP over a well-tuned autoencoder, while incurring the additional cost of multi-objective training (larger batches for CLIP, multi-crop augmentation for SSL). The paper provides no guidance on when VTP is worth the additional complexity versus when a simpler reconstruction-only tokenizer suffices.

The text-to-image experiments (Figure 8) provide some indirect evidence: CLIP loss specifically improves text rendering, which is a semantically demanding task (the model must understand that the word "cat" should render as the visual string "cat" in the output). This suggests VTP's benefits are indeed targeted to tasks requiring semantic understanding. But this is a single task type—text rendering—rather than a systematic difficulty stratification across the image distribution.

What evidence exists in the paper. The paper does not perform any difficulty-based or category-based analysis of generation quality. All reported metrics are aggregate across the full ImageNet validation set or the LAION test set. The linear probing accuracy of 85.7% on ImageNet (Table 2) indicates strong average semantic understanding, but does not reveal whether certain classes or image types are poorly represented. The reconstruction comparison with RAE (Figure 9a) shows qualitative examples favoring VTP's detail preservation, but these are selected examples, not a systematic analysis.

Mitigation status. Not addressed. The paper's evaluation protocol is standard for generative model papers (aggregate FID on ImageNet), and difficulty stratification is not expected by convention. However, given the paper's emphasis on understanding as the mediator of generation quality, a category-level or difficulty-stratified analysis would substantially strengthen the causal claims. Future work could bin ImageNet classes by semantic complexity (e.g., number of objects per image, intra-class visual diversity) and test whether VTP's advantage over autoencoders correlates with semantic complexity at the class level.

The Sequential Nature of Multi-Objective Training Increases Wall-Clock Time per Pre-Training Step; Latency and Hardware Requirements Are Not Discussed

The assumption or constraint. VTP jointly trains three objectives (reconstruction, SSL, CLIP) in each pre-training step, each with different batch sizes, augmentation pipelines, and forward-pass requirements. The SSL component requires multi-crop augmentation (generating multiple views per image) and an EMA teacher forward pass; the CLIP component requires a text encoder forward pass on 16K captions per step; the reconstruction component requires a decoder forward pass. The paper's batch sampling strategy (Section 3.6) mitigates some of this by using different batch sizes per objective, but the total per-step computation is substantially higher than reconstruction-only training at the same base batch size.

The paper reports FLOPs as the unit of compute for scaling experiments (Figures 5, 6, 7), which is appropriate for measuring total computational work. However, FLOPs abstract away two practical concerns: wall-clock time and hardware requirements. The multi-objective training may have lower hardware utilization (due to the heterogeneous computation within each step—some GPUs processing CLIP batches, some processing SSL crops, some processing reconstruction), may require more GPU memory (the EMA teacher, text encoder, and decoder all occupy memory simultaneously), and may be more difficult to parallelize across nodes (the CLIP loss requires a large batch for effective contrastive learning, which may exceed the memory of a single node, requiring inter-node communication for the contrastive loss computation).

The consequence. A practitioner considering VTP for tokenizer pre-training needs to know not just "how many total FLOPs to achieve a given gFID" but also "how many GPU-hours on what hardware configuration." The paper provides FLOPs numbers but not wall-clock time or hardware configuration details (GPU type, number of GPUs, memory per GPU, inter-node communication strategy). This makes it difficult to estimate the real-world cost of adopting VTP versus a simpler reconstruction-only approach. If VTP's per-step wall-clock time is 5× longer than an autoencoder at the same nominal FLOPs (due to lower GPU utilization or communication overhead), the effective cost of pre-training is higher than the FLOPs comparison suggests.

Additionally, the large batch size required for CLIP (16K image-text pairs) may be infeasible on smaller hardware setups. The paper's experiments likely used a large GPU cluster—the 277M-sample dataset and 1.1B-sample training runs imply industrial-scale infrastructure. A research lab or startup with a more modest GPU budget might not be able to run VTP at the batch sizes needed for effective contrastive learning, and the paper does not test whether VTP retains its scaling benefits at smaller CLIP batch sizes (e.g., 4K or 8K).

What evidence exists in the paper. The paper specifies the batch sizes (B_clip = 16K, B_ssl = 4K, B_rec = 2K) in Section 4.1, and reports training FLOPs are reported on the x-axes of Figures 5–7. No information about GPU type, count, memory, training throughput (samples/second), or total wall-clock time is provided. The paper does not ablate CLIP batch size to determine whether the 16K requirement is necessary or whether smaller batches would still yield scaling benefits. The text encoder architecture (12-layer, 768-dim) is specified, but the computational cost of the text encoder forward pass is not broken out from the total FLOPs.

Mitigation status. Not addressed. The paper treats FLOPs as the sole cost metric, which is standard in scaling-law research (Hoffmann et al., 2022; Kaplan et al., 2020) but insufficient for practical deployment planning. The paper's code release (linked in the abstract) may provide implementation details that allow practitioners to estimate wall-clock costs, but the paper itself does not discuss latency or hardware requirements. A throughput comparison (samples/second for VTP vs. reconstruction-only training at matched FLOPs) would substantially improve the practical utility of the scaling analysis.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper fundamentally reframes how the field should think about visual tokenizer pre-training for generative models. Before VTP, the dominant paradigm treated the tokenizer as a compression engine: its job was to learn a latent space that minimized reconstruction error, with the implicit assumption that better reconstruction would translate to better generation. This assumption was so deeply embedded that the standard recipe—train a VAE on L1 + perceptual + GAN losses, measure PSNR/rFID, deploy—had gone largely unquestioned despite accumulating evidence of a reconstruction-generation trade-off (Esser et al., 2024; Yao et al., 2025).

VTP demolishes this assumption and replaces it with a representation-learning paradigm. The core conceptual shift is to treat the visual tokenizer not as a compressor but as a representation learner whose downstream task is to provide a useful latent space for generation, not to faithfully reproduce pixels. This is more than a methodological tweak—it is a change in the evaluation criterion, the training objective, and the scaling narrative simultaneously.

The magnitude of the shift is substantial but bounded. This is not a paradigm shift on the scale of the original LDM proposal (Rombach et al., 2022), which created an entirely new architecture family. Rather, it is a reframing within the existing LDM paradigm that changes what practitioners optimize for and how they allocate compute. The two-stage architecture (tokenizer → diffusion model) remains intact. What changes is that the first stage is now understood as a representation-learning problem rather than a compression problem, which opens up an entirely different design space—self-supervised objectives, contrastive learning, multi-modal alignment—that the compression framing never suggested.

The field-level implications are threefold:

The pre-training scaling problem is now a named, diagnosable failure mode. By explicitly identifying and characterizing the phenomenon—reconstruction improves while generation degrades as compute scales—the paper creates a clear benchmark for future tokenizer work. Any new tokenizer pre-training method must now demonstrate not just fixed-budget improvements but positive scaling behavior: does generation improve as you invest more compute, parameters, and data in the tokenizer? A method that only shows gains at one scale but degrades or saturates at larger scales is now understood to be on the wrong scaling curve. This raises the methodological bar for the field from "our tokenizer gets better FID at one compute point" to "our tokenizer exhibits a scaling law."
Representation learning becomes a first-class design dimension for tokenizers. Prior work that injected semantic objectives (VA-VAE, REPA-E, RAE) did so through distillation from frozen, separately trained models. VTP's key move is to train the representation jointly with reconstruction, making the tokenizer itself the representation learner. This eliminates the teacher bottleneck (demonstrated starkly in Table 3, where RAE degrades at larger scales because the fixed DINOv2 teacher cannot provide improved targets) and means the latent space can scale with model capacity. The implication for future research is clear: distillation-based approaches are a dead end for scalable tokenizers; joint representation learning is the path forward.
Understanding and generation are unified under one optimization, not in tension. The paper provides strong evidence that semantic understanding is not merely correlated with generation quality but is the causal driver of it. Figure 2 shows a positive correlation; Figures 5 and 6 show that adding representation-learning objectives shifts the scaling curve from negative to positive; Figure 8b shows that CLIP loss specifically improves semantically demanding tasks like text rendering. This unification means that the field's long-standing narrative of a "trade-off" between reconstruction and generation is misdiagnosed—the real tension is between low-level pixel fidelity and high-level semantic structure, and the solution is to prioritize the latter. A tokenizer that achieves 0.36 rFID and 78.2% zero-shot accuracy simultaneously (Table 2) is not balancing two competing objectives; it is using semantic structure to improve reconstruction implicitly.

Which research directions become more attractive:

Joint representation-learning objectives for tokenizers now have a clear scaling justification. Researchers can explore new SSL objectives (e.g., DINOv3, IJEPA, data2vec), new contrastive formulations (e.g., SigLIP, BLIP-style captioning+contrastive), and new multi-modal alignments (image-text, image-audio, image-video) as tokenizer pre-training strategies, confident that the integration framework can scale.
Scaling studies on tokenizer pre-training become an important subfield. The paper's demonstration that tokenizer scaling matters—and that the scaling behavior differs qualitatively between objectives—means that tokenizer pre-training is no longer a fixed-cost upfront step but a resource to be optimized. Researchers can now ask: what is the compute-optimal allocation between tokenizer pre-training and diffusion model training?
Understanding as a proxy for generation quality opens the door to lightweight tokenizer evaluation without full DiT training. If linear probing accuracy on a validation set reliably predicts downstream gFID, practitioners can iterate on tokenizer design much faster.

Which directions become less attractive:

Distillation from frozen representation models is now known to hit a ceiling (Table 3). Investing further in this approach—e.g., better alignment losses, multi-teacher distillation, adaptive distillation weights—is unlikely to overcome the fundamental bottleneck that the teacher's capacity is fixed and not optimized for generation.
Reconstruction-only improvements (better perceptual losses, more sophisticated GAN training, multi-scale decoder designs) are now understood to be operating on the wrong scaling curve. These may improve reconstruction metrics but will not translate to generation gains beyond the saturation point the paper demonstrates in Figure 4. Research effort is better directed at representation-learning integration.
Architecture tweaks as primary contributions for tokenizer design lose salience. The paper shows that a standard ViT architecture, when trained with the right objectives, dramatically outperforms more complex architectural designs trained with reconstruction alone. The objective matters more than the architecture.

Reconciling prior contradictions. The paper explains why prior work found mixed results on the reconstruction-generation relationship. Methods that improved reconstruction at the expense of generation (the "trade-off") were operating in the reconstruction-dominated regime where additional compute overfits to pixel statistics. Methods that improved generation by adding semantic objectives (VA-VAE, REPA-E) were implicitly operating in the representation-learning regime but without the scaling benefits that joint training provides. VTP's framework unifies these observations: the key variable is the relative emphasis on reconstruction vs. representation learning, and as the representation-learning signal dominates (λ_rec = 0.1 vs. λ_ssl = λ_clip = 1.0), the scaling behavior shifts from negative to positive. The prior contradictions reflect different points on this spectrum, not genuinely conflicting findings.

One cautionary note on the magnitude of the shift. The paper's evidence is entirely within the ViT-LightningDiT ecosystem on ImageNet and LAION. The paradigm shift claim—that tokenizers should be trained as representation learners—is well-supported for this setting, but its generality to CNNs, to other generative architectures, to video, to other modalities, and to much larger scales is unproven. The paper opens a new research direction; it does not close the book on how tokenizers should be trained.

Follow-Up Research This Work Enables

Quantitative scaling laws for VTP pre-training across 2–3 orders of magnitude. The paper demonstrates qualitative positive scaling over ~1 order of magnitude in compute (Figures 5, 6). The natural next step is a systematic scaling law study: train VTP tokenizers at ViT-S, B, L, H scales with pre-training FLOPs spanning 10^10 to 10^12 or 10^13, fit a power-law or broken-power-law function gFID = a × C^(-α) + b where C is tokenizer pre-training FLOPs, and report the exponent α and irreducible loss b. This would tell practitioners whether VTP scaling is truly power-law (predictable marginal returns) or sigmoidal (saturating at some threshold), and would inform cost-benefit analysis for tokenizer pre-training investment. A strong study would also jointly model the scaling of tokenizer pre-training compute and DiT training compute, analogous to Chinchilla's joint model of parameters and data, to determine the compute-optimal allocation between the two stages under a total FLOPs budget.

VTP for convolutional VAE architectures: is the scaling behavior architecture-agnostic? Every VTP experiment uses a ViT encoder-decoder. The dominant deployed tokenizers (Stable Diffusion, SDXL, FLUX) use convolutional VAEs. A critical stress-test is to reimplement VTP's multi-objective training on a CNN VAE: attach SSL heads to the CNN's feature hierarchy (e.g., global average pooling for DINO, feature pyramid for MIM) and a contrastive head to a pooled embedding. Train at multiple scales and measure whether the scaling curve shifts from negative (reconstruction-only CNN) to positive (VTP CNN). If the scaling property transfers, VTP is a general framework. If not—if CNNs cannot effectively integrate representation-learning objectives without architectural modification—then the paradigm shift is ViT-specific and the field needs CNN-compatible representation-learning strategies. This experiment would also reveal whether the paper's ViT choice is a convenience or a necessity.

Difficulty-stratified analysis of VTP's generation benefits. The paper reports aggregate gFID on ImageNet but never asks: for which images does VTP help most? A natural extension is to bin ImageNet classes by semantic complexity—e.g., number of objects per image, intra-class visual diversity, presence of text or fine-grained structure—and compute per-bin gFID for VTP vs. autoencoder. The paper's hypothesis that "understanding drives generation" predicts that VTP's benefit should be largest for semantically complex images (multiple objects, compositional scenes) and smallest for texture-dominated classes (e.g., "volcano," "seashore") where pixel-level reconstruction fidelity matters more. A null result (uniform benefit across all bins) would weaken the causal claim about understanding; a strong interaction would strengthen it and provide practical guidance about when VTP is worth the additional training cost.

Does VTP eventually saturate, or does the scaling continue indefinitely? Figure 7b already shows diminishing returns: VTP-S → VTP-B → VTP-L yields gFID improvements of ~3.45 → ~1.71. The critical question is whether the next jump (ViT-H) would yield another ~0.8 improvement or whether performance has effectively saturated. Train VTP-H and VTP-G (if feasible) with matched pre-training data and measure downstream gFID under the same fixed DiT protocol. If gFID continues improving, fit a scaling law and extrapolate to practical upper bounds. If it saturates, determine whether the saturation is due to the tokenizer bottleneck (the latent space can only capture so much semantic structure) or the DiT bottleneck (the downstream model cannot exploit further improvements in the latent space). The latter case would motivate scaling the DiT alongside the tokenizer, or exploring end-to-end fine-tuning, rather than treating the tokenizer as a frozen upstream component.

Lightweight tokenizer quality predictors for checkpoint selection without full DiT training. The paper's evaluation requires training a DiT from scratch for each tokenizer checkpoint—prohibitively expensive for iterative development. A high-impact methodological contribution would be to train a small predictor model that takes tokenizer properties as input (linear probing accuracy on a validation set, rFID, latent-space metrics like cluster purity or nearest-neighbor consistency, and tokenizer architecture hyperparameters) and predicts downstream gFID. Train the predictor on the scaling data the paper has already generated (each data point = one tokenizer checkpoint + its measured gFID from a full DiT training run). Evaluate whether the predictor can rank unseen VTP checkpoints by generation quality with high rank correlation. If successful, this predictor would decouple tokenizer development from expensive DiT training, enabling much faster iteration. If not, it would reveal that the relationship between latent-space properties and generation quality is more complex than aggregate metrics capture, and that full downstream evaluation remains necessary.

VTP for video tokenizers and other modalities. The paper's framework is evaluated exclusively on images, but the principles—joint reconstruction and representation learning for a latent space that serves generation—apply directly to video tokenizers, which face an even more severe compression challenge and where semantic understanding (motion, object permanence, temporal coherence) is arguably more important than pixel-level reconstruction. A natural extension is to adapt VTP to video: replace image SSL objectives with video SSL (e.g., time-contrastive learning, masked frame prediction), replace image-text contrastive with video-text contrastive, and train on a large video dataset. Measure whether video VTP exhibits positive scaling for downstream video generation (e.g., on UCF-101 or Kinetics class-conditional generation, or text-to-video on WebVid). The video setting is a strong stress-test because the computational cost is higher, making the scaling question more practically urgent—if VTP can make video tokenizer pre-training more efficient, the practical impact is large. Similarly, adapting VTP to audio (mel-spectrogram tokenizers for audio generation) or 3D (point cloud or neural radiance field tokenizers) would test the generality of the representation-learning paradigm beyond 2D images.

Practical Applications and Downstream Use Cases

Cost-efficient batch image generation at scale. For organizations running large-scale image generation pipelines—stock photo synthesis, e-commerce product visualization, game asset generation—VTP offers a clear cost reduction path. The paper shows that a VTP-L tokenizer enables a DiT-B to achieve 2.03 gFID in 80 epochs without guidance, compared to 4.29 for VA-VAE (Table 2). At equal generation quality, VTP requires fewer DiT training epochs—a direct reduction in the most expensive part of the LDM pipeline. The 65.8% FID improvement from 10× tokenizer pre-training compute (Section 1) means that a modest investment in tokenizer pre-training (which is done once and amortized across all downstream models) yields disproportionate savings in repeated DiT training runs. For a team training hundreds of DiT variants (different resolutions, conditioning types, architectures), the one-time cost of pre-training a VTP tokenizer on 277M images pays back rapidly.

On-device and edge deployment with smaller diffusion models. The paper demonstrates that VTP latents enable faster DiT convergence: at 80 epochs, VTP achieves generation quality that would take substantially longer with autoencoder latents. This fast convergence is particularly valuable for fine-tuning diffusion models under tight compute budgets, such as personalizing a model on a user's photos on-device. The VTP tokenizer, once pre-trained, is frozen and can be deployed as a shared encoder-decoder on the edge. A small DiT fine-tuned on VTP latents for a few epochs could achieve quality that would require many more epochs (and more energy, draining battery) with a standard autoencoder. The f16d64 compression (16× spatial reduction, 64-dim latents) is already aggressive enough for efficient on-device inference; VTP's semantic structuring may further improve sample efficiency during personalization.

Data generation for self-improving vision models. The paper's finding that VTP achieves 78.2% zero-shot and 85.7% linear probing accuracy while enabling state-of-the-art generation (Table 2) makes VTP an attractive tokenizer for unified understanding-and-generation models. A single VTP encoder can serve both as a feature extractor for discriminative tasks (classification, retrieval, open-vocabulary detection) and as a tokenizer for generative tasks (image synthesis, inpainting, super-resolution). This unification is practically valuable for building vision systems that can both recognize and generate content without maintaining separate backbones. For instance, a robotic perception system could use VTP's encoder to recognize objects and scenes, then use the same latent space to generate hypothetical future states for planning, all within one model family. The 0.36 rFID (Table 2) means the reconstructions are faithful enough for human inspection.

Text-to-image generation with improved text rendering. The targeted benefit shown in Figure 8b—CLIP loss specifically improves text rendering in text-to-image generation—directly addresses a known weakness of diffusion models. Many deployed T2I systems struggle to generate readable text within images (logos, signage, captions). VTP pre-training with CLIP, by aligning the visual latent space with language representations, makes the tokenizer's latents inherently more compatible with text conditioning. A production T2I system that replaces its convolutional VAE with a VTP tokenizer pre-trained with CLIP+SSL+AE would likely see improved text rendering quality without any change to the downstream diffusion model architecture or training recipe. The paper's LAION experiments (Figure 8a) provide direct evidence that this benefit persists across different pre-training compute scales and DiT training budgets, suggesting it is a robust property rather than a small-scale artifact.