ArXiv: 2605.00503

🎯 Pitch

Image generation models typically keep their tokenizer frozen after pre-training, but this work shows that jointly training the tokenizer with the autoregressive generator slashes FID to 1.48 on ImageNet 256Γ—256β€”a new state‐of‐the‐art without guidance. The key is an extra reconstruction loss on the model’s own predictions, which prevents the latent codes from collapsing and lets the tokenizer learn to produce sequences that are genuinely easier to generate.

1. Executive Summary

This paper introduces EOSTok, an end-to-end training pipeline that jointly optimizes a 1D vision tokenizer and an autoregressive generative model for image generation, departing from the conventional two-stage paradigm where tokenizers are trained only for reconstruction. The approach addresses a fundamental misalignment β€” next-token-prediction loss on discrete tokens fails to reflect pixel-space generation quality β€” by designing an Autoregressive Prediction Reconstruction (APR) loss that decodes teacher-forcing predictions into pixels (providing end-to-end generative feedback) and by investigating implicit semantic alignment with vision foundation models (aligning hidden patch embeddings, not latent tokens, to DINOv2 representations to avoid leaking 2D spatial priors). On ImageNet 256Γ—256 generation, the EOSTok-H model with 644M parameters achieves a state-of-the-art FID of 1.48 without guidance, with the largest variant outperforming a ~14Γ— larger pretrained baseline on this metric, while the APR loss resolves latent codebook collapse and the implicit alignment strategy improves generation quality substantially over direct alignment approaches β€” establishing that 1D autoregressive image generation can match or exceed 2D methods only when the tokenizer receives end-to-end generative supervision and semantic guidance that respects its sequential structure.

2. Context and Motivation

The Core Problem: Autoregressive Image Generation Demands Better Tokenizers

The fundamental gap this paper addresses is simple to state but deep in its implications: conventional vision tokenizers are trained for reconstruction, not generation, and this mismatch cripples autoregressive image modeling. Autoregressive (AR) models have achieved remarkable success in language modeling β€” GPT-style models predict one token at a time, conditioning each prediction on all previously generated tokens. This paradigm works naturally for text because language is inherently sequential: words follow one another in a linear order, and the unidirectional dependencies of AR modeling align with the structure of the data.

Images, however, are fundamentally two-dimensional. When we apply the same AR paradigm to images, we face an immediate tension: the data has spatial structure that doesn't decompose naturally into a linear sequence. The standard approach has been to use 2D grid-structured tokenizers β€” typically VQ-VAE variants β€” that divide an image into a grid of patches (e.g., 16Γ—16), encode each patch into a discrete token, and then generate these tokens in raster-scan order (left-to-right, top-to-bottom). This produces a token sequence that preserves the spatial layout, but at a cost: tokens that are adjacent in the raster order may be far apart in the image (e.g., the last token of one row and the first token of the next row are adjacent in the sequence but spatially distant), while tokens that are spatially adjacent may be far apart in the sequence. The resulting token sequences contain bidirectional dependencies that are fundamentally misaligned with unidirectional AR factorization β€” the model must predict a token without access to tokens that are spatially above or to the right, even though those tokens contain relevant visual information.

This misalignment is not a minor implementation detail. It means the AR model is being asked to solve a harder problem than necessary, because the tokenizer has produced a representation that doesn't respect the structure the AR model can exploit. Prior work has tried to work around this in two ways: either design new generation paradigms that avoid strict unidirectional AR (masked models, multi-scale prediction), or build 1D tokenizers that compress images into truly sequential representations. The second approach is the most direct solution β€” if the tokenizer produces a sequence where the sequential order reflects meaningful visual structure, vanilla AR modeling becomes natural. But existing 1D tokenizers are trained only for reconstruction quality, not for how predictable their tokens are under AR modeling. This is the gap EOSTok fills.

Why This Matters: The Stakes for Generative Modeling

The importance of this problem extends across multiple dimensions:

Practical: closing the quality gap with diffusion models. Diffusion models currently dominate image generation benchmarks, but AR models offer several practical advantages: faster sampling (since they avoid iterative denoising steps and can use KV-caching), more straightforward scaling (following well-understood LLM scaling recipes), and natural compatibility with the unified multimodal architectures that the field is increasingly pursuing (models that process and generate both text and images). If AR models could match diffusion quality, they would become the default architecture for multimodal systems. The obstacle has been the tokenizer β€” as the MAGVIT-v2 paper famously put it, "tokenizer is key to visual generation" [46]. EOSTok's result of 1.48 FID without guidance (matching or exceeding the best diffusion models) demonstrates that the AR approach is genuinely competitive when the tokenizer is properly designed.

Theoretical: the reconstruction-generation dilemma. Training tokenizers for reconstruction and using them for generation creates a fundamental tension. A tokenizer optimized purely for pixel-level fidelity will preserve fine details that are perceptually important but statistically unpredictable β€” high-frequency textures, exact boundary positions, subtle color variations. The AR model must then learn to predict these details, which may be essentially random given the available context. Conversely, a tokenizer optimized purely for predictability would discard this variation, producing sequences that are easy to model but result in blurry or unrealistic reconstructions. This is the reconstruction-generation dilemma that the paper identifies and addresses: the two objectives are in tension, and the conventional two-stage approach (train tokenizer for reconstruction, then freeze it and train AR model on its tokens) gives all the weight to reconstruction, leaving the AR model to cope with a token space that wasn't designed for it.

Practical: enabling smaller, faster models. The paper's demonstration that EOSTok-H (644M parameters, generating 256 tokens) achieves state-of-the-art results has direct implications for deployment. As shown in Table 8, EOSTok-H requires approximately 552 GFLOPs per generated image versus roughly 59,900 GFLOPs for DiT-XL/2 with 250 diffusion steps β€” a 20-100Γ— speedup depending on the diffusion sampler. This isn't just about convenience; it makes high-quality image generation feasible on consumer hardware and in latency-sensitive applications. The tokenizer's design β€” compressing images into only 256 sequential tokens β€” is central to this efficiency.

Prior Approaches and Where They Fall Short

The paper positions itself against three broad classes of existing work, each of which has identifiable limitations:

2D discrete tokenizers with AR models (VQGAN, RQ-Transformer, LlamaGen). These tokenizers produce grid-aligned latent codes and generate them in raster-scan order. The core limitation is the mismatch between bidirectional spatial dependencies and unidirectional AR factorization, which the paper describes as "suboptimal" (Section 2). The AR model must predict each token without seeing its spatial neighbors, forcing it to implicitly model 2D structure through a 1D causal bottleneck. Empirically, models in this class (e.g., LlamaGen-XL at 2.62 FID with guidance, VQGAN at 15.78 FID) underperform comparable diffusion approaches. More critically, the tokenizer training procedure is entirely decoupled from the generative task β€” the tokenizer learns to optimize reconstruction metrics (L2, perceptual loss, GAN loss) with no signal about whether its tokens are predictable.

Modified generation paradigms (MaskGIT, MAR, VAR). To avoid the unidirectional limitation, these approaches change the generation procedure rather than the tokenizer. MaskGIT [4] uses bidirectional attention and generates tokens in multiple passes (predicting a subset of tokens, then refining). MAR [22] extends this to continuous latents with diffusion-style generation. VAR [37] predicts tokens at multiple scales (coarse-to-fine) rather than raster order. While these methods achieve strong results, they abandon the simplicity of vanilla AR modeling β€” they require specialized architectures, training procedures, and sampling strategies that don't benefit from the same scaling principles and infrastructure as language models. As the paper notes in Section 2, these are workarounds rather than solutions to the underlying tokenizer problem.

Existing 1D tokenizers (TiTok, FlexTok, Semanticist). These are the most direct predecessors to EOSTok. TiTok [48] introduced the core architectural idea: use learnable query tokens in a ViT encoder to compress 2D images into 1D latent sequences, eliminating the 2D spatial prior. However, TiTok was designed for aggressive compression (32 tokens), trading reconstruction quality for ease of generation. The paper argues that "the removal of 2D structural dependency paves the way for designing visual tokens that naturally support vanilla autoregressive modeling, which does not necessarily rely on aggressive compressions" (Section 1). In other words, TiTok's insight about removing spatial structure is correct, but its focus on extreme compression is unnecessary and limiting β€” a 1D tokenizer with more tokens (256 vs. 32) can achieve good reconstruction and generation quality if trained properly.

FlexTok [1] and Semanticist [41] added nested dropout to 1D tokenizers, enforcing that earlier tokens carry more important information. But like TiTok, they train tokenizers only for reconstruction in a first stage, then freeze them for AR training in a second stage. The tokenizer never receives feedback about whether its tokens are easy or hard to predict sequentially. The paper's key insight is that this feedback is essential β€” as demonstrated in Table 4, EOSTok's end-to-end training produces token sequences that are significantly more AR-friendly (gFID of 4.10 vs. 7.81 for random ordering), showing that the tokenizer learns to structure its latent space to facilitate sequential prediction.

Vision foundation model alignment (REPA, VA-VAE). A separate line of work has shown that aligning the internal representations of generative models to pre-trained vision foundation models (VFMs) like DINOv2 significantly improves generation quality. REPA [49] aligns intermediate diffusion transformer features to DINOv2 patch embeddings. VA-VAE [44] directly aligns the VAE latent space to DINOv2 features using interpolation to match dimensions. However, these approaches were developed for 2D-structured latent spaces. When applied to 1D tokenizers, the paper identifies a critical problem: "directly aligning the 1D sequential latent space to 2D VFM representations forces it to degenerate to a raster-ordered, patch-aligned sequence, leading to suboptimal performance" (Section 1). This is empirically confirmed in Table 2 (row "(a) Direct alignment"), where direct alignment improves reconstruction but worsens generation quality compared to the VFM-free baseline. The VFM's 2D structure leaks a spatial prior into the tokenizer, undermining the very property (freedom from 2D constraints) that makes 1D tokenizers valuable for AR modeling.

The Missing Pieces: Why Prior Work Doesn't Solve the Problem

Synthesizing these limitations, the paper identifies three specific gaps that prior work leaves unaddressed:

  1. No end-to-end generative supervision for tokenizers. Every prior tokenizer is trained with reconstruction objectives only. The AR model is trained separately on the frozen tokenizer's outputs. This means the tokenizer never learns which latent representations make generation easier or harder. The paper's Vanilla E2E experiment (Table 1) shows that simply adding next-token-prediction loss to tokenizer training is harmful β€” it causes codebook collapse as the tokenizer learns to use a tiny subset of tokens to make prediction trivially easy (30.2% accuracy vs. 11.8% baseline, but code usage drops from 99.8% to 51.8%). Something more sophisticated is needed: a way to provide generative supervision that doesn't incentivize degenerate solutions.

  2. No mechanism to bridge NTP loss and pixel-space quality. The next-token-prediction loss operates in a discrete token space that changes during training. A lower NTP loss doesn't guarantee better generated images, because the loss measures token-level accuracy while the end goal is perceptual quality. The paper shows this directly in Figure 5b: separate (two-stage) training achieves lower NTP loss than end-to-end training, but worse gFID. The gap between these two metrics β€” token prediction accuracy vs. image quality β€” is not acknowledged or addressed in prior tokenizer design.

  3. No principled way to inject semantic knowledge into 1D tokenizers. Vision foundation models encode rich semantic information (object categories, spatial relationships, visual concepts) that could guide the tokenizer to produce more meaningful latent representations. But prior alignment methods assume a 2D spatial correspondence between the tokenizer's latent space and the VFM's patch-level features. For 1D tokenizers, whose latent sequences have no fixed spatial mapping, this assumption breaks. The field lacked a method to distill VFM semantics into a 1D tokenizer without imposing the 2D structure that 1D tokenizers were designed to escape.

How EOSTok Positions Itself

EOSTok addresses these gaps through a unified framework with three innovations that work together:

End-to-end joint training (bridging gap 1 and 2). Rather than training tokenizer and AR model separately, EOSTok trains them simultaneously from scratch. The key enabling technique is the Autoregressive Prediction Reconstruction (APR) loss: take the teacher-forced predictions of the AR model during training, decode them directly to pixels using the tokenizer's decoder, and compute a reconstruction loss against the ground-truth image. This provides end-to-end generative supervision β€” the tokenizer learns to produce latent codes whose predicted continuations (from the AR model) decode to realistic images. The APR loss prevents the codebook collapse that vanilla joint training causes (Table 1: gFID drops from 8.01 to 3.32 with APR loss) by ensuring that the latent space remains expressive enough to reconstruct images, not just predict tokens.

Implicit semantic alignment (bridging gap 3). Instead of forcing 1D latent tokens to align with 2D VFM features (which leaks spatial structure), EOSTok aligns the hidden patch embeddings of the tokenizer's encoder to VFM representations (Figure 4c). The latent tokens themselves remain free from 2D constraints β€” they only interact with the aligned patch embeddings through the ViT's attention mechanism, indirectly absorbing semantic information. This is a crucial design choice: the paper empirically shows it outperforms direct alignment (Table 2: gFID 3.32 vs. 5.98) and direct substitution (gFID 3.32 vs. 4.89), while also improving AR prediction accuracy (11.9% vs. 8.5% for direct alignment), indicating the latent space becomes more generation-friendly.

Joint scaling of tokenizer and AR model. Unlike prior work where tokenizer and generator sizes are chosen independently, EOSTok scales both components together across four model sizes (S, B, L, H). Figure 5a shows consistent improvement with scale, and the largest model (EOSTok-H) benefits more from larger codebooks (4096 vs. 16384) than smaller models β€” the reconstruction-generation tradeoff in codebook size (Table 6) can be mitigated by scaling the model.

The paper's positioning is not that 1D tokenizers are universally better than 2D ones, nor that end-to-end training replaces all prior approaches. Rather, it argues that if you want to use vanilla AR modeling for images, you need a 1D tokenizer that was trained with generative feedback. The contributions are the specific techniques (APR loss, implicit alignment, joint scaling) that make this work in practice, and the empirical demonstration that this combination achieves state-of-the-art results while maintaining the simplicity and efficiency advantages of AR generation.

3. Technical Approach

3.1 Reader Orientation

EOSTok is a jointly trained system consisting of a 1D vision tokenizer (which compresses images into discrete token sequences) and an autoregressive transformer (which learns to generate new images by predicting those tokens one at a time), trained together from scratch rather than in separate stages. The system solves the fundamental problem that conventional tokenizers are optimized only for reconstruction quality, producing token sequences that are unnecessarily difficult for autoregressive models to predict β€” EOSTok's solution shape is to inject end-to-end generative feedback (via a loss that decodes the AR model's predictions back to pixels) and semantic knowledge from vision foundation models (via hidden-state alignment that avoids leaking 2D spatial structure into the 1D latent space), so that the tokenizer learns to produce latent representations that are simultaneously good for reconstruction and easy for sequential prediction.

3.2 Big-Picture Architecture (Diagram in Words)

The EOSTok system has four major components connected in a training pipeline (Figure 2):

  1. 1D ViT Encoder ($E_\phi$) : Takes image patches concatenated with learnable query tokens, processes them through a causal Vision Transformer, and outputs a 1D latent representation $z$ (plus discarded hidden patch embeddings). This component eliminates 2D spatial structure from the latent space.

  2. Vector Quantizer ($Q$) : Discretizes the continuous latent codes into a finite vocabulary using Index Backpropagation Quantization (IBQ) with a learned codebook. The output is quantized tokens $z_q$ that the AR model can treat as a discrete vocabulary.

  3. Autoregressive Transformer ($G_\theta$) : A decoder-only language model that predicts the next token in the quantized sequence $p(z_n \mid z_{<n})$, trained with standard next-token-prediction loss. During training it operates in teacher-forcing mode; during inference it generates tokens autoregressively.

  4. 1D ViT Decoder ($D_\psi$) : Takes the quantized tokens and learnable mask tokens, and reconstructs the original image through a symmetric ViT architecture. This decoder is used both for the standard reconstruction objective and for the novel APR loss (which decodes the AR model's predictions back to pixels).

Information flow during training: Image patches enter the encoder β†’ encoder produces latent codes $z$ β†’ quantizer discretizes to $z_q$ β†’ AR model predicts $\hat{z}$ (teacher-forced) β†’ decoder reconstructs $x$ from $z_q$ (for reconstruction loss) AND decodes $\hat{z}$ to pixels (for APR loss). Simultaneously, a frozen DINOv2 model extracts VFM features from the input image β†’ these align with hidden patch embeddings in the encoder (implicit alignment) and with hidden decoder features (decoder alignment). Four losses flow back: reconstruction loss, NTP loss, APR loss, and alignment loss β€” all gradients update the encoder, decoder, and AR model jointly.

3.3 Roadmap for the Deep Dive

  • First, the 1D ViT tokenizer architecture (Section 3.1 of the paper), because the tokenizer's causal, query-based design is the foundation that enables both the end-to-end training and the semantic alignment to work without imposing 2D constraints. We need to understand what the encoder outputs and how quantization works before we can understand the losses.

  • Second, the joint training framework and the core challenge that motivates the APR loss (Section 3.2), because the APR loss is the paper's central technical innovation β€” we need to see what goes wrong with naive joint training (codebook collapse) and how APR loss fixes it by bridging the gap between token-level NTP loss and pixel-space generation quality.

  • Third, the semantic representation injection strategies (Section 3.3), because the choice of where to align VFM features (hidden patch embeddings vs. latent tokens, encoder vs. decoder) is the key design decision that determines whether the 1D tokenizer benefits from semantic knowledge without being corrupted by 2D spatial priors. We need to understand all variants attempted and why implicit alignment wins.

3.4 Detailed, Sentence-Based Technical Breakdown

This is a methods and empirical analysis paper whose core idea is that a 1D vision tokenizer and its autoregressive generative model should be trained jointly with end-to-end generative feedback (via pixel-space reconstruction of AR predictions) and semantic guidance from vision foundation models (via alignment on hidden states rather than latent tokens, preserving the 1D tokenizer's freedom from 2D constraints).

1D Vision Transformer Tokenizer Architecture

The tokenizer follows the TiTok-style architecture: a ViT encoder with learnable query tokens that compresses a 2D image into a 1D sequence, followed by vector quantization and a symmetric ViT decoder. The critical difference from standard (2D) ViT tokenizers is that the latent representation has no fixed spatial correspondence to image patches β€” the query tokens are free to capture global, non-local information distributed across the entire image.

Encoder. The input image (256Γ—256 on ImageNet) is first patchified: divided into $P \times P$ patches (patch size 16, producing $16 \times 16 = 256$ patches total for a 256Γ—256 image). Each patch is linearly projected to a hidden dimension $D$ (768 for S/B/L models, 1024 for H). These $N = 256$ patch tokens $x_{\text{patch}} \in \mathbb{R}^{N \times D}$ are flattened into a sequence and concatenated with $L = 256$ learnable query tokens $q \in \mathbb{R}^{L \times D}$, producing an input sequence of length $N + L = 512$. Learnable positional embeddings are added to both the patch tokens and the query tokens.

The concatenated sequence passes through multiple transformer layers (12 for S/B/L encoder, 16 for H encoder). The attention pattern is hybrid: within the 2D patch tokens, attention is bidirectional (each patch can attend to all other patches), and among the 1D query tokens, attention is causal (each query token can only attend to previous query tokens and all patch tokens). Crucially, patch tokens cannot attend to query tokens β€” information flows from patches to queries but not back, forcing the queries to extract visual information from the patches without the patches being influenced by the queries. This causal design on the query side ensures the latent tokens are produced in a fixed order, which matters for the autoregressive model downstream.

The encoder outputs two things: $[h_{\text{Enc}}, z] = E_\phi([x_{\text{patch}}, q])$, where $h_{\text{Enc}} \in \mathbb{R}^{N \times D}$ is the hidden patch embedding (the encoder's representation of each image patch after attending to other patches) and $z \in \mathbb{R}^{L \times D}$ is the query token output. The hidden patch embedding $h_{\text{Enc}}$ is typically discarded (except for the implicit alignment loss, described later). The query tokens $z$ pass through a linear projection layer that maps them from dimension $D$ (768 or 1024) to a smaller latent dimension $d = 64$, producing $z \in \mathbb{R}^{L \times d}$.

Why 1D with causal queries? The paper argues that conventional 2D tokenizers "induce inherently bidirectional dependencies among tokens, which are fundamentally misaligned with the unidirectional factorization required by raster-order autoregressive modeling" (Section 1). By using causal query tokens that extract information from bidirectional patch features, the 1D architecture explicitly breaks the spatial structure: the latent tokens don't correspond to specific image locations, so the AR model doesn't need to implicitly model 2D spatial dependencies through a 1D causal bottleneck. The causal ordering on queries provides a natural sequential structure for AR modeling β€” token $z_n$ can only depend on $z_{<n}$, which matches how the AR model will generate them.

Quantization. The continuous latent codes $z \in \mathbb{R}^{L \times d}$ must be discretized into a finite vocabulary so the AR model can treat them as a classification problem (predicting which token comes next from a discrete set). EOSTok uses Index Backpropagation Quantization (IBQ) from Shi et al. [32], which is a variant of vector quantization that improves codebook utilization and gradient flow.

The quantizer maintains a learned codebook $C \in \mathbb{R}^{K \times D}$ where $K$ is the vocabulary size (4096 by default, with experiments up to 16384). To stabilize training, both the codebook vectors and the latent codes are $\ell_2$-normalized to unit length, and their similarity is computed as cosine similarity scaled by a temperature:

logits=[zTC1βˆ₯zβˆ₯2βˆ₯C1βˆ₯2,…,zTCKβˆ₯zβˆ₯2βˆ₯CKβˆ₯2],p=softmax(logits/Ο„)\text{logits} = \left[\frac{z^T C_1}{\|z\|_2 \|C_1\|_2}, \ldots, \frac{z^T C_K}{\|z\|_2 \|C_K\|_2}\right], \quad p = \text{softmax}(\text{logits} / \tau)

where $\tau = 1.0$ is the temperature, $z \in \mathbb{R}^{1 \times d}$ is a single latent token, and $C_k \in \mathbb{R}^{1 \times d}$ is the $k$-th codebook entry. The logits are a vector of $K$ cosine similarities; the softmax converts them to a probability distribution over the $K$ codes.

p=softmax(logits/Ο„)p = \text{softmax}(\text{logits} / \tau)

where $\text{logits}_k = \frac{z^T C_k}{\|z\|_2 \|C_k\|_2}$ is the cosine similarity between the normalized latent vector and the $k$-th codebook vector, and $\tau = 1.0$ is the temperature.

What it computes: a probability distribution over the $K$ codes in the codebook for each latent token, representing how similar that token is to each code.

Why this form: normalizing both vectors to the unit sphere ensures that the quantization is based on direction rather than magnitude, preventing degenerate solutions where the encoder outputs large-magnitude vectors that always map to the same code. The temperature controls the sharpness of the distribution; $\tau = 1.0$ is the standard softmax without scaling.

The key innovation of IBQ is in how it computes the one-hot index for the forward pass while maintaining differentiability for the backward pass. Standard argmax is not differentiable; straight-through estimators approximate the gradient by pretending the argmax was an identity function. IBQ refines this:

p=softmax(logits)p = \text{softmax}(\text{logits})

Ind=onehot(arg⁑max⁑(p))+[pβˆ’stopgrad(p)]\text{Ind} = \text{onehot}(\arg\max(p)) + [p - \text{stopgrad}(p)]

where $\text{onehot}(\arg\max(p)) \in \{0,1\}^K$ is the hard one-hot vector selecting the most probable code, $\text{stopgrad}(p)$ treats $p$ as constant during backpropagation, and $[p - \text{stopgrad}(p)]$ is zero in the forward pass but provides the gradient of $p$ in the backward pass.

What it computes: a one-hot index vector Ind that selects exactly one code from the codebook for the forward pass (ensuring discrete tokens for the AR model), but whose gradient with respect to the encoder parameters $\phi$ equals $\partial p / \partial \phi$ β€” the gradient flows through the softmax probabilities even though the forward pass uses hard selection.

Why this form: the $p - \text{stopgrad}(p)$ term is the standard straight-through gradient estimator applied to the probability vector rather than the logits. By applying it to $p$ (the softmax output) rather than the one-hot index directly, IBQ ensures that the gradient flows to all codebook entries proportional to their probabilities, not just the selected one. This is claimed to improve codebook utilization and prevent the collapse where only a few codes are ever used.

The quantized output is then $z_q = \text{Ind}^T C \in \mathbb{R}^{L \times d}$ β€” a weighted (actually, hard-selected) combination of codebook entries, one per latent token position.

Decoder. The decoder follows a symmetric design to the encoder. The quantized latent tokens $z_q \in \mathbb{R}^{L \times d}$ are first projected back to dimension $D$, then concatenated with $N = 256$ learnable mask tokens $m_{\text{patch}} \in \mathbb{R}^{N \times D}$ that serve as placeholders for the image patches to be reconstructed. This sequence passes through transformer layers (12 for S/B/L, 16 for H) with the same hybrid attention pattern: causal over the latent tokens, bidirectional over the mask tokens, with mask tokens attending to latent tokens but not vice versa.

The decoder outputs the transformed mask tokens, which then go through an "unpatchify" layer (rearranging the 1D sequence of patch embeddings back to a 2D grid) and a convolutional output layer to produce the reconstructed image $\hat{x}$. The reconstruction is $[βˆ…, \hat{x}] = D_\psi([z_q, m_{\text{patch}}])$ β€” the decoder also outputs something for the latent token positions (indicated by $βˆ…$), but only the mask token outputs are used.

Why a symmetric decoder with mask tokens? The mask tokens serve as queries that ask "what pixel values go here, conditioned on the latent tokens?" Since the latent tokens are 1D and have no spatial structure, the decoder must learn to map from this global, non-spatial representation back to a 2D image. This is a harder task than a 2D decoder where each latent token directly corresponds to a specific image region, but it preserves the 1D tokenizer's freedom from spatial constraints β€” the encoder doesn't need to assign latent tokens to specific locations because the decoder learns to extract spatial information from the globally-conditioned mask tokens.

Model size configurations (Table 9). The paper trains four model sizes (S, B, L, H) by jointly scaling both the tokenizer and the AR model:

ModelTokenizer ParamsTokenizer Layers (Enc/Dec)Hidden DimAR ParamsAR LayersAR Hidden Dim
EOSTok-S165M12/1276893M12768
EOSTok-B165M12/12768164M121024
EOSTok-L165M12/12768312M241024
EOSTok-H388M16/161024644M321280

The tokenizer is scaled from S/B/L (all using the same 165M configuration) to H (388M with deeper and wider architecture), while the AR model scales continuously across all four sizes. The total trainable parameters for EOSTok-H (tokenizer + AR model) is approximately 1 billion.

Autoregressive model architecture. The generative model is "based on LlamaGen [36], with an additional shared global AdaLN modulation with per-block learnable biases" (Section 4.1). This means it uses the standard Llama architecture (RMSNorm, SwiGLU activations, rotary or learned positional embeddings) but adds adaptive layer normalization conditioned on the class label (for class-conditional generation) or a learned embedding (for unconditional generation). Since the tokens are 1D sequences (not 2D grids), the paper replaces LlamaGen's 2D RoPE (rotary position embeddings designed for 2D spatial positions) with standard learnable positional embeddings. The AR model uses a causal attention mask so each token can only attend to previous tokens in the sequence.

Quantizer regularization. The VQ-VAE objective includes standard regularization terms for the quantizer. The paper mentions "commitment loss, entropy loss, and etc." as part of $\mathcal{L}_{\text{reg}}$. The commitment loss encourages the encoder to commit to codebook entries (penalizing large distances between $z$ and the selected code), while the entropy loss encourages uniform codebook usage (penalizing the distribution of selected codes from collapsing to a few entries). The specific regularization weight $\lambda_{\text{reg}} = 10^{-3}$ and entropy weight $0.01$ are listed in Table 9.

Joint Training of Reconstruction and Generation: The APR Loss

The core technical problem that EOSTok solves is: how do you provide useful generative supervision to a tokenizer without causing it to collapse to degenerate solutions? This section explains why naive joint training fails, how the APR loss fixes it, and the gradient propagation mechanism that makes end-to-end training possible.

The conventional two-stage paradigm. In the standard approach used by all prior tokenizer-based AR image models:

  • Stage 1: Train the tokenizer (encoder $E_\phi$, decoder $D_\psi$, codebook $C$) solely on reconstruction:

    LVQVAE(Ο•,ψ)=Lrecon(x,Dψ(zq))+Ξ»regLreg\mathcal{L}_{\text{VQVAE}}(\phi, \psi) = \mathcal{L}_{\text{recon}}(x, D_\psi(z_q)) + \lambda_{\text{reg}}\mathcal{L}_{\text{reg}}

    where $z_q = Q(E_\phi(x))$ is the quantized latent representation. $\mathcal{L}_{\text{recon}}$ combines L2 loss, LPIPS perceptual loss (computed with a VGG backbone), and a GAN loss with StyleGAN-T discriminator and LeCam divergence regularization. The GAN weight is $\lambda_{\text{GAN}} = 0.1$ and LeCam weight is $0.05$.

  • Stage 2: Freeze the tokenizer. Encode all training images into discrete token sequences. Train an AR model $G_\theta$ from scratch on next-token prediction:

    LNTP(ΞΈ)=βˆ’βˆ‘n=1Llog⁑pΞΈ(zq,n∣zq,<n)\mathcal{L}_{\text{NTP}}(\theta) = -\sum_{n=1}^{L} \log p_\theta(z_{q,n} \mid z_{q,<n})

    where $p_\theta$ is the AR model's predicted distribution over the $K$ codes at position $n$, conditioned on all previous tokens.

The problem with this decoupled approach is that the tokenizer receives no signal about whether its latent codes are easy or hard to predict sequentially. A tokenizer optimized purely for reconstruction might produce codes where fine-grained details (high-frequency textures, exact edge positions) are distributed across tokens in a way that's statistically unpredictable, forcing the AR model to model essentially random variation. The paper's key insight is that the tokenizer should be trained to make the AR model's job easier, and the AR model's performance should provide feedback to the tokenizer.

Vanilla end-to-end training and why it fails. The simplest way to provide this feedback is to add the NTP loss to the tokenizer's training objective:

LE2E(Ο•,ψ,ΞΈ)=LVQVAE(Ο•,ψ)+Ξ»NTPLNTP(Ο•,ΞΈ)\mathcal{L}_{\text{E2E}}(\phi, \psi, \theta) = \mathcal{L}_{\text{VQVAE}}(\phi, \psi) + \lambda_{\text{NTP}}\mathcal{L}_{\text{NTP}}(\phi, \theta)

where $\phi, \psi, \theta$ are parameters of the encoder, decoder, and AR model respectively, and $\lambda_{\text{NTP}}$ controls the weight of the generative objective (set to $0.1$ for S/B/L models, $0.01$ for H).

What it computes: the standard VQ-VAE reconstruction loss plus the AR model's cross-entropy loss on predicting the next discrete token at each position. The AR model and the tokenizer are trained simultaneously from scratch, with the NTP loss providing a learning signal to the encoder about how predictable its token sequences are.

Why this form: if the AR model can predict the next token easily (low NTP loss), the encoder is producing structured, predictable sequences β€” which should be good for generation. If the NTP loss is high, the sequences are unpredictable, and the encoder should adjust to make them more structured.

The codebook collapse problem. As shown in Table 1 and Figure 3, vanilla E2E training produces a catastrophic failure: the NTP loss hacks the tokenizer into using very few codes. The AR prediction accuracy jumps from 11.8% (two-stage baseline) to 30.2% β€” the AR model finds the tokens much easier to predict. But code usage (the fraction of the 4096-code vocabulary used with frequency > 5%/K on the validation set) plummets from 99.8% to 51.8%. Reconstruction FID degrades from 1.09 to 4.92, and generation FID worsens from 3.82 to 8.01.

What's happening mechanically. The NTP loss measures cross-entropy on a $K$-way classification problem at each position. If the encoder maps all images to the same few tokens, the AR model can achieve low NTP loss simply by learning a trivial distribution (always predict the same tokens in roughly the same order). The encoder learns to do this because it receives gradient from the NTP loss through the straight-through estimator in the quantizer β€” it can adjust its outputs to make the AR model's job trivially easy. Figure 3a visualizes this: the PCA of the codebook shows it distributes unevenly in the latent space, and the latent embeddings of images only match a small fraction of codes. Figure 3b shows the token frequency distribution is highly skewed toward a small subset.

This is a classic case of reward hacking or specification gaming: the NTP loss, which is intended as a proxy for "how structured is the latent space," can be minimized in ways that don't actually produce a useful latent space. The codebook collapse means the latent representation has very low information capacity β€” images are mapped to nearly identical token sequences regardless of content, so reconstruction quality degrades, and generation quality degrades because the AR model can only produce a limited variety of images.

The APR loss: bridging the gap. The paper's solution is to provide an end-to-end signal that directly measures generation quality in pixel space, bypassing the discrete token bottleneck. The key idea: during training, take the AR model's teacher-forced predictions (what it thinks the next token should be at each position, given the ground-truth previous tokens), decode these predicted tokens through the tokenizer's decoder, and compare the resulting image to the original:

LAPR(Ο•,ψ,ΞΈ)=βˆ₯xβˆ’Dψ(GΞΈ(zq))βˆ₯22\mathcal{L}_{\text{APR}}(\phi, \psi, \theta) = \|x - D_\psi(G_\theta(z_q))\|_2^2

where $x$ is the ground-truth image, $z_q = Q(E_\phi(x))$ is the quantized latent representation, $G_\theta(z_q)$ produces the AR model's teacher-forced predictions $\hat{z}_q$ (at each position, the AR model predicts the next token given the previous ground-truth tokens), and $D_\psi(\hat{z}_q)$ decodes these predicted tokens back to pixel space.

What it computes: the mean squared error between the original image and the image reconstructed from the AR model's predicted token sequence. This is a direct measure of generation quality: if the AR model can accurately predict the token sequence (and the decoder can reconstruct from it), the APR loss is low.

Why this form: unlike NTP loss, which measures token-level accuracy in a discrete space, APR loss measures perceptual quality in continuous pixel space. This means the encoder cannot hack it by collapsing to a few codes β€” using fewer codes would make it impossible to reconstruct diverse images from the AR model's predictions, increasing the APR loss. The APR loss thus provides a counterbalancing force that prevents the codebook collapse that vanilla E2E training causes.

Implementation detail: batched decoding. During training, the authors "concatenate the AR prediction $\hat{z}_q = G_\theta(z_q)$ with $z_q$ along the batch dimension, and pass them together to the decoder" (Section 3.2). This means the decoder processes both the ground-truth latent codes (for the reconstruction loss) and the AR-predicted latent codes (for the APR loss) in the same forward pass, sharing computation. The APR loss is enhanced with LPIPS perceptual loss in addition to L2, following the same pattern as the reconstruction loss.

APR loss prevents codebook collapse. As shown in Table 1, adding APR loss to the end-to-end training framework restores code usage to 99.7% (essentially full utilization, matching the 99.8% of the two-stage baseline), improves rFID from 4.92 back to 1.02 (even slightly better than the 1.09 baseline), and dramatically improves gFID from 8.01 to 3.32 (beating the 3.82 baseline). Figure 3a confirms visually that the codebook distributes uniformly in the latent space, and the latent embeddings of images match a broad range of codes.

Why this works: the APR loss as a regularizer. The NTP loss alone provides a myopic signal ("make the next token easy to predict") that can be satisfied by collapsing the latent space. The APR loss provides a holistic signal ("make the entire predicted sequence decode to a realistic image") that can only be satisfied by maintaining a rich, diverse latent space. The two losses together create a productive tension: the NTP loss pushes the latent space to be structured and predictable, while the APR loss pushes it to be expressive enough to reconstruct diverse images. The tokenizer must find a representation that satisfies both β€” structured enough for sequential prediction, expressive enough for reconstruction.

Gradient propagation through discrete tokens. A critical technical challenge in end-to-end training is that the AR model operates on discrete tokens, which are not naturally differentiable. The paper's solution is to modify the AR model's embedding layer: "instead of a look-up operation," the embedding layer takes the probability vector $\text{Ind} \in \mathbb{R}^{L \times K}$ from the IBQ quantizer and computes the embedding as $h = \text{Ind}^T \text{Embed}$, where $\text{Embed} \in \mathbb{R}^{K \times D}$ is the learned embedding matrix. Since Ind includes the gradient term $p - \text{stopgrad}(p)$ from IBQ, gradients from the NTP loss flow through the embedding layer, through Ind, and back to the encoder and codebook via the softmax probabilities $p$. This enables full end-to-end training with discrete tokens.

Training efficiency and APR loss weight. The paper ablates the APR loss weight $\lambda_{\text{APR}}$ in Table 10, finding that $\lambda_{\text{APR}} = 1.0$ achieves the best balance (gFID 3.32 vs. 4.09 at 0.0 and 3.57 at 4.0). This suggests that the APR loss needs to be strong enough to counteract the NTP loss's tendency to collapse the latent space, but not so strong that it overwhelms the reconstruction objective and degrades image quality.

Evidence that end-to-end training improves AR-friendliness. Table 4 provides a compelling demonstration. The authors take a tokenizer trained with end-to-end training (including NTP and APR losses), freeze it, and then train new AR models from scratch on token sequences with different orderings:

  • Original ordering: gFID 4.10, AR accuracy 10.3%
  • Reversed ordering: gFID 10.27, AR accuracy 9.5%
  • Random (fixed) ordering: gFID 7.81, AR accuracy 9.8%

The original ordering dramatically outperforms both alternatives, even though the tokenizer was never explicitly told to make the sequence sequential (the reconstruction loss doesn't depend on token order). This shows that the end-to-end training pipeline β€” specifically, the NTP and APR losses β€” has taught the tokenizer to structure its latent space such that tokens in the original order are much more predictable than tokens in any other order. The latent space has become genuinely "AR-friendly."

Semantic Representation Injection: How to Use Vision Foundation Models Without Breaking the 1D Structure

Vision foundation models (VFMs) like DINOv2 encode rich semantic information β€” object identities, spatial relationships, visual concepts β€” that could help the tokenizer produce more meaningful latent representations. However, VFM features are inherently 2D: they produce a feature vector for each patch in the image, preserving spatial layout. The challenge is injecting this semantic knowledge into a 1D tokenizer without imposing the 2D spatial structure that 1D tokenizers were designed to eliminate.

Why VFM alignment helps for 1D tokenizers. The paper hypothesizes that 1D decoding is inherently harder than 2D decoding because the decoder must recover pixels whose information "distributes globally, instead of locally aligns with, the latent token sequence. This is more similar to a conditional generation task, rather than a reconstruction task" (Section 3.3). In other words, a 2D decoder can use the spatial correspondence between latent codes and image regions to simplify reconstruction (each latent code corresponds to a specific patch, so the decoder only needs to fill in local details). A 1D decoder has no such correspondence β€” it must learn a global mapping from the sequential latent codes to the 2D image. This makes convergence harder, and VFM alignment provides a rich semantic signal that can guide the decoder's learning, much like how REPA [49] uses VFM alignment to accelerate diffusion model training.

The VFM model. EOSTok uses DINOv2-ViT-L [24] as the default vision foundation model. DINOv2 is a self-supervised ViT trained on a diverse dataset of images, producing features that capture semantic information (object parts, semantic correspondences) without being explicitly trained on labels. Given an input image $x$, the frozen DINOv2 model produces patch-level features $y = f(x) \in \mathbb{R}^{N \times D_{\text{VFM}}}$ where $N = 256$ is the number of patches (matching the tokenizer's patch grid) and $D_{\text{VFM}}$ is the DINOv2 feature dimension. A learnable MLP projector $h_\omega$ maps these features to the tokenizer's hidden dimension $D$.

Three encoder injection strategies compared (Figure 4). The paper systematically evaluates three ways to inject VFM representations into the 1D ViT encoder:

(a) Direct alignment (like VA-VAE [44]). This approach forces the 1D latent tokens $z$ to directly match the VFM features $y$ using a cosine similarity loss:

Ldirect(Ο‰,Ο•)=βˆ’1Lβˆ‘β„“=1Lsim(hΟ‰(z[β„“]),I(y)[β„“])\mathcal{L}_{\text{direct}}(\omega, \phi) = -\frac{1}{L} \sum_{\ell=1}^{L} \text{sim}(h_\omega(z[\ell]), \mathcal{I}(y)[\ell])

where $L = 256$ is the number of latent tokens, $z[\ell] \in \mathbb{R}^D$ is the $\ell$-th latent token (before projection to dimension $d$), $y \in \mathbb{R}^{N \times D_{\text{VFM}}}$ is the VFM's patch-level features (256 patches), $\mathcal{I}: \mathbb{R}^{N \times D_{\text{VFM}}} \to \mathbb{R}^{L \times D}$ is an interpolation operation that maps the 256 VFM patch features to 256 latent token positions (preserving spatial order), $h_\omega$ is a learnable MLP projector, and $\text{sim}$ measures cosine similarity.

What it computes: the average cosine similarity between each latent token and its spatially corresponding (interpolated) VFM feature. The encoder is trained to make its latent tokens similar to the VFM features at the same spatial positions, while the MLP projector learns to map VFM features into the tokenizer's representation space.

Why this form: cosine similarity focuses on the direction of the representations rather than their magnitude, which is appropriate for alignment since the encoder and VFM may operate at different scales. The interpolation step is necessary because the 1D latent tokens don't have a natural spatial correspondence β€” by interpolating the 2D VFM features to a 1D sequence (essentially rasterizing them), this approach imposes an artificial but consistent spatial ordering.

Why this fails for 1D tokenizers (Table 2): "This loss enforces the 1D latent codes $z$ to match the spatially aligned features $f(x)$, which inevitably leaks the 2D spatial prior to the 1D tokenizer" (Section 3.3). The interpolation forces each latent token to correspond to a specific (interpolated) spatial position in the VFM's feature map. The latent tokens are no longer free to capture global, non-local information β€” they are being pushed to represent specific spatial regions. The empirical result: direct alignment improves reconstruction slightly (rFID 0.98 vs. 1.75 baseline) but worsens generation quality (gFID 5.98 vs. 12.27 baseline β€” wait, this is still an improvement over the 1D baseline, but the paper claims it's worse than implicit alignment at 3.32, and importantly, Table 2 shows direct alignment has lower AR accuracy (8.5%) than implicit alignment (11.9%), indicating the latent space has become less AR-friendly despite better reconstruction).

(b) Direct substitution. An alternative approach that has been used in diffusion models [2, 5, 33, 54] is to replace the raw image patches with projected VFM features. Instead of feeding pixel patches into the ViT encoder, feed the corresponding DINOv2 features:

xpatch←MLP(f(x))x_{\text{patch}} \leftarrow \text{MLP}(f(x))

where $\text{MLP}$ projects the VFM features to the tokenizer's hidden dimension $D$. The learnable query tokens $q$ are concatenated with these semantic patch features rather than raw pixel features, and the encoder processes them as before.

What it computes: the encoder operates on pre-extracted semantic features rather than learning features from pixels. The query tokens extract information from these semantic features, producing latent codes that are grounded in the VFM's understanding of the image content.

Why this is partially effective: it provides strong semantic grounding without explicitly aligning the latent space to 2D structure β€” the latent tokens can still capture global information because they attend to all VFM patch features through the transformer's attention mechanism. Table 2 shows this improves both reconstruction (rFID 1.05) and generation (gFID 4.89) over the baseline, and significantly improves AR accuracy (12.1%, the highest among all variants).

(c) Implicit alignment (the winning approach). Instead of aligning the latent tokens to VFM features, align the hidden patch embeddings $h_{\text{Enc}}$ β€” the encoder's intermediate representation of each image patch β€” to the VFM features. The latent tokens $z$ are not directly aligned:

Limplicit(Ο‰,Ο•)=βˆ’1Nβˆ‘n=1Nsim(hΟ‰(hEnc[n]),y[n])\mathcal{L}_{\text{implicit}}(\omega, \phi) = -\frac{1}{N} \sum_{n=1}^{N} \text{sim}(h_\omega(h_{\text{Enc}}[n]), y[n])

where $N = 256$ is the number of patches, $h_{\text{Enc}}[n] \in \mathbb{R}^D$ is the encoder's hidden representation of the $n$-th image patch (after attention but before the final output projection), $y[n] \in \mathbb{R}^{D_{\text{VFM}}}$ is the VFM feature for the same spatial patch, $h_\omega$ is a learnable MLP projector, and $\text{sim}$ measures cosine similarity.

What it computes: the average cosine similarity between each patch's hidden representation in the encoder and the corresponding VFM feature for that patch. The encoder is trained to make its internal representation of each patch match the VFM's semantic understanding of that patch.

Why this form: because the alignment is on the hidden patch embeddings rather than the latent tokens, the 1D latent codes $z$ remain free from 2D spatial constraints. The latent tokens can still capture global, non-local information through their attention over the patch embeddings β€” they benefit from the enriched, semantically-aware patch features without being forced to correspond to specific spatial positions. The patch embeddings serve as a semantic "backbone" that the latent tokens extract information from, but the extraction process is learned through the reconstruction and generative objectives, not through explicit spatial alignment.

Why this wins (Table 2): implicit alignment achieve the best balance β€” good reconstruction (rFID 1.02), the best generation quality among the encoder alignment methods (gFID 3.32), and high AR accuracy (11.9%, second only to direct substitution). Critically, comparing the "Baseline" (rFID 1.75, gFID 12.27, AR Acc 7.8%) to "Baseline + Decoder alignment + Implicit alignment" (the full EOSTok recipe, though Table 2 shows them additively), the generation quality improves dramatically while AR accuracy increases from 7.8% to 11.9% β€” evidence that the latent space has become genuinely more AR-friendly through semantic guidance.

Decoder alignment. In addition to encoder alignment, the paper applies REPA-style alignment to the tokenizer's decoder. The motivation: "the reconstruction task of a 1D ViT decoder is much harder than a 2D ViT decoder" because it must recover pixels without spatial correspondence to latent codes, making it "more similar to a conditional generation task, rather than a reconstruction task" (Section 3.3). Just as REPA accelerates diffusion model training by aligning intermediate features to VFM representations, aligning the decoder's hidden states to VFM features could help the 1D decoder converge.

The decoder alignment loss is applied to the hidden features of the mask tokens from the $k$-th layer of the decoder (the specific layer index is not specified in the main paper, but following REPA's practice would likely be an early-to-middle layer):

Ldecoder-align(Ο‰,Ο•,ψ)=βˆ’1Nβˆ‘n=1Nsim(hΟ‰(hDec[n]),y[n])\mathcal{L}_{\text{decoder-align}}(\omega, \phi, \psi) = -\frac{1}{N} \sum_{n=1}^{N} \text{sim}(h_\omega(h_{\text{Dec}}[n]), y[n])

where $h_{\text{Dec}}[n]$ is the decoder's hidden representation of the $n$-th mask token at some intermediate layer, and all other symbols are as defined previously.

What it computes: cosine similarity between the decoder's intermediate mask token representations and the corresponding VFM patch features, encouraging the decoder to internally represent semantic information consistent with the VFM.

Why this matters empirically (Table 2): comparing the baseline 1D tokenizer (rFID 1.75, gFID 12.27) to the baseline with only decoder alignment (rFID 1.12, gFID 5.68), decoder alignment provides a massive improvement β€” rFID drops by 36% and gFID by 54%. Interestingly, AR accuracy barely changes (7.8% vs. 8.2%), meaning the improvement comes from better reconstruction and generation quality, not from making the latent space more predictable.

Full EOSTok objective. Putting everything together, the complete training objective is:

LEOSTok(Ο•,ψ,ΞΈ)=LVQVAE(Ο•,ψ)+Ξ»NTPLNTP(Ο•,ΞΈ)+Ξ»APRLAPR(Ο•,ψ,ΞΈ)+min⁑ω1,Ο‰2Ξ»sem(Limplicit(Ο‰1,Ο•)+Ldecoder-align(Ο‰2,Ο•,ψ))\mathcal{L}_{\text{EOSTok}}(\phi, \psi, \theta) = \mathcal{L}_{\text{VQVAE}}(\phi, \psi) + \lambda_{\text{NTP}}\mathcal{L}_{\text{NTP}}(\phi, \theta) + \lambda_{\text{APR}}\mathcal{L}_{\text{APR}}(\phi, \psi, \theta) + \min_{\omega_1, \omega_2} \lambda_{\text{sem}}\left(\mathcal{L}_{\text{implicit}}(\omega_1, \phi) + \mathcal{L}_{\text{decoder-align}}(\omega_2, \phi, \psi)\right)

where $\omega_1, \omega_2$ are the parameters of two separate learnable MLP projectors (one for encoder implicit alignment, one for decoder alignment), and $\min_{\omega_1, \omega_2}$ indicates that the projectors are trained to minimize the alignment loss (they learn to map the tokenizer's internal representations to match the VFM features), while the tokenizer is trained to make its representations alignable (by gradient through the $\min$ operation β€” the tokenizer gets gradient from the alignment loss after the projectors have been optimized for the current step).

What it computes: a weighted sum of four objectives: reconstruction quality (VQ-VAE loss), token-level prediction accuracy (NTP loss), pixel-level generation quality (APR loss), and semantic alignment (implicit encoder + decoder alignment). All components are trained jointly, with gradients flowing between them through the differentiable IBQ quantizer.

Why this form: each loss addresses a different aspect of the desired tokenizer behavior. The VQ-VAE loss ensures the tokenizer can compress and reconstruct images (expressiveness). The NTP loss ensures the latent space is structured for sequential prediction (predictability). The APR loss prevents the NTP loss from collapsing the latent space and provides end-to-end generation quality feedback (grounding). The alignment losses inject semantic knowledge from VFMs without imposing 2D constraints (semantic richness). The $\min$ over projectors is a standard technique from representation learning: the projectors are optimized to best align the representations, while the tokenizer is optimized to produce representations that can be aligned β€” this prevents the tokenizer from simply ignoring the alignment loss.

Loss weights (Table 9). The specific weights used are:

  • Reconstruction: L2 weight 1.0, LPIPS weight 1.0, GAN weight 0.1, LeCam weight 0.05
  • Implicit alignment: 1.0
  • APR loss: L2 weight 1.0, LPIPS weight 1.0
  • NTP loss: 0.1 (for S/B/L) or 0.01 (for H)
  • Semantic alignment (𝝀sem): not explicitly stated as a separate scalar in Table 9, appears to be 1.0 from the implicit alignment weight

Choice of VFM matters but isn't critical (Table 12). The paper also experiments with SigLIP2 [38] instead of DINOv2 for representation alignment. SigLIP2 provides richer global semantic information (it's a vision-language model trained with contrastive learning). Results show SigLIP2 slightly improves rFID (0.88 vs. 1.02) and gFID (3.02 vs. 3.32), demonstrating that the framework is robust to the choice of VFM β€” any strong semantic feature extractor works, and better semantic features lead to slightly better results.

Training and Optimization Details

Optimizer and schedule. Both the tokenizer and AR model use Adam optimizers with an initial learning rate of $1 \times 10^{-4}$ and a cosine learning rate schedule that decays to $1 \times 10^{-6}$ over 2 million iterations (approximately 400 epochs on ImageNet with batch size 256). The discriminator (for GAN loss) uses a separate Adam optimizer with a fixed learning rate of $1 \times 10^{-4}$. The Adam $\beta_1 = 0.9$ for all components, but $\beta_2$ differs: 0.999 for the tokenizer and 0.95 for the AR model β€” this is a deliberate choice following standard practice where generative models (AR, diffusion) often benefit from lower $\beta_2$ (less momentum on the second moment, meaning faster adaptation to changing gradient statistics), while autoencoders benefit from higher $\beta_2$ (more stable second-moment estimates for the reconstruction task). Training uses BF16 mixed precision on 8 H100 GPUs with a per-GPU batch size of 32 (total batch size 256).

Exponential moving average. An EMA with decay rate 0.9999 is applied to the tokenizer weights. EMA helps stabilize the tokenizer outputs during training, which is important because the AR model's training depends on the tokenizer's current outputs β€” if the tokenizer's latent space shifts too quickly, the AR model's predictions become stale.

Nested dropout. Following FlexTok [1] and Semanticist [41], the paper applies nested dropout to the latent tokens during training. With probability $p = 0.5$ (for S/B/L models) or $p = 1.0$ (for H model, Table 9), a random prefix of the latent token sequence is dropped β€” the decoder must reconstruct the image from only the remaining (later) tokens. This "enforces important information to be represented by earlier tokens" β€” since early tokens are more likely to be present (they're dropped with lower probability), the encoder learns to put the most critical information in the first few tokens, creating a coarse-to-fine structure in the latent sequence. The ablation in Table 11 shows that nested dropout significantly improves AR accuracy (from 10.2% at $p=0$ to 17.6% at $p=1.0$) by making the token sequence more predictable (early tokens carry coarse information that's easy to predict, later tokens refine details). However, aggressive dropout ($p=1.0$) slightly worsens gFID (3.50 vs. 3.32 at $p=0.5$) because the over-compression hurts reconstruction quality. The paper uses $p=0.5$ as the sweet spot.

Class dropout. For classifier-free guidance, the class conditioning is dropped with probability 0.1 during training β€” the AR model learns to generate both class-conditional and unconditional samples, enabling guidance at inference time.

Sampling at inference. During inference, images are generated by sampling from the AR model autoregressively with KV-caching (for efficiency). The sampling uses a temperature of 1.0 without top-k or top-p filtering β€” the model's learned distribution is used directly. For guidance, the paper uses AutoGuidance [17] instead of standard classifier-free guidance for the larger models (L and H). In AutoGuidance, a smaller, weaker version of the AR model is trained on the same tokenizer, and its predictions are used as the "unconditional" signal in the CFG formula: $\ell_g = \ell_u + s(\ell_c - \ell_u)$, where $\ell_c$ are the conditional logits from the main model, $\ell_u$ are the logits from the smaller "bad" model, and $s$ is the guidance scale. The paper notes "a diminishing effect on applying CFG sampling when scaling up our model" β€” as models get larger, their unconditional generation quality improves to the point where standard CFG provides little benefit. AutoGuidance, by using a deliberately weaker model as the unconditional baseline, restores the effectiveness of guidance.

Sequence Length and Codebook Size: The Reconstruction-Generation Tradeoffs

Sequence length tradeoff (Table 5). The 1D tokenizer can compress images into any sequence length $L$ by changing the number of learnable query tokens. The paper sweeps $L \in \{32, 64, 128, 192, 256\}$. Longer sequences consistently improve reconstruction (rFID drops from 17.50 at $L=32$ to 1.02 at $L=256$) because more tokens provide higher information capacity. However, generation quality follows a U-curve: gFID improves from 22.37 at $L=32$ to 3.04 at $L=192$, then slightly worsens to 3.32 at $L=256$. This establishes a reconstruction-generation tradeoff analogous to what VA-VAE [44] observed for latent channels in diffusion models: more tokens make reconstruction easier but generation harder (longer sequences are more difficult for the AR model to predict accurately). The paper uses $L=256$ for the main experiments, presumably because the slight gFID penalty is offset by better reconstruction.

Codebook size tradeoff (Table 6). The vocabulary size $K$ presents a similar tradeoff. Larger codebooks improve reconstruction (rFID drops from 1.18 at $K=1024$ to 0.96 at $K=16384$) because more codes provide finer quantization granularity. But larger codebooks also make the AR model's classification task harder β€” predicting one of 16384 codes is harder than predicting one of 1024 codes. The gFID worsens from 3.24 at $K=1024$ to 4.08 at $K=16384$. Crucially, the paper finds that scaling the model mitigates this tradeoff: Figure 5a shows that for EOSTok-H, the gap between $K=4096$ and $K=16384$ nearly closes (gFID 1.48 vs. 1.51), while for EOSTok-L the gap is larger (1.74 vs. 1.92). Larger AR models can handle larger vocabularies without degradation, suggesting that the optimal codebook size depends on model capacity.

Code usage across codebook sizes (Table 6). The paper's IBQ quantizer maintains high code usage even at large vocabulary sizes: 100% utilization at $K=1024$ and $K=2048$, 99.7% at $K=4096$ and $K=8192$, and 99.2% at $K=16384$. This is a significant improvement over standard VQ-VAE, where codebook collapse (many codes unused) is a persistent problem at large $K$. The $\ell_2$ normalization of both codebook vectors and latent codes, combined with the IBQ gradient formulation, appears to be the key to maintaining utilization.

4. Key Insights and Innovations

Innovation 1: End-to-end generative feedback resolves the latent codebook collapse that explicit token prediction causes

The field's default assumption β€” implicit across all two-stage tokenizer-then-AR training pipelines β€” is that a tokenizer trained purely for reconstruction produces representations that are at least adequate for generative modeling. EOSTok's diagnostic experiment with vanilla end-to-end training (Table 1) shatters this assumption: exposing the tokenizer to next-token prediction loss causes a catastrophic collapse to a tiny subset of the codebook vocabulary, because the NTP objective can be trivially satisfied that way. The tokenizer isn't merely unhelpful under generative supervision β€” it actively breaks.

This is not an incremental finding. Prior work treated "separate training" as a convenient engineering choice; EOSTok reveals it as a necessary safety mechanism that was masking a fundamental instability. The moment you let the tokenizer see the NTP loss, it exploits it β€” collapsing from 99.8% code usage to 51.8% while paradoxically improving AR prediction accuracy from 11.8% to 30.2%. This is the paper's central diagnostic contribution: NTP loss is not a valid proxy for latent space quality when the tokenizer has control over the latent space. The tokenizer can "hack" the metric by reducing the vocabulary to a handful of tokens, making prediction trivially easy at the cost of destroying information capacity.

What makes this insight broadly significant is that it identifies a specification gaming problem in a context where the field hadn't been looking for one. Reward hacking is well-known in RLHF; codebook collapse is well-known in VQ-VAE training. But the interaction between them β€” that generation loss actively incentivizes collapse, and that two-stage training was unknowingly preventing it β€” is novel. It reframes the reconstruction-generation dilemma from a tradeoff to a principal-agent problem: the NTP loss is an agent that will find the cheapest way to look good, and the tokenizer needs a counterbalancing objective that forces it to maintain representational capacity.

The APR loss itself is the mechanism (Section 3), but the insight is that you need an end-to-end pixel-space signal to ground the token-level objective. This explains why prior attempts at joint training (if any were tried informally) would have failed and been abandoned β€” without understanding the collapse mechanism, the natural reaction would be "joint training doesn't work" rather than "joint training requires a pixel-space regularizer." The fact that the paper's Vanilla E2E ablation (the first row of results showing collapse) is presented as a deliberate experiment reveals a diagnostic mindset that the field was missing.

Innovation 2: Removing spatial structure from the latent space is necessary but not sufficient β€” the training objective must respect that removal

This paper distinguishes itself from prior 1D tokenizer work (TiTok, FlexTok, Semanticist) through a conceptual move rather than an architectural one. All existing 1D tokenizers argued that removing the 2D spatial prior from the latent space was the key to making tokens suitable for AR modeling. EOSTok demonstrates that this is only half the story: you must also remove the 2D spatial prior from the training objective, or it leaks back in through the loss function.

The evidence for this is Table 2's comparison of alignment strategies. Direct alignment (Figure 4a) forces the 1D latent tokens to match spatially-interpolated DINOv2 features β€” the latent space is 1D in architecture but 2D in its training signal. The result: worse generation quality (gFID 5.98) than implicit alignment (gFID 3.32), despite better reconstruction (rFID 0.98 vs. 1.02). The latent space has been secretly re-spatialized through the loss, undermining the very property that makes 1D tokenizers valuable. Direct substitution (Figure 4b) avoids this by replacing pixel patches with VFM features, but it still ties the latent tokens' inputs to a fixed spatial grid. Only implicit alignment (Figure 4c) β€” aligning hidden patch embeddings rather than latent tokens β€” fully respects the 1D design: the latent tokens interact with semantically enriched but spatially-grounded representations through attention, free to learn their own non-spatial extraction patterns.

This is a principle with implications beyond EOSTok. Any method that introduces 2D-aligned supervision to a 1D tokenizer β€” whether through feature matching, distillation, or auxiliary losses β€” must contend with the risk of re-imposing the spatial structure that the architecture was designed to escape. The paper's table of alignment variants (Table 2) is essentially a controlled experiment demonstrating how different injection points leak different amounts of spatial bias, with the latent token space being the most sensitive and the hidden patch space being a safe intermediary.

The significance for the field is that it provides a concrete design rule: when incorporating external knowledge into a non-spatial latent representation, inject it at the intermediate feature level where spatial structure is natural, not at the latent level where spatial structure is actively harmful. This is not obvious a priori β€” one might reasonably think that aligning the latent space directly to VFM features would be the most effective way to transfer semantic knowledge. The paper shows that intuition is wrong for 1D tokenizers, and it explains why through the lens of spatial prior leakage rather than through a narrower claim about which loss function happens to work better.

Innovation 3: The reconstruction-generation tradeoff can be shifted by scaling, but its existence is structural β€” it's not merely a capacity limitation

Table 5 (sequence length) and Table 6 (codebook size) both show a U-shaped or negatively-sloped relationship between reconstruction quality and generation quality. This is not a new observation β€” VA-VAE [44] documented similar tradeoffs for latent diffusion models β€” but EOSTok deepens the understanding in two ways that weren't apparent from prior work.

First, the paper demonstrates that the tradeoff is movable but ineliminable. Figure 5a shows that scaling from EOSTok-L to EOSTok-H nearly closes the gap between codebook sizes 4096 and 16384 (gFID 1.48 vs. 1.51 for H, compared to 1.74 vs. 1.92 for L). This means the tradeoff is partly a capacity problem β€” larger AR models can handle larger vocabularies β€” but it's also structural: even the largest model shows a gap, and the optimal codebook size shifts with model scale. This reframes the tradeoff from "pick the right fixed configuration" to "the configuration and the model capacity are coupled hyperparameters that should be scaled together." Prior work treated tokenizer design (vocabulary size, sequence length) as a fixed choice to be inherited by the generative model; EOSTok shows they are co-dependent.

Second, the nested dropout results (Table 11) reveal a deeper tension: enforcing coarse-to-fine structure (via dropout) improves AR accuracy from 10.2% to 17.6% but eventually hurts generation quality (gFID 3.50 at p=1.0 vs. 3.32 at p=0.5). This means predictability and information content are in direct conflict β€” making tokens more predictable requires putting less information in them, but at some point the information loss outweighs the predictability gain. This is a genuine structural tradeoff, not a capacity limitation. The tokenizer can't just "learn better representations" to satisfy both objectives; making the sequence more predictable inherently means discarding variation that might be needed for reconstruction.

The insight is that the reconstruction-generation tradeoff in AR image models is not just about the number of tokens or codes, but about how information is distributed across the sequence. Nested dropout is a knob that controls this distribution β€” pushing information earlier in the sequence makes the early tokens easier to predict (coarse, stable information) at the cost of leaving less capacity for fine details. This connects the practical hyperparameter choices (sequence length, codebook size, dropout rate) to a deeper information-theoretic principle that prior work hadn't articulated.

Innovation 4: The tokenizer's sequential ordering is learned, not imposed β€” and this learning is what makes AR generation work

Table 4 is arguably the paper's most conceptually important result, beyond any specific metric. An EOSTok tokenizer, frozen after end-to-end training, produces token sequences where the original ordering achieves gFID 4.10, but reversed ordering achieves only 10.27 and random ordering achieves 7.81. The tokenizer was never explicitly trained to make one ordering better than another β€” the reconstruction loss is invariant to permutation, and the APR loss provides only implicit ordering information through the teacher-forcing process. Yet the learned latent space strongly prefers the original sequential order.

This is significant because it establishes that the 1D sequential structure is an emergent property of the training procedure, not an architectural imposition. The tokenizer doesn't have an inductive bias toward sequential structure beyond the causal attention mask on query tokens β€” and yet it learns to encode images such that information unfolds in a predictable order. This is a form of self-organization: the NTP and APR losses jointly teach the encoder to arrange visual information temporally, making earlier tokens predict later tokens in a way that naturally forms a coarse-to-fine or causally-structured representation.

Why does this matter for the field? Because it validates the core hypothesis of the 1D tokenizer approach: that a non-spatial latent representation, when trained with appropriate objectives, will spontaneously develop a structure that supports sequential prediction. This is not guaranteed β€” one could imagine a 1D tokenizer that simply scrambles information arbitrarily across tokens, achieving good reconstruction but being impossible to predict. The fact that end-to-end training produces an order-sensitive latent space (as Table 4 demonstrates) means the training objectives successfully communicated "be predictable in this order" to the tokenizer, and the tokenizer found a way to comply.

It also has negative implications for approaches that try to avoid the ordering problem entirely. VAR [37] and MaskGIT [4] change the generation paradigm partly because they assume raster-order is fundamentally misaligned with 2D structure. Table 4 suggests that any learned ordering can work, as long as the tokenizer is trained to make it work β€” the problem isn't the ordering itself, but the fact that conventional two-stage training never teaches the tokenizer to care about predictability. This reframes the debate between "change the tokenizer (1D)" and "change the generation paradigm (masked, multi-scale)" as potentially a training methodology question rather than an architectural one.

Innovation 5: Joining reconstruction and generation into a single stage changes the scaling behavior of the system

The observation that Figure 5a shows monotonic improvement with model scale is not surprising for a generative model. What's distinctive is the comparison in Figure 5b: separately-trained models achieve lower NTP loss (better token prediction accuracy) than end-to-end trained models at the same scale, but worse generation quality. The curves cross in a non-trivial way β€” the two-stage approach optimizes a metric (NTP loss) that doesn't track the true objective (gFID), while the end-to-end approach optimizes a noisier but more aligned signal.

This is a measurement problem masquerading as a training methodology choice. In two-stage training, the tokenizer is evaluated by rFID and the AR model by NTP loss β€” both are surrogates for the real goal (generation quality), and neither captures the interaction between tokenizer and AR model. End-to-end training, by incorporating APR loss, creates a feedback loop where the tokenizer's choices affect the AR model's pixel-space output, and the AR model's limitations inform the tokenizer's representations. The system converges to a different point in the joint parameter space β€” one with higher token-level entropy (higher NTP loss) but better pixel-level fidelity.

The scaling implication is that joint training changes what "better" means for each component. In two-stage training, a better tokenizer means lower rFID; a better AR model means lower NTP loss. In end-to-end training, a better tokenizer means one that helps the AR model produce better images β€” which might involve slightly worse reconstruction (accepting some rFID degradation) if it makes the latent space substantially more predictable. The components co-adapt in a way that's impossible under separate training. This is visible in Figure 5a's codebook size comparison: EOSTok-H with K=16384 narrows the gap to K=4096 not because either is inherently better, but because the scaled-up AR model can handle the larger vocabulary without losing prediction accuracy, and the tokenizer can exploit the finer quantization without sacrificing predictability. The optimal configuration emerges from the joint optimization, not from independently optimizing each stage.

5. Experimental Analysis

Evaluation Methodology

  • Dataset. All experiments use ImageNet-1K at 256Γ—256 resolution [7], consisting of approximately 1.28 million training images across 1,000 object categories. The standard ImageNet validation set (50,000 images) is used for evaluation. This is the canonical benchmark for class-conditional image generation, enabling direct comparison with the extensive prior work in Tables 3 and 13.

  • Base model(s). The paper trains four model sizes (EOSTok-S, B, L, H) from scratch, jointly scaling the 1D ViT tokenizer (165M parameters for S/B/L, 388M for H) and the autoregressive transformer (93M to 644M parameters). The tokenizer architecture follows TiTok [48] with IBQ quantization [32]; the AR model is based on LlamaGen [36] with added AdaLN modulation. The choice of jointly scaling both components (rather than fixing one and scaling the other) is deliberate β€” it tests whether the end-to-end training framework benefits from increased capacity in both the tokenizer and the generator simultaneously.

  • Metrics. Three standard metrics are reported: rFID (FrΓ©chet Inception Distance between original and reconstructed images, measuring reconstruction quality), gFID (FID between generated and real images, measuring generation quality), and Inception Score (IS) [29]. The evaluation code follows Dhariwal and Nichol [8]. All FID scores are computed without classifier-free guidance unless explicitly noted with "w/ guidance." The paper also reports AR Accuracy (the next-token prediction accuracy of the AR model on the validation set, measuring how predictable the latent token sequences are) and Code Usage (the fraction of the K-code vocabulary used with frequency > 5%/K on the validation set, measuring whether the codebook has collapsed).

  • Baselines. The paper compares against three categories of prior work (Table 3):

    2D continuous latent space methods: LDM-4 [28], DiT-XL/2 [25], REPA-XL/2 [49], Lightning-DiT-XL [44], MAR-L [22].

    2D discrete tokenization methods: VQGAN [10], RQ-Transformer [20], DQ-Transformer [15], MaskGIT [4], MAGVIT-v2 [46], LlamaGen-XL [36], RAR-L [47], IBQ-L [32], VAR-d20 [37], AliTok-L [42].

    1D tokenization methods: TiTok-L-32 [48], FlexTok d18-18 [1], Semanticist [41], GigaTok [43], SpectralAR-d20 [16], VFMTok [53], ResTok [52].

    For internal ablations, the paper uses a Baseline (two-stage training: tokenizer trained on reconstruction only, then AR model trained on frozen tokenizer outputs) and a Vanilla E2E baseline (joint training with only NTP loss, no APR loss).

  • Generation budget / compute accounting. The paper measures computation in GFLOPs (Table 7, Table 8). For fair comparison between methods, the key metric is inference-time GFLOPs per generated image. EOSTok-H requires approximately 552 GFLOPs per image (342 for the AR model generating 256 tokens, plus 210 for one decoder pass), compared to approximately 59,900 GFLOPs for DiT-XL/2 with 250 diffusion steps β€” a roughly 100Γ— reduction. The paper also reports training GFLOPs and peak memory (Table 7), noting that end-to-end training incurs a 15-19% overhead compared to two-stage training due to the extra decoder forward passes for the APR loss and the VFM feature extraction. Sampling speed: EOSTok-H generates approximately 10.5 images per second on a single H100 GPU with batched generation and KV-caching.

  • Cross-validation / statistical protocol. No explicit cross-validation is reported. The paper uses the standard ImageNet training/validation split with evaluation on the full 50K-image validation set. Training runs for a fixed 400 epochs (~2M iterations at batch size 256) across all model sizes, with gFID monitored throughout training (Figure 5a,c). The paper does not report confidence intervals or multiple training runs with different seeds β€” results are from single training runs at each configuration. This is standard practice for ImageNet-scale generative modeling experiments (where training costs make multiple runs prohibitive), but it means the reported FID differences of 0.1-0.2 points (e.g., 1.48 vs. 1.51 in the codebook size comparison) should be interpreted cautiously as they may fall within run-to-run variance.

Main Quantitative Results

ImageNet 256Γ—256 Generation: State-of-the-Art Without Guidance

The headline result (Table 3): EOSTok-H achieves a gFID of 1.48 without guidance and 1.38 with guidance, using 644M AR parameters plus a 388M-parameter tokenizer (approximately 1B total trainable parameters). This is the best reported FID among all methods in the table that do not use classifier-free guidance. With guidance, EOSTok-H (gFID 1.38, IS 265.7) is competitive with the best diffusion models (Lightning-DiT-XL: gFID 1.35, IS 295.3; AliTok-L: gFID 1.38, IS 326.2) while using approximately 100Γ— fewer inference GFLOPs (Table 8).

Scaling behavior is monotonic across all four model sizes:

  • EOSTok-S (93M AR): gFID 3.50 without guidance, 2.57 with guidance, IS 155.7
  • EOSTok-B (164M AR): gFID 2.38 without guidance, 1.98 with guidance, IS 185.6
  • EOSTok-L (312M AR): gFID 1.74 without guidance, 1.35 with guidance, IS 210.2
  • EOSTok-H (644M AR): gFID 1.48 without guidance, 1.38 with guidance, IS 239.5

The IS improves consistently with scale (155.7 β†’ 239.5 without guidance), indicating that larger models produce more diverse and recognizable images across the 1,000 ImageNet classes.

Comparison with the strongest 1D tokenizer baselines (all using separate two-stage training):

  • TiTok-L-32 [48] achieves gFID 3.15 without guidance but uses only 32 tokens (vs. EOSTok's 256) and a mask-based generator (177M params). The paper notes TiTok was designed for aggressive compression; EOSTok's result shows that 1D tokenizers can achieve much better quality at higher token counts when trained end-to-end.
  • FlexTok d18-18 [1] reports gFID 2.02 with guidance using a 1.33B-parameter AR model β€” substantially larger than EOSTok-H (644M) with worse guided FID.
  • Semanticist [41] achieves gFID 2.57 with guidance using an AR-diffusion hybrid generator (343M). EOSTok-L (312M) achieves gFID 1.35 with guidance β€” better quality with fewer parameters.
  • VFMTok [53] achieves gFID 2.11 without guidance (343M AR). EOSTok-L achieves 1.74 without guidance at similar scale (312M).

The reconstruction quality is also state-of-the-art among discrete 1D tokenizers: EOSTok-L achieves rFID 0.73 (Table 3), compared to TiTok-L-32 at 2.21 (using only 32 tokens), FlexTok at 1.61, and GigaTok at 0.81. This is notable because 1D tokenizers face a harder reconstruction task than 2D tokenizers (which benefit from spatial correspondence between latent codes and image patches). The paper attributes the strong rFID to decoder alignment with VFM features (Table 2: decoder alignment improves rFID from 1.75 to 1.12 for the 1D tokenizer).

Scaling Behavior Across Training Iterations (Figure 5)

Figure 5a plots gFID against training iterations (from 200K to 2M) for all model configurations. Key observations:

  • Consistent improvement with scale: at every iteration count, larger models achieve lower gFID. The EOSTok-H curves (both K=4096 and K=16384) lie below the EOSTok-L curves throughout training.
  • Codebook size gap narrows with scale: for EOSTok-L, the K=4096 model converges to approximately 1.74 gFID while K=16384 converges to approximately 1.92 β€” a gap of 0.18. For EOSTok-H, K=4096 converges to approximately 1.48 and K=16384 to 1.51 β€” a gap of only 0.03. This demonstrates that scaling the AR model's capacity mitigates the difficulty of predicting from a larger vocabulary.
  • No saturation at 2M iterations: all curves are still declining at 2M iterations, suggesting that longer training would yield further improvements. This is a limitation of the reported results β€” the models are not trained to convergence, so the final gFID numbers may understate the achievable performance at each scale.

Figure 5b plots the next-token prediction loss (LNTP) against training iterations. The separate (two-stage) training achieves lower NTP loss than end-to-end training for the same model size (EOSTok-L), converging to a lower value. However, as Table 1 shows, end-to-end training with APR loss achieves substantially better gFID (3.32 vs. 3.82) despite higher NTP loss. This is the paper's key evidence that NTP loss does not determine generation quality β€” the two metrics can move in opposite directions.

Figure 5c shows gFID curves for different semantic alignment strategies during the first 250K iterations. Implicit alignment (the chosen method) achieves the fastest initial convergence and the lowest final gFID, while direct alignment (which leaks 2D spatial structure) converges slower and plateaus higher. Direct substitution converges similarly to implicit alignment initially but is ultimately worse.

End-to-End Training Prevents Codebook Collapse (Table 1)

The quantitative evidence for the APR loss's necessity:

  • Baseline (two-stage training): rFID 1.09, gFID 3.82, AR Accuracy 11.8%, Code Usage 99.8%
  • Vanilla E2E (joint training with only NTP loss): rFID 4.92, gFID 8.01, AR Accuracy 30.2%, Code Usage 51.8%
  • + APR loss (full EOSTok): rFID 1.02, gFID 3.32, AR Accuracy 11.9%, Code Usage 99.7%

The Vanilla E2E results are a diagnostic: NTP loss alone drives the tokenizer to increase AR accuracy from 11.8% to 30.2% β€” a 2.6Γ— improvement in predictability β€” but destroys reconstruction quality (rFID degrades 4.5Γ—) and generation quality (gFID degrades 2.1Γ—) by collapsing the codebook to half its vocabulary. Adding APR loss restores full code usage (99.7%) and improves both reconstruction (rFID 1.02, slightly better than the 1.09 baseline) and generation (gFID 3.32, notably better than the 3.82 baseline).

The AR accuracy in the full EOSTok model (11.9%) is essentially unchanged from the baseline (11.8%), yet generation quality is substantially better (gFID 3.32 vs. 3.82). This means the improvement does not come from making token prediction easier β€” it comes from making the consequences of token prediction (pixel-space reconstruction) better. The latent space has been restructured to be more generation-friendly without becoming more predictable in the discrete token sense.

Learned Sequential Ordering (Table 4)

Using a frozen EOSTok-L tokenizer (trained end-to-end for 50 epochs), new AR models are trained from scratch on token sequences with different orderings:

  • Original order: gFID 4.10, AR Accuracy 10.3%
  • Reversed order: gFID 10.27, AR Accuracy 9.5%
  • Random (fixed) order: gFID 7.81, AR Accuracy 9.8%

The original ordering achieves 2.5Γ— better gFID than reversed and 1.9Γ— better than random, even though AR accuracy is similar across all three (9.5-10.3%). This demonstrates that the end-to-end training has taught the tokenizer to structure information along the sequential axis β€” the original order is genuinely easier to generate from, not just easier to predict token-by-token (since AR accuracy barely differs). The gFID degradation from reversing the order (10.27) is worse than random ordering (7.81), suggesting the learned structure has a directional asymmetry β€” reversing it doesn't just shuffle information but actively breaks the coarse-to-fine or causal organization.

Sequence Length and Codebook Size Tradeoffs (Tables 5-6)

Sequence length (Table 5): Sweeping L ∈ {32, 64, 128, 192, 256} with a EOSTok-L model trained for 50 epochs:

  • rFID improves monotonically: 17.50 β†’ 1.94 β†’ 1.32 β†’ 1.08 β†’ 1.02
  • gFID follows a U-curve: 22.37 β†’ 3.18 β†’ 3.09 β†’ 3.04 β†’ 3.32

The optimal generation quality occurs at L=192 (gFID 3.04), slightly better than L=256 (3.32). The paper uses L=256 for main experiments, trading a small gFID penalty for better reconstruction. At extreme compression (L=32), both reconstruction and generation are poor β€” the information bottleneck is too severe. At moderate lengths (64-192), generation quality is relatively flat while reconstruction improves substantially, suggesting a broad plateau where the AR model can handle the sequence length without difficulty.

Codebook size (Table 6): Sweeping K ∈ {1024, 2048, 4096, 8192, 16384} with EOSTok-L for 50 epochs:

  • rFID improves monotonically: 1.18 β†’ 1.07 β†’ 1.02 β†’ 0.98 β†’ 0.96
  • gFID worsens: 3.24 β†’ 3.20 β†’ 3.32 β†’ 3.68 β†’ 4.08
  • Code usage remains high throughout: 100% β†’ 100% β†’ 99.7% β†’ 99.7% β†’ 99.2%

The tradeoff is clear: larger codebooks provide finer quantization (better reconstruction) but increase the classification difficulty for the AR model (worse generation). The IBQ quantizer maintains near-perfect codebook utilization even at K=16384 (99.2%), which is a significant improvement over standard VQ-VAE where codebook collapse is common at large vocabulary sizes. However, as Figure 5a shows, this tradeoff narrows substantially when scaling to EOSTok-H β€” the reconstruction-generation dilemma in codebook size is partly a capacity problem that can be addressed by scaling the AR model.

Semantic Alignment Strategy Comparison (Table 2)

The controlled experiment uses a EOSTok-L model trained for 50 epochs with 256 latent tokens (to match 2D tokenizer sequence length for fair comparison). Results for 1D tokenization:

  • Baseline (no VFM): rFID 1.75, gFID 12.27, AR Accuracy 7.8%
  • + Decoder alignment only: rFID 1.12, gFID 5.68, AR Accuracy 8.2%
  • + Decoder alignment + Direct alignment (a): rFID 0.98, gFID 5.98, AR Accuracy 8.5%
  • + Decoder alignment + Direct substitution (b): rFID 1.05, gFID 4.89, AR Accuracy 12.1%
  • + Decoder alignment + Implicit alignment (c): rFID 1.02, gFID 3.32, AR Accuracy 11.9%

For reference, the same alignment strategies applied to 2D tokenization:

  • 2D Baseline: rFID 1.52, gFID 12.51, AR Accuracy 5.2%
  • 2D + Decoder & direct alignment: rFID 0.87, gFID 6.06, AR Accuracy 7.9%

Several patterns emerge:

  • Decoder alignment provides the largest single improvement: gFID drops from 12.27 to 5.68 (-54%) while rFID improves from 1.75 to 1.12 (-36%). AR accuracy barely changes, meaning the decoder alignment improves reconstruction and generation quality without making the latent space more predictable.
  • Direct alignment on the encoder hurts generation: Adding direct alignment to decoder alignment actually worsens gFID (5.68 β†’ 5.98) compared to decoder alignment alone, despite improving rFID (1.12 β†’ 0.98). This supports the paper's claim that enforcing 2D spatial structure on the latent space is detrimental to AR modeling.
  • Implicit alignment is decisively best: gFID 3.32 vs. 5.98 (direct alignment) and 4.89 (direct substitution). It also achieves the second-highest AR accuracy (11.9%, vs. 12.1% for direct substitution), indicating the latent space becomes genuinely more generation-friendly, not just easier to decode.
  • 1D tokenizers benefit more from VFM alignment than 2D tokenizers: The 1D baseline improves from gFID 12.27 to 3.32 with the full EOSTok recipe (-73%), while the 2D baseline improves from 12.51 to 6.06 with decoder + direct alignment (-52%). The relative gain is larger for 1D, consistent with the paper's hypothesis that 1D decoding is a harder task that benefits more from semantic guidance.
  • Direct substitution achieves the highest AR accuracy (12.1%) but worse gFID than implicit alignment (4.89 vs. 3.32), another instance where NTP accuracy and generation quality diverge.

FLOPs Efficiency vs. Diffusion Models

Table 8 provides the inference cost comparison:

  • EOSTok-H: AR model (256 tokens) 342.3 GFLOPs + Decoder 210.1 GFLOPs = 552.4 GFLOPs total per image
  • DiT-XL/2: Diffusion model 237.2 GFLOPs Γ— 250 steps + Decoder 622.2 GFLOPs β‰ˆ 59,900 GFLOPs total

The paper notes that DiT's GFLOPs are reported using a convention where 1 multiply-add = 1 operation (rather than 2), which halves the apparent count. Even accounting for this, EOSTok-H is approximately 20-100Γ— faster depending on the diffusion sampling steps. At batch size 1 with KV-caching, EOSTok-H generates approximately 10.5 images/second on a single H100 GPU. This efficiency advantage β€” while achieving comparable or better FID β€” is the practical motivation for pursuing AR image generation over diffusion.

Ablation Studies and Robustness Checks

APR loss weight (Table 10): Sweeping Ξ»_APR ∈ {0.0, 0.5, 1.0, 2.0, 4.0} with fixed Ξ»_NTP = 0.1 on EOSTok-L for 50 epochs. rFID is stable for Ξ»_APR ≀ 1.0 (1.02-1.03) and degrades at higher weights (1.12 at 4.0). gFID follows a U-curve: 4.09 (no APR) β†’ 3.52 (0.5) β†’ 3.32 (1.0) β†’ 3.34 (2.0) β†’ 3.57 (4.0). The optimal weight of 1.0 balances the APR loss's regularization against the reconstruction objective. At Ξ»_APR = 0 (equivalent to Vanilla E2E plus decoder/VFM alignment from Table 2), gFID is 4.09 β€” notably better than the 8.01 in Table 1's Vanilla E2E, suggesting the VFM alignment and decoder alignment themselves partially mitigate codebook collapse, but not fully (the jump from 4.09 to 3.32 with APR loss is still substantial).

Nested dropout rate (Table 11): Sweeping dropout probability p ∈ {0.0, 0.25, 0.5, 1.0} on EOSTok-L for 50 epochs. rFID degrades as dropout increases: 0.85 β†’ 0.94 β†’ 1.02 β†’ 1.24. AR accuracy improves dramatically: 10.2% β†’ 10.8% β†’ 11.9% β†’ 17.6%. gFID follows an inverted U: 3.70 β†’ 3.52 β†’ 3.32 β†’ 3.50. The optimal gFID occurs at p=0.5, where the tradeoff between predictability (improved by forcing coarse-to-fine structure) and reconstruction fidelity (degraded by information loss from dropping tokens) is balanced. The AR accuracy at p=1.0 (17.6%) is particularly striking β€” the latent space has become much more predictable β€” but the gFID penalty (3.50 vs. 3.32) shows that excessive predictability comes at the cost of generation quality. This is a clean demonstration that predictability and information content trade off against each other.

Choice of vision foundation model (Table 12): Comparing DINOv2 [24] vs. SigLIP2 [38] on EOSTok-L for 50 epochs. SigLIP2 achieves better rFID (0.88 vs. 1.02) and better gFID (3.02 vs. 3.32). The improvement is modest but consistent, indicating the framework is robust to the choice of VFM and can benefit from stronger semantic feature extractors. This is important for practical adoption β€” users can plug in whatever VFM is available or state-of-the-art without architectural changes.

Scalability to 512Γ—512 resolution (Table 13): EOSTok-L (using the same architecture, patch size 16, and sequence length 256) achieves gFID 1.98 without guidance on ImageNet 512. This outperforms DiT-XL/2 (12.03), MaskDiT [55] (10.79), and TiTok variants (TiTok-B-128 at 4.17, TiTok-L-64 at 3.99). The result demonstrates that EOSTok's training framework transfers to higher resolutions without modification β€” the sequence length stays at 256 while the patch size remains 16, so the effective compression ratio is higher (512Γ—512 images have 1024 patches vs. 256 for 256Γ—256, meaning the 256 latent tokens must compress 4Γ— more information). The strong result suggests the semantic VFM alignment helps the tokenizer extract more informative latent representations under higher compression.

EOSTok-H with larger codebook (Figure 5a, Table 3): EOSTok-H with K=16384 achieves gFID 1.51 without guidance and 1.38 with guidance, compared to 1.48 and 1.37 for K=4096. The near-identical performance at scale indicates that for sufficiently large AR models, codebook size becomes less critical β€” the model can handle the larger vocabulary without prediction accuracy degradation. This is an encouraging result for scaling: one does not need to carefully tune codebook size for each model scale; larger models naturally accommodate larger vocabularies.

Training overhead of end-to-end training (Table 7): The forward-pass GFLOPs for one training iteration on EOSTok-L is 597 (AR 162 + Encode 91 + 2Γ— Decode 91Γ—2 = 182 + VFM 162), compared to 518 for two-stage training (AR 162 + Encode 91 + Decode 91 + VFM 162). The overhead is 15.2% for EOSTok-L and 18.6% for EOSTok-H. The extra cost comes primarily from the second decoder forward pass (for APR loss) and from processing both ground-truth and predicted tokens through the decoder. The paper argues this overhead is acceptable given the generation quality improvements and the fact that inference cost (which dominates total cost in deployment) is unchanged.

Reconstruction metrics beyond FID (Table 14): EOSTok-L achieves PSNR 22.15, SSIM 0.67, LPIPS 0.231, and rFID 0.73. Compared to recent discrete tokenizers: IBQ-L (n=16384) achieves similar PSNR (22.01) but worse SSIM (0.61) and rFID (1.37); GigaTok-B-L achieves slightly better LPIPS (0.206) and rFID (0.81) but lower PSNR (21.21). The paper notes that EOSTok achieves "comparable performance on PSNR, SSIM, and LPIPS despite using compact 1D compression" β€” the 1D structure does not inherently limit reconstruction quality when VFM alignment is used.

Critical Assessment

Claim 1: "End-to-end training improves final generation quality and makes the latent space more autoregressive-predictable." The evidence is strong but nuanced. Table 1 clearly shows that end-to-end training with APR loss improves gFID over two-stage training (3.32 vs. 3.82) while maintaining comparable AR accuracy (11.9% vs. 11.8%). Table 4 demonstrates that the learned latent space is order-sensitive, establishing that the tokenizer has learned to structure information sequentially. However, the "more autoregressive-predictable" claim is trickier: AR accuracy does not increase (11.9% vs. 11.8% baseline, or 10.3% in Table 4's original ordering), and Figure 5b shows that two-stage training achieves lower NTP loss than end-to-end training. What improves is not raw predictability (measured by token-level accuracy) but generation-quality-per-unit-of-predictability β€” the tokenizer produces sequences where correct predictions matter more for pixel-space quality, even if the prediction task isn't easier. This is a more subtle claim than "more predictable," and the paper's own evidence shows that higher AR accuracy (Vanilla E2E at 30.2%) can accompany worse generation quality (gFID 8.01). The paper would benefit from acknowledging this distinction explicitly: what improves is the alignment between prediction accuracy and generation quality, not prediction accuracy itself.

Claim 2: "The APR loss bridges the gap between NTP loss and generation quality." Strongly supported by Table 1 and the ablation in Table 10. The diagnostic comparison (Baseline vs. Vanilla E2E vs. +APR) cleanly isolates the APR loss's effect: vanilla joint training with NTP loss causes codebook collapse and degrades all metrics; adding APR loss restores code usage and improves both reconstruction and generation over the two-stage baseline. Table 10 shows the effect is dose-dependent β€” too little APR weight (0.0-0.5) under-regularizes, too much (4.0) over-constrains. An experiment that would further strengthen this claim: measuring whether the APR loss correlates with gFID better than NTP loss does across training. The paper shows the NTP loss curves (Figure 5b) and gFID curves (Figure 5a) but doesn't directly correlate them, which would provide quantitative evidence for the "bridging" claim.

Claim 3: "Implicit semantic alignment outperforms direct alignment by avoiding 2D spatial prior leakage." The evidence for the empirical superiority of implicit alignment is clear (Table 2: gFID 3.32 vs. 5.98). The mechanism claim β€” that this is because direct alignment leaks 2D structure β€” is plausible but has weaker direct evidence. The paper's argument rests on: (a) direct alignment forces latent tokens to match spatially-interpolated VFM features, which imposes a 2D correspondence, and (b) this leads to worse AR accuracy (8.5% vs. 11.9%) and worse gFID. However, the paper doesn't provide a direct measurement of "spatial prior leakage" β€” it infers the mechanism from the combination of "direct alignment uses spatial interpolation" and "direct alignment performs worse." Alternative explanations are possible: direct alignment might simply be a harder optimization problem (forcing two differently-structured representation spaces to match), or the interpolation might introduce artifacts. The 2D tokenizer results in Table 2 provide partial corroboration: for 2D tokenizers where spatial structure is desirable, direct alignment helps (gFID 6.06 vs. 12.51 baseline). But a more targeted experiment β€” e.g., measuring whether direct alignment causes latent tokens to become more spatially localized (via attention map analysis) β€” would strengthen the mechanistic claim.

Claim 4: "EOSTok achieves state-of-the-art FID of 1.48 without guidance." This claim is true within the scope of Table 3's comparison, but with important qualifications:

(a) No guidance comparison is not standardized. The paper compares EOSTok's unguided FID (1.48) against other methods' unguided FIDs, but this is an apples-to-oranges comparison because most methods in the table are evaluated primarily with guidance (diffusion models typically report CFG results). The fair comparison is really between EOSTok-H with guidance (1.38) and other methods with guidance, where it is competitive but not clearly state-of-the-art over Lightning-DiT-XL (1.35) and REPA-XL/2 (1.42). The paper's emphasis on "without guidance" is partly a reflection of EOSTok's strength (its unguided quality is unusually high β€” the gap between unguided and guided is only 0.10 FID for EOSTok-H vs. much larger gaps for diffusion models) and partly a framing choice.

(b) Training budget differences are unaccounted for. EOSTok-H trains for 400 epochs on 8 H100s. Diffusion models like DiT-XL/2 typically train for much longer (DiT-XL/2 was trained for 7M iterations at batch size 256, approximately 1400 ImageNet epochs). The paper does not provide a FLOPs-matched training comparison, making it unclear whether EOSTok's efficiency advantage at inference time (Table 8) comes at the cost of higher training cost, or vice versa. The training overhead numbers in Table 7 (15-19% per iteration over two-stage training) only compare EOSTok variants to each other, not to diffusion model training costs.

(c) The NTP loss vs. gFID disconnect complicates model selection. Figure 5b shows that NTP loss continues to decrease while gFID improves (Figure 5a), but the paper also shows (Table 1, Figure 5b) that NTP loss and gFID can move in opposite directions. This means NTP loss on the validation set is not a reliable early-stopping criterion β€” practitioners cannot use validation NTP loss to decide when to stop training, and must periodically evaluate gFID (which requires generating images and computing FID against the full validation set). The paper doesn't address how model selection should be done in practice.

Missing experiments that would strengthen the paper:

  • Multiple training runs with error bars. The differences between EOSTok-H with K=4096 (gFID 1.48) and K=16384 (gFID 1.51) are small enough to potentially be within run-to-run variance. Without multiple seeds or confidence intervals, it's difficult to know whether observed differences (e.g., the 0.18 FID gap between codebook sizes for EOSTok-L, or the 0.26 gap between implicit and direct alignment gFID in Table 2) are reliable.

  • Ablation on the interaction between APR loss and VFM alignment. Table 2 shows decoder alignment + implicit alignment achieves gFID 3.32. What is the gFID of implicit alignment alone (without decoder alignment)? What about APR loss alone (without any VFM alignment)? These combinatorial ablations would clarify which components are additive vs. redundant. The current results show each component added to a baseline, but the baseline changes between tables (Table 1 baseline: gFID 3.82; Table 2 baseline: gFID 12.27), likely because Table 2 uses a different training duration (50 epochs vs. fully trained). A systematic ablation matrix would be more informative.

  • Measuring spatial structure in the latent space. The paper claims that direct alignment leaks 2D spatial prior while implicit alignment doesn't. This could be tested: measure the spatial localization of latent tokens by analyzing which image regions they attend to (via attention rollout or gradient-based attribution). If direct alignment causes latent tokens to focus on specific spatial regions (consistent with the interpolated VFM features) while implicit alignment doesn't, this would be direct evidence for the mechanism.

  • Training budget comparison with diffusion models. Given the paper's emphasis on inference efficiency (Table 8), a training cost comparison would be valuable. How many GPU-hours to train EOSTok-H vs. DiT-XL/2? If EOSTok requires substantially more training compute to achieve its inference advantage, the overall cost-benefit depends on the expected inference volume (number of images generated over the model's lifetime), which is an important practical consideration the paper doesn't address.

  • Diversity metrics beyond IS. The paper reports IS (which measures both quality and diversity) but doesn't report Precision/Recall or coverage metrics that would separately quantify sample diversity vs. fidelity. For AR models, which can potentially suffer from exposure bias (error accumulation during autoregressive sampling), diversity metrics are important to ensure the model isn't memorizing or producing limited variations of training examples.

Assessment of the codebook collapse diagnosis. The paper's identification of codebook collapse under NTP loss supervision (Table 1, Figure 3) is a genuine contribution β€” it's a failure mode that the field hadn't documented because two-stage training inadvertently prevented it. However, the diagnosis raises a question the paper doesn't fully answer: why does the APR loss prevent collapse? The paper's explanation (Section 3.2) is that APR loss "regulates next token prediction loss to be meaningful" by providing "end-to-end generative supervision to the tokenizer directly from the pixel space." But the mechanism isn't fully explained. The APR loss is essentially an additional reconstruction loss applied to the AR-predicted tokens. Why doesn't the tokenizer learn to ignore it β€” e.g., by making the AR model's predictions trivially easy while the decoder learns to reconstruct from a collapsed latent space? The answer may lie in the fact that the AR model's teacher-forced predictions G_ΞΈ(z_q) differ from the ground-truth z_q, so the APR loss sees a different input to the decoder than the reconstruction loss. If the latent space collapses to few tokens, the AR predictions will be nearly identical to the ground truth (since there are few possible tokens), and the APR loss provides no additional signal. The fact that APR loss helps suggests that maintaining a diverse latent space is the only way to satisfy the APR loss, because the decoder cannot reconstruct diverse images from AR predictions if the latent space is collapsed. This is plausible but warrants more analysis.

Assessment of scaling claims. Figure 5a demonstrates that larger models achieve better gFID, and Figure 5b shows that larger models achieve lower NTP loss. However, the paper doesn't fit a scaling law (power law relating gFID to model size or compute) β€” it only shows that the curves are ordered by size. Without a functional form, it's difficult to extrapolate whether further scaling would continue to yield improvements or whether diminishing returns have set in. The fact that the curves in Figure 5a don't appear to be saturating at 2M iterations suggests headroom remains, but how much is unclear. The codebook size results (Figure 5a) provide a more specific scaling insight: the gap between K=4096 and K=16384 narrows with scale, suggesting that vocabulary size is a bottleneck for small models but not for large ones. This is a practically useful finding β€” it means practitioners training large AR image models can use larger codebooks without penalty, simplifying hyperparameter choices.

6. Limitations and Trade-offs

The AR Model's Autoregressive Prediction Accuracy Is a Misleading Proxy for Generation Quality β€” and the Paper Offers No Reliable Surrogate for Model Selection

The paper's central diagnostic finding is that next-token-prediction (NTP) loss does not determine generation quality. This is demonstrated in multiple ways: Vanilla E2E training achieves 30.2% AR accuracy but gFID 8.01, while the full EOSTok method achieves 11.9% AR accuracy with gFID 3.32 (Table 1); separately-trained models achieve lower NTP loss than end-to-end trained models but worse gFID (Figure 5b); direct substitution achieves the highest AR accuracy (12.1%) but substantially worse gFID than implicit alignment (4.89 vs. 3.32, Table 2). The paper treats this disconnect as a problem it solves β€” the APR loss bridges the gap β€” but it does not address the practical consequence: if NTP loss does not track generation quality, how should practitioners perform model selection, early stopping, or hyperparameter tuning?

In standard AR language modeling, validation perplexity (or NTP loss) is the primary signal for deciding when to stop training, which checkpoint to deploy, and whether a training run is on track. The paper provides no replacement for this signal. The gFID metric (which requires generating thousands of images and computing FID against a reference set) is prohibitively expensive to compute frequently during training. The paper monitors gFID throughout training (Figure 5a,c) for experimental analysis, but this is a retrospective evaluation, not a practical training signal. Without a reliable validation metric that correlates with final generation quality, practitioners training EOSTok-style models face a model selection problem: they must either periodically evaluate gFID (expensive, and the paper does not specify how often or at what computational cost), or guess when to stop based on training iterations.

The paper's own evidence shows that this is not a theoretical concern. Table 10 shows that APR loss weight Ξ»_APR = 0.5 and 2.0 produce nearly identical rFID (1.02 vs. 1.05) and similar code usage (presumably high in both cases), but gFID differs meaningfully (3.52 vs. 3.34). A practitioner choosing between these hyperparameters without computing gFID would have no basis for the decision. Similarly, nested dropout at p=0.5 vs. p=1.0 (Table 11) shows a tradeoff between AR accuracy and gFID that cannot be resolved without the expensive gFID measurement.

The paper partially acknowledges the measurement problem in Section 3.2 ("the NTP objective is defined on discrete token space that is constantly changing during training, and it cannot reflect the final generation quality in the pixel space"), but treats it as a justification for the APR loss rather than as a residual limitation of the framework. The APR loss itself might serve as a validation metric β€” does lower APR loss during training correlate with better final gFID? The paper does not report this correlation. If APR loss tracks gFID reliably, it could serve as the missing model selection signal, but this is not established. If it does not, then end-to-end training has solved the training problem (preventing codebook collapse) but created a monitoring problem (no reliable validation metric) that two-stage training did not have (since in two-stage training, the tokenizer can be validated by rFID and the AR model by NTP loss separately, even if neither directly predicts gFID).

The mitigation status is: unaddressed. The paper provides no guidance on model selection during training, no correlation between APR loss and final gFID, and no practical recommendation for practitioners beyond "train for 400 epochs." Given that Figure 5a shows gFID still improving at 2M iterations, the choice of training duration is consequential, and practitioners with different compute budgets need a principled way to decide when to stop.

Difficulty Estimation for the Tokenizer's Own Training: The End-to-End Framework Assumes the AR Model Can Learn from Scratch Alongside a Moving Tokenizer Target

The paper's joint training approach trains the tokenizer and AR model simultaneously from random initialization. This means the AR model is learning to predict tokens from a latent space that is itself changing throughout training β€” the "vocabulary" and "grammar" of the token sequences shift as the tokenizer's encoder and codebook evolve. The paper demonstrates that this works empirically (the models converge and achieve strong results), but it does not address whether this non-stationarity imposes hidden costs: slower convergence, higher variance, or sensitivity to initialization and learning rate ratios between the tokenizer and AR model.

This is not a hypothetical concern. The paper uses different learning rate schedules and optimizer settings for the tokenizer and AR model (Adam Ξ²β‚‚ = 0.999 for tokenizer, 0.95 for AR model; Table 9), and different NTP loss weights for different model sizes (Ξ»_NTP = 0.1 for S/B/L, 0.01 for H; Table 9). These hyperparameters were presumably tuned, but the paper does not report the sensitivity of results to the ratio of tokenizer learning rate to AR model learning rate, or to the NTP loss weight relative to the reconstruction loss weight. If the tokenizer learns too quickly relative to the AR model, the AR model's gradients (through NTP and APR losses) may be computed against a tokenizer that has already moved on, providing noisy or stale signals. If the AR model learns too quickly, it may overfit to an immature latent space and provide poor feedback to the tokenizer.

The consequence is that practitioners attempting to reproduce or extend EOSTok face an undertested hyperparameter sensitivity: the relative learning dynamics of two jointly-trained components with different architectures, different loss landscapes, and different optimal learning rates. The paper's four model sizes (S, B, L, H) use different configurations (Table 9), but these are presented as fixed choices rather than the result of a systematic sweep. There is no ablation showing that results are robust to, say, halving or doubling the tokenizer learning rate relative to the AR model learning rate. If the framework is sensitive to these choices, transfer to new datasets, resolutions, or model architectures may require expensive re-tuning.

A related concern is training length. The paper trains for 400 epochs (~2M iterations), and Figure 5a shows gFID still declining at this point for all model sizes. It is unclear whether the joint training dynamic reaches a stable equilibrium or whether the tokenizer and AR model continue to co-adapt indefinitely. If they co-adapt at different rates, extended training might cause one component to overfit relative to the other, leading to degradation that would not occur in separate training (where the tokenizer is frozen, preventing degradation of the AR model's target distribution).

The paper partially acknowledges the computational overhead of joint training (Table 7: 15-19% more GFLOPs per iteration due to the extra decoder forward pass for APR loss and VFM feature extraction), but does not address the potential need for more training iterations to reach convergence (due to the non-stationary target) or the additional hyperparameter tuning burden. The mitigation status is: unaddressed. No sensitivity analysis on relative learning rates, no comparison of convergence speed (in terms of wall-clock time or total FLOPs to reach a given gFID) between joint and separate training, and no guidance on hyperparameter selection for new settings.

Single Benchmark, Single Dataset: All Results Are on ImageNet Class-Conditional Generation with No Evidence of Transfer to Other Domains or Tasks

Every experiment in the paper uses ImageNet-1K at 256Γ—256 resolution for class-conditional image generation. The 512Γ—512 experiment (Table 13) is still ImageNet, still class-conditional, and still the same model architecture with the same training recipe. The paper does not evaluate on text-to-image generation (which would require the AR model to condition on text embeddings, testing whether the end-to-end training framework extends to cross-modal conditioning), on other image datasets (e.g., FFHQ for faces, LSUN for scenes), on other visual generation tasks (e.g., image inpainting, super-resolution, or class-unconditional generation), or on other domains where 1D tokenization might be beneficial (e.g., video generation, where the temporal dimension could naturally align with the sequential latent structure).

This matters because ImageNet class-conditional generation at 256Γ—256 is a specific problem with specific properties: 1,000 relatively distinct categories, centered objects, limited compositional complexity, and a moderate resolution where patch-level details are important but global structure is relatively simple. The paper's key innovations β€” the APR loss to prevent codebook collapse, implicit semantic alignment to inject VFM knowledge without spatial leakage β€” may behave differently in settings where:

  • The base model's pass@1 is lower: On more complex datasets (e.g., text-to-image generation with diverse prompts), the AR model may struggle to achieve sufficient prediction accuracy for the APR loss to provide useful feedback. The APR loss relies on the AR model's teacher-forced predictions being at least partially correct; if the AR model's predictions are essentially random, the decoded images will be meaningless, and the APR loss may provide no useful gradient. The paper does not explore this regime.

  • The VFM's features are less informative: DINOv2 is trained on a diverse internet-scale dataset and provides strong semantic features for ImageNet classes. On specialized domains (medical imaging, satellite imagery, scientific data), available VFMs may provide weaker or misaligned features. The paper shows that SigLIP2 slightly outperforms DINOv2 (Table 12), but both are general-purpose VFMs trained on natural images β€” the robustness to domain shift in the VFM is untested.

  • Higher resolution requires longer sequences: The 512Γ—512 experiment (Table 13) keeps the sequence length at 256 and patch size at 16, meaning the latent tokens must compress 1024 patches into 256 tokens (4Γ— compression). The strong result (gFID 1.98) is encouraging, but it's unclear whether this scales further β€” 1024Γ—1024 images would require 16Γ— compression at 256 tokens, which may exceed the information capacity of the latent space regardless of training methodology.

The authors do not claim broader applicability β€” the paper is scoped to ImageNet generation β€” but the framing in Section 1 positions EOSTok as a general approach to 1D tokenizer training for autoregressive image generation. The gap between the claimed general approach and the single-dataset evaluation is significant for practitioners considering whether to adopt the method for other tasks.

Mitigation status: not addressed. The paper does not discuss domain transfer, does not evaluate on any non-ImageNet task, and the 512Γ—512 experiment is the only variation from the main experimental setup. The conclusion states that "our method is effective and easily scalable" (Section 1), but the scaling claim is supported only by scaling model size on the same dataset, not by scaling to new domains or tasks.

The VFM Computational Cost Is Substantial and Not Amortized in Efficiency Comparisons

Every training iteration of EOSTok requires a forward pass through a frozen DINOv2-ViT-L model to extract semantic features for representation alignment. As shown in Table 7, the VFM embedder costs 162 GFLOPs per forward pass β€” comparable to the AR model's 162 GFLOPs (for EOSTok-L) and the encoder's 91 GFLOPs. The VFM feature extraction accounts for 27% of the total training GFLOPs for EOSTok-L (162 out of 597) and 14% for EOSTok-H (162 out of 1130). This cost is purely an auxiliary training signal β€” the VFM is not used at inference time, so it provides no benefit during deployment.

The paper compares training overhead of end-to-end training vs. two-stage training (Table 7: 15-19% overhead), but this comparison includes the VFM cost in both the end-to-end and two-stage baselines (since the two-stage baseline in Table 2 includes decoder alignment, which also requires VFM features). The more relevant comparison for a practitioner deciding whether to adopt EOSTok is: what is the total training cost to achieve a given gFID, compared to methods that do not use VFM alignment? The paper does not provide this comparison. The "Baseline" in Table 2 (gFID 12.27, no VFM) and the various VFM-augmented configurations are all trained for 50 epochs at the same batch size, so the VFM cost is an additional per-iteration expense on top of the baseline. If a non-VFM method could achieve comparable gFID by training longer or with a larger model (trading VFM computation for additional model computation), the effective efficiency of the VFM alignment would be lower than the per-iteration comparison suggests.

During inference, the VFM is not needed β€” only the AR model and the tokenizer's decoder run (Table 8). So the training cost of the VFM is amortized over the number of images generated during deployment. For high-volume deployment (millions or billions of generated images), this amortization makes the VFM training cost negligible. But for research, fine-tuning, or low-volume deployment, the VFM training cost may dominate, and the paper provides no guidance on whether the VFM alignment is still worthwhile in these regimes.

A related concern: the choice of VFM is a fixed architectural dependency. The paper uses DINOv2-ViT-L, a specific pretrained model with a specific architecture and feature dimension. If a practitioner wants to use a different VFM (better, newer, or domain-specific), they must retrain EOSTok with that VFM's features. There is no evidence that the training framework is robust to VFM choice beyond the single SigLIP2 comparison (Table 12), which shows a modest improvement. Changing the VFM architecture (e.g., to a larger model, a different patch size, or a different training objective) might require re-tuning the alignment loss weight, the MLP projector architecture, or the layer at which decoder alignment is applied.

Mitigation status: partially acknowledged. The paper reports the VFM GFLOPs in Table 7, so the cost is transparent, but it does not discuss the amortization question, compare total training FLOPs to VFM-free methods, or evaluate how results change if a smaller/cheaper VFM is used. A practitioner reading the paper would know the VFM costs 162 GFLOPs per iteration but would not know whether that cost is necessary or whether a smaller VFM (e.g., DINOv2-ViT-S) would suffice.

The Framework Adds Multiple Loss Terms and Hyperparameters Without a Clear Recipe for Balancing Them on New Problems

The complete EOSTok objective (Equation 8) contains six distinct loss terms with associated weights: reconstruction L2 (Ξ»=1.0), reconstruction LPIPS (Ξ»=1.0), GAN loss (Ξ»=0.1), LeCam regularization (Ξ»=0.05), implicit alignment (Ξ»=1.0), decoder alignment (Ξ»=1.0, absorbed into Ξ»_sem), APR L2 (Ξ»=1.0), APR LPIPS (Ξ»=1.0), and NTP loss (Ξ»=0.1 for S/B/L, 0.01 for H). Additionally, there are hyperparameters for: nested dropout rate (0.5 or 1.0), sequence length (256), codebook size (4096), IBQ temperature (1.0), commitment loss weight (1e-3), entropy weight (0.01), learning rates for tokenizer, AR model, and discriminator, Adam Ξ²β‚‚ values for tokenizer vs. AR model, EMA decay rate, and training duration.

The paper ablates some of these choices β€” APR loss weight (Table 10), nested dropout rate (Table 11), sequence length (Table 5), codebook size (Table 6), choice of VFM (Table 12) β€” but the ablations are largely one-dimensional: vary one hyperparameter while holding others fixed at their presumably-optimal values. There is no evidence that the optimal values of these hyperparameters are independent. For example, the optimal APR loss weight might depend on the NTP loss weight, the codebook size, and the model scale; the optimal nested dropout rate might depend on the sequence length. The paper's finding that Ξ»_NTP drops from 0.1 to 0.01 when scaling from L to H (Table 9) is a concrete example of hyperparameter-scale interaction β€” it implies that the loss weights are not fixed across model sizes, but the paper provides no principle for how they should be adjusted.

This creates a practitioner burden: adopting EOSTok for a new dataset, resolution, or model architecture requires navigating a large hyperparameter space with potentially coupled dimensions. The paper's ablation tables provide starting points, but a practitioner cannot know whether the ImageNet-optimized hyperparameters transfer to a new setting without running their own expensive sweeps. The complexity is compounded by the model selection problem (Limitation 1): since NTP loss is not a reliable validation metric, hyperparameter sweeps must be evaluated using gFID, which is expensive to compute.

The paper does not attempt to simplify the loss landscape β€” for example, by showing that some loss terms can be removed without penalty, or that a single "generation quality" loss can replace the APR L2 + APR LPIPS + NTP combination. The NTP loss weight of 0.01 for EOSTok-H is so small that one might ask whether it could be zero (i.e., relying entirely on APR loss for generative supervision). The paper does not test this.

Mitigation status: partially addressed through ablations, but the ablations are incomplete (not combinatorial) and specific to the ImageNet 256 setting. The paper does not provide a hyperparameter selection methodology, a sensitivity analysis showing which hyperparameters matter most, or a reduced-complexity configuration for new users. The fact that EOSTok-H required a different NTP loss weight than EOSTok-L (discovered presumably through tuning) suggests that scaling to new model sizes will require additional tuning that the paper does not guide.

Inference Latency vs. Throughput: The Autoregressive Generation Paradigm Has Fundamental Serial Dependency That Limits Parallelism

EOSTok generates images by sampling 256 tokens sequentially from the AR model, where each token depends on all previously generated tokens. This sequential dependency is inherent to autoregressive generation and cannot be parallelized β€” generating token n requires having already generated tokens 1 through nβˆ’1. The paper emphasizes the FLOPs advantage over diffusion models (Table 8: EOSTok-H requires ~552 GFLOPs vs. ~59,900 GFLOPs for DiT-XL/2 with 250 steps, approximately a 100Γ— reduction), but total FLOPs is not the same as wall-clock latency. Diffusion models perform hundreds of sequential denoising steps, but each step applies the full model to all pixels in parallel. AR models perform fewer sequential steps (256 vs. 250-1000), but each step processes only one token and must wait for the previous step to complete.

The paper reports that "EOSTok-H can generate about 10.5 images per second on a single H100" with batched image generation and KV-caching (Section A.3). This is a throughput number (images per second in batch mode), not a latency number (time to generate one image from a cold start). The distinction matters for interactive applications: a user waiting for a single image experiences latency equal to the time to generate 256 tokens sequentially, which is bounded by the per-token inference time times 256 (minus any KV-cache speedup, but the sequential dependency remains). In contrast, a diffusion model with 250 steps might have higher total FLOPs but can potentially be optimized through techniques like progressive distillation or consistency models to fewer steps, reducing latency below what the FLOPs ratio would suggest.

The paper does not report single-image generation latency, does not compare latency to diffusion models at the same batch size, and does not discuss whether the throughput advantage (10.5 images/second in batch mode) translates to a latency advantage for interactive use cases. This is a significant omission for practitioners evaluating whether to deploy EOSTok in applications where users expect sub-second generation times.

Additionally, the AR model's token-by-token generation creates a memory bandwidth bottleneck during inference: each step loads the full model weights and KV-cache from memory, processes one token, and writes the updated KV-cache back. For large models (EOSTok-H has 644M AR parameters), the per-token overhead of weight loading may dominate over the actual computation, making the effective utilization of the GPU's compute capacity low. Diffusion models, by processing all pixels in parallel, can achieve higher compute utilization per step. The paper's FLOPs comparison (Table 8) assumes ideal utilization, which may not reflect real-world throughput on actual hardware.

Mitigation status: unaddressed. The paper provides a throughput number (10.5 images/second) and a FLOPs comparison (Table 8), but does not report single-image latency, does not discuss the latency-throughput tradeoff, and does not compare end-to-end generation time (including decoding) to diffusion model sampling time on the same hardware. The claim that EOSTok is "20 to 100 times faster than the DiT-XL/2 model" (Section A.3) is based on FLOPs, not measured wall-clock time, and should be interpreted as an upper bound on the speedup.

7. Implications and Future Directions

How This Work Changes the Landscape

EOSTok is best understood as a training methodology intervention rather than an architectural breakthrough β€” it demonstrates that the two-stage paradigm (train tokenizer for reconstruction, freeze it, train generative model on frozen tokens) was not a neutral engineering choice but an active constraint on what tokenizers could learn. The paper's diagnostic experiment with vanilla end-to-end training (Table 1) reveals that this constraint was masking a fundamental instability: the moment a tokenizer is exposed to next-token-prediction loss without counterbalancing pixel-space feedback, it collapses its codebook to a small subset of tokens, destroying both reconstruction and generation quality. The two-stage paradigm wasn't just suboptimal β€” it was a safety mechanism preventing a failure mode the field didn't know existed.

This changes the landscape in three specific ways:

1. It reframes the reconstruction-generation dilemma from a tradeoff to a principal-agent problem. Prior work (TiTok, FlexTok, Semanticist, VA-VAE) treated the tension between reconstruction fidelity and generation quality as an optimization tradeoff β€” more tokens improve reconstruction but make generation harder, so you choose a point on the Pareto frontier. EOSTok reveals a deeper dynamic: the NTP loss is an agent that will exploit any degree of freedom to minimize its objective, even if that means destroying the latent space's information capacity. The tokenizer is not passively struggling with a hard problem; it is actively gaming the metric when given control. This means the field's focus should shift from "how do we balance reconstruction and generation losses?" to "how do we design generation objectives that are ungameable?" β€” that is, objectives whose minimum genuinely corresponds to a useful latent space. The APR loss is one such design (pixel-space reconstruction cannot be satisfied by codebook collapse), but the principle generalizes: any generation objective that operates purely in a learned discrete space is vulnerable to this form of specification gaming.

2. It establishes that the 1D tokenizer's sequential structure is learned, not just architected, and that this learning is what matters for AR generation. Table 4's result β€” that a frozen EOSTok tokenizer produces token sequences where the original ordering achieves 2.5Γ— better gFID than reversed ordering β€” is not just an ablation. It demonstrates that end-to-end training causes the tokenizer to organize visual information along the temporal axis in a way that supports sequential prediction. This is an emergent property: the reconstruction loss is permutation-invariant, and the causal attention mask on query tokens provides only a weak inductive bias toward sequential structure. Yet the tokenizer learns to encode images such that earlier tokens carry information that genuinely helps predict later tokens, and reversing this order breaks the structure catastrophically (gFID jumps from 4.10 to 10.27 on reversed ordering).

This finding has two consequences for the field. First, it validates the entire 1D tokenizer research direction: the hypothesis that non-spatial latent representations can support AR modeling is correct, but only if the tokenizer is trained with objectives that reward sequential predictability. Second, it reframes the debate between "fix the tokenizer" (1D approaches) and "fix the generation paradigm" (MaskGIT, VAR, MAR). If a tokenizer can learn to make any ordering predictable (as Table 4 suggests), then the problem with 2D raster-scan AR generation isn't the raster order per se β€” it's that two-stage training never teaches the tokenizer to care about predictability in that order. End-to-end training might make 2D tokenizers with raster-order AR generation work substantially better, collapsing the distinction between the two camps. The paper's 2D tokenizer results with alignment (Table 2: gFID 6.06 with guidance-level methods) hint at this, but don't explore it systematically.

3. It provides a concrete mechanism β€” implicit semantic alignment β€” for transferring knowledge from vision foundation models to non-spatial latent representations without leaking spatial structure, which opens up a design space that was previously unexplored. Before EOSTok, the dominant paradigm for VFM alignment (REPA, VA-VAE) assumed a spatial correspondence between the generative model's internal representations and the VFM's patch-level features. This worked for 2D latent spaces (where the spatial correspondence exists naturally) but was inapplicable or harmful for 1D tokenizers (where imposing spatial structure undermines the architecture's purpose). EOSTok's implicit alignment strategy β€” aligning hidden patch embeddings rather than latent tokens β€” demonstrates that the injection point matters more than the alignment objective. The patch embeddings are a safe intermediary: they can absorb spatial semantic information without forcing the latent tokens (which attend to them through learned attention patterns) to inherit that spatial structure. The latent tokens remain free from 2D constraints while benefiting from semantically enriched input features.

This changes the conceptual toolkit for any research that combines non-spatial representations with spatially-structured external knowledge. The principle β€” inject spatial information at the intermediate feature level where it's natural, not at the latent level where it's actively harmful β€” extends beyond vision tokenizers. Multi-modal models that compress spatially-structured inputs (video, audio spectrograms, LiDAR) into sequential latent codes face the same tension, and the implicit alignment strategy provides a template for addressing it.

Reconciliation of prior contradictions. The paper resolves an apparent tension in the literature that wasn't explicitly articulated as a contradiction but was visible in results: why do some VFM alignment methods help generation (REPA, VA-VAE) while others don't? The answer, per Table 2, is that alignment at the latent space level (direct alignment, Figure 4a) can help reconstruction but hurt generation when the latent space is non-spatial, because it leaks a structural prior that the architecture was designed to escape. Earlier papers applying VFM alignment to diffusion models (which use 2D latent spaces where spatial structure is beneficial) saw uniform improvements; a naΓ―ve application of the same technique to 1D tokenizers would have produced negative results and been abandoned. EOSTok explains why and provides the fix.

Directions that become more attractive. The paper's success with end-to-end training makes joint optimization of tokenizer and generative model the default approach going forward β€” the two-stage paradigm now carries a burden of proof. It also makes 1D tokenization at moderate sequence lengths (64-256 tokens, not the aggressive 32-token compression of TiTok) a credible alternative to 2D tokenization, since the reconstruction-generation tradeoff can be managed through end-to-end feedback. This opens up AR image generation using standard language model architectures (the "AR model" in EOSTok is essentially a Llama) without the specialized generation paradigms (masked, multi-scale) that prior work resorted to.

Directions that become less attractive. The paper's finding that lookahead-style search (represented by the various alignment variants applied to the encoder, only one of which works well) is fragile compared to end-to-end pixel-space feedback suggests that complex auxiliary objectives on the latent space are unreliable compared to direct pixel-space supervision. The field should be skeptical of methods that optimize token-level metrics without pixel-space grounding β€” EOSTok shows that even the most natural token-level objective (NTP loss) can be catastrophically gamed. This also suggests that pure discrete-space training (e.g., training AR models on discrete tokens without any continuous decoder feedback) may have fundamental limitations that end-to-end approaches avoid.

Follow-Up Research This Work Enables

Does the APR loss correlate with gFID well enough to serve as a validation metric, solving the model selection problem? The paper's central finding is that NTP loss does not track generation quality, but it does not establish whether the APR loss β€” which measures pixel-space reconstruction from AR predictions β€” does. A targeted experiment would monitor APR loss on a held-out validation set throughout training for multiple model sizes (S through H) and compute the rank correlation (Spearman's ρ or Kendall's Ο„) between APR loss and final gFID at each checkpoint. If APR loss tracks gFID with high correlation (ρ > 0.9), it becomes the practical validation metric that the paper currently lacks β€” practitioners could use APR loss for early stopping and hyperparameter selection without expensive periodic FID evaluation. If the correlation is weak or inconsistent across model scales, it would indicate that even pixel-space feedback from teacher-forced predictions is an imperfect proxy for autoregressive sampling quality, pointing to exposure bias (the mismatch between teacher-forced training and autoregressive inference) as the residual gap. This experiment requires no architectural changes β€” only logging APR loss on a validation set during the training runs the paper already performs.

Can end-to-end training rescue 2D tokenizers with raster-scan AR generation, collapsing the distinction between 1D and 2D approaches? The paper's 1D tokenizer is designed to eliminate 2D spatial structure because that structure is "fundamentally misaligned with unidirectional AR factorization" (Section 1). But Table 4 suggests that end-to-end training can teach a tokenizer to make any ordering predictable. What happens if you take a standard 2D VQ-VAE tokenizer (e.g., the LlamaGen tokenizer producing a 16Γ—16 grid) and train it end-to-end with APR loss and implicit alignment? The hypothesis: the 2D tokenizer would learn to structure its 256 grid tokens such that raster-scan order becomes genuinely sequential (earlier tokens predict later tokens), despite the spatial adjacency structure that normally creates bidirectional dependencies. If this works β€” achieving gFID competitive with EOSTok while retaining the architectural simplicity of standard 2D tokenizers β€” it would unify the two lines of research and simplify deployment (no need for specialized 1D architectures). If it fails (gFID remains substantially worse than 1D), it would confirm that the 2D grid imposes an architectural constraint that even end-to-end training cannot overcome, validating the 1D approach as structurally necessary. The experiment requires re-implementing the EOSTok training pipeline with a standard 2D VQ-VAE tokenizer backbone β€” codebook collapse under NTP loss (the paper's Table 1 failure mode) would be the first diagnostic to check.

What is the training cost vs. inference cost Pareto frontier between EOSTok-style AR models and diffusion models, accounting for VFM training overhead amortized over deployment volume? The paper shows EOSTok-H has ~100Γ— lower inference FLOPs than DiT-XL/2 (Table 8), but the training cost comparison is not provided. A FLOPs-matched comparison would sweep: (a) EOSTok models trained for different durations (50, 100, 200, 400 epochs) vs. (b) DiT-style diffusion models trained for durations that equalize total training FLOPs (accounting for EOSTok's VFM overhead, ~162 GFLOPs/iteration, and the 15-19% end-to-end overhead from Table 7). For each training budget, measure gFID. Then, for a range of deployment volumes (1K, 1M, 1B generated images), compute total cost = training FLOPs + inference FLOPs Γ— deployment volume. Plot the Pareto frontier: which method achieves the best gFID at each total cost? This would answer the question the paper raises but doesn't settle: for what deployment scenarios is EOSTok's training overhead (VFM features, end-to-end joint optimization) justified by inference savings? If EOSTok dominates the frontier at all deployment volumes above some threshold, it becomes the default choice for production systems. If diffusion models dominate at low deployment volumes (where training cost matters more), the choice depends on expected usage. The experiment requires training diffusion baselines at matched FLOPs budgets, which is expensive but well-defined.

Does the implicit alignment strategy generalize to non-visual modalities where spatial/temporal structure exists but the latent representation should be non-spatial? The paper's key insight β€” inject spatially-structured external knowledge at the intermediate feature level, not the latent level β€” applies to any domain where a spatially- or temporally-structured input must be compressed into a sequential latent code for autoregressive modeling. Concrete candidates: (a) Video generation: video has spatial (per-frame) and temporal structure; a 1D tokenizer that compresses video frames into sequential tokens could use implicit alignment on per-frame patch embeddings to inject semantic information from a video VFM (e.g., VideoMAE features) without forcing the latent tokens into a fixed spatiotemporal grid. (b) Audio generation from spectrograms: spectrograms have time-frequency structure; implicit alignment on time-frequency patch embeddings from an audio VFM (e.g., CLAP audio encoder) could inject semantic information (phoneme content, speaker identity) without spatializing the 1D latent sequence. (c) Molecular generation: 3D molecular structures have spatial (atomic coordinates) and topological (bond) structure; implicit alignment on atom-level embeddings from a pretrained molecular VFM (e.g., a 3D GNN) could inject chemical knowledge into a 1D SMILES-like latent sequence. The hypothesis: in each domain, direct alignment (forcing latent tokens to match spatially-interpolated VFM features) will degrade generation quality compared to implicit alignment (aligning intermediate patch/atom embeddings), replicating the pattern in Table 2. A positive result would establish the implicit alignment strategy as a general principle for multi-modal autoregressive modeling; a negative result in some domains would identify boundary conditions (e.g., domains where the VFM features are too weak or misaligned for the alignment to help).

Can the learned sequential structure in the latent space (Table 4) be explicitly characterized β€” do earlier tokens encode coarse semantic categories, object identities, or global layout, and later tokens encode fine details, following a coarse-to-fine hierarchy? The paper shows that the latent space has learned order-sensitive structure (reversing order hurts gFID by 2.5Γ—), but does not characterize what that structure is. A probing experiment would: (a) train linear classifiers to predict ImageNet class labels from individual latent token positions (tokens 1-256), measuring whether earlier tokens are more class-predictive (suggesting they encode global semantic information); (b) ablate individual tokens or token ranges during decoding and measure the effect on reconstruction (e.g., zero out tokens 1-32 vs. 225-256 and measure which causes larger rFID degradation); (c) visualize what image regions each latent token attends to (via attention rollout from the decoder's cross-attention to latent tokens), testing whether earlier tokens attend to global structure and later tokens to local details. This would transform the qualitative claim ("the latent space becomes more AR-generation-friendly") into a quantitative characterization of the learned representation. If earlier tokens encode coarse semantic information and later tokens encode fine details, it explains why the sequential structure supports AR modeling (coarse information is more predictable and provides context for predicting fine details) and connects EOSTok to the nested dropout results (Table 11), where forcing coarse-to-fine structure improves AR accuracy. If the structure is different β€” e.g., tokens encode orthogonal basis functions or frequency components β€” it would reveal an unexpected organizational principle and suggest different regularization strategies.

What is the sensitivity of end-to-end training to the relative learning dynamics of tokenizer and AR model, and can the hyperparameter burden be reduced? The paper uses different Adam Ξ²β‚‚ values (0.999 for tokenizer, 0.95 for AR model) and different NTP loss weights (0.1 for S/B/L, 0.01 for H) without ablating these choices. A systematic sweep over: (a) the ratio of tokenizer learning rate to AR model learning rate (e.g., 0.25Γ—, 0.5Γ—, 1Γ—, 2Γ—, 4Γ—), (b) the NTP loss weight across scales (to test whether the 0.1 β†’ 0.01 shift for H is necessary or coincidental), and (c) the APR loss weight across scales, would establish how sensitive final gFID is to these choices. If the sensitivity is low (gFID varies by < 0.2 across a broad range), the framework is robust and the paper's fixed hyperparameters are reasonable defaults. If sensitivity is high, it would indicate that joint training requires careful balancing of learning dynamics, making adoption more difficult and motivating research into automated balancing methods (e.g., gradient normalization, adaptive loss weights). A negative result (high sensitivity) would also explain why prior informal attempts at joint training may have failed β€” not because the concept is wrong, but because the optimization is brittle β€” and would focus future work on stabilization techniques rather than new loss functions.

Practical Applications and Downstream Use Cases

Class-conditional image generation at scale with strict latency budgets. EOSTok-H generates approximately 10.5 images per second on a single H100 GPU with batched inference (Section A.3), while achieving gFID 1.48 without guidance β€” competitive with the best diffusion models that require 20-100Γ— more FLOPs per image. For applications that need to generate large volumes of class-conditional images (e.g., synthetic data generation for training downstream classifiers, data augmentation for rare classes, or content creation pipelines where a user specifies a category and expects rapid results), EOSTok's throughput advantage is directly actionable. A production system generating 1 million images would require approximately 26.5 GPU-hours on H100 with EOSTok-H (at 10.5 images/second), versus roughly 500-2,500 GPU-hours with DiT-XL/2 (depending on sampling steps). The cost differential β€” potentially 20-100Γ— β€” makes EOSTok the pragmatic choice for any high-volume class-conditional generation pipeline, provided the ImageNet class distribution matches the deployment need.

On-device or edge deployment of image generation models where model size and inference compute are constrained. EOSTok-S uses only 93M AR parameters plus a 165M tokenizer (total 258M parameters, though the tokenizer is only needed for decoding, not for the 256-step autoregressive sampling loop during generation β€” the decoder runs once at the end). At 552 GFLOPs for EOSTok-H but proportionally less for smaller variants, EOSTok models can generate images on hardware that cannot run diffusion models at acceptable latency. A mobile device or laptop GPU that can handle ~500 GFLOPs could run EOSTok-S or EOSTok-B (gFID 3.50 and 2.38 without guidance, respectively) for interactive applications, while diffusion models at comparable quality would require orders of magnitude more compute. The paper's scaling results (Figure 5a) show that smaller models still achieve respectable quality (EOSTok-S gFID 3.50 compares favorably to many diffusion baselines in Table 3), making the compute-quality tradeoff explicit and favorable for resource-constrained deployment.

Training data generation for self-improving AR models, leveraging the fact that EOSTok produces both token sequences and pixel-space reconstructions in the same forward pass. A self-improvement loop (generate images β†’ filter by quality β†’ retrain) requires both the generated image (for quality assessment, e.g., by a classifier or VFM) and the underlying token sequence (for retraining the AR model). EOSTok's joint architecture produces both naturally: the AR model generates token sequences, and the shared decoder reconstructs them to pixels. In contrast, a two-stage pipeline would require encoding the generated images back to tokens (adding computation and potential encoding errors) or storing both the tokens and the images. For large-scale data generation (e.g., generating millions of images for the next training iteration), the integrated pipeline reduces storage and computation overhead. The paper's finding that end-to-end training produces token sequences that are specifically optimized for the AR model's prediction capability (Table 4: original ordering outperforms random by 1.9Γ— in gFID) further suggests that self-generated data would be high-quality for retraining, potentially enabling a virtuous cycle where the tokenizer and AR model co-evolve.

Accelerating research on autoregressive visual generation by providing a stable, documented training recipe with known failure modes. Before EOSTok, a researcher wanting to train a 1D tokenizer with AR generation would need to: (a) implement a TiTok-style architecture, (b) train the tokenizer for reconstruction, (c) freeze it, (d) train an AR model, and (e) hope the latent space was AR-friendly β€” with no diagnostic for why it might not be. EOSTok provides a single-stage training pipeline with documented failure modes (codebook collapse under NTP loss, Table 1; spatial prior leakage under direct alignment, Table 2; the APR loss as the fix), known hyperparameter sensitivities (APR weight, Table 10; nested dropout, Table 11), and a clear recipe (Table 9) that achieves state-of-the-art results. This lowers the barrier to entry for research groups wanting to explore 1D tokenization for new domains (video, 3D, multi-modal) β€” they can start from the EOSTok recipe and adapt, rather than rediscovering the collapse and alignment problems from scratch. The training overhead (15-19%, Table 7) is modest enough that academic groups with limited GPU budgets can replicate and extend the work.