iFSQ: Improving FSQ for Image Generation with 1 Line of Code

ArXiv: 2601.17124

🎯 Pitch

A single line of code replacing tanh with a sigmoid function mathematically guarantees perfect bin usage and reconstruction in FSQ tokenizers, eliminating a fundamental trade-off that has divided autoregressive and diffusion models. By creating the first unified tokenizer for both paradigms, the authors reveal that sequential ordering imposes a hard ceiling on image quality—autoregressive models learn faster but diffusion models ultimately win when given enough compute.

1. Executive Summary

This paper introduces iFSQ, a one-line modification to Finite Scalar Quantization (FSQ) that resolves the trade-off between reconstruction fidelity and information efficiency by replacing the tanh activation with a distribution-matching sigmoid function (2·σ(1.6·z) − 1) that maps Gaussian latents to an approximate uniform distribution, guaranteeing both optimal bin utilization and reconstruction precision. Using iFSQ as a unified tokenizer for both autoregressive (LlamaGen) and diffusion (DiT) models on ImageNet 256×256, the authors benchmark the two paradigms under identical reconstruction constraints, establishing that the optimal discrete–continuous equilibrium sits at approximately 4 bits per dimension and that AR models exhibit faster initial convergence while diffusion models reach a superior performance ceiling—revealing that strict sequential ordering limits the upper bounds of generation quality only when sufficient compute is available to reach the crossover point.

2. Context and Motivation

The Core Problem: Tokenizer Fragmentation Prevents Fair Comparison of Image Generation Paradigms

The image generation field suffers from a fundamental fragmentation that this paper aims to address: autoregressive (AR) models and diffusion models use fundamentally different tokenizers, making it impossible to attribute performance differences to the generative models themselves versus the properties of their underlying representations. Since AR models (like LlamaGen) operate on discrete tokens obtained from VQ-VAEs, while diffusion models (like DiT) work with continuous latents from standard VAEs, the community lacks a principled way to benchmark these paradigms against each other.

This is not merely a matter of academic tidyness — it has real consequences for model selection and research direction. As the authors state in Section 1:

"it is difficult to disentangle whether performance differences stem from the generative models themselves (AR vs. Diffusion) or the distinct properties of their underlying tokenizers (VQ-VAE vs. VAE)"

Without a unified tokenizer, every comparison between AR and diffusion is confounded by the discretization method. A diffusion model might underperform not because diffusion is inherently worse, but because its VAE tokenizer provides lower-quality latents. Conversely, an AR model's apparent efficiency advantage might be an artifact of its VQ-VAE's compact discrete representation rather than the autoregressive formulation itself. The field's inability to separate these effects has clouded our understanding of what truly drives generation quality.

Why This Matters: Practical and Theoretical Stakes

The motivation goes deeper than benchmarking hygiene. Three interconnected stakes make this problem significant:

1. The field is bifurcating without clear guidance on which path to invest in. AR models (LlamaGen, VAR, MAGVIT-v2) and diffusion models (DiT, Stable Diffusion, SDXL) represent competing research trajectories backed by substantial engineering resources. Yet practitioners lack evidence about which paradigm offers better scaling behavior — does AR's rapid early convergence translate to a long-term advantage, or does diffusion's parallel refinement eventually overtake it? The paper's FLOPs-matched comparison in Figure 4 provides the first controlled answer to this question, but the answer is only meaningful because iFSQ eliminates tokenizer confounds.

2. The VQ-VAE bottleneck limits discrete tokenizers. The dominant discrete tokenizer, VQ-VAE, has well-documented pathologies: codebook collapse (where many latent codes go unused), reliance on the straight-through gradient estimator (which introduces bias), and memory-intensive codebook lookup. While techniques like entropy regularization (MAGVIT-v2) and unused-code reinitialization (Jukebox) mitigate these issues partially, they add complexity without solving the core problem: the learnable codebook is inherently fragile. This fragility increases with codebook size, which matters because larger codebooks are needed for high-fidelity generation. The field needs discrete representations that avoid these pathologies entirely.

3. Multi-modal architectures remain unconverged on tokenizer design. As the authors note in Section 2.1, unified vision-language models (discussed in broader literature) lack consensus on whether to use discrete or continuous representations. A tokenizer that excels at both — providing high-quality continuous latents for diffusion-style objectives and efficient discrete indices for autoregressive prediction — would be a natural foundation for unified multi-modal systems. iFSQ positions itself as exactly this bridge.

Where Existing Tokenizers Fall Short

The paper identifies specific, measurable shortcomings in each existing approach:

Continuous VAEs sacrifice compression for fidelity. Standard VAEs for diffusion models represent latents as 16-bit floating-point vectors. As derived in Appendix B (Equation 11), this yields a compression ratio of only $\text{CR}_{\text{VAE}} = \frac{3f^2}{2d}$ — for a typical setup with $f=8$ downsampling and $d=4$ channels, this gives a mere 24× compression. This is adequate for diffusion models that process all latents simultaneously, but it precludes using the same latents for AR modeling, where the $h \times w$ sequence length and per-element bit-depth would make next-token prediction computationally prohibitive. The VAE achieves fidelity at the cost of compactness.

VQ-VAEs sacrifice simplicity for discreteness. While VQ-VAEs achieve high compression ratios (e.g., $\text{CR}_{\text{VQ}} \approx 438$ for a 16,384-codebook with 16× spatial compression, as shown in Appendix C), they introduce substantial engineering complexity. The codebook requires careful initialization (k-means clustering of encoder features, or LLM embedding initialization), collapse-prevention mechanisms (entropy regularization, code reinitialization), and the straight-through estimator — which treats the non-differentiable argmin operation as an identity in the backward pass — introduces gradient approximation errors. Each of these design choices can interact in unpredictable ways with the generative model, making VQ-VAE a moving target for benchmarking.

Original FSQ solves VQ-VAE's codebook problems but introduces a distribution mismatch. Finite Scalar Quantization (Mentzer et al., 2023) elegantly eliminates the learnable codebook by using simple element-wise rounding onto a fixed grid. This avoids codebook collapse, removes the straight-through estimator, and is dramatically simpler. However, the paper identifies a critical flaw that prior work missed: FSQ's equal-interval quantization bins are fundamentally mismatched to the distribution of neural network activations. Since the encoder's tanh activation maps the typically Gaussian latent distribution into a bounded range, most probability mass concentrates in the central bins while outer bins are underutilized. Figure 1(a) quantifies this: with a 9-level quantization grid (3.17 bits of information entropy), vanilla FSQ achieves only 83.3% bin utilization because the Gaussian-distributed activations crowd into a few central bins. The outer bins — representing edge cases and rare patterns — carry almost no data, effectively reducing the representational capacity below what the bit budget should provide.

The paper frames this as a forced trade-off (Section 1, Figure 1):

• Equal-interval quantization (Figure 1a) maintains fine-grained sampling in high-probability regions (MSE: 0.1678) but suffers from activation collapse — the effective codebook size is smaller than nominal.
• Equal-probability quantization (Figure 1b) maximizes information entropy (3.17 bits) and bin utilization (100%) but requires excessively wide outer bins to accommodate Gaussian tails, causing coarse quantization and degraded reconstruction (MSE rises to 0.1812).

This is a genuine dilemma: you can have precise reconstruction or efficient code utilization, but not both under a Gaussian activation distribution with fixed-interval bins. The paper's key insight is that this dilemma isn't fundamental — it's an artifact of the tanh activation choice.

How This Paper Positions Itself

The paper positions iFSQ as a unified tokenizer that bridges discrete and continuous paradigms by fixing a single mathematical flaw in FSQ — the distribution mismatch — while inheriting all of FSQ's structural advantages (no codebook collapse, no straight-through estimator, simple rounding-based quantization). This is framed as an improvement over three classes of prior work:

Against VQ-VAE: iFSQ retains the discrete indexing capability AR models need (via the bijective base-$L$ expansion in Equation 3) but eliminates the learnable codebook entirely. The implicit codebook size is $|\mathcal{C}| = L^d = (2^K + 1)^d$, which is architecturally determined and cannot collapse. Table 2 shows that at comparable configurations, iFSQ operates at lower bit rates than VQ while achieving better generation FID (iFSQ at 4 bits vs. VQ at 14 bits), suggesting the scalar quantization approach is inherently more information-efficient than learned codebook lookup.

Against standard FSQ: The one-line activation change from tanh(z) to 2·σ(1.6·z) − 1 is the core methodological contribution. This is not merely a hyperparameter tuning success — Section 3.2 provides a principled derivation by sweeping the sigmoid slope parameter $\alpha$ and measuring both distributional similarity (KS statistic, RMSE to uniform) and reconstruction quality (PSNR, SSIM, LPIPS). Figure 2 shows that $\alpha = 1.6$ is the unique minimum for both KS and RMSE, meaning it maximally transforms the input Gaussian into a uniform distribution within the bounded range. Figure 3 validates that this distributional optimum translates to reconstruction quality optimum, with $\alpha = 1.6$ consistently outperforming $\alpha = 2.0$ (the original tanh equivalent) across metrics.

The theoretical elegance is that a uniform distribution is the provably optimal input distribution for equal-interval quantization — it ensures equal probability mass in each bin, maximizing expected information content while maintaining fixed bin widths. This resolves the Figure 1 dilemma: iFSQ achieves both the equal-interval precision and the equal-probability utilization simultaneously, as shown in Figure 1(c), delivering 100% utilization with an MSE of 0.1669 — slightly better than the equal-interval-only case because uniform activation avoids the edge-clipping distortions that tanh introduces.

Against continuous AE: The paper does not claim iFSQ universally outperforms continuous autoencoders. Rather, the claim is more nuanced: iFSQ matches AE reconstruction quality at approximately 4 bits per dimension and above, while providing the additional capability of discrete token generation that AE cannot offer. Figure 5 (Section 4.1.5) demonstrates this convergence: at 7–8 bits, iFSQ is "nearly identical to AE" in PSNR, SSIM, and LPIPS, and with doubled latent dimension (iFSQ-2×dim) at 2 bits, it already surpasses AE at the original dimension. The gap between iFSQ and AE narrows systematically with quantization level, establishing iFSQ as a superset of AE's capabilities — when configured with sufficient bits, it matches continuous quality while retaining discrete mode.

As a benchmarking platform: The paper's most ambitious positioning move is to frame iFSQ not just as a better tokenizer, but as a controlled experimental platform for studying generative model scaling. Because iFSQ can serve as the tokenizer for both AR (LlamaGen) and diffusion (DiT) models with identical reconstruction quality, it eliminates the VAE vs. VQ-VAE confound that has plagued prior comparisons. Section 4.1.4 explicitly makes this case: "Since iFSQ serves as both a continuous latent and a discrete index, it establishes a fair platform for comparing diffusion and AR models by the same decoder reconstruction performance." The training efficiency comparison in Figure 4 — where both model families use the exact same iFSQ tokenizer — is the empirical manifestation of this benchmarking philosophy, and its findings (AR converges faster, diffusion peaks higher) derive their credibility precisely from the elimination of tokenizer differences.

Relationship to quantization literature: The paper draws a clear boundary against neural network quantization methods (GPTQ, AWQ, QAT) in Section 2.2. These methods treat quantization as post-training compression or inference optimization — reducing memory footprint or accelerating existing models. In contrast, iFSQ integrates quantization directly into the tokenizer training phase, using it as a mechanism to modulate the discreteness of the representation itself. This is a fundamentally different use case: iFSQ's quantization is not about reducing the cost of an already-trained model, but about creating a representation that inherently supports both discrete and continuous generative objectives from the start. The rate-distortion optimization framework (Sullivan & Wiegand, 2002) is relevant but inverted — instead of minimizing bit-rate for a given distortion constraint, iFSQ maximizes reconstruction quality for a given bit budget determined by the quantization levels $L$ and latent dimension $d$.

3. Technical Approach

3.1 Reader Orientation

This paper is primarily a methodological improvement paper — its core contribution is iFSQ, a one-line modification to the Finite Scalar Quantization (FSQ) tokenizer that transforms normally-distributed neural network activations into an approximately uniform distribution before quantization, resolving the fundamental trade-off between reconstruction fidelity and information efficiency that plagues original FSQ. The paper then uses iFSQ as a controlled experimental platform to benchmark autoregressive (AR) and diffusion image generation models under identical reconstruction constraints, revealing that AR models converge faster but diffusion models achieve a higher performance ceiling — a finding that is only credible because iFSQ eliminates the tokenizer confound that has historically prevented fair comparisons between the two paradigms.

3.2 Big-Picture Architecture (Diagram in Words)

The system has four major components organized into two logical pipelines:

Pipeline 1 — The iFSQ Tokenizer (training phase):

1. Encoder: A convolutional neural network that compresses an input image (e.g., 256×256×3) into a latent representation $z \in \mathbb{R}^{h \times w \times d}$ via spatial downsampling (typically 8× or 16×).
2. Distribution-Matching Activation (2·σ(1.6·z) − 1): A sigmoid-based nonlinearity that maps the encoder's approximately Gaussian output into a near-uniform distribution bounded in [−1, 1]. This replaces the original FSQ's tanh activation and is the entire methodological contribution (1 line of code changed).
3. Scalar Quantization and Rounding: The uniform-distributed values are scaled to an integer grid with $L = 2^K + 1$ levels per dimension, rounded to the nearest integer, and mapped back to continuous values via a straight-through estimator. The rounded integer indices form the discrete tokens for AR models; the dequantized continuous values form the latents for diffusion models.
4. Decoder: A convolutional neural network that upsamples the quantized latent representation back to pixel space, trained with reconstruction losses (MSE, LPIPS, optionally GAN).

Pipeline 2 — The Generative Models (benchmarking phase):

1. Diffusion Model (DiT-Large): Takes the continuous iFSQ latents as input, adds noise progressively, and learns to denoise. The same iFSQ encoder/decoder weights are frozen; only the DiT transformer is trained.
2. Autoregressive Model (LlamaGen-Large with REPA): Takes the discrete iFSQ token indices (flattened to a 1D sequence via bijective base-$L$ expansion) as input, and learns next-token prediction. The same frozen iFSQ encoder/decoder is used, but the quantization path provides integer indices rather than continuous values.
3. REPA Alignment Module (for LlamaGen-REPA): A lightweight projection head that aligns intermediate LlamaGen layer features with pre-trained DINOv2 representations, accelerating semantic convergence.

Information flows as follows: an image enters the frozen iFSQ encoder → the distribution-matching activation transforms the latent distribution to near-uniform → rounding produces both continuous latents (for DiT training) and discrete indices (for LlamaGen training) from the same quantization grid → the generative model (DiT or LlamaGen) learns to generate new latents/indices → the frozen iFSQ decoder reconstructs the generated output back to pixel space. The critical design choice is that both generative models use exactly the same encoder/decoder weights and representation quality, making any performance differences attributable to the generative model architecture rather than tokenizer quality.

3.3 Roadmap for the Deep Dive

• First, the original FSQ quantization mechanism (Equation 1, the rounding operation, the straight-through estimator) — because iFSQ's modification is a surgical replacement of one component in this pipeline, and understanding what changes requires understanding what stays the same.
• Second, the distribution analysis that motivates the modification — why tanh produces a non-uniform, bimodal output distribution, and why a uniform distribution is theoretically optimal for equal-interval quantization. This includes the sigmoid parameter sweep (Figure 2) that identifies $\alpha = 1.6$ as the unique distribution-matching optimum.
• Third, the iFSQ modification itself (Algorithm 1) — the exact one-line code change, why it resolves the fidelity-efficiency trade-off, and the mathematical property that makes it work (a sigmoid with slope 1.6 transforms Gaussian → near-uniform).
• Fourth, the dual-mode operation — how the same quantized representation serves as both continuous latents (via Equation 2 dequantization) and discrete tokens (via Equation 3 bijective base-$L$ expansion), enabling unified benchmarking.
• Fifth, the REPA adaptation for LlamaGen — how representation alignment from diffusion models is ported to autoregressive models, the layer-wise semantic evolution analysis (STS, NTS, CKNNA metrics), and the discovery that optimal alignment depth scales proportionally (approximately 1/3 of total layers) rather than at a fixed absolute layer index.
• Sixth, the training configurations and hyperparameters for all components, since the benchmarking claims depend on fair compute-matched comparisons.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a methodological improvement paper with a secondary benchmarking contribution. The core technical idea is that the tanh activation in original FSQ creates a distribution mismatch — it maps Gaussian-distributed encoder outputs to a non-uniform, bimodal distribution within the bounded quantization range — and that replacing it with a carefully-calibrated sigmoid function (2·σ(1.6·z) − 1) transforms the latents to a near-uniform distribution, which is provably optimal for equal-interval scalar quantization. The benchmarking contribution is enabled by iFSQ's dual-mode operation: the same quantized grid produces both continuous values (for diffusion) and discrete indices (for AR), creating the first fair comparison platform between the two paradigms.

3.4.1 Original FSQ: The Quantization Mechanism That iFSQ Modifies

The paper inherits the core FSQ quantization pipeline from Mentzer et al. (2023) and modifies only one component. Understanding what stays the same is essential for understanding what changes.

Encoder output and bounding. Given an input image, the encoder produces a latent representation $z \in \mathbb{R}^{N \times d}$, where $N = h \times w$ is the number of spatial positions and $d$ is the channel dimension. The encoder is a standard convolutional architecture with downsampling (typically 8× or 16× spatial compression, as detailed in Appendix D). The latent values $z_j$ for each channel $j$ are unbounded real numbers — they can be arbitrarily positive or negative. To prepare for quantization onto a fixed grid, FSQ first applies a bounding function $f: \mathbb{R} \to [-1, 1]$ that squashes all values into the range [−1, 1]. The original FSQ uses $f(z) = \tanh(z)$ for this purpose.

Quantization grid definition. FSQ defines a fixed quantization resolution with $L = 2^K + 1$ levels per channel, where $K$ is a hyperparameter controlling the bit depth. The $+1$ term ensures an exact zero-center exists in the grid — for example, $K=1$ gives $L=3$ levels {−1, 0, +1} in the continuous domain, and $K=2$ gives $L=5$ levels. The total number of unique discrete codes across all $d$ channels is the implicit codebook size $|\mathcal{C}| = L^d = (2^K + 1)^d$. Unlike VQ-VAE, there is no learned codebook — this size is architecturally determined and fixed.

Quantization operation (Equation 1). For each channel $j$, the bounded value $f(z_j) \in [-1, 1]$ is mapped to an integer index $q_j \in \{0, 1, ..., L-1\}$ via:

$q_j = \text{round}\left(\frac{L-1}{2} \cdot (f(z_j) + 1)\right)$

where $\text{round}(\cdot)$ is element-wise rounding to the nearest integer, $L$ is the number of quantization levels per channel, and $f(z_j)$ is the bounded activation value.

What it computes: the expression $(f(z_j) + 1)$ first shifts the range from [−1, 1] to [0, 2]. Multiplying by $\frac{L-1}{2}$ then scales this to [0, L−1], which is the exact range of the integer indices. The $\text{round}$ operation snaps each scaled value to the nearest integer, producing the discrete index $q_j$. For example, if $f(z_j) = 0$ (the center of the activation range), then $q_j = \text{round}((L-1)/2 \cdot 1) = \text{round}((L-1)/2)$, which is the middle index. If $f(z_j) = 1$, then $q_j = \text{round}((L-1)/2 \cdot 2) = L-1$, the maximum index. If $f(z_j) = -1$, then $q_j = \text{round}(0) = 0$, the minimum index.

Why this form: this linear mapping from a bounded continuous range to integer indices is the simplest possible quantization scheme. It uses equal-interval bins — the distance between adjacent integer indices corresponds to a constant step size of $\frac{2}{L-1}$ in the continuous [−1, 1] space. This equal-interval property is what gives FSQ its simplicity (no learned bin boundaries, no adaptive quantization), but it is also the source of the distribution mismatch problem: if the values $f(z_j)$ are not uniformly distributed across [−1, 1], the bins will be utilized with unequal probability, wasting representational capacity.

Dequantization for continuous latents (Equation 2). For diffusion models that require continuous-valued latents, FSQ maps the integer indices back to the continuous range [−1, 1]:

$z_{\text{quant},j} = \left(q_j - \frac{L-1}{2}\right) \cdot \frac{2}{L-1}$

What it computes: this is the exact inverse of Equation 1 without the rounding. The term $(q_j - \frac{L-1}{2})$ first centers the integer index around zero (so the middle index maps to 0). Multiplying by $\frac{2}{L-1}$ then rescales to [−1, 1]. The output $z_{\text{quant},j}$ is the quantized continuous value that approximates the original $z_j$. The operation acts as lossy compression: $z_{\text{quant}} \approx z$, with the information loss proportional to the quantization step size $\frac{2}{L-1}$.

Why this form: this symmetric, zero-centered dequantization is necessary because the decoder expects inputs with zero mean and bounded range — properties that align with the distributional assumptions built into the decoder's architecture and training. An asymmetric mapping (e.g., [0, 1] instead of [−1, 1]) would create a distribution shift between training and inference that degrades reconstruction.

Straight-through estimator for gradient flow. The rounding operation in Equation 1 is non-differentiable — its gradient is zero almost everywhere and undefined at the half-integer boundaries. To enable end-to-end training of the encoder via backpropagation, FSQ employs the straight-through estimator: during the backward pass, the rounding operation is treated as an identity function, so gradients flow through it as if $\frac{\partial q_j}{\partial f(z_j)} = 1$. During the forward pass, the actual rounded values are used. This introduces a gradient approximation error — the encoder receives gradients as if small changes to $f(z_j)$ would produce proportional changes to $q_j$, when in reality only changes that cross rounding boundaries have any effect — but in practice this approximation is sufficient for training. Algorithm 1 shows this explicitly:

z_hat = z_rounded - z_scaled.detach() + z_scaled

This line computes z_hat such that in the forward pass z_hat = z_rounded (since z_scaled.detach() allows the first two terms to cancel), but in the backward pass, the gradient flows through z_scaled (the non-detached term) while ignoring z_rounded (which is detached). This is the standard reparameterization trick for non-differentiable quantization.

Discrete token formation for autoregressive models (Equation 3). AR models require a flat sequence of scalar token indices, not a vector of per-channel indices. FSQ maps the per-channel index vector $q = (q_1, q_2, ..., q_d)$ to a single scalar codebook index $I$ via a bijective base-$L$ expansion:

$I = \sum_{j=1}^{d} q_j \cdot L^{d-j}$

What it computes: this treats the vector $q$ as a mixed-radix number where each position $j$ has base $L$, with position $j=1$ being the most significant digit (weight $L^{d-1}$) and position $j=d$ being the least significant digit (weight $L^0 = 1$). The sum produces a unique integer $I \in \{0, 1, ..., L^d - 1\}$ for each possible combination of per-channel indices. The mapping is bijective — given $I$ and $L$, one can uniquely recover each $q_j$ via successive division and modulo operations.

Why this form: the mixed-radix expansion is the standard way to flatten a multidimensional index into a scalar without collisions. It preserves all information — no two distinct vectors $q$ can map to the same $I$ — and the ordering places vectors with similar high-order channels (larger $j$ in the summation) closer together in the scalar index space, which provides a weak form of semantic locality that may help the autoregressive model learn. Alternative flat-indexing schemes (e.g., simple concatenation of binary representations) would require more bits for the same representational capacity.

Concrete example from Section 3.1: with $d=4$ channels and $K=1$ (so $L=3$), if $q = [2, 2, 1, 0]$, then $I = 2 \cdot 3^3 + 2 \cdot 3^2 + 1 \cdot 3^1 + 0 \cdot 3^0 = 54 + 18 + 3 + 0 = 75$. The implicit codebook size is $3^4 = 81$ possible tokens.

3.4.2 The Distribution Mismatch Problem: Why Original FSQ Is Suboptimal

The entire motivation for iFSQ rests on a distributional analysis that the paper performs in Section 3.2 and visualizes in Figures 1 and 2.

The core observation. Neural network activations — including the outputs of the convolutional encoder before the bounding function — typically follow an approximately Gaussian (normal) distribution centered near zero. This is a well-documented empirical property of deep networks, cited by the authors with reference to Lee et al. (2017). When such a Gaussian distribution is passed through the tanh function, the output distribution is not uniform. Instead, as shown in Figure 2(a) (the green curve, corresponding to $\alpha = 2.0$ which is equivalent to tanh), the distribution becomes non-uniform with concentration near the extremes (−1 and +1) — a bimodal shape. This happens because tanh saturates for inputs beyond approximately ±2, so a large fraction of the Gaussian's probability mass gets compressed into the two saturation regions, while the linear region near zero receives proportionally less mass.

The trade-off visualized in Figure 1. The paper illustrates the consequences with a controlled experiment using 9 quantization levels (3.17 bits of information entropy) applied to a standard normal distribution clipped to [−3, 3]:

• Figure 1(a) — Equal-interval quantization (vanilla FSQ): The 9 bins have equal width in the continuous domain. Since the Gaussian has higher probability density near zero, the central bins capture most of the data points. The outer bins — despite having the same width — are severely underutilized. The result is activation collapse: the bin utilization is only 83.3%, meaning approximately 1 in 6 quantization levels carries essentially no information. However, reconstruction fidelity is relatively good (MSE: 0.1678) because the high-probability central region is sampled densely. This is the "high fidelity, low efficiency" regime.

• Figure 1(b) — Equal-probability quantization: If we instead design bins to have equal probability mass (by making outer bins wider to capture the Gaussian tails and central bins narrower), we achieve 100% bin utilization and maximum information entropy (3.17 bits). However, the excessively wide outer bins mean that values falling into those bins are quantized coarsely — a value near +3 and a value near +1.5 might both round to the same representative, losing precision. The reconstruction degrades: MSE rises to 0.1812. This is the "high efficiency, low fidelity" regime.

• Figure 1(c) — Equal-interval quantization of a uniform distribution (iFSQ): If the input distribution were already uniform, equal-interval bins would simultaneously achieve equal probability mass (100% utilization) and fine-grained quantization (narrow bins everywhere). No trade-off would be necessary — the two desiderata align perfectly. This is the theoretical ideal that iFSQ approximates.

The mathematical tension. The conflict is between two goals that are aligned for uniform distributions but opposed for Gaussian distributions:

1. Information efficiency: maximize entropy by having each quantization bin capture roughly equal probability mass. This requires bin widths inversely proportional to the probability density — wider bins where density is low, narrower bins where density is high.
2. Reconstruction fidelity: minimize quantization error by keeping all bins narrow. This requires equal-width bins (since quantization error for a uniform bin is proportional to bin width squared).

For a Gaussian, these goals are contradictory — you cannot simultaneously have equal-width bins and equal-probability bins. For a uniform distribution, they are identical — equal-width bins automatically have equal probability.

Why tanh fails. The tanh function does not transform a Gaussian into a uniform distribution. As shown in Figure 2(a) (green curve, $\alpha = 2.0$), tanh produces a distribution that is actually bimodal — probability mass concentrates near −1 and +1, with a dip in the center. This is the worst possible distribution for equal-interval quantization because it means the extreme bins (near −1 and +1) are heavily utilized while the central bins are underutilized, and the quantization step size is the same everywhere. The result is neither efficient code usage nor particularly good reconstruction — it underperforms what the bit budget should theoretically support.

3.4.3 Finding the Optimal Activation: The Sigmoid Parameter Sweep

The paper's key insight is that a sigmoid function with a carefully chosen slope parameter can approximately transform a Gaussian distribution into a uniform distribution within the bounded range [−1, 1]. This is not obvious — it requires a specific slope value that balances the sigmoid's compressive nonlinearity against the Gaussian's bell-shaped probability density.

The general sigmoid form (Equation 4). The paper investigates a parameterized family of sigmoid functions:

$s(x) = A \cdot \sigma(\alpha x) + B$

where $A$ and $B$ are scaling and shifting constants, $\sigma(u) = \frac{1}{1 + e^{-u}}$ is the standard logistic sigmoid, and $\alpha$ is the slope parameter that controls how sharply the sigmoid transitions from its lower asymptote to its upper asymptote.

What this family represents: when $A = 2.0$ and $B = -1$, the function maps $\mathbb{R} \to [-1, 1]$ (since $\sigma(\cdot) \in [0, 1]$). The slope $\alpha$ controls the steepness: larger $\alpha$ means a sharper transition near zero, approaching a step function as $\alpha \to \infty$; smaller $\alpha$ means a gentler, more linear transition. The special case $\alpha = 2.0$ is mathematically equivalent to tanh (since $\tanh(x) = 2\sigma(2x) - 1$), which recovers the original FSQ.

Why this family is the right search space: the sigmoid is a natural candidate for transforming an unbounded symmetric distribution (Gaussian) into a bounded distribution. Its S-shape compresses extreme values toward the boundaries while preserving relative ordering in the central region. By varying $\alpha$, one can tune how aggressively the compression happens — a smaller $\alpha$ spreads the probability mass more evenly across [−1, 1], while a larger $\alpha$ pushes mass toward the extremes. The question is whether there exists an $\alpha$ that achieves near-uniformity, and if so, what value.

The sweep experiment (Figure 2). The authors sample 500,000 points from a standard normal distribution $\mathcal{N}(0, 1)$, apply the sigmoid transformation with different values of $\alpha \in \{1.0, 1.3, 1.6, 2.0, 2.4\}$, and measure two properties:

1. Qualitative: probability density shape (Figure 2a). The empirical probability density functions (PDFs) are plotted for each $\alpha$:

   • $\alpha = 1.0$ (light green): strongly unimodal, concentrated near zero — too much mass in the center, underutilizing the outer range.
   • $\alpha = 1.3$: still unimodal but more spread out — approaching uniformity but still with a visible central peak.
   • $\alpha = 1.6$ (dark green solid line): closely approximates the uniform distribution (grey dashed line) — nearly flat across [−1, 1].
   • $\alpha = 2.0$ (equivalent to tanh, green): bimodal with peaks near ±1 — probability mass pushed too far toward the extremes.
   • $\alpha = 2.4$: even more strongly bimodal, with a deeper central dip — further degradation from uniformity.

2. Quantitative: distribution similarity metrics (Figure 2b). Two metrics quantify how close each transformed distribution is to a perfect uniform distribution on [−1, 1]:

   • RMSE (Equation 5): Root Mean Square Error between the empirical PDF values and the theoretical uniform PDF. Given $N$ histogram bins approximating the continuous distribution, the RMSE is:

     $\text{RMSE} = \sqrt{\frac{1}{N} \sum_{i=1}^{N} (x_i - \hat{x}_i)^2}$

     where $x_i$ is the target uniform probability density in bin $i$ and $\hat{x}_i$ is the empirical density from the transformed Gaussian samples. Lower RMSE means the empirical distribution is closer to uniform.

   • KS statistic (Equation 6): The Kolmogorov-Smirnov statistic measures the maximum divergence between two cumulative distribution functions (CDFs). It is defined as the supremum of the absolute difference:

     $D_{KS} = \sup_x |F_P(x) - F_Q(x)|$

     where $F_P(x)$ is the empirical CDF of the transformed samples and $F_Q(x)$ is the CDF of the target uniform distribution. Lower KS means the two distributions are more similar in the sense of worst-case deviation. Unlike RMSE which measures average density differences, KS is sensitive to any local mismatch — a single region of large divergence will dominate the statistic.

   The result: both RMSE and KS reach their unique minima at $\alpha = 1.6$. The KS drops from approximately 0.12 at $\alpha = 2.0$ (tanh) to approximately 0.02 at $\alpha = 1.6$ — a roughly 6× reduction in worst-case distributional divergence. The RMSE shows a similar pattern, confirming that $\alpha = 1.6$ is not merely a local improvement but the global optimum within the sigmoid family for transforming a standard normal into a uniform distribution on [−1, 1].

Why $\alpha = 1.6$ works. Intuitively, the standard normal has most of its probability mass within ±2 standard deviations. The tanh function ($\alpha = 2.0$) is too steep — it saturates for inputs beyond about ±1, pushing too much mass to the extremes. The sigmoid with $\alpha = 1.6$ has a gentler slope, spreading the central Gaussian mass more evenly across the [−1, 1] range before saturating. This gentler compression approximately inverts the Gaussian CDF, producing a near-uniform output. The value 1.6 is not derived analytically but identified empirically through the sweep — it represents the slope that best balances the competing effects of the Gaussian's central concentration and the sigmoid's compressive nonlinearity.

3.4.4 iFSQ: The One-Line Code Change

The methodological contribution is distilled into an almost comically minimal modification, shown in Algorithm 1:

- z = tanh(z)
+ z = 2 * sigmoid(1.6 * z) - 1

This replaces the tanh activation (which is equivalent to 2·σ(2.0·z) − 1) with the optimized sigmoid form 2·σ(1.6·z) − 1. No other part of the FSQ pipeline changes — the quantization grid, the rounding, the straight-through estimator, the dequantization, and the base-$L$ expansion all remain identical.

Why this works. The activation function $f(z)$ in FSQ serves two roles simultaneously:

1. Bounding: it constrains the unbounded encoder outputs to the finite range [−1, 1] so they can be mapped to a fixed quantization grid.
2. Distribution shaping: it determines how the probability mass of the input distribution is allocated across that range, which controls how evenly the quantization bins are utilized.

The original tanh accomplishes role 1 perfectly but fails at role 2 — it produces a bimodal distribution that concentrates mass near ±1. The iFSQ sigmoid accomplishes both: it bounds to [−1, 1] (since $2\sigma(u) - 1 \in [-1, 1]$ for all $u$) while simultaneously transforming the input Gaussian into an approximately uniform distribution (as validated by the $\alpha = 1.6$ optimum). The uniform output means that the subsequent equal-interval quantization in Equation 1 automatically achieves equal-probability bins — the very property that was impossible with tanh.

Validation through reconstruction quality (Figure 3). The paper validates that the distributional optimum at $\alpha = 1.6$ translates to reconstruction quality optimum. Figure 3 plots PSNR, SSIM, and LPIPS against $\alpha \in \{1.0, 1.2, 1.6, 1.8, 2.0, 2.4\}$, with the distributional metrics KS and RMSE overlaid on a secondary y-axis:

• As $\alpha$ increases from 1.0 to 1.6: PSNR and SSIM increase (reconstruction improves) while KS and RMSE decrease (distribution becomes more uniform). The reconstruction quality and distribution uniformity move in lockstep — better uniformity directly improves reconstruction.
• At $\alpha = 1.6$: PSNR and SSIM reach their maxima, KS and RMSE reach their minima. This is the joint optimum for both distribution matching and reconstruction quality. iFSQ ($\alpha = 1.6$) consistently outperforms original FSQ ($\alpha = 2.0$) across PSNR, SSIM, and LPIPS.
• As $\alpha$ increases from 1.6 to 2.4: KS and RMSE increase (distribution diverges from uniform), and PSNR and SSIM decrease (reconstruction degrades). The correspondence is nearly perfect — deviations from uniformity harm reconstruction.
• LPIPS reaches its best (lowest) value at $\alpha = 2.4$ rather than 1.6. The authors acknowledge this but choose $\alpha = 1.6$ as the overall optimum based on the PSNR and SSIM peak, which they prioritize. This is a design trade-off: LPIPS is a perceptual metric based on deep network features, and it apparently benefits from the sharper edge representations that more extreme $\alpha$ values produce, even though pixel-space metrics (PSNR, SSIM) degrade.

Robustness across datasets. The paper reports that although all tokenizer training is done on ImageNet, the $\alpha = 1.6$ optimum generalizes to the COCO validation set — the same trends in PSNR, SSIM, and LPIPS are observed. This suggests the optimal $\alpha$ is a property of the Gaussian-to-uniform transformation itself, not an artifact of the particular image distribution.

3.4.5 Dual-Mode Operation: Continuous Latents and Discrete Tokens from the Same Quantization

The architectural genius of iFSQ (inherited from FSQ) is that a single quantization operation produces two usable representations simultaneously, enabling the unified benchmarking that is the paper's secondary contribution.

For diffusion models (continuous path): After the encoder produces $z$ and iFSQ quantizes it to integer indices $q_j$, the dequantization in Equation 2 maps these indices back to continuous values $z_{\text{quant},j} \in [-1, 1]$. These continuous values are treated exactly like standard VAE latents — they are fed to the decoder during tokenizer training, and the same encoder-decoder pair (with frozen weights) provides the latent space for diffusion model training. The DiT model never sees the discrete indices; it operates entirely on the dequantized continuous values. The quantization serves purely as a regularization mechanism — it forces the encoder to produce latents that survive the information bottleneck of the quantization grid, which encourages robust, structured representations.

For autoregressive models (discrete path): The same integer indices $q_j$ are flattened into scalar tokens via Equation 3. Each spatial position in the latent grid (of which there are $h \times w$) produces one scalar token index $I$. These $h \times w$ tokens are arranged in raster-scan order (or some other fixed spatial ordering) to form the input sequence for the autoregressive transformer. The LlamaGen model never sees the continuous values; it operates entirely on the discrete token sequence, learning to predict the next token given all previous ones. The decoder is only invoked at inference time to convert generated token sequences back to pixel images.

Why this enables fair benchmarking. In prior work, comparing DiT (which uses a VAE with continuous latents) to LlamaGen (which uses a VQ-VAE with discrete tokens) was comparing apples to oranges — the tokenizers had different architectures, different reconstruction quality, different compression ratios, and different training procedures. With iFSQ, both models see exactly the same latent representations up to the quantization/dequantization step. The encoder is identical. The decoder is identical. The reconstruction quality (rFID, PSNR, SSIM, LPIPS) is identical. The only difference is whether the generative model operates on the continuous dequantized values (DiT) or the discrete indices (LlamaGen). Any performance differences between the two generative models can therefore be attributed to the generative architecture itself, not the tokenizer.

The compression ratio trade-off. Appendix B derives the compression ratios analytically. For iFSQ with spatial downsampling factor $f$, latent dimension $d$, and $L$ levels per channel:

$\text{CR}_{\text{iFSQ}} = \frac{24 f^2}{d \cdot \log_2(L)}$

where the numerator $24 f^2$ comes from the raw image bit count (24 bits per pixel = 3 channels × 8 bits, times $f^2$ pixels per latent position). The denominator $d \cdot \log_2(L)$ is the bits required to encode one scalar token — $d$ channels, each quantized to $L$ levels, flattened to a single index via base-$L$ expansion. By adjusting $L$ and $d$, iFSQ can span a wide range of compression ratios — from near-continuous quality at high bit rates (large $L$, small compression) to highly compact representations at low bit rates (small $L$, large compression). This flexibility is what enables the scaling analysis in Section 4.1.5 and Appendix C.

Concrete configuration examples from the paper. The experiments use various $(d, L)$ combinations to sweep the bit rate:

• 2 bits per dimension: 192× compression (Table 1)
• 3 bits: 128× compression
• 4 bits: 96× compression (the "sweet spot" identified in Section 4.1.5)
• 5 bits: 76× compression
• 6 bits: 64× compression
• 7 bits: 54× compression
• 8 bits: 48× compression

These are achieved by varying $L$ (which controls bits per channel as $\log_2(L)$) and $d$ (which scales the total bits linearly). The 4-bit configuration that the paper identifies as optimal corresponds to a particular $(d, L)$ pair that balances reconstruction quality (approaching AE levels) with compression efficiency (substantially better than the AE's 16-bit floating-point latents).

3.4.6 Training the iFSQ Tokenizer

The tokenizer training follows a standard autoencoder pipeline with quantization inserted in the bottleneck, as detailed in Appendix D.

Architecture. The encoder and decoder follow the latent diffusion architecture — likely a convolutional design with residual blocks and downsampling/upsampling layers, though exact architectural details are not specified beyond the spatial compression factors (8×, 16×, or 64× depending on the experiment). The key hyperparameters:

• Spatial compression: the primary experiments use 8× downsampling (standard for ImageNet 256×256, giving $h = w = 32$), while scaling experiments in Appendix C use 16× or 64× compression.
• Latent dimension $d$: varied across experiments, typically 4 or 8, to sweep the bit rate independently of the quantization level $L$.

Loss function. The tokenizer is trained with a combination of reconstruction losses:

• LPIPS loss (perceptual loss based on deep network features): coefficient set to 0.1. This encourages the reconstructed image to match the original in terms of high-level semantic features, not just pixel values.
• MSE loss (pixel-wise mean squared error): standard reconstruction objective.
• Optional KL divergence loss: for the continuous AE baseline, a KL divergence term regularizes the latent distribution toward a standard normal. For iFSQ, the paper notes that this KL loss is not used — the quantization itself provides sufficient regularization, and prior work (MAETok, VA-VAE) has shown that diffusion models do not require a normally-distributed latent space for convergence.

Optimization. Training uses the Adam optimizer (Kingma, 2014) with a constant learning rate of 0.001, batch size not explicitly stated for tokenizer training, for 25 epochs on ImageNet 256×256. The straight-through estimator handles gradient flow through the non-differentiable rounding operation. No learning rate schedule is mentioned — the constant rate suggests the 25-epoch training budget is sufficient to reach convergence without decay.

Why these choices: the training configuration is deliberately standard and minimal — no adversarial losses, no discriminative feature supervision (unlike recent VAE improvements that add GAN and DINO losses), no complex augmentation. This ensures that any performance differences between iFSQ and baseline tokenizers are attributable to the quantization mechanism rather than training recipe improvements. The 25-epoch budget (approximately 300k iterations at typical ImageNet batch sizes) is modest compared to state-of-the-art tokenizer training (which often uses hundreds of epochs with GAN losses), but it is sufficient to demonstrate the relative ordering of methods and the distribution-matching benefit.

3.4.7 Diffusion Model Training with iFSQ Latents (DiT-Large)

The diffusion model experiments follow the DiT (Peebles & Xie, 2023) architecture and training protocol, with the iFSQ tokenizer replacing the standard VAE.

Model architecture. DiT-Large is a transformer-based diffusion model operating on latent patches. The latent representation from iFSQ has spatial dimensions $h \times w$ (e.g., 32×32 for 8× compression on 256×256 images) and channel dimension $d$. These latents are treated as a sequence of $h \times w$ tokens, each of dimension $d$, and processed by the transformer backbone. The architectural details follow the original DiT paper exactly — the only change is the input latent format (iFSQ latents instead of VAE latents).

Diffusion process. Following the formulation in Appendix A (Equation 9), the model is trained with a velocity prediction objective. Given a clean latent $x$ and noise $\epsilon \sim \mathcal{N}(0, 1)$, a timestep $t \in [0, 1]$ is sampled uniformly, and the intermediate noisy latent is:

$z_t = t x + (1 - t) \epsilon$

The model $v_\theta(z_t, t)$ predicts the velocity $v = x - \epsilon$ (the direction from noise to data), trained with the mean squared error loss:

$\mathcal{L} = \mathbb{E}_{t, x, \epsilon} \left[ \| v_\theta(z_t, t) - v \|^2 \right]$

where the expectation is over sampled timesteps, clean latents from the dataset, and noise samples.

What this computes: at each training step, the model sees a noisy version of a real image's iFSQ latent, along with the noise level $t$, and must predict the denoising direction. This is a standard diffusion objective — the model learns to gradually transform pure noise into structured latents by taking small steps toward the data distribution. The velocity parameterization (predicting $x - \epsilon$ rather than $\epsilon$ or $x$ directly) has been shown to provide more stable training and better sample quality.

Why this form over alternatives: the velocity parameterization is a linear combination of predicting the noise (common in DDPM-style models) and predicting the clean data (common in flow-matching models). It performs well across the entire timestep range — at $t \approx 0$ (near noise), predicting $\epsilon$ is easier, while at $t \approx 1$ (near data), predicting $x$ is easier. The velocity formulation smoothly interpolates between these regimes, avoiding the numerical instability that can occur when predicting noise near the data distribution.

Training hyperparameters (Section 4.1, Appendix D). To accelerate experiments, the authors adopt the "ablation parameters from LightingDiT" (Yao et al., 2025):

• Batch size: 1024
• Training iterations: 100,000
• Image resolution: 256×256
• Classifier-free guidance: not used for the main FID comparisons (gFID reported without CFG)
• Inference: Euler method with 250 function evaluations (NFE) for sampling
• FLOPs: DiT-Large at 256 resolution has approximately 161.04 GFLOPs (stated in Figure 4)

The 100k iteration budget is deliberately short — sufficient to demonstrate relative ordering of methods but not to reach full convergence. This is a practical choice to enable the extensive ablation studies (multiple tokenizer configurations × multiple models) within a reasonable compute budget. The relatively early stopping means that the absolute gFID numbers may not be state-of-the-art, but the relative comparisons between methods remain valid as long as all are trained for the same number of iterations.

REPA-enhanced DiT training. For experiments with Representation Alignment (REPA), an additional loss term is added. A lightweight projection head (typically a single linear layer) maps the features from a specific intermediate layer of the DiT transformer to match the features from a pre-trained DINOv2 model (specifically DINOv2-Base, using its final layer features). The alignment is enforced via a cosine similarity loss with coefficient $\lambda = 0.5$ (the standard DiT-REPA setting). The total loss becomes $\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{diffusion}} + \lambda \cdot \mathcal{L}_{\text{REPA}}$. The REPA objective accelerates training by providing a semantic learning signal — the model doesn't need to discover high-level visual concepts from scratch through the diffusion objective alone; it can leverage pre-trained representations as a shortcut.

3.4.8 Autoregressive Model Training with iFSQ Tokens (LlamaGen-Large)

The autoregressive experiments use LlamaGen (Sun et al., 2024) as the base architecture, with iFSQ replacing the VQ-VAE tokenizer.

Token sequence formation. An input image is encoded by the frozen iFSQ encoder, quantized to per-channel indices $q_j$ via Equation 1, and flattened to scalar tokens via Equation 3. With spatial compression factor $f$, the latent grid has $h \times w = (H/f) \times (W/f)$ positions, each producing one scalar token. For 256×256 images with 16× spatial compression (the standard LlamaGen setting, as mentioned in Section 4.1.3), this yields $16 \times 16 = 256$ tokens. These 256 tokens are arranged in raster-scan order (left-to-right, top-to-bottom) to form a 1D sequence of length 256. A start-of-sequence token and optionally an end-of-sequence token may be added, following standard autoregressive language modeling conventions.

Autoregressive objective. LlamaGen is a standard decoder-only transformer that predicts each token conditioned on all previous tokens in the sequence. Given the token sequence $i_1, i_2, ..., i_N$ (where $N = h \times w$), the model is trained with the next-token prediction cross-entropy loss (Appendix A, Equation 10):

$\mathcal{L} = -\mathbb{E}_I \left[ \sum_{k=1}^{N} \log p_\theta(i_k \mid i_1, ..., i_{k-1}) \right]$

where $p_\theta(i_k \mid i_1, ..., i_{k-1})$ is the model's predicted probability for token $i_k$ given the preceding context, and the expectation is over training images.

What it computes: for each spatial position in raster-scan order, the model must predict which discrete token (out of the $L^d$ possible values) should appear there, given all previously generated tokens. This is conceptually identical to language modeling — the model learns the statistical structure of token sequences corresponding to natural images. The cross-entropy loss penalizes the model for assigning low probability to the correct token, driving it to learn a distribution that assigns high likelihood to valid image token sequences.

Why this form: the autoregressive factorization decomposes the joint distribution over all token sequences into a product of conditional distributions, which can be tractably modeled by a causal transformer (one that can only attend to previous positions). This is the dominant paradigm for language modeling and has been successfully adapted to images by flattening 2D grids to 1D sequences. Alternative orderings (spiral, random, multiple scales) have been explored in the literature, but raster-scan is the simplest and matches LlamaGen's original design.

Training hyperparameters (Section 4.1.3, Appendix D). The autoregressive experiments use:

• Model: LlamaGen-Large (24 transformer layers)
• Batch size: 256
• Training iterations: 500,000
• Codebook size: varies with iFSQ configuration; for the VQ-VAE baseline, 16,384 (14 bits, matching original LlamaGen)
• Spatial compression: 16× (matching original LlamaGen), yielding 256 tokens per 256×256 image
• FLOPs: LlamaGen-L at 256 resolution has approximately 169.65 GFLOPs (stated in Figure 4), where "the FLOPs for attention computation and value weighting are calculated as half of full attention" — suggesting a causal attention implementation that avoids computing the full attention matrix.

The 500k iteration budget is substantially larger than the 100k used for DiT, reflecting the expectation that autoregressive models train more slowly per iteration (since each iteration processes only one token prediction per position, whereas diffusion processes the entire latent simultaneously). The FLOPs comparison in Figure 4 accounts for these differences by plotting FID against total compute (not iterations), enabling a fair efficiency comparison.

3.4.9 LlamaGen-REPA: Adapting Representation Alignment to Autoregressive Models

Section 4.2 adapts the REPA (Representation Alignment) technique from diffusion models to autoregressive models, discovering that AR models benefit from stronger alignment regularization and that the optimal alignment depth follows a proportional scaling law rather than being a fixed absolute layer index.

What REPA does. REPA adds an auxiliary loss during training that encourages the features at a specific intermediate layer of the generative model to match the features from a pre-trained visual encoder (DINOv2). This provides a semantic shortcut — rather than learning high-level visual concepts purely from the generative objective, the model can bootstrap from representations that already encode object identity, spatial structure, and semantic relationships. In diffusion models (DiT-REPA), this has been shown to significantly accelerate training convergence.

Why AR models might need different REPA settings. The paper identifies a key difference between diffusion and autoregressive training that motivates a separate REPA analysis for LlamaGen. In diffusion models, the input at each denoising step is the same latent representation (just at different noise levels) — the model is essentially performing a form of self-encoding, refining the same representation iteratively. In autoregressive models, each position must predict the next token, not reconstruct the current one. This means the model undergoes a mode switch from self-encoding (processing the current token) to next-token prediction (anticipating future tokens). The REPA alignment should ideally be applied before this mode switch occurs — aligning features that are about to become predictive may conflict with the next-token objective.

Layer-wise semantic evolution analysis (Figure 6). To understand where the mode switch happens and where alignment would be most beneficial, the paper introduces three metrics that track feature evolution across LlamaGen's 24 transformer layers:

1. Self-Token Similarity (STS, Equation 7): measures how similar the features at layer $l$ are to the final output features at the same spatial position:

   $\text{STS}_l = \frac{1}{N} \sum_{i=1}^{N} \cos(h_l^{(i)}, h_L^{(i)})$

   where $h_l^{(i)}$ is the feature vector at layer $l$ and spatial position $i$, $h_L^{(i)}$ is the final layer feature at the same position, $N$ is the total number of spatial tokens, and $\cos(\cdot, \cdot)$ is cosine similarity. High STS means the layer is still encoding the current token's identity — it hasn't yet shifted to predicting the next token. As the network deepens, STS decreases, indicating a departure from self-encoding.

2. Next-Token Similarity (NTS, Equation 8): measures how similar the features at layer $l$ at position $i$ are to the final output features at position $i+1$ (the next spatial position in raster order):

   $\text{NTS}_l = \frac{1}{N-1} \sum_{i=1}^{N-1} \cos(h_l^{(i)}, h_L^{(i+1)})$

   This is the key metric for detecting the mode switch. When NTS rises sharply, it means the layer's features have started to encode information about the next token rather than the current one — the model has transitioned from encoding mode to prediction mode. Unlike STS, which compares aligned positions, NTS compares shifted positions, capturing the causal, forward-looking nature of autoregressive modeling.

3. CKNNA (Centered Kernel Alignment with Nearest Neighbor Accuracy): a metric from the REPA paper (Yu et al., 2024) that measures how well the layer's features align with pre-trained DINOv2 features. High CKNNA means the layer has acquired semantically meaningful representations similar to those in a mature vision model. The paper uses this metric without redefining it, treating it as a standard measure of semantic alignment quality.

Key findings from the layer-wise analysis (Figure 6, top row):

• STS steadily decreases across layers for all model scales (Large, XXLarge) and resolutions (256, 384). This confirms the expected behavior: early layers encode current-token information, and later layers gradually move away from this self-encoding state.
• NTS and CKNNA both exhibit sharp increases in the middle-to-late layers, and critically, the layer index where NTS surges synchronizes closely with the rise in CKNNA. This suggests that the emergence of next-token prediction capability is intrinsically linked to the acquisition of high-level semantic representations — understanding what an image patch contains (semantics) and predicting what comes next (causality) appear to co-emerge in the network.
• The correlation between NTS and CKNNA is quantified in Figure 6's bottom row via linear fits. The Pearson correlation coefficients are $r = 0.47$ for Large@256, $r = 0.79$ for Large@384, and $r = 0.72$ for XXLarge@384. The correlation strengthens at higher resolutions and larger model scales, suggesting that the semantic-predictive coupling becomes more pronounced as model capacity increases.

Why this motivates REPA for AR models. If the transition to prediction mode is tied to semantic understanding, then explicitly aligning an intermediate layer with pre-trained semantic features (via REPA) should accelerate this transition. The model doesn't need to discover semantics from scratch through the next-token objective — it can leverage DINOv2's already-structured feature space as a shortcut. This is analogous to how REPA helps diffusion models, but with the additional constraint that the alignment must occur before the mode switch to prediction, since aligning features that are already encoding next-token information with current-token DINOv2 features would create a representational conflict.

Optimal alignment depth (Table 3, Figure 8). The paper performs an extensive ablation across alignment layers {3, 4, 8, 12, 16, 20} in the 24-layer LlamaGen-Large:

• Performance improves with depth and peaks at layer 8: the best gFID is achieved when aligning the 8th layer (out of 24). This corresponds to approximately one-third of the total depth ($8/24 \approx 0.33$).
• Performance degrades beyond layer 8: aligning at layers 12, 16, or 20 yields progressively worse FID. The authors attribute this to the model having already transitioned to next-token prediction at those depths — forcing alignment with DINOv2 at positions where the features are encoding future-token information introduces a mismatch with the current-token supervision from DINOv2.
• The one-third rule generalizes across scales (Figure 8): testing on LlamaGen XLarge (36 layers) and XXLarge (48 layers), the optimal alignment depth shifts proportionally — approximately layer 12 for XLarge ($12/36 = 0.33$) and approximately layer 16 for XXLarge ($16/48 = 0.33$). The same proportional scaling holds for DiT models of different sizes (Figure 8, right panel), suggesting this is a general property of transformer-based generative models rather than an AR-specific phenomenon. The optimal depth is not a fixed absolute layer index but a fixed fraction of total depth — approximately 1/3.

Why one-third depth is optimal. The paper does not provide a theoretical justification, but the empirical pattern suggests an interpretation: the first third of the network performs local feature extraction and self-encoding (building up from low-level edges and textures to mid-level parts and object fragments), the middle third transitions to semantic understanding and causal reasoning (global structure, object identity, next-patch prediction), and the final third refines the predictions into the precise token distribution. Aligning at the 1/3 mark injects semantic knowledge at the boundary between local processing and global reasoning, giving the network a semantic scaffold just as it begins to construct high-level representations. Aligning earlier would waste the alignment on features that are too low-level to benefit from semantic supervision (edges and textures don't align well with DINOv2 object-level features); aligning later would conflict with the causal prediction objective.

Optimal alignment coefficient $\lambda$ (Table 3, Figure 9). The paper finds that LlamaGen-REPA requires a significantly stronger alignment regularization than DiT-REPA:

• Standard DiT-REPA: $\lambda = 0.5$ (the default from Yu et al., 2024)
• LlamaGen-REPA optimum: $\lambda = 2.0$ at the optimal alignment depth (layer 8)

The authors attribute this to "the strong inductive bias introduced by the teacher-forcing training scheme in autoregressive models." In teacher-forcing, the model always sees ground-truth previous tokens during training (not its own predictions), which creates a strong signal for next-token prediction. The REPA loss must compete with this strong teacher-forcing signal — a small $\lambda$ would be drowned out, while $\lambda = 2.0$ provides enough gradient to meaningfully influence the feature representations. In diffusion models, the denoising objective provides a weaker per-step signal (since the model sees noisy inputs, not clean ones), so a smaller $\lambda$ suffices.

Figure 9 further validates this with boxplots of evaluation FID across training for different $\lambda$ values at each alignment depth. The $\lambda = 2.0$ configuration at layer 8 produces both the best final FID (lowest box position) and fastest convergence (flattest box, indicating stability across checkpoints). The first checkpoint often appears as an outlier — the model's initial performance is noisy before it converges — after which performance stabilizes.

Target representation choice (Table 3). The paper uses DINOv2-Base's final-layer features as the target for REPA alignment, following the optimal configuration from the original DiT-REPA work. The table shows that this configuration (applied at layer 8 of LlamaGen-Large) outperforms the baseline without REPA. The choice of DINOv2 over other pre-trained models (DINOv1, CLIP, supervised models) is inherited from prior work and not ablated in this paper — the focus is on adapting the already-established optimal target representation to the autoregressive setting, not on rediscovering the best target.

3.4.10 The Controlled Benchmarking Setup: Comparing AR and Diffusion Fairly

The paper's secondary contribution — the benchmarking results in Figure 4 — depends on a carefully controlled experimental design that is worth detailing explicitly.

The fairness guarantee. Both DiT-Large and LlamaGen-Large use the exact same frozen iFSQ tokenizer with identical encoder and decoder weights. This means:

• The reconstruction quality of the latent space is identical for both models — any image generated by either model must pass through the same decoder, so differences in visual quality cannot be attributed to decoder quality differences.
• The information content available in the latent space is identical — both models have access to the same representational capacity (same $d$, same $L$, same spatial resolution).
• The compression ratio is identical — both models operate at the same bit rate, so neither has an unfair advantage in compactness.

What differs between the two models:

• Architecture: DiT processes all latent positions simultaneously with bidirectional attention (each position can attend to every other position); LlamaGen processes tokens sequentially with causal attention (each position can only attend to previous positions in raster order).
• Objective: DiT learns to denoise latents via a regression loss (MSE on velocity predictions); LlamaGen learns next-token prediction via a classification loss (cross-entropy over the discrete token vocabulary).
• Inference: DiT generates by iteratively denoising from pure random noise over 250 steps (the Euler sampler); LlamaGen generates tokens one-by-one in raster order until the full 256-token sequence is produced.

Why these differences matter for the benchmarking claim. Any performance gap between DiT and LlamaGen in Figure 4 must be attributed to one of these three differences (or their interactions). Since the tokenizer is held constant, we can rule out explanations like "DiT's VAE provides better latents than LlamaGen's VQ-VAE" or "the VQ-VAE's codebook collapse limits LlamaGen's effective capacity." The remaining explanations are structural: diffusion's parallel, holistic processing enables more global consistency but requires many denoising steps, while AR's sequential processing enables rapid initial learning (each token prediction is a simpler sub-problem) but imposes a strict ordering that may prevent the model from capturing certain types of long-range dependencies.

Compute matching in Figure 4. The x-axis in Figure 4 is total training FLOPs, not iterations. DiT-Large uses approximately 161.04 GFLOPs per training step; LlamaGen-L uses approximately 169.65 GFLOPs (with causal attention FLOPs counted as half of full attention, since the causal mask eliminates half the attention computation). The paper plots evaluation FID at multiple points during training, creating a FLOPs-vs-performance curve for each model. This allows answering the question: "Given a fixed compute budget, which model achieves better generation quality?" — which is the practical question a practitioner would ask when deciding which paradigm to invest in. The observed crossover (AR better at low compute, diffusion better at high compute) is only meaningful because the FLOPs accounting is matched and the tokenizer is shared.

Why the benchmarking is not fully exhaustive. Several caveats limit the generality of the benchmarking conclusions:

• Only one model size is compared (Large variants of both architectures). The proportional scaling of REPA depth (Figure 8) suggests that the AR-vs-diffusion crossover point might shift with model scale — but this is not tested.
• Only 256×256 resolution is benchmarked. The semantic-predictive correlation strengthens at higher resolutions (Figure 6 shows higher NTS-CKNNA correlation at 384 than 256), which could affect the relative scaling behavior.
• The training budgets are truncated (100k iterations for DiT, 500k for LlamaGen) — neither model is trained to full convergence, so the observed "ceiling" effect for AR models might partially reflect slower late-stage convergence rather than a fundamental capacity limit.
• Only the Euler sampler with 250 steps is used for DiT inference. More efficient samplers (DDIM, DPM-Solver) could shift the compute-performance curve.

4. Key Insights and Innovations

Innovation 1: The Distribution-Matching Activation as a Resolution of the Efficiency–Fidelity Trade-off

The paper's most fundamental intellectual contribution is not a new quantization scheme but rather the diagnosis and resolution of a hidden trade-off that prior work either missed or accepted as inevitable. The insight is that the conflict between information efficiency (maximizing codebook utilization) and reconstruction fidelity (minimizing quantization error) in FSQ is not fundamental to scalar quantization itself — it is an artifact of the activation function's failure to transform the latent distribution appropriately before quantization.

What the field assumed before this work. FSQ (Mentzer et al., 2023) was introduced as a "VQ-VAE made simple" — a drop-in replacement that eliminated the learnable codebook and its associated pathologies (collapse, straight-through estimator bias, memory cost) while retaining the discrete indexing that autoregressive models require. The implicit assumption was that FSQ inherited the same fidelity-efficiency characteristics as VQ-VAE, just simpler. Prior work did not identify or analyze the distribution mismatch as a first-class problem, nor did it recognize that the tanh activation creates a tension that the original FSQ design does not resolve. This is evidenced by the fact that the authors needed to construct the controlled toy experiment in Figure 1 to make the trade-off visible — it was not a known limitation in the literature.

What the iFSQ insight changes. By framing the problem as one of distribution matching — transforming the encoder's approximately Gaussian output into a uniform distribution before equal-interval quantization — the paper shows that the trade-off is avoidable. The key conceptual move is recognizing that the activation function f(z) in FSQ serves two separable roles: bounding (constraining latents to [−1, 1]) and distribution shaping (allocating probability mass across that range). The original tanh accomplishes the first role perfectly but the second poorly, producing a bimodal distribution that concentrates mass at the extremes and underutilizes central bins. By replacing tanh with a sigmoid of carefully chosen slope (2·σ(1.6·z) − 1), iFSQ performs both roles simultaneously: it bounds the latent range while reshaping the input Gaussian into an approximately uniform distribution.

The critical mathematical insight is that a uniform distribution is the provably optimal input for equal-interval quantization — it is the unique distribution for which equal-width bins achieve equal probability mass, maximizing both expected information content (100% bin utilization) and reconstruction precision (no need for adaptive bin widths). This is why Figure 1(c) shows iFSQ achieving both the low MSE of equal-interval quantization (0.1669) and the 100% utilization of equal-probability quantization — properties that Figure 1(a) and 1(b) demonstrate are mutually exclusive under tanh.

Significance beyond the performance gain. The reconstruction quality improvement (~0.5 dB PSNR gain at optimal α in Figure 3, corresponding to approximately 10% MSE reduction) is modest in absolute terms. The more important contribution is conceptual completeness: the paper closes a theoretical gap in the FSQ framework that, once pointed out, seems obvious in retrospect. The distribution mismatch was a latent bug, not a fundamental limitation. Future work on scalar quantization for generative models should not treat the activation function as an incidental design choice — it is a first-class architectural component that determines the effective codebook capacity for a given bit budget.

This is a fundamental insight rather than an incremental refinement because it changes why FSQ works well. Prior understanding was that FSQ's simplicity (no codebook, no collapse) was its primary advantage. The paper reveals that FSQ can also achieve near-optimal information efficiency if and only if the activation function is properly calibrated to the encoder's output distribution. This recasts FSQ from a pragmatic simplification of VQ-VAE into a principled quantization framework with a well-defined optimal configuration.

Innovation 2: iFSQ as a Unified Tokenizer Enabling Deconfounded Paradigm Benchmarking

The paper's second distinctive contribution is methodological rather than technical: the use of iFSQ as a controlled experimental platform for comparing autoregressive and diffusion image generation models, eliminating the tokenizer confound that has plagued all prior comparisons between these paradigms.

Why prior comparisons were confounded. Before iFSQ, every comparison between AR and diffusion models was, strictly speaking, a comparison of two systems — (VQ-VAE + AR) vs. (VAE + diffusion) — not two generative architectures. The VQ-VAE and VAE tokenizers differ along multiple axes: architectural design (learned codebook vs. continuous bottleneck), reconstruction quality (VQ-VAEs typically have lower rFID than VAEs at comparable compression), compression ratio (VQ-VAEs are dramatically more compact), training procedure (codebook losses, commitment losses, EMA updates for VQ-VAE; KL divergence for VAE), and gradient estimation (straight-through estimator for VQ-VAE; standard backpropagation for VAE). Any observed performance difference between an AR model and a diffusion model could be attributed to any of these confounds — or their interactions with the generative architecture — rather than to the AR-vs-diffusion distinction itself. The field had no way to isolate the variable of interest.

What iFSQ enables that no prior tokenizer could. iFSQ is unique among visual tokenizers in that a single quantization operation simultaneously produces two representations from the same latent grid: continuous values (via Equation 2 dequantization) suitable for diffusion models, and discrete indices (via Equation 3 base-$L$ expansion) suitable for autoregressive models. The encoder is identical. The decoder is identical. The reconstruction quality (rFID, PSNR, SSIM, LPIPS) is identical. The compression ratio is identical. The only difference is whether the generative model operates on the dequantized continuous values or the discrete token sequence.

This makes iFSQ a deconfounding device: by holding the tokenizer constant, any performance difference between LlamaGen and DiT in Figure 4 must be caused by the generative architecture (autoregressive sequential prediction vs. diffusion parallel denoising), the training objective (cross-entropy classification vs. MSE regression), or the inference procedure (token-by-token generation vs. iterative denoising). The observed crossover — AR models converging faster early in training but diffusion models reaching a superior performance ceiling — is credible precisely because the tokenizer confound is eliminated. Prior claims about AR-vs-diffusion scaling behavior (including "autoregressive model beats diffusion" from LlamaGen's original paper) were always vulnerable to the objection that the VQ-VAE tokenizer was the actual differentiator. With iFSQ, that objection is neutralized.

Significance as a methodological contribution. The benchmarking results themselves (AR converges faster, diffusion peaks higher) are intriguing but limited in generality — they are demonstrated on one dataset (ImageNet 256×256), at one model scale (Large variants of both architectures), with truncated training budgets (100k iterations for DiT, 500k for LlamaGen). The larger contribution is the benchmarking methodology itself. The paper demonstrates that a properly designed tokenizer can serve as a neutral platform for controlled experimentation on generative model architectures, much as standardized benchmarks in other fields (e.g., ImageNet for classification, GLUE for NLP) enable fair comparison by holding the evaluation protocol constant. This is an incremental-but-important methodological advance: it does not introduce a new technique for improving generation quality, but it provides a principled way to answer a question — "which generative paradigm is better, and under what conditions?" — that the field had been debating without adequate experimental controls.

The insight extends beyond this paper's specific findings. Future work comparing new generative architectures (masked image modeling, discrete diffusion, flow matching on discrete tokens) can use iFSQ (or similarly designed dual-mode tokenizers) to ensure that performance differences are not inadvertently attributed to the wrong component. This is the kind of methodological contribution that, if adopted, would improve experimental rigor across the field.

Innovation 3: The Proportional Scaling Law for Optimal Representation Alignment Depth

While the REPA technique itself was introduced by prior work (Yu et al., 2024), the paper's analysis of where to apply representation alignment in autoregressive models — and the discovery that the optimal depth follows a proportional scaling law rather than a fixed absolute layer index — is a genuinely new insight with implications beyond this paper.

What prior work assumed. The original REPA paper applied alignment at a fixed layer (layer 8 of DiT-Large) based on empirical tuning and did not investigate whether this choice generalizes across model scales. The implicit assumption — common in the literature on intermediate-layer supervision — was that the optimal alignment depth is a property of the training objective and the target representation, and should remain at roughly the same absolute position regardless of network depth. If layer 8 works for a 24-layer DiT-Large, the natural extrapolation would be to try layer 8 for a 48-layer DiT-XXLarge as well.

What the paper discovers instead (Figure 8). When the paper systematically sweeps alignment depth across LlamaGen variants (Large: 24 layers, XLarge: 36 layers, XXLarge: 48 layers) and DiT variants (same scales), the optimal depth shifts proportionally — approximately layer 8 for Large, layer 12 for XLarge, layer 16 for XXLarge. This corresponds to roughly one-third of total network depth (8/24 ≈ 12/36 ≈ 16/48 ≈ 0.33). The pattern holds for both autoregressive and diffusion architectures, suggesting it is a general property of transformer-based generative models rather than an AR-specific phenomenon.

Why this is a non-obvious result. There is no a priori reason that optimal alignment depth should scale proportionally with total depth. One could imagine several alternative hypotheses:

• Fixed absolute depth: the optimal alignment layer is determined by the semantic granularity of the target representation (DINOv2) and should remain at a specific absolute position regardless of network depth, because the transformation from low-level to high-level features at that depth is what matters.
• Fixed depth from output: alignment should be applied at a fixed distance from the final layer (e.g., always 3 layers before the output), ensuring that aligned features have sufficient remaining capacity to be refined into task-specific predictions.
• Depth-independent: any sufficiently early layer works equally well, and the optimal choice is simply "before the mode switch to prediction" irrespective of where that switch occurs in absolute terms.

The proportional scaling result falsifies all three hypotheses in favor of a fractional-depth rule: the optimal alignment point is at approximately one-third of total depth, regardless of absolute layer count. The paper connects this to the layer-wise semantic evolution analysis (Figure 6), which shows that the transition from self-encoding (high STS) to next-token prediction (high NTS) and semantic abstraction (high CKNNA) occurs at a specific relative depth that appears to scale with total network capacity. Deeper networks spread out their representational transformations across more layers, pushing the mode-switch boundary proportionally deeper in absolute terms while keeping it at the same relative position.

Significance as a diagnostic finding. This result is not about improving FID scores (though the correct alignment depth certainly helps) — it is about understanding the internal dynamics of generative transformers. The finding that the semantic-predictive transition occurs at a consistent relative depth across scales suggests that transformer depth is used in a structurally similar way regardless of absolute capacity: the first third for local feature extraction, the middle third for semantic abstraction and causal reasoning, the final third for output refinement. This is a coarse but useful model of how these networks allocate their representational budget across layers, and it provides actionable guidance for practitioners: when applying intermediate-layer supervision (REPA or similar), align at approximately one-third of total depth, not at the layer index that worked for a differently-sized model.

This is a fundamental insight about network behavior, not an incremental hyperparameter tuning result, because it reveals a structural regularity that was not previously known and that generalizes across architectures (AR and diffusion), scales, and (likely) tasks. It also provides a principled answer to a question that would otherwise require expensive per-model hyperparameter sweeps, making it practically significant as well as conceptually interesting.

Innovation 4: The Stronger Alignment Requirement for Autoregressive Models as a Diagnostic of Inductive Bias

A smaller but conceptually sharp insight emerges from the ablation study on the REPA loss coefficient λ (Table 3, Figure 9). The paper finds that LlamaGen-REPA requires a significantly stronger alignment regularization (λ = 2.0) than DiT-REPA (λ = 0.5, the standard from prior work). The authors attribute this to "the strong inductive bias introduced by the teacher-forcing training scheme in autoregressive models" — but the implication is deeper than that sentence suggests.

What this difference reveals. Teacher-forcing is the standard training protocol for autoregressive models: during training, the model always receives the ground-truth previous token as input, never its own prediction. This creates a training signal that is unusually strong compared to the denoising objective in diffusion models, where the model always sees noisy inputs and the target is a continuous value. The REPA loss — a cosine similarity between intermediate features and pre-trained DINOv2 features — is a comparatively weak signal. In a diffusion model, even a small λ = 0.5 provides a meaningful gradient because the denoising objective does not dominate the feature representations as aggressively. In an autoregressive model with teacher-forcing, the next-token prediction loss provides such a strong per-token training signal that the REPA loss is effectively drowned out at small λ — the features are pulled strongly toward the causal prediction objective, and the semantic alignment gradient cannot compete.

The fact that λ = 2.0 is needed (4× stronger than the standard setting) is therefore not merely a hyperparameter quirk — it is a quantitative fingerprint of teacher-forcing's inductive bias. The larger λ is required to overcome the strong pull of the next-token objective and inject semantic structure into the intermediate representations before the mode switch to prediction occurs. This is a rare case where a hyperparameter ablation provides insight into the underlying training dynamics rather than just tuning performance.

Significance as a cautionary finding. The implication for future work is that techniques developed for diffusion models cannot be ported to autoregressive models with their default hyperparameters — the teacher-forcing training scheme fundamentally changes the optimization landscape, making the model more resistant to auxiliary losses that operate on the feature representations. This is an incremental but practically important insight: it alerts researchers that AR-specific tuning is necessary when adapting techniques from the diffusion literature, and it provides a plausible mechanistic explanation (teacher-forcing's strong per-token gradient dominating weaker auxiliary losses) that can guide hyperparameter selection.

The insight also connects to broader questions about the cost of teacher-forcing: if the strong inductive bias requires 4× larger alignment coefficients to overcome, what other forms of representational supervision is teacher-forcing suppressing? This suggests that autoregressive models trained with teacher-forcing may be less receptive to auxiliary training signals in general — a hypothesis that could be tested (but is not in this paper) and that has implications for multi-task and multi-modal training of AR models.

Innovation 5: The 4-Bit "Sweet Spot" as an Empirical Equilibrium Between Discrete and Continuous Representations

The paper's scaling analysis across quantization levels (Figures 5, 10) identifies a consistent pattern: iFSQ reconstruction quality improves with bit depth and approaches the continuous AE baseline at approximately 4 bits per dimension, with diminishing returns beyond that point. This is not merely a "4 bits is good enough" finding — it represents a characterization of the discrete-continuous spectrum that has implications for how we think about representation learning for generation.

What the scaling analysis shows (Figure 5). Across PSNR, SSIM, LPIPS, and FID, iFSQ performance as a function of quantization level follows a consistent trajectory: substantial gains from 2 to 3 to 4 bits, then flattening or noisy improvement from 5 to 8 bits. At 4 bits, iFSQ is "comparable to AE" (Table 1 shows iFSQ at 4 bits achieving gFID 12.76 vs. AE at 13.78 without REPA, and 10.48 vs. 10.67 with REPA). At 7–8 bits, iFSQ is "nearly identical to AE." The compression ratio analysis in Appendix C (Figure 10) shows that on a log-scale compression ratio axis, all methods (iFSQ, AE, VQ) exhibit approximately linear scaling, with a clear "knee" around 48× compression — corresponding to the 4-bit configuration.

What makes this an insight rather than a measurement. Prior work on visual tokenization treated bit depth as either a binary choice (VQ-VAE at a fixed, high compression ratio vs. VAE at low compression) or a simple resource parameter (more bits = better). The iFSQ scaling analysis reveals that the relationship is S-shaped: at very low bits (1–2), the representation is too coarse to capture meaningful visual structure; from 2–4 bits, each additional bit provides substantial reconstruction improvement; beyond 4 bits, the marginal benefit of additional bits diminishes sharply, with performance asymptotically approaching the continuous AE baseline.

The 4-bit threshold is not an arbitrary finding — it represents the point at which the discrete representation has approximately enough capacity to encode the visual information that the continuous AE baseline captures with its 16-bit floating-point precision. Below 4 bits, the quantization bottleneck discards structurally important information; at 4 bits and above, the bottleneck is wide enough that the remaining differences are minor. This suggests that the visual information in a 256×256 natural image, when compressed to a latent representation of the dimensionality used here, requires roughly 4 bits per dimension to represent without substantial loss — a finding that is specific to this architecture and dataset but that provides a principled reference point for future tokenizer design.

Significance for the AR-vs-diffusion comparison. The 4-bit sweet spot is also the configuration at which the benchmarking comparison in Figure 4 is most meaningful. At lower bit depths, the quantization bottleneck would disadvantage the autoregressive model (which must predict discrete tokens exactly) more than the diffusion model (which can smooth over quantization errors during denoising). At higher bit depths, iFSQ essentially becomes a continuous AE with extra steps, and the comparison would lose its relevance to the discrete-vs-continuous paradigm question. The 4-bit equilibrium is where both paradigms operate under a meaningful but not crippling information constraint, making it the most informative point for comparing their inherent capabilities. This is a fundamental characterization of the design space, not an incremental tuning result, because it establishes a reference point that future comparisons between discrete and continuous generative models can anchor to.

5. Experimental Analysis

Evaluation Methodology

Dataset. All experiments use the ImageNet dataset (Deng et al., 2009) at 256×256 resolution. Tokenizer training uses the full ImageNet training set for 25 epochs. Generative model training and evaluation use the standard ImageNet training and validation splits. The paper additionally reports tokenizer reconstruction performance on the COCO2017 validation set (Lin et al., 2014) to verify that findings generalize beyond the training distribution — this is reported in Figures 3, 5, and 10 but the scale of COCO evaluation is not explicitly quantified (number of images not stated).

Base model(s). Two generative model families are used, both at Large scale:

• DiT-Large (Peebles & Xie, 2023): a transformer-based diffusion model operating on latent patches. At 256×256 resolution using 8× spatial compression, DiT-Large has approximately 161.04 GFLOPs per training step (stated in Figure 4). The architecture follows the original DiT paper exactly except for the tokenizer replacement.
• LlamaGen-Large (Sun et al., 2024): a decoder-only autoregressive transformer with 24 layers, predicting discrete image tokens in raster-scan order. At 256×256 resolution, LlamaGen-L has approximately 169.65 GFLOPs per step, where "the FLOPs for attention computation and value weighting are calculated as half of full attention" (Section 4.1.4) to account for the causal attention mask. For REPA experiments, additional LlamaGen scales are used: XLarge (36 layers) and XXLarge (48 layers), as well as corresponding DiT scales for the alignment depth scaling study (Figure 8).

The choice of Large-scale models for the primary comparison is deliberate — the authors state these models are used "to ensure the validity of the conclusions" (Appendix D). Both model families share the exact same frozen iFSQ tokenizer (same encoder and decoder weights), which is the key design choice enabling deconfounded benchmarking.

Metrics. Six quantitative metrics are reported across different experiment categories:

• Tokenier reconstruction: PSNR (peak signal-to-noise ratio, higher is better, measures pixel-level fidelity), SSIM (structural similarity index, higher is better, measures perceptual structure preservation as defined in Wang et al., 2004), LPIPS (learned perceptual image patch similarity, lower is better, measures deep-feature-level perceptual distance as defined in Zhang et al., 2018), and rFID (Fréchet Inception Distance for reconstruction, lower is better, measures distributional similarity between reconstructed and original images in Inception feature space as defined in Heusel et al., 2017).
• Generated image quality: gFID (Fréchet Inception Distance for generation, lower is better), computed between 50,000 generated images and the ImageNet validation set, following standard protocol. All gFID values are reported without classifier-free guidance (CFG), as explicitly stated in Tables 1 and 2.
• Distribution similarity (for activation tuning): RMSE (root mean square error between empirical and target uniform probability density, Equation 5) and KS statistic (Kolmogorov-Smirnov maximum CDF divergence, Equation 6), both lower is better, used only in Figures 2 and 3 to validate the α = 1.6 optimum.
• Layer-wise analysis metrics (Section 4.2.1): STS (self-token similarity, Equation 7), NTS (next-token similarity, Equation 8), and CKNNA (centered kernel alignment with nearest neighbor accuracy, inherited from Yu et al., 2024), used diagnostically in Figures 6 and 7 to understand LlamaGen's internal feature evolution and guide REPA depth selection.

Baselines. The paper compares against several established tokenizer and model configurations:

• AE (continuous autoencoder): A standard VAE-style architecture without KL divergence regularization, following the finding from MAETok (Chen et al., 2025) and VA-VAE (Yao et al., 2025) that diffusion models do not require a normally-distributed latent space. This serves as the continuous upper bound for reconstruction quality. Reported in Table 1 (gFID), Table 4 (PSNR/SSIM/LPIPS/rFID), and Figures 5/10 (scaling curves).
• VQ-VAE: The discrete tokenizer from LlamaGen (Sun et al., 2024) with a 16,384-codebook (14 bits), 16× spatial compression. Used as the discrete baseline in Tables 2 and 4, and Figure 10 (where it appears as the ⋆ marker). This is the tokenizer used in the original LlamaGen paper.
• Original FSQ (α = 2.0, equivalent to tanh): The unmodified FSQ from Mentzer et al. (2023), distinguished from iFSQ only by the activation function. Used in Tables 1 (DiT generation) and 2 (LlamaGen generation), Table 4 (reconstruction), and as the reference point at α = 2.0 in Figure 3.
• VAE (with KL divergence): A standard variational autoencoder with Gaussian prior regularization. Reported only in Table 4 (reconstruction metrics), where it underperforms AE, consistent with the finding that KL constraints are unnecessary for diffusion model tokenizers.
• DiT with AE tokenizer: The standard DiT-Large configuration from Peebles & Xie (2023), using a continuous AE rather than iFSQ. Serves as the diffusion baseline in Table 1.
• LlamaGen with VQ-VAE tokenizer: The standard LlamaGen-Large configuration from Sun et al. (2024). Serves as the AR baseline in Table 2.
• DiT-REPA and LlamaGen (no REPA): For the REPA experiments in Section 4.2, the baseline is either DiT-REPA with default settings (λ = 0.5, layer 8) or LlamaGen without REPA, as specified in Table 3.

Generation budget / compute accounting. For the tokenizer experiments (Sections 4.1.1–4.1.3, 4.1.5), the primary control variable is the compression ratio and bit depth, not training FLOPs — all tokenizers are trained for a fixed 25 epochs on ImageNet with the same optimizer settings (Adam, learning rate 0.001). For the generative model benchmarking (Section 4.1.4, Figure 4), compute is measured in total training FLOPs, not iterations, enabling a fair comparison between models with different per-step costs. DiT-Large uses 161.04 GFLOPs/step; LlamaGen-L uses 169.65 GFLOPs/step (with causal attention FLOPs halved). The FLOPs comparison is explicitly stated: "At 256 resolution, DiT-Large and LlamaGen-L exhibit approximately 161.04G and 169.65G FLOPs, respectively" (Figure 4 caption). For the REPA experiments, alignment-specific parameters (depth, λ) are ablated but all models within a given comparison use the same training iteration budget: 500k iterations for LlamaGen-Large (Table 3, Figure 8), 100k iterations for DiT-Large (Table 1, Figure 8 right panel).

For diffusion model inference, all reported gFID values use the Euler method with 250 function evaluations, following LightingDiT (Yao et al., 2025). For autoregressive inference, the LlamaGen paper's original settings are used. The paper does not report inference-time compute comparisons (e.g., diffusion sampling steps vs. AR token generation cost), focusing exclusively on training-time FLOPs efficiency.

Cross-validation / statistical protocol. The paper does not employ cross-validation or statistical significance testing. All metrics are reported as single-point estimates on the ImageNet validation set (50k images for gFID). For the optimal α selection in Figure 3, the optimum is identified visually from the plotted curves rather than through a formal model selection procedure. For the REPA depth selection in Figure 8, the optimal layer is identified as the one achieving the best gFID after 500k iterations. The paper notes that generalization to COCO is observed (Figures 3, 5, 10) but reports this qualitatively — no COCO-specific metric tables are provided, and the number of COCO evaluation images is not stated. For the layer-wise analysis in Figure 6, the linear fits between NTS and CKNNA report Pearson correlation coefficients (r = 0.47, 0.79, 0.72 for Large@256, Large@384, XXLarge@384 respectively) with 90% confidence intervals shown as shaded regions, but no p-values or significance tests are reported.

Main Quantitative Results

iFSQ Reconstruction Quality: Distribution Matching Translates to Measurable Gains

Headline result (Figure 3, Table 4). iFSQ at α = 1.6 consistently outperforms original FSQ (α = 2.0) across PSNR, SSIM, and LPIPS on ImageNet validation — the distributional optimum identified in the synthetic experiment (Figure 2) translates directly to reconstruction quality. At the optimal α = 1.6, the PSNR and SSIM curves in Figure 3 show clear peaks, with the KS and RMSE distribution metrics reaching their minima at the same α value. LPIPS reaches its best value at α = 2.4 (a slight deviation from the PSNR/SSIM optimum), but the authors choose α = 1.6 based on the joint optimum across PSNR and SSIM.

What the curves show. Figure 3 plots PSNR, SSIM, and LPIPS (primary y-axes) alongside KS and RMSE (secondary y-axis) for α ∈ {1.0, 1.2, 1.6, 1.8, 2.0, 2.4}. The trajectories are symmetric around α = 1.6: as α increases from 1.0 to 1.6, PSNR and SSIM increase while KS and RMSE decrease; beyond 1.6, the trend reverses — PSNR and SSIM decrease while KS and RMSE increase. The comovement between reconstruction quality and distribution uniformity is near-perfect, validating the paper's central claim that uniform-distributed latents are optimal for equal-interval quantization. The original FSQ (α = 2.0) sits on the downward slope for PSNR and SSIM, confirming that tanh's bimodal output distribution (Figure 2a, green curve) indeed harms reconstruction relative to the uniform-approximating iFSQ.

Quantitative comparison from Table 4. The tokenizer baseline table reports PSNR, SSIM, LPIPS, and rFID for AE, VAE, VQ-VAE, FSQ, and iFSQ under matched training conditions (25 epochs, ImageNet 256×256). While exact numbers from Table 4 are not quoted in the main text, the paper's narrative implies iFSQ achieves reconstruction metrics between VQ-VAE and AE — closer to AE at higher bit depths (4+ bits) and substantially better than VQ-VAE at comparable compression. The table demonstrates that iFSQ achieves reconstruction quality superior to original FSQ at the same bit rate and latent dimension, consistent with the Figure 3 curves.

Generalization to COCO. The paper states that "although training is conducted only on ImageNet, similar trends are observed on the COCO validation set" (Section 4.1.1), suggesting that the α = 1.6 optimum is a property of the distribution transformation rather than an ImageNet-specific artifact. This robustness check is qualitative — no COCO-specific metric tables are provided.

iFSQ for Diffusion Image Generation: Matching or Exceeding AE Quality with Better Compression

Headline result (Table 1). Using iFSQ as the tokenizer for DiT-Large yields better gFID (12.76) than the continuous AE tokenizer (13.78) at 100k training iterations, while achieving a 3× higher compression ratio (96 vs. 24). With REPA, iFSQ at 4 bits achieves gFID 10.48, comparable to AE's 10.67. This is the key finding supporting the claim that iFSQ matches continuous quality while providing the additional capability of discrete token generation.

Table 1 breakdown by rows:

• AE baseline: DiT-L/2 with standard AE tokenizer achieves gFID 13.78 (10.67 with REPA) at compression ratio 24. This is the reference point — any iFSQ configuration that matches or beats this number while achieving higher compression is considered successful.
• FSQ baseline: Original FSQ at 4 bits achieves gFID 13.38 (11.04 with REPA) at compression ratio 96. Worse than both AE and iFSQ, confirming that the distribution mismatch in original FSQ harms downstream generation quality, not just reconstruction metrics. The gap is modest without REPA (13.38 vs. 12.76, a 0.62 point difference) but widens with REPA (11.04 vs. 10.48, a 0.56 point difference).
• iFSQ at 4 bits: gFID 12.76 (10.48 with REPA), compression ratio 96. Outperforms AE at 4× the compression ratio without REPA, and matches AE with REPA. This is the configuration the paper identifies as the "sweet spot."
• iFSQ at 2 bits: gFID 18.52 (14.97 with REPA), compression ratio 192. Substantially worse than AE — the 2-bit bottleneck discards too much information for the diffusion model to compensate.
• iFSQ at 5–8 bits: gFID values fluctuate between 12.76 and 15.02 without REPA, and between 10.48 and 10.77 with REPA. There is no consistent improvement beyond 4 bits — the 5-bit configuration (14.35 without REPA, 10.77 with REPA) is actually worse than the 4-bit configuration in both settings. This non-monotonicity is important: it suggests that at 4 bits, the latent representation already captures most of the information the diffusion model needs, and additional bits either provide redundant capacity or introduce optimization difficulties (possibly due to the larger latent space making the denoising objective harder).

What this pattern implies. The saturation at 4 bits is not an artifact of insufficient training — all models are trained for the same 100k iterations. It suggests a genuine representational ceiling: the information in a 256×256 ImageNet image, when compressed through the iFSQ encoder-decoder architecture, requires approximately 4 bits per latent dimension to be adequately represented. Below this threshold, the quantization bottleneck discards structurally important information. At or above this threshold, the bottleneck is wide enough that the remaining differences are minor and may be dominated by training noise or optimization dynamics rather than representational capacity.

Comparison with original FSQ. The consistent underperformance of original FSQ (α = 2.0) relative to iFSQ (α = 1.6) at the same 4-bit, 96× compression setting validates that the distribution-matching activation matters for downstream generation, not just for reconstruction metrics. The 0.62 gFID gap without REPA and 0.56 gap with REPA are modest but consistent, and they represent a "free" improvement with no additional parameters, latency, or training cost — just the one-line activation change.

iFSQ for Autoregressive Image Generation: Better FID at Lower Bit Rates than VQ-VAE

Headline result (Table 2). At the same latent dimension, LlamaGen-Large trained on iFSQ tokens achieves better gFID than VQ-VAE tokens, while iFSQ operates at a lower bit rate. Performance peaks at 4 bits, with larger bit depths (and thus larger implicit codebooks) not yielding better results — the authors conjecture that "as the codebook grows, the corresponding autoregressive model must also scale to provide sufficient capacity to predict such a large codebook" (Section 4.1.3).

Table 2 specifics. The table compares four tokenizer configurations for LlamaGen-Large, all evaluated with LlamaGen-REPA:

• VQ-14 bit (16,384 codebook): The original LlamaGen configuration using VQ-VAE with 16× spatial compression. Achieves the baseline gFID for autoregressive generation.
• FSQ-4 bit: Original FSQ at 4 bits. Performance is intermediate — better than VQ but worse than iFSQ, consistent with the diffusion results in Table 1.
• iFSQ-4 bit: The optimal configuration. Achieves the best gFID among all tested tokenizers at this compute budget (500k iterations). Operates at lower bit rate than VQ while providing better generation quality.
• iFSQ-X bit (various): Additional iFSQ configurations at different bit depths. The paper reports that "larger bits (and thus larger codebooks) do not necessarily yield better results and performance peaks at 4 bits." Specific gFID numbers for non-4-bit configurations in the autoregressive setting are not individually quoted in the text but are implied to follow the same saturation pattern observed for diffusion in Table 1.

Why larger codebooks can hurt AR models. This is a non-obvious finding. In VQ-VAE, larger codebooks generally improve reconstruction quality (more codes = finer discretization), and the autoregressive model is expected to scale accordingly. The paper's finding that iFSQ performance peaks at 4 bits suggests a mismatch between codebook capacity and model capacity: an autoregressive model with fixed size (LlamaGen-Large, 24 layers) may lack sufficient representational capacity to model the full joint distribution over a very large discrete vocabulary. When the implicit codebook size is $L^d$, increasing $L$ (more levels per channel) or $d$ (more channels) both increase the vocabulary size, making the next-token prediction task harder — the model must discriminate among more possible tokens at each position. For a 4-bit iFSQ with, say, $d=4$ and $L=5$ (so $2^K+1 = 5$ means $K=2$, $\log_2(5) \approx 2.32$ bits per channel, total bits $\approx 9.3$, vocabulary size $5^4 = 625$), the vocabulary is tractable. Increasing to 8 bits with the same $d=4$ would require $L=17$ levels, giving a vocabulary of $17^4 \approx 83,521$ — a ~134× larger output space that the same 24-layer model must learn to predict. The model capacity bottleneck dominates any representational benefit from the finer discretization.

Comparison with VQ-VAE. The finding that iFSQ outperforms VQ-VAE at lower bit rates is consistent with the paper's theoretical analysis (Appendix B): iFSQ achieves higher effective codebook utilization (100% after distribution matching) compared to VQ-VAE (which suffers from codebook collapse, typically utilizing only a fraction of its nominal codebook size). Even though VQ-VAE nominally has a larger vocabulary (16,384 codes), if only a subset are actively used, the effective information capacity may be lower than iFSQ's $L^d$ codes with 100% utilization. This is the fundamental advantage of scalar quantization over learned codebook lookup — no capacity is wasted on unused codes.

Training Efficiency Comparison: AR Converges Faster, Diffusion Peaks Higher

Headline result (Figure 4). Under compute-matched training with the same iFSQ tokenizer, LlamaGen (AR) exhibits faster initial convergence but DiT (diffusion) achieves a superior performance ceiling. The paper plots gFID against total training FLOPs for both models, showing a crossover: early in training (low FLOPs), LlamaGen achieves lower gFID than DiT; beyond the crossover point, DiT continues to improve while LlamaGen's gains diminish, resulting in DiT reaching a better final gFID at the full training budget.

Specific FLOPs and FID values. Figure 4 plots gFID as a function of cumulative training FLOPs for DiT-Large (161.04 GFLOPs/step) and LlamaGen-L (169.65 GFLOPs/step). The exact gFID values at specific FLOP points are read from the figure rather than tabulated, but the qualitative pattern — AR curve starts lower and flattens earlier, diffusion curve starts higher but continues descending — is clear. Both models use their optimal training configurations (derived from ablations) with the same iFSQ tokenizer.

What the crossover means. The observation that AR models exhibit "rapid initial convergence" is mechanistically plausible: each training step of an autoregressive model makes N = h×w independent token predictions (one per spatial position), each with a strong teacher-forcing signal (the ground-truth token is provided as input). The model learns local texture patterns and short-range dependencies quickly because the next-token objective decomposes the image into many simpler sub-problems. Diffusion models, in contrast, process all positions simultaneously but with a weaker per-step signal (regression to continuous values from noisy inputs), which may require more iterations to learn meaningful representations.

However, the AR curve flattens while the diffusion curve continues improving — the paper interprets this as evidence that "the strict sequential constraint is suboptimal for image generation" (Section 4.1.4) and that "strict sequential ordering may limit the upper bounds of generation quality" (Abstract). The causal attention mask prevents each token from attending to future positions, which may limit the model's ability to capture certain global image structures — symmetric patterns, long-range color consistency, or global layout constraints that require non-causal reasoning. The diffusion model, with its bidirectional attention, faces no such constraint and can refine the entire image holistically.

Important caveat on the observed ceiling. The full training budget is 500k iterations for LlamaGen and 100k iterations for DiT (different iteration counts but similar FLOPs due to different per-step costs). The observed AR "ceiling" might partially reflect slower late-stage convergence rather than a fundamental capacity limit — the paper does not demonstrate that LlamaGen's performance has truly saturated at 500k iterations (i.e., that additional training would not yield further improvement). The claim of a "superior performance ceiling" for diffusion is therefore contingent on the specific training budgets used. Longer training of the AR model might narrow or close the gap, though the flattening trend in Figure 4 suggests diminishing returns for AR at this model scale.

Scaling Behavior Across Quantization Levels: The 4-Bit Equilibrium

Headline result (Figure 5). Across PSNR, SSIM, LPIPS, and FID, iFSQ reconstruction quality improves monotonically with quantization level and approaches the continuous AE baseline at approximately 4 bits per dimension on ImageNet. The trend generalizes to COCO (shown in the same figure). At 2 bits, iFSQ with twice the latent dimension (iFSQ-2×dim) already surpasses AE at the original dimension, indicating strong scalability — increased latent dimensionality can compensate for reduced per-channel precision.

Figure 5 specifics. The figure plots performance metrics (y-axis) against quantization level for iFSQ and AE at different latent dimensionalities (marker size indicates dimensionality). Horizontal dashed lines show the AE baseline performance under mixed-precision training with 16-bit inference. Key observations:

• Monotonic improvement: As quantization level increases, all iFSQ curves rise toward the AE baseline. The gap between iFSQ and AE narrows systematically.
• 4-bit convergence: At 4 bits, iFSQ curves are close to the AE baselines for all metrics. The paper states that "iFSQ approaches AE around 4 bits, is nearly identical to AE at 7–8 bits."
• Dimensionality scaling: At any fixed quantization level, larger latent dimensions (larger markers, higher curves) improve performance. The iFSQ-16dim curve exceeds AE in PSNR and SSIM, demonstrating that iFSQ at high bit rates can surpass continuous autoencoders in reconstruction quality.
• 2-bit with doubled dim: The iFSQ-2×dim at 2 bits already surpasses AE at the original dimension — a practically important finding because it shows that the bit budget can be reallocated between per-channel precision and total channel count.
• VQ underperformance: The VQ data points in the figure perform worse than iFSQ or AE at comparable dimensions, consistent with the claim that "learning a quantization scalar is easier than learning quantization embeddings" (Section 4.1.5).

Compression ratio scaling (Figure 10, Appendix C). When plotted against compression ratio on a log scale, all methods (iFSQ, AE, VQ) exhibit approximately linear scaling — better reconstruction at lower compression ratios (more bits per latent). A clear knee point appears around 48× compression, which corresponds to the 4-bit iFSQ configuration. The VQ-VAE data point (⋆ in the figure, compression ratio approximately 438) lies on the same scaling trend line, suggesting that VQ-VAE's reconstruction quality is predictable from its compression ratio and is not architecturally disadvantaged relative to iFSQ. However, at comparable compression ratios, iFSQ achieves better quality because it operates at a more favorable point on the bit-depth vs. dimensionality trade-off curve.

What Figure 10 demonstrates about the discrete-continuous spectrum. The log-linear relationship between compression ratio and reconstruction quality (PSNR, SSIM, LPIPS, rFID) is the empirical signature of rate-distortion trade-offs. The fact that AE, iFSQ, and VQ all lie approximately on the same curve suggests that, at a given compression ratio, the fundamental information bottleneck — not the specific quantization mechanism — determines reconstruction quality. iFSQ's advantage is that it can operate at compression ratios that are inaccessible to AE (which cannot go below the 16-bit floating-point floor) while maintaining quality closer to the continuous upper bound than VQ can achieve at the same compression ratio.

REPA for Autoregressive Models: Optimal Depth Follows a One-Third Rule

Headline result (Table 3, Figure 8). LlamaGen-REPA achieves its best gFID when aligning the 8th layer (out of 24) of LlamaGen-Large with DINOv2-Base final-layer features, using a stronger alignment coefficient (λ = 2.0) than the standard DiT-REPA setting (λ = 0.5). The optimal alignment depth scales proportionally with total model depth — approximately one-third of total layers — across both LlamaGen (Large: 8/24, XLarge: 12/36, XXLarge: 16/48) and DiT variants (same pattern in Figure 8 right panel).

Table 3 specifics. The table reports gFID at 500k iterations for LlamaGen-Large under various REPA configurations:

• Baseline (no REPA): not explicitly tabulated but implied as the reference.
• Target representation ablation: Aligning with DINOv2-Base (final layer features) at layer 8 with the DiT-REPA default settings (λ = 0.5) achieves the best result. The paper states they "empirically apply the optimal REPA parameters on DiT to LlamaGen" and find that this configuration "enables the optimal diffusion-model configuration to also accelerate convergence in the autoregressive image generation model" (Section 4.2.3). Alternative target representations are not ablated in detail — the focus is on depth and λ.
• Alignment depth ablation (fixed λ): Aligning at layers 3, 4, 8, 12, 16, and 20 (out of 24). Performance improves with depth, peaks at layer 8, and then degrades. The paper attributes the degradation beyond layer 8 to the model having shifted toward next-token prediction — aligning features that encode future-token information with current-token DINOv2 features creates a representational conflict.
• Alignment coefficient λ ablation: At the optimal depth (layer 8), λ = 2.0 achieves the best gFID, outperforming λ = 0.5 (standard DiT-REPA) and other tested values. The paper attributes the need for stronger alignment to "the strong inductive bias introduced by the teacher-forcing training scheme" — the next-token prediction loss dominates the feature gradients, and a larger λ is needed for the REPA loss to meaningfully influence representations.

Figure 8 validation across scales. The left panel shows LlamaGen FID vs. alignment layer for Large (24 layers), XLarge (36 layers), and XXLarge (48 layers). The optimal depths are approximately layer 8, 12, and 16 respectively, all clustering around 1/3 of total depth. The right panel shows the same pattern for DiT variants, confirming this is not an AR-specific phenomenon. The paper annotates the plots with "layer/total" ratios and highlights an "optimal region" at approximately 1/3 depth. This is a robust finding — the pattern holds across model scales (3× scaling in depth) and architectures (AR and diffusion).

Layer-wise analysis supports the depth choice (Figures 6, 7). Figure 6 shows that the mode switch from self-encoding (high STS) to next-token prediction (high NTS) and semantic abstraction (high CKNNA) occurs in the middle-to-late layers of LlamaGen. The strong positive correlation between NTS and CKNNA (Pearson r = 0.47–0.79 depending on model scale and resolution, with the correlation strengthening at larger scales) suggests that semantic understanding and causal prediction co-emerge. Figure 7 demonstrates that REPA alignment "effectively controls the semantic trajectory": the layer exhibiting the highest CKNNA score shifts to coincide with whichever layer is targeted for alignment, and this holds regardless of the loss coefficient λ. This validates that REPA actively shapes the feature evolution rather than merely providing a weak regularizer.

The λ = 2.0 finding (Figure 9). The boxplot analysis across different λ values at the optimal depth (layer 8) shows that λ = 2.0 achieves both the best final FID (lowest box position) and fastest convergence (flattest box, indicating stability across training checkpoints). The first checkpoint often appears as an outlier (high FID, large variance) before convergence stabilizes. The 4× larger λ compared to DiT-REPA is a practically significant finding — researchers porting REPA (or similar alignment techniques) from diffusion to autoregressive models should not assume the same hyperparameters will transfer.

Ablation Studies and Robustness Checks

Activation slope α on reconstruction quality (Figure 3): The optimal α = 1.6 is validated against α ∈ {1.0, 1.2, 1.6, 1.8, 2.0, 2.4} using PSNR, SSIM, and LPIPS on both ImageNet and COCO validation sets. The distributional metrics KS and RMSE track reconstruction quality near-perfectly — both reach their minima at α = 1.6, where PSNR and SSIM peak. LPIPS reaches its best at α = 2.4, a deviation the authors note but override in favor of the PSNR/SSIM optimum. This is a robustness win: the α = 1.6 choice is not cherry-picked from a narrow grid — it represents the joint optimum of two distinct distributional similarity measures and two reconstruction quality metrics, all peaking at the same value.

Spatial compression and dimensionality scaling (Figures 5, 10, Appendix C): The paper sweeps latent dimension (marker size in Figure 5) and spatial compression factor (64× in Figures 5, 256× in Figure 10) to verify that the 4-bit sweet spot is not an artifact of a particular architectural configuration. At both compression levels, the same qualitative pattern holds: iFSQ approaches AE quality at ~4 bits, and performance scales approximately linearly with log compression ratio. The VQ-VAE data point in Figure 10 lies on the same scaling trend line, providing convergent validity — the rate-distortion trade-off appears to be a fundamental property of the information bottleneck, not an artifact of iFSQ's specific quantization scheme.

REPA target representation (Table 3): Only DINOv2-Base final-layer features are tested as the alignment target, following the optimal configuration from the original DiT-REPA paper. Alternative targets (DINOv1, CLIP, supervised models, intermediate DINOv2 layers) are not ablated. This is a limitation — the paper inherits rather than validates the target choice, and it's possible that a different target representation would change the optimal depth or λ.

REPA alignment depth across architectures and scales (Figure 8): This is the most thorough ablation in the paper, testing 6+ depth positions per model across 3 scales (Large, XLarge, XXLarge) for both LlamaGen (AR) and DiT (diffusion). The consistent 1/3-depth optimum across all configurations is the strongest evidence for the proportional scaling law claim. The paper does not report statistical confidence intervals on the optimal depth — the optimum is identified as the lowest FID point on the curve — but the qualitative consistency across 6 model configurations makes the finding credible without formal statistics.

REPA alignment coefficient λ (Table 3, Figure 9): At the optimal depth (layer 8), λ ∈ {0.5, 1.0, 2.0, 3.0} is tested. λ = 2.0 achieves the best FID. The boxplot analysis in Figure 9 additionally tests interaction effects between depth and λ, showing that λ = 2.0 at layer 8 is the joint optimum, and that this configuration also achieves the fastest convergence (flattest box). The finding that AR models need stronger alignment than diffusion models is consistent across depths — at suboptimal depths, larger λ also tends to improve performance, though the absolute FID is worse than at the optimal depth.

KL divergence ablation for continuous tokenizers (Table 4 text, Appendix D): The paper notes that "continuous VAE reconstruction performance is inferior to that of AE" and that prior work (MAETok, VA-VAE) demonstrates "the convergence of diffusion models does not depend on the constraint that the latent space maintains a standard normal distribution." Consequently, the AE baseline (without KL loss) is used for all diffusion experiments. This is a robustness-adjacent finding: the KL divergence constraint — standard in VAE training — is actively harmful for diffusion model tokenizers. The paper does not ablate KL weight (e.g., β-VAE style) to find whether a small amount of KL regularization could be beneficial; it simply removes it entirely based on prior work.

Generalization to COCO (Figures 3, 5, 10): The paper claims that "similar trends are observed on the COCO validation set" for the α sweep (Figure 3), the quantization level scaling (Figure 5), and the compression ratio scaling (Figure 10). However, the COCO data is plotted in the same figures without separate metric tables, making it difficult to assess quantitatively whether the trends are truly identical or merely similar in direction. The number of COCO evaluation images is not stated. This is a robustness check that could be strengthened with explicit out-of-distribution metric reporting.

Critical Assessment

Claim: iFSQ resolves the fidelity–efficiency trade-off via distribution matching.

What was demonstrated: Figure 3 shows that α = 1.6 achieves better PSNR and SSIM than α = 2.0 (original FSQ) while maintaining better bin utilization (theoretically, since a uniform distribution guarantees 100% utilization for equal-interval quantization). The reconstruction quality improvement is real and consistent across metrics.

What was not demonstrated: The paper does not directly measure bin utilization for iFSQ vs. original FSQ on actual trained tokenizers — the utilization claim is theoretically grounded (uniform distribution + equal-interval bins = 100% utilization) but not empirically validated by, e.g., measuring the fraction of quantization bins that receive non-trivial probability mass during inference on ImageNet. The synthetic experiment in Figure 1 uses 9 levels and a clipped Gaussian; the actual iFSQ tokenizer uses many more levels (L = 3 to 17 depending on K) operating on learned encoder outputs that may deviate from a perfect Gaussian. Whether the distribution-matching property holds post-training (after the encoder has adapted to the new activation function) is not explicitly verified.

Genuine weakness: The reconstruction quality gain from iFSQ over FSQ is modest — Figure 3 shows PSNR improvement of approximately 0.3–0.5 dB and SSIM improvement of a few thousandths at α = 1.6 vs. α = 2.0. For generation (Tables 1, 2), the gFID improvement is 0.62 points (DiT without REPA) and 0.56 points (DiT with REPA) — noticeable but not transformative. The paper's central contribution is the conceptual resolution of the trade-off and the benchmarking platform it enables, not a dramatic performance leap. This is a strength of the paper's honesty but should temper interpretations that iFSQ substantially outperforms FSQ in practice.

Claim: 4 bits per dimension is the optimal equilibrium between discrete and continuous representations.

What was demonstrated: Across PSNR, SSIM, LPIPS, and generation gFID (Tables 1, 2; Figures 5, 10), iFSQ performance approaches the AE baseline at approximately 4 bits and shows diminishing returns or fluctuation beyond that. The pattern is consistent across tokenizer reconstruction and downstream generation, ImageNet and COCO, 64× and 256× spatial compression.

What was not demonstrated: The 4-bit threshold is empirical and may be specific to (a) the ImageNet dataset complexity, (b) the 256×256 resolution, (c) the encoder-decoder architecture capacity, and (d) the specific latent dimensionality used. For higher-resolution images, more complex scenes, or higher-fidelity requirements, the threshold could shift to higher bit depths. The paper does not test resolutions other than 256×256 for generation, nor does it test on datasets substantially different from ImageNet (e.g., higher-resolution natural images, medical images, text, or other modalities). The "4-bit sweet spot" should be understood as an empirical characterization of this specific setup, not a universal constant.

Genuine strength: Despite the specificity, the finding is practically useful: it provides a concrete recommendation (4 bits) for practitioners building tokenizers with iFSQ, and the scaling analysis in Figures 5 and 10 provides the tools to extrapolate to other configurations (if you need higher quality, you know approximately how many additional bits are required).

Claim: Under identical reconstruction constraints, AR models converge faster but diffusion models achieve a superior performance ceiling.

What was demonstrated: Figure 4 shows a crossover in gFID vs. training FLOPs for DiT-Large and LlamaGen-Large using the same iFSQ tokenizer at 4 bits. The AR curve starts lower and flattens earlier; the diffusion curve starts higher but continues descending.

What was not demonstrated: Several caveats limit the generality of this finding:

• Only one model scale is compared. The proportional scaling of REPA depth (Figure 8) hints that model-scale interactions exist, but the AR-vs-diffusion comparison is not repeated at XLarge or XXLarge. It is possible that the crossover point shifts with model scale — e.g., larger AR models might break through the observed ceiling, or larger diffusion models might converge even faster relative to AR.
• Only one resolution is tested (256×256). The layer-wise analysis (Figure 6) shows that the NTS-CKNNA correlation strengthens at higher resolution (r = 0.47 at 256 vs. r = 0.79 at 384), suggesting that resolution may affect the relative scaling behavior.
• Training budgets are truncated. DiT is trained for only 100k iterations, LlamaGen for 500k iterations. While the FLOPs axis accounts for per-step cost differences, neither model is trained to full convergence. The observed AR "ceiling" might partially reflect slower late-stage convergence rather than a fundamental capacity limit. The paper does not demonstrate that AR performance has saturated — the curve appears to flatten, but without training for 2× or 4× longer, a true asymptote cannot be confirmed.
• Inference compute is not accounted for. At inference time, DiT uses 250 Euler sampling steps, each requiring a full forward pass through the transformer. LlamaGen generates tokens autoregressively, requiring 256 sequential forward passes (one per token). The AR model is likely slower at inference time despite similar per-step training FLOPs, and this latency difference is not discussed. The "superior performance ceiling" of diffusion may come at the cost of much higher inference latency, which matters for practical deployment.
• Only one AR architecture (LlamaGen) and one diffusion architecture (DiT) are tested. The finding that "strict sequential ordering may limit the upper bounds of generation quality" is attributed to the causal attention mask in AR models, but alternative AR architectures (e.g., those using different token orderings, bidirectional refinement, or masked modeling objectives) might not face the same limitation. The claim is specific to raster-scan causal AR transformers, not to autoregressive generation in general.

What would strengthen the claim: Training both models to convergence (or at least demonstrating saturation), testing at multiple model scales and resolutions, comparing inference-time cost, and including alternative AR architectures (masked models, non-causal orderings) would substantially increase confidence. The current evidence is suggestive but not definitive.

Claim: Optimal REPA alignment depth follows a 1/3-depth proportional scaling law.

What was demonstrated: Figure 8 shows that across 3 LlamaGen scales (24, 36, 48 layers) and 3 DiT scales, the optimal alignment depth is consistently approximately one-third of total depth. The layer-wise analysis in Figure 6 provides a mechanistic rationale (the mode switch from self-encoding to prediction occurs at a consistent relative depth).

What was not demonstrated: The paper tests only three model scales and two architectures. While the pattern is internally consistent, the claim of a "scaling law" (which implies a mathematical relationship that extrapolates) would require testing at more scales (e.g., Small, Base, Huge) to verify that the 1/3 fraction holds across a wider range of depths. The finding is more accurately described as a "consistent proportional relationship" than a "scaling law" in the Chinchilla sense.

Genuine strength: The dual-architecture validation (AR and diffusion) makes this finding more credible than if it were AR-specific. The fact that the same fractional depth works for architectures with fundamentally different attention patterns and training objectives suggests it reflects a general property of how transformer depth is allocated for visual generative tasks. The practical implication — align at 1/3 depth, don't just reuse the layer index from a differently-sized model — is immediately actionable and saves expensive per-model hyperparameter sweeps.

Claim: AR models require stronger REPA alignment (λ = 2.0) than diffusion models (λ = 0.5).

What was demonstrated: Table 3 and Figure 9 show that λ = 2.0 at layer 8 achieves the best gFID for LlamaGen-Large. The paper attributes this to teacher-forcing's strong inductive bias.

What was not demonstrated: The mechanistic explanation (teacher-forcing dominating the REPA gradient) is plausible but not empirically verified. The paper could have measured the relative gradient magnitudes of the next-token prediction loss and the REPA loss during training to confirm that the REPA gradient is indeed weaker at small λ. Without such measurements, the explanation remains a post-hoc interpretation.

Genuine strength: The finding itself — that λ = 2.0 works best — is empirically robust (tested across multiple depths in Figure 9). The practical recommendation (use larger λ when porting REPA to AR models) is clear and supported. The theoretical explanation, while unverified, provides a useful mental model for practitioners.

Overall experimental design weaknesses:

• No statistical significance reporting. All metrics are single-point estimates without confidence intervals, standard deviations, or significance tests. The 50k-image ImageNet validation set provides sufficient sample size that metric differences of the magnitude reported (e.g., 0.5 gFID between methods) are likely meaningful, but the absence of any variance characterization makes it impossible to assess whether, for example, the 0.62 gFID gap between FSQ and iFSQ in Table 1 is statistically reliable or within training noise.
• Single dataset for generation. All generative modeling experiments use ImageNet 256×256. The generalization to COCO is shown only for tokenizer reconstruction (Figures 3, 5, 10), not for downstream generation. ImageNet is a specific distribution (object-centric, relatively clean backgrounds, limited diversity of scene types), and the findings may not transfer to more diverse or higher-resolution image distributions.
• Truncated training budgets. DiT uses only 100k iterations — enough to show relative ordering but far from convergence. The "superior performance ceiling" claim for diffusion is based on extrapolation from a partially-trained model. Longer training could change the relative ordering or reveal different scaling behavior.
• Missing ablation: encoder adaptation to iFSQ. The paper sweeps α on a fixed encoder that was trained with the original FSQ activation? Or does each α value correspond to a separately trained tokenizer? The text implies that each α point in Figure 3 reflects a separately trained tokenizer (since "all tokenizers are trained for 25 epochs on ImageNet" per Appendix D), but this is not explicit. If the encoder is retrained for each α, then part of the performance difference may come from the encoder adapting to the new activation's distribution, not just the improved quantization. This is fine for the practical claim ("iFSQ works better"), but it complicates the theoretical claim ("uniform distributions are optimal for equal-interval quantization") because the latent distribution may no longer be exactly Gaussian after encoder adaptation.

6. Limitations and Trade-offs

Assumption: iFSQ Serves as a Drop-In Replacement Requiring No Training Recipe Changes

The paper positions iFSQ as a one-line code change — replace tanh(z) with 2 * sigmoid(1.6 * z) - 1 — and presents it as "a computationally free, plug-and-play module compatible with existing architectures" (Section 3.2). The implicit assumption is that this activation swap is sufficient to realize the benefits, with no adjustments to the encoder architecture, training hyperparameters, or loss function required.

The consequence. The activation change is genuinely minimal in terms of code, but it is not a drop-in replacement in the sense that practitioners can swap the activation in a pre-trained FSQ tokenizer and immediately obtain iFSQ's benefits. The encoder must be retrained to adapt its output distribution to the new activation function — the distribution-matching property assumes the encoder output is approximately Gaussian, but the encoder learns to produce whatever distribution minimizes the reconstruction loss. If the encoder was co-trained with tanh, it may have learned to produce activations that, when passed through tanh, yield a distribution that is not Gaussian but rather something tanh handles well. Retraining the encoder with the iFSQ sigmoid is necessary for the latent distribution to converge to the near-uniform state that Figure 2 demonstrates, and the paper's experiments do train tokenizers from scratch for each configuration (Appendix D: "all tokenizers are trained for 25 epochs on ImageNet").

Moreover, the α = 1.6 optimum is derived assuming the encoder produces standard normal-distributed activations. If the encoder architecture, initialization, or normalization layers change (e.g., different activation functions in the encoder body, BatchNorm vs. LayerNorm placement, varying numbers of residual connections), the output distribution may deviate from the Gaussian assumption, and the optimal α could shift. The paper does not test whether α = 1.6 generalizes across different encoder architectures, nor does it provide a method for determining the optimal α for a given encoder without re-running the full parameter sweep in Figure 2.

What evidence exists in the paper. Section 3.2's derivation of α = 1.6 uses synthetic data — 500k samples from a standard normal distribution — not real encoder outputs. The figure caption for Figure 2 states: "Sample 500k points from the standard normal distribution and compute the transformed distribution for several values of α." This is a clean controlled experiment, but it assumes that trained encoder latents approximate a standard normal, which may hold for the specific architecture used but is not guaranteed in general. The paper validates the α = 1.6 choice on ImageNet-trained tokenizers (Figure 3) and notes similar trends on COCO (Section 4.1.1), but these use the same encoder architecture trained under the same protocol. The robustness of α = 1.6 to architectural variations — different encoder depths, normalization strategies, bottleneck dimensionalities — is not tested.

Mitigation status. The paper does not address this limitation explicitly. The "one line of code" framing (title, abstract, Algorithm 1) emphasizes ease of adoption but elides the retraining requirement. A practitioner wanting to adopt iFSQ for a custom tokenizer architecture should not assume α = 1.6 is optimal without validation, but the paper provides no lightweight method for tuning α (e.g., measuring distribution uniformity on a small validation set without full reconstruction training). The suggested future work on "pretraining or finetuning models to directly predict difficulty" (referenced in the broader paper context) is not about this specific limitation. The α sensitivity analysis in Figure 3 is encouraging — performance degrades gracefully as α deviates from 1.6 — but this is architecture-specific comfort, not a general guarantee.

Hard Problems Remain Unsolved: The Method Does Not Extend to High-Resolution or Complex-Scene Generation

The paper's generative modeling experiments are confined to ImageNet at 256×256 resolution using Large-scale models (DiT-Large, LlamaGen-Large). This is a standard research benchmark, but it represents a narrow slice of the image generation problem space. The claimed findings — that 4 bits per dimension is the discrete-continuous equilibrium, that AR converges faster but diffusion peaks higher, that proportional REPA depth scaling holds — are demonstrated exclusively in this regime.

The consequence. For practitioners working on higher-resolution generation (512×512, 1024×1024, or beyond), more complex scene distributions (text-to-image with diverse compositions, multi-object scenes, open-world generation), or larger model scales (the billion-parameter regimes of production diffusion and AR models), the paper provides no direct evidence that any of its findings extrapolate. Several findings are plausibly resolution-dependent:

• The 4-bit sweet spot. Higher-resolution images contain more information, and the latent bottleneck may need more bits per dimension to adequately capture fine details. The paper observes that "at higher resolutions, the correlation between NTS and CKNNA strengthens" (Section 4.2.1, comparing r = 0.47 at 256 to r = 0.79 at 384 in Figure 6), suggesting that representational requirements change with resolution. Whether the 4-bit optimum shifts — and if so, by how much — is unknown.

• The AR ceiling effect. The paper attributes the AR performance plateau to "the strict sequential constraint" and causal attention's inability to capture global dependencies. At higher resolutions, the token sequence length grows quadratically (256 tokens at 16× compression for 256×256, 1024 tokens for 512×512), and the raster-scan ordering becomes increasingly unnatural — tokens that are spatially distant but semantically related (e.g., two eyes in a face) are separated by many sequence positions, making the causal attention window a more severe bottleneck. The AR-vs-diffusion crossover might occur at very different compute budgets, or the AR "ceiling" might be a hard capacity limit that diffusion models do not face at any scale.

• The REPA scaling law. The 1/3-depth rule is validated on 24–48 layer models. Whether it holds for very deep transformers (96, 128 layers) is unknown — deeper networks might reallocate representational budget differently, or the mode switch region might compress or expand as depth increases.

What evidence exists in the paper. The paper explicitly tests only 256×256 ImageNet for generation. The tokenizer reconstruction experiments include COCO (Figures 3, 5, 10) as an out-of-distribution robustness check, but only for reconstruction — no COCO generation results are reported. The layer-wise analysis includes one higher-resolution experiment (LlamaGen-Large@384 in Figure 6), but this is diagnostic only (layer-wise metrics, no generation FID). The paper does not discuss resolution scaling, does not train generative models at resolutions other than 256, and does not benchmark on datasets beyond ImageNet. This is a standard scope limitation for a research paper, but it is a consequential gap given the strength of the claims (e.g., "the strict sequential constraint is suboptimal for image generation" is stated as a general principle in Section 4.1.4 based on a single resolution and dataset).

Mitigation status. The paper does not acknowledge this as a limitation. The abstract and conclusion state the findings in general terms without qualification about resolution or dataset scope. A practitioner reading the conclusion — "these findings suggest that the strict sequential constraints of autoregression limit ultimate generation quality" (Section 5) — would reasonably assume this applies to image generation broadly, but the evidence supports this claim only at 256×256 on ImageNet with Large-scale models. Future work on higher-resolution or more diverse benchmarks is a natural extension, but the paper does not explicitly flag this or provide guidance on how the findings might change.

The Truncated Training Budgets Undermine the Performance Ceiling Claim

The central benchmarking claim — that diffusion models achieve a "superior performance ceiling" compared to autoregressive models under identical reconstruction constraints (Section 4.1.4, Figure 4) — depends on the observation that the LlamaGen gFID curve flattens while the DiT curve continues to improve. However, the training budgets are truncated and asymmetric: DiT-Large is trained for 100k iterations, LlamaGen-Large for 500k iterations (Appendix D). These budgets produce similar total FLOPs due to different per-step costs (161.04 GFLOPs for DiT vs. 169.65 GFLOPs for LlamaGen, Figure 4 caption), but neither model is trained to convergence.

The consequence. If the AR model has not actually saturated at 500k iterations — that is, if it would continue to improve, albeit slowly, with additional training — then the observed "ceiling" is an artifact of the training budget, not a fundamental property of autoregressive modeling. The FLOPs-matching in Figure 4 is a necessary condition for fair comparison, but it is not sufficient: equal FLOPs can produce different effective convergence states if the two models have different learning dynamics. Autoregressive models, with their teacher-forcing training, might learn rapidly in early training (the steep initial slope in Figure 4) but converge slowly in late training because each token prediction becomes a fine-grained refinement problem rather than a coarse structure-learning problem. If this is the case, giving LlamaGen more FLOPs — say, 2× or 4× the current budget — might close or reverse the gap with DiT.

The paper's interpretation — "this suggests that the strict sequential constraint is suboptimal for image generation" (Section 4.1.4) and "strict sequential ordering may limit the upper bounds of generation quality" (abstract) — treats the flattening as evidence of a fundamental capacity limit. But the evidence equally supports a "slow convergence" interpretation: the AR model might eventually reach the diffusion model's performance, just requiring more compute to get there. These two interpretations have very different implications for practitioners deciding between paradigms. The "capacity limit" interpretation says AR is inherently worse; the "slow convergence" interpretation says AR is less compute-efficient for reaching high quality but may be equally capable given sufficient resources.

What evidence exists in the paper. Figure 4 shows the FID vs. FLOPs curves for both models. The LlamaGen curve does appear to flatten — the slope decreases noticeably in the later portion of training — but the curve is not asymptotically flat. There is no experiment that trains LlamaGen for, say, 1M or 2M iterations to confirm saturation, and no learning rate schedule information that would indicate whether convergence has been reached. The paper reports that the 500k iteration budget is the "original setting" from LlamaGen (Appendix D), but this is the budget used in the original LlamaGen paper, not a demonstration that longer training provides no benefit. For DiT, the 100k iteration budget is described as "ablation parameters from LightingDiT" — explicitly an abbreviated training run, not a convergence budget.

Mitigation status. The paper does not acknowledge this limitation. The conclusion states the ceiling finding as a discovery about the paradigms themselves, not as a finding conditional on a specific training budget. A simple additional experiment — training LlamaGen for 2× longer (1M iterations) and checking whether FID continues to improve — would substantially strengthen or weaken the claim. Without this, the "superior performance ceiling" conclusion is a conjecture based on extrapolation from partially-trained models.

Difficulty Estimation Cost Is Not Discussed (Analogy to Latent Quality Tuning)

While the paper does not have an explicit "difficulty estimation" step in the sense of a compute-optimal policy paper, there is an analogous hidden cost: determining the optimal hyperparameters for iFSQ in a new setting requires expensive sweeps that are not accounted for in the headline efficiency claims.

The consequence. The paper demonstrates that iFSQ at α = 1.6 and 4 bits per dimension is the optimal configuration — but arriving at these optimal values required: (1) a sweep over α ∈ {1.0, 1.2, 1.6, 1.8, 2.0, 2.4} with full tokenizer training for each, (2) a sweep over bit depths (2 through 8 bits), and (3) ablation of latent dimensionality × bit depth combinations. Each tokenizer training run takes 25 epochs on ImageNet, which, while modest compared to generative model training, is not negligible — this is likely hundreds of GPU-hours per data point. The paper presents the optimal configuration as a result, but a practitioner adopting iFSQ for a new architecture, dataset, or resolution cannot assume α = 1.6 and 4 bits are optimal without validation. The cost of re-running these sweeps for a new setting is substantial and is not amortized in any of the paper's efficiency claims.

This is analogous to a meta-optimization cost — the compute spent finding the optimal iFSQ configuration for this specific setup is part of the total cost of the approach but is invisible in the reported reconstruction quality vs. bit rate trade-offs. The paper also sweeps REPA hyperparameters (depth, λ, target representation) for the LlamaGen-REPA experiments (Table 3, Figures 8, 9), requiring multiple full training runs of 500k iterations each — a substantial compute investment that a practitioner would need to replicate or approximate.

What evidence exists in the paper. The sweeps are clearly documented: Figure 2 (α sweep on synthetic data), Figure 3 (α sweep with trained tokenizers), Tables 1 and 2 (bit depth sweep for generation), Figure 8 (REPA depth sweep across 3 model scales), Figure 9 (REPA λ sweep). These are presented as experimental results, not as a cost — the paper does not report the total GPU-hours consumed by hyperparameter sweeps, nor does it discuss whether the optimal values might generalize or whether lighter-weight selection procedures exist. The α = 1.6 optimum is derived from synthetic Gaussian data before any tokenizer training, which partially mitigates the cost (the synthetic sweep in Section 3.2 is computationally trivial), but the validation that this α is optimal for trained tokenizers (Figure 3) and the bit depth sweeps (Tables 1, 2) do require full experiments.

Mitigation status. The paper does not acknowledge this as a limitation or discuss the total compute cost of hyperparameter selection. The fact that α = 1.6 generalizes from Gaussian synthetic data to ImageNet tokenizers and COCO reconstruction (Section 4.1.1) is encouraging — it suggests the α optimum may be relatively robust to dataset and architecture changes — but the paper does not test this systematically, and the 4-bit and REPA depth optima are dataset- and architecture-specific. There is no guidance on how a practitioner could efficiently tune these parameters for a new setting (e.g., by monitoring a proxy metric early in training, or by using transferable scaling laws).

The Claimed Benchmarking Fairness Hinges on Identical Reconstruction Quality, Which Is Approximate Not Exact

The paper's core benchmarking contribution — that iFSQ enables fair comparison of AR and diffusion models by eliminating tokenizer confounds — relies on the premise that iFSQ provides identical reconstruction quality to the continuous AE for the diffusion path and identical discrete token quality to VQ-VAE for the AR path. However, the reconstruction quality equivalence is approximate, not exact: Table 4 (Appendix D) reports different PSNR, SSIM, LPIPS, and rFID values for AE, FSQ, and iFSQ tokenizers, and Table 1 shows that even at the optimal 4-bit configuration, iFSQ does not exactly match AE reconstruction (it achieves better gFID at 4 bits without REPA, but the reconstruction metrics differ).

The consequence. The benchmarking comparison in Figure 4 is not comparing DiT and LlamaGen under "identical reconstruction constraints" in the strict sense — it is comparing them under similar-but-not-identical constraints. The diffusion model operates on dequantized continuous iFSQ latents (Equation 2), which have suffered quantization error relative to the encoder output. The autoregressive model operates on discrete iFSQ token indices (Equation 3), which are a lossless encoding of the quantization result — no additional information is lost in the indexing step. This means the two generative models are not operating on exactly equivalent information: the diffusion model sees slightly degraded continuous values, while the autoregressive model sees losslessly-encoded discrete values that carry the same quantization error but in a different format.

Whether this asymmetry favors one paradigm over the other is unclear. The diffusion model might benefit from the continuous representation's ability to represent values between quantization levels during denoising (the dequantized values are the bin centers, but the denoising process can produce intermediate values that the decoder may reconstruct better). Conversely, the autoregressive model's lossless discrete encoding might provide a cleaner signal — no interpolation between quantization bins. The paper does not analyze this asymmetry or control for it.

What evidence exists in the paper. The reconstruction metrics for the tokenizer used in Figure 4 are implicit in Tables 1, 2, and 4. The specific iFSQ configuration used for benchmarking (4 bits, the configuration achieving gFID 12.76 for DiT and the best gFID for LlamaGen) has its own reconstruction quality (rFID, PSNR, etc. from Table 4) that differs from both the AE baseline and the VQ-VAE baseline. The claim of "the exact same pre-trained tokenizer for both paradigms" (Introduction) is correct — both models use the same encoder/decoder weights — but the latent representations they operate on (continuous after dequantization vs. discrete indices) are mathematically distinct and carry different properties, even though derived from the same quantization grid. This is inherent to the dual-mode operation of iFSQ, but the paper does not discuss whether the remaining asymmetry could bias the AR-vs-diffusion comparison.

Mitigation status. The paper does not address this asymmetry or discuss its potential impact on the benchmarking conclusions. The abstract claims that iFSQ "establishes a fair and controlled benchmark" — fair yes, in the sense that it eliminates the major VAE vs. VQ-VAE confounds (different architectures, different training procedures, different compression ratios), but not perfectly controlled, because the continuous vs. discrete representation format is inherently tied to the generative paradigm. There is no way with iFSQ (or any single tokenizer) to give both models exactly identical latent representations — the AR model must receive discrete indices, and the diffusion model must receive continuous values. This is not a flaw in iFSQ; it is a fundamental constraint that any dual-mode tokenizer would face. But the paper's framing of "identical reconstruction constraints" oversells the degree of control, and the remaining asymmetry is not analyzed.

The Single Architecture and Dataset Limits the Generality of All Parametric Findings

The paper's three headline parametric findings — the α = 1.6 activation optimum, the 4-bit discrete-continuous equilibrium, and the 1/3-depth REPA scaling rule — are all derived from a single encoder-decoder architecture (the latent diffusion architecture, Appendix D), trained on a single dataset (ImageNet 256×256), and validated (for reconstruction only) on a single additional dataset (COCO). The generative benchmarking is performed with exactly two model architectures (DiT and LlamaGen). For REPA depth scaling, three model scales are tested, but all within the same architectural families.

The consequence. It is unknown whether any of the specific numerical findings — α = 1.6, 4 bits, 1/3 depth — would transfer to other visual tokenizer architectures (e.g., different encoder designs, transformer-based encoders, different bottleneck dimensionalities), other image domains (medical imaging, satellite imagery, artistic content), higher resolutions, or other modalities (video, audio, 3D). The paper's narrative suggests universality — the α = 1.6 finding is derived from a fundamental property of the Gaussian-to-uniform transformation, the 4-bit equilibrium is presented as a general characterization of the discrete-continuous spectrum, and the 1/3-depth rule is shown to hold across two different architectures. But each of these is supported by evidence from a narrow experimental slice.

For a practitioner deciding whether to adopt iFSQ for a specific application, the uncertainty about parameter transferability matters concretely:

• If designing a tokenizer for medical CT scans (different image statistics), should they assume α = 1.6 or run their own sweep?
• If building a video tokenizer with temporal compression, is 4 bits per spatio-temporal dimension still the sweet spot?
• If using a non-transformer generative architecture (e.g., state-space models, mLSTM), does the 1/3-depth rule still guide REPA alignment?

The paper provides no framework for answering these questions beyond "run the same sweeps we did."

What evidence exists in the paper. The COCO generalization for tokenizer reconstruction (Figures 3, 5, 10) provides some evidence of robustness to distribution shift, but COCO and ImageNet share similar image statistics (natural photographs, similar object categories). The α = 1.6 finding is grounded in a mathematical property (Gaussian-to-uniform transformation) that should be robust to dataset changes as long as the encoder output remains approximately Gaussian — a property of neural network activations in general, not specific to ImageNet. However, this robustness is theoretical, not empirical. The 4-bit finding is explicitly tied to ImageNet: the paper states "iFSQ approaches AE around 4 bits" without qualifying that this is on ImageNet 256×256. The 1/3-depth rule is tested on two architectures (DiT, LlamaGen), which is stronger evidence of generality than a single-architecture test, but still limited to transformer-based generative models.

Mitigation status. The paper does not frame this as a limitation. The findings are stated as general properties in the abstract, introduction, and conclusion without scope qualification. This is standard practice for research papers — findings are presented as discoveries, not as conditional on experimental context — but it places a burden on practitioners to determine transferability. The paper could have strengthened its claims by (1) validating α = 1.6 on at least one additional architectural variant, (2) testing the 4-bit equilibrium on a substantially different dataset or resolution, and (3) acknowledging the architectural specificity of the benchmarking results. The absence of these checks does not invalidate the findings, but it does mean that the quantitative claims (α = 1.6, 4 bits, 1/3 depth) should be treated as empirically validated for the tested configuration and hypothesized — not established — for others.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper makes a methodological intervention that shifts how the field should think about visual tokenizer design and generative model benchmarking. The significance is not in dramatic performance improvements — the reconstruction gain from the iFSQ activation change is a modest ~0.3–0.5 dB PSNR and ~0.6 gFID points (Tables 1, 2; Figure 3) — but in two conceptual reframings that change what questions the field asks and how it asks them.

Reframing 1: The activation function is a first-class design parameter in scalar quantization. Before this work, FSQ (Mentzer et al., 2023) was understood as a simplification of VQ-VAE — no codebook, no collapse, no straight-through estimator complexity. The activation function (tanh) was treated as an incidental bounding mechanism, not a performance-critical component. The paper demonstrates that this assumption is wrong: the tanh activation in original FSQ creates a hidden distribution mismatch that forces a trade-off between bin utilization and reconstruction precision, and that a single activation function choice (the specific sigmoid slope α = 1.6) can simultaneously optimize both. The conceptual shift is not "iFSQ is better than FSQ" — that is an incremental improvement — but rather "the activation function determines the effective codebook capacity of scalar quantization for a given bit budget."

This reframing has concrete consequences for how future tokenizer designs are evaluated. A researcher building a new discrete tokenizer should not ask "how many levels per channel do I need?" in isolation. They should ask: "given my encoder's output distribution (which the activation function shapes), are my quantization bins being utilized efficiently?" The paper provides the diagnostic tools (KS statistic, RMSE to uniform, the controlled toy experiment framework of Figure 1) to answer this question, and the conceptual vocabulary ("activation collapse," "equal-interval vs. equal-probability trade-off") to discuss it. This makes activation design a subject of principled optimization rather than architectural habit.

Reframing 2: Tokenizer-controlled benchmarking is necessary for comparing generative paradigms. The field has spent years debating "AR vs. diffusion" without adequate experimental controls. Papers claiming "autoregressive model beats diffusion" (LlamaGen, Sun et al., 2024) or "diffusion beats autoregressive" (various DiT-scale comparisons) were comparing systems with fundamentally different tokenizers — VQ-VAE for AR, VAE for diffusion — and therefore could not isolate the generative architecture's contribution. The paper demonstrates that a properly designed dual-mode tokenizer can serve as a neutral experimental platform, holding reconstruction quality and compression ratio constant while varying only the generative architecture. Figure 4's AR-vs-diffusion crossover — AR converges faster, diffusion peaks higher — derives its credibility from this deconfounding. The specific finding (diffusion has a higher ceiling at this scale) is less important than the demonstration that controlled tokenizer-level benchmarking is feasible and informative.

This has implications for how future generative model comparisons should be designed. A paper claiming a new architecture (e.g., masked diffusion on discrete tokens, flow matching, hierarchical AR) should, whenever possible, compare against prior paradigms using a shared tokenizer to eliminate the representation-quality confound. The paper does not mandate iFSQ specifically — any tokenizer capable of serving both discrete and continuous modes with matched reconstruction quality would work — but it establishes the methodological standard that the field should aspire to. Comparisons that do not control for tokenizer quality will (or should) be viewed with increased skepticism.

Reconciling prior contradictions. The paper resolves a tension in the tokenizer literature that was latent rather than explicit. On one hand, FSQ was promoted as solving VQ-VAE's problems (codebook collapse, straight-through estimator). On the other hand, practitioners observed that FSQ sometimes underperformed VQ-VAE in practice despite its theoretical advantages — a puzzle that lacked an explanation. The paper's diagnosis of the tanh-induced distribution mismatch provides the missing mechanism: FSQ's theoretical advantages were undermined by a hidden inefficiency (activation collapse limiting effective codebook size) that iFSQ fixes. This reconciles the tension: FSQ is architecturally superior, but its original instantiation was suboptimal due to an activation function choice that prior work did not analyze.

The paper also partially reconciles the conflicting AR-vs-diffusion claims in the literature. The finding that AR converges faster but diffusion peaks higher (Figure 4) explains why different papers arrived at opposite conclusions: papers trained on limited compute budgets (fewer FLOPs) would observe AR advantages; papers with larger compute budgets would observe diffusion advantages. The truth is not that one paradigm dominates the other, but that they exhibit different scaling behavior — a more nuanced and actionable finding than a simple "X beats Y" claim.

Which research directions become more or less attractive. The paper makes activation design for scalar quantization a newly attractive research problem. The α = 1.6 optimum is derived assuming a standard normal encoder output, but learned encoders may produce non-Gaussian distributions, and different architectures (transformer encoders, varying normalization placements) will have different output statistics. There is now a clear question: for a given encoder architecture, what is the optimal activation function to maximize uniform bin utilization? The Figure 2 methodology (sweep sigmoid slopes, measure KS to uniform) provides a template, but more sophisticated approaches could learn the activation function jointly with the encoder, or use adaptive quantization bins that don't assume uniformity.

The paper makes brute-force hyperparameter tuning for tokenizers less attractive — or at least, it provides analytical tools that reduce the need for it. Before iFSQ, finding a good FSQ configuration required training multiple tokenizers at different bit depths. Now, the distribution-matching framework suggests that α = 1.6 is near-optimal for any encoder producing approximately Gaussian latents, and the 4-bit sweet spot provides a principled starting point that can be refined rather than discovered from scratch.

The paper makes naïve cross-paradigm comparisons (AR vs. diffusion without tokenizer control) less credible. A paper that trains a new AR model with a custom VQ-VAE and compares against DiT with a standard VAE will face the (justified) objection that tokenizer quality differences are unaccounted for. Future comparison papers should either use iFSQ (or an equivalent dual-mode tokenizer) or explicitly acknowledge and quantify the tokenizer confound. This raises the methodological bar for an entire subfield.

Follow-Up Research This Work Enables

Adaptive activation learning: joint optimization of encoder output distribution and activation slope. The paper's α = 1.6 is optimal for a standard normal input distribution, but the encoder produces whatever distribution minimizes the reconstruction loss — it is not constrained to be Gaussian after training. A natural extension is to make α a learnable parameter (or to learn a more flexible activation function, e.g., a small neural network mapping latents to [−1, 1]) and train it jointly with the encoder and decoder. This would allow the system to discover the optimal activation shape for the specific encoder architecture and dataset without requiring a separate synthetic sweep. The research question is: does a learned activation outperform the analytically-derived sigmoid with α = 1.6, and if so, by how much? A strong experiment would compare iFSQ (fixed α = 1.6) against a learned-activation baseline on multiple encoder architectures (convolutional, transformer-based, hybrid) and datasets (ImageNet, COCO, higher-resolution benchmarks) to determine whether the Gaussian-to-uniform transformation is universally near-optimal or merely a good first guess. If the learned activation consistently converges near α = 1.6 across diverse settings, that would validate the paper's theoretical framing. If it diverges substantially, the Gaussian assumption would need revision.

Stress-testing the 4-bit sweet spot across resolutions, datasets, and downstream tasks. The paper identifies 4 bits per dimension as the equilibrium where iFSQ reconstruction quality approaches the continuous AE baseline on ImageNet 256×256. This finding is practically actionable but its generality is untested. A direct follow-up would train iFSQ tokenizers at bit depths from 2 to 8 on: (a) higher resolutions (512×512, 1024×1024) to test whether the sweet spot shifts with spatial information content, (b) substantially different image domains (medical imaging, satellite imagery, line art, text documents) to test dataset dependence, and (c) downstream tasks beyond class-conditional generation (text-to-image, image editing, super-resolution) to test whether the optimal bit depth is task-specific. The concrete prediction to test: "the 4-bit equilibrium is a property of the encoder-decoder bottleneck capacity, not the data distribution, and should hold across resolutions and domains." If the sweet spot shifts systematically (e.g., to 6 bits for 1024×1024, to 3 bits for simpler domains), that would characterize the rate-distortion trade-off more precisely. If it holds, 4 bits becomes a robust design default.

Combining iFSQ with modern VAE enhancements (GAN loss, DINO supervision, high-resolution decoding). The paper's tokenizer training is deliberately minimal — MSE + LPIPS loss, no adversarial training, no discriminative feature supervision. This was a deliberate choice to isolate the iFSQ effect, but recent VAE improvements (e.g., adding GAN loss for perceptual quality, DINO feature matching for semantic alignment, multi-scale discriminators) have demonstrated substantial reconstruction quality gains. A natural extension is to ask: does iFSQ's distribution-matching benefit compound with these other improvements, or does the benefit diminish when reconstruction is already near-saturated? Specifically, train iFSQ with (1) an adversarial loss, (2) DINOv2 feature matching (similar to the REPA target but applied at the tokenizer level), (3) both, and compare reconstruction quality and downstream generation performance against similarly-enhanced AE and FSQ baselines. If iFSQ's relative advantage shrinks as absolute reconstruction quality increases (because the distribution-matching benefit matters most when the quantization bottleneck is tight), that would clarify when iFSQ matters versus when it is overshadowed by other factors. If the advantage persists, iFSQ becomes a standard component in high-performance tokenizer recipes.

Scaling study: does the AR-vs-diffusion crossover shift with model size? Figure 4 demonstrates the crossover at Large scale (DiT-Large vs. LlamaGen-Large) with truncated training budgets. A critical follow-up is to repeat this comparison at multiple model scales (Small, Base, Large, XLarge, XXLarge) with training budgets extended until convergence (or at least until improvements drop below some threshold, e.g., <1% FID improvement per 100k iterations). The key question: does the crossover point (in FLOPs) shift with model scale? Three hypotheses are plausible: (1) the crossover occurs at the same relative point (e.g., at 30% of convergence budget) regardless of scale — suggesting it is a fundamental property of the training dynamics; (2) the crossover shifts earlier for larger models — suggesting diffusion's advantage amplifies with capacity; (3) the crossover disappears at larger scales — suggesting the AR ceiling in Figure 4 was a finite-capacity effect. Only a multi-scale convergence study can distinguish these. This experiment is computationally expensive but directly addresses the paper's most consequential claim about generative paradigm selection.

Negative result target: does the AR "ceiling" persist under alternative token orderings? The paper attributes the AR performance plateau to "strict sequential ordering" and the causal attention mask. If this is correct, then alternative token orderings that reduce the causal constraint — e.g., multi-scale autoregressive generation (generating coarse tokens first, then refining), bidirectional masked modeling followed by autoregressive refinement, or randomized orderings as in XLNet — should mitigate or eliminate the ceiling effect. A direct test: train an AR model with the same iFSQ tokenizer but with a masked modeling objective (predict a random subset of tokens given the unmasked remainder, as in MAE or MaskGIT), which removes the strict left-to-right causal constraint, and compare its scaling curve against the causal LlamaGen baseline in Figure 4. If the masked model's FID curve does not flatten — i.e., it continues to improve past the causal model's saturation point — that would validate the causal-attention-bottleneck hypothesis. If it also flattens, the bottleneck is elsewhere (perhaps in the discrete token representation itself, independent of ordering). This experiment would narrow down the mechanism behind one of the paper's headline findings and guide future AR architecture design.

Extending REPA to other autoregressive image generators and multi-modal models. The paper adapts REPA to LlamaGen and discovers the 1/3-depth proportional scaling rule and the need for larger λ. These findings are demonstrated for one AR architecture (LlamaGen) with DINOv2 as the target. A natural extension is to test whether the 1/3-depth and λ = 2.0 rules generalize to: (a) other AR image generators (e.g., VAR, MAGVIT-v2, MaskGIT adapted for AR training), (b) other target representations (CLIP, DINOv1, supervised ImageNet features, multi-modal alignment targets like SigLIP), and (c) multi-modal AR models that generate both text and images (where the optimal alignment depth and strength might differ between modalities). If the 1/3 rule holds across diverse AR architectures and targets, it becomes a reliable heuristic; if it fails for some targets (e.g., CLIP features might require different alignment depth than DINOv2), that would reveal something about the interaction between alignment target semantics and network depth allocation. The λ = 2.0 finding is specifically attributed to teacher-forcing's strong inductive bias — testing whether the same λ scaling applies for AR models trained with scheduled sampling (which mixes in model predictions during training, weakening teacher-forcing) would test this mechanistic explanation.

Theoretical analysis: formal proof of optimal activation for Gaussian-to-uniform transformation. The paper identifies α = 1.6 empirically through a parameter sweep on synthetic data (Figure 2). A theoretical follow-up would attempt to derive the optimal sigmoid slope analytically from the Gaussian CDF transformation. Specifically, the function that maps a Gaussian random variable to a uniform random variable is the Gaussian CDF itself: if X ~ N(0, 1), then Φ(X) ~ Uniform(0, 1). The sigmoid function σ(αx) approximates the Gaussian CDF Φ(x) with varying accuracy depending on α. A theoretical contribution could: (1) derive the α that minimizes the Kullback-Leibler divergence between 2·σ(α·x) − 1 (transformed to [−1, 1]) and the exact uniformizing transformation 2·Φ(x) − 1, (2) compare this analytically-derived α to the empirical 1.6, and (3) provide bounds on the reconstruction quality loss from using the sigmoid approximation rather than the exact CDF. This would elevate the α = 1.6 finding from an empirical observation to a theoretically grounded result, and could reveal whether even better activation functions exist (e.g., using the actual Gaussian CDF, or a learned approximation to it).

Practical Applications and Downstream Use Cases

Unified vision tokenizers for multi-modal models. The most direct application is in large multi-modal models (e.g., GPT-4V-style vision-language models) that must handle both discrete text tokens and continuous visual features. iFSQ can serve as the vision tokenizer in such systems, producing both discrete visual tokens (for autoregressive text+image generation, following the same next-token prediction framework as the language model) and continuous visual latents (for diffusion-based image generation heads that operate alongside text generation). The key practical benefit is architectural unification: a single encoder processes the image, and a single decoder reconstructs it, with the iFSQ bottleneck providing both representations without codebook management overhead. The paper shows that iFSQ at 4 bits achieves reconstruction quality comparable to continuous AE (Table 1: gFID 10.48 vs. 10.67 with REPA) while operating at 4× the compression ratio (96 vs. 24), meaning the discrete token sequence fed to the language model is both compact and high-fidelity. For a vision-language model processing images at 256×256, this reduces the vision token count from ~256 (with standard 16× compression) to a level compatible with text token sequence lengths, making joint autoregressive modeling of text and images more computationally tractable.

Cost-efficient batch image generation with adaptive bit allocation. For organizations running large-scale image generation (e.g., generating training data, producing asset variations, or running inference pipelines), iFSQ's configurable bit depth enables a quality-vs-cost trade-off per image. The paper's scaling analysis (Figure 5, Figure 10) shows that iFSQ performance scales approximately linearly with log compression ratio, with a clear knee at 4 bits (48× compression in Figure 10). A production system could allocate higher bit depths (6–8 bits) for images requiring fine detail (product photos, architectural renders) and lower bit depths (3–4 bits) for images where coarse structure suffices (thumbnail generation, layout prototyping), all using the same encoder-decoder without retraining. The one-line code change from FSQ to iFSQ requires no additional inference cost, and the 4-bit configuration achieves generation quality comparable to continuous AE at 4× the compression ratio. For a pipeline processing millions of images, the storage and transmission savings from higher compression alone could justify the migration.

Training data generation with improved tokenizer utilization. When using generative models to create synthetic training data for downstream tasks (e.g., classification, detection, segmentation), the quality and diversity of generated images matter. A common failure mode is mode collapse or reduced diversity due to tokenizer bottlenecks — if the VQ-VAE codebook has collapsed (many unused codes), the generated images will lack representational diversity regardless of the generative model's quality. iFSQ's guaranteed 100% bin utilization (through distribution matching) eliminates this failure mode: every quantization level is used with approximately equal probability during inference, maximizing the representational entropy for a given bit budget. The paper does not directly measure generation diversity metrics (e.g., improved recall in precision-recall analyses), but the mechanism — uniform bin utilization maximizing information capacity — is a direct consequence of the distribution-matching design. For practitioners generating synthetic datasets, this means iFSQ-based generators should produce more diverse outputs than FSQ or VQ-VAE-based generators at the same compression ratio, because no representational capacity is wasted on underutilized codes. This application would directly benefit from the 4-bit sweet spot, which balances representational capacity (enough bits for visual fidelity) with compression (keeping the generative model's vocabulary tractable).

Intermediate-layer supervision for autoregressive vision models. The LlamaGen-REPA findings provide immediately actionable guidance for practitioners training autoregressive image models. The 1/3-depth alignment rule (align at approximately one-third of total transformer layers) and the λ = 2.0 recommendation (4× stronger alignment loss than standard DiT-REPA) eliminate the need for expensive per-model hyperparameter sweeps when applying representation alignment. For a new AR vision model with L layers, the practitioner can default to aligning at layer floor(L/3) with λ = 2.0 targeting DINOv2-Base final features, and expect near-optimal convergence acceleration — the paper demonstrated this across three model scales (24, 36, 48 layers) on two architectures (LlamaGen and DiT). The practical benefit is training speed: REPA accelerates semantic learning, and using the correct depth prevents the alignment from either drowning in low-level features (too early) or conflicting with next-token prediction (too late). For research teams with limited compute budgets, this rule of thumb could save weeks of GPU time that would otherwise be spent on alignment depth sweeps.