ELT: Elastic Looped Transformers for Visual Generation

ArXiv: 2604.09168

🎯 Pitch

Looping the same transformer blocks repeatedly in a generative model usually fails—intermediate outputs are incoherent junk until the final iteration. ELT fixes this by forcing early exits to match the final output during training, yielding a single model that generates crisp images at any loop count you choose, matching a 446M-parameter baseline with just 111M parameters.

---

1. Executive Summary

This paper introduces Elastic Looped Transformers (ELT), a parameter-efficient class of visual generative models that uses iterative, weight-shared transformer blocks—looping a small set of unique layers repeatedly within each generation sampling step—to drastically reduce parameter counts while maintaining synthesis quality. Evaluated on class-conditional ImageNet 256×256 and UCF-101 using both masked generative transformers (MaskGIT/MAGVIT) and diffusion transformers (DiT), ELT's core training innovation is Intra-Loop Self Distillation (ILSD)—a dual-path training scheme where the full-loop teacher provides a distillation target for stochastically sampled intermediate-loop students, ensuring that representations at early exits remain meaningful rather than waiting until the final loop to produce coherent outputs (enabling Any-Time inference where practitioners can vary the loop count at test time without retraining). ELT achieves a competitive FID of 2.0 on ImageNet with 4× fewer parameters than the MaskGIT-XL baseline under iso-inference-compute settings—matching the 446M-parameter model's performance with just 111M parameters—and an FVD of 72.8 on UCF-101, surpassing the MAGVIT baseline while using only 76M parameters, establishing that recursive depth via looping can substitute for model width as a precision-for-compute lever so long as a minimum block size provides sufficient architectural expressiveness within each iteration.

2. Context and Motivation

The Core Problem: Generative Models Are Too Large to Deploy Efficiently

The fundamental tension this paper addresses is simple but consequential: high-quality visual generative models require enormous parameter counts, but large models are expensive to deploy—especially on resource-constrained hardware. A standard MaskGIT-XL model for 256×256 ImageNet generation uses 446 million parameters; a comparable DiT model uses over 2 billion. These parameters must be stored, loaded into accelerator memory, and shuttled between different memory hierarchies during every inference step. For real-time or on-device applications—mobile photo editing, video generation on edge devices, interactive creative tools—this memory footprint is prohibitive.

The standard response to this problem—making models smaller by reducing layer count or hidden dimension—directly degrades generation quality. There is an inherent depth-quality tradeoff: deeper transformer stacks produce better images (more layers = more representational capacity = more sequential refinement of features), but each additional unique layer adds parameters that must be stored and transferred. The paper's opening framing captures this: "Traditional techniques to increase compute capacity in deep learning models, such as stacking deeper layers or increasing network width, inevitably lead to a proportionally larger memory footprint" (Section 1). This is the parameter-compute coupling problem: you cannot increase the model's effective depth (how many transformations it applies) without also increasing its parameter count (how many unique weights it stores).

Why This Problem Matters: The Memory Wall

This parameter inefficiency is not merely an academic concern—it directly limits where and how generative models can be deployed. The paper identifies a specific hardware bottleneck: the memory wall (Section 1). Modern accelerators (GPUs, TPUs) can perform matrix multiplications far faster than they can read weights from memory. In a standard deep transformer, each unique layer's parameters must be transferred from high-bandwidth memory (HBM) to on-chip SRAM for computation. As models grow, the time spent moving weights increasingly dominates the time spent computing with them. The paper's throughput measurements in Table 3 quantify the practical consequence: when the base model is small enough to fit entirely on-chip (the Scale B case, 768-dimensional), there is no transfer penalty, so ELT provides no throughput gain. But as models grow to scales L, XL, and H (1024-1280 dimensional), ELT's shared parameter block stays compact enough to remain largely on-chip, avoiding repeated transfers and yielding 2.9× to 3.5× throughput improvements over the baseline at the same inference compute budget.

This has direct implications for deployment economics. Cloud inference costs scale with both FLOPs and memory bandwidth consumption; on-device inference is often memory-bandwidth-limited. A model that performs the same number of operations with fewer unique parameters can run faster, consume less energy, and fit on smaller devices—without sacrificing quality.

Prior Approaches and Where They Fall Short

The paper positions itself against three broad categories of existing work, each with identifiable limitations:

1. Architecture-Specific Efficiency Techniques (Pruning, Separable Convolutions, Width Reduction)

The most direct approach to making generative models smaller is to prune redundant components or replace expensive operations with cheaper ones. MobileDiffusion (Zhao et al., 2024) prunes residual blocks in U-Nets and substitutes standard convolutions with separable ones, producing a ~400M parameter model for on-device generation. EdgeFusion (Castells et al., 2024) combines a lightweight Stable Diffusion variant with step distillation. E-DiT (Wang et al., 2026) introduces adaptive block skipping and MLP width reduction to allow variable compute budgets.

Where they fall short: These methods are architecture-specific and often require manual redesign for each model family. More fundamentally, they reduce the model's effective capacity—pruning removes parameters that might have been useful, width reduction limits representational expressiveness per layer. They trade quality for compactness in a way that cannot be dynamically adjusted: once you prune a model, you cannot recover the lost capacity at inference time by spending more compute. The quality-compute tradeoff is fixed at design time.

2. Nested and Matryoshka-Style Architectures

A different approach, represented by MaGNeTS (Goyal et al., 2025) and the Matryoshka family (Kusupati et al., 2022; Devvrit et al., 2024), trains nested sub-networks within a larger model—essentially, a single training run produces models of different sizes that can be extracted at deployment. MaGNeTS specifically applies this to visual generation, using schedules of different model sizes over the generation process without increasing total parameter count.

Where they fall short: Nested architectures still maintain the fundamental coupling between depth and unique parameters. A MaGNeTS model that extracts a "small" subnetwork at early sampling steps and a "large" subnetwork at later steps is using different unique parameters for each tier—the total parameter count (the largest subnetwork) must be stored in memory. These methods provide model size elasticity (you can choose a different-sized model) but not compute-depth elasticity (you cannot vary the effective depth of a given set of parameters). They also typically require specialized training procedures and architectural constraints.

3. Recurrent/Looped Architectures in Other Domains

The idea of recursively applying shared parameters—looping—has a long history. Universal Transformers (Dehghani et al., 2018) introduced weight-tied transformer layers that iterate until a halting condition is met. Deep Equilibrium Models (Bai et al., 2019) formulate the output as the fixed point of a nonlinear transformation, using black-box solvers rather than explicit unrolling. More recently, looped transformers have shown promise in language domains: Saunshi et al. (2025) demonstrated their power for reasoning tasks, Geiping et al. (2025) showed that scaling test-time compute via recurrent depth enables complex latent reasoning in language models, and Gatmiry et al. (2024) proved that looped transformers can implement multi-step gradient descent for in-context learning.

Where they fall short for visual generation: The paper identifies two critical gaps. First, looping for visual generation is underexplored—"its potential for high-fidelity visual generation remains largely untapped" (Section 1). The computational and representational demands of image and video synthesis (high-dimensional continuous or discrete token spaces, spatial coherence requirements, fine-grained texture) differ substantially from language tasks, and it was unclear whether recursive architectures could meet these demands. Second, and more fundamentally, standard looped transformers produce interpretable outputs only at the final loop. As Figure 2 illustrates, intermediate loop states in a vanilla recurred transformer remain "uninterpretable"—they do not correspond to meaningful, high-quality outputs that could be used for early exit. This is the fixed-depth problem: the model is trained to produce a correct output at exactly $L_{\text{max}}$ loops, and its internal trajectory is essentially a black box that only resolves into a coherent result at the very end.

The DEQ contrast is instructive. Unlike Deep Equilibrium Models, which rely on black-box solvers to find an analytical fixed point and typically run to convergence, ELT explicitly optimizes unrolled intermediate states. The paper draws a clear distinction (Section 2): "Unlike DEQs that rely on black-box solvers for an analytical fixed point, our ELT framework explicitly optimizes unrolled intermediate states via Intra-Loop Self Distillation (ILSD), retaining the flexibility of Any-Time inference without requiring the network to reach a strict analytical fixed point." This means ELT can exit at any loop count with meaningful output—it does not need to converge to a fixed point first.

4. Few-Step and Consistency Models (The Orthogonal Axis)

The paper explicitly positions few-step diffusion models (consistency models, progressive distillation) as solving an orthogonal problem (Section 2, "Relation to Few-Step and Consistency Models"). These methods reduce the number of sampling steps (inter-step acceleration)—going from 250 denoising steps to, say, 4 or 1. ELT reduces compute within each sampling step by varying the loop count $L$ (intra-step acceleration). The paper argues these axes are complementary: "we can combine ELT with few-step methods to achieve further efficiency gains." Moreover, in one-step generative paradigms (consistency models, drifting models), there is only a single sampling step, meaning the loop count $L$ becomes the sole lever for controlling the compute-quality tradeoff at inference time—a point the paper returns to in its conclusion as a key future direction.

Conflicting Demands: Parameter Efficiency vs. Any-Time Flexibility vs. Quality

The paper's framing reveals a specific, underexplored tension in the design space. Existing parameter-efficient methods sacrifice quality or flexibility. Existing looped architectures lack Any-Time capability—they are fixed-depth. Existing elastic inference methods (like E-DiT's block skipping) require architectural modifications that are specific to non-recurrent designs and do not leverage the parameter-sharing benefits of recurrence. No prior method simultaneously provides:

1. Extreme parameter efficiency (4× fewer parameters than baselines at iso-quality)
2. Any-Time inference (variable compute at test time without retraining)
3. High synthesis quality (matching or approaching non-recurrent baselines)
4. Generality across generation paradigms (working for both masked generative models and diffusion models)

This is the specific gap ELT aims to fill.

How ELT Positions Itself

The paper frames ELT not as a single architectural tweak but as a design philosophy that decouples model size from computational depth. The key insight, stated in Section 4, is: "This looping architecture decouples physical model size from computational depth. The parameter count (Θ) is constrained by the number of unique blocks ($N$), while representational capacity and depth ($D$) scale with loop count ($L$)." In a standard transformer, effective depth $D = N \times 1$ (each layer is unique). In ELT, effective depth $D = N \times L$ (the same $N$ layers repeat $L$ times). This means you can increase the model's sequential processing capacity—how many transformations it applies to the latent representation—without adding a single parameter.

But simply looping without modification creates the fixed-depth problem. The paper's central methodological contribution—Intra-Loop Self Distillation (ILSD)—is what transforms a vanilla looped transformer into an elastic one. ILSD is not just a regularization trick; it fundamentally changes what the shared parameters learn. Instead of learning transformations that only produce meaningful output after exactly $L_{\text{max}}$ applications, the shared block learns a progressive refinement process: each additional loop incrementally improves the representation, and any intermediate loop count produces a usable output. Figure 2 captures this visually: in the left panel (vanilla looping), only $X_k^{\text{max}}$ reaches the solution space—the intermediate states $X_k^{\text{int}}$ drift away from useful representations. In the right panel (ELT with ILSD), $X_k^{\text{int}}$ is pulled toward the solution space, meaning the model can be exited early without catastrophic quality loss.

This positioning also connects to a broader theme in the paper: test-time compute as a precision-for-compute lever. Just as recent work in language models (Geiping et al., 2025) shows that recurrent depth enables scaling test-time compute for reasoning, ELT shows that loop count $L$ provides a similar lever for visual generation—you can spend more compute (more loops) to get better quality, or less compute (fewer loops) for faster inference, all from a single set of trained parameters. This contrasts with traditional architectures where the only way to get better quality is to train a bigger model with more unique layers.

The Practical Deployment Motivation

Although the paper does not extensively discuss deployment scenarios in the introduction, the motivation is clear from the experiments and the emphasis on Any-Time inference. The key deployment insight is that different use cases have different compute budgets. A cloud-based image generation API serving professional users might afford 256 loops per sampling step for maximum quality. A mobile app generating thumbnails in real-time might afford only 2-4 loops. A video generation pipeline might need to balance per-frame quality against total generation time. Traditionally, serving these different tiers required training and deploying different models—a small fast model for mobile, a large slow model for cloud. ELT's promise is a single trained model that can serve all tiers by simply adjusting $L$ at inference time, traversing the quality-compute Pareto frontier dynamically.

This is the "Any-Time inference" paradigm the paper inherits from the broader literature on anytime algorithms (Zilberstein, 1996) but applies specifically to visual generation: "allowing to traverse the pareto frontier of quality versus compute at test-time without retraining. This allows to serve diverse deployment tiers: from latency-critical on-device generation (few loops) to high-fidelity cloud rendering (more loops)" (Section 1).

Summary of the Gap

The paper identifies a specific, well-motivated gap: visual generative models need to be simultaneously parameter-efficient, high-quality, and flexible in their compute-quality tradeoff, but existing approaches satisfy at most two of these three desiderata. Standard models deliver quality and flexibility (via different model sizes) but are parameter-inefficient. Pruned/downscaled models deliver efficiency but sacrifice quality or flexibility. Vanilla looped transformers deliver efficiency and quality at a fixed depth but lack flexibility—intermediate loops produce garbage. ELT with ILSD aims to hit all three: parameter efficiency via shared weights, quality via sufficient effective depth and ILSD-guided training, and flexibility via Any-Time inference enabled by ILSD's progressive refinement learning.

3. Technical Approach

This is primarily a methods paper whose core idea is that weight-shared recurrent transformer blocks, combined with a novel intra-loop distillation training scheme, can produce a single visual generative model that achieves both extreme parameter efficiency and Any-Time inference—the ability to dynamically trade compute for quality at test time by varying how many times the shared block repeats.

3.1 Reader Orientation

ELT is a training and architecture framework for visual generative models that replaces a deep stack of unique transformer layers with a shallow block of unique layers applied repeatedly (looped) within each generation sampling step. The system solves the parameter-compute coupling problem: in standard transformers, increasing effective depth—how many sequential transformations the model applies to its latent representation—requires adding more unique layers, each with its own parameters that must be stored and transferred from memory. ELT decouples effective depth from parameter count by reusing the same small set of weights multiple times, and ensures that every intermediate repetition produces meaningful output (not just the final one) through a self-distillation procedure that treats the full-depth forward pass as a teacher for shallower, stochastically sampled student configurations.

3.2 Big-Picture Architecture (Diagram in Words)

The ELT system has four major components that interact during a single training step:

1. Shared Transformer Block ($g_\Theta$): A composite block of $N$ unique transformer layers with parameters $\Theta = \{\theta_1, \theta_2, \ldots, \theta_N\}$. This block is the only learnable component—everything else is a fixed procedure that operates on its outputs. At inference, this block is applied $L$ times to the input, where $L$ is chosen at test time. At training time, the block is applied $L_{\text{max}}$ times (the maximum loop budget).

2. Teacher Path: The full $L_{\text{max}}$-loop forward pass through $g_\Theta$, producing a final representation $F^{(N, L_{\text{max}})}(x)$. This path provides the high-quality distillation target because more loops = more sequential refinement = better representations.

3. Stochastic Student Path: A randomly sampled intermediate loop count $L_{\text{int}} \sim \mathcal{U}(L_{\text{min}}, L_{\text{max}})$, producing an intermediate representation $F^{(N, L_{\text{int}})}(x)$. Because the student computation is a strict prefix of the teacher computation—the intermediate state at loop $L_{\text{int}}$ is exactly what the teacher path computed at that point—there is no additional forward pass required. The student receives supervision both from ground-truth labels and from the teacher's output via distillation loss.

4. Task-Specific Prediction Head: A shared head (e.g., a masked language modeling head for discrete tokens, or a denoising prediction head for continuous latents) that maps any intermediate or final representation to a prediction. This head is applied identically regardless of which loop count produced the representation.

Information flow during training: Input $x$ (masked tokens for MaskGIT, noised latents for diffusion) → the shared block $g_\Theta$ is applied iteratively, with the intermediate state at a randomly chosen loop $L_{\text{int}}$ captured along the way → both the intermediate state (student) and final state after $L_{\text{max}}$ loops (teacher) are fed through the prediction head → the teacher's prediction receives ground-truth supervision → the student's prediction receives both ground-truth supervision (weighted by a curriculum parameter $\lambda$) and distillation supervision (weighted by $1-\lambda$) encouraging it to match the teacher's output → gradients from both paths flow back through the single shared $\Theta$, updating the block's parameters to produce useful representations at all loop depths.

Information flow during inference: Input $x$ → the shared block $g_\Theta$ is applied $L$ times, where $L$ is any integer chosen by the practitioner → the output after $L$ loops is fed through the prediction head → the prediction is used for the current sampling step (token unmasking or denoising).

3.3 Roadmap for the Deep Dive

• First, the looping mechanism itself—the notation, the effective depth formula, and the precise architectural composition of the shared block $g_\Theta$—because everything else builds on this parameter-sharing structure.
• Second, the Intra-Loop Self Distillation (ILSD) objective—the joint loss function, the stochastic student sampling procedure, and the curriculum schedule—because this is the core training innovation that enables Any-Time inference.
• Third, the loss formulations for masked generative models and diffusion models separately, because the exact forms of $L_{\text{GT}}$ and $L_{\text{dist}}$ differ between discrete-token and continuous-latent paradigms, including the specific handling of masked positions, vocabulary distributions, and denoising targets.
• Fourth, the training algorithm as a whole—the dual-path forward pass, the gradient flow, and the justification for why this specific design avoids the overhead of traditional two-model distillation while still providing a rich training signal.
• Finally, the inference algorithm and the design choices that make Any-Time inference possible, including why the shared prediction head is critical and what happens when $L$ exceeds $L_{\text{max}}$.

3.4 Detailed, Sentence-Based Technical Breakdown

---

The Looping Mechanism: Decoupling Depth from Parameters

The fundamental architectural unit in ELT is the shared composite block $g_\Theta$. In a standard transformer, a model with effective depth $D$ has $D$ unique transformer layers, each with its own set of parameters. ELT replaces this with $N$ unique layers that are applied $L$ times, producing the same effective depth $D = N \times L$ but with only $N$ layers' worth of parameters.

Let $f_{\theta_i}(x)$ denote a single transformer layer with parameters $\theta_i$. A standard transformer layer includes multi-head self-attention followed by a feedforward network, with residual connections and layer normalization. The paper does not modify the internal structure of these layers—they are conventional transformer blocks. The composite block groups $N$ such layers sequentially:

$g_\Theta(x) = f_{\theta_N}(f_{\theta_{N-1}}(\cdots f_{\theta_1}(x)))$

where $\Theta = \{\theta_1, \theta_2, \ldots, \theta_N\}$ is the set of all parameters across the $N$ unique layers in the block.

What it computes: a single application of the composite block takes an input representation $x$ and passes it through $N$ sequential transformer layers, each with its own distinct parameters within the block. The output is a transformed representation of the same dimensionality. When applied once ($L=1$), the effective depth is $N$; when applied twice ($L=2$), the effective depth is $2N$, with the second application reusing the exact same $\Theta$.

Why this form: the grouping into a composite block of $N > 1$ layers (rather than looping a single layer) is deliberate. The paper's ablation in Table 2 demonstrates that $N=1$ (a single unique transformer layer looped 32 times, configuration $1N \times 32L$) achieves an FID of 10.30 versus 3.43 for a standard 32-layer DiT—dramatically worse despite the same effective depth. The authors state this "reveals that a single unique transformer layer, despite 32 effective passes, lacks the representational capacity for high-fidelity generation, highlighting that a minimum block size $N$ is necessary to provide sufficient architectural expressiveness within each iteration" (Section 5.2). A single transformer layer has only one attention mechanism and one feedforward network; looping it means the same transformation is applied sequentially with no diversity of operations within a loop. Multiple unique layers within the block provide varied transformations (different attention patterns, different feedforward projections) at each recursion, giving the looped process more expressive power per iteration.

The full effective transformation after $L$ loops is:

$F^{(N, L)}(x) = \underbrace{g_\Theta(g_\Theta(\cdots g_\Theta(x)))}_{L \text{ loops}} \equiv g_\Theta^L(x)$

What it computes: the input $x$ passes through the composite block $L$ times sequentially. After the first pass, the output representation becomes the input to the second pass (same parameters, same block structure), and so on, for a total of $L$ applications. The effective total number of transformer layer applications is $N \times L$.

Why this form: this is standard function composition. The key property is that $F^{(N, L)}$ and $F^{(N, L_{\text{max}})}$ share the same $\Theta$—the only difference is how many times the composition is applied. This means (a) the parameter count is independent of $L$, (b) any intermediate state $F^{(N, L_{\text{int}})}(x)$ for $L_{\text{int}} < L_{\text{max}}$ is a strict prefix of the computation of $F^{(N, L_{\text{max}})}(x)$, and (c) at inference time, $L$ can be any positive integer without loading new parameters. Property (b) is what makes ILSD computationally cheap: the student path does not require a separate forward pass.

The paper uses the notation $N \times L$ throughout to specify configurations. For example, ELT-XL with $7N \times 4L$ means 7 unique layers in the block, looped 4 times per sampling step, for an effective depth of 28 transformer layer applications per sampling step, using only 7 layers' worth of parameters (111M total). The MaskGIT-XL baseline has 28 unique layers with no looping—same effective depth, 4× more parameters (446M).

---

Intra-Loop Self Distillation (ILSD): The Core Training Innovation

ILSD is a training procedure, not a change to the model architecture. It operates on the existing looped transformer and alters only the loss function and the forward-pass bookkeeping. The goal is to ensure that the shared block $g_\Theta$ learns transformations that produce useful representations at every loop count, not just at $L_{\text{max}}$.

The problem ILSD solves. In a standard looped transformer trained only with supervision at $L_{\text{max}}$, the intermediate states $F^{(N, L_{\text{int}})}(x)$ for $L_{\text{int}} < L_{\text{max}}$ receive no direct training signal. The model is free to use those intermediate loops for any computation—it might gradually build up a representation that only becomes coherent at the final loop, or it might route information in ways that make early exits meaningless. Figure 2 (left panel) illustrates this: intermediate states drift away from the solution space. Empirically, this manifests as catastrophic quality degradation when inference uses $L \neq L_{\text{max}}$, as shown in Figure 1 (right panel) and Figures 8a, 8b (the "w/o ILSD" curves).

How ILSD works: dual-path training with shared parameters. During each training step, the model executes $L_{\text{max}}$ loops through $g_\Theta$. At a randomly sampled intermediate loop count $L_{\text{int}}$, the current representation is saved. This gives two representations from a single forward pass:

• $F^{(N, L_{\text{max}})}(x)$ — the teacher, produced by the full $L_{\text{max}}$ loops
• $F^{(N, L_{\text{int}})}(x)$ — the student, produced by only $L_{\text{int}}$ loops

Both representations are fed through the same prediction head to produce predictions. The loss combines three terms:

$\mathcal{L}_{\Theta}^{\text{ILSD}} = \underbrace{\mathcal{L}_{\text{GT}}(F^{(N, L_{\text{max}})}(x), y)}_{\text{(1) Ground-truth for teacher}} + \underbrace{\lambda \mathcal{L}_{\text{GT}}(F^{(N, L_{\text{int}})}(x), y)}_{\text{(2) Ground-truth for student}} + \underbrace{(1 - \lambda) \mathcal{L}_{\text{dist}}(F^{(N, L_{\text{int}})}(x), \text{sg}(F^{(N, L_{\text{max}})}(x)))}_{\text{(3) Intra-Loop Self Distillation}}$

where:

• $x$ is the input (masked tokens or noised latents, depending on the generation paradigm)
• $y$ is the ground-truth target (original unmasked tokens or the noise/clean latent)
• $\lambda \in [0, 1]$ is a curriculum weight controlling the ground-truth vs. distillation balance for the student
• $\text{sg}(\cdot)$ is the stop-gradient operator, preventing gradients from flowing through the teacher when it serves as a distillation target
• $\mathcal{L}_{\text{GT}}$ is the standard task loss (cross-entropy for discrete tokens, weighted MSE for continuous latents)
• $\mathcal{L}_{\text{dist}}$ is the distillation loss measuring discrepancy between student and teacher outputs

What it computes, operationally: For every training example, three scalar losses are computed. Term (1) is the standard loss on the full-depth output—this alone would train a vanilla fixed-depth looped transformer. Term (2) is the standard loss on the intermediate-depth output, weighted by $\lambda$—this directly supervises the student to produce correct outputs even with fewer loops. Term (3) is the distillation loss encouraging the student's output distribution (or predicted latent) to match the teacher's, weighted by $1 - \lambda$—this provides a dense signal about how the teacher refines its predictions beyond what the ground-truth alone provides. All three gradient contributions are summed and backpropagated through the single shared $\Theta$.

Why this form: the three-term decomposition has several deliberate properties:

1. The teacher always receives ground-truth supervision (term 1). This anchors the entire system to the actual task. Without it, the teacher might drift, and the student would be distilling from a deteriorating target. It also ensures that the final output at $L_{\text{max}}$ remains optimal—distillation improves intermediate exits without sacrificing the upper bound.

2. The student receives both ground-truth and distillation signals (terms 2 and 3). Ground-truth supervision provides an unbiased learning signal—it tells the student the correct answer regardless of what the teacher thinks. Distillation provides a richer signal—it tells the student to match the teacher's full output distribution or predicted latent, which encodes information about relative token probabilities or fine-grained latent structure that a one-hot ground-truth label does not capture. The curriculum weight $\lambda$ balances these: early in training when the teacher is unreliable, $\lambda$ is high (mostly ground-truth); later when the teacher produces good outputs, $\lambda$ is low (mostly distillation).

3. Stop-gradient on the teacher in term (3). Without stop-gradient, the teacher would be pulled toward the student (since both share $\Theta$), creating a collapsing effect where the deep and shallow outputs converge to the same (potentially suboptimal) representation. Stop-gradient ensures information flows only from teacher to student—the teacher shows the student what a better representation looks like, but the student's limitations do not drag the teacher down.

4. Single shared $\Theta$ receiving gradients from all paths. This is what forces the shared block to learn transformations that are incrementally useful. A transformation that works well at $L_{\text{int}}$ but fails when composed further (because it destroys information needed by later loops) would produce poor teacher outputs, incurring loss from term (1). A transformation that only produces good outputs at $L_{\text{max}}$ but leaves intermediate states chaotic would produce poor student outputs, incurring loss from terms (2) and (3). The shared parameters must satisfy all constraints simultaneously.

---

Stochastic Student Sampling ($S^3$)

The student loop count $L_{\text{int}}$ is not fixed—it is randomly sampled at each training step:

$L_{\text{int}} \sim \mathcal{U}(L_{\text{min}}, L_{\text{max}})$

where $L_{\text{min}} \leq L_{\text{int}} < L_{\text{max}}$. $L_{\text{min}}$ constrains the sampling distribution, preventing the student from being sampled at trivially shallow depths (e.g., $L_{\text{int}} = 1$ when $L_{\text{max}} = 8$) where the model has insufficient effective depth to produce meaningful output. The paper does not specify exact $L_{\text{min}}$ values used in experiments, but the principle is clear: the student should represent a meaningful reduction in compute (fewer loops) without being impossibly shallow.

What it computes: At each training step, a random integer between $L_{\text{min}}$ and $L_{\text{max}} - 1$ (inclusive) is drawn uniformly. The forward pass proceeds for $L_{\text{max}}$ loops. At loop $L_{\text{int}}$, the current hidden state is saved. After the full $L_{\text{max}}$ loops complete, both the intermediate state (from loop $L_{\text{int}}$) and the final state (from loop $L_{\text{max}}$) are used to compute the ILSD loss.

Why this form: uniform sampling ensures that the model is trained to produce good outputs at every intermediate depth, not just a few hand-picked ones. If the student were always $L_{\text{int}} = L_{\text{max}} / 2$, the model might learn to produce good outputs at depth $L_{\text{max}}/2$ and $L_{\text{max}}$ but fail at $L_{\text{max}}/4$ or $3L_{\text{max}}/4$. Uniform sampling provides coverage across the entire loop spectrum, which is essential for true Any-Time inference where the practitioner might choose any integer $L \in [1, L_{\text{max}}]$ at test time. The exclusion of $L_{\text{int}} = L_{\text{max}}$ from the student sampling is because that case is already covered by the teacher path (term 1)—there is no need to distill from the teacher to a student of identical depth.

The computational cost of this stochastic sampling is minimal. Because the student path ($L_{\text{int}}$ loops) is a strict prefix of the teacher path ($L_{\text{max}}$ loops), the intermediate state is available "for free" during the forward pass. The only overhead is storing the intermediate hidden state at the randomly chosen step and computing the additional loss terms—no extra forward passes through the model. This is a crucial practical advantage over traditional knowledge distillation, which requires separate forward passes through both teacher and student models.

---

Curriculum Schedule for $\lambda$

The weight $\lambda$ controlling the ground-truth vs. distillation balance for the student follows a linear decay schedule:

$\lambda(t) = 1 - \frac{t}{T}$

where $t$ is the current training step and $T$ is the total training steps. The paper states: "we linearly decay it from 1 to 0 as training progresses. This initially anchors the student to reliable ground-truth labels while the teacher is still untrained and gradually shift to mimicking the teacher's predictions once they have matured" (Section 4).

What it computes: At the start of training ($t = 0$), $\lambda = 1$, meaning the student receives only ground-truth supervision (term 2 dominates, term 3 is zero). The teacher is randomly initialized and produces poor outputs, so distillation would be harmful—the student would learn to mimic a bad teacher. At the end of training ($t = T$), $\lambda = 0$, meaning the student receives only distillation supervision (term 3 dominates, term 2 is zero). The teacher now produces high-quality outputs (having been trained on ground-truth via term 1 throughout), so the student benefits from matching the teacher's refined predictions.

Why this form: The linear schedule is simple and effective. The paper reports: "We found this linear schedule to be effective across all our experiments and did not observe sensitivity to the decay rate, as long as the transition is gradual" (Section 4). The insensitivity to the exact decay rate is noteworthy—it suggests the key principle is the monotonic transition from ground-truth to distillation, not the precise rate. A gradual transition prevents abrupt changes in the training signal that could destabilize learning. The choice to end at $\lambda = 0$ rather than some positive floor means the final training phase is pure distillation, where the student learns entirely from the teacher's internal representations. This aligns with the goal of making intermediate states match the full-depth output distribution.

---

Loss Formulation for Masked Generative Models

Masked generative models (MaskGIT, MAGVIT) operate on discrete tokens from a learned vocabulary $\mathcal{V}$. At each sampling step, some tokens are masked, and the model predicts the original tokens for the masked positions. The input $x_{\text{mask}}$ is a sequence of tokens where a subset are replaced with a special [MASK] token. The ground-truth $y = \{y_i\}_{i \in \text{Mask}}$ is the original tokens at the masked positions.

Ground-truth loss (cross-entropy):

$\mathcal{L}_{\text{GT}} = -\sum_{i \in \text{Mask}} \log P^{(N, L)}(y_i \mid x_{\text{mask}})$

where $P^{(N, L)}(y_i \mid x_{\text{mask}})$ is the predicted probability assigned to the correct token $y_i$ at masked position $i$, after $L$ loops through the shared block. The sum is over all masked positions (typically a fraction of the total sequence, determined by the cosine masking schedule).

What it computes: the standard cross-entropy loss for masked token prediction. For each masked position, the model outputs a probability distribution over the vocabulary (size $|\mathcal{V}| = 1024$ for the tokenizers used). The loss is the negative log probability of the correct token, summed across all masked positions, producing a scalar that is low when the model assigns high probability to the correct tokens.

Why this form: cross-entropy is the standard maximum-likelihood objective for categorical distributions. It encourages the model to place all probability mass on the correct token (a one-hot target). This is appropriate for the teacher path (which needs to learn the correct answer) and for the student path early in training when the student needs unbiased ground-truth signal.

Distillation loss (KL divergence between token distributions):

$\mathcal{L}_{\text{dist}} = -\sum_{i \in \text{Mask}} \sum_{v \in \mathcal{V}} P^{(N, L_{\text{max}})}(v \mid x_{\text{mask}}) \log P^{(N, L_{\text{int}})}(v \mid x_{\text{mask}})$

where $P^{(N, L_{\text{max}})}(v \mid x_{\text{mask}})$ is the teacher's predicted probability for vocabulary token $v$ at masked position $i$, and $P^{(N, L_{\text{int}})}(v \mid x_{\text{mask}})$ is the student's predicted probability for the same token.

What it computes: the cross-entropy between the teacher's full output distribution and the student's output distribution, summed over all masked positions. This is equivalent to the KL divergence $D_{\text{KL}}(P_{\text{teacher}} \parallel P_{\text{student}})$ up to an additive constant (the teacher's entropy). For each masked position, the teacher produces a probability vector over 1024 tokens; the student produces another. The loss penalizes the student for assigning low probability to tokens that the teacher considers likely, and for assigning high probability to tokens the teacher considers unlikely.

Why this form: this is the standard distillation loss for classification tasks (Hinton et al., 2015). Unlike the ground-truth loss which only provides signal about the single correct token, the distillation loss provides a dense signal across the entire vocabulary. The teacher might assign, say, 0.6 probability to the correct token, 0.2 to a semantically similar token, 0.1 to a plausible alternative, and near-zero to unrelated tokens. The student learns to match this full distribution, which encodes richer information about token relationships, uncertainty, and the relative plausibility of alternatives. This is especially valuable for visual generation where multiple tokens might produce visually similar outputs. The stop-gradient on the teacher (via $\text{sg}(F^{(N, L_{\text{max}})}(x))$ in the ILSD objective) ensures gradients only flow through the student's predictions—the teacher's distribution is treated as a fixed target.

Why the switch from one-hot to soft targets matters: In early training ($\lambda \approx 1$), the student sees mostly one-hot ground-truth targets, learning the basic task. In late training ($\lambda \approx 0$), the student sees only the teacher's soft distributions, learning to match the nuanced predictions of a more-refined model. This progression mirrors the logic of standard knowledge distillation but is achieved within a single model and single forward pass, without a separate pretrained teacher.

---

Loss Formulation for Diffusion Models

Diffusion models operate on continuous latent vectors rather than discrete tokens. At each denoising step, the model takes a noised latent $x_t$ (produced by adding Gaussian noise to the clean latent $x_0$ according to the noise schedule) and predicts either the added noise, the clean latent, or a velocity (depending on the parameterization). The paper uses $v$-prediction (velocity prediction) parameterization.

Ground-truth loss (weighted MSE):

$\mathcal{L}_{\text{GT}} = w(t) \left\| F^{(N, L)}(x_t) - x_0 \right\|_2^2$

where $x_t$ is the noised latent at diffusion timestep $t$, $x_0$ is the clean target latent, $F^{(N, L)}(x_t)$ is the model's prediction after $L$ loops, and $w(t)$ is a time-dependent weighting term.

What it computes: the squared Euclidean distance between the model's prediction and the clean latent $x_0$, weighted by $w(t)$. The model takes a noised latent as input and directly predicts the clean latent (rather than the noise, which would be $\epsilon$-prediction). The weighting $w(t)$ is a sigmoid function of the noise level, following Hoogeboom et al. (2025). The result is a scalar that is low when the prediction is close to $x_0$ across all spatial positions and latent channels.

Why this form: the weighted MSE is standard for diffusion models with $v$-prediction or $x_0$-prediction. The sigmoid weighting $w(t)$ upweights timesteps where the signal-to-noise ratio is intermediate—where the model's prediction is neither trivially easy (very low noise) nor impossibly hard (pure noise). The paper uses this weighting because it has been shown to improve sample quality (Hoogeboom et al., 2025). Note that unlike the masked generative case, the ground-truth loss here uses a soft target (continuous vectors) rather than hard one-hot targets, which is inherent to diffusion.

Distillation loss (MSE between predictions):

$\mathcal{L}_{\text{dist}} = w(t) \left\| F^{(N, L_{\text{max}})}(x_t) - F^{(N, L_{\text{int}})}(x_t) \right\|_2^2$

where $F^{(N, L_{\text{max}})}(x_t)$ is the teacher's predicted clean latent (after $L_{\text{max}}$ loops) and $F^{(N, L_{\text{int}})}(x_t)$ is the student's prediction (after $L_{\text{int}}$ loops).

What it computes: the squared Euclidean distance between the teacher's prediction and the student's prediction, weighted by the same $w(t)$. This directly encourages the student to predict the same clean latent as the teacher, making the student's output after fewer loops approximate the more-refined output after more loops.

Why this form: in continuous latent spaces, MSE between predictions is the natural distillation loss. It is simpler than the cross-entropy/KL used for discrete tokens because the outputs are vectors in $\mathbb{R}^d$ rather than probability distributions over a vocabulary. The weighting $w(t)$ ensures that distillation focuses on timesteps where the prediction matters most for sample quality. An alternative would be to distill in noise-prediction space ($\epsilon$-prediction), but since the model already predicts $x_0$, directly matching $x_0$ predictions keeps the distillation loss aligned with the ground-truth loss. The stop-gradient on the teacher (via the $\text{sg}(\cdot)$ in the ILSD objective) ensures the teacher's prediction is a fixed target.

Key difference from masked generative distillation: In masked generation, the distillation loss operates on probability distributions, capturing the teacher's uncertainty across tokens. In diffusion, the distillation loss operates directly in latent space, capturing the teacher's refined estimate of the clean signal. Both serve the same purpose—making the student's output match the teacher's—but the mathematical form adapts to the output space (discrete vs. continuous).

---

Training Algorithm: Dual-Path Forward Pass with Shared Gradients

The complete training procedure is formalized in Algorithm 1 in the paper (Appendix, page 24). Here is a detailed walkthrough of a single training step:

Step 1: Sample student depth. An integer $L_{\text{int}}$ is randomly drawn from $\mathcal{U}(L_{\text{min}}, L_{\text{max}})$. This determines which intermediate loop will serve as the student.

Step 2: Execute the forward pass with state tracking. The input $x$ is fed through the shared block $g_\Theta$ iteratively. At each loop $i \in \{0, 1, \ldots, L_{\text{max}}-1\}$, the current hidden state $F_{\text{curr}}$ is updated by applying $g_\Theta$. When the loop index $i$ equals $L_{\text{int}} - 1$ (0-indexed), the current state is saved as the student representation $F_{\text{int}}$. After $L_{\text{max}}$ total applications, the final state $F_{\text{max}}$ is the teacher representation.

The paper's pseudocode uses jax.lax.fori_loop for this iteration, with a jax.lax.cond to conditionally save the intermediate state. This is an implementation detail for JAX compatibility, not a conceptual requirement—any framework that supports iterative application with conditional state saving would work.

Step 3: Compute predictions. Both $F_{\text{max}}$ and $F_{\text{int}}$ are passed through the same prediction head to produce predictions $\hat{y}_{\text{max}}$ and $\hat{y}_{\text{int}}$. The prediction head is a learned mapping (e.g., a linear projection to vocabulary size for masked models, or a linear projection to latent dimensionality for diffusion) that is part of the parameters $\Theta$ and is trained jointly with the transformer block.

Step 4: Compute the three loss terms.

• $\mathcal{L}_{\text{GT}}^{\text{max}} = \mathcal{L}_{\text{GT}}(\hat{y}_{\text{max}}, y)$ — teacher ground-truth loss
• $\mathcal{L}_{\text{GT}}^{\text{int}} = \lambda \cdot \mathcal{L}_{\text{GT}}(\hat{y}_{\text{int}}, y)$ — student ground-truth loss (weighted by curriculum)
• $\mathcal{L}_{\text{distill}} = (1 - \lambda) \cdot \mathcal{L}_{\text{dist}}(F_{\text{int}}, \text{sg}(F_{\text{max}}))$ — student distillation loss (weighted by curriculum)

The total loss is the sum: $\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{GT}}^{\text{max}} + \mathcal{L}_{\text{GT}}^{\text{int}} + \mathcal{L}_{\text{distill}}$.

Step 5: Backpropagate through shared parameters. Gradients of $\mathcal{L}_{\text{total}}$ with respect to $\Theta$ are computed and used to update the parameters via the optimizer (AdamW). Because all three loss terms depend on $g_\Theta$ (the teacher through the full $L_{\text{max}}$ applications, the student through the first $L_{\text{int}}$ applications, and the distillation through both), the gradient update reflects all three objectives simultaneously.

Why this algorithm is efficient: The critical observation is that computing $F_{\text{int}}$ requires no additional forward passes. In traditional knowledge distillation, you would run the student model separately (forward pass through a shallower network), then run the teacher model separately (forward pass through a deeper network), then compute distillation loss between their outputs. In ILSD, the student computation is a strict prefix of the teacher computation: the hidden state after $L_{\text{int}}$ loops is exactly what the teacher's forward pass computed at that point. You simply save it along the way. The only additional cost versus training a vanilla looped transformer is: (a) the memory to store the intermediate hidden state, (b) the computation of the additional loss terms $\mathcal{L}_{\text{GT}}^{\text{int}}$ and $\mathcal{L}_{\text{distill}}$, and (c) the corresponding backward pass computation. These are negligible compared to the cost of the forward and backward passes through $g_\Theta$ itself.

Why the shared prediction head matters: Both $F_{\text{max}}$ and $F_{\text{int}}$ are fed through the same prediction head. If different heads were used for different loop counts, the model could learn to produce outputs in different representational formats at different depths, with the heads translating them to predictions. By sharing the head, the model is forced to produce representations that are directly interpretable by the same readout at any depth—the representation must be "prediction-ready" at every loop. This is a stronger constraint that pushes the shared block to produce incrementally refined versions of the same kind of representation, rather than qualitatively different representations at different depths.

---

Inference Algorithm: Any-Time Compute via Variable Loop Count

At inference time, ELT operates as described in Algorithm 2 (paper, Appendix, page 25). For a single sampling step (one masking/denoising iteration in the outer generation loop):

Step 1: Choose loop budget $L$. The practitioner selects any positive integer $L$. This is the only decision that varies between deployment scenarios. No retraining, no loading different parameters, no architectural changes.

Step 2: Iteratively apply the shared block. Starting from input $x$ (the masked/noised latent for the current sampling step), apply $g_\Theta$ exactly $L$ times, feeding each output as the input to the next application.

Step 3: Produce prediction. After $L$ loops, the final hidden state is fed through the shared prediction head to produce the output for this sampling step (token logits for masking, clean latent prediction for diffusion).

Step 4: Apply sampling step logic. Use the prediction according to the generation paradigm: for MaskGIT, sample and unmask the top-$p$ most confident tokens; for diffusion, apply the denoising update to produce $x_{t-1}$.

Why Any-Time inference works: The ILSD training ensures that $g_\Theta$ has learned to produce a meaningful, incrementally refined representation at every loop count. If training only supervised $L_{\text{max}}$, the model might produce representations that are unrecognizable by the prediction head until the final loop—e.g., the head might expect a certain statistical structure in the hidden state that only emerges after many iterations. ILSD forces the block to produce head-compatible representations at every depth, so choosing $L < L_{\text{max}}$ yields a slightly less refined but still coherent output rather than garbage.

What about $L > L_{\text{max}}$? The paper reports an interesting empirical finding (Section 5.4, Figure 7): on UCF-101 video generation, a model trained with $L_{\text{max}} = 4$ achieves peak FVD of 69.20 at $L = 6$ during inference—two loops beyond its training depth. The authors note this "suggests that ILSD regularizes the iterative process sufficiently for modest extrapolation beyond training depth" and that "this extrapolation behavior warrants further investigation across datasets and scales" (Section 5.4). This extrapolation capability is not guaranteed—it is an emergent property of ILSD's regularization, not a designed feature. The mechanism is presumably that ILSD encourages the shared block to learn an incremental refinement process that generalizes: if $g_\Theta$ has learned to take a representation and make it slightly better, applying it additional times may continue to yield improvements until hitting diminishing returns or instability. However, the paper also notes in the Limitations (Appendix B) that "when loop count $L$ significantly exceeds $L_{\text{max}}$ at inference, quality can deteriorate as the shared block over-iterates beyond its trained convergence regime."

The relationship between $L$, quality, and compute. For a fixed model configuration ($N$ unique layers, specific hidden dimension $d$), increasing $L$ monotonically improves quality (to a point) while linearly increasing compute. This creates the Pareto frontier shown in Figure 4: each faded curve shows how FID improves as $L$ increases for a fixed $(N, d)$ configuration, and the black envelope shows the best achievable FID for any given GFLOPs budget (by choosing the optimal combination of $N$, $d$, and $L$). The key practical insight: for a given deployment compute budget, you select $L$ to hit your quality target, and the same model checkpoint serves all budgets.

---

Design Choice: Why Looping Instead of Model Ensembling or Multi-Scale Architectures?

The paper's core design choice—looping with shared weights rather than alternative approaches to parameter efficiency—rests on several justifications:

1. Memory bandwidth benefits. As discussed in Section 1 and quantified in Table 3, shared parameters that fit on-chip avoid repeated HBM-to-SRAM transfers. This is not true of ensembling (which requires storing multiple models) or multi-scale architectures (which still have many unique parameters, just organized hierarchically).

2. Compute-depth elasticity without architectural complexity. Looping naturally provides a compute-quality tradeoff: more loops = more sequential processing = higher quality. This does not require block-skipping logic (E-DiT), sub-network extraction (MaGNeTS), or search procedures (EvoSearch). The practitioner simply chooses $L$ at inference time.

3. Composability with the refinement nature of generative sampling. Both masked generation and diffusion involve iterative refinement over multiple sampling steps. Each sampling step already embodies the idea "take a partial/noisy input and make it better." Looping within each step extends this philosophy: "take the current representation and make it better, then make it better again, then make it better again." The architecture mirrors the task structure.

4. Training simplicity. ILSD adds minimal overhead to standard training loops—no separate teacher model, no architectural modifications, no reinforcement learning or adversarial training. The student path is a strict prefix of the teacher path, making implementation straightforward.

---

Design Choice: Why Not Deeper Equilibrium Models (DEQs)?

The paper explicitly contrasts ELT with DEQs (Section 2). DEQs define the output as the fixed point of a nonlinear transformation and use black-box root-finding (typically Broyden's method or Anderson acceleration) to compute that fixed point. This has theoretical elegance—the model is defined by its equilibrium, not by a specific number of iterations. However, the paper identifies practical limitations for visual generation:

1. No Any-Time capability: DEQs must run to convergence. You cannot exit a DEQ early and get a meaningful result, because the intermediate states during root-finding are algorithm-dependent and not trained to be useful. ELT explicitly optimizes intermediate states.

2. Black-box solver instability: Root-finding can be numerically unstable or slow to converge, especially for high-dimensional visual representations. ELT uses a fixed, explicit number of loops ($L$), which is deterministic and predictable in terms of compute cost and latency.

3. Training complexity: DEQs require implicit differentiation through the fixed-point equation, which involves solving a linear system at each training step. This is computationally expensive and can be numerically delicate. ELT uses standard backpropagation through the unrolled loop, which is simple and well-supported by deep learning frameworks.

The paper's approach retains the parameter-sharing benefits of recurrence while avoiding the complexity and rigidity of equilibrium formulations.

---

Summary of Key Design Choices and Their Justifications

• Composite block of $N > 1$ layers rather than looping a single layer: provides sufficient per-iteration expressiveness; empirically, $N=1$ fails catastrophically (FID 10.30 vs. 3.43 for $N=16$, Table 2).
• Uniform sampling of student depth $L_{\text{int}}$: provides coverage across the full loop spectrum, enabling true Any-Time inference rather than optimizing for a few specific depths.
• Linear decay of $\lambda$ from 1 to 0: gradually transitions student supervision from ground-truth (when teacher is unreliable) to distillation (when teacher is mature), with insensitivity to exact decay rate making it robust across settings.
• Stop-gradient on teacher in distillation loss: prevents the collapsing effect where shared parameters cause teacher and student to converge to an intermediate representation, preserving the teacher's quality advantage.
• Shared prediction head across all loop counts: forces representations to be head-compatible at every depth, preventing the model from producing qualitatively different representations at different loop counts that would only work with specialized readout layers.
• Separate loss formulations for masked and diffusion paradigms: adapts the general ILSD principle to the specific output space (discrete categorical distributions vs. continuous vectors), using cross-entropy/KL for the former and weighted MSE for the latter.
• No separate teacher model: the student path is a prefix of the teacher path, making ILSD a single-model, single-forward-pass procedure with negligible overhead beyond standard training—a practical advantage over traditional two-model knowledge distillation.

4. Key Insights and Innovations

Innovation 1: ILSD Reframes Looped Transformers from Fixed-Depth Black Boxes to Incremental Refiners

Prior work on looped transformers—from Universal Transformers (Dehghani et al., 2018) through the recent language reasoning results (Saunshi et al., 2025; Geiping et al., 2025)—treated the looping mechanism as a way to increase effective depth for a fixed parameter budget, period. The fundamental assumption was: train at some depth $L_{\text{max}}$, inference at exactly $L_{\text{max}}$, and the intermediate loop states are implementation details that happen to exist but are not meaningful representations. This is the "black box internal trajectory" problem the paper identifies (Section 4): the model learns to eventually produce a good output, but nobody—not the practitioner, not the model itself in any explicit sense—cares about what happens at loop 3 versus loop 7.

ILSD fundamentally reframes this. It says: the internal loop states are not accidental byproducts of depth-scaling; they are incremental refinement steps that can and should be optimized to be independently useful. The innovation is conceptual, not architectural. ILSD does not change the block structure, the attention mechanism, or the prediction head. It changes what problem the training objective solves. Instead of "produce the correct output after $L_{\text{max}}$ applications of $g_\Theta$," the objective becomes "produce correct and head-compatible outputs after every application of $g_\Theta$, with each additional application making the output strictly more refined." This reframes the shared block from a depth-scaling trick into an iterative refinement operator.

The diagnostic evidence for why this reframing matters is Figure 8: without ILSD, a model trained at $L_{\text{max}} = 8$ produces garbage (FID 41.7) at $L = 2$ and degrades (FID 6.8) at $L = 6$; with ILSD, the same model produces FID 6.0 at $L = 2$ and 2.4 at $L = 6$, a smooth quality gradient that makes early exit meaningful. This is not a small training tweak that nudges a curve slightly upward—it transforms a model that is catastrophically depth-brittle into one that is depth-elastic. The collapse in vanilla looping (Figure 1 right panel vs. left panel) is qualitative: images are unrecognizable noise at $L \neq L_{\text{max}}$. ILSD changes the model's behavioral class from fixed-depth to Any-Time.

What makes this distinctive from prior distillation approaches is that the teacher is not a separately trained model, a larger architecture, or even a different set of parameters—it is the exact same model with more compute applied. The paper calls it "self distillation" but it is more precisely compute-scaling distillation: the student is the teacher at an earlier point in its own computation. This collapses the student-teacher hierarchy into a single model and a single forward pass, which is both a practical efficiency gain and a conceptual reframing. Standard distillation says "a larger model teaches a smaller model"; ILSD says "more compute teaches less compute."

This is a fundamental shift in how to think about looped transformer training. It converts looping from a parameter-efficiency technique that happens to work at a fixed depth into a compute-efficiency lever with continuous quality control. Whether this shift is the "main" contribution of the paper or "merely" the mechanism that enables Any-Time inference is beside the point: it is the intellectual move that makes everything else possible.

---

Innovation 2: Decoupling Parameters from Compute Depth Is a Design Philosophy, Not Just an Architecture Trick

Most parameter-efficient vision techniques—pruning (MobileDiffusion), width reduction (E-DiT), separable convolutions—operate within the paradigm of permanently removing capacity. You design a smaller model, you train it, and that smaller model is all you have. If at deployment you find you have extra compute budget (a faster chip, a less latency-sensitive application), you cannot "add back" the pruned layers or the reduced width. The quality ceiling is fixed.

Looping decouples the parameter count (which is fixed) from the effective computational depth (which is variable). But the paper's insight goes deeper than stating this fact. It identifies that this decoupling creates a new axis in the design space that is orthogonal to model width scaling. Figure 5 makes this concrete: for a given FID target, you can reach it either by increasing model width $d$ (more parameters, fixed shallow depth) or by increasing loop count $L$ (same parameters, more sequential compute). The ELT-H configuration ($d = 1280$, 8 unique layers, $L = 4$) achieves roughly the same FID as a baseline model with 4× more parameters. This is not a one-off observation—Figure 4 shows a smooth Pareto frontier across model scales and loop counts, with a fitted power law (FID $\propto \text{GFLOPs}^{-0.95}$) governing the tradeoff.

What makes this an insight rather than an obvious consequence of looping is the non-trivial interaction between $N$ and $L$. The paper shows that $N = 1$ (looping a single layer) fails catastrophically (FID 10.30 for $1N \times 32L$ vs. 3.43 for baseline DiT-32, Table 2) despite identical effective depth. This means looping does not substitute for architectural diversity within a block—there is a minimum $N$ threshold below which additional loops are wasted. Conversely, Figures 4 and 5 show that for fixed $N$, increasing $L$ follows diminishing returns that eventually make increasing $d$ (model width) more effective. The design space has a genuine structure: $d$, $N$, and $L$ are not freely interchangeable but follow a hierarchy where width dominates, then $N$, then $L$.

This matters for the field because it reframes the conversation about efficient generative models. Before this paper, the question was "how small can we make the model while preserving quality?" ELT reframes it as "given a parameter budget and a compute budget (which may be different at different deployment tiers), what is the optimal configuration of width, block depth, and loop count?" This is a richer optimization problem that maps more naturally onto heterogeneous deployment scenarios—cloud vs. edge vs. on-device—than the binary "large model or pruned model" paradigm.

The innovation is framing-level rather than method-level. The specific architecture (looping) is not new, and the specific training trick (ILSD) is described in Section 3. What is new is the systematic investigation of the $(N, L, d)$ design space as a unified efficiency frontier, the empirical characterization of its structure (minimum $N$, diminishing returns on $L$, width as primary driver), and the argument that this frontier enables a deployment philosophy—train once, serve anywhere by adjusting $L$—that fundamentally differs from the dominant "train one model per deployment tier" approach. This is an incremental advance in the sense that it uses known building blocks, but a fundamental reframing in how it positions those blocks as a design space rather than a single architecture.

---

Innovation 3: ILSD Enables Extrapolation Beyond Training Depth—an Emergent Property with Practical Significance

The paper reports a surprising empirical finding that the authors themselves flag as warranting further investigation: models trained with ILSD can perform effectively at loop counts exceeding their training depth $L_{\text{max}}$. On UCF-101 video generation, a model trained with $L_{\text{max}} = 4$ achieves peak FVD of 69.20 at $L = 6$—two loops beyond what it was explicitly trained for—compared to FVD of 72.9 at the trained depth $L = 4$ (Figure 7). This is not a designed capability; it is an emergent consequence of ILSD's training objective.

Why does this happen? The paper offers a hypothesis but does not prove the mechanism: "ILSD regularizes the iterative process sufficiently for modest extrapolation beyond training depth" (Section 5.4). The intuition is that ILSD forces $g_\Theta$ to learn a transformation with a certain contractive or refinement property: applying it to a representation makes it better (closer to a teacher-refined target), and this property generalizes because the block learns the operation of refinement rather than memorizing a specific number of applications. If $g_\Theta$ has genuinely learned "take a representation and improve it slightly," then applying it more times than during training will continue to yield improvements until the representation converges or the block's limitations (finite capacity, accumulated errors) cause degradation.

This is distinctive because it inverts the typical relationship between training and inference in deep learning. Standard practice: the model is trained for a specific compute budget, and exceeding that budget (e.g., generating tokens beyond the training sequence length in language models) leads to degradation. Here, training for $L_{\text{max}}$ loops produces a model that can be safely deployed at higher compute budgets, extracting additional quality from the same parameters. This means the training budget $L_{\text{max}}$ is not a ceiling on inference quality but rather a minimum quality guarantee—you get at least the $L_{\text{max}}$ quality, and often more by spending extra loops.

The practical significance is substantial: it means practitioners can train with a conservative $L_{\text{max}}$ (reducing training cost, which scales with loop count) and still benefit from larger $L$ at inference time. It also connects ELT to the broader "inference-time compute scaling" literature—just as Geiping et al. (2025) showed that recurrent depth enables language models to perform more complex reasoning by spending more test-time compute, ELT shows that visual generative models can improve sample quality by spending more loops at inference, even beyond what was seen during training.

The caveats are important: the paper explicitly limits this claim to "modest extrapolation" and notes in the Limitations (Appendix B) that "when loop count $L$ significantly exceeds $L_{\text{max}}$ at inference, quality can deteriorate." The degree of extrapolation possible likely depends on dataset, model scale, and $L_{\text{max}}$. This is a negative result with positive implications—the existence of a degradation regime beyond some extrapolation threshold is expected (all models have finite capacity), but the existence of an improvement regime before that threshold is novel and not guaranteed by the training procedure. It suggests ILSD is doing something qualitatively different from standard depth-limited training: it is learning a refinement process rather than a fixed-depth computation.

This innovation is incremental in mechanism (it falls out of ILSD without additional design) but fundamental in implication: it means $L$ is a genuine test-time precision-for-compute lever akin to the number of diffusion sampling steps, not just a deployment-time choice among pre-trained operating points. A single ELT checkpoint provides a continuous quality spectrum extending beyond its training configuration.

---

Innovation 4: Throughput Gains from Parameter Sharing Are Quantified as a Memory-Bandwidth Effect, Not Just Parameter Counting

Most parameter-efficient architecture papers report success as "we achieve X quality with Y% fewer parameters." This framing implicitly suggests that parameter count reduction translates directly to speed or memory savings, but it papers over a crucial hardware reality: smaller parameter count helps throughput only when it changes the memory bandwidth bottleneck profile. A model with half the parameters might still be memory-bandwidth-bound if both the original and the smaller model are too large to fit on-chip, or it might bring no throughput gain if the original was already small enough to fit on-chip.

Section 5.2 and Table 3 present what is effectively a hardware-aware efficiency analysis that distinguishes parameter-count reduction from throughput improvement. The result structure is telling: for Scale B (768-dimensional, the smallest model), ELT achieves a throughput ratio of 1.0—no speedup over the baseline despite having fewer parameters—because the baseline already fits entirely within on-chip memory, so parameter reduction does not change the memory-transfer profile. For Scale H (1280-dimensional), the throughput ratio jumps to 3.5× because the compact shared parameters of ELT remain largely on-chip while the baseline requires repeated HBM-to-SRAM transfers for its much larger unique parameter set.

This is a diagnostic contribution rather than a method contribution. It tells the field something about why looped transformers can be faster, not just that they are. The mechanism is specifically the memory wall: "ELT utilizes a compact set of shared parameters and maintain its major weight footprint closer to the accelerator computation unit. This avoids the cost of repeated memory transfers typically required in standard models" (Section 5.2). The throughput gains are not from reduced FLOPs (ELT performs the same number of operations as an iso-inference-compute baseline) but from reduced memory traffic.

The practical significance of this analysis is that it gives practitioners a criterion for when ELT will provide speed benefits: the baseline model must be large enough that its unique parameters cannot all fit in on-chip memory during a forward pass. For very small models already fitting on-chip, ELT provides parameter efficiency (smaller checkpoint, less memory for storing the model) but not latency improvements. For large models, it provides both. This is a more nuanced and actionable deployment guideline than "uses fewer parameters so it must be faster."

This insight is incremental in that the memory-wall phenomenon is well-known in systems research, and the throughput measurements are straightforward benchmarking. But it is distinctive because the paper does the work of connecting architectural properties (shared weights, compact parameter footprint) to hardware consequences (on-chip residency, reduced memory transfers, measured throughput gains) rather than leaving it as an implicit assumption. Table 3, with its explicit throughput ratios broken down by model scale and the explanation of why Scale B shows no gain, anchors the theoretical parameter-efficiency advantage in measured hardware behavior. This makes the efficiency argument falsifiable and hardware-dependent—which is a feature, not a bug, because it tells practitioners when to use ELT versus when standard architectures are already sufficient.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper uses ImageNet 1k at 256×256 resolution (Deng et al., 2009) for class-conditional image generation experiments, with the standard 1.28M training images and 50K validation images, evaluated on the 50K validation set for FID computation. For class-conditional video generation, it uses UCF-101 (Soomro et al., 2012), a dataset of 13,320 videos across 101 action classes, which the authors describe as a data-constrained setting with ~13.7M training tokens—making it a testbed for evaluating whether looped transformers exhibit robustness against overfitting.

• Base model(s). The paper uses two generation paradigms with different base architectures. For masked generative transformers, the base framework is MaskGIT (Chang et al., 2022) for images and MAGVIT (Yu et al., 2023) for videos, both using BERT-style (Devlin et al., 2019) transformer backbones with pretrained VQ tokenizers at codebook size 1024. The authors experiment at five model scales—B (d_model=768), L (1024), XL (1152), H (1280), and G (1664)—with architecture details in Appendix Table 6. For diffusion transformers, the base is DiT (Peebles and Xie, 2023) operating on latents from a pretrained Stable Diffusion v1.4 VAE (Rombach et al., 2022), mapping 256×256 images to a 32×32×4 continuous latent space, with experiments at two scales: Small (16 layers, d_model=2048) and Large (32 layers, d_model=2048). The base models were chosen because they represent established, well-characterized baselines for their respective paradigms, allowing clear isolation of the effect of replacing unique layers with looped shared blocks.

• Metrics. For image generation quality: Fréchet Inception Distance (FID) (Heusel et al., 2017; Dhariwal and Nichol, 2021) computed on 50K generated samples vs. the ImageNet validation set, and Inception Score (IS) (Salimans et al., 2016). For video generation quality: Fréchet Video Distance (FVD) (Unterthiner et al., 2018). For efficiency: inference GFLOPs (total floating-point operations per generated sample, accounting for all sampling steps and loops within each step) and throughput (samples generated per second, measured on a Google Cloud TPU v6e with 1×1 topology and inference batch size of 8). Throughput is reported as the ratio of ELT throughput to baseline throughput at the same GFLOPs budget. The paper also reports # parameters (total trainable parameters, which for ELT equals the unique parameters in the shared block plus the prediction head).

• Baselines. The paper compares against an extensive set of prior work, organized by generation paradigm:

  Masked generative baselines (Table 1): MaskGIT (Chang et al., 2022) at both L (303M params, 24 steps) and XL (446M params, 24 steps) scales—the latter being the direct architectural baseline for ELT; MaskBit (Weber et al., 2024) with classifier-free guidance (305M params, 64 steps); PAR-4× and PAR-16× (Wang et al., 2024) as autoregressive alternatives; VQGAN (Esser et al., 2021) and VQVAE-2 (Razavi et al., 2019) as earlier VQ-based methods. The paper also includes GAN-based and diffusion-based baselines for broader context (BigGAN-deep, StyleGAN-XL, ADM, LDM-4, DiT-XL/2, MDT, MaskDiT, CDM, RIN, Simple Diffusion, VDM++, EDiff, LPDM-ADM, MAR).

  Diffusion baselines (Table 2): Standard DiT at 16 layers (1.1B params) and 32 layers (2.1B params), both trained under identical conditions (same noise schedule, same weighting, same training steps) as the ELT-based diffusion models, enabling apples-to-apples comparison of parameter count vs. effective depth.

  Video generation baselines (Table 4): MAGVIT-L (Yu et al., 2023) at 306M parameters, 12 sampling steps—the direct architectural baseline; PAR-4× and PAR-16× (Wang et al., 2024); MaGNeTS (Goyal et al., 2025); and several prior video generation methods (RaMViD, StyleGAN-V, DIGAN, DVD-GAN, Video Diffusion, TATS, CCVS+StyleGAN, Make-A-Video, CogVideo).

• Generation budget / compute accounting. Compute is measured in two complementary ways, both essential for fair comparison. First, the paper uses the notation $N \times L$ (unique layers × loops) with the constraint that total effective depth D = N × L is matched between ELT and baselines under iso-inference-compute settings—meaning the total number of transformer layer applications per sampling step is identical. For example, in Table 1, ELT-XL with 7N × 4L (effective depth 28, 111M params) is compared to MaskGIT-XL (28 unique layers, effective depth 28, 446M params) at the same 24 sampling steps and same GFLOPs (~3.9K). Second, inference GFLOPs is reported directly (Figure 4, Table 1) to account for any differences in hidden dimension or MLP expansion that would cause models with the same effective depth to have different actual FLOP counts. For diffusion models, the accounting is explicit: a DiT-32 (32 unique layers) and an ELT 16N×2L (16 unique layers × 2 loops = 32 effective depth) have the same effective depth and roughly the same inference compute per sampling step, but the ELT variant uses 2× fewer parameters. For beam search and lookahead variants (not applicable here—the paper does not use search over completions), the budget would include the cost of generating multiple candidates, but since ELT generates deterministically from a single latent trajectory per sampling step, the budget is simply $L$ × (FLOPs per $g_\Theta$ application) × (number of sampling steps).

• Cross-validation / statistical protocol. The paper does not use cross-validation for model selection in the standard sense, but it does report FVD with standard deviations (Table 4: e.g., "72.8 ± 2.5" for ELT $6N \times 4L$ on UCF-101) indicating multiple evaluation runs. For ImageNet FID and IS, no confidence intervals are reported—standard practice in the generative modeling literature, but a limitation for assessing whether the observed differences (e.g., FID 2.0 vs. 2.1) are statistically significant. The Pareto frontier fit in Figure 4 is a simple power law (FID = 1922.5 · G⁻⁰·⁹⁵ + 1.48) fitted to the set of Pareto-optimal configurations; no cross-validation or uncertainty quantification on this fit is reported.

Main Quantitative Results

Image Generation with Masked Generative Transformers (Table 1, Figures 4-5)

Headline result: ELT matches MaskGIT-XL's FID of 2.0 with 4× fewer parameters at iso-inference-compute. Table 1 reports the direct comparison: MaskGIT-XL achieves FID 2.0, IS 294.8 using 446M parameters, 24 sampling steps, and 3.9K GFLOPs. ELT-XL in its $7N \times 4L$ configuration (effective depth 28 matches MaskGIT-XL's 28 unique layers) achieves FID 2.0, IS 266.1 using only 111M parameters—a 75% reduction—at the same 3.9K GFLOPs. This is the paper's central efficiency claim: matched quality at 4× parameter compression.

The result is not cherry-picked across a single configuration. ELT-L ($8N \times 3L$) achieves FID 2.2 with 101M parameters (vs. MaskGIT-L's 303M, also ~3× fewer). ELT-L ($12N \times 2L$) achieves FID 2.1 with 152M parameters. The paper also trains a larger ELT-G model ($8N \times 3L$, 1536-dimensional, trained with $L_{\text{max}} = 4$) that achieves FID 1.9 (reported in Figure 12 caption), surpassing MaskGIT-XL, though no direct iso-parameter baseline is provided at this scale.

The iso-inference-compute constraint is critical to the comparison. The paper explicitly states (Table 1 caption) that "# steps refers to the number of neural network runs" and GFLOPs are matched. ELT configurations are chosen so that $N \times L$ (effective depth) equals the baseline's number of unique layers, and the generation uses the same number of sampling steps (24). This means the only variable is parameter count—the compute budget, effective depth, and number of sampling steps are held constant.

Guided vs. unguided results. The main ELT results in Table 1 use classifier-free guidance (CFG), indicated by the superscript $g$. The MaskGIT baselines at L and XL scales are also reported with CFG (FID 2.1 and 2.0 respectively). The MaskGIT results without guidance (the $\Delta$ row from Chang et al., 2022) show FID 6.2 at 227M parameters with 8 steps—a different operating point that is not directly comparable to the guided ELT results. This is standard practice but means the guided-unguided gap is not isolated for ELT specifically.

Image Generation with Diffusion Transformers (Table 2, Figure 6)

Headline result: ELT-based diffusion outperforms the baseline DiT at iso-inference-compute while using fewer parameters. Table 2 shows:

• DiT-32 (32 unique layers): FID 3.43, 2.1B parameters
• ELT $16N \times 2L$ (effective depth 32, same compute): FID 2.83, 1.1B parameters — 2× fewer parameters, better FID
• ELT $8N \times 4L$ (effective depth 32): FID 3.16, 539M parameters — 4× fewer parameters, better FID than the 32-layer baseline
• ELT $4N \times 8L$ (effective depth 32): FID 3.96, 271M parameters — 8× fewer parameters, slightly worse FID
• ELT $1N \times 32L$ (effective depth 32): FID 10.30, 69M parameters — a catastrophic degradation demonstrating the minimum-$N$ threshold

The non-monotonic relationship between parameter count and quality is notable. Going from 1.1B (16N×2L) to 539M (8N×4L) to 271M (4N×8L) parameters, FID goes 2.83 → 3.16 → 3.96. This is not a simple "fewer parameters = worse" trend—the 16N×2L configuration actually outperforms the 2.1B baseline in FID despite having half the parameters, suggesting that the looped structure with sufficient per-loop expressiveness ($N=16$) can exceed the quality of unique-layer baselines. The paper attributes this to the iterative refinement property enabled by ILSD, though the mechanism is not ablatively isolated (e.g., no comparison to a 16N×2L model trained without ILSD at this scale in Table 2).

The minimum-block-size finding is stark. The $1N \times 32L$ result (FID 10.30) versus the baseline DiT-32 (FID 3.43) and versus ELT $4N \times 8L$ (FID 3.96) demonstrates that a single looped layer—even with 32× the sequential applications—cannot approach the quality of a model with at least 4 unique layers in the block. The effective depth is identical (32 applications), but the architectural diversity within each loop is insufficient. This is a genuinely non-obvious finding: one might expect that 32 applications of the same layer, each with the same attention mechanism and feedforward network, would accumulate enough representational power through the nonlinearity of repeated application. The result shows this is not the case—the model needs different transformations (different attention patterns, different feedforward projections) at each step within a loop to effectively refine the representation.

Pareto Frontier of Quality vs. Compute (Figure 4)

Headline result: ELT establishes a smooth, predictable tradeoff between FID and inference GFLOPs across model scales and loop counts, with a fitted power law FID = 1922.5 · G⁻⁰·⁹⁵ + 1.48.

Figure 4 plots FID against inference GFLOPs (log-scale on GFLOPs axis) for a large sweep of ELT configurations spanning five model scales (B through G) and multiple $(N, L)$ combinations per training run. Each faded curve connects points from the same training run (same $N$, same $L_{\text{max}}$, same model scale $d$) evaluated at different inference loop counts $L$. The black envelope curve is the fitted Pareto front—the locus of configurations that achieve the best FID for a given GFLOPs budget.

Several structural properties emerge:

1. Within a training run, increasing $L$ consistently improves FID (faded curves slope downward-left). For a fixed $(N, d, L_{\text{max}})$ configuration, doing more loops at inference always helps—to a point. The curves eventually flatten and sometimes show slight degradation (visible in the rightmost points of some curves where $L$ exceeds $L_{\text{max}}$).

2. Across model scales, the Pareto front shifts systematically. Smaller models (B, d=768) cannot reach the lowest FIDs regardless of loop count—their curves bottom out around FID 3-4. Larger models (G, d=1664) achieve FID ~2 at high compute budgets. Moving from one model scale to the next (e.g., B→L, L→XL) shifts the achievable FID range downward by roughly 0.5-1.0 FID points.

3. The fitted power law has exponent -0.95, meaning FID improves roughly as GFLOPs⁻¹. The additive constant 1.48 is the asymptotic floor—the lowest FID achievable as GFLOPs → ∞ under this model family. This asymptote is consistent with the best ELT results (FID 1.9 for ELT-G, reported in Figure 12) and approaches the best reported results in the literature (MaskBit: FID 1.7, MAR: FID 1.8).

4. The Pareto frontier is populated by configurations from multiple model scales. At ~2000 GFLOPs, the best configuration is from the L scale ($6N \times 3L$). At ~4000 GFLOPs, XL scale ($7N \times 4L$). At ~7000 GFLOPs, H scale ($8N \times 4L$). This means the optimal strategy is not "use the biggest model you can afford" but rather "match model scale to compute budget"—a configuration with moderate $d$ and high $L$ can outperform a configuration with high $d$ and low $L$ at certain points on the frontier.

The paper does not report confidence intervals for the Pareto fit parameters, nor does it test whether the power-law form is statistically preferred over alternatives (e.g., exponential decay). The fit is descriptive rather than inferential—it summarizes the observed tradeoff rather than making claims about the functional form of the underlying scaling law.

Parameter Scaling (Figure 5)

Headline result: Increasing model width ($d$) is the primary driver of quality, but looping allows parameter-constrained models to match much larger baselines.

Figure 5 plots best-achievable FID against parameter count (log scale on the parameter axis) for four model scales (L, XL, H, G), with each ELT point representing the best $(N \times L)$ inference configuration chosen from a training run at that scale. The baseline points (MaskGIT-B, -L, -XL, -H at their standard configurations) are shown for reference.

The key structural finding is the power-law relationship between FID and parameter count: as parameters increase (moving right on the log-scale x-axis), FID decreases smoothly. The ELT points for a given model width $d$ lie along approximately the same vertical band (since all configurations at that width have similar parameter counts, differing only in $N$), and the best FID within that band improves with width.

The labeled annotations quantify the parameter savings: "67% fewer" for ELT-L $8N \times 3L$ vs. Baseline-L, "75% fewer" for ELT-XL $7N \times 4L$ and ELT-H $8N \times 4L$ vs. their respective baselines. These are the same configurations reported in Table 1.

The figure also shows iso-inference-compute connections: the horizontal dashed lines connect ELT configurations to baseline configurations at approximately the same FID. For example, ELT-XL $7N \times 4L$ (111M params) achieves roughly the same FID as Baseline-XL (446M params) at the same inference compute—this is the 4× parameter reduction claim visualized. Similarly, ELT-H $8N \times 4L$ (a model with 1280-dimensional hidden states, 8 unique layers, looped 4 times) approaches Baseline-H quality with ~75% fewer parameters.

An important detail: the ELT points in Figure 5 are the best inference configuration chosen from each training run. This means for a model trained with $L_{\text{max}}$, the reported FID might come from an inference $L$ that differs from $L_{\text{max}}$—e.g., a model trained with $L_{\text{max}} = 4$ might achieve its best FID at $L = 5$ or $L = 6$ (as observed in the UCF-101 extrapolation result, Figure 7). The figure does not indicate which $L$ was used for each point, which slightly obscures the relationship between training configuration and final quality.

Training Convergence Speedup (Figure 6)

Headline result: ELT-based diffusion models converge 1.4× to 2.0× faster than standard DiT at iso-inference-compute.

Figure 6 plots FID (log scale) against training steps for two comparisons:

• Left panel: DiT-32 (2.1B params, 32 unique layers) vs. ELT $16N \times 2L_{\text{max}}$ (1.1B params, 2× fewer). The ELT curve reaches FID ~3 at approximately 250K steps, while the DiT baseline requires ~500K steps to reach the same FID—a 2.0× speedup.

• Right panel: Same DiT-32 baseline vs. ELT $8N \times 4L_{\text{max}}$ (539M params, 4× fewer). The ELT curve reaches FID ~3.5 at approximately 350K steps vs. ~500K for DiT—a 1.4× speedup.

The speedup is measured as the ratio of training steps to reach equivalent FID, not wall-clock time. Since each training step for ELT involves $L_{\text{max}}$ applications of $g_\Theta$ (same effective depth as the baseline), and the per-step cost is similar (same GFLOPs, but ELT may have faster per-step wall-clock time due to memory bandwidth benefits), the training step speedup should translate approximately to wall-clock speedup, though the paper does not report wall-clock training times.

The mechanism for faster convergence is not explicitly ablated, but the paper attributes it to the parameter-sharing inductive bias: with fewer unique parameters to optimize, the model may find better solutions faster because the parameter space is smaller and the shared block receives richer gradient signals (from multiple loop depths via ILSD). This is a practically significant finding independent of the inference-time benefits—even if you only need a fixed-depth model, training a looped architecture with ILSD may reach good quality faster than training a standard unique-layer architecture.

Throughput Gains (Table 3)

Headline result: ELT achieves 2.9× to 3.5× higher throughput than baselines at model scales L through H on TPU v6e, but no gain at the smallest scale.

Table 3 reports the ratio of ELT throughput to baseline throughput (images/sec) at matched inference GFLOPs on a Google Cloud TPU v6e with 1×1 topology and inference batch size 8:

• Scale B (d=768, $6N \times 2L$): 1.0× — no improvement
• Scale L (d=1024, $8N \times 3L$): 2.9×
• Scale XL (d=1152, $7N \times 4L$): 3.3×
• Scale H (d=1280, $8N \times 4L$): 3.5×

The paper explains the scale-B result explicitly: "the baseline (MaskGIT) is small enough to fit entirely within device memory capacity, eliminating the transfer penalty." At scale B, the baseline's parameters already fit on-chip, so reducing parameter count via looping does not reduce memory transfers—both ELT and the baseline keep all weights in SRAM throughout the forward pass. At scales L and above, the baseline's parameter count exceeds on-chip capacity, requiring repeated HBM-to-SRAM transfers as different layers are loaded. ELT's compact shared block stays on-chip, avoiding these transfers and yielding the measured speedups.

This is a conditional result: the throughput gains depend on the baseline being memory-bandwidth-bound. For very small models or for hardware with very large on-chip memory, the gains would be smaller or absent. For large models or memory-constrained hardware, the gains would be larger. The paper does not test on other hardware (GPUs, different TPU topologies, edge devices), so the reported throughput ratios are specific to the TPU v6e configuration.

The throughput measurement uses the best-performing ELT inference configuration from each model scale (as identified in Figure 5), meaning the $N \times L$ choice has been optimized for quality. It is not a sweep over all possible $(N, L)$ at each scale, so the reported throughput ratios may not be the maximum achievable—they represent the throughput at the quality-optimal configuration.

Video Generation (Table 4, Figure 7)

Headline result: A 76M-parameter ELT model matches the MAGVIT baseline (306M params) with slightly better FVD (72.8 vs. 76) under iso-inference-compute on UCF-101.

Table 4 shows the direct comparison: MAGVIT-L achieves FVD 76 ± 2, IS 89.27 ± 0.15 using 306M parameters and 12 sampling steps. ELT ($6N \times 4L$) achieves FVD 72.8 ± 2.5, IS 88.27 ± 0.33 using only 76M parameters (4× fewer) at the same 12 sampling steps and ~4.3K GFLOPs. This is a stronger result than the ImageNet masked generation case because ELT not only matches but surpasses the baseline in FVD while maintaining comparable IS.

The paper emphasizes the data-constrained nature of UCF-101: "This suggests that looped transformers can exhibit robustness against overfitting in data-constrained regimes like UCF-101, effectively regularizing the learning process while maintaining the expressive capacity for high-quality generation" (Section 5.3). With only ~13.7M training tokens (vs. hundreds of millions for ImageNet), a standard 306M-parameter transformer might overfit; the 76M-parameter ELT, with its strong inductive bias from weight sharing, may generalize better. This is a hypothesis, not an ablated finding—the paper does not compare ELT against a 76M-parameter non-looped baseline to isolate the regularization effect of looping from the effect of simply having fewer parameters.

Scaling loop count at inference further improves FVD. The ELT ($6N \times 6L$) configuration achieves FVD 60.8 ± 2.7 using 24 sampling steps (double the baseline's 12) and ~13K GFLOPs. This demonstrates the test-time compute lever: spending more loops per sampling step and more sampling steps overall improves quality substantially (72.8 → 60.8 FVD). The paper does not report whether the MAGVIT baseline would also improve with more sampling steps, so the 60.8 vs. 76 comparison is not iso-inference-compute.

ILSD Efficacy for Elasticity (Figures 7, 8)

Headline result: ILSD is the essential ingredient for Any-Time inference—without it, looped transformers collapse catastrophically at $L \neq L_{\text{max}}$.

Figure 8a (masked generative models) compares a model trained with $L_{\text{max}} = 10$ with and without ILSD, evaluated at loop counts $L = 1$ through $L = 10$:

• With ILSD: FID improves smoothly from 22.6 at $L=1$ to 6.0 at $L=2$, then gradually to 2.2 at $L=7-8$, with a slight degradation to 2.4 at $L=10$. The curve is monotonic and smooth—every additional loop helps, and there is no cliff.
• Without ILSD: FID is 41.7 at $L=2$ (catastrophic), improves to 10.5 at $L=3$, 6.6 at $L=4$, then degrades to 6.9 at $L=5$, 6.4 at $L=6$, before dropping to 5.2 at $L=7$ and 2.4 at $L=9$. The curve is non-monotonic and erratic—quality yo-yos as $L$ changes, and only approaches the ILSD-trained quality near $L_{\text{max}}$.

The key takeaway: without ILSD, quality is acceptable only in a narrow band around $L_{\text{max}}$ (roughly $L = 7-10$ in this case). With ILSD, quality is good at every $L \geq 2$, enabling true Any-Time inference.

Figure 8b shows the same comparison for diffusion models, trained with $L_{\text{max}} = 8$:

• With ILSD: FID is ~10 at $L=4$, drops smoothly to ~3.5 at $L=6$, ~2.8 at $L=8$, and continues improving slightly at $L=9-10$. Smooth, monotonic.
• Without ILSD: The model trained at $L_{\text{max}}=8$ and evaluated at $L=4$ (the only comparison point shown explicitly) shows drastically worse FID—the paper does not give the exact numbers for all $L$ in the diffusion case, but the qualitative result is clear from Figure 8b's curve separation.

Figure 7 extends this analysis to video generation on UCF-101, trained with $L_{\text{max}} = 4$, evaluated at $L = 1$ through $L = 10$:

• With ILSD: FVD improves from 317.8 at $L=1$ to 124.6 at $L=2$, 72.9 at $L=4$ (training depth), and reaches a minimum of 69.2 at $L=6$ (two loops beyond training). At $L=10$, FVD has risen to 113.3—still better than $L=2$ (124.6) but worse than the peak. The curve is smooth and shows an extrapolation benefit before eventual degradation.
• Without ILSD: FVD is 580.0 at $L=2$, 400.8 at $L=3$, 156.9 at $L=4$ (training depth, where it roughly matches ILSD), then degrades to 77.1 at $L=5$, 91.0 at $L=6$, and collapses to 1958.1 at $L=10$.

The extrapolation behavior is the most striking result: ILSD enables the model to improve beyond its training depth, reaching better FVD at $L=6$ than at $L=4$. Without ILSD, even going one loop beyond training ($L=5$) degrades quality (FVD 77.1 vs. 72.9 at $L=4$). The paper does not explain why the peak shifts to $L > L_{\text{max}}$, but the phenomenon is consistent across datasets (observed in both Figure 7 for video and Figure 8 for images, though the image curves show the optimum at or near $L_{\text{max}}$ rather than beyond it).

Ablation Studies and Robustness Checks

Effect of ILSD vs. no ILSD on Any-Time capability (Figures 7, 8): This is the central ablation. Without ILSD, models are brittle to loop count changes—FID/FVD degrades sharply when inference $L$ differs from training $L_{\text{max}}$. With ILSD, quality is stable across the full loop spectrum. Quantitatively, for masked generation at $L=2$, ILSD achieves FID 6.0 vs. 41.7 without ILSD (Figure 8a); for video at $L=2$, ILSD achieves FVD 124.6 vs. 580.0 without ILSD (Figure 7). The ablation establishes ILSD as necessary (not just helpful) for Any-Time inference.

Effect of block size $N$ on minimum viable expressiveness (Table 2): The $1N \times 32L$ configuration (FID 10.30) vs. $4N \times 8L$ (FID 3.96) at the same effective depth 32 demonstrates that $N=1$ is insufficient. The paper reports this as an ablation of the architectural hyperparameter $N$, showing that there is a threshold (somewhere between 1 and 4 unique layers) below which looping cannot compensate for lack of architectural diversity. The exact threshold is not pinpointed—configurations with $N=2, 3$ are not reported for this comparison—but the qualitative finding is clear: "a minimum block size $N$ is necessary to provide sufficient architectural expressiveness within each iteration" (Section 5.2).

Effect of model width $d$ on attainable quality (Figure 5): By plotting best FID against parameter count for scales L through G, Figure 5 ablates the effect of hidden dimension while controlling for the looping architecture. Wider models consistently achieve better FID, with the ELT-G (d=1664) reaching FID ~1.9 vs. ELT-L (d=1024) at ~2.5. This confirms that width remains a primary quality driver even in looped architectures—looping provides a compute lever but does not replace the need for sufficient per-layer capacity.

Effect of sampling steps and loop count on video generation (Table 4): Moving from 12 to 24 sampling steps (and $L=4$ to $L=6$ loops per step) improves FVD from 72.8 to 60.8, demonstrating that both inter-step and intra-step compute scaling contribute to quality. This is not a controlled ablation (both sampling steps and loop count change simultaneously), but it shows the combined effect.

Effect of model scale on throughput gains (Table 3): The scale-B configuration shows 1.0× throughput ratio, while scales L-H show 2.9-3.5×. This ablates the interaction between model size and memory bandwidth: throughput gains from parameter sharing only materialize when the baseline model exceeds on-chip memory capacity. For models small enough to fit on-chip, ELT provides no throughput benefit despite parameter savings.

Effect of training curriculum ($\lambda$ schedule) on ILSD effectiveness: The paper states that it "did not observe sensitivity to the decay rate, as long as the transition is gradual" (Section 4). However, no quantitative ablation is provided—no comparison of linear vs. cosine decay, no comparison of different decay rates, and no comparison to a fixed $\lambda$ (e.g., $\lambda = 0.5$ throughout training). This limits the strength of the claim that the schedule is robust; the robustness is asserted but not demonstrated experimentally.

Effect of $L_{\text{min}}$ on student sampling distribution: The paper mentions that $L_{\text{min}}$ constrains the student sampling distribution ("preventing the student from being sampled at trivially shallow depths"), but no ablation over $L_{\text{min}}$ values is reported. The choice of $L_{\text{min}}$ likely affects the tradeoff between early-exit quality (lower $L_{\text{min}}$ provides better coverage of very shallow exits) and training stability (very shallow exits may produce noisy gradients that destabilize the shared parameters). This is an unexamined hyperparameter.

Effect of CFG scale on ELT performance: The paper states that CFG scales were swept for both ELT and baseline, but no ablation over CFG values is provided. The reported results use the optimal CFG scale found via sweep. Since CFG is known to significantly affect FID-IS tradeoffs, the sensitivity of ELT to CFG compared to baselines is unknown.

Extrapolation behavior beyond $L_{\text{max}}$ across configurations: Figure 7 shows extrapolation for UCF-101 ($L_{\text{max}}=4$, peak at $L=6$), but the paper does not systematically study extrapolation across different $L_{\text{max}}$ values, model scales, or datasets. The UCF-101 finding is presented as a notable observation rather than a robust phenomenon, and the paper acknowledges this warrants further investigation. The ImageNet results in Figures 8a and 8b do not show clear extrapolation benefits (peak FID at or near $L_{\text{max}}$), suggesting the phenomenon may be dataset-dependent or scale-dependent.

Critical Assessment

The experimental section is comprehensive in scope—spanning two generation paradigms (masked and diffusion), two data modalities (images and video), five model scales, and multiple efficiency metrics (parameter count, GFLOPs, throughput, training convergence speed). However, the breadth comes at some cost to depth in specific analyses, and several of the paper's central claims require careful qualification.

Claim: "ELT achieves a competitive FID of 2.0 on class-conditional ImageNet 256×256... representing a 4× reduction in parameters."

This claim is directly supported by Table 1: ELT-XL $7N \times 4L$ (111M params) achieves FID 2.0, matching MaskGIT-XL (446M params, FID 2.0). The iso-inference-compute condition is satisfied (same effective depth 28, same 24 sampling steps, same ~3.9K GFLOPs). The claim is supported for this specific configuration.

However, several qualifications apply. First, the "4× reduction" is measured against the specific MaskGIT-XL baseline. The paper's own MaskGIT-L baseline (303M params, FID 2.1) is only 3× larger than ELT-XL, and some baselines in Table 1 (StyleGAN-XL: 166M, LPDM-ADM: not reported) use comparable or fewer parameters. The 4× figure is the maximum reduction against the direct architectural baseline, not an average reduction across all baselines. Second, the claim of "matching" quality relies on FID difference of 0.0 (both 2.0), but FID values are typically reported with some variance across runs; the paper does not report confidence intervals, so "2.0 vs. 2.0" might be "2.0 ± 0.05 vs. 2.0 ± 0.05" or might reflect a genuine match. Third, the MaskGIT-XL baseline uses the original MaskGIT training recipe (270 epochs, as stated in Section 5.1: "We train all models for 270 epochs unless otherwise specified"), but the paper acknowledges that "using a superior tokenizer or optimized training & inference configurations can further boost ELT's performance"—which implies the reported baselines may not be the strongest possible. If MaskGIT-XL were retrained with the same optimizations available to ELT, the quality gap might change.

Claim: "ELT achieves FVD of 72.8 on class-conditional UCF-101."

Supported by Table 4, with the qualification that the FVD is reported with standard deviation (72.8 ± 2.5), and the MAGVIT baseline achieves 76 ± 2. The two intervals overlap (72.8 ± 2.5 = [70.3, 75.3]; 76 ± 2 = [74, 78]), so the difference may not be statistically significant under a standard t-test, though it is directionally favorable. The paper does not claim statistical significance, so this is a fair presentation.

Claim: "This enables Any-Time inference capability with dynamic trade-offs between computational cost and generation quality."

Strongly supported by Figures 7, 8, and the qualitative results in Figures 1, 9, 10. The ILSD vs. no-ILSD ablation demonstrates that ELT with ILSD produces smooth, monotonic quality improvements as loop count increases, while standard looping produces catastrophic degradation away from $L_{\text{max}}$. The Any-Time capability is the paper's most robustly demonstrated contribution—it holds across masked generative models (Figure 8a), diffusion models (Figure 8b), and video generation (Figure 7), and the qualitative examples in Figures 1, 9, 10 viscerally show the difference between ILSD-enabled elastic generation and vanilla looping's fixed-depth brittleness.

The primary limitation of this claim is that the paper does not characterize the shape of the quality-vs-loops curve in a way that would guide practical deployment. For a practitioner deciding on $L$, Figures 7-8 show that quality improves with $L$, but they do not identify the point of diminishing returns (where additional loops yield negligible improvement) or the knee of the curve (where most of the benefit is achieved for a fraction of the maximum compute). This is information a deployment engineer would need to set $L$ appropriately; the paper provides the curves but does not analyze them in these terms.

Claim: "ELT significantly shifts the efficiency frontier for visual synthesis."

This claim is broader and harder to assess from the reported experiments. The paper demonstrates that ELT achieves competitive quality at reduced parameter count on two benchmarks (ImageNet 256×256, UCF-101) using two generation frameworks (MaskGIT/MAGVIT, DiT). This is a meaningful demonstration but falls short of "shifting the frontier" in a field-wide sense. The paper does not compare against the strongest published results at the time of writing (e.g., MaskBit achieves FID 1.7 at 305M params; MAR achieves FID 1.8 at 479M params; both are better than ELT's FID 2.0 at 111M params, though at higher parameter counts). ELT shifts the frontier for parameter-constrained generation—it achieves better quality per parameter than prior work—but not necessarily for unconstrained quality. This is a genuine contribution, but the framing as "shifting the efficiency frontier" is most accurate when qualified with "under parameter constraints."

Missing experiments that would strengthen the paper:

1. Comparison to a non-looped model at ELT's parameter count. The baseline comparisons in Tables 1 and 2 use models with 4× more parameters than ELT. What would a standard MaskGIT model with 111M parameters achieve? If such a model achieves FID 3.0, then ELT's FID 2.0 represents a 1.0 FID improvement from looping at fixed parameter count—a strong result. If such a model achieves FID 2.1, then looping provides only a 0.1 FID improvement, and most of the benefit comes from the ILSD training rather than the architecture. The paper does not provide this critical ablation.

2. Comparison of ILSD to alternative intermediate supervision strategies. ILSD uses distillation from the full-depth teacher to the stochastic student. Would simpler strategies work? For example: (a) directly supervising all intermediate loops with ground-truth (no distillation), (b) using a fixed rather than stochastically sampled $L_{\text{int}}$, (c) using a different distillation target (e.g., the average of multiple teacher depths). The paper ablates ILSD vs. no ILSD but not ILSD vs. alternative intermediate supervision schemes. This makes it difficult to attribute the benefit specifically to the self-distillation mechanism rather than to the general principle of supervising intermediate representations.

3. Wall-clock training time measurements. Figure 6 shows faster convergence in terms of training steps, but ELT training steps may be slower or faster than baseline steps depending on implementation. If each ELT step takes 20% longer due to the ILSD bookkeeping, the 2.0× step speedup might translate to only a 1.67× wall-clock speedup—still significant but less dramatic. The paper does not report this.

4. Memory bandwidth analysis on diverse hardware. The TPU v6e throughput measurements in Table 3 are informative but specific to one hardware configuration. Measurements on GPUs (which have different memory hierarchies and bandwidth characteristics) and on edge devices (where the memory wall is even more severe) would substantially strengthen the practicality argument.

5. Difficulty-stratified or class-stratified analysis. The paper evaluates on entire test sets (50K images for ImageNet, all of UCF-101). Does ELT's quality improvement over baselines vary by image category, complexity, or difficulty? This would help practitioners understand when ELT is most beneficial—analogous to the difficulty-bin analysis that is central to inference-time compute scaling papers.

6. Formal ablation of the $\lambda$ curriculum. The paper's claim that performance is insensitive to the decay rate is unsubstantiated. Even a simple comparison of linear decay at two different rates (fast vs. slow) or linear vs. constant $\lambda = 0.5$ would provide evidence for this claim. Without it, the claim is conjecture.

Weaknesses in evaluation methodology:

• No uncertainty quantification for FID/IS on ImageNet. FID is typically computed on 50K samples, which provides a reasonably stable estimate, but the absence of error bars or multiple-run statistics makes it impossible to assess whether FID 2.0 vs. 2.1 is a meaningful difference. The video results do report standard deviations on FVD, making the omission for images conspicuous.

• Single VAE/tokenizer for each paradigm. All masked generative experiments use the MaskGIT tokenizer (codebook size 1024); all diffusion experiments use the SD v1.4 VAE. If ELT's benefits interact with tokenizer quality—as the paper suggests when noting that "using a superior tokenizer can further boost ELT's performance"—then the reported results are conditional on these specific tokenizers.

• The $L_{\text{min}}$ hyperparameter is not specified. The paper states that $L_{\text{min}}$ constrains the student sampling distribution but never reports the values used in experiments. This makes the training procedure not fully reproducible and leaves open the question of how sensitive results are to this choice.

• No comparison to other elastic inference methods on the same benchmarks. E-DiT (Wang et al., 2026) also provides Any-Time inference for diffusion models via block skipping and width reduction. A direct comparison—E-DiT vs. ELT at the same parameter count and same compute budget—would contextualize ELT's advantages (parameter sharing vs. block skipping). Without such a comparison, the claim that ELT is a superior approach to elastic inference is asserted rather than demonstrated.

Conditional validity of claims:

The paper's efficiency claims hold when the baseline model is large enough to benefit from parameter sharing. For small models (scale B, d=768), ELT provides parameter savings but no throughput improvement (Table 3). For larger models, both parameter savings and throughput gains materialize. This conditionality is clearly documented.

The Any-Time inference claim holds for loop counts within and modestly beyond the training range $[1, L_{\text{max}} + \text{small margin}]$. The paper demonstrates this for up to $L = L_{\text{max}} + 2$ in the UCF-101 case (Figure 7). For $L \gg L_{\text{max}}$, quality degrades (the UCF-101 curve rises at $L=8, 10$). The claim is qualified by the paper's own Limitations section: "when loop count $L$ significantly exceeds $L_{\text{max}}$ at inference, quality can deteriorate."

The faster training convergence claim holds in terms of training steps (Figure 6) but is not validated for wall-clock time. This is a meaningful but incomplete efficiency claim.

The parameter-efficiency claim holds at iso-inference-compute against the specific MaskGIT and DiT baselines. Against stronger baselines (MaskBit, MAR), the quality gap closes or reverses at higher parameter counts, though ELT maintains a parameter-efficiency advantage. The claim is accurate but scope-limited.

6. Limitations and Trade-offs

The Cost of Difficulty Estimation Is Not Amortized in the Headline Efficiency Numbers

The assumption or constraint. ELT's Any-Time inference capability means the practitioner can choose $L$ at deployment based on available compute. However, the paper provides no mechanism for automatically selecting the optimal $L$ for a given input or compute budget—the choice is left to the practitioner, who must either guess or calibrate offline. More concretely, the paper does not account for the cost of determining which $L$ to use. In the experiments, the best $L$ for each configuration is found by exhaustive evaluation (sweeping all $L$ values and picking the one that minimizes FID on the validation set). In deployment, a practitioner would need a similar procedure to map their compute budget and quality target to the appropriate $L$.

The consequence. The headline efficiency gains—4× parameter reduction at matched quality, smooth quality-vs-compute curves—assume the practitioner already knows the right $L$ for their scenario. The cost of determining this mapping (sweeping $L$ on a representative sample of inputs, measuring quality, and selecting the operating point) is not included in any of the reported efficiency metrics. For a cloud deployment serving millions of heterogeneous requests, this calibration may be a one-time cost that amortizes well. For a practitioner deploying a single model on a new hardware platform, the calibration overhead could be significant. Moreover, the optimal $L$ likely varies per input—some images may require fewer loops to reach acceptable quality than others—but ELT applies a uniform $L$ across all inputs. An adaptive mechanism that varies $L$ per input based on some difficulty signal could extract further efficiency, but no such mechanism exists.

What evidence exists in the paper. The paper demonstrates the existence of a quality-vs-loops tradeoff (Figures 7, 8) but does not provide guidance on operating point selection. The Pareto frontier in Figure 4 identifies the best $(N, d, L)$ combination for each GFLOPs budget, but this frontier is constructed post-hoc from exhaustive sweeps—it is a descriptive result, not a decision procedure. The paper acknowledges this implicitly by not providing an input-dependent loop selection mechanism.

Mitigation status. Not addressed. The paper frames Any-Time inference as a feature (the practitioner can choose $L$) but does not address the companion problem of how to choose $L$ optimally. This is analogous to the difficulty estimation problem in inference-time compute scaling: the theoretical gains exist, but realizing them requires a practical difficulty estimator that the paper does not provide.

---

No Comparison to Models at ELT's Own Parameter Count—the Baseline Is Always Much Larger

The assumption or constraint. All parameter-efficiency claims in the paper are measured against baselines with substantially larger parameter counts. MaskGIT-XL (446M params) vs. ELT-XL $7N \times 4L$ (111M params) shows matched FID at 4× parameter reduction. DiT-32 (2.1B params) vs. ELT $8N \times 4L$ (539M params) shows better FID (3.16 vs. 3.43) at 4× parameter reduction. MAGVIT-L (306M params) vs. ELT $6N \times 4L$ (76M params) shows better FVD (72.8 vs. 76) at 4× parameter reduction. These comparisons demonstrate that looping with ILSD can substitute for unique layers, which is the paper's central architectural claim.

The consequence. Missing from the evaluation is a comparison to a non-looped model at the same parameter count as ELT. If a standard 111M-parameter MaskGIT model (with, say, 7 unique layers and a narrower hidden dimension, or some other configuration matching ELT's parameter budget) achieves FID 2.5, then ELT's FID 2.0 represents a 0.5 FID improvement attributable to looping + ILSD—a solid but moderate gain. If such a model achieves FID 5.0, then the gain is dramatic. The paper provides no evidence one way or the other. This matters because it conflates two distinct effects: (a) the benefit of looping shared parameters vs. unique parameters (the architectural contribution), and (b) the benefit of ILSD training vs. standard training (the training contribution). Without the iso-parameter, non-looped baseline, we cannot attribute the quality improvements to the architecture, the training method, or their interaction.

What evidence exists in the paper. The closest the paper comes to this ablation is the $1N \times 32L$ result in Table 2 (FID 10.30 at 69M params), which demonstrates that extreme parameter sharing (single layer) fails. But this is an ablation of the minimum $N$ threshold within the looped architecture class, not a comparison to non-looped architectures at matched parameter count. The scaling curves in Figure 5 show ELT configurations at various parameter counts, but there are no non-looped baselines at those same parameter counts—the baseline points are at much higher parameter counts (MaskGIT-B at 227M, MaskGIT-L at 303M, etc.).

Mitigation status. Not addressed. The paper consistently uses "iso-inference-compute" comparisons (matching effective depth and GFLOPs) but never "iso-parameter-count" comparisons (matching total trainable parameters between looped and non-looped architectures). This is a significant gap because it leaves open the possibility that a well-tuned non-looped model with ELT's parameter budget could achieve similar or even better quality. The absence of this baseline weakens the claim that looping specifically (rather than ILSD or other training choices) drives the efficiency gains.

---

Extrapolation Beyond Training Depth Is Empirically Observed but Not Characterized or Guaranteed

The assumption or constraint. The paper reports that with ILSD, models can achieve better quality at inference loop counts exceeding $L_{\text{max}}$ than at $L_{\text{max}}$ itself. The UCF-101 result in Figure 7 is the clearest example: trained with $L_{\text{max}} = 4$, peak FVD of 69.2 occurs at $L = 6$. This is presented as an emergent property of ILSD's regularization—the shared block learns a refinement operation that generalizes beyond its training horizon.

The consequence. This extrapolation behavior is not guaranteed, not characterized, and not controllable. The paper acknowledges this explicitly: "we note that this extrapolation behavior warrants further investigation across datasets and scales" (Section 5.4). Practitioners cannot rely on it: for a given model, dataset, and $L_{\text{max}}$, there is no way to predict whether extrapolation will help, how far it will extend before degrading, or whether it will occur at all. The ImageNet results (Figure 8a, 8b) do not show clear extrapolation benefits—the FID curves bottom out near $L_{\text{max}}$ rather than beyond it—suggesting the phenomenon may be dataset-dependent or scale-dependent. Moreover, the Limitations section (Appendix B) warns that "when loop count $L$ significantly exceeds $L_{\text{max}}$ at inference, quality can deteriorate as the shared block over-iterates beyond its trained convergence regime." The paper provides no criterion for distinguishing "modest extrapolation" (which may help) from "significant extrapolation" (which hurts).

What evidence exists in the paper. One dataset (UCF-101), one $L_{\text{max}}$ value (4), one model configuration ($6N$). The extrapolation from $L=4$ to $L=6$ yields a 3.7 FVD improvement (72.9 → 69.2). At $L=8$, FVD rises to 78.8 (worse than $L=4$); at $L=10$, it rises further to 113.3. So the useful extrapolation window is narrow (2 loops) and the degradation beyond that is substantial. For ImageNet masked generation (Figure 8a, $L_{\text{max}}=10$), FID at $L=10$ is 2.4 vs. 2.2 at $L=7-8$—no extrapolation benefit, and slight degradation at $L > L_{\text{max}}$. For ImageNet diffusion (Figure 8b, $L_{\text{max}}=8$), FID continues improving slightly at $L=9-10$ but the effect is small.

Mitigation status. The paper flags this as an area for future investigation (Section 5.4, Appendix B) but provides no mitigation. The practical advice implied by the results is: do not count on extrapolation. If deployment may require $L > L_{\text{max}}$, train with a larger $L_{\text{max}}$. The paper does not provide a method for predicting the extrapolation window or for extending it.

---

Throughput Gains Are Hardware-Dependent and Conditional on the Baseline Exceeding On-Chip Memory

The assumption or constraint. The paper's throughput measurements (Table 3) demonstrate 2.9×–3.5× speedups over baselines at model scales L through H on a Google Cloud TPU v6e. The mechanism is reduced memory transfers: ELT's compact shared parameter block fits on-chip, avoiding repeated HBM-to-SRAM transfers that dominate the baseline's runtime. The paper is admirably transparent about the conditionality: at model scale B (d=768), the throughput ratio is 1.0× because "the baseline (MaskGIT) is small enough to fit entirely within device memory capacity, eliminating the transfer penalty" (Section 5.2).

The consequence. The reported throughput gains are not universal properties of ELT—they are properties of ELT relative to a specific baseline on specific hardware. On hardware with larger on-chip memory (e.g., future accelerators with more SRAM, or edge devices with different memory hierarchies), the crossover point where ELT becomes faster would shift upward (only larger baselines would benefit). On hardware with smaller on-chip memory (e.g., older GPUs, mobile processors), even the Scale B baseline might not fit on-chip, and ELT would show throughput gains at all scales. On hardware with fundamentally different memory architectures (e.g., processors with unified memory where there is no HBM-to-SRAM distinction), the throughput advantage might disappear entirely or be replaced by a different bottleneck.

The paper reports results on a single hardware configuration (TPU v6e, 1×1 topology, batch size 8). Practitioners deploying on different hardware—NVIDIA GPUs with different memory bandwidths, Apple Silicon with unified memory, mobile NPUs, edge TPUs—cannot directly apply these throughput ratios. The 3.5× speedup is an existence proof (looping can provide throughput gains under the right conditions) but not a portable performance guarantee.

What evidence exists in the paper. Table 3, with its explicit documentation of the 1.0× ratio at Scale B and the explanation for why, is the relevant evidence. The paper's discussion of the memory wall in Section 1 and the throughput analysis in Section 5.2 provides the mechanistic explanation. The limitation is not that the paper hides the conditionality—it is commendably upfront—but that it does not characterize the conditionality boundary (at what parameter count or memory footprint does ELT begin to show gains on this hardware? What about other hardware?).

Mitigation status. Partially addressed through transparency. The paper's clear documentation of the mechanism (memory wall, on-chip residency) gives practitioners the conceptual tools to reason about whether ELT will help on their hardware. However, the paper provides no characterization across hardware platforms, no methodology for estimating throughput gains on new hardware, and no measurements on GPUs or edge devices. The throughput ratios in Table 3 should be interpreted as "up to 3.5× on the tested hardware" rather than "3.5× in general."

---

The Effect of ILSD Is Not Isolated from Other Training Choices—Multiple Factors Co-Vary

The assumption or constraint. The central ablation in the paper compares "ELT with ILSD" vs. "vanilla looped transformers without ILSD" (Figures 7, 8). The vanilla looped baseline is trained with supervision only at $L_{\text{max}}$—the standard approach for looped transformers. The ILSD-trained model differs from the vanilla baseline in several simultaneous ways: (a) it receives supervision at intermediate depths (stochastic student sampling), (b) that intermediate supervision combines ground-truth and distillation losses, (c) the distillation target is the full-depth teacher with stop-gradient, and (d) the ground-truth/distillation balance follows a curriculum schedule ($\lambda$ decaying from 1 to 0).

The consequence. When ILSD outperforms vanilla training (which it does, dramatically), we cannot attribute the improvement to any specific component of ILSD. Would directly supervising all intermediate loops with ground-truth alone (without distillation) work almost as well? Would a fixed sampling scheme (e.g., always supervising $L_{\text{int}} = L_{\text{max}}/2$) work comparably to stochastic sampling? Would a different curriculum (e.g., cosine decay, constant $\lambda = 0.5$) change the results? Would a simple regularization scheme—such as adding an auxiliary loss that encourages intermediate representations to remain close to the final representation in cosine similarity—achieve similar elasticity without the full distillation apparatus?

The paper does not answer any of these questions. The ablation is binary (ILSD vs. no ILSD), which establishes that something about the approach works but does not identify what matters. From a scientific perspective, this limits our understanding of the mechanism. From a practical perspective, it means a practitioner implementing ELT does not know which components of ILSD are essential and which are incidental—they must implement the full procedure, which includes stochastic sampling, dual loss terms, stop-gradient, and a curriculum schedule.

What evidence exists in the paper. Figures 7 and 8 show the ILSD vs. no-ILSD comparison. The paper states that it "did not observe sensitivity to the decay rate, as long as the transition is gradual" (Section 4), but provides no data to support this claim—no ablation over decay rates or curriculum types. The $L_{\text{min}}$ hyperparameter is mentioned but never specified or ablated. The choice of uniform sampling for $L_{\text{int}}$ is justified conceptually but not compared to alternatives. The stop-gradient on the teacher is motivated conceptually (preventing collapse) but not ablated (e.g., what happens without stop-gradient?).

Mitigation status. Not addressed. The paper treats ILSD as a monolithic training method and evaluates it as such. This is reasonable for a paper introducing a new method, but it limits the strength of the mechanistic claims. A more thorough ablation—even a single additional experiment comparing ILSD to "ground-truth supervision at all intermediate loops with no distillation"—would substantially strengthen the argument that the distillation component specifically (rather than intermediate supervision generally) is necessary.

---

Single Domain (Class-Conditional Generation), Single Modality Family (Visual), No Open-Ended or Text-Conditional Tasks

The assumption or constraint. All experiments are on class-conditional generation: ImageNet 256×256 (1000 classes) and UCF-101 (101 action classes). The generation is conditioned on a discrete class label, not on text prompts, input images, or other complex conditioning signals. The base architectures are MaskGIT/MAGVIT (masked generative transformers) and DiT (diffusion transformers)—both operating on compressed latent representations from pretrained autoencoders (VQ-VAE tokenizers or Stable Diffusion VAE).

The consequence. It is unknown whether ELT's benefits transfer to the most practically important generative modeling scenarios: text-to-image generation (e.g., Stable Diffusion-style models), text-to-video generation, image-to-image translation, inpainting, super-resolution, or open-ended image generation without conditioning. Text conditioning introduces cross-attention layers that interact with the transformer's self-attention blocks; looping these cross-attention layers (which take external text embeddings as input) may behave differently from looping self-attention blocks alone. The paper's class-conditional setup uses a simple class embedding added to the input; this is a much weaker conditioning signal that does not test the interaction between looping and rich conditioning information.

Additionally, the tasks are exclusively visual generation. The paper's claims about "parameter efficiency," "Any-Time inference," and "scaling behavior" are demonstrated for two visual datasets and two architectural paradigms within the visual generation domain. The paper does not test on language tasks, audio generation, or other modalities where looped transformers have shown promise (Saunshi et al., 2025; Geiping et al., 2025). This is not a failure of the paper—it is scoped to visual generation—but it means the generality of the ILSD approach to other modalities is unproven.

What evidence exists in the paper. Tables 1, 2, and 4 cover ImageNet 256×256 (masked and diffusion) and UCF-101 (masked). The paper does not experiment with text conditioning, larger resolutions (e.g., 512×512 ImageNet), or non-generative visual tasks. The related work section discusses looped transformers in language domains (Saunshi et al., 2025; Geiping et al., 2025) but provides no cross-domain experiments.

Mitigation status. The paper does not claim applicability beyond visual generation—the title and abstract specify "Visual Generation." However, the paper does not discuss how text conditioning, larger resolutions, or other modalities might interact with the looping mechanism. A practitioner interested in applying ELT to text-to-image generation (the dominant use case for generative models in practice) would need to conduct substantial additional experiments to determine whether ILSD remains effective when cross-attention layers are present in the looped block, how the text-conditioning signal propagates through repeated loops, and whether the optimal $N$, $L_{\text{max}}$, and curriculum schedule differ from the class-conditional case. The paper provides no guidance on these questions.

7. Implications and Future Directions

How This Work Changes the Landscape

ELT shifts the conversation around parameter efficiency in visual generation from architecture-specific pruning and width reduction toward recurrent depth as a first-class design axis. This is a reframing rather than a paradigm shift: the individual components (looping, distillation, shared-weight transformers) are established, but the paper's contribution is demonstrating that these components, combined via ILSD, can produce a single trained model that serves an entire spectrum of deployment scenarios without retraining.

The conceptual shift is from "train the smallest model that meets your quality target" to "train one elastic model and select the operating point at inference time." This is analogous to the shift in language models from fixed-size deployments to mixture-of-experts and adaptive computation—but realized through a fundamentally simpler mechanism (weight sharing + self-distillation) rather than architectural complexity. The paper demonstrates that this shift is viable for visual generation at competitive quality levels (FID 2.0 on ImageNet, FVD 72.8 on UCF-101), which was not obvious before: prior looped transformer work had shown promise in language reasoning (Saunshi et al., 2025; Geiping et al., 2025) but visual generation imposes different demands (spatial coherence, fine texture, high-dimensional continuous or discrete token spaces) that might have been incompatible with severe parameter sharing.

The paper also resolves a latent tension in the literature about whether looped architectures can handle high-fidelity generation. The $1N \times 32L$ failure case (FID 10.30) demonstrates that naive extreme parameter sharing fails catastrophically—a finding that might have led researchers to conclude that looping is unsuitable for visual generation. But ELT shows this failure is a property of insufficient per-loop expressiveness ($N$ too small), not of looping itself. With appropriate $N$ (4–16 unique layers) and ILSD training, looped architectures match or exceed unique-layer baselines. This provides a boundary condition: looping works for visual generation when $N \geq 4$ and training includes intermediate supervision; it fails when $N = 1$. This is a practical diagnostic that prior work had not established.

The throughput analysis (Table 3) reframes parameter efficiency from an abstract metric (parameter count) to a hardware-grounded one (throughput gains conditional on memory bandwidth bottlenecks). This is important because it tells the field when parameter reduction actually matters for latency: only when the baseline model's parameters exceed on-chip memory capacity. The 1.0× throughput ratio at Scale B is as informative as the 3.5× at Scale H—it establishes that parameter count reduction is necessary but not sufficient for speed improvements. This should influence how future efficiency papers report their results: parameter count reduction alone is not a performance claim; it must be accompanied by throughput measurements on relevant hardware.

The paper also makes few-step generation and one-step generation paradigms (consistency models, drifting models) more attractive by providing the missing intra-step compute lever. In one-step generation, the loop count $L$ becomes the only knob for controlling quality at inference time—the paper explicitly notes this in the conclusion. This means ELT is not just an alternative to few-step methods but a complement that can make those methods more versatile. The implication for the field: researchers developing one-step generative models should consider looped architectures rather than standard unique-layer transformers, because looping provides the inference-time flexibility that one-step models otherwise lack.

Follow-Up Research This Work Enables

Adaptive per-input loop allocation using a lightweight difficulty predictor. The paper demonstrates that quality improves monotonically with $L$, but applies a uniform $L$ across all inputs. In practice, some images are easier to generate than others—a simple scene with a single object against a uniform background may need only 2 loops, while a complex scene with multiple interacting objects and fine textures may benefit from 8 or more. A natural extension is to train a lightweight "difficulty predictor" that takes the initial noisy or masked latent as input and predicts the minimum $L$ needed to reach a target quality threshold. This predictor could be a small MLP trained on the ILSD teacher's quality estimates at different loop counts, or learned via a budgeted loss that penalizes both quality degradation and excess compute. The paper's ILSD training procedure already produces quality estimates at all loop counts during training (since $L_{\text{int}}$ is stochastically sampled), providing the necessary training signal essentially for free. A strong follow-up would measure the additional compute savings from per-input allocation over uniform allocation on ImageNet, stratified by image complexity (e.g., using the number of objects, scene clutter metrics, or Inception-based difficulty proxies).

Direct comparison of ILSD to alternative intermediate supervision strategies. The paper ablates ILSD vs. no intermediate supervision (Figures 7, 8) but does not compare ILSD to simpler strategies: (a) ground-truth supervision at all intermediate loops without distillation (no teacher matching, just cross-entropy or MSE at every $L$), (b) distillation without ground-truth for the student ($\lambda = 0$ throughout training), (c) a fixed student depth (e.g., always supervising $L_{\text{int}} = L_{\text{max}}/2$) rather than stochastic sampling. Any of these ablations would clarify whether the benefit of ILSD comes from intermediate supervision in general (which would mean simpler methods work almost as well), from the distillation component specifically (which would mean teacher matching is essential), or from the stochastic sampling specifically (which would mean coverage across depths matters). An experiment training ELT-L on ImageNet with these three ablations and measuring FID-vs-$L$ curves would take approximately the same compute as the paper's existing experiments and would substantially strengthen the mechanistic understanding. The paper's claim that "we did not observe sensitivity to the decay rate" also needs empirical backing—a simple comparison of linear decay at rates 2× faster and 2× slower than the default would test this.

Combining ELT with few-step diffusion distillation for fully elastic one-step generation. The paper positions ELT as orthogonal to few-step methods (Section 2) and suggests combining them. The most impactful version of this combination would be: take a consistency model or progressive-distillation model that generates images in 1–4 sampling steps, replace its internal transformer layers with an ELT-style looped block, and train with ILSD. The result would be a model where both inter-step and intra-step compute are elastic—the practitioner could choose (number of sampling steps, loops per step) independently. For one-step generation (the limiting case), $L$ becomes the sole quality lever. A strong follow-up would train such a model on ImageNet 512×512, measure the two-dimensional Pareto frontier of (sampling steps × loops) vs. FID, and compare to a standard few-step model of matched total GFLOPs. The key question: does adding intra-step elasticity to few-step models improve the Pareto frontier, or do the benefits saturate because few-step models already operate near the quality ceiling? The paper's own results (Figure 4 showing diminishing returns in $L$) suggest the gains may be modest for high-quality few-step models but substantial at lower compute budgets.

Testing ELT on text-to-image generation with cross-attention. The paper's experiments are class-conditional only. Text-to-image generation (e.g., Stable Diffusion-style models) introduces cross-attention layers that attend to text embeddings. These cross-attention layers would need to be part of the looped block $g_\Theta$, raising questions: does the text-conditioning signal degrade or get reinforced through repeated loops? Does the optimal $N$ change when cross-attention is present? Does ILSD's distillation between loop depths cause the model to "forget" the text conditioning at intermediate depths? A direct experiment would take a DiT-based text-to-image architecture (e.g., PixArt-$\alpha$ or a Stable Diffusion 3-style model), replace the transformer blocks with ELT looped blocks, train on a text-image dataset (e.g., MS-COCO or a larger internal dataset), and measure both FID and CLIP score as functions of $L$. The key comparison is ELT vs. a non-looped baseline at matched parameter count and GFLOPs, evaluated on text alignment metrics (CLIP score, PickScore) in addition to FID. This would determine whether ELT generalizes beyond class conditioning, which is practically important since text-to-image is the dominant use case for generative models.

Characterizing the extrapolation beyond $L_{\text{max}}$ and developing methods to extend it. The UCF-101 result (FVD improves from 72.9 at $L=4$ to 69.2 at $L=6$, trained with $L_{\text{max}}=4$) is a tantalizing but isolated observation. A systematic study would train ELT models at $L_{\text{max}} \in \{2, 4, 8, 16\}$ on ImageNet (masked and diffusion) and UCF-101, then evaluate FID/FVD at $L$ up to $2 \times L_{\text{max}}$, measuring both the peak improvement beyond $L_{\text{max}}$ and the loop count at which degradation begins. This would establish whether extrapolation is a general phenomenon (occurs across datasets and $L_{\text{max}}$ values) or specific to data-constrained regimes like UCF-101. If extrapolation proves general, a follow-up would explore training modifications to extend it—for example, training with a small probability of sampling $L_{\text{int}} > L_{\text{max}}$ (extrapolatory student sampling) to explicitly train for beyond-$L_{\text{max}}$ behavior, or adding a regularization term that penalizes large changes in the representation between consecutive loops (encouraging the iterative process to converge smoothly rather than oscillate). The paper's hypothesis that ILSD regularizes the iterative process enough for modest extrapolation is plausible but untested mechanistically; an experiment measuring representation change between loops (e.g., cosine similarity between $F^{(N, L)}(x)$ and $F^{(N, L+1)}(x)$) could reveal whether ILSD-trained models converge more smoothly than vanilla looped models.

Scaling study of the $(N, d, L)$ design space to produce actionable deployment guidelines. The paper sweeps configurations across five model scales and reports the Pareto frontier (Figure 4), but the analysis is post-hoc and descriptive. A more systematic study would fix a total parameter budget (e.g., 100M, 200M, 500M) and, for each budget, train multiple $(N, d, L_{\text{max}})$ configurations that use that budget, then measure the full FID-vs-$L$ curve for each. This would produce a lookup table: "If you have X million parameters and Y GFLOPs at inference, use configuration Z." Such a study would also characterize the interaction between $d$ and the minimum viable $N$—does a wider model (larger $d$) allow smaller $N$ because each layer has more capacity? The paper's $1N \times 32L$ failure uses a specific hidden dimension; would $1N \times 32L$ with $d=4096$ (much wider single layer) perform better? Answering this would help practitioners allocate their parameter budget between width and depth. The study would need to control for total training compute (not just parameter count) to ensure fair comparison, since wider models train slower per step.

Practical Applications and Downstream Use Cases

On-device image generation with dynamic quality control. Mobile devices (phones, tablets, AR/VR headsets) have severe memory and power constraints but varying latency requirements depending on the use case—a real-time camera filter needs sub-100ms generation, while an offline image editing tool can afford several seconds. With ELT, a single model checkpoint (e.g., ELT-L at $8N \times 3L$, 101M parameters) can serve both scenarios: the real-time filter runs at $L=1$ (lower quality but fast), while the editing tool runs at $L=4$ (higher quality). The paper's throughput measurements (Table 3) and quality-vs-loops curves (Figures 7, 8) provide the calibration data: at Scale L, throughput is 2.9× the baseline, and FID improves from ~6 at $L=2$ to ~2.2 at $L=8$ (Figure 8a), so the engineer can pick the $L$ that hits their latency budget. The key practical advantage over storing multiple models (a fast low-quality model and a slow high-quality model) is reduced storage (one checkpoint vs. two) and seamless quality transitions (the user can adjust a quality slider that maps directly to $L$).

Cost-optimized cloud video generation pipelines. Cloud-based video generation for professional content creation (marketing videos, animated shorts, visual effects previsualization) involves generating many candidate videos and selecting the best. The generation budget is typically fixed (e.g., total GPU-hours per project), and the challenge is allocating that budget across candidate generations. ELT enables a two-stage pipeline: generate a large number of candidates at low $L$ (fast, lower quality) to identify promising directions, then regenerate the top candidates at high $L$ (slow, higher quality). The paper's UCF-101 results (Table 4) quantify this tradeoff: $L=4$ gives FVD 72.8 at ~4.3K GFLOPs, while $L=6$ gives FVD 60.8 at ~13K GFLOPs. If the pipeline generates 100 candidates at $L=2$ (fast scan), selects the top 5 by some heuristic (e.g., motion consistency, user preference model), and regenerates those 5 at $L=6$, the total cost is $100 \times \text{cost}(L=2) + 5 \times \text{cost}(L=6)$ rather than $100 \times \text{cost}(L=6)$ for a brute-force high-quality search—a substantial savings while maintaining final output quality. The single-model property is crucial here: the $L=2$ and $L=6$ generations come from the same model, so the low-quality preview is a faithful preview of what the high-quality regeneration will produce, unlike using two separate models where the fast model might make different "creative choices" than the slow model.

Data generation for training visual models with quality-tiered datasets. Many visual understanding models (object detectors, segmenters, image classifiers) benefit from training on synthetic data generated by generative models. The quality requirements vary: for pre-training, large quantities of medium-quality synthetic images may be sufficient; for fine-tuning, smaller quantities of high-quality images are preferred. ELT enables a single generative model to produce both tiers: run at low $L$ for bulk pre-training data (prioritizing throughput and diversity) and at high $L$ for fine-tuning data (prioritizing per-sample quality). The paper's ImageNet results (Figure 4) show that FID improves from ~5 to ~2 as GFLOPs scale from ~1000 to ~4000, so the quality gap between tiers is substantial and controllable. This is more efficient than training and maintaining two separate generative models (one fast, one high-quality) and ensures that the two data tiers share the same generative distribution (same model weights, same latent space structure), avoiding distribution shift between pre-training and fine-tuning data.