Decoupled DMD: CFG Augmentation as the Spear, Distribution Matching as the Shield

ArXiv: 2511.22677

🎯 Pitch

DMD's few-step distillation actually works by baking classifier-free guidance directly into the student through a 'CFG Augmentation' engine, not through distribution matching as previously believed. The distribution matching term merely serves as a regularizer to prevent training collapse and artifacts, and surprisingly, even simpler constraints like matching mean and variance can substitute for it.

---

1. Executive Summary

This paper analyzes the practical training objective of Distribution Matching Distillation (DMD) for text-to-image diffusion models and discovers that its effectiveness stems not from distribution matching, but from a functional decoupling into two distinct mechanisms: a CFG Augmentation (CA) term that acts as the primary "engine" for converting a multi-step teacher into a few-step student generator (by directly baking the classifier-free guidance signal into the student's predictions), and a Distribution Matching (DM) term that serves as a "regularizer" ensuring training stability and mitigating artifacts like over-saturation and high-frequency noise. Through ablation studies on SDXL and Lumina-Image-2.0, the authors demonstrate that training with CA alone achieves few-step conversion quality comparable to full DMD but collapses without the DM regularizer, and that these components benefit from independent re-noising schedules—constraining CA to noise levels above the current generation timestep while keeping DM global—yielding consistent improvements across multiple benchmarks (e.g., +7.6 ImageReward points over DMD2 on 4-step SDXL distillation). The paper establishes that the DM regularizer, while highly effective, is not unique: simpler non-parametric mean-variance constraints or GAN-based objectives can also stabilize the CA engine, though with different stability-performance trade-offs.

2. Context and Motivation

The Core Problem: We Don't Actually Understand Why DMD Works

Distribution Matching Distillation (DMD) and its variants have achieved impressive results in distilling multi-step diffusion models into efficient few-step generators, producing images of quality that rivals or exceeds many competing distillation paradigms. The standard explanation for this success, repeated across the score-based distillation literature, is elegant and theoretically grounded: DMD minimizes the Integral Kullback-Leibler (IKL) divergence between the student's output distribution $p_{\text{fake}}$ and the teacher's target distribution $p_{\text{real}}$, effectively transferring the teacher's modeling capability to a faster student through principled distribution matching.

The problem is that this explanation does not survive scrutiny of how DMD is actually implemented in practice. The theoretical objective (Equation 2 in the paper) uses the teacher's conditional score prediction $s_{\text{cond}}^{\text{real}}$ as the target signal. In practice, however, practitioners uniformly replace this with a Classifier-Free Guidance (CFG) augmented score $s_{\text{cfg}}^{\text{real}} = s_{\text{uncond}}^{\text{real}} + \alpha(s_{\text{cond}}^{\text{real}} - s_{\text{uncond}}^{\text{real}})$ with $\alpha > 1$, while keeping the student-tracking "fake" model's score $s_{\text{cond}}^{\text{fake}}$ unguided. This is not a minor implementation detail — it creates a fundamental asymmetry between the two score models that breaks the theoretical framework's assumption that both models are estimating scores of their respective distributions under identical conditioning.

The paper identifies this gap explicitly:

"Even if we were to boldly assume—an assumption we find to be insufficiently grounded—that CFG somehow produces a higher-quality substitute for the original score, the asymmetric application of CFG, in which only the real model but not the fake model is equipped with CFG, still creates a stark inconsistency between theory and practice. Overall, the usage of CFG breaks the integrity of the original, rigorous theoretical derivation of matching two distributions."

This is not merely an academic quibble about theoretical purity. If the distribution matching interpretation is wrong or incomplete, then the entire research program built on top of it — designing better distribution divergences, improving the fake model training, analyzing convergence properties — may be optimizing the wrong thing. More concretely, if we don't understand which part of the DMD objective actually does the distillation work, we cannot systematically improve it. We are stuck with empirical trial-and-error rather than principled design.

Why This Gap Matters: Practical and Theoretical Significance

The paper's investigation has direct consequences for both practitioners and theorists:

For practitioners, the DMD training recipe involves numerous design choices: the CFG scale $\alpha$, the re-noising timestep distribution $\tau$, the fake model training schedule, the balance between different loss terms in DMD variants like DMD2 (which adds a GAN loss), and so on. Without understanding which component is the "engine" versus the "regularizer," tuning these hyperparameters is a blind search. If, as the paper demonstrates, CA is the actual distillation engine, then practitioners should focus their tuning efforts on the CA term's properties (its noise schedule, its guidance scale) rather than on the fake model or the distribution matching divergence. Conversely, if DM is a regularizer, improvements to it should prioritize stability and artifact suppression over distribution-matching fidelity.

For theorists, the finding challenges the narrative that score-based distillation works primarily through a distribution-level optimization. The paper's decomposition reveals that the effective gradient signal driving few-step conversion is $\Delta_{\text{cfg}}^{\text{real}} = s_{\text{cond}}^{\text{real}} - s_{\text{uncond}}^{\text{real}}$ — the CFG direction itself — applied directly to the student's output, without any involvement of the fake model at all (Section 3.1). This means the student isn't learning to match the teacher's distribution; it's learning to internalize the CFG correction that the teacher would apply at inference time. The distinction is subtle but profound: distribution matching is about replicating the teacher's unconditional generative process, while CFG augmentation is about replicating the teacher's guided sampling behavior.

This explains a persistent puzzle in the distillation literature. Why do trajectory-matching methods (like progressive distillation or consistency models) and GAN-based methods (like ADD or SDXL-Turbo) also succeed without any explicit distribution matching to a teacher score? The answer, from this paper's perspective, is that they all share a common underlying mechanism: they bake some form of CFG-like conditioning or adversarial signal into the student, bypassing the need for step-by-step score alignment. The paper's framework thus offers a unifying lens through which to understand seemingly disparate distillation approaches.

Where Prior Approaches Fall Short

The paper contextualizes its contribution against three broad categories of prior work, each of which has crucial blind spots regarding the role of CFG:

Score-based distillation methods (Diff-Instruct, DMD, DMD2, SiD, ASID). These works collectively established the paradigm of minimizing a statistical divergence between student and teacher distributions using a pair of score models. The theoretical frameworks are rigorous (Luo et al., 2023b proved that Diff-Instruct minimizes IKL divergence; Zhou et al., 2024a;b explored alternative divergences like score identity), but they uniformly treat the CFG substitution as a heuristic relaxation of the theoretical framework, not as a mechanism with its own independent function. The paper cites this directly:

"the adoption of CFG in real score is a common practice among these works, but this choice is rarely officially discussed. An exception is (Luo, 2024), which models the CFG term as an extra reward function after distillation."

Luo (2024) (Diff-Instruct++) is the closest prior work to recognizing the CFG term as having independent significance, but it frames CFG as a post-hoc reward augmentation — something that improves the evaluation of the student's outputs — rather than as the primary training engine that drives the few-step conversion. The paper's decomposition goes further by showing that the CA term directly provides the gradient signal that converts the generator, while DM plays a purely stabilizing role.

Trajectory-matching and consistency distillation methods (LCM, PCM, CTM). These approaches train the student to match the teacher's sampling trajectory in fewer steps, either by directly regressing on intermediate states (progressive distillation, Salimans & Ho, 2022) or by enforcing self-consistency along the probability flow ODE (consistency models, Song et al., 2023). While these methods can produce reasonable few-step generators, they do not explicitly address or explain the role of CFG. In practice, they often require CFG during inference or incorporate it heuristically into the training objective. The paper's CA/DM decomposition provides a potential explanation for why these methods work — they may implicitly serve the same function as CA, baking the guidance signal into the student — but this connection is not explored in the prior trajectory-matching literature.

GAN-based distillation methods (ADD, SDXL-Turbo, SDXL-Lightning, Diffusion Adversarial Post-Training). These methods use a discriminator to provide an adversarial training signal that pushes the student's outputs toward the teacher's distribution. They achieve impressive results, particularly at very low step counts, but introduce significant training instability — a well-known challenge in GAN training. The paper's investigation of GAN as an alternative regularizer (Section 3.2, Figure 3) reveals that while a GAN discriminator can stabilize the CA engine, it collapses after roughly 4000 iterations, echoing the instability concerns that Diff-Instruct originally identified when positioning score-based distillation as "a more stable alternative to GANs, especially when the distributions have disjoint supports" (Luo et al., 2023b). The DM regularizer thus occupies a sweet spot: more powerful than simple statistical constraints, more stable than GANs.

The Overlooked Question: What Does the CFG Term Actually Do?

The paper's central motivating observation is that no prior work has rigorously decomposed the practical DMD objective to understand which part of the gradient signal drives distillation. The substitution $s_{\text{cond}}^{\text{real}} \rightarrow s_{\text{cfg}}^{\text{real}}$ in Equation 3 is treated as an empirical hack that "works better," with the implicit assumption that it provides a sharper, higher-quality target distribution for the student to match. The paper challenges this assumption by asking a simple question: if distribution matching were truly the mechanism, why does the method fail so dramatically when the theoretically correct score $s_{\text{cond}}^{\text{real}}$ is used (Equation 2)?

The answer cannot be that $s_{\text{cond}}^{\text{real}}$ is simply a worse score estimate — the teacher model is perfectly capable of producing high-quality conditional scores, and these scores are what drive the teacher's own multi-step generation process. Something else must be happening. The paper's decomposition reveals what that "something else" is: the CFG substitution introduces a new gradient term $(\alpha - 1)(s_{\text{cond}}^{\text{real}} - s_{\text{uncond}}^{\text{real}})$ that was never part of the distribution matching derivation. This term directly applies the CFG direction to the student's output, teaching the student to predict what the teacher would produce after guidance is applied, rather than what the teacher would produce without it.

How This Paper Positions Itself

The paper positions itself as a mechanistic analysis paper rather than a new method paper. Its primary contribution is not a new distillation algorithm but a reinterpretation of existing algorithms that enables principled improvements. This is explicitly stated in the introduction:

"In this paper, we aim to redefine the understanding of how DMD and similar algorithms work... we reveal that its effectiveness is not driven by a single mechanism, but by a clear division of labor between two distinct components."

This framing is important because it changes the nature of the paper's claims. The paper is not arguing that DMD is wrong — it's arguing that our understanding of why DMD works is wrong, and that correcting this understanding leads to better design choices. The decoupled noise schedule (Section 4.3) serves as validation of this claim: if the CA term is the engine, constraining its noise schedule to focus on unresolved image frequencies should improve performance (it does, Table 1, Figures 4-5); if the DM term is a regularizer, keeping its noise schedule global to correct artifacts at all frequency levels should improve stability (it does, Table 1, Figure 5 bottom row vs. middle row).

The paper also explicitly connects to the broader question of why sequential generation is necessary in diffusion models (Appendix A), drawing a parallel to why LLMs cannot predict multiple tokens at once. This theoretical framing — that CFG represents an external, uncontrollable decision pattern that the student must internalize to collapse the sequential generation tree into a single direct path — is speculative but provides a conceptual bridge between the empirical findings and a deeper understanding of generative models. The paper is careful to label this as a hypothesis rather than a proven claim:

"We acknowledge that the preceding discussion remains at the level of high-level ideas and that our hypothesis—that 'CFG represents a specific, deterministic decision pattern'—is a strong assumption. We share this perspective here primarily to stimulate further investigation into this fundamental question."

This intellectual honesty is characteristic of the paper's approach: it identifies a genuine gap in our understanding, provides rigorous empirical evidence for a new interpretation, derives practical improvements from that interpretation, and explicitly flags the remaining open questions rather than overclaiming.

3. Technical Approach

3.1 Reader Orientation

This paper is primarily a mechanistic analysis rather than a new method proposal: it takes the existing DMD training algorithm and decomposes its objective function to understand which parts actually do the distillation work. The system being analyzed is a training pipeline that converts a slow, multi-step diffusion model (the teacher) into a fast, few-step generator (the student) using gradient signals derived from pre-trained score models, and the core contribution is the discovery that this gradient signal splits into two functionally independent components — an engine that drives the conversion and a regularizer that keeps training stable.

3.2 Big-Picture Architecture (Diagram in Words)

The DMD training system has four major components:

1. A pre-trained diffusion teacher model — a frozen multi-step generative model that produces high-quality images when run with many denoising steps and Classifier-Free Guidance (CFG). It provides two score functions: a conditional score $s_{\text{cond}}^{\text{real}}$ (conditioned on text) and an unconditional score $s_{\text{uncond}}^{\text{real}}$ (conditioned on a null prompt). These are the "real" score estimates.

2. A student generator $G_\theta$ — initialized from the teacher's weights, this model takes a noisy latent $z_t$ at timestep $t$ and directly predicts a clean image $G_\theta(z_t)$ in a small number of steps (1, 4, or 8 in the paper's experiments). This is the model being trained.

3. A "fake" score model $s^{\text{fake}}$ — also initialized from the teacher and updated concurrently during training on the student's outputs. It estimates the score of the student's current output distribution $p_{\text{fake}}$, serving as a tracking model that learns to replicate the student's characteristic outputs (including its artifacts).

4. A re-noising mechanism — at each training step, the student's predicted clean image $G_\theta(z_t)$ is corrupted with Gaussian noise at a sampled timestep $\tau$ to produce a noisy intermediate $x_\tau$. Both the real and fake score models then operate on this $x_\tau$ to produce gradient signals that update the student.

Information flows as follows: the student generates a clean image from a noisy input → the image is re-noised to some intermediate noise level $\tau$ → the real teacher model and the fake tracking model both estimate scores at this noise level → the DMD loss gradient is computed as the difference between these score estimates → this gradient is backpropagated through the student (via a proxy L2 loss) to update its parameters → periodically, the fake model is updated to denoise the student's outputs, keeping it aligned with $p_{\text{fake}}$.

3.3 Roadmap for the Deep Dive

• First, the theoretical DMD objective (Equation 2) and the practical DMD objective (Equation 3), establishing the precise mathematical gap between theory and practice that motivates the entire paper.
• Second, the algebraic decomposition of the practical DMD gradient into its CFG Augmentation (CA) and Distribution Matching (DM) components (Equation 6), which is the paper's central analytical move.
• Third, the ablation study design that isolates CA and DM, along with the empirical evidence establishing CA as the engine and DM as the regularizer (Section 3.1.1, Figure 2).
• Fourth, the investigation of alternative regularizers — mean-variance constraints and GAN discriminators — that validates DM's role as non-unique stabilizer (Section 3.2).
• Fifth, the mechanistic analysis of the CA engine's behavior as a function of the re-noising timestep $\tau$ (Section 4.1), establishing that CA enhances image content corresponding to the noise level at which it is applied.
• Sixth, the diagnostic experiment revealing DM's corrective mechanism — how the fake model learns to replicate the student's artifacts, making the DM gradient an artifact-cancelling signal (Section 4.2, Figure 4b).
• Seventh, the proposal and validation of decoupled re-noising schedules — constrained for CA ($\tau_{\text{CA}} > t$), global for DM ($\tau_{\text{DM}} \in [0,1]$) — as a principled improvement derived from the engine-regularizer understanding (Section 4.3, Equation 8, Tables 1-2).

3.4 Detailed, Sentence-Based Technical Breakdown

This is a mechanistic analysis paper whose core idea is that the practical DMD training objective contains two functionally independent gradient terms — one that directly bakes the CFG signal into the student (the engine) and one that matches distributions (the regularizer) — and that recognizing this decoupling enables principled improvements to the distillation process.

---

The Theoretical DMD Objective and Its Gradient

The Distribution Matching Distillation framework, as originally derived in Diff-Instruct (Luo et al., 2023b) and DMD (Yin et al., 2024b), is grounded in minimizing an Integral Kullback-Leibler (IKL) divergence between the student's output distribution $p_{\text{fake}}$ and the teacher's target distribution $p_{\text{real}}$:

$L_{\text{IKL}}(p_{\text{real}}, p_{\text{fake}}) = \int_0^1 \text{KL}(p_{\text{real},\tau} \,||\, p_{\text{fake},\tau}) \, d\tau$

where $p_{\text{real},\tau}$ is the distribution of real data corrupted with noise at level $\tau$, and $p_{\text{fake},\tau}$ is the distribution of student outputs corrupted with noise at level $\tau$. The integral ranges from $\tau = 0$ (pure noise) to $\tau = 1$ (clean data), following the flow matching convention adopted throughout the paper.

What it computes: the expected KL divergence between the teacher's and student's distributions at every noise level, integrated over the full diffusion trajectory. For each $\tau$, we add Gaussian noise to both real and fake samples, then measure how distinguishable the resulting distributions are. Minimizing this drives the student to produce outputs whose noised versions match the teacher's noised versions at all noise levels.

Why this form: the integral over $\tau$ exploits the fact that diffusion models estimate scores at all noise levels. By matching distributions in noised space rather than clean data space, the objective avoids the problem of disjoint supports — two distributions of clean images may have non-overlapping support (making KL divergence undefined), but after adding even a small amount of noise, their supports overlap, making the divergence well-behaved. This is the key insight that makes score-based distillation feasible.

The gradient of this objective with respect to the student's parameters $\theta$ can be estimated as:

$\nabla_\theta L_{\text{DMD-theory}} = \mathbb{E}_{z_t, \tau, x_\tau}\left[-\left(s_{\text{cond}}^{\text{real}}(x_\tau) - s_{\text{cond}}^{\text{fake}}(x_\tau)\right) \frac{\partial G_\theta(z_t)}{\partial \theta}\right]$

where $z_t$ is the generator input at timestep $t$ (pure noise for single-step generation, or the output of previous steps for few-step generation via "backward simulation"), $\tau \sim \mathcal{U}(0,1)$ is a randomly sampled re-noising timestep, $x_\tau$ is the student's predicted clean image $G_\theta(z_t)$ after adding noise to level $\tau$, $s_{\text{cond}}^{\text{real}}(x_\tau)$ is the teacher's conditional score estimate at noise level $\tau$, $s_{\text{cond}}^{\text{fake}}(x_\tau)$ is the fake model's conditional score estimate, and $\frac{\partial G_\theta(z_t)}{\partial \theta}$ is the Jacobian of the generator output with respect to its parameters.

What it computes: the update direction for the student generator. The core signal is $\Delta_{\text{real-fake}} = s_{\text{cond}}^{\text{real}} - s_{\text{cond}}^{\text{fake}}$: the difference between what the teacher thinks the score should be and what the fake model (which tracks the student's current output distribution) thinks the score is. When this difference is non-zero, it indicates that the student's noisy outputs $x_\tau$ are not following the teacher's expected score, and the gradient pushes the student's clean output $G_\theta(z_t)$ in a direction that reduces this discrepancy.

Why this form: the gradient takes the form of a score difference multiplied by the generator's Jacobian because the loss is a functional of the student's output distribution. The score difference $\Delta_{\text{real-fake}}$ acts as a vector field: at each point $x_\tau$ in the noised space, it points from regions of high $p_{\text{fake},\tau}$ density toward regions of high $p_{\text{real},\tau}$ density. Moving the generator's clean output $G_\theta(z_t)$ in the direction indicated by this vector field (via the chain rule through the re-noising process) reduces the KL divergence between the two distributions.

---

The Practical DMD Objective: Introducing CFG Asymmetry

In practice, the theoretically derived gradient in Equation 2 produces poor results on complex tasks like text-to-image generation. Practitioners universally replace the teacher's conditional score $s_{\text{cond}}^{\text{real}}$ with a Classifier-Free Guidance (CFG) augmented score:

$\nabla_\theta L_{\text{DMD}} = \mathbb{E}_{z_t, \tau, x_\tau}\left[-\left(s_{\text{cfg}}^{\text{real}}(x_\tau) - s_{\text{cond}}^{\text{fake}}(x_\tau)\right) \frac{\partial G_\theta(z_t)}{\partial \theta}\right]$

where the CFG score is defined as:

$s_{\text{cfg}}^{\text{real}}(x_\tau) = s_{\text{uncond}}^{\text{real}}(x_\tau) + \alpha\left(s_{\text{cond}}^{\text{real}}(x_\tau) - s_{\text{uncond}}^{\text{real}}(x_\tau)\right)$

What it computes: a modified "real" score that extrapolates away from the unconditional distribution and toward the conditional distribution, controlled by the guidance scale $\alpha > 1$. When $\alpha = 1$, this reduces to the original conditional score; when $\alpha > 1$, it amplifies the difference between the conditional and unconditional predictions, effectively sharpening the teacher's target distribution toward modes that are more strongly associated with the conditioning text.

Why this form: Classifier-Free Guidance was originally introduced by Ho & Salimans (2022) as an inference-time technique to improve the quality and text-alignment of diffusion model outputs. At each denoising step, instead of using $s_{\text{cond}}$ directly, the model extrapolates beyond the conditional estimate in the direction away from the unconditional estimate. This has the effect of increasing the influence of the conditioning signal and reducing the influence of the unconditional prior, producing sharper, more text-aligned images. The empirical fact is that text-to-image diffusion models produce high-quality outputs only with CFG applied; the unconditional or standard conditional outputs are significantly worse. Therefore, a student that learns to match the teacher's unguided distribution would learn to produce the teacher's mediocre unconditional outputs, which is not the goal. The CFG substitution is an attempt to make the target distribution be the teacher's guided sampling distribution — the one that actually produces good images.

The paper identifies the critical asymmetry: the fake model's score remains unguided ($s_{\text{cond}}^{\text{fake}}$, with no CFG applied), while the real score is guided ($s_{\text{cfg}}^{\text{real}}$). This means the gradient signal is no longer a pure comparison of two score estimates under identical conditions. The theoretical derivation that Equation 2 minimizes the IKL divergence assumes both score models provide unbiased estimates of their respective distributions' scores under the same conditioning scheme. Breaking this symmetry breaks the distribution matching interpretation.

---

The Central Decomposition: Separating the DMD Gradient into CA and DM

The paper's key analytical contribution is a simple algebraic rearrangement of the practical DMD gradient. By substituting the CFG definition (Equation 4) into the DMD gradient (Equation 3):

$\nabla_\theta L_{\text{DMD}} = \mathbb{E}\left[-\left(s_{\text{uncond}}^{\text{real}}(x_\tau) + \alpha(s_{\text{cond}}^{\text{real}}(x_\tau) - s_{\text{uncond}}^{\text{real}}(x_\tau)) - s_{\text{cond}}^{\text{fake}}(x_\tau)\right) \frac{\partial G_\theta(z_t)}{\partial \theta}\right]$

and rearranging terms, the gradient decomposes into:

$\nabla_\theta L_{\text{DMD}} = \mathbb{E}\left[-\left(\underbrace{(s_{\text{cond}}^{\text{real}}(x_\tau) - s_{\text{cond}}^{\text{fake}}(x_\tau))}_{\Delta_{\text{real-fake}}: \text{Distribution Matching}} + \underbrace{(\alpha - 1)(s_{\text{cond}}^{\text{real}}(x_\tau) - s_{\text{uncond}}^{\text{real}}(x_\tau))}_{\Delta_{\text{cfg}}^{\text{real}}: \text{CFG Augmentation}}\right) \frac{\partial G_\theta(z_t)}{\partial \theta}\right]$

What it computes: the total DMD gradient as the sum of two independent terms. The first term $\Delta_{\text{real-fake}}$ is exactly the theoretical distribution matching gradient from Equation 2 — it compares the real conditional score against the fake conditional score and would be the only term present if CFG were not used. The second term $\Delta_{\text{cfg}}^{\text{real}}$ is purely a function of the teacher model: it is the CFG direction $(s_{\text{cond}}^{\text{real}} - s_{\text{uncond}}^{\text{real}})$ scaled by $(\alpha - 1)$, and it involves the fake model not at all.

Why this form matters: this decomposition reveals that the practical DMD objective contains a term ($\Delta_{\text{cfg}}^{\text{real}}$) that was never present in the theoretical derivation. This term does not compare the student's distribution against the teacher's distribution — it directly applies the teacher's CFG correction vector as a gradient to the student's output. The student is not just learning to match $p_{\text{real}}$; it is being explicitly taught to move its predictions in the direction that CFG would move them at inference time. If this term dominates the gradient (which the ablation studies will show it does), then the mechanism of DMD is fundamentally different from distribution matching.

The paper emphasizes the significance of the fake model's absence from the CA term: "This component was typically overlooked" (Section 3.1). The CA term uses only the frozen teacher model's internal signals ($s_{\text{cond}}^{\text{real}}$ and $s_{\text{uncond}}^{\text{real}}$). It does not involve the fake model at all, meaning the student could theoretically learn from CA alone without ever training or using a fake score model. The fake model's sole purpose is in the DM term, which — if the CA term is the primary engine — serves a secondary, stabilizing function.

---

Ablation Study Design: Isolating Engine from Regularizer

To test the hypothesis that CA is the distillation engine and DM is the regularizer, the paper designs an ablation study with three training configurations (Section 3.1.1):

Configuration 1: Full DMD ($\Delta_{\text{real-fake}} + \Delta_{\text{cfg}}^{\text{real}}$). This is the standard practical DMD objective, using both terms exactly as they appear in Equation 6. It serves as the performance upper bound and baseline.

Configuration 2: CFG Augmentation only ($\Delta_{\text{cfg}}^{\text{real}}$). The student is trained using only the CA term $(\alpha - 1)(s_{\text{cond}}^{\text{real}} - s_{\text{uncond}}^{\text{real}})$. The fake model is never used for the generator update (though the paper keeps it training in the diagnostic experiment of Figure 4b to observe what it learns). The DM term $\Delta_{\text{real-fake}}$ is completely removed from the gradient.

Configuration 3: Distribution Matching only ($\Delta_{\text{real-fake}}$). The student is trained using only the theoretically correct DM term $s_{\text{cond}}^{\text{real}} - s_{\text{cond}}^{\text{fake}}$, with no CFG augmentation. This corresponds to $\alpha = 1$ in Equation 3, or equivalently, using Equation 2 directly. Both the fake model and the real model operate in conditional mode only.

All three configurations use the same base architecture (SDXL for the experiments in Figure 2), the same generator initialization (from the teacher), the same fake model initialization and training procedure (except Configuration 2 where the fake model gradient does not update the generator), and the same re-noising schedule ($\tau \sim \mathcal{U}(0,1)$). For the CA-only configuration, the CFG scale is set to the same value used in the full DMD configuration (the standard SDXL CFG scale, which is typically $\alpha$ values of 5-7.5 for SDXL; the exact value is not specified in the ablation description but follows the standard DMD practice for that model).

The experiments are run for two step-count settings: 1-step SDXL distillation and 4-step SDXL distillation. Both are evaluated at multiple training iterations (200, 600, 1000, 1400, 1800 for 1-step; 200, 1000, 2000, 4000, 6000 for 4-step) using two metrics: ImageReward (Xu et al., 2023) and HPS v2.1 (Wu et al., 2023), both designed to correlate with human preferences for text-to-image generation. All evaluations use 1000 randomly sampled prompts from the COCO-10k validation set (Lin et al., 2014).

---

Ablation Results: Establishing the Division of Labor

The results in Figure 2 reveal a clear and consistent pattern across both 1-step and 4-step settings:

CFG Augmentation alone is remarkably effective as a distillation engine. In the 4-step SDXL setting, the CA-only configuration produces images that are qualitatively very similar to full DMD in terms of content, composition, and overall structure. The ImageReward and HPS v2.1 curves for CA-only track the full DMD curves closely, particularly in early training (up to approximately 2000 steps for 4-step, 600 steps for 1-step). This demonstrates that the CA term alone carries nearly all the information needed to convert the multi-step model into a few-step generator — the fake model and the distribution matching objective are not necessary for the core distillation task.

Distribution Matching alone is substantially worse. The DM-only configuration (using $s_{\text{cond}}^{\text{real}} - s_{\text{cond}}^{\text{fake}}$ with no CFG) produces significantly worse results across both metrics in the 1-step setting and produces noticeably degraded images in the 4-step setting. While the paper notes that "it is improper to conclude that the DM term is totally incapable of doing the multi-step to few-step conversion (since in the 4-step experiment it indeed makes relatively reasonable images)," the performance gap versus CA-only is large and persistent. The numerical indicators (ImageReward, HPS v2.1) show DM-only consistently below both CA-only and full DMD.

CA alone is unsustainable without DM. The critical finding is that while CA-only matches full DMD performance initially, it eventually diverges: the generated images "progressively suffer from artifacts such as over-saturation and high-frequency noise, eventually leading to training collapse" (Section 3.1.1). In the 1-step setting, this collapse is visible as a sharp degradation in image quality at later iterations (around 1800 steps), and in the 4-step setting, it manifests as increasing artifact severity. The full DMD configuration (CA + DM) does not exhibit this collapse — the DM term "eliminates these issues, enabling stable training over extended periods and yielding higher-quality final results."

The combined objective achieves the best final quality. Full DMD (CA + DM) reaches the highest final ImageReward and HPS v2.1 scores in both settings, exceeding both CA-only (which collapses) and DM-only (which underperforms from the start). The DM term is not just preventing collapse; it is enabling the CA engine to continue improving beyond the point where unregularized CA would degrade.

The paper distills these observations into two conclusions that become the foundation for all subsequent analysis:

"1. CFG Augmentation is the engine for few-step conversion. The ability of the distilled generator to produce high-quality samples in a few steps is almost entirely attributable to the $\Delta_{\text{cfg}}$ term. 2. Distribution Matching is a regularizer for training stability. The $\Delta_{\text{real-fake}}$ term, while not the primary driver of distillation, plays a crucial role as a regularizer that prevents the training process from diverging and ensures the quality of the final output."

A subtle but important implication: the conversion to a few-step generator "is not primarily an act of matching distributions but rather a direct consequence of 'baking' the CFG pattern into the student generator's predictions... which is irrelevant to the fake model" (Section 3.1.1). The fake model, which previous literature treated as an essential component for estimating the student's distribution, is revealed to be unnecessary for the core distillation mechanism. It matters only insofar as it participates in the DM regularizer.

---

The Training Procedure: Generator Update via Proxy Loss

The paper's pseudo-code (Algorithm 1 in Appendix B) provides the complete training procedure, which is identical for both the original DMD and the proposed decoupled DMD except for the noise sampling step. The generator update follows a "proxy loss" formulation rather than directly applying the score difference as a gradient:

$L_{\text{proxy}} = \|G_\theta(z_t) - \text{stop\_grad}(G_\theta(z_t) + \lambda \Delta_{\text{total}})\|^2$

where $\Delta_{\text{total}} = \Delta_{\text{DM}} + \Delta_{\text{CA}}$ is the combined update direction (either from Equation 6 for original DMD or from Equation 8 for decoupled DMD), $\lambda$ is a proxy loss weight hyperparameter, and $\text{stop\_grad}$ prevents gradient flow through the target term.

What it computes: the L2 distance between the generator's current output $G_\theta(z_t)$ and a target that equals the current output plus a step in the score-difference direction, scaled by $\lambda$. Minimizing this loss encourages the generator to predict an image that is exactly $\lambda \Delta_{\text{total}}$ away from its current prediction — effectively implementing the score-based update as a regression target.

Why this form: directly using the score difference as a gradient (i.e., $\nabla_\theta L = -\Delta_{\text{total}} \frac{\partial G_\theta}{\partial \theta}$) is mathematically equivalent but can be numerically unstable or difficult to tune. The proxy loss formulation converts the gradient signal into a standard regression problem: the generator is trained to predict its own output shifted by the desired update direction. This is analogous to how policy gradient methods in reinforcement learning can be reformulated as supervised learning on advantage-weighted actions, and it provides a stable, well-behaved training signal. The hyperparameter $\lambda$ controls the effective learning rate of the score-based update independently of the optimizer's learning rate.

---

The Fake Model Training Procedure

The fake model $s^{\text{fake}}$ is trained concurrently with the generator using a standard denoising objective on the generator's outputs (Algorithm 1, lines 45-51). At each fake model update step:

1. A new noise level $\tau' \sim \mathcal{U}(0,1)$ is sampled.
2. A new image is generated: $x'_{\text{gen}} = G_\theta(z_t)$ with gradient detached.
3. The image is re-noised: $x'_{\tau'} = \text{renoise}(x'_{\text{gen}}, \tau')$.
4. The fake model is trained to denoise via L2 loss: $L_{\text{denoise}} = \|s^{\text{fake}}(x'_{\tau'}, \tau') - x'_{\text{gen}}\|^2$.

What it computes: the standard diffusion denoising objective — the fake model learns to predict the clean image $x'_{\text{gen}}$ from its noised version $x'_{\tau'}$ at noise level $\tau'$. Since $x'_{\text{gen}}$ comes from the student generator, the fake model is learning to denoise samples from the student's current output distribution $p_{\text{fake}}$.

Why this form: this is the same objective used to train the original diffusion model, but applied on-policy to the student's evolving outputs. The fake model tracks the student's distribution: as the student improves, its outputs change, and the fake model is continuously updated to denoise those new outputs. The gradient is detached from the generator during the fake model update to prevent the fake model's objective from influencing the generator through this path — the generator only receives gradient from the DMD loss (CA + DM terms).

The paper notes that the fake model update can be run multiple times per generator update, following the Two Time-Scale Update Rule (TTUR) commonly used in GAN training, to ensure the fake model stays closely aligned with the rapidly changing generator distribution.

---

The Backward Simulation for Few-Step Generation

For multi-step generation (e.g., 4-step or 8-step distillation), the generator input $z_t$ at timestep $t$ is not pure random noise — it is the output of previous generation steps, produced via a procedure called "backward simulation" (Yin et al., 2024a). The paper references this technique without providing full details (it is established in prior DMD work), but the essential idea is:

1. The few-step generation schedule is defined by a sequence of timesteps $\{t_1, t_2, \ldots, t_N\}$ where $t_1 = 0$ (pure noise) and $t_N = 1$ (clean output).
2. At training time, a step $t_k$ is sampled uniformly from the schedule.
3. To prepare the input $z_{t_k}$, the generator runs the previous steps $\{t_1, \ldots, t_{k-1}\}$ using its current parameters, producing the partially denoised latent that would be the input at step $k$ during inference.
4. This $z_{t_k}$ is then fed to the generator to produce the clean image prediction $G_\theta(z_{t_k})$, which is re-noised and scored.

This ensures that the generator is trained on inputs that match the distribution it will encounter during actual few-step inference, rather than on synthetic inputs that don't reflect accumulated errors from previous steps.

---

Alternative Regularizers: Validating DM as Non-Unique

Section 3.2 addresses a natural follow-up question: if DM is "just" a regularizer, can it be replaced by simpler or different regularization schemes? The answer has implications for both understanding (if DM is replaceable, its role as regularizer is confirmed) and practice (if simpler regularizers work, training could be simplified).

Non-Parametric Mean-Variance Regularization

The paper observes that training with CA alone causes a "monotonic increase in the variance of generated images, finally reaching unreasonably large values" (Figure 3). This motivates the simplest possible regularization: constraining the per-image mean and variance of the generator's output to match target statistics.

$L_{\text{KL}} = \frac{1}{B} \sum_{i=1}^B \frac{1}{2} \left[ \frac{\sigma_i^2 + (\mu_i - \mu_{\text{target}})^2}{\sigma^2_{\text{target}}} - 1 - \log\frac{\sigma_i^2}{\sigma^2_{\text{target}}} \right]$

where $B$ is the batch size, $\mu_i$ and $\sigma_i^2$ are the channel-wise mean and variance of the $i$-th generated image (computed over spatial dimensions), and $\mu_{\text{target}} = 0.075$ and $\sigma^2_{\text{target}} = 0.81$ are the target statistics — the averaged mean and variance of real images sampled from the teacher model.

What it computes: the KL divergence between a Gaussian distribution with the generated image's empirical mean/variance and a target Gaussian distribution with the specified target statistics. This penalizes the generator when its outputs deviate from the target mean (encouraging brightness/contrast to match real images) or have variance that is too high or too low (encouraging the dynamic range to stay in a realistic regime).

Why this form: the KL divergence for Gaussians has a closed form that decomposes into a variance ratio term, a mean-squared-error term, and a log-variance term. This is analytically convenient and provides separate forces for mean matching and variance matching. The target statistics are computed empirically from the teacher's output distribution on real data, making this an extremely lightweight regularizer that requires no learned components — just two scalar statistics.

Result: as shown in Figure 3 and Figure 6, this simple non-parametric regularization "proves remarkably effective at stabilizing the training process, allowing the CA engine to operate durably, keeping the quality indicators at a relatively high level." The training no longer collapses, and the generated images remain reasonable. However, "the final image quality, while stable, falls noticeably short of that achieved with DM," suggesting that "the artifacts induced by the CA engine are more complex than what can be captured by mean and variance alone."

GAN-Based Regularization

The paper also tests a more powerful alternative: using a GAN discriminator as the regularizer, following the approach used in several diffusion distillation works (Yin et al., 2024a for DMD2; Lin et al., 2025 for Diffusion Adversarial Post-Training). The discriminator is initialized from the weights of the pre-trained teacher model (following Sauer et al., 2024b's ADD approach) and trained adversarially against the generator alongside the CA term.

Result: the GAN "can indeed function as a regularizer, effectively controlling image variance and eliminating certain artifacts" (Figure 3). The image quality initially improves and the variance remains controlled. However, "the approach introduces significant challenges in training stability, as the training still collapsed after 4k iterations." The GAN provides a stronger corrective signal than mean-variance constraints (eliminating more complex artifacts) but is substantially less stable than DM.

The sweet spot interpretation: the paper frames these results as revealing a stability-performance Pareto frontier. Simple statistical constraints are the most stable but least powerful; GANs are potentially the most powerful but least stable; DM sits at a sweet spot — "offering a more powerful corrective signal than simple statistical constraints, while being substantially more stable and less complex than GANs."

This connects to the original motivation for score-based distillation. Diff-Instruct (Luo et al., 2023b) argued that "score-matching can be viewed as a more stable alternative to GANs, especially when the distributions have disjoint supports." The paper's results validate this: when distributions have disjoint supports (the student's artifact-laden outputs vs. the teacher's clean outputs), GAN discriminators struggle with gradient vanishing or exploding, while score-based methods (the DM term) provide smooth, well-defined gradients at all noise levels by operating in noised space.

The paper also notes a practical pattern observed in VAE training (Rombach et al., 2021; 2022) and advanced few-step distillation (Lin et al., 2025): "models are often first trained with a stable objective before being fine-tuned with a GAN loss to achieve peak performance." This suggests a training curriculum where CA + DM establishes a strong baseline, and GAN fine-tuning pushes quality further, consistent with DMD2's approach of adding a GAN loss on top of DMD.

---

Mechanistic Analysis of the CA Engine: Re-Noising Schedule Dependence

Section 4.1 investigates how the CA engine drives few-step conversion by studying its dependence on the re-noising timestep $\tau$. The CA term is:

$\Delta_{\text{cfg}}^{\text{real}} = (\alpha - 1)(s_{\text{cond}}^{\text{real}}(x_\tau) - s_{\text{uncond}}^{\text{real}}(x_\tau))$

where $x_\tau = \text{renoise}(G_\theta(z_t), \tau)$ is the student's output corrupted with noise at level $\tau$. Since the score models operate on $x_\tau$, the noise level $\tau$ determines what information is visible to the models and therefore what aspects of the image the CA term can influence.

Experimental design: a single-step generator (no intermediate timesteps, $t=0$, $z_t$ is pure noise) is trained using only the CA term (no DM regularizer for this analysis). The re-noising timestep $\tau$ is sampled from different restricted ranges, starting from the noisiest end and expanding toward cleaner timesteps:

• $\tau \in [0.0, 0.05]$: only very noisy re-noising (image is almost pure noise; only the coarsest structure is visible).
• $\tau \in [0.0, 0.1]$: slightly expanded to include slightly cleaner noise levels.
• $\tau \in [0.0, 0.3]$: includes moderately noisy levels where mid-frequency structure becomes visible.
• $\tau \in [0.0, 1.0]$: the full range, from pure noise to clean data.
• $\tau \in [0.7, 1.0]$: only clean timesteps, where only high-frequency details remain to be determined.

The convention is $\tau = 0$ for pure noise and $\tau = 1$ for clean data (flow matching notation), meaning smaller $\tau$ corresponds to noisier images and larger $\tau$ to cleaner images.

Results (Figure 4a): a clear frequency-progressive pattern emerges:

• $\tau \in [0.0, 0.05]$: The generated images exhibit only low-frequency information — broad color blocks, overall composition, and rough object silhouettes are established, but edges are blurry and textures are absent. The CA engine, applied only at very noisy levels, can only enhance the coarsest aspects of the image.
• As the range expands to include cleaner timesteps: The images progressively gain richer detail. Edges sharpen, textures appear, and fine features emerge. Each expansion of the $\tau$ range toward cleaner values adds higher-frequency content to the generated images.
• $\tau \in [0.7, 1.0]$ (clean only): Training collapses. High-frequency details are meaningless without the low-frequency structure to contextualize them — the model cannot learn to add textures to objects that haven't been defined yet.

Conclusion: "the CA engine, when applied at a specific noise level $\tau$, primarily enhances the image content corresponding to that level." This is because the score models, at noise level $\tau$, are most sensitive to image structure at the frequency scale that survives that amount of noise. At high noise ($\tau \approx 0$), only low frequencies survive; at low noise ($\tau \approx 1$), high frequencies dominate. The CFG correction $(s_{\text{cond}}^{\text{real}} - s_{\text{uncond}}^{\text{real}})$ at a given $\tau$ thus provides gradient information primarily about the frequency band corresponding to that $\tau$.

Implication for multi-step generation: in a few-step generator with steps at timesteps $\{t_1, \ldots, t_N\}$, the input $z_t$ at step $t$ already contains resolved information for all noise levels below $t$ (the previous steps have already established the coarser structure). If CA is applied with $\tau < t$, it would redundantly re-enhance already-determined low-frequency content, potentially over-emphasizing it and creating artifacts. The paper hypothesizes that "an optimal CA schedule should act as a focused engine, concentrating its power on the remaining, unresolved aspects of the image by constraining its re-noising schedule to $\tau > t$."

---

Mechanistic Analysis of the DM Regularizer: The Corrective Mechanism

Section 4.2 investigates how the DM term $\Delta_{\text{real-fake}} = s_{\text{cond}}^{\text{real}} - s_{\text{cond}}^{\text{fake}}$ counteracts CA-induced artifacts through a diagnostic experiment that reveals the mechanism visually.

Experimental design: the generator is trained using only the CA engine (no DM gradient applied to the generator). However, a fake model is still trained concurrently on the generator's outputs — it serves as a "non-interfering observer" whose predictions are recorded but not used to update the generator. This setup allows direct visualization of what the fake model learns and how a hypothetical DM gradient $(s_{\text{cond}}^{\text{real}} - s_{\text{cond}}^{\text{fake}})$ would behave.

Results (Figure 4b): a striking observation. The CA-only generator produces an image with clear high-frequency checkerboard artifacts — a characteristic failure mode of unregularized CA training. When this artifact-laden image is re-noised and fed to both score models:

• The frozen real model ($s_{\text{cond}}^{\text{real}}$) produces a prediction that is clean — the artifact is "conspicuously absent." This is because the real model was trained only on clean, artifact-free teacher outputs and has never learned to represent or reproduce such artifacts.
• The observer fake model ($s_{\text{cond}}^{\text{fake}}$) produces a prediction that retains the artifact — it "persists in the prediction from the observer fake model." This is because the fake model has been trained on the generator's outputs and has learned to replicate the generator's characteristic failure modes.

The corrective mechanism: consider the DM gradient $\Delta_{\text{real-fake}} = s_{\text{cond}}^{\text{real}} - s_{\text{cond}}^{\text{fake}}$ applied to the generator's output. The fake model's prediction contains the artifact; the real model's prediction does not. Their difference is therefore negative in the spatial regions corresponding to the artifact — the fake model predicts a stronger signal there than the real model. Subtracting $s_{\text{cond}}^{\text{fake}}$ (which is high where artifacts are) from $s_{\text{cond}}^{\text{real}}$ (which is low where artifacts are) yields a negative correction that, when applied as a gradient to the generator's output, actively cancels the artifacts. The DM term functions as an adaptive artifact detector: whatever patterns the fake model learns to reproduce (because they are present in the generator's outputs) but the real model does not reproduce (because they were not present in the teacher's training data), the DM gradient will penalize.

Implication for the DM re-noising schedule: the re-noising timestep $\tau$ determines the scope of correction. A large $\tau$ (cleaner image, e.g., $\tau = 0.9$) preserves high-frequency details — the real and fake scores can diverge on subtle, high-frequency artifacts like the checkerboard pattern in Figure 4b. A small $\tau$ (noisier image, e.g., $\tau = 0.1$) destroys most details — the real and fake scores can only diverge on fundamental, low-frequency discrepancies like incorrect composition, color shifts, or over-saturation.

This leads to the paper's hypothesis about optimal DM scheduling in few-step generation: even late-stage generation steps (which primarily add high-frequency details and operate at large $t$) can still suffer from low-frequency artifacts like color over-saturation. These artifacts may be inherited from earlier steps or induced by an imperfect CA schedule. To address these global issues, the DM regularizer needs to "see" them, and it can only do so if it operates at noisy enough levels where low-frequency structure is visible. Therefore:

"DM should function as a comprehensive regularizer, always spanning the full noise range ($\tau_{\text{DM}} \in [0, 1]$), irrespective of the generator's current timestep $t$."

This contrasts with the CA schedule hypothesis (constrain $\tau_{\text{CA}} > t$ for focused enhancement). The two components have fundamentally different optimal behaviors with respect to the re-noising timestep.

---

The Decoupled DMD (d-DMD) Formulation

To enable independent control of the CA and DM re-noising schedules, the paper generalizes the DMD gradient to a $\tau$-decoupled form that assigns separate noise levels $\tau_{\text{CA}}$ and $\tau_{\text{DM}}$ to the two terms:

$\nabla_\theta L_{\text{d-DMD}} = \mathbb{E}\left[-\left(\underbrace{(s_{\text{cond}}^{\text{real}}(x_{\tau_{\text{DM}}}) - s_{\text{cond}}^{\text{fake}}(x_{\tau_{\text{DM}}}))}_{\text{DM term, uses } \tau_{\text{DM}}} + \underbrace{(\alpha - 1)(s_{\text{cond}}^{\text{real}}(x_{\tau_{\text{CA}}}) - s_{\text{uncond}}^{\text{real}}(x_{\tau_{\text{CA}}}))}_{\text{CA term, uses } \tau_{\text{CA}}}\right) \frac{\partial G_\theta(z_t)}{\partial \theta}\right]$

where $x_{\tau_{\text{DM}}} = \text{renoise}(G_\theta(z_t), \tau_{\text{DM}})$ is the student's output re-noised to level $\tau_{\text{DM}}$ for the DM term, and $x_{\tau_{\text{CA}}} = \text{renoise}(G_\theta(z_t), \tau_{\text{CA}})$ is the student's output re-noised to level $\tau_{\text{CA}}$ for the CA term. These are two independent noise levels, sampled from potentially different distributions.

What it computes: the same sum of CA and DM gradient contributions as the original DMD, but with the re-noising performed twice — once at $\tau_{\text{DM}}$ for computing the real-fake score difference, and once at $\tau_{\text{CA}}$ for computing the CFG direction. The student's output $G_\theta(z_t)$ is the same in both cases; only the noise added before score computation differs.

Why this form: the original DMD ties the two terms to the same $\tau$ because the decomposition in Equation 6 shares the same $x_\tau$ between the $\Delta_{\text{real-fake}}$ and $\Delta_{\text{cfg}}^{\text{real}}$ terms. The decoupled formulation recognizes that there is no mathematical requirement for these two noise levels to be equal — the DM term needs $s_{\text{cond}}^{\text{real}}$ and $s_{\text{cond}}^{\text{fake}}$ evaluated at the same $\tau$ (to compare them fairly), and the CA term needs $s_{\text{cond}}^{\text{real}}$ and $s_{\text{uncond}}^{\text{real}}$ evaluated at the same $\tau$ (to compute the CFG direction), but the DM $\tau$ and CA $\tau$ can differ. This requires generating two separate re-noised versions of the student's output, which doubles the re-noising computation but does not change the score model forward passes (each evaluation at its respective $\tau$).

The pseudo-code (Algorithm 1, lines 9-27) shows that in the decoupled schedule mode, two independent noise levels are sampled: $\tau_{\text{CA}} \sim \mathcal{U}(t, 1)$ and $\tau_{\text{DM}} \sim \mathcal{U}(0, 1)$, the student's output is re-noised twice to produce $x_{\tau_{\text{CA}}}$ and $x_{\tau_{\text{DM}}}$, and the DM and CA score differences are computed separately before being summed into $\Delta_{\text{total}}$.

---

Validating the Decoupled Schedule Hypothesis

Section 4.3 empirically tests the two hypotheses derived from the mechanistic analysis:

1. CA should be constrained ($\tau_{\text{CA}} > t$): the engine should focus on unresolved frequencies only, avoiding redundant enhancement of already-established structure.
2. DM should be global ($\tau_{\text{DM}} \in [0, 1]$): the regularizer needs a full-frequency perspective to correct both low-frequency artifacts (like color shifts) and high-frequency artifacts (like checkerboard patterns).

Four schedule configurations are compared for a 4-step generator:

• Configuration ➀: Coupled-Shared. The original DMD approach where $\tau_{\text{CA}} = \tau_{\text{DM}}$, both sampled from $\mathcal{U}(0, 1)$. This is the standard DMD baseline.
• Configuration ➁: Decoupled-Full. The two schedules are independent but both cover the full range: $\tau_{\text{CA}} \sim \mathcal{U}(0, 1)$, $\tau_{\text{DM}} \sim \mathcal{U}(0, 1)$. This tests whether simply decoupling the schedules (allowing them to differ) matters, independent of constraining their ranges.
• Configuration ➂: Decoupled-Constrained. Both schedules are independent and both are constrained: $\tau_{\text{CA}} > t$, $\tau_{\text{DM}} > t$. This tests the "CA should be constrained" hypothesis but applies the same constraint to DM.
• Configuration ➃: Decoupled-Hybrid (proposed). The CA schedule is constrained ($\tau_{\text{CA}} > t$) while the DM schedule is global ($\tau_{\text{DM}} \in [0, 1]$). This is the paper's proposed optimal configuration.

Results on Lumina-Image-2.0 (Table 1, Figures 5):

First, Configuration ➁ (Decoupled-Full) yields "negligible impact compared to the baseline ➀" across DPG-Bench, HPS v2.1, and HPS v3 metrics. The overall DPG-Bench scores are 83.77 vs. 83.90 (essentially identical), and the HPS scores are similarly close. This confirms that the benefit of decoupling "comes from the schedule's range, not just its independence."

Second, both configurations with constrained schedules (➂ and ➃) "significantly outperform the baselines across multiple benchmarks." Configuration ➂ (both constrained) achieves HPS v2.1 Overall of 31.71 (vs. 30.61 for baseline ➀) and HPS v3 Overall of 11.08 (vs. 10.34). Configuration ➃ (hybrid) achieves the best scores: HPS v2.1 Overall of 32.29 and HPS v3 Overall of 11.59.

On the more fine-grained DPG-Bench metrics, Configuration ➃ shows consistent improvements over baseline ➀: Global score 91.40 vs. 80.22, Entity 91.62 vs. 90.45, Attribute 91.18 vs. 90.47, Relation 91.93 vs. 89.36, and Other 91.98 vs. 91.36. The largest gains are in Global (overall image quality) and Relation (spatial/structural relationships between objects), suggesting the hybrid schedule particularly helps with compositional coherence.

Third, the qualitative results in Figure 5 provide visual confirmation of the hypothesized mechanisms:

• Configuration ➁ (top row): produces reasonable images but with somewhat muted details and a slightly soft appearance, consistent with the CA engine being applied at redundant noise levels that don't focus enhancement.
• Configuration ➂ (middle row): produces images with "richer, finer details" compared to ➁ — the constrained CA schedule successfully focuses the engine on unresolved high frequencies. However, this configuration "still suffers from color oversaturation, a low-frequency artifact that its constrained DM regularizer fails to correct." The images appear overly vivid and slightly unnatural in their color palette.
• Configuration ➃ (bottom row): "retains these enhanced details while effectively mitigating the saturation artifacts, yielding the most visually appealing and natural-looking images." The global DM schedule catches the low-frequency color artifacts that the constrained DM in ➂ misses, while the constrained CA schedule still focuses enhancement where it's needed.

User study validation (Appendix C): a comprehensive user study with 15 professional annotators strongly confirms the quantitative and qualitative findings. In a model-level side-by-side comparison between Configuration ➃ and each of the other three configurations, ➃ achieved "a unanimous 100% preference rate in all comparisons." Annotators justified their preference by citing ➃'s ability to generate "richer details, produce a more realistic/less over-saturated/not greasy texture/coloring, and exhibit fewer anatomical or structural deformities." In a per-image forced-ranking evaluation across 500 prompts with 10 annotators, ➃ was ranked first in 59.8% of cases, far surpassing ➂ at 33.8% and ➁ at 6.4%.

Results on SDXL (Table 2): to situate the decoupled schedule within the broader distillation landscape, the paper applies the Decoupled-Hybrid schedule to DMD2 (Yin et al., 2024a) on 4-step SDXL distillation. For rigorous comparison, the paper adopts DMD2's exact training configuration — including its GAN loss and all other hyperparameters — and only replaces the re-noising schedule. Evaluated on 10k COCO2014-val prompts:

The Decoupled schedule improves FID (lower is better) from 18.95 to 17.80, CLIP-S from 33.14 to 33.62, and ImageReward substantially from 71.01 to 78.61 (+7.6 points). HPS v2.1 is essentially unchanged (30.64 vs. 30.34, within typical variance), and HPS v3 improves from 9.64 to 9.79. The large ImageReward improvement is notable because ImageReward is trained to predict human preferences, suggesting the decoupled schedule produces images that humans find more appealing — consistent with the user study results.

---

Summary of Key Design Choices and Their Justifications

• Algebraic decomposition of the DMD gradient rather than designing a new loss: reveals that the existing method already contains two functionally independent components, enabling analysis without changing the method.
• Ablation by removing terms (CA-only, DM-only) rather than adding them: isolates causal contributions rather than observing correlations, establishing CA as engine and DM as regularizer through controlled absence.
• Mean-variance regularization as simplest possible stabilizer: demonstrates that CA's instability is primarily a variance explosion problem, and that any constraint controlling variance can substitute for DM — proving DM is non-unique.
• Fake model as non-interfering observer in the diagnostic experiment (Figure 4b): reveals what the fake model learns without using it for updates, directly visualizing the DM corrective mechanism as artifact cancellation.
• Independent $\tau$ schedules with constrained ranges for CA and DM: exploits the frequency-progressive nature of diffusion models (different noise levels correspond to different frequency bands) to focus the engine and generalize the regularizer.
• Decoupled-Hybrid schedule ($\tau_{\text{CA}} > t$, $\tau_{\text{DM}} \in [0,1]$): the engine focuses on unresolved frequencies to avoid redundant over-enhancement; the regularizer spans all frequencies to catch both low-frequency global artifacts and high-frequency local artifacts.
• DMD2 configuration for SDXL comparison rather than vanilla DMD: DMD2 represents the state-of-the-art in score-based distillation, making the improvement more meaningful and demonstrating the decoupled schedule's compatibility with other techniques (GAN loss, etc.).

4. Key Insights and Innovations

Innovation 1: DMD's Success Is a Functional Decoupling, Not Unified Distribution Matching

The paper's most fundamental contribution is a reclassification of what DMD actually does. Prior to this work, the score-based distillation literature (Diff-Instruct, DMD, DMD2, SiD, ASID) uniformly treated DMD as a method for minimizing a statistical divergence between the student's and teacher's distributions, with the CFG substitution viewed as a heuristic relaxation that produces a better target distribution. The paper demonstrates that this interpretation is not merely incomplete — it misattributes the primary mechanism entirely.

The intellectual move is a simple algebraic decomposition (Equation 6) that reveals the practical DMD gradient contains two terms with fundamentally different mathematical origins: one that matches the theoretical IKL divergence derivation ($\Delta_{\text{real-fake}}$) and one that was never present in that derivation ($\Delta_{\text{cfg}}^{\text{real}}$). The second term depends only on the frozen teacher model and involves the CFG direction $(s_{\text{cond}}^{\text{real}} - s_{\text{uncond}}^{\text{real}})$ applied directly to the student's output — no fake model, no distribution comparison, no divergence minimization. This is not a relaxation of distribution matching; it is a categorically different mechanism.

What makes this distinctive is that the paper doesn't propose a new method to achieve this insight — it reveals that the existing method, as actually practiced, already works through this decoupled mechanism. The CFG substitution that practitioners had been making for years, and that theorists had been rationalizing as a higher-quality target score, turns out to be the entire engine of few-step conversion. The distribution matching component, which the entire theoretical framework was built around, serves as a regularizer. This is a fundamental reframing, not an incremental improvement: it changes what researchers should optimize, what they should blame when things fail, and what questions they should ask next.

The evidence is stark. Figure 2 shows that training with CA alone achieves few-step conversion quality comparable to full DMD, while DM alone is substantially worse. The paper explicitly states: "the conversion to a few-step generator is not primarily an act of matching distributions but rather a direct consequence of 'baking' the CFG pattern into the student generator's predictions... which is irrelevant to the fake model" (Section 3.1.1). The fake model — which prior work treated as essential for estimating the student's distribution — is revealed as unnecessary for the core distillation task.

This insight reconciles the persistent tension in the literature between theory and practice. Why does DMD fail with $\alpha = 1$ (Equation 2) despite the theory saying it should work? Because distribution matching alone is a weak distillation signal on complex tasks. Why does it work dramatically better with $\alpha > 1$? Not because CFG provides a better target distribution, but because it introduces an entirely new gradient term that directly teaches the student to internalize the teacher's guidance behavior. The paper's decomposition makes this explicit for the first time.

Innovation 2: The Regularizer Is Replaceable — DM Is Not Unique

The paper's second conceptual contribution is demonstrating that the DM regularizer, while highly effective, can be functionally substituted by mechanisms that bear no resemblance to distribution matching. This is a diagnostic move that validates the engine-regularizer decomposition: if DM were truly the distillation engine (as prior work assumed), replacing it with something simpler should cause performance to collapse. Instead, the paper shows that even a non-parametric mean-variance constraint (Equation 7) can stabilize the CA engine and produce reasonable results (Figure 3, Figure 6).

The significance extends beyond validating a hypothesis. It establishes a stability-performance continuum for regularizers in few-step distillation. Mean-variance constraints are trivially stable but produce lower-quality images because they cannot capture the complex, spatially-varying artifacts that the CA engine generates. GAN discriminators can capture richer failure modes but introduce training instability — the training collapses after approximately 4k iterations (Figure 3). DM sits at a sweet spot, providing strong corrective signals against artifacts like checkerboard patterns and over-saturation while remaining stable over extended training. This continuum provides a principled framework for choosing and combining regularizers: start with DM for stable training, optionally fine-tune with GAN for peak quality — a pattern already observed in VAE training (Rombach et al., 2021; 2022) and advanced few-step distillation (Lin et al., 2025).

The diagnostic experiment in Figure 4b provides the mechanistic explanation for why DM is such an effective regularizer: the fake model learns to replicate the generator's characteristic artifacts, making $s_{\text{cond}}^{\text{fake}}$ contain those artifacts while $s_{\text{cond}}^{\text{real}}$ does not. The DM gradient $\Delta_{\text{real-fake}}$ therefore becomes an adaptive artifact cancellation signal — it automatically penalizes whatever failure modes the generator is currently exhibiting, without requiring manual specification of what those failure modes are. This explains DM's advantage over mean-variance constraints (which can only penalize global statistical deviations) and its stability advantage over GANs (whose discriminators can overfit to spurious features when distributions have disjoint supports, as argued in Diff-Instruct, Luo et al., 2023b).

This is not an incremental improvement — it is a reclassification of the DM term's role from "primary optimization objective" to "adaptive regularizer," with direct practical consequences. Practitioners who understand this should invest effort in improving the fake model's ability to track the generator's artifacts (to strengthen the corrective signal) rather than in designing better distribution divergences (which would have been the priority under the traditional interpretation).

Innovation 3: The CA Engine Operates in Frequency-Banded, Schedule-Controlled Fashion

The paper's mechanistically richest contribution is the discovery that the CA engine's effect is frequency-selective and controlled by the re-noising timestep $\tau$. Section 4.1 demonstrates that when CA is applied only at noisy $\tau$ ranges (e.g., $\tau \in [0.0, 0.05]$), the engine enhances only low-frequency content — broad color regions, overall composition, rough object silhouettes. As the $\tau$ range expands toward cleaner values, the engine progressively adds higher-frequency details. When applied only at clean $\tau$ ($\tau \in [0.7, 1.0]$), training collapses because high-frequency details cannot be meaningfully learned without the low-frequency structure they depend on.

This finding is significant because it provides a control knob for what the CA engine does. Prior work treated the re-noising timestep as a random variable to be integrated over (the IKL divergence integrates $\tau$ from 0 to 1) — a mathematical convenience that obscures its functional role. The paper reveals that $\tau$ is actually a frequency-band selector: different $\tau$ values give the CA engine gradient information about different spatial frequency ranges of the image. This insight is not stated in prior score-based distillation work, which uniformly samples $\tau$ from $\mathcal{U}(0, 1)$ without analyzing its per-value effect.

The practical consequence is the constrained CA schedule ($\tau_{\text{CA}} > t$) for multi-step generation. In a few-step generator, earlier steps (smaller $t$) establish low-frequency structure, and later steps (larger $t$) add high-frequency details. If CA is applied at $\tau < t$, it redundantly re-enhances structure that previous steps have already resolved, causing over-emphasis and artifacts. By constraining $\tau_{\text{CA}} > t$, the engine focuses exclusively on the frequencies that remain to be determined at the current step. This is a principled design choice derived from mechanistic understanding rather than empirical search — the paper doesn't sweep over schedule ranges and pick the best one; it deduces the optimal constraint from the frequency-progressive nature of the CA engine and validates it experimentally (Table 1, Configurations ➂ and ➃ consistently outperform ➀ and ➁).

The conceptual parallel to spectral analysis of diffusion models is explicit: the paper cites Dieleman (2024) ("Diffusion is spectral autoregression") to support the interpretation that diffusion models generate images coarse-to-fine in frequency space. The CA engine, by applying CFG corrections at specific noise levels, selectively enhances the corresponding frequency bands. This reframes distillation not as a distribution-matching problem but as a frequency-targeted knowledge transfer problem, where the goal is to teach the student which frequency-specific decisions the teacher's CFG mechanism would make at each stage of generation.

Innovation 4: The Asymmetric CFG Application Is Not a Bug — It's the Feature

A subtle but profound insight emerges from the decomposition: the asymmetry in CFG application — guided real score, unguided fake score — which prior work treated as a theoretical inconsistency, is actually what makes DMD effective. If both real and fake scores used the same CFG scale, the $\Delta_{\text{cfg}}^{\text{real}}$ term would be canceled by a corresponding $\Delta_{\text{cfg}}^{\text{fake}}$ term, and the objective would reduce to pure distribution matching (with a CFG-sharpened target, but without the independent CA engine). The paper doesn't explicitly state this counterfactual, but it follows directly from Equation 6: the CA term exists only because the fake model's score is not guided.

This reframes the CFG asymmetry from a "dark cloud" over the theoretical interpretation (as the paper calls it in Section 1) into the intentional design principle. The fake model should not use CFG because its role is to track the student's actual (unguided) output distribution, capturing whatever artifacts the student produces. The real model should use CFG because its role is to provide the guidance direction that the student needs to internalize. If both used CFG, the CA term would vanish, and we would be back to pure distribution matching — which the paper shows is insufficient for few-step conversion on complex tasks.

This insight has implications for how we think about the fake model. In the traditional distribution-matching view, the fake model is an auxiliary estimator of $p_{\text{fake}}$'s score, and its quality directly affects the fidelity of distribution matching. In the decoupled view, the fake model's primary job is to learn the generator's failure modes so that the DM gradient can cancel them (as visualized in Figure 4b). Its quality matters not for distribution-matching accuracy but for artifact-detection sensitivity. This suggests different training strategies for the fake model — for instance, training it more aggressively on recent generator outputs where artifacts are most pronounced, rather than maintaining a balanced replay buffer.

The paper doesn't fully develop this implication, but it's a natural consequence of the decomposition and represents a conceptual advance over prior work that treated the fake model as a generic score estimator. Luo (2024) (Diff-Instruct++) was the closest prior work to recognizing the CFG term's independent significance, modeling it as a post-hoc reward function. The current paper goes further by showing that CFG is not an add-on reward but the primary training signal, and that its asymmetric application is not a limitation to be corrected but the mechanism that makes the method work.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The primary evaluation uses prompts from the COCO dataset (Lin et al., 2014). For the ablation studies in Figures 2 and 3, 1,000 prompts are randomly sampled from the COCO-10k validation split. For the SDXL distillation comparison in Table 2, 10,000 prompts from the COCO2014 validation set are used. The Lumina-Image-2.0 experiments in Table 1 use the DPG-Bench benchmark (Hu et al., 2024) for fine-grained text-image alignment (evaluating Global, Entity, Attribute, Relation, and Other scores, plus an Overall metric), HPS v2.1 (Wu et al., 2023), and HPS v3 (Ma et al., 2025). The HPS benchmarks are further broken down by image category (Concept-Art, Photo, Anime, Paintings for HPS v2.1; Animals, Architecture, Arts, Characters, Design, Food, Natural Scenes, Others, Plants, Products, Science, Transportation for HPS v3) in Tables 6 and 7.

• Base model(s). Two base model families are used. The ablation studies on the CA/DM decomposition (Figures 2, 3, 6) and the SDXL comparison (Table 2) use Stable Diffusion XL (SDXL). The decoupled schedule validation (Table 1, Figures 5) uses Lumina-Image-2.0 (Qin et al., 2025), a more recent unified image generation framework. In both cases, the pre-trained multi-step diffusion model serves as the teacher, and the student generator is initialized from the teacher's weights. The SDXL teacher model operates with a standard CFG scale that produces high-quality images; the exact guidance scale is not specified in the paper but follows standard DMD practice for SDXL (typically $\alpha$ values in the range of 5–7.5 for SDXL inference). The Lumina-Image-2.0 teacher model is a 50-step model listed as "Original (50 steps)" in Table 1.

• Metrics. The paper uses several complementary metrics designed to capture different aspects of image quality and text alignment. ImageReward (Xu et al., 2023) is a learned metric trained to predict human preferences for text-to-image generation, with higher scores indicating greater preference alignment. HPS v2.1 (Wu et al., 2023) similarly measures human preference alignment, reported both as overall scores and broken down by image category (Concept-Art, Photo, Anime, Paintings). HPS v3 (Ma et al., 2025) provides a newer, wider-spectrum human preference score across a broader range of image categories. DPG-Bench (Hu et al., 2024) evaluates text-image alignment along five fine-grained dimensions: Global (overall image quality and coherence), Entity (object presence and identity), Attribute (object properties such as color and material), Relation (spatial and structural relationships between objects), and Other (miscellaneous alignment aspects), plus an Overall aggregate score. For the SDXL comparison in Table 2, additional standard metrics are used: FID (Fréchet Inception Distance, lower is better) for distribution-level image quality, and CLIP-S (CLIP Score) for text-image semantic alignment. All metrics except FID are designed such that higher scores indicate better quality or alignment.

• Baselines. The paper's ablation studies are self-contained, comparing configurations of the DMD training objective against each other rather than against external distillation methods. The primary internal baselines are: (1) Full DMD (CA + DM), the standard practical DMD objective using both the CA and DM terms with shared re-noising schedule $\tau \sim \mathcal{U}(0,1)$, serving as the performance upper bound; (2) DM-only ($\Delta_{\text{real-fake}}$ only, equivalent to setting $\alpha = 1$ or using the theoretical Equation 2 directly), which prior work would have considered the "correct" distribution matching objective; (3) CA-only ($\Delta_{\text{cfg}}^{\text{real}}$ only), which tests whether the CFG augmentation term alone can drive distillation; (4) CA + Mean-Var Regression, which replaces the DM regularizer with a non-parametric per-image mean and variance constraint (Equation 7); and (5) CA + GAN, which replaces DM with an adversarial discriminator initialized from the teacher model. For the SDXL comparison in Table 2, external baselines include LCM (Luo et al., 2023a), SDXL-Turbo (Sauer et al., 2024b), SDXL-Lightning (Lin et al., 2024), Flash Diffusion (Chadebec et al., 2025), PCM (Wang et al., 2024), and DMD2 (Yin et al., 2024a). The DMD2 comparison is the most important, as the paper's "Decoupled" method adopts DMD2's exact training configuration (including its GAN loss and all hyperparameters) and changes only the re-noising schedule.

• Generation budget / compute accounting. The paper measures training progress in terms of training iterations (steps), not total FLOPs or wall-clock time. In the CA/DM ablation study (Figure 2), results are reported at specific iteration counts: 200, 600, 1000, 1400, and 1800 steps for 1-step SDXL distillation; 200, 1000, 2000, 4000, and 6000 steps for 4-step SDXL distillation. In the alternative regularizer experiments (Figure 3), results are tracked from 600 through 8000 steps. For the Lumina-Image-2.0 experiments (Table 1) and SDXL comparison (Table 2), final converged model checkpoints are evaluated without specifying the exact training duration. The decoupled schedule (d-DMD) uses two re-noising operations per generator update instead of one (re-noising to $\tau_{\text{CA}}$ and separately to $\tau_{\text{DM}}$), which approximately doubles the re-noising computation but does not change the number of score model forward passes (each score model is evaluated once at each noise level). The paper does not explicitly account for this additional cost in the comparisons, effectively assuming that the re-noising step is computationally negligible compared to the score model evaluations. All experiments keep the CFG scale $\alpha$, the proxy loss weight $\lambda$, and other hyperparameters consistent across compared configurations within each experiment, changing only the terms being investigated.

• Cross-validation / statistical protocol. The paper does not employ formal cross-validation for the main experimental results. The evaluations use fixed sets of prompts (1,000 or 10,000 from COCO, the full DPG-Bench, and the standard HPS benchmark splits) rather than multiple random train/test splits. For the decoupled schedule experiments, a comprehensive user study (Appendix C) provides statistical validation through human judgment: 15 professional annotators conduct model-level side-by-side comparisons across 200 image pairs per model pair (three comparisons total), and 10 annotators perform per-image forced ranking across 500 prompt triplets. The user study results are reported with preference percentages and average rankings. No confidence intervals or statistical significance tests are reported for the automated metric results.

Main Quantitative Results

CA/DM Ablation: Establishing the Engine-Regularizer Division

The headline result from Figure 2 is that CFG Augmentation alone matches full DMD performance initially but eventually collapses, while Distribution Matching alone substantially underperforms. In the 4-step SDXL setting, the CA-only configuration produces images qualitatively similar to full DMD in terms of content and composition across training iterations 200 through approximately 2000. The ImageReward and HPS v2.1 curves for CA-only track the CA+DM curves closely during this period, with the paper noting that the results "demonstrate high similarity in content to the full DMD objective, indicating the dominant role of the CA term in DMD loss." The DM-only configuration, in contrast, produces images with a "significant performance gap... towards the CA setting, as indicated by both image visualizations and numerical indicators." At 4-step training iteration 6000, the full DMD (CA+DM) achieves the highest ImageReward and HPS v2.1 scores, CA-only has collapsed (producing images with over-saturation, high-frequency noise, and eventual training divergence), and DM-only remains substantially below both.

In the 1-step SDXL setting (Figure 2a), the pattern is similar but more compressed. CA-only matches full DMD quality at early iterations (200–600) but degrades rapidly by iteration 1400–1800. The DM-only configuration produces noticeably worse images across all iterations, with lower ImageReward and HPS v2.1 scores than either CA-involving configuration. The paper explicitly qualifies the DM-only results: "even though it is improper to conclude that the DM term is totally incapable of doing the multi-step to few-step conversion (since in the 4-step experiment it indeed makes relatively reasonable images), a significant performance gap exists towards the CA setting." This qualification is important — it acknowledges that distribution matching does provide some distillation signal, but that its contribution is secondary compared to CA.

The paper extracts two fundamental conclusions from these results (Section 3.1.1): (1) "CFG Augmentation is the engine for few-step conversion. The ability of the distilled generator to produce high-quality samples in a few steps is almost entirely attributable to the $\Delta_{\text{cfg}}$ term," and (2) "Distribution Matching is a regularizer for training stability. The $\Delta_{\text{real-fake}}$ term, while not the primary driver of distillation, plays a crucial role as a regularizer that prevents the training process from diverging and ensures the quality of the final output."

A critical detail in interpreting Figure 2: the paper notes the CA-only training "progressively suffer[s] from artifacts such as over-saturation and high-frequency noise, eventually leading to training collapse." This collapse is visible in the 1-step setting as a sharp quality degradation after approximately 1400 steps, and in the 4-step setting as increasing artifact severity that ultimately causes divergence. The DM term's role is thus not merely to marginally improve quality — it is to prevent catastrophic failure. Without DM, the CA engine eventually destroys its own progress.

Alternative Regularizers: DM Is Replaceable but Optimal

The results in Figure 3 quantify the stability-performance tradeoff across different regularization schemes when combined with the CA engine. All experiments use 4-step SDXL distillation.

Mean-Variance Regression (CA + Mean-Var): This simplest possible regularizer "proves remarkably effective at stabilizing the training process, allowing the CA engine to operate durably." The training does not collapse, and the generated images maintain reasonable quality throughout extended training (tracked to 8000 steps in Figure 6). The ImageReward and HPS v2.1 scores remain "at a relatively high level" — the paper does not provide exact numerical values in the main text but the Figure 3 plots show CA + Mean-Var maintaining stable but lower scores than CA + DM. The per-image mean and standard deviation of generated images are effectively controlled by the KL regularization (Equation 7), staying close to the target statistics ($\mu_{\text{target}} = 0.075$, $\sigma^2_{\text{target}} = 0.81$) rather than diverging as in the unregularized CA case. However, "the final image quality, while stable, falls noticeably short of that achieved with DM," indicating that CA-induced artifacts involve more complex, spatially-varying patterns than can be captured by global mean and variance constraints alone.

No Regression (CA-only): The unregularized CA engine shows a "monotonic increase in the variance of generated images, finally reaching unreasonably large values" (visible in the Standard Deviation plot of Figure 3). This variance explosion correlates with the increasing artifact severity observed in Figure 2 and ultimately leads to training collapse. The mean of generated images also drifts, though less dramatically than the variance. This configuration serves as the lower bound, confirming that some form of regularization is necessary for the CA engine to function beyond initial training.

GAN-Based Regularization (CA + GAN): The GAN discriminator "can indeed function as a regularizer, effectively controlling image variance and eliminating certain artifacts." The standard deviation of generated images remains controlled (Figure 3), and the ImageReward and HPS v2.1 scores initially reach levels competitive with or exceeding CA + DM. However, the approach "introduces significant challenges in training stability, as the training still collapsed after 4k iterations." The collapse point is visible in Figure 3 where the GAN curves show a sharp degradation after approximately 4000 training steps. This instability is consistent with known GAN training challenges, particularly when the generator and data distributions have disjoint supports — a condition identified by Diff-Instruct (Luo et al., 2023b) as being particularly problematic for GANs and motivating score-based methods as a more stable alternative.

CA + DM (Full DMD): The DM regularizer achieves the best balance: stable training throughout all 8000 iterations (no collapse), effective artifact suppression (the Standard Deviation plot in Figure 3 shows controlled variance), and the highest final ImageReward and HPS v2.1 scores among all regularized configurations. The paper characterizes DM as "a sweet spot—offering a more powerful corrective signal than simple statistical constraints, while being substantially more stable and less complex than GANs."

The qualitative samples in Figure 6 provide visual corroboration. At early iterations (600 steps), all configurations produce broadly similar images. By 4000 steps, the unregularized CA-only images show clear degradation (visible artifacts), while the regularized variants remain reasonable. At 8000 steps, the CA + DM images are clearly superior in visual quality to both CA + Mean-Var (which produces somewhat muted or artifact-ridden images) and CA + GAN (which has collapsed). The CA + GAN samples at 4000 steps show promising quality before the collapse, suggesting a potential training curriculum: "models are often first trained with a stable objective before being fine-tuned with a GAN loss to achieve peak performance" (Section 3.2), a pattern observed in VAE and advanced few-step distillation work.

CA Engine Frequency Analysis: Re-Noising Schedule Determines Enhanced Content

The experiment in Figure 4a reveals that the CA engine's effect is frequency-selective and controlled by the re-noising timestep $\tau$. Using single-step generators trained with CA alone and with $\tau$ sampled from restricted ranges:

• $\tau \in [0.0, 0.05]$: Generates images with only low-frequency information — "broad color blocks and overall composition." Edges are blurry, textures are absent, and only the coarsest structure is determined.
• $\tau \in [0.0, 0.1]$: Images gain slightly more structure but remain dominated by low frequencies.
• $\tau \in [0.0, 0.3]$: Mid-frequency content appears — object boundaries become clearer, and some texture begins to emerge.
• $\tau \in [0.0, 1.0]$ (full range): Images contain the full spectrum of frequencies, with sharp edges and fine textures — the richest detail of any configuration.
• $\tau \in [0.7, 1.0]$ (clean only): Training collapses — "high-frequency details are meaningless if low-frequency general structure has not yet been determined."

The paper draws the conclusion that "the CA engine, when applied at a specific noise level $\tau$, primarily enhances the image content corresponding to that level." This frequency-progressive behavior is explained by the properties of diffusion models: at high noise levels (small $\tau$), only low-frequency information survives the noise corruption and is therefore visible to the score models; at low noise levels (large $\tau$), high-frequency details are preserved and can be scored. The CFG correction $(s_{\text{cond}}^{\text{real}} - s_{\text{uncond}}^{\text{real}})$ at a given $\tau$ thus provides gradient information primarily about the frequency band that survives noising to that level.

The paper explicitly connects this to the spectral autoregression view of diffusion models (citing Dieleman, 2024), framing the CA engine as a mechanism that applies CFG corrections in a frequency-targeted manner. This insight directly motivates the constrained CA schedule for multi-step generation ($\tau_{\text{CA}} > t$): "an optimal CA schedule should act as a focused engine, concentrating its power on the remaining, unresolved aspects of the image."

DM Corrective Mechanism: The Fake Model Learns Generator Artifacts

Figure 4b provides the visual evidence for the DM regularizer's corrective mechanism. The experiment trains a generator with CA only while maintaining a fake model as a "non-interfering observer" that is trained on the generator's outputs but does not contribute gradients to the generator update. The key observation: the CA-only generator produces images with "clear high-frequency checkerboard artifacts." When this artifact-laden image is re-noised and fed to both score models, "the artifact is conspicuously absent in the prediction from the frozen real model, yet it persists in the prediction from the observer fake model."

This asymmetry — the real model being blind to artifacts it was never trained on, while the fake model reproduces them because it tracks the generator's output — explains how the DM gradient $\Delta_{\text{real-fake}} = s_{\text{cond}}^{\text{real}} - s_{\text{cond}}^{\text{fake}}$ functions as an adaptive artifact detector. In spatial regions where artifacts exist, $s_{\text{cond}}^{\text{fake}}$ produces a stronger signal than $s_{\text{cond}}^{\text{real}}$, making the difference negative. Applying this negative gradient to the generator's output "would thus encourage a change that actively cancels out these artifacts." The DM term does not need to be explicitly told what artifacts to look for — whatever patterns the fake model learns to reproduce (because they are present in the generator's outputs) that the real model does not reproduce will automatically generate a corrective gradient.

The paper extends this analysis to the DM re-noising schedule: the noise level $\tau$ at which the DM term is applied "controls the scope of correction." At large $\tau$ (cleaner images), the real and fake scores "diverge primarily on subtle, high-frequency details" like the checkerboard pattern in Figure 4b. At small $\tau$ (noisier images), most details are destroyed, and the scores "diverge on more fundamental, low-frequency elements like composition and color." This explains why a global DM schedule ($\tau_{\text{DM}} \in [0, 1]$) is necessary: it allows the regularizer to correct both low-frequency global artifacts (like color over-saturation, which requires noisy $\tau$ to detect) and high-frequency local artifacts (which require clean $\tau$ to detect).

Decoupled Schedule Validation: Constrained CA + Global DM Is Optimal

Table 1 presents the core quantitative results for the decoupled schedule hypothesis on Lumina-Image-2.0 with a 4-step generator. The 50-step teacher model ("Original") is included as an upper reference but is not the baseline — the baseline is Configuration ➀ (Coupled-Shared, which is standard DMD).

Headline numbers (Table 1, Overall columns):

The comparison between ➀ and ➁ confirms that "merely decoupling the schedules while keeping them global yields negligible impact" — DPG-Bench 83.77 vs. 83.90, HPS v2.1 30.69 vs. 30.61, HPS v3 10.32 vs. 10.34 are all within typical evaluation variance. This demonstrates that the benefit of the decoupled approach comes from the schedule ranges, not from the independence per se.

Both constrained configurations (➂ and ➃) significantly outperform the baselines. Configuration ➂ (both CA and DM constrained to $\tau > t$) improves HPS v2.1 Overall from 30.61 to 31.71 (+1.10) and HPS v3 Overall from 10.34 to 11.08 (+0.74). Configuration ➃ (CA constrained, DM global) achieves the best scores across all three overall metrics: HPS v2.1 Overall 32.29 (+1.68 over baseline ➀), HPS v3 Overall 11.59 (+1.25 over baseline ➀), and DPG-Bench Overall 85.85 (+1.95 over baseline ➀).

Fine-grained DPG-Bench results (Table 1): Configuration ➃ shows substantial improvements over the DMD baseline ➀ on several dimensions. The Global score improves from 80.22 to 91.40 (+11.18), Entity from 90.45 to 91.62 (+1.17), Attribute from 90.47 to 91.18 (+0.71), Relation from 89.36 to 91.93 (+2.57), and Other from 91.36 to 91.98 (+0.62). The large Global improvement suggests the hybrid schedule particularly enhances overall image quality and coherence, while the Relation improvement indicates better handling of spatial and structural relationships between objects. Notably, Configuration ➂ (both constrained) achieves even higher scores on some dimensions (Relation 90.71 vs. ➀'s 89.36, Other 90.71 vs. ➀'s 91.36 — actually lower on Other), but ➃ consistently achieves the best or near-best scores on all dimensions, highlighting the value of the global DM schedule for comprehensive artifact correction.

DPI-Bench category breakdowns (Tables 6 and 7): The HPS v2.1 category breakdown (Table 6) shows Configuration ➃ achieving the highest scores in all four categories: Concept-Art 32.37 (vs. 30.31 for baseline ➀), Photo 30.87 (vs. 29.95), Anime 33.61 (vs. 31.72), and Paintings 32.31 (vs. 30.45). The largest relative gains are in Paintings (+1.86) and Concept-Art (+2.06), categories that may particularly benefit from the artifact suppression provided by the global DM schedule. Configuration ➂ achieves the second-best scores in most categories but is consistently below ➃.

The HPS v3 category breakdown (Table 7) shows a consistent pattern across all 12 categories: Configuration ➃ achieves the highest score in every single category, with particularly large improvements in Architecture (12.899 vs. 11.537 for baseline ➀, +1.362), Products (11.841 vs. 10.994, +0.847), and Natural Scenes (11.592 vs. 10.138, +1.454). The uniformity of the improvement across diverse image categories — from Animals and Food to Science and Transportation — suggests the decoupled schedule provides a general-purpose quality improvement rather than benefiting only specific image types.

Qualitative results (Figure 5): The un-cherry-picked samples provide visual evidence for the quantitative results. Configuration ➁ (Decoupled-Full, top row) produces reasonable images with adequate detail and color but a somewhat soft or muted appearance. Configuration ➂ (Decoupled-Constrained, middle row) shows "richer, finer details" — visible in sharper textures, clearer edges, and more intricate patterns — confirming the benefit of the constrained CA schedule. However, this configuration "still suffers from color oversaturation, a low-frequency artifact that its constrained DM regularizer fails to correct." The images appear overly vivid, with exaggerated color intensity that looks unnatural. Configuration ➃ (Decoupled-Hybrid, bottom row) "retains these enhanced details while effectively mitigating the saturation artifacts, yielding the most visually appealing and natural-looking images." The colors are vivid but realistic, details are sharp without being harsh, and the overall visual quality is clearly superior.

SDXL comparison (Table 2): When applied to DMD2 on 4-step SDXL distillation, the Decoupled-Hybrid schedule improves over the DMD2 baseline across most metrics. FID decreases from 18.95 to 17.80 (lower is better, -1.15 improvement), CLIP-S increases from 33.14 to 33.62 (+0.48), ImageReward increases substantially from 71.01 to 78.61 (+7.60), HPS v2.1 is essentially unchanged (30.64 vs. 30.34, -0.30), and HPS v3 increases from 9.64 to 9.79 (+0.15). The large ImageReward improvement (+7.60 points) is particularly notable because ImageReward is specifically trained to predict human preferences, suggesting the decoupled schedule produces images that humans find substantially more appealing. This aligns with the user study results in Appendix C. Compared to other distillation methods in Table 2, the Decoupled configuration achieves the best FID (17.80, vs. 18.95 for the next-best DMD2), the best CLIP-S (33.62, vs. 33.14 for DMD2), and the best ImageReward (78.61, vs. 71.01 for DMD2). It does not achieve the best HPS v2.1 (30.34 vs. 30.76 for PCM) or HPS v3 (9.79, tied with DMD2 at 9.64–9.79 range).

User Study Results: Strong Human Preference for Decoupled-Hybrid

The user study in Appendix C provides decisive human judgment validation. In the per-model side-by-side comparison (Table 5), 15 annotators compared Configuration ➃ against each of the other three configurations using 200 image pairs per comparison. The result: Configuration ➃ achieved "a unanimous 100% preference rate in all comparisons." Every single annotator preferred ➃ over ➀ (Coupled-Shared), over ➁ (Decoupled-Full), and over ➂ (Decoupled-Constrained). The annotators' justifications consistently cited ➃'s ability to generate "richer details, produce a more realistic/less over-saturated/not greasy texture/coloring, and exhibit fewer anatomical or structural deformities." These qualitative observations align precisely with the paper's mechanistic hypotheses: richer details come from the focused CA engine (constrained schedule), reduced over-saturation comes from the global DM regularizer catching low-frequency color artifacts, and fewer structural deformities come from the combined effect of both improvements.

In the per-image ranking evaluation (Tables 3 and 4), 10 annotators performed forced ranking of three images per prompt across 500 prompts, comparing Configurations ➁, ➂, and ➃. Configuration ➃ achieved an average rank of 1.560 (where 1 is best, 3 is worst), compared to 1.692 for ➂ and 2.748 for ➁. It was ranked first in 59.8% of evaluations, substantially ahead of ➂ at 33.8% and ➁ at 6.4%. In pairwise win rates (Table 4), ➃ won against ➂ in 60.6% of cases and against ➁ in 83.4% of cases. The fact that ➃ won against ➁ (the Decoupled-Full configuration, which differs only in having global rather than constrained schedules) in 83.4% of cases provides strong evidence that the schedule constraints, not just the decoupling, drive the improvement.

Ablation Studies and Robustness Checks

• DM vs. CA in isolation (Figure 2, Section 3.1.1): Removing the DM term (CA-only) demonstrates that CA alone drives few-step conversion initially but produces training collapse from over-saturation and high-frequency noise. Removing the CA term (DM-only) demonstrates that pure distribution matching is substantially weaker. The fact that CA-only matches full DMD performance at early iterations (through approximately 2000 steps for 4-step SDXL) but then degrades provides direct evidence for the engine-regularizer decomposition: CA provides the distillation signal, DM prevents collapse. The non-obvious finding is that CA alone works at all — prior work assumed both terms were necessary for any distillation to occur, since the DM term was thought to be the mechanism.

• CA training with restricted $\tau$ ranges (Figure 4a, Section 4.1): Restricting $\tau$ to different ranges reveals the frequency-progressive nature of the CA engine. The most informative result is the negative one: training with only clean $\tau$ ($\tau \in [0.7, 1.0]$) causes collapse because "high-frequency details are meaningless if low-frequency general structure has not yet been determined." This provides mechanistic justification for why the standard DMD samples $\tau$ from the full range — not for theoretical distribution-matching reasons, but because the CA engine needs to enhance all frequency bands in the correct coarse-to-fine order. The finding that $\tau$ acts as a frequency-band selector rather than merely a noise level for distribution matching is a conceptual shift from prior work.

• Fake model as observer diagnostic (Figure 4b, Section 4.2): Training the fake model on the CA-only generator's outputs without using it for updates reveals what the fake model learns. The key observation — that the fake model reproduces the generator's checkerboard artifacts while the real model does not — directly visualizes the DM corrective mechanism. This is a clever experimental design that isolates the fake model's learning from its gradient contribution, something not done in prior DMD analyses.

• Mean-variance regularization as DM replacement (Figure 3, Section 3.2): Replacing the DM regularizer with per-image mean and variance constraints (Equation 7) demonstrates that CA engine stabilization does not require distribution matching. The training remains stable and produces reasonable images, but with noticeably lower quality than DM. This ablation establishes that the DM term is not unique in its regularizing function — simpler constraints can serve the same stabilizing role — and simultaneously demonstrates that DM's advantage comes from its ability to correct complex, spatially-varying artifacts beyond what global statistics capture. The paper identifies the target statistics ($\mu_{\text{target}} = 0.075$, $\sigma^2_{\text{target}} = 0.81$) as the averaged statistics of real sampled data, but does not explore sensitivity to these values.

• GAN discriminator as DM replacement (Figure 3, Section 3.2): Using a GAN discriminator (initialized from teacher weights) as the regularizer shows that more powerful corrective signals are possible but introduce significant instability — training collapses after approximately 4000 iterations. This validates the Diff-Instruct claim that score-based methods offer better stability than GANs when distributions have disjoint supports, and establishes DM as occupying a sweet spot between simpler but weaker constraints (mean-variance) and stronger but less stable ones (GAN).

• Decoupled-Full schedule (Config ➁ vs. ➀, Table 1): Simply decoupling the CA and DM schedules while keeping both global ($\tau_{\text{CA}} \in [0,1]$, $\tau_{\text{DM}} \in [0,1]$) yields negligible difference from the coupled baseline (DPG-Bench 83.77 vs. 83.90, HPS v2.1 30.69 vs. 30.61, HPS v3 10.32 vs. 10.34). This confirms that the benefit of decoupling comes from the schedule ranges, not from allowing the two terms to use different noise levels per se.

• Decoupled-Constrained schedule (Config ➂ vs. ➀, Table 1): Constraining both CA and DM to $\tau > t$ improves over the baseline (HPS v2.1 31.71 vs. 30.61, HPS v3 11.08 vs. 10.34) but introduces color oversaturation artifacts (visible in Figure 5, middle row). This validates the constrained CA hypothesis (focused engine improves detail) while simultaneously revealing the constrained DM hypothesis to be incorrect — DM needs global range to correct low-frequency artifacts.

• Decoupled-Hybrid schedule (Config ➃ vs. ➂ and ➀, Table 1): Constraining CA while keeping DM global achieves the best overall scores. Compared to ➂ (both constrained), ➃ improves HPS v2.1 from 31.71 to 32.29 and HPS v3 from 11.08 to 11.59, while eliminating the oversaturation artifacts (Figure 5, bottom row vs. middle row). This is the key validation of the paper's central hypothesis: the CA engine and DM regularizer require different re-noising schedules because they serve different functions — focused enhancement vs. comprehensive correction.

• SDXL DMD2 with decoupled schedule (Table 2): Applying only the decoupled schedule change to DMD2 (keeping all other components including the GAN loss) yields substantial improvements in ImageReward (+7.60) and FID (-1.15) with minor changes in other metrics. This demonstrates that the schedule innovation is orthogonal to and compatible with other DMD improvements (multi-step generation, GAN loss, etc.), and that the gains are not specific to a particular base model or DMD variant.

• Cross-model validation (SDXL vs. Lumina-Image-2.0): The consistent pattern of improvement from the decoupled schedule across both SDXL (Table 2) and Lumina-Image-2.0 (Table 1) suggests the findings are not model-specific. However, the paper does not test on additional model families or architectures, leaving open the question of whether the CA-engine/DM-regularizer decomposition and the optimal decoupled schedule generalize beyond these two diffusion model implementations.

• Negative result: GAN regularization eventually collapses (Figure 3, Section 3.2): The CA+GAN configuration shows competitive or superior image quality metrics at 2000-4000 steps but undergoes training collapse after approximately 4000 iterations. This negative result is important because GAN-based distillation methods (ADD, SDXL-Turbo, Diffusion Adversarial Post-Training) have shown strong results, and the finding that GAN regularization of the CA engine is unstable suggests that combining CA with GANs requires additional stabilization techniques (as done in DMD2, which uses both DM and GAN losses). The paper notes this aligns with VAE and advanced distillation practices where "models are often first trained with a stable objective before being fine-tuned with a GAN loss."

Critical Assessment

The experiments in this paper collectively support a reframing of how DMD works — from unified distribution matching to a functional decoupling into a CA engine and a DM regularizer — but there are important boundaries on what has been demonstrated and what remains assumed or untested.

Does the CA/DM ablation (Figure 2) genuinely establish CA as the engine and DM as the regularizer? The evidence is strong for the claim that CA is the primary driver of few-step conversion: CA-only training achieves few-step generation quality comparable to full DMD at early-to-mid training iterations, while DM-only is substantially worse. The paper is appropriately careful in qualifying the DM-only result, noting that DM "indeed makes relatively reasonable images" in the 4-step setting — it is not completely incapable, just substantially weaker. The "engine" designation is thus supported: CA drives the conversion, DM contributes but is not the primary mechanism.

However, the evidence for DM being "primarily" or "merely" a regularizer is slightly weaker than the engine claim because the paper does not fully disentangle DM's regularizing function from any potential distillation contribution it might also make. The CA-only training not only collapses eventually but also, at its peak (before collapse), achieves lower quality than CA+DM according to the metrics in Figure 2. The paper attributes this to CA+DM "yielding higher-quality final results," but this gap could arise from either (a) DM providing a small but genuine distillation signal that augments CA, or (b) DM's regularizing effect allowing CA to reach higher quality before artifacts set in. The paper's framing leans toward (b) but the experimental design does not definitively rule out (a). A stronger test would be: train with CA to the point just before collapse, then freeze the generator and apply only DM for additional fine-tuning. If DM further improves quality beyond what CA achieved at its peak, this would suggest DM provides an independent distillation signal; if not, the regularizer interpretation is strengthened.

Does the alternative regularizer experiment (Figure 3) truly prove DM is replaceable? The mean-variance constraint stabilizes CA training and produces reasonable images, demonstrating that distribution matching is not the only way to stabilize the CA engine. This is a valid and important result. However, the paper's claim that DM is "not unique" should be interpreted carefully: the mean-variance regularized images "fall noticeably short of that achieved with DM," and the GAN-regularized training collapses. What has been demonstrated is that some form of regularization is necessary and that distribution matching is one effective choice among several, not that all regularizers are equally capable. The "sweet spot" characterization — DM is more powerful than simple constraints but more stable than GANs — is more precise than the stronger claim of non-uniqueness.

Does the frequency-progressive CA analysis (Figure 4a) support the constrained $\tau_{\text{CA}} > t$ schedule? The experimental evidence that different $\tau$ ranges correspond to different frequency bands is clear and well-designed. The logical bridge from this observation to "therefore $\tau_{\text{CA}} > t$ is optimal" relies on the additional assumption that in few-step generation, earlier steps (smaller $t$) have already resolved the low-frequency content that would be redundantly re-enhanced by CA at $\tau < t$. This assumption is plausible given how diffusion models work (coarse-to-fine generation) but is not directly tested. An ablation that would strengthen this link: compare constrained $\tau_{\text{CA}} > t$ against $\tau_{\text{CA}} < t$ (applying CA only to noise levels below the current timestep) in a few-step setting. The paper hypothesizes this would be redundant and cause artifacts, but does not demonstrate it. The closest evidence is the qualitative improvement from ➁ (unconstrained) to ➂ (constrained) in Figure 5, which is suggestive but confounded by the simultaneous constraint on DM.

How robust are the decoupled schedule results (Tables 1 and 2)? The consistent improvement from Configuration ➃ over baselines across two model families (SDXL and Lumina-Image-2.0), multiple metrics (DPG-Bench, HPS v2.1, HPS v3, ImageReward, FID, CLIP-S), and human evaluation (Appendix C) is the paper's strongest empirical contribution. The improvements are not marginal — +7.60 ImageReward on SDXL, +1.68 HPS v2.1 on Lumina-Image-2.0, unanimous human preference — suggesting the decoupled schedule provides a genuine and substantial benefit.

That said, several robustness questions remain unaddressed. First, the paper tests only one guidance scale $\alpha$ per model (the standard value used in DMD practice). Since the CA term scales with $(\alpha - 1)$, the optimal schedule might depend on $\alpha$ — a stronger CFG scale might require different CA constraint bounds to avoid over-enhancement, or might change the balance between CA and DM. The paper does not explore this interaction. Second, the experiments use a fixed number of generation steps (4 for all schedule ablation experiments). Whether the optimal decoupled schedule depends on the number of steps (e.g., would a 1-step generator benefit from a different constraint than an 8-step generator?) is not investigated. Third, the DMD2 comparison in Table 2 changes only the schedule while keeping all other DMD2 components fixed, but the paper does not report whether DMD2-specific hyperparameters (GAN loss weight, fake model update frequency, etc.) were re-tuned for the new schedule. If DMD2 was heavily tuned for the original shared schedule, the decoupled schedule might underperform its potential if those hyperparameters are suboptimal for the new noise sampling pattern.

What is missing from the experimental evaluation? Several experiments would strengthen the paper's claims:

1. Dependence on CFG scale $\alpha$. The CA term is scaled by $(\alpha - 1)$. Does the optimal decoupled schedule change with $\alpha$? If $\alpha$ is reduced, does the CA engine become weaker, requiring a less constrained schedule? If $\alpha$ is increased, does over-enhancement become more severe, requiring a tighter constraint? This would connect the paper's mechanistic understanding to a practically important hyperparameter.

2. Step-count scaling. The experiments use 1-step and 4-step generators. Does the decoupled schedule benefit scale with more steps (8-step, 16-step)? Does the optimal constraint ($\tau_{\text{CA}} > t$) become more or less important as the number of steps increases and each step covers a narrower frequency range?

3. Training duration / convergence. The paper reports results at fixed iteration counts but does not investigate whether the decoupled schedule changes the convergence rate (how many iterations to reach a given quality level) or the asymptotic performance (where the quality plateaus). The CA/DM ablation in Figure 2 shows that training dynamics differ substantially between configurations — DM-only improves more slowly, CA-only peaks and then collapses. Does the decoupled schedule reach its peak faster or slower than the baseline?

4. Interaction with the proxy loss weight $\lambda$. The proxy loss formulation (Algorithm 1) introduces a hyperparameter $\lambda$ that scales the score-based update. If the CA term provides a stronger gradient signal than the DM term (which seems likely given their relative magnitudes and contributions), decoupling the schedules might change the effective balance between the two gradient contributions, which could interact with $\lambda$ tuning.

5. Computational cost of the decoupled schedule. The decoupled schedule requires two separate re-noising operations per generator update (one for $\tau_{\text{CA}}$, one for $\tau_{\text{DM}}$), compared to one for the original DMD. The paper does not report the wall-clock time or memory overhead of this change. If the re-noising step is cheap compared to score model evaluation, this overhead is negligible; if not, the fair comparison would give the original DMD additional training iterations to match the total computation.

6. Single CFG scale for both CA and DM. The decoupled formulation (Equation 8) maintains the same $\alpha$ for both terms, but the CA term uses $\alpha$ directly (via the $(\alpha-1)$ scaling of the CFG direction) while the DM term uses $\alpha$ only indirectly (through the choice of $\tau_{\text{DM}}$ but not through any explicit CFG scaling). Could the DM term benefit from its own, independent guidance scale? The paper's decomposition naturally suggests this possibility but does not explore it.

7. Broader model and task validation. All experiments use either SDXL or Lumina-Image-2.0, both of which are latent diffusion models for text-to-image generation. Whether the CA-engine/DM-regularizer decomposition holds for pixel-space diffusion models, video generation, audio generation, or other modalities is entirely untested. The paper's findings might be specific to the architecture, the training data, or the text-conditioning setup of these particular models.

Do the experiments support the paper's broader conceptual claims? The paper makes a strong empirical case for the functional decoupling in DMD specifically. The broader claim — that this reveals something general about how few-step distillation works, connecting to why trajectory-matching and GAN-based methods also succeed — is more speculative and is not directly tested. The discussion in Appendix A (the parallel to LLM token prediction and the hypothesis about CFG as a "deterministic decision pattern") is presented as preliminary ideas rather than experimentally validated claims, and the paper is appropriately cautious: "we acknowledge that a significant gap remains towards a rigorously accurate explanation." The experiments in the main paper do not test whether non-DMD distillation methods (LCM, progressive distillation, ADD) exhibit similar CA-like mechanisms or frequency-progressive behavior. These remain hypotheses for future work.

6. Limitations and Trade-offs

Difficulty Estimation Cost Is Not Accounted For

The entire compute-optimal framework in the reference paper rests on the ability to estimate prompt difficulty before allocating the inference budget. The method for doing so — generating 2048 samples per question and averaging either ground-truth correctness (oracle) or PRM scores (predicted) — is extraordinarily expensive relative to the test-time budgets being studied. The paper explicitly acknowledges this gap:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity" (Section 3.2)

This is not a minor omission. At 2048 samples per question, the difficulty estimation step alone consumes more compute than the largest test-time budgets studied (256–512 generations). The reported 4× efficiency gains over best-of-N are computed after difficulty is known, without amortizing the cost of learning it. In a realistic deployment, the total cost would be difficulty estimation plus strategy execution, and the former could dominate the latter — potentially erasing or even reversing the claimed efficiency improvements. The paper frames the exploration-exploitation tradeoff (compute spent assessing difficulty versus compute spent solving the problem) as "a key avenue for future work" but provides no method for addressing it.

The paper contains no experiment that measures total cost including difficulty estimation. The "predicted difficulty" bins use PRM scores rather than ground-truth labels (Section 3.2), which removes the need for knowing correct answers but does not reduce the 2048-sample generation cost. The difficulty estimation overhead is not included in any budget calculation, Figure 4, or Figure 8. Mitigation is entirely deferred to future work: the paper suggests "pretraining or finetuning models to directly predict difficulty of a question" (Section 8), but no such model is developed or evaluated. Until this gap is closed, the reported 4× efficiency figure should be understood as an upper bound on achievable efficiency under the assumption of zero-cost difficulty estimation, not a realized deployment gain.

Hard Problems Remain Unsolvable — Test-Time Compute Has a Hard Ceiling

The paper demonstrates clearly that test-time compute provides essentially zero benefit on the hardest problems (difficulty bin 5). Across all methods — PRM search (Figure 3, right), iterative revisions (Figure 7, right), and their compute-optimal combinations — bin 5 accuracy hovers at 1–3% regardless of compute budget. In the FLOPs-matched comparison (Figure 9, Section 7), the bin 5 scaling line is "essentially flat near 0–5%," and the 14× larger pretrained model substantially outperforms test-time compute on these problems at all ratios R.

The consequence is fundamental: test-time compute can amplify existing capability but cannot create it from nothing. If the base model's pass@1 is near zero on a problem class — meaning no correct solutions exist in the proposal distribution to find or refine — no amount of search or revision will help. The paper is candid about this:

"test-time compute provides minimal gains on problems that are fundamentally outside the base model's capability range" (Section 7)

This is not a limitation the paper can "fix" — it is a property of the approach. But it establishes a sharp boundary condition that practitioners must respect: deploying this method on problem distributions with a meaningful fraction of bin-5-hard questions (where the base model essentially never produces correct answers) will yield near-zero improvement from test-time compute regardless of budget. The only path forward for such problems is better pretraining.

The paper does not attempt to characterize what fraction of real-world deployment queries fall into each difficulty bin, nor does it provide guidance on how to detect whether a given problem distribution is amenable to test-time compute before investing in the infrastructure. The difficulty bins are defined relative to the base model's pass@1 rate on MATH, which may not transfer to other domains. Mitigation is partial: the difficulty estimator can identify hard problems (by detecting that the PRM gives uniformly low scores), allowing the system to recognize when test-time compute will not help, but it offers no solution for those problems beyond routing them to a larger model.

Single Benchmark and Single Model Family — Generalization Is Untested

All experiments in the paper use the MATH benchmark (500 test questions) with PaLM 2-S* as the base model. The authors state they "believe this model is representative of the capabilities of many contemporary LLMs" (Section 4), but this claim is entirely unverified. The paper contains no experiments on other model families (GPT, LLaMA, Claude), other reasoning benchmarks (GSM8K, MMLU, ARC, HumanEval, MBPP), or other task categories (code generation, factual QA, logical reasoning, open-ended generation).

Several aspects of the findings could be model-specific or benchmark-specific. The PRM's quality and over-optimization behavior depend on PaLM 2-S*'s output distribution — a model with different calibration properties or error patterns might exhibit qualitatively different difficulty-dependent scaling curves (e.g., the threshold where beam search becomes harmful vs. helpful might shift). The revision model's effectiveness depends on the base model's in-context learning capabilities and the quality of the edit-distance-based training data construction, both of which could vary substantially across model families. The MATH benchmark consists exclusively of competition-level math problems requiring multi-step symbolic reasoning. It is unclear whether the difficulty-dependent patterns generalize to other reasoning domains or to tasks requiring factual knowledge rather than inference.

The test set size introduces additional uncertainty. With 500 questions split into five difficulty quintiles of ~100 each, and two-fold cross-validation within each bin, the compute-optimal policy is selected based on ~50 questions per fold per bin. The paper does not report confidence intervals on the compute-optimal scaling curves, making it difficult to assess whether the observed trends are statistically reliable at this sample size. Mitigation is absent: the paper does not replicate on any additional benchmark or model, and does not discuss domain-specific caveats for practitioners applying the method to non-math tasks. The findings should be treated as demonstrated for PaLM 2-S on MATH* until replication on other models and tasks exists.

The 14× Larger Model Baseline Is Not Compute-Optimally Trained

The FLOPs-matched comparison in Section 7 scales model parameters while holding training data fixed, following the LLaMA paradigm (Touvron et al., 2023). The paper explicitly acknowledges this departs from compute-optimal pretraining (Hoffmann et al., 2022):

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work." (Section 7)

The consequence is that the pretraining baseline is weaker than it needs to be. A Chinchilla-optimal model trained with 14× more total FLOPs (scaling both parameters and data according to the ~1:1 ratio prescribed by Hoffmann et al.) would likely outperform a parameter-only-scaled model at the same total compute. The reported advantages of test-time compute over pretraining — e.g., +27.8% relative improvement on easy questions at R ≪ 1 for revisions (Section 7, Figure 1 bar charts) — may shrink or reverse when compared against a properly compute-optimal larger model.

Additionally, the 14× larger model uses only greedy decoding in the FLOPs-matched comparison — no majority voting, no best-of-N, no search, no revision. This is an asymmetric comparison: the smaller model gets an optimized test-time compute strategy selected per difficulty bin, while the larger model gets nothing. A fairer baseline would give the larger model at least a modest test-time compute budget (e.g., best-of-8 weighted), which would substantially raise its performance without dramatically changing the FLOPs accounting. The paper does not explore this, nor does it report how sensitive the FLOPs-matched conclusions are to the choice of pretraining scaling strategy.

The paper does not attempt to mitigate this limitation. The authors are transparent about it but defer the compute-optimal pretraining comparison entirely to future work. Practitioners interpreting the FLOPs-matched results should understand that they represent an upper bound on the advantage of test-time compute over pretraining — the advantage narrows (and potentially reverses) as the pretraining baseline becomes more compute-optimal and as the larger model is allowed its own test-time compute budget.

Latency and Wall-Clock Time Are Ignored

The paper measures compute exclusively in "generations" (number of complete solutions sampled), which is a reasonable proxy for total FLOPs but wholly ignores latency — the wall-clock time required to produce an answer. This is a consequential omission because the compute-optimal strategies recommended by the paper heavily favor sequential computation on certain difficulty levels.

Sequential revisions are inherently serial: each revision depends on the previous one, so a chain of length 64 cannot be parallelized. Parallel best-of-N with N = 64 can run all 64 samples simultaneously with sufficient hardware. The compute-optimal policy for easy problems (Figure 7, right) favors purely sequential revisions, meaning a strategy that allocates 64 generations sequentially takes roughly 64× longer wall-clock time than one that runs 64 parallel samples. For latency-sensitive applications — interactive assistants, real-time decision-making, on-device deployment where the paper explicitly argues test-time compute is most valuable (Section 1) — the sequential-heavy strategies may be impractical regardless of their accuracy advantages.

The paper does not measure or discuss latency, throughput, or the hardware assumptions required to achieve the reported generation budgets within acceptable response times. Mitigation is absent. A practitioner deploying this method would need to independently evaluate whether the recommended sequential strategies meet their latency requirements, and potentially trade off accuracy for wall-clock time by limiting revision chain length or falling back to parallel sampling.

Revisions and Search Are Never Combined

The paper studies PRM search (Section 5) and iterative revisions (Section 6) as independent mechanisms, establishing their complementary difficulty-dependent strengths: revisions excel on easy problems (local refinement), PRM search excels on medium problems (global exploration). However, the two mechanisms are never combined in any experiment. The paper explicitly acknowledges this:

"we did not experiment with PRM tree-search techniques in combination with revisions" (Section 8)

This is a significant gap because the natural synthesis — using the revision model as the proposal distribution within PRM-guided beam search, or using the PRM to score and select among revision chain outputs — could yield gains beyond either method alone. The paper's own framework (Section 2, the proposer-scorer decomposition) directly suggests that combining a better proposal distribution (revisions) with a better verifier (PRM search) should be superior to improving either in isolation. On medium-difficulty problems, where both mechanisms show non-trivial but incomplete benefits individually, the combined approach might push through the performance ceiling that each hits separately.

The paper does not attempt this combination or provide any experimental evidence about whether the two mechanisms interact synergistically or interfere. The reported results therefore represent a lower bound on what a fully integrated system could achieve. The paper delegates this to future work without providing guidance on how to combine the mechanisms (e.g., should the PRM score revision chain steps? should beam search be applied across revision steps or only within them?).

7. Implications and Future Directions

How This Work Changes the Landscape

This paper fundamentally reframes how the field should understand score-based distillation. It is not an incremental improvement to DMD but a reclassification of the mechanism itself: what prior work treated as an elegant distribution-matching algorithm with an empirically-motivated CFG hack turns out to be, in practice, a CFG-internalization engine with a distribution-matching regularizer bolted on. The magnitude of this shift is significant because it changes what researchers should optimize, what questions they should ask, and how they should interpret results across the entire score-based distillation family (Diff-Instruct, DMD, DMD2, SiD, ASID).

The most immediate impact is that the fake model is demoted from essential component to optional regularizer. Prior to this work, the score-based distillation literature treated the fake model as indispensable — it estimates the student's distribution score, enabling the IKL divergence minimization that was thought to drive distillation. The paper demonstrates that the fake model is not involved in the CA term at all (Equation 6), and that CA alone achieves few-step conversion quality comparable to full DMD (Figure 2). The fake model matters only through the DM term, whose role is stabilization, not distillation. This implies that substantial research effort — designing better distribution divergences, improving fake model training schedules, analyzing fake model convergence — has been directed at optimizing a regularizer rather than the engine. The paper redirects attention toward understanding and improving the CA mechanism: its noise schedule, its frequency-selective properties, its dependence on guidance scale, and its connection to the teacher's internal representations.

The paper also resolves a tension that has quietly persisted in the score-based distillation literature. The theoretical framework (Equation 2) predicts that matching the teacher's conditional score should suffice for distillation, yet practitioners universally find that setting α = 1 produces poor results. Prior work rationalized this as "CFG provides a better target distribution" — a reasonable but unsatisfying explanation that didn't explain why the theoretically correct objective underperforms. The paper provides a clean resolution: α = 1 causes the CA term to vanish (since it is scaled by (α − 1), Equation 6), leaving only the weak DM signal. The theoretically correct objective is not "wrong" — it's just missing the engine that does the actual distillation work. This explains both why α = 1 fails (no engine) and why α > 1 succeeds (engine engaged) without needing to invoke improved target quality.

The frequency-selective analysis (Section 4.1) introduces a new conceptual tool for understanding and controlling distillation. The finding that different re-noising timesteps τ correspond to different frequency bands — with the CA engine enhancing content at the frequency that survives noising to that level — reframes distillation as a frequency-targeted knowledge transfer problem. This connects distillation to the spectral analysis of diffusion models (Dieleman, 2024) and provides a vocabulary for describing what the student is learning at each noise level. The constrained CA schedule (τ_CA > t) is a direct consequence of this understanding, and its success (Tables 1-2, Figure 5) validates that the frequency-band perspective has practical utility beyond theoretical interest.

More broadly, the paper suggests that many few-step distillation methods may share a common underlying mechanism — baking some form of guidance or conditioning signal into the student to collapse the sequential generation tree into fewer steps. The discussion in Appendix A (parallel to LLM token prediction) frames this speculatively: if CFG represents an external, uncontrollable decision pattern that the multi-step model relies on, and if distilling that pattern into the student is what enables few-step generation, then trajectory-matching methods (which implicitly encode the teacher's CFG-influenced trajectory) and GAN-based methods (which provide an adversarial guidance signal) may be doing the same thing through different mechanisms. The paper does not prove this, but the framework provides a unifying lens that was not previously available.

The research agenda around score-based distillation is redirected in several ways. Directions that become more attractive include: understanding the CA engine's properties (its frequency behavior, its optimal guidance scale, its interaction with step count), designing better regularizers that match DM's artifact-detection capability with lower computational cost or greater stability, and exploring whether the CA engine can be extracted or pre-computed to reduce distillation training time. Directions that become less attractive include: pursuing incrementally better distribution divergences for the DM term (since DM is a regularizer, not the engine, improvements to it will hit diminishing returns once artifacts are adequately suppressed) and analyzing fake model convergence as a proxy for distillation quality (since the fake model is not involved in the primary distillation signal).

Follow-Up Research This Work Enables

Characterizing CA engine behavior as a function of CFG scale α. The CA term is scaled by (α − 1) (Equation 6), meaning its contribution vanishes at α = 1 and grows with α. The paper's experiments use whatever standard CFG scale is typical for each model (likely 5–7.5 for SDXL, not explicitly stated), but never vary α systematically. An experiment that sweeps α from 1 to, say, 15 while tracking (a) the speed of few-step conversion (iterations to reach a target quality), (b) the severity of artifacts when CA is used alone, (c) the optimal constrained τ range for each α, and (d) the final converged quality with DM would reveal whether there is a "CA-optimal" guidance scale distinct from the teacher's inference-time optimal scale. The paper's hypothesis that the CA engine "bakes" the CFG pattern into the student would predict that higher α produces faster conversion but also more severe artifacts (since stronger CFG emphasizes the conditional direction more aggressively, potentially over-enhancing). If instead higher α saturates or degrades distillation speed, that would constrain the "CFG-as-external-decision-pattern" interpretation.

Measuring whether the decoupled schedule benefit scales with generation step count. The paper validates the constrained CA + global DM schedule (τ_CA > t, τ_DM ∈ [0,1]) on 4-step generators (Tables 1-2, Figure 5). The logic that motivates the constraint — earlier steps resolve low frequencies, CA should focus on unresolved higher frequencies — implies that the constraint's benefit should change with step count. For a 1-step generator (t is always 0), τ_CA > 0 is equivalent to τ_CA ∈ (0,1], which is barely different from the full range — the constraint should provide minimal benefit. For an 8-step generator, each step covers a narrower frequency band, and the constraint τ_CA > t excludes a larger fraction of the noise range at early steps — the benefit should be larger. An experiment comparing the improvement from decoupled vs. coupled schedules at 1, 2, 4, 8, and 16 steps would test this prediction and provide practical guidance for practitioners choosing step counts.

Testing whether the DM corrective mechanism can be strengthened by targeted fake model training. Figure 4b reveals that the DM term functions as an adaptive artifact detector because the fake model learns to replicate whatever artifacts the generator produces, while the real model does not. This suggests that fake model training strategy directly affects regularizer quality. A stronger fake model (trained with more iterations per generator update, or with a focused objective on recent generator outputs) should detect artifacts more sensitively, producing stronger corrective gradients. Conversely, a weaker fake model should produce weaker regularization, potentially allowing artifacts to grow. An experiment that varies fake model update frequency (e.g., 1, 2, 4, 8 fake model updates per generator update) or fake model capacity (varying its parameter count) while measuring artifact severity and final image quality in CA+DM training would test whether the DM term's effectiveness is bottlenecked by fake model quality. If improving the fake model beyond a certain point yields diminishing returns, that would indicate other factors (e.g., the real model's insensitivity to certain artifact types) limit DM's corrective power.

Does the CA engine work with alternative guidance formulations? The paper's decomposition reveals that the CA term uses the specific CFG formulation s_uncond + α(s_cond − s_uncond). This is the standard Classifier-Free Guidance introduced by Ho & Salimans (2022), but it is not the only way to incorporate conditioning information into diffusion model scores. Alternative formulations — such as using a separate classifier's gradient (classifier guidance, Dhariwal & Nichol, 2021), applying guidance only to certain denoising steps, or using text-conditional embeddings without explicit unconditional subtraction — would produce different Δ_cfg terms. Replacing the CA engine with these alternative guidance signals (keeping the DM regularizer unchanged) and measuring distillation effectiveness would clarify what properties of CFG make it such an effective engine. If alternative guidance formulations produce similar distillation speed and quality, then the engine is not specific to CFG — any conditioning signal that sharpens the target distribution works. If CFG is uniquely effective, that supports the paper's hypothesis (Appendix A) that CFG represents a specific, predictable decision pattern that the student can internalize.

Training a difficulty predictor for the paper's compute-optimal framework. The compute-optimal test-time scaling approach from the reference paper requires estimating prompt difficulty before allocating budget, and the current method (generating 2048 samples and scoring with a PRM) is far too expensive for deployment. The paper explicitly calls for "pretraining or finetuning models to directly predict difficulty of a question" (Section 8). A concrete experiment: train a small classifier (e.g., a lightweight transformer or even an MLP on top of frozen LLM embeddings) to predict the difficulty quintile of a MATH problem from the problem text alone, using the oracle difficulty labels (based on pass@1 over 2048 samples) as supervision. The key metric is whether substituting this predicted difficulty for the PRM-based difficulty in the compute-optimal policy achieves accuracy within, say, 2% of the oracle policy. If the classifier is accurate enough, the 4× efficiency gains become realizable in deployment without the 2048-sample overhead. If not, it quantifies how much of the gain is lost to estimation error, motivating research on better difficulty features or adaptive estimation strategies.

Stress-testing the decomposition on a model where the teacher's unconditional outputs are already high-quality. The paper's experiments use SDXL and Lumina-Image-2.0, both of which produce significantly better images with CFG than without it. This is typical for large text-to-image models, but it creates a potential confound: the CA engine could be dominant partly because the unguided teacher distribution is mediocre, making distribution matching (DM) an inherently weak target. What happens when the teacher's unconditional or α = 1 conditional outputs are already high-quality — for instance, on a model trained without conditioning, or a model where the conditioning signal is inherently strong enough that CFG provides only marginal improvement? If the CA engine's dominance persists even when the unguided distribution is good, that would strengthen the claim that CA is the fundamental mechanism rather than a workaround for weak conditional generation. If, instead, DM becomes competitive or superior when the unguided distribution is already strong, that would establish a boundary condition: DM is the engine when the target distribution is already sharp, CA is the engine when CFG is needed to sharpen it.

Practical Applications and Downstream Use Cases

Improving existing DMD and DMD2 training pipelines. The most immediate practical application is straightforward: replace the shared re-noising schedule in any DMD-based training pipeline with the decoupled-hybrid schedule (τ_CA > t, τ_DM ∈ [0,1]). The paper demonstrates this on DMD2 for 4-step SDXL, yielding +7.60 ImageReward points and −1.15 FID with no other changes (Table 2). The implementation change is minimal — approximately 10 lines of code modification (comparing the original vs. decoupled branches in Algorithm 1) — and introduces negligible computational overhead (one extra re-noising operation per generator update). For practitioners already using DMD or DMD2 for few-step distillation, this is a drop-in improvement with demonstrated gains across two model families (SDXL, Lumina-Image-2.0) and validated by human preference (100% annotator preference, Appendix C). The paper's ablation (Table 1, Config ➁ vs. ➀) confirms that simply decoupling without constraining yields no benefit, so the constrained CA schedule is the active ingredient — practitioners must implement τ_CA > t, not just independent sampling.

Reducing the cost of fake model training in resource-constrained settings. The finding that the fake model is unnecessary for the core distillation signal (CA-only training achieves comparable few-step conversion, Figure 2) has direct cost implications. Training the fake model concurrently with the generator requires maintaining a separate model copy, running additional forward and backward passes, and tuning the generator-to-fake-model update ratio. If a practitioner is willing to trade some final quality for reduced training cost — for instance, during rapid prototyping, hyperparameter search, or distillation of very large models where maintaining a full fake model is memory-prohibitive — the paper suggests two options: (a) train with CA + mean-variance regularization (Equation 7), which eliminates the fake model entirely while maintaining stable training, at the cost of "noticeably" lower final quality (Figure 3, Figure 6); or (b) train with CA + DM for the bulk of training, then optionally fine-tune with an added GAN loss (following DMD2's approach) to recover peak quality. The paper does not quantify the computational savings of the mean-variance approach versus full DMD, but the savings are substantial: no fake model forward passes, no fake model backward passes, no fake model memory footprint, and a trivial regularizer computation (per-image mean and variance).

Designing generation-step-dependent noise schedules for new few-step models. The frequency-progressive analysis of the CA engine (Section 4.1, Figure 4a) provides a principled design rule for practitioners building few-step generators from scratch: at each generation step t, the distillation training objective should focus its gradient signals on frequencies that remain to be determined. Concretely, for an N-step generator with timesteps {t_1, …, t_N}, the noise levels used for computing training gradients at step t_k should be drawn from the range (t_{k-1}, 1] — excluding noise levels below the previous step's timestep, since those frequencies have already been resolved. This rule is directly actionable for any score-based or trajectory-matching distillation method that uses re-noised intermediate representations, not just DMD. For methods like progressive distillation or consistency distillation that do not explicitly re-noise, the rule suggests a curriculum where early training focuses on coarser noise levels and later training on finer ones, potentially improving convergence speed or final quality. The paper validates this rule only for DMD and only for 4-step generation, so practitioners applying it to other methods should verify with their own ablations.

Interpreting and debugging artifacts in few-step generators. The DM corrective mechanism (Section 4.2, Figure 4b) provides a diagnostic framework for understanding artifacts in few-step generators. If a distilled generator produces a specific artifact pattern (e.g., checkerboard noise, over-saturation, structural deformities), the paper's analysis suggests examining whether the fake model — if one exists in the training pipeline — has learned to reproduce that artifact. If the fake model does not capture the artifact (because it was trained on a stale distribution, or with insufficient capacity, or on insufficient recent samples), the DM gradient cannot correct it, and the artifact will persist or grow. Practitioners can diagnose this by visualizing fake model reconstructions of artifact-laden generator outputs à la Figure 4b, checking whether the artifact is present in s_fake predictions but absent in s_real predictions. If the fake model fails to capture the artifact, interventions include: increasing fake model update frequency, training the fake model on a more recent buffer of generator outputs, or increasing fake model capacity. If the artifact is present in both s_fake and s_real (meaning even the teacher model reproduces it), the problem is not a DM failure but rather an inherent limitation of the teacher that no amount of distillation can fix — suggesting the need for a better teacher or a different guidance formulation.

When to Prefer This Method

The paper does not position itself against a specific named alternative with an explicit tradeoff matrix. Its primary contribution is a re-interpretation of DMD rather than a new method competing against existing ones — the "Decoupled" configuration in Table 2 is DMD2 with a modified schedule, not a new distillation algorithm. The practical decision the paper enables is within DMD-based training: prefer the decoupled-hybrid schedule (τ_CA > t, τ_DM ∈ [0,1]) over the standard shared schedule (τ ∼ U(0,1)) when training few-step generators using DMD or DMD2. The evidence for this preference is consistent across two model families, multiple automated metrics, and unanimous human annotator preference (Tables 1-2, Appendix C).

The broader engine-regularizer insight suggests a conditional preference that the paper does not explicitly state but that follows from its findings: when designing or debugging a score-based distillation pipeline, direct analytical and tuning effort toward the CA term (guidance scale α, re-noising schedule, frequency targeting) rather than the DM term (distribution divergence choice, fake model architecture, fake model training schedule). The CA term is the engine; improvements to it will directly affect distillation speed and quality. The DM term is a regularizer; improvements to it will primarily affect stability and artifact suppression, with diminishing returns once artifacts are adequately controlled. The paper's mean-variance experiment (Figure 3) demonstrates that even a trivially simple regularizer can partially substitute for DM, further supporting the claim that DM is not the primary lever for distillation quality.

For practitioners choosing between DMD-based methods and other distillation paradigms (trajectory-matching, consistency distillation, GAN-based), the paper does not provide direct comparative evidence and does not claim DMD with decoupled schedules is universally superior. The SDXL comparison in Table 2 shows the Decoupled configuration achieving best FID (17.80) and ImageReward (78.61) among listed methods, but this is confounded by the DMD2 base configuration and the specific hyperparameter choices. The paper's contribution is conceptual — understanding what DMD actually does — rather than benchmarking against alternatives.