CDLM: Consistency Diffusion Language Models For Faster Sampling

ArXiv: 2511.19269

🎯 Pitch

Diffusion language models can now match auto-regressive decoding speed without sacrificing quality. By training the model to finalize multiple tokens per step and enabling KV caching, CDLM cuts latency by up to 14.5× while preserving accuracy on math and coding benchmarks—even beating auto-regressive baselines in throughput.

1. Executive Summary

This paper introduces CDLM (Consistency Diffusion Language Models), a training-based acceleration method for diffusion language models that simultaneously tackles two core inference bottlenecks—excessive refinement steps and the inability to use standard KV caching—by integrating consistency modeling with block-wise causal attention fine-tuning. CDLM trains a block-wise causal student to finalize multiple tokens per step by distilling from a fully bidirectional teacher and enforcing temporal consistency between less-informed and more-informed states along the decoding trajectory (operationalized via forward KL divergence on still-masked positions), achieving 3.6×–14.5× lower latency and 3.4×–7.9× fewer refinement steps across math (GSM8K, MATH) and coding (HumanEval, MBPP) benchmarks with Dream-7B-Instruct and LLaDA-8B-Instruct. On GSM8K-CoT, CDLM–Dream achieves 44.1 total steps compared to the baseline’s 256 (a 5.8× reduction) while maintaining accuracy at 78.8 versus 79.1, and on MBPP-Instruct it cuts latency from 21.7 seconds to 1.5 seconds with improved accuracy (51.8 → 53.0). The method also surpasses equal-size autoregressive baselines in throughput, establishing that few-step parallel refinement with native KV caching can match or exceed autoregressive decoding speed while preserving quality only when the underlying DLM backbone is sufficiently strong and the distillation corpus covers the target domain.

2. Context and Motivation

The Core Problem: Diffusion Language Models Are Too Slow for Practical Use

Diffusion language models (DLMs) represent a fundamentally different approach to text generation than the autoregressive (AR) paradigm that dominates modern LLMs. Instead of generating tokens one by one from left to right under a causal mask, DLMs iteratively refine a sequence of masked or noisy tokens into coherent text through repeated parallel passes over the entire sequence—a process inspired by the success of diffusion models in image, audio, and video synthesis (Ho et al., 2020). Each denoising step updates all token positions simultaneously, which in principle breaks the sequential dependency that bottlenecks AR decoding.

This parallel-generation paradigm carries two compelling promises. First, it could dramatically improve inference throughput. Closed-source diffusion-based models from major labs have already reported up to 10× higher throughput than autoregressive models while preserving output quality on code generation tasks (DeepMind Gemini Diffusion, Mercury, SeedDiffusion; cited in Section 1). If open-source DLMs could approach these speeds, they would reshape the economics of LLM deployment—offering an alternative to the dominant decoder-only Transformer that is faster, more parallelizable, and inherently better at tasks requiring bidirectional context (text infilling, document editing, global planning).

Second, bidirectional attention enables capabilities that causal attention forbids. Because DLMs do not enforce left-to-right temporal order during generation, they can attend to the entire sequence context at every step. This is valuable for tasks where the model benefits from seeing both past and future context simultaneously—structured document completion, code repair, or compositional reasoning where later parts of the output constrain earlier decisions.

However, a sharp gap separates promise from reality. The authors are clear about the problem: open-source DLMs remain significantly slower than their AR counterparts in practice (Section 1). The paper identifies two specific, interacting bottlenecks that cause this:

Excessive refinement steps. To achieve high-quality text, standard DLMs typically require a number of denoising steps proportional to the target sequence length (Kim et al., 2025; cited in Section 1). For a 256-token generation—a typical length for a math solution or code snippet—models like Dream-7B and LLaDA-8B use 256 refinement steps by default. Each step requires a full forward pass through the Transformer, making the total FLOP cost enormous relative to an AR model that generates the same 256 tokens with 256 forward passes but amortizes the cost over sequentially generated tokens. Worse, as the ablation in Section 5.3.2 demonstrates (Table 4), simply reducing the step budget without retraining destroys accuracy—Dream's accuracy on GSM8K-CoT drops from 79.1 to 41.8 when forced to use 48 steps instead of 256. The model has been trained to rely on many small, incremental refinements and cannot seamlessly transition to taking larger jumps.
Inability to use standard KV caching. The causal attention mask in autoregressive Transformers is what enables KV caching: once a key-value pair is computed for a past token, it never changes because that token can only attend to positions before it. DLMs, by contrast, use fully bidirectional attention at every step. Every token attends to every other token, and because the sequence changes at each refinement step (masked positions get filled in), there is no stable set of KV states that can be cached across steps. Each denoising step requires recomputing the entire sequence's attention from scratch. This is catastrophic for efficiency: the cost per step is quadratic in sequence length with no caching amortization.

These two bottlenecks compound each other. Many steps × no caching = very slow inference. The paper reports baseline Dream-7B-Instruct latencies of 23.5 seconds for GSM8K-CoT and 13.4 seconds for HumanEval-Instruct on A100 GPUs (Table 1)—far too slow for interactive applications.

The Landscape of Existing Solutions and Their Gaps

The paper situates itself within an active and rapidly evolving area of research aimed at accelerating DLM inference. The authors categorize prior work along two axes (Section 2.2): training-free (inference-only) methods and training-based (fine-tuning) methods.

Training-Free Acceleration: Patching Symptoms Without Fixing the Root Cause

Approximate caching approaches. Methods like dLLM-Cache (Liu et al., 2025) and Fast-dLLM's dual-cache KV (Wu et al., 2025) attempt to reduce per-step computation by caching KV states for regions of the sequence that change slowly and recomputing only the regions where tokens are actively being finalized. These are training-free: they work by cleverly managing model state during inference without modifying the model's weights. The results in Tables 1 and 2 show these methods provide meaningful speedups—dLLM-Cache reduces baseline Dream latency from 23.5s to 12.6s on GSM8K-CoT, and Fast-dLLM with dual-cache brings it down to 2.5s.

However, these methods have fundamental limitations. They are approximate—the caching heuristics risk accumulating errors over long refinement chains. More importantly, they only address the caching bottleneck, not the step-count bottleneck. dLLM-Cache keeps the step budget fixed at 256, so while per-step cost drops, the model still runs hundreds of forward passes. The speedup is multiplicative (lower cost per step) but bounded by the step count.

Parallel sampling with confidence thresholds. Fast-dLLM's parallel decoding (Wu et al., 2025) attacks the step-count problem by revealing multiple tokens per iteration based on a confidence threshold: at each step, any masked token whose predicted probability exceeds a threshold is finalized immediately rather than waiting for its scheduled step. This is training-free because it works with the base model's existing confidence estimates. Tables 1 and 2 show this provides substantial acceleration—Fast-dLLM (Parallel) reduces Dream's GSM8K-CoT steps from 256 to 53.7 with only a minor accuracy change (79.1 → 79.9).

The limitation is that these methods are heuristic overlays on a model that was trained to make many small refinement steps. The base model generates predictions expecting that only a small fraction of tokens will be unmasked per step; forcing multi-token finalization through thresholding can introduce instability or quality degradation. The effectiveness depends heavily on the threshold hyperparameter, which has to be tuned per-task, and there is no guarantee that the model's confidence estimates are well-calibrated when used outside the training distribution (making larger jumps than it was trained for).

Combined approaches. Fast-dLLM's Parallel + Dual Cache variant combines confidence-thresholded multi-token decoding with approximate KV caching. This yields the best training-free results (e.g., 2.5s latency on Dream GSM8K-CoT at 77.3 accuracy), nearly matching CDLM's latency but at higher step counts and with heuristic caching rather than native KV cache support.

Training-Based Acceleration: The Right Direction, but Gaps Remain

The paper identifies two existing training-based approaches, each addressing one bottleneck:

Block-wise causal fine-tuning for caching. D2F (Wang et al., 2025) and Fast-dLLMv2 (Wu et al., 2025) fine-tune DLMs to adopt a block-wise causal attention mask, illustrated in Figure 2 (right). Under this mask, the sequence is divided into fixed-size blocks. The current block can attend to the prompt and all previously completed blocks (causal across blocks), plus all positions within the current block (bidirectional within-block). This enables native KV caching: once a block is finalized, its attention context is frozen, and its KV states can be cached and reused for all subsequent blocks without recomputation. This is a genuine architectural fix rather than a heuristic—the model is trained to operate under this attention pattern, so there is no approximation error from caching.

The gap is that these approaches do not address the step-count bottleneck. D2F is trained with a target generation length of 512 tokens (Section 5.1), requiring many refinement steps within each block. The model still expects incremental refinement; the causal mask just makes each step cheaper.

Objectives for faster convergence. D-Parallel (Chen et al., 2025) designs training objectives that encourage the model to make higher-confidence predictions earlier in the refinement process, promoting faster convergence. This addresses step count but not caching.

What's missing: a unified solution. No existing training-based method tackles both bottlenecks simultaneously through a single fine-tuning procedure. A model that is both block-wise causal (enabling native KV caching) and trained to take large jumps between states (reducing step counts) would achieve multiplicative speedups: fewer steps × lower cost per step.

How Consistency Modeling Provides the Missing Piece

The key conceptual insight of this paper is to adapt consistency modeling—a technique originally developed for continuous diffusion models in vision (Song et al., 2023)—to the discrete token-level refinement process of DLMs. The consistency principle is deceptively simple: any intermediate state along a diffusion trajectory should map directly to the final clean state, regardless of how much noise it contains. In the vision domain, this enabled generative models to produce high-quality images in one or two steps instead of hundreds.

Adapting this to discrete language modeling is non-trivial for several reasons. Discrete token trajectories do not follow the probability flow ODE that underpins continuous consistency models. The "noise" is not Gaussian but rather a pattern of [MASK] tokens with structured unmasking schedules. The learning signal must be defined over probability distributions on a discrete vocabulary rather than over continuous pixel values.

The paper's approach, detailed in Section 4, is to define a token-level decoding trajectory $\mathcal{T}_x$ (the sequence of partially refined states the teacher model visits during its standard 256-step generation process) and train a student to make jumps directly from earlier states to later states within this trajectory. Crucially, the student is simultaneously trained with three objectives:

Distillation: The student learns to predict the same token identities and probabilities that the teacher would assign at the block-completion state, supervised through forward KL divergence on the teacher's reconstructed logits (Equation 4). This is the "what should this token be?" signal.
Consistency: The student's predictions at a less-informed state (earlier in the block) are forced to match its own predictions at a more-informed state (the block-completion state) for positions that are still masked at both states (Equation 5). This is the "stay stable in your predictions across the refinement interval" signal—it teaches the model that it can safely make larger unmasking jumps without its predictions drifting.
DLM loss: The standard masked-denoising objective is retained as an auxiliary term (Equation 6) to prevent catastrophic forgetting of the base model's mask-prediction capability.

The block-wise causal attention mask (Figure 2, right) is applied during this fine-tuning. This means the student learns to operate under the attention pattern it will use at inference time—causal across blocks, bidirectional within the current block—and the consistency and distillation objectives are computed under this mask. There is no train-inference mismatch.

How This Paper Positions Itself

The paper positions CDLM as filling a specific gap in the DLM acceleration landscape (Section 1, contributions bullet points):

Relative to training-free methods: CDLM addresses both bottlenecks (steps and caching) through architectural and objective-level changes, not inference-time heuristics. The ablation in Table 4 demonstrates that step reduction through training is necessary—naive truncation without consistency training destroys quality.
Relative to prior training-based methods: CDLM is the first to combine block-wise causal fine-tuning (enabling native KV caching) with consistency-based multi-step jumping (reducing step counts) in a single, unified training procedure. D2F provides caching but not step reduction; D-Parallel encourages faster convergence but not caching; CDLM provides both.
Relative to AR models: The paper explicitly compares against AR throughput (Figures 3 and 4), showing CDLM can match or exceed equal-size AR models in tokens-per-second while maintaining competitive (though not always superior) accuracy. This is a deliberate positioning: CDLM is not claiming to beat AR models on absolute accuracy, but to make DLMs fast enough to be a viable alternative in deployment scenarios where throughput matters more than marginal accuracy differences, or where bidirectional context provides task-specific advantages.
Relative to the broader consistency modeling literature: The paper adapts the consistency concept to a setting—discrete token-level diffusion—where it has not been successfully applied before. Prior work explored consistency for Jacobi-style parallel decoding in AR models (Kou et al., 2024) and masked diffusion frameworks (Xu et al., 2025), but CDLM is the first to integrate it with block-wise causal DLMs and to demonstrate end-to-end speedups on practical math and coding benchmarks.

The paper's ambition is not to propose an entirely new model architecture but to show that a relatively lightweight fine-tuning procedure (8–16 hours on 4× A100s) can transform standard open-source DLMs from impractically slow to competitive with AR models, unlocking the parallel-generation benefits that have so far been confined to proprietary systems.

3. Technical Approach

3.1 Reader Orientation

CDLM is a fine-tuning recipe that converts a standard, fully-bidirectional diffusion language model (which is slow because it requires hundreds of refinement steps and cannot use KV caching) into a block-wise causal model that generates text much faster by finalizing multiple tokens per step and natively reusing cached key-value states across blocks. The core idea is to train a student model—initialized from the teacher's weights but with a causal-across-blocks attention mask—to make larger, stable jumps along the teacher's original refinement trajectory by simultaneously learning from teacher supervision (distillation), enforcing self-consistency between early and late states, and preserving the base model's masked-token prediction capability.

3.2 Big-Picture Architecture (Diagram in Words)

The CDLM system has five major components operating in two phases (training and inference):

Training Phase:

Teacher DLM (fully bidirectional): A standard, pre-trained DLM (Dream-7B-Instruct or LLaDA-8B-Instruct) with full bidirectional attention. It runs its normal, high-quality 256-step block-wise decoding on a collection of prompts to generate trajectories—sequences of partially unmasked states. For each prompt, it also records a hidden-state buffer capturing the last-layer representation at the moment each token is finalized.
Trajectory Dataset: The collection of $(x, \hat{\mathbf{y}}, \mathcal{T}_x, \mathbf{H}_x)$ quadruples, where $x$ is a prompt, $\hat{\mathbf{y}}$ is the ground-truth answer, $\mathcal{T}_x$ is the sequence of intermediate unmasking states the teacher visited, and $\mathbf{H}_x$ is the hidden-state buffer for logit reconstruction.
Student DLM (block-wise causal): Initialized from the teacher's weights but trained with a modified attention mask (Figure 2): causal across blocks (each block attends only to the prompt and previously completed blocks) but bidirectional within the current block. This is the model that will be deployed.
Three-Objective Training Loop (Algorithm 2): The student is fine-tuned on the trajectory dataset by jointly minimizing:
- Distillation loss: On tokens newly unmasked between a sampled state $y$ and its block-completion state $y^\star$ , the student's predicted distribution is pushed toward the teacher's distribution (reconstructed from the hidden-state buffer) via forward KL divergence.
- Consistency loss: On tokens still masked at both $y$ and $y^\star$ , the student's prediction at $y$ is forced to match its own (stop-grad) prediction at the more-informed $y^\star$ , again via forward KL divergence.
- DLM loss: A standard masked-denoising objective on ground-truth text, applied independently to prevent catastrophic forgetting.

Inference Phase: 5. Block-Wise Parallel Decoder with Confidence Thresholding: The trained student generates text block by block. For each block, starting from a fully masked state, it repeatedly runs the model and finalizes (unmasks) any token whose predicted probability exceeds a confidence threshold $\tau_{\text{conf}}$ . KV states for completed blocks are cached and reused. Generation terminates early if an <endoftext> token appears within a block.

3.3 Roadmap for the Deep Dive

First, the teacher trajectory collection procedure (Section 4.1): We need to understand what data the student learns from—the specific decoding configuration, the block-wise trajectory structure, and how hidden states are stored compactly for logit reconstruction. This is the foundation that the training objectives operate on.
Second, the three training objectives in detail (Section 4.2): We walk through each loss term—distillation, consistency, and DLM—defining the notation, stating the equations, explaining what each computes operationally and why the authors chose this specific form. This is the core technical contribution.
Third, the block-wise causal attention mask: We explain the architecture change that enables native KV caching and how it interacts with the training objectives. This is not a separate section but is woven into the discussion of the student model and the attention pattern.
Fourth, the inference procedure (Section 4.3): We describe how the trained student generates text—the block-wise loop, the confidence-thresholded parallel finalization within each block, and the early-stopping mechanism.
Finally, key design choices and hyperparameters: We summarize the critical decisions (forward vs. reverse KL, logit-space vs. embedding-space distillation, temperature augmentation, block size, loss weights) and their justifications.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical systems paper whose core idea is that a block-wise causal DLM fine-tuned with distillation and consistency objectives can achieve large reductions in inference latency and step count while preserving quality, addressing both the caching and step-count bottlenecks of standard DLMs.

4.1 Trajectory Collection for Diffusion Language Models

The student is trained on offline trajectories generated by the teacher DLM, not on on-policy rollouts. This is a critical design choice: on-policy generation during training would require running the slow DLM sampling loop inside the training loop, which the authors note is currently infeasible ("slow generation limits the feasibility of closed-loop training," Appendix B). Instead, trajectories are pre-computed once and stored.

Teacher Decoding Configuration

The teacher DLM (Dream-7B-Instruct or LLaDA-8B-Instruct) runs with a specific configuration designed to be its "most performant operating point" (Section 4.1):

Generation length $L_g = 256$ : The model generates up to 256 tokens. This is the target length used in all experiments and matches the baseline DLM configurations in Section 5.
Total refinement steps $N = L_g = 256$ : The teacher uses exactly one step per token of generation. This is the default high-quality setting for these models; the paper states that "the number of sampling steps equal to the generation length... places the teacher at its most performant operating point and yields higher-quality trajectories" (Section 4.1).
Block size $B = 32$ : Decoding is block-wise—the sequence is divided into blocks of 32 tokens, and within each block, tokens are unmasked one per step following the standard low-confidence remasking policy. This means exactly one token in the current block is finalized at each step, and the teacher only operates on the current block. The block structure is necessary because allowing the model to unmask tokens anywhere in the sequence can lead to unnatural early generation (e.g., generating an <endoftext> token at position 10 when the answer is 200 tokens long). Block-wise decoding constrains the generation to proceed from left to right at the block level, even though it is bidirectional within the block.
Temperature augmentation $\tau \in \{0.0, 0.5\}$ : For each prompt, two trajectories are generated—one with greedy decoding ( $\tau = 0.0$ ) and one with modest stochasticity ( $\tau = 0.5$ ). The paper notes that in DLMs, temperature "not only influences token selection... [it] also changes the order in which tokens are revealed" (Section 4.1, Data Augmentation). This is a DLM-specific property: because the model decides which tokens to unmask based on confidence (which changes with temperature), the same prompt can produce different unmasking orders at different temperatures, providing trajectory diversity without changing the final answer. Temperature $\tau = 1.0$ was found to "often destabilize the reasoning chain and yield incorrect conclusions" (Appendix A.1, Figure 5), so it is excluded.

Trajectory Structure

For a single prompt $x$ , the teacher produces a trajectory $\mathcal{T}_x$ , which is a sequence of $(N+1)$ states:

$\mathcal{T}_x = (\mathbf{x}_{t_0}, \mathbf{x}_{t_1}, \ldots, \mathbf{x}_{t_N}), \quad t_k = 1 - \frac{k}{N}$

where $\mathbf{x}_{t_0} = \mathbf{x}_1$ is the fully masked answer sequence at diffusion time $t=1$ (step 0), $\mathbf{x}_{t_N} = \mathbf{x}_0$ is the fully unmasked clean text at $t=0$ (step $N$ ), and each intermediate $\mathbf{x}_{t_k}$ is a partially refined sequence with exactly $k$ tokens finalized (since the teacher finalizes exactly one token per step under $N = L_g$ ).

Each $\mathbf{x}_{t_k}$ is a sequence of length $L_g$ (plus prompt length) where some token positions contain actual tokens and others contain [MASK] tokens. The set of unmasked positions grows monotonically: once a token is unmasked, it stays unmasked and its identity never changes. This is a defining property of masked diffusion models—there is no "re-noising" or re-masking of tokens.

Hidden-State Buffer for Logit Reconstruction

For white-box distillation, we need the teacher's output distribution (logits or probabilities) at specific states. Storing the full logits for every step would be prohibitively expensive: the logit dimensionality is $|V| \approx 128,000$ (vocabulary size), and we have $256$ steps $\times$ thousands of prompts, yielding terabytes of data.

The paper uses a compact alternative: instead of storing logits, store the teacher's last-layer hidden state at the moment each token is finalized (Appendix A.1, "Hidden-state buffer"). The logic is:

The teacher's output logits are computed as $\ell = \text{lm\_head}(\mathbf{h})$ , where $\mathbf{h} \in \mathbb{R}^d$ is the last hidden state and $\text{lm\_head}$ is the final linear projection (frozen during fine-tuning).
When a token at position $i$ is finalized during the teacher's decoding, we record the hidden state $\mathbf{h}_i \in \mathbb{R}^d$ at that position and write it into a buffer $\mathbf{H}_x \in \mathbb{R}^{L_g \times d}$ .
During training, when the student needs the teacher's distribution at a particular token position, we reconstruct it by applying the same $\text{lm\_head}$ to the stored hidden state: $\ell_i^{(T)} = \text{lm\_head}(\mathbf{H}_x[i])$ , $p_i^{(T)} = \text{softmax}(\ell_i^{(T)})$ .

Since $d \approx 4,096$ and $|V| \approx 128,000$ , this yields roughly a 30× storage reduction compared to storing full logits (the paper states "using hidden states... gives roughly a 30× reduction in storage," Appendix A.1). The reconstructed logits are exact (no approximation), assuming the $\text{lm\_head}$ is deterministic and frozen—which it is, since only the Transformer backbone is fine-tuned.

Data Augmentation and Dataset Composition

The final training dataset $\mathcal{D}$ consists of quadruples $(x, \hat{\mathbf{y}}, \mathcal{T}_x, \mathbf{H}_x)$ , where:

$x$ : the prompt
$\hat{\mathbf{y}}$ : the ground-truth answer (used only for the DLM auxiliary loss, not for distillation)
$\mathcal{T}_x$ : the teacher's 257-state trajectory (including initial fully masked and final fully unmasked)
$\mathbf{H}_x$ : the hidden-state buffer of shape $L_g \times d$

For CDLM–Dream, the prompts come from two HuggingFace datasets (lansechen_easy_2025 and lansechen_hard_2025), which are filtered subsets of Bespoke-Stratos-17k with Qwen2.5-7B responses as ground truth. Prompts with length > 512 tokens are filtered out. With two temperature settings, this yields approximately 15k trajectory pairs.

For CDLM–LLaDA, an additional 7.5k math-style prompts from the DParallel dataset are included (with Qwen2.5-7B generating the reference answers), bringing the total to approximately 30k trajectory pairs. The paper notes that "dataset selection is crucial; overly rigid formats (e.g., multiple-choice math) tend to hurt the model's ability to generalize" (Appendix A.1).

4.2 The Three Training Objectives

The student $q_\phi(\cdot \mid x)$ is initialized from the teacher's weights but trained with a block-wise causal attention mask (Figure 2, right). This mask divides the (prompt + generation) sequence into blocks of size $B = 32$ . The attention pattern is:

Within the current block: full bidirectional attention—every token in the block attends to every other token in the same block.
Across blocks: causal—the current block attends to all previous blocks (and the prompt) but NOT to future blocks. This mirrors autoregressive causality but at the block level rather than the token level.
Prompt: the prompt is visible to all blocks (it is treated as "block 0").

This mask is applied during both training and inference, so the student learns to operate under the exact attention pattern it will use at deployment. There is no train-inference mismatch.

The training loop (Algorithm 2) proceeds as follows for each sampled quadruple:

Sample a trajectory $\mathcal{T}_x$ from the dataset.
Sample a starting step $t_{\text{start}}$ (corresponding to a state $y = \mathbf{x}_{t_{\text{start}}}$ ).
Compute the ending step $t_{\text{end}} = \min(N, \lceil t_{\text{start}} / B \rceil \cdot B)$ , which is the step at which the block containing $t_{\text{start}}$ is fully completed. Let $y^\star = \mathbf{x}_{t_{\text{end}}}$ be the block-completion state.
Retrieve the teacher's hidden states $\mathbf{H}_x$ for positions that are newly unmasked between $y$ and $y^\star$ .
Compute the three losses and update $\phi$ .

The key sampling detail is that $t_{\text{end}}$ is always the end of the current block, meaning $y$ and $y^\star$ are at most $B = 32$ steps apart. This means the consistency objective operates within a single block's refinement window, not across blocks. The model learns to jump from an early state within a block to the block's completion state.

Notation for the Training Objectives

We first establish the shared notation (Section 4.2, "Notation"):

$y$ : a state sampled from the teacher's trajectory $\mathcal{T}_x$ , representing a partially unmasked sequence at some intermediate step.
$y^\star$ : the block-completion state—the state obtained by fully unmasking the block that $y$ belongs to. $y$ and $y^\star$ are at most $B = 32$ steps apart along the trajectory.
$\mathcal{U}_y = \{i \mid y_i = \texttt{[MASK]}, y^\star_i \neq \texttt{[MASK]}\}$ : the set of token positions that are newly unmasked between $y$ and $y^\star$ —positions that were masked in $y$ but get filled in by the time the block is complete.
$\mathcal{S}_y = \{i \mid y_i = \texttt{[MASK]}, y^\star_i = \texttt{[MASK]}\}$ : the set of token positions that are still masked at $y^\star$ —positions that are not in the current block and won't be finalized until later blocks.
$q_\phi(\cdot \mid y, x)_i$ : the student's predicted probability distribution over the vocabulary at position $i$ , conditioned on the partially unmasked state $y$ and prompt $x$ .
$p_i^{(T)} = \text{softmax}(\text{lm\_head}(\mathbf{H}_x[i]))$ : the teacher's predicted distribution at position $i$ , reconstructed from the stored hidden state.
$q_{\phi^-}(\cdot \mid y^\star, x)_i$ : the student's predicted distribution at position $i$ conditioned on the block-completion state $y^\star$ , with the gradient detached (stop-grad). This means $q_{\phi^-}$ is treated as a constant target; the loss only updates $\phi$ through the $q_\phi(\cdot \mid y, x)$ term.

Distillation Loss (Equation 4)

$\mathcal{L}_{\text{Distillation}} = \mathbb{E}_{(x,\mathcal{T}_x,\mathbf{H}_x) \sim \mathcal{D}} \, \mathbb{E}_{y \sim \mathcal{T}_x} \left[ \frac{1}{|\mathcal{U}_y|} \sum_{i \in \mathcal{U}_y} D_{\mathrm{KL}}\!\left( p_i^{(T)} \,\big\|\, q_\phi(\,\cdot \mid y, x)_i \right) \right]$

where $D_{\mathrm{KL}}(P \| Q) = \sum_{v} P(v) \log \frac{P(v)}{Q(v)}$ is the forward KL divergence from the teacher distribution $P$ to the student distribution $Q$ , and the outer expectations are over the dataset and over states uniformly sampled from the trajectory.

What it computes: For every position $i$ that transitions from [MASK] to unmasked between state $y$ and block-completion $y^\star$ , we compute the KL divergence between the teacher's predicted distribution at that position (reconstructed from the hidden-state buffer) and the student's predicted distribution—both conditioned on the same less-informed state $y$ . We average these per-position divergences across $\mathcal{U}_y$ (the positions the student should be trying to finalize), then average over randomly sampled states $y$ and over the dataset. The loss is zero if the student exactly matches the teacher's distribution at every newly unmasked position; it is positive otherwise.

In operational terms: given the same partially unmasked input $y$ , the student is being trained to output the same probabilities as the teacher would for the tokens that are about to be finalized. This directly supervises the multi-token finalization that the student needs to perform at inference time—the student sees $y$ and learns to predict what the block-completion state $y^\star$ will look like at those positions.

Why this form (forward KL, logit space, and $\mathcal{U}_y$ masking):

Forward KL: The authors state that "the forward KL divergence yielded more stable, better-calibrated training dynamics and more monotonic convergence than the reverse KL divergence for distribution matching" (Appendix A.2). Forward KL ( $D_{\mathrm{KL}}(p \| q)$ ) is "mean-seeking"—it penalizes the student strongly when the teacher assigns high probability to a token the student assigns low probability to, encouraging the student to cover all plausible tokens the teacher considers. Reverse KL ( $D_{\mathrm{KL}}(q \| p)$ ) is "mode-seeking" and would encourage the student to collapse onto a single high-probability token, which is undesirable for a model that needs calibrated probabilities for confidence-based decoding.
Logit-space distillation: The paper also notes that "distillation in logit space outperformed embedding-space distillation using mean squared error" (Appendix A.2). This is expected—logit-space distillation via KL divergence provides richer gradient information about the relative probabilities of all tokens, while MSE on hidden states only captures a point estimate of the representation.
Restriction to $\mathcal{U}_y$ : The loss is computed ONLY on positions that are [MASK] in $y$ but not in $y^\star$ —the positions the student should be learning to unmask. Positions that are already unmasked in $y$ have deterministic, known values (they never change in masked diffusion) and don't need supervision. Positions in $\mathcal{S}_y$ (still masked at $y^\star$ ) are handled by the consistency loss instead. This clean separation between "positions I should finalize now" (distillation) and "positions I should think about consistently" (consistency) is a key architectural insight.
Hidden-state reconstruction: The teacher's distribution $p_i^{(T)}$ is NOT computed by running the teacher model on $y$ (which would be expensive). Instead, it is reconstructed directly from the pre-stored hidden-state buffer. This means the distillation loss provides multi-token finalization supervision WITHOUT requiring the teacher to be in the training loop—a crucial efficiency that makes the approach practical.

Consistency Loss (Equation 5)

$\mathcal{L}_{\text{Consistency}} = \mathbb{E}_{(x,\mathcal{T}_x) \sim \mathcal{D}} \, \mathbb{E}_{y \sim \mathcal{T}_x} \left[ \frac{1}{|\mathcal{S}_y|} \sum_{i \in \mathcal{S}_y} D_{\mathrm{KL}}\!\left( q_{\phi^-}(\,\cdot \mid y^\star, x)_i \,\big\|\, q_\phi(\,\cdot \mid y, x)_i \right) \right]$

where $q_{\phi^-}$ denotes the student's prediction with a stop-gradient applied (detached from the computation graph), and $D_{\mathrm{KL}}$ is again the forward KL divergence.

What it computes: For every position $i$ that is STILL masked in BOTH $y$ and $y^\star$ (positions outside the current block), we compute the forward KL divergence between the student's prediction at the more-informed state $y^\star$ (treated as a fixed target via stop-grad) and the student's prediction at the less-informed state $y$ . We average across $\mathcal{S}_y$ (the still-masked positions), then average over states and the dataset.

In operational terms: the student looks at a partially completed state $y$ and makes predictions about all still-masked positions. It then looks at the more-complete state $y^\star$ (where the current block is fully unmasked, providing additional context) and makes predictions about the same still-masked positions. The consistency loss says: "Your predictions at $y$ should match your predictions at $y^\star$ for the positions you haven't filled in yet." This teaches the student that its predictions about future tokens should be stable across the refinement of the current block—it can safely make larger jumps without its predictions about unmasked positions drifting unpredictably.

Why this form (consistency on still-masked positions with stop-grad target):

Restriction to $\mathcal{S}_y$ : The paper explicitly states that "since DLMs are trained to predict only [MASK] tokens, restricting the loss to still-masked indices avoids ill-defined supervision" (Section 4.2). DLMs are never trained to predict tokens that are already unmasked—those are deterministically known and don't need prediction. Including already-unmasked positions would provide a meaningless or harmful training signal. Similarly, positions in $\mathcal{U}_y$ (newly unmasked between $y$ and $y^\star$ ) are handled by the distillation loss; the consistency loss focuses on a complementary set.
Stop-gradient on $y^\star$ : This follows the standard practice from consistency models (Song et al., 2023). If gradients were allowed to flow through $q_{\phi^-}(\cdot \mid y^\star)$ , the model could trivially minimize the loss by collapsings BOTH predictions to a degenerate distribution (e.g., uniform) rather than learning genuine temporal consistency. The stop-grad makes $q_{\phi^-}$ an anchor—the model must pull its less-informed prediction toward a more-informed target without changing the target itself.
Forward KL (student-II-to-student-I, not vice versa): The ordering matters. The loss is $D_{\mathrm{KL}}( \text{target} \| \text{current} )$ , where the target is the more-informed prediction (which should be more accurate) and the current is the less-informed prediction (which we are trying to improve). This "mean-seeking" direction encourages the student at $y$ to cover all modes of its own more-informed prediction, preventing it from becoming overconfident about tokens it hasn't yet seen in context.
Why consistency helps step reduction: The core problem with naive step reduction (forcing the model to finalize multiple tokens per step without retraining) is that the model's predictions at early states are poorly calibrated for tokens it wasn't planning to unmask yet—it was trained to make incremental, one-token-at-a-time decisions. The consistency loss directly addresses this: by enforcing that predictions stay stable as context improves, the model learns that it CAN finalize tokens earlier because its predictions won't change much anyway. The distillation loss tells it WHAT to predict; the consistency loss tells it WHEN it's safe to predict it early.

The paper's ablation (Table 3, row 2) demonstrates that consistency alone is catastrophic: with $w_{\text{distill}} = 0$ and $w_{\text{cons}} = 1.0$ , GSM8K accuracy collapses to 6.9% and HumanEval-Instruct to 0.0%. Consistency without teacher supervision leads to degenerate self-consistency where the model learns to produce self-consistent but wrong predictions. The synergy comes from coupling consistency WITH distillation—the teacher provides the "what" and consistency provides the "when."

DLM Loss (Equation 6)

$\mathcal{L}_{\text{DLM}} = -\mathbb{E}_{(x,\hat{\mathbf{y}}) \sim \mathcal{D}} \, \mathbb{E}_t \left[ \frac{1}{t} \sum_{i=1}^{L_g} \mathbf{1}\!\big[\hat{y}_{t,i} = \texttt{[MASK]}\big] \, \log q_\phi\big(\hat{y}_i \mid \hat{\mathbf{y}}_t, x\big) \right]$

where $t \sim \mathcal{U}[0,1]$ is a randomly sampled masking ratio, $\hat{\mathbf{y}}_t$ is the ground-truth answer with each token independently masked with probability $t$ , $\mathbf{1}[\cdot]$ is the indicator function selecting only the masked positions, and $\log q_\phi(\hat{y}_i \mid \hat{\mathbf{y}}_t, x)$ is the student's log-probability assigned to the correct token $\hat{y}_i$ given the masked input.

What it computes: This is the standard masked-denoising (MDM) pre-training objective from Dream and LLaDA, applied during fine-tuning. For each ground-truth answer $\hat{\mathbf{y}}$ , we randomly mask a fraction $t$ of the tokens, feed the partially masked sequence to the student, and compute the cross-entropy loss on predicting the identities of the masked tokens, weighted by $1/t$ (the inverse of the masking rate to normalize across different $t$ values). The loss encourages the student to maintain the fundamental "fill in blanks" capability that all DLMs are built on.

In operational terms: this is an auxiliary regularizer. The distillation and consistency losses train the student to make fast, stable jumps along the teacher's specific refinement trajectories, which could cause it to forget the general mask-prediction skill it was pretrained with. The DLM loss, computed independently on ground-truth text with random masking (NOT on trajectory states), keeps this base capability intact.

Why this form:

Independent from trajectory data: The DLM loss uses only the ground-truth answer $\hat{\mathbf{y}}$ and the prompt $x$ , NOT the teacher trajectory $\mathcal{T}_x$ . This is important because it prevents the student from overfitting to the specific unmasking order the teacher happened to produce—the DLM loss provides a general-purpose signal about what tokens are plausible given arbitrary masking patterns.
Weighting by $1/t$ : At small masking rates (most tokens visible), the model has abundant context and the loss per position is small (easy task, many positions). At large masking rates (few tokens visible), the task is hard but there are many masked positions. The $1/t$ normalization ensures that all masking ratios contribute roughly equally to the gradient regardless of the number of masked tokens, preventing the objective from being dominated by high- $t$ regimes.
Small weight in the combined objective: The DLM loss weight is small relative to distillation: $w_{\text{dlm}} = 0.01$ for Dream and $0.1$ for LLaDA (the LLaDA weight is higher because "the DLM loss on LLaDA has a smaller absolute scale," Section 5.3.1). The distillation loss is the primary training signal; DLM loss is fine-tuning stabilization. The ablation in Table 3 (rows 3 vs. 4) shows that removing the DLM loss raises HumanEval-Instruct (42.7 → 48.2) but lowers GSM8K (74.1 → 73.3), indicating that the DLM loss helps preserve math reasoning ability at a small cost to code generation; the authors note that "general coding performance can be recovered with longer training" (Section 5.3.1).

Combined Objective (Equation 7)

$\mathcal{L}(\phi) = w_{\text{distill}} \, \mathcal{L}_{\text{Distillation}} + w_{\text{cons}} \, \mathcal{L}_{\text{Consistency}} + w_{\text{dlm}} \, \mathcal{L}_{\text{DLM}}$

The final loss is a weighted sum of the three components. The default weights are $(w_{\text{distill}}, w_{\text{cons}}, w_{\text{dlm}}) = (1.0, 0.5, 0.01)$ for Dream and $(1.0, 0.5, 0.1)$ for LLaDA.

The choice of $w_{\text{cons}} = 0.5$ (half the distillation weight) is empirical: the authors "find that $w_{\text{cons}} = 0.5$ [provides] a good balance between speed and quality" (Section 5.3.1). The distillation loss is the primary driver of multi-token finalization accuracy; the consistency loss stabilizes the predictions so that the multi-token finalization can happen safely. Too much consistency weight relative to distillation would encourage the model to be self-consistent but drift from the teacher's distribution (as seen in the consistency-only collapse).

4.3 Inference

At inference time, the trained CDLM student decodes text using a block-wise procedure with confidence-thresholded parallel finalization:

Block-wise loop: The generation target is divided into blocks of size $B = 32$ . The model processes one block at a time, starting from block 1 (the first 32 answer tokens). Within each block:
- Initialize all positions in the current block to [MASK].
- The prompt and all previously completed blocks are fully unmasked, and their KV states are cached. The current block attends bidirectionally to itself and causally to all prior content (Figure 2, right).
Confidence-thresholded parallel finalization: At each refinement step within the current block:
- Run the model on the current state to get predicted probability distributions $q_\phi(\cdot \mid \text{current\_state}, x)_i$ for all [MASK] positions in the current block.
- For each masked position $i$ , extract the model's most confident prediction: $p_{\max}^i = \max_v q_\phi(v \mid \text{current\_state}, x)_i$ and $\hat{y}_i = \arg\max_v q_\phi(v \mid \text{current\_state}, x)_i$ .
- Threshold: if $p_{\max}^i \geq \tau_{\text{conf}}$ , finalize position $i$ by replacing [MASK] with $\hat{y}_i$ . If $p_{\max}^i < \tau_{\text{conf}}$ , leave it masked.
- The default threshold is $\tau_{\text{conf}} = 0.9$ , chosen as "a robust default that balances speed and quality across tasks" (Section 5.3.3).
- Repeat until either (a) all positions in the current block are unmasked, or (b) an <endoftext> token appears (see early stopping below).
Block transition: Once the current block is fully processed (all positions finalized or block capacity reached), its KV states are cached for subsequent blocks. The model moves to the next block.
Early stopping: If the model generates an <endoftext> token within the current block, generation terminates immediately—no further blocks are processed. This is "analogous to AR early stopping" (Section 4.3) and prevents the model from generating padding or trailing content after the natural answer endpoint. This is enabled by the block-wise causal structure: once an <endoftext> appears, there is no need to continue because the model has signaled completion.

The confidence threshold $\tau_{\text{conf}}$ directly controls the speed-quality tradeoff. The ablation in Table 5 shows:

$\tau_{\text{conf}} = 0.95$ (conservative): higher accuracy (e.g., 51.2 on HumanEval), lower throughput (34.4 TPS), higher latency (2.8s).
$\tau_{\text{conf}} = 0.85$ (aggressive): slightly lower accuracy (48.2 on HumanEval), higher throughput (47.3 TPS), lower latency (2.0s).
The paper notes this is a "monotonic speed trend" (Section 5.3.3): increasing threshold → fewer tokens finalized per step → more steps → slower but higher quality.

This inference procedure is deliberately simple: "We intentionally avoid additional heuristics such as inter-block parallelism... which introduce extra hyperparameters whose optimal values are task- and domain-dependent" (Section 4.3). The goal is to have a single, robust inference algorithm with one tunable parameter ( $\tau_{\text{conf}}$ ) rather than a complex multi-hyperparameter scheme.

Summary of Key Design Choices and Justifications

Offline trajectory collection over on-policy generation: On-policy training is currently infeasible due to DLM sampling speed; offline trajectories can be reused across epochs. The static dataset also enables careful curation and temperature augmentation.
Hidden-state buffer over logit storage: 30× storage reduction makes the dataset manageable on a single server (25–30 GiB per shard for 15k samples).
Forward KL over reverse KL: Forward KL is mean-seeking and produces better-calibrated distributions for confidence-based decoding.
Logit-space distillation over embedding-space MSE: Richer gradient information about relative token probabilities.
Block size $B = 32$ : Matches the baseline block-wise decoding convention; enables at most 32× step reduction per block. The paper does not ablate this choice.
Loss weight ratio $w_{\text{distill}}:w_{\text{cons}}:w_{\text{dlm}} = 1.0:0.5:0.01$ (Dream): Distillation is the primary driver; consistency stabilizes; DLM prevents forgetting. Ablation in Table 3 supports this balance.
Consistency restricted to $\mathcal{S}_y$ and distillation to $\mathcal{U}_y$ : Clean separation between "positions I'm about to finalize" (learn from teacher) and "future positions I need to think about stably" (self-consistency).
Block-wise causal mask applied during both training and inference: No train-inference mismatch; the model learns to operate under exactly the attention pattern used at deployment.
Confidence threshold $\tau_{\text{conf}} = 0.9$ : Empirically robust default; Table 5 shows 0.90 balances speed and accuracy across GSM8K-CoT and HumanEval-Instruct.
LoRA fine-tuning rather than full fine-tuning: Enables training on 4× A100 GPUs in 8–16 hours; the configurations (rank 32/64, targeting attention and MLP modules) are standard parameter-efficient fine-tuning choices.

4. Key Insights and Innovations

Innovation 1: Consistency Modeling as a Unified Solution to Both DLM Inference Bottlenecks

The central intellectual move in CDLM is recognizing that two seemingly independent DLM acceleration problems—excessive refinement steps and the inability to use KV caching—can be addressed simultaneously through a single training paradigm by integrating consistency modeling with block-wise causal attention.

The conceptual gap this fills. Prior work treated these as separate problems requiring separate solutions. D2F (Wang et al., 2025) and Fast-dLLMv2 (Wu et al., 2025) restructured attention for caching but did nothing about step counts. Fast-dLLM's parallel decoding (Wu et al., 2025) and D-Parallel (Chen et al., 2025) reduced step counts but operated on fully bidirectional architectures that remained incompatible with KV caching. The dominant assumption was that you pick your bottleneck and optimize for it: either make each step cheaper (caching) or make fewer steps (parallel decoding), but not both through a single architectural change.

CDLM's insight is that block-wise causality and multi-step jumping are synergistic when trained jointly. The block-wise causal mask creates natural boundaries where KV caching becomes possible (once a block is complete, its context is frozen). The consistency objective teaches the model to cross those boundaries in fewer steps. Neither alone would achieve the full speedup: block-wise causality without consistency still requires many steps per block (D2F); consistency without causality eliminates steps but still requires recomputing attention for the full sequence at each step. The results in Tables 1 and 2 bear this out: CDLM achieves multiplicative speedups (e.g., 11.2× latency reduction on Dream GSM8K-CoT) that exceed what either axis achieves alone—Fast-dLLM (Parallel) provides step reduction (4.8×) but no caching, dLLM-Cache provides caching but no step reduction (1.0× step change), while CDLM provides both (5.8× steps + caching).

This is more than an engineering combination. It represents a diagnostic insight: the two bottlenecks are not independent problems to be patched sequentially but manifestations of a shared underlying cause—the model was trained to take many small steps under full bidirectional attention. Address the training objective and the attention pattern together, and both symptoms resolve. The key evidence is Table 4: forcing a standard DLM to use CDLM-level step budgets without retraining causes catastrophic accuracy collapse (Dream drops from 79.1 → 41.8 on GSM8K-CoT), confirming that step reduction requires training, not just inference heuristics, and that the consistency+distillation objectives specifically teach the model to make those larger jumps stably.

Fundamental vs. incremental. This is a fundamental integration rather than a novel primitive. Neither consistency modeling (Song et al., 2023) nor block-wise causal attention (Wang et al., 2025) is new individually. The intellectual contribution is recognizing that coupling them creates a regime where each amplifies the other's benefit, and demonstrating that the coupling works through a specific three-objective training formulation (Equations 4–6) rather than through independent modifications.

Innovation 2: Trajectory-Based Supervision as a Bridge Between Continuous Consistency and Discrete Token Sequences

Adapting consistency modeling from continuous diffusion (where it operates on probability flow ODEs over real-valued states) to discrete masked diffusion (where states are token sequences with [MASK] tokens and transitions are deterministic token unmasking events) requires rethinking what "consistency" means. The paper's key conceptual move is to ground consistency not in solving an ODE but in a specific, concrete data structure: the teacher's decoding trajectory.

What the field did before. Prior work on discrete consistency (Kou et al., 2024; Xu et al., 2025) for language modeling focused on Jacobi-style parallel decoding in autoregressive models, where intermediate states are sequences of draft tokens that get refined. The trajectory in those settings is implicit and defined by the inference algorithm, not by a pre-existing diffusion process. In masked DLMs, however, there is a natural, pre-defined trajectory: the sequence of partially unmasked states the teacher visits during its standard 256-step generation. Each step unmaskes exactly one token in the current block when run under the $N = L_g$ configuration.

CDLM's innovation is to make this trajectory an explicit training resource rather than an abstract probabilistic model. The student learns from concrete $(y, y^\star)$ pairs extracted from real teacher decoding runs, where $y$ and $y^\star$ are actual states the teacher visited (separated by at most $B = 32$ steps). This has two major advantages over abstract formulations:

The supervision is grounded in realistic state transitions. The student learns to jump between states that actually occur during high-quality generation, not between arbitrarily chosen noise levels. The teacher's block-wise decoding with $N = L_g$ places it at "its most performant operating point" (Section 4.1), so the trajectories represent near-optimal refinement paths. The student learns to compress these optimal paths into fewer steps.
The discrete structure of blocks creates natural consistency intervals. Unlike continuous consistency, where the step size between $y$ and $y^\star$ is a continuous parameter, CDLM's consistency intervals are defined by block boundaries: $y$ is any state within the current block's refinement, and $y^\star$ is always the block-completion state. This makes the consistency target well-defined and interpretable—"learn to predict the block-completion state from any intermediate state within the block." The distillation loss handles the tokens that change within this interval ( $\mathcal{U}_y$ ); the consistency loss handles tokens that don't yet change ( $\mathcal{S}_y$ ).

The significance beyond performance. This trajectory-as-training-data framing has a practical consequence that the paper only hints at: it makes the training procedure largely independent of the specific DLM architecture. Any DLM that produces trajectories can be used as a teacher; any student with a compatible vocabulary and tokenizer can be trained. The paper demonstrates this by applying the identical procedure to two different model families (Dream and LLaDA) with different architectures, different pre-training procedures, and different sensitivities to fine-tuning. The CDLM recipe transfers despite these differences, suggesting the trajectory-based consistency framework captures something general about DLM refinement rather than being tied to a specific model's behavior.

The hidden-state buffer for logit reconstruction (Section 4.1) is a practical enabler of this framing, but its intellectual significance is that it decouples the teacher from the training loop without approximation error. The student gets exact teacher distributions without the teacher being present during training—a form of lossless compression of teacher knowledge that makes trajectory-based supervision scalable.

Fundamental vs. incremental. This is a conceptual reframing of consistency modeling for discrete sequences. The mathematical machinery (forward KL, stop-gradient targets) is inherited from continuous consistency models, but the decision to anchor the consistency objective in real teacher trajectories—and to structure the consistency interval around block boundaries—is a genuine intellectual contribution that makes consistency practically applicable to masked DLMs where it had not previously been demonstrated at scale.

Innovation 3: Diagnostic Decomposition of the Speed-Quality Tradeoff Through Loss Ablation

The ablation study in Table 3 is not merely a hyperparameter sweep—it is a diagnostic experiment that decomposes what each training objective contributes and reveals a previously undocumented failure mode: consistency without distillation leads to complete model collapse.

The diagnostic finding. When the consistency loss is used in isolation ( $w_{\text{distill}} = 0$ , $w_{\text{cons}} = 1.0$ ), the student learns to produce self-consistent predictions (its early-state predictions match its late-state predictions) but these predictions bear no relation to correct answers. GSM8K accuracy collapses to 6.9% and HumanEval-Instruct to 0.0%. This is not a gradual degradation—it is catastrophic failure. The model achieves perfect temporal consistency for wrong predictions.

This negative result is intellectually significant because it reveals that self-consistency is not an intrinsic good in discrete token spaces. In continuous diffusion models, consistency along the probability flow ODE guarantees that the endpoint is the data distribution (Song et al., 2023). In discrete masked diffusion, there is no analogous guarantee: the model can learn to map every intermediate state to an arbitrary (but self-consistent) final state. The distillation loss provides the grounding—it anchors the self-consistency to the teacher's actual token identity decisions, ensuring that "consistent" also means "correct."

Why this matters beyond this paper. This finding has implications for anyone attempting to apply consistency-style objectives to discrete domains. It suggests that pure self-consistency losses (without external supervision signal) are likely to fail in discrete token spaces, and that distillation or ground-truth supervision is not optional but essential. The paper does not make this theoretical claim explicitly, but the empirical evidence in Table 3 is stark enough to serve as a warning for future work.

The synergy insight. The coupling of distillation and consistency (rows 3 and 5 in Table 3) produces both faster convergence (fewer steps to completion) and stable or improved accuracy compared to distillation alone (row 1). For GSM8K: distillation-only gets 73.2% in 46.7 steps; distillation+consistency gets 74.1% in 49.4 steps ( $w_{\text{cons}} = 1.0$ ) or 75.1% in 48.0 steps ( $w_{\text{cons}} = 0.1$ ). The consistency loss is not just preventing collapse—it is actively improving the quality of multi-token finalization by making predictions more stable across the refinement window. This is a non-trivial synergy: the combination performs better than either component alone, and the mechanism (temporal stability enabling larger, more confident unmasking jumps) is distinct from what either loss would achieve independently.

Fundamental vs. incremental. This is a diagnostic contribution rather than a methodological one. The loss decomposition experiment qualifies as a genuine insight because it characterizes a novel failure mode (consistency collapse) and establishes a necessary condition for discrete consistency training to work (grounding through distillation). The finding is robust because it appears in both tested models and across both math and code benchmarks.

Innovation 4: Block-Wise Causality as a Native, Not Heuristic, Solution to DLM Caching

The distinction between CDLM's native KV caching (achieved through block-wise causal attention fine-tuning) and prior approximate caching methods (dLLM-Cache, Fast-dLLM Dual Cache) is not just an implementation detail—it represents a fundamentally different philosophy about how to solve the caching problem.

Two philosophies. Training-free caching methods (Section 2.2) adopt a patching philosophy: take a model trained for full bidirectionality and modify its inference behavior to simulate causality through heuristics—periodically refreshing cached regions, approximating attention over stale KV states, or maintaining dual caches for stable and active regions. These methods work because the sequence changes slowly during refinement (most tokens, once unmasked, are stable), so stale KV states are usually good enough. But they introduce approximation error that can accumulate over long refinement chains and require careful tuning of refresh schedules, block sizes for caching, and staleness thresholds.

CDLM adopts an architectural philosophy: modify the model's attention pattern during training so that genuine causality emerges as a learned property. The block-wise causal mask (Figure 2, right) is applied during both training and inference. The model never learns to depend on future blocks for its within-block predictions, so there is no approximation error when those future blocks are masked or absent at inference time. The KV cache is exact—not approximate—because the attention pattern is inherently causal across blocks.

Evidence for the benefit of native over heuristic caching. The comparison in Tables 1 and 2 supports this distinction. Fast-dLLM (Parallel + Dual Cache) achieves latency reductions comparable to CDLM on some benchmarks (e.g., 2.5s vs. 2.1s on Dream GSM8K-CoT) but does so with more refinement steps (60.8 vs. 44.1) and slightly lower accuracy (77.3 vs. 78.8). The dual-cache heuristic reduces per-step cost but cannot reduce the step count—the base model still expects many small refinements. CDLM's native caching enables both lower per-step cost (through exact KV reuse) AND fewer steps (through consistency training), and the two mechanisms compound. The fact that the step count drops by an additional ~16 steps when moving from Fast-dLLM (Par.+D.C.) to CDLM, despite both using confidence-thresholded decoding, is direct evidence that training-based consistency (not just caching) is driving the additional step reduction.

The training decision that makes this possible. The authors' choice to apply the block-wise causal mask during the entirety of CDLM training—not just during inference—is conceptually important. It means the distillation and consistency objectives are computed under the same attention constraints the model will face at deployment. The student learns to predict token identities and maintain temporal consistency given only access to the current and previous blocks, not future ones. This eliminates a common failure mode in model compression and distillation: the student learns to exploit information that will be available during training but not at inference (here, future-block context that the fully bidirectional teacher can see but the block-wise causal student cannot).

Some information loss is inevitable: a block-wise causal student cannot attend to future blocks, so it has strictly less context than the fully bidirectional teacher. The distillation objective compensates by providing teacher-level supervision for tokens in the current block, effectively transferring some of the cross-block reasoning into the student's within-block predictions. The fact that CDLM maintains or exceeds teacher accuracy on several benchmarks (HumanEval-Instruct: 48.2 → 50.0 for Dream; MATH: 24.1 → 28.3 for LLaDA) suggests this knowledge transfer is effective—the student learns to make accurate predictions without needing to see future blocks, potentially because the consistency training encourages it to make stable predictions that don't require future context to resolve.

Fundamental vs. incremental. This is best characterized as an architectural integration. Individual components (block-wise causal masking, native KV caching) existed in prior work. The innovation is integrating block-wise causality into a consistency+distillation training pipeline, demonstrating that native caching and step reduction can be achieved simultaneously without the train-inference mismatches or approximation errors that plague training-free caching methods. The resulting system is simpler at inference time (no caching heuristics, one hyperparameter $\tau_{\text{conf}}$ ) while being more effective, which is a hallmark of a good architectural decision.

5. Experimental Analysis

Evaluation Methodology

Dataset. The primary data sources for CDLM fine-tuning are two HuggingFace datasets, lansechen_easy_2025 and lansechen_hard_2025, which are post-processed subsets of Bespoke-Stratos-17k. These datasets are filtered to include prompts with a maximum length of 512 tokens and contain Qwen2.5-7B responses treated as ground-truth answers. For LLaDA, an additional 7.5k math-style prompts from the DParallel dataset are included to improve math performance. The resulting training sets contain approximately 15k trajectory pairs for Dream (generated at temperatures 0.0 and 0.5) and 30k pairs for LLaDA. Evaluation is conducted on standard math and coding benchmarks: GSM8K (8-shot for Dream, 4-shot for LLaDA), GSM8K-CoT (8-shot), MATH (4-shot), HumanEval (0-shot), HumanEval-Instruct (0-shot), and MBPP-Instruct (0-shot).
Base model(s). The paper trains CDLM variants from two open-source diffusion language models: Dream-7B-Instruct and LLaDA-8B-Instruct. These models represent the current state of open-source DLMs at the 7–8B parameter scale. They were chosen because they are publicly available and provide a strong baseline for evaluating the proposed acceleration method on practical reasoning tasks. For autoregressive throughput comparisons, Qwen2.5-7B-Instruct is paired with Dream, and Llama-3.1-8B-Instruct is paired with LLaDA, ensuring equal-size comparisons.
Metrics. The paper evaluates both efficiency and quality. Efficiency metrics include throughput measured in tokens-per-second (TPS), end-to-end latency measured in seconds, and the average total number of refinement steps executed per sample. Quality metrics depend on the benchmark: exact match on GSM8K, math-verify scores on MATH, and pass@1 on HumanEval, HumanEval-Instruct, and MBPP-Instruct after standard post-processing. All efficiency measurements are taken as per-sample averages over the evaluation set on 4× NVIDIA A100 (80 GB) GPUs with batch size 1 under data parallelism.
Baselines. The paper evaluates against five distinct baselines that isolate different acceleration axes. The vanilla DLM is the original Dream-7B-Instruct or LLaDA-8B-Instruct model running under its official inference setting with block-wise decoding (Dream does not natively support block-wise decoding, so the authors extend the baseline accordingly). dLLM-Cache (Liu et al., 2025) is a training-free method targeting KV caching via adaptive feature caching but keeping the step budget fixed at 256. Fast-dLLM (Parallel) (Wu et al., 2025) targets step reduction through confidence-thresholded parallel decoding without addressing caching. Fast-dLLM (Parallel + Dual Cache) (Wu et al., 2025) combines both training-free acceleration axes via approximate dual-cache KV caching. The autoregressive baselines are Qwen2.5-7B-Instruct (for Dream comparisons) and Llama-3.1-8B-Instruct (for LLaDA comparisons). D2F (Wang et al., 2025) is noted as a training-based baseline but is not directly compared because it is configured for Dream-7B-Base with a generation length of 512 rather than the Instruct models with $L_g = 256$ used here.
Generation budget / compute accounting. All generation-length-based comparisons use a fixed target generation length of $L_g = 256$ tokens with a block size of $B = 32$ . The vanilla DLM baseline uses exactly $N = L_g = 256$ refinement steps (one step per token of generation). For CDLM and all training-free accelerated baselines, the effective step count varies depending on how many tokens are finalized per iteration, with no artificial step budget imposed. The confidence threshold is set to $\tau_{\text{conf}} = 0.9$ as the default for all methods that support parallel decoding. For autoregressive models, greedy decoding (temperature 0.0) is used with the same maximum generation length. Throughput comparisons are made at the same hardware configuration (4× A100 GPUs, batch size 1) to ensure fair measurement.
Cross-validation / statistical protocol. The paper does not employ cross-validation or multiple random seeds for the main experimental results. The training procedure runs for a fixed 16 epochs, and the checkpoint corresponding to the best validation performance is selected for evaluation. Ablation studies on loss weights (Table 3) are conducted with only 4 training epochs and a constant learning rate, evaluated on a single model per configuration. The authors acknowledge that dataset selection is crucial and report that "overly rigid formats (e.g., multiple-choice math) tend to hurt the model's ability to generalize," but no formal statistical testing is performed. This is a notable limitation: with evaluation sets of varying sizes (GSM8K has 1,319 test examples, HumanEval has 164, MATH has 5,000, MBPP has roughly 500), the reliability of small absolute accuracy differences (e.g., 78.8 vs. 79.1 on GSM8K-CoT) cannot be assessed without confidence intervals.

Main Quantitative Results

CDLM–Dream: Step Reduction, Latency, and Accuracy

Table 1 reports results for CDLM–Dream across GSM8K-CoT (8-shot), HumanEval-Instruct (0-shot), MATH (4-shot), and MBPP-Instruct (0-shot). The headline finding is that CDLM–Dream achieves 4.1×–7.7× step reduction compared to the baseline 256 steps across all four benchmarks, with latency reductions of 6.1×–14.5× while maintaining or improving accuracy on three of four tasks.

Step counts. The baseline Dream-7B-Instruct uses 256.0 steps for every benchmark, since $N = L_g = 256$ is the fixed operating point. CDLM–Dream reduces this to 44.1 steps on GSM8K-CoT (5.8× reduction), 49.6 steps on HumanEval-Instruct (5.2× reduction), 63.2 steps on MATH (4.1× reduction), and 33.2 steps on MBPP-Instruct (7.7× reduction). To put these numbers in context, the training-free Fast-dLLM (Parallel) method, which also performs multi-token finalization via confidence thresholding, achieves 53.7, 61.6, 87.1, and 35.3 steps respectively. CDLM consistently requires fewer steps than the training-free approach, despite using the same confidence threshold of $\tau_{\text{conf}} = 0.9$ . This gap—roughly 10–25 fewer steps depending on the benchmark—is attributable to the consistency training teaching the model to make larger, more stable unmasking jumps that the base model cannot make without sacrificing quality.

Latency. End-to-end latency on GSM8K-CoT drops from 23.5 seconds (vanilla Dream) to 2.1 seconds (CDLM–Dream), an 11.2× reduction. On MBPP-Instruct, latency drops from 21.7 seconds to 1.5 seconds, a 14.5× reduction. These gains exceed all baselines. Fast-dLLM (Parallel + Dual Cache), the strongest training-free baseline, achieves 2.5 seconds on GSM8K-CoT (9.4× reduction) and 1.9 seconds on MBPP-Instruct (11.4× reduction). CDLM–Dream's additional latency reduction over this already-optimized baseline comes from two sources: fewer refinement steps (44.1 vs. 60.8 on GSM8K-CoT) and native KV caching that eliminates the approximation overhead of the dual-cache heuristic.

Accuracy. On GSM8K-CoT, CDLM–Dream achieves 78.8, a negligible drop from the baseline's 79.1. On HumanEval-Instruct, accuracy improves from 48.2 to 50.0. On MBPP-Instruct, it rises from 51.8 to 53.0. The only degradation is on MATH, where accuracy drops from 38.0 to 32.4. The paper attributes this to two factors: the small training mixture (~7.5k prompts for Dream) provides limited exposure to the advanced problem-solving skills MATH requires, and the 256-token generation budget may be insufficient for complex multi-step reasoning. However, it is worth noting that this 5.6-point drop represents a 14.7% relative decrease, which is substantial. The paper acknowledges this limitation and suggests training with a 512-token budget as a natural next step.

Throughput. CDLM–Dream achieves the highest TPS on three of four tasks. On the standout MBPP-Instruct benchmark, TPS jumps from 2.3 (vanilla) to 48.1, a 20.9× increase. The baseline Fast-dLLM (Par.+D.C.) achieves 25.4 TPS on the same benchmark, meaning CDLM–Dream is 1.89× faster than the best training-free method. The exception is HumanEval-Instruct, where CDLM–Dream records 43.3 TPS versus 79.9 TPS for Fast-dLLM (Par.+D.C.). The paper explains this anomaly through generation length: CDLM–Dream produces much shorter outputs (96.9 tokens average) compared to Fast-dLLM (Par.+D.C.) at 200.9 tokens. Since TPS is computed as total tokens divided by total time, and both methods produce correct answers (CDLM's pass@1 is 50.0 vs. 46.3), the shorter generation indicates CDLM–Dream "emits fewer redundant tokens while maintaining or improving solution quality." This is a subtle but important observation: CDLM's consistency training makes it more confident in early finalization, leading it to produce concise outputs that terminate sooner via the block-wise early-stopping mechanism, which depresses the measured TPS but not the actual utility.

CDLM–LLaDA: Step Reduction, Latency, and Accuracy

Table 2 reports results for CDLM–LLaDA across GSM8K (4-shot), HumanEval (0-shot), MATH (4-shot), and MBPP-Instruct (0-shot). CDLM–LLaDA achieves 3.4×–7.9× step reduction and 3.6×–8.6× latency reduction with a mixed accuracy picture: improvements on HumanEval and MATH, a slight decline on MBPP-Instruct, and a clear degradation on GSM8K.

Step counts. From the baseline 256.0 steps, CDLM–LLaDA reduces to 57.7 steps on GSM8K (4.4× reduction), 32.3 on HumanEval (7.9×), 75.3 on MATH (3.4×), and 58.0 on MBPP-Instruct (4.4×). These are the largest step reductions among all methods compared. Fast-dLLM (Parallel), the most aggressive training-free alternative, requires 77.5, 100.3, 96.9, and 59.1 steps respectively. The consistency training again yields a clear advantage in step compression, particularly on HumanEval where CDLM–LLaDA uses less than one-third the steps of the training-free method.

Latency. GSM8K latency drops from 28.3 seconds to 3.3 seconds (8.6× reduction). HumanEval latency drops from 11.3 to 1.9 seconds (5.9× reduction). These are the best latency numbers across all methods on these benchmarks. The KV caching contribution is visible when comparing LLaDA's baseline (28.3 seconds, 256 steps) to dLLM-Cache (12.3 seconds, same 256 steps)—roughly halving latency purely through approximate caching. CDLM–LLaDA's additional latency gain over dLLM-Cache (3.3 vs. 12.3 seconds on GSM8K) comes from step reduction (57.7 vs. 256), demonstrating the compounding benefit of addressing both bottlenecks.

Accuracy. This is where the results are most nuanced. CDLM–LLaDA improves HumanEval from 37.8 to 40.2 and MATH from 24.1 to 28.3. The MATH improvement is particularly striking: a 4.2-point absolute gain (17.4% relative improvement) while using 3.4× fewer steps. MBPP-Instruct shows a modest decline (40.8 → 38.4) but CDLM–LLaDA still outperforms Fast-dLLM (Par.+D.C.) which scores 35.0. The clear negative is GSM8K, where accuracy drops from 77.1 to 73.9. The paper provides a detailed diagnosis: training CDLM–LLaDA on a small ~7.5k prompt mix initially caused a more severe drop (77.1 → 72.1), and augmenting with 7.5k math-focused prompts recovered about half the loss (72.1 → 73.9). The authors note that "even simple SFT tends to harm [LLaDA's] math ability unless the training data are carefully curated and sufficiently large" and "expect that further scaling and balancing of the dataset would reduce the remaining GSM8K gap." This sensitivity is an important practical finding: LLaDA is substantially more brittle to fine-tuning than Dream, and CDLM's success on LLaDA requires more careful data curation.

Throughput. CDLM–LLaDA shows throughput gains of 3.4×–6.9× over the vanilla baseline. On HumanEval, the TPS advantage over the vanilla model is 6.9× (7.4 → 50.9), and it dramatically exceeds Fast-dLLM (Par.+D.C.)'s 18.9 TPS. The effective tokens-per-step for CDLM–LLaDA ranges from approximately 2.0 on GSM8K (177.3 tokens / 57.7 steps ≈ 3.1 tokens per step—though this calculation is approximate given Table 2 data) to higher ratios on other benchmarks, confirming that the consistency training enables genuine multi-token finalization rather than just faster sequential processing.

Comparison with Autoregressive Models

Figures 3 and 4 present throughput comparisons between CDLM and equal-size AR models, with accuracy numbers discussed in the text. CDLM–Dream vs. Qwen2.5-7B-Instruct (Figure 3): On GSM8K-CoT, CDLM–Dream achieves 1.21× the throughput of Qwen2.5-7B (51.7 TPS vs. approximately 42.7 TPS, estimated from the bar chart). On MBPP-Instruct, CDLM–Dream reaches 1.12× the throughput (48.1 TPS vs. approximately 43 TPS). In terms of accuracy, CDLM–Dream outperforms Qwen2.5-7B on GSM8K-CoT (78.8 vs. 73.8) but substantially underperforms on MBPP-Instruct (53.0 vs. 81.7). CDLM–LLaDA vs. Llama-3.1-8B-Instruct (Figure 4): On GSM8K, CDLM–LLaDA achieves 1.27× the throughput (54.3 TPS vs. an estimated ~42.8 TPS from the bar chart). On HumanEval, the advantage is dramatic at 4.17× (50.9 TPS vs. ~12.2 TPS). The accuracy tradeoff, however, goes against CDLM–LLaDA on both benchmarks: 73.9 vs. 80.3 on GSM8K and 40.2 vs. 60.4 on HumanEval.

The paper frames these results honestly: "Although throughput is our focus, accuracy may be lower than that of AR models because CDLMs remain bounded by the strength of their DLM backbones." This is a critical caveat. The throughput advantage demonstrates that CDLM makes DLMs viable from a speed perspective, but the accuracy gap—particularly on MBPP-Instruct and HumanEval—shows that the underlying Dream and LLaDA models are simply weaker than comparably-sized AR models on these tasks. The paper's implicit argument is that as stronger DLM backbones emerge, CDLM's acceleration recipe should apply, yielding competitive speed AND accuracy. This is a forward-looking but currently unverified claim.

Effective Step Reduction vs. Naive Truncation

Table 4 provides the crucial ablation demonstrating that training-based step reduction is necessary. When naive Dream-7B-Instruct is forced to use 48 refinement steps (matching CDLM–Dream's 44.1-step budget), accuracy on GSM8K-CoT collapses from 79.1 to 41.8. Similarly, naive LLaDA-8B-Instruct at 56 steps drops from the CDLM–LLaDA-comparable performance to 60.3 on GSM8K. In both cases, CDLM achieves dramatically higher accuracy (78.8 and 73.9 respectively) at similar or lower step counts. The paper states that "consistency training is necessary for stable multi-token refinement." The comparison also shows that CDLM with KV caching reduces latency by roughly half relative to the naive DLM at comparable iteration counts—CDLM–Dream achieves 2.1 seconds at 44.1 steps, while naive Dream achieves 4.4 seconds at 48 steps. The latency gap here is attributable to caching: the naive model recomputes attention over the full sequence at every step, while CDLM reuses cached KV states.

Ablation Studies and Robustness Checks

Loss-weight composition (Table 3): Distillation is essential; consistency without distillation causes collapse; coupling yields synergy. The ablation varies $(w_{\text{distill}}, w_{\text{cons}}, w_{\text{dlm}})$ and evaluates on GSM8K (4-shot) and HumanEval-Instruct (0-shot) after 4 epochs of CDLM–Dream training. Distillation-only (row 1: $w_{\text{distill}} = 1.0$ , $w_{\text{cons}} = 0$ , $w_{\text{dlm}} = 0.01$ ) achieves 73.2 on GSM8K and 42.7 on HumanEval, converging in 46.7 and 61.0 steps respectively. This serves as the anchor. Consistency-only (row 2: $w_{\text{distill}} = 0$ , $w_{\text{cons}} = 1.0$ , $w_{\text{dlm}} = 0.01$ ) causes catastrophic failure: GSM8K drops to 6.9 and HumanEval-Instruct to 0.0, with the model requiring many steps (100.6 and 124.3)—it never learns to generate coherent solutions. This confirms that self-consistency without teacher grounding is not merely ineffective but actively destructive. The combination of distillation + consistency (row 3: both at 1.0) improves GSM8K to 74.1 (from 73.2) at similar steps (49.4 vs. 46.7) and maintains HumanEval at 42.7 at lower steps (49.4 vs. 61.0). Reducing the consistency weight to 0.1 (row 5) yields the best GSM8K result (75.1 at 48.0 steps) but reduces HumanEval from 42.7 to 45.7 at slightly higher steps (59.9 vs. 49.4 for row 3). This non-uniform behavior across tasks motivates the final choice of $w_{\text{cons}} = 0.5$ as a compromise between the extremes. Removing the DLM auxiliary loss (rows 4 and 6) improves HumanEval (42.7 → 48.2; 45.7 → 50.6) but hurts GSM8K (74.1 → 73.3; 75.1 → 73.7), indicating a tradeoff between math reasoning preservation and code generation performance. The small DLM weight helps retain the model's pre-existing math capabilities during fine-tuning.
Token-level confidence threshold (Table 5): A monotonic speed-quality tradeoff exists, with $\tau_{\text{conf}} = 0.90$ providing a robust default. Sweeping $\tau_{\text{conf}} \in \{0.85, 0.90, 0.95\}$ on CDLM–Dream for GSM8K-CoT and HumanEval-Instruct reveals the expected pattern: higher thresholds produce more conservative decoding (fewer tokens finalized per step), which increases accuracy at the cost of throughput and latency. On GSM8K-CoT, moving from $\tau = 0.85$ to $\tau = 0.95$ changes TPS/latency/score from 57.7/1.9s/78.4 to 51.7/2.1s/78.8 to 42.7/2.5s/78.8. The accuracy is essentially flat between 0.90 and 0.95 (both 78.8), suggesting 0.90 is the knee of the curve—further conservative gains yield no quality improvement but cost speed. On HumanEval-Instruct, the range is wider: 48.2 (0.85) → 50.0 (0.90) → 51.2 (0.95). Here, the quality improvement continues, with a 3-point gap between the most aggressive and most conservative settings. The paper's choice of 0.90 as the default is reasonable for general use but the sensitivity on HumanEval suggests that task-specific tuning could yield meaningful accuracy improvements.
Teacher trajectory temperature (Appendix A.1, Figure 5): Trajectory collection is restricted to low temperatures because higher stochasticity destabilizes reasoning. The paper generates teacher trajectories at $\tau \in \{0.0, 0.5\}$ , deliberately excluding $\tau = 1.0$ . The qualitative example in Figure 5 shows why: at $\tau = 1.0$ , LLaDA-8B-Instruct produces an incorrect final answer (marked red) despite the lower-temperature runs producing correct answers (marked blue). The paper notes that $\tau = 1.0$ "often destabilizes the reasoning chain and can yield incorrect conclusions." This is a practical constraint on trajectory diversity: the augmentation benefit of multiple temperatures is bounded by the fact that high-temperature trajectories may contain errors that would poison the student's training. The paper does not explore whether filtering correct trajectories at high temperature (keeping only the ones that happen to be right) would provide useful diversity.
Hidden-state buffer vs. logit storage (Appendix A.1): The 30× compression via hidden-state storage is lossless for logit reconstruction. The paper's approach of storing last-layer hidden states rather than full logits reduces storage from approximately 750 GiB to 25–30 GiB per 15k-sample shard. Since the $\text{lm\_head}$ transformation is linear (a frozen matrix multiply), the reconstructed logits are exact—there is no information loss. This is a design choice that makes the approach practical without any accuracy penalty. The paper does not ablate this choice (e.g., comparing against approximate logit compression), but the exactness of the reconstruction is a mathematical property rather than an empirical finding.
Distillation space: logit-space vs. embedding-space (Appendix A.2): Logit-space distillation with forward KL outperforms embedding-space MSE. The authors report that forward KL divergence in logit space "yielded more stable, better-calibrated training dynamics and more monotonic convergence" than alternative formulations. Specifically, they tested reverse KL divergence (which was less stable) and mean-squared error in embedding space (which underperformed logit-space KL). No quantitative results are provided for this comparison; it is stated as an empirical observation guiding methodological choices. A quantitative ablation of this design decision would strengthen the paper, as the choice of distillation loss has been shown to significantly impact results in other KD settings.
Validation-based checkpoint selection (Section 5.1, Appendix A.2): The best epoch is selected by validation performance with no formal early-stopping criterion. Training runs for 16 epochs, but the best checkpoint is reported at epoch 12 for both Dream and LLaDA. The paper does not describe the validation metric used for checkpoint selection or provide learning curves showing stability across epochs. Given the sensitivity of LLaDA to fine-tuning (documented in the GSM8K degradation), the validation-based selection could be introducing an implicit form of hyperparameter optimization on the test set if the validation distribution is similar to the test distribution.

Critical Assessment

The experiments demonstrate convincingly that CDLM achieves substantial step reduction and latency improvement over baseline DLMs. Tables 1 and 2 uniformly show CDLM requiring fewer refinement steps than any other method while maintaining or improving accuracy on the majority of tested benchmarks. The ablation in Table 4 rules out the possibility that these gains come purely from inference heuristics—naive step truncation destroys accuracy, confirming that the consistency training is essential.

Does CDLM genuinely solve both DLM inference bottlenecks simultaneously? The evidence supports this claim with precision. CDLM reduces step counts (the step-count bottleneck) AND eliminates per-step recomputation via native KV caching (the caching bottleneck). The latency numbers isolate these effects: comparing CDLM–Dream (2.1s at 44.1 steps) against Fast-dLLM (Par.+D.C.) (2.5s at 60.8 steps) shows that CDLM wins on both axes—fewer steps AND lower per-step cost via native caching. The training-free caching baselines (dLLM-Cache, Fast-dLLM Dual Cache) show that caching alone cannot compensate for high step counts; the training-based caching in CDLM enables caching without the approximation overhead. This is the central engineering contribution and the experiments unambiguously support it.

Does consistency modeling specifically enable the step reduction, or would distillation alone suffice? The loss-weight ablation in Table 3 provides a nuanced answer. Distillation alone (row 1) achieves reasonable step reduction (46.7 steps on GSM8K vs. CDLM's 44.1–49.4 depending on weight configuration), suggesting that teacher supervision on newly unmasked tokens is the primary driver of multi-token finalization. Consistency adds about 2–3% accuracy at similar step counts when coupled with distillation (73.2 → 74.1–75.1 on GSM8K), with the exact gain depending on the consistency weight. This is a moderate improvement, not a transformative one. The genuine contribution of consistency appears to be stability and quality preservation at aggressive step budgets, not raw step reduction. The paper would be strengthened by an ablation showing step-reduction capability with and without consistency at different target accuracy thresholds.

Can CDLM-trained models match autoregressive models end-to-end? The answer depends on the metric. For throughput, CDLM–Dream matches or exceeds Qwen2.5-7B (1.2× on GSM8K-CoT, 1.1× on MBPP-Instruct), and CDLM–LLaDA exceeds Llama-3.1-8B (1.3× on GSM8K, 4.2× on HumanEval). These are genuine throughput advantages at the same hardware budget. For accuracy, the picture is sobering: CDLM–Dream trails Qwen2.5-7B by 28.7 points on MBPP-Instruct (53.0 vs. 81.7), and CDLM–LLaDA trails Llama-3.1-8B by 20.2 points on HumanEval (40.2 vs. 60.4). The paper's claim that CDLM "surpasses equal-size autoregressive LLMs in tokens-per-second" is well-supported and correct. The implicit stronger claim—that this makes DLMs a practical alternative to AR models—is NOT supported by the current results, because the accuracy gap on several benchmarks is far too large to be acceptable in deployment. This is not a weakness of CDLM per se (the method cannot improve the base DLM's raw capability) but it is a critical boundary condition on the paper's practical impact: CDLM makes DLMs fast enough to be considered, but the underlying DLMs are not yet strong enough to be chosen over AR models for many tasks.

How robust is CDLM to model family and data regime? The dual evaluation on Dream-7B and LLaDA-8B is a genuine strength. The method transfers across architectures, with both models showing substantial speedups. However, the transfer reveals important sensitivity: LLaDA is significantly more brittle to fine-tuning than Dream, with the GSM8K degradation requiring explicit data augmentation to partially mitigate. The paper's diagnostic that "even simple SFT tends to harm [LLaDA's] math ability" is an important practical finding and suggests that CDLM's success on a given DLM depends on that model's robustness to weight modification. The data scale sensitivity is partially explored: the 7.5k-prompts-for-Dream and 15k-prompts-for-LLaDA configurations work, but the LLaDA experience shows that domain coverage matters. What is missing is a scaling experiment showing how CDLM performance varies with dataset size—would 30k prompts for Dream further close the MATH gap? Would 5k prompts for LLaDA cause wider degradation than observed? Without a scaling curve, it's unclear whether the current dataset sizes are near-optimal or substantially suboptimal.

What is not tested that should be? Several experiments would significantly strengthen the claims:

Scaling to longer generation lengths ( $L_g = 512$ or $1024$ ): The paper trains and evaluates only at $L_g = 256$ . The D2F method uses $L_g = 512$ , and practical applications often require longer outputs. The fact that CDLM-Dream's accuracy drops on MATH—attributed partly to insufficient generation budget—suggests that testing at longer lengths is essential. Does the step-count advantage scale proportionally (e.g., 7.9× step reduction at 512 tokens as well), or does it degrade with longer sequences?
Block size ablation: The paper fixes $B = 32$ throughout. This is a critical hyperparameter: larger blocks allow more tokens to be finalized per step but increase the maximum jump size the consistency objective must learn. Smaller blocks make consistency easier but reduce the upper bound on step reduction. Understanding this tradeoff would help practitioners choose $B$ for new models.
Comparison with D2F and Fast-dLLMv2: The paper acknowledges D2F (Wang et al., 2025) as the closest training-based caching work but does not compare directly because D2F is trained on Dream-7B-Base with $L_g = 512$ . A fair comparison (e.g., retraining D2F on Dream-7B-Instruct with $L_g = 256$ , or training CDLM in the D2F setting) would clarify whether the step reduction gains are genuinely attributable to the consistency+distillation objectives or could be achieved by other training-based approaches.
On-policy vs. offline trajectory comparison: The paper discusses online trajectory generation as a future direction (Appendix B) but provides no data on how much the offline nature limits performance. An experiment comparing offline trajectories from the base teacher against trajectories from an intermediate CDLM checkpoint (even if expensive) would quantify the train-inference distribution gap.
Statistical significance: With evaluation sets ranging from 164 examples (HumanEval) to 5,000 (MATH), and accuracy differences of 1–5 points, confidence intervals are necessary to distinguish real improvements from noise. The 4.2-point MATH improvement for CDLM–LLaDA (24.1 → 28.3) on 5,000 examples is likely significant, but the 4-point GSM8K degradation (77.1 → 73.9) on 1,319 examples may also be significant in the opposite direction.

What claims are conditionally true and what are the conditions? The claim that CDLM achieves "3.6×–14.5× lower latency" is true under the specific evaluation conditions: $L_g = 256$ , $\tau_{\text{conf}} = 0.90$ , $B = 32$ , 4× A100 GPUs, batch size 1. The claim of accuracy preservation is conditionally true: it holds for Dream on GSM8K-CoT, HumanEval-Instruct, and MBPP-Instruct, but fails on MATH (32.4 vs. 38.0). For LLaDA, it holds on HumanEval and MATH (both improve), partially holds on MBPP-Instruct (minor decline, outperforms baselines), and fails on GSM8K (73.9 vs. 77.1). The throughput advantage over AR models holds on the measured benchmarks but the accuracy gap makes this a hollow victory on tasks like MBPP-Instruct where CDLM–Dream is 28.7 points behind Qwen2.5-7B. The paper is unusually candid about these conditions, explicitly stating that "accuracy may be lower than that of AR models because CDLMs remain bounded by the strength of their DLM backbones." This is both a limitation and a roadmap: improve the backbone, and CDLM's acceleration should transfer.

6. Limitations and Trade-offs

Limitation 1: CDLM Performance Is Strictly Bounded by Teacher Quality, and the Underlying DLMs Are Substantially Weaker Than Comparable AR Models

The assumption or constraint. CDLM is a fine-tuning method that distills knowledge from a teacher DLM into a block-wise causal student. The student cannot exceed the teacher's capabilities because all supervision—distillation targets for newly unmasked tokens and the trajectory path itself—originates from the teacher. The paper states this explicitly: "CDLM's performance is ultimately bounded by the teacher: a bidirectional DLM distilled into a block-causal student cannot exceed the teacher's knowledge" (Appendix B). The teachers are Dream-7B-Instruct and LLaDA-8B-Instruct, which are known to lag behind comparably sized autoregressive models on standard benchmarks.

The consequence. Even with optimal acceleration, CDLM cannot close the capability gap between current open-source DLMs and AR models. The results in Section 5.2.3 quantify this: CDLM–Dream trails Qwen2.5-7B-Instruct by 28.7 points on MBPP-Instruct (53.0 vs. 81.7) and CDLM–LLaDA trails Llama-3.1-8B-Instruct by 20.2 points on HumanEval (40.2 vs. 60.4). The paper's headline throughput advantages over AR models (1.1×–4.2×) are genuine but exist in a regime where the accuracy gap makes direct substitution unacceptable for most deployment scenarios. A practitioner choosing between an AR model and a CDLM-accelerated DLM faces a tradeoff the paper does not resolve: take the faster model with lower accuracy, or the slower model with higher accuracy. On tasks where the teacher DLM is already strong (Dream on GSM8K-CoT: 79.1 baseline, CDLM achieves 78.8), CDLM is compelling. On tasks where the teacher is weak, CDLM inherits that weakness.

What evidence exists in the paper. The accuracy comparisons in Tables 1 and 2 directly show the capability gaps. Figures 3 and 4 show throughput advantages but the accompanying text (Section 5.2.3) provides the accuracy numbers that contextualize them. The MATH degradation for CDLM–Dream (38.0 → 32.4, Table 1) and the GSM8K degradation for CDLM–LLaDA (77.1 → 73.9, Table 2) further demonstrate that the fine-tuning process itself can degrade certain capabilities, particularly when the training data distribution does not match the evaluation distribution.

Mitigation status. The paper acknowledges this limitation candidly but does not address it. The authors suggest that "distillation from stronger AR teachers is a natural next step" (Appendix B) and note that "as stronger DLM backbones emerge, applying our consistency-based fine-tuning should yield further throughput gains with improved quality" (Section 5.2.3). These are forward-looking statements, not mitigations within the current work. The paper does not experiment with AR-to-DLM distillation or with stronger DLM teachers. The limitation is fundamental to the current evaluation but potentially resolvable with better base models.

Limitation 2: Training Data Scale and Domain Coverage Are Insufficient for Robust Generalization, Particularly for LLaDA

Assumption or constraint. CDLM training relies on static, offline trajectories collected from the teacher on a relatively small, domain-specific prompt corpus. The Dream training set contains approximately 15k trajectory pairs (7.5k unique prompts × 2 temperatures) derived from a filtered subset of Bespoke-Stratos-17k. The LLaDA training set doubles this to approximately 30k pairs by adding 7.5k math-focused prompts from DParallel. Both datasets are constructed from a single source distribution (Bespoke-Stratos-17k, which predominantly contains math word-style reasoning problems) and use Qwen2.5-7B outputs as ground-truth answers. The paper acknowledges that "dataset selection is crucial; overly rigid formats (e.g., multiple-choice math) tend to hurt the model's ability to generalize" (Appendix A.1).

The consequence. The limited data scale and narrow domain coverage create two failure modes. First, the student can overfit to the teacher's specific unmasking patterns and output styles on the training distribution, degrading performance on out-of-distribution benchmarks—the paper explicitly warns that "the student can still overfit to the teacher's priors because supervision does not adapt during training" (Appendix B). Second, the data insufficiency directly causes accuracy degradation on under-represented tasks, most visibly MATH for CDLM–Dream (38.0 → 32.4) and GSM8K for CDLM–LLaDA (77.1 → 73.9). The paper's own diagnosis for the MATH drop is that "the small training mixture (~7.5k prompts) limits exposure to the advanced problem-solving skills required by MATH" (Section 5.2.1). For LLaDA, CDLM training on only the Bespoke-derived prompts caused an even more severe GSM8K degradation (77.1 → 72.1, reported in Section 5.2.2), which was only partially recovered by adding 7.5k math-focused prompts (72.1 → 73.9). This sensitivity—where a moderate fraction of the total training data determines whether a benchmark degrades by 5 points or 7 points—indicates the model is operating in a data-starved regime where out-of-distribution generalization is fragile.

What evidence exists in the paper. The benchmark-specific accuracy degradation patterns provide direct evidence: MATH drops 5.6 points for Dream (Table 1), GSM8K drops 3.2 points for LLaDA (Table 2). The paper's ablation on LLaDA training data (Section 5.2.2) explicitly quantifies how adding 7.5k math prompts shifts GSM8K accuracy—this is a partial data scaling result, though it only measures recovery from degradation rather than improvement over the baseline. The paper also reports an empirical finding in Appendix A.1 that "overly rigid formats (e.g., multiple-choice math) tend to hurt the model's ability to generalize," indicating sensitivity to data format, not just domain.

Mitigation status. The paper acknowledges this limitation and proposes "scaling beyond our current 15k-prompt corpus and broadening it to more diverse domains" as "straightforward directions for future work" (Section 6). For the LLaDA case, data augmentation with DParallel prompts was attempted and partially mitigated the GSM8K degradation, showing that data scaling helps but is insufficient at the current scale. The paper does not provide a scaling curve showing how accuracy varies with dataset size, leaving practitioners uncertain about how much data would be needed to achieve robust generalization. Without such a curve, adopting CDLM for a new DLM or a new domain would require extensive trial-and-error data collection.

Limitation 3: The Training Paradigm Is Entirely Offline, Creating a Train-Inference Distribution Gap That Cannot Be Closed Without Faster DLM Sampling

Assumption or constraint. All CDLM training uses pre-collected, static trajectories generated by the teacher model running its standard 256-step block-wise decoding before any fine-tuning begins. The student is never exposed to its own on-policy trajectories during training—it learns exclusively from states the teacher visited, under the teacher's unmasking schedule and confidence estimates. The paper states: "Training currently relies on offline, static trajectories. Despite careful dataset selection, the student may overfit to the teacher's prior because supervision does not adapt during learning" (Section 6). The reason for this design choice is practical: "slow generation limits the feasibility of closed-loop training without prior trajectory materialization" (Appendix B).

The consequence. There is a systematic mismatch between the states the student encounters during training and the states it encounters during its own inference. At inference time, CDLM uses confidence-thresholded parallel finalization, which can unmask multiple tokens per step in an order determined by the student's own confidence estimates—not the teacher's. The student's unmasking order, token probabilities, and the sequence of partially refined states it visits may diverge from the teacher's trajectories, especially as consistency training changes the student's prediction calibration. When the student encounters states that are out-of-distribution relative to its training data, the distillation and consistency objectives provide no guidance—the student must generalize from teacher-visited states to its own self-generated states. This is the standard train-inference distribution shift problem in distillation, but it is amplified here because the student is explicitly trained to take larger jumps (via consistency) than the teacher ever demonstrated (the teacher took exactly one token per step). The largest jumps the student makes at inference time—potentially unmasking 5–10 tokens in a single step under low confidence thresholds—correspond to state transitions that have no direct analogue in the teacher's trajectory, where at most $B = 32$ steps separate $y$ and $y^\star$ but each intermediate step unmaskes exactly one token.

What evidence exists in the paper. The paper does not directly measure the train-inference distribution gap. There is no experiment comparing offline-trained CDLM against a hypothetical online-trained variant, and no analysis of how the student's unmasking order or state distribution differs from the teacher's. The paper acknowledges this as a limitation (Section 6, Appendix B) but provides no empirical quantification. The degradation patterns on MATH (Dream) and GSM8K (LLaDA) could be partially attributable to this distribution shift—the student encounters reasoning patterns during its own inference that were under-represented in the teacher's trajectories—but the paper does not attempt to disentangle data insufficiency from distribution shift as causal factors. The ReST^EM experiment that degraded performance (referenced in the original paper's context about revision models in Appendix K) is not present in this paper, so there is no analogous negative result demonstrating distribution-shift-induced failure in the CDLM setting.

Mitigation status. The paper proposes moving "from offline supervision to on-the-fly generation" where "the student generates trajectories during training and the teacher verifies or refines them online" (Appendix B) as a future direction. It correctly identifies that this is "currently constrained by DLM sampling speed" and that "improving DLM inference throughput is therefore a prerequisite for large-scale online training." This is a candid acknowledgment but not a mitigation: the core problem is that online training would require the very acceleration that CDLM is trying to provide, creating a circular dependency. A partial mitigation not explored in the paper would be to collect trajectories from an intermediate CDLM checkpoint (once it has achieved some speedup) and use those for subsequent training iterations—a form of iterative self-distillation that could incrementally reduce the distribution gap without requiring full online training speed.

Limitation 4: The Block Size, Confidence Threshold, and Generation Length Are Fixed Hyperparameters with No Ablation or Guidance for Practitioners Adapting CDLM to New Settings

The assumption or constraint. CDLM's training and inference are configured with three critical hyperparameters that are set to specific values and never varied in the main experiments: block size $B = 32$ , confidence threshold $\tau_{\text{conf}} = 0.90$ , and generation length $L_g = 256$ . The block size determines the maximum distance between $y$ and $y^\star$ in the consistency objective (at most 32 steps), the granularity of KV caching (caches are flushed at block boundaries), and the upper bound on tokens finalized per step (at most $B$ tokens can be unmasked per block). The confidence threshold controls the aggressiveness of multi-token finalization at inference time and is only ablated in isolation (Table 5), not in interaction with other parameters. The generation length sets the total refinement budget and limits the maximum output length.

The consequence. A practitioner adapting CDLM to a new DLM, a different generation length, or a different task distribution has no guidance on how to set these parameters. The consequences of each choice are specific and non-trivial:

Block size ( $B$ ): Larger blocks allow more tokens to be unmasked per step (higher potential speedup) but increase the maximum jump the consistency objective must learn (up to $B$ steps between $y$ and $y^\star$ ). If $B$ is too large, the consistency loss may be too weak to enforce stable predictions across the full block, and the student may fail to learn multi-token finalization. If $B$ is too small, the step reduction upper bound is tight (e.g., with $B = 8$ and $L_g = 256$ , at minimum 32 blocks × 1 step = 32 steps are required). The paper does not explore whether consistency training becomes harder or easier as $B$ varies, nor whether the optimal $B$ depends on the teacher's refinement behavior (how much token predictions change within a block).
Confidence threshold ( $\tau_{\text{conf}}$ ): Table 5 shows this parameter meaningfully affects the speed-quality tradeoff: on HumanEval-Instruct, moving from $\tau = 0.85$ to $\tau = 0.95$ changes accuracy by 3 points (48.2 → 51.2) and latency by 0.8 seconds (2.0 → 2.8). However, this ablation is performed only on CDLM–Dream for two benchmarks. It is unknown whether the same threshold values are optimal for CDLM–LLaDA, for other tasks, or for different block sizes. A practitioner tuning CDLM for a new deployment has to run their own $\tau_{\text{conf}}$ sweep without knowing whether the sensitivity is task-dependent.
Generation length ( $L_g$ ): All experiments use $L_g = 256$ . The paper mentions that D2F uses $L_g = 512$ but does not evaluate CDLM at this length. CDLM–Dream's accuracy drop on MATH is partially attributed to "this shorter thinking budget" (Section 5.2.1), suggesting that tasks requiring longer reasoning chains would benefit from larger $L_g$ . However, longer $L_g$ means more blocks, more total refinement steps, and potentially different optimal $B$ and $\tau_{\text{conf}}$ values. The step reduction factor observed at $L_g = 256$ (e.g., 5.8× on Dream GSM8K-CoT) may not scale linearly to $L_g = 512$ or $L_g = 1024$ —the paper provides no evidence either way.

What evidence exists in the paper. The $\tau_{\text{conf}}$ ablation (Table 5) is the only hyperparameter sensitivity experiment. The block size $B$ and generation length $L_g$ are not ablated at all. The paper does not report experiments varying $B$ (e.g., 16 vs. 32 vs. 64) to show how it affects convergence speed, final accuracy, or inference throughput. The training configuration tables (Tables 6 and 7) list these as fixed values with no justification beyond matching the baseline block-wise decoding convention for $B = 32$ .

Mitigation status. The paper does not address this limitation. It provides a single robust default ( $\tau_{\text{conf}} = 0.90$ ) based on limited ablation and treats $B$ and $L_g$ as fixed architectural choices inherited from prior work. A sensitivity analysis for $B$ across a range of values and guidance on how to select it for new models would substantially improve the paper's practical utility. The paper also does not report whether the optimal $B$ depends on the teacher's training configuration—for instance, whether a teacher trained with a different block size would require a matching student block size for effective consistency learning.

Limitation 5: CDLM Is Evaluated on a Single Hardware Configuration with Batch Size 1, Leaving Multi-User Throughput and Hardware Scalability Uncharacterized

Assumption or constraint. All latency, throughput, and step-count measurements in the paper (Tables 1 and 2, Figures 3 and 4) are collected on 4× NVIDIA A100 (80 GB) GPUs with data parallelism and batch size 1 per GPU. The paper states: "All efficiency measurements are taken on 4× NVIDIA A100 (80 GB) GPUs with batch size 1 under data parallelism" (Section 5.1). The evaluation is single-sample: each GPU processes exactly one prompt at a time, and the reported TPS and latency are per-sample averages. This configuration measures single-stream inference speed—the time to generate one complete answer for one user query—rather than serving throughput under concurrent load.

The consequence. The reported latency and throughput numbers may not generalize to deployment scenarios where multiple users or queries are processed simultaneously (batched inference) or where different hardware (fewer GPUs, consumer-grade GPUs, different GPU generations) is used. Several specific concerns arise:

Batched inference scaling is unknown. A key claimed advantage of DLMs is that they can process multiple tokens in parallel, which, in principle, enables better utilization of GPU compute than the sequential token generation of AR models. Under batch size 1, this parallelism advantage is partially wasted because the GPU may not be saturated by processing a single sequence. The paper's TPS comparisons against AR models (Figures 3 and 4) use batch size 1 for both, which is fair for single-stream latency but does not measure whether CDLM's throughput advantage grows or shrinks under batched inference. AR models benefit substantially from continuous batching (processing multiple requests simultaneously by interleaving their token generation), while DLM batching behavior is less well-understood.
KV cache memory pressure under batching is not characterized. CDLM's native KV caching stores key-value states for all completed blocks. Under single-stream inference with $L_g = 256$ , this cache is modest. Under batched inference with many concurrent sequences of varying lengths, the aggregate KV cache memory grows linearly with batch size, potentially exceeding GPU memory. The paper provides no memory footprint measurements for the KV cache or any analysis of how memory constraints would limit maximum batch size. This is particularly relevant because one motivation for DLMs is high-throughput serving; without batching characterization, the throughput numbers cannot be extrapolated to production serving scenarios.
Hardware sensitivity is untested. All experiments use A100 (80 GB) GPUs, which have high memory bandwidth and large compute capacity. It is unknown whether CDLM's relative speedup over baselines holds on lower-end GPUs (e.g., A10, T4, consumer RTX cards) where memory bandwidth is a tighter bottleneck. The paper also does not report whether CDLM's training (8–16 hours on 4× A100s) can be performed on smaller GPU configurations with gradient accumulation or reduced batch sizes.
The "data parallelism" evaluation setup may obscure per-GPU variance. The paper reports per-sample averages computed by summing wall-clock time across all 4 GPUs and dividing by the total number of evaluated samples. This assumes perfect load balancing across GPUs under data parallelism, which may not hold if some prompts produce longer/shorter generations than others. The paper does not report per-GPU latency variance or discuss whether outlier samples (very long or very short generations) skew the averages.

What evidence exists in the paper. All reported efficiency numbers are under the single configuration described above. The paper does not include any batched inference experiments, memory footprint analysis, or multi-GPU scaling studies. The training hardware requirements (4× A100 or 8× RTX A6000) are reported in Appendix A.2, but inference hardware is only specified for the 4× A100 configuration used in evaluation. There is no ablation varying inference batch size, GPU type, or the number of GPUs used for serving.

Mitigation status. The paper does not address this limitation, nor does it frame it as a limitation. The single-stream, batch-size-1 evaluation is standard for DLM acceleration papers (Fast-dLLM, dLLM-Cache, and D2F all use similar measurement protocols), so CDLM is consistent with prior work. However, this convention means the practical throughput gains in a production serving environment—where batching and concurrent request handling dominate—remain unquantified. A practitioner considering CDLM for a high-throughput serving deployment would need to run their own batching experiments to determine real-world performance. The paper would be strengthened by even a simple ablation showing how TPS scales with batch size for both CDLM and AR baselines.

Limitation 6: The Consistency Training Approach Has No Theoretical Guarantee of Stable Convergence in Discrete Token Spaces, and the Observed "Consistency Collapse" Failure Mode Is Not Fully Characterized

Assumption or constraint. CDLM adapts consistency modeling from continuous diffusion models, where consistency along the probability flow ODE has a theoretical guarantee: mapping any intermediate state to the data distribution's endpoint is well-defined under the learned score function (Song et al., 2023). In discrete masked diffusion, there is no analogous ODE, no continuous trajectory, and no theoretical guarantee that enforcing self-consistency between $y$ and $y^\star$ will produce a meaningful mapping. The paper's adaptation is empirical: it defines a token-level trajectory based on the teacher's unmasking order, defines consistency intervals bounded by block completion, and optimizes a forward KL objective with a stop-gradient target. The paper states this framing explicitly: "Conceptually, our objective can be viewed as an empirical generalization of the original consistency objective defined over continuous ODE trajectories, adapted to discrete, token-level diffusion processes" (Section 2.4).

The consequence. Because the approach lacks theoretical grounding in discrete spaces, there are failure modes whose existence and severity were discovered only empirically. The most dramatic is the "consistency collapse" documented in Table 3 (row 2): when consistency is trained without distillation ( $w_{\text{distill}} = 0$ , $w_{\text{cons}} = 1.0$ ), the student achieves perfect self-consistency (its early-state predictions match its late-state predictions) but these predictions are completely wrong—GSM8K accuracy collapses to 6.9% and HumanEval-Instruct to 0.0%. The paper does not explain why this collapse occurs, only that it does. In continuous consistency models, consistency alone (without distillation) can work because the trajectory endpoint is constrained to be the data distribution. In discrete token space, the student can apparently learn a degenerate mapping where every intermediate state maps to an arbitrary but self-consistent final state, with no force pushing that final state toward the correct answer distribution.

This empirical observation raises broader concerns about the stability of the approach:

Interaction between consistency weight and data quality is unknown. The ablation in Table 3 shows that the optimal consistency weight depends on the task: $w_{\text{cons}} = 0.1$ produces the best GSM8K result (75.1), while $w_{\text{cons}} = 1.0$ produces the best HumanEval-Instruct result within the "coupled with distillation" rows (though the comparison is muddled by the DLM weight variation). This suggests the consistency loss's benefit-to-risk ratio varies with the training data distribution and target task, but the paper provides no analysis of this interaction.
The trajectory sampling procedure (uniformly sampling $y$ from $\mathcal{T}_x$ ) may over-weight easy states. The paper samples $y$ uniformly from all 257 states in the trajectory. States near the beginning of a block (where most tokens are still masked and predictions are uncertain) may provide a much stronger consistency signal than states near the end of a block (where most tokens are already finalized and predictions are nearly identical to $y^\star$ ). A non-uniform sampling strategy (e.g., biasing toward earlier states) might improve consistency learning or reduce the risk of collapse, but this is not explored.
The forward KL direction for the consistency loss may encourage undesirable mode averaging. Forward KL ( $D_{\mathrm{KL}}(q_{\phi^-} \| q_{\phi})$ ) is "mean-seeking"—it encourages $q_{\phi}(y)$ to cover all modes of $q_{\phi^-}(y^\star)$ . If $q_{\phi^-}$ has multiple plausible modes for a still-masked token (which is common in early refinement), the consistency loss pushes $q_{\phi}(y)$ to be uncertain rather than to commit to the mode that will eventually be correct. This might explain why consistency alone fails: it encourages the student to remain non-committal, and without the distillation loss providing the "correct" target, the student learns to output uniform-like distributions that are self-consistent but uninformative.

What evidence exists in the paper. Table 3 is the primary evidence for the consistency collapse phenomenon. The diagnostic is convincing but narrow: it shows the failure for one configuration ( $w_{\text{cons}} = 1.0$ , $w_{\text{distill}} = 0$ , $w_{\text{dlm}} = 0.01$ ) on two benchmarks after four epochs. The paper does not explore whether the collapse is gradual (accuracy declining epoch by epoch) or sudden, whether it occurs at other consistency weights, or whether it can be prevented by alternative formulations (e.g., reverse KL, different stop-gradient strategies, or adding entropy regularization). The paper also does not verify whether the collapsed model is actually self-consistent (i.e., whether $D_{\mathrm{KL}}(q_{\phi^-} \| q_{\phi})$ is close to zero) or whether the collapse is due to some other pathological training dynamic—the accuracy numbers demonstrate the outcome but not the mechanism.

Mitigation status. The paper mitigates the collapse by always coupling consistency with distillation, using a moderate consistency weight ( $w_{\text{cons}} = 0.5$ , half the distillation weight), and including the DLM loss as an additional regularizer. This combination empirically prevents collapse across all reported experiments. However, this is a practical workaround, not a resolution of the underlying theoretical gap. A practitioner adapting CDLM to a new setting—different DLM architecture, significantly different data distribution, or different loss weight ratios—has no way to predict whether consistency collapse will occur without running the experiment. The paper does not provide diagnostic criteria (e.g., monitoring the consistency loss magnitude relative to the distillation loss) that would serve as early warning signals during training. The suggestion in Appendix B to move toward online training could either mitigate or exacerbate this problem—on-policy trajectories might be more diverse and less prone to collapse, or they might amplify the degenerate self-consistency if the student's own predictions reinforce its errors. Without theoretical understanding of why collapse happens, the safety of the approach in new regimes is unknown.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper changes the conversation around diffusion language model acceleration from one where step reduction and KV caching are treated as independent, competing research directions to one where they are recognized as synergistic objectives that should be pursued jointly through architectural fine-tuning. Before CDLM, the DLM acceleration field was bifurcated: training-free methods (Fast-dLLM, dLLM-Cache) applied inference-time heuristics to patch specific symptoms, and training-based methods (D2F, Fast-dLLMv2) addressed exactly one bottleneck through architectural modification but left the other untouched. CDLM demonstrates that both bottlenecks share a common root cause—the model was trained to take many small steps under full bidirectionality—and that a single fine-tuning procedure coupling block-wise causality with consistency-based multi-step jumping resolves both simultaneously. The magnitude of this shift is best characterized as a methodological reframing with practical consequences: it does not introduce fundamentally new mathematical primitives (consistency models and block-wise causal attention each existed), but it demonstrates that their combination produces multiplicative speedups (e.g., 11.2× latency reduction on Dream GSM8K-CoT, Table 1) that neither approach could achieve alone, establishing a new baseline expectation for what DLM acceleration should target.

The work also provides a diagnostic contribution that clarifies the failure conditions for discrete consistency. The consistency-without-distillation collapse (Table 3, row 2: GSM8K drops to 6.9%, HumanEval-Instruct to 0.0%) demonstrates that self-consistency in discrete token spaces is not intrinsically meaningful—unlike continuous consistency models where following the probability flow ODE guarantees convergence to the data distribution, discrete masked diffusion trajectories can collapse to self-consistent but arbitrary endpoints. This negative result serves as an important boundary condition for anyone adapting consistency-style objectives to language: teacher grounding through distillation is not optional but essential. The paper does not resolve why this collapse occurs at a theoretical level, but the empirical finding itself is sufficiently stark to redirect research away from pure self-consistency formulations in discrete domains.

A subtler but practically significant reframing concerns the caching philosophy. Training-free caching methods (dLLM-Cache, Fast-dLLM Dual Cache) treat caching as an inference-time optimization over a frozen model—an approximation problem where stale KV states are managed through refresh schedules and heuristics. CDLM treats caching as a training objective: the block-wise causal attention mask is applied during fine-tuning, so the model learns to operate under the exact attention constraints it will face at deployment. This eliminates the approximation error inherent in heuristic caching and simplifies inference (one hyperparameter, $\tau_{\text{conf}}$ , vs. the multi-parameter configurations of training-free methods). The performance consequence is visible in the latency gap between CDLM and Fast-dLLM (Par.+D.C.) at similar accuracy levels—CDLM achieves lower latency with fewer steps (e.g., 2.1s at 44.1 steps vs. 2.5s at 60.8 steps on Dream GSM8K-CoT, Table 1), demonstrating that native caching compounds with step reduction in a way approximate caching cannot.

Regarding the broader DLM-vs-AR debate, CDLM's contribution is narrowing the speed gap but widening the visibility of the capability gap. Prior to CDLM, the default assumption was that open-source DLMs were impractically slow, making capability comparisons moot—even if a DLM could theoretically match an AR model on accuracy, nobody would use it because inference was an order of magnitude slower. CDLM eliminates that excuse: with latency reductions of 6.1×–14.5× and throughput matching or exceeding equal-size AR models (Figures 3, 4), DLMs are now fast enough to be seriously compared. But this sharper comparison reveals that current open-source DLMs (Dream-7B, LLaDA-8B) are substantially weaker than comparable AR models on several benchmarks—CDLM–Dream trails Qwen2.5-7B by 28.7 points on MBPP-Instruct (53.0 vs. 81.7), and CDLM–LLaDA trails Llama-3.1-8B by 20.2 points on HumanEval (40.2 vs. 60.4). The paper does not solve this gap, but by making the speed question answerable, it redirects attention to the capability question: the bottleneck is no longer inference speed but model quality. This has the salutary effect of focusing future DLM research on improving base model performance rather than on acceleration gimmicks, confident that acceleration techniques like CDLM can be applied once the base models improve.

Follow-Up Research This Work Enables

Scaling CDLM to longer generation lengths and evaluating whether step reduction factors are length-invariant. All CDLM experiments use $L_g = 256$ . The DLM baseline and CDLM are both configured with exactly 256 refinement steps to generate 256 tokens, and CDLM's step reduction (4.1×–7.7×) is measured against this fixed budget. A natural question is whether the same reduction factor holds at longer generation lengths: if $L_g = 512$ or $L_g = 1024$ , does CDLM achieve similar proportional step reduction (e.g., ~5×), or does the consistency objective become harder as blocks become a smaller fraction of the total sequence? The paper hints at this issue by noting that D2F uses $L_g = 512$ and that CDLM–Dream's MATH degradation may be partly due to insufficient generation budget. A concrete follow-up would train CDLM–Dream at $L_g = 512$ with appropriately scaled trajectory data (generating 512-step teacher trajectories) and measure whether the step reduction factor, latency reduction factor, and accuracy preservation from Tables 1 and 2 generalize. This experiment would also reveal whether the block size $B = 32$ remains appropriate at longer lengths or whether consistency training benefits from proportionally larger blocks (e.g., $B = 64$ at $L_g = 512$ ).

Characterizing the train-inference distribution gap through iterative self-distillation. The paper acknowledges that offline trajectory training creates a mismatch between the states the student encounters during training (teacher-visited) and during inference (self-generated under confidence-thresholded parallel decoding). A direct experiment would quantify this gap: train CDLM–Dream on the standard offline trajectories, then collect new trajectories from the trained CDLM checkpoint itself (which generates faster but potentially visits different states), and compare the distribution of partially unmasked states (measured via token-level entropy, unmasking order correlations, or representation similarity) between the teacher trajectories and the student's self-generated trajectories. If the gap is large, a second experiment would attempt iterative self-distillation: use the CDLM checkpoint from epoch 12 to generate new offline trajectories, then fine-tune from epoch 12 for additional epochs on these on-policy trajectories, and measure whether accuracy improves (suggesting the gap was harmful) or degrades (suggesting the teacher provided essential grounding that on-policy data lacks). This would directly inform whether Appendix B's proposal for online training is worth pursuing or whether offline training with strong teachers is sufficient.

Block size ablation to characterize the consistency difficulty-speedup tradeoff. The paper fixes $B = 32$ throughout without justification beyond matching baseline block-wise decoding conventions. The block size is a critical hyperparameter that controls the maximum jump the consistency objective must learn and the upper bound on tokens per step. A systematic sweep of $B \in \{8, 16, 32, 64, 128\}$ at fixed $L_g = 256$ , measuring both training convergence (epochs to reach stable accuracy) and inference performance (step count, latency, accuracy), would reveal whether there is an optimal $B$ and whether it depends on the teacher's behavior. The hypothesis to test: smaller $B$ makes consistency easier (shorter jumps) but reduces maximum speedup (more blocks → more block transitions → more total steps), while larger $B$ enables greater speedup but risks consistency collapse if the student cannot learn to stay stable across long intervals. The paper's consistency collapse finding (Table 3) suggests this risk is real; understanding how $B$ interacts with collapse susceptibility would directly inform the safety of pushing to larger blocks for greater speedup.

Distillation from autoregressive teachers to break the DLM quality ceiling. The paper's most significant practical limitation is that CDLM cannot exceed its teacher's quality, and current DLM teachers (Dream-7B, LLaDA-8B) are substantially weaker than comparable AR models. The paper suggests distilling from AR teachers as a future direction (Appendix B). A concrete experiment would use Qwen2.5-7B-Instruct (which outperforms Dream-7B on MBPP-Instruct by 28.7 points) as the teacher for a Dream-7B student. The challenge is that AR models do not produce DLM-style trajectories—they generate tokens autoregressively, not through iterative unmasking. A feasible approach: use the AR model to generate final answers (the ground-truth text $\hat{\mathbf{y}}$ ), then run the DLM teacher on these answers to produce the trajectory $\mathcal{T}_x$ and hidden-state buffer $\mathbf{H}_x$ as usual, but replace the ground-truth $\hat{\mathbf{y}}$ with the AR model's output. The DLM teacher provides the trajectory structure; the AR model provides the answer quality. This would test whether CDLM can inherit the AR model's stronger capabilities through the distillation loss while maintaining the DLM's parallel generation advantages. The key metric: does CDLM–Dream trained on Qwen2.5-7B outputs close the accuracy gap on MBPP-Instruct (53.0 vs. 81.7) while maintaining the throughput advantage? If successful, this would make CDLM a bridge between AR model quality and DLM inference speed.

Combining CDLM with inter-block parallelism to further reduce wall-clock latency. CDLM's inference procedure is deliberately simple, processing blocks sequentially with caching across block boundaries. The paper explicitly notes that inter-block parallelism (as in D2F) could be layered on top (Section 6). A direct follow-up would implement D2F-style inter-block speculation on top of CDLM: while the model is processing the current block (bidirectionally), speculatively decode the next block using the partially completed current block as context, accepting or rejecting the speculative tokens once the current block is finalized. This would overlap the latency of sequential block processing, potentially reducing end-to-end latency below what step reduction alone achieves. The experiment would measure the additional latency reduction from inter-block parallelism on top of CDLM, characterizing whether the speedup multiplies (each technique independently reduces latency) or saturates (diminishing returns from overlapping already-fast operations). The comparison against D2F alone at equivalent block sizes would also provide the head-to-head training-based caching comparison currently missing from the paper.

Testing CDLM on DLMs trained with different pre-training objectives or architectures to assess generality. The paper evaluates on two models (Dream, LLaDA) that both use encoder-decoder-style Transformer architectures with masked diffusion pre-training. The consistency+distillation+DLM objective may transfer differently to other DLM architectures—e.g., models using continuous embedding-space diffusion (Diffusion-LM) rather than discrete masked diffusion, or models with different block-wise decoding policies. A stress-test experiment would apply CDLM to a DLM with a fundamentally different generation procedure (e.g., one that stochastically remasks tokens rather than deterministically unmasking them, or one that uses learned scheduling rather than fixed block-wise schedules) and measure whether the step reduction and accuracy preservation properties hold. A negative result (CDLM fails to generalize beyond masked diffusion with deterministic unmasking) would be scientifically valuable: it would reveal that CDLM's consistency formulation is specifically tied to the monotonic, irreversible unmasking property of MDMs, which would narrow the scope of applicability but deepen the theoretical understanding.

Practical Applications and Downstream Use Cases

Batch inference for code generation with quality filtering. CDLM–Dream achieves a 20.9× throughput increase on MBPP-Instruct (2.3 → 48.1 TPS, Table 1) while matching or exceeding the baseline Dream's accuracy (51.8 baseline vs. 53.0 CDLM). In a batch code generation pipeline—for instance, generating candidate solutions to programming problems and filtering by test-case execution—CDLM's throughput advantage directly reduces hardware cost per candidate. With 4× A100 GPUs and CDLM–Dream, a system could generate approximately 48 tokens per second per sample, meaning a batch of 100 problems with 256-token solutions would complete in roughly 530 seconds of GPU time (100 × 256 / 48), versus approximately 11,130 seconds for the vanilla Dream baseline (100 × 256 / 2.3). The key caveat is that this extrapolation assumes single-stream evaluation; batched inference with concurrent requests would likely change the absolute numbers, but the relative advantage (CDLM being ~20× faster than the baseline) should persist because CDLM's step reduction benefits all samples uniformly.

Interactive math tutoring with sub-3-second response latency. CDLM–Dream achieves 78.8 accuracy on GSM8K-CoT (8-shot) with just 2.1 seconds latency (Table 1), compared to 23.5 seconds for the vanilla Dream baseline. For an interactive math tutoring application where a student types a problem and expects a step-by-step solution within a few seconds, CDLM brings DLM-based generation from unacceptably slow (nearly half a minute) to competitive with AR model response times. The accuracy of 78.8 on GSM8K-CoT is slightly behind Qwen2.5-7B-Instruct's 73.8 (Section 5.2.3), meaning CDLM–Dream actually outperforms the equal-size AR model on this specific task while matching its speed. For a tutoring system that prioritizes accuracy on grade-school math over code generation ability, CDLM–Dream on GSM8K-style problems represents a deployable configuration today: sub-3-second latency with state-of-the-art accuracy for the model size.

On-device or edge deployment of smaller DLMs with CDLM acceleration. Although the paper evaluates 7–8B parameter models on datacenter GPUs, the CDLM recipe is architecture-agnostic and relies on LoRA fine-tuning (rank 32–64), which adds only a small fraction of trainable parameters. A plausible deployment scenario is taking a smaller DLM (e.g., 1–3B parameters) fine-tuned with CDLM for a specific domain (e.g., structured data extraction, form completion) and running it on a consumer GPU or even a high-end mobile SoC. The latency reduction factors in the paper (6.1×–14.5×) suggest that a model that was borderline too slow for interactive use (e.g., 5–10 seconds per generation) could become viable (sub-1-second) with CDLM, without the memory overhead of a larger model. The key enabling property is that CDLM's speedups come from training-based architectural changes (block-wise causality + consistency), not from inference-time heuristics that might require per-hardware tuning. The block-wise causal mask and confidence threshold of $\tau_{\text{conf}} = 0.9$ provide a single, portable configuration that should transfer across hardware without recalibration (though the paper does not test this directly).

When to Prefer This Method

The paper positions CDLM primarily for scenarios where DLM inference speed is the bottleneck and the underlying DLM backbone is already sufficiently accurate for the target task. The decision rule is implicit in the results but not stated as a formal recommendation. Based on the evidence:

Prefer CDLM fine-tuning over vanilla DLM inference when the deployment requires interactive or high-throughput generation (latency targets under ~5 seconds, throughput targets above ~20 TPS on A100-class hardware) and the task is one where the base DLM already achieves acceptable accuracy—on GSM8K-CoT for Dream (79.1 baseline accuracy, CDLM achieves 78.8 with 11.2× lower latency) or on HumanEval/MATH for LLaDA (where CDLM improves accuracy while cutting latency). CDLM provides speed with negligible or positive accuracy impact on three of four Dream benchmarks (Table 1) and two of four LLaDA benchmarks (Table 2).
Prefer CDLM over training-free acceleration methods (Fast-dLLM, dLLM-Cache) when the additional 8–16 hours of GPU training time (on 4× A100s) is acceptable and the goal is maximizing both step reduction and caching efficiency simultaneously, rather than optimizing one axis at the cost of the other. The results in Tables 1 and 2 show CDLM achieving lower step counts AND lower latency than any training-free method across all benchmarks.
Prefer CDLM over training a larger DLM from scratch when the base DLM's accuracy is already sufficient for the task and the primary constraint is inference cost or latency—CDLM provides a multiplicative speedup for a fixed model size with only lightweight LoRA fine-tuning, avoiding the pretraining cost of a larger model.
Avoid CDLM (or invest in additional data curation) when the base DLM has marginal accuracy on the target task and the training data for CDLM does not cover the target domain—the LLaDA GSM8K degradation (77.1 → 73.9 despite data augmentation, Table 2) and Dream MATH degradation (38.0 → 32.4, Table 1) demonstrate that CDLM fine-tuning can harm under-represented capabilities. In these cases, either use the vanilla DLM (accepting slower inference), invest in broader training data before applying CDLM, or use an AR model if accuracy is paramount and throughput is secondary.