Training Methods2025~107 minarXiv:2512.14067

Efficient-DLM: From Autoregressive to Diffusion Language Models, and Beyond in Speed

score 9Yonggan Fu, Lexington Whalen, Zhifan Ye +11NVIDIA↑ 16Dec 16, 2025cs.CL

🎯 Pitch

Turning a pretrained autoregressive LLM into a diffusion model can make it 4.5× faster than the leading diffusion model while also beating its original autoregressive accuracy—provided you never let the model look ahead during conversion training and you bias masking toward later tokens to match how inference actually runs.

1. Executive Summary

This paper studies how to convert pretrained autoregressive (AR) language models into diffusion language models (dLMs) that achieve higher inference throughput while preserving task accuracy. The authors identify limitations in the attention patterns and training objectives of prior AR-to-dLM conversion methods and introduce a continuous pretraining scheme built around two mechanisms: a block-wise attention pattern with clean context (concatenating noisy and clean tokens so each block conditions only on already-decoded context during training, unlike the fully bidirectional attention used in Dream) and position-dependent token masking (assigning higher masking probabilities to later tokens within a block as denoising approaches completion, closing the gap between uniform training-time masking and the left-to-right tendency observed during confidence-based sampling at inference). The resulting Efficient-DLM 8B achieves +5.4% higher accuracy with 4.5× higher throughput compared to Dream 7B and +2.7% higher accuracy with 2.7× higher throughput compared to Qwen3 4B, while matching or slightly exceeding the accuracy of Qwen3 8B, establishing that AR-to-dLM conversion can produce models that surpass both prior dLMs and comparably-sized AR models on accuracy–throughput trade-offs only when the attention pattern preserves block-wise causality and the masking distribution aligns with the inference-time decoding tendency.

2. Context and Motivation

The Core Problem: dLMs Promise Speed but Can't Compete with AR Models on Accuracy

The fundamental tension this paper addresses is one of asymmetric maturity in language model architectures. Autoregressive (AR) language models—think Qwen, Llama, GPT—have been trained at enormous scale, with trillions of tokens and billions of parameters, producing models that are remarkably capable across coding, math, reasoning, and factual knowledge. But they suffer from a well-known bottleneck: tokens must be generated one at a time, left to right, which limits throughput, particularly in memory-bounded inference scenarios where the model cannot amortize the cost of loading weights across a large batch of concurrent requests.

Diffusion language models (dLMs) offer a compelling escape from this bottleneck. Instead of generating tokens sequentially, dLMs start with a fully masked (or partially masked) sequence and iteratively denoise it—predicting multiple tokens in parallel at each step. In principle, this parallelism should translate into higher throughput: if a dLM can denoise 3 tokens per forward pass, it needs roughly one-third the number of sequential model calls compared to an AR model generating the same sequence length.

The reality, however, is less rosy. As the paper notes in Section 1:

"despite their conceptual appeal, most existing dLMs have not delivered faster speed than AR models in practice [nie2025large, ye2025dream], due to the limited compatibility with key-value (KV) caching and the limited parallelism during decoding."

This is the core gap: dLMs theoretically enable parallel generation but practically underperform AR models on both accuracy and throughput when compared at meaningful scales. LLaDA 8B, for example, achieves only 54.92% average accuracy across 12 tasks with a throughput of 25 tok/sec on an H100—compared to Qwen3 8B's 71.58% accuracy at 42.51 tok/sec (Table 3). Dream 7B reaches 65.30% accuracy at 28.11 tok/sec. These are not competitive numbers; they represent dLMs that are simultaneously less accurate and slower than their AR counterparts, despite their architectural promise of parallelism.

Why This Matters: The Training Cost Barrier

If the only problem were accuracy, the obvious solution would be to train better dLMs from scratch—larger models, more data, longer training. But this runs into a second-order problem that the paper identifies:

"successful scaling of dLMs to larger model sizes has been restricted by prohibitive training costs [nie2024scaling]. This is because AR models learn only left-to-right modeling, while dLMs learn all possible permutations [xue2025any], which is more difficult and requires longer training."

The key insight here is about learning difficulty. An AR model has a single, fixed decomposition of the joint probability: $p(x_1, x_2, ..., x_L) = \prod_{\ell=1}^L p(x_\ell \mid x_{<\ell})$ . The model only ever needs to predict $x_\ell$ given all preceding tokens—one conditional distribution per position. A dLM, by contrast, must be able to denoise any subset of positions—it needs to model $p(x_i \mid \text{context}, \text{some subset of other tokens})$ for all possible corruption patterns. This is combinatorially many more conditional distributions, making the learning problem fundamentally harder. From-scratch dLM training therefore requires substantially more compute than AR training to reach comparable quality, which is economically prohibitive at large scale.

This is where AR-to-dLM conversion enters as the strategic alternative. Rather than training a dLM from scratch, start with a pretrained AR model that already possesses strong language understanding and reasoning capabilities, then adapt it to the diffusion paradigm through additional training. If done correctly, the adaptation requires far less compute than from-scratch training (tens to hundreds of billions of tokens rather than trillions) while preserving the pretrained model's capabilities. The paper frames this explicitly:

"This work leverages pretrained AR models for initialization and systematically explores how to continuously pretrain them into dLMs that achieve high generation speed while preserving task accuracy."

This matters for real-world deployment because most organizations deploying LLMs are not training from scratch—they are fine-tuning, adapting, or otherwise building on existing pretrained models. A recipe for converting those models into faster dLMs without substantial accuracy loss would be immediately actionable.

Prior Approaches and Their Shortcomings

The paper engages with three threads of prior work, each with specific limitations that motivate the proposed approach.

Fully bidirectional dLM training from scratch (LLaDA, MDLM variants). The original generation of scaled dLMs—including LLaDA 8B [nie2025large] and MDLM-style models [sahoo2024simple]—trained from scratch with fully bidirectional attention, where every token can attend to every other token during denoising. While conceptually clean (bidirectional context should help with denoising), this design has two practical problems. First, it is incompatible with KV caching: because the attention pattern changes at every denoising step (tokens become unmasked, changing what should be attended to), standard KV caching techniques developed for AR models don't apply, forcing recomputation of keys and values for the entire sequence at each step. Second, from-scratch training struggles to match AR accuracy at equivalent model sizes and training budgets, as discussed above.

AR-initialized dLMs with fully bidirectional attention (Dream). Dream [ye2025dream] introduced the idea of initializing dLM training from pretrained AR weights and training with fully bidirectional attention plus a token shift mechanism (predicting the next token after a masked position, mimicking AR behavior). This is a significant step forward—it reduces training cost and improves accuracy over from-scratch dLMs. However, the paper identifies specific limitations in this approach (Section 2.1):

Fully bidirectional attention still prevents native KV caching, limiting practical throughput.
The context is "overly corrupted": during training, when the entire sequence is randomly masked, tokens that should condition on largely clean left context in practice (because left context was already decoded during inference) instead see heavily masked context during training. This creates a train–test mismatch that degrades accuracy.
Bidirectional attention diverges from the causality of AR initialization, causing larger weight drifts from the pretrained model and thus losing more of the pretrained capabilities.

The empirical evidence for this limitation is stark. Table 1, Row (b) shows that applying Dream's training scheme to Qwen2.5 1.5B yields an average accuracy of only 18.10% across six generation tasks—a dramatic drop from the original AR model's 41.79%. Even removing the token shift (Row c) only improves this to 19.29%. Something about the fully bidirectional training regime is fundamentally damaging the pretrained model's capabilities.

Block diffusion trained from scratch (Block Diffusion). The block-wise attention pattern—where the sequence is partitioned into blocks, attention remains causal across blocks, and bidirectional within each block—was introduced by Arriola et al. [arriola2025block] for dLMs trained from scratch. This design enables native KV caching (keys and values from completed blocks are preserved and reused) and brings the attention pattern closer to the AR paradigm. However, this prior work only demonstrated results on small-scale models (e.g., 110M parameters) and trained from scratch. The paper observes that "successful scaling of dLMs to larger model sizes has been restricted by prohibitive training costs" [nie2024scaling], and block diffusion from scratch at scale had not been demonstrated. Moreover, prior block diffusion work did not condition each block on clean context during training—a detail that Section 2.2 shows is critical, contributing a 9.46% accuracy improvement.

Uniform token masking. Existing dLMs uniformly sample mask tokens based solely on a noise level $t$ , independent of position. The paper identifies a training–test gap in this approach (Section 3.1): during confidence-based sampling at inference, tokens are decoded with a clear left-to-right tendency—earlier tokens in a block are denoised and gain confidence first, while later tokens remain masked longer. Figure 6(a) shows the average number of denoising steps at each position, demonstrating that "the average number of denoising steps increases with the positions in a block, exhibiting a notable left-to-right tendency due to the autoregressive nature of language." Training with uniform masking means the model never learns that later positions are more likely to be masked when the denoising process is nearly complete—a distribution shift between training and inference that degrades accuracy, particularly under aggressive parallel decoding (fewer denoising steps, more tokens per forward).

How This Paper Positions Itself

The paper frames its contribution not as proposing a single novel method but as systematically studying the design space of AR-to-dLM conversion and identifying principles that make it work. The key claims are:

Attention pattern matters enormously, and the choice between fully bidirectional, block-wise without clean context, and block-wise with clean context is not a minor implementation detail—it is the dominant factor determining whether conversion succeeds. The block-wise attention with clean context pattern (Figure 2d) is presented as a "win–win in accuracy and efficiency": it enables KV caching for throughput, better preserves pretrained weights for accuracy, and aligns training and inference attention patterns.
Token masking distribution is an overlooked design factor. The training–test gap in mask token positions is not just a theoretical concern—it has measurable impact on downstream accuracy, with position-dependent masking yielding up to +4.38% improvement on the most aggressive parallel decoding settings (Table 2, $\lambda=0.1$ vs. uniform).
Training dynamics reveal that accuracy–throughput trade-offs improve with longer training. Section 4 shows that extended continuous pretraining (on the order of 100B+ tokens) improves not just raw accuracy but specifically the model's ability to generate tokens in parallel with fewer denoising steps—a dimension of dLM quality that prior work had not systematically characterized.

The paper positions Efficient-DLM as a practical recipe rather than a fundamental architectural innovation, aimed at guiding the community toward building dLMs that actually deliver on the promise of faster inference:

"Beyond serving as a practical recipe for AR-to-dLM conversion, our results highlight the broader opportunity to rethink pretraining, masking, and decoding strategies for dLMs in order to realize their promise as alternatives to AR models." (Section 7)

Relative to prior work, the paper makes a deliberate shift from training dLMs from scratch (LLaDA, MDLM) or minimal adaptation of AR models with bidirectional attention (Dream) to careful adaptation with attention patterns that respect the causal structure of the pretrained model and masking distributions that match inference-time behavior. The empirical evidence argues that this shift is not incremental—Row (b) vs. Row (g) in Table 1 shows a jump from 18.10% to 38.41% average accuracy on exactly the same base model, purely through better training design choices.

3. Technical Approach

3.1 Reader orientation (approachable technical breakdown)

This paper develops a continuous pretraining recipe for converting already-trained autoregressive (AR) language models into diffusion language models (dLMs) that generate tokens in parallel blocks rather than one-at-a-time. The core problem is that existing conversion methods drop the pretrained model's accuracy substantially because they force it into a fully bidirectional attention pattern that clashes with the causal structure it learned during AR training. The solution involves two design choices: (1) a block-wise attention pattern where each block attends to clean (already-decoded) context and uses bidirectional attention only within its own boundaries, preserving the causal flow across blocks that the AR model expects, and (2) a position-dependent token masking strategy that makes training-time corruption patterns match the left-to-right decoding tendency that emerges during confidence-based inference.

3.2 Big-picture architecture (diagram in words)

The system has four major components:

Pretrained AR base model (Qwen2.5-1.5B, Qwen3-4B, or Qwen3-8B) — the starting point, trained with standard causal language modeling. Its weights encode strong reasoning, coding, and factual knowledge, but it generates tokens one at a time left-to-right.
Block-wise attention mask with clean context — a modified attention pattern applied during continuous pretraining that partitions each sequence into fixed-size blocks, keeps attention causal across blocks (each block sees previous blocks' clean tokens), and enables full bidirectional attention within each block. This mask is applied by concatenating the noisy tokens of block $b$ with the clean tokens of blocks $1$ through $b-1$ and constructing an attention mask that prevents noisy tokens from attending to future blocks while letting clean tokens attend freely to all prior clean context.
Position-dependent token masking module — a sampling procedure that determines which positions within a block receive mask tokens at each training step. Rather than uniformly sampling $k$ positions to mask (where $k = \lfloor t L' \rceil$ for block size $L'$ and noise level $t$ ), this module assigns higher masking probability to later positions within the block as $t \to 0$ , producing mask distributions that concentrate toward the block end when the sequence is nearly clean.
Confidence-based parallel decoder (inference only) — at test time, the trained dLM iteratively denoises a fully masked prompt. At each denoising step, the model predicts token probabilities for all currently-masked positions; tokens whose predicted probability exceeds a confidence threshold $\tau$ are "decoded" (fixed to their predicted values) and removed from the mask set for subsequent steps, while tokens below the threshold remain masked for further refinement. The number of tokens decoded per forward pass determines the effective parallel speed-up.

Information flows as follows: a pretrained AR model enters the pipeline → during continuous pretraining, sequences are partitioned into blocks of size $L'$ → for each block, $k = \lfloor t L' \rceil$ positions are selected for masking using the position-dependent distribution $w_i(t)$ → the noisy block tokens are concatenated with clean context from previous blocks → the model sees this concatenated sequence with the block-wise attention mask and predicts the original clean tokens for the masked positions (without token shift) → the loss is computed as the negative log-likelihood of the ground-truth tokens at masked positions, weighted by $1/t$ → after training, the model generates by iteratively denoising blocks with confidence-based sampling, producing variable numbers of tokens per forward pass.

3.3 Roadmap for the deep dive

First, the training objective (Equation 1) — what loss the model minimizes during continuous pretraining, how the $1/t$ weighting works, and how the block-conditional form differs from prior work. This defines what the model is being trained to do.
Second, the attention pattern — the block-wise mask with clean context illustrated in Figure 2(d), why it requires concatenating noisy and clean tokens, how KV caching is enabled, and the empirical evidence (Table 1, Figure 2e) for why this pattern preserves pretrained weights better than fully bidirectional attention.
Third, the token masking strategy — the position-dependent weighting function $w_i(t) = \exp[\beta(1-t)i]$ , the half-life parameterization $\lambda$ , the Gumbel-top- $k$ sampling procedure, and how this bridges the gap between uniform training-time masking and the left-to-right decoding pattern observed during inference (Figure 6).
Fourth, block size analysis — how the training block size and evaluation block size interact (Figure 3), why larger models tolerate larger blocks, and the practical guideline for selecting block sizes during conversion.
Fifth, training dynamics — how accuracy on likelihood tasks and generation tasks evolves with token budget (Figure 8), and why longer training specifically improves the model's ability to generate tokens in parallel with fewer denoising steps.
Sixth, inference procedure — confidence-based sampling with the block-wise attention pattern, how tokens per forward (TPF) is controlled, and how the model achieves variable accuracy–throughput trade-offs from a single checkpoint.

3.4 Detailed, sentence-based technical breakdown

This is primarily an empirical analysis and systems design paper whose core idea is that successful AR-to-dLM conversion depends critically on two design factors: an attention pattern that preserves the pretrained model's causal structure, and a token masking distribution that matches the inference-time left-to-right decoding tendency. The paper systematically ablates these factors and provides a complete recipe.

Training Objective: Block-Conditional Masked Denoising

The model is trained to denoise individual blocks of tokens, where each block is conditioned on the clean (ground-truth, unmasked) tokens from all preceding blocks. This is formalized in Equation 1:

$\mathcal{L}(\theta) = \mathbb{E}_{t \sim \mathcal{U}[0,1]} \; \mathbb{E}_{\tilde{\mathbf{x}}^b_t \sim q(\cdot \mid \mathbf{x}^b)} \Bigg[ -\frac{1}{t} \sum_{b=1}^{B} \log p_\theta(\mathbf{x}^b \mid \tilde{\mathbf{x}}^b_t, \mathbf{x}^{<b}) \Bigg]$

where $\theta$ represents the model parameters being optimized, $t \in [0, 1]$ is the noise level sampled uniformly at each training step, $B$ is the total number of blocks in the sequence, $\mathbf{x}^b$ is the ground-truth token sequence for block $b$ , $\tilde{\mathbf{x}}^b_t$ is the corrupted (partially masked) version of block $b$ at noise level $t$ , $\mathbf{x}^{<b}$ is the clean token sequence for all blocks preceding block $b$ , $q(\cdot \mid \mathbf{x}^b)$ is the forward corruption process that randomly selects and applies mask tokens, and $p_\theta(\mathbf{x}^b \mid \tilde{\mathbf{x}}^b_t, \mathbf{x}^{<b})$ is the model's predicted probability of the ground-truth tokens at the masked positions given the corrupted block and the clean context.

What it computes: for each block in the sequence, the model receives (1) the corrupted version of that block, where $k = \lfloor t L' \rceil$ tokens have been replaced with mask tokens, and (2) the clean tokens from all previous blocks. The model must predict the original tokens at the masked positions. The loss is the negative log-likelihood of the correct tokens, summed across all blocks and weighted by $1/t$ . The expectation is taken over both the noise level $t$ (sampled uniformly from $[0,1]$ ) and the specific mask token placement (sampled from the corruption process $q$ , which itself depends on the position-dependent masking strategy described in Section 3.2).

Why this form: the $1/t$ weighting is a standard technique in diffusion models that up-weights losses at low noise levels, where the model must make fine-grained distinctions between nearly-correct and correct tokens. Without this weighting, the loss would be dominated by high noise levels (where many tokens are masked and the prediction task is relatively easy), and the model would under-invest in learning the precise denoising needed when only a few tokens remain masked. The block-conditional form $\log p_\theta(\mathbf{x}^b \mid \tilde{\mathbf{x}}^b_t, \mathbf{x}^{<b})$ is the critical departure from prior work: Dream and LLaDA use $\log p_\theta(\mathbf{x}^b \mid \tilde{\mathbf{x}}^{<b}_t)$ or similar, where previous blocks are also corrupted during training. By replacing the corrupted context $\tilde{\mathbf{x}}^{<b}$ with clean context $\mathbf{x}^{<b}$ , the training objective exactly matches the inference-time scenario where all previous blocks have already been fully decoded. The authors explicitly state the alternative formulation for comparison:

"the block-wise attention variant in Fig. 2(c) without using clean context can be formulated by replacing the clean context $\mathbf{x}^{<b}$ in Eq. 1 with the corrupted context $\tilde{\mathbf{x}}^{<b}$ ."

This seemingly small distinction—clean versus corrupted context—produces a 9.46% accuracy difference in Table 1 (Row d vs. Row f), making it one of the largest single-factor improvements in the paper.

The training objective is applied to a model initialized from a pretrained AR model that was originally trained with the standard autoregressive loss $\mathcal{L}_{\text{AR}}(\theta) = -\sum_{\ell=1}^{L} \log p_\theta(x_\ell \mid x_{<\ell})$ . This initialization is what makes the conversion possible at modest token budgets (tens of billions of tokens rather than trillions): the model already knows how language works and how to reason; the continuous pretraining only needs to teach it the new attention pattern and the denoising task.

Token shift is explicitly removed. Prior work (Dream, Gong et al.) preserved a "token shift" mechanism where the model predicts the next token after a masked position, mimicking AR behavior. Efficient-DLM predicts the masked token itself directly. Removing token shift improves accuracy across settings (Table 1, Rows b vs. c and f vs. g), with the authors hypothesizing that "predicting the mask token itself (without token shift) is easier than predicting the next token of a masked position" because the latter forces the model to simultaneously infer the masked token and predict the following one.

Block-Wise Attention Pattern with Clean Context

The attention pattern is the most consequential design choice in the paper. Rather than allowing every token to attend to every other token (fully bidirectional, as in Dream and LLaDA), or allowing bidirectional attention within blocks but with corrupted context (Figure 2c), the paper uses the attention mask shown in Figure 2(d).

Construction of the attention mask. For a given block $b$ , the input to the model consists of two concatenated segments:

Clean context tokens: the ground-truth tokens $\mathbf{x}^{<b}$ for all blocks preceding block $b$ . These tokens contain no mask tokens and represent the already-decoded prefix that would be available at inference time.
Noisy block tokens: the corrupted tokens $\tilde{\mathbf{x}}^b_t$ for block $b$ , where a subset of positions have been replaced with mask tokens.

The attention mask is then constructed as follows:

Clean context to clean context: full bidirectional attention among clean context tokens (they can all see each other).
Clean context to noisy tokens: no attention — clean context tokens cannot see into the current noisy block.
Noisy tokens to clean context: full attention — every noisy token can attend to all clean context tokens. This is the "conditioning on clean context" that gives the approach its name.
Noisy tokens to noisy tokens: full bidirectional attention within block $b$ — every noisy token can attend to every other noisy token in the same block.
Noisy tokens to future blocks: no attention — noisy tokens in block $b$ cannot see any tokens from blocks $b+1$ and beyond.

The paper's notation describes this compactly: " " denotes attention among noisy tokens, " " denotes attention from noisy tokens to clean-context tokens, and " " denotes attention within the clean context (Figure 2d caption).

How this enables KV caching. At inference time, the model generates blocks sequentially. Once block $b$ is fully decoded (all its tokens have been denoised and fixed), the keys and values computed for those tokens will never change—they are clean tokens that future blocks will attend to via the "noisy tokens to clean context" connection. These key-value pairs can be cached and reused across all denoising steps of future blocks, avoiding recomputation. This is not possible with fully bidirectional attention (Figure 2b), where every token's keys and values depend on the entire sequence's state, which changes at every denoising step as tokens become unmasked. The block-wise pattern localizes the changing computation to the current block only, making the rest cacheable.

Why this preserves AR weight distributions. The key empirical finding is shown in Figure 2(e), which visualizes the magnitude of weight changes (measured as the Frobenius norm difference between initial and final weights) in both attention and feed-forward network (FFN) layers after training under different attention patterns. Fully bidirectional attention (Figure 2b) causes the largest weight changes in both layer types—the model must substantially rewire its internal representations to handle the new attention pattern. Block-wise attention without clean context (Figure 2c) causes intermediate weight changes. Block-wise attention with clean context (Figure 2d) causes the smallest weight changes, particularly in FFN layers.

The mechanism behind this preservation is the block-wise causality: in the pretrained AR model, every token attends causally to all previous tokens in the sequence. In the block-wise attention pattern, the causal flow across blocks is preserved—block $b$ still sees blocks $1$ through $b-1$ in their clean, ground-truth form, just as an AR model would see the preceding context. The only difference is that within block $b$ , attention becomes bidirectional rather than left-to-right. This structural similarity means the pretrained weights—which encode the ability to use left context for prediction—can be largely reused, with adaptation needed only for the within-block bidirectionality. The authors frame this as:

"this approach can better preserve pretrained AR models' weight distributions than the fully bidirectional modeling used in prior work such as Dream, in addition to its known benefit of enabling KV caching, and leads to a win–win in accuracy and efficiency" (Section 1).

Training sequence length and block granularity. Because the input now concatenates clean context and noisy block tokens (rather than processing only the noisy sequence as in fully bidirectional training), the effective sequence length during training is larger. The paper accounts for this in its comparisons: Row (e) of Table 1 trains the block-wise-without-clean-context variant with $2\times$ the token budget to match the increased sequence length of the clean-context variant. Even with the doubled budget, this variant (27.73% average accuracy) substantially underperforms the clean-context variant (37.69%, Row f), demonstrating that the benefit comes from the presence of clean context, not merely from seeing more tokens.

The attention pattern used during inference is identical to training. During block-wise decoding at test time, the model processes one block at a time: block $b$ 's tokens start fully masked, the model iteratively denoises them (with full bidirectional attention within the block and attention to all clean preceding blocks), and once denoising completes, the tokens become clean context for block $b+1$ . This is exactly the scenario the training objective simulates, closing the training–test gap that would exist if training used corrupted context (Figure 2c) but inference used clean context.

Position-Dependent Token Masking

The second major design innovation addresses a mismatch between how masks are placed during training and where they actually appear during inference. This gap exists because of how confidence-based sampling works in practice.

The training–test gap identified. During training, existing dLMs (LLaDA, Dream) uniformly sample $k = \lfloor t L' \rceil$ positions to mask within each block, independently of position. For a noise level $t = 0.3$ and block size $L' = 64$ , roughly 19 positions are masked, and each of the 64 positions has equal probability of being selected. During inference with confidence-based sampling, however, tokens are decoded with a strong left-to-right bias. Figure 6(a) visualizes this: the average number of denoising steps required at each token position in a block shows a clear increasing trend from left (position 1) to right (position 64). Earlier tokens are decoded in fewer steps—they gain confidence quickly because they have clean left context and are part of the natural left-to-right flow of language. Later tokens require more steps because they depend on earlier tokens being decoded first.

The consequence is a distribution shift: during training, mask tokens are equally likely to appear at any position. During inference, as the denoising process approaches completion (low effective $t$ ), the remaining mask tokens are concentrated at the right end of each block. The model was never trained on this distribution—it never saw examples where, at low noise levels, most masks are at the block end. This mismatch degrades accuracy, particularly when the model is asked to decode aggressively (low number of denoising steps, high tokens per forward), because the later denoising steps—where the positional distribution matters most—are precisely where the training distribution is most wrong.

Position-dependent masking weights. To bridge this gap, the paper introduces a weighting function that assigns higher masking probability to later positions as the noise level decreases:

$w_i(t) = \exp\!\big[\beta\,(1-t)\,i\big]$

where $i \in [1, L']$ is the relative token position within a block (1 = first token, $L'$ = last token), $t \in [0,1]$ is the noise level, and $\beta \geq 0$ is a hyperparameter controlling the strength of the positional bias.

What it computes: for each position $i$ in a block and each noise level $t$ , this function outputs a scalar weight. When $t \to 1$ (high noise, many masked tokens), $(1-t) \to 0$ , so $w_i(t) \to 1$ for all positions—the distribution becomes approximately uniform. When $t \to 0$ (low noise, few masked tokens), $(1-t) \to 1$ , so $w_i(t) = \exp(\beta i)$ —positions with larger $i$ (further right in the block) get exponentially higher weights. When $t=0$ exactly (which never occurs during training since $t \sim \mathcal{U}[0,1]$ , but represents the extreme), the weights are maximally tilted toward the right.

Why this form: the $\exp(\cdot)$ function ensures positivity and provides a smooth interpolation between uniform ( $t \to 1$ ) and right-weighted ( $t \to 0$ ) distributions controlled by a single parameter $\beta$ . The $(1-t)$ factor is the key: it couples the positional bias to the noise level, so the distribution automatically becomes more position-dependent as the denoising process progresses, matching the inference-time pattern. Alternative forms—such as a step function that switches from uniform to right-weighted at some threshold $t$ —would introduce a discontinuity that the model might exploit or that might cause training instability.

Half-life parameterization for practical tuning. Rather than directly tuning $\beta$ (which would have units of 1/position, making it hard to interpret), the paper introduces a reparameterization using the half-life ratio $\lambda$ :

$\lambda = \frac{\ln 2}{\beta L'} \in (0, 1]$

where $L'$ is the block size. The value $\lambda$ is the fraction of a block over which, under maximal tilt ( $t \to 0$ ), the positional weight changes by a factor of two. For example, $\lambda = 0.1$ and $L' = 64$ means the weight doubles every $0.1 \times 64 = 6.4$ positions. A smaller $\lambda$ means a stronger positional prior (weights increase more rapidly from left to right). $\lambda \to \infty$ recovers $\beta = 0$ , i.e., uniform masking. $\lambda \to 0$ recovers $\beta \to \infty$ , i.e., the extreme case of always masking the rightmost tokens.

Sampling mask tokens from the distribution. Given the weights $w_i(t)$ for all positions in a block, the set of $k = \lfloor t L' \rceil$ mask positions is drawn using Gumbel-top- $k$ sampling [huijben2022review]. This procedure adds independent Gumbel noise to each log-weight $\log w_i(t)$ and selects the $k$ positions with the largest perturbed values. Gumbel-top- $k$ is equivalent to sampling without replacement from the categorical distribution defined by the normalized weights $w_i(t) / \sum_j w_j(t)$ , but is computationally more efficient for large $L'$ .

Average masking probabilities (Figure 7). The figure shows the empirical masking probability for each position within a block of size $L' = 64$ , averaged over the entire training distribution $t \sim \mathcal{U}[0,1]$ . For uniform masking ( $\lambda \to \infty$ ), all positions have equal probability. For $\lambda = 0.25$ , there is a slight upward slope from left to right. For $\lambda = 0.1$ , the slope is more pronounced—later positions are masked approximately $1.5\times$ more often than earlier positions. For $\lambda = 0.05$ , the tilt is even stronger. For right-to-left masking ( $\lambda \to 0$ ), only the rightmost $k$ positions are ever masked—a distribution that is too extreme and performs poorly (Table 2, "right-to-left" row) because the model never learns to denoise earlier positions and misses the benefits of bidirectional context.

Why position-dependent masking helps most under aggressive parallel decoding. Table 2 compares masking schemes at different levels of parallel decoding, measured in tokens per forward (TPF)—the average number of tokens decoded at each denoising step. Higher TPF means more aggressive parallelism and fewer total denoising steps. At TPF=1 (one token per step, essentially AR-like decoding), $\lambda = 0.1$ provides only a +1.75% improvement over uniform masking. At TPF=5.6 (the most aggressive setting), $\lambda = 0.1$ provides a +4.38% improvement. This gradient is consistent with the mechanism: aggressive decoding concentrates mask tokens at the block end in the final denoising steps, exactly where the position-dependent training distribution provides the largest benefit. At TPF=1, the distribution mismatch is less severe because tokens are decoded in small numbers and the mask token distribution never becomes strongly skewed.

Block Size Selection and Interaction

The block size $L'$ is a critical hyperparameter that affects both training dynamics and inference behavior. Larger blocks provide richer context (more tokens can see each other bidirectionally) but introduce more corruption (tokens at the end of a large block see a longer prefix of potentially-noisy tokens). The paper studies the interaction between training block size and evaluation block size in Figure 3.

Training block size sweep. The authors train diffusion Qwen2.5 1.5B (50B tokens) and Qwen3 4B (25B tokens) with training block sizes of $[4, 8, 16, 32, 64, 128]$ . For each trained model, they evaluate with multiple evaluation block sizes and report average accuracy across six generation tasks.

Key finding 1: small models have a clear sweet spot. For Qwen2.5 1.5B, the optimal training block size is 16. Training with smaller blocks (4, 8) provides insufficient context richness—the model cannot leverage enough bidirectional information to denoise effectively. Training with larger blocks (32, 64, 128) introduces too much corruption—tokens at the end of the block see noisy prefixes, making denoising harder. The sweet spot at 16 balances context richness against corruption.

Key finding 2: larger models tolerate larger blocks. For Qwen3 4B, training block sizes of 64 and 128 perform comparably, both outperforming smaller blocks. Larger models have greater capacity to handle the increased corruption in larger blocks while benefiting from the richer bidirectional context. This suggests a scaling property: optimal block size grows with model capacity.

Key finding 3: training generalizes across evaluation block sizes. A model trained with a single block size (e.g., 16) can be evaluated with different block sizes and achieve reasonable accuracy. For example, the Qwen2.5 1.5B model trained with block size 16 achieves strong accuracy when evaluated with block sizes 8, 16, 32, and even 64. This generalization occurs because the block-wise attention mask naturally exposes the model to varying effective sequence lengths during training: within a fixed block size of 16, the model sees attention patterns where different numbers of tokens participate (some positions are masked, some are clean), which approximates the variability it would encounter at different evaluation block sizes. This contrasts with fully bidirectional attention (Figure 2b), where the model always sees the same number of tokens participating in attention, necessitating additional techniques like random sequence length truncation to generalize to other sequence lengths, as noted:

"the model trained with a single block size can see varying numbers of tokens participating in the attention mechanism. This differs from the fully bidirectional attention in Fig. 2(b), where the model always sees the same number of tokens participating in attention, necessitating additional techniques such as random sequence length truncation [nie2025large] to generalize to other sequence lengths."

Weight change patterns across block sizes (Figure 4). Larger training block sizes lead to larger weight changes in both attention and FFN layers. There exists a sweet-spot block size where the trade-off between preserving pretrained capabilities (small weight changes) and adapting to the new attention pattern (sufficient weight changes) is balanced. For Qwen2.5 1.5B, this sweet spot is block size 16.

Evaluation block size and parallel decoding (Figure 5). When performing aggressive parallel decoding (low NFE, high tokens per forward), larger evaluation block sizes yield higher accuracy. The mechanism: with fewer denoising steps, the model must decode more tokens per step, and larger blocks provide more candidate positions for parallel decoding—if 10 tokens need to be decoded in one step, a block of size 64 has 64 possible positions to choose from, while a block of size 16 only has 16, making it more likely that the 10 highest-confidence tokens are spread across positions rather than concentrated. At higher NFEs (more denoising steps, fewer tokens per step), the evaluation block size matters less, and moderate block sizes all perform comparably.

Practical guideline. The paper's final models use block size 16 for Efficient-DLM 1.5B (trained from Qwen2.5 1.5B) and block size 64 for Efficient-DLM 4B and 8B (trained from Qwen3 4B and 8B), with evaluation block sizes of 16 and 32 respectively. The training block size is selected as the optimal for each model scale; the evaluation block size is selected to balance accuracy and throughput.

Training Dynamics and Token Budget Scaling

Section 4 studies how model performance evolves over the course of continuous pretraining, tracking both likelihood-based task accuracy and generation task accuracy (with and without parallel decoding) as a function of training tokens consumed.

Experimental setup. Qwen2.5 1.5B is trained for 200B tokens with the optimal configuration from Sections 2 and 3 (block-wise attention with clean context, block size 16, no token shift, position-dependent masking). The model is evaluated periodically on:

Likelihood tasks: multiple-choice tasks where accuracy is computed by estimating and selecting the largest likelihood among candidate answers.
Generation tasks: coding and math tasks where the model generates free-form responses, evaluated at different NFE levels to measure the accuracy–throughput trade-off.

Finding 1: rapid recovery of task accuracy. With relatively low training cost (on the order of 10B tokens), the converted dLM largely recovers the task accuracy of the original AR model. Figure 8(a) shows accuracy on likelihood tasks rising quickly in the first ~25B tokens and then improving more gradually. This rapid recovery is enabled by the initialization from pretrained AR weights and the block-wise attention pattern that preserves those weights.

Finding 2: likelihood estimation improves monotonically. Accuracy on likelihood tasks continues to improve throughout the 200B token training run, without signs of plateauing. The model becomes progressively better at estimating the probability of correct tokens under noisy conditions.

Finding 3: generation accuracy improves with fluctuations. On generation tasks (Figure 8b-d), accuracy at the rightmost point of each curve (equivalent to TPF=1, one token per step) also improves over time, though with more task-specific fluctuation than likelihood tasks.

Finding 4: parallel decoding ability improves with longer training. The most important finding from the training dynamics analysis is that longer training improves the accuracy–NFE trade-off: at a given NFE budget, a model trained for 200B tokens achieves higher accuracy than a model trained for 50B or 100B tokens. This is visible in Figure 8(b-d) as the upward shift of the entire accuracy–NFE curve with more training tokens. The mechanism: stronger likelihood estimation produces more accurate and reliable confidence scores. When confidence-based sampling uses these scores to decide which tokens to decode, more accurate scores mean the model is less likely to prematurely decode tokens that are actually wrong (high confidence but incorrect prediction) and less likely to leave tokens masked that are actually correct (low confidence but correct prediction). This directly enables more aggressive parallel decoding without proportionally sacrificing accuracy.

Practical implication. This finding motivates the paper's decision to train Efficient-DLM 1.5B/4B for 300B tokens and Efficient-DLM 8B for 500B tokens—substantially more than the 25-50B token budgets used in the ablation studies. The extra training tokens translate not just to higher raw accuracy but specifically to better accuracy under aggressive parallel decoding, which is exactly the regime where dLMs' throughput advantage over AR models is maximized.

Inference: Confidence-Based Parallel Decoding with Block-Wise Attention

At inference time, the trained dLM generates text through an iterative denoising process that produces a variable number of tokens per forward pass, controlled by a confidence threshold.

Initialization. The model receives a prompt (which may be empty for unconditional generation). The prompt tokens are treated as clean context (block 0, already decoded). The first block to be generated is initialized as a sequence of $L'$ mask tokens, where $L'$ is the evaluation block size (16 for Efficient-DLM 1.5B, 32 for Efficient-DLM 4B/8B).

Iterative denoising loop for a single block. For the current block being generated:

Forward pass: the model processes the concatenation of clean context (all previously completed blocks) and the current block's tokens (a mix of mask tokens and already-decoded tokens). The attention pattern follows Figure 2(d): current-block tokens attend bidirectionally to each other and causally to clean context; clean context attends bidirectionally to itself and not to current-block tokens.
Token prediction: for every position that is currently a mask token, the model outputs a probability distribution over the vocabulary. The predicted token for each masked position is $\hat{x}_i = \arg\max_{v} p_\theta(v \mid \text{current block state}, \text{clean context})$ .
Confidence-based filtering: for each predicted token $\hat{x}_i$ , the model's predicted probability $p_\theta(\hat{x}_i \mid \ldots)$ is compared to a confidence threshold $\tau$ . If the probability exceeds $\tau$ , the token is "decoded"—it is fixed to $\hat{x}_i$ and removed from the mask set. If the probability is below $\tau$ , the position remains masked for the next denoising step. This implements the intuition that the model should only commit to predictions it is confident about.
KV cache update: as tokens are decoded, their keys and values can be cached. Decoded tokens within the current block have fixed values, so their KV entries will not change in subsequent denoising steps. Clean context tokens are already cached. Only the still-masked tokens require recomputation of keys and values at each step.
Termination: the denoising loop for the block continues until all positions are decoded or a maximum number of steps is reached. The completed block is then appended to the clean context, and generation proceeds to the next block.

The loop repeats for successive blocks until the model generates an end-of-sequence token or reaches a maximum generation length.

Tokens per forward (TPF). This is the paper's primary metric for parallel decoding efficiency. TPF is computed as the total number of generated tokens divided by the total number of forward passes across all denoising steps. For example, if a sequence of 128 tokens is generated in 50 forward passes, TPF = 128 / 50 = 2.56. Higher TPF means more parallelism and therefore higher throughput. TPF is controlled by the confidence threshold $\tau$ : lower $\tau$ means the model is willing to accept less-confident predictions, leading to more tokens decoded per step (higher TPF) but potentially lower accuracy; higher $\tau$ means the model only decodes tokens it is very confident about, leading to fewer tokens per step (lower TPF, closer to AR-like decoding) but higher accuracy.

The accuracy–throughput trade-off from a single checkpoint. A key advantage of the dLM paradigm is that a single trained model can operate at different points on the accuracy–throughput curve simply by changing $\tau$ . Figure 9 shows this: a single Efficient-DLM 8B checkpoint spans from high-accuracy/low-throughput (rightmost points) to lower-accuracy/high-throughput (leftmost points) across four tasks. This "one-for-all" flexibility contrasts with AR models, which have a fixed accuracy–throughput point determined by their architecture (they always generate one token per forward). To achieve a different trade-off with AR models, one must switch to a different model size entirely (e.g., from Qwen3 8B to Qwen3 4B).

Throughput measurement. Throughput (tokens per second) is measured on an NVIDIA H100 GPU with a batch size of 1, using confidence-based sampling with the block-wise attention pattern and KV caching. For Efficient-DLM 8B at TPF=1.00 (most conservative, similar to AR-like decoding), throughput is 39.99 tok/sec. At TPF=2.57, throughput rises to 103.89 tok/sec. At TPF=3.10, throughput reaches 126.43 tok/sec. The trade-off is visible in Table 3: at TPF=3.10, average accuracy drops from 71.62% (TPF=1.00) to 70.65%, a loss of 0.97 percentage points for a $3.16\times$ throughput improvement.

Batch size effects (Appendix B, Figure 11). The throughput advantage of dLMs over AR models is most pronounced at small batch sizes (1–4), which correspond to memory-bounded inference scenarios where hardware utilization is low for AR models. At larger batch sizes (16–32), the advantage diminishes because AR models can better amortize weight-loading costs across multiple concurrent sequences. At batch size 32, Efficient-DLM 8B actually falls behind Qwen3 1.7B in throughput. The paper acknowledges this limitation: "the efficiency benefits of dLMs over AR models are more pronounced at small batch sizes, which correspond to more memory-bounded scenarios, and these benefits begin to diminish at larger batch sizes." This is flagged as a direction for future work, with potential solutions including adaptive block sizes and combining dLMs with linear attention to improve large-batch efficiency.

Parameter-Efficient Conversion (LoRA)

Appendix E explores whether AR-to-dLM conversion can be achieved with parameter-efficient fine-tuning rather than full-model training, motivated by the relatively small weight changes observed under the optimal attention pattern.

Setup. LoRA [hu2022lora] is applied to all linear layers in attention and FFN modules of Qwen2.5 1.5B, with ranks 16 and 64. All other parameters are frozen except for the embedding layer, normalization operators, and the final model head, which the authors find must remain trainable for effective adaptation. Training uses the best configuration from Section 2.2: block-wise attention with clean context, no token shift.

Results (Table 7, Rows h-i). LoRA with rank 64 achieves 30.78% average accuracy across six tasks, compared to 38.41% for full-model training and 19.29% for the equivalent fully-bidirectional full-model baseline. This means LoRA can recover a substantial fraction of the full-model performance at much lower training cost (only the LoRA adapters and a few layers are trained), but a 7.63% gap remains. Rank 16 underperforms rank 64, achieving 27.93%, suggesting that sufficient adapter capacity is needed. The authors conclude that "full-model training remains necessary to obtain strong dLMs" but note that "even parameter-efficient tuning can yield competitive dLMs" for resource-constrained settings.

Summary of Design Choices and Their Justifications

Block-wise attention with clean context over fully bidirectional: preserves pretrained AR weight distributions (smaller weight drifts, Figure 2e), enables native KV caching, and closes the training–test gap (training conditions each block on clean context, matching inference where previous blocks are already decoded). Justified by +19.12% improvement over bidirectional attention in Table 1 (Row g vs. Row c).
Removal of token shift: predicting the masked token directly rather than the next token after a masked position is an easier learning task that doesn't require the model to simultaneously infer the mask and predict the following token. Justified by consistent improvements across settings (Rows b vs. c and f vs. g in Table 1).
Position-dependent token masking with $\lambda = 0.1$ over uniform masking: bridges the training–test gap where inference-time mask tokens concentrate at block ends, especially important for aggressive parallel decoding. Justified by up to +4.38% improvement at TPF=5.6 (Table 2).
Gumbel-top- $k$ sampling for mask token selection: efficient sampling without replacement from the position-weighted distribution, avoiding the bias that would come from independent Bernoulli sampling at each position (which could produce fewer or more than $k$ masks).
$1/t$ loss weighting: standard diffusion technique that prevents high-noise steps from dominating the training signal and forces the model to learn precise denoising at low noise levels.
Cosine learning rate schedule from 1e-5 to 3e-6 with AdamW: the sweet spot identified in Appendix C (Table 6) that balances preserving pretrained abilities (lower LR avoids large weight drifts) and adapting to new attention patterns (higher LR enables sufficient weight changes). LR=1e-4 causes accuracy collapse; LR=1e-6 fails to adapt.
Block size 16 for 1.5B, 64 for 4B/8B: selected based on the per-model-scale optimum from Figure 3, balancing context richness against corruption. Larger models can handle larger blocks.
Extended training (300B–500B tokens): justified by the training dynamics in Section 4 showing that longer training specifically improves the accuracy–NFE trade-off, enabling more aggressive parallel decoding without proportional accuracy loss.

4. Key Insights and Innovations

Innovation 1: The Attention Pattern Is the Dominant Lever in AR-to-dLM Conversion — Not Just an Implementation Detail, but the Primary Determinant of Whether Pretrained Capabilities Survive

Prior to this work, the field's approach to AR-to-dLM conversion treated attention pattern choice as a secondary implementation decision, subordinate to training objectives and architectural modifications. Dream [ye2025dream] and contemporaneous efforts [gong2025scaling] adopted fully bidirectional attention by default, focusing their innovations on token shift mechanisms and training recipes while inheriting the bidirectional assumption from from-scratch dLM training (LLaDA [nie2025large], MDLM [sahoo2024simple]). The implicit consensus was that bidirectional attention — allowing every token to attend to every other token during denoising — was necessary for dLMs to leverage contextual information from both directions.

This paper fundamentally challenges that consensus. The central empirical finding of Section 2 is that the attention pattern is not a secondary detail but the primary factor determining conversion success. Comparing Row (c) (fully bidirectional without token shift, 19.29% accuracy) to Row (g) (block-wise with clean context, no token shift, 38.41% accuracy) in Table 1 — both trained on the same base model with the same token budget, differing only in attention pattern and a clean-context conditioning variable — reveals a 19.12 percentage point gap. This is an enormous effect size that dwarfs the impact of other design choices the paper studies (token shift contributes ~1%, position-dependent masking contributes up to ~4%, learning rate contributes ~4%). The attention pattern alone accounts for the majority of the accuracy differential between a failed conversion and a successful one.

What makes this finding intellectually distinctive is that it reframes AR-to-dLM conversion from a training objective problem to a weight preservation problem. The paper's diagnostic contribution is the visualization in Figure 2(e), showing that fully bidirectional attention causes substantially larger weight changes in both attention and FFN layers compared to block-wise attention with clean context. This is not merely an observation — it explains why prior approaches like Dream underperformed. The pretrained AR model's internal representations — shaped by trillions of tokens of causal language modeling — are not just being fine-tuned; they are being substantially restructured under fully bidirectional attention. The block-wise pattern, by preserving the causal flow across blocks, allows the model to reuse its pretrained left-context processing capabilities while only adapting to within-block bidirectionality. The conceptual move is from "how should we train a dLM?" to "how can we minimize how much we need to change the pretrained AR model while still enabling diffusion?"

This also reframes the debate about KV caches and bidirectional attention in dLMs. Prior work treated KV cache compatibility as an inference-time engineering concern — nice to have for efficiency, but orthogonal to model quality. The paper shows that attention patterns enabling KV caching (block-wise with causal flow) are also the patterns that best preserve pretrained capabilities. This is not a coincidence: both benefits derive from the same structural property — preserving causal information flow across segments of the sequence. The win–win framing ("both accuracy and efficiency") is genuinely novel because it unifies what were treated as separate concerns.

The finding is fundamental rather than incremental. Prior AR-to-dLM work operated under the assumption that any attention pattern could work if paired with the right training recipe (token shift, careful learning rate schedules, etc.). This paper establishes that the attention pattern itself is a first-order constraint: some patterns are structurally incompatible with preserving pretrained AR capabilities, and no amount of training recipe tuning can fully compensate. This changes how future researchers and practitioners should approach dLM design — attention pattern selection should be the starting point, not an afterthought.

Evidence anchors: Table 1 (Rows b–g), showing the 19.12% gap; Figure 2(e), showing weight change magnitudes across attention patterns; Table 5, showing the +14.42% jump when adding block-wise attention with clean context to Dream's baseline.

Innovation 2: The Training–Test Gap in Masking Distributions Is a Genuine Bottleneck — and Position-Dependent Masking Is a Simple, Theoretically Motivated Fix

The paper identifies a mismatch between training-time and inference-time mask token distributions that prior dLM work had not systematically characterized. The default approach — uniform random masking of positions within a block, independent of token position — implicitly assumes that during inference, mask tokens will be uniformly distributed across positions at every denoising step. The paper demonstrates that this assumption is false.

Section 3.1 provides what amounts to a diagnostic tool for dLM behavior: measuring the average number of denoising steps required at each token position during confidence-based inference. Figure 6(a) reveals a clear left-to-right gradient — earlier tokens in a block require fewer denoising steps because they gain confidence from clean left context and the natural left-to-right structure of language. Later tokens require more steps because they depend on preceding tokens being decoded first. As the denoising process approaches completion, the remaining mask tokens concentrate at the right end of each block.

This diagnostic reveals a distribution shift: the model was trained with uniform masking at all noise levels but encounters highly non-uniform masking during inference, particularly in the final denoising steps that are most consequential for accuracy. This is analogous to exposure bias in sequence models — the mismatch between training-time teacher forcing (conditioning on ground-truth context) and inference-time autoregressive decoding (conditioning on model-generated context). In the dLM setting, the mismatch is between training-time uniform mask placement and inference-time position-concentrated mask distributions.

What distinguishes this from a simple observation is that the paper develops a principled, mathematically simple correction rather than an ad-hoc heuristic. The position-dependent weighting function $w_i(t) = \exp[\beta(1-t)i]$ couples the positional bias to the noise level $t$ in exactly the right way: when $t$ is high (many masks), the distribution is approximately uniform, matching the need for diverse corruption patterns during early denoising training; when $t$ is low (few masks), the distribution tilts toward later positions, matching the inference-time pattern. The coupling through $(1-t)$ ensures the transition is smooth rather than a discontinuous regime switch. The reparameterization using the half-life ratio $\lambda$ makes the hyperparameter interpretable — it directly translates to "over what fraction of the block length does the masking probability double?" — which is rare in diffusion model design spaces.

The significance extends beyond the specific weighting function. The conceptual contribution is identifying mask token distribution as a first-class design dimension in dLMs. Prior work treated the mask sampling procedure as a given — uniformly random, determined only by the total number of masks. This paper establishes that where masks are placed matters independently of how many masks are placed, and that this choice interacts non-trivially with the inference-time decoding strategy. The position-dependent approach is a specific instantiation of a broader principle: training-time corruption patterns should anticipate the inference-time denoising trajectory.

The finding that position-dependent masking provides the largest benefits under aggressive parallel decoding (up to +4.38% at TPF=5.6 vs. +1.75% at TPF=1, Table 2) is theoretically satisfying. It confirms the mechanism: the training–test gap is most severe when the model is asked to decode many tokens per step, because the mask token distribution at later denoising steps is maximally non-uniform. A model that was trained on uniform masking performs worst exactly where dLMs' throughput advantage is greatest. This means position-dependent masking is not just an accuracy improvement — it specifically enables the high-throughput operating regime that makes dLMs practically competitive with AR models.

This is an incremental innovation in mechanism (the weighting function is a straightforward extension of existing masking approaches) but a fundamental contribution in problem identification. The paper names and characterizes a bottleneck that the field had overlooked, provides diagnostic tools to measure it, and demonstrates that addressing it yields meaningful gains. Future work on dLM masking strategies — adaptive schemes, learned masking distributions, masking conditioned on content rather than just position — can build directly on this framing.

Evidence anchors: Figure 6(a–b), demonstrating the left-to-right inference time pattern; Table 2, showing the λ=0.1 vs. uniform comparison across TPF levels; Figure 7, showing the average masking probability curves for different λ values.

Innovation 3: Extended Continuous Training Improves Not Just Accuracy, but Specifically the Accuracy–Throughput Pareto Frontier — Redefining What "dLM Quality" Means

The standard evaluation paradigm for language models measures accuracy at a fixed decoding strategy — for AR models, greedy or temperature-sampled decoding with one token per step; for dLMs, iterative denoising with either one token per step (matching AR) or a fixed parallel decoding setting. Under this paradigm, the only question is "how accurate is the model?" The training dynamics analysis in Section 4 introduces a more nuanced evaluation: how does the accuracy–throughput Pareto frontier shift with additional training?

Figure 8 shows that as continuous pretraining proceeds from 25B to 200B tokens, the entire accuracy–NFE curve shifts upward. A model trained for 200B tokens achieves higher accuracy at a given NFE budget (i.e., at a given throughput level) than a model trained for 100B tokens, which in turn outperforms a model trained for 50B tokens. Critically, the gains are not uniform across the NFE spectrum — longer training disproportionately improves accuracy at low NFE (aggressive parallel decoding). This means additional training tokens are not just improving the model's raw denoising ability; they are specifically improving the quality of confidence estimates that enable reliable parallel decoding.

This finding reframes how the field should think about dLM training budgets. The conventional view would treat training as a means to improve benchmark accuracy, with parallel decoding ability as a fixed property of the architecture. The paper demonstrates that parallel decoding ability is itself a trainable capability that improves with additional compute investment. This has practical implications: if a practitioner cares about throughput (which is the entire point of deploying a dLM), they should allocate training budget not just to maximize accuracy but to push the accuracy–throughput frontier outward. The optimal training budget might be larger than what would be suggested by accuracy-saturation alone.

The conceptual contribution is establishing a multi-dimensional evaluation framework for dLMs where model quality is a curve (accuracy vs. throughput), not a point. Under this framework, two models with identical single-token-per-step accuracy can have meaningfully different quality if one maintains higher accuracy under aggressive parallel decoding. This parallels the way the computer architecture community evaluates processors on performance-per-watt curves rather than peak performance alone. Prior dLM evaluations (including Dream and LLaDA) did not systematically report accuracy–throughput trade-off curves as a function of training budget.

The paper's mechanism hypothesis — that improved likelihood estimation yields more reliable confidence scores, which enables more aggressive parallel decoding — is plausible and consistent with the evidence, but the paper does not prove it causally. The claim that "stronger likelihood estimation produces more accurate and reliable confidence scores" is asserted rather than demonstrated through targeted interventions (e.g., showing that improvements in likelihood calibration metrics correlate with improvements in accuracy–NFE trade-offs). This is a limitation of the analysis but does not diminish the importance of establishing the phenomenon itself.

This innovation is incremental in method (running training for longer is not novel) but fundamental in perspective — it changes what it means for a dLM to be "well-trained." The paper's decision to train Efficient-DLM models for 300B–500B tokens rather than the 25B–50B tokens used in ablation studies is directly motivated by this finding, making it central to the practical recipe.

Evidence anchors: Figure 8(b–d), showing the upward shift of accuracy–NFE curves with training budget; Figure 8(a), showing monotonic likelihood accuracy improvement.

Innovation 4: AR-to-dLM Conversion Succeeds Because the Block-Wise Pattern Reduces Necessary Weight Changes — Turning a Parameter-Efficiency Observation into a Design Principle

The paper makes a subtle but consequential conceptual move: it reframes AR-to-dLM conversion from a capability acquisition problem to a capability preservation problem. The implicit model in prior work (Dream, Gong et al.) was that pretrained AR weights provide a useful initialization, but substantial retraining is needed to teach the model the fundamentally new skill of bidirectional denoising. Under this model, some accuracy loss during conversion is inevitable — the model trades off previously-learned AR capabilities for newly-acquired dLM capabilities.

The weight-change visualizations in Figures 2(e) and 4 challenge this framing. They show that under the optimal attention pattern (block-wise with clean context), the weight changes are surprisingly small — the model does not need to be substantially retrained. The key insight is that the block-wise attention pattern with clean context is structurally similar enough to AR attention that the pretrained model's internal computations can be largely reused. The within-block bidirectionality requires adaptation, but the cross-block causal flow — which constitutes the majority of the attention computation for any given token — is unchanged. The model's pretrained ability to use left context for prediction transfers directly, because the training setup ensures that each block's left context (the clean tokens from previous blocks) is exactly the same as what an AR model would see.

This has deep implications for how to think about dLM architectures. It suggests that the design space for successful dLMs should be constrained by compatibility with pretrained AR representations, not just by what might theoretically enable the best bidirectional denoising. The fully-bidirectional approach, while mathematically elegant (every token can in principle use information from every other token), is architecturally incompatible with AR pretraining in ways that no amount of additional training can fully overcome. The block-wise approach accepts a restricted form of bidirectionality (only within blocks) in exchange for preserving the pretrained model's capabilities.

The LoRA experiments in Appendix E provide converging evidence for this interpretation. The fact that parameter-efficient fine-tuning with rank-64 LoRA can recover a substantial fraction (30.78% vs. 38.41%) of full-model training performance suggests that the required weight changes are indeed low-rank — the model doesn't need to learn entirely new computations, just adapt existing ones. If the conversion required learning fundamentally new capabilities, low-rank adaptation would be insufficient. The ~7.6% gap between LoRA and full-model training indicates that some full-rank changes are needed (likely for the within-block bidirectional attention computations), but the majority of the adaptation is low-rank.

The significance of this innovation is that it provides a design principle rather than just an empirical finding. When designing future dLM architectures or conversion methods, the question to ask is not "what attention pattern maximizes bidirectional context?" but "what attention pattern maximizes compatibility with AR pretraining while providing sufficient bidirectionality?" This principle can guide decisions about block sizes (Section 2.3), evaluation block sizes, and potentially extensions to other model architectures.

This is a fundamental reframing rather than an incremental improvement. It explains why the block-wise approach works rather than just demonstrating that it works, and the explanation generalizes beyond the specific implementation choices in this paper.

Evidence anchors: Figure 2(e), comparing weight change magnitudes across attention patterns; Figure 4, showing the relationship between block size and weight changes; Table 7, showing LoRA vs. full-model training results; the fact that LoRA can achieve competitive (if not fully equal) performance.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper evaluates on 12 tasks spanning four categories: math (GSM8K, Minerva Math), coding (HumanEval, HumanEval Plus, MBPP, MBPP Plus), factual knowledge (MMLU), and commonsense reasoning (ARCC, ARCE, Hellaswag, PIQA, Winogrande). Ablation studies in Sections 2–4 use a subset of six generation tasks (HumanEval, HumanEval Plus, MBPP, MBPP Plus, GSM8K, Minerva Math). Text embedding experiments use 15 datasets from the MTEB benchmark across six categories (retrieval, reranking, clustering, pair classification, classification, semantic textual similarity), following the ablation setup from LLM2Vec. All evaluations use lm-evaluation-harness with shot settings following Dream: 8-shot for GSM8K, 4-shot for Minerva Math, 0-shot for HumanEval/HumanEval+, 3-shot for MBPP/MBPP+, and 0-shot for commonsense reasoning tasks. Maximum generation length is 512 tokens for all tasks except GSM8K (256 tokens).
Base model(s). The paper uses three model families as AR initialization: Qwen2.5 1.5B (ablation studies and Efficient-DLM 1.5B), Qwen3 4B (ablation studies and Efficient-DLM 4B), and Qwen3 8B (Efficient-DLM 8B). The Qwen family is chosen as a representative strong pretrained AR model; the three scales enable studying how conversion behavior changes with model size. The pretrained AR weights serve as initialization for continuous pretraining into dLMs.
Metrics. The primary metrics are: (1) accuracy on each benchmark, computed by comparing model outputs to ground truth using standard evaluation harness procedures; (2) average accuracy across the relevant task set; (3) tokens per forward (TPF), the average number of tokens decoded per denoising step, computed as total generated tokens divided by total forward passes; (4) throughput in tokens per second, measured on an NVIDIA H100 GPU with a batch size of 1 (unless otherwise noted). The accuracy–throughput trade-off is the central evaluative framework: models are compared by their Pareto frontier rather than by accuracy alone. For likelihood tasks, accuracy is computed by estimating and selecting the largest likelihood among multiple-choice answers.
Baselines. The paper compares against three categories of baselines: (1) AR models — Llama3.2 1B, SmolLM2 1.7B, Qwen2.5 0.5B, Qwen2.5 1.5B, Qwen3 1.7B, Qwen3 4B, Qwen3 8B; (2) from-scratch dLMs — LLaDA 8B [nie2025large]; (3) AR-to-dLM conversion methods — Dream 7B [ye2025dream] (fully bidirectional attention, token shift). In ablation studies, the primary baseline is Row (b) of Table 1, which replicates Dream's training scheme (bidirectional attention, token shift, 50B tokens) on Qwen2.5 1.5B. For the position-dependent masking study, the baseline is uniform token masking ( $\lambda \to \infty$ ). Internal ablation baselines are clearly specified per experiment (e.g., "bidirectional attention" in Table 1, "uniform masking" in Table 2).
Generation budget / compute accounting. Training cost is measured in tokens of continuous pretraining data. Ablation studies use 25B or 50B tokens; final Efficient-DLM models use 300B tokens (1.5B, 4B) or 500B tokens (8B). Inference cost is measured in number of function evaluations (NFEs) and tokens per forward (TPF), controlled by varying the confidence threshold $\tau$ in confidence-based sampling. All throughput measurements are on NVIDIA H100 GPUs with batch size 1 unless otherwise specified. For the block-wise with clean context vs. without clean context comparison (Table 1, Rows d–f), the paper accounts for the increased sequence length of the clean-context variant by training the no-clean-context variant with $2\times$ the token budget (Row e). All models in Table 1 use the same 50B token budget for fair comparison except where explicitly noted.
Cross-validation / statistical protocol. The paper does not use cross-validation for strategy selection. Instead, it performs a systematic sweep over design factors (attention pattern, token shift, block size, masking strategy, learning rate) and reports results on standard benchmark test sets. The 12-task evaluation in Section 5.1 uses the standard test splits from lm-evaluation-harness. For block size selection, each trained model is evaluated across multiple evaluation block sizes and the results are reported as heatmaps (Figure 3). No statistical significance testing or confidence intervals are reported; results are presented as point estimates. The test set for the six-task ablation studies consists of the standard test splits of HumanEval (164 problems), HumanEval+ (164), MBPP (500), MBPP+ (500), GSM8K (1,319), and Minerva Math (272), totaling approximately 2,900 test instances. The paper does not discuss test-set contamination or data leakage from the continuous pretraining dataset to the evaluation benchmarks.

Main Quantitative Results

5.1 Attention Pattern Comparison (Table 1)

The core finding is that attention pattern choice dominates all other design factors in determining AR-to-dLM conversion success. Table 1 reports accuracy on six generation tasks for Qwen2.5 1.5B trained with 50B tokens under different attention pattern configurations.

Headline numbers. The original AR Qwen2.5 1.5B achieves 41.79% average accuracy (Row a). Dream's bidirectional scheme with token shift collapses to 18.10% (Row b)—a 23.69 percentage point drop. The best Efficient-DLM configuration (block-wise with clean context, no token shift) achieves 38.41% (Row g), recovering 92% of the original AR accuracy. The gap between the worst bidirectional variant (Row b, 18.10%) and the best block-wise variant (Row g, 38.41%) is 20.31 percentage points.

Bidirectional vs. block-wise. Even without clean context and with token shift removed, block-wise attention (Row d, 28.23%) substantially outperforms bidirectional attention (Row c, 19.29%), a gap of 8.94 percentage points. This demonstrates that the block-wise structure itself—independent of clean context—provides a significant advantage over fully bidirectional attention.

Clean context contribution. Adding clean context to block-wise attention improves average accuracy from 28.23% (Row d, no clean context) to 37.69% (Row f, with clean context, token shift retained), a gain of 9.46 percentage points. Doubling the training tokens for the no-clean-context variant (Row e, 27.73%) does not close this gap, demonstrating that the benefit comes from the presence of clean context rather than from seeing more tokens overall.

Token shift is detrimental. Across both bidirectional (Row b vs. c: 18.10% vs. 19.29%) and block-wise settings (Row f vs. g: 37.69% vs. 38.41%), removing token shift consistently improves accuracy, though the effect is modest (+0.72 to +1.19 percentage points) compared to attention pattern changes. The final Efficient-DLM configuration (Row g) omits token shift.

Task-level detail. The accuracy improvements are not uniform across tasks. On coding tasks (HumanEval, HumanEval+, MBPP, MBPP+), the best configuration (Row g) actually exceeds the original AR model on HumanEval (39.02% vs. 36.59%) and HumanEval+ (34.76% vs. 29.88%), while being slightly behind on MBPP (34.0% vs. 43.6%) and MBPP+ (48.15% vs. 59.52%). On math tasks, Row g reaches 52.99% on GSM8K (vs. AR's 54.74%) and 21.56% on Minerva Math (vs. AR's 26.40%). The dLM conversion preserves coding ability particularly well, while math reasoning shows a larger gap.

Weight change visualization (Figure 2e). The weight change magnitude measured across attention and FFN layers directly correlates with accuracy preservation. Fully bidirectional attention causes the largest weight changes in both layer types. Block-wise without clean context causes intermediate changes. Block-wise with clean context causes the smallest changes, especially in FFN layers. This provides a mechanistic explanation for why the attention pattern matters: smaller weight changes mean less disruption of pretrained representations.

KV caching note. Rows (d–g) all support KV caching, while Rows (b–c) do not. The throughput implications are reported separately in Table 3.

5.2 Block Size Analysis (Figures 3–5)

The block size study examines how the number of tokens per block affects conversion quality and how training block size interacts with evaluation block size.

Training block size sweep (Figure 3). For Qwen2.5 1.5B trained with 50B tokens, the optimal training block size is 16. Training with block size 4 yields visibly lower accuracy (the heatmap shows lighter colors in the block size 4 row). Training with block size 128 also degrades accuracy compared to block size 16. For Qwen3 4B trained with 25B tokens, the optimal training block sizes are 64 and 128, which perform comparably. This demonstrates that larger models can tolerate (and benefit from) larger blocks—a scaling property the paper identifies.

Evaluation block size interaction (Figure 3). A model trained with a given block size can transfer to different evaluation block sizes. In the heatmaps, rows are not restricted to their diagonal—the block size 16 training row (Qwen2.5 1.5B) shows good accuracy across evaluation block sizes 8, 16, 32, and even 64. This generalization occurs because the block-wise attention mask exposes the model to varying numbers of participating tokens during training.

Weight changes vs. block size (Figure 4). Larger training block sizes produce larger weight changes. For Qwen2.5 1.5B, block size 4 causes the smallest weight changes; block size 128 causes the largest. The accuracy sweet spot (block size 16) represents a balance between sufficient adaptation (enough weight change to learn the within-block bidirectional attention) and preservation (not so much change that pretrained capabilities are lost).

Parallel decoding and evaluation block size (Figure 5). At low NFE (aggressive parallel decoding), larger evaluation block sizes produce higher accuracy. For the diffusion Qwen2.5 1.5B model (trained with block size 16) evaluated at NFE ≈ 10, block size 64 achieves visibly higher accuracy than block size 8. At high NFE (near one token per step), all moderate block sizes perform comparably. For the diffusion Qwen3 4B model, the pattern is similar: larger block sizes help most at low NFE, and the benefit diminishes as NFE increases.

Practical selections. Efficient-DLM 1.5B uses training block size 16 and evaluation block size 16. Efficient-DLM 4B and 8B use training block size 64 and evaluation block size 32. These are chosen based on the per-model-scale optima from Figure 3 and the throughput–accuracy trade-off from Figure 5.

5.3 Position-Dependent Token Masking (Table 2, Figures 6–7)

Table 2 compares token masking strategies on Qwen3 4B trained for 25B tokens, evaluated at four parallel decoding settings (TPF = 1, 2.8, 4, 5.6).

Headline numbers. At the most aggressive parallel decoding setting (TPF = 5.6), position-dependent masking with $\lambda = 0.1$ achieves 38.37% average accuracy compared to 33.99% for uniform masking—a +4.38 percentage point improvement. At TPF = 1 (one token per step, least aggressive), the same comparison shows a +1.75 percentage point improvement (62.02% vs. 60.27%). The benefit grows monotonically with decoding aggressiveness.

Lambda sweep. $\lambda = 0.1$ is the best setting, outperforming $\lambda = 0.25$ and $\lambda = 0.05$ at all TPF levels. At TPF = 5.6, $\lambda = 0.1$ (38.37%) outperforms $\lambda = 0.25$ (34.55%) by 3.82 points and $\lambda = 0.05$ (37.38%) by 0.99 points. This demonstrates a sweet spot: too little positional bias ( $\lambda = 0.25$ , near-uniform) doesn't close the training–test gap enough; too much positional bias ( $\lambda = 0.05$ ) over-constrains the model to right-side masking; $\lambda = 0.1$ balances the two.

Right-to-left masking fails. The extreme case of always masking the rightmost $k$ tokens ( $\lambda \to 0$ , "right-to-left") performs poorly across all TPF levels, with accuracy dropping from 60.27% (uniform, TPF=1) to 38.21%—a catastrophic 22.06 point decline. At TPF=5.6, right-to-left masking achieves only 14.13%. The paper attributes this to the model being "forced to train only on the hard cases at the block end without learning to exploit the bidirectional context."

Average masking probabilities (Figure 7). The figure visualizes the empirical masking probability for each position within a block of size 64, averaged over $t \sim \mathcal{U}[0,1]$ . Uniform masking shows a flat line. $\lambda = 0.25$ shows a mild upward slope. $\lambda = 0.1$ shows a more pronounced slope, with later positions masked approximately 1.5× more often than earlier positions. $\lambda = 0.05$ shows a strong slope. Right-to-left masking shows a step function—only the rightmost positions are ever masked.

Inference-time left-to-right tendency (Figure 6a–b). Figure 6(a) visualizes the average number of denoising steps required at each token position in a block on GSM8K, using the trained diffusion Qwen2.5 1.5B. With parallel decoding (confidence threshold applied), the number of steps increases from left to right—earlier tokens decode in fewer steps. Without parallel decoding (one token per step), all positions require one step. Figure 6(b) shows a concrete example: confidence scores within a block across denoising steps, with red boxes marking decoded tokens. Tokens gain confidence as their neighbors are decoded, and the pattern is predominantly left-to-right. These visualizations establish the empirical basis for the position-dependent masking strategy.

Per-position loss (Figure 6c). The average loss at each token position within a block, averaged over 200 samples, shows that later tokens incur higher loss. This indicates that later tokens are genuinely harder to predict (due to more corrupted context) and thus would benefit from more training emphasis—which the position-dependent masking provides by masking them more frequently.

5.4 Training Dynamics (Figure 8)

Figure 8 tracks model performance over 200B tokens of continuous pretraining for Qwen2.5 1.5B.

Likelihood task accuracy (Figure 8a). Accuracy on likelihood-based tasks improves monotonically with training tokens, rising from approximately 32% at 0B tokens (the pretrained AR model evaluated as a dLM without any adaptation) to approximately 38% at 200B tokens. The curve shows no sign of plateauing at 200B tokens.

Generation task accuracy–NFE trade-off (Figure 8b–d). Three generation tasks are shown: HumanEval (b), MBPP (c), and GSM8K (d). For each, the accuracy–NFE curve shifts upward as training progresses from 25B to 50B to 100B to 200B tokens. At a given NFE, the 200B model consistently outperforms the 100B model, which outperforms the 50B model, which outperforms the 25B model. The rightmost point of each curve (TPF=1) also shows accuracy improvement with training, though with task-specific fluctuations.

Disproportionate benefit at low NFE. The upward shift of the curves is not uniform—longer training provides larger relative gains at low NFE (aggressive parallel decoding). For example, on HumanEval at NFE=5, the gap between the 200B and 25B models appears larger than at NFE=20. This indicates that extended training specifically improves the quality of confidence estimates used to guide parallel decoding.

Interpretation. The paper argues that "improved likelihood estimation allows for more aggressive parallel token generation, as reflected in the enhanced accuracy–NFE trade-off with longer training." The monotonic improvement in likelihood task accuracy provides indirect evidence for this mechanism, though the paper does not directly measure confidence calibration metrics.

5.5 Benchmark Against SOTA Models (Table 3, Figure 1, Figure 9)

Table 3 compares Efficient-DLM 1.5B, 4B, and 8B against AR models (Llama3.2, SmolLM2, Qwen2.5, Qwen3) and dLMs (LLaDA 8B, Dream 7B) on 12 tasks.

Efficient-DLM 8B vs. Dream 7B. At TPF=1.00 (most conservative decoding), Efficient-DLM 8B achieves 71.62% average accuracy with 39.99 tok/sec throughput. Dream 7B achieves 65.30% accuracy with 28.11 tok/sec. The accuracy gap is +6.32 percentage points (the paper reports +5.4% in the abstract using a different TPF setting; at TPF=2.57 with 103.89 tok/sec, Efficient-DLM 8B achieves 70.93%, which is +5.63 points over Dream's 65.30% at 28.11 tok/sec). The throughput advantage at comparable accuracy is approximately 3.7× (103.89 vs. 28.11 tok/sec). The paper's abstract number (+5.4% accuracy, 4.5× throughput) is computed from the higher-throughput Efficient-DLM configuration.

Efficient-DLM 8B vs. Qwen3 4B. At TPF=2.57 (103.89 tok/sec), Efficient-DLM 8B achieves 70.93% accuracy. Qwen3 4B achieves 67.97% accuracy at 47.13 tok/sec. This represents +2.96 percentage points higher accuracy with 2.2× higher throughput. The abstract's +2.7% accuracy and 2.7× throughput is a similar comparison at a slightly different operating point.

Efficient-DLM 8B vs. Qwen3 8B. Efficient-DLM 8B at TPF=1.00 achieves 71.62% accuracy, slightly higher than Qwen3 8B's 71.58%. At TPF=2.57, accuracy drops to 70.93% (a 0.69 point decline) but throughput increases from 39.99 to 103.89 tok/sec (2.6×). At TPF=3.10, accuracy further drops to 70.65% (0.97 points below TPF=1.00) with 126.43 tok/sec throughput (3.2× over TPF=1.00, 3.0× over Qwen3 8B's 42.51 tok/sec).

Efficient-DLM 4B vs. Qwen3 1.7B. Efficient-DLM 4B at TPF=2.52 achieves 67.39% accuracy with 119.33 tok/sec throughput. Qwen3 1.7B achieves 59.39% accuracy with 71.59 tok/sec. The accuracy advantage is +8.00 percentage points with 1.67× throughput.

LLaDA 8B comparison. LLaDA 8B achieves only 54.92% accuracy with 25.04 tok/sec throughput, substantially below all Efficient-DLM variants and both Qwen3 4B and 8B. This baseline illustrates the gap between from-scratch dLM training and AR-to-dLM conversion.

Task-level breakdown (Table 3). On coding tasks, Efficient-DLM 8B at TPF=1.00 achieves 67.36% average accuracy vs. Qwen3 8B's 68.45%—a 1.09 point gap. On math, Efficient-DLM 8B achieves 69.22% vs. 69.87%—a 0.65 point gap. On MMLU, Efficient-DLM 8B achieves 77.22% vs. 76.93%—a 0.29 point advantage. On commonsense reasoning, Efficient-DLM 8B achieves 74.88% vs. 73.71%—a 1.17 point advantage. The dLM matches or exceeds the AR model on knowledge and reasoning while trailing slightly on coding and math. Under aggressive parallel decoding (TPF=2.57), the gaps widen modestly: coding drops to 65.64% (2.81 points below Qwen3 8B), math to 68.52% (1.35 points below).

One-for-all flexibility (Figure 9). A single Efficient-DLM 8B checkpoint spans an accuracy–throughput frontier across four tasks (GSM8K, Minerva Math, HumanEval, MBPP). The frontier lies above the individual points of the Qwen3 family (1.7B, 4B, 8B)—each of which provides only a single accuracy–throughput operating point. The dLM frontier shows the characteristic trade-off shape: accuracy declines gradually as throughput increases, with the decline accelerating at the highest throughput levels.

Throughput at larger batch sizes (Figure 11, Appendix B). At batch sizes 1–4, Efficient-DLM 8B maintains throughput advantages over AR models. At batch size 8, the throughput gap narrows. At batch size 16, Efficient-DLM 8B remains competitive. At batch size 32, Efficient-DLM 8B falls behind Qwen3 1.7B in throughput—the smaller AR model, despite lower accuracy, achieves higher tokens-per-second due to better large-batch scaling. The paper explicitly notes this as a limitation: "the efficiency benefits of dLMs over AR models are more pronounced at small batch sizes, which correspond to more memory-bounded scenarios, and these benefits begin to diminish at larger batch sizes."

5.6 Text Embedding Results (Table 4)

Table 4 compares Efficient-DLM 1.5B and 4B against their AR counterparts (Qwen2.5 1.5B, Qwen3 4B) on 15 MTEB datasets across six categories.

Headline numbers. Efficient-DLM 1.5B achieves 37.25% average score compared to Qwen2.5 1.5B's 29.54%—a +7.71 percentage point advantage. Efficient-DLM 4B achieves 40.70% compared to Qwen3 4B's 30.79%—a +9.91 percentage point advantage.

Category-level detail. The dLM advantage is concentrated in pair classification (Efficient-DLM 1.5B: 56.76% vs. Qwen: 24.59%, a 32.17 point gap) and semantic textual similarity (49.14% vs. 39.33%, a 9.81 point gap at 1.5B; 47.27% vs. 40.56%, a 6.71 point gap at 4B). For retrieval, the AR models hold a small advantage at 1.5B (20.69% vs. 18.67%) but Efficient-DLM 4B pulls ahead (20.17% vs. 19.46%). For clustering, the gap is modest (23.58% vs. 21.42% at 1.5B; 23.91% vs. 21.77% at 4B). The pattern is attributed to bidirectional attention enabling richer sequence representations, consistent with the paper's claim that "dLMs are more promising than AR models for tasks requiring high-quality text embeddings."

Methodology note. Both AR and dLM models are evaluated in a zero-shot setting without fine-tuning. For Qwen, causal attention is used (switching to bidirectional degraded performance). For Efficient-DLM, bidirectional attention is used. Embeddings are obtained by mean pooling over the last layer's hidden states.

5.7 Component Ablation Study (Table 5)

Table 5 ablates each component of Efficient-DLM by progressively adding them to the Dream baseline (bidirectional attention, 25B tokens) on Qwen3 4B, reporting accuracy on six math and coding tasks.

Progression. Starting from bidirectional attention (44.59%), adding block-wise attention with clean context yields 59.01% (+14.42 points). Removing token shift raises this to 60.27% (+1.26 points). Adding position-dependent masking raises it to 62.02% (+1.75 points). Scaling training to 300B tokens raises it to 64.05% (+2.03 points). The total improvement from baseline to final configuration is +19.46 percentage points.

Contribution breakdown. Block-wise attention with clean context is the dominant component, accounting for 74% of the total improvement (14.42 / 19.46). Position-dependent masking contributes 9% (1.75 / 19.46). Removing token shift contributes 6% (1.26 / 19.46). Longer training contributes 10% (2.03 / 19.46). The ordering is consistent with the paper's emphasis: attention pattern first, masking strategy second, training budget third.

Per-task variation. The impact of each component varies by task. On HumanEval, the progression is 39.02 → 53.66 → 56.10 → 60.37 → 60.98 (total +21.96 points). On GSM8K, the progression is 67.40 → 78.39 → 82.87 → 81.12 → 86.43 (total +19.03 points). Note that position-dependent masking causes a regression on GSM8K (82.87 → 81.12) and Minerva Math (47.02 → 45.92), but the subsequent training scale-up recovers and exceeds these levels. This suggests the masking strategy may interact with task characteristics in ways the paper does not fully explore.

Ablation Studies and Robustness Checks

Learning rate sensitivity (Table 6, Appendix C). The initial learning rate for continuous pretraining significantly affects conversion quality. Evaluated on Qwen3 4B trained for 25B tokens: 1e-4 yields 50.95% average accuracy; 3e-5 yields 56.59%; 1e-5 yields 60.63% (best); 3e-6 yields 60.11%; 1e-6 yields 54.56%. The U-shaped curve confirms that both too-high LRs (causing excessive weight drift) and too-low LRs (failing to adapt to the new attention pattern) degrade performance. The sweet spot of 1e-5 is used throughout the paper. The magnitude of the effect (up to 10 points between best and worst) confirms learning rate is a material design factor.

LoRA fine-tuning for AR-to-dLM conversion (Table 7, Appendix E). Applying LoRA with rank 64 to all linear layers (plus trainable embeddings, norms, and head) achieves 30.78% average accuracy—competitive with full-model bidirectional training (19.29%) and block-wise without clean context (28.23%), but 7.63 points below full-model training with the best attention pattern (38.41%). LoRA rank 16 achieves 27.93%. The finding that parameter-efficient methods can achieve reasonable performance supports the paper's claim that the required weight changes are modest, but the remaining gap indicates full-model training is needed for optimal quality. The result also demonstrates that the conversion is sensitive to which parameters are trained: "the embedding layer, normalization operators, and the final model head... must remain trainable for effective adaptation."

Token shift removal (Tables 1 and 5). Across both bidirectional and block-wise settings, removing token shift consistently improves accuracy by 0.7–1.3 percentage points. This is a small but reliable effect. The paper's hypothesis is that predicting the masked token directly is an easier task than simultaneously inferring the mask and predicting the next token. This ablation validates the claim that token shift—which prior work (Dream) considered essential—is unnecessary and mildly harmful.

Block size sweep (Figure 3, Section 2.3). The interaction between training and evaluation block sizes is explored across two model scales (1.5B and 4B). Key robustness finding: models trained with a single block size transfer to other evaluation block sizes, avoiding the sequence-length generalization issues that fully bidirectional models face. The optimal training block size increases with model scale (16 for 1.5B, 64 for 4B), indicating a scaling property. Very small blocks (4) underperform at both scales. Very large blocks (128) show degraded accuracy for 1.5B but not for 4B. This ablation justifies the per-model-scale block size selection used in the final Efficient-DLM family.

Clean context necessity (Table 1, Rows d–f). The clean context variant (37.69%) substantially outperforms the corrupted context variant (28.23%) under identical token budgets. Doubling the token budget for corrupted context (27.73%) does not match the clean context result. This demonstrates that clean context is not merely providing more tokens to learn from—it is qualitatively changing what the model learns by aligning training context with inference context. This is a strong result that validates the design choice.

Masking strategy comparison (Table 2). Position-dependent masking with $\lambda = 0.1$ outperforms uniform masking across all TPF levels, with the advantage growing from +1.75 points at TPF=1 to +4.38 points at TPF=5.6. Right-to-left masking catastrophically fails (14.13% at TPF=5.6). The U-shaped performance curve across $\lambda$ values (uniform → 0.25 → 0.1 → 0.05 → right-to-left) confirms a genuine optimum rather than a monotonic relationship.

Training budget scaling (Figure 8, Section 4). Extended training from 25B to 200B tokens continuously improves both likelihood accuracy and the accuracy–NFE trade-off. The curves show no saturation at 200B tokens, motivating the final models' 300B–500B token training budgets. This ablation validates the paper's claim that longer training specifically improves parallel decoding capability.

Benchmark with Fast-dLLM acceleration (Figure 10, Appendix B). To verify that Efficient-DLM's advantage over Dream and LLaDA is not an artifact of decoding method, the paper applies Fast-dLLM (dual cache + parallel decoding) to Dream and LLaDA and plots their accuracy–throughput curves against Efficient-DLM 8B on GSM8K. Efficient-DLM's frontier lies above both baselines across the throughput range. The gap is largest at high throughput, where Efficient-DLM maintains higher accuracy than the accelerated Dream/LLaDA baselines.

Negative result: ReST-style training degrades revision quality. Not present in this paper—no self-improvement loops or RL fine-tuning of dLMs are attempted. The paper focuses exclusively on supervised continuous pretraining.

Critical Assessment

Claim 1: "Block-wise attention with clean context leads to a win–win in accuracy and efficiency"

This claim is supported by consistent evidence across multiple experimental axes. Table 1 demonstrates the accuracy advantage: block-wise with clean context (Row g, 38.41%) vs. bidirectional (Row c, 19.29%) on the same base model and token budget—a 19.12 point gap. The KV caching capability is an architectural property, not a measured outcome, but Table 3 confirms throughput: Efficient-DLM 8B achieves 103.89 tok/sec (TPF=2.57) compared to Dream 7B's 28.11 tok/sec—a 3.7× improvement that depends on KV caching.

Where the claim holds unconditionally: the accuracy advantage of block-wise with clean context over bidirectional attention appears across all six generation tasks in Table 1, across two model scales (1.5B and 4B) in Figure 3, and in the progressive ablation of Table 5. The weight change visualizations in Figures 2e and 4 provide mechanistic evidence consistent with the preservation hypothesis. The LoRA results in Appendix E provide converging evidence that the required adaptation is modest.

Limitations and boundary conditions: the experiments demonstrate the advantage at two specific model scales (1.5B, 4B) from the Qwen family. Whether the finding generalizes to other model architectures (e.g., non-Transformer models, models with different pretraining objectives) is untested. The clean context concatenation increases training sequence length, which the paper controls for (Row e with 2× token budget) but which still changes the computational characteristics of training in ways that might interact with hardware efficiency—a practical concern not addressed. The block-wise pattern constrains bidirectionality to within-block only, which could limit performance on tasks requiring long-range bidirectional reasoning (e.g., document-level understanding where relevant context spans multiple blocks). The paper's text embedding results (Table 4) hint that bidirectionality within blocks is sufficient for representation quality improvements, but this is not tested on tasks specifically requiring cross-block bidirectional dependencies.

Missing experiments: a direct comparison between block-wise dLMs and AR models at matched inference latency (not just throughput) would strengthen the efficiency claim. The paper measures tokens per second but not time-to-first-token or per-request latency. Since dLMs require multiple forward passes per block, the latency for generating a short response could be higher than an AR model even if throughput (tokens per second over many requests) is favorable. The batch size 1 throughput numbers are relevant for interactive applications, but Figure 11 shows the advantage erodes at larger batch sizes—the paper does not characterize at what batch size the advantage fully disappears, only that it diminishes. A throughput break-even batch size analysis would clarify the practical deployment envelope.

Claim 2: "Position-dependent token masking narrows the training–test gap and improves accuracy, especially under aggressive parallel decoding"

This claim is supported by the empirical pattern in Table 2: position-dependent masking with $\lambda = 0.1$ improves accuracy over uniform masking, with the improvement growing from +1.75 points at TPF=1 to +4.38 points at TPF=5.6. The diagnostic visualizations in Figure 6 establish the existence of the training–test gap (left-to-right decoding tendency during inference). The loss-per-position data in Figure 6c provides a supplementary justification (later tokens are harder and benefit from more masking emphasis).

Where the claim holds: the improvement is consistent across the four TPF levels tested, across all six generation tasks (the average in Table 2 aggregates all six), and across multiple λ values (showing a clear optimum). The monotonic relationship between TPF aggressiveness and improvement magnitude matches the mechanistic explanation: more aggressive decoding concentrates masks at block ends more severely, making the training distribution mismatch more consequential.

Limitations: the experiment is conducted on a single model scale (Qwen3 4B) with 25B training tokens. Whether the optimal λ generalizes to other model scales, block sizes, or training budgets is not tested. The λ parameter is manually tuned rather than learned or derived from the data—a learned masking schedule might outperform the simple exponential weighting. The paper evaluates only four discrete TPF levels—a continuous sweep would provide a more complete picture of where the benefit is maximized. The mechanism claim ("narrows the training–test gap") is supported by correlation (the improvement pattern matches the predicted mechanism) but not by direct causal evidence (e.g., measuring the KL divergence between training and inference mask distributions and showing it correlates with accuracy).

Missing experiments: the paper does not evaluate position-dependent masking at TPF values between 1 and 2.8 or above 5.6, leaving the shape of the accuracy–TPF curve partially characterized. The paper does not test whether the masking strategy interacts with block size—intuitively, larger blocks have more room for positional effects, so λ might need to be block-size-dependent. The paper does not evaluate position-dependent masking on models without clean context—it is possible that the benefit would be even larger when the model cannot rely on clean left context. The paper does not ablate the exponential functional form against alternatives (e.g., linear weighting, sigmoid) to determine whether the specific shape matters or just the existence of a left-to-right gradient.

Claim 3: "Extended continuous training improves the accuracy–throughput Pareto frontier"

This claim is supported by Figure 8, which shows the upward shift of accuracy–NFE curves as training progresses from 25B to 200B tokens. The shift is visible across three generation tasks (HumanEval, MBPP, GSM8K).

Where the claim holds: the trend is consistent across all three tasks shown. The monotonic improvement on likelihood tasks (Figure 8a) provides convergent evidence. The final model training budgets (300B–500B tokens) are motivated by this finding, and the performance in Table 3 is consistent with the claim.

Limitations: the paper does not establish why the Pareto frontier improves. The hypothesis—"stronger likelihood estimation produces more accurate and reliable confidence scores"—is plausible but untested. No confidence calibration metrics (expected calibration error, reliability diagrams) are reported that would directly test this mechanism. It is possible that the improvement comes from better raw token prediction accuracy rather than better confidence estimates per se, or from an interaction between the two. The training dynamics are measured at only one model scale (1.5B)—whether the same curves hold at 4B and 8B is not shown. The paper evaluates at four training budgets (25B, 50B, 100B, 200B)—more frequent evaluation points would provide a finer-grained picture of the scaling trajectory and whether saturation is approaching. There is no theoretical model or scaling law fitted to the data that would predict the accuracy–NFE trade-off as a function of training tokens, limiting the ability to determine optimal training budgets.

Missing experiments: the most significant omission is the absence of confidence calibration metrics. The paper's central mechanism hypothesis would be directly testable by measuring expected calibration error (ECE) at different training checkpoints and correlating it with accuracy–NFE trade-off quality. An experiment comparing two models with identical single-token accuracy but different calibration quality, and showing that the better-calibrated model achieves superior accuracy under aggressive parallel decoding, would establish causality. The paper also does not evaluate whether training on data with more diverse sequence lengths improves the Pareto frontier (since sequence length affects how denoising steps distribute across positions). Finally, the paper does not test whether the training dynamics finding depends on the position-dependent masking strategy—it is possible that the Pareto frontier improvement from longer training is partially driven by better handling of the masking distribution, and the effect might be smaller or larger with different λ.

Claim 4: "Efficient-DLM outperforms both AR and dLM baselines in accuracy–throughput trade-offs"

This is demonstrated in Table 3 and Figures 1, 9, and 10. Efficient-DLM 8B at TPF=1.00 achieves 71.62% accuracy (matching Qwen3 8B's 71.58%) while at TPF=2.57 achieves 70.93% with 2.4× the throughput of Qwen3 8B. Compared to Dream 7B, Efficient-DLM 8B shows +5.4% to +6.3% higher accuracy with 3.7–4.5× higher throughput depending on the operating point.

Where the claim holds: the advantage over dLM baselines (Dream, LLaDA) is unambiguous and large. The advantage over comparably-sized AR models (Qwen3 4B vs. Efficient-DLM 4B) is clear. The comparison to Qwen3 8B is more nuanced: Efficient-DLM 8B matches or slightly exceeds its accuracy at TPF=1.00 while offering 2.4–3.2× higher throughput at moderately reduced accuracy. Whether this constitutes a net "win" depends on the user's accuracy–throughput preference, which is precisely the "one-for-all flexibility" the paper emphasizes.

Limitations and boundary conditions: the advantage is measured at batch size 1. At larger batch sizes, the advantage shrinks (Figure 11) and may reverse. The paper does not establish at what batch size the crossover occurs. The comparison is to specific AR models (Qwen3 family) and dLMs (Dream, LLaDA)—whether the advantage holds against other model families (Llama, Mistral, Gemma) is not tested. The evaluation tasks are weighted toward reasoning (math, coding) and knowledge—generation quality on open-ended tasks (summarization, dialogue, creative writing) is not evaluated, which matters because dLMs may exhibit different quality characteristics than AR models for long-form generation. The paper uses the base AR model's accuracy as a reference point but does not report results where the AR model is given an equivalent FLOP budget at inference time (e.g., by using speculative decoding or multiple samples with voting)—comparisons that would be needed to establish that dLMs are strictly more efficient rather than trading accuracy for throughput.

Missing experiments: the most important missing comparison is an AR model with equivalent inference-time FLOPs. For example, if Efficient-DLM 8B uses 3 forward passes per token at TPF=3.0, it consumes approximately the same FLOPs as an AR model generating 3 tokens. Giving the AR model a best-of-3 or majority-voting budget would equalize the FLOP comparison. The paper's comparison (TPF=3.10 for dLM vs. TPF=1.00 for AR) is a throughput comparison, not a FLOPs-matched one—the dLM achieves higher throughput by doing fewer forward passes per token, but whether this is a fundamental advantage or simply a different operating point on a shared accuracy–compute curve is not established. Additionally, the paper does not report latency (time per request for a fixed-length generation), which differs from throughput in the dLM setting because the number of forward passes varies per block. Finally, the paper does not evaluate whether the Efficient-DLM 8B accuracy advantage over Qwen3 8B on MMLU and commonsense reasoning is statistically significant or within the range of benchmark noise (the gap is 0.29 and 1.17 points, respectively—small enough to potentially be noise without confidence intervals).

Cross-cutting assessment

Strengths of the experimental design: the paper's ablation methodology is systematic and hierarchical, starting from a well-defined baseline (Dream's configuration), varying one factor at a time (attention pattern, token shift, masking strategy, training budget), and reporting the cumulative impact in Table 5. The use of weight-change visualization (Figures 2e and 4) provides mechanistic evidence beyond accuracy metrics. The multi-dimensional evaluation (accuracy vs. throughput curves, not just point estimates) is a methodological improvement over prior dLM work. The progressive component ablation (Table 5) clearly quantifies each factor's contribution and validates that the factors are complementary (each adds to the previous).

Weaknesses: the paper evaluates on a single model family (Qwen), limiting claims of generality. The test sets are relatively small for some tasks (HumanEval: 164 problems; HumanEval+: 164)—random variation could account for the smaller accuracy differences. No confidence intervals or significance tests are reported. The training data for continuous pretraining is a mixed dataset whose composition is not disclosed in detail, making it difficult to assess whether the results are contaminated by benchmark data leakage. The throughput measurements are at a single hardware configuration (H100, batch size 1) and the scaling to other hardware or batch sizes is only partially characterized (Figure 11 goes up to batch size 32). The most fundamental limitation is the absence of a FLOPs-matched comparison between dLMs and AR models—the paper compares throughput (tokens per second of wall time) but not total computational work, which would be the appropriate metric for establishing whether dLMs are more than just a latency-vs-throughput trade-off.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Remains Unaccounted for in the Efficiency Gains

The assumption or constraint. The entire compute-optimal framework rests on estimating each prompt's difficulty before deciding how to allocate the inference budget. In this paper, difficulty estimation requires generating 2,048 samples per question and scoring them with the PRM (Section 3.2). The authors acknowledge this cost explicitly but do not include it in their headline efficiency calculations:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

The consequence. The reported 4× efficiency gains over best-of-N are computed after difficulty is known, without amortizing the cost of learning it. Generating 2,048 samples per question costs more compute than the largest test-time budgets studied (256–512 generations). In a realistic deployment, the total cost would be difficulty estimation plus strategy execution, and the former could dominate the latter — potentially erasing the 4× advantage entirely. For use cases with few repeated inferences per prompt (e.g., interactive chat), the upfront cost is prohibitive. Even for batch processing, the difficulty estimation cost must be amortized over a large number of similar queries to be economical.

What evidence exists in the paper. The paper provides indirect evidence that the cost matters: the "predicted" difficulty bins (using PRM scores rather than ground-truth labels) perform similarly to oracle bins (Figures 4 and 8), but both methods still require the 2,048-sample generation step. The paper does not measure total compute including difficulty estimation, does not report how difficulty accuracy degrades with fewer than 2,048 samples, and does not evaluate whether difficulty estimation could be done with a smaller sample budget (e.g., 16 or 32 samples) while preserving the 4× efficiency claim.

Mitigation status. The authors flag this as a key direction for future work (Section 3.2, Section 8), suggesting "pretraining or finetuning models to directly predict difficulty of a question" and noting the exploration–exploitation tradeoff between compute spent assessing difficulty versus solving the problem. No mitigation is implemented or evaluated in the current paper. The 4× figure should be understood as an upper bound conditional on already knowing the difficulty, not a realized deployment gain.

Hard Problems Remain Fundamentally Outside the Reach of Test-Time Compute

The assumption or constraint. The entire compute-optimal framework assumes that the base model's pass@1 is non-zero — that there exist correct solutions in the proposal distribution to find or refine. The paper is explicit about this:

"On the hardest questions (bin 5), no method makes meaningful progress — the base model simply lacks the capability to produce correct solutions regardless of how budget is allocated." (Section 5.3)

The consequence. Across all methods — search, revisions, and their compute-optimal combinations — difficulty bin 5 shows near-zero improvement regardless of how much inference compute is applied. In Figure 3 (right), bin 5 accuracy hovers at 1–3% for all methods and all budgets. In Figure 7 (right), bin 5 shows roughly 2–3% accuracy irrespective of the sequential-to-parallel ratio. In the FLOPs-matched comparison (Figure 9, Section 7), the bin 5 scaling line is effectively flat near 0–5% for both revisions and PRM search, and the ≈14× larger pretrained model consistently outperforms test-time compute on these problems.

This means the approach offers no path forward for genuinely novel or out-of-distribution reasoning that exceeds the base model's training distribution. If the model cannot produce a correct answer even once in 2,048 attempts, no allocation of test-time compute will help. For such problems, additional pretraining — scaling model capacity and training data — remains the only viable path. This is not a limitation the paper can solve with better methods; it is a fundamental ceiling on what test-time compute can achieve.

What evidence exists in the paper. The evidence is extensive and consistent: Figure 3 (right), Figure 7 (right), Figure 9, and the negative FLOPs-matched comparison results on hard problems in Figure 1 (bottom-right bar chart, showing −37.2% to −52.9% disadvantage for test-time compute vs. the larger model on hard problems at high $R$ ). The authors are transparent about this boundary condition in Section 7:

"test-time compute amplifies existing capability but does not create it from nothing."

Mitigation status. None — this is a fundamental property of the approach, not a fixable bug. The paper frames it correctly as a boundary condition (Section 7, Section 8) rather than claiming universality. For practitioners, the actionable implication is that deployment should route hard problems to larger models or human review rather than spending inference compute on problems the base model cannot solve.

The ~14× Larger Model Baseline Is Not Compute-Optimally Trained, Inflating the Apparent Advantage of Test-Time Compute

The assumption or constraint. The FLOPs-matched comparison in Section 7 scales model parameters while holding training data fixed, following the LLaMA paradigm (Touvron et al., 2023). The authors acknowledge this departure from compute-optimal pretraining:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work." (Section 7)

The consequence. A Chinchilla-optimal model trained with ~14× more total FLOPs — scaling both parameters and data equally — would likely outperform a parameter-only-scaled model. This makes the pretraining baseline weaker than it could be. The reported advantages of test-time compute over pretraining (e.g., +27.8% on medium questions at $R \ll 1$ for revisions) may shrink or reverse against a properly compute-optimal larger model. Additionally, the larger model uses only greedy decoding with no test-time compute augmentation of its own — no majority voting, no best-of-N, no search. This creates an asymmetric comparison: the smaller model gets sophisticated inference-time optimization while the larger model operates at its weakest inference configuration. A fairer baseline would give the larger model at least some test-time compute budget (e.g., best-of-8 or majority voting), which would narrow or potentially reverse the reported advantages.

What evidence exists in the paper. The FLOPs-matched results are in Figure 9 and the Figure 1 bar charts. The paper does not provide results with a Chinchilla-optimal pretraining baseline, nor does it give the larger model any test-time compute budget. The sensitivity of the conclusions to these design choices is not explored.

Mitigation status. The authors acknowledge this limitation in Section 7 and scope it as future work. The caveat is noted but not resolved. Practitioners should interpret the FLOPs-matched results as evidence for the potential of test-time compute substitution, not as precise guidance on when to prefer it over pretraining in a compute-optimal regime.

The Revision Model Has a 38% Correct-to-Incorrect Reversion Rate with No Principled Solution

The assumption or constraint. The revision model is trained on sequences where all in-context answers are incorrect, followed by a correct target (Section 6.1). This is a pragmatic training data construction choice — correct-to-correct revisions (where the model should recognize the answer is already correct and leave it unchanged) never appear in the training data. The authors report:

"approximately 38% of correct answers get converted back to incorrect ones using a naive approach" (Section 6.1)

The consequence. During a sequential revision chain at inference time, the model will occasionally produce a correct answer at step $k$ and then "revise" it to an incorrect answer at step $k+1$ . This means the revision chain is not monotonically improving — accuracy oscillates — and the system cannot simply take the final revision output as the answer. The paper mitigates this with majority voting or verifier-based selection across the chain, but these are post-hoc patches: they require evaluating every revision in the chain and selecting the best one, which wastes compute on revisions that degrade the answer. A more principled solution — such as training the model to output a "no revision needed" token when the current answer is already correct — is not explored. The 38% reversion rate also means that longer revision chains eventually reach a steady state where improvements and degradations approximately balance, capping the benefit of additional sequential compute.

What evidence exists in the paper. The 38% figure is reported in Section 6.1 without a dedicated table — it appears in the prose as a motivation for the within-chain selection mechanism. Figure 6 (left) shows the revision model's pass@1 gradually improving through the chain but not saturating at a high accuracy, which is consistent with the oscillation effect. The ReST $^{EM}$ experiment (Appendix K, Figure 16) shows that attempting to optimize the revision model with RL-style training made performance substantially worse with sequential revisions, demonstrating the fragility of the revision training recipe.

Mitigation status. Partially mitigated. The within-chain selection mechanism (majority voting or verifier-based selection) prevents the reversion problem from destroying end-to-end accuracy, but it does not solve the underlying issue. The paper does not propose or evaluate methods to reduce the reversion rate directly (e.g., training on correct-to-correct examples, adding a "keep" action, or using the PRM to detect when a revision is degrading the answer). This is a practical pain point for anyone deploying sequential revisions in production.

Search and Revisions Are Studied Independently, Not Combined Despite Complementary Strengths

The assumption or constraint. The paper studies two complementary mechanisms — PRM-guided tree search (Section 5) and iterative revisions (Section 6) — but never combines them. Section 8 explicitly acknowledges this gap:

"we did not experiment with PRM tree-search techniques in combination with revisions"

The consequence. The paper's results represent a lower bound on what a fully integrated system could achieve. The two mechanisms have complementary strengths: revisions improve the proposal distribution (generating better candidates by conditioning on previous attempts), while PRM search improves candidate selection (finding the best among generated candidates). Applying beam search to revision model outputs — or using the PRM's per-step scores to decide when a revision is on track versus when to restart from scratch — could yield gains beyond either method alone. Since the paper establishes that revisions work best on easy problems (Figure 7, right) and PRM search works best on medium problems (Figure 3, right), a combined system could potentially route problems to the best combination of both mechanisms rather than choosing between them. The current independent analysis leaves this synergy unexplored.

What evidence exists in the paper. The independent results for search (Section 5) and revisions (Section 6) demonstrate the complementary difficulty-dependent strengths. Figure 3 (right) shows search helping primarily on medium problems. Figure 7 (right) shows revisions helping primarily on easy problems with a balanced sequential-parallel ratio optimal for harder problems. The paper's compute-optimal policy selects between search algorithms or between sequential-parallel ratios, but never combines search with revisions as interacting components. No experiment tests whether beam search with a revision model as the proposal distribution outperforms either method alone.

Mitigation status. Acknowledged as future work in Section 8. The paper's framework and analysis provide the intellectual scaffolding for such combination, but a practitioner wanting to deploy a combined system today must determine the interaction between search and revisions through their own experimentation. The absence of combination experiments also means the 4× efficiency gains are measured separately per axis — it is unknown whether combining them would yield multiplicative gains, additive gains, or diminishing returns.

All Results Are on a Single Benchmark (MATH) with a Single Model Family (PaLM 2-S*), Limiting Generality

The assumption or constraint. All experiments in Sections 5–7 use the MATH benchmark (500 test questions, high-school competition math) with PaLM 2-S* as the base model. The authors state they "believe this model is representative of the capabilities of many contemporary LLMs" (Section 4), but this is asserted rather than demonstrated.

The consequence. Several aspects of the findings could be model-specific or benchmark-specific. The PRM's quality and over-optimization behavior depend on PaLM 2-S*'s output distribution — a model with different calibration properties or different error patterns might exhibit different difficulty-dependent scaling curves. The revision model's ability to learn from edit-distance-paired incorrect examples depends on the base model's in-context learning capabilities, which vary across model families. MATH consists exclusively of symbolic reasoning problems with exact-match ground truth — it is unclear whether the difficulty-dependent patterns (beam search hurting easy problems, revisions helping easy problems) generalize to code generation, logical reasoning, scientific QA, or tasks requiring factual recall rather than multi-step inference. The test set of 500 questions, split into five quintiles of ~100 each and then further split by two-fold cross-validation, means compute-optimal strategies are selected based on ~50 questions per fold per bin — a small sample that introduces variance in the selected policies.

What evidence exists in the paper. All figures and tables in Sections 5–7 are based on MATH with PaLM 2-S*. The paper does not report results on any secondary benchmark (e.g., GSM8K, HumanEval, MMLU) for the main scaling analysis—the MATH benchmark is used for everything from PRM training to difficulty estimation to strategy selection to final evaluation. No ablation tests whether the optimal strategies selected on MATH generalize to other math benchmarks, let alone other domains.

Mitigation status. The authors acknowledge the single-benchmark limitation in Section 4 but do not mitigate it with additional experiments. The argument that MATH is "representative" is speculative. A practitioner wanting to apply compute-optimal test-time scaling to non-math tasks must replicate the entire difficulty estimation, strategy selection, and evaluation pipeline on their domain of interest. The paper provides the methodology but not the evidence that the methodology transfers.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper fundamentally reframes the problem of building diffusion language models from "how do we train dLMs from scratch to match AR accuracy?" to "how do we minimally adapt pretrained AR models to unlock their parallel decoding potential?" The magnitude of this shift is substantial — it is not an incremental refinement but a change in the default strategy that the field should use when attempting to build practical dLMs.

The reframing rests on a single empirical observation with far-reaching consequences: the attention pattern used during AR-to-dLM conversion is the dominant factor determining whether pretrained capabilities survive, accounting for ~74% of the total accuracy recovery in the paper's ablation study (Table 5). Prior work (Dream, Gong et al.) implicitly treated attention pattern as a secondary implementation choice, defaulting to the fully bidirectional pattern inherited from from-scratch dLM training (LLaDA, MDLM). This paper shows that fully bidirectional attention is structurally incompatible with AR pretraining — it causes large weight changes (Figure 2e) that destroy pretrained reasoning capabilities, and no amount of training recipe tuning (token shift, learning rate schedules, masking strategies) can fully compensate. The conceptual move is from "train a dLM" to "preserve as much of the pretrained AR model as possible while enabling within-block bidirectionality."

This shift has immediate consequences for what research directions become attractive and which become less so. More attractive: work on attention patterns that maximize compatibility with AR pretraining while providing sufficient bidirectionality for parallel decoding — the block-wise design in this paper is one instantiation, but the principle generalizes to other structural constraints (e.g., block sizes that adapt to sequence content, patterns that mix causal and bidirectional attention at different granularities). Work on token masking distributions that anticipate inference-time decoding trajectories becomes newly important, since the paper demonstrates that the training–test gap in mask placement is a real bottleneck, particularly for the high-throughput operating regime that makes dLMs practically competitive. Work on parameter-efficient AR-to-dLM conversion (Appendix E) is validated as a viable direction — LoRA recovers a substantial fraction of full-model performance — opening the door to converting models that are too large to fully fine-tune.

Less attractive: work that assumes fully bidirectional attention is necessary for dLM quality and focuses primarily on training objectives or decoding strategies without questioning the attention pattern. The paper's evidence (Tables 1 and 5) demonstrates that training recipe improvements within a fully bidirectional paradigm yield only modest gains compared to the leap achieved by switching to block-wise attention with clean context. From-scratch dLM training at scale (LLaDA-style) also becomes less compelling as a primary research direction, since AR-to-dLM conversion with the block-wise pattern achieves substantially higher accuracy at lower training cost — Efficient-DLM 8B (converted from Qwen3 8B with 500B tokens of continuous pretraining) achieves 71.62% accuracy vs. LLaDA 8B's 54.92% (trained from scratch). The paper does not prove that from-scratch training cannot eventually match or exceed conversion quality given enough compute, but it establishes that conversion is the more practical path given current resource constraints.

The paper also reconciles a latent contradiction in prior dLM research: why Dream's AR-to-dLM conversion showed promise (it recovered some pretrained capabilities) but ultimately underperformed AR models at comparable sizes (Dream 7B: 65.30% vs. Qwen3 8B: 71.58%). The answer, according to this paper's analysis, is that Dream's fully bidirectional attention pattern caused unnecessary weight drift that limited how much of the pretrained model could be preserved. The block-wise pattern with clean context — which preserves AR weight distributions and closes the training–test attention gap — resolves this contradiction by showing that conversion can match or exceed AR accuracy when the attention pattern is compatible with the pretrained structure. This is not a small correction; it explains a gap of 6.3+ percentage points between prior AR-to-dLM work and this paper's results.

A diagnostic contribution that may prove influential is the weight-change visualization as a proxy for conversion quality (Figures 2e, 4). By measuring the Frobenius norm difference between pretrained and post-conversion weights in attention and FFN layers, the paper provides a lightweight metric for evaluating attention pattern compatibility without requiring full downstream evaluation. This diagnostic could guide future architectural searches: proposed attention patterns can be screened by measuring their induced weight changes on a small adaptation budget (e.g., 1B tokens), with smaller changes predicting better capability preservation. The paper does not formalize this as a metric or validate its predictive power across model families, but the concept is clearly present in the analysis.

Follow-Up Research This Work Enables

Measuring confidence calibration to test the parallel-decoding mechanism directly. The paper's central hypothesis for why extended training improves the accuracy–throughput trade-off (Section 4) is that "stronger likelihood estimation produces more accurate and reliable confidence scores." This hypothesis is untested. A direct follow-up would measure expected calibration error (ECE) at different training checkpoints (25B, 50B, 100B, 200B, 300B tokens) for Efficient-DLM models and correlate ECE with the accuracy–NFE trade-off quality. The prediction is that improved calibration (lower ECE) enables more aggressive parallel decoding — tokens with genuinely high accuracy get high confidence scores and are decoded early, while uncertain tokens remain masked for further refinement. An experiment comparing two models with matched single-token accuracy (TPF=1) but different ECE, and showing that the better-calibrated model achieves superior accuracy at TPF=3.0, would establish causality. This is newly tractable because Efficient-DLM provides dLMs with strong enough base accuracy that calibration differences are measurable — prior dLMs (Dream, LLaDA) may have had accuracy too low for calibration to be the binding constraint.

Learned masking distributions that adapt to content, not just position. The position-dependent masking strategy in Section 3 uses a fixed exponential weighting that depends only on noise level $t$ and relative position $i$ . A natural extension is a learned masking policy — a small auxiliary model (or the dLM itself, in a self-supervised loop) that predicts, for each token position given the current sequence context, how likely that position is to be masked at the current denoising step during inference. This could be trained by collecting statistics from actual inference runs (tracking which positions remain masked at each step across many examples) and using these as targets. The paper's diagnostic visualizations (Figure 6a–b) provide the data format: for each denoising step, record which positions are still masked. A learned policy could capture interactions between content and masking (e.g., tokens that are part of a formula might remain masked longer than tokens in boilerplate text) that the simple position-dependent scheme misses. The paper makes this tractable by establishing that masking distribution matters and providing the inference statistics to train against.

Evaluating block-wise dLMs on tasks requiring long-range bidirectional reasoning. The block-wise pattern restricts bidirectionality to within-block only — tokens in block $b$ cannot attend to tokens in block $b+1$ during denoising. This raises the question: are there tasks where cross-block bidirectional attention is necessary, and does the block-wise pattern's constraint cause measurable accuracy degradation on those tasks? Candidate tasks include document-level question answering (where relevant information spans paragraph boundaries that fall in different blocks), multi-step reasoning where the model generates a plan that must be revised based on later steps, and code generation where a function definition in block 3 might inform what should be generated in block 2. The experiment would compare Efficient-DLM against an idealized dLM with fully bidirectional attention across the entire sequence (accepting the throughput penalty) on such tasks, measuring whether the accuracy gap grows relative to standard benchmarks. The paper's text embedding results (Table 4) hint that within-block bidirectionality is sufficient for representation quality, but generation tasks requiring bidirectional dependencies are not tested.

FLOPs-matched comparison between dLMs and AR models with equivalent inference compute budgets. The paper compares dLM throughput (tokens per wall-clock second) against AR throughput, but does not match total inference FLOPs. A dLM generating at TPF=3.0 uses approximately 3× fewer forward passes than an AR model generating the same sequence, but the comparison would be fairer if the AR model received an equivalent FLOPs budget — e.g., best-of-3 sampling with majority voting, or speculative decoding with a draft model. The experiment would compare Efficient-DLM 8B at TPF=3.0 (126.43 tok/sec, 70.65% accuracy) against Qwen3 8B with speculative decoding or best-of-N, measuring accuracy at matched total FLOPs per generated token. If the dLM still achieves higher accuracy at matched FLOPs, the throughput advantage is a genuine algorithmic improvement rather than a different operating point on a shared compute–accuracy curve. If the AR model matches or exceeds dLM accuracy at matched FLOPs, the dLM advantage is primarily a latency–throughput trade-off, not a compute-efficiency one. The paper enables this comparison by providing detailed throughput and accuracy numbers at multiple TPF levels (Table 3).

Investigating whether the block-size scaling property generalizes across model families. The paper observes that larger models (Qwen3 4B) tolerate larger training block sizes (64–128) than smaller models (Qwen2.5 1.5B, optimal at 16). Is this a general scaling law? The experiment would replicate the block-size sweep (Figure 3) on a different model family — Llama-3 1B/3B/8B, or Mistral variants — and test whether the optimal block size as a function of model parameters follows a predictable relationship. If it does, this becomes a design principle: for a model of size $N$ parameters, use block size $f(N)$ . If the relationship is model-family-specific or non-monotonic, the finding is architecture-dependent. The paper makes this tractable by providing the experimental protocol (train with multiple block sizes, evaluate across evaluation block sizes, report as heatmap) and the approximate budgets needed (25–50B tokens per configuration for small models).

Negative result: stress-testing AR-to-dLM conversion on models with fundamentally different pretraining objectives. The paper converts standard causal language models (Qwen2.5, Qwen3) trained with next-token prediction. Would the conversion succeed on models pretrained with different objectives — e.g., models with fill-in-the-middle training (CodeLlama), models with instruction tuning, or models with retrieval-augmented pretraining? The prediction from the paper's framework is that any pretraining objective that produces representations compatible with the causal-flow-preserving block-wise attention should convert successfully, while objectives that rely on unusual attention patterns or non-causal dependencies might fail. Training a fill-in-the-middle model with block-wise attention and clean context and measuring accuracy recovery would test this boundary. A negative result (conversion fails on instruction-tuned models) would narrow the applicability of the paper's recipe; a positive result would expand it.

Practical Applications and Downstream Use Cases

Memory-bounded inference for interactive applications. The most directly actionable use case is deploying Efficient-DLM for applications where batch size is small (1–4) due to interactive latency requirements — chatbots, code assistants, real-time translation. In this regime, AR models are memory-bounded: the cost of loading model weights dominates, and the sequential token generation cannot amortize this cost. Efficient-DLM 8B at TPF=2.57 achieves 103.89 tok/sec on an H100 with batch size 1 (Table 3), compared to Qwen3 8B's 42.51 tok/sec — a 2.4× improvement in tokens delivered per second. For a code assistant generating 200-token responses, this reduces wall-clock time from ~4.7 seconds to ~1.9 seconds per request, crossing a usability threshold for interactive applications. The throughput advantage is largest at batch size 1 and diminishes at larger batch sizes (Figure 11), making this use case the sweet spot for the current technology.

Text embedding and retrieval systems. Table 4 demonstrates that Efficient-DLM substantially outperforms comparably-sized AR models on text embedding tasks: +7.71% at 1.5B scale and +9.91% at 4B scale across 15 MTEB datasets. The largest gains are in pair classification and semantic textual similarity — tasks that directly benefit from bidirectional context. A retrieval system that uses Efficient-DLM as the embedding model could achieve higher retrieval quality than an AR-based embedder of the same parameter count, without any fine-tuning. This is a drop-in improvement: the embedding extraction process (mean pooling over last-layer hidden states with bidirectional attention) is a single forward pass per document, so the dLM's iterative denoising is not involved — the advantage comes purely from the bidirectional attention enabled by the conversion. Organizations currently using AR models for embeddings (e.g., Qwen-based retrieval pipelines) could convert them to Efficient-DLM variants using the paper's recipe and immediately improve retrieval metrics.

One-for-all model serving with dynamic accuracy–throughput adjustment. A single Efficient-DLM checkpoint can operate at multiple points on the accuracy–throughput Pareto frontier simply by changing the confidence threshold (Figure 9). This enables a deployment pattern where the same model serves both high-accuracy/low-throughput and lower-accuracy/high-throughput requests depending on the use case or current load. For example, a cloud API serving a dLM could offer a "quality" tier (TPF=1.0, highest accuracy) and a "speed" tier (TPF=3.0, 3.2× higher throughput at ~1% lower accuracy), with the same model weights loaded in GPU memory. This is structurally different from AR models, where switching accuracy–throughput trade-offs requires switching model sizes entirely (loading different weights into GPU memory, with associated latency and memory costs). The one-for-all flexibility matters for cost-efficient serving: rather than provisioning separate GPU instances for high-accuracy and high-throughput model variants, a single instance can handle both traffic patterns by adjusting the confidence threshold per request.

Small-model deployment on edge devices. Efficient-DLM 1.5B at TPF=2.69 achieves 184.48 tok/sec (Table 3) with accuracy (51.77%) that is competitive with AR models in the 1–2B parameter range (Qwen3 1.7B: 59.39% at 71.59 tok/sec; Qwen2.5 1.5B: 54.47% at 73.03 tok/sec). For edge deployment scenarios where throughput is limited by memory bandwidth and batch size is inherently 1, the 2.5× throughput advantage over the AR Qwen2.5 1.5B (from which it was converted) could be the difference between usable and unusable generation speed. The conversion cost (50B tokens of continuous pretraining, ~200B additional tokens for the final model) is modest compared to training a new model from scratch, making this a practical path for organizations that already have a deployed small AR model and want to improve its inference throughput without changing hardware.

When to Prefer This Method

The paper positions AR-to-dLM conversion with block-wise attention as a direct alternative to both AR decoding and prior dLM approaches (fully bidirectional from-scratch training or fully bidirectional AR-to-dLM conversion). The decision rules below are based on conditions the paper explicitly evaluates:

Prefer AR-to-dLM conversion with block-wise attention over from-scratch dLM training (LLaDA-style) when: you have access to a pretrained AR model of the target size, you need strong accuracy on reasoning and knowledge tasks (the conversion preserves pretrained capabilities, while from-scratch dLM training at comparable scale substantially underperforms — 71.62% vs. 54.92% average accuracy at 8B), and your training budget is on the order of hundreds of billions of tokens rather than trillions (300B–500B tokens for conversion vs. the trillions needed for from-scratch dLM training to match AR accuracy).
Prefer AR-to-dLM conversion with block-wise attention over fully bidirectional AR-to-dLM conversion (Dream-style) when: throughput matters (block-wise attention enables native KV caching, yielding 4.5× higher throughput at batch size 1 compared to Dream 7B) and accuracy preservation is important (the block-wise pattern with clean context causes substantially smaller weight changes than fully bidirectional training, recovering 92% of the original AR model's accuracy vs. Dream's 44% recovery on the same base model at the same token budget).
Prefer AR-to-dLM conversion over keeping the original AR model when: inference throughput at small batch sizes (1–4) is the primary constraint, the deployment scenario is memory-bounded (interactive applications with low concurrency), and the accuracy loss under parallel decoding is acceptable (Efficient-DLM 8B at TPF=2.57 achieves 70.93% vs. Qwen3 8B's 71.58% — a 0.65 point drop for 2.4× throughput). The conversion becomes less attractive at large batch sizes where AR models achieve better amortization (Figure 11 shows Efficient-DLM 8B falling behind Qwen3 1.7B in throughput at batch size 32) and when the application requires the absolute highest accuracy regardless of throughput (the TPF=1.0 setting still provides some throughput advantage through KV caching but the gap is narrower — 39.99 vs. 42.51 tok/sec at 8B).

Dependency tree

Papers this one builds on, walking backwards through the same domain and shared topics.

Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding2025· inference-optimization← LLM-judged similar
- SageAttention2++: A More Efficient Implementation of SageAttention22025· inference-optimization← in-domain predecessor
- ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference2024· serving← topic thread
- DMax: Aggressive Parallel Decoding for dLLMs2026· inference-optimization← similar topic
- Stochastic KV Routing: Enabling Adaptive Depth-Wise Cache Sharing2026· inference-optimization← similar topic
Directional Textual Inversion for Personalized Text-to-Image Generation2025· training-methods← in-domain predecessor
- Zero-Shot Text-to-Image Generation2021· architecture← LLM-judged similar
- QwenLong-L1.5: Post-Training Recipe for Long-Context Reasoning and Memory Management2025· training-methods← in-domain predecessor
Beyond Single Tokens: Distilling Discrete Diffusion Models via Discrete MMD2026· training-methods← similar topic
- A Survey on Diffusion Language Models2025· llm-systems← LLM-judged similar
- Nemotron-Cascade 2: Post-Training LLMs with Cascade RL and Multi-Domain On-Policy Distillation2026· training-methods← in-domain predecessor
Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion2025· training-methods← similar topic
- Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning2025· training-methods← in-domain predecessor
- TreePO: Bridging the Gap of Policy Optimization and Efficacy and Inference Efficiency with Heuristic Tree-based Modeling2025· training-methods← similar topic

What it influenced

Later papers in the same chain. Direction reversed.

Nemotron-Math: Efficient Long-Context Distillation of Mathematical Reasoning from Multi-Mode Supervision2025· training-methods→ in-domain predecessor
- Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Reward2025· training-methods→ in-domain predecessor
- Hybrid Linear Attention Done Right: Efficient Distillation and Effective Architectures for Extremely Long Contexts2026· architecture→ topic thread
- The Lessons of Developing Process Reward Models in Mathematical Reasoning2025· training-methods→ similar topic
- Black-Box On-Policy Distillation of Large Language Models2025· training-methods→ similar topic
- QwenLong-L1.5: Post-Training Recipe for Long-Context Reasoning and Memory Management2025· training-methods→ similar topic
Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion2025· training-methods→ similar topic
- Self-Forcing++: Towards Minute-Scale High-Quality Video Generation2025· diffusion→ LLM-judged similar
- BlockVid: Block Diffusion for High-Quality and Consistent Minute-Long Video Generation2025· diffusion→ LLM-judged similar
- End-to-End Training for Autoregressive Video Diffusion via Self-Resampling2025· diffusion→ LLM-judged similar
- Knot Forcing: Taming Autoregressive Video Diffusion Models for Real-time Infinite Interactive Portrait Animation2025· diffusion→ LLM-judged similar
- DreaMontage: Arbitrary Frame-Guided One-Shot Video Generation2025· vision→ LLM-judged similar
- LIVE: Long-horizon Interactive Video World Modeling2026· uncategorized→ LLM-judged similar
- Mode Seeking meets Mean Seeking for Fast Long Video Generation2026· vision→ LLM-judged similar
- Helios: Real Real-Time Long Video Generation Model2026· vision→ LLM-judged similar
- HiAR: Efficient Autoregressive Long Video Generation via Hierarchical Denoising2026· vision→ LLM-judged similar
- SparseLoRA: Accelerating LLM Fine-Tuning with Contextual Sparsity2025· training-methods→ in-domain predecessor
- TreePO: Bridging the Gap of Policy Optimization and Efficacy and Inference Efficiency with Heuristic Tree-based Modeling2025· training-methods→ similar topic
Beyond Single Tokens: Distilling Discrete Diffusion Models via Discrete MMD2026· training-methods→ similar topic
- RoboAlign: Learning Test-Time Reasoning for Language-Action Alignment in Vision-Language-Action Models2026· training-methods→ in-domain predecessor
- Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning2025· training-methods→ similar topic