LLaDA2.0: Scaling Up Diffusion Language Models to 100B

🎯 Pitch

LLaDA2.0 shows you can convert a pretrained 100B-parameter autoregressive model into a diffusion model that matches its performance, while running over 2× faster at inference. The key is a staged training recipe that progressively teaches the model to denoise entire sequences, bypassing the catastrophic costs of training a frontier-scale diffusion model from scratch.

1. Executive Summary

This paper introduces LLaDA2.0, a family of discrete diffusion language models scaling to 100B total parameters through systematic conversion from pre-trained autoregressive (AR) models rather than training from scratch. The core mechanism is a Warmup–Stable–Decay (WSD) continual pre-training strategy that progressively transforms AR models into block diffusion language models — progressively increasing block size from 1 to full-sequence diffusion (warmup), large-scale training under the full-sequence masked diffusion objective (stable), and reverting to compact block sizes for inference efficiency (decay) — combined with a document-level attention mask to prevent cross-document interference and complementary masking during supervised fine-tuning to accelerate convergence. The resulting instruction-tuned models, LLaDA2.0-mini (16B) and LLaDA2.0-flash (100B), achieve competitive performance with similarly sized AR models (73.18 vs. 73.60 average score against Qwen3-30B-A3B-Instruct-2507 for the flash variant), with the 100B model demonstrating a 2.1× inference speed advantage (535 tokens/second vs. 256 tokens/second) over AR baselines on code and math benchmarks when augmented with confidence-aware parallel training. The results establish that diffusion language models can match or exceed AR counterparts in complex structured domains like code generation and agentic tool use, but only when initialized from AR checkpoints through a carefully staged conversion process that preserves pretrained knowledge while progressively introducing bidirectional denoising capabilities.

2. Context and Motivation

The Core Problem: Diffusion Language Models Cannot Scale to Frontier Sizes

The fundamental problem this paper tackles is deceptively simple: discrete diffusion language models (dLLMs) exist, but none operate at the 100B+ parameter scale at which modern autoregressive models achieve practical deployment. This is not a minor gap — it is a categorical difference between experimental research models and frontier-scale systems. The paper explicitly frames this as the primary frontier for the field:

"Bridging this scale difference to the hundreds of billions of parameters seen in the leading mainstream AR models is a primary frontier for enabling diffusion models to fully capture complex linguistic patterns for practical deployment." (Section 1)

The magnitude of this gap matters. Prior to LLaDA2.0, all existing diffusion language models — including LLaDA (8B), LLaDA-MoE, Dream-7B, DiffusionLLaMA, and even block diffusion models like BDLMs — capped out at roughly 8B parameters when trained from scratch, or at most 30B when initialized from AR checkpoints (as with SDAR on Qwen-3). This is a ~3–12× scale deficit compared to the 100B+ MoE architectures that dominate production deployment (DeepSeek-V3, Qwen3-30B-A3B, Llama 4). The paper's own assessment is blunt:

"one key limitation across all existing methods is their restricted model scale—ranging only from 7B to 30B parameters—leaving the feasibility and scalability of AR-initialized diffusion models largely unexplored at larger scales." (Section 2.2)

This scale gap is not merely an academic concern about chasing parameter counts. It reflects a structural limitation in the current approach to diffusion language models: training from scratch at scale is prohibitively expensive, and the naïve conversion methods that work at 7B scale may break down at 100B due to optimization instability, catastrophic forgetting, or inefficiencies in the training objective.

Why Scale Matters Practically and Theoretically

The paper's motivation operates on multiple levels.

Practical deployment viability. Diffusion language models offer a compelling advantage over autoregressive models: parallel decoding during inference. In AR generation, each token must be produced sequentially, creating an inherent latency bottleneck that cannot be parallelized. Diffusion models, by contrast, can generate multiple tokens simultaneously through iterative denoising — in principle, they can decode an entire sequence in a fixed number of refinement steps regardless of sequence length. This is not a minor efficiency tweak; it represents a fundamentally different computational profile at inference time, one that could make large language model deployment viable in latency-constrained settings (interactive applications, real-time systems) where AR models struggle.

However, this advantage is meaningless if diffusion models cannot reach the capability frontier established by AR models. A fast but inaccurate model has no deployment value. The practical motivation for scaling dLLMs is therefore to demonstrate that the parallel decoding advantage can be realized without sacrificing output quality — that the diffusion architecture is not merely faster but genuinely competitive on capability. This paper's inference speed results (535 tokens/second at ~73% average benchmark score) represent the first evidence that this tradeoff can be resolved favorably at production scale.

The evidence gap in scaling laws. AR models benefit from well-established scaling laws (Chinchilla, Hoffmann et al., 2022) that allow practitioners to predict performance as a function of model size, data volume, and compute. The dLLM community has no comparable body of evidence. As the paper notes (Section 2.1):

"established training practices and hyperparameter recipes from the AR domain are often suboptimal for MDLMs. To address this gap, recent efforts such as Quakka and OpenMoE2 have begun investigating the scaling properties and optimal training strategies specifically tailored for MDLMs, laying the groundwork for principled scaling in this emerging paradigm."

LLaDA2.0 provides a critical data point in this empirical scaling landscape: it demonstrates that a 100B dLLM can be built, trained stably, and deployed competitively. This is not a scaling law in the formal sense, but it is the necessary first demonstration that scaling is even feasible — a prerequisite for anyone to invest in deriving scaling laws for diffusion architectures.

The architectural frontier. Beyond practical deployment and scaling evidence, there is a deeper theoretical motivation: understanding whether the diffusion generation paradigm has inherent capability ceilings that differ from the AR paradigm. AR models are constrained by their causal attention pattern — they cannot look ahead during generation, which some hypothesize makes them suboptimal for tasks requiring bidirectional reasoning, holistic understanding, or constraint satisfaction across the output. Diffusion models, which see the full sequence context during denoising, might excel in these domains even at equal parameter counts. The paper's finding that LLaDA2.0-flash shows "clear advantages in complex generative tasks" — specifically coding (HumanEval 94.51 vs. 93.29 for Qwen3-30B-A3B), agent capabilities (BFCL v3 75.43 vs. 73.19), and advanced mathematics (AIME 2025 60.00 vs. 61.88) — provides some of the first large-scale evidence that the architectural hypothesis might hold. The authors frame this cautiously:

"These may have opened a new door to future work in the agentic LLM era while solidifying a gaugeable potential of dLLM for test-time scaling." (Section 8)

Prior Approaches and Where They Fall Short

The paper identifies three categories of prior work, each with specific scaling limitations.

1. Training dLLMs from scratch

Several works have demonstrated that masked diffusion language models can be trained entirely from random initialization and achieve respectable performance. LLaDA (Nie et al., 2025) showed an 8B dense MDLM competitive with similarly sized AR models. LLaDA-MoE (Zhu et al., 2025) extended this to Mixture-of-Experts architectures. Quakka (Ni et al., 2025) and OpenMoE2 (Ni & team, 2025) began investigating MDLM-specific scaling properties.

The failure point is straightforward: training from scratch is too expensive to reach frontier scale. The paper states this explicitly:

"from-scratch trained MDLMs still lag behind state-of-the-art AR models in overall performance. This gap can be largely attributed to the disparity in training data volume and the maturity of infrastructure support—factors that have been extensively optimized over years of development for AR models." (Section 2.1)

The economic reality is that AR models benefit from years of accumulated optimization: pretraining recipes, hyperparameter schedules, data curation pipelines, and infrastructure (Megatron-LM, FSDP, etc.) have been refined through thousands of training runs at massive scale. Replicating this investment for diffusion models from scratch would require comparable resources — billions of dollars of compute — with no guarantee of comparable results. The paper's observation that scratch-trained MDLMs "are typically limited in model scale (≤8B)" is not a coincidence; it reflects the practical ceiling of what research groups can afford to train from scratch without the accumulated infrastructure optimizations of the AR ecosystem.

2. Initializing dLLMs from AR checkpoints

Recognizing the prohibitive cost of training from scratch, several recent works have explored converting pre-trained AR models into diffusion language models. This is the approach LLaDA2.0 inherits and extends.

DiffusionLLaMA (Gong et al., 2025) and Dream-7B (Ye et al., 2025) both employ mask annealing strategies — gradually transitioning from causal to bidirectional attention during training — combined with CART (Continuous-Aware Reweighted Training) loss reweighting to balance token-level learning dynamics. CART adjusts the per-token loss based on how frequently that token position is masked, addressing the imbalance where some positions are masked more often than others under the noise schedule.

RND1 (Keshigeyan et al., 2025) takes a more aggressive approach: immediately converting the AR model's causal attention to bidirectional upon initialization, with the key insight that knowledge-intensive capabilities are preserved by constraining updates to the model's dense layers to prevent catastrophic forgetting.

SDAR (Cheng et al., 2025) leverages the Qwen-3 series to train block diffusion language models with improved efficiency, exploring various block sizes and optimization strategies.

The critical failure of these approaches is not performance at their tested scale — they work well at 7B–30B parameters — but the absence of evidence that they generalize to larger models. The paper explicitly identifies this gap:

"whether such initialization strategies can effectively generalize to models beyond the 30B scale remains an open question." (Section 2.2)

This uncertainty is well-founded. Several factors could cause conversion methods to break at larger scales: (1) catastrophic forgetting becomes harder to manage as the model encodes more knowledge in its weights; (2) optimization instability (gradient explosion, loss spikes) becomes more likely due to the larger parameter space and the distributional mismatch between AR and diffusion objectives; (3) training efficiency becomes a bottleneck — block diffusion models, in particular, suffer from low data utilization because the block-wise objective only trains on a fraction of tokens at each step.

The paper's own diagnosis of the efficiency problem is precise:

"Although the BDLM formulation partially reduces this gap through blockwise masked reconstruction, it suffers from low data utilization, limiting the effective exploitation of large-scale corpora." (Section 1)

In a block diffusion model with block size $L_B$ and total sequence length $L$ , only $L_B / L$ tokens are masked and learned from at each training step. For small $L_B$ (e.g., 32), this means the model processes only ~1% of tokens per forward pass. For a 100B model training on trillions of tokens, this inefficiency is catastrophic — it multiplies the already-enormous training cost by orders of magnitude.

3. Post-training for dLLMs

The paper also surveys post-training efforts for diffusion models, which are still nascent:

SFT for domain adaptation: Dream-Coder (Xie et al., 2025) fine-tunes a 7B dLLM for code generation, demonstrating "sketch-then-fill" strategies for complex algorithms. Dream-7B (Ye et al., 2025) achieves general-purpose performance competitive with top AR models. Seed-Diffusion (Song et al., 2025) uses two-stage curriculum learning for high-speed code generation.
Reinforcement learning: Standard policy gradient methods are intractable for dLLMs because exact log-likelihoods cannot be computed — the generation process involves multiple denoising steps with masked token sampling, making the likelihood of a particular output sequence a sum over all possible denoising trajectories. SPG (Wang et al., 2025a) proposes a Sandwich Policy Gradient that maximizes an evidence lower bound for good samples and minimizes an upper bound for bad ones. TraceRL (Wang et al., 2025d) aligns the training objective with the multi-step generation trajectory, producing the TraDo series — the first dLLM capable of long chain-of-thought reasoning.
Inference acceleration: DPad (Chen et al., 2025a) offers training-free acceleration using dynamic scratchpads and pruning. D2F (Wang et al., 2025c) introduces a hybrid AR-diffusion paradigm enabling KV-cache-based acceleration that, for the first time, surpasses equivalently sized AR models in inference speed.

The failure point across all these efforts is lack of systematic integration and scaling. The paper notes:

"the field of dLLM post-training is still nascent. Systematic exploration of how these techniques—SFT, RL, and acceleration—interact with one another, and how they scale to models with hundreds of billions of parameters, remains an open and critical area for future research." (Section 2.3)

Individual techniques (complementary masking, confidence-aware training, DPO adaptation) have been demonstrated at small scale, but no prior work has assembled them into a complete post-training pipeline for a 100B model, nor demonstrated that they remain effective when applied together at that scale.

How This Paper Positions Itself

LLaDA2.0 positions itself not as a fundamentally new architecture or a novel training objective, but as a systematic recipe for scaling diffusion language models through principled conversion from AR checkpoints. The contribution is an engineering and methodological one: assembling and extending existing ideas (block diffusion, document-level attention masking, complementary masking, confidence-aware training, DPO for diffusion) into a coherent, documented pipeline that demonstrably works at 100B scale.

The paper's positioning involves several strategic choices:

Knowledge inheritance over training from scratch. The paper explicitly rejects the scratch-training approach, instead leveraging "existing AR checkpoints as the foundation for a systematic conversion process that preserves linguistic knowledge while introducing diffusion capabilities" (Section 1). This is a pragmatic choice — it acknowledges the economic reality that frontier-scale AR models represent billions of dollars of invested compute and that discarding that investment to train diffusion models from scratch is unrealistic for the foreseeable future. The paper frames this as a design principle: "knowledge inheritance, progressive adaptation, and efficiency-aware design" (Section 3).

Progressive adaptation over abrupt conversion. The WSD strategy addresses the central tension in AR-to-diffusion conversion: the model's internal representations are optimized for left-to-right causal generation, and abruptly switching the objective and attention pattern disrupts these representations, causing optimization instability and catastrophic forgetting. Prior work (RND1's immediate conversion, DiffusionLLaMA's mask annealing) attempted to manage this disruption, but the WSD approach is more structured: it explicitly decomposes the conversion into three phases with different objectives, each building on the previous one. The warmup phase (progressive block size increase from 1 to 4096) is particularly important — by starting from block size 1 (which is equivalent to AR generation) and gradually increasing, the model adapts its internal geometry incrementally rather than experiencing a sudden distributional shock.

Full-sequence diffusion as a training accelerator, not just an inference mechanism. A key insight in WSD is that the stable phase — training under full-sequence MDLM (block size equal to full sequence length) — is not primarily about inference (since inference uses small block sizes for efficiency) but about training efficiency. The paper explains:

"Once the block size reaches 4096 and the model transitions to the MDLM pattern, the 'clean' part of the attention computation no longer needs to be maintained. This significantly reduces the computational cost of attention, allowing data to be processed far more efficiently under the MDLM paradigm." (Section 4.1)

In block diffusion, attention must compute both block-diagonal attention within the noisy sequence and block-causal attention across the clean sequence (see Equation 3). Under MDLM, the entire sequence is one noisy block, so the clean-attention component disappears entirely. This reduces the attention computation from roughly $O(2L^2)$ to $O(L^2)$ , nearly halving the attention cost per training step while simultaneously training on all tokens (since all positions are masked at some rate). This is the solution to the low data utilization problem that plagues block diffusion training — under MDLM, the model learns from every token position at every step, dramatically improving training throughput.

Efficiency-aware design as a first-class requirement. The paper treats efficiency not as an optimization detail but as a design constraint that shapes the entire pipeline. The decay phase (reverting from full-sequence MDLM to small block sizes) exists specifically to recover inference efficiency — KV-cache reuse and variable-length generation capabilities that are impossible under full-sequence diffusion. The confidence-aware parallel training addresses the observation that diffusion models often produce diffuse, uncertain predictions that slow down decoding because tokens fail to meet the confidence threshold for early acceptance. The choice of a threshold decoder with hybrid acceptance (confident tokens accepted immediately, low-confidence tokens filled by top-k selection) is motivated by the practical need for steady generation progress without stalling.

The paper's ambition, stated plainly, is to provide "a practical recipe for the community to leverage AR stability while achieving diffusion parallelism, opening new possibilities for efficient large-scale language modeling" (Section 1). The word "recipe" is instructive — this is not a paper claiming a single breakthrough innovation, but one documenting a working system, including the infrastructure choices (Megatron-LM backend, cuDNN attention for flexible masks, zig-zag partitioning for load balancing), stability fixes (adding Gaussian noise to masked token embeddings to prevent gradient explosion), and post-training tricks (complementary masking, mask ratio bandwidth) that together make 100B-scale diffusion language models practical for the first time.

3. Technical Approach

3.1 Reader Orientation

The LLaDA2.0 system is a pipeline for converting a pre-trained autoregressive language model into a diffusion language model that generates text by iteratively denoising masked sequences rather than predicting tokens left-to-right. It solves the problem of scale: prior diffusion language models topped out at ~30B parameters because training from scratch is prohibitively expensive and naïve AR-to-diffusion conversion methods break down at larger scales due to optimization instability, catastrophic forgetting, and training inefficiency. The solution is a carefully staged conversion recipe — Warmup–Stable–Decay continual pre-training, followed by specialized post-training — that preserves the AR model's knowledge while progressively introducing bidirectional denoising capabilities, ultimately producing a 100B-parameter diffusion model that matches AR performance and exceeds AR inference speed.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major stages, organized as a sequential pipeline that transforms an AR model into a deployable diffusion language model:

AR Base Model (Ling-mini-2.0 / Ling-flash-2.0) — the starting checkpoint: a fully trained autoregressive Mixture-of-Experts language model at 16B or 100B total parameters. It serves as the "knowledge foundation" whose linguistic capabilities will be preserved and repurposed.
Continual Pre-Training (CPT) via Warmup–Stable–Decay — the core conversion stage. It takes the AR model and progressively trains it with a block diffusion objective, where the block size (the number of tokens denoised jointly) is first increased from 1 to the full sequence length (4096), then trained extensively at full-sequence scale, then decreased back to a small efficient size (32). This stage uses a specialized document-level attention mask to prevent cross-document interference in packed training sequences and a top-k checkpoint merge to produce the final base diffusion model.
Block Diffusion Supervised Fine-Tuning (SFT) — instruction-tuning the converted base model on a curated dataset of prompts and responses, using the block diffusion objective conditioned on the prompt. This stage employs complementary masking (training on both a random mask and its logical complement to guarantee every token is seen unmasked) and a mask ratio bandwidth (restricting the noise level to informative ranges) to accelerate convergence and stabilize training.
Confidence-Aware Parallel (CAP) Training (optional) — an additional fine-tuning stage that adds an auxiliary loss to sharpen the model's predictive confidence, enabling more aggressive parallel decoding at inference time. Tokens are accepted based on a confidence threshold; sharper predictions mean more tokens pass the threshold per denoising step, increasing throughput.
Direct Preference Optimization (DPO) — aligning the model with human preferences by maximizing the margin between the model's ELBO-based log-probability estimates for preferred and dispreferred responses, using a frozen reference model regularizer.

The output is an instruction-tuned diffusion language model (LLaDA2.0-mini or LLaDA2.0-flash) that generates responses through iterative block-wise denoising, conditioned on both the user prompt and previously generated blocks.

3.3 Roadmap for the Deep Dive

First, the high-level training paradigm (CPT → Block Diffusion SFT → DPO) — the design philosophy of knowledge inheritance and progressive adaptation, establishing why the pipeline is staged this way rather than attempting direct conversion.
Second, the Warmup–Stable–Decay (WSD) continual pre-training strategy — the centerpiece mechanism. This includes the block diffusion training objective (Equation 1), the three phases and their block-size schedules, the full-sequence MDLM objective (Equation 2), and the rationale for why progressive block-size expansion and contraction is necessary for stability, efficiency, and final inference performance.
Third, the document-level attention mask — the mechanism that enables training on packed heterogeneous documents without semantic contamination. This includes the formal mask definition (Equation 3) and the simplified MDLM variant (Equation 4), plus the connection to the block diffusion vectorized forward pass.
Fourth, the top-k checkpoint merge and numerical stability fix — the post-CPT model averaging strategy and the Gaussian noise injection trick that prevents gradient explosion when converting from AR to diffusion.
Fifth, the supervised fine-tuning stage — the block-diffusion conditional objective (Equation 5), the complementary masking strategy, the mask ratio bandwidth, and the data curation philosophy.
Sixth, the confidence-aware parallel training — the auxiliary entropy loss (implicit in Equation 6) and its role in making parallel decoding practical.
Seventh, the DPO adaptation for diffusion models — the block diffusion ELBO (Equation 7), the DPO loss (Equation 8), and the rationale for using ELBO surrogates rather than exact log-likelihoods.
Eighth, the inference mechanism — the block-wise threshold decoder with hybrid acceptance, connecting the training design choices to their deployment consequences.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and methodology paper whose core contribution is a documented, working recipe for converting AR language models into diffusion language models at 100B scale, along with the infrastructure adaptations required to make training and inference practical. The technical approach is best understood as a sequence of design decisions, each addressing a specific failure mode that prevents simpler conversion methods from scaling.

The Training Paradigm: Knowledge Inheritance and Progressive Adaptation

The paper frames its approach through three explicit design principles (Section 3): knowledge inheritance (leveraging pre-trained AR weights rather than training from scratch), progressive adaptation (gradually introducing diffusion capabilities rather than abrupt objective switching), and efficiency-aware design (ensuring the resulting model is practically deployable, not just accurate).

The overall pipeline follows a three-stage progression shown in Figure 2:

Stage 1: Continual Pre-Training from AR to MDLM. The AR base model — which can be viewed as a special case of a Block Diffusion Language Model with block size $L_B = 1$ (each token is its own "block," and generation is strictly left-to-right) — is progressively trained to denoise larger and larger blocks of masked text until it operates as a full-sequence Masked Diffusion Language Model (MDLM) where the entire sequence is one block. This stage uses the Warmup–Stable–Decay schedule and the document-level attention mask.

Stage 2: Block Diffusion Pre-Training. The full-sequence MDLM is then converted back to a block diffusion model with a small block size (e.g., 32) for efficient inference. This is the "Decay" phase of WSD, but the paper treats it as a distinct pre-training step because it establishes the final architecture for all downstream use.

Stage 3: Post-Training for Alignment. The block diffusion base model undergoes supervised fine-tuning (SFT) on instruction–response pairs with complementary masking, optional confidence-aware parallel training to sharpen predictions, and Direct Preference Optimization (DPO) to align with human preferences.

The key philosophical choice is the separation of concerns: the CPT stage handles the fundamental architectural conversion and knowledge preservation, while the post-training stages handle task alignment and deployment optimization. This means each stage can use different training objectives, data mixtures, and hyperparameters optimized for its specific goal, rather than trying to solve everything simultaneously in a single training run.

Warmup–Stable–Decay (WSD) Continual Pre-Training

The WSD strategy is the central mechanism for converting an AR model into a diffusion language model while preserving pretrained knowledge. It decomposes the conversion into three coordinated phases, each with a specific block-size schedule and training objective.

The Block Diffusion Training Objective

Before explaining the phases, it is essential to understand the objective function that governs training during the warmup and decay phases (when block size $L_B$ is less than the total sequence length). The model operates under the Block Diffusion Language Model (BDLM) paradigm: the input sequence of length $L_{\text{total}}$ is divided into $K = L_{\text{total}} / L_B$ contiguous blocks. Blocks are processed in order; the model sees clean (unmasked) text for all preceding blocks and a noisy (partially masked) version of the current block, and must predict the original tokens for the masked positions in the current block.

The training loss, given in Equation 1 as:

$\mathcal{L}_{\text{BDLM}}(\theta) = -\mathbb{E}_{t, \mathbf{x}_0, \mathbf{x}_t}\left[\frac{\alpha'_t}{1 - \alpha_t} \sum_{k=1}^{K} \sum_{i=1}^{L_B} \mathbb{1}[x^i_{t,k} = \texttt{[MASK]}] \log p_\theta(x^i_{0,k} \mid \mathbf{x}_{0,<k}, \mathbf{x}_{t,k})\right]$

where:

$t \sim \text{Uniform}(0, 1)$ is the diffusion timestep, controlling how much noise is added;
$\mathbf{x}_0$ is the clean (original) token sequence;
$\mathbf{x}_t$ is the corrupted sequence, where each token is independently replaced with a [MASK] token with probability $1 - \alpha_t$ ;
$\alpha_t$ is the noise schedule function — it starts at 1 (no noise) and decreases to 0 (fully masked) over the diffusion process; its specific form follows standard discrete diffusion conventions (a cosine or linear schedule, though the paper does not specify the exact functional form);
$\alpha'_t$ is the derivative of $\alpha_t$ with respect to $t$ , which appears as part of the time-weighting term derived from the diffusion ELBO;
$K = L_{\text{total}} / L_B$ is the number of blocks;
$L_B$ is the block size (number of tokens per block);
$x^i_{t,k}$ is the $i$ -th token in the $k$ -th block of the corrupted sequence;
$\mathbb{1}[x^i_{t,k} = \texttt{[MASK]}]$ is an indicator that is 1 only for masked positions (predictions are only made and penalized where the original token was replaced with [MASK]);
$p_\theta(x^i_{0,k} \mid \mathbf{x}_{0,<k}, \mathbf{x}_{t,k})$ is the model's predicted probability distribution over the vocabulary for the original token at position $i$ in block $k$ , conditioned on the clean text of all preceding blocks ( $\mathbf{x}_{0,<k}$ ) and the noisy version of the current block ( $\mathbf{x}_{t,k}$ ).

What it computes: For a single training example, the procedure is: (1) sample a timestep $t$ ; (2) independently mask each token in the input sequence with probability $1 - \alpha_t$ to produce $\mathbf{x}_t$ ; (3) feed the pair $(\mathbf{x}_0, \mathbf{x}_t)$ through the model with the block diffusion attention mask (described below); (4) for every [MASK] token in the current block, compute the cross-entropy between the model's predicted token distribution and the ground-truth token; (5) weight each token's loss by the time-dependent factor $-\alpha'_t / (1 - \alpha_t)$ ; (6) average over all blocks and all masked positions.

The output is a single scalar loss value per training example. Over many examples and timesteps, minimizing this loss teaches the model to denoise text at all noise levels, from nearly clean (small $t$ , few masks) to nearly fully masked (large $t$ , many masks).

Why this form: The objective is derived from the evidence lower bound (ELBO) for discrete diffusion models. The time-weighting term $-\alpha'_t / (1 - \alpha_t)$ arises from the change-of-variables in the ELBO derivation — it ensures that the loss is a proper variational bound on the negative log-likelihood of the data, meaning that minimizing it also minimizes the model's perplexity. The block-wise conditioning structure $\mathbf{x}_{0,<k}$ is what makes this a block diffusion model rather than a full-sequence diffusion model: the model only sees clean text for blocks it has already "generated" (in inference order), while the current block is denoised jointly. This enables KV-cache reuse during inference (since the clean blocks' key-value pairs don't change) and variable-length generation (since blocks can be generated sequentially). The indicator function $\mathbb{1}[x^i_{t,k} = \texttt{[MASK]}]$ ensures the loss is only computed on tokens that were actually corrupted — predicting already-clean tokens would be trivial and would dilute the learning signal.

Phase 1: Progressive Block Size Warmup

The warmup phase addresses the fundamental challenge: the AR model's internal representations are shaped by a causal, left-to-right attention pattern, but diffusion denoising requires bidirectional attention within each block. An abrupt switch would cause a severe distributional mismatch between the model's learned representations and the new objective, leading to optimization instability and catastrophic forgetting of pretrained knowledge.

The solution is to start from where the AR model already is and gradually expand the model's receptive field. The AR model can be viewed as a BDLM with block size $L_B = 1$ : each token is its own block, and the "diffusion" process is trivial — the model always sees all preceding tokens (clean blocks) and predicts the next token (a "block" of size 1). The warmup phase progressively increases $L_B$ through the schedule: $1 \rightarrow 4 \rightarrow 32 \rightarrow 64 \rightarrow 4096$ .

At each step, the model is trained on "moderate-scale data" (the paper does not specify exact token counts, but the term implies enough data for the model to adapt its attention patterns and output distributions to the new block size without overfitting). The requirement that "the sequence length [be] divisible by the current block size" (Section 4.1) ensures that no block is fragmented — every block has exactly $L_B$ tokens, which simplifies the attention mask construction and ensures uniform denoising difficulty across blocks.

When $L_B$ reaches 4096 (the full sequence length), the BDLM becomes equivalent to a standard Masked Diffusion Language Model (MDLM): there is exactly one block ( $K = 1$ ), and the model denoises the entire sequence jointly with global bidirectional attention. The "clean" context $\mathbf{x}_{0,<k}$ is empty (there are no preceding blocks), so the conditioning reduces to just $\mathbf{x}_{t,1}$ , the noisy version of the full sequence.

Why progressive warmup works: By increasing block size in small increments, the model adapts its attention patterns incrementally. At $L_B = 4$ , the model learns to jointly denoise small 4-token windows while still relying on left context from preceding blocks — a small departure from the AR pattern. At $L_B = 32$ , the joint denoising window expands to roughly sentence-length, requiring the model to develop bidirectional representations at the phrase level. At $L_B = 64$ , the window expands further. By the time $L_B = 4096$ is reached, the model has been gradually conditioned to handle large bidirectional contexts, and the final transition is less jarring than a direct $1 \rightarrow 4096$ jump would be.

Phase 2: Large-Scale Stable Training

Once the model operates at $L_B = 4096$ (full-sequence MDLM), the training objective simplifies significantly. The "clean" attention component — the block-causal attention over $\mathbf{x}_0$ in Equation 3 — disappears because there are no clean preceding blocks to attend to. The entire sequence is one noisy block. This has two critical consequences for training efficiency:

Reduced attention computation. Under block diffusion, the attention mask includes three components: self-attention within the noisy block, cross-attention from noisy to clean blocks, and causal self-attention within clean blocks. Under MDLM, only the self-attention within the single noisy block remains. The paper states this "significantly reduces the computational cost of attention" (Section 4.1), effectively nearly halving the attention FLOPs per training step.
Full data utilization. Under block diffusion with small $L_B$ , only $L_B$ tokens per sequence are masked and learned from at each step (since the loss is only computed on the current noisy block). Under MDLM, all $L$ tokens in the sequence are masked (at rate $1 - \alpha_t$ ) and all contribute to the loss. This means every training step provides a learning signal from every token position, dramatically improving data efficiency. This directly addresses the "low data utilization" limitation of block diffusion that the paper identifies as a barrier to scaling.

The MDLM objective (Equation 2) is:

$\mathcal{L}_{\text{MDLM}}(\theta) = -\mathbb{E}_{t, \mathbf{x}_0, \mathbf{x}_t}\left[\frac{\alpha'_t}{1 - \alpha_t} \sum_{i=1}^{L} \mathbb{1}[x^i_t = \texttt{[MASK]}] \log p_\theta(x^i_0 \mid \mathbf{x}_t)\right]$

where $L$ is the full sequence length and the conditioning is simply on $\mathbf{x}_t$ , the fully noisy sequence. There is no block structure — every masked token is predicted using global bidirectional context from all other tokens (both masked and unmasked) in the sequence.

The stable phase trains the model extensively on large-scale corpora under this MDLM objective. The block size is fixed at 4096 throughout. The goal is to "deepen its understanding of diffusion dynamics" (Section 4.1) — that is, to give the model enough exposure to the full-sequence denoising task that it becomes proficient at leveraging bidirectional context for reconstruction, developing the internal representations needed for high-quality generation.

Phase 3: Block Size Decay

After extensive MDLM training, the model is a capable full-sequence diffusion model, but full-sequence generation is inefficient at inference time: it requires denoising the entire output in one shot, which prevents KV-cache reuse (since the full sequence changes at each denoising step) and does not support variable-length generation. The decay phase addresses this by converting the model back to a block diffusion model with a small block size.

The decay proceeds by gradually reducing $L_B$ from 4096 to a target size (e.g., 32), with intermediate steps (e.g., 4096 → 2048 → ... → 32). The paper describes this as "distilling the global contextual knowledge learned during MDLM into a compact blockwise structure" (Section 4.1). By decreasing the block size step-by-step rather than abruptly, the model smoothly adapts from global to local conditioning, preserving its semantic understanding — learned through full-sequence attention — while regaining BDLM's practical benefits:

KV-cache reuse: Clean blocks' key-value pairs can be cached and reused across denoising steps, since they don't change once generated.
Variable-length generation: Blocks are generated sequentially, so the model can stop after producing an end-of-sequence token in a block, rather than being forced to generate a fixed-length sequence.
Efficient attention: Each block's self-attention is $O(L_B^2)$ rather than $O(L^2)$ , significantly reducing per-step computation.

Summary of WSD Rationale

The three-phase design reflects a careful analysis of the competing demands on the model. The warmup phase prioritizes stability — preventing catastrophic forgetting by making only incremental changes to the model's attention patterns. The stable phase prioritizes efficiency and capability — using the computationally cheaper and more data-efficient MDLM objective to train the model at scale, building strong bidirectional denoising capabilities. The decay phase prioritizes deployability — converting the model's learned capabilities into an architecture that supports fast, flexible inference. Attempting to skip any phase (e.g., training directly with a small block size from the start) would either cause optimization instability (no warmup), suffer from low data utilization and slow training (no stable phase), or produce a model that is expensive to run at inference time (no decay phase).

Document-Level Attention Mask

The WSD strategy governs what the model learns; the attention mask governs how the model processes input sequences to enable efficient, semantically coherent training.

The Packing Problem

To maximize hardware utilization during training, sequences are formed by packing multiple heterogeneous documents into fixed-length segments (e.g., concatenating document A, document B, and document C into one 4096-token training example). This is standard practice in AR training — the causal attention mask naturally prevents documents from attending to each other because each token only sees tokens to its left, and documents are separated by end-of-text tokens.

For diffusion models, this packing creates a serious problem. Under bidirectional attention (needed for denoising), tokens from document A could attend to tokens from document B, forming spurious cross-document dependencies. A token in a physics textbook might attend to a token in a recipe, learning associations that are semantically meaningless and degrading the model's ability to learn coherent document-level representations. The paper describes this as "contextual confusion" that "significantly [hinders] the model's ability to perform robust bidirectional modeling crucial for denoising" (Section 4.2).

The Document-Level Mask Solution

The solution is to modify the attention mask to enforce document boundaries: each token can only attend to other tokens within the same document. This preserves the computational efficiency of packed training (no wasted padding tokens) while ensuring semantic coherence (no cross-document interference).

Block Diffusion Attention Mask (for WSD Warmup and Decay)

During block diffusion training (warmup and decay phases), the attention mask must handle a more complex structure. The input sequence is constructed as a concatenation of the noisy sequence $\mathbf{x}_t$ followed by the clean sequence $\mathbf{x}_0$ , producing a full input of length $2L$ . The block diffusion vectorized forward pass processes multiple blocks in parallel by carefully controlling attention patterns.

Given that tokens $i$ and $j$ belong to the same document (enforced by an initial document-level mask — a simple block-diagonal mask that restricts attention within document boundaries), the attention mask $\mathbf{M} \in \{0, 1\}^{2L \times 2L}$ is defined in Equation 3 as:

$\mathbf{M}_{ij} = \begin{cases} \mathbf{1}_{b(i)=b(j)} & \text{if } i \in \mathbf{x}_t \text{ and } j \in \mathbf{x}_t \\ \mathbf{1}_{b(i)>b(j-L)} & \text{if } i \in \mathbf{x}_t \text{ and } j \in \mathbf{x}_0 \\ \mathbf{1}_{b(i-L)\geq b(j-L)} & \text{if } i \in \mathbf{x}_0 \text{ and } j \in \mathbf{x}_0 \\ 0 & \text{otherwise} \end{cases}$

where:

$i, j \in \{0, 1, \ldots, 2L-1\}$ are token indices in the concatenated $[\mathbf{x}_t; \mathbf{x}_0]$ sequence;
$b(k) = \lfloor k / L_B \rfloor$ maps a token index to its block index within its respective half (for $k < L$ , the block index is relative to $\mathbf{x}_t$ ; for $k \geq L$ , subtract $L$ to get the block index relative to $\mathbf{x}_0$ );
$\mathbf{1}_{\text{condition}}$ is the indicator function, returning 1 when the condition is true and 0 otherwise;
$L_B$ is the current block size.

What it computes: The mask is a binary matrix of size $2L \times 2L$ where $\mathbf{M}_{ij} = 1$ means token $j$ can be attended to by token $i$ (query $i$ , key $j$ ) and $\mathbf{M}_{ij} = 0$ means the attention is blocked. The four cases specify four attention patterns:

Case 1 ( $i \in \mathbf{x}_t$ , $j \in \mathbf{x}_t$ ): Self-attention within the noisy sequence. The condition $\mathbf{1}_{b(i)=b(j)}$ implements block-diagonal attention: a token in noisy block $k$ can only attend to other tokens in the same noisy block $k$ . This means each block is denoised independently using information from its own tokens — there is no cross-block attention within $\mathbf{x}_t$ . This is crucial because different noisy blocks represent different generation steps during inference; they should not have access to each other's partially denoised state.

Case 2 ( $i \in \mathbf{x}_t$ , $j \in \mathbf{x}_0$ ): Cross-attention from the noisy sequence to the clean sequence. The condition $\mathbf{1}_{b(i)>b(j-L)}$ means a token in noisy block $k$ can attend to tokens in clean blocks with indices less than $k$ (strictly preceding blocks). The subtraction $j - L$ converts from full-sequence index $j$ to the clean-sequence-relative index. This is "block-causal" cross-attention: the noisy block can see clean text from earlier blocks (which provide context) but not from later blocks (which haven't been "generated" yet in the inference order).

Case 3 ( $i \in \mathbf{x}_0$ , $j \in \mathbf{x}_0$ ): Self-attention within the clean sequence. The condition $\mathbf{1}_{b(i-L)\geq b(j-L)}$ implements block-causal attention: a token in clean block $k$ can attend to tokens in its own block and all preceding clean blocks ( $\leq k$ ). This preserves the causal structure within the clean context, which is important because during inference, clean blocks are generated left-to-right.

Case 4 (otherwise): This covers attention from $\mathbf{x}_0$ to $\mathbf{x}_t$ , which is explicitly blocked (0 everywhere). Clean tokens should not attend to noisy tokens because during inference, clean tokens represent finalized output and should not incorporate information from the still-malleable noisy block.

The result is a structured attention matrix (illustrated in Figure 2, right panel) with three zones: a block-diagonal pattern within $\mathbf{x}_t$ , an offset block-causal pattern from $\mathbf{x}_t$ to $\mathbf{x}_0$ , and a block-causal pattern within $\mathbf{x}_0$ . All attention is further constrained by the document boundary mask, so tokens in document A never attend to tokens in document B, regardless of their block indices.

Why this form: This mask design enables efficient vectorized training by allowing multiple blocks to be processed in a single forward pass. Instead of processing each block sequentially (which would require $K$ separate forward passes), the model processes all blocks simultaneously with the attention mask ensuring each block only sees the information it is allowed to see. The block-diagonal pattern within $\mathbf{x}_t$ prevents information leakage between different noisy blocks (which represent independent generation tasks in the batch). The block-causal cross-attention and self-attention preserve the sequential dependency structure that makes block diffusion equivalent to autoregressive generation at the block level. The document-level constraint prevents semantically meaningless cross-document attention.

MDLM Attention Mask (for WSD Stable)

During the stable phase with full-sequence MDLM ( $L_B = L$ ), the mask simplifies dramatically. There is only one block, so the block-diagonal and block-causal patterns collapse. The mask $\mathbf{M} \in \{0, 1\}^{L \times L}$ is defined in Equation 4 as:

$\mathbf{M}_{ij} = \begin{cases} 1, & \text{if } i, j \text{ belong to the same document}, \\ 0, & \text{otherwise}. \end{cases}$

What it computes: Every token can attend to every other token within the same document — full bidirectional attention with document-boundary constraints. There is no separation into noisy and clean halves because the MDLM objective directly denoises the full sequence.

Why this form: Full bidirectional attention is the defining feature of MDLM — it allows the model to use context from all positions when predicting each masked token, which is what gives diffusion models their potential advantage in tasks requiring holistic understanding. The document-level constraint remains critical because packed sequences still contain multiple documents; without it, the model would learn spurious cross-document associations that degrade its document-level semantic modeling.

Comparison with Alternative Approaches

The paper notes that the authors experimented with alternative techniques like random-length training (Xie et al., 2025) and CART reweighting (Ye et al., 2025) but found that "the document-level attention mask is more fundamental in CPT training compared to these techniques, and it consistently achieves superior performance" (Section 4.2). Random-length training varies the sequence length during training to improve length generalization, and CART adjusts per-token loss weights based on masking frequency. While these may provide incremental benefits, the document-level mask addresses a more fundamental problem — semantic coherence — that directly affects the quality of the representations the model learns during bidirectional training.

Top-k Checkpoint Merge and Numerical Stability

Top-k Checkpoint Merge

After completing the BDLM pre-training (the full WSD pipeline), the paper applies a post-hoc model averaging technique to improve generalization. Rather than using the final checkpoint, the authors identify the top $k$ best-performing checkpoints (selected based on validation perplexity from different training steps near the end of training) and arithmetically average their parameters (weights and biases) to form a single merged model.

The paper references the WSM scheduler (Tian et al., 2025) for this approach and highlights its key advantage: it is optimizer-agnostic, meaning it can be applied as a post-hoc step without modifying the training pipeline. Unlike Exponential Moving Average (EMA), which continuously averages parameters throughout training and ends up heavily weighted toward the final steps, the top-k merge explicitly selects distinct, high-performing states from different points in training and averages them equally. This "ensembles diverse 'knowledge' captured by the model at various optimal or near-optimal training states" (Section 4.3), smoothing the parameter landscape and producing a more robust model. The paper does not specify the value of $k$ or the exact selection criterion beyond "validation metrics like perplexity."

Numerical Stability: Gaussian Noise Injection for Masked Embeddings

A subtle but critical implementation detail: when converting from AR to diffusion training, the model can experience gradient explosion, "especially at high mask ratios within a document" (Section 7.1). The root cause is that during AR training, masked token embeddings are never used (since AR models never see [MASK] tokens), so their embedding vectors are never updated from their initial values. Over the course of AR pre-training, the optimizer (e.g., AdamW) updates the embedding matrix, but the rows corresponding to the [MASK] token remain at their random initialization. Meanwhile, the optimizer's internal state (first and second moments) for these rows accumulates based on zero gradients. When diffusion training begins and the [MASK] embeddings suddenly receive gradients, the combination of zero-valued parameters and stale optimizer state can produce disproportionately large updates, causing gradient norms to explode.

The straightforward fix — randomly reinitializing the masked token embeddings when loading the AR checkpoint — is rejected because it "may disrupt other well-trained parameters, potentially causing catastrophic forgetting" (Section 7.1). Reinitializing one row of the embedding matrix changes the embedding of the [MASK] token, but because the [MASK] token appears frequently in diffusion training, this random vector interacts with the rest of the model's parameters (which are optimized for the old [MASK] embedding) and can produce large, destabilizing gradients.

The paper's solution is a compromise: add independent Gaussian noise to the output of the embedding layer for each masked token during the initial iterations of training. Specifically, rather than changing the embedding vector itself, the model adds noise to the embedding output (after the lookup, before it enters the transformer layers) only for positions that are masked. The noise is chosen such that the L2 norm of the masked token's representation remains "significant" — large enough that the subsequent layers' weight matrices, which have been trained to handle representations of that magnitude, don't produce vanishing or exploding activations. This prevents gradient explosion without permanently altering the embedding matrix, allowing the model to gradually adapt the [MASK] embedding through normal gradient updates over the course of training. The paper describes this as applied "during the initial iterations of training" (Section 7.1), implying it is a temporary measure that can be phased out once the [MASK] embedding has been updated to a reasonable value.

Supervised Fine-Tuning with Block Diffusion

After CPT produces a base block diffusion model, the SFT stage (Section 5.1) adapts it to follow instructions. The core objective is the block diffusion loss conditioned on a user prompt.

The Conditional Block Diffusion SFT Objective

The SFT loss (Equation 5) extends the BDLM objective to incorporate a prompt $c$ :

$\mathcal{L}_{\text{SFT}}(\theta) = -\mathbb{E}_{t, (c, \mathbf{x}_0), \mathbf{x}_t}\left[\frac{\alpha'_t}{1 - \alpha_t} \sum_{k=1}^{K} \sum_{i=1}^{L_B} \mathbb{1}[x^i_{t,k} = \texttt{[MASK]}] \log p_\theta(x^i_{0,k} \mid c, \mathbf{x}_{0,<k}, \mathbf{x}_{t,k})\right]$

where $c$ is the instruction prompt, $\mathbf{x}_0$ is the target response, and all other symbols have the same meaning as in Equation 1.

What it computes: This is identical to the BDLM pre-training loss except for the added conditioning on $c$ . For each training example: (1) concatenate the prompt $c$ and the clean response $\mathbf{x}_0$ ; (2) mask tokens only in the response portion according to the diffusion noise schedule (the prompt is always fully visible); (3) the model predicts the original response tokens conditioned on the prompt, the preceding clean blocks of the response, and the noisy current block.

Why this form: By masking only the response and keeping the prompt clean, the model learns to generate responses given instructions. The block diffusion structure ensures that generation during inference can proceed block-by-block, with each block conditioned on the prompt and previously generated blocks — identical to the training setup.

Padding Strategies and Mask Ratio Bandwidth

Two practical optimizations improve SFT training dynamics:

Padding to block boundaries. Because the block diffusion attention mask requires exact block divisions, each response sequence is padded to a length that is a multiple of the block size $L_B$ . The paper calls this the "effective length" — the original sequence length rounded up to the nearest multiple of $L_B$ . Tokens in the padded region are excluded from the loss computation.

Mask ratio bandwidth. Standard discrete diffusion samples the mask probability $1 - \alpha_t$ uniformly across the full range $[0, 1]$ by sampling $t \sim \text{Uniform}(0, 1)$ . However, extreme mask ratios provide minimal useful signal: when very few tokens are masked ( $\alpha_t \approx 1$ ), the denoising task is trivially easy (most tokens are just copied from the input); when almost all tokens are masked ( $\alpha_t \approx 0$ ), the denoising task reduces to unconditional generation (predicting the data distribution without context, since there is essentially no visible context). Both extremes produce high-variance gradients that destabilize training without contributing meaningful learning.

The mask ratio bandwidth strategy clips the noise schedule to a bounded interval $[\alpha_{\text{min}}, \alpha_{\text{max}}]$ , restricting mask probabilities to an intermediate range where the denoising task is neither trivial nor impossible. The paper references Arriola et al. (2025) for this technique and states that it "focuses the training objective on the noise regimes that provide the most informative gradients, thereby stabilizing convergence and improving the model's generative perplexity" (Section 5.1). The specific values of $\alpha_{\text{min}}$ and $\alpha_{\text{max}}$ are not provided in the paper.

Complementary Masking

Complementary masking (Section 5.1, attributed to Li et al., 2025) is a data efficiency technique that doubles the effective utilization of each training sample. The core idea is to generate two training instances from a single clean sequence $\mathbf{x}_0$ :

A primary noisy sequence $\mathbf{x}_t$ , created by applying a random mask (each token is independently masked with probability $1 - \alpha_t$ ).
A complementary noisy sequence $\mathbf{x}'_t$ , created by applying the logical inverse of that mask — every token that was NOT masked in $\mathbf{x}_t$ IS masked in $\mathbf{x}'_t$ , and vice versa.

By including both $\mathbf{x}_t$ and $\mathbf{x}'_t$ in the same training batch, the method provides a deterministic guarantee: every token position in the original sequence appears in its uncorrupted (clean) state exactly once across the pair. In $\mathbf{x}_t$ , the masked positions get a [MASK] token, and the model must predict the clean token for those positions — but it learns nothing about the unmasked positions in $\mathbf{x}_t$ . In $\mathbf{x}'_t$ , the previously unmasked positions are now masked, so the model learns to predict those tokens. Together, the pair covers the entire sequence.

What this accomplishes: Without complementary masking, a single training example provides learning signal for only the masked positions (fraction $1 - \alpha_t$ of the sequence). With complementary masking, every token position is masked and learned from in at least one of the two instances, guaranteeing 100% token coverage per pair. This "entirely eliminates token-level sampling bias" (Section 5.1) — no token position is systematically under-trained because of how the random masks happen to fall. The result is faster convergence and more robust learning.

Why it is only used in post-training: The paper notes an interesting empirical finding: complementary masking "only works fine on corpus less than 100B tokens, while it does not show advantages with more training data" (Section 5.1, footnote). The hypothesis is that with very large datasets, the random masking process already provides sufficient coverage of all token positions across different training examples, so the explicit guarantee of complementary masking becomes unnecessary. SFT datasets are typically much smaller than pre-training corpora (millions of examples rather than billions), so the technique provides meaningful benefit in the post-training regime.

Data Recipe Curation

The SFT dataset is organized into three "pillars": Reasoning (mathematics, code generation — builds analytical and logical capabilities), General (creative writing, dialogue — builds linguistic richness and social intelligence), and Industrial (domain-specific workflows with real-world constraints — builds applied problem-solving). The paper describes this as an "integrated methodology" that "ensures a holistic skill profile, preventing capability skew and enabling fluid shifts between abstract reasoning and applied problem-solving" (Section 5.1). No specific dataset sizes, sources, or proportions are provided.

Confidence-Aware Parallel (CAP) Training

The bottleneck in diffusion language model inference is the number of denoising steps required to produce acceptable output quality. Each step, the model must perform a full forward pass, and the total inference time scales linearly with the number of steps. Standard diffusion models typically require many steps because the model's predictions at intermediate noise levels are uncertain — the probability distribution over possible tokens at each masked position is diffuse, meaning few tokens exceed the confidence threshold for early acceptance.

The Mechanism

CAP training (Section 5.2, inspired by dParallel, Chen et al., 2025b) adds an auxiliary loss that sharpens the model's predictive distribution for correctly predicted tokens. The primary SFT loss $\mathcal{L}_{\text{SFT}}$ ensures the model assigns high probability to the correct token at each position, but it provides "diminishing incentive to sharpen the predictive distribution for tokens that are already correctly predicted" — once the model's top prediction matches the ground truth, the cross-entropy loss is already low, and further increasing the probability of the correct token (from, say, 0.6 to 0.9) yields minimal additional reduction in the loss.

The confidence loss $\mathcal{L}_{\text{conf}}$ addresses this by selectively minimizing the entropy of the model's output distribution $p_\theta(x_0 \mid x_t, c)$ , but only for the subset of tokens that are correctly predicted in a given step. For incorrectly predicted tokens, the confidence loss is not applied — the model should not become more confident about wrong answers. The final training objective (Equation 6) is:

$\mathcal{L}(\theta) = \mathcal{L}_{\text{SFT}}(\theta) + \lambda \mathcal{L}_{\text{conf}}(\theta)$

where $\lambda$ is a hyperparameter balancing the two objectives (value not specified in the paper).

Why this matters for inference: During inference, LLaDA2.0 uses a hybrid acceptance strategy (Section 5.4). At each denoising step, the model produces a probability distribution for every remaining masked position. Tokens whose predicted probability exceeds a predefined confidence threshold (set to 0.95 in experiments) are immediately accepted — their predicted token is "locked in" and they become unmasked context for future denoising steps. If an insufficient number of tokens meet the threshold, a "low-confidence fallback" accepts a fixed number of the most probable tokens regardless of their absolute confidence. A sharper predictive distribution means more tokens exceed the threshold at each step, reducing the total number of steps needed and thus increasing throughput.

The empirical effect is shown in Figure 3: on four code and math benchmarks, LLaDA2.0-flash-CAP achieves 535 tokens/second (tokens per second of decoded output, measured as total decoding tokens divided by total inference time) compared to 383 tokens/second for LLaDA2.0-flash without CAP — a 40% speedup — while benchmark scores slightly improve (78.57 vs. 76.85 average). The CAP model also achieves higher tokens-per-forward (TPF), meaning more tokens are accepted per denoising step on average (approximately 4.65 vs. 3.18), which directly translates to fewer total steps and lower latency.

Direct Preference Optimization (DPO) for Diffusion Models

After SFT, the model is further aligned with human preferences using DPO (Section 5.3). The adaptation requires addressing a fundamental incompatibility: standard DPO assumes the model can compute exact log-likelihoods of sequences, but diffusion models only provide a variational lower bound (ELBO) because the generation process involves multiple stochastic denoising steps.

The Block Diffusion ELBO

The paper defines a conditional Block Diffusion ELBO (Equation 7) that serves as a surrogate for the intractable log-likelihood:

$\mathcal{B}_{\text{BDLM}}(\theta, x \mid c) = \mathbb{E}_{t, \mathbf{x}_t}\left[\frac{\alpha'_t}{1 - \alpha_t} \sum_{k=1}^{K} \sum_{i=1}^{L_B} \mathbb{1}[x^i_{t,k} = \texttt{[MASK]}] \log p_\theta(x^i_k \mid c, \mathbf{x}_{<k}, \mathbf{x}_{t,k})\right]$

What it computes: For a given response $x$ and prompt $c$ , this estimates the model's log-probability of generating $x$ using a single Monte Carlo sample over timesteps $t$ and noise $\mathbf{x}_t$ . The inner expression is identical to the SFT loss (Equation 5) but without the negative sign — it is the sum of log-probabilities of the correct tokens at masked positions, weighted by the time-dependent factor. This is a stochastic estimate because a single $(t, \mathbf{x}_t)$ pair is sampled rather than integrating over all possible noise levels and mask patterns.

Why ELBO rather than exact likelihood: Computing the exact likelihood of a sequence under a diffusion model requires marginalizing over all possible denoising trajectories — all sequences of intermediate states $\mathbf{x}_{t_1}, \mathbf{x}_{t_2}, \ldots$ that could lead from the fully masked initial state to the final clean sequence. This sum is intractable for sequences of non-trivial length (the number of trajectories grows exponentially). The ELBO provides a tractable lower bound that can be estimated efficiently with a single sample per training step. For DPO, using the ELBO as a surrogate for log-likelihood is a standard technique in diffusion model alignment — the assumption is that models with higher ELBO scores for preferred responses will also (on average) assign higher true likelihood to those responses.

The DPO Objective

Given a preference pair $(x_w, x_l)$ — a preferred ("winning") response and a dispreferred ("losing") response for the same prompt — the DPO loss (Equation 8) is:

$\mathcal{L}_{\text{DPO}}(\theta) = -\mathbb{E}_{(c, x_w, x_l) \sim \mathcal{D}}\left[\log \sigma\left(\beta \left[\Delta\mathcal{B}(x_w \mid c) - \Delta\mathcal{B}(x_l \mid c)\right]\right)\right]$

where:

$\mathcal{D}$ is a dataset of 1.5 million preference pairs spanning general knowledge, mathematics, and instruction following;
$\sigma(\cdot)$ is the logistic sigmoid function $\sigma(z) = 1/(1 + e^{-z})$ , which maps the margin to a probability in $[0, 1]$ ;
$\beta = 0.1$ is a hyperparameter controlling the deviation from the reference policy;
$\Delta\mathcal{B}(x \mid c) = \mathcal{B}_{\text{BDLM}}(\theta, x \mid c) - \mathcal{B}_{\text{BDLM}}(\theta_{\text{ref}}, x \mid c)$ is the ELBO advantage: the difference between the policy model's ELBO for response $x$ and the reference model's ELBO for the same response;
$\theta_{\text{ref}}$ is the frozen reference policy, initialized from the post-SFT model (before DPO training begins).

What it computes: For each preference pair, the model computes the ELBO for both responses under both the current policy $\theta$ and the frozen reference $\theta_{\text{ref}}$ . The advantage $\Delta\mathcal{B}$ measures how much better (in ELBO terms) the current policy is at generating each response compared to the reference. The difference $\Delta\mathcal{B}(x_w \mid c) - \Delta\mathcal{B}(x_l \mid c)$ is the margin by which the policy prefers the winning response over the losing response, relative to the reference. The sigmoid converts this margin to a probability, and the negative log-sigmoid loss encourages the margin to be large and positive — meaning the policy assigns much higher relative likelihood to the preferred response.

Why this form: This is a direct adaptation of the standard DPO loss (Rafailov et al., 2023) with the implicit reward defined as $\beta \cdot \Delta\mathcal{B}$ . The key departure is replacing exact log-probabilities with ELBO estimates. The $\beta$ parameter controls how far the policy can deviate from the reference: small $\beta$ keeps the policy close to the SFT model (conservative updates), while large $\beta$ allows more aggressive optimization toward the preference signal. The value $\beta = 0.1$ is a standard conservative choice that prioritizes maintaining the SFT model's general capabilities while still incorporating preference information.

Implementation detail: The learning rate for DPO is initialized to be the same as the final learning rate from the SFT stage, ensuring a smooth transition without the optimization shock that could occur if the learning rate were reset to a higher value.

Inference: Block-wise Threshold Decoding with Hybrid Acceptance

The inference procedure (Section 5.4) connects the training objectives to the deployment behavior. Generation proceeds block-by-block, with each block undergoing multi-step iterative denoising.

Step-by-step procedure:

Block initialization: A new block of $L_B$ tokens is initialized as fully masked (all [MASK] tokens).
Iterative denoising: At each denoising step, the model takes as input the prompt $c$ , all previously generated (clean) blocks $\mathbf{x}_{<k}$ , and the current partially denoised block $\mathbf{x}_{t,k}$ . The model outputs a probability distribution over the vocabulary for every remaining masked position.
Hybrid acceptance: For each masked position, the model's predicted probability of its top token is compared to a predefined confidence threshold (set to 0.95 in all experiments). If the probability exceeds the threshold, the token is accepted — the [MASK] is replaced with that token, and the position becomes unmasked context for subsequent steps. If an insufficient number of tokens meet the threshold (i.e., the model is not confident enough about any token to make meaningful progress), the "low-confidence fallback" activates: a fixed number of the highest-probability tokens are accepted regardless of their absolute confidence.
Repeat or advance: Steps 2–3 repeat until all positions in the block are unmasked. The completed block is then added to the sequence of clean blocks, and generation advances to the next block.
Termination: Generation stops when the model produces an end-of-sequence token or a maximum number of blocks is reached.

Why hybrid acceptance: The dual mechanism (threshold-based + fallback) addresses a practical failure mode of pure threshold-based decoding. If the threshold is high (e.g., 0.95) and the model is uncertain (e.g., sampling in a low-probability region of the data distribution), it is possible that zero tokens exceed the threshold in a given step. Without a fallback, the decoding process would stall — no progress would be made, and the inference would hang. The fixed-number fallback guarantees "steady generation progress" (Section 5.4) by ensuring that at least some tokens are unmasked at every step, even when the model lacks confidence.

The block size $L_B = 32$ and threshold 0.95 used in the main experiments were chosen through the hyperparameter analysis in Section 6.3. The analysis found that a block size of 32 provides the best balance: it achieves nearly the same accuracy as the slower $L_B = 16$ setting (average score 70.15 vs. 70.26 on the analysis subset) while providing substantially higher throughput (2.55 TPF vs. 2.44 TPF, where TPF is tokens-per-forward, a measure of how many tokens are accepted per denoising step). A larger block size of 64 degraded both quality and speed. Similarly, the threshold 0.95 achieved the highest quality score (70.15) compared to lower thresholds (0.90: 69.56, 0.85: 67.90), though at the cost of lower TPF (2.55 vs. 2.93 vs. 3.31). The choice of 0.95 reflects a design priority for output quality over maximum speed — the CAP training partially compensates for the speed penalty by sharpening predictions so that more tokens meet the high threshold.

Connection to training design: The inference procedure is the reason for the decay phase in WSD. Training with a small block size ( $L_B = 32$ ) during the decay phase ensures that the model's representations are optimized for the exact inference setup — denoising blocks of 32 tokens with conditioning on preceding clean blocks. The confidence-aware parallel training directly optimizes the model for the threshold-based acceptance criterion by making its predictions sharper, increasing throughput without changing the inference algorithm. The DPO alignment ensures that the model's generation preferences align with human judgments, closing the loop from pre-training to deployment.

4. Key Insights and Innovations

Innovation 1: Knowledge Inheritance as a First-Class Design Principle for Diffusion Language Models

The most distinctive conceptual move in this paper is not any single training technique, but the elevation of knowledge inheritance from a pragmatic shortcut to a foundational design principle. Prior work on diffusion language models fell into two camps: train from scratch and accept the resulting capability gap (as with LLaDA, Dream-7B, and Quakka, all capped at ~8B parameters), or initialize from AR checkpoints as a cost-saving measure but treat the conversion as a one-time initialization hack rather than a principled preservation problem (as with RND1's immediate attention conversion or DiffusionLLaMA's mask annealing). The unstated assumption in both approaches was that the AR model's knowledge is incidental — a convenient starting point that can be overwritten — rather than essential — a hard-won capability that must be actively preserved throughout conversion.

LLaDA2.0 reframes the problem entirely. The paper treats the AR checkpoint not as a warm start to be consumed and discarded, but as a knowledge asset whose preservation constrains every design decision in the pipeline. This is visible in three places:

The Warmup phase is designed around preservation, not just stability. Progressive block size expansion (1 → 4 → 32 → 64 → 4096) is not merely an optimization trick to avoid gradient spikes — it is a deliberate strategy to keep the model's internal representations close to their AR-optimized geometry at each step. The model only moves as fast as it can without forgetting. This is fundamentally different from RND1's approach of immediate conversion with constrained dense-layer updates, which treats forgetting as a damage-control problem (limit the damage) rather than a prevention problem (don't cause the damage).

The rejection of embedding reinitialization reveals the depth of the commitment. When the paper encounters gradient explosion from stale [MASK] embeddings, the obvious fix — randomly reinitializing those embedding rows — is explicitly rejected because it might disrupt other well-trained parameters and cause catastrophic forgetting. Instead, the paper adds temporary Gaussian noise to the embedding outputs, a fix whose entire justification is that it preserves the AR model's knowledge while solving the numerical problem. A less preservation-oriented team would have reinitialized and moved on; the paper's refusal to do so signals that knowledge preservation is not negotiable.

The top-k checkpoint merge is retrospective preservation. Rather than trusting the final checkpoint to have retained everything, the paper explicitly ensembles the best-performing states from across training. This is an admission that even with careful conversion, some knowledge may drift during training, and the best solution is to combine snapshots from different points. This is not a standard practice in diffusion model training — it is borrowed from the AR scaling literature (Tian et al., 2025) and adapted specifically to address the preservation problem.

The significance of this reframing extends beyond this paper. It implies that future work on diffusion language models should not ask "how do we train diffusion models from scratch?" but rather "how do we convert the best available AR models while retaining what they know?" This inverts the relationship between the two paradigms: AR models become the foundation, and diffusion models become an upgrade path rather than an independent lineage. It also implies that investment in better AR models directly benefits the diffusion ecosystem — a better starting checkpoint yields a better final diffusion model — which makes the diffusion research program less zero-sum with AR development than it initially appeared.

This is a fundamental reframing, not an incremental refinement. Prior work treated AR initialization as an implementation detail; LLaDA2.0 treats it as the central organizing principle.

Evidence: The entire WSD strategy (Section 4.1) is structured around preservation. The paper's competitive performance at 100B scale (Table 2) — matching a 30B AR model with a diffusion model of comparable total parameters — would not be achievable if substantial knowledge were lost during conversion. The ablation that found document-level attention masking "more fundamental... than [random-length training or CART]" (Section 4.2) supports the claim that architectural choices serving preservation (preventing cross-document contamination during bidirectional training) matter more than training-curve optimizations.

Innovation 2: The Warmup–Stable–Decay Schedule as a Unified Solution to the Scale Trilemma

The paper's second conceptual contribution is the recognition that scaling diffusion language models faces a trilemma: you cannot simultaneously have (1) stable optimization during AR-to-diffusion conversion, (2) efficient training on large-scale corpora, and (3) practical inference speed — at least not with any single training regime. Each of the three WSD phases resolves one leg of this trilemma by temporarily sacrificing the others, and the overall schedule sequences these tradeoffs so that the final model inherits the benefits of all three.

To see why this is novel, consider the alternatives the paper explicitly rejects:

Training only with small block sizes (like a standard BDLM) would give efficient inference (leg 3) but catastrophic data utilization (leg 2 fails) — only a fraction of tokens contribute to the loss at each step, making training on trillions of tokens prohibitively expensive.
Training only with full-sequence MDLM would give efficient training (leg 2) but no inference-time KV-cache reuse or variable-length generation (leg 3 fails) — the model would have to denoise the entire output in one shot.
Directly switching from AR to MDLM (skipping warmup) would give efficient training (leg 2) but optimization instability and catastrophic forgetting (leg 1 fails) — the distributional shock would destroy the AR knowledge the conversion is meant to preserve.
Using mask annealing or CART reweighting (as in DiffusionLLaMA, Dream-7B) addresses stability (leg 1) but doesn't solve the training efficiency problem — these methods still operate under block diffusion or full-sequence objectives but don't dynamically switch between them to exploit the computational advantages of MDLM.

The WSD schedule is ingenious because it temporally decouples the three objectives. The warmup phase sacrifices training efficiency (small blocks, low data utilization) in favor of stability — it uses just enough data at each block size to adapt the model without forgetting, accepting the inefficiency as the price of safe conversion. The stable phase sacrifices inference practicality (the model becomes a full-sequence MDLM, useless for efficient decoding) in favor of training throughput — it exploits the halved attention cost and full token coverage of MDLM to train on massive corpora efficiently. The decay phase sacrifices some of the global context capabilities built during the stable phase (the model must adapt back to local conditioning) in favor of inference efficiency — it distills the MDLM's learned representations into a blockwise architecture that supports KV-cache reuse and fast decoding.

Prior work did not recognize this as a trilemma at all. Methods like DiffusionLLaMA and Dream-7B tried to solve stability and efficiency simultaneously with a single training regime (mask annealing + CART), which is why they topped out at 7B–30B parameters — the compromises required to maintain stability during conversion imposed efficiency ceilings that made larger-scale training infeasible. SDAR (Cheng et al., 2025) explored block size as a variable but treated it as a hyperparameter to be optimized for a fixed training run, not as a schedule to be dynamically varied.

The WSD framework is significant because it provides a template for scaling beyond 100B. If someone wants to build a 500B diffusion model, the recipe scales: start from a 500B AR checkpoint, use a longer warmup with finer-grained block size increments, train extensively under MDLM with the full 500B parameters, and then decay back to an efficient block size. The phases can be independently scaled — more warmup steps for larger models, more stable-phase tokens for better capability, smaller final block sizes for faster inference — without redesigning the entire pipeline.

This is a fundamental insight (the trilemma framing) that enables an incremental advance (the WSD schedule itself is a specific instantiation of the trilemma-aware design, but the trilemma concept generalizes). The paper does not explicitly name the trilemma, but the three-phase structure and the explicit discussion of why each phase exists (stability, efficiency, deployability) makes the underlying logic clear.

Evidence: The training infrastructure discussion (Section 7.1) reports that the cuDNN-based attention implementation achieves "more than 1.3× end-to-end speedup and over 90% memory savings" compared to unfused attention, but this is during block diffusion training. The stable phase's MDLM training would see even larger gains because the attention computation is further simplified (no clean-attention component). The paper does not report ablation of the WSD schedule itself (e.g., skipping the decay phase and comparing block-diffusion-trained vs. WSD-trained models), which is a limitation — but the trilemma logic is internally consistent and the competitive final performance (Table 2) is consistent with all three phases contributing as designed.

Innovation 3: Diffusion Models Can Match AR Models at Frontier Scale — But the Advantage Is Task-Specific, Not Uniform

The paper's most consequential empirical finding is not that LLaDA2.0 achieves competitive overall benchmark scores (73.18 vs. 73.60 for the flash variant against Qwen3-30B-A3B, Table 2), but rather where and how it achieves parity and advantage. The pattern of results suggests that diffusion architectures have inherent strengths in structured, multi-step generation tasks that become visible only at sufficient scale — and that these strengths coexist with continued weaknesses in knowledge-intensive tasks.

Consider the disaggregated results for LLaDA2.0-flash (Table 2) compared to Qwen3-30B-A3B-Instruct-2507:

Coding tasks show consistent diffusion advantage. HumanEval: 94.51 vs. 93.29. MBPP: 88.29 vs. 86.65. MultiPL-E: 74.87 vs. 70.67. This is not a single-outlier story — the advantage appears across three independent coding benchmarks. The gap is small (1–4 points) but consistent.
Agent and tool-use tasks also favor diffusion. BFCL v3: 75.43 vs. 73.19. CodeIF-Bench: 58.00 vs. 54.00. Nexus FC: 50.45 vs. 49.93. Again, small but consistent margins across multiple benchmarks.
Knowledge and reasoning tasks are mixed or AR-favoring. MMLU: 87.69 vs. 87.13 (essentially tied). GPQA-Diamond: 61.98 vs. 57.34 (diffusion advantage). But HellaSwag: 84.97 vs. 86.31 (AR advantage). AIME 2025: 60.00 vs. 61.88 (AR advantage). There is no clear pattern in either direction.
The aggregate score (73.18 vs. 73.60) masks this task-type structure.

What makes this finding intellectually distinctive is that it provides the first large-scale evidence for a hypothesis that was previously only theoretical: diffusion models' bidirectional context during generation may confer structural advantages for tasks requiring constraint satisfaction, multi-step planning, and holistic consistency, while their denoising-based training may lag behind AR models' left-to-right causal reasoning on tasks that benefit from sequential logical chains. Coding and agentic tool use are precisely the domains where global consistency matters — a function definition must be consistent with its call sites, a tool call must respect the API's argument structure, and the entire output must form a coherent program — and these are exactly where LLaDA2.0-flash pulls ahead.

Prior work at smaller scales could not test this hypothesis because the overall capability gap between diffusion and AR models was too large. LLaDA (8B) was "competitive with similarly sized AR counterparts" in aggregate but did not show task-type-specific patterns. Dream-7B achieved "performance on par with top-tier AR models" but at a scale where coding and agent benchmarks are less informative (7B models score much lower on these tasks, compressing the measurement range). LLaDA2.0-flash, at 100B total parameters evaluated against a similarly sized AR MoE model, is in the regime where differential strengths can emerge from the noise.

The paper does not overclaim this finding — it is presented cautiously as a pattern that "may have opened a new door" (Section 8) — but the implication is significant. If diffusion models are genuinely better at structured generation, then the optimal model architecture may be task-dependent: AR for sequential reasoning, diffusion for constraint-heavy generation, hybrid approaches for mixed workloads. This is a more nuanced future than the "diffusion replaces AR" narrative that some early dLLM work implied.

This finding is empirically novel (first evidence at scale) but the underlying hypothesis is theoretically pre-existing (the bidirectional advantage has been speculated since the first MDLMs). The contribution is providing the scale necessary to test the hypothesis credibly.

Evidence: Table 2, with the task-type pattern described above. The inference speed results (Figure 3) reinforce the practical significance: the coding/agent advantage coexists with 2.1× faster inference, making the tradeoff unambiguously favorable in those domains. The hyperparameter analysis (Figure 4) confirms that the block size and threshold choices are near-optimal, reducing the concern that the results are artifacts of suboptimal AR inference configuration.

Innovation 4: The Document-Level Attention Mask as a Diagnostic for a Deeper Problem

At first glance, the document-level attention mask (Section 4.2) appears to be a minor implementation detail — an engineering fix for the well-known packing problem in language model training. But the paper's treatment of it reveals something more interesting: it is a diagnostic tool that identifies cross-document contamination as a qualitatively different problem for diffusion models than for AR models, and the fact that it "consistently achieves superior performance" over alternative techniques like random-length training and CART reweighting (Section 4.2) suggests that semantic coherence during bidirectional training is a deeper challenge than the field has recognized.

In AR training, packing multiple documents into a single sequence is safe because the causal attention mask naturally prevents cross-document attention — tokens in document B cannot attend to tokens in document A because document A is to their right. The end-of-text token between documents provides a clean semantic boundary, and the model never needs to learn which tokens belong to which document because the attention pattern inherently separates them.

In diffusion training, this protection disappears. Under bidirectional attention, tokens in document B can attend to tokens in document A, and if they do, the model learns to use information from the physics textbook to predict tokens in the recipe — forming spurious correlations that degrade its ability to model either document independently. This is not a minor data-quality issue; it fundamentally undermines the model's document-level semantic representations because the training signal is systematically contaminated.

What makes this finding intellectually distinctive is that it identifies a previously overlooked failure mode of bidirectional training at scale. Prior work on packing for diffusion models (e.g., LLaDA, Dream-7B) either did not discuss the issue, handled it implicitly through separate-sequence batching (wasting compute on padding), or assumed that random-length training variations would prevent the model from overfitting to cross-document patterns. The paper's explicit ablation — testing random-length, CART, and document-level masking and finding the mask to be "more fundamental" — demonstrates that cross-document contamination is not solved by training-curve tweaks; it requires architectural intervention.

The broader implication is that bidirectional training is more vulnerable to data-organization artifacts than causal training, and that scaling diffusion models to massive heterogeneous corpora requires more careful attention to data boundaries than the AR literature would suggest. This is a transferable insight: any future work training large diffusion language models on web-scale data (where documents are diverse and unrelated) should adopt document-level attention masking as a default, not an optional optimization.

This is an incremental contribution in mechanism (document-level masking is a known technique in other contexts) but a fundamental diagnostic in implication: it surfaces a scaling-specific failure mode that was invisible at the ≤8B scales where most prior dLLM work operated.

Evidence: The paper states that the document-level mask "consistently achieves superior performance" in CPT compared to alternative techniques (Section 4.2). While no specific ablation numbers are reported for this comparison, the fact that the authors tested multiple alternatives and concluded that the mask is "more fundamental" suggests a non-trivial performance gap. The stability benefit is also indirect: preventing cross-document contamination reduces gradient noise and semantic confusion, contributing to the overall stability that enables WSD to work at 100B scale — a claim that would be difficult to isolate experimentally but is consistent with the paper's emphasis on the mask throughout all training phases.

5. Experimental Analysis

Evaluation Methodology

Dataset. The primary evaluation is conducted across a suite of 47 benchmarks organized into five dimensions: Knowledge (10 benchmarks including MMLU, MMLU-Pro, GPQA-Diamond, ARC, CMMLU), Reasoning (12 benchmarks including SQuAD 2.0, DROP, HellaSwag, BIG-Bench Hard, MuSR), Coding (13 benchmarks including HumanEval, MBPP, MultiPL-E, LiveCodeBench, Spider), Math (8 benchmarks including GSM8K, MATH, OlympiadBench, AIME 2025), and Agent & Alignment (4 benchmarks including BFCL, IFEval, CodeIF-Bench). The benchmarks collectively span standard academic QA, competition mathematics, code generation, function calling, and instruction following. No single "test split" is described; the paper reports results on each benchmark's standard evaluation set using the benchmark's native grading protocol.
Base model(s). Two base models are evaluated: LLaDA2.0-mini (16B total parameters, a MoE variant with unspecified active parameters) and LLaDA2.0-flash (100B total parameters, MoE). Both are the instruction-tuned outputs of the full LLaDA2.0 training pipeline (WSD CPT + SFT + optional CAP + DPO), initialized from the Ling family of AR models — Ling-mini-2.0 for the 16B variant and Ling-flash-2.0 for the 100B variant. The models are chosen to represent two deployment scales: resource-constrained (mini) and high-performance (flash).
Metrics. The primary metric is benchmark-specific accuracy (or equivalent score — e.g., pass@1 for code generation, exact match for math, task-specific grading for BFCL). For cross-benchmark aggregation, the paper reports an unweighted average score across all 47 benchmarks, equivalently weighting each benchmark regardless of its native scale or number of examples. A secondary metric used in the inference hyperparameter analysis is Tokens Per Forward (TPF) — the average number of tokens accepted per denoising step, where higher TPF indicates faster decoding. For end-to-end speed comparisons, Tokens Per Second (TPS) is reported, defined as total decoding tokens divided by total inference time.
Baselines. The paper compares against strong open-source AR models of comparable scale. For LLaDA2.0-mini, the baselines are: Qwen3-8B (no-think mode — a dense 8B AR model) and Ling-mini-2.0 (the AR base model from which LLaDA2.0-mini was converted, also ~16B total parameters MoE). The Qwen3-8B comparison tests cross-family competitiveness; the Ling-mini-2.0 comparison tests whether the diffusion conversion preserves the original model's capabilities. For LLaDA2.0-flash, the baselines are: Qwen3-30B-A3B-Instruct-2507 (a 30B total, 3B active parameter MoE AR model) and Ling-flash-2.0 (the AR base model, ~100B total parameters MoE). The Qwen3-30B-A3B is chosen as the closest open-source peer in capability scale. The paper also reports an intermediate LLaDA2.0-flash-preview checkpoint to show progression during development. For inference speed comparisons (Figure 3), additional baselines are the AR models' inference throughput when deployed with SGLang under consistent generation settings.
Generation budget / compute accounting. For the main evaluation, all LLaDA2.0 models use a fixed generation configuration: temperature 0.0 (greedy decoding within the denoising process), block size 32, and decoding threshold 0.95. The number of denoising steps is not fixed — it varies per example based on how quickly tokens meet the confidence threshold, but the block size and threshold settings determine the expected compute per output token. For the inference speed comparison (Figure 3), the budget is implicitly measured as wall-clock throughput (tokens/second), with all models evaluated on the same hardware under their respective optimized serving frameworks (dInfer for diffusion models, SGLang for AR models). There is no explicit FLOP counting or generation budget comparison — the paper relies on the fixed inference hyperparameters to provide a consistent per-model compute profile, and the speed comparison uses real-world serving throughput rather than theoretical FLOP counts.
Cross-validation / statistical protocol. There is no cross-validation or statistical significance testing reported. The main results (Tables 1 and 2) are single-point evaluations — each model is run once on each benchmark with the specified configuration, and the resulting accuracy is reported. The hyperparameter analysis (Section 6.3) sweeps threshold values (0.85, 0.90, 0.95) and block sizes (16, 32, 64) for LLaDA2.0-mini on a "representative subset" of benchmarks (unspecified which subset or size), with results reported as point estimates. The context length analysis (Figure 5) evaluates both models on the RULER benchmark at context lengths of 4k, 8k, 16k, 32k, and 64k (the latter using YaRN dynamic RoPE scaling with factor 2.0), again as single-point evaluations.

Main Quantitative Results

LLaDA2.0-mini (16B): Competitive with AR Peers, with Early Signals in Reasoning and Coding

The LLaDA2.0-mini model achieves an average score of 64.34 across all 47 benchmarks (Table 1), positioning it between Qwen3-8B (63.42) and Ling-mini-2.0 (65.77). The LLaDA2.0-mini-preview (an earlier checkpoint) scores 54.67, indicating substantial improvement during the final stages of development.

The headline comparison is against the AR base model Ling-mini-2.0 (65.77) — the model LLaDA2.0-mini was converted from. The 1.43-point gap (64.34 vs. 65.77) represents the net effect of the WSD conversion plus post-training: the diffusion model retains ~97.8% of the original AR model's average performance. This is notably strong evidence for the knowledge-preservation claims, since the conversion changes the model's fundamental generation paradigm and attention patterns.

The aggregate masks differential performance across domains. On Reasoning benchmarks, LLaDA2.0-mini shows genuine strength relative to the AR baseline: SQuAD 2.0 (86.50 vs. 75.56, a +10.94 point advantage), OCNLI (64.51 vs. 60.17, +4.34), and HellaSwag (79.01 vs. 69.02, +9.99). However, this pattern is not uniform across reasoning tasks — KOR-Bench shows a substantial deficit (50.40 vs. 62.72, -12.32), as does ZebraLogic (64.20 vs. 79.85, -15.65), both of which test complex multi-step logical deduction. This suggests that the bidirectional advantage in reasoning is task-structure-dependent: tasks requiring holistic understanding of a passage may benefit from bidirectional context, while tasks requiring sequential logical chaining may still favor the AR model's native inference pattern.

On Coding benchmarks, LLaDA2.0-mini is competitive but not dominant: HumanEval (86.59 vs. 85.98, essentially tied), MultiPL-E (67.46 vs. 67.09, tied), MBPP (81.50 vs. 84.07, slightly behind). This is notable because the mini model is the smaller variant — the coding advantage becomes clearer at the flash scale. On Math, LLaDA2.0-mini trails Ling-mini-2.0 on the hardest benchmarks: AIME 2025 (36.67 vs. 47.66), OlympiadBench (67.70 vs. 72.30), Omni-MATH (41.70 vs. 48.80), though GSM8K and MATH-MATH are nearly tied. On Agent & Alignment, LLaDA2.0-mini shows a clear advantage on IFEval (80.78 vs. 76.16, +4.62) and BFCL v3 (70.90 vs. 53.98, +16.92) — the function-calling benchmark — which the paper attributes to diffusion models' structural advantages in constraint satisfaction.

Compared to the external baseline Qwen3-8B (63.42 average), LLaDA2.0-mini (64.34) holds a small aggregate lead. The advantage is concentrated in specific areas: HumanEval (86.59 vs. 84.76), BFCL v3 (70.90 vs. 70.08, essentially tied on this version of the benchmark), and IFEval (80.78 vs. 86.90, a deficit — the Qwen model outperforms on strict instruction following). The comparison is somewhat confounded by the scale difference: LLaDA2.0-mini is a 16B-total MoE model (active parameters unspecified), while Qwen3-8B is a 8B dense model, making the parameter comparison ambiguous.

LLaDA2.0-flash (100B): Parity with Frontier AR Models, with Coding and Agent Advantages

The LLaDA2.0-flash model achieves an average score of 73.18 across all 47 benchmarks (Table 2), essentially tied with Qwen3-30B-A3B-Instruct-2507 at 73.60 — a difference of 0.42 points, or less than 0.6% relative. Against the AR base model Ling-flash-2.0 (72.15), LLaDA2.0-flash holds a 1.03-point aggregate advantage, meaning the diffusion conversion plus post-training actually improves the overall benchmark performance relative to the original AR model. This is the paper's strongest quantitative claim: at 100B scale, a properly converted diffusion model can match or slightly exceed its AR progenitor.

The domain-level breakdown reveals the task-type specificity discussed in Section 4 (Key Insight 3). On Coding benchmarks, LLaDA2.0-flash shows consistent advantages over Qwen3-30B-A3B: HumanEval (94.51 vs. 93.29, +1.22), MBPP (88.29 vs. 86.65, +1.64), MultiPL-E (74.87 vs. 70.67, +4.20), HumanEval+ (87.80 vs. 88.41, essentially tied within 0.61), and LiveCodeBench (42.29 vs. 41.63, +0.66). On 7 of the 12 coding benchmarks, LLaDA2.0-flash matches or exceeds the Qwen model. Compared to the AR base Ling-flash-2.0, the coding advantage is clearer: HumanEval (94.51 vs. 85.98, +8.53), MBPP (88.29 vs. 85.01, +3.28), MultiPL-E (74.87 vs. 65.76, +9.11). These are non-trivial gaps on standard benchmarks.

On Agent & Alignment benchmarks, LLaDA2.0-flash also shows consistent leads over Qwen3-30B-A3B: BFCL v3 (75.43 vs. 73.19, +2.24), CodeIF-Bench (58.00 vs. 54.00, +4.00), Nexus FC (50.45 vs. 49.93, tied). IFEval (81.70 vs. 84.29) is an exception — the Qwen model maintains an advantage on strict instruction following. The function-calling BFCL result is particularly notable because it involves structured JSON generation with specific API constraints, exactly the type of task where bidirectional context during generation might confer an advantage over left-to-right decoding.

On Knowledge benchmarks, the results are mixed with no clear directional pattern: MMLU (87.69 vs. 87.13, tied), GPQA-Diamond (61.98 vs. 57.34, +4.64 — a notable advantage), ARC-c (95.93 vs. 95.81, tied), TriviaQA (66.88 vs. 65.61, tied). On Reasoning benchmarks, the pattern is similarly inconsistent: HellaSwag (84.97 vs. 86.31, slight AR advantage), MuSR (80.48 vs. 79.15, tied), DROP (87.90 vs. 87.57, tied), ZebraLogic (82.30 vs. 90.97, a substantial AR advantage — -8.67 points). The ZebraLogic deficit mirrors the mini model's pattern and suggests that tasks requiring strict sequential logical constraint propagation remain challenging for diffusion architectures. On Math benchmarks, the comparison is close: AIME 2025 (60.00 vs. 61.88, essentially tied), MATH-MATH (95.44 vs. 96.70, slight AR advantage), GSM8K (96.06 vs. 96.36, tied), OlympiadBench (74.07 vs. 77.59, AR advantage).

The progression from LLaDA2.0-flash-preview (65.97 average) to LLaDA2.0-flash (73.18) — a +7.21 point improvement — is reported but not attributed to specific training interventions. This gap likely reflects the cumulative effect of CAP training, DPO alignment, and possibly additional SFT data or training steps. The paper does not provide an ablation decomposition of this improvement, making it unclear which post-training stage contributes how much.

Inference Efficiency: 2.1× Speedup Over AR Models on Code and Math Benchmarks

The inference speed comparison (Figure 3, right panel) reports throughput in tokens per second (TPS) on four benchmarks: HumanEval, MBPP, GSM8K, and CRUXEval. The key numbers:

LLaDA2.0-flash-CAP: 535 TPS
LLaDA2.0-flash (without CAP): 383 TPS
Ling-flash-2.0 (AR baseline, served via SGLang): 256 TPS
Qwen3-30B-A3B-Instruct-2507 (AR baseline, served via SGLang): 237 TPS

LLaDA2.0-flash-CAP achieves a 2.09× speedup over Ling-flash-2.0 and a 2.26× speedup over Qwen3-30B-A3B. Even without CAP training, LLaDA2.0-flash (383 TPS) is 1.50× faster than Ling-flash-2.0 and 1.62× faster than Qwen3-30B-A3B. This speed advantage coexists with the competitive benchmark scores reported above — LLaDA2.0-flash-CAP achieves an average score of 78.57 on the 12-benchmark subset used for the speed-quality tradeoff analysis (Figure 3, left panel), compared to 76.85 for LLaDA2.0-flash without CAP, indicating that the confidence sharpening does not degrade output quality and may slightly improve it.

The Tokens Per Forward (TPF) metric provides insight into the mechanism of the speedup. LLaDA2.0-flash-CAP achieves approximately 4.65 TPF compared to 3.18 TPF for LLaDA2.0-flash without CAP (Figure 3, left). This means CAP training increases the average number of tokens accepted per denoising step by ~46%, directly reducing the total number of forward passes needed per output sequence — and hence the total inference time.

Important caveat on the speed comparison: The AR baselines are run with SGLang, while the diffusion models use dInfer — different serving frameworks with potentially different levels of optimization maturity. The paper notes that "more mature features in dInfer are undergoing to transport to SGLang" (Section 7.3), suggesting that the dInfer framework may have inference optimizations that SGLang lacks for this particular model class. The comparison is described as "fair performance comparison in real inference environments" (Section 7.3), but the framework difference means the speedup should be interpreted as an end-to-end system-level comparison rather than a pure architectural efficiency ratio.

Hyperparameter Analysis: Threshold and Block Size Trade-offs

The hyperparameter analysis (Section 6.3, Figure 4) sweeps two key inference parameters on LLaDA2.0-mini using a "representative subset" of benchmarks (not specified which or how many). The results inform the configuration choices used in the main evaluation.

Denoising Threshold sweep (block size fixed at 32):

Threshold 0.95: Score = 70.15, TPF = 2.55
Threshold 0.90: Score = 69.56, TPF = 2.93
Threshold 0.85: Score = 67.90, TPF = 3.31

The highest threshold (0.95) yields the best quality but slowest decoding; lowering the threshold to 0.85 increases throughput by ~30% (2.55 → 3.31 TPF) at a 2.25-point accuracy cost. The gradient of the quality-speed tradeoff is not linear — the degradation from 0.95 to 0.90 is modest (0.59 points), while the drop from 0.90 to 0.85 is steeper (1.66 points). This suggests a threshold near 0.90–0.95 is near-optimal for quality-sensitive applications, with 0.95 chosen for the main evaluation to maximize accuracy.

Block Size sweep (threshold fixed at 0.95):

Block size 16: Score = 70.26, TPF = 2.44
Block size 32: Score = 70.15, TPF = 2.55
Block size 64: Score and TPF both suboptimal (exact values not given, but described as "degraded both score and speed relative to the size-32 setting")

Block size 32 is selected as the "most compelling choice, offering a significant speed-up for a negligible performance cost" — the accuracy loss from 16 to 32 is 0.11 points while TPF increases from 2.44 to 2.55. Block size 64 is worse on both axes, which the paper attributes to the interaction between larger block size and the confidence-threshold decoder — larger blocks may require more denoising steps because tokens have less context per step, offsetting the throughput gain from processing more tokens in parallel.

Important limitation: The hyperparameter analysis is conducted only on LLaDA2.0-mini (not LLaDA2.0-flash) on a "representative subset" of benchmarks (not the full 47-benchmark suite). The paper extrapolates the optimal settings to LLaDA2.0-flash without verifying that the optimal block size and threshold generalize across model scales. Given that LLaDA2.0-flash is ~6× larger in total parameters and likely has different calibration properties (potentially more confident predictions, different optimal block sizes), this is a meaningful extrapolation.

Context Length Analysis: Robust to 32k, Degradation at 64k

The RULER benchmark evaluation (Figure 5) tests long-context capabilities:

LLaDA2.0-flash: scores of >93 across 4k, 8k, 16k, and 32k context lengths, with the 32k score described as "above 93" (exact values not provided in the text beyond 4k = 93.29 for mini).
LLaDA2.0-mini: 4k = 93.29, 32k = 83.94 — a degradation of ~9.4 points across the context range, showing more sensitivity to sequence length.
At 64k (using YaRN dynamic RoPE scaling with factor 2.0, beyond the native training window of 32k), both models show "performance degradation" (exact values not given).

The flash model's flat performance curve from 4k to 32k suggests that the document-level attention mask and the block diffusion training have not impaired long-context modeling — the model can effectively leverage context across its entire native window. The mini model's more pronounced degradation suggests that the smaller model's capacity limits its ability to maintain consistent quality across context lengths, a pattern consistent with typical AR scaling behavior. The 64k extrapolation results are presented as a flexibility feature (YaRN scaling enables processing longer sequences "albeit with a predictable performance cost") rather than as a primary capability claim.

Ablation Studies and Robustness Checks

CAP Training on inference efficiency and quality: Figure 3 (left panel) compares LLaDA2.0-flash with and without CAP training on a 12-benchmark subset. With CAP: average score 78.57, TPF 4.65. Without CAP: average score 76.85, TPF 3.18. The CAP model achieves both higher quality (+1.72 points) and higher throughput (+46% TPF). This is a non-obvious finding — one might expect a quality-speed tradeoff where sharper predictions risk overconfidence errors, but the results suggest the entropy minimization loss improves both calibration (fewer tokens with middling probabilities) and accuracy (the model's top predictions are more often correct). The paper does not provide a mechanistic explanation for the quality improvement, but it is consistent with the hypothesis that encouraging the model to commit to predictions early in the denoising process acts as a form of implicit regularization that reduces error accumulation across steps.

WSD schedule ablation: The paper does NOT provide an ablation of the WSD schedule. There is no comparison between: (1) WSD-trained model vs. directly-trained BDLM (skip warmup and stable phases), (2) WSD with vs. without the stable phase (warmup directly to small block size, skip MDLM training), or (3) WSD with vs. without the decay phase (stop at MDLM, compare inference efficiency). This is a significant omission — the WSD schedule is the paper's central methodological contribution, but its individual phases are never experimentally validated. The trilemma framing (Section 4, Key Insight 2) provides a conceptual justification, but the empirical evidence for the specific phase contributions is absent.

Document-level attention mask vs. alternatives: The paper reports (Section 4.2) that the document-level attention mask "consistently achieves superior performance" compared to random-length training and CART reweighting, with the mask described as "more fundamental in CPT training." However, no quantitative results are provided — no table, no figure, no specific accuracy or perplexity numbers. This makes it impossible to assess the magnitude of the advantage or whether the mask's benefit is primarily in training stability (enabling successful completion of CPT at all) or in final model quality (higher benchmark scores). Given that random-length and CART are techniques from prior work (DiffusionLLaMA, Dream-7B) that those papers reported as beneficial, the absence of a quantitative comparison weakens the claim of superiority.

Complementary masking effectiveness: The paper reports (Section 5.1, footnote) that complementary masking "only works fine on corpus less than 100B tokens, while it does not show advantages with more training data." This is a negative result — a technique that helps at small data scales does not transfer to large-scale training — but it is reported without quantitative evidence (no learning curves, no ablation table). The finding is attributed to the hypothesis that with very large datasets, random masking provides sufficient token coverage across examples, but this hypothesis is not tested (e.g., by measuring token coverage rates with and without complementary masking at different data scales). The decision to use complementary masking only in post-training (where datasets are smaller) is reasonable given the reported scaling behavior, but the evidence supporting this decision is anecdotal.

ReST^EM-based revision model performance: The paper does not include RL-based post-training experiments for LLaDA2.0, but references prior work (TraceRL, SPG) in the related work section (Section 2.3). This is not an ablation of LLaDA2.0 components but rather an acknowledgment of a direction not yet integrated. The paper positions RL training (specifically reasoning-capable dLLMs with long chain-of-thought) as future work (Section 8).

FLOPs-matched comparison against pretraining: The paper does NOT conduct a FLOPs-matched comparison between scaling test-time compute (i.e., the diffusion model's inference budget) and scaling pretraining compute (i.e., training a larger AR model). This is a notable absence given that such comparisons are a standard feature of scaling analysis papers. The closest the paper comes is the side-by-side benchmark evaluation against similarly-sized AR models (Tables 1-2), but this compares models with different training recipes, data mixtures, and inference costs — it is not a controlled FLOPs-matched experiment. The inference speed results (Figure 3) provide a throughput comparison but do not account for the total training FLOPs invested in each model.

Block size sweep at flash scale: The hyperparameter analysis (Section 6.3) sweeps block size and threshold only on LLaDA2.0-mini. No equivalent analysis is reported for LLaDA2.0-flash. Given that the optimal block size might depend on model scale (larger models may have different calibration properties, different optimal tradeoffs between block size and denoising steps), this extrapolation is an untested assumption.

Ablation of data recipe components: The SFT data curation is described as spanning "three principal pillars: Reasoning, General, and Industrial" (Section 5.1), but no ablation is reported on the contribution of each pillar or the effect of data mixture ratios. It is unclear whether all three pillars are necessary, whether the Industrial pillar (domain-specific workflows) contributes to general benchmark performance, or whether a simpler data mixture would yield comparable results.

Critical Assessment

The experimental results broadly support the paper's central narrative — diffusion language models can be scaled to 100B parameters through AR-to-diffusion conversion, and the resulting models are competitive with similarly sized AR models — but with important qualifications about what the experiments actually demonstrate versus what they claim.

Claim 1: LLaDA2.0 achieves competitive performance with similarly sized AR models. This claim is supported by the aggregate benchmark scores in Tables 1 and 2. LLaDA2.0-flash (73.18) is within 0.42 points of Qwen3-30B-A3B (73.60) and exceeds Ling-flash-2.0 (72.15), its AR progenitor. However, "competitive" is doing substantial work here. The aggregate score masks domain-level variance: LLaDA2.0-flash leads on coding and agent tasks, trails on some reasoning and knowledge tasks, and the pattern is not uniform within domains. A practitioner choosing between LLaDA2.0-flash and Qwen3-30B-A3B would need to consider their specific task distribution — the diffusion model is not a drop-in replacement for the AR model in all settings.

Moreover, the "similarly sized" framing warrants scrutiny. LLaDA2.0-flash is a 100B-total-parameter MoE model; Qwen3-30B-A3B-Instruct-2507 is a 30B-total, 3B-active MoE model. The total parameter counts differ by ~3.3×. If active parameters were used as the comparison metric, the interpretation would shift — LLaDA2.0-flash might be viewed as using more compute per forward pass to achieve similar quality. The paper does not report active parameter counts for its MoE models, making this comparison impossible to evaluate. This is a significant transparency gap for a paper that emphasizes "efficiency-aware design."

Claim 2: Diffusion models show advantages in complex structured domains like code generation and agentic tool use. The evidence is suggestive but not conclusive. LLaDA2.0-flash leads Qwen3-30B-A3B on coding benchmarks (HumanEval: +1.22, MBPP: +1.64, MultiPL-E: +4.20) and agent benchmarks (BFCL v3: +2.24, CodeIF-Bench: +4.00). The margins are small enough that they could be attributable to differences in training data mixtures, SFT recipes, or random variation across benchmarks rather than to the diffusion architecture per se. The paper does not control for these confounds — LLaDA2.0-flash and Qwen3-30B-A3B were trained on different datasets with different post-training procedures, so the coding/agent advantage could reflect data choices rather than architectural benefits. A within-family comparison (LLaDA2.0-flash vs. Ling-flash-2.0) partially addresses this, and here the coding advantage is larger (HumanEval: +8.53, MBPP: +3.28, MultiPL-E: +9.11), but this comparison confounds the architectural change with the additional CPT and post-training stages — the Ling base model was not instruction-tuned on the same data as LLaDA2.0-flash.

A stronger test of the architectural advantage hypothesis would require: (a) training an AR model and a diffusion model on identical data with identical hyperparameter budgets, and comparing them on the same benchmarks; or (b) ablating the diffusion-specific components (bidirectional attention, denoising objective) against an AR baseline with matched training compute. Neither experiment is present.

Claim 3: The WSD strategy enables a smooth and data-efficient conversion from AR to diffusion models. This claim is supported by the fact that the conversion succeeded at 100B scale and the final model performs competitively, but the specific contributions of the WSD phases are never isolated. There is no ablation showing that training without warmup (direct conversion to small block size) causes instability or forgetting at this scale, no comparison of training efficiency with and without the stable MDLM phase, and no evidence that the decay phase improves inference relative to a model trained entirely at the target block size. The WSD schedule's benefits are asserted based on conceptual reasoning (the trilemma framing) rather than demonstrated experimentally.

The numerical stability fix (Gaussian noise injection for masked embeddings) and the top-k checkpoint merge are also presented without ablation — we do not know whether gradient explosion would have occurred without the noise injection, or whether the top-k merge meaningfully improves over the final checkpoint alone. These are implementation details that the paper treats as important for reproducibility, but their actual contribution to the final result is unquantified.

Claim 4: The document-level attention mask is more fundamental than alternative techniques like random-length training and CART. The paper asserts this but provides no quantitative evidence. For a claim of relative superiority, the absence of numbers is a significant gap. The reader is asked to accept that one technique "consistently achieves superior performance" over others without knowing by what margin, on which metrics, or at what scale. This claim would be substantially strengthened by a table showing the performance of the CPT model with different attention mask configurations.

Specific weaknesses in the experimental design:

No ablation of the central WSD mechanism. The three-phase schedule is the paper's signature contribution, but its empirical validation is entirely indirect — we see that the final model works, but not that each phase contributed. A minimal ablation suite would include: (a) WSD vs. direct block diffusion training from the AR checkpoint (skip warmup and stable), (b) WSD with stable vs. WSD with extended warmup (skip stable, test whether full-sequence training is actually necessary), (c) WSD with decay vs. WSD without decay (test inference efficiency at the MDLM block size vs. the decayed block size). The absence of any such ablation makes it impossible to assess whether the WSD complexity is justified or whether a simpler schedule would work equally well.
No FLOPs-matched training comparison. The paper compares inference speed (Figure 3) but does not account for total training cost. Training a 100B-parameter diffusion model via WSD CPT from an AR checkpoint requires: (a) all the training FLOPs of the original AR model (Ling-flash-2.0), plus (b) the CPT FLOPs for warmup, stable, and decay phases, plus (c) the SFT, CAP, and DPO FLOPs. A fair comparison against training a larger AR model from scratch would need to account for this total training budget. The paper does not report training FLOPs, token counts, or training time for any phase, making it impossible to evaluate whether the conversion approach is computationally efficient relative to simply training a larger AR model.
The benchmark suite is extensive but not systematic in its coverage of the paper's claims. The paper claims diffusion advantages in "complex, structured domains like code generation and agentic tool use," but does not include benchmarks specifically designed to test the mechanisms hypothesized to drive this advantage (e.g., benchmarks isolating constraint satisfaction, planning depth, or bidirectional context utilization from other factors like factual knowledge or reasoning length). Most coding benchmarks (HumanEval, MBPP) primarily test functional correctness of short programs, which conflates code generation ability with the specific architectural advantage claimed. Agent benchmarks (BFCL) test structured output generation, but performance on these benchmarks also depends heavily on the model's instruction-following training data, not just its architecture.
Single evaluation per benchmark. The paper reports point estimates for each benchmark without confidence intervals, standard deviations, or multiple evaluation runs. For benchmarks like AIME 2025 with a small number of problems, sampling variance can be substantial. Without uncertainty quantification, small differences between models (e.g., the +1.22 HumanEval advantage for LLaDA2.0-flash over Qwen3-30B-A3B) cannot be distinguished from noise.
The active parameter count is not reported for the MoE models. LLaDA2.0-mini is described as 16B total parameters, and LLaDA2.0-flash as 100B total parameters, but the expert configuration (number of experts, number active per token) is never specified. This makes it impossible to compare the models' per-token inference cost against the baselines. If LLaDA2.0-flash activates a larger fraction of its parameters per token than Qwen3-30B-A3B (which uses 3B active out of 30B total), then the TPS comparison in Figure 3 understates the architectural speed advantage — the diffusion model might be faster in tokens/second despite using more FLOPs per forward pass.
Extrapolation of hyperparameters from mini to flash. The inference hyperparameters (block size 32, threshold 0.95) were optimized on LLaDA2.0-mini and applied unchanged to LLaDA2.0-flash. Whether these settings are near-optimal for the 100B model is untested. Given the substantial difference in model scale, the flash model may have different calibration curves (predictive confidence distributions), which could shift the optimal threshold, and different compute-to-context tradeoffs, which could shift the optimal block size. The paper's decision not to re-tune for the larger model is a practical compromise but introduces uncertainty about whether the reported flash results represent the model's full potential.
The preview-to-final improvement is unexplained. LLaDA2.0-flash-preview scores 65.97; LLaDA2.0-flash scores 73.18 — a +7.21 point jump. The paper does not specify which training stages, data changes, or hyperparameter adjustments occurred between these checkpoints. This makes it difficult to assess whether the improvement reflects fundamental training progress (more CPT tokens, more SFT data) or optimization tuning (better hyperparameters, data mixture adjustments) that might partially reflect overfitting to the benchmark distribution.

What would strengthen the paper:

A WSD phase ablation quantifying the contribution of warmup (stability), stable (training efficiency / data utilization), and decay (inference efficiency) to the final model quality and speed.
Training FLOPs / token counts for each phase of the pipeline, enabling a FLOPs-matched comparison against AR models.
Active parameter counts for the MoE models, enabling meaningful per-forward-pass FLOP comparisons.
Multiple evaluation runs with confidence intervals, especially for small benchmarks like AIME and LiveCodeBench.
A benchmark or analysis specifically isolating the mechanism hypothesized to give diffusion models an advantage (e.g., constrained generation tasks, tasks with varying degrees of bidirectional dependency).
A replication of the hyperparameter sweep on LLaDA2.0-flash rather than extrapolating from mini.
A direct training data comparison — SFT an AR model on the same instruction dataset used for LLaDA2.0 and evaluate on the same benchmarks, to control for data effects when attributing performance differences to architecture.

6. Limitations and Trade-offs

LLaDA2.0 Requires an Existing Frontier-Scale AR Checkpoint — This Is a Dependency, Not an Independence

LLaDA2.0 is not a method for training diffusion language models from scratch at scale. It is a method for converting a pre-existing, fully trained autoregressive model into a diffusion model. The paper is explicit about this dependency from the outset, framing it as a deliberate design choice:

"Rather than attempting to train diffusion models from scratch, we leverage existing AR checkpoints as the foundation for a systematic conversion process that preserves linguistic knowledge while introducing diffusion capabilities." (Section 1)

This means that the entire pipeline is parasitic on the AR ecosystem: every advance in LLaDA2.0's capability is bounded by the quality of the AR checkpoint it starts from. If no suitable AR model exists for a target domain (e.g., a low-resource language, a specialized scientific corpus, a multimodal modality), there is no LLaDA2.0 pathway to a diffusion model for that domain without first training an AR model — which defeats the purpose of avoiding scratch training.

Consequence: The practical implication is that LLaDA2.0 does not reduce the total training compute required to obtain a frontier-scale language model relative to the AR paradigm — it only converts the model's inference paradigm after the AR training investment has already been made. The full cost of producing LLaDA2.0-flash includes (a) all pretraining FLOPs for Ling-flash-2.0 (the 100B AR base model), (b) all CPT FLOPs for the Warmup, Stable, and Decay phases, and (c) all post-training FLOPs for SFT, CAP training, and DPO. The paper does not report the token counts or FLOPs for any of these stages (Section 7.1 describes the infrastructure but not the total compute budget), making it impossible to assess whether the total training investment — AR pretraining plus diffusion conversion — is competitive with simply training a larger AR model. If the CPT stages require, say, 30% as many tokens as the original AR pretraining, then the total cost to obtain a competitive 100B diffusion model might exceed the cost of training a 130B AR model, which could achieve similar or better benchmark scores without the conversion overhead.

Evidence in the paper: The paper reports no training FLOPs, token counts, or training durations for any stage. The only cost-related figure is the inference speed comparison (Figure 3), which addresses deployment cost but not total cost of ownership. The benchmark results (Tables 1-2) show that LLaDA2.0-flash (73.18 average) is competitive with Qwen3-30B-A3B (73.60), but the comparison does not account for the possibility that the total training compute for LLaDA2.0-flash (AR base + CPT + post-training) may far exceed that for Qwen3-30B-A3B, making the apparent parity less impressive when normalized by training FLOPs.

Mitigation status: The paper does not address this limitation. It frames the AR dependency as a feature ("knowledge inheritance") rather than a cost, and does not discuss the total training budget or compare it against alternatives. This is a reasonable framing for the paper's stated goal — providing a practical recipe for the community to leverage existing AR checkpoints — but it means the work does not answer the question of whether diffusion language models, as a paradigm, can be cost-competitive with AR models in total training compute. The open-sourcing of the models partially mitigates the practical impact: downstream users can use LLaDA2.0-flash without paying the training cost, making the dependency on the AR pretraining someone else's problem. But for anyone considering whether to invest in training a diffusion model versus an AR model, the total-cost question remains unanswered.

The WSD Schedule Is the Central Methodological Contribution, but None of Its Three Phases Are Experimentally Validated

The Warmup–Stable–Decay strategy is the paper's signature technical innovation — the mechanism that ostensibly makes 100B-scale AR-to-diffusion conversion possible. Yet the paper provides no ablation study isolating the contribution of any individual phase. There is no comparison between:

Full WSD vs. Warmup only (skip Stable and Decay — train with progressively increasing block sizes, then stop at a target block size): does the Stable phase's full-sequence MDLM training actually improve model quality, or would extended warmup with the target block size achieve comparable results?
Full WSD vs. Stable only (skip Warmup — directly switch to full-sequence MDLM from the AR checkpoint): does the progressive block size increase actually prevent catastrophic forgetting and optimization instability at 100B scale, or would the AR model adapt directly?
Full WSD vs. Warmup + Stable (skip Decay — keep the model at full-sequence MDLM for inference): what is the actual inference speed penalty of using the full-sequence model compared to the decayed block-wise model?

The paper provides conceptual justification for each phase (stability, training efficiency, deployability) organized around the trilemma framing discussed in Section 4, but these are theoretical arguments. Without ablation, a reader cannot determine whether the three-phase schedule is genuinely necessary, whether a simpler two-phase schedule would work, or whether the phases interact in ways the conceptual model does not capture.

Consequence: The paper's central claim — that WSD "enables a smooth and data-efficient conversion from AR to dLLMs" (Section 4) — is supported only by the fact that the conversion succeeded at all at 100B scale, not by evidence that any particular element of WSD was causal. A failed conversion (e.g., catastrophic forgetting, training instability) without warmup would provide strong evidence for the warmup phase's necessity. A successful conversion with a simpler schedule would suggest WSD is over-engineered. Without either, the contribution stands as an existence proof — "we did it this way and it worked" — rather than a validated recipe where each step's function is understood.

This is particularly significant for practitioners who might want to adapt the LLaDA2.0 recipe to different model families, scales, or domains. If the warmup phase is critical for stability only above some threshold scale (say, 50B parameters), then a team converting a 7B model could skip it and save substantial training cost. If the stable phase's efficiency benefit is only realized at very large token counts (say, >500B tokens), then a team with a smaller data budget might train directly with block diffusion. Without ablation, everyone must replicate the full WSD schedule, incurring its full cost, because no one knows which parts are load-bearing.

Evidence in the paper: There is no ablation table, no phase-wise learning curve, no comparison of WSD against alternative schedules. The paper reports that the document-level attention mask is "more fundamental... than [random-length training and CART reweighting]" in CPT (Section 4.2) but provides no quantitative evidence even for that claim. The numerical stability fix (Gaussian noise injection for masked embeddings, Section 7.1) and the top-k checkpoint merge (Section 4.3) are similarly presented as important without quantitative ablation. The paper's ablation philosophy appears to be that final benchmark performance (Tables 1-2) validates all design choices collectively, but this is a weak form of evidence — many component choices could be suboptimal or unnecessary without noticeably affecting the aggregate score against similarly noisy baselines.

Mitigation status: Not addressed. The paper does not acknowledge the absence of phase-level ablation as a limitation. There is no suggestion for future work to systematically ablate the WSD phases or to develop principled criteria for when each phase is necessary. The WSD recipe is presented as a complete unit, and the reader is implicitly asked to trust that all three phases contribute meaningfully.

Difficulty Estimation During Inference Is Not Modeled, and the Computational Cost Could Dominate the Problem-Solving Budget

This is not directly applicable to LLaDA2.0, which does not use difficulty estimation or adaptive compute allocation — it generates every output with a fixed block size and confidence threshold. However, a structurally analogous limitation exists: the confidence-threshold decoding mechanism (Section 5.4) has no guarantee of steady progress, and the low-confidence fallback is a heuristic with no theoretical grounding. When the model is uncertain — as it may be on difficult problems, out-of-distribution inputs, or adversarial prompts — the threshold-based acceptance may stall, requiring repeated fallback activations that degrade the parallel decoding advantage and may produce lower-quality output.

The paper's hyperparameter analysis (Section 6.3, Figure 4) shows that lowering the threshold from 0.95 to 0.85 increases throughput by ~30% (2.55 → 3.31 TPF) at a 2.25-point accuracy cost. This suggests that the threshold mechanism forces a quality-speed tradeoff that is sensitive to the chosen value, but the optimal value depends on the difficulty distribution of the input prompts. The paper's benchmarks are drawn from standard evaluation suites that may not represent the difficulty profile of real-world deployment prompts. On a distribution skewed toward harder problems (where the model is less confident), the threshold decoder might require more fallback steps, reducing throughput below the reported figures and potentially degrading quality due to the heuristic fallback acceptance.

More fundamentally, the threshold decoder assumes that confidence (predicted probability of the top token) correlates with correctness. This assumption is not validated in the paper beyond the aggregate TPF and score metrics, which show that CAP training improves both confidence and accuracy simultaneously (Figure 3). However, this aggregate correlation does not guarantee per-token calibration — it is possible that the model is systematically overconfident on certain token types (e.g., rare tokens, tokens in long-tail contexts) and underconfident on others, leading to uneven generation quality that aggregate benchmarks do not capture.

Consequence: In deployment, LLaDA2.0's inference speed and quality may degrade on inputs that differ from the benchmark distribution in ways that affect model confidence. This includes out-of-distribution prompts, adversarial inputs designed to induce uncertainty, and long-form generation tasks where error accumulation across many blocks compounds the effects of imperfect calibration. The reported 2.1× speedup over AR models (535 vs. 256 TPS, Figure 3) was measured on four code and math benchmarks (HumanEval, MBPP, GSM8K, CRUXEval) — tasks where the model has high confidence and the output length is relatively short. On longer, more open-ended generation tasks, the speed advantage may shrink because each block requires more denoising steps when the model is less certain about its predictions.

Evidence in the paper: The hyperparameter sweep (Figure 4) demonstrates the quality-speed tradeoff but only for LLaDA2.0-mini on a "representative subset" of benchmarks — the same behavior may or may not hold for LLaDA2.0-flash on a different input distribution. The CAP training results (Figure 3) show that confidence sharpening improves throughput, but the improvement is measured on the same four benchmarks used for the speed comparison, creating a potential circularity: the benchmarks where CAP training helps may be precisely those where the model is already well-calibrated. There is no evaluation of how the threshold decoder performs on inputs specifically designed to probe calibration (e.g., out-of-distribution detection benchmarks, adversarial prompts, or tasks with deliberately ambiguous or underspecified instructions).

Mitigation status: The paper includes the low-confidence fallback specifically to address the stalling problem, ensuring "steady generation progress" (Section 5.4) even when few tokens exceed the threshold. However, this is a heuristic patch, not a solution to the underlying calibration problem. When the fallback activates, it accepts a fixed number of tokens based on relative probability rather than absolute confidence — these tokens are precisely those the model is least certain about, so fallback acceptance likely introduces errors that propagate to subsequent denoising steps within the block and to subsequent blocks. The paper does not analyze the frequency of fallback activation across different input types, the error rate of fallback-accepted tokens, or the downstream quality impact of fallback-driven generation.

The Coding and Agent Advantages Are Plausibly Attributable to Training Data, Not Architecture

The paper's most interesting empirical finding is that LLaDA2.0-flash shows consistent advantages over Qwen3-30B-A3B on coding benchmarks (HumanEval: +1.22, MBPP: +1.64, MultiPL-E: +4.20) and agent benchmarks (BFCL v3: +2.24, CodeIF-Bench: +4.00). The paper attributes this pattern to the diffusion architecture's potential strengths in structured generation and constraint satisfaction:

"This suggests that as diffusion models scale, their inherent strengths in structured generation and tool use become increasingly apparent." (Section 6.2)

However, this inference conflates architectural effects with training data effects. LLaDA2.0-flash and Qwen3-30B-A3B were trained on completely different datasets with different curation strategies, different SFT mixtures, and different preference alignment procedures. The paper's own SFT data recipe (Section 5.1) includes an "Industrial pillar" that "embeds domain-specific expertise by simulating end-to-end workflows under real-world constraints" — a data component specifically designed to improve performance on structured, tool-use-oriented tasks. If Qwen3-30B-A3B's training data had a different emphasis (e.g., more general dialogue, less industrial workflow simulation), the observed coding/agent advantage could reflect data composition, not architecture.

The within-family comparison (LLaDA2.0-flash vs. Ling-flash-2.0, its AR progenitor) partially controls for base model pretraining data but not for post-training data. LLaDA2.0-flash underwent SFT, CAP training, and DPO on the LLaDA2.0-specific data mixture, while Ling-flash-2.0's post-training recipe is not described in the paper and likely differs. The coding advantage over Ling-flash-2.0 is larger (HumanEval: +8.53, MBPP: +3.28, MultiPL-E: +9.11) than over Qwen3-30B-A3B, which could reflect either the architectural benefit of diffusion, the additional CPT stages improving the model's coding capabilities, or the post-training data emphasizing coding tasks more heavily than Ling-flash-2.0's original post-training.

Consequence: The claim that diffusion architectures have "inherent strengths in structured generation" remains a hypothesis, not an established finding. A practitioner choosing between LLaDA2.0-flash and an AR model for a code generation or agent deployment cannot confidently attribute the reported benchmark differences to the diffusion architecture and therefore cannot predict whether these advantages will generalize to their specific task distribution or persist as AR models improve their own structured generation capabilities (e.g., through better tool-use training data). The finding is also vulnerable to a confound: if the LLaDA2.0 team invested more effort in curating coding and agent training data (because these are domains where they hypothesized a diffusion advantage), the observed pattern could be a self-fulfilling prophecy driven by data investment, not architecture.

Evidence in the paper: The paper does not compare LLaDA2.0-flash against an AR model trained on the same SFT and DPO data. The Ling-flash-2.0 baseline is the closest such comparison, but it uses Ling's original post-training, not the LLaDA2.0 post-training recipe. The paper does not provide a data composition breakdown that would allow readers to assess whether the coding/agent benchmarks were overrepresented in LLaDA2.0's training data relative to standard AR post-training mixtures. The SFT data description (Section 5.1) is qualitative ("Reasoning pillar hones analytical and logical faculties... Industrial pillar embeds domain-specific expertise") without quantitative proportions or examples.

Mitigation status: The paper does not acknowledge this confound. The interpretation of the coding/agent advantage as an architectural effect is presented without qualification. The closest the paper comes to addressing it is the acknowledgment in Section 8 that the advantages "may have opened a new door to future work," but this hedging is about the strength of the evidence, not about the potential confound. A proper control — training an AR model on the identical post-training data and comparing against LLaDA2.0-flash — is not suggested as future work.

LLaDA2.0's Inference Speed Advantage Coexists with Unreported Active Parameter Counts, Making FLOPs-Normalized Comparisons Impossible

The paper reports a 2.1× inference speedup for LLaDA2.0-flash-CAP (535 TPS) over AR baselines (256 and 237 TPS for Ling-flash-2.0 and Qwen3-30B-A3B respectively) on four code and math benchmarks (Figure 3). This is presented as evidence that diffusion models can achieve faster inference than equivalently capable AR models. However, the paper does not report the active parameter count for either LLaDA2.0 model, making it impossible to determine whether the speed advantage reflects architectural efficiency or simply using more FLOPs per forward pass in a way that amortizes favorably across denoising steps.

LLaDA2.0-mini is described as a 16B-total-parameter MoE model; LLaDA2.0-flash as a 100B-total-parameter MoE model. But a MoE model's inference cost per token is determined by its active parameters — the number of parameters actually computed during a forward pass, which depends on the number of experts and the top-k routing configuration. If LLaDA2.0-flash activates, say, 10B parameters per token while Qwen3-30B-A3B activates 3B parameters per token (as its name suggests), then LLaDA2.0-flash is performing ~3.3× more FLOPs per forward pass. The fact that it achieves only 2.1× higher tokens-per-second means its FLOPs-per-output-token efficiency is actually worse than the AR baseline — the speed advantage in tokens/second is more than offset by the higher per-forward-pass cost.

Conversely, if LLaDA2.0-flash activates a similar number of parameters per token as Qwen3-30B-A3B (despite its larger total parameter count, due to more aggressive expert sparsity), then the speed advantage genuinely reflects architectural efficiency. Without the active parameter count, the reader cannot distinguish these scenarios.

Consequence: The paper's efficiency claims are uninterpretable without active parameter information. The statement that LLaDA2.0 offers "2.1× speed-up over the AR baselines" (Section 7.3) could mean any of: (a) the diffusion architecture is genuinely more FLOP-efficient per output token, (b) LLaDA2.0 uses more FLOPs per forward pass but fewer forward passes, with the net effect being faster wall-clock time on the specific hardware used, or (c) the speed advantage is partly or wholly attributable to the dInfer vs. SGLang serving framework difference, with dInfer being better optimized for the specific hardware configuration. The paper acknowledges the framework difference but frames the comparison as "fair performance comparison in real inference environments" (Section 7.3), which conflates architectural efficiency with engineering optimization.

This limitation is particularly significant because it undermines the paper's central value proposition. The primary motivation for diffusion language models is inference-time parallelism — the ability to generate multiple tokens simultaneously to reduce latency and increase throughput. If the speed advantage is achieved by using a larger active parameter count (and thus more compute per token), it is not a proof of the diffusion paradigm's efficiency but rather a demonstration that the speed penalty of the denoising process can be masked by throwing more compute at each step — a strategy that AR models could also employ (e.g., by using a larger draft model in speculative decoding).

Evidence in the paper: The paper never states the active parameter count, number of experts, or expert routing configuration for either LLaDA2.0-mini or LLaDA2.0-flash. The model architecture is described only as "Mixture-of-Experts (MoE)" (Abstract, Section 1). The inference speed comparison (Figure 3) reports tokens/second but not FLOPs/second or FLOPs/output token. The TPF metric (tokens per forward pass) measures decoding efficiency in terms of model forward passes, but without the cost per forward pass (in FLOPs or time), TPF alone does not indicate total computational efficiency.

Mitigation status: Not addressed. The paper does not discuss the active parameter count, does not report per-token FLOPs, and does not acknowledge the ambiguity created by its omission. The open-sourcing of the models (HuggingFace link) means that the community can eventually determine the active parameter count from the model configuration files, but this information should be in the paper itself as it is central to the efficiency claims. The framework difference (dInfer vs. SGLang) is mentioned as a contextual detail but not discussed as a potential confound for the speed comparison.

The Benchmark Suite Is Broad but Not Deep — Task-Specific Architectural Advantages Are Claimed Without Task-Specific Benchmarks

The paper's evaluation suite of 47 benchmarks (Section 6.1) is extensive in coverage, spanning knowledge, reasoning, coding, math, and agent tasks. However, the benchmarks are standard evaluation tools designed to measure general model capability, not to isolate the mechanisms hypothesized to give diffusion models an advantage. The paper claims that diffusion architectures offer benefits for "complex, structured domains like code generation and agentic tool use" (Section 6.2) and attributes this to bidirectional context enabling better constraint satisfaction and holistic consistency. But none of the coding or agent benchmarks in the suite are designed to test constraint satisfaction independent of other capabilities.

For example, HumanEval and MBPP primarily test whether a model can generate a functionally correct function given a docstring and a few test cases. Performance on these benchmarks depends on multiple factors: understanding the problem specification, recalling relevant programming patterns, generating syntactically correct code, and ensuring logical correctness. The diffusion model's bidirectional context might help with some of these (e.g., ensuring the function body is consistent with the docstring's type hints) but is irrelevant to others (e.g., knowing the right algorithm). Without benchmarks that isolate constraint satisfaction — such as tasks where the output must satisfy explicit, measurable constraints that are provided in the prompt and do not depend on external knowledge — the paper cannot determine whether the coding advantage is driven by the hypothesized mechanism or by other factors (training data, better post-training, scale).

Similarly, the agent benchmark BFCL v3 tests whether the model can generate correct function calls given API documentation. Performance depends on instruction following, schema understanding, and output formatting. Diffusion models might be better at formatting (producing valid JSON with correct field types) due to bidirectional consistency, or they might be better at schema understanding for reasons unrelated to architecture. The benchmark does not disentangle these.

Consequence: The paper's central empirical narrative — that diffusion models have inherent strengths in structured generation — rests on correlational evidence from general-purpose benchmarks that were not designed to test this hypothesis. A skeptic could argue that the coding/agent advantages simply reflect LLaDA2.0 having better post-training data for these domains, or that the diffusion architecture's bidirectional context provides a minor formatting advantage that inflates scores on benchmarks with strict output parsing requirements without reflecting deeper capability improvements. Without benchmarks that specifically manipulate the degree of bidirectional dependency in the task (e.g., varying whether later parts of the output constrain earlier parts), the architectural advantage hypothesis cannot be tested.

Evidence in the paper: The benchmark descriptions in Section 6.1 list standard evaluation tools with citations but do not discuss which benchmarks test which specific capabilities or how they relate to the diffusion architecture's hypothesized strengths. The results analysis (Section 6.2) points to coding and agent benchmark scores as evidence of diffusion advantages but does not analyze the error types, the nature of the generated outputs, or any qualitative differences between LLaDA2.0 and AR model outputs that would reveal the mechanism. There is no error analysis, no qualitative comparison of generated code or function calls, and no breakdown of which subtasks within each benchmark drive the score differences.

Mitigation status: Not addressed. The paper does not discuss the gap between its hypothesized mechanism (bidirectional context enabling better constraint satisfaction) and its empirics (standard benchmark scores). Future work on "test-time scaling" for diffusion models is mentioned (Section 8), but the need for mechanism-specific evaluation is not identified. The qualitative examples that would help substantiate the architectural advantage hypothesis — e.g., side-by-side comparisons of LLaDA2.0 and AR outputs on tasks with explicit constraints — are not provided.

7. Implications and Future Directions

How This Work Changes the Landscape

LLaDA2.0 shifts the diffusion language model field from a research curiosity at small scale to a credible deployment pathway at frontier scale. Prior to this work, the implicit consensus — reinforced by every prior dLLM paper capping out at ≤30B parameters — was that diffusion models could demonstrate interesting properties in controlled experiments but could not compete with production autoregressive models where it mattered. This paper breaks that ceiling not by a small increment but by a categorical leap: from ~30B to 100B total parameters, from "competitive with similarly sized AR counterparts" to matching the Qwen3-30B-A3B frontier model on aggregate benchmarks (73.18 vs. 73.60, Table 2) while delivering 2.1× inference throughput on code and math tasks (535 vs. 256 tokens/second, Figure 3). This changes the conversation from "can diffusion models work?" to "where should we use them?"

The paper's most consequential reframing is its treatment of AR models not as competitors to be beaten but as foundations to be upgraded. By establishing knowledge inheritance as a first-class design principle — backed by a concrete, documented conversion pipeline — LLaDA2.0 effectively inverts the dependency relationship between the two paradigms. Before this work, training a diffusion language model meant starting over, discarding the billions of dollars of invested compute embedded in existing AR checkpoints. After this work, every AR checkpoint is a potential diffusion model waiting to be converted. This has two immediate effects on the research landscape:

First, it makes diffusion model research cumulative with AR progress. Every improvement in AR pretraining — better architectures, larger datasets, more efficient training recipes — directly benefits the diffusion ecosystem because those improved AR checkpoints become better starting points for WSD conversion. A team that spends $100M training a better AR model has, knowingly or not, also produced a better future diffusion model. This aligns incentives across the two communities rather than forcing zero-sum competition for scarce training compute.

Second, it changes the risk calculus for adopting diffusion architectures in deployment. The primary barrier to production use of diffusion language models was not performance per se — LLaDA (8B) was already "competitive" at its scale — but the capability cliff between research-scale diffusion models and the 70B–400B AR models that power real applications. By demonstrating that conversion can bridge this gap while preserving (and in some domains improving) the original model's capabilities, LLaDA2.0 reduces the perceived risk of investing in diffusion-specific inference infrastructure. A deployment team can now reason: "We'll train or obtain the best AR model we can, convert it to diffusion, and get faster inference at comparable quality on our task distribution." This is a much easier sell than "we'll train a diffusion model from scratch and hope it catches up."

The paper also provides a reconciliation mechanism for the conflicting narratives around diffusion model capabilities at small scale. Prior work showed tantalizing hints — Dream-7B's planning advantages, LLaDA-MoE's MoE scalability, Dream-Coder's sketch-then-fill strategies — but none of these translated into a model that could sit alongside production AR systems. LLaDA2.0 suggests that these early signals were not false positives but scale-limited previews of capabilities that become practically meaningful only at 100B+ parameters. The coding advantage that was a 1–2 point curiosity at 7B becomes a 4-point lead on MultiPL-E at 100B (74.87 vs. 70.67 for Qwen3-30B-A3B, Table 2). The agentic tool-use advantage that was a theoretical possibility becomes a 2.2-point lead on BFCL v3 (75.43 vs. 73.19). This reframes prior small-scale dLLM work not as a dead end but as an early-stage exploration whose findings needed scale to become decisive.

However, the paper does NOT establish that diffusion architectures are universally preferable to AR architectures. The aggregate parity (73.18 vs. 73.60) masks domain-level variance: LLaDA2.0-flash trails Qwen3-30B-A3B on HellaSwag (84.97 vs. 86.31), ZebraLogic (82.30 vs. 90.97), and IFEval (81.70 vs. 84.29), all by non-trivial margins. The paper's contribution is therefore more specific and more useful than "diffusion beats AR": it provides evidence that the optimal architecture may be task-dependent, with diffusion models favored for structured generation (code, function calls, tool use) and AR models retaining advantages for certain types of sequential reasoning and strict instruction following. This is a more nuanced future — one of architectural pluralism rather than paradigm replacement — and it redirects research attention toward understanding which tasks benefit from bidirectional generation rather than trying to make diffusion models win everywhere.

Follow-Up Research This Work Enables

WSD phase ablation at 100B scale. The most urgent follow-up is to determine whether all three WSD phases are necessary, or whether a simpler schedule would suffice. A strong experiment would: (1) replicate the AR-to-diffusion conversion at 100B scale (using the same Ling-flash-2.0 checkpoint) with three variants — full WSD, Warmup-only (skip Stable, train directly at the target block size after progressive warmup), and Stable-only (skip Warmup, directly switch to full-sequence MDLM from the AR checkpoint) — and compare final benchmark scores, training stability (gradient norm trajectories, loss spike frequency), and inference throughput. The null result that all three schedules produce similar final models would dramatically simplify future conversion efforts; the result that Warmup-only catastrophically forgets or Stable-only diverges would validate the trilemma framing and provide practitioners with clear guidance on when each phase is load-bearing. The paper's open-source release makes this experiment feasible for any group with access to the Ling checkpoints and sufficient compute.

Training-data-controlled architectural comparison. The paper attributes LLaDA2.0-flash's coding and agent advantages to the diffusion architecture, but the training data confound (different SFT mixtures, different DPO data) makes this attribution speculative. A clean follow-up would: take Ling-flash-2.0 (the AR base), apply the LLaDA2.0 post-training pipeline (SFT with complementary masking + CAP + DPO on the identical data mixture) to produce an AR instruction-tuned model, and compare against LLaDA2.0-flash on the full 47-benchmark suite. Any remaining performance difference can then be attributed to the architectural conversion (WSD CPT + diffusion decoding) rather than to post-training data. If the coding/agent advantage disappears or reverses, the paper's architectural narrative would need substantial revision — the advantages would be revealed as data effects. If the advantage persists, it would be the strongest evidence to date for a genuine architectural benefit of diffusion in structured generation.

Difficulty-calibrated threshold decoding. The confidence-threshold decoder (Section 5.4) uses a fixed threshold (0.95) regardless of input difficulty, but the model's calibration almost certainly varies across problem types. A natural extension is to train a lightweight confidence predictor — a small model that takes the prompt and initial denoising trajectories as input and predicts a per-example optimal threshold or fallback strategy — and use it to dynamically adjust the quality-speed tradeoff. The experiment would: (1) collect per-token confidence and correctness data across the benchmark suite for multiple threshold values, (2) train a predictor to estimate per-example expected accuracy and throughput at each threshold, (3) deploy the predictor to select thresholds that maximize throughput subject to an accuracy constraint. The evaluation would compare static-threshold LLaDA2.0-flash against adaptive-threshold decoding on held-out benchmarks, measuring both average throughput and worst-case accuracy degradation. This directly addresses the unmodeled calibration problem identified in Section 6 while requiring only inference-time changes (no retraining of the base model).

RL-based reasoning for diffusion models at 100B scale. The paper explicitly identifies RL and chain-of-thought reasoning as future work (Section 8), and the TraDo series (Wang et al., 2025d) has already demonstrated that RL-trained dLLMs can produce long chain-of-thought reasoning at smaller scales. The obvious next step is to apply TraceRL or SPG to LLaDA2.0-flash and evaluate on reasoning-intensive benchmarks (AIME, OlympiadBench, ZebraLogic) where the current model trails AR competitors. A strong experiment would compare: (1) LLaDA2.0-flash with SFT + DPO (current), (2) LLaDA2.0-flash with SFT + TraceRL, and (3) the AR base model (Ling-flash-2.0) with equivalent RL training. If RL-trained LLaDA2.0-flash closes or reverses the ZebraLogic gap (82.30 vs. 90.97 for Qwen3-30B-A3B, Table 2), it would demonstrate that the diffusion architecture's current reasoning weaknesses are trainable rather than inherent. If the gap persists even with RL, it would suggest a fundamental limitation of diffusion-based generation for sequential logical deduction — a negative result that would be equally informative for the field.

Block-size scaling laws for diffusion language models. The paper sweeps block sizes {16, 32, 64} only on LLaDA2.0-mini and selects 32 as near-optimal (Figure 4), but this analysis is a point estimate on an unspecified benchmark subset. A systematic study would: measure LLaDA2.0-flash's accuracy and throughput across block sizes from 4 to 256 on the full 47-benchmark suite, fit scaling curves relating block size to both quality and speed, and determine whether the optimal block size depends on model scale (does a 100B model benefit from larger blocks than a 16B model?), task type (do coding tasks want different block sizes than knowledge tasks?), or output length (does longer-form generation favor larger blocks?). The finding that the optimal block size is task-dependent would motivate adaptive block-size scheduling during inference, similar to how the WSD schedule varies block size during training. The finding that optimal block size increases with model scale would inform architectural choices for future even-larger diffusion models (500B+).

Hybrid AR-diffusion decoding for mixed workloads. The paper's results suggest that diffusion models excel on structured generation while AR models retain advantages on sequential reasoning and instruction following. Rather than choosing one paradigm, a hybrid system could route prompts to the appropriate decoder based on predicted task type: diffusion for coding and function-calling queries, AR for logical reasoning and open-ended dialogue. The experiment would: (1) train a lightweight task classifier on prompt embeddings to predict whether a query falls into a "structured generation" or "sequential reasoning" category, (2) deploy LLaDA2.0-flash with both diffusion and AR decoding heads (the AR head is available from the base checkpoint), (3) route each query to the decoder that maximizes expected accuracy subject to a latency budget. The evaluation would compare hybrid routing against pure-diffusion and pure-AR decoding on a held-out mixture of coding, reasoning, and agent tasks, measuring both aggregate accuracy and tail latency. This direction is enabled by LLaDA2.0's unique position as a model that contains both AR and diffusion capabilities (the AR knowledge is preserved through the conversion, even if the model is deployed as a diffusion model).

Practical Applications and Downstream Use Cases

Low-latency code completion in IDEs. A deployment scenario where LLaDA2.0's 2.1× throughput advantage (535 vs. 256 tokens/second, Figure 3) translates directly to user experience is real-time code completion. In an IDE setting, the model must generate completions within a few hundred milliseconds to feel responsive; every millisecond of latency reduces developer acceptance. LLaDA2.0-flash's parallel decoding means it can produce a complete function body in roughly half the wall-clock time of an equivalently capable AR model, while its coding benchmark scores (HumanEval 94.51, MBPP 88.29, Table 2) indicate that the quality of those completions matches or exceeds AR alternatives. The block-wise generation also maps naturally to code structure — a block of 32 tokens corresponds roughly to a line or two of code — meaning the model can generate coherent multi-line completions in a single block without the line-by-line latency of AR decoding. The primary deployment consideration would be the first-token latency: diffusion models require a full forward pass before any tokens are produced, while AR models can stream the first token with lower initial latency. For short completions (single-line), the AR model might still win on time-to-first-token, but for multi-line completions where total generation time dominates, LLaDA2.0's throughput advantage would be decisive.

High-throughput batch inference for synthetic data generation. Organizations generating training data through LLM inference — distilling larger models, creating SFT datasets, or running reinforcement learning from model feedback — operate in a throughput-dominated regime where per-query latency is secondary to total tokens-per-dollar. LLaDA2.0-flash's 2.1× throughput advantage (Figure 3) means that a fixed GPU cluster can generate roughly twice as many training examples per hour compared to an AR model of comparable quality. With CAP training, the advantage grows to 2.26× (535 vs. 237 TPS against Qwen3-30B-A3B). At the scale of modern data generation pipelines — which routinely process millions of examples — this translates to halved infrastructure cost or doubled data volume for the same budget. The use case is particularly compelling for code and function-calling data generation, where LLaDA2.0-flash's benchmark advantages (HumanEval 94.51, BFCL 75.43, Table 2) suggest the generated data would be high quality. The main risk is that the diffusion model's calibration properties might produce systematically different error distributions than AR models, potentially biasing the synthetic data in ways that affect downstream training. A validation step comparing downstream model performance when trained on LLaDA2.0-generated vs. AR-generated synthetic data would be advisable before full-scale adoption.

Agentic tool-use deployments requiring structured output. In production agent systems — where an LLM must generate valid JSON function calls, respect API schemas, and produce syntactically correct structured output — LLaDA2.0-flash offers two advantages over AR alternatives. First, the bidirectional context during generation means the model can simultaneously ensure that the function name, parameter types, and argument values are mutually consistent — a JSON object's closing braces are generated with full knowledge of its opening structure, unlike AR models which must anticipate structure before seeing it. Second, the benchmark results (BFCL v3 75.43, Nexus FC 50.45, Table 2) indicate that these theoretical advantages translate into measurable performance improvements on function-calling benchmarks. In a production setting where malformed function calls cause downstream errors (failed API invocations, retry logic, degraded user experience), even a small reduction in formatting error rate compounds across millions of calls. The deployment would use LLaDA2.0-flash exclusively for the structured output generation step — the system prompt and conversation history can be processed by any model, but the final function call JSON is generated by the diffusion decoder with a high confidence threshold (0.95) to maximize output validity. The block size of 32 tokens is well-suited to typical function call sizes (which often fit in 1–3 blocks), meaning the generation completes in a small, predictable number of denoising steps.

🎯 Pitch

1. Executive Summary

2. Context and Motivation

The Core Problem: Diffusion Language Models Cannot Scale to Frontier Sizes

Why Scale Matters Practically and Theoretically

Prior Approaches and Where They Fall Short

1. Training dLLMs from scratch

2. Initializing dLLMs from AR checkpoints

3. Post-training for dLLMs

How This Paper Positions Itself

3. Technical Approach

3.1 Reader Orientation

3.2 Big-Picture Architecture (Diagram in Words)

3.3 Roadmap for the Deep Dive

3.4 Detailed, Sentence-Based Technical Breakdown

The Training Paradigm: Knowledge Inheritance and Progressive Adaptation

Warmup–Stable–Decay (WSD) Continual Pre-Training

Document-Level Attention Mask

Top-k Checkpoint Merge and Numerical Stability

Supervised Fine-Tuning with Block Diffusion

Confidence-Aware Parallel (CAP) Training

Direct Preference Optimization (DPO) for Diffusion Models

Inference: Block-wise Threshold Decoding with Hybrid Acceptance

4. Key Insights and Innovations

Innovation 1: Knowledge Inheritance as a First-Class Design Principle for Diffusion Language Models

Innovation 2: The Warmup–Stable–Decay Schedule as a Unified Solution to the Scale Trilemma

Innovation 3: Diffusion Models Can Match AR Models at Frontier Scale — But the Advantage Is Task-Specific, Not Uniform

Innovation 4: The Document-Level Attention Mask as a Diagnostic for a Deeper Problem

5. Experimental Analysis

Evaluation Methodology

Main Quantitative Results

LLaDA2.0-mini (16B): Competitive with AR Peers, with Early Signals in Reasoning and Coding

LLaDA2.0-flash (100B): Parity with Frontier AR Models, with Coding and Agent Advantages

Inference Efficiency: 2.1× Speedup Over AR Models on Code and Math Benchmarks

Hyperparameter Analysis: Threshold and Block Size Trade-offs

Context Length Analysis: Robust to 32k, Degradation at 64k

Ablation Studies and Robustness Checks

Critical Assessment

6. Limitations and Trade-offs

LLaDA2.0 Requires an Existing Frontier-Scale AR Checkpoint — This Is a Dependency, Not an Independence

The WSD Schedule Is the Central Methodological Contribution, but None of Its Three Phases Are Experimentally Validated

Difficulty Estimation During Inference Is Not Modeled, and the Computational Cost Could Dominate the Problem-Solving Budget

The Coding and Agent Advantages Are Plausibly Attributable to Training Data, Not Architecture

LLaDA2.0's Inference Speed Advantage Coexists with Unreported Active Parameter Counts, Making FLOPs-Normalized Comparisons Impossible

The Benchmark Suite Is Broad but Not Deep — Task-Specific Architectural Advantages Are Claimed Without Task-Specific Benchmarks

7. Implications and Future Directions

How This Work Changes the Landscape

Follow-Up Research This Work Enables

Practical Applications and Downstream Use Cases

Adjacent in domain

End-to-End Training for Autoregressive Video Diffusion via Self-Resampling

Knot Forcing: Taming Autoregressive Video Diffusion Models for Real-time Infinite Interactive Portrait Animation