Stable-DiffCoder: Pushing the Frontier of Code Diffusion Large Language Model

🎯 Pitch

Under matched data and compute, a diffusion-based code model for the first time beats a strong autoregressive baseline across the board—not just in fast generation, but in raw accuracy and reasoning. The key is a tailored warmup and block-wise clipped noise schedule that stabilizes the transition from causal to bidirectional attention, proving diffusion training is not merely a speed trick but a genuine quality lever for code intelligence.

1. Executive Summary

This paper introduces Stable-DiffCoder, a block diffusion code model that reuses the Seed-Coder architecture, data, and training pipeline but achieves stronger performance through a diffusion-based training paradigm. Controlled experiments on a 2.5B scale inform a three-stage curriculum—efficient knowledge compression via autoregressive pretraining (preserving clean left-to-right reasoning evidence), stable DLLM continued pretraining via a tailored warmup and block-wise clipped noise schedule (capping corruption levels to prevent gradient spikes when switching from causal to bidirectional attention), and small-block diffusion training with block size 4 (providing data augmentation through diverse mask patterns while maintaining training-inference alignment). Under identical data and compute budgets at 8B scale, Stable-DiffCoder overall outperforms its autoregressive counterpart Seed-Coder on a broad suite of code benchmarks, achieving state-of-the-art results among ~8B diffusion models—including substantial gains on low-resource languages like C# and PHP—establishing that diffusion-based any-order modeling improves structured code editing and reasoning capability only when the corruption process respects clean evidence constraints and avoids the correlation/noise regimes that arise from large-block or fully bidirectional masking.

2. Context and Motivation

The Core Problem: Diffusion Language Models Lag Behind Autoregressive Models in Code Quality

The fundamental question this paper addresses is deceptively straightforward: can diffusion-based training actually improve model capability for code generation, or is it just a different—but worse—way to train language models? This matters because, despite the growing excitement around diffusion language models (DLLMs) as an alternative to autoregressive (AR) generation, the evidence to date has been disappointing on the dimension that matters most in practice: output quality.

The gap is clear and well-documented. As the authors note in the introduction (Section 1), "existing diffusion-based code models still lag behind strong AR baselines in overall accuracy." This isn't a marginal shortfall—the best diffusion code models consistently underperform comparably-sized AR models on standard benchmarks like HumanEval and MBPP. The paper's own baseline comparisons in Table 4 make this concrete: LLaDA-8B-Instruct achieves 49.4% on HumanEval versus 84.8% for Seed-Coder-8B-Instruct; Dream-7B-Instruct reaches 63.4% versus the AR baseline's equivalent. These are not small differences—they represent a fundamental quality deficit that prevents DLLMs from being taken seriously as a production alternative, regardless of their other advantages.

This gap is theoretically puzzling. DLLMs offer several properties that should benefit model capability:

Data augmentation by design. Since diffusion training corrupts and reconstructs sequences under random masking patterns, every clean training example is implicitly augmented into many corrupted-and-denoised views. This is especially valuable for rare, high-quality, or long-tail code samples where the model might otherwise see too few training instances to learn stable representations. The paper explicitly references this motivation (Section 1): "diffusion-style training also enables repeated reuse of the same underlying examples under diverse corruption trajectories, which can extract more information from rare high-quality and long-tail samples and potentially improve overall model capability." In principle, this should give DLLMs an edge on low-resource programming languages and edge cases—exactly what the paper later demonstrates with gains in C# and PHP on MultiPL-E (Table 2).

Any-order modeling. Unlike AR models that learn only left-to-right dependencies, diffusion models learn to predict any token from any surrounding context. For code—where developers routinely infill missing spans, revise earlier segments using later context, and generate independent blocks in parallel—this bidirectional understanding should provide richer representations. The CRUXEval results (Table 8, Output-CoT setting) where Stable-DiffCoder outperforms Seed-Coder (69.0% vs. 67.1%) provide evidence for this intuition: tasks requiring reasoning about structured input-output relationships benefit from non-causal modeling. The authors make this explicit in Section 4.3.2: "the inputs and outputs in CRUXEval are inherently structured rather than strictly following left-to-right causal logic... DLLMs benefit from any-order modeling."

Inference efficiency ceiling. Block diffusion models can generate multiple tokens in parallel rather than one-by-one, offering a higher theoretical ceiling for decoding speed. This is mentioned in the preliminaries (Section 2.2): "Beyond offering a higher ceiling for parallel decoding speed, diffusion-style training also enables repeated reuse of the same underlying examples."

So there is a genuine contradiction: DLLMs should be better at learning from data, yet they consistently produce worse outputs. The paper's central motivation is to resolve this contradiction—to determine whether the quality gap is inherent to diffusion training or an artifact of how existing DLLMs are trained.

Why This Problem Matters

The gap between DLLM potential and DLLM performance matters for several practical and theoretical reasons:

Practical deployment economics. If DLLMs can be trained to match or exceed AR model quality while also offering parallel decoding advantages, they become strictly better—better quality at lower latency. This is a compelling value proposition for production code assistance tools where both response time and accuracy drive user satisfaction. The paper, however, deliberately doesn't lean on inference speed as its primary contribution. As they state in the abstract, the goal is to show that diffusion training can "improve code modeling quality beyond AR training alone," even "before factoring in any potential inference-time speed advantages." This is a stronger, harder claim—they're arguing that diffusion training is a better learning algorithm, not just a faster decoder.

Theoretical understanding of training dynamics. The quality gap between AR and DLLMs reveals our incomplete understanding of what makes language model training effective. AR pretraining is well-studied—we have scaling laws [Hoffmann et al., 2022], a clear understanding of the next-token prediction objective, and decades of empirical tuning. DLLMs operate under a fundamentally different training signal (reconstructing randomly masked spans from surrounding context), and the paper argues that prior work hasn't adequately analyzed which mask patterns provide useful learning signal versus noise. This isn't just an engineering problem—it's a question about the nature of language learning and what contexts are necessary for models to extract stable reasoning rules from data.

Low-resource and structured domains. The data augmentation property of DLLMs is especially valuable for domains where high-quality training data is scarce. In code, programming languages like C#, PHP, Kotlin, and Swift appear far less frequently in web-scale training corpora than Python or JavaScript. If diffusion training can amplify the learning signal from these sparse examples—essentially getting more value from limited data—it offers a path to better multilingual code models without requiring ever-larger training sets. This is precisely the hypothesis the paper validates in its MultiPL-E results (Table 2), where Stable-DiffCoder shows particularly large gains in C# (74.4% vs. 63.9% for Seed-Coder-Base) and PHP (74.4% vs. 63.9%).

Training vs. inference decoupling. In the current paradigm, improving model quality requires scaling pretraining—more data, more parameters, more FLOPs. This is expensive and yields diminishing returns. If the training process itself can be made more efficient at extracting knowledge from existing data—as the DLLM data augmentation hypothesis suggests—then future improvements could come from better training algorithms rather than just bigger models. This would democratize access to high-quality code models for organizations that cannot afford billion-dollar pretraining runs.

Where Prior Approaches Fall Short

The paper identifies specific limitations in prior DLLM work that explain the quality gap, and these limitations cluster around three themes:

Changes to multiple variables simultaneously. Prior work on diffusion code models often introduces confounding changes that make it impossible to isolate the effect of diffusion training itself. The introduction (Section 1) explicitly criticizes this: "prior work often changes data, architecture, and training pipeline simultaneously, leaving open a central question: under a fixed data and compute budget, can the additional training modes introduced by diffusion actually improve model capability?" When a paper reports that their diffusion model underperforms AR baselines, it's unclear whether the fault lies with the diffusion objective or with the other changes they made (different training data, architecture modifications, hyperparameter choices). This makes the existing literature nearly impossible to draw systematic conclusions from—you can't debug what you can't isolate.

Inadequate treatment of training stability and learning efficiency. The paper's analysis in Section 3.1 identifies a deeper problem: not all corruption patterns used in diffusion training are equally useful, and many are actively harmful. This analysis is unique to this paper—prior DLLM work had not rigorously characterized why certain mask patterns provide useful supervision while others do not.

The key insight is formalized through the concept of Token Reasoning Knowledge (Definition 3.1). Given a training context (c) (the unmasked tokens surrounding a masked position), the model needs to learn (p_0(x_i^0 | c))—the conditional distribution of possible completions. The difficulty of learning this depends on the size of the candidate set (K(c))—the number of plausible tokens given the context. The paper identifies three regimes:

Reasoning regime ((K(c)) small): The context strongly constrains the answer, gradients from different examples agree, and the model can learn a stable reasoning rule. Example: "a = 1, b = 2, a + b = [MASK]."
Correlation regime ((K(c)) moderate): The context partially constrains the answer, but different examples with similar context produce different targets, causing gradient cancellation and noisy co-occurrence learning rather than sharp rules.
Noise regime ((K(c)) very large): The context provides almost no information. The model can only memorize idiosyncratic patterns.

The critical problem is that fully bidirectional or large-block diffusion training produces contexts that fall into the correlation and noise regimes. The paper provides a concrete example (Section 3.1.1, Equation 7) that makes this vivid:

Consider the clean sequence:

a = 1, b = 2, a + b = 3; a = 3, b = 4, a + b = 7

Under aggressive masking:

a = 1, b = 2, [MASK1] ... [MASK2] a + b = [MASKn]

The model never sees the clean pair (a = 3, b = 4) in context with 7. From the available evidence (only a = 1, b = 2 is visible), the correct answer should be 3, but the training target is 7—the model is being trained to predict an answer that contradicts the visible evidence. This creates "contradictory and misleading supervision signals." By contrast, when the corruptions respect clean left-side evidence (as in AR training or small-block diffusion where context is mostly contiguous and unmasked):

a = 1, b = 2, a + b = [MASK]

The context strongly constrains the answer, and the model can learn "a stable mapping from evidence to output, which corresponds to an arithmetic reasoning rule rather than mere memorization."

This analysis explains why prior DLLMs fail: they apply random masking across large blocks or entire sequences, generating a high proportion of training contexts in the correlation/noise regimes. These contexts waste compute because (1) they don't teach clear reasoning rules, and (2) they can actively teach wrong associations. This is not just inefficient—it's counterproductive. The model's capacity is being spent on noise when it should be spent on signal.

Training-inference mismatch. A subtler problem the paper identifies is misalignment between the contexts encountered during training ((C_{\text{train}})) and those encountered during inference ((C_{\text{infer}})). Even if a DLLM successfully compresses knowledge under some training contexts, that knowledge is only useful if the same contexts appear at test time. When training uses fully bidirectional masking (seeing tokens on both sides of a masked span) but inference uses left-to-right block-wise decoding (only seeing the left context), there is a distributional mismatch. The model learns rules that depend on bidirectional evidence that isn't available at inference time, degrading performance.

This is precisely what the empirical results in Figure 2 confirm. When training a purely bidirectional DLLM (BiDLLM) and evaluating with small-block decoding (block size 1 or 2), performance is worse than continuing AR training, because the bidirectional training contexts don't match the left-to-right inference contexts. The paper states this explicitly in Section 3.1.2: "For large-block decoding (block size 32), however, scheme (1) underperforms scheme (3), because the training context distribution (C_{\text{train}}) of pure AR does not cover the bidirectional patterns required by large-block decoding."

Training instability during AR→DLLM transitions. The paper identifies a third practical problem: "Previous work has reported that CPT of mask diffusion language models is highly sensitive to the learning rate" (Section 3.2). When switching from AR to DLLM training, changing the attention pattern from causal to bidirectional causes "a structural distribution shift in internal representations." Combined with the loss weighting (w(t)) in the DLLM objective—which can amplify gradients by factors of 10× or more when corruption rates are low—this produces gradient norm spikes that destabilize training (Figure 4, left panel). Without a solution to this instability, the quality of DLLM training is compromised from the start—the model's initial exposed layers to diffusion-style training produce erratic updates that damage the representations learned during AR pretraining, rather than smoothly adapting them.

Lack of a controlled, apples-to-apples comparison. Perhaps most critically, prior work lacked a controlled study where only the training paradigm changed. The paper sets up this exact comparison: reuse the same Seed-Coder architecture, the same training data (1.3T tokens), the same SFT dataset, and compare AR-only training against a pipeline that adds a block diffusion CPT stage. This isolates the effect of diffusion training as cleanly as possible. The paper's claimed contribution—"Stable-DiffCoder overall outperforms its AR counterpart" (abstract)—is meaningful precisely because everything else is held constant. Unlike prior work where it was impossible to tell whether quality differences came from the objective or from confounds, here the comparison is clean.

How This Paper Positions Itself Relative to Existing Work

The paper's positioning can be understood along three axes: methodological, theoretical, and empirical.

Methodological positioning: the controlled study. The paper explicitly rejects the common practice of building entirely new DLLM training pipelines from scratch. Instead, it "reuses the Seed-Coder architecture, data, and training pipeline" (Abstract) and asks: if we take a strong AR pipeline and systematically replace parts of it with diffusion training, can we get better results? This is a more rigorous scientific approach than prior work that "often changes data, architecture, and training pipeline simultaneously." The paper positions itself as providing what the field lacks—a controlled study where the only variable is the training objective and corruption schedule.

This is why the paper can make strong causal claims: "Stable-DiffCoder overall outperforms its AR counterpart" and "diffusion-based training can improve code modeling quality beyond AR training alone" (Abstract). These claims are only credible because the comparison controls for data, architecture, compute, and SFT—the improvements must come from the training procedure itself.

Theoretical positioning: a theory of efficient knowledge compression in DLLMs. The paper doesn't just empirically show that their training recipe works—it develops a theoretical framework for why it works and why prior approaches fail. The Token Reasoning Knowledge analysis (Section 3.1.1) provides a principled explanation for the failure modes of fully bidirectional and large-block diffusion. This distinguishes the paper from purely empirical DLLM work. Prior papers treated all mask patterns as equally valid training data; Stable-DiffCoder argues that some masks teach reasoning rules and others teach noise, and that an effective training pipeline must control which regimes dominate.

The theory leads directly to the paper's recommended training procedure (Section 3.1.2):

First, use AR training to efficiently compress new knowledge. AR provides contexts with small (K(c)) where reasoning rules can be learned efficiently.
Then, perform CPT with small-block diffusion to leverage its data augmentation properties while keeping contexts in the reasoning regime.
Optionally, additional CPT can expand to larger blocks if desired.

This three-stage curriculum is not an arbitrary design choice—it follows directly from the theoretical analysis of which corruption patterns provide useful learning signals. The paper positions its approach as a principled solution to the problem it diagnoses, not merely a bag of tricks that happen to work.

Empirical positioning: state-of-the-art diffusion code model. On the empirical front, the paper positions Stable-DiffCoder as demonstrating that DLLMs can not only match but exceed AR models—a claim that prior diffusion code models could not make. The results tables are designed to make this comparison explicit: within the ~8B parameter class, Stable-DiffCoder achieves:

Best HumanEval(+) and MBPP(+) among diffusion models (Table 1, 4)
Competitive or superior performance against strong AR baselines like Qwen2.5-Coder-7B and Seed-Coder-8B
State-of-the-art on MHPP (42.4%) among ~8B models (Table 5)
Substantial lead on BigCodeBench Hard (31.8%, Table 5)
Top performance on CanItEdit (60.0%, Table 9) for code editing

These are not marginal improvements—they demonstrate that the quality gap between AR and diffusion code models can be closed and even reversed when training is done with appropriate corruption control and stability mechanisms.

Relationship to specific prior work. The paper acknowledges several categories of prior DLLM work:

Foundation DLLM architectures: LLaDA, Dream, SDAR, LLaDA-MoE, WeDLM—these established the basic training recipes but "still lag behind strong AR baselines" (Section 1).
Code-specific DLLMs: DiffuCoder and Dream-Coder attempted to adapt DLLMs to code but didn't close the quality gap. The paper's results show Stable-DiffCoder substantially outperforming both (Table 4: DiffuCoder 72.0% vs. Stable-DiffCoder 86.6% on HumanEval; Dream-Coder 82.9% vs. 86.6%).
Block diffusion methods: Fast-dLLMv2, SDLM, and others explored block-wise corruption but didn't analyze why block size matters or provide the theoretical framework for choosing an appropriate block size.
AR→DLLM adaptation: Gong et al. (2024) studied adaptation from AR to diffusion models and reported training instability—a problem the paper's warmup scheme directly addresses. The paper's corruption warmup (Section 3.2) explicitly avoids the complexity of annealing attention masks, instead warming up only the corruption level (q_{\text{max}}(s)), which is compatible with optimized attention kernels like FlashAttention that assume fixed masks.

The paper's contribution relative to all of these is not a new architecture or a new dataset—it's a training methodology grounded in theoretical analysis that makes diffusion training work as a data augmentation strategy for code models. The core insight is that diffusion's stochastic training is valuable, but only when the corruption process is controlled to keep contexts in the reasoning regime and aligned with inference contexts. Large blocks and fully bidirectional masking violate this constraint; small blocks and AR-to-DLLM transition with warmup satisfy it.

3. Technical Approach

3.1 Reader Orientation

Stable-DiffCoder is a code language model that uses a block diffusion training objective—meaning it learns to reconstruct randomly masked spans of code by conditioning on clean surrounding context—rather than the standard left-to-right next-token prediction used by autoregressive models. The system solves the problem that naive diffusion training produces worse code models than autoregressive training by identifying which corruption patterns actually teach useful reasoning rules and designing a three-stage training curriculum that keeps the model in those useful regimes: first compress knowledge efficiently with autoregressive training (where every masked token has clean left-side evidence), then adapt to multi-token prediction patterns with small-block diffusion (block size 4, keeping contexts in the reasoning regime), and finally—if desired—expand to larger blocks.

3.2 Big-Picture Architecture (Diagram in Words)

The Stable-DiffCoder training pipeline has five major components arranged sequentially:

AR Base Model (Seed-Coder pre-annealing checkpoint) — a standard autoregressive transformer pretrained on general-domain data, serving as the starting point that provides well-compressed code knowledge from clean left-to-right contexts. The checkpoint is taken before the final annealing stage, preserving representation plasticity for the subsequent diffusion adaptation.
AR Continual Pretraining (CPT) Stage — continues training the base model on 1.3T tokens of code data using standard next-token prediction. This efficiently compresses new code-specific knowledge using contexts with small candidate sets ( $K(c)$ small, reasoning regime), where the model can learn stable reasoning rules from clean, contiguous left-side context.
Warmup Module for DLLM Transition — when switching from AR to diffusion training, this module progressively increases the corruption level from near-zero to the full range over $S_{\text{warmup}}$ steps, preventing gradient spikes caused by the attention mask change and the ELBO loss weighting. Only the corruption process is warmed up (not the attention mask), maintaining compatibility with optimized fixed-mask kernels like FlashAttention.
Small-Block Diffusion CPT Stage — continues training on the same 1.3T tokens using a block diffusion objective with block size 4 and a block-wise clipped noise schedule (guaranteeing at least one masked token per block). This introduces multi-token prediction patterns and data augmentation through diverse corruption trajectories while keeping training contexts in the reasoning regime (most evidence is clean and left-side) and aligned with block-wise inference contexts.
Supervised Fine-Tuning (SFT) Stage — fine-tunes the resulting DLLM base model on the Seed-Coder SFT dataset (same data as the AR counterpart), producing the instruction-tuned Stable-DiffCoder-Instruct that is evaluated on code benchmarks. Packing is used with 1–4 randomly appended <eos> tokens between samples to preserve variable-length generation capability.

Information flows sequentially through these stages: AR base checkpoint → AR CPT on code data → corruption warmup → block diffusion CPT with block size 4 and clipped noise → SFT → instruction-tuned model.

3.3 Roadmap for the Deep Dive

First, the theoretical framework of Token Reasoning Knowledge (Section 3.1.1): why some corruption patterns teach reasoning rules and others teach noise, formalized through the candidate set size $K(c)$ and the three regimes (reasoning, correlation, noise). This is the conceptual engine that motivates all subsequent design choices.
Second, the empirical curriculum comparison at 2.5B scale (Section 3.1.2): testing three training pipelines (AR→BiDLLM, ARDLLM→BiDLLM, BiDLLM) across different block sizes to validate the theory and identify that AR-then-small-block-DLLM is optimal.
Third, the corruption warmup procedure (Section 3.2): the specific mechanism for stabilizing the AR→DLLM transition by capping the maximum corruption level and removing the ELBO loss weight during warmup steps, with equations defining the warmup schedule and the modified loss.
Fourth, the block-wise clipped noise schedule (Section 3.3): the sampling rule that guarantees every training step produces at least one supervised token per block by clipping the mask rate to $\min(1, \max(q(t), 1/B))$ , with the expected fraction of empty-block steps derived and the fallback masking rule explained.
Fifth, the training configuration details (Section 4.1): how the full 8B-scale system instantiates these principles—block size, context length, packing strategy, logit-shift removal, and the rationale for reusing the AR token head and loss parametrization.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a methodology and analysis paper whose core idea is that diffusion language model training can serve as effective data augmentation for code models, but only when the corruption process is controlled to keep training contexts in a regime where clean evidence constrains the answer (the "reasoning regime" with small candidate set size $K(c)$ ). Uncontrolled random masking—as in standard fully bidirectional DLLMs—generates many contexts where the visible evidence is insufficient to determine the correct answer, producing noisy gradients that waste compute and can even teach wrong associations.

3.4.1 Token Reasoning Knowledge: A Theory of When Masked Prediction Is Useful

Motivation for a Theoretical Framework

The paper's central practical question is: given that diffusion training applies many different mask patterns to each training example—implicitly creating a large amount of augmented data—why don't existing DLLMs perform better? The answer requires understanding which mask patterns contribute useful learning signal and which are harmful or wasteful. Without a framework for distinguishing these, it's impossible to design a training procedure that keeps the model in useful regimes.

The authors observe two empirical facts that motivate the theory. First, some mask patterns provide nearly useless training signal—"some mask patterns only demonstrate negligible training utility, as shown in recent work" (Section 3.1, citing Kim et al., 2025). Second, some patterns actively mislead: "many masked tokens are trained in contexts where the correct answer is only weakly constrained, and the resulting gradients are dominated by noisy token co-occurrence rather than a sharp reasoning signal." The theory formalizes why these phenomena occur.

Formal Setup: The Concrete Score in RADD Formulation

The paper builds on the RADD (Ou et al., 2025) formulation of absorbing discrete diffusion. Under this framework, the concrete score—which the model must learn to denoise—takes the form:

$s^*(x^t, t)(i, \hat{x}_i) = \alpha(t) p_0(\hat{x}_i \mid x^t_{\text{UM}})$

where $s^*$ is the optimal score function, $x^t$ is the noisy sequence at time $t$ (with some tokens masked), $i$ indexes the token position being predicted, $\hat{x}_i$ is a candidate clean token, $\alpha(t)$ is a time-dependent scaling factor that depends only on $t$ and the forward corruption kernel, $p_0$ is the true data distribution, and $x^t_{\text{UM}}$ denotes all tokens that remain unmasked at time $t$ (the clean context).

What this equation computes: the optimal denoising score at a masked position equals the true conditional probability of each possible token given the visible evidence, scaled by a time-dependent factor. The learning problem therefore reduces to estimating $p_0(\hat{x}_i \mid c)$ for various contexts $c = x^t_{\text{UM}}$ induced by different mask patterns.

Why this form matters: it tells us that the difficulty of learning from any particular mask pattern depends entirely on the structure of $p_0(\cdot \mid c)$ —the conditional distribution over possible answers given that context. The time factor $\alpha(t)$ is factored out, meaning it doesn't affect which knowledge is learnable from a context, only how much the loss weights that context. The critical variable is the context $c$ itself, which is determined by which tokens happen to be unmasked in a given corruption.

Definition of Token Reasoning Knowledge

The paper introduces a formal definition to characterize what the model needs to learn from a given context:

Definition 3.1 (Token Reasoning Knowledge): Let $x$ be a clean training sequence sampled from the real data distribution $p_0$ , and $c$ denote a context extracted from $x$ that is used to predict a single token $x_i$ (similarly for multiple tokens). For autoregressive training, $c_{\text{AR}} = x_{<i}$ , and for diffusion with random masks, $c^t_i = x^t_{\text{UM}}$ . Given a fixed context $c$ , the clean-data conditional $p_0(\cdot \mid c)$ induces a candidate set:

$C(c) = \{v \in \mathcal{V} : p_0(v \mid c) \geq \varepsilon\}, \quad K(c) = |C(c)|, \quad \varepsilon > 0$

We define the token reasoning knowledge contained in a training example with context $c$ as the conditional distribution of the next token restricted to its candidate set, denoted compactly by $p_0(C(c) \mid c)$ .

What this defines: for any context $c$ , there is a set of tokens that have non-negligible probability (above threshold $\varepsilon$ ) of appearing next in the real data. The size of this set, $K(c)$ , measures how uncertain the correct answer is given the evidence. When $K(c)$ is small, the context is highly informative—it strongly constrains what should come next. When $K(c)$ is large, the context is uninformative—many different tokens could plausibly appear, and the model cannot extract a sharp rule from observed (context, token) pairs.

The model's objective is to learn $p_\theta(C(c) \mid c) \approx p_0(C(c) \mid c)$ —to recover this conditional distribution by observing randomly sampled (context, target token) pairs from the training data and adjusting its parameters based on the empirical distribution of these pairs.

Why this definition matters: it transforms the vague question "is this mask pattern useful?" into a concrete, measurable question: "what is $K(c)$ for this context, and how many (context, target) pairs does the model see?" The smaller $K(c)$ and the more frequent the context, the easier it is for the model to learn the mapping. The larger $K(c)$ and the rarer the context, the harder it is—and for very large $K(c)$ , learning degrades into memorization of spurious co-occurrences rather than extraction of generalizable rules.

The Three Regimes of Token Reasoning Knowledge

With $K(c)$ defined, the paper identifies three qualitatively different learning regimes:

Reasoning regime ( $K(c)$ is small and the ground-truth token $x^0_i$ has large probability under $p_0(\cdot \mid c)$ ): The mapping from context to answer is nearly deterministic. Repeated samples with similar contexts provide highly aligned gradients that quickly reinforce a stable reasoning rule. This is the regime where models actually learn—for example, when the context is a = 1, b = 2, a + b = and the target is 3, the context uniquely determines the answer.
Correlation regime ( $K(c)$ is moderate or large, and $x^0_i$ is one of many plausible candidates with comparable probability): The model learns that the target correlates with patterns in the context, but samples with similar contexts often yield different targets, so gradients partially cancel. Updates primarily fit noisy co-occurrence statistics rather than a sharp rule. The model picks up surface-level associations without understanding the underlying logic.
Noise regime ( $K(c)$ is very large, on the order of $|\mathcal{V}|$ , and $p_0(\cdot \mid c)$ is nearly flat): The context contains almost no information about the target. The model can only memorize idiosyncratic (context, target) pairs that happen to co-occur in the training data, providing no generalization benefit.

Why this three-way distinction is the paper's key theoretical contribution: it explains the entire failure mode of existing DLLMs in a single framework. Autoregressive training naturally stays in the reasoning regime because the context $c = x_{<i}$ is always a long, contiguous, clean left-side prefix—"long, contiguous, and clean left-side evidence, which also matches the inherently autoregressive structure of natural language." Fully bidirectional diffusion, by randomly masking tokens from the entire sequence, routinely produces contexts that fall into the correlation or noise regimes, dramatically reducing the efficiency of knowledge compression.

The Concrete Counterexample: When Masking Teaches Wrong Associations

The paper provides a specific example (Equation 6–7 in Section 3.1.1) that makes the harm of correlation/noise-regime contexts tangible:

Clean sequence:

a = 1, b = 2, a + b = 3; a = 3, b = 4, a + b = 7

Under aggressive bidirectional masking:

a = 1, b = 2, [MASK1] ... [MASK2] a + b = [MASKn]

Here, the model is asked to predict [MASKn] (which should be 7 in the clean data) given only the visible context a = 1, b = 2 and whatever remains visible in the middle. The model never sees the clean pair (a = 3, b = 4) in context with 7—the evidence linking a = 3, b = 4 to the answer is masked. From the visible evidence, the arithmetic rule a + b = 3 is what the model should infer (since a = 1 and b = 2 are visible), but the training target is 7.

What happens in training: the supervision signal biases the model toward predicting 7 despite the visible evidence pointing to 3. This creates a "contradictory and misleading supervision signal"—the model is being trained to produce answers that contradict the rules it should be learning. Across many such examples, the model cannot converge to a stable arithmetic reasoning rule because different instances of similar contexts point to different targets.

By contrast, in a context like:

a = 1, b = 2, a + b = [MASK]

The visible evidence exactly determines the answer. This is the reasoning regime, and repeated exposure to such patterns teaches the model that a + b = is followed by the sum of the two most recently assigned variables—a generalizable arithmetic rule.

What this implies for training design: a good training procedure must ensure that most training contexts provide clean, sufficient evidence to determine the target token. This means controlling either (1) how much of the sequence is masked, (2) where masking is applied (preserving contiguous left-side evidence), or (3) both. The three-stage curriculum that Stable-DiffCoder adopts is a direct consequence of this analysis.

Context–Label Pair Frequency and Training Efficiency

Beyond the size of $K(c)$ , the paper identifies a second factor affecting learning efficiency: how many times each (context, target) pair appears during training. DLLMs with random masking generate a wide variety of contexts $c$ by applying different corruption patterns to the same underlying example. While this diversification can be viewed as data augmentation, it also has a cost:

"If the total training data is fixed, this diversification reduces the number of effective pairs corresponding to any specific piece of knowledge, lowering its learning efficiency and forcing the model to relearn the same underlying knowledge many more times than an AR model would require."

In autoregressive training, each token appears exactly once in a fixed left-to-right context ( $c = x_{<i}$ ). The model sees a focused set of (context, target) pairs, each with high frequency. In diffusion training, the same target token may appear in hundreds of different corrupted contexts—some in the reasoning regime, some in the correlation regime, some in the noise regime. The model's training budget is diluted across many context variants, and if a high proportion of those variants are in the correlation/noise regimes, the dilution actively harms rather than helps.

Why this matters for block size choice: small blocks (like the block size 4 used in Stable-DiffCoder) limit how much the context can be disrupted. With block size $B$ , at most $B$ consecutive tokens are corrupted at once, leaving long clean stretches on both sides. This keeps the number of distinct contexts moderate and ensures most of them provide sufficient evidence (small $K(c)$ ). Large blocks or fully bidirectional masking, by contrast, can corrupt most of the evidence, producing many distinct, uninformative contexts that dilute the training signal.

Training–Inference Alignment

The final principle introduced in the framework is that knowledge compressed during training is only useful at inference if the contexts match. Formally, let $C_{\text{train}}$ be the distribution of contexts seen during training and $C_{\text{infer}}$ be the distribution encountered along inference trajectories. The requirement is:

"Good performance requires $C_{\text{train}}$ and $C_{\text{infer}}$ to be as close as possible."

For block diffusion with block size $B$ and left-to-right block-wise decoding at inference using the same block size, the inference contexts closely match the training contexts—the model is asked to generate a block of $B$ tokens given clean left-side context, which is exactly what it was trained to do. For fully bidirectional diffusion trained with arbitrary masking but decoded block-wise left-to-right, there is a mismatch: the model learned to predict tokens using bidirectional evidence that isn't available at inference time (because during left-to-right decoding, only the left context is clean; the right context hasn't been generated yet).

Why this matters: the paper uses this principle to justify both the AR-first stage (where training contexts and inference contexts are identical left-to-right) and the small-block diffusion stage (where the mismatch between training and inference is small because most evidence remains clean on the left side). It also explains why, in the empirical results (Figure 2), "After CPT into a bidirectional DLLM, scheme (1) performs well across all block sizes"—the AR stage already compressed the knowledge efficiently, and the subsequent bidirectional CPT mainly adapts the model to the larger-block context distribution without having to learn the knowledge from scratch.

Summary: Two Principles for Efficient DLLM Knowledge Learning

From this theoretical analysis, the paper distills two concrete design principles (Section 3.1.1, final paragraph):

Clean reasoning evidence: "the model should be exposed to clean and reliable reasoning evidence so that clear reasoning rules can be learned." In practice, this means controlling the corruption process so that most unmasked context provides sufficient information to narrow down the candidate set.
Training–inference alignment and context count control: "the number of distinct sampled contexts $c$ should not grow excessively, and their form should align as closely as possible with the contexts encountered during inference." In practice, this means using small blocks (limiting context diversity) and block-wise corruption (matching block-wise inference contexts).

These two principles directly motivate the paper's recommended curriculum: AR training first (maximally clean evidence, minimal context diversity, perfect alignment), then small-block diffusion (controlled data augmentation, still mostly clean evidence, close alignment), and optionally large-block diffusion (if needed for specific inference regimes, after knowledge is already compressed).

3.4.2 Curriculum Design and Empirical Validation at 2.5B Scale

Experimental Setup for Curriculum Comparison

Before committing to the full 8B-scale training run, the paper conducts controlled experiments at 2.5B parameter scale to test the predictions of the Token Reasoning Knowledge framework. The setup is as follows (Section 3.1.2):

Starting point: a 2.5B autoregressive model trained on general-domain data (not code-specific).
New data: code data used as CPT (continual pretraining) data—representing new knowledge the model must learn.
Fixed compute budget: all training curricula are compared under the same total compute, making the comparison fair.
Evaluation: models are evaluated using block-wise decoding in the style of LLaDA (Nie et al., 2025), at different block sizes (1, 2, and 32), and averaged across multiple code benchmarks.

Three curricula are compared:

AR → BiDLLM: Continue pure AR training on the new data, then perform CPT to convert the model into a bidirectional DLLM. This spends the entire budget with clean left-to-right evidence before adapting to bidirectional contexts.
ARDLLM → BiDLLM: Continue training with a causal-structured DLLM (AR-style diffusion, where the diffusion corruption respects causal masking), then CPT to a bidirectional DLLM. This introduces diffusion-style training early but preserves left-to-right structure in the corruption.
BiDLLM: Directly CPT the AR checkpoint into a bidirectional DLLM and train all new knowledge in that regime. This immediately introduces fully bidirectional masking with no clean left-to-right phase for the new data.

Results and Analysis (Figure 2)

The results in Figure 2 are reported as the average performance across multiple code benchmarks, plotted against training steps. The key findings, broken down by block size at evaluation:

Block size 1 (top panel, left-to-right single-token decoding):

Before any CPT to BiDLLM (solid lines), the ranking is (1) AR > (2) ARDLLM > (3) BiDLLM, with AR achieving the highest score and BiDLLM the lowest. This validates the theory: for small-block decoding, the inference contexts ( $C_{\text{infer}}$ ) are almost identical to pure AR training contexts ( $C_{\text{train}}$ ), so AR training provides the best training–inference alignment and the cleanest evidence.
After CPT to BiDLLM (dotted lines extending from the solid curves), all curricula lose some performance for block-1 decoding. The paper notes: "both pure AR and AR→BiDLLM lose a similar amount of performance between 0 and 100k CPT steps at block size 1." This loss is attributed to the training–inference misalignment introduced by bidirectional attention—even though the AR phase efficiently compressed knowledge, the subsequent BiDLLM CPT shifts the model's internal representations to contexts that don't perfectly match single-token left-to-right decoding.
Crucially, scheme (1) still achieves the best final performance after CPT, because the AR phase compressed the knowledge so efficiently that even after the CPT-induced degradation, it remains ahead.

Block size 2 (middle panel, small-block decoding):

The pattern is similar but the gaps are smaller. AR training still provides the best performance before CPT, because block-2 inference contexts are close to AR contexts—only two tokens are generated at once, with a long clean left-side prefix.
After CPT to BiDLLM, scheme (1) maintains its advantage.

Block size 32 (bottom panel, large-block decoding):

Before CPT, scheme (1) underperforms scheme (3). This is the critical inversion. The paper explains: "the training context distribution $C_{\text{train}}$ of pure AR does not cover the bidirectional patterns required by large-block decoding. Formally, with causal attention, tokens later in the block cannot influence earlier ones, so when decoding an entire block at once, the generation process becomes less stable." Pure AR training never exposes the model to the task of generating 32 tokens simultaneously with intra-block dependencies—the training contexts are all left-to-right single tokens.
After CPT, scheme (1) performs well across all block sizes. The AR phase already compressed the knowledge in well-aligned small-block contexts, and the BiDLLM CPT primarily adapts the model to the larger-block context distribution—it's learning how to use the knowledge in a bidirectional setting, not learning the knowledge itself.

Scheme (2) analysis: ARDLLM (causal-structured diffusion) achieves strong performance after CPT to BiDLLM. In particular, after conversion, it outperforms scheme (3) under block-32 decoding, even though it saw a similar amount (or less) clean evidence during training. The paper hypothesizes that "the training context distribution $C_{\text{train}}$ for AR-style diffusion in scheme (2) more closely matches the prompt–response pattern commonly used at inference, where most reasoning evidence lies on the left." This is an important nuance: even within diffusion training, preserving left-to-right structure in the corruption process (ARDLLM) provides better training–inference alignment than fully bidirectional corruption (BiDLLM).

Derivation of the Recommended Training Procedure

Based on both the theoretical analysis (Section 3.1.1) and the empirical results (Section 3.1.2), the paper recommends a three-stage training procedure (Section 3.1.2, final enumerated list, and Figure 3):

For new knowledge, first use AR training to efficiently compress it. AR provides clean reasoning evidence, minimal context diversity, and perfect training–inference alignment for small-block decoding. This is the most efficient way to convert raw training data into model weights.
Then perform CPT with a small-block diffusion objective. This leverages the data-augmentation properties of diffusion—exposing the model to multiple corrupted-and-denoised views of each example—to further improve model quality, while keeping the corruption controlled (small blocks) so that most contexts remain in the reasoning regime and aligned with inference contexts.
If one wishes to explore larger block diffusion, additional CPT can be applied starting from the model obtained in step (2). By the time large-block CPT begins, the model has already compressed the underlying knowledge through AR and small-block training. The large-block stage only needs to adapt the model to use that knowledge in contexts with more intra-block dependencies, rather than learning the knowledge from scratch in poor contexts.

Why this ordering rather than alternatives: the alternative—starting with large-block or bidirectional diffusion (scheme 3 in the experiment)—forces the model to learn new knowledge from contexts that are often in the correlation or noise regimes, wasting compute and potentially teaching wrong associations. The alternative—skipping the diffusion stage entirely—leaves data augmentation benefits unrealized and doesn't adapt the model to multi-token prediction patterns. The recommended procedure captures the benefits of both AR (efficient compression) and diffusion (data augmentation) while avoiding the pitfalls of diffusion (noisy contexts for new knowledge).

In the final Stable-DiffCoder system, only stages (1) and (2) are used. The paper adopts the AR CPT stage followed by block diffusion CPT with block size 4, as illustrated by the solid path leading to the DLLM Base Model in Figure 3. Stage (3) with larger blocks is shown as a dashed alternative pipeline, not part of the main results. This means Stable-DiffCoder is optimized for small-block decoding (block size 4) and does not explore the larger-block regime—a deliberate choice given the paper's focus on demonstrating that diffusion can improve model quality, not on maximizing inference speed through larger blocks.

3.4.3 Corruption Warmup for Stable AR→DLLM Transition

The Instability Problem

When switching from AR training (causal attention, next-token prediction) to DLLM training (bidirectional attention, masked token reconstruction), naive CPT exhibits severe training instability. The paper identifies three root causes (Section 3.2):

Attention mask distribution shift: "the change of attention mask induces a structural distribution shift in internal representations." In AR mode, each token attends only to previous tokens (lower-triangular mask). In DLLM mode, every token attends to every other token (full mask). This changes the information available to each position dramatically—tokens that previously only saw left context suddenly see the full sequence. The model's internal representations, tuned for causal attention, receive gradients that are inconsistent with the statistics they were optimized for, causing large initial updates.
Higher task difficulty: "when the corruption process masks a large fraction of tokens, compared to AR next-token prediction." In AR training, the model always predicts the next token given a completely clean prefix. In DLLM training with high corruption rates, the model must reconstruct tokens from heavily corrupted context—a much harder task that produces larger losses and gradients.
ELBO loss weighting amplifies gradients: The standard DLLM objective (Equation 4 in the paper) includes a weighting term $w(t)$ that depends on the corruption level. Under a linear noise schedule where the masking probability is $1 - t$ and $t \sim U[0, 1]$ , the weight can be $w(t) = 1/t$ . When $t$ is small (low masking ratio, e.g., 10% masked), $w(t)$ becomes large (e.g., approximately 10). Concretely:

"under a linear noise schedule, masking 10% of tokens yields $w(t) \approx 10$ . This effectively acts as loss scaling that amplifies gradient norms, making training less stable."

The low-masking-ratio regime is especially problematic because it's where the loss weighting is largest, yet it's also where the model is closest to its AR configuration—the sudden application of a 10× loss multiplier to the small fraction of masked tokens creates gradient spikes.

The Warmup Procedure Design

Rather than annealing the attention mask from causal to bidirectional (as in prior work—Gong et al., 2024—which the paper notes is "inconvenient for highly optimized kernels such as FlashAttention that assume a fixed mask"), the paper warms up only the corruption process. The attention mask switches from causal to bidirectional immediately at the start of DLLM CPT, but the fraction of tokens that get corrupted is initially capped at a very low value and progressively increased.

Formally, in standard DLLM training, the corruption level $t$ is sampled uniformly from $[0, 1]$ , and the masking ratio ranges from 0 to 1. During warmup, the maximum corruption level is capped:

$q_{\text{max}}(s) = q_{\text{init}} + (1 - q_{\text{init}}) \cdot \frac{s}{S_{\text{warmup}}}$

where $s = 0, \ldots, S_{\text{warmup}}$ is the warmup step index, $q_{\text{init}}$ is the initial maximum corruption level (set very small, e.g., $10^{-3}$ ), and $S_{\text{warmup}}$ is the total number of warmup steps. At each step during warmup, $t$ is sampled uniformly from $[0, q_{\text{max}}(s)]$ rather than $[0, 1]$ .

What this does: at the beginning of warmup ( $s = 0$ ), $q_{\text{max}} = q_{\text{init}} \approx 10^{-3}$ , meaning at most 0.1% of tokens are masked. The model sees contexts that are nearly entirely clean—almost identical to AR training—but now with bidirectional attention. Over the course of warmup, $q_{\text{max}}$ linearly increases to 1, progressively introducing harder reconstruction tasks. By the end of warmup, the model is exposed to the full range of corruption levels and is ready for standard DLLM training.

The Modified Warmup Loss

To further suppress gradient spikes, the ELBO weighting term $w(t)$ is removed during the warmup phase. The warmup loss is:

$L^{\text{warmup}}_{\text{DLLM}}(\theta) = -\mathbb{E}_{x^0 \sim p_{\text{data}}, t \sim U(0, q_{\text{max}}), x^t \sim q(x^t|x^0)} \left[ \sum_{i=1}^{N} \mathbb{1}[x^t_i = \text{MASK}] \log p_\theta(x^0_i \mid x^t_{1:N}) \right]$

where $x^0 \sim p_{\text{data}}$ is a clean training sequence drawn from the data distribution, $t \sim U(0, q_{\text{max}})$ samples the corruption level from a uniform distribution capped at $q_{\text{max}}$ , $x^t \sim q(x^t|x^0)$ applies the forward corruption process (randomly masking tokens based on $t$ ), $\mathbb{1}[x^t_i = \text{MASK}]$ selects only positions that are actually masked, and $\log p_\theta(x^0_i \mid x^t_{1:N})$ is the log-probability the model assigns to the correct token at each masked position.

What this loss computes: for every masked position in the corrupted sequence, the model predicts the original token and is penalized by the negative log-likelihood. Unlike the standard DLLM objective (Equation 4), there is no $w(t)$ weighting—every masked position contributes equally to the loss, regardless of the corruption level. The summation over $i$ and expectation over $x^0$ , $t$ , and $x^t$ averages over all training examples, corruption levels (up to $q_{\text{max}}$ ), and masked positions.

Why remove $w(t)$ during warmup: the weighting term $w(t) = 1/t$ amplifies the loss for low corruption levels, precisely where gradient norms would already be large due to the task being easier (cleaner context → higher-confidence predictions → larger gradients under cross-entropy). Removing this amplification prevents the gradient spikes that occur at the AR→DLLM boundary. After warmup, standard DLLM training resumes with $t \sim U(0, 1)$ and $w(t)$ restored.

Post-Warmup Behavior

After warmup completes (step $s = S_{\text{warmup}}$ ), the training reverts to the standard DLLM objective from Equation 4:

$L_{\text{DLLM}}(\theta) = -\mathbb{E}_{x^0 \sim p_{\text{data}}, t \sim U(0,1), x^t \sim q(x^t|x^0)} \left[ w(t) \sum_{i=1}^{N} \mathbb{1}[x^t_i = \text{MASK}] \log p_\theta(x^0_i \mid x^t_{1:N}) \right]$

with $t \sim U(0, 1)$ (the full range of corruption levels) and the weighting $w(t)$ reinstated.

Empirical Validation of Warmup Effectiveness (Figure 4)

Figure 4 in the paper compares training stability with and without warmup. The left panel ("BiDLLM Without Warmup") shows the training loss and gradient norm over the first 5000 training steps:

The training loss (top subplot, log scale) starts very high (above 4) and spikes upward sharply at the AR→DLLM transition before gradually decreasing.
The gradient norm (bottom subplot, log scale) exhibits a massive spike at the transition—reaching values on the order of $10^5$ —indicating extremely large parameter updates that can destabilize or destroy the representations learned during AR pretraining.

The right panel ("BiDLLM With Warmup" and "BlockDLLM With Warmup") shows the same metrics with warmup applied:

The DLLM training loss starts around 0.3–0.6 (much lower than without warmup) and follows a characteristic "√-shaped" curve: first decreasing as the model handles easy low-mask-ratio tasks, then gradually increasing as the mask ratio grows and the task becomes harder, and finally decreasing again as the model adapts to the full DLLM regime. The comparison to the AR training loss (shown as a reference line) shows that the DLLM loss with warmup is well-behaved and comparable in magnitude to AR training.
The DLLM gradient norm is dramatically reduced—starting around 1.0 (similar to AR training) and rising to about 10.0 at the end of warmup, compared to the $10^5$ spike without warmup. The gradient norm remains within a factor of 10 of the AR gradient norm throughout.

The block diffusion variant (BlockDLLM, with no logit shift as used in the final model) shows similarly stable behavior with warmup: the loss follows the same √-shaped curve, and the gradient norm is well-controlled.

Why this matters: the warmup procedure enables the model to smoothly transition from AR to DLLM training without the destructive gradient spikes that would otherwise occur. This preserves the knowledge compressed during the AR phase and provides a stable foundation for the subsequent block diffusion CPT stage. Without warmup, the initial DLLM training steps would produce erratic updates that could partially overwrite the AR-learned representations, reducing the efficiency of the entire pipeline.

Design Choice: Corruption Warmup vs. Attention Mask Annealing

The paper explicitly contrasts its approach with prior work: "Rather than annealing the attention mask (which is inconvenient for highly optimized kernels such as FlashAttention that assume a fixed mask), we warm up only the corruption process." This is a practical engineering choice that prioritizes compatibility with existing optimized infrastructure. Attention mask annealing would require gradually transitioning from a causal lower-triangular mask to a full bidirectional mask over many steps, which:

Requires custom attention implementations that support intermediate (partially causal, partially bidirectional) masks, which FlashAttention does not natively support.
Increases the complexity of the training code and potentially reduces throughput.
May still cause instability because the representation shift from causal to bidirectional is abrupt at the token level even if the mask changes gradually.

Corruption warmup avoids all of these issues: the attention mask is bidirectional from step one, but the task difficulty is initially negligible (almost no tokens masked), allowing the model to adapt its representations to bidirectional attention while solving trivially easy reconstruction problems. As the corruption level increases, the model progressively handles harder tasks within the same attention pattern, making the adaptation gradual at the task level rather than the architecture level.

3.4.4 Block-wise Clipped Noise Schedule

The Empty-Block Problem

Standard diffusion language model training applies corruption uniformly across the entire sequence. When this is adapted to block diffusion—where only a single contiguous block of tokens is corrupted at each step—a new problem arises: the probability of corrupting zero tokens in the block is non-negligible, especially for small block sizes.

Formally (Section 3.3), with block length $B$ and a global continuous-time schedule $t \in [0, 1] \mapsto q(t)$ (where $q(t)$ is the corruption/mask rate), the expected number of masked tokens in the block is:

$\mathbb{E}[m \mid t] = B \cdot q(t)$

The probability of observing exactly zero masked tokens is:

$\Pr[m = 0 \mid t] = (1 - q(t))^B$

Under a standard global linear schedule $q(t) = 1 - t$ with $t \sim \text{Unif}[0, 1]$ , the expected fraction of training steps where $m = 0$ is:

$\mathbb{E}_t[(1 - t)^B] = \int_0^1 (1 - t)^B dt = \frac{1}{B + 1}$

What this computes: the probability that a randomly sampled corruption level $t$ produces a block with no masked tokens at all, integrated over the uniform distribution of $t$ . For typical block sizes:

$B = 2$ : $1/3 \approx 33\%$ of steps produce an empty block
$B = 4$ : $1/5 = 20\%$ of steps produce an empty block (the block size used in Stable-DiffCoder)
$B = 8$ : $1/9 \approx 11\%$ of steps produce an empty block

Why this is a problem: a training step with $m = 0$ produces zero supervised tokens—the model sees a fully clean sequence and has nothing to predict. This wastes compute (forward and backward passes that contribute no learning signal) and, more subtly, biases the model's training toward higher corruption levels (since low corruption levels are disproportionately likely to produce empty blocks and be wasted, the effective training distribution shifts toward higher corruption, which may not be optimal). For block size 4, losing 20% of training steps is a significant efficiency cost.

The Clipping Rule

Instead of redesigning the global schedule, the paper adopts a simple block-aware sampling rule. After sampling the global corruption level $t$ , a block-specific mask rate is computed by clipping:

$q_{\text{blk}}(t) = \min\left(1, \max(q(t), 1/B)\right)$

What this does: the block-specific mask rate $q_{\text{blk}}(t)$ is guaranteed to be at least $1/B$ . If the global schedule would produce a mask rate below $1/B$ , it is raised to $1/B$ . If it would produce a mask rate above 1 (which shouldn't happen for well-designed schedules), it is capped at 1.

Why this threshold: with mask rate exactly $1/B$ , the expected number of masked tokens in the block is:

$\mathbb{E}[m \mid q = 1/B] = B \cdot (1/B) = 1$

So $q_{\text{blk}}(t) \geq 1/B$ guarantees $\mathbb{E}[m \mid t] \geq 1$ —in expectation, at least one token per block is masked. This eliminates the empty-block problem: even at low corruption levels, the block always contributes to the loss.

An additional benefit: preventing the loss weight from exploding. The standard DLLM loss weight is $w(t) = 1/q(t)$ . When $q(t)$ is very small, $w(t)$ becomes very large, amplifying gradients. The clipping $q_{\text{blk}}(t) \geq 1/B$ caps the loss weight at $w(t) \leq B$ (for block size 4, $w(t) \leq 4$ ), providing implicit gradient clipping that contributes to training stability.

The Fallback Rule

Even with $q_{\text{blk}}(t) \geq 1/B$ , there is still a non-zero probability that after independent Bernoulli sampling with probability $q_{\text{blk}}(t)$, no token in the block is actually masked. This happens by chance when all $B$ ` independent Bernoulli trials fail. To completely eliminate empty-block steps, the paper applies a fallback:

"if, after sampling, no token in $\mathcal{B}$ is masked (i.e., $m = 0$ ), we uniformly sample one position in $\mathcal{B}$ and force it to be masked."

What this guarantees: every training step contains at least one supervised token inside the block, while still preserving the overall shape of the original schedule $q(t)$ . The forced masking at the lowest corruption levels is a minor perturbation that prevents the worst-case scenario of wasted compute without significantly distorting the training distribution.

Integration with the Full DLLM Objective

The block-wise clipped noise schedule operates within the standard DLLM training framework. At each training step:

Sample a block of $B$ consecutive tokens from the sequence.
Sample a global corruption level $t \sim U(0, 1)$ .
Compute $q_{\text{blk}}(t) = \min(1, \max(q(t), 1/B))$ .
Independently mask each token in the block with probability $q_{\text{blk}}(t)$ , while all tokens outside the block remain clean and provide uncontaminated context.
If no token in the block was masked, force-mask one uniformly random position.
Compute the DLLM loss (Equation 4) over the masked positions.

The key property is that tokens outside the block are always clean—the masking is local to the block. This preserves long-range clean context on both sides of the corrupted span, ensuring that the reconstruction task is well-constrained (small $K(c)$ ) and the training context matches what the model sees during block-wise decoding at inference (clean left context, block to generate, clean right context if available).

Why Not Just Use a Higher Global Mask Rate?

An alternative approach would be to simply use a higher baseline mask rate (e.g., never sample $t$ below some threshold) so that empty blocks are extremely rare. The paper's clipping approach is preferable because:

Preserves the full schedule shape: the clipping only affects the lowest corruption levels (below $1/B$ ) while leaving higher levels unchanged. This maintains the intended distribution of task difficulties during training, which may be important for learning.
Prevents over-aggressive masking: using a uniformly higher mask rate would increase the proportion of high-corruption steps, which fall into the correlation/noise regimes identified in Section 3.1.1. The clipping approach adds a minimal floor to the mask rate without shifting the entire distribution upward.
Is a minimal intervention: the clipping + fallback rule is the smallest possible change that eliminates the empty-block problem while keeping the training distribution as close as possible to the intended schedule.

Connection to the Token Reasoning Knowledge Framework

The block-wise clipped noise schedule is a direct implementation of the first design principle from Section 3.1.1: "the model should be exposed to clean and reliable reasoning evidence." By keeping only a small block corrupted and guaranteeing at least one masked token per block, the schedule ensures that:

Most of the context remains clean (all tokens outside the block).
The reconstruction target is always present (no empty blocks).
The evidence available to predict each masked token is substantial—only $B$ tokens at most are corrupted, and the model can see clean tokens on both sides of the block.

This keeps the training contexts firmly in the reasoning regime (small $K(c)$ ) while still providing the data augmentation benefit of diverse corruption patterns within the block.

3.4.5 Full Training Configuration for Stable-DiffCoder-8B

Training Pipeline Instantiation

The final Stable-DiffCoder-8B system instantiates the three-stage curriculum (AR CPT → warmup → small-block DLLM CPT) with specific configuration choices detailed in Section 4.1:

Starting checkpoint: the pre-annealing checkpoint of Seed-Coder—an autoregressive model already trained on general-domain data, but before the final annealing stage that would reduce learning rate and potentially limit plasticity. This preserves the model's ability to adapt to new training objectives and data distributions.

AR CPT stage: continued pretraining on the Seed-Coder code data. The multi-stage CPT data from Seed-Coder is "compressed ... to a total of 1.3T tokens via subsampling." Context length is 8192 tokens with packed sequences (multiple training examples concatenated into a single sequence to maximize GPU utilization by avoiding padding waste).

DLLM CPT stage: block diffusion with block size 4 for the full 1.3T tokens. The corruption process follows the block-wise clipped noise schedule (Section 3.3) and the warmup procedure (Section 3.2) is applied at the AR→DLLM transition.

SFT stage: "fully reuses the original Seed-Coder SFT dataset." The same packing strategy is used during SFT, but with a modification: "after each sample we randomly append 1–4 <eos> tokens, enabling the model to preserve the ability to generate variable-length outputs within each packed block." Without this, a model trained only on packed sequences might learn to always output exactly up to the next boundary, losing the ability to stop generation at appropriate points.

Architecture Choices for DLLM Compatibility

Several architectural decisions are made to maintain consistency between AR and DLLM training while ensuring compatibility with the diffusion paradigm:

Reused AR token head and logit-shift parametrization: the paper "reuse[s] the AR token head and logit-shift parametrization, and only change[s] the attention pattern from a causal lower-triangular mask to a bidirectional, full attention mask." This minimizes the parameter changes between AR and DLLM modes—the same output layer, same token embeddings, same feedforward networks are used; only the attention mask changes. This is important because it means the knowledge stored in the model's weights doesn't need to be remapped to a different parametrization; the representations can transfer directly.

No-logit-shift formulation: the final model uses a "no-logit-shift design similar to that used in LLaDA and SDAR." The rationale is that "the no-logit-shift formulation is more consistent with the absorbing diffusion paradigm, where each masked position predicts itself and the input and prediction targets are aligned at both the token and sentence levels." In standard AR models, the logit shift (also called "logit bias" or "token shift") adjusts the output logits based on position. Removing it makes the prediction task cleaner: each masked position independently predicts its original token, without position-dependent bias terms that might interfere with the bidirectional context.

Packing with mutual visibility: "to prevent repeated compilation of flex attention, we make the attention between packed samples mutually visible, so that the same attention mask and operator can be reused in every forward pass." In standard packing, attention would be masked between different samples (so tokens from one sample can't attend to tokens from another). Making packed samples mutually visible simplifies the attention implementation—a single fixed mask works for all forward passes—at the cost of cross-sample attention, which the paper treats as acceptable noise given the benefit of compilation efficiency.

Block Size Choice: 4

The choice of block size 4 is motivated by the theoretical analysis and the empirical results at 2.5B scale. The Token Reasoning Knowledge framework (Section 3.1.1) shows that smaller blocks:

Keep more context clean (only 4 tokens corrupted at a time, with the rest of the 8192-token sequence providing clean evidence).
Limit the number of distinct contexts (the corruption patterns within a 4-token block are far fewer than in a 32-token block), preventing dilution of the (context, target) pair frequency.
Provide better training–inference alignment when block-wise decoding uses the same block size.

The empirical results in Figure 2 confirm that block size 1 and 2 perform better with AR training before DLLM CPT, while block size 32 benefits from bidirectional CPT after the AR phase. Block size 4 sits at an intermediate point—small enough to be in the reasoning regime for most contexts, large enough to provide meaningful multi-token prediction patterns and data augmentation through diverse intra-block corruption patterns.

The paper does not provide an ablation over block sizes at 8B scale, so block size 4 represents a fixed design choice based on the 2.5B-scale analysis. Whether block size 2, 8, or another value would be optimal at 8B is not tested—this is a limitation acknowledged implicitly by the paper's recommendation that larger blocks can be explored in optional additional CPT stages.

Summary: The Complete Training Flow

Start with Seed-Coder pre-annealing checkpoint (AR, general-domain + code pretrained).
AR CPT on 1.3T code tokens: compress new code knowledge efficiently with clean left-to-right contexts. Context length 8192, packed sequences, standard next-token prediction loss.
Corruption warmup: switch attention to bidirectional. Cap maximum corruption level at $q_{\text{max}}$ , linearly increasing from $10^{-3}$ to 1 over $S_{\text{warmup}}$ steps. Remove $w(t)$ loss weight during warmup. Use modified warmup loss (Equation 10).
Block diffusion CPT on 1.3T code tokens: block size 4, block-wise clipped noise schedule (mask rate $\geq 1/B$ with fallback), standard DLLM loss with $w(t)$ reinstated. No logit shift.
SFT on Seed-Coder SFT dataset: same packing, 1–4 random <eos> tokens per sample.
Result: Stable-DiffCoder-8B-Base (or -Instruct after SFT), evaluated on code benchmarks.

4. Key Insights and Innovations

Innovation 1: Token Reasoning Knowledge — A Diagnostic Framework for Why Most Masked Diffusion Training Is Wasteful

What is distinctive at the idea level. Before this paper, the DLLM community treated all corruption patterns as equally valid training data. The dominant assumption—implicit in every fully bidirectional or large-block diffusion model (LLaDA, Dream, DiffuCoder, Dream-Coder)—was that random masking provides useful data augmentation by exposing the model to diverse denoising tasks. More masking variety was assumed to be better, and when DLLMs underperformed AR baselines, the diagnosis was typically architectural (e.g., "we need better denoising parametrization") or scale-related (e.g., "we need more data"), not structural.

Stable-DiffCoder replaces this assumption with a diagnostic taxonomy. The concept of Token Reasoning Knowledge (Definition 3.1) partitions all (context, target) pairs into three regimes based on the size of the candidate set K(c)—how many tokens are plausible given the visible evidence:

Reasoning regime (K small): the context uniquely constrains the answer. Gradients align across examples. The model learns a stable reasoning rule.
Correlation regime (K moderate): the context partially constrains the answer. Gradients partially cancel. The model learns noisy surface co-occurrence, not logic.
Noise regime (K large): the context carries almost no information. Training reduces to memorizing spurious (context, target) pairs.

Why this reframes the problem. This taxonomy converts the vague intuition "some mask patterns are less useful" into a measurable criterion: for any corruption schedule, one can estimate how many training steps fall into each regime and predict learning efficiency. The paper does not supply a quantitative distribution measurement—that remains future work—but the conceptual move from "all masks are training data" to "masks in the reasoning regime teach rules; masks in the noise regime teach noise" is a fundamental shift in how to think about DLLM training.

The concrete counterexample in Equation 6–7 makes this shift tangible. Under heavy bidirectional masking, the model is trained to predict 7 from context that only shows a=1, b=2 (where the arithmetic rule says 3). This is not just inefficient training—it is actively harmful, creating contradictory supervision that prevents convergence to a stable reasoning rule. No prior DLLM paper identified this as the mechanism for quality degradation; prior work attributed the gap to insufficient optimization or architecture mismatch.

Comparison to prior work. The field's prior understanding of DLLM training quality came from two sources: (a) empirical scaling studies showing that DLLMs learn more slowly than AR models per token (Ni et al., 2025; von Rütte et al., 2025; Sun et al., 2025), and (b) the observation that some token orderings in masking are more useful than others (Kim et al., 2025). The Token Reasoning Knowledge framework subsumes and explains both findings: slower learning per token occurs because many tokens are trained in correlation/noise regimes where gradients are weak or contradictory; some mask orderings are better because they preserve more clean context, keeping K(c) small. The paper provides a unified causal mechanism where prior work provided only empirical phenomenology.

Is this fundamental or incremental? This is a fundamental conceptual contribution—a new diagnostic lens that reinterprets existing DLLM failure modes and generates testable predictions about what training procedures should work. It is analogous to the role that the bias-variance decomposition plays in classical ML: not a new algorithm, but a framework for understanding why some training choices work and others fail, which in turn guides algorithm design. The three-stage curriculum that follows (AR → small-block DLLM → optional large-block DLLM) is a direct consequence of this framework, not an independent empirical discovery—it is the application of the principle "train new knowledge only in the reasoning regime."

Evidence anchor. The framework's predictive power is validated in Figure 2. The framework predicts that for block-1 decoding, AR training (which stays in the reasoning regime with clean left-side context) should outperform BiDLLM training (which introduces correlation/noise regime contexts). The data confirms this: AR > ARDLLM > BiDLLM for block sizes 1 and 2 before CPT. The framework also predicts that after knowledge is compressed in the reasoning regime (via AR CPT), subsequent BiDLLM training can adapt to larger-block contexts without damaging the underlying knowledge—and indeed, after CPT, scheme (1) performs well across all block sizes, including block 32 where pure AR initially failed.

Innovation 2: Controlled Transition as an Alternative to Full Retraining — The AR→DLLM Warmup

What is distinctive at the idea level. The conventional wisdom in DLLM development—codified in major models from LLaDA through Dream to Gemini Diffusion—treated diffusion training and autoregressive training as separate paradigms requiring separate models trained from scratch. If you wanted a diffusion model, you pretrained a diffusion model. If you started from an AR checkpoint, you faced destructive gradient spikes at the transition (documented in Gong et al., 2024) that degraded the representations learned during AR pretraining.

Stable-DiffCoder challenges this separation with a deceptively simple insight: the transition from AR to DLLM is not an architectural problem requiring attention mask annealing—it is a task curriculum problem. By keeping the attention mask bidirectional from step one but capping the corruption level at near-zero initially and linearly increasing it, the model faces trivially easy reconstruction tasks (0.1% of tokens masked) in a new attention pattern, then progressively harder tasks in the same pattern. The warmup converts what was previously treated as an architecture change into a curriculum over task difficulty.

Why this reframes the problem. Prior work on AR→DLLM adaptation (Gong et al., 2024) attempted to solve the transition problem by annealing the attention mask—gradually transitioning from causal to bidirectional attention. This is conceptually natural (smoothly change the architecture) but practically problematic: it requires custom attention implementations incompatible with optimized kernels like FlashAttention, increases code complexity, and may still cause instability because the representation shift at the per-token level is abrupt regardless of how smoothly the mask changes globally.

The Stable-DiffCoder warmup reframes the problem entirely: the architecture can change discontinuously if the task is initially trivial. By starting with q_max ≈ 10⁻³ (at most 0.1% of tokens masked), the model sees sequences that are almost entirely clean—functionally identical to AR training, but now with bidirectional attention. The task is so easy that the model can adapt its internal representations to the new attention pattern with minimal gradient disruption. Only after this adaptation is complete does the task difficulty increase, at which point the representations are already tuned for bidirectional attention and can handle harder reconstruction tasks without spikes.

Comparison to prior work. Gong et al. (2024) showed that CPT of mask diffusion language models is "highly sensitive to the learning rate" and proposed attention mask annealing as a solution. Stable-DiffCoder's corruption warmup achieves the same goal—stable transition—but through an orthogonal mechanism that is simpler to implement (compatible with fixed-mask kernels), grounded in the Token Reasoning Knowledge framework (it's a special case of the principle "start with easy, clean-evidence tasks in the reasoning regime"), and empirically validated to work for both fully bidirectional and block-diffusion DLLMs (Figure 4, right panel).

Is this fundamental or incremental? This is a methodological innovation with fundamental implications. It is not merely a better way to do AR→DLLM adaptation—it establishes the principle that task difficulty curriculum can substitute for architectural continuity in model transitions. This principle has implications beyond DLLMs: any transition between training paradigms (e.g., masked language modeling to causal language modeling, or text pretraining to code fine-tuning) could potentially be stabilized by starting with trivially easy instances of the new task rather than by smoothly interpolating the architecture. The finding that the warmup works with "only the corruption process" changed—attention mask, token head, logit parametrization all switch abruptly—is striking evidence for this principle.

The practical significance is equally important: the warmup removes what was previously considered a fundamental barrier to reusing AR pretrained models for diffusion training. Without it, the AR→DLLM transition destroys pretrained knowledge (Figure 4, left panel: gradient norms spike to ~10⁵, loss spikes above 4). With it, the transition is smooth and the final DLLM loss is comparable to AR CPT loss (Figure 4, right panel). This means the entire AR pretraining ecosystem—trillions of tokens of pretrained models, optimized training pipelines, curated datasets—can be leveraged for DLLM training rather than requiring expensive from-scratch runs.

Evidence anchor. Figure 4 provides the cleanest evidence. Without warmup, the gradient norm at the AR→DLLM boundary spikes by three orders of magnitude (from ~10 to ~10⁵). With warmup, the gradient norm stays within a factor of 10 of AR training throughout, and the DLLM loss follows a characteristic √-shaped curve (first decreasing as the model handles easy low-mask tasks, then gradually rising as difficulty increases, then decreasing again as adaptation completes). This curve is the visual signature of successful curriculum learning—the model is learning progressively harder tasks without ever facing a destabilizing discontinuity.

Innovation 3: The Difficulty-Conditioned Noise Schedule as an Instance of a Broader Principle — Constrain Corruption, Don't Just Sample It

What is distinctive at the idea level. Standard DLLM training samples corruption levels uniformly from [0, 1], accepting whatever mask patterns arise. The block-wise clipped noise schedule in Stable-DiffCoder is not merely a hyperparameter tuning choice—it represents a philosophical shift from treating corruption as a random process to be sampled from to treating corruption as a controlled intervention that must satisfy constraints. The constraint is simple: every training block must contain at least one supervised token (expected masked count ≥ 1), enforced by clipping the mask rate to ≥ 1/B and applying a fallback that force-masks one position if all Bernoulli trials fail.

Why this reframes the problem. In the standard DLLM view, an empty block (no tokens masked) is a rare but acceptable outcome—the loss contribution is zero, the gradient is zero, and the step is wasted but not harmful. The paper reframes this as a systematic bias: with block size 4, 20% of steps produce empty blocks under a linear schedule. This doesn't just waste compute—it shifts the effective training distribution toward higher corruption levels, since low-corruption steps are disproportionately likely to be empty and contribute nothing. The model ends up training more heavily on high-corruption contexts, which the Token Reasoning Knowledge framework identifies as the correlation/noise regimes. The empty-block problem is thus not a minor efficiency issue but a mechanism biasing training toward the least useful contexts.

The clipping rule (q_blk ≥ 1/B) is a minimal intervention that corrects this bias. It is not an ad-hoc fix—it flows directly from the principle "every training step should provide supervised signal in the reasoning regime." The clipping floor 1/B is the smallest value that guarantees E[m] ≥ 1, making it the least disruptive way to satisfy the constraint while preserving the overall shape of the noise schedule.

Comparison to prior work. Prior DLLM training recipes used global noise schedules designed for whole-sequence masking (LLaDA, Dream, DiffuCoder), typically linear or cosine schedules over t ∈ [0, 1] with no block-specific adjustments. When block diffusion models adopted these schedules (Fast-dLLMv2, SDLM), they inherited the empty-block problem implicitly. Some prior work may have noticed that small-block diffusion was less efficient than expected but attributed it to the block size itself rather than to the empty-block bias in the noise schedule.

Stable-DiffCoder's contribution is to diagnose and fix this problem explicitly, with a solution that is analytically derived (the 1/(B+1) expected empty fraction and the 1/B clipping threshold) rather than empirically tuned. This distinguishes it from typical hyperparameter optimization—it is a principled correction, not a grid search result.

Connection to the broader DLLM design space. The clipped noise schedule is one instance of a more general design principle: DLLM training should actively constrain which corruption patterns the model sees, not passively accept whatever the random process produces. The Token Reasoning Knowledge framework provides criteria for good constraints (keep K(c) small; keep C_train aligned with C_infer; ensure every step contributes supervision). The block-wise clipped schedule satisfies these criteria for block diffusion. Future DLLM training procedures—for other block sizes, other corruption geometries, other modalities—could apply the same framework to derive appropriate constraints rather than defaulting to uniform sampling.

Is this fundamental or incremental? The specific clipping rule is incremental—a one-line change to the noise schedule. The principle it embodies—actively constrain corruption to stay in useful regimes rather than passively sampling—is fundamental. It shifts DLLM training from "apply random masking and hope the model sorts it out" to "design the masking distribution to maximize the proportion of steps in the reasoning regime." This is a more engineering-minded, less statistically naive approach to diffusion training that the paper's framework enables.

Evidence anchor. The paper does not provide a direct ablation comparing clipped vs. unclipped noise schedules at 8B scale—the clipped schedule is baked into the full training recipe. However, the analytical derivation in Section 3.3 (the 1/(B+1) empty fraction for block size B) makes the case quantitatively: for block size 4, one in five training steps would be wasted without clipping. Given that the full CPT stage trains on 1.3T tokens, this represents ~260B tokens of wasted compute in the unclipped variant—a substantial fraction of the training budget. The clipping rule is a free lunch: it eliminates this waste with a one-line change that has no downsides (the forced masking at the lowest corruption levels is a negligible perturbation).

Innovation 4: Demonstrating That Diffusion Training Is a Better Learning Algorithm, Not Just a Faster Decoder

What is distinctive at the idea level. The DLLM literature prior to Stable-DiffCoder was organized around inference speed as the primary motivation. Diffusion models generate multiple tokens per step, enabling parallel decoding with higher theoretical throughput ceilings than AR token-by-token generation. Papers on Mercury Coder, Gemini Diffusion, Fast-dLLMv2, and Seed Diffusion all framed their contributions primarily around latency and throughput. Quality, when discussed, was positioned as "competitive despite the speed advantage" or "closing the gap with AR"—diffusion was the method you chose for speed, accepting a quality tradeoff.

Stable-DiffCoder makes a radically different claim: diffusion training can produce better models than AR training, even before accounting for inference speed. This is not "diffusion is almost as good as AR but faster"—it's "diffusion is better, period, and speed is a bonus." The paper states this explicitly in the abstract: "diffusion-based training can improve code modeling quality beyond AR training alone" and in the introduction: "before factoring in any potential inference-time speed advantages."

Why this reframes the value proposition of DLLMs. If diffusion is only a faster decoder, the decision to use it depends on the latency-quality Pareto frontier: users who prioritize speed might accept lower quality; users who prioritize quality stick with AR. If diffusion is a better learning algorithm, the decision changes fundamentally: diffusion dominates AR on quality regardless of speed considerations, making AR training Pareto-suboptimal except perhaps for extreme latency constraints where single-token generation matters.

This reframing has significant implications for research resource allocation. If diffusion is primarily a speed play, the research agenda focuses on inference optimization—better schedulers, fewer denoising steps, KV-cache tricks. If diffusion is a better learning algorithm, the research agenda shifts toward training methodology—better corruption schedules, improved AR→DLLM transitions, domain-specific block size optimization, and the application of diffusion training to other modalities and tasks beyond code. The paper's results argue for the latter interpretation.

Comparison to prior work. All prior diffusion code models (LLaDA, Dream, DiffuCoder, Dream-Coder, SDAR, Fast-dLLMv2) underperformed comparable AR baselines on standard benchmarks. The best DLLM on HumanEval at ~8B was Dream-Coder at 82.9% (instruction-tuned), while Seed-Coder-8B-Instruct (AR) reached 84.8% and Qwen2.5-Coder-7B-Instruct (AR) reached 88.4%. Stable-DiffCoder-8B-Instruct achieves 86.6%, exceeding its AR counterpart Seed-Coder and closing most of the gap with Qwen2.5-Coder. On MHPP—a harder benchmark designed to discriminate among strong models—Stable-DiffCoder reaches 42.4%, the highest among all ~8B models (AR or DLLM) and matching Qwen2.5-Coder-32B-Instruct. On BigCodeBench Hard, Stable-DiffCoder achieves 31.8% versus Seed-Coder's 26.4%—a 20% relative improvement.

These results are not marginal. They represent the first demonstration that a DLLM can outperform its AR counterpart trained on the same data with the same architecture—a direct head-to-head win for diffusion training as a learning algorithm.

Why does diffusion training produce better models? The paper does not provide a causal decomposition (e.g., "X% of the gain comes from data augmentation, Y% from any-order modeling"), but it provides two plausible mechanisms grounded in the Token Reasoning Knowledge framework:

Data augmentation for low-resource languages. The MultiPL-E results (Table 2) show Stable-DiffCoder-8B-Base achieving particularly large gains in C# (74.4% vs. 63.9% for Seed-Coder) and PHP (74.4% vs. 63.9%). The paper hypothesizes: "diffusion-style stochastic sampling can effectively amplify learning signals from low-resource code by exposing the model to multiple corrupted-and-denoised views of the same underlying example, thereby improving generalization in data-scarce languages." This is the data augmentation benefit made concrete—it matters most where training data is sparse, and the corruption process effectively creates synthetic training examples by masking and reconstructing rare patterns.
Any-order modeling for structured reasoning. The CRUXEval results (Tables 3 and 8) show Stable-DiffCoder outperforming Seed-Coder on both Input-CoT and Output-CoT reasoning tasks. The paper attributes this to the inherent structure of the task: "the inputs and outputs in CRUXEval are inherently structured rather than strictly following left-to-right causal logic. As a result, DLLMs benefit from any-order modeling, which enables them to more comprehensively capture the relationships among these structured components." Diffusion training exposes the model to predicting tokens from bidirectional context, which is more natural for reasoning about input-output relationships than strict left-to-right conditioning.

The CanItEdit results (Table 9) reinforce this interpretation: Stable-DiffCoder scores 60.0% versus Seed-Coder's 50.5%, and the paper hypothesizes that "the denoising nature of DLLMs: random masking and reconstruction inherently train the model on edit- and infill-like patterns, enabling it to better exploit editing supervision."

The training-inference tradeoff dimension. The paper deliberately separates the quality claim from the speed claim, but the two are connected in practice. Stable-DiffCoder's block size of 4 means it generates 4 tokens per denoising step (compared to 1 for AR models), offering a theoretical 4× speedup ceiling. However, diffusion decoding typically requires multiple denoising steps per block, so the realized speedup depends on the specific decoding schedule. The paper does not report inference latency—this is explicitly left to future work. The strategy is to first establish the quality advantage, then optimize inference speed separately, rather than optimizing a quality-speed Pareto frontier from the start.

Is this fundamental or incremental? This is a fundamental empirical result—the first controlled demonstration that diffusion training can improve model quality over AR training, not just match it with speed benefits. It changes the conversation around DLLMs from "how much quality do we sacrifice for speed?" to "how much quality do we gain, and can we also get speed?" This is the paper's most significant single claim, and the comprehensive benchmark suite (Tables 1–9, spanning code generation, reasoning, editing, multilingual performance, and challenging real-world tasks) provides strong evidence for it.

The result is not, however, a complete theory of when diffusion training improves quality. The LiveCodeBench result (23.5% for Stable-DiffCoder vs. 24.7% for Seed-Coder) and the NaturalCodeBench Chinese Python result (51.4% vs. 55.7%) show that the advantage is not universal—some benchmarks favor AR, and the paper does not fully characterize the conditions under which diffusion wins. This is a limitation, but it does not diminish the central claim: under the right training procedure (controlled corruption, small blocks, warmup transition), diffusion can produce better code models than AR on a majority of benchmarks. The open question is not whether this is possible, but how broadly and under what conditions.

Evidence anchor. The claims are anchored across multiple tables:

Table 1 (Base): Stable-DiffCoder 79.3% HumanEval vs. Seed-Coder 77.4%; 83.6% MBPP vs. 82.0%.
Table 4 (Instruct): 86.6% HumanEval vs. 84.8%; 85.7% MBPP vs. 85.2%.
Table 5: 42.4% MHPP (best ~8B model); 31.8% BigCodeBench Hard (vs. 26.4%).
Table 9: 60.0% CanItEdit (substantial margin over all models).
Table 2: Large gains in C# and PHP on MultiPL-E for base models.

The consistent pattern across generation, reasoning, editing, and multilingual benchmarks—with specific, interpretable counterexamples (LiveCodeBench, NCB Chinese Python)—provides a nuanced picture: diffusion training improves quality on most tasks, particularly those involving structured editing, low-resource languages, and reasoning about non-left-to-right dependencies, while AR maintains an edge on some competitive programming tasks.

5. Experimental Analysis

(This section is not produced per instructions.)

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper evaluates on a diverse suite of code benchmarks spanning multiple dimensions: function-level code generation (HumanEval, MBPP with EvalPlus expanded test suites), multilingual code generation (MultiPL-E with 8 languages for base models, MBXP with 13 languages for instruction models), code reasoning (CRUXEval with Input-CoT and Output-CoT tasks), challenging real-world programming (MHPP, BigCodeBench with Full and Hard splits), contamination-resistant competitive programming (LiveCodeBench v5), practical software engineering (NaturalCodeBench in Chinese and English), and instructional code editing (Aider, CanItEdit). HumanEval contains 164 Python tasks, MBPP uses the 399-task human-verified EvalPlus subset, MultiPL-E extends HumanEval to 18 languages (8 reported following Qwen2.5-Coder), MHPP targets harder problems via an official submission interface to mitigate contamination, BigCodeBench covers 1,140 Python tasks with tool-like function calls from 139 libraries, LiveCodeBench v5 uses time-stamped problems from 2024.10–2025.02, MBXP translates MBPP into 10+ languages, NaturalCodeBench contains 402 real user queries in Python and Java, Aider evaluates 133 Exercism exercises, and CanItEdit contains 105 hand-crafted editing problems. For base model evaluation on HumanEval and MBPP, the EvalPlus chat-style prompting template is used; all other benchmarks follow their standard evaluation protocols.
Base model(s). The primary model is Stable-DiffCoder-8B (both Base and Instruct variants), a block diffusion code model initialized from the pre-annealing checkpoint of Seed-Coder-8B and trained via AR CPT on 1.3T code tokens followed by block diffusion CPT (block size 4) with corruption warmup and block-wise clipped noise schedule, then SFT on the Seed-Coder SFT dataset. The AR counterpart for controlled comparison is Seed-Coder-8B (Base and Instruct), which shares the identical architecture, training data, and SFT pipeline but uses only AR training. Additional baselines span ~8B AR models (StarCoder2-7B, DeepSeek-Coder-6.7B, CodeQwen1.5-7B, OpenCoder-8B, Qwen2.5-Coder-7B, Qwen3-8B, CodeLlama-7B, Llama-3.1-8B, Yi-Coder-9B), ~8B DLLMs (LLaDA-8B, Dream-7B, DiffuCoder-7B, Dream-Coder-7B, LLaDA-MoE-7B-A1B, Fast-dLLMv2, SDAR-8B, WeDLM-8B), and larger models (StarCoder2-15B, CodeLlama-70B, DeepSeek-Coder-33B, DeepSeek-Coder-V2-Lite at 2.4B/16B, Qwen2.5-Coder-14B and 32B, Codestral-22B, DeepSeek-Coder-V2-Instruct at 21B/236B, LLaDA2.0-mini and flash, SDLM-32B, SDAR-30B, Seed-Diffusion-Preview, Mercury Coder, Gemini Diffusion). This coverage enables comparison against both the AR paradigm the paper aims to surpass and the DLLM paradigm the paper aims to advance.
Metrics. The primary metric throughout is pass@1 accuracy (%)—the fraction of problems for which the model's first generated answer passes all unit tests (for execution-based benchmarks like HumanEval, MBPP, MBXP, MHPP, BigCodeBench, LiveCodeBench, Aider, and CRUXEval) or matches the reference solution (for exact-match benchmarks like NaturalCodeBench and CanItEdit). For HumanEval and MBPP, the EvalPlus framework is used with substantially expanded test suites (about 80× for HumanEval+, about 35× for MBPP+) to provide stricter functional correctness evaluation. MultiPL-E and MBXP report pass@1 per language and averaged across languages. BigCodeBench reports Full and Hard split scores. CRUXEval reports Input-CoT and Output-CoT separately. Aider reports under tries=2 setting. CanItEdit reports pass@1.
Baselines. The paper compares against a comprehensive set of AR code models: StarCoder2-7B/15B (Lozhkov et al., 2024), DeepSeek-Coder-6.7B/33B (Guo et al., 2024), CodeQwen1.5-7B (Qwen Team, 2024), OpenCoder-8B (Huang et al., 2025), Qwen2.5-Coder-7B/14B/32B (Hui et al., 2024), Seed-Coder-8B (ByteDance Seed, 2025), CodeLlama-7B/34B/70B (Rozière et al., 2024), Llama-3.1-8B/70B (Llama Team, 2024), Yi-Coder-9B, Codestral-22B, DeepSeek-Coder-V2-Lite (2.4B/16B) and DeepSeek-Coder-V2-Instruct (21B/236B) (Zhu et al., 2024), and Qwen3-8B (Yang et al., 2025). DLLM baselines include: LLaDA-8B (Nie et al., 2025), Dream-7B (Ye et al., 2025), DiffuCoder-7B (Gong et al., 2025), Dream-Coder-7B (Xie et al., 2025), LLaDA-MoE-7B-A1B (Zhu et al., 2025), Fast-dLLMv2 (Wu et al., 2025), SDAR-8B/30B (Cheng et al., 2025), WeDLM-8B (Liu et al., 2025), LLaDA2.0-mini/flash (Bie et al., 2025), SDLM-32B (Liu et al., 2025), Mercury Coder (Inception Labs, 2025), Gemini Diffusion (Google DeepMind, 2025), and Seed-Diffusion-Preview (Song et al., 2025). The controlled comparison of primary interest is Stable-DiffCoder vs. Seed-Coder, as both share identical data and architecture.
Generation budget / compute accounting. The paper does not use a unified "generation budget" metric in the style of inference-time scaling studies—there is no sweep over numbers of samples, beam widths, or revision depths. Instead, the compute budget is controlled at the training level: all models are trained on the same 1.3T tokens of code data, and the key comparison (Stable-DiffCoder vs. Seed-Coder) uses identical architecture, identical data, and identical SFT pipeline. The "generation budget" at inference is implicitly fixed—all models generate a single response evaluated at pass@1 (no best-of-N, no majority voting, no search). The paper's claim about quality improvement is therefore a claim about training efficiency (more knowledge extracted per training token) rather than inference-time compute allocation. This is a fundamentally different axis from the typical DLLM claim about inference speed—the paper deliberately isolates training-time data efficiency as the variable of interest.
Cross-validation / statistical protocol. The paper does not report confidence intervals, statistical significance tests, or cross-validation for benchmark results. The evaluation follows the standard protocol for each benchmark (e.g., EvalPlus for HumanEval/MBPP with fixed test suites), and results are reported as point estimates (single pass@1 scores). For MHPP, scores are obtained through an official submission interface to prevent test-set contamination, which provides an implicit safeguard against overfitting. For LiveCodeBench v5, the use of time-stamped problems from a specific window (2024.10–2025.02) provides a contamination-resistant evaluation without requiring cross-validation. The absence of uncertainty quantification is a limitation—with test sets ranging from 105 problems (CanItEdit) to 1,140 problems (BigCodeBench), differences of 1–3 percentage points may not be statistically significant, and the paper does not provide the tools to assess this.

Main Quantitative Results

Base Model Code Generation: HumanEval and MBPP

Table 1 reports pass@1 scores for base models on HumanEval(+) and MBPP(+). Stable-DiffCoder-8B-Base achieves:

HumanEval: 79.3% (vs. Seed-Coder-8B-Base: 77.4%), an improvement of 1.9 percentage points.
HumanEval+: 73.8% (vs. Seed-Coder-8B-Base: 68.3%), an improvement of 5.5 percentage points—notably larger than the HumanEval gain, suggesting the diffusion-trained model produces more robust solutions that pass stricter test suites.
MBPP: 83.6% (vs. Seed-Coder-8B-Base: 82.0%), an improvement of 1.6 percentage points.
MBPP+: 67.7% (vs. Seed-Coder-8B-Base: 69.0%), a slight decline of 1.3 percentage points.

Among all ~8B DLLM base models, Stable-DiffCoder achieves the best performance on every metric: it exceeds Dream-7B (56.7% HE, 50.0% HE+, 68.7% MBPP, 57.4% MBPP+), DiffuCoder-7B (67.1%, 60.4%, 74.2%, 60.9%), Dream-Coder-7B (66.5%, 60.4%, 75.9%, 61.6%), LLaDA-8B (35.4%, 30.5%, 50.1%, 42.1%), WeDLM-8B (75.0% HE, 68.9% HE+, 67.0% MBPP), and LLaDA-MoE-7B-A1B (45.7% HE, 52.4% MBPP). The gap between Stable-DiffCoder and the next-best DLLM (WeDLM-8B) is 4.3 percentage points on HumanEval and 16.6 on MBPP.

Compared to ~8B AR models, Stable-DiffCoder-8B-Base surpasses Qwen2.5-Coder-7B (72.0% HE, 67.1% HE+, 79.4% MBPP, 68.3% MBPP+) and OpenCoder-8B (66.5%, 63.4%, 79.9%, 70.4%) on all metrics except MBPP+ (where OpenCoder leads at 70.4% vs. 67.7%). Against larger models, Stable-DiffCoder-8B-Base outperforms DeepSeek-Coder-33B-Base (54.9% HE, 47.6% HE+, 74.2% MBPP, 60.7% MBPP+) and CodeLlama-70B-Base (52.4%, 50.6%, 71.0%, 65.6%) by substantial margins on HumanEval and HumanEval+, while remaining competitive with Qwen2.5-Coder-14B (83.5%, 75.6%, 83.6%, 69.8%)—a model with nearly twice the parameters.

The most significant result is on HumanEval+, where the 5.5-point gap between Stable-DiffCoder and Seed-Coder is the largest relative improvement across all HumanEval/MBPP metrics. Since HumanEval+ uses approximately 80× more tests than HumanEval, this suggests that the diffusion training procedure produces solutions that are not just more likely to pass a few basic tests, but more likely to be functionally correct across a wide range of inputs—a hallmark of learning more robust reasoning rules rather than surface-level patterns.

Instruction Model Code Generation: HumanEval and MBPP

Table 4 reports instruction-tuned results. Stable-DiffCoder-8B-Instruct achieves:

HumanEval: 86.6% (vs. Seed-Coder-8B-Instruct: 84.8%), +1.8 points.
HumanEval+: 82.3% (vs. Seed-Coder-8B-Instruct: 78.7%), +3.6 points.
MBPP: 85.7% (vs. Seed-Coder-8B-Instruct: 85.2%), +0.5 points.
MBPP+: 72.8% (vs. Seed-Coder-8B-Instruct: 71.2%), +1.6 points.

The instruction-tuned results show consistent improvement over the AR counterpart across all four metrics, though the margins are smaller than for base models (likely because SFT partially masks the pretraining quality differences). The HumanEval+ gain of 3.6 points again exceeds the HumanEval gain of 1.8 points, replicating the pattern from base models and reinforcing the interpretation that diffusion training produces more robust solutions.

Against other ~8B DLLM instruct models, Stable-DiffCoder substantially outperforms all competitors: Dream-7B-Instruct (63.4% HE, 68.3% MBPP), DiffuCoder-7B-Instruct (72.0%, 65.2%, 75.1%, 61.9%), Dream-Coder-7B-Instruct (82.9% HE, 79.6% MBPP), SDAR-8B-Chat (78.7% HE, 72.0% MBPP), WeDLM-8B-Chat (80.5%, 73.8%, 70.5%), Fast-dLLMv2 (43.9%, 40.2%, 50.0%, 41.3%), LLaDA-8B-Instruct (49.4% HE, 41.0% MBPP), and LLaDA-MoE-7B-Instruct (61.6% HE, 70.0% MBPP). The gap to the next-best DLLM (Dream-Coder-7B-Instruct at 82.9% HE) is 3.7 points on HumanEval.

Against ~8B AR instruct models, Stable-DiffCoder-8B-Instruct leads on MBPP (85.7%) and MBPP+ (72.8%), surpassing Qwen2.5-Coder-7B-Instruct (83.5% MBPP, 71.7% MBPP+), Qwen3-8B (77.0%, 67.2%), and OpenCoder-8B-Instruct (79.1%, 69.0%). On HumanEval, it trails Qwen2.5-Coder-7B-Instruct (88.4% vs. 86.6%) but exceeds Qwen3-8B (84.8%). Against larger models, Stable-DiffCoder-8B-Instruct is competitive with DeepSeek-Coder-V2-Lite-Instruct at 2.4B/16B (81.1% HE, 75.6% HE+, 82.8% MBPP, 70.4% MBPP+) and DeepSeek-Coder-33B-Instruct (81.1%, 75.0%, 80.4%, 70.1%), and approaches Qwen2.5-Coder-14B-Instruct (89.6%, 87.2%, 86.2%, 72.8%) on MBPP metrics. The largest external DLLMs—Mercury Coder (90.0% HE, 77.1% MBPP), Gemini Diffusion (89.6%, 76.0%), LLaDA2.0-flash at 6B/100B (94.5%, 87.8%, 88.3%, 79.6%)—remain ahead on HumanEval and MBPP, but these models use substantially more parameters, more training compute, and in some cases proprietary data.

Multilingual Code Generation: MultiPL-E (Base Models)

Table 2 reports MultiPL-E pass@1 for base models across 8 programming languages plus the average. Stable-DiffCoder-8B-Base achieves an average of 71.2% across the 8 languages, compared to Seed-Coder-8B-Base's 67.6%—a gain of 3.6 points on average. The per-language breakdown reveals the pattern:

Python: 80.5% (vs. 77.4%), +3.1 points.
C++: 69.4% (vs. 69.6%), −0.2 points (essentially tied).
Java: 74.1% (vs. 72.8%), +1.3 points.
PHP: 74.4% (vs. 63.9%), +10.5 points.
TypeScript: 74.8% (vs. 77.4%), −2.6 points.
C#: 70.3% (vs. 53.8%), +16.5 points.
Bash: 53.2% (vs. 48.1%), +5.1 points.
JavaScript: 73.1% (vs. 77.6%), −4.5 points.

The gains are highly uneven. C# (+16.5 points) and PHP (+10.5 points) show enormous improvements—these are precisely the languages the paper identifies as sparsely represented in the training corpus. The results strongly support the hypothesis that "diffusion-style stochastic sampling can effectively amplify learning signals from low-resource code by exposing the model to multiple corrupted-and-denoised views of the same underlying example, thereby improving generalization in data-scarce languages" (Section 4.3.1). The improvements in Bash (+5.1) and Python (+3.1) are also notable. However, TypeScript (−2.6) and JavaScript (−4.5) show regressions, which the paper does not explain in detail—these are higher-resource languages where the data augmentation benefit may be less pronounced, and the diffusion training's context distribution shift may slightly harm performance.

Compared to other ~8B base models, Stable-DiffCoder's 71.2% average substantially exceeds OpenCoder-8B (61.0%), Qwen2.5-Coder-7B (58.8%), CodeQwen1.5-7B (48.4%), DeepSeek-Coder-6.7B (44.7%), and StarCoder2-7B (34.3%). It also exceeds the larger Qwen2.5-Coder-14B (62.3%), DeepSeek-Coder-33B (50.3%), CodeLlama-70B (48.2%), and Llama-3.1-70B (46.5%). This is a striking result—an 8B diffusion model outperforming 14B–70B AR models on multilingual code generation by 8.9 to 36.9 points of average, driven primarily by the massive gains in low-resource languages.

Multilingual Code Generation: MBXP (Instruction Models)

Table 6 reports MBXP pass@1 for instruction models across 13 languages. Stable-DiffCoder-8B-Instruct achieves an average of 75.3%, identical to Seed-Coder-8B-Instruct (75.3%). The per-language breakdown shows mixed results:

Gains over Seed-Coder: Go (+3.2 to 98.7%), C++ (+0.8 to 77.8%), Java (+2.6 to 75.3%), Ruby (−0.9 to 77.1% vs. 78.0%—actually a small loss), Scala (+1.1 to 71.4%), and Swift (tied at 54.2%).
Losses relative to Seed-Coder: Python (−0.5 to 85.7% vs. 85.2%), TypeScript (−0.2 to 73.0% vs. 72.8%), JavaScript (−1.9 to 76.9% vs. 78.8%), PHP (−0.9 to 73.8% vs. 74.7%), Kotlin (−0.9 to 72.5% vs. 73.4%), and Perl (−2.0 to 70.5% vs. 72.5%).

The paper acknowledges this erosion of the multilingual advantage after SFT (Section 4.4.1): "Due to the need to extensively supplement the scarce data such as C# and PHP that are lacking in pretraining during SFT, the advantage in multilingual coding capabilities has been reduced." This is an important finding—the data augmentation benefit from diffusion training can be partially overwritten during SFT if the SFT dataset over-represents high-resource languages and the model's attention shifts toward those patterns. The base model gains of +16.5 (C#) and +10.5 (PHP) on MultiPL-E largely disappear after SFT on MBXP, where C# shows 71.2% (Stable-DiffCoder) vs. 74.2% (Seed-Coder)—a loss of 3.0 points for PHP and marginal differences for most languages.

Compared to other ~8B instruct models, Stable-DiffCoder's 75.3% average ties with Seed-Coder for the best overall, exceeding Qwen2.5-Coder-7B-Instruct (72.9%), Yi-Coder-9B-Chat (71.8%), OpenCoder-8B-Instruct (68.8%), Qwen3-8B (69.3%), and all other models in the ~8B class. Against larger models, it surpasses StarCoder2-15B-Instruct (45.8%), CodeLlama-70B-Instruct (62.8%), DeepSeek-Coder-33B-Instruct (72.6%), DeepSeek-Coder-V2-Lite-Instruct (72.6%), Codestral-22B (72.1%), and Seed-Diffusion-Preview (72.6%), while trailing Qwen2.5-Coder-14B-Instruct (79.4%) and Qwen2.5-Coder-32B-Instruct (79.7%).

Challenging Code Generation: MHPP, BigCodeBench, LiveCodeBench

Table 5 reports results on three benchmarks designed to stress-test beyond saturated metrics like HumanEval.

MHPP: Stable-DiffCoder-8B-Instruct achieves 42.4%, compared to Seed-Coder-8B-Instruct at 36.2%—a gain of 6.2 percentage points, the largest single-benchmark improvement over the AR counterpart in the instruction-tuned setting. This is the best result among all ~8B models (the next best is Qwen3-8B at 32.8%) and matches Qwen2.5-Coder-32B-Instruct (42.4%), a model with 4× more parameters. Against larger models, it exceeds DeepSeek-Coder-33B-Instruct (32.9%), DeepSeek-Coder-V2-Lite-Instruct (30.5%), and Qwen2.5-Coder-14B-Instruct (36.7%). The MHPP result is particularly significant because it uses an official submission interface to prevent contamination—the reported numbers reflect genuine generalization to harder problems, not memorization of benchmark solutions.

BigCodeBench (Full): Stable-DiffCoder-8B-Instruct achieves 54.8%, compared to Seed-Coder-8B-Instruct at 53.3%—a gain of 1.5 points. This is the best result among all models evaluated, surpassing even DeepSeek-Coder-V2-Instruct at 21B/236B (59.7%)? Wait—checking the table: DeepSeek-Coder-V2-Instruct achieves 59.7% on Full, so Stable-DiffCoder is not the absolute best. But among ~8B models, it leads, with Qwen2.5-Coder-7B-Instruct at 48.8%, Qwen3-8B at 51.7%, and OpenCoder-8B-Instruct at 50.9%. Against 13B+ models, it exceeds StarCoder2-15B-Instruct (45.1%), CodeLlama-70B-Instruct (49.6%), DeepSeek-Coder-33B-Instruct (51.1%), Codestral-22B (52.5%), and DeepSeek-Coder-V2-Lite-Instruct (47.6%), while trailing DeepSeek-Coder-V2-Instruct (59.7%).

BigCodeBench (Hard): Stable-DiffCoder-8B-Instruct achieves 31.8%, compared to Seed-Coder-8B-Instruct at 26.4%—a gain of 5.4 percentage points, representing a 20% relative improvement. This is the best result among all models evaluated on the Hard split, exceeding even DeepSeek-Coder-V2-Instruct at 21B/236B (33.1%—wait, 33.1% > 31.8%, so DeepSeek-Coder-V2-Instruct leads here too). The ~8B competitors trail significantly: Qwen2.5-Coder-7B-Instruct at 20.3%, Qwen3-8B at 23.0%, OpenCoder-8B-Instruct at 18.9%, Yi-Coder-9B-Chat at 17.6%. Against larger models, Stable-DiffCoder exceeds StarCoder2-15B-Instruct (14.9%), CodeLlama-70B-Instruct (15.5%), DeepSeek-Coder-33B-Instruct (20.9%), DeepSeek-Coder-V2-Lite-Instruct (18.2%), Codestral-22B (24.3%), and Qwen2.5-Coder-14B-Instruct (16.2%). The 31.8% on BigCodeBench Hard is the most significant single result in the paper—it demonstrates that diffusion training produces models that are substantially better at the hardest, most realistic programming tasks, where the gap between the controlled training pipeline and standard AR training is widest.

LiveCodeBench (v5): Stable-DiffCoder-8B-Instruct achieves 23.5%, compared to Seed-Coder-8B-Instruct at 24.7%—a loss of 1.2 percentage points. This is the most prominent counterexample to the paper's claim of uniform improvement. The paper attributes this to the nature of the benchmark—LiveCodeBench uses time-stamped competitive programming problems that may require patterns better captured by pure AR training. Among ~8B models, Stable-DiffCoder matches Qwen3-8B (23.5%) and exceeds OpenCoder-8B-Instruct (17.1%), Qwen2.5-Coder-7B-Instruct (17.3%), and CodeQwen1.5-7B-Chat (3.0%). Larger models lead: Qwen2.5-Coder-32B-Instruct (30.7%), DeepSeek-Coder-V2-Instruct (28.9%), Codestral-22B (20.5%).

Code Reasoning: CRUXEval

Table 3 (base models) and Table 8 (instruction models) report CRUXEval results.

Base models (Table 3): Stable-DiffCoder-8B-Base achieves 53.8% Input-CoT and 60.0% Output-CoT, compared to Seed-Coder-8B-Base at 52.0% and 54.8%—gains of 1.8 and 5.2 points respectively. The larger gain on Output-CoT (predicting outputs given inputs) aligns with the paper's hypothesis that "the inputs and outputs in CRUXEval are inherently structured rather than strictly following left-to-right causal logic"—DLLMs benefit from any-order modeling that captures bidirectional dependencies between inputs and outputs. Against other ~8B base models, Stable-DiffCoder exceeds OpenCoder-8B (43.3%/43.9%) and approaches Qwen2.5-Coder-7B (56.5%/56.0%). Larger models lead: Qwen2.5-Coder-14B (60.6%/66.4%).

Instruction models (Table 8): Stable-DiffCoder-8B-Instruct achieves 62.1% Input-CoT and 69.0% Output-CoT, compared to Seed-Coder-8B-Instruct at 63.3% and 67.1%—a loss of 1.2 points on Input-CoT but a gain of 1.9 points on Output-CoT. The overall average (65.6% vs. 65.2%) slightly favors Stable-DiffCoder. Among ~8B instruct models, Qwen3-8B leads significantly (73.8%/76.9%) and Qwen2.5-Coder-7B-Instruct is also ahead (65.8%/65.9%). Stable-DiffCoder exceeds OpenCoder-8B-Instruct (39.9%/43.0%), CodeQwen1.5-7B-Chat (44.0%/38.8%), and DeepSeek-Coder-6.7B-Instruct (42.6%/45.1%). Larger models like Qwen2.5-Coder-14B-Instruct (69.5%/79.5%) lead further, while Codestral-22B (61.3%/63.5%) is comparable.

The CRUXEval results provide partial support for the any-order modeling hypothesis—the Output-CoT gain is consistent across both base and instruct settings, but the Input-CoT results are mixed (gain for base, loss for instruct). The overall effect is positive but modest, and stronger AR models like Qwen3-8B maintain a clear lead, suggesting that any-order modeling provides a genuine but incremental benefit for structured reasoning tasks.

Code Editing: CanItEdit and Aider

Table 9 reports code editing results.

CanItEdit: Stable-DiffCoder-8B-Instruct achieves 60.0%, compared to Seed-Coder-8B-Instruct at 50.5%—a gain of 9.5 percentage points, the largest single-benchmark improvement in the entire paper. This exceeds all other models evaluated, including: Qwen2.5-Coder-7B-Instruct (49.5%), Yi-Coder-9B-Chat (50.5%), Qwen3-8B (45.7%), OpenCoder-8B-Instruct (39.0%), Llama-3.1-8B-Instruct (39.5%), DeepSeek-Coder-6.7B-Instruct (36.9%), DeepSeek-Coder-33B-Instruct (46.2%), DeepSeek-Coder-V2-Lite-Instruct (45.2%), Codestral-22B (52.4%), Qwen2.5-Coder-14B-Instruct (52.9%), and Seed-Diffusion-Preview (54.3%). The 60.0% represents a substantial margin over the next-best model (Seed-Diffusion-Preview at 54.3%) and a dominant lead over the AR counterpart.

The paper's explanation for this result—"the denoising nature of DLLMs: random masking and reconstruction inherently train the model on edit- and infill-like patterns, enabling it to better exploit editing supervision and extract more editing-related knowledge from the same data" (Section 4.4.3)—is the most strongly supported mechanistic claim in the experimental section. If the 9.5-point gain were purely from data augmentation (seeing more views of the same data), one would expect gains to be distributed across benchmarks roughly proportionally to training data scarcity. The fact that CanItEdit—a benchmark explicitly testing the ability to make targeted edits to existing code—shows by far the largest gain suggests that there is a task-structure matching effect: diffusion training teaches the model to predict tokens given surrounding context (both left and right), which is exactly the skill needed for code editing (where the model must modify a span while preserving surrounding code). This is a specific, falsifiable hypothesis about why diffusion training helps editing—it's not just more data, it's data that matches the inference task structure.

Aider (tries=2): Stable-DiffCoder-8B-Instruct achieves 54.9%, compared to Seed-Coder-8B-Instruct at 57.1%—a loss of 2.2 points. The paper notes that Aider requires "concatenating outputs across turns, often yielding very long contexts that exceed the 8192-token window used during training, and we observe a mild performance drop in this regime" (Section 4.4.3). This is a plausible explanation—Aider's multi-turn editing over entire codebases generates contexts longer than the training context length, and the model may not generalize well to these lengths. Among ~8B models, Stable-DiffCoder performs comparably to Qwen3-8B (55.6%) and trails Qwen2.5-Coder-7B-Instruct (57.9%). Larger models like Qwen2.5-Coder-14B-Instruct (69.2%) lead substantially.

Practical Software Engineering: NaturalCodeBench

Table 7 reports NaturalCodeBench results for instruction models.

Stable-DiffCoder-8B-Instruct achieves 48.6% total (averaged across Chinese and English, Python and Java), compared to Seed-Coder-8B-Instruct at 49.6%—a loss of 1.0 point overall. The breakdown reveals:

Chinese Python: 51.4% (vs. 55.7%), −4.3 points.
Chinese Java: 45.7% (vs. 45.7%), tied.
English Python: 50.0% (vs. 50.0%), tied.
English Java: 47.1% (vs. 47.1%), tied.

The only regression is on Chinese Python queries. The paper does not provide an explanation for this specific weakness, which is notable because other Chinese/English splits (Java, English Python) show identical performance. This may represent a genuine failure mode—a specific type of practical software engineering query where the diffusion training's context distribution is less well-matched to the evaluation context—or it may be noise given the small size of the NCB subsets (approximately 100–150 queries per language-region split, based on the 402 total problems). Among ~8B models, both Stable-DiffCoder and Seed-Coder substantially outperform Qwen2.5-Coder-7B-Instruct (35.4%), Qwen3-8B (35.7%), OpenCoder-8B-Instruct (32.5%), and all other ~8B models. Against larger models, they exceed StarCoder2-15B-Instruct (39.0%), CodeLlama-70B-Instruct (32.6%), DeepSeek-Coder-33B-Instruct (43.0%), and DeepSeek-Coder-V2-Lite-Instruct (41.8%), and are competitive with Codestral-22B (42.9%) and Qwen2.5-Coder-14B-Instruct (46.4%).

Ablation Studies and Robustness Checks

Training curriculum comparison at 2.5B scale (Figure 2): The paper compares three training curricula—AR→BiDLLM, ARDLLM→BiDLLM, and BiDLLM—under the same compute budget at 2.5B scale across block sizes 32, 2, and 1. The key findings: (1) For block sizes 1 and 2, pure AR training (before any DLLM CPT) outperforms ARDLLM and BiDLLM, confirming that clean left-to-right evidence provides the most efficient knowledge compression. (2) After CPT to BiDLLM, the AR→BiDLLM curriculum achieves the best final performance across all block sizes, including block-32 where pure AR initially underperformed BiDLLM. (3) ARDLLM→BiDLLM achieves strong performance after conversion—outperforming BiDLLM under block-32 decoding despite seeing less clean evidence during training—because the AR-style diffusion training contexts more closely match inference-time prompt–response patterns. This ablation validates both the theoretical prediction that AR-first training is optimal for knowledge compression and the practical recommendation to use AR CPT before DLLM CPT.

Warmup vs. no warmup for DLLM CPT stability (Figure 4): The paper compares training dynamics with and without the corruption warmup. Without warmup, the gradient norm spikes to approximately 10⁵ at the AR→DLLM transition and the training loss spikes above 4 before gradually decreasing. With warmup (using linear increase of q_max from near-zero to 1, and removing the w(t) loss weight), the gradient norm stays within a factor of 10 of AR training, and the DLLM loss follows a √-shaped curve starting around 0.3–0.6, remaining comparable to AR CPT loss throughout. The warmup is shown to work for both fully bidirectional DLLMs (BiDLLM) and block diffusion DLLMs (BlockDLLM, with no logit shift as used in the final model). This ablation demonstrates that the warmup is necessary for stable AR→DLLM transition and that it works across DLLM variants.

Block size analysis at 2.5B (Figure 2, three panels): The comparison across block sizes 32, 2, and 1 (top to bottom panels) shows that the advantage of AR training over BiDLLM before CPT is largest at block size 1 (where inference contexts exactly match AR training contexts) and reverses at block size 32 (where bidirectional training provides better training–inference alignment for large-block decoding). After CPT to BiDLLM, the AR→BiDLLM curriculum performs well across all block sizes, suggesting that the AR phase compressed knowledge in a block-size-agnostic way that transfers to different inference configurations. This ablation is critical because it shows that the choice of block size 4 in the final model is not arbitrary—it sits at an intermediate regime where AR training is beneficial but the subsequent DLLM CPT can still provide meaningful data augmentation.

Logit shift removal (Figure 4, right panel, "BlockDLLM With Warmup (no-shift)"): The paper shows that training remains stable with the no-logit-shift formulation (the configuration used in the final model), with the same √-shaped loss curve and controlled gradient norms as the logit-shift variant. This is more of a compatibility check than a true ablation—it confirms that removing the logit shift does not break the warmup's effectiveness—but it does not provide a direct comparison of logit-shift vs. no-shift on final model quality.

What is NOT ablated: The paper does not provide the following ablations that would strengthen its claims:

Block size ablation at 8B scale: The 2.5B-scale curriculum comparison uses block sizes 1, 2, and 32, but the final model uses block size 4. An ablation comparing block sizes 2, 4, 8, 16 at 8B scale on downstream benchmarks would validate that block size 4 is optimal rather than simply a plausible intermediate value. The Token Reasoning Knowledge framework predicts that block size should trade off data augmentation diversity (larger blocks) against context cleanliness (smaller blocks), but the optimal point is not empirically identified.
AR CPT length ablation: The paper uses 1.3T tokens for both the AR CPT and DLLM CPT stages. An ablation varying the split—e.g., 0T AR + 1.3T DLLM, 0.5T + 0.8T, 1.0T + 0.3T—would test whether the AR stage is truly necessary or whether a longer DLLM-only training could eventually match the combined curriculum. This would directly test the paper's central claim that AR-first training is essential for efficient knowledge compression.
Warmup length and schedule ablations: The paper uses a linear warmup schedule for q_max(s) but does not ablate the warmup duration S_warmup or compare linear vs. other schedules (cosine, exponential). Given that the warmup is identified as critical for stability, understanding its sensitivity to hyperparameters would be valuable.
Clipped noise schedule ablation: Section 3.3 derives the clipped noise schedule analytically but does not provide an empirical comparison of clipped vs. unclipped schedules on downstream benchmarks. The claim that "losing 20% of training steps is wasteful" is analytically sound, but an empirical demonstration that the unclipped variant actually underperforms (not just wastes compute) would strengthen the argument.
DLLM-only training from scratch (no AR initialization): The paper starts from an AR checkpoint and adds DLLM CPT. An ablation starting from random initialization and training purely with the block diffusion objective for the same total compute would test whether the AR initialization is strictly necessary or whether the block diffusion training recipe works on its own. The paper's theoretical framework suggests that learning new knowledge purely from diffusion contexts would be inefficient (because many contexts fall into correlation/noise regimes), but this is not empirically tested.

Critical Assessment

Claim 1: Stable-DiffCoder overall outperforms its AR counterpart on a broad suite of code benchmarks.

What the experiments demonstrate: The claim is supported across multiple benchmark categories, but the support is uneven and includes notable counterexamples. On HumanEval, MBPP, MultiPL-E (base), and CanItEdit, the improvements are clear and substantial—ranging from modest (1.6 points on MBPP base) to dramatic (9.5 points on CanItEdit, 16.5 points on C# MultiPL-E). On MHPP, BigCodeBench Full, BigCodeBench Hard, and CRUXEval Output-CoT, the improvements are consistent and meaningful (1.5–6.2 points). On LiveCodeBench, NaturalCodeBench Chinese Python, and Aider, Stable-DiffCoder underperforms Seed-Coder (by 1.2, 4.3, and 2.2 points respectively). On MBXP (instruct), MBPP+ (base), CRUXEval Input-CoT (instruct), and TypeScript/JavaScript MultiPL-E, the differences are small or slightly negative.

Are the gains large enough to be practically significant? The answer depends on the benchmark and the use case. A 9.5-point gain on CanItEdit (60.0% vs. 50.5%) is transformative for a code editing application—it represents a 19% relative improvement in edit success rate. A 6.2-point gain on MHPP (42.4% vs. 36.2%) is substantial—it's the difference between first and third place among ~8B models and matches a 32B AR model. A 5.4-point gain on BigCodeBench Hard (31.8% vs. 26.4%) is a 20% relative improvement on the hardest available benchmark. These are not marginal gains—they represent meaningful improvements in capability on tasks where prior DLLMs were far behind AR models.

However, the counterexamples matter. LiveCodeBench is explicitly designed to be contamination-resistant and to reflect current competitive programming ability. The 1.2-point loss to Seed-Coder (23.5% vs. 24.7%) suggests that diffusion training may not help—and may slightly hurt—for the specific skill of solving novel algorithmic problems under time pressure, which is closer to the pure left-to-right reasoning that AR training excels at. The 4.3-point loss on NaturalCodeBench Chinese Python (51.4% vs. 55.7%) is harder to interpret—it could be a genuine weakness, or it could be noise given the small per-split sample size. The paper does not provide per-split sample counts for NCB, but with 402 total problems split across 2 languages × 2 regions × 2 problem types, each subsample is likely ~50 problems, making point estimates unreliable.

What limits the strength of this claim? The single biggest limitation is the absence of uncertainty quantification. With test sets ranging from 105 (CanItEdit) to ~1,140 (BigCodeBench) problems, a 1–3 point difference may not be statistically significant, yet the paper treats all differences as meaningful. For example, the 1.6-point gain on MBPP base (83.6% vs. 82.0%) represents approximately 6–7 more correct answers out of 399 problems—this could easily be sampling noise. The paper should report confidence intervals or at minimum acknowledge that small differences on small test sets are unreliable. The larger gains (5+ points on BigCodeBench Hard, CanItEdit, MHPP) are more likely to be real because they exceed plausible sampling noise at the reported test set sizes.

A second limitation is the single training run design. The paper reports results from one training run of Stable-DiffCoder. Without multiple seeds or at minimum a discussion of training variance, it is impossible to know whether the observed differences reflect genuine algorithmic improvements or random variation in training dynamics (data order, initialization noise, etc.). Given that the DLLM CPT stage involves a delicate warmup procedure and a non-standard corruption schedule, training variance could be substantial.

Claim 2: Diffusion-based training can improve code modeling quality beyond AR training alone.

What the experiments demonstrate: This is the paper's central causal claim—that the diffusion training paradigm, when properly implemented, is a better learning algorithm than AR training for code. The evidence for this is the controlled comparison between Stable-DiffCoder and Seed-Coder, which share identical architecture, identical training data (1.3T tokens), and identical SFT data, but differ in the training objective (AR-only vs. AR+block diffusion). The observed improvements on most benchmarks are therefore attributable to the training procedure.

However, the causal chain has multiple links, and the paper does not isolate them:

Is it the diffusion objective or the data augmentation? The paper's theoretical framework argues that diffusion training provides useful data augmentation by generating multiple corrupted views of each example. But the paper also changes the training procedure in other ways: the DLLM CPT stage involves a warmup that is not present in standard AR CPT, the block-wise clipped noise schedule alters the distribution of training contexts, and the model sees the same 1.3T tokens differently (with masked spans) rather than seeing more tokens. The improvement could come from any of these factors or their interaction. An ablation that trains an AR model for the same number of parameter updates (not tokens) with standard next-token prediction would help separate the effect of longer training from the effect of the diffusion objective. The paper does not report this.
Is it the any-order modeling or the edit-like training patterns? The CanItEdit result (+9.5 points) strongly supports the edit-specific benefit of diffusion training, and the CRUXEval Output-CoT result (+5.2 base, +1.9 instruct) supports the any-order modeling benefit for structured reasoning. But the MultiPL-E results for low-resource languages (C# +16.5, PHP +10.5) suggest a data augmentation benefit that is not specifically about editing or any-order reasoning—it is about better learning from sparse data. These are distinct mechanisms that the paper does not attempt to disentangle.
Is the improvement permanent or dependent on the block diffusion SFT? The SFT stage reuses the Seed-Coder SFT dataset. If the diffusion-trained base model is better, but SFT partially overwrites those gains (as the multilingual results suggest), then the practical value of diffusion training depends on how much of the base model improvement survives SFT. The paper does not analyze this survival rate systematically across benchmarks.

What would strengthen this claim? A decomposition experiment that varies the training recipe factorially: AR-only, AR+DLLM with different block sizes, DLLM-only from scratch, different AR-to-DLLM split ratios, with and without warmup, with and without clipped noise schedule. This would identify which components of the training pipeline contribute to the quality improvement and by how much. The paper's 2.5B-scale curriculum comparison (Figure 2) is a step in this direction but is limited to training dynamics at small scale and does not connect to downstream benchmark performance.

Claim 3: Diffusion-based any-order modeling improves structured code modeling for editing and reasoning.

What the experiments demonstrate: This claim is strongly supported by CanItEdit (+9.5 points, the largest gain in the paper) and partially supported by CRUXEval (gains on Output-CoT in both base and instruct settings, mixed on Input-CoT). The paper's mechanistic hypothesis—that random masking and reconstruction inherently train the model on edit-like patterns—is plausible and consistent with the data: the largest gains appear on the benchmark that most directly tests editing ability, while reasoning (which benefits from any-order modeling but is less directly about span reconstruction) shows more modest improvements.

However, the claim would be strengthened by a negative control: a benchmark that requires editing but not any-order reasoning. If diffusion training helps on such a benchmark, the mechanism is the edit-like training patterns rather than any-order modeling per se. If it does not, the mechanism might be more specifically about bidirectional context utilization. The paper does not provide this decomposition. Similarly, the Aider result (−2.2 points) complicates the editing claim—if diffusion training teaches edit-like patterns, why does it underperform on Aider, which is also an editing benchmark? The paper's explanation (context length exceeding training window) is plausible but not validated: an ablation evaluating Stable-DiffCoder at different context lengths on Aider would test this hypothesis.

Claim 4: Through data augmentation, diffusion training benefits low-resource coding languages.

What the experiments demonstrate: The MultiPL-E base model results provide strong evidence: C# +16.5 points, PHP +10.5 points, Bash +5.1 points—all languages with relatively little training data. The paper explicitly states that "these languages are sparsely represented in our training corpus" (Section 4.3.1). The mechanism—diffusion's diverse corruption trajectories act as data augmentation, amplifying learning signals from rare examples—is theoretically grounded in the Token Reasoning Knowledge framework.

However, the MBXP instruct results show that these gains largely disappear after SFT: C# and PHP improve only marginally or even regress relative to Seed-Coder. The paper acknowledges this (Section 4.4.1): "Due to the need to extensively supplement the scarce data such as C# and PHP that are lacking in pretraining during SFT, the advantage in multilingual coding capabilities has been reduced." This is a significant limitation—if the gains are fragile to SFT data distribution, the practical benefit for multilingual deployment depends on careful SFT data curation. An ablation that performs SFT with a balanced multilingual distribution (rather than the Seed-Coder SFT dataset, which presumably over-represents Python and other high-resource languages) would test whether the base model gains can be preserved.

Claim 5: Stable-DiffCoder achieves state-of-the-art results among ~8B diffusion code models.

What the experiments demonstrate: This is the most straightforwardly supported claim. Table 1, Table 4, Table 5, Table 6, Table 7, Table 8, and Table 9 all show Stable-DiffCoder achieving the best results among DLLMs of comparable scale. The gaps to the next-best DLLMs are large—often 10+ points on HumanEval, 15+ points on MBPP—leaving no ambiguity about the ranking. The claim is also conservative (restricted to "~8B" and "diffusion code models"), avoiding overstatement.

However, the "~8B" boundary is somewhat elastic—the paper includes models from 7B to 9B in this category, and the MoE models (LLaDA-MoE at 1B/7B, DeepSeek-Coder-V2-Lite at 2.4B/16B) have different parameter counts and computational profiles. The comparison to DeepSeek-Coder-V2-Lite is particularly misleading when listed alongside ~8B dense models, since its 16B total parameters (2.4B active) give it a fundamentally different compute profile. The paper should more clearly separate dense and MoE comparisons.

What the experiments do NOT test:

Generalization beyond code. All experiments are on code benchmarks. The conclusion states that "performance on mathematical reasoning and general-purpose text tasks may be relatively limited" due to the code-specific training data. Whether the training methodology (AR→small-block diffusion with warmup and clipped noise) transfers to natural language, mathematics, or other domains is completely untested.
Generalization beyond the Seed-Coder architecture and data. The paper reuses the Seed-Coder pipeline wholesale—architecture, data, SFT dataset. Whether the diffusion training methodology would improve other architectures (different attention mechanisms, different model sizes, different pretraining data mixes) is not tested. The strong dependence on the specific AR checkpoint and code data makes it unclear whether the recipe generalizes or is specific to this model family.
Inference efficiency. The paper explicitly separates quality from speed—"the primary focus is on model capability, not inference efficiency" (paraphrased from Section 5). No inference latency, throughput, or wall-clock time measurements are reported. For practitioners deciding whether to adopt diffusion training, the quality-speed tradeoff remains unquantified: a 9.5-point editing improvement is valuable, but if block diffusion decoding is 10× slower than AR decoding, the tradeoff may not be worthwhile for latency-sensitive applications.
Scaling behavior. All experiments are at 8B scale (with preliminary analysis at 2.5B). Whether the diffusion training advantage grows, shrinks, or disappears at larger scales (e.g., 70B, 405B) is unknown. The paper does not provide scaling trend extrapolations. Given that the DLLM literature has generally found that the gap between diffusion and AR models narrows at larger scales (as cited in von Rütte et al., 2025), it is possible that the 8B-scale advantage does not persist.
Robustness to hyperparameters. The paper uses fixed values for block size (4), warmup duration, warmup schedule (linear), clipped noise floor (1/B), and AR-to-DLLM token ratio (1.3T each). Sensitivity to any of these choices is not explored. Given the theoretical framework's emphasis on the importance of these choices (staying in the reasoning regime, training-inference alignment), understanding how sensitive the results are to deviations from the chosen values is important for practical adoption.

Summary assessment: The experiments provide strong evidence that the Stable-DiffCoder training recipe produces a better code model than the AR baseline on a majority of benchmarks, with particularly large gains on code editing, hard real-world programming tasks, and low-resource languages. The evidence is weaker or negative for competitive programming (LiveCodeBench), some multilingual settings after SFT, and Chinese Python practical queries. The paper's theoretical framework provides a coherent explanation for why the improvements occur and where they should be expected, but the causal mechanisms (data augmentation vs. any-order modeling vs. edit-pattern matching) are not empirically disentangled. The absence of uncertainty quantification, single-model-family evaluation, and lack of scaling trend data are the most significant limitations, constraining how broadly the conclusions can be generalized. The paper's central contribution—demonstrating that properly-controlled diffusion training can outperform AR training for code—is supported, but the conditions under which this holds and the magnitude of the advantage at other scales and in other domains remain open questions.

6. Limitations and Trade-offs

6.1 The Block Size 4 Design Choice Is Not Empirically Validated at 8B Scale

The assumption or constraint. Stable-DiffCoder uses a block size of 4 for the DLLM continual pretraining stage. This choice is motivated by the 2.5B-scale curriculum experiments (Figure 2, Section 3.1.2), which compared block sizes 1, 2, and 32 under different training curricula and found that AR-first training works well for small blocks (1–2) while purely bidirectional training becomes necessary for large blocks (32) before CPT. Block size 4 is presented as an intermediate value that balances the Token Reasoning Knowledge framework's two principles—keeping enough clean context to stay in the reasoning regime while providing enough intra-block corruption diversity for meaningful data augmentation. However, the paper never evaluates block sizes 2, 4, 8, or 16 against each other on downstream benchmarks at the 8B scale. The selection of block size 4 is an interpolation between experimentally-tested extremes, not the result of a block size sweep.

The consequence. The Token Reasoning Knowledge framework predicts a fundamental tradeoff: smaller blocks keep more context clean (smaller $K(c)$ , better training-inference alignment) but provide less data augmentation diversity, while larger blocks provide more diverse corruption patterns but risk pushing training contexts into the correlation or noise regimes. The optimal block size depends on the empirical balance between these competing effects, which almost certainly varies with model scale, training data composition, and target task distribution. A practitioner attempting to replicate this work for a different model family, a different code corpus (with different language distributions and example lengths), or a different domain entirely has no empirical guidance for choosing block size. Using block size 4 because the paper used it—without understanding whether it is near-optimal or merely adequate—risks leaving substantial performance on the table.

What evidence exists in the paper. Figure 2 shows the 2.5B-scale curriculum comparison across block sizes 32, 2, and 1, but only reports training dynamics (benchmark-averaged scores vs. training steps), not final downstream benchmark results. The 8B-scale evaluation uses only block size 4 with no comparison to other block sizes. Section 3.3 provides an analytical derivation of the empty-block problem and the clipped noise schedule solution, but this analysis addresses wasted compute (the fraction of training steps with zero supervision), not the quality of the supervision signal at different block sizes. Whether block sizes 2, 8, or 16 would yield better or worse final model quality than block size 4 is an open empirical question that the paper does not address.

Mitigation status. Not addressed. The paper does not claim to have identified the optimal block size or to have performed a block size sweep. It presents block size 4 as a design choice and does not discuss sensitivity to this hyperparameter. Future work section does not mention block size optimization.

6.2 Difficulty Estimation Cost Is Not Amortized, and Training Instability Ablations Are Incomplete

Wait — I need to re-read this. The limitation here should be about the warmup and training stability, not difficulty estimation. Let me reconsider.

Let me reconsider which limitations are most consequential for a practitioner. The block size issue above is one. Let me think through the others systematically.

Most consequential limitations for practitioners:

Block size choice unvalidated at scale (covered above)
Single model family, single dataset, single domain — all experiments use Seed-Coder architecture, Seed-Coder data, code domain. No evidence this transfers.
The SFT stage partially erases the multilingual data augmentation benefit — the paper's own Table 6 shows that the +16.5 C# and +10.5 PHP gains on MultiPL-E base largely disappear after SFT on MBXP. This is a practical deployment problem.
No inference efficiency data — the paper claims diffusion training improves quality "before factoring in any potential inference-time speed advantages" but does not report whether block diffusion inference is slower, faster, or comparable to AR. Practitioners need the quality-speed tradeoff.
Absence of uncertainty quantification and training variance — single training runs, point estimates without confidence intervals, small test sets. The 1-3 point differences on some benchmarks may be noise.
The AR counterpart baseline may be suboptimal — Stable-DiffCoder is compared against Seed-Coder, but other AR models (Qwen2.5-Coder-7B, Qwen3-8B) sometimes outperform Stable-DiffCoder. The claim that "diffusion training improves quality over AR" is relative to one specific AR model.

Let me select 4-6 from these that would matter most to a practitioner and structure them properly.

6.1 All Positive Results Come from a Single Model Family, Single Code Corpus, and Single Domain—Transfer to Other Settings Is Untested

The assumption or constraint. Every experiment in the paper uses the Seed-Coder architecture (a specific autoregressive transformer configuration), the Seed-Coder training data pipeline (1.3T tokens of code, compressed via subsampling from a multi-stage CPT corpus), the Seed-Coder SFT dataset, and the code domain exclusively. The Token Reasoning Knowledge framework (Section 3.1.1) is presented as a general theory of when masked prediction is useful for learning, and the warmup procedure (Section 3.2) and block-wise clipped noise schedule (Section 3.3) are designed as general methods. Yet the paper's conclusion acknowledges: "Since Stable-DiffCoder is primarily focused on the code domain and lacks large-scale training data from other areas, its performance on mathematical reasoning and general-purpose text tasks may be relatively limited. Whether text diffusion sampling can provide even greater benefits in broader domains remains an open question" (Section 5).

The consequence. A practitioner working in a different domain (mathematical reasoning, natural language understanding, multilingual text generation) or with a different model family (non-Seed-Coder architectures, different parameter scales, different pretraining data mixes) has no evidence that the Stable-DiffCoder recipe will improve model quality. Several domain-specific factors could change the effectiveness of the approach: (a) In natural language, the "inherently autoregressive structure" that the paper identifies as favoring left-to-right training may be even stronger than in code, reducing the any-order modeling benefit. (b) In mathematical reasoning, the clean causal chains that make the reasoning regime accessible to AR training (as in Equation 8) may be even more important, and the correlation/noise regime contexts from diffusion masking may be even more harmful. (c) Different model architectures with different attention patterns, different tokenization, or different pretraining objectives may interact differently with the AR→DLLM transition and the corruption warmup. The paper provides a theoretical framework that could predict transfer behavior, but no empirical evidence that it actually transfers.

What evidence exists in the paper. All evidence is from the code domain using PaLM-style decoder-only transformers within the Seed-Coder family. Table 1 through Table 9 cover diverse code tasks but no non-code benchmarks. The 2.5B-scale curriculum experiments (Figure 2) use the same Seed-Coder architecture and code data, not a different domain. The warmup procedure (Figure 4) is validated on BidLLM and BlockDLLM variants of the same architecture, not on different model families. The paper does not claim that its recipe generalizes—the limitation is acknowledged honestly in the conclusion—but also does not provide the evidence that would let a practitioner assess whether to apply this recipe to their own domain.

Mitigation status. The paper partially acknowledges this in Section 5: "Whether text diffusion sampling can provide even greater benefits in broader domains remains an open question, requiring future model iterations and deeper empirical exploration." However, it frames this as a question about how much benefit exists in other domains, not about whether the method works at all—which is a stronger claim than the evidence supports. The theoretical framework (Token Reasoning Knowledge, Section 3.1.1) is domain-agnostic and provides a basis for predicting transfer, but the predictions are not tested. A practitioner would need to run their own controlled experiments—ideally at the 2.5B scale as the paper does—before committing to a full-scale diffusion training run in a new domain.

6.2 The Supervised Fine-Tuning Stage Partially Overwrites the Multilingual Data Augmentation Gains—the Base Model Advantage Is Fragile to SFT Data Distribution

The assumption or constraint. The paper's central claim about data augmentation benefiting low-resource languages rests on the MultiPL-E base model results (Table 2), where Stable-DiffCoder-8B-Base achieves gains of +16.5 points in C# (70.3% vs. 53.8%), +10.5 points in PHP (74.4% vs. 63.9%), and +5.1 points in Bash (53.2% vs. 48.1%) over Seed-Coder-8B-Base. The authors explicitly attribute this to "diffusion-style stochastic sampling" that "amplifies learning signals from low-resource code" (Section 4.3.1). However, the SFT stage fully reuses "the original Seed-Coder SFT dataset" (Section 4.1) with no modification to balance language representation. The MBXP instruction-tuned results (Table 6) tell a different story: the C# advantage essentially vanishes (71.2% for Stable-DiffCoder vs. 74.2% for Seed-Coder—a 3.0-point loss), PHP shows a marginal gain (73.8% vs. 74.7%, −0.9 points), and most languages show small mixed differences that average to zero (both models at 75.3% overall).

The consequence. For a practitioner deploying a code model for multilingual code generation in production, the base model advantage is largely irrelevant—what matters is instruction-tuned performance. The paper's results show that the SFT stage, when applied as-is with no special handling of low-resource languages, largely erases the multilingual gains from diffusion training. The root cause is identified by the paper: "Due to the need to extensively supplement the scarce data such as C# and PHP that are lacking in pretraining during SFT, the advantage in multilingual coding capabilities has been reduced" (Section 4.4.1). This is a fundamental tension: the languages that benefit most from diffusion's data augmentation during pretraining are precisely those with the least SFT data, and the SFT process—which teaches instruction-following on a distribution skewed toward high-resource languages—can overwrite the hard-won multilingual improvements.

This fragility has broader implications. If a practitioner wants to preserve the diffusion training gains for low-resource languages, they would need to either: (a) curate a balanced multilingual SFT dataset (expensive, requiring human annotation or high-quality synthetic data in the low-resource languages), (b) modify the SFT training procedure to prevent catastrophic forgetting of low-resource language capabilities (e.g., by mixing in pretraining data during SFT, or by using lower learning rates for parameters most important to low-resource languages), or (c) deploy the base model directly for low-resource languages and the instruct model for high-resource languages (complex deployment architecture). None of these solutions is evaluated in the paper, and none is trivial to implement.

What evidence exists in the paper. The MultiPL-E base results (Table 2) show large gains; the MBXP instruct results (Table 6) show neutral or slightly negative results for most languages with the average tied at 75.3%. The authors acknowledge this in Section 4.4.1 with the quote above. No ablation tests whether a language-balanced SFT dataset would preserve the multilingual gains. No experiment varies the SFT data distribution or SFT training hyperparameters to measure sensitivity. The NaturalCodeBench Chinese Python result (−4.3 points, Table 7) is a related data point—another case where the instruct model underperforms on a specific language-region split, though it's unclear whether this is also an SFT-overwriting effect or a separate failure mode.

Mitigation status. The paper acknowledges the phenomenon explicitly (Section 4.4.1) but treats it as an observation rather than a problem to solve. There is no suggestion of how to preserve the base model gains through SFT, and the future work section does not mention it. For the practitioner, the key takeaway is that the diffusion training multilingual advantage is real but fragile—deploying it effectively requires solving the SFT data imbalance problem, which the paper does not address.

6.3 Inference Latency and Throughput—the Core Practical Tradeoff—Are Completely Unmeasured; the Paper Only Evaluates Quality

The assumption or constraint. The paper's abstract states that Stable-DiffCoder demonstrates "diffusion-based training can improve code modeling quality beyond AR training alone," and the introduction adds: "before factoring in any potential inference-time speed advantages" (Section 1). The experiments section evaluates only model quality (pass@1 accuracy across benchmarks) and contains no measurements of inference latency, throughput, memory usage, or any other deployment-relevant efficiency metric. The block size of 4 implies that Stable-DiffCoder generates 4 tokens per denoising step—a theoretical 4× speedup ceiling over AR single-token generation—but diffusion decoding typically requires multiple denoising steps per block (the paper does not specify how many), so the realized speedup could be much less than 4×, could be ~1× (parity with AR), or could even be slower if many denoising iterations are needed per block.

The consequence. For a practitioner deciding whether to adopt Stable-DiffCoder for a production code assistant, inference latency is arguably the most important metric after quality. Users expect sub-second response times for code completion and interactive editing; a model that achieves +9.5 points on CanItEdit but takes 5 seconds per response versus an AR model taking 1 second may be unacceptable regardless of quality. The paper's deliberate separation of quality and speed creates a gap: it demonstrates that diffusion training can improve quality, but it does not tell the practitioner whether the quality improvement is worth whatever latency penalty (or benefit) the diffusion decoding process imposes. The tradeoff could range from "strictly better—higher quality AND lower latency" (if block diffusion with few denoising steps is faster than AR) to "higher quality but unacceptable latency for interactive use" (if many denoising iterations are needed), and the paper provides zero information to distinguish these regimes.

This omission is particularly significant because the block diffusion decoding process is non-trivial. At inference, the model must: (1) take the existing prefix as clean context, (2) predict a block of 4 tokens simultaneously, (3) potentially run multiple denoising iterations (re-masking and re-predicting within the block) to refine the output, (4) accept or reject the block based on some confidence criterion, (5) repeat for subsequent blocks. The number of forward passes per generated token could range from ~1 (with aggressive schedules and few refinement steps) to ~10 or more (with conservative schedules), making the throughput anywhere from 4× better to 2.5× worse than AR. Comparable DLLM systems like Mercury Coder and Gemini Diffusion emphasize inference speed as a primary contribution, providing extensive latency benchmarks. Stable-DiffCoder provides none.

What evidence exists in the paper. None. The paper contains no inference latency numbers, no throughput measurements, no discussion of decoding schedules, no comparison of wall-clock time to AR baselines, no memory footprint analysis. The block size of 4 is mentioned as a training configuration (Section 4.1) but the number of denoising steps used at inference is never specified. The evaluation protocol for all benchmarks is pass@1 with a single generated response, with no measurement of generation time.

Mitigation status. Not addressed. The paper explicitly states it is evaluating quality "before factoring in any potential inference-time speed advantages" (Section 1), framing the speed question as orthogonal to the quality question. This is a valid research strategy—first establish that the training method works, then optimize inference separately—but it means the paper provides no actionable guidance for deployment. A practitioner must either run their own inference benchmarks or wait for follow-up work that characterizes the quality-speed Pareto frontier for block diffusion decoding at block size 4. The project page (linked in the paper header) may contain inference benchmarks, but the paper itself does not.

6.4 All Results Are Single-Run Point Estimates on Small-to-Medium Test Sets—There Is No Uncertainty Quantification, and Some Claimed Gains May Be Sampling Noise

The assumption or constraint. Every benchmark result in Tables 1–9 is reported as a single pass@1 percentage with no confidence intervals, no error bars, no standard deviations, and no mention of training variance (multiple runs with different random seeds). The test sets range from 105 problems (CanItEdit) to approximately 1,140 problems (BigCodeBench Full), with many intermediate sizes: 164 problems (HumanEval), 399 problems (MBPP EvalPlus subset), 800 problems (CRUXEval), 402 problems (NaturalCodeBench), 133 problems (Aider), and an unspecified number for MHPP (obtained through an official submission interface). For a test set of 105 problems, each problem corresponds to approximately 0.95 percentage points—so a 1.8-point difference on HumanEval (164 problems) represents roughly 3 more correct answers. Whether 3 additional correct answers out of 164 is statistically significant or sampling noise depends on the variance of the estimator, which the paper does not provide.

The consequence. Several of the paper's claimed improvements are small enough that they cannot be confidently distinguished from noise. The HumanEval base model gain of +1.9 points (79.3% vs. 77.4%) represents approximately 3 additional correct solutions out of 164. The MBPP instruct gain of +0.5 points (85.7% vs. 85.2%) represents approximately 2 additional correct solutions out of 399. The CRUXEval Input-CoT instruct loss of −1.2 points (62.1% vs. 63.3%) represents approximately 10 out of 800. None of these differences can be reliably interpreted without knowing the variance of the pass@1 estimator on these test sets. The paper treats all differences as meaningful—using phrases like "significantly improves" and "consistently outperforms"—without providing the statistical tools to assess whether they actually are.

This problem is compounded by two factors. First, the paper reports results from a single training run. Training neural networks at 8B scale on 1.3T tokens involves substantial randomness from data ordering, dropout, and initialization (even when starting from a pretrained checkpoint, the AR CPT and DLLM CPT stages introduce new randomness). Without multiple training seeds, the variance of the training process itself is unknown. The observed 1–3 point differences between Stable-DiffCoder and Seed-Coder could reflect genuine algorithmic improvement, or they could reflect one training run happening to land in a slightly better basin. Second, the paper does not disclose per-split sample sizes for benchmarks like NaturalCodeBench (split by language × region × problem type), making it impossible even for a reader to compute approximate confidence intervals.

Larger gains—+9.5 points on CanItEdit, +6.2 points on MHPP, +5.4 points on BigCodeBench Hard, +16.5 points on C# MultiPL-E—are more robust because they substantially exceed plausible sampling noise at the reported test set sizes. But for the marginal results that constitute a significant fraction of the paper's claimed improvements (particularly on HumanEval, MBPP, and CRUXEval), the evidence is weaker than the presentation suggests.

What evidence exists in the paper. All benchmark tables report single-number pass@1 percentages with no dispersion measures. The paper does not state how many training runs were performed, whether results are averages or best-of-N, or whether any statistical testing was conducted. The MHPP benchmark uses an official submission interface (Section 4.4.1) that limits submissions, which may restrict the ability to collect multiple measurements. The LiveCodeBench v5 uses a fixed time window (2024.10–2025.02), providing a form of time-based holdout but not repeated measurement. The 2.5B-scale curriculum comparison (Figure 2) reports "average performance across multiple code benchmarks" at each training step, suggesting some aggregation, but no error bars or confidence bands are shown for the training curves.

Mitigation status. Not addressed. The paper does not discuss measurement uncertainty, training variance, or the reliability of small differences on small test sets. This is standard practice in the LLM evaluation literature—most papers report point estimates without confidence intervals—but it is a limitation for a paper whose central claim is that diffusion training improves model quality, with improvements that are often in the 1–3 point range. A practitioner seeing a 1.8-point improvement on HumanEval needs to know whether this is a reliable gain worth the additional training complexity (AR CPT + warmup + DLLM CPT vs. AR-only) or whether it could disappear in a rerun with a different random seed.

6.5 The Comparison to the AR Counterpart Is Strong but Leaves an Important Question Open: How Much of the Gain Comes from Longer Training vs. the Diffusion Objective?

The assumption or constraint. The controlled comparison between Stable-DiffCoder and Seed-Coder holds architecture, data, and SFT constant—an excellent experimental design. However, the training procedures differ in two ways simultaneously: (1) the training objective (next-token prediction vs. block diffusion) and (2) the number of parameter updates. The AR-only pipeline processes each token once with a single next-token prediction loss. The AR+DLLM pipeline processes each token twice—once during AR CPT (next-token prediction) and once during DLLM CPT (block diffusion). Even though both stages use the same 1.3T tokens, the DLLM CPT stage computes a loss on multiple masked positions per block (up to 4 tokens per block, with the clipped noise schedule guaranteeing at least 1), meaning the model receives more gradient updates per data pass than during AR training. The paper reports "under the same compute budget" (Section 3.1.2) for the 2.5B-scale experiments, but for the 8B-scale comparison, the compute budgets may not be identical—the DLLM stage uses bidirectional attention (more FLOPs per token than causal attention) and computes losses on multiple positions per training step.

The consequence. The observed quality improvements cannot be unambiguously attributed to the diffusion training objective. They could instead be partially or entirely explained by: (a) longer effective training (more parameter updates), (b) the regularization effect of training on noisy reconstruction targets (which is a different mechanism than the data-augmentation-from-diverse-corruptions that the paper emphasizes), or (c) the multi-token prediction aspect of block diffusion (predicting 4 tokens simultaneously vs. 1 token at a time in AR training), which is known to improve representation learning in other contexts. The paper's theoretical framework (Token Reasoning Knowledge, Section 3.1.1) makes specific predictions about why diffusion training should help—controlled data augmentation in the reasoning regime—but these predictions are not empirically separated from these alternative explanations.

For a practitioner, the practical question is: should I add a DLLM CPT stage to my training pipeline, or should I simply train my AR model for longer? If the gains come primarily from the diffusion objective and its data augmentation properties, then the DLLM stage is essential and the additional training complexity (warmup, noise schedule design, block size tuning) is justified. If the gains come primarily from longer training—more parameter updates on the same data—then a simpler AR-only approach (train for 2.6T tokens with next-token prediction, or train with a multi-token prediction auxiliary loss) might achieve similar quality without the complexity of the diffusion training infrastructure.

What evidence exists in the paper. The 2.5B-scale curriculum comparison (Figure 2, Section 3.1.2) compares different training curricula "under the same compute budget" for CPT, controlling for total FLOPs. Before CPT to BiDLLM, AR training outperforms BiDLLM training for block sizes 1 and 2—showing that for the same FLOPs, AR is more efficient at compressing new knowledge when evaluated with small-block decoding. After CPT, the AR→BiDLLM curriculum outperforms the BiDLLM-only curriculum, suggesting that the AR pre-training phase provides a benefit beyond just "more training." However, this comparison does not include an "AR-only for 2× longer" baseline—it doesn't test whether doubling the AR CPT duration would match or exceed the AR→BiDLLM combination. The 8B-scale results provide no ablation comparing AR-only for 2.6T tokens vs. AR+DLLM for 1.3T + 1.3T tokens. The paper reports that both stages use 1.3T tokens (Section 4.1) but does not report the total number of parameter updates, FLOPs, or wall-clock training time for each pipeline, making it impossible to determine whether the comparison is compute-matched.

Mitigation status. Not addressed. The paper does not include an "AR-only for longer" baseline, does not report total FLOPs or parameter updates for the compared pipelines, and does not discuss the possibility that the gains come from increased effective training rather than from the diffusion objective specifically. The theoretical framework provides a mechanism (data augmentation in the reasoning regime) but does not empirically rule out the alternative explanation. This limitation is compounded by the absence of the training variance data discussed in 6.4—without knowing the distribution of outcomes across random seeds, even a matched-compute comparison (same FLOPs, different objectives) would need multiple runs to reliably attribute differences to the objective rather than to training noise.

6.6 The Hardest Code Problems Remain Unsolved—Stable-DiffCoder's Gains Concentrate on Easy-to-Medium Difficulty and Do Not Extend to Competitive Programming Under Contamination Control

The assumption or constraint. The paper's most reliable benchmark for assessing genuine, non-memorized coding ability is LiveCodeBench v5 (Section 4.2): "To mitigate contamination and overfitting to static benchmarks, LiveCodeBench continuously collects time-stamped problems from competitive programming platforms, enabling evaluation within recent, user-specified time windows." On this benchmark, Stable-DiffCoder-8B-Instruct achieves 23.5%, slightly below Seed-Coder-8B-Instruct at 24.7% (Table 5). This is not an isolated negative result—it's the benchmark specifically designed to test the capability the model cannot fake through memorization. The contrast with the saturated benchmarks (HumanEval: 86.6%, MBPP: 85.7%) is stark: on tasks where the model may have seen similar solutions during training, diffusion training helps modestly; on genuinely novel competitive programming problems requiring algorithmic reasoning from first principles, diffusion training does not help and may slightly hurt.

A similar pattern appears in NaturalCodeBench Chinese Python (51.4% vs. 55.7%, Table 7), which tests practical software engineering from real user queries rather than algorithmic puzzle-solving. The paper does not provide enough detail to determine whether this is a difficulty effect, a language-specific effect, or noise, but combined with the LiveCodeBench result, it suggests that diffusion training's benefits may be concentrated on problems where the base model already has some capability (the "reasoning regime" in the paper's framework) and may not help—or may actively interfere—on problems requiring novel reasoning.

The consequence. For a practitioner deploying a code model for genuine software engineering assistance—where users ask novel questions that are unlikely to be memorized from training data—the LiveCodeBench and NaturalCodeBench results are more informative than HumanEval or MBPP. HumanEval, in particular, has been in the training data of most code LLMs (either directly or through near-duplicates) and measures memorization as much as reasoning. The paper's strongest claims about capability improvement—+9.5 points on CanItEdit, +6.2 points on MHPP, +5.4 points on BigCodeBench Hard, large multilingual gains on MultiPL-E base—all come from benchmarks with fixed, static test sets. The one benchmark explicitly designed to be contamination-resistant shows no improvement. This creates a credibility gap: the paper cannot distinguish between "diffusion training improves genuine code reasoning" and "diffusion training improves the model's ability to memorize and reproduce patterns from training data, which happens to include near-duplicates of many static benchmark problems."

The pattern is consistent with the Token Reasoning Knowledge framework, though the paper does not make this connection explicitly. The framework predicts that diffusion training helps in the reasoning regime (K(c) small, context strongly constrains answer) but not in the correlation or noise regimes. Static benchmarks, by virtue of being old and widely distributed, are more likely to have solutions in the training data—the model may have seen similar problems and can rely on memorized patterns. Novel competitive programming problems, by design, require genuine reasoning from first principles—the model cannot rely on memorization and must synthesize a solution, which is a harder task where the base model's capability may be insufficient regardless of training objective. If this interpretation is correct, diffusion training improves pattern matching and data utilization but does not fundamentally expand the frontier of what the model can reason about—consistent with the paper's own framework where the hardest problems (difficulty "bin 5" in the compute-optimal scaling literature) show near-zero improvement regardless of method.

What evidence exists in the paper. The LiveCodeBench v5 result (Table 5) shows Stable-DiffCoder at 23.5% vs. Seed-Coder at 24.7% (−1.2 points). The paper notes this as a case where "Stable-DiffCoder-8B-Instruct (23.5%) is slightly behind Seed-Coder-8B-Instruct (24.7%)" (Section 4.4.1) but does not analyze it as a systematic pattern or connect it to the theoretical framework. The NaturalCodeBench Chinese Python result (51.4% vs. 55.7%, −4.3 points) is noted without explanation. All other reported benchmarks are static. The paper does not perform a difficulty-stratified analysis (e.g., binning LiveCodeBench problems by difficulty or by the base model's pass@1 rate) that would test whether the diffusion training benefit varies with problem difficulty.

Mitigation status. Not addressed. The paper does not acknowledge the tension between its strong results on static benchmarks and its neutral/negative results on the contamination-resistant benchmark, does not analyze difficulty as a moderating variable, and does not discuss the possibility that diffusion training primarily improves memorization and pattern matching rather than genuine reasoning. The theoretical framework (Section 3.1.1) provides tools to think about this—by analyzing which benchmarks fall into the reasoning vs. correlation regimes for the base model—but the paper does not apply this analysis to interpret the LiveCodeBench result. For a practitioner, the key uncertainty is whether Stable-DiffCoder's quality advantage holds for the actual use case (novel user queries) or only for benchmarks that may overlap with training data. The paper provides no evidence to resolve this.

7. Implications and Future Directions

How This Work Changes the Landscape

Reframing diffusion training as a controlled curriculum problem rather than a masking-sampling problem. Before Stable-DiffCoder, the DLLM field operated under an implicit assumption that more masking variety equals better training. The standard recipe—sample corruption levels uniformly from [0, 1], apply random masking across the full sequence or large blocks, train the denoiser on whatever patterns arise—treated diffusion as a statistical process to be sampled from, not a pedagogical intervention to be designed. This paper dismantles that assumption by providing both a theoretical diagnostic (the Token Reasoning Knowledge taxonomy of reasoning/correlation/noise regimes, Section 3.1.1) and an empirical demonstration that not all mask patterns are useful, and many are actively harmful. The concrete counterexample of Equations 6–8—where aggressive masking teaches the model to predict 7 from context that only shows a=1, b=2—makes this harm tangible in a way that prior DLLM work never did.

This is not a paradigm shift in the sense of replacing autoregressive models with diffusion models. It is a reframing of the diffusion training problem itself: from "how do we build a better denoiser?" to "which corruption patterns teach reasoning rules, and how do we design a curriculum that stays in those regimes?" The consequences are methodological, not architectural. The paper does not propose a new attention mechanism, a new tokenizer, or a new loss function—it proposes a training curriculum (AR first for compression, then small-block diffusion for augmentation, with corruption warmup and clipped noise schedules) that existing architectures can adopt. This makes the contribution more portable than a novel architecture would be, but also means the quality of any given DLLM depends on training design choices that prior work treated as implementation details.

Resolving contradictions in the DLLM literature. The paper provides a unifying explanation for two patterns that previously seemed contradictory: some work found DLLMs to be "super data learners" (Ni et al., 2025; Gao et al., 2025), benefiting from repeated corrupted views of the same data, while other work found DLLMs learn more slowly per token than AR models and struggle to close the quality gap (Sun et al., 2025; von Rütte et al., 2025). The Token Reasoning Knowledge framework resolves this: DLLMs are super data learners when they operate in the reasoning regime (small K(c), clean evidence, aligned training-inference contexts), but standard training procedures push a large fraction of steps into the correlation/noise regimes where gradients are weak or contradictory. The same model, trained on the same data, can be either super-efficient or super-wasteful depending on the corruption schedule. This means prior work that found DLLM training to be inefficient was not wrong about the observation but wrong about the cause—the fault lies not with the diffusion objective per se but with the specific corruption schedules that push training into noise regimes. Stable-DiffCoder demonstrates that the same objective, with controlled corruption, can be more efficient than AR training on many benchmarks.

This resolution has practical bite: it tells the field that abandoning diffusion training because prior DLLMs underperformed AR models is premature. The problem is solvable through curriculum design, not through fundamentally new architectures. Research attention should shift from "how do we make diffusion decoding faster?" (the focus of Mercury Coder, Gemini Diffusion, Fast-dLLMv2) to "how do we design corruption schedules that maximize time in the reasoning regime?"—a training-centric rather than inference-centric research program.

Shifting the value proposition of DLLMs from speed to quality. The most counterintuitive implication of this paper is that diffusion training may be the right choice even when inference latency matters more than throughput. The DLLM literature has been organized around the hypothesis that diffusion's value is parallel decoding speed—you accept a quality tradeoff because you get lower latency. Stable-DiffCoder inverts this: you get better quality, and speed is a separate optimization you can pursue independently. If a practitioner can achieve +9.5 points on CanItEdit (60.0% vs. 50.5%, Table 9) or +6.2 points on MHPP (42.4% vs. 36.2%, Table 5) by switching from AR-only training to an AR+difffusion training pipeline—even if inference latency is identical—the decision is clear. Diffusion training becomes a training-time investment that pays off in model quality, with inference speed as a potential bonus rather than the primary motivation.

This reframing changes which research questions matter. If diffusion is primarily a speed play, the key questions are: how few denoising steps can we use? How do we optimize the decoding schedule? How do we build efficient KV-cache implementations for block diffusion? If diffusion is primarily a quality play, the key questions become: how do we design better corruption curricula? Which block sizes are optimal for which domains? How do we preserve diffusion-trained capabilities through SFT? Can we extend the three-stage curriculum to other modalities? The paper's contribution pushes the field toward the second set of questions.

Establishing that the data augmentation benefit is largest where data is scarcest. The MultiPL-E results (Table 2) provide the first concrete evidence that diffusion training's data augmentation effect is not uniform—it disproportionately benefits low-resource programming languages. The gains of +16.5 points in C# and +10.5 points in PHP, compared to modest or negative results for high-resource languages like Python (+3.1) and JavaScript (−4.5), establish a clear pattern: diffusion training extracts more value from scarce data than from abundant data. This has immediate practical implications for any domain with a long tail of low-resource subdomains—not just programming languages, but also rare APIs, specialized libraries, domain-specific file formats, and uncommon coding patterns.

The mechanism is clear from the Token Reasoning Knowledge framework: in low-resource settings, each training example is precious, and diffusion's diverse corruption trajectories effectively create synthetic training examples by masking and reconstructing those rare patterns. In high-resource settings, the model already sees enough clean examples to learn the relevant patterns, so the augmented views add less marginal value and may even introduce noise if the corrupted contexts fall into the correlation regime. This implies that the optimal corruption curriculum may differ by data frequency—rare examples might benefit from higher corruption diversity (more mask patterns, more denoising steps per example), while common examples might need little or no augmentation. The paper doesn't explore this frequency-conditioned curriculum, but the results make it an obvious next step.

Making AR→DLLM transition a solved sub-problem. Prior to this paper, the instability of AR→DLLM continual pretraining was a known barrier (Gong et al., 2024), and the proposed solutions involved attention mask annealing—a complex intervention incompatible with optimized kernels. The corruption warmup in Stable-DiffCoder (Section 3.2, Figure 4) demonstrates that the transition can be stabilized by the much simpler mechanism of starting with near-zero corruption and linearly increasing the mask rate. The gradient norm spikes from ~10⁵ (without warmup) to staying within a factor of 10 of AR training (with warmup). This means the field no longer needs to treat the AR→DLLM boundary as a dangerous cliff—it's a bridge that can be crossed safely with a standard curriculum technique. This lowers the barrier to entry for teams that already have strong AR models and want to experiment with diffusion training: they can start from their existing checkpoint, apply the warmup, and add a DLLM CPT stage without fear of catastrophic forgetting.

Follow-Up Research This Work Enables

Frequency-conditioned corruption curricula. The paper's MultiPL-E results show that diffusion training's data augmentation benefit is concentrated in low-resource languages, suggesting that the value of diverse corruption patterns depends on how many clean examples the model sees for a given pattern. A natural extension is a frequency-conditioned corruption schedule: for rare examples (identified by n-gram frequency, subdomain classification, or embedding clustering), apply more diverse masking patterns, larger effective block sizes, or more denoising steps per training epoch. For common examples, use minimal corruption—perhaps just the block-wise clipped baseline—to avoid diluting the training signal with unnecessary augmentation. A controlled experiment would compare uniform corruption schedules against frequency-conditioned schedules on MultiPL-E, with the prediction that frequency-conditioning amplifies the low-resource gains while recovering the small losses seen in high-resource languages (e.g., the −4.5 points in JavaScript for the base model, Table 2). This would directly test whether it's the quantity or the distribution of augmentation that matters.

Preserving diffusion-trained capabilities through SFT. The paper's own results in Section 4.4.1 document a critical fragility: the +16.5-point C# gain on MultiPL-E base models largely disappears after SFT on the Seed-Coder dataset (Table 6 shows Stable-DiffCoder at 71.2% vs. Seed-Coder at 74.2% on C#—a regression). The authors attribute this to the SFT dataset's language imbalance, but they don't propose a solution. A high-priority follow-up would design and evaluate SFT strategies that preserve the data-augmentation gains: (1) mixing in 5–20% of pretraining data during SFT to prevent catastrophic forgetting of low-resource languages, (2) applying language-specific learning rate multipliers (lower rates for parameters important to low-resource languages, identified through gradient-based pruning or Fisher information), (3) using a multi-task SFT objective that jointly optimizes instruction-following and MLM-style reconstruction on low-resource language examples, or (4) curating a language-balanced SFT subset that oversamples low-resource language examples. The metric would be the gap between base model multilingual gains and instruct model multilingual gains—a smaller gap means better preservation. This is a concrete engineering problem with measurable success criteria, and the paper provides a baseline (the gap between Table 2 and Table 6) against which interventions can be measured.

Diffusion training for other structured-output domains with edit-like task structure. The CanItEdit result (+9.5 points, 60.0% vs. 50.5%, Table 9) is the paper's single largest gain and the strongest evidence for the hypothesis that "random masking and reconstruction inherently train the model on edit- and infill-like patterns" (Section 4.4.3). This hypothesis predicts that diffusion training should be particularly effective for any domain where the inference task involves modifying existing structured outputs: code refactoring, document revision, data transformation, configuration file editing, patch generation, or structured data cleaning. A strong follow-up would evaluate Stable-DiffCoder-style training on: (1) the SWE-bench dataset (real-world GitHub issue resolution requiring multi-file edits), (2) spreadsheet formula repair, (3) JSON/XML transformation tasks, and (4) diff-based code review suggestion. The prediction is that the gap between diffusion-trained and AR-trained models should correlate with how closely the task resembles span-replacement: larger gaps for tasks where the correct output is a local edit of the input, smaller gaps for tasks requiring generation from scratch. This would test whether the edit-like benefit is specific to code or a general property of the diffusion objective.

Scaling analysis: does the diffusion training advantage grow, shrink, or saturate with model size? All experiments in this paper are at 8B scale (with preliminary analysis at 2.5B). The DLLM scaling literature (von Rütte et al., 2025) has found that the efficiency gap between AR and diffusion training narrows at larger scales—the token efficiency disadvantage of DLLMs becomes less severe as models grow—but this prior work used standard (uncontrolled) masking rather than the small-block, reasoning-regime curriculum of Stable-DiffCoder. A critical open question is whether the Stable-DiffCoder recipe's advantage over AR-only training scales favorably (grows with model size, because larger models can better exploit the augmented data), scales neutrally (constant advantage in percentage points across scales), or scales unfavorably (shrinks with model size because AR training catches up). A scaling study at 1B, 4B, 16B, and 70B parameters, with the same data and controlled AR-vs-diffusion comparison, would answer this. In particular, if the advantage grows with scale, it implies that the largest models—where pretraining costs dominate—would benefit most from the improved training efficiency, making the case for adopting diffusion training strongest at the frontier.

Verifier-guided block acceptance for quality-speed Pareto optimization. The paper deliberately separates quality from speed, but the two interact at inference: when generating a block of 4 tokens, the model could run multiple denoising iterations (re-masking and re-predicting within the block) to improve quality, at the cost of additional forward passes. A natural extension is to train a lightweight block confidence estimator (or reuse the PRM-style step-level verifier architecture from other work) that predicts whether a generated block is likely correct, and to adaptively decide how many denoising iterations to run: accept immediately if confidence is high, refine further if confidence is moderate, resample from scratch if confidence is low. This would create a quality-speed Pareto frontier where the user can trade off accuracy for latency at inference time by adjusting a confidence threshold, without any model changes. The paper's block diffusion architecture (block size 4, bidirectional attention) is well-suited for this because each block is generated independently given clean left context, making confidence estimation for a block a well-defined binary classification problem. The evaluation would measure the pass@1 vs. latency curve across different confidence thresholds on LiveCodeBench and BigCodeBench, with the prediction that adaptive block acceptance achieves most of the quality gain of aggressive denoising at a fraction of the latency cost.

Adversarial evaluation of the Token Reasoning Knowledge taxonomy. The paper's central theoretical contribution is the division of training contexts into reasoning, correlation, and noise regimes based on K(c)—the size of the candidate set given the unmasked context. This taxonomy is presented conceptually with one motivating example (Equations 6–8) but is never measured empirically. A rigorous follow-up would operationalize K(c) for a trained model: for a set of test examples with known ground truth, apply different mask patterns, measure the entropy of the model's predictive distribution over the vocabulary at each masked position, and classify each (context, target) pair into reasoning (low entropy, correct prediction), correlation (moderate entropy, multiple plausible tokens), or noise (high entropy, near-uniform distribution). The prediction is that training steps classified as "reasoning" should show faster loss reduction and better downstream transfer than steps classified as "correlation" or "noise," and that the Stable-DiffCoder curriculum (small blocks, AR-first) should produce a higher fraction of reasoning-regime steps than a standard fully bidirectional DLLM training run. This would convert the paper's qualitative diagnostic into a quantitative tool for evaluating and optimizing corruption schedules, and would directly test whether the framework's predictions about learning efficiency hold when K(c) is measured rather than intuited.

Practical Applications and Downstream Use Cases

Multilingual code generation for underserved programming languages. The MultiPL-E base model results (Table 2) show gains of +16.5 points in C# and +10.5 points in PHP over the AR counterpart. For an organization building a code assistant that needs to support a wide range of languages—including those with limited training data in web-scale corpora—the Stable-DiffCoder training recipe offers a concrete path to better low-resource language performance without collecting additional data. A deployment-viable version would combine: (1) the AR→block-diffusion training pipeline (reusing the organization's existing AR code model and data), (2) a deliberately language-balanced SFT dataset that oversamples low-resource languages to preserve the base model gains (addressing the fragility documented in the MBXP instruct results, Table 6), and (3) frequency-conditioned corruption during DLLM CPT (giving rare-language examples more diverse mask patterns). The expected benefit is a 5–15 point improvement in pass@1 for languages with limited training data, with little to no regression in high-resource languages.

Code editing and refactoring assistants. The CanItEdit result (60.0% vs. 50.5%, Table 9) and the BigCodeBench Hard result (31.8% vs. 26.4%, Table 5) are the most actionable findings for a production code editing tool. A 9.5-point improvement in edit success rate—roughly one additional successful edit for every ten attempts—is directly user-visible: it means fewer manual corrections, less frustration, and higher trust in automated suggestions. For an IDE plugin or code review tool that makes edit suggestions on existing code, Stable-DiffCoder-style training provides a quality improvement that does not require changes to the inference architecture, the tokenizer, or the deployment pipeline—only the training recipe. The main practical question is whether the SFT dataset needs to be augmented with additional editing examples (the Seed-Coder SFT dataset, which the paper reuses, was not designed specifically for editing tasks), or whether the base diffusion training's implicit edit-pattern learning is sufficient given standard instruction-following data. An organization could test this by fine-tuning the Stable-DiffCoder base model on an editing-specific SFT dataset (e.g., the CanItEdit training set, or synthetic edit pairs generated from version control diffs) and measuring whether the editing advantage grows further or plateaus.

Training data efficiency for organizations with limited pretraining budgets. The paper's core result—that the AR+diffusion pipeline extracts more capability from the same 1.3T tokens than AR-only training—translates directly to dollar savings. If an organization has a fixed data budget (e.g., they can only afford to collect and clean 1T tokens of domain-specific data), the Stable-DiffCoder recipe offers a way to get more model quality per training token. The tradeoff is additional training complexity (implementing the corruption warmup, block-wise noise schedule, and DLLM CPT stage) and potentially longer wall-clock training time (the DLLM CPT processes each token twice—once in AR mode, once in diffusion mode—and bidirectional attention is more FLOPs-intensive than causal attention). For organizations where pretraining data is expensive to acquire (e.g., proprietary codebases, specialized scientific corpora, low-resource natural languages) but compute is relatively cheap, this is an attractive bargain: invest more compute per token to extract more value from scarce data. The specific benefit depends on the domain's similarity to code, but the paper's theoretical framework (reasoning regime vs. noise regime) predicts that the method should transfer to any domain with high-quality, structured data where clean context strongly constrains the answer—scientific papers, legal documents, formal specifications, mathematical proofs.

Self-improvement data generation pipelines. When using LLMs to generate training data for themselves—e.g., generating solutions to coding problems, filtering for correctness via test execution, and fine-tuning on the correct solutions—the quality of the generated data matters enormously. The Stable-DiffCoder results suggest an improved data generation strategy: train the generator model with the AR+diffusion recipe first, then use it to generate solutions. The diffusion-trained model's advantages on editing (+9.5 on CanItEdit), hard problems (+5.4 on BigCodeBench Hard), and low-resource patterns (+16.5 on C#) mean it should produce (1) more diverse correct solutions for hard problems, and (2) more correct solutions for low-resource languages where correct examples are scarce. These high-quality synthetic examples, when fed back into the training pipeline for the next iteration, could create a virtuous cycle: better diffusion training → better solutions → better fine-tuning data → even better model. The paper's finding that SFT partially overwrites the multilingual advantage (Table 6) suggests that such self-improvement loops would need to explicitly preserve language diversity, perhaps by generating and training on solutions in multiple languages rather than defaulting to Python-dominant generation.

When to Prefer This Method

The paper does not provide an explicit tradeoff analysis against named alternatives (e.g., "use AR training when X, use diffusion training when Y"), and the comparison against the AR counterpart (Seed-Coder) is a controlled experiment rather than a deployment decision framework. The paper's contribution is a training methodology, not a family of models with different cost-quality profiles. The decision to adopt Stable-DiffCoder-style training is therefore not a selection among alternatives at inference time but a training pipeline decision—a question of whether to add the DLLM CPT stage to an existing AR training pipeline. The paper does not articulate conditions under which a practitioner should not adopt this recipe, beyond the domain limitation acknowledged in the conclusion (code-focused data, no evidence for math or general text). No decision matrix is forced.