DIFFA-2: A Practical Diffusion Large Language Model for General Audio Understanding

🎯 Pitch

A diffusion-based audio language model, DIFFA-2, matches top autoregressive systems on general audio benchmarks while using only 1.1% trainable parameters and fully open-source data. Critically, it achieves this by treating speech and sound as iterative denoising problems, demonstrating that next-token prediction is not a prerequisite for competitive multimodal understanding.

1. Executive Summary

This paper introduces DIFFA-2, a practical diffusion-based large audio language model (dLLM) for general audio understanding that transforms the prior DIFFA proof-of-concept into a competitive system through comprehensive upgrades. The model is built around a frozen Whisper-Large-V3 encoder, a dual-adapter audio interface (a semantic adapter for content-aligned features and an acoustic Q-former adapter for paralinguistic cues), and an LLaDA-8B-Instruct diffusion backbone trained with a progressive four-stage training curriculum (semantic alignment via ASR, joint semantic–acoustic alignment, LoRA-based backbone fine-tuning, and variance-reduced preference optimization). On the MMSU, MMAU, and MMAR audio understanding benchmarks, DIFFA-2 achieves an overall MMSU accuracy of 60.45 — surpassing comparable open AR-based LALMs such as Kimi-Audio (59.28) and Qwen2.5-Omni (59.09) with only ~1.1% trainable parameters and 14.8k hours of open-source data — while its predecessor DIFFA improves by +4.41 points on MMSU and nearly +20 points on MMAU Test-mini, establishing that diffusion-based backbones can serve as viable alternatives to autoregressive models for audio understanding under realistic data and training budgets, though performance on conversational dialogue benchmarks such as VoiceBench remains behind heavily instruction-tuned omni models.

2. Context and Motivation

The Core Problem: Scaling Audio Language Models Beyond Autoregressive Decoding

The fundamental question this paper tackles is whether diffusion-based large language models (dLLMs) can be turned into competitive and practical audio understanding backbones that rival today's dominant autoregressive (AR) large audio language models (LALMs). The vast majority of state-of-the-art systems — Qwen-2.5-Omni, Qwen3-Omni, Kimi-Audio, GPT-4o-Audio, Gemini 2.0 Flash — all rely on AR decoders that generate tokens strictly left-to-right. While these models achieve strong results on audio understanding benchmarks, the paper identifies three intertwined problems with this AR monopoly:

AR models are expensive to scale in data and computation. Training an AR LALM from scratch requires enormous quantities of high-quality audio–text supervision across speech, sound, and music — data that is far more expensive to collect and curate than text-only corpora. The sequential nature of AR pretraining also means that model quality improvements are tightly coupled to the scale and diversity of the training data.
Strictly sequential decoding limits inference efficiency. AR decoders produce one token at a time with no parallelism within a generation step. This becomes a practical bottleneck for long-form audio understanding (e.g., multi-turn dialogues, long audio descriptions) and interactive applications where latency matters. While techniques like speculative decoding can partially mitigate this, they add engineering complexity and still fundamentally rely on sequential token generation.
The research community lacks a clear answer on whether dLLMs can scale for audio. Diffusion models have shown promise in text domains — LLaDA demonstrated that dLLMs can match AR models on language understanding tasks, and recent work by Ni et al. (2025) showed that dLLMs are "super data learners" that continue improving even when training data is limited, leveraging dense compute and implicit Monte Carlo-style data augmentation. These properties are theoretically appealing for audio, where data scarcity is a persistent challenge. However, no prior work had demonstrated whether these advantages translate to multimodal audio understanding at a competitive scale.

This gap is significant because audio understanding sits at an intersection of constraints that make the dLLM promise particularly compelling:

Data scarcity: High-quality audio–text pairs covering speech, environmental sounds, and music are orders of magnitude less abundant than text-only data. If dLLMs genuinely make more effective use of limited unique data — as Ni et al. (2025) suggest — this could lower the data barrier for building capable audio models.
Latency concerns: Audio is inherently temporal and often requires processing long sequences (multi-minute recordings, continuous conversations). The ability of dLLMs to update multiple tokens in parallel during iterative denoising could translate to lower wall-clock latency compared to strictly sequential AR decoding, especially for long outputs.
Bidirectional context: Audio understanding tasks often require reasoning that integrates information across an entire utterance — for instance, identifying a speaker's emotional state from prosody patterns distributed throughout a recording, or answering a question about a specific event in a multi-minute environmental sound clip. AR models can only condition on past tokens, while dLLMs naturally exploit full bidirectional context through their mask-and-reconstruct paradigm.

Prior Approach: DIFFA as a Proof of Concept

Before this work, the only attempt to apply dLLMs to audio was DIFFA (Zhou et al., 2025). DIFFA took a straightforward approach: replace the AR backbone in a standard LALM architecture with a diffusion counterpart (LLaDA), keeping the audio encoder and adapter design largely unchanged. The results were encouraging — under matched data, adapter design, and training recipes, the diffusion variant showed substantial gains on audio understanding benchmarks like MMAU and MMSU compared to its AR equivalent.

However, DIFFA remained fundamentally a proof of concept. The paper identifies several critical limitations that prevented it from being taken seriously as a practical alternative to AR LALMs:

Weak acoustic representation. DIFFA used a relatively small Whisper encoder and a single projection-based adapter. This limited its ability to capture fine-grained acoustic details — prosody, emotion, speaker characteristics, and environmental sound textures — that are essential for tasks beyond simple speech recognition.
Frozen diffusion backbone. The LLaDA backbone was kept completely frozen during training. Only the adapter was updated. This meant the model could never adapt its language modeling capabilities to the specific demands of audio-conditioned generation — understanding that "the speaker sounds anxious" should influence subsequent reasoning, or that a background siren is contextually relevant to a question about the audio scene.
Speech-centric supervision. DIFFA was trained mainly on automatic speech recognition (ASR) data with limited exposure to the broader audio understanding tasks (environmental sound classification, music understanding, paralinguistic analysis) that define modern LALM benchmarks. This narrow focus prevented it from developing general audio understanding capabilities.
No preference alignment or instruction tuning at scale. Contemporary AR LALMs benefit from large-scale supervised fine-tuning (SFT) on diverse audio QA data and, increasingly, from preference optimization techniques (RLHF, DPO) that align model outputs with human judgments of quality and helpfulness. DIFFA used neither, leaving it far behind the state of the art in terms of instruction-following ability and response quality.
No practical inference acceleration. DIFFA used standard LLaDA iterative decoding without any optimization for wall-clock latency. Since diffusion models require multiple denoising passes (typically 128–256 steps), naive decoding is substantially slower than AR generation, undercutting the theoretical parallelism advantage.

The consequence was that DIFFA, while conceptually promising, scored 56.04 on MMSU — behind essentially all contemporary open AR LALMs of comparable size — and did not demonstrate that dLLMs could actually compete in practical settings.

Where Existing AR LALMs Fall Short

Even setting aside the dLLM alternative, the paper implicitly critiques several limitations of the dominant AR paradigm that create an opening for diffusion-based approaches:

Data inefficiency in AR pretraining. AR models learn through next-token prediction, an objective that only provides supervision signal from the immediately preceding context. This is known to be sample-inefficient for learning long-range dependencies — a model must see many examples of a particular discourse structure or reasoning pattern before it reliably captures the relevant relationships. In the audio domain, where the total volume of training data is constrained, this inefficiency translates directly to reduced model quality. The dLLM's mask-and-reconstruct objective, by contrast, provides denser supervision: at each training step, the model must predict many masked tokens simultaneously using bidirectional context from both past and future tokens. This is hypothesized — and shown by Ni et al. (2025) in text — to extract more learning signal per training example.

The sequential decoding bottleneck for long audio. Consider a task where the model must produce a 200-word description of a 3-minute audio recording. An AR model must generate all ~300 output tokens one at a time, with each token requiring a full forward pass through the decoder. A dLLM using block-wise semi-autoregressive decoding (as DIFFA-2 does) can generate multiple tokens per forward pass, with the parallelism factor determined by model confidence. While this doesn't eliminate the need for multiple denoising steps, it creates a fundamentally different latency profile — one that can be tuned via the decoding factor to trade off speed and accuracy.

Difficulty incorporating multimodal bidirectional context. AR models are architecturally constrained to attend only to past tokens when generating. This means that when answering a question about an audio clip, the model must encode all relevant information from the audio into the prefix that precedes the response. If the model "realizes" mid-generation that it missed an acoustic detail, there is no mechanism to go back and re-interpret the audio — the generation is strictly feed-forward. dLLMs, by performing iterative denoising over the full response, can theoretically revise their interpretation of the audio input across multiple refinement steps, with each step having access to the entire partially-generated response and the full audio context.

How DIFFA-2 Positions Itself

DIFFA-2 is not a novel architectural contribution in the sense of inventing a new model class. Rather, it is an engineering and training methodology contribution that systematically addresses every limitation of the DIFFA proof-of-concept to answer the question: "Can dLLMs be turned into competitive and practical audio backbones that match AR LALMs under realistic data and latency budgets?"

The paper's positioning is explicit and honest about its scope. It does not claim to surpass state-of-the-art AR models — Qwen3-Omni (30B-A3B) still outperforms DIFFA-2 by ~5 points on MMSU, and GPT-4o-Audio and Gemini 2.0 Flash remain ahead on MMAU. Instead, the paper argues that DIFFA-2 demonstrates parity with comparably-sized open AR LALMs, establishing dLLMs as a viable alternative backbone rather than an experimental curiosity. The goal is to shift the assumption that AR decoding is the only realistic choice for production audio models.

Several design choices reflect this pragmatic positioning:

Fully open-source data only. DIFFA-2 uses ~11,000 hours of ASR data and ~3,767 hours of SFT data, all from publicly available corpora. This stands in contrast to commercial AR LALMs that likely use proprietary and web-scale audio datasets. By demonstrating competitiveness under these constraints, the paper makes the case that dLLMs' data efficiency — if real — could be a practical advantage for organizations without access to massive proprietary audio collections.
Minimal trainable parameters (1.1%). Only the dual adapters and LoRA modules are updated (99M out of 8.77B total parameters). This extreme parameter efficiency underscores the claim that the dLLM backbone's pretrained capabilities transfer effectively to audio tasks without extensive retraining — a property that aligns with the "data learner" hypothesis, since the backbone can leverage its text-domain knowledge while the lightweight adapters handle modality alignment.
Explicit focus on audio understanding (not dialogue). The paper is careful to scope DIFFA-2 as an audio understanding model, not a conversational voice assistant. It acknowledges that performance on VoiceBench — a dialogue-centric benchmark — is mid-range because the training pipeline deliberately prioritizes fine-grained audio reasoning over chitchat alignment. This honesty about limitations strengthens the core claim: diffusion backbones work for the specific problem of understanding audio content, even if further work is needed for dialogue.
Inference efficiency as a tunable knob. Rather than claiming diffusion is universally faster, the paper shows that factor-based parallel decoding (from fast-dLLMs) provides a spectrum of accuracy–latency tradeoffs. At one end, standard decoding achieves maximum accuracy but higher latency; at the other, aggressive parallelism reduces steps substantially with minimal accuracy loss (Table 4 shows RTF dropping from 0.6792 to 0.0820 on Librispeech-clean with factor-based decoding while WER increases only from 2.72 to 3.05). This positions DIFFA-2 as a flexible system rather than a one-size-fits-all solution.

Reconciling Conflicting Signals from Prior Work

The paper is motivated by a productive tension in the research landscape:

On one hand, dLLMs have shown compelling properties in text domains: LLaDA matches AR models on language understanding benchmarks, and Ni et al. (2025) demonstrate that dLLMs continue to improve with additional training even when data is fixed — a regime where AR models saturate. The mask-and-reconstruct objective provides richer per-example supervision and enables bidirectional context modeling that AR decoders cannot match. VRPO (Zhu et al., 2025) further demonstrates that preference optimization — critical for aligning models to human expectations — can be adapted to the dLLM framework through variance-reduced ELBO estimation.

On the other hand, the audio domain has seen zero adoption of dLLMs beyond the DIFFA proof-of-concept. Every major LALM — whether open (Qwen2-Audio, SALMONN, Kimi-Audio) or proprietary (GPT-4o-Audio, Gemini) — uses AR decoding. The community's default assumption is that AR models are the safe, proven choice, and that any alternative would require demonstrating substantial advantages to justify the switching cost.

DIFFA-2 aims to break this deadlock by showing that the gap is not fundamental but rather an artifact of under-investment. By giving dLLMs the same treatment that successful AR LALMs receive — strong acoustic encoders, multi-adapter architectures, large-scale SFT, preference optimization, and inference acceleration — the paper demonstrates that performance rises to competitive levels. The implication is that the field has been ignoring a viable model class not because it doesn't work, but because no one had done the engineering to make it work.

Theoretical Motivation: Why Diffusion Might Be Well-Suited to Audio

The paper does not present a formal theoretical analysis, but its architectural choices are motivated by intuitive alignment between dLLM properties and audio understanding demands:

Corruption–reconstruction as an audio prior. The LLaDA training objective — randomly mask tokens and learn to reconstruct them from context — bears a conceptual resemblance to how humans process degraded audio. We fill in missing words in noisy speech using linguistic context and acoustic cues; we infer an occluded sound event from surrounding temporal context. By training the backbone to perform this reconstruction task over text, DIFFA-2 may develop representations that transfer naturally to audio-conditioned reconstruction, where the audio input provides additional "unmasked context" that guides the denoising process.

Bidirectional context for holistic audio reasoning. Many audio understanding tasks require integrating information distributed across an entire clip. For example, determining that "the speaker becomes increasingly frustrated over the course of the conversation" requires attending to prosodic patterns at both the beginning and end of the recording. AR models must encode all relevant audio information into an autoregressive prefix; if the model fails to include a key feature, it cannot recover. dLLMs can, in principle, use bidirectional attention over both the audio prefix and the partially-generated response to revisit audio features at any denoising step.

Iterative refinement as multiple "looks" at the audio. Standard AR decoding produces the response in a single pass; the model never gets a second chance to re-interpret the audio in light of what it has written. dLLM iterative denoising effectively gives the model multiple opportunities to attend to the audio and refine its answer, with each denoising step having access to the full (partially completed) response and the complete audio context. This could be particularly valuable for tasks where the appropriate response depends on subtle acoustic details that may not be fully captured in a single encoding pass.

The data learner hypothesis and audio data scarcity. Ni et al. (2025) demonstrated that when training data is limited, dLLMs can surpass AR models by leveraging what they call "super-dense compute" — training for more epochs with aggressive data augmentation essentially equivalent to the masking process itself, which creates an exponential number of corrupted variants from each original example. The authors do not explicitly test this hypothesis in DIFFA-2 (they do not compare training data scaling curves between diffusion and AR backbones), but the strong performance with only 14.8k hours of data is consistent with the idea that dLLMs extract more learning signal per unique example.

Taken together, these motivations position DIFFA-2 not as a replacement for AR LALMs, but as an alternative that may be preferable under specific constraints: limited training data, latency-sensitive applications where parallel decoding can be leveraged, and tasks that benefit from iterative bidirectional reasoning over long audio contexts. The paper's contribution is establishing that this alternative is now practically viable, not just theoretically appealing.

3. Technical Approach

3.1 Reader Orientation

DIFFA-2 is a large audio language model that, given an audio recording (speech, environmental sounds, or music) and a text question, generates a text answer by iteratively refining a masked text response using a diffusion-based language model backbone. The system solves the problem of building a competitive audio understanding model using a diffusion rather than autoregressive backbone — the "shape" of the solution is a frozen audio encoder feeding two complementary adapters into a mostly-frozen diffusion LLM trained through a progressive four-stage curriculum that sequentially addresses semantic alignment, acoustic enrichment, backbone adaptation, and preference alignment.

3.2 Big-Picture Architecture

DIFFA-2 consists of five major components arranged in a feed-forward then iterative pipeline:

Audio Input → Frozen Whisper-Large-V3 Encoder: converts raw audio waveforms into 50 Hz frame-level representations (one feature vector every 20 ms). This encoder is never updated during training — it serves as a fixed, high-quality acoustic feature extractor.

Dual-Adapter Bridge (two parallel pathways from the encoder output):

Semantic Adapter: a two-layer convolution subsampling module followed by a two-layer linear projection. It reduces the temporal resolution from 50 Hz to 12.5 Hz (4× subsampling) and projects the temporally aggregated features into the textual embedding space of the diffusion backbone. This stream carries what is being said — the linguistic content temporally aligned with the text modality.
Acoustic Adapter: a two-layer Q-former with 64 trainable query vectors that cross-attend to intermediate encoder states. It produces a compact fixed-length summary vector capturing how it is being said — prosody, emotion, speaker characteristics, environmental sound textures, and musical attributes.

Frozen LLaDA-8B-Instruct Diffusion Backbone: a pretrained 8-billion-parameter diffusion language model that has been instruction-tuned on text. During DIFFA-2 training, most backbone parameters remain frozen; only LoRA adapters (14.7M parameters) are updated in Stage 3. The backbone takes the concatenation of audio embeddings (from both adapters) and text prompt tokens as its fully-visible prefix, then performs iterative denoising over masked response tokens.

LoRA Modules (rank 8, α=16): low-rank adaptation matrices inserted into the backbone's attention layers, updated only during Stage 3 and Stage 4. They allow the backbone to adapt its text-generation behavior to audio-conditioned generation without catastrophic forgetting of its pretrained knowledge.

Training Pipeline Stages (sequential curriculum): Stage 1 trains only the semantic adapter on ASR data; Stage 2 jointly trains both adapters on synthesized audio QA data; Stage 3 unfreezes the backbone via LoRA on the same SFT data; Stage 4 applies variance-reduced preference optimization on curated preference pairs.

Information flow at inference: Audio → Whisper encoder → (semantic adapter, acoustic adapter) → concatenated audio embeddings + text prompt → backbone prefix (fully visible, never masked) → initialized fully-masked response sequence → iterative denoising over T steps (block-wise semi-autoregressive with factor-based parallel decoding) → final answer text.

3.3 Roadmap for the Deep Dive

First, the LLaDA diffusion framework that DIFFA-2 inherits — how its mask-and-reconstruct objective works, how it enables bidirectional context, and why this paradigm is architecturally different from autoregressive decoding. This is the generative engine that all subsequent components depend on.
Second, the dual-adapter architecture — why DIFFA-2 uses two separate adapters rather than one, the specific structure of each (convolution subsampling for semantics, Q-former cross-attention for acoustics), and what information each pathway captures. This is the modality bridge that connects continuous audio to discrete text.
Third, the progressive four-stage training curriculum — the rationale for training in stages rather than jointly, what is frozen/unfrozen at each stage, and the data composition at each stage. This is the optimization strategy that enables parameter-efficient training with heterogeneous data.
Fourth, the supervised fine-tuning objective under the LLaDA framework — how the standard diffusion loss is adapted to audio-conditioned generation, what gets masked and what stays visible, and the role of the <endoftext> token. This is the core learning signal.
Fifth, the variance-reduced preference optimization (VRPO) procedure — how DPO-style preference learning is adapted to dLLMs, why standard DPO suffers from high variance in ELBO estimates, and the antithetic sampling strategy that VRPO introduces to stabilize training. This is the alignment mechanism.
Sixth, the inference procedure — the semi-autoregressive block-wise decoding strategy, the iterative denoising process, and the factor-based parallel decoding acceleration that adaptively trades off speed and accuracy. This is the deployment mechanism.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an engineering and training methodology paper whose core idea is that a diffusion-based LLM backbone, when equipped with a dual-adapter audio interface and trained through a carefully staged curriculum combining semantic alignment, acoustic enrichment, backbone fine-tuning, and preference optimization, can match or surpass comparably-sized autoregressive audio language models on audio understanding benchmarks.

The LLaDA Diffusion Framework: What DIFFA-2 Inherits

Before diving into DIFFA-2's audio-specific components, we must understand the generative engine it builds upon. LLaDA (Large Language Diffusion Models, Nie et al., 2025) is a non-autoregressive language modeling framework based on discrete random masking. Unlike autoregressive models that factorize the joint probability of a token sequence $x_0 = (x_0^1, ..., x_0^L)$ as $\prod_{i=1}^L p(x_0^i | x_0^{1:i-1})$ (each token conditioned only on previous tokens), LLaDA uses a corruption-then-reconstruction paradigm.

The forward masking process. During training, LLaDA takes a clean target sequence $x_0$ and independently replaces each token with a special mask token M with probability $t \in (0, 1]$ , producing a corrupted sequence $x_t$ . The masking probability $t$ is sampled uniformly for each training example, meaning the model sees everything from lightly corrupted sequences (few masks, easy reconstruction) to heavily corrupted sequences (many masks, difficult reconstruction requiring global reasoning). The masking is independent per position — there is no sequential dependency in how tokens are corrupted, unlike autoregressive models where each token's probability depends on all previous tokens.

The pre-training objective. The mask predictor $p_\theta(x_0 | x_t)$ — parameterized by a standard Transformer decoder (yes, a decoder architecture, but one that can attend bidirectionally because the full corrupted sequence is provided at once rather than generated left-to-right) — is trained to reconstruct the original tokens at masked positions:

$\mathcal{L}(\theta) \triangleq -\mathbb{E}_{t, x_0, x_t} \left[ \frac{1}{t} \sum_{i=1}^{L} \mathbb{I}[x_t^i = \mathrm{M}] \log p_\theta(x_0^i \mid x_t) \right]$

where $\mathbb{I}[x_t^i = \mathrm{M}]$ is 1 if position $i$ is masked, 0 otherwise; $L$ is the sequence length; $t$ is the masking probability; and $p_\theta(x_0^i \mid x_t)$ is the model's predicted distribution over the vocabulary for position $i$ given the corrupted sequence $x_t$ .

What this equation computes. For each masked position in the corrupted sequence, the model predicts a probability distribution over all possible tokens in the vocabulary, and we take the negative log probability of the true token. These losses are summed across all masked positions, then divided by $t$ (the masking rate). The division by $t$ is crucial: it up-weights the loss when few tokens are masked, because each masked position carries more information about the reconstruction task when the context is mostly clean. When $t$ is small and only a few tokens are masked, the $1/t$ factor makes those few positions count more, compensating for the fact that most positions contribute zero loss. The expectation $\mathbb{E}_{t, x_0, x_t}$ averages over training examples, random masking rates, and random masking patterns.

Why this form. This objective is derived as a tractable upper bound on the negative log-likelihood of the data under a continuous-time discrete diffusion process (Shi et al., 2024; Ou et al., 2025). Unlike autoregressive training, which only provides supervision from the left context, this objective provides dense bidirectional supervision: to reconstruct a masked token at position $i$ , the model can attend to both the left context (tokens $1...i-1$ ) and the right context (tokens $i+1...L$ ), since neither is masked (except for other independently masked positions). The $1/t$ weighting ensures the objective remains a valid likelihood bound regardless of the masking rate, and the uniform sampling of $t$ ensures the model learns to handle all corruption levels.

Supervised fine-tuning under LLaDA. When adapting LLaDA to instruction-following tasks, the framework introduces a crucial distinction between the prompt and the response. Given a prompt–response pair $(p_0, r_0)$ , only the response $r_0$ is subject to random masking — the prompt $p_0$ remains fully visible throughout:

$-\mathbb{E}_{t, p_0, r_0, r_t} \left[ \frac{1}{t} \sum_{i=1}^{L'} \mathbb{I}[r_t^i = \mathrm{M}] \log p_\theta(r_0^i \mid p_0, r_t) \right]$

where $L'$ is the response length and $r_t$ is the masked response. This means the model always has full access to the instruction/prompt when reconstructing masked response tokens, mirroring how inference will work (the prompt is known, only the response needs generation).

Inference by iterative denoising. At inference time, LLaDA starts with a fully masked response sequence $r_T$ (all tokens are M) and iteratively denoises it. At each step $t \rightarrow s$ (where $s < t$ , moving toward less masking), the model:

Predicts the most likely token at each currently-masked position using $\arg\max p_\theta(r_0 \mid p_0, r_t)$ .
Selects a fraction of the lowest-confidence predictions and re-masks them (replaces the predicted token back with M), producing $r_s$ .
Repeats until all masks are resolved (typically over a fixed schedule of steps, e.g., 128 or 256).

The key insight is that the model never commits to a prediction until the final step — tokens predicted in early steps can be re-masked and revised in later steps, allowing the model to use bidirectional context to refine its output. This is fundamentally different from AR decoding, where each token is generated once and never revisited. The paper adopts a semi-autoregressive block-wise strategy: the response is generated in left-to-right blocks (ensuring sequential coherence), but within each block, tokens are predicted in parallel and selectively re-masked across diffusion steps.

The Dual-Adapter Audio Interface

DIFFA-2's most significant architectural innovation over its predecessor is the dual-adapter design, which provides the diffusion backbone with two complementary views of the audio signal. The core design principle is that audio carries two fundamentally different types of information that should be processed through different pathways:

Type 1: Linguistic/semantic content — what words are spoken, what a sound means, what musical structure exists. This information is temporally structured: the meaning of speech unfolds over time in a way that aligns with the structure of text. To capture this, the model needs a temporally-resolved representation that can be mapped to textual semantics.

Type 2: Paralinguistic/acoustic qualities — how something is said (emotion, prosody, speaker identity), what texture a sound has (reverberation, timbre), what mood music conveys. This information is often temporally diffuse: emotion is conveyed through prosodic patterns across entire utterances, not individual time steps. A compact holistic summary may be more useful than a frame-by-frame representation.

The dual adapters reflect these two information types.

The Semantic Adapter: Temporally-Aligned Content Encoding

The semantic adapter is designed to produce a sequence of embeddings that is temporally aligned with the text modality — each output vector corresponds to roughly 80 ms of audio (12.5 Hz) and captures the linguistic content in that time window.

Structure. The adapter consists of two sequential components:

Two-layer convolution subsampling module. This takes the Whisper encoder's output at 50 Hz (one 1280-dimensional vector every 20 ms) and applies two convolutional layers with stride, reducing the temporal resolution by a factor of 4 to 12.5 Hz. The convolutions aggregate local acoustic patterns into higher-level features. The choice of 4× subsampling (from 50 Hz to 12.5 Hz) means each output frame summarizes roughly 80 ms of audio — enough to capture phoneme-level information but not so coarse that word boundaries are lost.
Two-layer linear projection. The subsampled features are projected through two feed-forward layers into the embedding dimensionality of the LLaDA backbone. This projection maps from the Whisper encoder's representation space (which is optimized for ASR) to the backbone's token embedding space (which represents semantic meaning). The two layers provide sufficient capacity for a non-linear transformation while remaining parameter-efficient.

Why convolution subsampling + linear projection. An alternative would be a single linear projection from the full 50 Hz encoder output, but this would produce sequences that are 4× longer, dramatically increasing the backbone's computational cost (since the audio prefix is part of every forward pass during both training and inference). Another alternative would be a Q-former (as used for the acoustic adapter), but Q-formers produce a fixed-length summary that loses temporal structure. For semantic content, preserving the temporal dimension is important because the model needs to know when specific words or sounds occur to answer temporally-grounded questions (e.g., "what did the speaker say after the door slammed?").

The Acoustic Adapter: Compact Paralinguistic Summarization

The acoustic adapter is designed to produce a compact, fixed-length representation that captures global acoustic and paralinguistic properties of the entire audio input — information that is not necessarily tied to specific time points but characterizes the audio as a whole.

Structure. The adapter is implemented as a two-layer Q-former (Li et al., 2023, originally from BLIP-2 for vision-language models). A Q-former uses a small set of learnable "query" vectors that cross-attend to the encoder's hidden states, extracting relevant information into a fixed-size output:

64 trainable query vectors (each with the backbone's hidden dimensionality) are randomly initialized and learned during training. These queries are analogous to "questions" the model learns to ask about the audio — e.g., "what emotion is conveyed?", "what is the speaker's gender?", "is there background noise?", "what is the tempo?".
Cross-attention to intermediate Whisper encoder states. Rather than attending to the final encoder output (which is optimized for ASR and may have discarded paralinguistic information), the Q-former attends to intermediate layer outputs of the Whisper encoder. Earlier layers of Whisper are known to retain more acoustic detail (prosody, speaker characteristics) that gets abstracted away in later layers optimized for word recognition. By attending to intermediate states, the Q-former can extract acoustic features that ASR-focused training might otherwise discard.
Two transformer layers with cross-attention and feed-forward sublayers process the queries, producing 64 output embeddings. The exact architecture follows the standard Q-former design: each query vector independently cross-attends to all encoder hidden states, aggregates relevant information, and is transformed through feed-forward layers.

Why a Q-former. Several design alternatives were available and rejected:

Global average pooling over the encoder output would be extremely cheap but would equally weight all time steps, losing the ability to focus on salient acoustic events (e.g., a sudden change in pitch indicating surprise).
Using the Whisper encoder's own [CLS] token (if available) would require the encoder to have been trained with a pooling objective, which Whisper-Large-V3 was not.
A single projection from all encoder states would produce a variable-length sequence, not a compact summary — it would blur the distinction between the semantic pathway (temporally resolved) and the acoustic pathway (holistic).
Directly using the semantic adapter output would force one representation to serve both purposes, likely degrading both. Content words and emotional prosody occupy different representational subspaces, and optimizing a single adapter for both would create interference.

The Q-former's cross-attention mechanism allows the 64 query vectors to dynamically extract different acoustic attributes — one query might learn to attend to pitch contours, another to energy patterns, another to spectral features associated with environmental sounds. The fixed-size output (64 vectors) keeps the acoustic prefix compact regardless of audio length, preventing the backbone's context window from being dominated by audio tokens.

Information complementarity. The semantic adapter provides a high-temporal-resolution content stream; the acoustic adapter provides a compact holistic summary. Together, they give the backbone both the "what" (via semantics) and the "how" (via acoustics). For a question like "Is the speaker angry?", the semantic adapter provides the words (which might be neutral), while the acoustic adapter provides prosodic cues (raised pitch, increased energy, faster speech rate) that signal anger. For a question like "What sound occurs at 0:35?", the semantic adapter's temporal resolution lets the model localize the event, while the acoustic adapter provides the texture information that distinguishes a door slam from a gunshot.

Both adapters are frozen during Stage 1-2 backbone freezing and updated during Stage 3-4. This staged training ensures that the adapters are first aligned to the frozen backbone's representation space before the backbone is allowed to adapt to the audio modality, preventing the backbone from drifting away from its pretrained knowledge before the adapters have learned a stable mapping.

The Progressive Four-Stage Training Curriculum

DIFFA-2's training is organized into four sequential stages, each with a specific purpose, data composition, and parameter update strategy. The staged approach is motivated by the need to progressively introduce complexity: if all parameters were trained jointly from the start on heterogeneous data (ASR + diverse audio QA + preference pairs), the adapters would receive conflicting gradients (e.g., ASR requires precise word-level transcription, while emotion recognition focuses on global prosody), the backbone would drift from its text pretraining before the adapters could provide stable audio representations, and the preference optimization would be applied to a model that hasn't yet learned basic audio understanding.

Which parameters are updated at each stage. The paper summarizes this in Table B.3:

Whisper-Large-V3 Encoder (637M): never trained (frozen throughout all stages)
Semantic Adapter (36.4M): trained in Stages 1–4
Acoustic Adapter (47.9M): trained in Stages 2–4
dLLM Backbone (8.03B): frozen in Stages 1–2, updated via LoRA in Stages 3–4
LoRA Modules (14.7M): inserted before Stage 3, trained in Stages 3–4
Total trainable parameters: 99.0M out of 8.77B (~1.1%)

Why freeze the backbone until Stage 3? The paper does not explicitly justify this, but the reasoning follows from transfer learning principles. The LLaDA-8B-Instruct backbone was pretrained on text and instruction-tuned on text tasks. Its internal representations encode linguistic knowledge (grammar, reasoning patterns, factual knowledge) that is valuable for audio understanding (the model still needs to generate coherent text responses). If the backbone were fine-tuned from Stage 1 on ASR data — which has a very narrow distribution (transcription of spoken words) — it would likely overfit to ASR-style outputs and lose its general language understanding capabilities (catastrophic forgetting). By keeping the backbone frozen during Stages 1–2, the adapters learn to map audio into the backbone's existing text-compatible representation space. Only once this mapping is stable does Stage 3 allow the backbone to adapt slightly (via low-rank LoRA) to the audio modality, ensuring that any backbone updates are in response to audio-specific demands (e.g., learning to attend more to paralinguistic cues) rather than compensating for poor adapter alignment.

Stage 1: Semantic Alignment on ASR

Purpose. Train the semantic adapter to produce representations that the frozen diffusion backbone can decode into accurate transcriptions. This stage addresses the most basic form of audio understanding — speech recognition — and ensures the content pathway is functioning before introducing more complex tasks.

Data. LibriSpeech (960 hours) and GigaSpeech (10,000 hours), totaling approximately 11,000 hours of transcribed speech. Each transcript is converted into an instruction-following format by applying 25 distinct prompt templates generated by Qwen3-32B. Examples include "Please transcribe the following audio..." and "What is the exact content of this recording?" (full list in Figure A.1). This transforms plain ASR into instruction-following ASR, teaching the model to understand that transcription is a task to be performed in response to a prompt — a format consistent with later stages that will use diverse instructions.

What is trained. Only the semantic adapter. The acoustic adapter does not yet exist (or is not trained). The diffusion backbone is frozen.

Training configuration. Learning rate $1 \times 10^{-4}$ , batch size 1280, 1000 warmup steps, 12 epochs (Table B.2). The high batch size (1280) and learning rate (the highest of any stage) reflect the simplicity of the task — ASR is a well-defined problem with clean supervision, and the adapter can be trained aggressively without risk of overfitting given 11,000 hours of data.

Objective. The standard LLaDA supervised fine-tuning loss applied to audio-conditioned transcription (Equation 2, adapted for audio input):

$\mathcal{L}_{\text{stage1}} = -\mathbb{E}_{t, a, p, r_0, r_t} \left[ \frac{1}{t} \sum_{i=1}^{L'} \mathbb{I}[r_t^i = \mathrm{M}] \log p_\theta(r_0^i \mid a, p, r_t) \right]$

where $a$ is the audio input (encoded by Whisper and projected through the semantic adapter), $p$ is the text prompt ("Please transcribe..."), $r_0$ is the ground-truth transcription, and $r_t$ is its masked version. The model learns to reconstruct masked transcription tokens given the audio embedding and prompt.

What this stage accomplishes. By the end of Stage 1, the semantic adapter's output has been aligned to the backbone's text embedding space — the backbone "understands" these audio-derived vectors as if they were semantically meaningful text tokens. The paper reports the WER achieved at this stage in Table 4: 2.72 on LibriSpeech-clean and 5.34 on LibriSpeech-other, indicating the adapter successfully bridges audio to text for the ASR task. This stage also provides the backbone with extensive exposure to speech patterns, even though the backbone itself isn't updated — the backbone's forward pass processes audio-prompt-transcription triplets, and while gradients don't flow to the backbone, the adapter learns to produce representations that the backbone's existing text knowledge can decode.

Stage 2: Joint Semantic–Acoustic Alignment

Purpose. Introduce the acoustic adapter and train both adapters jointly on diverse audio understanding tasks that require paralinguistic, environmental, and music understanding — capabilities that pure ASR does not teach. The backbone remains frozen, forcing the adapters to learn representations that enable the existing text model to answer diverse audio questions.

Data. Three categories of supervised fine-tuning data, totaling approximately 3,767 hours:

Category 1: Caption-grounded Audio QA (582,666 samples from Sound/General Audio + 331,406 from Music). The paper collects a diverse set of audio datasets with existing captions or annotations (AudioCaps, Clotho, ESC-50, TACOS, VocalSound, WavCaps, FMA_medium, LP-MusicCaps-MTT, MusicCaps, Nsynth, ParaSpeechCaps, AccentDB, EmoV-DB, IEMOCAP, MELD, VCTK, Meta FAIR ASR, VoxCeleb1, and more — see Table A.1). For each dataset, the existing captions/annotations describe what is in the audio (e.g., "A dog barking followed by a car horn", "A female speaker with a British accent expressing sadness"). An LLM (Qwen3-32B) is prompted to generate question–answer pairs grounded in these captions, using the prompt template shown in Figure A.2. Crucially, the LLM is instructed to generate diverse question types and incorporate paralinguistic/annotation details into both questions and answers. This automated data generation approach ensures broad coverage of audio phenomena without requiring human annotation.

Category 2: Direct Audio QA via TTS (text QA pairs synthesized as speech). To expose the model to general question-answering in spoken form, the paper takes text-only QA pairs from standard datasets (Alpaca, NaturalQuestions, TriviaQA, WebQuestions — see Section A.2 Part 2) and converts the questions to speech using CosyVoice2 TTS with randomly sampled LibriSpeech speaker prompts. This produces audio where the content is a general-knowledge question (e.g., "What is the capital of France?") spoken in a synthetic voice. While the questions themselves don't require audio understanding (they could be answered from text alone), this data teaches the model to process spoken questions and respond in text — a format that bridges pure ASR and audio understanding.

Simple vs. complex split. The paper categorizes samples based on answer length: short answers (simple QA) and long answers (complex QA), using different instruction templates for each (Figure A.3). This prevents the model from always producing terse answers or always producing verbose ones — it learns to match answer length to question type.
Empathetic QA. The English subset of OpenS2S is additionally included, providing examples of emotionally-aware responses to spoken input.

Category 3: ASR Subset (5% of Stage 1 data). A small fraction of ASR data is retained to prevent catastrophic forgetting of the basic transcription capability acquired in Stage 1. Without this, the diverse audio QA data might cause the semantic adapter to drift away from precise word-level alignment — the model might become good at answering "what emotion?" but worse at transcribing "what words?".

What is trained. Both the semantic adapter and the acoustic adapter. The diffusion backbone remains frozen. This is the first stage where the acoustic adapter is introduced and trained.

Training configuration. Learning rate $5 \times 10^{-5}$ (half of Stage 1), batch size 196 (much smaller than Stage 1), 1000 warmup steps, 10 epochs. The lower learning rate and batch size reflect the greater diversity and complexity of the data — with many different task types, larger batch sizes would mix incompatible gradients, and higher learning rates could cause oscillation between competing objectives.

Objective. Same LLaDA SFT loss as Stage 1, but now conditioning on both audio adapters:

$\mathcal{L}_{\text{stage2}} = -\mathbb{E}_{t, a, p, r_0, r_t} \left[ \frac{1}{t} \sum_{i=1}^{L'} \mathbb{I}[r_t^i = \mathrm{M}] \log p_\theta(r_0^i \mid a_{\text{sem}}, a_{\text{ac}}, p, r_t) \right]$

where $a_{\text{sem}}$ is the semantic adapter output and $a_{\text{ac}}$ is the acoustic adapter output. Both are concatenated into the prefix that conditions response generation.

What this stage accomplishes. Table 5 shows the results after Stage 2: DIFFA-2 (S2) achieves 63.90 on MMAU Overall and 56.43 on MMSU Overall. Comparing to the AR baseline LLaMA-Audio (S2) trained on identical data — which achieves 60.80 on MMAU and 43.71 on MMSU — the diffusion backbone already shows a clear advantage (+3.1 points MMAU, +12.72 points MMSU). This suggests that the corruption–reconstruction training objective enables more effective use of the diverse audio QA data than AR next-token prediction, even before the backbone is fine-tuned. The acoustic adapter, trained for the first time here, enables the model to answer questions about paralinguistic properties (MMSU Phonology and Paralinguistics scores) that would be inaccessible from purely semantic features — the semantic adapter alone cannot convey "sadness" or "British accent" or "door slamming sound."

Stage 3: Unfreezing the Diffusion Backbone with LoRA

Purpose. Allow the diffusion backbone to adapt its internal representations to the audio modality, now that the adapters have been trained to provide stable audio embeddings. This stage addresses the "frozen backbone" limitation of DIFFA — the backbone can now learn to attend differently to audio features, adjust its generation behavior for audio-conditioned tasks, and specialize its text knowledge for the audio domain.

Method. Low-Rank Adaptation (LoRA, Hu et al., 2022) is applied to the diffusion backbone. LoRA inserts trainable low-rank matrices into the backbone's attention layers: for a weight matrix $\mathbf{W} \in \mathbb{R}^{d \times k}$ , LoRA adds $\Delta\mathbf{W} = \mathbf{B}\mathbf{A}$ where $\mathbf{B} \in \mathbb{R}^{d \times r}$ , $\mathbf{A} \in \mathbb{R}^{r \times k}$ , and $r \ll \min(d, k)$ . The original weight $\mathbf{W}$ is frozen; only $\mathbf{A}$ and $\mathbf{B}$ are trained. The forward pass computes $\mathbf{h} = \mathbf{W}\mathbf{x} + \mathbf{B}\mathbf{A}\mathbf{x}$ .

DIFFA-2 uses rank $r = 8$ and scaling factor $\alpha = 16$ . The scaling factor controls the magnitude of the LoRA update relative to the base weights (effective update is scaled by $\alpha/r = 2$ ). This yields 14.7M trainable LoRA parameters added to the 8.03B backbone (0.18% of backbone parameters). The low rank constrains the adaptation to be in a low-dimensional subspace, preventing overfitting and catastrophic forgetting — the backbone can adjust its behavior but cannot fundamentally change its representation structure.

Data. Same SFT data as Stage 2, with the addition of Category 4: Multiple-choice AQA data (AudioMCQ). The AudioMCQ corpus provides multiple-choice questions derived from audio captioning datasets (AudioCaps, Clotho, CompA-R, MusicCaps, LP-MusicCaps, SpeechCraft, TACOS). Each sample presents an audio clip, a question, and several answer options, requiring fine-grained discrimination between similar audio attributes. The paper uses the "without chain-of-thought" version, meaning the model must directly select the correct answer rather than reasoning step-by-step. This data teaches the model to attend to subtle acoustic differences (e.g., distinguishing "a dog barking" from "a dog whining") that are essential for the benchmark evaluations.

What is trained. Both adapters (semantic and acoustic) continue training, and the LoRA modules in the backbone are trained for the first time. All other backbone parameters remain frozen.

Training configuration. Same learning rate as Stage 2 ( $5 \times 10^{-5}$ ), same batch size (196), same warmup (1000 steps), same epochs (10). The consistency with Stage 2 suggests that the addition of LoRA doesn't fundamentally change the optimization dynamics — the adapters are already well-aligned, and the LoRA updates are relatively small.

Objective. Same LLaDA SFT loss as Stages 1–2. The only difference is that $\theta$ now includes the LoRA parameters.

What this stage accomplishes. Table 5 shows the jump from Stage 2 to Stage 3: MMAU improves from 63.90 to 68.20 (+4.30), and MMSU Overall improves from 56.43 to 59.41 (+2.98). The gains are particularly notable in MMSU Perception (+5.62 points, from 38.54 to 44.16), especially semantic perception (52.91 to 59.53, +6.62) and paralinguistic perception (31.32 to 40.63, +9.31). This pattern suggests that LoRA fine-tuning primarily improves the model's ability to attend to and interpret the adapter outputs — the acoustic information was already present in Stage 2, but the frozen backbone wasn't fully utilizing it. Once the backbone can adapt, it learns to weight acoustic features more heavily for paralinguistic tasks, producing the substantial perception gains.

Comparing to LLaMA-Audio (S3) — the AR baseline trained identically — DIFFA-2 (S3) achieves 68.20 vs. 67.40 on MMAU (+0.80) and 59.41 vs. 55.31 on MMSU (+4.10). The MMSU gap is particularly striking (+4.10 overall, driven by +6.43 in reasoning), suggesting the diffusion backbone benefits more from backbone fine-tuning than the AR backbone does under identical data.

Stage 4: Preference Optimization with VRPO

Purpose. Refine the model's responses to be more accurate and faithful to subtle audio cues, using a preference-based objective rather than supervised fine-tuning. This stage addresses the "no preference alignment" limitation of DIFFA and aligns the model with human-like judgments of response quality.

Why preference optimization for audio. Supervised fine-tuning teaches the model to produce correct answers, but it doesn't explicitly teach the model to avoid plausible-sounding but incorrect answers. In audio understanding, many errors are subtle: the model might correctly identify that a speaker is emotional but misidentify the emotion (sadness vs. frustration); it might recognize that a sound event occurred but misidentify the sound (door slam vs. object dropping). These errors are not penalized by SFT, which only provides positive examples. Preference optimization provides contrastive pairs: a correct answer (chosen) and a plausible but incorrect answer (rejected), teaching the model to discriminate between them.

Preference data construction. Starting from the SFT data (which contains audio, questions, and correct answers), the paper generates "rejected" responses by prompting a language model to produce fluent, superficially reasonable answers that contain subtle audio-related factual errors. The prompt template (Figure A.4) instructs the LLM to introduce errors in specific categories: incorrect gender of speaker, incorrect emotion, incorrect rhythm/tempo for music, incorrect sound event identity, incorrect number of speakers, etc. Only pairs where the correct answer is "unambiguously superior" are retained, yielding approximately 3,000 preference pairs for Stage 4. This is a relatively small preference dataset — typical text-domain DPO uses tens of thousands of pairs — reflecting the difficulty of automatically generating high-quality audio-specific preference data.

The challenge: DPO for diffusion models has high variance. Standard Direct Preference Optimization (DPO, Rafailov et al., 2023) works by implicitly representing the reward function as the log-ratio of policy and reference model probabilities:

$r(x, y) = \beta \log \frac{p_\theta(y \mid x)}{p_{\text{ref}}(y \mid x)}$

where $p_\theta$ is the policy model, $p_{\text{ref}}$ is a frozen reference model (the Stage 3 checkpoint), and $\beta$ controls preference strength. The DPO loss then optimizes:

$\mathcal{L}_{\text{DPO}} = -\log \sigma\left(\beta \left[ \log \frac{p_\theta(y^+ \mid x)}{p_{\text{ref}}(y^+ \mid x)} - \log \frac{p_\theta(y^- \mid x)}{p_{\text{ref}}(y^- \mid x)} \right]\right)$

where $\sigma$ is the sigmoid function, $y^+$ is the chosen response, and $y^-$ is the rejected response.

This works for AR models because $p_\theta(y \mid x)$ can be computed exactly via the chain rule of probability. For diffusion models, the exact likelihood is intractable — we only have a lower bound (ELBO). Replacing the exact likelihood with an ELBO estimate introduces variance: different masking patterns produce different ELBO estimates for the same sequence, and the DPO loss amplifies this variance because it depends on the difference between two estimated log-ratios.

Variance-Reduced Preference Optimization (VRPO, Zhu et al., 2025). VRPO addresses this by using multiple Monte Carlo samples with shared masking patterns between the policy and reference models. The procedure is:

Estimate log-likelihoods via Monte Carlo ELBO. For each response $y$ , sample $K = 4$ independent masking patterns (each with a random masking rate $t \sim \text{Uniform}(0, 1]$ and random mask positions). For each pattern, compute the ELBO — the model's ability to reconstruct the masked tokens given the prompt, audio, and unmasked response tokens. Average these $K$ estimates:

$\widehat{\log p_\theta}(y \mid x, a) = \frac{1}{K} \sum_{k=1}^{K} \mathrm{ELBO}^{(k)}_\theta(y \mid x, a)$

where $\mathrm{ELBO}^{(k)}_\theta$ is computed as the average negative reconstruction loss over masked positions for the $k$ -th masking pattern:

$\mathrm{ELBO}^{(k)}_\theta(y \mid x, a) = -\frac{1}{t_k} \sum_{i=1}^{L'} \mathbb{I}[y_{t_k}^i = \mathrm{M}] \log p_\theta(y_0^i \mid x, a, y_{t_k})$

Share masking patterns between policy and reference models. Crucially, when computing $\widehat{\log p_{\text{ref}}}(y \mid x, a)$ , the exact same $K$ masking patterns are used as for $\widehat{\log p_\theta}(y \mid x, a)$ . This implements antithetic sampling: the positive correlation between the two estimates (since they use the same random masks) means that their difference has lower variance than if independent masks were used. If the policy model is better than the reference model at reconstructing masked tokens in a particular pattern, this advantage is consistently measured across both models rather than being obscured by different mask realizations.
Compute DPO-style preference loss on the estimated log-ratios. Define the score difference:

$s_\theta(y) = \widehat{\log p_\theta}(y \mid x, a) - \widehat{\log p_{\text{ref}}}(y \mid x, a)$

Then the VRPO loss is:

$\mathcal{L}_{\text{VRPO}} = -\log \sigma\left(\beta \left[ s_\theta(y^+) - s_\theta(y^-) \right]\right)$

where $\beta$ controls preference strength (the paper does not report the specific $\beta$ value used). The loss encourages the policy model's estimated log-probability (relative to the reference) to be higher for chosen responses than for rejected responses.

What this equation computes. For a pair of responses $(y^+, y^-)$ to the same audio input and question:

Estimate how much more (or less) likely $y^+$ is under the policy model compared to the reference model: $s_\theta(y^+)$ .
Estimate the same for $y^-$ : $s_\theta(y^-)$ .
Compute the difference: $s_\theta(y^+) - s_\theta(y^-)$ . If this is positive and large, the policy model strongly prefers the chosen response over the rejected one relative to the reference.
Apply the sigmoid and negative log: if $s_\theta(y^+) \gg s_\theta(y^-)$ , the sigmoid is near 1 and the loss is near 0 (good). If $s_\theta(y^+) \ll s_\theta(y^-)$ , the sigmoid is near 0 and the loss is large (bad — the policy model prefers the wrong response).

Why this form (VRPO over standard DPO). Standard DPO with ELBO estimates would use $K=1$ (a single masking pattern) and independent masks for policy and reference. This produces high-variance gradient estimates because:

A single ELBO estimate can vary substantially depending on which tokens happen to be masked — masking token positions that are easy vs. hard to predict changes the ELBO by a large amount.
With independent masks, the difference $\widehat{\log p_\theta} - \widehat{\log p_{\text{ref}}}$ has the sum of variances from two independent estimates, making it even noisier.
The DPO loss uses the difference of two such noisy estimates, and the gradient of the sigmoid is largest when the difference is near zero (where noise has the most impact).

VRPO's shared masking patterns create positive correlation: if a particular mask pattern is "easy" (most masked tokens are predictable from context), both the policy and reference models will have high ELBO estimates, and the difference primarily reflects which model is genuinely better at reconstruction. The $K=4$ samples further reduce variance through averaging. The result is more stable preference learning, particularly for long audio-conditioned sequences where the ELBO variance is naturally higher due to sequence length.

What is trained. Both adapters and LoRA modules continue training. The reference model $p_{\text{ref}}$ is a frozen copy of the Stage 3 checkpoint.

Training configuration. Learning rate $5 \times 10^{-6}$ (an order of magnitude lower than Stage 3), batch size 4 (dramatically smaller than Stage 3's 196), 200 warmup steps, 1 epoch. The tiny batch size and learning rate reflect the precision required for preference optimization — each preference pair provides a specific signal about relative response quality, and large batches would dilute this signal. The single epoch also prevents overfitting to the small preference dataset (3,000 pairs).

What this stage accomplishes. Table 5 shows the Stage 3 → Stage 4 improvement: MMAU from 68.20 to 69.60 (+1.40), MMSU Overall from 59.41 to 60.45 (+1.04). The gains are distributed across categories:

MMSU Perception: 44.16 → 45.58 (+1.42)
MMSU Reasoning: 75.70 → 76.40 (+0.70)
MMAU Sound (Test-mini): 74.77 → 76.28 (+1.51)
MMAU Music (Test-mini): 62.57 → 63.47 (+0.90)
MMAU Speech (Test-mini): 67.27 → 69.07 (+1.80)

The improvements are modest but consistent, and importantly they don't come at the cost of regression on any category (compare to the ReST $^{EM}$ experiment in DIFFA where sequential revision performance degraded). The VRPO procedure successfully sharpens the model's sensitivity to the subtle audio cues that distinguish correct from plausible-but-wrong answers, without destabilizing the SFT-learned capabilities.

The Supervised Fine-Tuning Objective for DIFFA-2

The core learning signal across Stages 1–3 is a diffusion-style masked prediction objective that adapts the LLaDA framework to audio-conditioned generation. Here is the formal specification:

$\mathcal{L}_{\text{sft-a}} = -\mathbb{E}_{t, a, p, r_0, r_t} \left[ \frac{1}{t} \sum_{i=1}^{L'} \mathbf{1}[r_t^i = \mathrm{M}] \log p_\theta(r_0^i \mid a, p, r_t) \right]$

where:

$t \in (0, 1]$ is the masking rate, sampled uniformly per training example
$a$ is the audio input, encoded by Whisper and projected through both adapters (semantic + acoustic) into the backbone's prefix
$p$ is the text prompt (the question or instruction)
$r_0$ is the ground-truth text response, with length $L'$
$r_t$ is the masked version of $r_0$ , where each token is independently replaced by M with probability $t$
$\mathbf{1}[r_t^i = \mathrm{M}]$ is 1 if position $i$ is masked, 0 otherwise
$p_\theta(r_0^i \mid a, p, r_t)$ is the model's predicted probability for the true token at masked position $i$

What this loss computes, operationally: For a single training example:

The audio $a$ passes through Whisper → semantic adapter → embedding sequence and Whisper → acoustic adapter → 64-vector summary. Both are concatenated with the text prompt $p$ to form the fully-visible prefix.
The response $r_0$ is corrupted by independently masking each token with probability $t$ . Tokens replaced by M become the prediction targets.
The model processes the concatenation $[a_{\text{prefix}}, p, r_t]$ through all transformer layers, attending bidirectionally over both the prefix (audio + prompt) and the corrupted response.
At each masked position, the model outputs a probability distribution over the vocabulary. We extract the probability assigned to the true token $r_0^i$ .
The negative log probability $-\log p_\theta(r_0^i \mid a, p, r_t)$ is the loss for that position. Positions that were not masked contribute zero loss.
Losses are summed over masked positions, divided by $t$ .
This is averaged over the expectation (i.e., the training batch).

Critical detail: the <endoftext> token. The paper notes that "the special token <endoftext> is used both as padding and as the end-of-sequence marker, and the model is required to predict it." This means the response $r_0$ includes the <endoftext> token at its end, and this token can be masked during training. By learning to predict <endoftext>, the model learns when to stop generating — during inference, once the model predicts <endoftext> at all unmasked positions in a block, generation terminates. This is analogous to the EOS token in AR models.

Why $1/t$ reweighting and uniform $t$ sampling. The $1/t$ factor compensates for the fact that when $t$ is small, fewer tokens are masked, so each masked token contributes more to the total loss. Without this reweighting, the objective would be dominated by high- $t$ examples (many masks → many loss terms). The $1/t$ ensures all masking rates contribute equally to the expected gradient. Uniform sampling of $t$ ensures the model is trained on the full spectrum: from few masks (where reconstruction requires detailed local knowledge) to many masks (where reconstruction requires global semantic understanding), making the model robust across all denoising schedules at inference.

What stays visible vs. what gets masked. A critical design choice that the paper makes explicit: "Audio embeddings (from the adapters) and text prompt tokens remain fully visible and are never masked; only the response tokens are corrupted and denoised." This means:

The audio prefix (semantic adapter output sequence + acoustic adapter output vectors) is always fully visible. The model has complete access to all audio information at every denoising step.
The text prompt (the question/instruction) is always fully visible. The model always knows what it's being asked.
Only the response is corrupted. The model must reconstruct the answer given the fixed question and audio.

This design is natural for instruction-following models but has an important implication: the model never learns to handle uncertainty about the audio or the question. During inference, if the audio is partially corrupted or the question ambiguous, the model has no training signal for handling this — it always assumes perfect input. This is a standard limitation of current LALMs, not specific to DIFFA-2.

Inference: Iterative Denoising with Factor-Based Parallel Decoding

At inference time, DIFFA-2 generates text responses through an iterative refinement process that balances generation quality and computational cost through two complementary mechanisms: semi-autoregressive block-wise decoding and factor-based parallel decoding.

Step 1: Prefix construction. The audio input is encoded through Whisper and both adapters, producing the semantic embedding sequence and acoustic summary vectors. These are concatenated with the text prompt tokens to form the fully-visible prefix. The response is initialized as a sequence of mask tokens M of length equal to the target answer length. The concatenated sequence $[a_{\text{prefix}}, p, r_T]$ (where $r_T$ is all masks) is the initial input to the backbone.

Step 2: Semi-autoregressive block-wise decoding. Following LLaDA's strategy, DIFFA-2 does not attempt to denoise the entire response at once. Instead, it generates the response in left-to-right blocks:

The first block of response tokens (of length $B$ , the block size) is the initial denoising target. All tokens beyond this block remain fully masked.
Within the first block, the model performs $T$ iterative denoising steps (e.g., 16 steps for MMSU/MMAU/MMAR with block size 16 — see Table B.4).
At each denoising step $t \rightarrow s$ , the model predicts tokens at all masked positions within the current block, then re-masks the lowest-confidence fraction proportional to the step schedule.
Once the first block is fully denoised (all tokens unmasked), the model moves to the second block. The first block's tokens now become part of the visible prefix, and the second block is initialized as all masks.
The process repeats until the <endoftext> token is predicted or the maximum answer length is reached.

Why block-wise rather than full-sequence denoising. Denoising the full response at once would require $O(L^2)$ attention computation per denoising step (where $L$ is the total response length), making long responses expensive. Block-wise decoding constrains attention to the current block plus prefix, reducing per-step cost while still allowing each block to attend to all previously generated blocks (since they're part of the visible prefix). The left-to-right progression ensures coherence — later blocks can condition on earlier ones, preventing the model from generating contradictory information in different parts of the response.

Inference hyperparameters by benchmark (Table B.4):

MMSU, MMAU, MMAR: Answer length 16, block length 16, 16 denoising steps. These benchmarks use short multiple-choice answers (typically a single letter), so a single block of 16 tokens suffices.
LibriSpeech (ASR): Answer length 128, block length 128, 128 steps. Transcription requires longer output, so a larger block is used.
AlpacaEval, CommonEval: Answer length 128, block length 32, 128 steps. These dialogue tasks require longer responses, so multiple blocks of 32 tokens are generated sequentially.

The paper synchronizes the answer length, block length, and number of steps for multiple-choice benchmarks (all 16), meaning a single block is generated with 16 denoising steps. For longer-form tasks, block length is smaller than answer length, requiring multiple sequential blocks.

Step 2b (optional): Factor-based parallel decoding acceleration. To reduce inference latency, DIFFA-2 optionally incorporates the factor-based parallel decoding strategy from fast-dLLMs (Wu et al., 2025). This method adaptively determines how many tokens to decode in parallel within each block, replacing the standard fixed-threshold re-masking.

Standard re-masking (without factor-based decoding): At each denoising step, predict all masked tokens, rank them by confidence, and re-mask a fixed fraction (determined by the step schedule). Low-confidence tokens are re-masked; high-confidence tokens are kept. The fraction is predetermined and doesn't adapt to the model's actual confidence distribution.

Factor-based re-masking: Given the model's confidence scores $c_1, c_2, ..., c_n$ for the $n$ masked tokens in the current block (sorted in descending order: $c^{(1)} \geq c^{(2)} \geq ... \geq c^{(n)}$ ), the algorithm selects the largest number of tokens $n$ to keep (i.e., not re-mask) such that:

$(n+1)(1 - c^{(n)}) < f$

where $c^{(n)}$ is the $n$ -th highest confidence, and $f$ is the decoding factor hyperparameter (set to 1.0 in DIFFA-2's experiments). The remaining $n_{\text{total}} - n$ lowest-confidence tokens are re-masked.

What this criterion computes. For each possible number $n$ of tokens to keep (starting from keeping all tokens and working downward), the algorithm checks: is the expected number of errors among the kept tokens (approximated by $(n+1)(1 - c^{(n)})$ ) less than the factor $f$ ? The factor $f$ acts as an error tolerance:

If $f = 0$ , the model keeps no tokens (all are re-masked) — maximum accuracy, minimum speed.
As $f$ increases, the model keeps more tokens per step (accepting more potential errors), reducing the total number of denoising steps needed — minimum accuracy, maximum speed.
DIFFA-2 uses $f = 1.0$ , meaning the model aggregates some parallelism while still being relatively conservative.

Why this form. The term $(1 - c^{(n)})$ is the estimated error probability for the $n$ -th most confident token. Multiplying by $(n+1)$ gives an approximate expected count of errors among the top $n$ tokens. The criterion keeps the expected error count below $f$ , adapting to the model's actual confidence distribution: when the model is very confident (high $c^{(n)}$ for large $n$ ), many tokens are kept; when uncertain, few are kept. This is superior to a fixed threshold because it doesn't require tuning a confidence cutoff that may be appropriate for some inputs but not others.

Impact on accuracy and latency (Table 4). On LibriSpeech ASR:

Standard DIFFA-2 (S1): WER 2.72 (clean) / 5.34 (other), RTF 0.6792 / 0.7489
DIFFA-2 with factor-based decoding: WER 3.05 (clean) / 5.68 (other), RTF 0.0820 / 0.0867

The RTF (real-time factor — seconds of computation per second of audio) drops by 8.3× on clean speech (0.6792 → 0.0820) while WER increases by only 0.33 absolute percentage points (2.72 → 3.05). This demonstrates the practicality of the tradeoff: for applications where small accuracy degradation is acceptable, factor-based decoding brings dLLM latency into a competitive range with AR models (LLaMA-Audio achieves RTF 0.1402 on clean speech, and DIFFA-2 with factor-based decoding is actually faster at 0.0820, though with slightly higher WER: 3.05 vs. 2.43).

Impact on benchmark accuracy (Tables 1, 2, 3). Across MMSU, MMAU, and MMAR, the "w/ FPD" (factor-based parallel decoding) variant consistently achieves accuracy within 0.3–0.6 points of the standard variant:

MMSU: 60.45 → 60.10 (−0.35)
MMAU Test-mini: 69.60 → 68.30 (−1.30)
MMAR: 50.80 → 50.20 (−0.60)

The slightly larger drop on MMAU Test-mini (−1.30 vs. −0.35/0.60) may reflect MMAU's greater reliance on fine-grained discrimination (multiple-choice questions with subtle distractors), where token-level confidence estimation is more challenging. Nevertheless, the overall pattern confirms that factor-based decoding provides a practical accuracy–latency tradeoff without fundamentally compromising the model's audio understanding capabilities.

4. Key Insights and Innovations

Innovation 1: The Dual-Adapter as a Conceptual Decomposition of Audio Information

The most distinctive conceptual move in DIFFA-2 is not the specific adapters themselves — Q-formers and convolution subsampling are borrowed from vision-language and speech processing, respectively — but the explicit separation of audio into two complementary information streams: a temporally-resolved content stream (what is being said) and a compact holistic paralinguistic stream (how it is being said). This is not a small engineering detail. It represents a diagnosis of what was missing from the first-generation DIFFA and, by implication, what may be missing from other LALMs that use single adapters.

Prior to this work, the dominant adapter design in open LALMs was a single projection layer or a simple downsampling module — Qwen2-Audio uses a single convolution + pooling adapter, SALMONN uses a single Q-former, and the original DIFFA used one projection-based adapter. The implicit assumption was that a single representation pathway could serve all downstream audio tasks: the Whisper encoder extracts features, a single adapter maps them to the LLM's embedding space, and the LLM figures out which aspects of the representation are relevant. This approach has the virtue of simplicity and has proven sufficient for many tasks. But it has a subtle failure mode: the representational interference problem.

When a single adapter must simultaneously preserve temporal structure for content words ("the cat sat on the mat") and distill global acoustic properties (the speaker's emotional state, the room's reverberation characteristics), the two objectives pull the adapter's parameters in conflicting directions. Temporal structure requires preserving the sequence dimension; holistic summarization benefits from collapsing it. Content words are best represented in a subspace that aligns with text semantics; paralinguistic features occupy a different representational subspace that may be orthogonal to text meaning. A single adapter trained on mixed objectives may end up doing both poorly — or, more commonly, optimizing for the dominant signal (content words, which have stronger gradients in ASR-style training) at the expense of the weaker one (paralinguistics).

DIFFA-2's dual-adapter design is not the first use of Q-formers in audio (SALMONN uses one), nor the first use of temporal pooling (standard in ASR adapters). What is novel is the framing of the problem as information decomposition rather than feature extraction. The two adapters are not merely different architectural choices for doing the same thing — they are designed to capture fundamentally different types of information that should not be forced through the same representational bottleneck. The semantic adapter handles temporally-resolved content aligned with text; the acoustic adapter handles temporally-diffuse qualities that characterize the audio as a whole. This is a conceptual advance over treating adapter design as a purely architectural hyperparameter search.

The evidence for this decomposition being meaningful rather than decorative comes from the ablation in Table 5. After Stage 2 (adapters trained, backbone frozen), DIFFA-2 (S2) substantially outperforms LLaMA-Audio (S2) — which uses the same dual-adapter architecture but with an AR backbone — on MMSU Paralinguistic perception (31.32 vs. 30.23, a modest +1.09) and MMSU Reasoning (75.50 vs. 55.45, +20.05). The fact that the same dual adapters produce much larger reasoning gains with a diffusion backbone than an AR one suggests the decomposition is not merely providing more features, but providing a structured representation that the diffusion backbone's bidirectional context can exploit more effectively than AR left-to-right attention. This interaction between the adapter architecture and the generative paradigm is a finding that prior work on adapters — which tested them only with AR backbones — could not have uncovered.

A counterargument is that the dual-adapter design simply adds more parameters (36.4M + 47.9M = 84.3M for both adapters) and more capacity inevitably improves performance. The paper does not include an ablation with a single adapter of comparable total size, which would be needed to definitively attribute the gains to decomposition rather than capacity. This is a legitimate limitation. However, the qualitative pattern — that paralinguistic and reasoning gains are disproportionately large relative to content-based metrics — is consistent with the decomposition hypothesis, since a single adapter with equivalent capacity but no inductive bias for separating content from paralinguistics would likely allocate the extra capacity to the dominant content signal.

Innovation 2: Recovering Diffusion's Data Efficiency Advantage for Audio Through Staged Training

The paper's second conceptual contribution is demonstrating that the "super data learner" property of dLLMs — observed by Ni et al. (2025) in text-only settings — transfers to multimodal audio understanding when paired with an appropriate training curriculum. This is not obvious. The data learner hypothesis says that diffusion models extract more learning signal per unique example because the corruption–reconstruction objective creates an implicit combinatorial data augmentation: every training example is seen at many masking rates and patterns, effectively multiplying the training set. But this hypothesis was established in a pure text setting where the model is trained and evaluated on the same modality. Whether it would survive the addition of a separate audio encoder and cross-modal adapters — which introduce their own learning dynamics and potential bottlenecks — was an open question.

DIFFA-2 provides affirmative evidence: using only 14.8k hours of open-source data (a fraction of what commercial LALMs likely use), it matches or surpasses comparably-sized AR models trained on equivalent or larger datasets. The comparison in Table 5 is particularly telling because it controls for data and training recipe: LLaMA-Audio (S3) — the AR equivalent trained on identical data with the same staged curriculum — achieves 55.31 on MMSU Overall, while DIFFA-2 (S3) achieves 59.41. This +4.10 gap cannot be attributed to data volume, adapter design, or training stages — all are held constant. It must arise from the generative paradigm itself.

The intellectual contribution here is not the raw performance number but the demonstration of a cross-modal transfer of the data efficiency property. The dLLM backbone was pretrained on text only; the audio modality is introduced entirely through the adapters and LoRA fine-tuning. Yet the diffusion backbone still extracts more value from limited audio-text pairs than the AR backbone does. This suggests that the data efficiency stems from the training objective (mask-and-reconstruct with random corruption rates) rather than from any architectural property of the backbone itself, since both backbones are similarly sized Transformer decoders (LLaDA-8B vs. LLaMA 3.1-8B). The objective, not the model, is what makes dLLMs "super data learners."

This has significant implications for how the field thinks about scaling audio-language models. The dominant narrative has been that audio understanding requires massive proprietary datasets — the Qwen and Kimi-Audio papers emphasize their large-scale training data as a key advantage. DIFFA-2 suggests an alternative path: with a diffusion backbone, competitive audio understanding may be achievable with datasets that are feasible for academic and open-source teams to curate. The 14.8k hours used here is large but not prohibitive — it's within the range that a well-resourced academic lab can assemble from existing public corpora. If this data efficiency result generalizes to other audio tasks and larger scales, it could lower the barrier to entry for LALM development and reduce the field's dependence on proprietary data moats.

A limitation of this claim is that the paper does not include explicit data scaling curves — it trains at one data volume (14.8k hours) and compares performance, rather than systematically varying data size and showing that the diffusion advantage grows as data decreases. Such curves would be stronger evidence for the data learner hypothesis. The current evidence is suggestive but not conclusive; it could also be that dLLMs simply work better than AR models at this specific data volume for reasons unrelated to data efficiency (e.g., better pretraining of the LLaDA backbone). The single-data-point comparison limits the strength of the claim.

Innovation 3: VRPO as a Bridge Between Preference Optimization and Diffusion Models for Multimodal Tasks

The adaptation of variance-reduced preference optimization (VRPO) to audio-conditioned diffusion models represents a modest but practically significant methodological contribution that addresses a gap in the dLLM ecosystem. When DIFFA-2 was developed, the dLLM literature had established supervised fine-tuning (via LLaDA's corrupted-response objective) and inference acceleration (via fast-dLLMs), but no prior work had demonstrated preference-based alignment for diffusion models in a multimodal setting. The VRPO paper (Zhu et al., 2025) introduced the method for text-only dLLMs; DIFFA-2 is the first to apply it to a model with audio input.

This matters because preference optimization has become a standard component of the LLM training pipeline — essentially all production models from GPT-4 to Llama 3 to Qwen undergo RLHF or DPO after SFT. If dLLMs could not support preference optimization, they would be fundamentally limited in their ability to produce aligned, helpful, and safe outputs, regardless of their raw accuracy on benchmarks. DIFFA-2's VRPO integration closes this gap, showing that the variance-reduction techniques developed for text dLLMs (shared masking patterns between policy and reference models, K-sample Monte Carlo ELBO estimation) transfer straightforwardly to the audio setting without modification.

The conceptual insight — modest but real — is that the bottleneck for dLLM preference optimization is estimation variance, not model architecture. One might have worried that the addition of audio inputs (which produce variable-length prefix sequences) would interact badly with the ELBO estimation procedure, since longer prefixes mean each denoising step processes more tokens and ELBO estimates might become noisier. The success of VRPO with only 3,000 preference pairs (an order of magnitude fewer than typical text-domain DPO datasets) and a single epoch of training suggests the variance reduction is sufficient to stabilize learning even with multimodal inputs and limited preference data.

The evidence for VRPO's effectiveness is in Table 5: the Stage 3 → Stage 4 transition improves MMSU Overall from 59.41 to 60.45 (+1.04) and MMAU Test-mini from 68.20 to 69.60 (+1.40), with consistent gains across all categories and no regressions. These are small improvements — preference optimization typically yields diminishing returns after strong SFT — but they are consistent and, critically, they don't damage the model's existing capabilities. This is not a breakthrough result; it is a reliability demonstration that rounds out the dLLM training toolkit.

A potential criticism is that VRPO might not be necessary — the SFT-only DIFFA-2 (S3) already matches or surpasses AR baselines on most metrics, and the +1 point gain from VRPO might not justify the additional complexity of preference data construction and optimization. The paper does not address this cost-benefit question, and for practitioners with limited resources, skipping Stage 4 might be a reasonable tradeoff. However, for applications where response quality and alignment matter (e.g., avoiding plausible-but-wrong answers about audio, handling ambiguous questions gracefully), VRPO provides a path forward that simply did not exist for diffusion-based audio models before this work.

Innovation 4: Repositioning the AR-vs-Diffusion Debate from "Is It Possible?" to "Under What Conditions Is Each Preferable?"

The paper's most significant rhetorical and intellectual contribution may be its reframing of the autoregressive versus diffusion debate in audio language modeling. Prior to DIFFA-2, the field's implicit question was: "Can diffusion models work at all for audio understanding?" The answer from DIFFA was a tentative "yes, in a proof-of-concept setting." DIFFA-2 moves the conversation forward by asking a more nuanced and productive question: "Given that both AR and diffusion backbones are viable, under what conditions — data budgets, latency requirements, task types — should a practitioner choose one over the other?"

This reframing is supported by the paper's habit of presenting diffusion not as universally superior but as offering a different tradeoff profile. The Stage 1 ASR results (Table 4) are honest about AR's advantage for pure transcription: LLaMA-Audio achieves lower WER (2.43 vs. 2.72 on clean speech) because strictly left-to-right decoding is naturally suited to monotonic speech-to-text mapping. But the paper immediately complicates this picture: with factor-based parallel decoding, the diffusion model's RTF drops below the AR baseline (0.0820 vs. 0.1402), offering a latency advantage at the cost of a small accuracy penalty (WER 3.05 vs. 2.43). The message is not "diffusion wins" or "AR wins" but "you can tune the dial depending on whether you care more about latency or accuracy."

Similarly, the difficulty-dependent patterns in audio understanding — where diffusion shows larger gains on MMSU Reasoning (+4.10 over AR at Stage 3) but smaller gaps on Perception — suggest that the generative paradigm matters more for tasks requiring integration of multiple acoustic cues than for tasks requiring precise local recognition. This is consistent with the architectural strengths of diffusion (bidirectional context, iterative refinement) and AR (efficient sequential processing), but the paper does not belabor this theoretical connection — it lets the data speak.

The VoiceBench results (Appendix C, Table C.1) further reinforce the reframing by showing where DIFFA-2 is clearly not the right choice: on dialogue-style spoken interaction, it scores 59.63 versus GPT-4o-Audio's 86.43 and Qwen2.5-Omni's 74.04. Rather than treating this as a failure, the paper positions it as a scope clarification: DIFFA-2 is designed for audio understanding, not conversational dialogue. An AR model heavily tuned for dialogue will outperform it on dialogue benchmarks; a diffusion model optimized for fine-grained audio reasoning may excel on understanding tasks. The insight is that the "best" generative paradigm is task-dependent, not universal.

This is a more mature and useful framing than the "AR is dead" or "diffusion is a gimmick" narratives that sometimes characterize paradigm-shift debates. It acknowledges the strengths of both approaches and points toward a future where model selection is guided by the specific requirements of the deployment context rather than by a one-size-fits-all preference for a particular generative paradigm. The paper's contribution is not proving diffusion's superiority but establishing it as a legitimate point on the tradeoff frontier, enabling practitioners to make informed choices rather than defaulting to AR out of habit or necessity.

5. Experimental Analysis

Evaluation Methodology

Dataset. The primary evaluations are on three audio understanding benchmarks: MMSU (5,000 audio–question–answer triplets across 47 tasks covering linguistic and paralinguistic perception and reasoning, from Wang et al., 2025b), MMAU (human-annotated multiple-choice questions over speech, music, and environmental sounds emphasizing high-level reasoning, from Sakshi et al., 2025, with results reported on both Test-mini and Test splits), and MMAR (multi-task audio reasoning benchmark with single-modality and mixed-modality subsets, from Ma et al., 2025). A supplementary evaluation is performed on VoiceBench (Chen et al., 2024b), which measures semantic dialogue ability rather than audio understanding and is included only as an auxiliary assessment.
Base model(s). The core generative backbone is LLaDA-8B-Instruct, an 8-billion-parameter diffusion large language model pretrained on text and instruction-tuned. It serves as the diffusion backbone that DIFFA-2 adapts for audio-conditioned generation. The audio encoder is a frozen Whisper-Large-V3 (637M parameters). For the ablation comparing diffusion and autoregressive backbones under matched conditions (Table 5), an LLaMA 3.1-8B is used as the AR backbone, trained with identical data and curriculum to produce the LLaMA-Audio baseline.
Metrics. The primary metric across all audio understanding benchmarks is accuracy (%) — the fraction of questions for which the model's selected answer matches the ground truth. On MMSU, results are further broken down by domain (Semantics, Phonology, Paralinguistics) and by task type (Perception vs. Reasoning), providing fine-grained insight into which audio capabilities improve. On MMAU, results are broken down by modality (Sound, Music, Speech) for both Test-mini and Test splits. On MMAR, results are reported for three single-modality subsets (Sound, Music, Speech), three two-modality mixtures (Sound–Music, Sound–Speech, Music–Speech), and one three-modality mixture (All), plus an overall average. For the Stage 1 ASR analysis (Table 4), the metric is Word Error Rate (WER%) on LibriSpeech-clean and LibriSpeech-other test sets, alongside Real-Time Factor (RTF) — seconds of computation per second of audio, measuring inference latency.
Baselines. The paper compares against both proprietary and open-source models spanning a wide range of scales and architectures. Proprietary baselines include GPT-4o-Audio (OpenAI et al., 2024) and Gemini 2.0 Flash (Team et al., 2025). Among open-source models, the primary competitors at comparable scale (7–8B parameters) are Qwen2.5-Omni (Xu et al., 2025a), Kimi-Audio (Ding et al., 2025), MiniCPM-O (Team, 2025), Qwen2-Audio (Chu et al., 2024), and the first-generation DIFFA (Zhou et al., 2025). Larger models include Qwen3-Omni (30B-A3B, Xu et al., 2025b). Additional baselines evaluated on specific benchmarks include Phi-4-multimodal, Baichuan-Omni-1.5, Baichuan-Audio, GLM-4-Voice, Qwen-Audio-Chat, Salmonn (7B and 13B), LTU, and several omni/voice models (Step-Audio, LLaMA-Omni, Slam-Omni, Freeze-Omni, Mini-Omni/Mini-Omni2, Moshi, DiVA, VITA) for VoiceBench. The full list of 23 baseline models appears in Table B.1.
Generation budget / compute accounting. The paper does not use a unified "generation budget" metric as in text-domain diffusion-vs-AR comparisons. Instead, compute is accounted for implicitly through model size and training data volume: DIFFA-2 uses an 8B backbone with only ~1.1% trainable parameters (99M out of 8.77B) and 14.8k total hours of training data (11k ASR + 3.8k SFT). Comparisons to other models are based on reported benchmark scores at their respective scales, with attention to whether models are of comparable size (7–11B) or substantially larger. For Stage 1 ASR analysis, inference efficiency is compared via RTF, measuring the wall-clock time per second of audio on identical hardware (single NVIDIA A100 GPU). The factor-based parallel decoding experiments explicitly trade off accuracy (WER, benchmark scores) against latency (RTF, total denoising steps), with the decoding factor f = 1.0 used throughout.
Cross-validation / statistical protocol. The paper does not employ cross-validation or report confidence intervals. All benchmark evaluations are performed on the standard test splits of each dataset (MMSU: 5,000 triplets; MMAU Test-mini and Test; MMAR single and mixed modality subsets; VoiceBench test split). The Stage 1 ASR evaluation uses the standard LibriSpeech clean/other test sets. The multi-stage ablation (Table 5) reports results at each training stage on the same benchmark splits, allowing comparison of intermediate checkpoints but without statistical testing of the observed differences. The absence of confidence intervals or significance tests means the reported score differences of 0.3–1.4 points between model variants should be interpreted as suggestive rather than statistically confirmed, particularly given the relatively small per-category sample sizes (MMSU breaks 5,000 questions across 47 tasks and 6 cross-domain categories, yielding roughly 80–800 questions per cell).

Main Quantitative Results

MMSU: Fine-Grained Spoken Language Understanding

Table 1 reports a comprehensive breakdown on MMSU across perception and reasoning dimensions in Semantics, Phonology, and Paralinguistics domains. DIFFA-2 achieves an overall accuracy of 60.45, which places it as the best-performing open model of comparable size: Kimi-Audio (7B) scores 59.28, Qwen2.5-Omni (7B) scores 59.09, and MiniCPM-O (8B) scores 56.53. The gap to the much larger proprietary Qwen3-Omni (30B-A3B, 65.63) is approximately 5 points, while DIFFA-2 surpasses GPT-4o-Audio (56.38) and Gemini 2.0 Flash (51.03) by 4.07 and 9.42 points respectively.

The perception and reasoning breakdown reveals where DIFFA-2's strengths lie. On Perception (average across semantics, phonology, and paralinguistics), DIFFA-2 achieves 45.58 — the highest among all models in Table 1, including the larger Qwen3-Omni (53.20) and GPT-4o-Audio (39.67). The paralinguistic perception score of 41.92 is particularly notable: it exceeds Qwen2.5-Omni (33.20) by 8.72 points, Kimi-Audio (35.74) by 6.18 points, and even Qwen3-Omni (38.85) by 3.07 points. This suggests the acoustic adapter's Q-former design is successfully extracting prosodic, emotional, and speaker-specific cues that other models — including larger AR ones — may be missing or underweighting. Semantic perception (60.63) is competitive with Qwen2.5-Omni (61.11) and substantially ahead of the first-generation DIFFA (52.67), confirming that the semantic adapter's temporal subsampling and projection are effective for content-level understanding.

On Reasoning (average across the three domains), DIFFA-2 achieves 76.40 — again the highest among open models, surpassing Kimi-Audio (76.03) and Qwen2.5-Omni (75.21), though trailing Qwen3-Omni (78.88) by 2.48 points. The semantic reasoning subscore (85.29) and phonological reasoning subscore (77.58) are the strongest among all models regardless of size, exceeding Qwen3-Omni's 86.64 and 82.19 respectively by a hair in phonology while matching closely in semantics. However, paralinguistic reasoning (43.58) — while competitive with open models — lags behind Kimi-Audio (55.22) by 11.64 points, identifying a clear weakness in DIFFA-2's ability to reason about paralinguistic information even though it perceives it well (paralinguistic perception: 41.92 vs. Kimi-Audio's 35.74). This perception–reasoning gap for paralinguistics suggests that the model can detect acoustic cues but struggles to integrate them into higher-level inferences — a pattern that the preference optimization in Stage 4 was specifically designed to address but evidently didn't fully close.

The first-generation DIFFA scores 56.04 overall, placing it behind essentially all contemporary open AR LALMs. DIFFA-2's improvement represents a +4.41 point gain over DIFFA overall, with the largest contributions coming from semantic perception (+7.96), paralinguistic perception (+6.80), and phonological reasoning (+4.90). The improved acoustic encoder (Whisper-Large-V3 vs. the original DIFFA's smaller Whisper variant), the dual-adapter design, and the four-stage training curriculum together produce gains across all nine subcategories with no regressions.

The factor-based parallel decoding variant (DIFFA-2 w/ FPD) scores 60.10 overall, a −0.35 point drop from standard decoding. The degradation is concentrated in semantic reasoning (−0.27) and paralinguistic reasoning (−1.49), consistent with the hypothesis that factor-based decoding's adaptive token acceptance introduces slight errors that propagate more in tasks requiring multi-step inference than in direct perception tasks.

MMAU: Multi-Task Audio Understanding Across Modalities

Table 2 reports results on MMAU, which tests understanding across sound, music, and speech through human-annotated multiple-choice questions. DIFFA-2 achieves 67.00 average on the Test split and 69.60 on Test-mini, establishing it as the top open model on this benchmark: Qwen2.5-Omni scores 66.64 (Test), Kimi-Audio scores 64.40, and MiniCPM-O scores 66.70. The gap to larger proprietary models is modest — Qwen3-Omni leads at 70.77, Gemini 2.0 Flash at 67.03, and GPT-4o-Audio at 60.82 — with DIFFA-2 actually surpassing Gemini on Test-mini (69.60 vs. 70.50, a narrower gap) and essentially tying it on the full Test split (67.00 vs. 67.03).

Breaking down by modality on the Test split, DIFFA-2 excels at sound and speech:

Sound (Test): 70.83 — the highest among all models, exceeding Qwen3-Omni's 73.70 by 2.87 points (though note Qwen3-Omni's 73.70 is on Test-mini; the Test split shows 78.68 on Test-mini vs. DIFFA-2's 76.28, but on Test DIFFA-2's 70.83 edges out Qwen2.5-Omni's 69.53 and Kimi-Audio's 70.70). The strong sound performance suggests the dual-adapter design is effectively capturing both semantic content (what sound is occurring) and acoustic texture (the qualities that distinguish similar sounds).
Speech (Test): 70.18 — second only to Gemini 2.0 Flash (72.87) among all models and ahead of both Qwen2.5-Omni (67.93) and Kimi-Audio (56.57, an unexpectedly low score that may indicate Kimi-Audio's design optimizes for different speech tasks). The +10.72 point advantage over DIFFA's original speech performance (59.46 on Test-mini, with Test split not reported for DIFFA) confirms that the upgraded encoder and training curriculum dramatically improve speech understanding.
Music (Test): 60.10 — competitive but not leading, falling behind Kimi-Audio (65.93), MiniCPM-O (65.57 on Test-mini, Test split not reported), and Qwen2.5-Omni (62.50). The paper notes that DIFFA-2 achieves this "without any music-specialized design," meaning the same dual-adapter architecture handles music through the general acoustic pathway. The 16.69-point gain over DIFFA (43.41 on Test-mini, Test not reported) is substantial but leaves room for music-specific adaptation strategies.

The Test-mini vs. Test split pattern reveals where DIFFA-2 is most robust: on Sound, the drop from Test-mini to Test is 76.28 → 70.83 (−5.45); on Speech, 69.06 → 70.18 (+1.12, actually improving); on Music, 63.47 → 60.10 (−3.37). The speech improvement from mini to full test is unusual and may reflect differences in question difficulty distribution between the two splits. The music drop, while present, is proportionally similar to other models' drops (Qwen3-Omni: 69.46 → 72.22 on mini to test for music, actually gaining — suggesting Test-mini music may be harder than Test for some benchmarks, an anomaly worth noting).

The factor-based parallel decoding variant achieves 66.34 on Test (−0.66) and 68.30 on Test-mini (−1.30), showing that the accuracy–latency tradeoff is slightly more pronounced on MMAU than on MMSU. This aligns with MMAU's emphasis on fine-grained multiple-choice discrimination, where even small perturbations in token-level confidence estimation can flip an answer choice.

Compared to DIFFA, the improvement is dramatic across the board. On Test-mini, DIFFA-2 scores 69.60 versus DIFFA's 49.71 — a +19.89 point gain. The per-modality gains are: Sound 76.28 vs. 46.25 (+30.03), Music 63.47 vs. 43.41 (+20.06), Speech 69.06 vs. 59.46 (+9.60). The asymmetry — sound improving most, speech improving least — is consistent with the hypothesis that DIFFA's speech-centric training left sound and music understanding severely underdeveloped, and that Stage 2's incorporation of diverse audio QA data plus the acoustic adapter's design disproportionately benefit non-speech modalities.

Table 3 evaluates a more challenging setting: reasoning over single audio modalities and their mixtures. DIFFA-2 achieves an overall average of 50.80%, substantially improving over DIFFA (37.20%, +13.60 points) and outperforming MiniCPM-O (48.60%), Baichuan-Omni-1.5 (40.70%), and Qwen2-Audio (30.00%). However, it trails Qwen2.5-Omni (51.40%) by 0.60 points and both Qwen3-Omni (65.90%) and Gemini 2.0 Flash (65.60%) by approximately 15 points, reflecting the difficulty of MMAR's compositional reasoning tasks.

On single-modality subsets, DIFFA-2 performs competitively:

Sound (54.55%): Slightly behind Qwen2.5-Omni (55.76%) and Gemini (61.21%), but ahead of all other open models. The gain over DIFFA is 54.55 − 37.58 = +16.97 points.
Music (41.75%): Essentially tied with Qwen2.5-Omni (41.75%) — an exact match — and ahead of MiniCPM-O (36.41%) and DIFFA (31.07%). The gain over DIFFA is +10.68 points. The parity with Qwen2.5-Omni on music is notable given DIFFA-2's lack of music-specific design.
Speech (53.40%): Slightly behind Qwen2.5-Omni (54.42%) and well behind Gemini (72.11%) and Qwen3-Omni (70.41%). The gain over DIFFA is 53.40 − 39.46 = +13.94 points. Speech on MMAR lags more relative to AR competitors than on MMSU or MMAU, suggesting that MMAR's speech tasks may require capabilities that the AR sequential decoding advantage (precise local transcription) benefits more than the diffusion bidirectional context.

On two-modality mixtures, DIFFA-2 shows particular strength:

Sound–Speech (58.26%): Exceeds Qwen2.5-Omni (55.96%) and all other open models, approaching GPT-4o-Audio (72.48%).
Sound–Music (45.45%): Ties Qwen2.5-Omni exactly and exceeds other open models, though far behind Qwen3-Omni (90.91%) — a score that suggests Qwen3-Omni may have been trained on data very similar to these mixture types.
Music–Speech (54.88%): Competitive with but behind Qwen2.5-Omni (57.32%).

The key weakness emerges in the three-modality mixture (All) category: DIFFA-2 scores only 37.50%, substantially below Qwen2.5-Omni (54.17%), Qwen3-Omni (70.83%), Gemini (70.83%), and GPT-4o-Audio (75.00%). The paper acknowledges this shortfall explicitly: "complex three-way mixtures remain a challenging regime compared with the strongest AR-based LALMs" and attributes it to "the lack of mixed-modality supervision in the training data." This is a concrete limitation: the training data consists of single-modality audio datasets (speech corpora, sound event datasets, music collections) and two-modality audio QA pairs, but no explicit training on audio containing simultaneous speech, music, and environmental sounds. The model must compose its single-modality knowledge to handle these mixtures, and the 37.50% score suggests this composition is far from perfect.

A curious pattern: DIFFA-2 outperforms Qwen2.5-Omni on Sound–Speech (58.26 vs. 55.96) and equals it on Sound–Music (45.45 vs. 45.45), but performs worse on the three-way All mixture (37.50 vs. 54.17). This suggests the difficulty is not simply additive — having three modalities simultaneously does not just triple the chance of error but introduces interaction effects that DIFFA-2's training distribution did not cover. Qwen2.5-Omni's advantage on All despite similar two-way scores suggests its training may have included multi-source audio data.

The factor-based parallel decoding variant scores 50.20 overall (−0.60), with the largest drops on Music single-modality (41.75 → 36.41, −5.34) and Sound–Music mixed (45.45 → 36.36, −9.09) — categories where fine-grained acoustic discrimination matters most and where confidence-based token selection may introduce errors that compound across the music-reasoning pipeline.

Stage 1 ASR: Diffusion vs. Autoregressive Transcription

Table 4 provides an ablation on a pure ASR task using only Stage 1 training, comparing diffusion and AR backbones. The LLaMA-Audio (S1) AR baseline achieves WER 2.43 on clean and 5.09 on other, while DIFFA-2 (S1) scores 2.72 on clean and 5.34 on other — a small but consistent AR advantage of 0.29 and 0.25 WER points respectively. This is expected: strictly left-to-right decoding aligns naturally with monotonic speech-to-text mapping. However, on latency, the picture reverses with factor-based decoding: DIFFA-2 (S1, standard) has RTF 0.6792 (clean) / 0.7489 (other), much slower than LLaMA-Audio's 0.1402 / 0.1418. But DIFFA-2 (S1 w/ FPD) achieves RTF 0.0820 / 0.0867 — faster than the AR baseline by 1.7× on clean speech and 1.6× on other — while WER increases only to 3.05 / 5.68. This demonstrates that with appropriate parallel decoding, the diffusion backbone's latency can be lower than AR for ASR at a modest accuracy cost.

The ASR analysis serves two purposes: (1) it establishes a performance floor — even with a frozen backbone and semantic adapter only, DIFFA-2 achieves reasonable ASR quality; (2) it quantifies the accuracy–latency tradeoff that factor-based decoding enables, which is not available to AR models (whose latency is fundamentally limited by the per-token sequential generation bottleneck). The paper does not claim diffusion is better for ASR; rather, it shows diffusion is competitive enough to serve as the foundation for more complex audio understanding tasks while offering latency flexibility that AR cannot match.

Multi-Stage Training Progression: Diffusion vs. AR Under Matched Conditions

Table 5 is the paper's most controlled experiment, comparing DIFFA-2 and LLaMA-Audio (an AR equivalent with LLaMA 3.1-8B backbone) at identical training stages with identical data. The results trace the performance trajectory from Stage 2 (adapter-only alignment, backbone frozen) through Stage 3 (LoRA backbone fine-tuning) to Stage 4 (VRPO preference optimization for DIFFA-2 only).

At Stage 2, before any backbone adaptation, DIFFA-2 already outperforms LLaMA-Audio on both benchmarks:

MMAU Overall: DIFFA-2 (S2) 63.90 vs. LLaMA-Audio (S2) 60.80 (+3.10)
MMSU Overall: 56.43 vs. 43.71 (+12.72)

The MMSU gap is dramatic — 12.72 points overall, driven by a 20.05-point advantage in MMSU Reasoning (75.50 vs. 55.45) and a 5.85-point advantage in MMSU Perception (38.54 vs. 32.69). The frozen backbone setting means both models have identical adapter architectures and identical training data; the performance difference arises purely from the generative paradigm. This is the strongest evidence in the paper for the "data learner" hypothesis: the diffusion backbone's corruption–reconstruction pre-training extracts more useful representations from the same audio-text training data than the AR backbone's next-token prediction pre-training.

Breaking down Stage 2 MMSU Perception more finely reveals where the diffusion advantage concentrates:

Semantic perception: 52.91 vs. 39.21 (+13.70)
Phonological perception: 36.58 vs. 30.91 (+5.67)
Paralinguistic perception: 31.32 vs. 30.23 (+1.09)

The semantic perception gap (+13.70) is massive. This suggests the diffusion backbone's bidirectional context is particularly helpful for extracting linguistic content from the semantic adapter's temporally-resolved output — intuitively, having access to both past and future audio context when reconstructing masked transcription tokens provides stronger supervision than left-to-right prediction. The much smaller paralinguistic advantage (+1.09) suggests that both backbones struggle similarly with the acoustic adapter output when frozen — extracting prosodic information may require the backbone to learn adapter-specific attention patterns, which only becomes possible in Stage 3 when LoRA is introduced.

At Stage 3, both models are fine-tuned with LoRA on the SFT data:

MMAU Overall: DIFFA-2 (S3) 68.20 vs. LLaMA-Audio (S3) 67.40 (+0.80)
MMSU Overall: 59.41 vs. 55.31 (+4.10)

The MMSU gap narrows somewhat from Stage 2 (12.72 → 4.10) as the AR backbone catches up through fine-tuning, but a substantial advantage remains. The MMSU Perception gap actually widens: 44.16 vs. 41.22 (+2.94), driven by continued improvement in semantic perception (59.53 vs. 50.39, +9.14) and paralinguistic perception (40.63 vs. 36.67, +3.96). This widening of the perception gap suggests that the diffusion backbone benefits more from LoRA fine-tuning for perceptual tasks than the AR backbone does — the LoRA modules may be learning to attend specifically to the acoustic adapter output in ways that complement the bidirectional pre-training.

The reasoning gap narrows: 75.70 vs. 70.33 (+5.37), down from +20.05 at Stage 2. The AR backbone catches up substantially in reasoning when fine-tuned, suggesting that left-to-right generation is not inherently disadvantaged for reasoning once the backbone has adapted to audio inputs — the Stage 2 gap may have reflected poor adapter alignment more than a fundamental reasoning limitation.

On MMAU, the modality-specific Stage 2 → Stage 3 improvements for DIFFA-2 are: Sound 72.07 → 74.77 (+2.70), Music 52.69 → 62.57 (+9.88), Speech 66.97 → 67.27 (+0.30). Music benefits disproportionately — a 9.88-point gain — suggesting that the acoustic adapter's music-relevant features are not fully utilized until the backbone learns (via LoRA) to attend to them. Speech barely improves, indicating that the semantic adapter already provides sufficient speech information at Stage 2 and backbone fine-tuning adds little.

At Stage 4, DIFFA-2 applies VRPO while LLaMA-Audio has no equivalent stage. The improvements are:

MMAU Overall: 68.20 → 69.60 (+1.40)
MMSU Overall: 59.41 → 60.45 (+1.04)

The gains are modest but consistent across all categories, with no regressions anywhere. The fact that these gains appear on MMAU Sound (74.77 → 76.28, +1.51) and Speech (67.27 → 69.06, +1.79) but only marginally on Music (62.57 → 63.47, +0.90) aligns with the paper's preference data construction: subtle errors in sound event identification and speaker characteristics are easier for the LLM to simulate when generating rejected responses than music-specific errors (tempo, key, instrumentation), so the preference pairs may be higher-quality for sound and speech than for music.

The key takeaway from Table 5 is that the diffusion backbone consistently outperforms the AR backbone at every stage under matched conditions, with the advantage being largest before fine-tuning (+12.72 MMSU at Stage 2), narrowing but remaining meaningful after LoRA (+4.10 at Stage 3), and extending further with preference optimization. This pattern supports the paper's central claim that dLLMs are viable alternatives to AR models for audio understanding, while also honestly showing that the advantage is not infinite — well-tuned AR models narrow the gap, and on some metrics (MMAU Stage 3, where the gap is only +0.80), the difference is small enough that other factors (inference latency, deployment complexity) would dominate the choice.

Ablation Studies and Robustness Checks

Diffusion vs. autoregressive backbone on pure ASR (Table 4, Stage 1): Under identical ASR-style training, the AR backbone (LLaMA-Audio S1) achieves slightly lower WER than the diffusion backbone (DIFFA-2 S1) on both LibriSpeech-clean (2.43 vs. 2.72) and LibriSpeech-other (5.09 vs. 5.34), confirming that strictly left-to-right decoding retains an advantage for monotonic transcription. However, the diffusion backbone with factor-based parallel decoding achieves substantially lower RTF than the AR baseline (0.0820 vs. 0.1402 on clean), demonstrating that the latency advantage can outweigh the accuracy penalty in latency-sensitive ASR applications.

Factor-based parallel decoding across three benchmarks (Tables 1, 2, 3; "w/ FPD" rows): Enabling factor-based parallel decoding degrades accuracy by small and consistent margins: −0.35 on MMSU (60.45 → 60.10), −1.30 on MMAU Test-mini (69.60 → 68.30), and −0.60 on MMAR (50.80 → 50.20). The larger drop on MMAU Test-mini may reflect that benchmark's reliance on fine-grained multiple-choice discrimination where token-level confidence estimation is more error-prone. The overall pattern confirms that the factor f = 1.0 provides a practical accuracy–latency tradeoff without fundamentally compromising the model's capabilities, and that the accuracy impact varies by task type.

Multi-stage training progression comparing diffusion and AR (Table 5, LLaMA-Audio vs. DIFFA-2): This is the central ablation establishing that the diffusion backbone, not just the dual adapters or training data, drives DIFFA-2's performance. Under identical data and training stages, DIFFA-2 (S2) outperforms LLaMA-Audio (S2) by 3.10 points on MMAU and 12.72 points on MMSU before any backbone fine-tuning, demonstrating that the diffusion pre-training objective alone provides an advantage for audio understanding. After LoRA fine-tuning (S3), DIFFA-2 maintains leads of 0.80 points on MMAU and 4.10 on MMSU, showing the advantage persists but narrows. The VRPO stage (S4) adds +1.40 on MMAU and +1.04 on MMSU for DIFFA-2.

Stage-by-stage within DIFFA-2 (Table 5, DIFFA-2 rows only): Tracing DIFFA-2's own progression, Stage 2 → Stage 3 (unfreezing backbone with LoRA) provides the largest single-stage gain: MMAU from 63.90 to 68.20 (+4.30), MMSU from 56.43 to 59.41 (+2.98). Within MMSU, the perception gain is 38.54 → 44.16 (+5.62), driven predominantly by paralinguistic perception (31.32 → 40.63, +9.31) — the category most dependent on the acoustic adapter. This confirms that LoRA fine-tuning primarily improves the model's ability to utilize the acoustic adapter's output, which the frozen backbone could not effectively process. Stage 3 → Stage 4 (VRPO) adds smaller but consistent gains: MMAU +1.40, MMSU +1.04.

Impact of backbone adaptation on different modalities (Table 5, MMAU): From Stage 2 to Stage 3, Music shows the largest gain (+9.88, from 52.69 to 62.57), Sound shows moderate gain (+2.70, from 72.07 to 74.77), and Speech shows minimal gain (+0.30, from 66.97 to 67.27). This asymmetry suggests that the semantic adapter already provides sufficient speech information at Stage 2, leaving little room for improvement, while the acoustic adapter's music-relevant features require backbone fine-tuning to be effectively utilized — consistent with music understanding depending more on holistic acoustic properties (timbre, harmony, rhythm) that the Q-former acoustic adapter captures but the frozen backbone cannot initially interpret.

VoiceBench as a domain transfer test (Appendix C, Table C.1): DIFFA-2 scores 59.63 on VoiceBench — a conversational dialogue benchmark — substantially below heavily instruction-tuned omni models (GPT-4o-Audio: 86.43, Qwen2.5-Omni: 74.04) but competitive with or better than open-source baselines that also lack extensive dialogue tuning (GLM-4-Voice: 55.99, DiVA: 55.70, Qwen2-Audio: 55.34). The gap between DIFFA-2's strong MMSU/MMAU/MMAR performance and mid-range VoiceBench performance validates the paper's scoping: DIFFA-2 is designed for audio understanding, not dialogue, and the training data reflects this priority. The +11.41 point improvement over DIFFA (48.22 → 59.63) shows that the enhanced training pipeline incidentally improves dialogue ability even without targeted dialogue data, but the large gap to omni models confirms that dialogue-specific data and tuning are necessary for conversational voice assistant applications — a limitation the paper explicitly acknowledges.

Scaling of the acoustic adapter (by design, not through explicit ablation): The paper does not ablate different acoustic adapter sizes or architectures (e.g., varying the number of Q-former query vectors from 64, testing a simple pooling alternative). However, the strong paralinguistic perception results on MMSU (41.92, best among all models including larger ones) provide indirect evidence that the 64-query Q-former design is effective for capturing paralinguistic cues. A missing ablation that would strengthen the dual-adapter claim is a comparison to a single adapter with equivalent total capacity — this would isolate whether the performance gain comes from the decomposition into semantic/acoustic pathways or simply from having more adapter parameters.

VRPO preference data quality (by construction, not explicit ablation): The paper constructs preference pairs automatically by prompting an LLM to generate responses with subtle audio-related errors, keeping only pairs where the reference is "unambiguously superior." The 3,000 retained pairs is a small number relative to typical text-domain DPO datasets (often 10,000–100,000 pairs). The paper does not ablate different preference data sizes or quality thresholds. The consistent but small Stage 4 gains (+1.04–1.40 points) could reflect either the inherent difficulty of improving an already-strong SFT model through preference optimization, or the limited size/noise in the preference data — distinguishing these would require varying the preference dataset size, an experiment not performed.

Critical Assessment

Does DIFFA-2 genuinely demonstrate that diffusion backbones are "competitive" with AR LALMs?

Yes, with qualifications about what "competitive" means. On MMSU (Table 1), DIFFA-2's 60.45 exceeds Qwen2.5-Omni's 59.09 and Kimi-Audio's 59.28 — the two strongest open AR LALMs of comparable size. On MMAU (Table 2), DIFFA-2's 67.00 Test average leads all open models. On MMAR (Table 3), DIFFA-2's 50.80 trails Qwen2.5-Omni's 51.40 by a negligible 0.60 points. These are genuine competitive results, placing DIFFA-2 at the top of the 7–8B open LALM leaderboard across three diverse audio understanding benchmarks.

However, the paper's scope is carefully delimited to audio understanding, and the VoiceBench results (Table C.1) make clear that DIFFA-2 is not competitive for spoken dialogue — it scores 59.63 versus Qwen2.5-Omni's 74.04 and GPT-4o-Audio's 86.43. This is not a failure given the paper's explicit focus, but it means the "competitive" claim applies only to the subset of audio tasks the model was designed and trained for. An AR omni model like Qwen2.5-Omni is competitive on both audio understanding and dialogue, while DIFFA-2 is competitive only on the former. The paper honestly scopes its contribution, but readers should not interpret "competitive" as "equally capable across all audio tasks."

Additionally, the paper does not compare to a compute-matched AR baseline — that is, an AR model trained with equivalent total FLOPs (including the extra computation from the diffusion backbone's iterative training with multiple masking rates). DIFFA-2's 8B diffusion backbone was pretrained on text with a certain compute budget; LLaMA-Audio's 8B AR backbone was pretrained with a potentially different budget. The comparison in Table 5 controls for training data in Stages 1–3 but does not control for pre-training compute. If LLaDA-8B required more pre-training FLOPs than LLaMA-3.1-8B to reach comparable text performance, then the Stage 2 advantage might partially reflect better pre-training rather than any inherent diffusion property. The paper does not discuss the pre-training compute budgets, making the "data learner" interpretation (diffusion extracts more from limited fine-tuning data) confounded with possible pre-training differences.

Is the "dual-adapter decomposition" claim supported by the evidence?

Partially. The ablation evidence shows that DIFFA-2 with dual adapters substantially outperforms DIFFA (which used a single projection-based adapter) — +4.41 MMSU, +19.89 MMAU Test-mini, +13.60 MMAR. This is strong evidence that something about the upgraded architecture and training improves performance. But the paper does not include an ablation with a single adapter of comparable total parameter count (e.g., a single larger Q-former or a single convolution + projection with equivalent capacity to the 84.3M combined adapters). Without this, we cannot distinguish between two hypotheses: (1) the dual-pathway decomposition into semantic and acoustic streams is the causal factor, or (2) having more adapter parameters (84.3M vs. DIFFA's presumably smaller single adapter) is sufficient. The paralinguistic perception gains (41.92 on MMSU, best among all models) are suggestive of Hypothesis 1 — a single adapter with equivalent capacity might struggle to simultaneously preserve temporal structure for content and distill global acoustic properties — but this is not experimentally isolated.

A further missing control: the paper does not test whether providing the acoustic adapter's 64-vector output to the AR backbone (LLaMA-Audio) would close the Stage 2 MMSU gap. Table 5 shows LLaMA-Audio uses the same dual adapters but performs much worse at Stage 2 (43.71 vs. 56.43 MMSU). This suggests the diffusion backbone benefits more from the dual-adapter design than the AR backbone, but the mechanism is unclear. Is it that bidirectional context lets the diffusion model integrate the acoustic summary more effectively? Or that the corruption–reconstruction pre-training incidentally learns to handle multiple heterogeneous prefix types? The paper does not analyze this interaction.

Does the four-stage curriculum matter, or would joint training work equally well?

The paper's staged approach is well-motivated (avoid catastrophic forgetting, stabilize adapter alignment before backbone fine-tuning), but no ablation compares staged vs. joint training. We see that Stage 2 (adapters only) → Stage 3 (add LoRA) → Stage 4 (add VRPO) each improve performance, but we do not know what would happen if Stages 2 and 3 were combined (train adapters and LoRA jointly from the start), or if VRPO were applied immediately after Stage 2 (skipping LoRA fine-tuning). It is possible that the staged curriculum is unnecessary — that end-to-end training with appropriate learning rate scheduling would achieve the same or better results in less wall-clock time. The paper's claim that the curriculum is "progressive" is descriptive, not experimentally validated as necessary or optimal.

A related concern: the Stage 1 ASR training uses 11,000 hours of data and 12 epochs (Table B.2), while Stages 2–3 use 3,767 hours and 10 epochs. This means the semantic adapter receives substantially more total gradient updates from ASR than from diverse audio QA. Could this imbalance bias the adapter toward transcription-style representations, limiting its ability to handle paralinguistic and music tasks that require different feature weightings? The Stage 3 results (where paralinguistic perception jumps by +9.31 when the backbone is unfrozen) suggest the frozen backbone was under-utilizing acoustic information that was actually present in the adapter outputs — possibly because the ASR-heavy Stage 1 trained the semantic adapter to prioritize content words over prosodic features. An ablation varying the Stage 1/Stage 2 data ratio would test whether ASR pre-training helps (by providing a stable initialization) or hurts (by biasing representations toward transcription).

Does the VRPO stage provide value beyond SFT?

The Stage 4 gains are small and consistent (+1.04 MMSU, +1.40 MMAU) with no regressions, which is the expected pattern for preference optimization after strong SFT. However, with only 3,000 preference pairs — an order of magnitude fewer than typical DPO datasets — the paper does not establish that VRPO's variance reduction is necessary for these gains. An ablation comparing VRPO to standard DPO (with ELBO estimates, no shared masking) at the same data scale would strengthen the claim that VRPO specifically enables stable preference learning for diffusion models. Without this ablation, we cannot rule out that standard DPO would work equally well or better on this small dataset, and VRPO is included because the authors (of the VRPO paper) developed it for dLLMs.

Moreover, the small gain might not justify the engineering complexity of VRPO for practitioners. The preference data construction pipeline (prompting an LLM to generate plausible-but-incorrect audio answers, filtering for unambiguous superiority) is itself error-prone — the "rejected" responses are generated by a text-only LLM that cannot hear the audio, so it may introduce errors that are not actually plausible given the audio content. This could cap the quality of the preference signal regardless of the optimization algorithm. The paper does not analyze the quality of the generated preference pairs or provide examples of chosen–rejected pairs.

Is the factor-based parallel decoding claim of "practical inference acceleration" fully substantiated?

For ASR, yes: Table 4 shows an 8.3× RTF reduction on LibriSpeech-clean (0.6792 → 0.0820) with only a 0.33 WER increase, and the RTF actually drops below the AR baseline (0.0820 vs. 0.1402). For audio understanding benchmarks, the evidence is partial: the paper reports benchmark scores with factor-based decoding (Tables 1–3, "w/ FPD" rows) and notes accuracy drops of 0.35–1.30 points, but does not report the corresponding latency reductions for these benchmarks. Without knowing the RTF improvement on MMSU, MMAU, and MMAR (which have different answer lengths and block configurations than ASR — Table B.4 shows 16-step decoding vs. 128-step for ASR), we cannot assess whether the accuracy–latency tradeoff is practically attractive. If factor-based decoding reduces MMSU latency by only 10–20% (plausible given the already-small 16-step budget), the 0.35 accuracy drop might not be worthwhile. If it reduces latency by 5×, the tradeoff is compelling. The paper's claim that factor-based decoding provides a "practical knob" is supported in principle but quantitatively incomplete.

What experiments are missing that would strengthen the paper?

Data scaling curves. Train DIFFA-2 and LLaMA-Audio at multiple data volumes (e.g., 25%, 50%, 100% of the 14.8k hours) and plot benchmark performance vs. data size. This would directly test the "data learner" hypothesis — if DIFFA-2's advantage grows as data decreases, that is strong evidence. Currently, the single-data-point comparison is suggestive but not conclusive.
Single-adapter ablation of equivalent capacity. Replace the dual adapters with a single adapter (e.g., a larger Q-former with 128 queries or a deeper convolution + projection network) with approximately 84.3M parameters and retrain. This would isolate the architectural decomposition from the parameter count.
Compute-matched AR baseline. Estimate the total pre-training FLOPs for LLaDA-8B and compare to an AR model (e.g., LLaMA-3.1-8B) with similar total FLOPs rather than similar parameter count. This would control for the possibility that LLaDA-8B's pre-training was simply more compute-intensive.
Difficulty-stratified analysis. Like the reference example's analysis by difficulty bin, the paper could analyze whether DIFFA-2's advantage over AR models varies by question difficulty. Do the gains concentrate on "easy" questions (where audio content is clearly discriminable) or "hard" ones (where subtle acoustic distinctions matter more)?
Combined search + revision analysis for audio. The reference example emphasizes that search and revisions were studied independently, not combined. For DIFFA-2, the analogous question is whether iterative diffusion denoising can be combined with explicit search strategies (e.g., generating multiple candidate answers and selecting via the PRM-like confidence scores from the denoising process) to further boost performance. The current inference is purely iterative denoising without explicit candidate selection or revision chains.
Statistical significance testing. With 5,000 MMSU questions and 0.35–1.40 point differences between variants, standard errors are likely in the 0.3–0.7 percentage point range (depending on the specific subcategory). Without confidence intervals, a 0.35-point difference (DIFFA-2 vs. DIFFA-2 w/ FPD on MMSU) could be noise.
Latency measurements on audio understanding benchmarks. The factor-based decoding table (Tables 1–3 "w/ FPD" rows) shows accuracy drops; the corresponding latency improvements would complete the tradeoff picture.

Summary assessment

DIFFA-2 successfully demonstrates that diffusion-based LALMs can match or exceed comparably-sized AR models on audio understanding benchmarks. The evidence is strongest on MMSU and MMAU, where DIFFA-2 leads the open-model leaderboard, and weakest on VoiceBench (where the dialogue deficit is acknowledged) and MMAR three-way mixtures (where a 16.67-point gap to Qwen2.5-Omni reveals a clear capability boundary). The paper's central insight — that the dual-adapter decomposition and staged curriculum enable the diffusion backbone's data efficiency advantage to transfer to audio — is supported by controlled ablation (Table 5) but incompletely isolated from confounds like pre-training compute and adapter capacity. The factor-based parallel decoding provides a genuine accuracy–latency tradeoff for ASR but lacks benchmark-specific latency measurements for audio understanding tasks. The paper achieves its stated goal of establishing dLLMs as viable audio backbones; the open questions are whether the dual-adapter design is optimal (vs. single large adapter), whether VRPO's variance reduction is necessary (vs. standard DPO), and whether the diffusion advantage persists under compute-matched AR baselines and across wider data scales — all appropriate directions for future work.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Makes the Headline Efficiency Gains Unrealistic for Deployment

The assumption or constraint. The compute-optimal framework that DIFFA-2's training curriculum and architecture embody — particularly the staged progression from ASR to diverse SFT — implicitly assumes that the model can be trained once on a carefully curated data mixture and then deployed. However, the paper makes no estimate of the computational cost of its own training pipeline relative to the performance gains achieved. This is not merely a missing budget calculation. The paper's central narrative is that dLLMs are "super data learners" that extract more value from limited unique data, making them practical for teams without massive proprietary datasets. But the training compute required to realize this data efficiency is substantial: Stage 1 uses 11,000 hours of ASR with batch size 1280 over 12 epochs on 64 A100 GPUs; Stages 2–3 use 3,767 hours of SFT with batch size 196 over 10 epochs each; and then Stage 4 applies VRPO with a separate optimization procedure. The paper reports that "the entire training pipeline takes approximately 5 days to complete" on 64 A100 GPUs (Appendix B.3), which is roughly 7,680 A100-hours. This is a non-trivial compute budget, and the paper never compares this training cost to what AR LALMs require for equivalent performance.

The consequence. A practitioner evaluating whether to adopt DIFFA-2's approach faces an incomplete picture. The paper demonstrates that with 14.8k hours of data, DIFFA-2 achieves competitive results. But if an AR LALM could achieve the same performance with 20k hours of data and half the training compute, the practical advantage of dLLMs' "data efficiency" evaporates. The cost that matters for deployment decisions is total cost: data curation + training compute + inference latency. The paper addresses only the first factor (data volume) and partially the third (inference latency for ASR), while ignoring the second entirely. Furthermore, the 5-day training time is for DIFFA-2's lightweight fine-tuning (only 1.1% of parameters updated). A full-scale dLLM LALM trained from scratch — which would be needed to truly realize the data learner benefits on much larger datasets — would require an entirely different order of compute. The paper's results on data efficiency are established in a fine-tuning regime; whether they transfer to pre-training-scale dLLM audio models is unknown.

What evidence exists in the paper. The training configuration is documented in Table B.2 and Appendix B.3, which provides learning rates, batch sizes, epochs, and hardware count. But no training cost analysis appears anywhere in the paper. There is no comparison of total FLOPs spent training DIFFA-2 vs. training the AR baselines (LLaMA-Audio, Qwen2.5-Omni, Kimi-Audio). The paper reports that DIFFA-2 uses only 14.8k hours of data, but this is a data volume metric, not a compute metric — and the paper's own thesis is that dLLMs use more compute per data point (the "super-dense compute" hypothesis from Ni et al., 2025). Without reporting total training FLOPs, the paper cannot claim that dLLMs are more "practical" — only that they use less unique data, which is a different claim with different implications for practitioners who may be compute-constrained rather than data-constrained.

Mitigation status. The paper does not acknowledge this limitation. It emphasizes parameter efficiency (1.1% trainable) and data volume (14.8k hours) as proxies for practical efficiency, but neither captures the training compute budget. Future work on training-aware comparisons between diffusion and AR LALMs would need to match total FLOPs (not just data volume) and report training curves to establish whether dLLMs' data efficiency translates to compute efficiency.

The Model Is Not Competitive on Dialogue or Spoken Interaction Tasks, Limiting Its Scope to a Narrow Slice of Audio Applications

The assumption or constraint. The paper explicitly scopes DIFFA-2 as an audio understanding model, not a conversational voice assistant. Section 7 (Limitations) states:

"our training objectives and data curation are geared toward fine-grained audio understanding rather than open-domain spoken dialogue. Consequently, DIFFA-2 is exposed to only limited conversational and alignment-style supervision, which is reflected in its mid-range performance on VoiceBench compared with heavily instruction-tuned AR Omni-models."

This is an honest acknowledgement, but it has a deeper implication than the paper fully explores. The audio understanding benchmarks on which DIFFA-2 excels — MMSU, MMAU, MMAR — evaluate a specific capability: answering multiple-choice or short-answer questions about audio clips. They do not test sustained dialogue, multi-turn interaction, instruction following in open-ended spoken contexts, or safety alignment for voice-based queries. Meanwhile, the dominant use case for LALMs in production is precisely spoken dialogue — voice assistants, customer service bots, interactive tutoring systems. The models DIFFA-2 is competing with (Qwen2.5-Omni, Kimi-Audio, GPT-4o-Audio) are designed for this dialogue setting and evaluated on benchmarks that include it.

The consequence. DIFFA-2's strong benchmark results on MMSU/MMAU/MMAR may not translate to the deployment scenarios that practitioners actually care about. A user evaluating whether to build on DIFFA-2 vs. Qwen2.5-Omni needs to know: does the model handle multi-turn conversations where context accumulates? Does it follow complex instructions delivered via speech? Does it refuse harmful requests appropriately? DIFFA-2's VoiceBench score of 59.63 — versus Qwen2.5-Omni's 74.04 and GPT-4o-Audio's 86.43 — provides a stark answer: no, not yet. The +4.41 MMSU advantage over Qwen2.5-Omni must be weighed against the −14.41 VoiceBench disadvantage, and for most production voice assistant applications, the VoiceBench gap is far more consequential.

Furthermore, the paper's framing — "DIFFA-2 is competitive with AR LALMs on audio understanding" — could be misinterpreted as "DIFFA-2 is a viable drop-in replacement for AR LALMs." It is not. It is a viable replacement for the audio understanding component of an LALM, but production systems need dialogue capability, safety alignment, and instruction following that DIFFA-2 lacks. A complete voice assistant built on DIFFA-2 would require substantial additional dialogue fine-tuning, the effectiveness of which is unproven.

What evidence exists in the paper. Table C.1 (VoiceBench) shows DIFFA-2's overall score of 59.63, which places it below not only proprietary omni models but also several open-source baselines (Kimi-Audio: 76.93, Qwen2.5-Omni: 74.04, Phi-4-multimodal: 63.69). The paper acknowledges this in the Limitations section and the main text (Section 5.1, VoiceBench subsubsection) but treats it as a scope clarification rather than a limitation that fundamentally restricts the model's practical applicability. The gap is +11.41 over DIFFA (48.22 → 59.63), showing the training improvements transfer partially to dialogue, but the ceiling remains far below dialogue-specialized models.

Mitigation status. The paper partially mitigates by being explicit about scope and suggesting future work: "Designing a more balanced training recipe that jointly targets audio understanding and spoken dialogue is an important direction for future work." However, no experiments mix dialogue and understanding data to test whether the dual strengths can coexist, and the paper does not analyze whether the dialogue deficit reflects missing data (solvable) or an inherent tension between audio-understanding-focused adapters and dialogue requirements (harder). The acoustic adapter's 64-query Q-former design captures global paralinguistic properties useful for emotion recognition, but dialogue requires turn-taking awareness, speaker state tracking, and pragmatic reasoning that may need different architectural support.

Factor-Based Parallel Decoding Latency Gains Are Not Quantified on the Benchmarks That Matter Most

The assumption or constraint. The paper introduces factor-based parallel decoding (from fast-dLLMs, Wu et al., 2025) as a practical inference acceleration mechanism and reports two pieces of evidence for its effectiveness: (1) ASR RTF improvements in Table 4, and (2) benchmark accuracy with factor-based decoding in Tables 1–3 ("w/ FPD" rows). The assumption is that the ASR latency gains generalize to audio understanding benchmarks, making the accuracy–latency tradeoff a "practical knob." But this assumption is untested: the paper reports no latency measurements for MMSU, MMAU, or MMAR with or without factor-based decoding.

The consequence. A practitioner cannot make an informed decision about deploying DIFFA-2 with factor-based decoding for audio understanding tasks. They know the accuracy cost: −0.35 MMSU, −1.30 MMAU Test-mini, −0.60 MMAR. But they do not know the latency benefit. Is decoding 1.2× faster? 5× faster? The answer determines whether the tradeoff is worth accepting. For ASR, factor-based decoding provides an 8.3× RTF speedup (0.6792 → 0.0820) — a dramatic improvement that would clearly justify the 0.33 WER increase for many applications. But audio understanding benchmarks use much shorter responses than ASR (answer length 16 vs. 128, block length 16 vs. 128, 16 denoising steps vs. 128 — Table B.4). With fewer steps, the relative speedup from adaptive token acceptance is likely much smaller than for ASR — the fixed overhead of encoding the audio prefix and running initial backbone forward passes dominates when denoising steps are few. The ASR latency gains may not transfer to understanding tasks, and the paper provides no evidence either way.

Furthermore, the paper does not report wall-clock latency for standard DIFFA-2 inference on any benchmark. Even without factor-based decoding, how long does DIFFA-2 take to answer a question compared to Qwen2.5-Omni or Kimi-Audio? The inference configurations (Table B.4) specify 16 denoising steps for MMSU/MMAU/MMAR, but 16 steps of a forward pass through an 8B model with audio prefix may still be slower or faster than 16 autoregressive steps through a 7B model — the number of operations per step differs (bidirectional attention over full response vs. causal attention over prefix + generated tokens). Without latency measurements, the paper's claim that dLLMs offer "practical inference acceleration" is supported only for ASR, not for the audio understanding tasks that are the paper's main contribution.

What evidence exists. Table 4 provides ASR RTF measurements with and without factor-based decoding. Tables 1–3 provide accuracy with factor-based decoding. No table provides RTF or wall-clock latency for any audio understanding benchmark. The inference hyperparameters in Table B.4 include denoising steps for each benchmark, which is a proxy for computational cost (more steps = more forward passes), but does not translate directly to latency without measurements on specific hardware. The paper states only that "All inference experiments are conducted on a single NVIDIA A100 GPU" (Appendix B.4), which is insufficient for latency comparisons across models.

Mitigation status. The paper acknowledges this implicitly by treating factor-based decoding as a "systems-level design choice" and suggesting that "adapting more advanced training-free acceleration methods from text dLLMs to audio is a promising but orthogonal direction for future work." However, it does not explicitly state that audio understanding latency is unmeasured, nor does it qualify its claims about "practical inference" with this caveat. A latency table for MMSU/MMAU/MMAR — with and without factor-based decoding, and ideally compared to AR baselines — would substantially strengthen the practical deployment case.

The Paper Provides No Evidence That the Dual-Adapter Decomposition Is Causal, Not Merely Correlated with Higher Capacity

The assumption or constraint. The paper's architectural contribution — the dual-adapter design with separate semantic and acoustic pathways — is presented as a key innovation enabling DIFFA-2's strong performance, particularly on paralinguistic tasks. The assumption is that decomposing audio information into temporally-resolved content (semantic adapter) and holistic paralinguistic summary (acoustic adapter) causes the observed improvements. However, the paper never tests this causal claim against the simpler alternative: that the gains come from having more adapter parameters (84.3M total for both adapters) regardless of architecture.

The consequence. A practitioner building their own LALM faces a design decision: invest in a dual-adapter architecture (which adds engineering complexity, requires training two separate modules, and increases the prefix length the backbone must process), or simply increase the capacity of a single adapter. The paper provides no guidance on this choice. If a single Q-former with 128 queries (matching the parameter count) would achieve equivalent performance, the dual-adapter design is unnecessary complexity. If the decomposition is essential, practitioners need to know this to replicate the results.

The evidence the paper does provide is confounded. DIFFA-2 substantially outperforms DIFFA (which used a single projection-based adapter), but DIFFA also used a different (smaller) Whisper encoder, different training data, no backbone fine-tuning, and no preference optimization. The gains cannot be cleanly attributed to the dual-adapter design versus any of these other changes. The comparison to LLaMA-Audio in Table 5 controls for training data and curriculum, but both models use the same dual adapters, so this comparison establishes that the diffusion backbone benefits from the dual adapters, not that the dual adapters are better than a single adapter. No ablation replaces the dual adapters with a single adapter of comparable capacity and retrains.

What evidence exists. The paralinguistic perception results on MMSU (Table 1) are suggestive: DIFFA-2 achieves 41.92, the highest among all models including larger ones, and 8.72 points above Qwen2.5-Omni (which uses a single adapter). This is consistent with the acoustic Q-former providing paralinguistic information that single-adapter designs miss. However, without an ablation, this remains correlation. The Stage 2 → Stage 3 MMSU Perception jump (38.54 → 44.16, +5.62), driven heavily by paralinguistic perception (31.32 → 40.63, +9.31), shows that the backbone must be fine-tuned to utilize acoustic adapter outputs — this is consistent with the acoustic adapter carrying useful information, but again doesn't isolate the dual-pathway design from total capacity.

Mitigation status. The paper does not acknowledge this as a limitation. The dual-adapter design is presented as an architectural contribution without a capacity-matched single-adapter ablation. Future work would need to train DIFFA-2 variants with: (1) a single larger Q-former (e.g., 128 queries, matching the combined parameter count) that cross-attends to both intermediate and final encoder states; (2) a single deeper convolution + projection network with equivalent parameters; and (3) the current dual-adapter design. Only such a comparison could establish whether the decomposition is causal or whether the paralinguistic gains reflect increased capacity and the Q-former cross-attention mechanism rather than the specific separation into two pathways.

The assumption or constraint. DIFFA-2's training data consists of single-modality audio datasets (speech corpora, sound event collections, music recordings) and two-modality audio QA pairs. The paper acknowledges that "the lack of mixed-modality supervision in the training data" limits performance on complex three-way mixtures (Section 5.1, MMAR discussion). The implicit assumption is that single-modality and two-modality training suffices to generalize to three-modality compositions — an assumption the MMAR results directly contradict.

The consequence. On MMAR's "All" category (simultaneous sound, music, and speech), DIFFA-2 scores 37.50%, compared to Qwen2.5-Omni's 54.17% and Qwen3-Omni's 70.83% (Table 3). This is a 16.67-point gap to the closest AR competitor of comparable size. The failure mode is not merely incremental degradation — the model essentially cannot handle three simultaneous audio sources at a level that approaches usefulness. This is a capability boundary: no amount of further fine-tuning on single-modality data will teach the model to disentangle and reason about three overlapping audio streams. The architecture may be capable of it (bidirectional context could, in principle, help separate sources), but the training distribution does not provide the necessary supervision.

This limitation matters for practical deployment because many real-world audio scenarios involve overlapping sources: a conversation in a cafe (speech + environmental sounds + background music), a video soundtrack (dialogue + sound effects + score), or an emergency call (speech + alarms + background noise). DIFFA-2's 37.50% accuracy on three-way mixtures means it would be unreliable in these common situations. The benchmark results that place DIFFA-2 at the top of the open-model leaderboard (MMSU, MMAU) mask this specific weakness because those benchmarks do not heavily emphasize mixed-source audio — MMAR was specifically designed to test this capability, and it reveals a gap that the other benchmarks miss.

What evidence exists. Table 3 breaks down MMAR performance by mixture type. DIFFA-2 performs well on two-way mixtures (Sound–Speech: 58.26, exceeding Qwen2.5-Omni's 55.96; Sound–Music: 45.45, tying Qwen2.5-Omni; Music–Speech: 54.88, competitive). But the three-way All score of 37.50% is dramatically lower than Qwen2.5-Omni's 54.17% — a disproportionate drop that is not explained by the model's single-modality or two-way performance. This pattern suggests a fundamental generalization failure, not just an across-the-board weakness. The paper explicitly attributes this to training data ("likely reflecting the lack of mixed-modality supervision") but does not explore whether this is solvable with more data or reflects a deeper architectural limitation of how the adapters encode audio — if the semantic and acoustic adapters produce representations that conflate information from different sources when they overlap, no amount of training data would fix the problem without architectural changes.

Mitigation status. The paper acknowledges this limitation in the MMAR analysis and the Limitations section but does not propose a solution or analyze the failure mode. The straightforward fix — including three-way mixture data in training — is mentioned implicitly but not tested. Whether the dual-adapter architecture can handle this with appropriate training data, or whether the adapter design itself creates a bottleneck for source separation (since the acoustic adapter produces 64 fixed vectors regardless of how many sources are present), is an open question the paper does not address.

The LLaDA-8B Backbone's Pre-Training Compute Is Unaccounted For, Confounding the "Data Efficiency" Claim

The assumption or constraint. The paper's central claim is that dLLMs are more data-efficient than AR models for audio understanding — they extract more learning signal per unique training example. The evidence comes from Table 5, where DIFFA-2 (S2) substantially outperforms LLaMA-Audio (S2) under matched fine-tuning data and curriculum. The implicit assumption is that the two backbones start from comparable positions before audio fine-tuning: both are 8B-parameter pre-trained text models (LLaDA-8B-Instruct vs. LLaMA-3.1-8B), so any performance differences during audio training reflect the generative paradigm's interaction with the audio modality, not pre-training quality.

The consequence. This assumption is almost certainly false in ways that matter for the conclusion. LLaDA-8B and LLaMA-3.1-8B were pre-trained by different teams with different data mixtures, different tokenizers, different total FLOPs, and different training objectives. The paper provides no information about the pre-training compute budgets, data volumes, or text-domain performance of these two backbones. If LLaDA-8B was pre-trained with 2× more FLOPs than LLaMA-3.1-8B, its advantage during audio fine-tuning might reflect better pre-trained representations (more factual knowledge, better reasoning patterns, more robust language understanding) rather than any property of the diffusion objective. The "data learner" narrative — that dLLMs' corruption–reconstruction training inherently extracts more from limited multimodal data — is confounded with possible differences in pre-training scale and quality.

This is not a hypothetical concern. The paper's thesis (citing Ni et al., 2025) is that dLLMs use "super-dense compute" — they train for more epochs with masking-based augmentation. If LLaDA-8B used more training compute than LLaMA-3.1-8B to reach a given text performance level, then the controlled comparison in Table 5 is actually comparing a better-pre-trained model against a worse-pre-trained one, and the performance gap during audio fine-tuning may simply reflect this pre-training gap rather than any audio-specific diffusion advantage. A practitioner choosing between backbones needs to know: for equivalent pre-training compute, does the diffusion backbone still outperform AR on audio tasks? The paper provides no evidence.

What evidence exists. The paper reports text-domain benchmark results for LLaDA-8B-Instruct only in passing (it is described as "instruction-tuned on text" without scores), and provides no benchmark scores for LLaMA-3.1-8B. There is no table comparing the two backbones' text capabilities before audio fine-tuning. The only controlled evidence in Table 5 compares them after identical fine-tuning, which conflates pre-training quality with fine-tuning efficiency. The paper does not even report the pre-training data or compute for either backbone, making it impossible for a reader to assess whether the comparison is fair.

Mitigation status. The paper does not acknowledge this confound. It treats "LLaDA-8B-Instruct" and "LLaMA-3.1-8B" as interchangeable 8B base models — an assumption that is standard in the LALM literature (where backbones are often treated as interchangeable "LLM components") but which undermines the specific claim about diffusion architecture being the causal factor. To isolate the diffusion advantage, future work would need to: (1) report text-domain performance of both backbones to establish a baseline; (2) ideally, pre-train a diffusion and AR model from scratch with identical compute budgets, then fine-tune both on identical audio data; or (3) at minimum, control for text-domain capability by matching backbones on text benchmarks before audio fine-tuning. The current evidence is strongly suggestive but not conclusive about whether dLLMs' audio advantage reflects the generative paradigm or differences in the specific backbone implementations used.

7. Implications and Future Directions

How This Work Changes the Landscape

DIFFA-2 shifts the conversation around audio language model backbones from "AR is the only viable option" to "diffusion is a legitimate point on the tradeoff frontier." This is not a paradigm overthrow — AR models remain dominant and DIFFA-2 does not surpass the strongest AR systems (Qwen3-Omni leads by ~5 points on MMSU, Gemini and GPT-4o-Audio lead on several metrics). Rather, it is a reframing of the design space: before this work, a practitioner building an audio understanding system would default to an AR backbone without serious consideration of alternatives. After DIFFA-2, the decision becomes a genuine engineering tradeoff — one that depends on data availability, latency requirements, and the specific mix of audio tasks targeted.

The magnitude of this shift is moderate but real. The paper's most important finding is not any single benchmark score but the controlled comparison in Table 5, which shows that under matched data and curriculum, the diffusion backbone (DIFFA-2 S2) outperforms the AR backbone (LLaMA-Audio S2) by 12.72 points on MMSU Overall before any backbone fine-tuning. This is a large effect that cannot be attributed to data volume, adapter design, or training stages — all are held constant. The gap narrows to 4.10 points after LoRA fine-tuning (S3), remaining meaningful but not dominant. The implication is that dLLMs' corruption–reconstruction pre-training objective provides a head start for audio understanding — it extracts representations that transfer more effectively to audio-conditioned generation than AR next-token prediction does — which LoRA fine-tuning can partially but not fully close.

This finding should redirect attention in the LALM research community toward understanding why the pre-training objective matters for cross-modal transfer. Currently, the field treats the LLM backbone as a largely interchangeable component — swap in LLaMA, Qwen, or LLaDA, add an audio encoder and adapter, fine-tune, and expect similar results modulo backbone quality. DIFFA-2's evidence suggests this assumption is false: the pre-training paradigm (AR vs. diffusion) has a first-order effect on downstream audio performance, independent of backbone scale or quality. This should encourage more systematic comparisons of backbone architectures for multimodal tasks, rather than the current practice of using whatever text LLM is most convenient.

Reconciling prior contradictions. Before DIFFA-2, the evidence on dLLMs for audio was sparse and inconclusive. DIFFA (Zhou et al., 2025) had shown that replacing an AR backbone with a diffusion one could improve audio understanding, but the proof-of-concept scale — frozen backbone, speech-centric data, weak encoder — left open the possibility that the gains were an artifact of the specific (weak) AR baseline or the narrow training distribution. Skeptics could reasonably argue that diffusion's advantages would evaporate once the model was properly trained with diverse audio data and backbone fine-tuning, as AR LALMs are. DIFFA-2 resolves this ambiguity: the diffusion advantage persists and in some dimensions grows when the model receives the full modern LALM treatment (dual adapters, large-scale SFT, backbone fine-tuning, preference optimization). The gains are not an artifact of weak baselines.

At the same time, DIFFA-2 provides evidence against diffusion being universally superior. The Stage 1 ASR results (Table 4) show the AR backbone achieving lower WER (2.43 vs. 2.72 on clean speech), confirming that strictly left-to-right decoding retains an edge for monotonic transcription. The VoiceBench results (59.63 vs. 74.04 for Qwen2.5-Omni) show a large dialogue deficit that is not merely a data issue — Qwen2.5-Omni was trained with more dialogue data, but the fact that the gap is 14.41 points rather than a few points suggests fundamental differences in how the two paradigms handle multi-turn conversational context. The MMAR three-way mixture results (37.50 vs. 54.17 for Qwen2.5-Omni) show a hard generalization failure that may reflect how the adapters encode overlapping audio sources. Together, these results paint a nuanced picture: diffusion backbones excel at tasks requiring integration of heterogeneous acoustic cues (paralinguistic perception: 41.92, best among all models; MMSU Reasoning: 76.40, leading open models), while AR backbones retain advantages for temporally-structured sequential tasks (transcription, possibly certain dialogue patterns).

Research directions that become more attractive. The paper's findings should increase interest in several areas:

Hybrid AR–diffusion architectures. If AR excels at local sequential tasks and diffusion at holistic integration tasks, a model that dynamically switches between generation paradigms — AR for transcription-like outputs, diffusion for reasoning-heavy responses — could combine strengths. The difficulty-estimation framework from the reference example (compute-optimal test-time scaling) could serve as a blueprint: estimate whether a query requires precise local decoding (AR) or global reasoning (diffusion) and route accordingly.
Pre-training objective design for multimodal transfer. The paper provides some of the first evidence that a language model's pre-training objective (mask-and-reconstruct vs. next-token) has substantial effects on downstream multimodal performance. This should motivate systematic studies comparing different pre-training objectives — masked language modeling, denoising autoencoding, prefix language modeling, diffusion — specifically for their cross-modal transfer properties, not just text-domain performance.
Adapter design informed by generative paradigm. The finding that the dual-adapter architecture benefits the diffusion backbone more than the AR backbone (Table 5, Stage 2 MMSU: +12.72 diffusion advantage, but the advantage is smaller on perception than reasoning) suggests that adapter design and backbone architecture interact. Adapters optimized for AR models (which the field currently designs by default) may be suboptimal for diffusion backbones, and vice versa. Research on backbone-aware adapter design could yield gains beyond what either component can achieve in isolation.

Research directions that become less attractive. The paper's results suggest that further incremental improvements to AR-specific adapter architectures may have diminishing returns if the field is open to switching backbones. The dual-adapter design, which required careful engineering (two separate pathways, Q-former for acoustics, convolution for semantics), produced gains that are comparable to or smaller than the gain from simply switching the backbone from AR to diffusion in the Stage 2 comparison (Table 5: the AR→diffusion backbone switch improves MMSU by 12.72 points, while adding the acoustic adapter to the diffusion model improves paralinguistic perception from presumably near zero to 31.32 at Stage 2 — both are large effects, but the backbone switch is free in terms of architectural complexity). This suggests that backbone selection may be a higher-leverage decision than adapter architecture for teams building LALMs, and that extensive adapter engineering on AR backbones may be less efficient than simply trying a diffusion backbone with a simpler adapter.

Follow-Up Research This Work Enables

Data scaling curves matching compute, not just data volume. The paper establishes that with 14.8k hours of data, DIFFA-2 outperforms an AR equivalent trained on identical data (Table 5). But the "super data learner" hypothesis from Ni et al. (2025) predicts that this advantage should grow as data volume decreases — diffusion models should extract more learning signal per unique example, so their relative advantage should be largest in low-data regimes. A direct follow-up would train DIFFA-2 and LLaMA-Audio at multiple data volumes (e.g., 25%, 50%, 75%, 100% of the current 14.8k hours) and compare performance. If the diffusion advantage is +12.72 MMSU points with full data but +20+ points with 25% data, that would strongly confirm the data learner hypothesis and establish a practical threshold below which diffusion backbones are clearly preferable. If the advantage is constant (always ~12 points), the effect is additive (better pre-training) rather than multiplicative (better data efficiency). This experiment requires ~5 days of training per data point on 64 A100s — feasible for a well-resourced lab.

Single-adapter capacity-matched ablation to isolate the dual-pathway contribution. The paper's dual-adapter design is presented as an architectural innovation, but the evidence for the decomposition being causal (rather than just higher parameter count) is missing. A clean follow-up would train three variants: (1) the current dual-adapter DIFFA-2 (84.3M adapter parameters across two pathways); (2) a single Q-former with 128 query vectors (matching parameter count, attending to both intermediate and final encoder states); and (3) a single deeper convolution + projection network with equivalent parameters. Train all three through Stages 1–3 and compare on MMSU's paralinguistic perception and reasoning subtasks. If the dual-adapter design substantially outperforms the capacity-matched single-adapter variants on paralinguistic tasks specifically, the decomposition hypothesis is supported. If the single Q-former matches or exceeds the dual adapters, the gain is from capacity and cross-attention, not decomposition — simplifying future dLLM-based LALM designs.

Difficulty-stratified analysis to identify where diffusion and AR diverge. The reference example uses difficulty quintiles to show that test-time compute strategies have qualitatively different effects at different difficulty levels. A similar analysis for DIFFA-2 would bin MMSU and MMAU questions by difficulty (estimated via the base model's pass@1 rate or average PRM confidence, analogous to the reference paper's procedure) and compare DIFFA-2 vs. Qwen2.5-Omni performance per bin. The hypothesis: diffusion's advantage concentrates on hard questions that require integrating multiple acoustic cues (e.g., "identify the speaker's emotion and explain which prosodic features indicate it"), while AR's advantage concentrates on easy questions requiring precise local recognition (e.g., "transcribe the third word"). This would transform the paper's binary "diffusion is competitive" claim into a more actionable "diffusion is preferable for these specific question types" guideline, directly informing model selection for practitioners.

Pre-training compute-matched comparison to isolate the objective from the implementation. The current comparison (Table 5) uses LLaDA-8B-Instruct and LLaMA-3.1-8B — two models pre-trained by different teams with unknown compute budgets. A rigorous follow-up would pre-train a diffusion model and an AR model from scratch with identical compute budgets (same total FLOPs, same data, same model architecture except for the attention mask and training objective), then fine-tune both on identical audio data using the DIFFA-2 pipeline. This would isolate the effect of the pre-training objective (mask-and-reconstruct vs. next-token) from confounds like pre-training data quality, tokenizer differences, and total FLOPs. If the diffusion backbone still shows a substantial advantage, the case for dLLMs as data-efficient multimodal learners is much stronger. If the advantage shrinks or disappears, the current results may reflect LLaDA-8B being simply a better-pre-trained model than LLaMA-3.1-8B, not a fundamental property of diffusion. This experiment is expensive (pre-training two 8B models from scratch) but is the only way to definitively answer the "data learner" question.

Mixed-modality training data augmentation to address the MMAR three-way mixture gap. The paper identifies the 37.50% score on MMAR three-way mixtures (vs. 54.17% for Qwen2.5-Omni) as a clear capability boundary caused by absent mixed-modality supervision. A direct follow-up would construct synthetic three-way mixture training data: overlay speech, music, and sound event audio clips with individual annotations, create QA pairs about the composite audio (e.g., "What emotion does the speaker convey, and what instrument is playing in the background?"), and fine-tune DIFFA-2 on this data. The experiment would measure: (1) does the MMAR All score improve proportionally to the amount of mixture data added? (2) does the improvement come at the cost of single-modality performance (catastrophic interference)? (3) is there a data volume threshold beyond which DIFFA-2 matches Qwen2.5-Omni on three-way mixtures? This would establish whether the gap is a training data issue (solvable) or a deeper architectural limitation of how the dual adapters handle overlapping sources (requiring architectural changes).

VRPO ablation: standard DPO vs. VRPO at matched preference data scale. The paper uses VRPO with 3,000 preference pairs, claiming that variance reduction is necessary for stable preference optimization in diffusion models. But no ablation compares VRPO to standard DPO (with ELBO estimates, no shared masking, single or multiple samples). A simple follow-up would train Stage 4 variants using: (1) standard DPO with K=1 sample and independent masking; (2) standard DPO with K=4 samples and independent masking; (3) VRPO with K=4 and shared masking (the current approach); (4) a no-preference-optimization baseline (Stage 3 only). Compare on MMSU and MMAU, and also measure training stability (loss variance across steps). If VRPO substantially outperforms standard DPO at the same sample budget, the variance reduction claim is supported and VRPO becomes the recommended approach for future dLLM preference optimization. If standard DPO performs similarly, the added complexity of shared masking is unnecessary, simplifying future dLLM alignment pipelines.

Practical Applications and Downstream Use Cases

Low-resource audio understanding for academic and open-source teams. DIFFA-2's most immediate practical value is for teams that lack access to massive proprietary audio datasets. The model achieves competitive results — 60.45 MMSU, 67.00 MMAU, 50.80 MMAR — using only publicly available corpora totaling 14.8k hours. A university lab or startup could replicate this pipeline: collect the same open-source audio datasets (AudioCaps, Clotho, LibriSpeech, GigaSpeech, FMA, etc. — all listed in Table A.1), fine-tune LLaDA-8B-Instruct with the four-stage curriculum on 4–8 A100 GPUs over ~5 days, and obtain an audio understanding model competitive with commercial AR APIs costing per-query. The 1.1% trainable parameter count means the model can be distributed as a small LoRA adapter + adapter weights (~99M parameters, ~400MB) paired with publicly available Whisper and LLaDA checkpoints, dramatically lowering the barrier to deploying capable audio understanding in research and prototyping settings.

On-device audio understanding with tunable latency–accuracy tradeoffs. The factor-based parallel decoding results (Table 4: RTF 0.0820 on LibriSpeech, Tables 1–3: 0.35–1.30 point accuracy drops with factor-based decoding) suggest a deployment pattern where the same model checkpoint serves multiple use cases with different latency requirements. For real-time applications (voice-controlled interfaces, live captioning), aggressive parallel decoding (lower factor, higher parallelism) reduces latency while accepting a small accuracy penalty. For offline batch processing (audio archival indexing, podcast transcription for search), standard decoding maximizes accuracy. A single DIFFA-2 checkpoint can serve both scenarios by toggling the decoding factor, eliminating the need to deploy separate fast and accurate models. This is a genuine practical advantage over AR models, whose latency is fundamentally bounded by per-token sequential generation regardless of the deployment context — an AR model cannot choose to "go faster" at the cost of accuracy in the same way.

Audio understanding component in multimodal pipelines. DIFFA-2 is scoped to text-out audio understanding, making it suitable as the audio module in larger multimodal systems. For example, a video understanding pipeline could use DIFFA-2 to answer questions about the audio track (speaker identity, emotion, sound events, music genre) while a separate vision model handles the visual content, with a fusion module integrating both. DIFFA-2's strong paralinguistic perception (41.92 on MMSU, best among all models) and sound event understanding (70.83 on MMAU Test Sound, leading open models) make it particularly valuable for applications where audio carries information not visible in the video — speaker emotion, off-screen sounds, musical mood. The open-source release of both training and inference pipelines enables integration without depending on proprietary audio APIs.