Understanding and Harnessing Sparsity in Unified Multimodal Models

🎯 Pitch

In unified vision–language models, image generation quality collapses under even light static pruning, revealing critical sensitivity missed in prior efficiency studies. Yet the very same models can match full performance while activating only half their generation parameters—if sparsity is instead routed dynamically through a Mixture-of-Experts design.

1. Executive Summary

This work systematically analyzes the slimness and sparsity of unified multimodal models—architectures that integrate heterogeneous components for both understanding and generation within a single framework—using training-free pruning as a probing methodology on BAGEL, Ming-Omni, and Qwen-Image. The study reveals that the understanding component exhibits high compressibility in both understanding and generation tasks (with neuron partitioning preserving generation quality at 50% width reduction), while generation components are dramatically sensitive to static compression, where even moderate pruning causes catastrophic quality degradation due to dynamic activation patterns that vary across samples and timesteps. To address the generation component's compression sensitivity, the paper introduces a Mixture-of-Experts (MoE) Adaptation that partitions MLP neurons into shared and routed experts with sparse activation, achieving performance comparable to the full model while activating only about half of its parameters, as demonstrated by the adapted BAGEL model matching the baseline on GenEval (0.88 vs. 0.86 overall score with only ~5B activated generation parameters), establishing that generation quality can be preserved under sparsity only when the model can dynamically route activations rather than relying on static parameter removal.

2. Context and Motivation

The Core Problem: Unified Models Are Architecturally Efficient but Computationally Wasteful

The fundamental tension this paper addresses is one that emerges naturally from the recent trajectory of multimodal AI research: unified multimodal models achieve impressive versatility by integrating understanding and generation into a single architecture, but this integration introduces substantial inference inefficiencies that remain unexplored and uncharacterized.

To appreciate why this matters, we need to understand what "unified" means concretely. Traditional multimodal research bifurcated into two largely separate tracks. On one side, multimodal large language models (MLLMs) like LLaVA (Liu et al., 2023) extended language model backbones to handle visual inputs, producing text outputs for tasks like visual question answering, captioning, and reasoning. On the other side, generative models like Diffusion Transformers (DiT; Peebles & Xie, 2023) were purpose-built for image or audio synthesis, operating through iterative denoising processes conditioned on text prompts. These two families of models shared little architectural common ground — they used different backbones, different training objectives, and different inference procedures.

The shift toward unified models — represented by BAGEL (Deng et al., 2025), Janus (Wu et al., 2024), Ming-Omni (AI et al., 2025), Qwen-Image (Wu et al., 2025), and others — represents a genuine architectural convergence. These models integrate heterogeneous components within a single framework: a shared vision encoder that processes visual inputs, a language backbone (often derived from Qwen-Instruct or similar) that handles both textual understanding and serves as a conditional feature extractor for generation, and modality-specific decoders (diffusion transformers or autoregressive generators) that produce non-text outputs. The unified architecture can seamlessly switch between answering questions about images and generating new images from text prompts.

However, this unification has a critical, underexamined downside: many tasks or input samples do not require the full knowledge or capacity of the unified model, but rather rely on a much slimmer or more sparsely activated sub-architecture. The paper identifies three complementary patterns that motivate this inefficiency concern (Section 1):

1. Component-wise redundancy. The understanding and generation components follow distinct computation patterns and serve fundamentally different functional roles. The understanding component processes multimodal inputs and produces language representations autoregressively — token by token, with each step conditioning on all previous tokens. The generation component, in contrast, operates through iterative refinement (in diffusion-based generators) or through a different decoding pathway. These components are not equally utilized across tasks: when the model is generating an image, the understanding component's full language decoding capability sits largely idle; when the model is answering a question, the generation component's denoising machinery is entirely unused. This structural heterogeneity suggests that different components may exhibit different levels of redundancy — some may be deeply overparameterized for their actual task demands.

2. Task-specific activation. Different tasks activate different parameter subsets. The paper's neuron partition analysis (Figure 2) provides direct evidence for this: when the authors identify the top 50% of most-important neurons for understanding tasks and separately for generation tasks, the overlap between these two sets is surprisingly low. This means that neurons critical for visual question answering are often distinct from neurons critical for generating coherent images. A model serving both tasks is therefore carrying parameters that are dormant for any single task — a form of task-conditional sparsity that static architectures cannot exploit.

3. Input variability. Even within the same task, different input queries activate different portions of the model. A prompt asking about the color of an object activates different neural pathways than one asking about spatial relationships. A generation request for "a realistic broccoli on a plain surface" triggers different feature combinations than "dolphins swimming through abandoned subway cars." This sample-level activation diversity means that the set of parameters genuinely needed for any single inference is a small, query-dependent subset of the full model.

These three patterns collectively point to a central inefficiency: unified models are carrying, on every forward pass, parameters that are irrelevant to the current task and the current input. This is the gap the paper sets out to characterize and address.

Why This Problem Is Important

The importance of this efficiency problem is both practical and conceptual.

Practical stakes. Unified multimodal models are large and computationally expensive. The BAGEL model, for instance, carries 7.62B parameters in its understanding component and another 7.62B in its generation component — a total of over 15B parameters that must be loaded, activated, and computed through for every inference, regardless of whether the task is understanding-only, generation-only, or both. For deployment scenarios — edge devices, real-time applications, high-throughput serving — this uniform full-model activation imposes a cost that scales with the model's maximum capability rather than the actual demands of each query. If a significant fraction of parameters could be safely deactivated per-task or per-sample without quality loss, the inference cost savings would be substantial and directly translate to reduced latency, lower energy consumption, and broader deployment feasibility.

Theoretical significance. The paper's investigation taps into a deeper question about how multimodal knowledge is represented and accessed in unified architectures. The finding that understanding and generation tasks activate largely non-overlapping neuron subsets (Figure 2) suggests that unified models are not achieving deep representational sharing between modalities — rather, they appear to be allocating distinct parameter subspaces to different tasks, essentially co-locating task-specialized sub-networks within a shared architecture. If true, this has implications for how we think about model capacity: unified architectures may achieve their versatility not through genuine cross-modal generalization but through capacity partitioning, which is inherently less parameter-efficient than true multi-task representation learning. The paper does not fully resolve this question, but the activation pattern analysis provides suggestive evidence that motivates deeper investigation.

Methodological gap. Prior compression research has focused almost exclusively on unimodal models (LLMs, vision transformers) or on vision-language understanding models where the output is always text, so the sensitivity of generation-specific components to compression was unknown. The paper establishes that the compressibility profiles of understanding and generation components are starkly different, which means that compression strategies developed for unimodal LLMs cannot be naively transferred to unified models. Understanding this differential sensitivity is prerequisite to designing any efficient inference system for unified architectures.

Where Existing Approaches Fall Short

The paper positions itself against two relevant but insufficient strands of prior work: model compression research and unified multimodal model design.

Model compression has been studied almost exclusively on unimodal or understanding-only models. The literature on network pruning (Cheng et al., 2024; Liu et al., 2019) has developed sophisticated techniques for identifying and removing redundant structures — depth pruning (Gromov et al., 2025; Men et al., 2024), width reduction (Ma et al., 2023; Xia et al., 2023), and unstructured weight pruning (Sun et al., 2024). These methods have been validated extensively on unimodal LLMs (Mistral, LLaMA) and, more recently, on vision-language understanding models (Lin et al., 2024; Sung et al., 2024). However, the authors explicitly note:

"While the uni-modal compression techniques can be transferred to Vision-Language models that take multi-modal inputs and output the language responses via language models, it is unclear whether such methods still work in unified models."

The critical distinction is that unified models contain generation components with fundamentally different computational characteristics than the autoregressive text decoders studied in prior work. Diffusion-based generators, for instance, operate through iterative denoising steps where the set of active parameters may change across timesteps — a dynamic that static pruning methods cannot accommodate. Prior compression research provides no guidance on how generation-specific modules respond to parameter removal.

Unified multimodal model research has focused on capability, not efficiency. The papers introducing BAGEL, Janus, Ming-Omni, and Qwen-Image are primarily concerned with demonstrating that understanding and generation can coexist within a single architecture without catastrophic interference, and with achieving competitive performance on both types of tasks. Efficiency considerations — beyond high-level architecture design choices like Mixture-of-Transformers (Liang et al., 2025) — have not been systematically studied. The authors observe that "the increased architectural complexity poses new challenges for efficiency, which remains underexplored." This is the gap they step into: the first systematic characterization of where and how redundancy manifests across the heterogeneous components of a unified model.

Specific limitations of prior compression approaches when applied to unified models:

Depth pruning assumptions break for generation. Layer dropping relies on the observation that deeper layers in autoregressive LLMs often contribute marginal transformations (quantified via input-output cosine similarity). This metric was developed and validated on text generation, where each layer processes a sequence of token representations. In generation components, the computational dynamics are different — diffusion models iterate over noise levels, and autoregressive image generators process spatial token sequences with potentially different redundancy patterns. The paper's results (Section 5.3, Appendix A) confirm that depth pruning catastrophically degrades generation quality, even at modest reduction ratios.
Static pruning cannot accommodate dynamic activation. Standard pruning selects a fixed subset of parameters to retain and discards the rest permanently. This works when the importance ranking of parameters is stable across inputs — an assumption that holds reasonably well for many unimodal LLM layers. However, the paper's analysis (Figure 3, Section 4.2) reveals that generation components exhibit sample-dependent activation patterns: only a small subset of neurons remain consistently active (ranked in the top 50% by activation scores) across all inputs and timesteps, while most neurons are active for some inputs and inactive for others. A static pruning mask would inevitably delete neurons that are critical for some inputs but not others, causing selective degradation that a dynamic activation mechanism could avoid.
Calibration data sensitivity is amplified in multimodal settings. Pruning methods typically use a small calibration dataset to compute importance scores. In unimodal LLMs, the choice of calibration data matters but the effects are often modest because the model's parameter importance distribution is relatively stable across similar text distributions. In unified models, the paper demonstrates (Figures 5, 12, 13, Appendix E) that calibration data from understanding tasks versus generation tasks leads to substantially different retained parameter sets and downstream performance — using understanding calibration for a generation task causes visible degradation in image quality (Figure 5: distorted structures, mismatched semantics). This means that compression strategies must be task-aware, selecting calibration data aligned with the target use case, which adds a layer of complexity absent from unimodal pruning.
No prior work studies generation component compressibility. The paper's finding that generation components are "highly sensitive to compression" with "performance deteriorating sharply even under moderate compression ratios" (Section 5.3) is itself a novel empirical result. Prior compression literature simply has no data on how image decoders, diffusion transformers, or autoregressive image generators respond to structured pruning, because those components didn't exist in the models being studied.

How This Paper Positions Itself

The paper frames its contribution not as proposing a single new compression method, but rather as systematically characterizing the efficiency landscape of unified models and then designing a targeted solution for the hardest part of that landscape. This is both an analytical contribution and a methodological one.

Analytical contribution: using pruning as a probing tool. The authors explicitly adopt the methodology of training-free pruning as an analytical instrument:

"We first adopt training-free pruning as a probing methodology, as it enables us to infer structural importance without retraining by removing structures and observing the resulting performance change."

This framing is important. The goal is not merely to compress models for deployment — it is to understand how these models allocate and utilize their capacity across heterogeneous components and tasks. By observing what breaks when different structures are removed, the authors map the functional anatomy of unified models: which components are robust to parameter removal (understanding), which are fragile (generation), and how activation patterns differ across tasks and samples. This is a diagnostic approach that generates insights about model organization independent of any particular compression technique.

Methodological contribution: MoE Adaptation as a response to dynamic sparsity. The analytical finding that generation components suffer under static compression but exhibit dynamic, input-dependent activation patterns directly motivates the MoE Adaptation approach. Rather than trying to make static pruning work better (which would require fighting against the inherent dynamics of generation), the paper embraces the dynamic nature of generation computation and designs a mechanism — Mixture-of-Experts with learned routing — that preserves the model's ability to activate different parameter subsets for different inputs while keeping the average number of activated parameters low.

The paper positions MoE Adaptation as a bridge between two observations: (1) static compression fails for generation components because it cannot accommodate activation diversity across samples and timesteps, and (2) the same component, when allowed to dynamically route activations through partitioned experts, can recover full-generation quality with only ~50% of parameters activated per forward pass. The approach is not claimed to be the only possible solution, but rather a natural consequence of the observed activation patterns.

Relationship to prior MoE work. The paper builds on the MoE design principles established in DeepSeek-MoE (Dai et al., 2024) — specifically, the separation of shared and routed experts, with shared experts capturing universally useful features and routed experts specializing for input-dependent computation. However, the paper's application context is distinct: prior MoE work focuses on scaling up model capacity while keeping inference cost manageable (DeepSeek-V3, Mixtral), whereas this paper applies MoE as a retrofit to an already-trained dense model to recover quality lost during compression. The expert partition is not trained from scratch but derived from importance scores computed on the pretrained dense weights, and the adaptation process includes a crucial "expert-frozen tuning" warmup phase that mitigates catastrophic forgetting while the router learns to select among frozen expert partitions.

The positioning is therefore: learn from the MoE literature's architecture design principles (shared + routed experts, top-k gating), but apply them in a novel regime — post-hoc conversion of dense generation components to sparse activation — that is motivated by empirical findings about differential compressibility across components.

What this paper is and is not. The paper is a systematic efficiency analysis of unified multimodal models that reveals component-specific compressibility profiles and proposes a targeted solution for the most compression-sensitive component. It is not: a new unified architecture, a general compression algorithm claiming to work across all model types, or a deployment-ready system with end-to-end latency measurements. The authors are transparent about this scope: the work uncovers "substantial optimization space for improving parameter efficiency" but the difficulty estimation, calibration data sensitivity, and limited model coverage all indicate this is an initial characterization rather than a final solution.

3. Technical Approach

3.1 Reader Orientation

This paper develops a two-phase methodology for understanding and exploiting parameter efficiency in unified multimodal models: first, training-free pruning techniques serve as diagnostic probes to map where redundancy exists across heterogeneous model components, and second, a training-aware Mixture-of-Experts (MoE) adaptation mechanism enables sparse activation in compression-sensitive generation components to recover quality lost under static pruning. The approach solves the problem that unified multimodal models — which integrate understanding and generation into a single architecture — carry parameters irrelevant to the current task or input on every forward pass, but existing compression methods developed for unimodal LLMs fail because (a) understanding and generation components have fundamentally different compressibility profiles, and (b) generation components exhibit dynamic, sample-dependent activation patterns that static pruning cannot accommodate.

3.2 Big-Picture Architecture

The system has five major components:

A pretrained unified multimodal model (BAGEL, Ming-Omni, or Qwen-Image) — the target to be analyzed and compressed. It contains an understanding component (a vision-language model backbone that processes multimodal inputs and produces text autoregressively) and a generation component (a diffusion transformer or autoregressive decoder that synthesizes images conditioned on features from the understanding component).
Training-free compression probes — analytical tools that remove structures (layers, neurons, attention heads) without any retraining and measure the resulting performance change. These include depth pruning (dropping entire transformer layers based on input-output similarity) and neuron partitioning (identifying and removing low-importance MLP neurons based on activation-weighted weight norms).
A calibration dataset — a small set of task-specific examples used solely to compute importance scores for pruning decisions. No ground-truth annotations are required, and no training occurs.
A Mixture-of-Experts (MoE) adaptation pipeline — a training-aware mechanism applied specifically to generation components. It partitions MLP neurons into shared experts (always active, capturing universal features) and routed experts (selectively activated by a learned router), then trains in two stages: expert-frozen tuning (warmup with frozen expert weights) followed by full end-to-end MoE training.
Evaluation benchmarks — GenEval for generation quality assessment and MME/MMBench/MMMU/MMVP for understanding performance measurement.

Information flows as follows: a unified model enters the analysis phase → training-free probes measure component-specific compressibility and activation patterns → findings reveal that understanding components are compressible but generation components are not → MoE adaptation is applied retroactively to generation components → the adapted model achieves sparse activation (~50% of parameters activated per forward pass) while matching dense model quality.

3.3 Roadmap for the Deep Dive

First, the formal definition of unified multimodal models (Equations 1–2), which establishes the mathematical framework distinguishing understanding from generation and defines the parameter groups $\theta_{\text{und}}$ and $\theta_{\text{gen}}$ that will be the targets of compression.
Second, the training-free compression strategies — depth pruning via layer dropping and width reduction via neuron partitioning — including the importance metrics, the calibration procedure, and the mechanistic justification for why these specific probes were chosen.
Third, the task-specific activation analysis that emerges from neuron partitioning, explaining how calibration data alignment determines which parameters are retained and providing empirical evidence that unified models allocate distinct neuron subsets to different tasks.
Fourth, the Mixture-of-Experts adaptation mechanism — expert partition based on cumulative importance scores, the router design with zero-initialized gating, and the two-stage training protocol (expert-frozen tuning → full MoE adaptation).
Fifth, the design choices and their justifications, including why shared experts constitute one-sixteenth of total experts, why activation ratio is set to 50%, and why the first and last layers are excluded from MoE conversion.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical analysis paper whose core idea is that different components of unified multimodal models exhibit fundamentally different compressibility profiles, and that generation components require dynamic sparse activation — not static pruning — to maintain quality under parameter reduction.

Formal Definition of Unified Multimodal Models

The paper first defines the mathematical framework for unified models to establish clear notation for which parameters belong to which component and what computational pathways are involved in understanding versus generation tasks.

Understanding pathway. For a multimodal input $x$ and a corresponding textual output $y$ , the understanding component predicts tokens autoregressively:

$p(y_{\text{und}} \mid x; \theta_{\text{und}}) = \prod_{t=1}^{T} p(y_t \mid y_{<t}, x; \theta_{\text{und}})$

where $\theta_{\text{und}}$ denotes the parameters of the understanding component, $y_{\text{und}}$ is the sequence of output tokens $(y_1, y_2, \ldots, y_T)$ , and $T$ is the total number of generated tokens.

What it computes: For a given multimodal input $x$ (e.g., an image paired with a text question), the understanding component generates text tokens one at a time, with each token's probability conditioned on all previously generated tokens $y_{<t}$ and the original input $x$ . The product over $t$ from 1 to $T$ gives the joint probability of the full output sequence under the model. This is standard autoregressive language modeling applied to multimodal inputs.

Why this form: The autoregressive factorization reflects how these models actually operate at inference — they cannot produce the entire answer at once but must decode token-by-token, with each step depending on all prior steps. This has implications for compression: because errors in early tokens propagate to later tokens (an error accumulation phenomenon the paper observes in Section 5.2), the understanding component is sensitive to any compression that introduces small per-step deviations. A model that produces slightly degraded token probabilities at each step may compound those degradations across $T$ steps and collapse entirely, which explains why depth pruning catastrophically fails on understanding tasks even when it works for generation tasks (where the understanding component's output is a fixed feature vector, not a decoded token sequence).

Generation pathway. For generation tasks, the model uses a two-stage process. The understanding component first processes an instructional input $x_{\text{inst}}$ (e.g., a text prompt and reference images) to produce conditional features $f_{\text{und}}(x_{\text{inst}}; \theta_{\text{und}})$ . The generative component, parameterized by $\theta_{\text{gen}}$ , then synthesizes the output $y_{\text{gen}}$ conditioned on both these features and an additional generative input $z$ :

$y_{\text{gen}} \sim p(y_{\text{gen}} \mid f_{\text{und}}(x_{\text{inst}}; \theta_{\text{und}}), z; \theta_{\text{gen}})$

where $z$ represents task-specific generative inputs — for diffusion models, this is random noise that gets iteratively denoised; for autoregressive image generators, this could be a start-of-sequence token.

What it computes: The understanding component acts as a conditional feature extractor for the generation component — it encodes the user's instruction into a representation $f_{\text{und}}$ that captures the semantic content of what should be generated. The generation component then takes this representation and transforms it, through its own parameters $\theta_{\text{gen}}$ , into a non-text output (an image, in the studied models). Crucially, the generation component does not interact with the raw input $x_{\text{inst}}$ directly; it sees only the processed features from the understanding component. This means that compression applied to the understanding component can affect generation quality indirectly — by degrading the quality of $f_{\text{und}}$ — even though the generation parameters $\theta_{\text{gen}}$ are untouched.

Why this two-stage form: This factorization captures the heterogeneous nature of unified models. The understanding component is shared across both tasks (it processes inputs whether the output is text or an image), while the generation component is invoked only when non-text outputs are needed. This architectural asymmetry is precisely what the paper exploits: because the understanding component feeds the generation component, understanding compression affects generation quality, but the reverse is not true (generation compression does not affect understanding). The two parameter groups $\theta_{\text{und}}$ and $\theta_{\text{gen}}$ are therefore the natural units of analysis for the paper's compression experiments.

Important note on parameter counts. Table 1 provides concrete numbers. BAGEL has $\theta_{\text{und}}$ = 7.62B parameters and $\theta_{\text{gen}}$ = 7.62B parameters (the generation component reuses the Qwen-Instruct backbone, so both components have identical parameter counts despite serving distinct functions — this is a Mixture-of-Transformers design where understanding and generation modules interact through cross-attention at every layer). Qwen-Image has $\theta_{\text{und}}$ = 7.62B and $\theta_{\text{gen}}$ = 20.42B (the generation component uses a much larger MMDiT-based generator). Ming-Omni has $\theta_{\text{und}}$ = 17.12B (an MoE-based backbone) and $\theta_{\text{gen}}$ = 2.51B (a relatively small multi-scale DiT block). These differences in parameter allocation across models are important because they predict differential sensitivity to compression — models with smaller generation components (like Ming-Omni) depend more heavily on high-quality features from the understanding component, making understanding compression more consequential for generation quality.

Training-Free Compression Strategies

The paper uses two complementary training-free pruning techniques, both serving as analytical probes more than deployment tools. The core idea is: remove structures without any retraining, then observe the performance change. The pattern of what breaks and what survives reveals where redundancy lives.

Depth Pruning via Layer Dropping

This method removes entire transformer layers from the understanding component and measures how performance degrades. The paper builds on prior work showing that deeper layers in LLMs often contribute marginal transformations (Gromov et al., 2025; Men et al., 2024).

The layer redundancy metric. For each layer $l$ , the paper computes:

$S_l = \text{CosineSim}(x_l, y_l)$

where $x_l$ is the input to layer $l$ and $y_l$ is its output. Cosine similarity measures the angle between these two vectors in the model's hidden space; values near 1 indicate the layer's output is nearly identical to its input (the layer is doing very little transformation), while values near 0 indicate substantial transformation.

What it computes: For every layer in the model, this metric produces a single scalar between -1 and 1 (in practice, between approximately 0.9 and 1.0 for transformer hidden states, which are high-dimensional and tend to be positively correlated). A layer with $S_l \approx 1.0$ can be interpreted as mostly computing the identity function — its removal should cause minimal disruption because the signal passing through it is nearly unchanged. A layer with lower $S_l$ is performing more substantial computation and its removal should hurt more.

Why this form: Cosine similarity is scale-invariant — it measures directional alignment independent of magnitude. This matters because transformer hidden states can vary in norm across layers (due to residual connections and layer normalization), but what matters for information preservation is whether the representation points in the same direction before and after the layer. Euclidean distance or MSE would be sensitive to norm changes that residual connections can compensate for, making those metrics overly pessimistic about layer importance.

Why depth pruning fails for understanding tasks. The paper reports (Section 5.2) that removing 50% of MLP layers in BAGEL's understanding component causes MME perception scores to drop from 1684.8 to 304.5 and cognition from 696.7 to 127.1 — a near-total collapse. The explanation, articulated in Section 3.4.1, is error accumulation in autoregressive decoding: understanding tasks require generating long token sequences where each token conditions on all previous tokens. If layer removal introduces even small per-token deviations in the probability distribution, these errors compound across the sequence. A token that should have been "the" might become "a," which changes the context for the next token, which changes the next token further, and after 5–10 steps the model has completely diverged from coherent output (visible in Figure 11, Appendix C, where the depth-reduced model degenerates into repeating a single word). This is a fundamental limitation of depth pruning for autoregressive decoders — it cannot be fixed by removing different layers or using better importance metrics, because the problem is the fragility of the decoding process itself, not the layer selection criterion.

Why depth pruning works for generation tasks. The same 50% layer removal in the understanding component preserves generation quality for BAGEL and Qwen-Image (Figure 4, Section 5.2). The reason is that for generation tasks, the understanding component's output is not a decoded token sequence but rather a fixed-length feature vector $f_{\text{und}}(x_{\text{inst}}; \theta_{\text{und}})$ that is fed once to the generation component. There is no iterative conditioning — the generation component receives this vector and uses it as a conditioning signal throughout its own denoising or autoregressive process. Modest degradation in this feature vector (from removed layers) acts as a slight perturbation to the conditioning signal, which generation models (particularly diffusion models, which are trained to be robust to input variation) can partially compensate for. The generation component is not recursively conditioning on its own potentially degraded outputs, so there is no error accumulation.

The Ming-Omni exception. Depth pruning is less effective for Ming-Omni specifically because its generation component is relatively small (2.51B vs. 7.62B for BAGEL and 20.42B for Qwen-Image). A smaller generator has less capacity to compensate for degraded input features, so it depends more critically on high-quality features from the understanding component. This is an architectural interaction that the paper identifies but does not deeply explore — it suggests that the tolerance of generation quality to understanding compression is proportional to the relative capacity of the generation component.

Width Reduction via Neuron Partition

Unlike depth pruning, which removes entire layers, neuron partitioning selectively removes individual neurons within MLP layers — a finer-grained compression that can preserve critical functionality within each layer while reducing overall width. This is the paper's primary training-free probe and the analytical tool that reveals task-specific activation patterns.

The MLP structure. The paper focuses on "Gate-Up-Down" MLPs, the standard design in Qwen-family models. Given an input $x \in \mathbb{R}^{s \times d}$ (a sequence of $s$ tokens, each of dimension $d$ ), the MLP computes:

$h = \left(\text{SiLU}(x W_g^{\top})\right) \odot (x W_u^{\top})$

$y = h W_d^{\top}$

where $W_g, W_u \in \mathbb{R}^{d_m \times d}$ are the gate-projection and up-projection matrices that expand from dimension $d$ to $d_m$ (the MLP intermediate dimension), $\text{SiLU}$ is the Sigmoid Linear Unit activation function applied element-wise, $\odot$ is element-wise multiplication, $h \in \mathbb{R}^{s \times d_m}$ is the gated hidden activation (containing $d_m$ individual neuron activations), and $W_d \in \mathbb{R}^{d \times d_m}$ is the down-projection matrix that collapses back to dimension $d$ .

What this computes: The input $x$ is projected twice in parallel: once through $W_g$ followed by a non-linear gating function (SiLU, which is $x \cdot \sigma(x)$ where $\sigma$ is the logistic sigmoid) to produce gating coefficients, and once through $W_u$ to produce candidate activation values. These two projections are multiplied element-wise — the gate controls how much of each candidate value passes through. The result $h$ then passes through $W_d$ which mixes across the $d_m$ hidden neurons to produce the output. Crucially, each of the $d_m$ hidden neurons corresponds to one column of $W_d$ and one row of both $W_g$ and $W_u$ — removing a neuron means deleting the corresponding column from $W_d$ and the corresponding rows from $W_g$ and $W_u$ .

Why this gating structure: The SiLU-gated design (also called SwiGLU) is the dominant MLP formulation in modern LLMs because it outperforms standard ReLU or GELU activations. The separate gate and up-projection paths allow the model to learn complex non-linear interactions — the gate can selectively suppress or amplify specific feature dimensions before they are mixed by the down-projection. This makes individual neuron importance more meaningful: a neuron with consistently low gate activation is genuinely contributing little, while a neuron with high and variable gate activation is important for the model's representational capacity.

The neuron importance metric. To decide which neurons to remove, the paper derives an importance score that combines activation magnitude with weight magnitude, inspired by Wanda (Sun et al., 2024) but adapted from unstructured weight pruning to structured neuron removal.

The contribution of neuron $i$ to the MLP output is:

$\Delta y_i = h_i W_{d,i}^{\top}$

where $h_i$ is the activation value of neuron $i$ (a scalar for each token in the sequence) and $W_{d,i}$ is the $i$ -th column of the down-projection matrix $W_d$ — a vector of length $d$ representing how neuron $i$ 's activation is distributed across output dimensions.

What this equation states: The entire contribution of neuron $i$ to the layer's output is the product of a scalar activation $h_i$ (which varies per token and per input) and a fixed weight vector $W_{d,i}$ . If $h_i = 0$ , the neuron contributes nothing regardless of its weight vector. If $W_{d,i}$ is the zero vector, the neuron contributes nothing regardless of its activation. So importance depends on both.

If this neuron is pruned, the induced output error norm is approximated by:

$\|\Delta y\|_2 \approx \|h_i W_{d,i}^{\top}\|_2$

This is an approximation rather than an exact equality because it ignores potential interactions between neurons (the model might partially compensate for a removed neuron through other neurons' activations), but it provides a first-order estimate of per-neuron impact.

Why this approximation is reasonable: In a wide MLP with hundreds or thousands of hidden neurons, individual neurons are largely independent in their contribution to the output — the down-projection is a linear combination of neuron activations, and removing one term from this sum primarily affects the output through the magnitude of that term. Second-order effects (other neurons adjusting to compensate) are real but require retraining to manifest, and in the training-free setting, we have no mechanism to observe or induce compensation. The first-order approximation is the best we can do without gradient computation.

The aggregated importance score. To account for variability across inputs, the paper averages over a calibration dataset $D$ :

$s_i = \mathbb{E}_{x \sim D}\left[|h_i| \cdot \|W_{d,i}\|_2\right]$

where $|h_i|$ is the absolute activation value of neuron $i$ (averaged across all tokens in the sequence, then across all sequences in the calibration set), and $\|W_{d,i}\|_2$ is the $\ell_2$ norm (Euclidean length) of the down-projection weight vector associated with neuron $i$ .

What it computes: For each neuron, this metric produces a single non-negative scalar. The term $|h_i|$ captures how strongly the neuron fires on average — neurons that rarely activate get low scores regardless of their weights. The term $\|W_{d,i}\|_2$ captures how much influence the neuron has when it does fire — a neuron connected to many output dimensions with large weights gets a high score even if its average activation is modest. The product means that both low-activation, high-weight neurons and high-activation, low-weight neurons can be important — importance is the product, not the sum.

Why this form over alternatives: The paper explicitly contrasts with gradient-based metrics (such as those used in LLM-Pruner, Ma et al., 2023). Gradient methods approximate the change in loss from removing a parameter by computing $|w \cdot \nabla_w \mathcal{L}|$ — the product of the weight magnitude and the gradient of the loss with respect to that weight. While principled, this requires: (1) labeled data to compute the loss, (2) a backward pass to compute gradients, and (3) careful handling of the loss function (which task's loss?). The activation-weight product used here requires only a forward pass on unlabeled calibration data, making it simpler and faster to deploy. Table 7 (Appendix D) shows that neuron partition achieves competitive overall GenEval scores with LLM-Pruner (0.71 vs. 0.70 for Ming-Omni) despite being computationally cheaper. The paper's metric is also naturally interpretable: $|h_i| \cdot \|W_{d,i}\|_2$ has a direct geometric meaning (the expected magnitude of the neuron's output vector), while $|w \cdot \nabla_w \mathcal{L}|$ is harder to interpret outside the specific loss landscape.

The structured pruning operation. Unlike unstructured pruning, which zeros out individual weights in $W_d$ , $W_g$ , and $W_u$ while keeping the matrix dimensions unchanged, structured pruning at the neuron level physically removes column $i$ from $W_d$ (reducing its dimension from $d \times d_m$ to $d \times (d_m - 1)$ ) and row $i$ from both $W_g$ and $W_u$ (reducing each from $d_m \times d$ to $(d_m - 1) \times d$ ). This yields genuine computational savings because the reduced matrices require fewer FLOPs for every forward pass — no sparse matrix multiplication or masking overhead.

Concrete pruning procedure. Given a target sparsity ratio (e.g., 50%, meaning half the neurons should be removed):

Run the calibration dataset through the model, collecting for each MLP layer the activations $h$ for every input.
For each neuron $i$ in the layer, compute $s_i = \text{mean}_{x \in D}(|h_i|) \cdot \|W_{d,i}\|_2$ .
Sort all $d_m$ neurons in the layer by $s_i$ in ascending order.
Remove the bottom $\lfloor \text{sparsity} \times d_m \rfloor$ neurons — deleting their corresponding columns/rows from the weight matrices.
The model is now structurally narrower by the specified ratio, with no retraining needed.

How calibration data selection works. The importance scores depend on which inputs are in the calibration set because $|h_i|$ varies with the input distribution. The paper's key insight (Section 4.1, Figure 2) is that different tasks activate different neuron subsets, so calibration from understanding tasks versus generation tasks leads to different retained neurons. Figure 2 quantifies this: the authors identify the top 50% of neurons by importance for understanding tasks and separately for generation tasks, then measure the overlap. The "relatively low overlap ratio" means that many neurons crucial for understanding are dispensable for generation and vice versa. The practical implication is that the calibration dataset should match the target task — if you want to compress the understanding component for use in generation, calibrate with generation examples; if for understanding, calibrate with understanding examples. Figures 5 and 13 empirically demonstrate the consequences of mismatched calibration: using understanding calibration for generation tasks produces distorted, semantically inconsistent images.

How this differs from Wanda. Wanda (Sun et al., 2024) uses per-weight importance $|W_{ij}| \cdot \|x_j\|_2$ to zero out individual weights within a matrix, achieving unstructured sparsity. The paper extends this idea in three ways: (1) it aggregates importance across the neuron dimension rather than individual weights, enabling structured (hardware-friendly) pruning; (2) it uses the neuron's activation $|h_i|$ rather than the input norm $\|x\|_2$ , which is more directly relevant for measuring the neuron's contribution to the output; and (3) it pairs this with a task-aware calibration strategy that accounts for the multimodal, multi-task nature of unified models. The extension from weight-level to neuron-level is non-trivial: Wanda's metric works because individual weights within a matrix are largely independent, but neurons involve coordinated weight vectors ( $W_{d,i}$ , $W_{g,i}$ , $W_{u,i}$ ) that must be pruned together.

Task-Specific Activation Analysis

This is not a separate algorithm but rather an analytical finding that emerges from applying neuron partitioning with different calibration sets. It is described in Section 4.1 and visualized in Figures 2 and 3.

Figure 2: neuron overlap analysis. The procedure:

For each layer in the understanding component, compute importance scores $s_i$ using an understanding-task calibration set (MME samples).
Identify the top 50% of neurons by these scores — call this set $U$ (neurons important for understanding).
Repeat with a generation-task calibration set (GenEval samples), producing set $G$ (neurons important for generation).
Compute three disjoint subsets: $U \setminus G$ (understanding-specific), $G \setminus U$ (generation-specific), and $U \cap G$ (shared).

Figure 2 plots, for each of the 28 layers in the understanding component, the number of neurons in each of these three categories. The key observation is that the overlap region (shared neurons) is small relative to the task-specific regions — in most layers, understanding-specific and generation-specific neurons together dominate the shared neurons.

What this reveals. Unified models are not achieving deep representational sharing between understanding and generation — they are allocating largely distinct parameter subspaces to different tasks. This is consistent with the Mixture-of-Transformers architecture of BAGEL, where understanding and generation modules are architecturally separated and interact only through cross-attention. The language backbone is genuinely being used differently for the two tasks: when doing understanding, it activates one set of features; when providing conditioning signals for generation, it activates a largely different set. This has practical implications: a single static compression (pruning the same neurons regardless of task) will inevitably delete neurons important for one task if calibrated on the other.

Figure 3: dynamic activation in generation components. A separate analysis applied specifically to the generation component's MLP layers. The procedure:

For each layer in the generation component, run multiple generation prompts through the model.
For each prompt and each denoising timestep, record which neurons are in the top 50% by activation magnitude.
Classify each neuron as:
- Saturated (consistently active): the neuron is in the top 50% for ALL prompts and timesteps.
- Inactive: the neuron NEVER enters the top 50% for any prompt or timestep.
- Selectively active (neither category): the neuron is in the top 50% for some prompts or timesteps but not others.

Figure 3 plots, for each of the 28 layers in the generation component, the proportion of neurons that are saturated versus inactive. The result: only a small fraction of neurons are consistently active (the "saturated" category), and a non-trivial fraction are never among the top activations (the "inactive" category). The majority of neurons fall into the middle category — they are important for some inputs or timesteps but not others.

What this reveals. The generation component exhibits sample-dependent and timestep-dependent activation patterns. There is no single set of neurons that can be statically retained and guarantee quality across all inputs — any static pruning mask will delete neurons that are critical for some subset of prompts. This is the central motivation for MoE adaptation: instead of permanently removing neurons (static compression), the model needs a mechanism to selectively activate different neuron subsets for different inputs (dynamic sparsity). MoE architectures are explicitly designed for this — the router learns which experts (neuron groups) are relevant for the current input and activates only those.

MoE Adaptation for Generation Component Sparsity

This is the paper's primary methodological contribution: a training-aware procedure that converts the dense MLP layers in the generation component to a Mixture-of-Experts architecture, enabling sparse activation while preserving generation quality. The approach is described in Section 4.2.

Motivation. The training-free analysis revealed that generation components cannot be statically compressed without catastrophic quality loss (Figure 6, Section 5.3: even 50% width reduction produces "distorted structures and unrealistic textures"). However, the dynamic activation patterns in Figure 3 suggest that at any given moment, only a subset of neurons are truly needed. The challenge is to enable the model to choose which subset per input — a dynamic routing problem that MoE architectures solve naturally.

Stage 1: Expert Partition

The first stage converts a dense MLP layer into an MoE layer without any training — it is a purely analytic repartitioning of existing weights.

Shared vs. routed experts. The paper separates the $d_m$ neurons in each MLP into two categories: shared experts $E_s$ (always active, capturing features that are consistently beneficial across samples) and routed experts $E_r^{(1)}, \ldots, E_r^{(n)}$ (selectively activated by a router, capturing features that are sample-dependent). The design follows the DeepSeek-MoE architecture (Dai et al., 2024), which demonstrated that separating shared and routed experts improves both training stability and specialization.

Selection of shared experts. For each neuron $i$ in the layer, compute the cumulative importance score $s_i$ from Equation 7 using a calibration dataset of generation examples. Sort all neurons by $s_i$ in descending order. The neurons with scores above a threshold are designated as shared experts. The paper specifies (Section 5.1) that shared experts constitute one-sixteenth of the total number of experts — so if there are 16 total experts, 1 is shared and 15 are routed; if 32 total, 2 are shared and 30 are routed.

Why one-sixteenth: This ratio is adopted from DeepSeek-V3 (DeepSeek-AI et al., 2025) and DeepSeek-MoE, where it was found to provide a good balance between universal representation (shared experts capture features that all inputs need) and specialized computation (routed experts capture input-specific features). Too many shared experts wastes capacity on features that could be specialized; too few shared experts forces the router to redundantly select the same experts for common features across all inputs. The paper does not ablate this ratio — it inherits the design choice from prior work.

Allocation of routed experts. The remaining neurons (those not selected as shared) are distributed among $n$ routed experts. The distribution procedure ensures balanced total importance across experts:

Sort the remaining neurons in descending order by $s_i$ .
Assign neurons to experts $E_r^{(1)}, E_r^{(2)}, \ldots, E_r^{(n)}$ in forward order.
When reaching $E_r^{(n)}$ , reverse direction and assign back to $E_r^{(1)}$ .
Continue alternating forward and reverse assignment until all neurons are allocated.

Why alternating assignment: Without this balancing, a naive round-robin allocation (neuron 1 to expert 1, neuron 2 to expert 2, etc.) would concentrate the highest-importance neurons in early experts, making some experts much more important than others. The alternating assignment distributes high-importance neurons approximately evenly: expert 1 gets neurons 1, $2n$ , $2n+1$ , $4n$ , etc.; expert $n$ gets neurons $n$ , $n+1$ , $3n$ , $3n+1$ , etc. This ensures the router has meaningful choices — all experts contain a mix of high- and medium-importance neurons.

Expert parameter count. The paper experiments with 16, 32, and 64 experts per MoE layer (Section 5.1). With 16 experts, each expert contains approximately $d_m / 16$ neurons. The activation ratio is set to 50% per layer, meaning that for any given input, 50% of the $d_m$ neurons are active. Since shared experts are always active (they contain $1/16 \approx 6.25\%$ of neurons), the router must select additional routed experts to reach 50% total activation. With 16 total experts, this means the shared expert plus approximately 7 routed experts (7/16 = 43.75%, total ≈ 50%) are active per input.

Layer exclusions. The first and last layers of the generation component are not converted to MoE. The paper states this is because they are "essential for preserving input encoding and output generation quality" — following standard MoE design practice where input embedding and output projection layers are kept dense to avoid routing instability near the model's interfaces.

Result after partition. At this stage, the model's MLP layers have been restructured into shared + routed expert groups, but no training has occurred. The experts contain the original pretrained weights, partitioned as described. The paper reports this "Expert Partition" baseline in Table 4 as "Zeroshot" (no training after partition): for BAGEL's generation component, this yields a GenEval overall score of 0.62 (down from 0.86 for the dense baseline), demonstrating that the partition alone — without training — causes significant degradation because the model has never learned to operate under sparse activation.

Stage 2: MoE Layer Formulation

The MoE layer's forward pass is defined as:

$\text{MoE}(x) = f_S(x) + \sum_{j \in \text{Top-}k(G)} G_j \cdot f_{R_j}(x)$

where $G$ denotes the gating function (router), $\text{Top-}k(G)$ selects the indices of the $k$ routed experts with the highest gating scores, $G_j$ is the scalar gating score for the $j$ -th selected expert, $f_S$ represents the transformation of the shared expert (always applied), and $f_{R_j}$ represents the transformation of the $j$ -th routed expert.

What it computes: For each input $x$ , the shared expert $f_S(x)$ is computed unconditionally — providing a baseline representation that captures universally useful features. Simultaneously, the router $G$ evaluates the input and produces a score for each routed expert. The top $k$ experts (by score) are selected, their transformations $f_{R_j}(x)$ are computed, each is multiplied by its gating score $G_j$ , and the weighted sum is added to the shared expert's output. Experts not in the top $k$ are not computed at all — their parameters remain dormant, saving FLOPs.

Why additive combination: The shared and routed expert outputs are summed rather than concatenated or otherwise combined. This preserves the residual structure of the transformer — the MoE output has the same dimension as the input, and can be cleanly added to the residual stream. The gating scores $G_j$ modulate how much each routed expert contributes, allowing the router to express varying degrees of confidence in different experts.

Gating reparameterization. The paper introduces a modification to standard MoE gating to enable smooth transition from dense to sparse activation. The gating score for each expert is reparameterized as:

$G_j = 1 + \text{Router}_j(x)$

where $\text{Router}_j(x)$ is the raw output of the router network for expert $j$ , and the router network's weight and bias are initialized to zero.

What this formulation does: When training begins, $\text{Router}_j(x) = 0$ for all $j$ , so $G_j = 1$ for all experts. The MoE layer initially behaves like the original dense MLP (all experts active with uniform weight 1), because the sum over all routed experts with weight 1 produces the same output as the original dense MLP. As training progresses and the router learns non-zero outputs, the gating scores diverge from 1, and the top- $k$ selection creates genuine sparsity.

Why zero-initialized gating: This initialization scheme is crucial for training stability. Standard MoE gating (with random initialization) would immediately introduce sparse activation before the experts have adapted to the sparse regime, causing a large initial quality drop that may be difficult to recover from. The $1 + \text{Router}(x)$ parameterization with zero initialization ensures a smooth warmup: the model starts at the dense solution and gradually introduces sparsity as the router learns. This is conceptually similar to the "router-tuning" approach in He et al. (2025) but applied to MoE conversion rather than dynamic depth.

Why no sum-to-one constraint: Standard MoE gating often uses a softmax over expert scores, ensuring $\sum_j G_j = 1$ . The paper explicitly relaxes this constraint, allowing the gating scores to sum to an arbitrary value. This is important because the shared expert is already contributing unconditionally — forcing the routed expert contributions to sum to 1 would effectively cap the total routed contribution, which is unnecessary when the shared expert provides a baseline. The unnormalized gating gives the model more flexibility to decide how much additional computation each input needs from the routed experts.

Stage 3: Expert-Frozen Tuning

This is a warmup phase that trains the model to operate under sparse activation while keeping all expert weights frozen — only the router and other non-expert parameters (e.g., layer normalization, attention projections) are trainable.

What is trained: The router network (which computes $\text{Router}_j(x)$ for each expert), layer normalization parameters, and any other parameters not part of the partitioned MLP experts. The shared and routed expert weights ( $W_d$ , $W_g$ , $W_u$ for each expert) remain exactly as they were after the expert partition stage.

Why freeze expert weights: The experts already contain high-quality pretrained representations. If all parameters were trainable from the start, the sparse activation (only top- $k$ experts active) would create a strong distribution shift — the selected experts would receive gradient updates while unselected experts would not, potentially causing the unselected experts to drift into irrelevance (a form of catastrophic forgetting where rarely-selected experts lose their specialized knowledge). Freezing the experts prevents this: the router must learn to select among fixed, high-quality expert representations rather than having the experts themselves adapt to the router's initial (possibly poor) choices.

Training procedure: The paper reports (Section 5.4, Figure 7) that only a few steps of expert-frozen tuning produce a substantial decline in MSE loss (the training objective for generation). With 64 experts, the loss drops from approximately 0.390 to 0.365 within 100 steps. The training uses whatever generation training data is available for each model — "high-quality image-text pairs, complemented by a small amount of synthetic data generated by existing text-to-image models" (Section 5.1). The exact optimizer, learning rate, and batch size are not specified in the paper.

What this achieves: Table 4 shows that for BAGEL with MoE applied to the generation component only ("Gen."), expert-frozen tuning improves the GenEval overall score from 0.62 (Expert Partition, zeroshot) to 0.78 — recovering a substantial fraction of the gap to the dense baseline (0.86). Visual quality improves markedly (Figure 8): before tuning, the model generates "noisy, low-detail images that fail to capture fine-grained semantics"; after expert-frozen tuning, images are more coherent and better aligned with prompts.

Router learning dynamics: During expert-frozen tuning, the router is the primary locus of learning. It must discover: (1) which routed experts contain representations relevant to different types of inputs (e.g., experts that specialize in object textures vs. spatial layouts vs. color rendering), and (2) how to balance the contributions of shared and routed experts through the gating scores. The training signal comes from the generation loss (MSE between generated and ground-truth images in the latent space), which provides a clear optimization target for router improvement — better expert selections produce lower reconstruction error.

Stage 4: Full MoE Adaptation

After expert-frozen tuning establishes a viable routing policy, the paper unfreezes the expert weights and continues training end-to-end. This allows the experts to specialize further under sparse activation.

What changes from expert-frozen tuning: All parameters — router, shared experts, routed experts, and non-expert components — are now trainable. The top- $k$ gating mechanism remains, meaning only selected experts receive gradient updates for any given input. This creates a natural specialization dynamic: experts that are frequently selected for certain types of inputs receive more gradient updates on those inputs, further adapting their representations to those input types, which in turn makes them more likely to be selected by the router for similar inputs in the future — a positive feedback loop that drives expert specialization.

Why additional training helps: Expert-frozen tuning produces a working routing policy but the experts themselves are still the original pretrained weights, partitioned into groups that were never optimized to work together as independent experts. Full MoE adaptation allows:

Expert specialization: Each routed expert can fine-tune its weights to better serve the subset of inputs for which it gets selected, developing more distinctive and complementary representations.
Compensating for missing experts: Since only a subset of routed experts are active per input, each active expert must learn to compensate for the absence of inactive experts that might have contributed useful features in the dense model.
Router refinement: The router can further optimize its selection policy based on how the adapting experts are changing, potentially discovering better expert-input assignments than were possible with frozen experts.

Training scale: The paper does not report the number of training steps, total training data volume, or computational cost for full MoE adaptation. This is a notable omission — the training cost is important for assessing whether the method is practical relative to alternatives (e.g., simply training a smaller dense model from scratch, or using distillation).

Performance outcome: Table 4 reports that full MoE adaptation on BAGEL's generation component achieves a GenEval overall score of 0.88, slightly exceeding the dense baseline of 0.86. The activated parameter count for the generation component is 4.96B (down from 7.62B), meaning the model matches or exceeds the dense baseline while activating approximately 65% of the generation component's parameters — or, stated differently, about half of the total model's parameters when both understanding and generation are considered.

Extension to the understanding component. The paper also explores applying MoE to both components ("Und. & Gen." in Table 4), but with a crucial asymmetry: the understanding component's experts are frozen after partition and never trained. This is because understanding tasks are more sensitive to compression, and the paper found that training understanding experts under sparse activation risks degrading understanding quality. The understanding component is kept dense for understanding tasks (all experts active) but sparse for generation tasks (only $k$ experts active, saving compute when the understanding component is used as a feature extractor for generation). This two-tier approach — frozen sparse for generation, dense for understanding — attempts to capture efficiency gains without sacrificing the understanding capability that was shown to be fragile under depth pruning.

Configuration details for the adapted BAGEL model. The generation component has MoE applied to all MLP layers except the first and last. With 16 experts (1 shared, 15 routed) and 50% activation ratio, approximately 8 experts are active per layer (the shared expert plus 7 routed experts). The activated parameter count of 4.96B for the generation component in Table 4 reflects this — 4.96 / 7.62 ≈ 65% of generation parameters active. The understanding component, when also converted to MoE, has 4.96B activated parameters out of 7.62B total, but note that for understanding tasks these experts must all be active (dense mode) to preserve quality, so the sparsity benefit only materializes during generation tasks.

Design Choices and Their Justifications

The paper makes several non-obvious design choices, each with specific justification.

Why neuron partition rather than gradient-based pruning for the training-free analysis: The paper explicitly compares with LLM-Pruner (Table 7), which uses gradient-based importance metrics. Neuron partition achieves comparable performance (0.71 vs. 0.70 overall on GenEval for Ming-Omni at 50% sparsity) while being simpler: no backward pass, no labeled data, no loss function specification. For a probing methodology where the goal is rapid iteration across models and sparsity levels, this practical simplicity matters more than marginal accuracy improvements. Gradient methods also require choosing which loss function to compute gradients with respect to — the understanding loss or the generation loss — which introduces a task assumption that the paper's activation-based method avoids (it can compute importance for any task by selecting calibration data, with no loss function specification needed).

Why depth pruning fails for understanding but not generation: This is an empirical finding rather than a design choice, but it motivates the paper's focus on width reduction for understanding components. The mechanism is error accumulation in autoregressive decoding (Section 5.2): understanding tasks require the model to generate token sequences where each token conditions on all previous tokens. Depth pruning introduces small per-token deviations that compound across the sequence, eventually causing the model to collapse into repetitive or nonsensical outputs (Figure 11 shows the model degenerating to repeating "portraying" after depth reduction). Generation tasks use the understanding component as a fixed feature extractor — its output is a single vector $f_{\text{und}}$ fed once to the generation component, with no iterative conditioning on the understanding component's own potentially degraded outputs. This means that understanding compression for generation-use only is much more forgiving than understanding compression for understanding-use, which is a subtle but important design consideration for any deployment optimization strategy.

Why 50% activation ratio: The paper sets a uniform 50% activation ratio across all MoE-converted layers. This choice balances two competing concerns: (1) sparsity (lower activation means more compute savings) and (2) capacity (higher activation means more representational power). The paper does not ablate this ratio — we don't know whether 40% or 60% would be better. The choice of 50% is likely because it provides a clear, interpretable benchmark ("matching dense performance while activating only half the parameters") and because architectures like DeepSeek-MoE have demonstrated that this ratio provides a good efficiency-quality tradeoff in language modeling contexts.

Why include shared experts: The shared expert mechanism ensures that there is always a baseline of universally useful computation active, regardless of the router's decisions. Without shared experts, a poorly initialized router could select experts that are all suboptimal for the current input, producing severely degraded outputs with no fallback. The shared expert provides a safety net — even if the router fails, the shared expert alone captures the most consistently important features (since it was constructed from the highest-importance neurons). This is particularly important during early training when the router is untrained and making essentially random expert selections.

Why expert-frozen tuning before full adaptation: This two-stage curriculum mirrors successful strategies in continual learning and parameter-efficient fine-tuning, where freezing pretrained weights during an initial adaptation phase prevents catastrophic forgetting. In the MoE adaptation context, the specific risk is that the router, initialized to zero, will initially make poor expert selections. If all parameters were trainable, the selected experts would receive gradient updates tailored to those (possibly random) inputs while unselected experts receive no updates, creating a vicious cycle: poorly selected experts adapt to inputs they weren't suited for, while well-suited experts remain frozen and become increasingly mismatched to the evolving router. Expert-frozen tuning breaks this cycle by forcing the router to learn to work with the existing expert representations before those representations are modified.

Why first and last layers excluded from MoE: The first layer processes the raw input embedding and the last layer produces the final output representation. These layers sit at the boundaries of the model and have a closer coupling to the input/output format than intermediate layers. Converting them to MoE would introduce routing decisions at these critical interfaces, potentially destabilizing the model's basic ability to encode inputs and decode outputs. This is standard practice in MoE literature — DeepSeek-MoE and other MoE architectures typically keep the embedding and output projection layers dense.

Why MoE adaptation but not distillation or retraining a smaller model: The paper does not explicitly justify MoE adaptation over alternative approaches to efficiency, but the implicit logic is: (1) distillation (training a smaller student model to mimic the larger teacher) would require a separate training pipeline and would lose the pretrained knowledge embedded in the original weights; (2) retraining a smaller dense model from scratch would be extremely expensive and might not converge to the same quality; (3) MoE adaptation retrofits sparsity into the existing pretrained model, preserving learned representations while introducing dynamic activation. The approach leverages the investment already made in pretraining rather than starting over, which is a practical advantage for organizations that have already trained large unified models.

Why calibration data choice matters more for unified models than for unimodal LLMs: In unimodal LLMs, the input distribution is text and the output is text — calibration data from different text domains (news, code, dialogue) produces similar importance rankings because the model's parameter utilization is relatively stable across text types. In unified models, inputs can be images, text, or both, and outputs can be text (autoregressive) or images (diffusion/autoregressive). These fundamentally different modalities activate fundamentally different computational pathways, so the calibration data's modality alignment with the target task is critical — more so than in any prior compression context. The paper demonstrates this empirically (Figures 5, 12, 13) and treats it as a first-order design consideration rather than a minor implementation detail.

4. Key Insights and Innovations

Innovation 1: Pruning as a Diagnostic Instrument, Not Just a Compression Tool

The paper's most distinctive conceptual move is repurposing training-free pruning from a deployment optimization technique into a systematic analytical methodology for understanding model organization. This is a genuine reframing of what pruning is for: rather than asking "how much can we remove while preserving accuracy?", the paper asks "what does the pattern of breakage under structured removal reveal about how different components allocate and utilize their capacity?"

This diagnostic framing is novel because prior work in both the pruning literature (Cheng et al., 2024; Sun et al., 2024; Ma et al., 2023) and the unified model literature (Deng et al., 2025; Wu et al., 2024) treated compression and capability as separate concerns — pruning was something you did after building a model to make it deployable, not a tool for understanding the model's internal organization. The paper inverts this: the compression experiments are the primary scientific instrument, and the efficiency gains from MoE adaptation are downstream of the insights generated.

The diagnostic power comes from the comparative anatomy approach: apply the same pruning operation (neuron partition, depth reduction) to different components (understanding vs. generation), at different granularities (layer-level vs. neuron-level), under different calibration conditions (understanding data vs. generation data), and observe the differential effects. This produces a functional map of the model: understanding components are robust to width reduction (Table 2: neuron partition at 50% sparsity preserves reasonable understanding performance) but catastrophically fragile to depth reduction (MME perception drops from 1684.8 to 304.5 with 50% layer removal); generation components are fragile to both width and depth reduction (Figure 6, Appendix A Figure 9); calibration data alignment with the target task determines which neurons are preserved and whether downstream performance holds (Figures 5, 12, 13). Each of these findings is a fact about how the model works, not merely a compression recipe.

This diagnostic methodology is what makes the paper more than an engineering report on compressing BAGEL. It demonstrates that the differential compressibility of model components is itself an informative signal about functional specialization — a principle that generalizes beyond the specific models and tasks studied. Future work on any heterogeneous architecture (multi-modal, multi-task, multi-lingual) could adopt this same probing methodology to map where capacity is genuinely shared versus merely co-located.

Innovation 2: The Discovery That Generation Components Are Fundamentally Incompatible with Static Compression

This is the paper's central negative result, and negative results with clear mechanistic explanations are genuine scientific contributions. Prior to this work, the compressibility of generation-specific components in unified models — diffusion transformers, autoregressive image decoders — was entirely unknown. The implicit assumption from the unimodal compression literature was that structured pruning techniques validated on LLMs would transfer with minor adaptation, since the underlying architectural building blocks (transformer layers, MLPs, attention) are the same.

The paper decisively refutes this assumption. The evidence is stark and multi-faceted: 50% width reduction in the generation component produces "distorted structures and unrealistic textures" (Figure 6, Section 5.3); depth reduction of even 14% of generation MLP layers causes "catastrophic degradation" (Appendix A, Figure 9); attention head pruning beyond 10% causes "noticeable performance drops" (Appendix A, Figure 10). These are not marginal regressions — they are qualitative collapses in output fidelity.

What makes this finding intellectually significant beyond the empirical observation is the mechanistic explanation: the dynamic activation analysis in Figure 3 shows that generation components exhibit fundamentally different parameter utilization patterns than the text decoders where pruning has been successful. Only a small fraction of neurons are "saturated" (consistently in the top 50% by activation across all inputs and timesteps), while most neurons are selectively active — important for some prompts or denoising timesteps but not others. Static pruning, by definition, makes a single binary keep/remove decision per neuron that must apply to all inputs. But in a component where the set of important neurons changes with the input and the timestep, any static mask will inevitably delete neurons critical for some subset of queries.

This is conceptually distinct from the standard overparameterization argument for why pruning works. The standard argument (Frankle & Carbin, 2018; Gromov et al., 2025) is that models contain redundant parameters — multiple neurons or layers that compute similar functions — so removing some fraction doesn't hurt because the remaining parameters can approximate the removed ones' function. The generation component's problem is different: it's not that neurons are redundant in the sense of being interchangeable, but rather that they are conditionally essential — each neuron may be genuinely necessary for some specific subset of inputs, even if it's dispensable on average. This is a fundamentally harder compression problem because it requires input-conditional parameter selection, not just identification of a universally-important subset.

This finding has important implications beyond this paper. It suggests that the compressibility of a neural network component depends not just on its architecture (transformer, CNN, MLP) but on the temporal dynamics of its computation. Autoregressive text decoders have relatively stable activation patterns across tokens (each token's processing looks broadly similar), making static pruning viable. Diffusion models iterate over noise levels with qualitatively different computations at early vs. late denoising steps, making static pruning fundamentally misaligned with the computational structure. This is a principle that should guide future compression research on any model with iterative refinement dynamics.

Innovation 3: Task-Aware Calibration as a First-Order Design Constraint for Multimodal Compression

The paper demonstrates that calibration data selection — typically treated as a minor implementation detail in pruning work — becomes a first-order design decision in multimodal settings because different modalities and tasks activate fundamentally different parameter subsets. This insight is crisply captured in Figure 2: when the authors identify the top 50% most-important neurons for understanding tasks and separately for generation tasks, the overlap between these two sets is small — meaning that a compression mask optimized for one task will delete neurons critical for the other.

The empirical demonstration in Figure 5 makes this concrete: when the understanding component's neurons are partitioned using generation-task calibration, the model produces faithful images (broccoli, scissors, dolphins) matching the prompts. When the same component is partitioned using understanding-task calibration — keeping neurons important for visual question answering — the generated images exhibit "distortions and mismatches," producing semantically inconsistent outputs. The quantitative ablations in Appendix E (Figures 12, 13) confirm the pattern: calibration aligned with the target task consistently outperforms mismatched calibration.

What makes this a genuine innovation rather than an obvious observation is that it overturns a working assumption from the unimodal pruning literature. In LLM pruning (Frantar et al., 2022; Sun et al., 2024; Men et al., 2024), calibration data is typically drawn from a generic text corpus (C4, WikiText) and the specific choice has modest effects because text-to-text models have relatively stable parameter importance distributions across text domains. The unified multimodal setting breaks this assumption: an image-and-text input for visual question answering activates different neural pathways than a text prompt for image generation, even within the same understanding component. The calibration data is not just a source of activation statistics — it selects which task's functional subnetwork gets preserved.

This insight has practical implications that the paper doesn't fully explore but that are logically downstream: if a unified model will be deployed primarily for generation tasks, its understanding component can be compressed much more aggressively (calibrating on generation data, which the paper shows tolerates 50-70% sparsity while maintaining generation quality — Table 3) than if it will be used for understanding tasks. This task-conditional compression budget represents a new degree of freedom in model deployment that the unimodal literature never had to consider.

Innovation 4: MoE Adaptation as a Response to Dynamic Sparsity, Not Just a Capacity-Scaling Mechanism

Mixture-of-Experts architectures have been primarily developed and justified as capacity-scaling mechanisms — ways to increase total parameter count while keeping per-input FLOPs manageable (Shazeer et al., 2017; Fedus et al., 2022; Dai et al., 2024; DeepSeek-AI et al., 2025). The standard narrative is: "we want a bigger model, but we can't afford to activate all parameters for every input, so we route inputs to different expert subsets." The paper uses MoE for a fundamentally different purpose: retrofitting dynamic activation into an already-trained dense model to recover quality that static compression destroyed.

This is a conceptual inversion. The standard MoE story is forward-looking: design the architecture with sparsity in mind from the start. The paper's story is backward-looking: take a dense model with known, high-quality weights, discover through analysis that its generation component cannot be statically compressed (Innovation 2), and then introduce dynamic routing as a mechanism to preserve the component's representational capacity while reducing average activation. The MoE is not enabling a larger model — it's enabling a sparser model that retains the original model's quality.

The specific design choices that make this inversion work are non-obvious: (1) expert partition based on importance scores from the pretrained weights rather than random initialization, ensuring each expert starts with high-quality, specialized representations; (2) the $1 + \text{Router}(x)$ gating reparameterization with zero initialization, which ensures the MoE layer initially computes the exact dense function and only gradually introduces sparsity; (3) the expert-frozen tuning phase, which prevents the catastrophic forgetting that would occur if expert weights were immediately trainable under sparse activation. These are not standard MoE design patterns — they are specific adaptations required when converting a dense model to sparse activation post-hoc rather than training sparse from scratch.

The results validate the approach: Table 4 shows that Expert Partition alone (zeroshot, no training) drops GenEval overall from 0.86 to 0.62 — the partitioned model without routing adaptation performs poorly because it has never learned to operate under sparse activation. Expert-frozen tuning recovers to 0.78. Full MoE adaptation reaches 0.88, slightly exceeding the dense baseline. The trajectory — 0.62 → 0.78 → 0.88 — demonstrates that the MoE mechanism is genuinely restoring lost quality, not merely providing additional capacity. The model activates only 4.96B of its 7.62B generation parameters (roughly 65%) while matching or exceeding the performance of the full dense model with all 7.62B parameters active.

This reframing of MoE as a compression recovery mechanism rather than a capacity-scaling mechanism is the paper's most transferable methodological insight. It suggests a general paradigm: when a component resists static compression due to dynamic, input-dependent activation patterns, introduce learned routing to preserve the component's ability to activate different parameter subsets for different inputs, and train the routing policy to approximate the dense model's behavior under sparse constraints. This could apply to any model component where the set of important parameters varies substantially across the input distribution.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper uses GenEval for generation quality assessment and MME, MMBench, MMMU, and MMVP for understanding performance. GenEval evaluates text-to-image generation across six sub-tasks: single object, two objects, counting, colors, position, and color attributes, producing an overall score as the average of sub-task scores. MME (Fu et al., 2023) separately measures perception and cognition capabilities through a comprehensive benchmark of multimodal large language models. MMBench, MMMU, and MMVP are standard multimodal understanding benchmarks covering diverse visual reasoning, multi-discipline knowledge, and visual perception respectively. The paper does not specify exact dataset sizes for these benchmarks, nor does it describe any custom data splits — the standard test sets for each benchmark are used as-is. For calibration datasets in training-free compression, the paper uses "a small number of examples drawn directly from the target task" (Section 5.1), explicitly noting that these samples are used solely to compute activation scores and require no ground-truth annotations. For MoE adaptation training data, the paper uses "high-quality image–text pairs, complemented by a small amount of synthetic data generated by existing text-to-image models" — exact dataset names, sizes, and sources are not specified.
Base model(s). The paper studies three unified multimodal models selected to span different architectural design choices: BAGEL (Deng et al., 2025) with a 7.62B-parameter VLM-based understanding component and a 7.62B-parameter LLM-based generation component (using Mixture-of-Transformers, where understanding and generation modules interact through cross-attention at every layer); Qwen-Image (Wu et al., 2025) with a 7.62B-parameter VLM understanding component and a 20.42B-parameter MMDiT-based generator; and Ming-Omni (AI et al., 2025) with a 17.12B-parameter MoE-VLM understanding component and a 2.51B-parameter multi-scale DiT block generation component. All three models use Qwen-Instruct (Yang et al., 2024) as the backbone for multimodal understanding. This selection is motivated by coverage of architectural diversity: BAGEL represents the Mixture-of-Transformers paradigm where understanding and generation share the same backbone architecture, Qwen-Image represents the case where the generation component dominates parameter count, and Ming-Omni represents the case where the understanding component uses MoE and the generation component is relatively small — enabling analysis of how architectural design choices interact with compressibility. For the FLOPs comparison, no separate pretraining-scaled baseline model is used (unlike the scaling analysis in the example paper); the comparison is purely between compressed and uncompressed versions of the same models.
Metrics. For generation quality, the primary metric is GenEval overall score — the average accuracy across six sub-tasks measuring whether generated images correctly reflect specified object counts, colors, positions, and attributes. For understanding quality, the paper reports MME perception score, MME cognition score, MMMU accuracy, MMBench accuracy, and MMVP accuracy — all standard metrics in the multimodal understanding literature where higher values indicate better performance. For MoE adaptation training, MSE loss (mean squared error between generated and ground-truth images in latent space) is used as the training objective. The paper does not report confidence intervals, standard deviations, or statistical significance tests for any of these metrics.
Baselines. The paper primarily compares against the dense (uncompressed) baseline — the original, unmodified unified model — across all compression experiments. Within training-free compression experiments, the paper also compares (1) neuron partition against depth reduction (layer dropping) to contrast width vs. depth compression strategies, (2) neuron partition using different calibration data sources (understanding vs. generation task samples) to isolate the effect of calibration alignment, and (3) neuron partition against LLM-Pruner (Ma et al., 2023), a gradient-based structured pruning method, to validate the activation-based importance metric (Appendix D, Table 7). Within MoE adaptation experiments, the baselines are: (1) Expert Partition zeroshot — the model immediately after expert partitioning with no training, (2) Dense Finetuning — the model compressed via neuron partition and then fine-tuned at the equivalent activated parameter budget, and (3) the dense baseline at full parameter count. The paper also compares neuron partition against 4-bit AWQ quantization (Lin et al., 2023) applied to the understanding component (Appendix D, Table 8).
Generation budget / compute accounting. For training-free compression, the "compute budget" is measured in sparsity ratio — the fraction of parameters (neurons or layers) removed. For depth pruning, ratios of 50%, 25%, and 14% are tested. For neuron partition, ratios of 25%, 50%, and 70% are tested for understanding components, and 50% for generation components. For MoE adaptation, the compute metric is activated parameters — the number of parameters actually used in a forward pass, which is lower than the total parameter count due to sparse expert activation. The activation ratio is set to 50% per MoE layer, meaning that for each MoE-converted layer, approximately half the neurons are active per input. Total activated parameters are reported in Table 4 for both the understanding and generation components in the format "Und. Param. & Gen. Param." (e.g., "4.96B + 4.96B" means 4.96 billion activated parameters in the understanding component and 4.96 billion in the generation component). The paper does not report wall-clock latency, FLOPs counts, or throughput measurements — all comparisons are in terms of activated parameter counts.
Cross-validation / statistical protocol. No cross-validation, statistical significance testing, or confidence interval computation is reported. The paper conducts ablations by varying calibration datasets (Section 5.2, Appendix E), expert counts (Figure 7: 16, 32, 64 experts), and compression ratios — but each configuration is reported as a single-point estimate without error bars or variance characterization. This is a notable methodological gap given that the test sets for benchmarks like GenEval and MME contain finite numbers of samples, and the calibration dataset selection introduces additional variance that could meaningfully affect results.

Main Quantitative Results

Training-Free Compression of Understanding Components

Headline finding for depth pruning on generation tasks (Figure 4, Section 5.2). Removing 50% of layers in the understanding component preserves generation quality for BAGEL and Qwen-Image but fails for Ming-Omni. The paper reports GenEval overall scores for three types of layer removal — blocks, MLP layers, and attention layers — with a compression ratio of 50%. BAGEL and Qwen-Image maintain high overall scores (exact values not tabulated but shown as bars near the top of the y-axis in Figure 4, with BAGEL presumably near 0.86 and Qwen-Image near 0.92 — the dense baselines from Table 3), while Ming-Omni drops substantially (from 0.82 dense to a lower value for all three removal types). The paper attributes Ming-Omni's greater sensitivity to its smaller generation component (2.51B vs. 7.62B for BAGEL and 20.42B for Qwen-Image), which "depends more heavily on precise features encoded by the understanding component."

Headline finding for depth pruning on understanding tasks (Table 6, Appendix C). Depth pruning catastrophically degrades understanding performance. Halving the number of MLP layers in BAGEL's understanding component causes MME perception to drop from 1684.8 to 304.5 and MME cognition from 696.7 to 127.1 — a near-total collapse representing roughly 82% and 82% relative degradation respectively. MMBench drops from 88.1 to 18.6, and MMMU from 65.0 to 16.7. For Ming-Omni, the same 50% depth reduction drops MME perception from 1584.3 to 1197.2 and MME cognition from 670.4 to 308.2 — a smaller but still severe degradation (roughly 24% and 54% relative drops). The paper's mechanistic explanation (Section 5.2) is error accumulation in autoregressive decoding: "Language-response-oriented understanding tasks rely on autoregressive decoding, which is inherently an error accumulation process, where the deviation of previous timesteps can propagate through subsequent decoding steps and ultimately cause the model to collapse within just a few steps." Figure 11 in Appendix C qualitatively demonstrates this collapse: the depth-reduced model degenerates into repeating "portraying" after a few tokens, while the neuron-partitioned model at the same compression ratio produces a coherent answer.

Headline finding for neuron partition on generation tasks (Table 3, Section 5.2). Neuron partition in the understanding component preserves generation quality remarkably well, with significant differences across models:

Model	Sparsity	Understanding Params.	GenEval Overall
BAGEL	0%	7.62B	0.86
BAGEL	25%	6.19B	0.84
BAGEL	50%	4.76B	0.63
Qwen-Image	0%	7.62B	0.92
Qwen-Image	50%	4.76B	0.90
Qwen-Image	70%	3.62B	0.82
Ming-Omni	0%	17.12B	0.82
Ming-Omni	50%	8.55B	0.79
Ming-Omni	70%	5.61B	0.71

Qwen-Image is the most robust: at 50% sparsity, GenEval overall drops only from 0.92 to 0.90 — a degradation of approximately 2.2%. At 70% sparsity, it still achieves 0.82, outperforming Ming-Omni's dense baseline (0.82 vs. 0.82 — equal at one-third the understanding parameters). Ming-Omni is moderately robust: 50% sparsity drops GenEval from 0.82 to 0.79 (3.7% relative degradation), while 70% sparsity drops to 0.71 (13.4% relative). BAGEL shows the steepest decline: 25% sparsity preserves performance (0.84 vs. 0.86), but 50% sparsity causes a sharp drop to 0.63 (26.7% relative degradation). The paper attributes BAGEL's greater sensitivity to its Mixture-of-Transformers architecture, "in which components interact more frequently through cross-attention at every layer" — more extensive interaction between understanding and generation components means understanding degradation propagates more severely to generation quality.

Headline finding for neuron partition on understanding tasks (Table 2, Section 5.2). Unlike depth pruning, width reduction preserves meaningful understanding capability even at aggressive ratios:

For BAGEL at 25% sparsity: MME perception drops from 1684.8 to 1558.1 (7.5% relative), MME cognition from 696.7 to 681.7 (2.2% relative), MMBench from 88.1 to 85.7 (2.7% relative), and MMMU from 65.0 to 60.1 (7.5% relative). At 50% sparsity: MME perception drops to 1392.6 (17.3% relative), MME cognition to 528.9 (24.1% relative), MMBench to 79.2 (10.1% relative), and MMMU to 56.7 (12.8% relative). These are substantial degradations but are orders of magnitude better than the near-complete collapse observed with depth pruning at the same 50% ratio (Table 6: MME perception 304.5, MMBench 18.6).

For Ming-Omni at 25% sparsity: MME perception drops from 1584.3 to 1578.5 (0.4% relative — negligible), MME cognition from 670.4 to 560.4 (16.4% relative), and MMMU from 66.7 to 56.7 (15.0% relative). At 50% sparsity: MME perception drops to 1269.0 (19.9% relative) and MME cognition to 317.9 (52.6% relative — a more severe degradation suggesting cognition tasks are more sensitive to width reduction in Ming-Omni's architecture).

Comparison of neuron partition vs. depth pruning for understanding tasks (Tables 2 vs. 6). At 50% sparsity on BAGEL, neuron partition achieves MMBench 79.2 while depth pruning achieves 18.6 — a 4.3× difference. The authors explain this by noting that "although certain layers exhibit substantial redundancy, aggressively removing them would also eliminate the small subset of weights that are critical to task performance. In contrast, neuron partition selectively preserves important neurons within each layer that are most relevant to the target task, thereby achieving more fine-grained and reliable compression." This is a granularity argument: layer-level removal is too coarse because every layer contains some critical neurons, while neuron-level removal can surgically excise only the low-importance neurons within each layer.

Calibration Data Alignment Effects

Headline finding (Figures 5, 12, 13; Section 5.2). The choice of calibration data for neuron partition strongly affects downstream performance, and the effect is task-asymmetric — generation tasks are more sensitive to calibration misalignment than understanding tasks.

Quantitative evidence for understanding tasks (Figure 12, Appendix E). When compressing the understanding component of BAGEL and Qwen-Image at 50% sparsity and evaluating on MME, using understanding-task calibration (MME samples) versus generation-task calibration (GenEval samples) produces notably different results. For BAGEL, the understanding-calibrated model achieves higher MME perception and cognition scores (exact values not tabulated but shown as taller bars in Figure 12 for both "Und. Calibrated" vs. "Gen. Calibrated" conditions), confirming that calibration aligned with the evaluation task is superior. For Qwen-Image, the same pattern holds, though the absolute difference between calibration choices appears smaller.

Quantitative evidence for generation tasks (Figure 13, Appendix E). The effect is more pronounced: for BAGEL, generation-calibrated compression achieves a substantially higher GenEval overall score than understanding-calibrated compression. The paper also reports a "test-time few-shot compression" variant that "directly uses a few test samples for calibration" and "achieves competitive performance" — suggesting that in deployment, a small number of examples from the target task can serve as calibration data without requiring a separate calibration dataset.

Qualitative evidence (Figure 5, Section 5.2). Figure 5 provides a visual side-by-side comparison of images generated after neuron partition with different calibration choices. Each row shows three outputs for the same prompt: the unmodified model (left), the model after neuron partition using generation calibration (middle), and using understanding calibration (right). For the prompt "A realistic broccoli sits upright on a plain surface," generation-calibrated compression produces a coherent, recognizable broccoli, while understanding-calibrated compression produces a distorted, barely-recognizable green blob. For "A pair of scissors lies on a flat surface," the generation-calibrated output shows recognizable scissors with correct metallic texture, while the understanding-calibrated output shows a distorted shape with incorrect coloring. For "dolphins swim through abandoned subway cars," the generation-calibrated output captures the aquatic-urban juxtaposition, while the understanding-calibrated output loses coherent structure. For "a fruit bowl consisting of fruits and miniature planets," the generation-calibrated output maintains the surreal composition, while the understanding-calibrated output degrades into visual noise.

The paper concludes: "task-aligned calibration data yields better performance, while mismatched data degrades generation quality. The effect is particularly critical for unified models, where both input and output types vary in different combination of modalities." This finding establishes calibration alignment as a prerequisite for any compression of unified models — a constraint that does not exist in unimodal settings where input and output modalities are fixed.

Generation Component Compression Sensitivity

Headline finding (Section 5.3, Figure 6). The generation component is fundamentally intolerant of static compression, with even moderate pruning causing catastrophic quality degradation. Figure 6 shows a qualitative comparison between the dense BAGEL model's generation output and the output after 50% width reduction in the generation component specifically. For the prompt "The word START," the dense model produces a clearly legible rendering of the word with appropriate styling. The compressed model produces barely-legible text with severe distortions — the letters lose their shape, the styling collapses, and semantic consistency with the prompt is lost. The paper states the compressed models "often produce distorted structures and unrealistic textures, deviating from the intended semantics."

Depth reduction in generation components (Appendix A, Figure 9). The paper extends the analysis to depth pruning, finding similarly catastrophic effects. Figure 9 shows generated outputs when removing 14 (50%), 7 (25%), 4 (14%), or 0 layers from the generation component's MLP blocks. At 50% depth reduction (14 layers removed), the output is essentially random noise — no coherent structure, no recognizable objects. At 25% depth reduction (7 layers removed), some vague structural elements emerge but are heavily distorted. At 14% depth reduction (4 layers removed), the output begins to show recognizable forms but with significant artifacts. The paper concludes that "removing entire layers also causes catastrophic degradation in generated outputs" and that "static compression along either width or depth alone struggles to preserve the performance of the original model."

Attention pruning in generation components (Appendix A, Figure 10). Attention head or layer pruning is even more sensitive: "more than a 10% reduction results in noticeable performance drops." This is more severe than MLP pruning, suggesting that the attention mechanism in generation components is particularly fragile — likely because attention layers are responsible for spatial coherence and cross-token interactions that are critical for generating images with correct structure.

Comparison with understanding component compressibility. The paper draws a stark contrast: whereas understanding components maintain generation quality at 50-70% width reduction (Table 3: Qwen-Image at 0.90 GenEval with 50% sparsity), generation components cannot tolerate even 25% width reduction without severe quality loss. This asymmetric compressibility is the paper's central empirical finding and the primary motivation for the MoE adaptation approach.

MoE Adaptation Results

Headline finding (Table 4, Figure 8, Section 5.4). MoE adaptation enables the generation component to match or exceed dense model performance while activating only approximately 65% of its parameters. The full trajectory from expert partition through MoE adaptation on BAGEL:

Method	Adapt. Comp.	Activated Params.	GenEval Overall
Baseline (dense)	N/A	7.62B + 7.62B	0.86
Expert Partition (zeroshot)	Gen.	7.42B + 4.96B	0.62
Dense Finetuning	Gen.	7.42B + 4.96B	0.82
Expert-Frozen Tuning	Gen.	7.42B + 4.96B	0.78
MoE Adaptation	Gen.	7.42B + 4.96B	0.88

The Expert Partition zeroshot configuration (no training after expert partitioning) achieves only 0.62 GenEval overall — a 27.9% relative degradation from the 0.86 dense baseline. This confirms that simply partitioning neurons into experts without teaching the model to route among them is insufficient; the model has never learned to operate under sparse activation.

Dense Finetuning — neuron partition to 4.96B parameters followed by fine-tuning — recovers to 0.82, demonstrating that some of the lost quality can be regained through training even without dynamic routing. However, this is a static compression approach (all remaining parameters are always active), and it still underperforms the dense baseline by 0.04 absolute (4.7% relative).

Expert-Frozen Tuning improves to 0.78 — actually slightly worse than Dense Finetuning (0.82). The paper does not explicitly discuss this comparison, but the implication is that expert-frozen tuning alone (where the router is learning but experts are frozen) provides less representational flexibility than dense fine-tuning (where all weights can adapt). However, expert-frozen tuning serves as a necessary warmup for the subsequent full MoE adaptation stage.

Full MoE Adaptation reaches 0.88, exceeding the dense baseline of 0.86 by 0.02 absolute — a 2.3% relative improvement — while activating only 4.96B of the generation component's 7.62B parameters (approximately 65%). This is the paper's headline result: the MoE-adapted model activates about half of the total unified model's parameters (understanding + generation = 7.42B + 4.96B ≈ 12.38B out of 15.24B total, or about 81% of total — but focused on the generation component where the sparsity is 4.96/7.62 ≈ 65%) and matches or exceeds the full dense model's generation quality.

Extension to understanding component (Table 4, "Und. & Gen." rows). When MoE is applied to both components:

Method	Adapt. Comp.	Activated Params.	GenEval Overall
Baseline	N/A	7.62B + 7.62B	0.86
Expert Partition (zeroshot)	Und. & Gen.	4.96B + 4.96B	0.28
Dense Finetuning	Und. & Gen.	4.96B + 4.96B	0.81
Expert-Frozen Tuning	Und. & Gen.	4.96B + 4.96B	0.63
MoE Adaptation	Und. & Gen.	4.96B + 4.96B	0.85

The zeroshot degradation is catastrophic — 0.28 GenEval overall, representing a 67.4% relative drop from 0.86. This is substantially worse than the generation-only zeroshot (0.62), confirming that applying untrained sparse activation to the understanding component (which feeds features to the generation component) compounds the quality loss. Expert-frozen tuning recovers to 0.63, dense fine-tuning to 0.81, and full MoE adaptation to 0.85 — slightly below the dense baseline (0.86) but within approximately 1.2% relative difference. Notably, this configuration activates only 4.96B parameters in BOTH components (9.92B total out of 15.24B, or approximately 65%), achieving near-dense quality with roughly two-thirds of the total parameters active.

Training dynamics (Figure 7, Section 5.4). Expert-frozen tuning with different numbers of experts (16, 32, 64) shows that finer-grained expert partitioning leads to lower training loss. At 100 training steps, the 64-expert configuration achieves an MSE loss of approximately 0.365, compared to approximately 0.370 for 32 experts and approximately 0.374 for 16 experts. The loss curves decrease rapidly in the first 20-40 steps and then plateau, suggesting that most of the routing learning happens quickly. The paper interprets the lower loss for more experts as evidence that "finer-grained expert partitioning enables more flexible activation combinations."

Qualitative progression (Figure 8, Section 5.4). Figure 8 shows generated images across the adaptation stages for four prompts of varying complexity. The "Zeroshot w/o SE" (without shared experts) condition produces noisy, low-detail outputs with poor semantic alignment — for "A famous flower that symbolizes wealth in China," the output is an indistinct red blob; for "Old analog picture of parked car on side street, quiet night," the output is random color patches with no recognizable objects. Adding shared experts ("Zeroshot w/ SE") improves structure slightly but still produces low-quality outputs. Expert-frozen tuning produces a qualitative leap: the flower becomes recognizable (though still coarse), the Easter activity gains thematic elements, the night street scene has visible structure, and the astronaut painting scene captures the concept. Full MoE adaptation further refines details: the flower gains petal definition, the Easter activity gains clearer object boundaries, the night scene gains sharper car and street elements, and the astronaut scene gains better color and spatial composition. The paper states that "these additional training stages enable the experts to refine their internal representations and develop stronger specialization, improving both structural coherence and semantic consistency in generated outputs."

Cross-model generalizability. The MoE adaptation results are reported only on BAGEL. The paper does not present MoE adaptation results for Qwen-Image or Ming-Omni, making it unclear whether the approach transfers to different generation component architectures (MMDiT, multi-scale DiT). This is a significant scope limitation — the MoE adaptation's effectiveness may depend on BAGEL's specific Mixture-of-Transformers design where the generation component is an LLM backbone, and may not apply to the dedicated image generator architectures in Qwen-Image and Ming-Omni.

Ablation Studies and Robustness Checks

Expert count in expert-frozen tuning (Figure 7, Section 5.4). Configurations with 16, 32, and 64 experts per MoE layer are compared during the expert-frozen tuning phase on BAGEL's generation component. The 64-expert configuration achieves the lowest training MSE loss (approximately 0.365 at 100 steps), 32 experts achieve intermediate loss (approximately 0.370), and 16 experts achieve the highest loss (approximately 0.374). The loss curves decrease rapidly in the first ~20 steps and largely plateau by step 60-80 across all configurations. This suggests that finer expert granularity provides more flexible activation combinations, but the marginal benefit diminishes as expert count increases (the gap between 64 and 32 is smaller than between 32 and 16). The paper does not report the effect of expert count on final generation quality after full MoE adaptation — only training loss during the warmup phase is shown, making it unclear whether the loss advantage of 64 experts translates to improved GenEval scores.

Shared expert presence (Figure 8, Section 5.4). The qualitative comparison in Figure 8 includes "Zeroshot w/o SE" (without shared experts) and "Zeroshot w/ SE" (with shared experts, where SE constitutes one-sixteenth of total experts) conditions. The shared expert condition produces visibly better outputs than the no-shared-expert condition across all four prompts shown — the images have more structure and better color coherence even before any training. However, quantitative GenEval scores for this ablation are not reported, and the comparison is only shown for the zeroshot (pre-training) condition, not for expert-frozen tuning or full MoE adaptation stages. This leaves open the question of whether shared experts remain important after training, or whether the router can learn to compensate for their absence.

Calibration data source for neuron partition (Figures 5, 12, 13; Appendix E). This is the most extensively ablated design choice. For understanding tasks (Figure 12): compressing BAGEL and Qwen-Image using understanding-task calibration (MME samples) vs. generation-task calibration (GenEval samples) and evaluating on MME shows that understanding-calibrated compression yields higher MME perception and cognition scores. For generation tasks (Figure 13): the same comparison shows generation-calibrated compression yields higher GenEval scores. The qualitative comparison in Figure 5 (discussed above) reinforces this with visual evidence of degraded outputs under mismatched calibration. The paper also reports that "test-time few-shot compression" — using a few test samples as calibration data — "seamlessly adapts to downstream tasks and achieves competitive performance," though no quantitative comparison between few-shot calibration and task-matched calibration is provided.

Depth pruning granularity in generation components (Appendix A, Figure 9). Removing 14 (50%), 7 (25%), 4 (14%), and 0 MLP layers from the generation component shows a monotonic degradation: 0 layers produces the baseline quality, 4 layers introduces mild artifacts, 7 layers causes substantial structural distortion, and 14 layers produces near-random noise. This confirms that even moderate depth reduction (14% of layers) causes noticeable quality loss, and 50% reduction is catastrophic.

Attention pruning in generation components (Appendix A, Figure 10). The paper separately tests width reduction (pruning attention heads within layers) at ratios of 50%, 25%, 10%, and 0%. Even 10% reduction causes "noticeable performance drops" — the generation component's attention mechanism appears to be even more compression-sensitive than its MLP layers, consistent with attention being responsible for spatial coherence that is critical for image generation.

Neuron partition vs. gradient-based pruning (Appendix D, Table 7). At 50% sparsity on Ming-Omni's understanding component, neuron partition achieves a GenEval overall score of 0.71 vs. 0.70 for LLM-Pruner (Ma et al., 2023) — a gradient-based method. The sub-task breakdown: LLM-Pruner scores higher on counting (0.72 vs. 0.58 for neuron partition), while neuron partition scores higher on position (0.49 vs. 0.47) and color attributes (0.56 vs. 0.55). The overall scores are nearly identical (0.71 vs. 0.70), demonstrating that the simpler activation-based metric is competitive with gradient-based methods. The paper emphasizes that neuron partition "eliminates the dependence on labeled data and explicit gradient computation," making it more practical for rapid analysis — though this advantage is partly offset by the need for careful calibration data selection, which gradient methods may be less sensitive to.

Neuron partition vs. quantization (Appendix D, Table 8). Comparing 50% neuron partition against 4-bit AWQ quantization (Lin et al., 2023) applied to Qwen-Image's understanding component: neuron partition achieves GenEval overall 0.90 vs. 0.88 for 4-bit quantization. The sub-task breakdown reveals that neuron partition substantially outperforms on color attributes (0.87 vs. 0.70) while being slightly worse on position (0.76 vs. 0.79) and counting (0.94 vs. 0.93). The paper notes this as a noteworthy finding: "This stands in contrast to traditional LLM compression, where pruning at similar ratios typically leads to noticeable performance degradation and remains significantly weaker than 4-bit quantization." The implication is that multimodal understanding components are unusually amenable to structured pruning compared to unimodal LLMs, possibly because the visual processing pathways introduce additional redundancy.

Depth reduction granularity in understanding components (Appendix C, Table 6, Figure 11). The paper reports depth reduction at 50% for both Ming-Omni and BAGEL on understanding tasks. The degradation is severe for both but substantially worse for BAGEL: MME perception drops from 1684.8 to 304.5 (82% relative) for BAGEL vs. 1584.3 to 1197.2 (24% relative) for Ming-Omni. Figure 11 provides a qualitative example: the depth-reduced BAGEL model, when asked to explain a meme, collapses after a few tokens into repeating the word "portraying" indefinitely, while the neuron-partitioned model at the same compression ratio produces a coherent, multi-sentence explanation. This ablation establishes that depth and width compression are not interchangeable — width reduction preserves functional integrity even when depth reduction destroys it — and that this holds at the same overall sparsity ratio.

Attention head partition in understanding components (Appendix B, Table 5). Applying neuron partition to attention heads (rather than MLP neurons) at 50% sparsity per layer in BAGEL's understanding component: when compressing layers 3-27, GenEval overall drops from 0.86 to 0.67; when compressing only layers 4-27 (preserving layer 3), overall score improves to 0.72. This suggests that early attention layers are more important for generation quality than middle or late layers, and that leaving the first few attention layers uncompressed is beneficial. The sub-task breakdown reveals that position and color attributes are the most affected by attention head pruning (position drops from 0.72 to 0.33 for layers 3-27), indicating that spatial reasoning and attribute binding rely heavily on attention mechanisms in the understanding component.

MoE adaptation components (Table 4). Comparing MoE adaptation applied only to the generation component ("Gen.") vs. applied to both components ("Und. & Gen."): the generation-only configuration achieves 0.88 GenEval overall (exceeding the 0.86 dense baseline), while the both-components configuration achieves 0.85 (slightly below baseline). The both-components configuration activates fewer total parameters (4.96B + 4.96B = 9.92B vs. 7.42B + 4.96B = 12.38B for generation-only), representing a more aggressive efficiency gain at a small quality cost. The paper notes that for the both-components configuration, "the experts in the understanding component are kept frozen, remaining fully activated for understanding and only sparsely activated for generation, since generation tasks are more tolerant to sparsity in this component." This asymmetric treatment — dense for understanding-use, sparse for generation-use — is an important design choice that the paper does not ablate (e.g., what happens if the understanding component is also sparsely activated for understanding tasks?).

Training stages of MoE adaptation (Table 4, Figure 8). The progression from Expert Partition (zeroshot) → Expert-Frozen Tuning → Full MoE Adaptation is tracked both quantitatively (Table 4) and qualitatively (Figure 8). For generation-only adaptation: 0.62 → 0.78 → 0.88. The jump from 0.62 to 0.78 represents a 25.8% relative improvement from expert-frozen tuning, and the jump from 0.78 to 0.88 represents an additional 12.8% relative improvement from unfreezing expert weights. The first stage (learning to route among frozen experts) provides roughly twice the improvement of the second stage (fine-tuning expert weights), suggesting that establishing a viable routing policy is the primary challenge, and expert specialization provides smaller but still meaningful additional gains.

Critical Assessment

Claim 1 from the executive summary: "the understanding component exhibits high compressibility in both understanding and generation tasks." The evidence for this claim is strong but qualified in important ways. For generation tasks, the evidence is unambiguous: Table 3 shows Qwen-Image's understanding component can be compressed to 50% sparsity with GenEval overall dropping only from 0.92 to 0.90 (2.2% relative), and Ming-Omni to 50% with a drop from 0.82 to 0.79 (3.7% relative). Even 70% sparsity on Qwen-Image achieves 0.82, matching Ming-Omni's dense performance. For BAGEL, the claim is weaker: 25% sparsity preserves generation quality (0.84 vs. 0.86), but 50% sparsity causes a sharp drop to 0.63 (26.7% relative). The paper attributes this to BAGEL's Mixture-of-Transformers architecture, which is a plausible explanation but also means the claim does not hold uniformly — architectural design choices mediate compressibility.

For understanding tasks, the evidence supports moderate compressibility (25% sparsity) more than aggressive compressibility (50% sparsity). At 25% neuron partition on BAGEL: MME perception drops only 7.5% relative, MME cognition 2.2%, MMBench 2.7%. These are modest degradations consistent with "high compressibility." But at 50% sparsity: MME perception drops 17.3%, MME cognition 24.1%, MMMU 12.8% — these are substantial degradations that would be problematic in many deployment scenarios. The paper's language of "high compressibility" should be understood as "compressible relative to generation components" and "compressible relative to depth pruning," not "compressible with negligible quality loss at all ratios."

A missing analysis: the paper does not report neuron partition results for understanding tasks at sparsity ratios between 25% and 50% (e.g., 35%, 40%), making it impossible to identify the "knee" in the quality-sparsity curve — the point beyond which degradation accelerates. This is important for practical deployment decisions.

Claim 2 from the executive summary: "generation components are dramatically sensitive to static compression, where even moderate pruning causes catastrophic quality degradation." This claim is strongly supported, though the evidence is primarily qualitative rather than quantitative. Figure 6 shows a single example of 50% width reduction causing severe degradation, but no quantitative GenEval scores are reported for generation component compression — only the qualitative statement that "compressed models often produce distorted structures and unrealistic textures." Appendix A Figures 9 and 10 provide additional qualitative evidence for depth reduction and attention pruning, but again without quantitative metrics. The paper would be strengthened by reporting GenEval scores at different generation component compression ratios, analogous to Table 3 for the understanding component. The claim that "moderate pruning causes catastrophic quality degradation" is visually evident but the threshold of "moderate" is not precisely characterized — does 10% width reduction cause noticeable degradation? 25%? The paper states that attention pruning beyond 10% causes noticeable drops, but MLP width reduction thresholds are not quantified.

A more fundamental limitation: the generation component compression experiments apply the same neuron partition metric (Equation 7) that was developed for and validated on understanding components. The paper does not investigate whether a different importance metric might work better for generation components specifically — perhaps the dynamic activation patterns require a metric that accounts for variance across inputs rather than mean activation. This is an important ablation that is missing.

Claim 3 from the executive summary: "MoE Adaptation achieves performance comparable to the full model while activating only about half of its parameters." This claim is well-supported for BAGEL but its generalizability is untested. Table 4 shows full MoE adaptation on BAGEL's generation component achieves GenEval overall 0.88 vs. 0.86 baseline — exceeding the dense model while activating 4.96B of 7.62B generation parameters (~65%). The "about half" language in the executive summary is slightly misleading: the generation component activates ~65%, not 50%, and when considering the full model (understanding + generation), the activation ratio is even higher (12.38B out of 15.24B ≈ 81% for the Gen.-only configuration). Only in the "Und. & Gen." configuration does total activated parameter count approach 50% (9.92B out of 15.24B ≈ 65%), and in that case GenEval drops to 0.85, slightly below baseline. So "about half" is an overstatement — the actual activation ratios range from 65% to 81% depending on configuration, and the configuration that most closely approaches 50% global sparsity slightly underperforms the dense baseline.

Critical missing evidence: the paper does not report MoE adaptation results for Qwen-Image or Ming-Omni. Since these models have fundamentally different generation component architectures (MMDiT, multi-scale DiT) than BAGEL (LLM-based), it is entirely unknown whether MoE adaptation transfers. The paper's claim is effectively "MoE adaptation works for BAGEL's generation component" — a single-model, single-architecture result. This is a significant scope limitation that the paper does not adequately acknowledge.

Another missing analysis: the paper does not compare MoE adaptation against alternative approaches that could achieve similar efficiency gains, such as (1) training a smaller dense generation component from scratch (which would avoid the complexity of expert partitioning and routing), (2) distillation of the generation component into a smaller model, or (3) simply using a lower-rank approximation of the MLP weight matrices. Without these baselines, it's unclear whether MoE adaptation is the most effective or practical approach to generation component sparsity, or merely one that works.

The calibration data sensitivity finding. This is one of the paper's most robust and well-demonstrated findings, with quantitative (Figures 12, 13), qualitative (Figure 5), and mechanistic (Figure 2) evidence all converging on the same conclusion: calibration data must align with the target task because different tasks activate different neuron subsets. This is a genuine contribution that should influence future compression work on multimodal models. However, the practical implications are somewhat ambiguous: in a deployment setting where the model will serve both understanding and generation requests, which calibration data should be used? The paper suggests task-specific compression (different masks for different tasks) but does not implement or evaluate this approach, and it's unclear how the overhead of maintaining and switching between multiple compression masks would compare to the efficiency gains.

Statistical rigor and reproducibility concerns. The paper reports no confidence intervals, standard deviations, or statistical tests for any metric. The test sets (GenEval, MME, MMBench, etc.) have finite sizes, and the variance in scores — particularly for broken-out sub-tasks with smaller sample counts — could be substantial. The calibration data is described as "a small number of examples" without specifying exact sizes, which makes the experiments difficult to reproduce. The MoE adaptation training data is described vaguely as "high-quality image–text pairs, complemented by a small amount of synthetic data" — without dataset names, sizes, or filtering criteria. These omissions are significant for a paper that presents MoE adaptation as its primary methodological contribution; reproducibility requires knowing the training recipe.

What would strengthen the paper. Several experiments would address the limitations discussed above: (1) Quantitative GenEval scores for generation component compression at multiple sparsity ratios, creating a proper compressibility curve analogous to Table 3; (2) MoE adaptation results on Qwen-Image and Ming-Omni to establish cross-architecture generalizability; (3) a comparison of MoE adaptation against training a smaller dense generation component or distillation; (4) an ablation on the shared expert ratio (one-sixteenth) and activation ratio (50%) to determine sensitivity to these hyperparameters; (5) a reported standard deviation or confidence interval for the main metrics; (6) explicit calibration dataset sizes and MoE training dataset specifications; and (7) a task-switching experiment where a single model uses task-specific compression masks for understanding vs. generation requests, measuring both quality and the overhead of mask switching.

Overall assessment. The paper's primary contributions are analytical rather than methodological: the demonstration that understanding and generation components have fundamentally different compressibility profiles, the identification of dynamic activation patterns as the mechanism underlying generation component sensitivity, and the calibration data alignment requirement for multimodal compression. These are well-supported by the experiments, though the quantitative evidence for generation component sensitivity is thinner than for understanding component compressibility. The MoE adaptation method is a promising direction motivated by the analytical findings, but the evidence for it is limited to a single model (BAGEL) with a specific architecture, and important baselines and ablations are missing. The paper succeeds as a diagnostic study that opens up a new problem space — efficient inference for unified multimodal models — and provides initial characterization and a prototype solution, but falls short of establishing MoE adaptation as a general, validated method.

6. Limitations and Trade-offs

Limitation 1: MoE Adaptation Is Validated on a Single Architecture (BAGEL Only)

The assumption or constraint. The paper's primary methodological contribution — MoE Adaptation for recovering generation quality under sparse activation — is demonstrated exclusively on BAGEL (Section 5.4), a model whose generation component adopts a Mixture-of-Transformers design that reuses the Qwen-Instruct LLM backbone. The authors do not apply MoE Adaptation to Qwen-Image (whose generation component is a 20.42B MMDiT-based generator) or Ming-Omni (whose generation component is a 2.51B multi-scale DiT block). The paper provides no justification for this restriction and does not explicitly acknowledge it as a scope limitation in the main text, though Table 1 documents the architectural diversity of the three models studied.

The consequence. The central finding that MoE Adaptation "achieves performance comparable to the full model while activating only about half of its parameters" is, at present, a single-model result. There is no evidence that the approach transfers to the fundamentally different generation architectures used by Qwen-Image and Ming-Omni — dedicated image generators (MMDiT, DiT) that do not share the LLM backbone architecture with their understanding components. MMDiT-based generators operate through joint text-image attention and iterative denoising with architectural features (e.g., dual-stream attention, adaLN conditioning) that have no analog in BAGEL's autoregressive LLM-based generator. The dynamic activation patterns documented in Figure 3 are measured on BAGEL's generation component specifically; it is unknown whether MMDiT and DiT generators exhibit similar sample-dependent neuron activation patterns or whether the expert partitioning strategy (which assumes MLP neurons can be grouped by cumulative importance) generalizes to these architectures. A practitioner evaluating this work for deployment on Qwen-Image or Ming-Omni — or any unified model not based on the Mixture-of-Transformers paradigm — has no empirical basis to estimate MoE Adaptation's effectiveness.

What evidence exists in the paper. Only BAGEL MoE Adaptation results appear (Table 4, Figures 7, 8). The training-free analysis covers all three models (Tables 2, 3; Figures 4, 5), demonstrating that the diagnostic methodology generalizes, but the therapeutic methodology does so only for one architecture. Table 1 documents that the three models' generation components are architecturally heterogeneous — 7.62B LLM (BAGEL), 20.42B MMDiT (Qwen-Image), 2.51B DiT (Ming-Omni) — which underscores this gap.

Mitigation status. The paper does not acknowledge this as a limitation or propose future work to validate MoE Adaptation on other unified architectures. The paper's claim scope ("the adapted BAGEL model achieves performance comparable to the full model") is technically accurate in naming BAGEL, but the broader framing of MoE Adaptation as a general solution for generation component sparsity is not supported by the experimental coverage.

Limitation 2: "About Half" Activation Claim Overstates Achieved Sparsity

The assumption or constraint. The paper's framing — both in the abstract and Section 5.4 — emphasizes that MoE Adaptation activates "only about half of its parameters" or enables the model to match the full model "while activating only about half of neurons." However, the actual activation ratios reported in Table 4 tell a more nuanced story. For the best-performing configuration (MoE Adaptation applied to the generation component only, "Gen." row), the activation counts are 7.42B (understanding) + 4.96B (generation) = 12.38B out of 15.24B total, or approximately 81% of total parameters active. The generation component specifically activates 4.96B of 7.62B, or approximately 65%, not 50%. Only in the "Und. & Gen." configuration — where both components are MoE-converted — does the total activation drop to 4.96B + 4.96B = 9.92B out of 15.24B, or approximately 65% of total parameters. At no point in the paper's results does a configuration activate "about half" (i.e., roughly 50%) of total parameters while matching dense performance. The 50% figure references the per-layer activation ratio set during MoE conversion (Section 5.1: "The overall activation ratio is set to 50% per layer"), but this per-layer ratio does not translate to 50% global activation because (1) the first and last layers remain dense, (2) shared experts are always active regardless of routing, and (3) the understanding component in the "Gen." configuration remains fully dense.

The consequence. The "about half" language in the abstract and conclusion creates a misleading impression of the achieved efficiency gain. A reader who encounters the statement that the model "activates only about half of its parameters" would reasonably expect a ~2× reduction in per-inference compute, which is not supported by the reported numbers (the reduction is closer to 1.2–1.5× depending on configuration). The actual efficiency improvement — activating ~65-81% of parameters rather than 100% — is substantially more modest than the framing suggests. This matters for downstream deployment planning: a 20-35% reduction in activated parameters translates to a proportionally smaller latency/throughput improvement than a 50% reduction would, and may not justify the engineering complexity of implementing MoE routing.

What evidence exists in the paper. Table 4 provides the activated parameter counts. The generation-only MoE configuration shows 7.42B + 4.96B = 12.38B activated vs. 15.24B total = 81.2% activation. The both-components configuration shows 9.92B vs. 15.24B = 65.1% activation. The per-layer 50% activation ratio is stated in Section 5.1. The discrepancy between per-layer ratio and global activation is never explicitly calculated or discussed in the paper.

Mitigation status. Not addressed. The paper does not compute global activation ratios explicitly, does not acknowledge the gap between per-layer and global sparsity, and uses the "about half" phrasing in both the abstract and conclusion without qualification. A more precise characterization — e.g., "activating approximately two-thirds of total parameters while matching dense performance" — would accurately reflect the data.

Limitation 3: Difficulty Estimation or Analogous Pre-Computation Cost Is Not Accounted for

The assumption or constraint. The MoE Adaptation pipeline requires several pre-computation and training stages whose costs are not quantified or included in any efficiency accounting: (1) collecting a calibration dataset and computing per-neuron importance scores $s_i$ (Equation 7) across all MLP layers in the generation component to perform Expert Partition; (2) expert-frozen tuning (Figure 7 shows ~100 training steps, but the computational cost in GPU-hours or FLOPs is unreported); and (3) full MoE Adaptation training (duration, data volume, and cost unreported). Additionally, the training-free analysis stage — running the three models through calibration datasets to characterize compressibility and activation patterns — represents a one-time research cost, but any deployment of MoE Adaptation on a new unified model would need to repeat the Expert Partition and training stages, making them part of the method's effective cost.

The consequence. The headline efficiency gain — matching dense performance with fewer activated parameters — is a per-inference metric that ignores the upfront cost of enabling that sparse activation. This is analogous to reporting a compressed model's inference speedup without accounting for the time and compute spent on pruning and fine-tuning. For a deployment where the same model serves many queries, the upfront cost may amortize well. For a deployment with limited queries or where the model must be frequently updated (and thus re-adapted), the upfront cost could dominate. Without any quantification of the training cost, practitioners cannot make this amortization calculation. The paper also does not report whether the MoE adaptation training is stable across random seeds, what volume of training data is needed, or whether the training is sensitive to hyperparameters — all of which affect the practical reproducibility and cost of the approach.

What evidence exists in the paper. Section 5.1 mentions using "high-quality image–text pairs, complemented by a small amount of synthetic data generated by existing text-to-image models" for MoE adaptation training, but dataset names, sizes, and sources are unspecified. Figure 7 shows training loss curves over 100 steps but reports no wall-clock time, GPU configuration, or FLOPs count. The Expert Partition procedure (computing importance scores from calibration data) is described algorithmically in Section 4.2 but its computational cost relative to standard inference is not discussed.

Mitigation status. Not addressed. The paper does not acknowledge the unaccounted training cost as a limitation, does not report training resource requirements, and does not discuss the amortization tradeoff. This is a significant gap for a paper whose primary claimed contribution is an efficiency-improving method.

Limitation 4: Generation Component Compressibility Is Not Quantitatively Characterized

The assumption or constraint. The paper's central empirical finding — that generation components are "dramatically sensitive to static compression" where "even moderate pruning causes catastrophic degradation" (Sections 5.3, 6) — is supported almost entirely by qualitative evidence. Figure 6 shows a single example of degraded generation after 50% width reduction in BAGEL's generation component. Appendix A Figures 9 and 10 provide additional qualitative examples for depth reduction and attention pruning. However, the paper never reports quantitative GenEval scores for generation component compression at any sparsity ratio. In contrast, the understanding component's compressibility is extensively quantified in Table 3 (GenEval scores at 0%, 25%, 50%, 70% sparsity) and Table 2 (MME, MMBench, MMMU, MMVP scores at 0%, 25%, 50% sparsity). The generation component results remain anecdotal.

The consequence. Without quantitative metrics, several critical questions cannot be answered: (1) At what sparsity ratio does generation component degradation become "catastrophic"? Is it 10%, 25%, or 50%? The paper states that attention pruning beyond 10% causes "noticeable performance drops," but MLP width reduction thresholds are unspecified. (2) How does degradation vary across GenEval sub-tasks? Some sub-tasks (e.g., counting, position) may be more sensitive than others. (3) Is the degradation a smooth function of sparsity or a sharp phase transition? The answer determines whether any amount of static compression is safe. (4) Does the sensitivity vary across the three models (BAGEL, Qwen-Image, Ming-Omni)? Given that understanding component compressibility varies substantially across models (Table 3: BAGEL degrades to 0.63 at 50% while Qwen-Image stays at 0.90), generation component sensitivity likely varies too, but this is untested. This gap makes it impossible to determine whether MoE Adaptation is solving a problem that exists for all unified models or one specific to BAGEL's architecture.

What evidence exists in the paper. Qualitative figures only: Figure 6 (one prompt, one sparsity ratio), Appendix A Figure 9 (one prompt, four depth reduction ratios), Appendix A Figure 10 (one prompt, four attention pruning ratios). No GenEval scores, no FID/CLIP-score/other image quality metrics, no multi-model comparison, no sub-task breakdown. The paper's claim about generation component sensitivity is visually plausible but quantitatively unsubstantiated.

Mitigation status. Not addressed. The paper treats the qualitative examples as sufficient evidence for the claim that generation components cannot be statically compressed, despite having the methodology (GenEval scoring) and infrastructure (all three models loaded) to produce the quantitative characterization. This is a notable asymmetry in the paper's analytical rigor between the understanding and generation components.

Limitation 5: Calibration Data Sensitivity Creates an Unresolved Deployment Tension

The assumption or constraint. The paper demonstrates convincingly that calibration data alignment with the target task is critical for neuron partition quality (Figures 5, 12, 13; Section 5.2). Using understanding-task calibration for generation tasks causes visible image quality degradation; using generation-task calibration for understanding tasks causes lower benchmark scores. The paper suggests test-time few-shot compression as a mitigation: "directly uses a few test samples for calibration" which "seamlessly adapts to downstream tasks and achieves competitive performance" (Appendix E). However, this suggestion is not implemented or evaluated in the context of MoE Adaptation — the Expert Partition stage of MoE Adaptation uses a fixed calibration dataset to determine which neurons become shared vs. routed experts, and this partitioning is a one-time architectural change, not something that can be switched per-task at inference time.

The consequence. In a deployment where the unified model serves both understanding and generation requests (which is the whole point of a unified model), there is a fundamental tension: the Expert Partition for MoE Adaptation must be based on some calibration dataset, but that dataset will inevitably be better aligned with either understanding or generation. If calibrated for generation (to maximize the quality of sparse generation), the understanding component's expert partitioning may be suboptimal for understanding tasks — potentially degrading the very capability the unified model is supposed to preserve. The paper's asymmetric solution for the "Und. & Gen." configuration (Section 5.4: "the experts in the understanding component are kept frozen, remaining fully activated for understanding and only sparsely activated for generation") partially addresses this by keeping the understanding component dense for understanding tasks, but this means there is no efficiency gain for understanding queries — only generation queries benefit from sparsity. The calibration problem thus translates into an efficiency problem: you can have sparse activation for generation OR for understanding, but not both simultaneously without risking quality degradation in the less-calibrated task.

What evidence exists in the paper. Figures 12 and 13 quantify the calibration-task mismatch effect on downstream performance. Table 4 shows that the "Und. & Gen." configuration, which must make a calibration choice for the understanding component's expert partition, achieves GenEval 0.85 (slightly below the 0.86 baseline) while activating 4.96B understanding parameters — but this configuration's understanding-task performance is not reported, so we cannot assess whether understanding quality is preserved. The paper does not evaluate a task-switching deployment scenario where a single MoE-adapted model serves both understanding and generation queries.

Mitigation status. Partially acknowledged but not resolved. The paper notes calibration sensitivity as a finding (Section 5.2) and proposes task-specific calibration as a solution, but does not extend this to the MoE adaptation setting where expert partitioning is a fixed architectural change. The asymmetric activation strategy (dense for understanding, sparse for generation) is a practical workaround but sacrifices efficiency gains on understanding tasks, which may dominate the query mix in many deployments. The paper does not discuss the possibility of task-conditional expert partitioning (different expert groupings for different tasks), which would be a natural extension.

Limitation 6: Static Difficulty Binning and No Dynamic or Continuous Adaptation

The assumption or constraint. The MoE Adaptation pipeline uses a fixed, one-time expert partition based on cumulative importance scores from a calibration dataset (Section 4.2). Once the experts are partitioned and the router is trained, the expert groupings and the routing policy are static — the same shared experts are always active, and the same top-k selection mechanism applies regardless of the input's characteristics (beyond what the router can express through its input-dependent scoring). There is no mechanism for dynamically adjusting the sparsity level based on input difficulty, for reallocating neurons between experts based on deployment-time usage patterns, or for falling back to denser activation when the router is uncertain. This is in contrast to the adaptive, difficulty-conditioned approach described in the reference paper (Section 3.2), where the compute-optimal policy selects different strategies for different difficulty levels.

The consequence. The fixed sparsity ratio of 50% per layer is applied uniformly regardless of input characteristics. However, the paper's own activation analysis (Figure 3) reveals that different inputs and timesteps activate different neuron subsets — some inputs may genuinely require more active neurons than others. A uniform 50% activation ratio may be insufficient for complex prompts (causing quality degradation on hard cases) while being unnecessarily conservative for simple prompts (leaving efficiency on the table). The trained router provides some input-conditional selection within the fixed budget, but the budget itself is not adaptive. This limits the achievable efficiency-quality Pareto frontier: a system that could dial sparsity up or down per-input could achieve better average efficiency at the same quality, or better quality at the same average efficiency.

What evidence exists in the paper. The fixed 50% ratio is stated as a design choice in Section 5.1 without ablation or justification of why this specific value was chosen. The activation pattern analysis in Figure 3 suggests substantial input-dependent variation in neuron utilization, which implicitly argues against a uniform activation budget but this tension is not discussed. No experiments vary the activation ratio or implement adaptive sparsity.

Mitigation status. Not addressed. The paper does not discuss input-adaptive sparsity, does not ablate the activation ratio, and does not suggest dynamic budget allocation as future work. The MoE Adaptation approach as presented is a static-sparsity method (fixed per-layer activation count) with dynamic routing (which experts are active), rather than a fully dynamic method where both the number and identity of active experts vary per input. This is a natural extension that the paper's own analysis motivates but does not pursue.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not propose a new unified architecture or a general compression algorithm. Instead, it makes a more foundational contribution: it establishes that the compressibility of a model component is diagnostic of its functional role and computational dynamics, and that unified multimodal models — precisely because they integrate heterogeneous components — cannot be treated as monolithic targets for efficiency optimization. This is less a paradigm shift than a corrective reframing of how the field should approach efficiency in multimodal systems.

The primary conceptual shift is methodological. Prior work on model compression (pruning, quantization, distillation) treated architectural heterogeneity as an implementation detail — the same techniques were applied uniformly across all layers, with the assumption that transformer layers are transformer layers regardless of what task they serve. This paper demonstrates that this assumption fails catastrophically in unified models: the understanding component tolerates 50% width reduction with minimal generation quality loss (Table 3: Qwen-Image GenEval 0.92 → 0.90), while the generation component cannot survive even moderate static compression without qualitatively obvious degradation (Figure 6, Appendix A Figures 9–10). The implication is that compressibility is not a property of the architecture (transformer, MLP, attention) but of the computational dynamics that architecture supports — autoregressive token decoding creates one compressibility profile, iterative denoising creates another, and they are not interchangeable.

This reframing matters because it redirects research attention. Before this work, a researcher studying efficient multimodal models might reasonably ask: "Which pruning method works best for unified models?" After this work, the question becomes: "How does the compressibility profile differ across components, what mechanisms explain those differences, and how should efficiency strategies be tailored per-component?" The paper's diagnostic methodology — using training-free pruning as a probe rather than a deployment tool — provides a template for answering this question on any new heterogeneous architecture.

The paper resolves a latent tension in the multimodal efficiency literature. Prior work on compressing vision-language models (Lin et al., 2024; Sung et al., 2024) demonstrated that understanding-focused multimodal models could be pruned following similar principles to unimodal LLMs. Naively, one might extrapolate: since unified models contain an understanding component architecturally similar to those compressed VLMs, the same techniques should transfer. The paper shows this extrapolation is half-right: the understanding component is compressible (confirming the VLM compression finding), but the generation component is not (a qualitatively new finding). The tension between "multimodal models are compressible" (the VLM literature) and "generation models are fragile" (implicit in the difficulty of compressing diffusion models) is resolved by recognizing that unified models inherit both properties in different components. The paper's component-wise analysis provides the framework for reasoning about this heterogeneity rather than treating it as a contradiction.

The paper also reframes MoE architectures in a novel role. The dominant narrative around Mixture-of-Experts has been capacity scaling: use MoE to build larger models while keeping per-input FLOPs constant (Shazeer et al., 2017; Fedus et al., 2022; Dai et al., 2024). This paper demonstrates a different use case: retrofitting dynamic sparsity into an already-trained dense model to recover quality that static compression destroyed. The expert partition is not initialized randomly but derived from importance scores on pretrained weights; the routing is not trained from scratch but gradually introduced via zero-initialized gating and expert-frozen warmup. This "compression recovery" framing of MoE is distinct from the capacity-scaling framing and opens a new application domain for MoE techniques: converting any dense model component that resists static pruning into a sparsely-activated equivalent. This is particularly relevant as the field trains increasingly large, expensive models where retraining from scratch with a new architecture is cost-prohibitive, but post-hoc architectural conversion may be feasible.

Research directions that become more attractive:

Component-aware efficiency analysis on any heterogeneous architecture (multi-modal, multi-task, multi-lingual, encoder-decoder). The paper's probing methodology — apply structured removal, observe differential breakage, infer functional organization — is portable.
Dynamic sparsity for iterative refinement models. The finding that diffusion-based generators exhibit sample-dependent and timestep-dependent activation patterns (Figure 3) suggests that video diffusion, audio diffusion, and other iterative generative models may share this property and similarly resist static compression.
Post-hoc MoE conversion as an alternative to training sparse models from scratch, particularly for large pretrained models where the pretraining investment dominates total cost.

Research directions that become less attractive:

Uniform compression recipes for unified models. The paper makes clear that any single sparsity ratio or pruning strategy applied across all components will be either too aggressive for generation or too conservative for understanding.
Depth pruning for multimodal models with autoregressive decoding. The catastrophic collapse of understanding performance under depth reduction (Table 6: MME perception 1684.8 → 304.5 at 50% depth pruning for BAGEL) is sufficiently severe that depth pruning should be considered a non-starter for the understanding components of unified models, at least without architectural modifications to mitigate error accumulation.
Calibration-agnostic pruning. The demonstration that calibration data alignment with the target task is a first-order effect (Figures 5, 12, 13) means that future compression work on multimodal models must report and justify calibration data choices, not treat them as an implementation detail.

Follow-Up Research This Work Enables

1. MoE Adaptation on non-LLM generation architectures (MMDiT, DiT, MAR). The paper validates MoE Adaptation only on BAGEL, whose generation component reuses a Qwen-Instruct LLM backbone. The most urgent follow-up is applying the same pipeline — expert partition by cumulative importance, zero-initialized routing, expert-frozen warmup, full adaptation — to Qwen-Image's MMDiT-based generator (20.42B parameters) and Ming-Omni's multi-scale DiT generator (2.51B parameters). The key open question is whether the dynamic activation patterns observed in BAGEL's generation component (Figure 3) are specific to autoregressive LLM-based generators or are a general property of iterative generative models. If MMDiT/DiT generators exhibit similar sample-dependent neuron activation, the MoE adaptation should transfer; if they show more uniform activation (suggesting less dynamic specialization), static pruning might actually work better for those architectures, and the paper's central claim about generation component sensitivity would need to be qualified as architecture-specific. A strong follow-up would report GenEval scores at multiple sparsity ratios for static pruning of the MMDiT/DiT components (providing the quantitative characterization missing from the current paper), then apply MoE adaptation and compare against both the dense baseline and a dense-finetuning baseline at the equivalent activated parameter budget. Negative results (MoE adaptation fails on MMDiT) would be equally informative, delimiting the boundary conditions of the approach.

2. Input-adaptive sparsity budgets with difficulty-conditioned expert selection. The paper uses a fixed 50% per-layer activation ratio regardless of input characteristics, but Figure 3 demonstrates that neuron activation patterns vary across inputs and timesteps — some inputs likely require more active experts than others. A natural extension is to make the number of selected experts k in the top-k gating mechanism a function of input difficulty, estimated either by the router's confidence (e.g., the gap between the k-th and (k+1)-th expert scores) or by a separate lightweight difficulty predictor. The experiment would: (1) collect per-input GenEval sub-task scores at different activation ratios to characterize the quality-sparsity curve per sample; (2) train a difficulty predictor (small MLP or probing classifier on the router's pre-softmax scores) to estimate which inputs need more experts; (3) implement an adaptive budget allocation where the activation ratio varies per input but averages to a target budget across a batch; and (4) compare against the fixed-ratio baseline at the same average sparsity. The paper's own findings predict that this should improve the quality-efficiency Pareto frontier — simple prompts (e.g., "a red apple") should need fewer active experts than complex prompts (e.g., "three blue spheres stacked on a yellow cube with a green pyramid to the left") — and quantifying that gap would directly inform deployment resource allocation strategies.

3. Task-conditional expert partitioning for unified models serving mixed query streams. The paper identifies calibration data alignment as critical (Section 5.2) but does not resolve the tension that arises in deployment: a unified model serving both understanding and generation queries must choose a single expert partition that may be suboptimal for one task type. A follow-up would implement and evaluate task-conditional routing, where the expert partition depends on the task being performed. Specifically: (1) compute separate importance scores for the understanding component using understanding-task calibration and generation-task calibration; (2) create two sets of expert partitions — one optimized for understanding queries, one for generation queries; (3) at inference time, select the partition based on the query type; (4) measure both task-specific quality and the overhead of maintaining two sets of expert groupings (memory for dual weight organizations, latency of partition switching). A more ambitious variant would train a single set of experts with a task-conditioned router — the router receives a task embedding as additional input and learns to select different experts for understanding vs. generation even within the same expert pool. The paper's Figure 2 (low neuron overlap between tasks) suggests this is feasible because the tasks naturally activate different neuron subsets. The key metric would be whether task-conditional routing achieves better understanding-task performance than the paper's asymmetric strategy (dense for understanding, sparse for generation) while maintaining generation quality.

4. Comparing MoE adaptation against distillation and low-rank approximation for generation component compression. The paper does not compare MoE adaptation against alternative approaches to reducing generation component compute. Three baselines are needed: (1) Distillation: train a smaller dense generation component (same activated parameter count as the MoE configuration) to mimic the full generation component's outputs, using the same training data the MoE adaptation uses; (2) Low-rank approximation: decompose the generation component's MLP weight matrices (W_g, W_u, W_d) via SVD and truncate to achieve the same parameter reduction, with optional fine-tuning of the low-rank factors; (3) Width pruning + longer fine-tuning: take the statically-pruned generation component (which performs poorly in the paper's training-free evaluation) and fine-tune it for the same number of steps as the MoE adaptation receives. The comparison would use identical training data, identical compute budget for adaptation (GPU-hours), and identical activated parameter count at inference. This would establish whether MoE adaptation's dynamic routing provides benefits beyond what simpler methods with equivalent training compute can achieve, or whether the gains come primarily from the additional training rather than the architectural change. A negative result (distillation matches MoE adaptation) would suggest that the paper's diagnostic findings are correct but the therapeutic solution is unnecessarily complex.

5. Understanding the error accumulation mechanism in depth-pruned autoregressive decoders. The paper observes that depth pruning catastrophically degrades understanding performance (Table 6) and attributes this to error accumulation in autoregressive decoding — small per-token deviations compound across the sequence. But the paper does not actually measure this accumulation or characterize its dynamics. A controlled experiment would: (1) depth-prune the understanding component at varying ratios (10%, 20%, 30%, 40%, 50%); (2) for each ratio, measure not just final task accuracy but per-token metrics — the KL divergence between the pruned and unpruned model's next-token distributions at each decoding step, and how this divergence grows with sequence position; (3) determine whether the collapse is gradual (a steady increase in divergence) or abrupt (a phase transition at a particular sequence length); (4) test whether the collapse can be mitigated by techniques that reduce error propagation, such as resetting the key-value cache periodically, increasing the temperature to add stochasticity that breaks repetitive loops, or using nucleus sampling instead of greedy decoding. This would clarify whether depth pruning of autoregressive decoders is fundamentally impossible or merely requires different inference procedures. The paper's qualitative example (Figure 11: the pruned model degenerates into repeating "portraying") suggests a specific failure mode — the model enters an absorbing state in token space — that could be systematically characterized and potentially circumvented.

6. Extending the diagnostic methodology to training dynamics, not just final models. The paper uses training-free pruning as a probe on fully-trained unified models. An interesting extension would be to apply the same probing methodology at multiple checkpoints during pretraining to understand when and how component-wise compressibility emerges during training. Do understanding and generation components develop their distinct compressibility profiles early in training (suggesting they are architecturally determined) or late (suggesting they emerge from task-specific specialization)? Does the neuron overlap between understanding and generation tasks (Figure 2) increase or decrease over training? Such an analysis would inform whether unified models could be trained with built-in sparsity from the start — e.g., if the dynamic activation patterns in generation components emerge early, MoE architectures could be used from the beginning of pretraining rather than retrofitted post-hoc, potentially achieving better quality at the same sparsity. The experiment would require access to intermediate pretraining checkpoints for BAGEL or a similar unified model, and would measure the same metrics (per-neuron importance scores, task-specific neuron overlap, layer redundancy via cosine similarity) at each checkpoint to trace the developmental trajectory of the model's functional organization.

Practical Applications and Downstream Use Cases

On-device or edge deployment of unified multimodal assistants. A unified model that can both answer questions about images and generate images from text prompts is attractive for mobile and edge devices (smartphones, AR/VR headsets, robotics), but the full parameter count of models like BAGEL (15.24B total) is prohibitive for on-device inference with acceptable latency. The paper's finding that the understanding component can be compressed to 50% width with modest quality loss on understanding tasks (Table 2: BAGEL neuron partition at 50% yields MME perception 1392.6 vs. 1684.8 dense, MMBench 79.2 vs. 88.1) while MoE adaptation on the generation component recovers dense-equivalent generation quality with ~65% activation (Table 4: GenEval 0.88 vs. 0.86 dense) suggests a deployment configuration where the understanding component is statically pruned for both task types, and the generation component uses MoE with sparse activation. The total activated parameters would be approximately 4.76B (understanding, 50% pruned) + 4.96B (generation, MoE-adapted) = 9.72B out of 15.24B, or roughly 64% of the original — a meaningful reduction that could make the difference between cloud-only and on-device deployment. The calibration-then-prune workflow is compatible with device-specific optimization: the understanding component could be pruned using calibration data representative of the expected on-device query distribution.

Cost-efficient batch generation pipelines for synthetic data creation. Organizations generating large volumes of images from text prompts for training data, content creation, or data augmentation (e.g., generating product images from descriptions, creating training data for downstream vision models) would benefit from reducing per-image generation cost. The paper's MoE-adapted BAGEL model matches the dense model's GenEval score (0.88 vs. 0.86) while activating only 4.96B of 7.62B generation parameters — roughly a 35% reduction in generation-component FLOPs per image. For a batch of one million images, this translates directly to ~35% lower generation cost. The exact savings depend on the fraction of total inference time spent in the generation component versus the understanding component (which is not MoE-adapted in the "Gen." configuration and runs at full cost), but since image generation typically dominates the compute for high-resolution outputs, the savings would be substantial. A practitioner would perform the MoE adaptation once on their base model using in-domain calibration data (ensuring the expert partition is aligned with their specific image distribution), then deploy the adapted model for batch inference with reduced per-image cost.

Unified model deployment with task-specific compression masks. In a production setting where a single unified model serves both understanding and generation API endpoints, the paper's calibration data sensitivity finding (Section 5.2) suggests a deployment strategy: maintain two copies of the understanding component's weight matrices — one pruned using understanding-task calibration (for the understanding endpoint) and one pruned using generation-task calibration (for the generation endpoint). Since the understanding component is shared across both tasks but activates different neuron subsets for each (Figure 2), task-specific pruning masks could achieve better quality than a single mask at the same sparsity. The storage overhead is only the pruned weight matrices (not the full model), and loading the appropriate weights per request type adds minimal latency relative to model inference time. The paper provides the evidence that this is beneficial (Figures 12, 13: task-aligned calibration outperforms mismatched calibration) but does not implement the switching system. This is a straightforward engineering extension that could immediately improve quality-efficiency tradeoffs in deployed unified models.

When to Prefer This Method

The paper does not explicitly position MoE adaptation against named alternatives (distillation, training a smaller dense model from scratch, low-rank approximation) with head-to-head comparisons. It does, however, establish a clear logical boundary for when the approach is applicable versus when it is not, based on the diagnostic findings:

Prefer MoE adaptation for generation component sparsity when:

The generation component exhibits dynamic, sample-dependent activation patterns (Figure 3) — i.e., different inputs and timesteps activate different neuron subsets. In such cases, static pruning will inevitably delete neurons critical for some inputs, making dynamic routing necessary.
The base model is already pretrained and retraining from scratch is cost-prohibitive. MoE adaptation retrofits sparsity into existing weights rather than requiring a new pretraining run.
The target sparsity is moderate (~60-65% activation rather than ~50% or lower). The paper only demonstrates quality preservation at these activation ratios; more aggressive sparsity is untested and may degrade regardless of routing.
The generation component uses MLP-based transformer blocks where the expert partition operation (Section 4.2) is well-defined. The approach has not been validated on other generation architectures (MMDiT, DiT).

Prefer static width pruning for the understanding component when:

The target tasks are generation-only (the understanding component is used solely as a feature extractor for the generation component). Table 3 shows Qwen-Image maintains GenEval 0.90 at 50% sparsity and 0.82 at 70% — remarkable tolerance that makes MoE adaptation's additional training cost unnecessary.
Calibration data can be aligned with the target task, as calibration misalignment degrades quality (Figures 5, 12, 13).
The target sparsity is moderate rather than aggressive; 25% sparsity preserves near-dense understanding quality (Table 2: MME perception 1558.1 vs. 1684.8 for BAGEL), while 50% sparsity introduces more substantial degradation.

Avoid depth pruning entirely for understanding components in autoregressive decoding settings. The paper's evidence of catastrophic collapse (Table 6, Figure 11) is sufficiently strong that depth pruning should be considered a non-viable strategy for the autoregressive understanding pathway in unified models, unless combined with architectural modifications that interrupt error accumulation (e.g., reset mechanisms, multi-pass decoding with verification).