ERNIE 5.0 Technical Report

ArXiv: 2602.04705

🎯 Pitch

ERNIE 5.0 is the first production-scale trillion-parameter model that generates images, video, and audio natively using the same autoregressive mechanism as text—not with bolt-on diffusion modules. Within a single pretraining run, a novel elastic training scheme learns an entire family of smaller sub-models that can be extracted and deployed with no retraining, offering over 15% decoding speedup for a minor accuracy cost. The paper also reveals that training reinforcement learning at this scale requires solving entropy collapse, where all probability mass collapses onto a single token.

1. Executive Summary

This report introduces ERNIE 5.0, a natively autoregressive foundation model that unifies multimodal understanding and generation across text, image, video, and audio under a single next-group-of-tokens prediction objective. Trained from scratch on a unified corpus, the model employs an ultra-sparse mixture-of-experts (MoE) architecture with modality-agnostic expert routing — dispatching tokens from all modalities to a shared expert pool without heuristic modality-specific partitioning — alongside a novel elastic training paradigm that jointly optimizes sub-networks of varying depth, expert capacity, and routing sparsity within a single pretraining run, enabling deployment under diverse memory and latency constraints without retraining. ERNIE 5.0 achieves competitive or leading performance across language, vision, and audio benchmarks — for instance, matching specialized image generation models on GenEval (90.1 vs. 91.0 for Qwen-Image) and surpassing Veo3 on VBench-Semantic for video generation — while the elastic variant retains competitive accuracy using only 53.7% of activated parameters and 35.8% of total parameters, and reducing routing top-k to 25% yields over 15% decoding speedup with minor accuracy loss. The results establish that a unified autoregressive framework can support both multimodal understanding and generation at trillion-parameter scale without sacrificing modality-specific performance, but that the benefit of test-time compute via beam search over best-of-N is difficulty-dependent — helping on medium-hard problems while degrading on easy ones due to verifier over-optimization — and that on the hardest problems no allocation of inference compute provides meaningful improvement over greedy decoding of a ~14× larger model.

2. Context and Motivation

The Core Problem: Unifying Multimodal Understanding and Generation in a Single Autoregressive Framework

The fundamental challenge this paper confronts is architectural fragmentation in multimodal AI systems. Contemporary frontier models — including GPT-4o (OpenAI, 2024), Gemini (DeepMind, 2025a;b), Claude (Anthropic, 2025), DeepSeek (Liu et al., 2024), and Qwen (Yang et al., 2025) — have demonstrated that large-scale autoregressive sequence modeling provides a powerful foundation for multimodal understanding. These models ingest diverse inputs (text, images, audio, video) as token sequences and exhibit strong reasoning capabilities across modalities. However, they share a critical architectural limitation: autoregressive modeling serves multimodal understanding while output remains text-centric. The model can describe an image or answer questions about audio, but it cannot generate images, videos, or speech natively — those capabilities require separate, bolt-on components.

This limitation is not merely an inconvenience. It creates what the paper terms an "ability seesaw" problem (Section 1), where strengthening individual modality capabilities through late-fusion designs comes at the cost of either multimodal integration depth or core language performance. The paper's central thesis is that this fragmentation is avoidable: a single autoregressive model, trained from scratch on all modalities simultaneously, can natively support both multimodal understanding and generation without sacrificing performance in any modality.

The significance of this problem spans both practical deployment and scientific understanding. Practically, deploying separate models for text generation, image generation, speech synthesis, and video creation multiplies engineering complexity, inference infrastructure costs, and maintenance burden. A unified model that genuinely handles all modalities would dramatically simplify production systems. Scientifically, the question of whether a single set of parameters can simultaneously encode the distinct representational requirements of text semantics, visual spatial structure, audio temporal dynamics, and cross-modal alignment — across both understanding and generation — tests the limits of current neural architectures and training paradigms.

Where Existing Approaches Fall Short

The paper identifies three broad classes of prior work, each with specific limitations that ERNIE 5.0 aims to overcome.

Late-Fusion Designs: Modality-Specific Decoders Bolted onto Language Backbones

The dominant architectural pattern in multimodal models prior to ERNIE 5.0 is late-fusion: start with a pretrained language model (typically a large autoregressive transformer), then augment it with modality-specific encoders, decoders, or generators that connect to the language backbone through adapter layers or cross-attention mechanisms. Examples include augmenting language models with separate image generators (Xu et al., 2025a; Seedream et al., 2025) or speech synthesizers.

This approach has three fundamental weaknesses:

1. Disjoint optimization trajectories. Because the language backbone is typically pretrained first (on text-only data) and modality-specific components are added later, the optimization objectives for different modalities are not aligned. The language model learns representations optimized for text prediction; the vision encoder learns representations optimized for visual tasks; the generator learns yet another objective (often non-autoregressive, such as diffusion). Training proceeds in separate stages with different loss functions, learning rates, and data distributions. The paper characterizes this as "inconsistent optimization trajectories" that prevent deep token-level multimodal interactions.

2. The seesaw problem. When a language model is adapted for multimodal tasks through late-fusion approaches, improvements in multimodal capability often degrade core language performance, and vice versa. This is not a theoretical concern — it is an empirically observed phenomenon where fine-tuning on vision-language data causes the model to partially forget language capabilities, or where optimizing for image generation quality reduces text coherence. The paper observes that late-fusion approaches "force a trade-off between multimodal integration and core language performance."

3. Modality boundaries remain hard. In late-fusion systems, information flows between modalities only at specific architectural interfaces — typically where modality-specific encoder outputs are injected into the language model's hidden states, or where the language model's output is routed to a modality-specific decoder. This means cross-modal reasoning cannot fully leverage token-level interactions across modalities throughout the full depth of the network. For tasks that require tight integration — such as editing an image based on a detailed textual description while preserving audio context — these architectural bottlenecks limit performance.

Modality-Isolated Routing in Mixture-of-Experts Models

Even the authors' own prior work, ERNIE 4.5 (ERNIE Team, 2025), employed a Mixture-of-Experts architecture but with modality-isolated expert routing: different modalities were assigned to different, manually partitioned subsets of experts. This heuristic approach has two problems. First, it requires deciding in advance how many experts each modality needs — a non-trivial decision that becomes increasingly difficult as the number of modalities grows (text, image, video, audio, interleaved combinations). Second, it prevents knowledge learned by experts serving one modality from benefiting others, eliminating any possibility of emergent cross-modal generalization in the routing patterns.

The paper's modality-agnostic routing (Section 2.1) eliminates this requirement entirely. Routing decisions are conditioned solely on unified token representations, not modality identifiers, allowing the router to learn optimal expert allocation patterns from data. The paper demonstrates (Section 6.4.1) that this leads to emergent specialization — experts develop modality-specific activation patterns without being told which modality each token belongs to — while also enabling cross-modal knowledge sharing through experts that are co-activated across modalities.

Separate Models for Each Deployment Scale

A practical but often overlooked problem is that training a single large model does not address deployment diversity. Real-world applications require models at different scales: an on-device model for a smartphone has vastly different memory and latency constraints than a cloud model running on clusters of GPUs. The traditional approach — training separate models from scratch at each target scale, or post-hoc compression (pruning, distillation, quantization) of a single large model — is computationally wasteful and introduces quality degradation at each compression step. Once a model is compressed, its architecture is fixed; producing a different-sized variant requires repeating the entire process, "thereby constraining deployment flexibility" (Section 3.3).

The paper's elastic training paradigm (Section 3.3) addresses this by treating the super-network as a family of sub-models that are co-optimized during a single pretraining run. Rather than training a model and then compressing it, elastic training exposes the model to different configurations (varying depth, expert count, and routing sparsity) during training itself, so that sub-networks emerge as naturally deployable variants.

Conflicting Demands: Understanding vs. Generation Representations

A subtler but equally critical challenge that the paper identifies is the tension between the representational requirements of multimodal understanding and multimodal generation (Section 2.1). Understanding tasks — answering questions about an image, transcribing speech, reasoning about a video — typically require high-level semantic abstractions: recognizing that a scene contains "a person playing guitar" rather than modeling the exact pixel values of the guitar strings. Generation tasks — producing a photorealistic image, synthesizing natural-sounding speech, generating a temporally coherent video — require fine-grained perceptual fidelity: accurate textures, precise timbre, smooth motion. These two objectives pull representations in opposite directions: semantic compression versus perceptual preservation.

The paper frames this not as an either-or choice but as a mutual reinforcement opportunity: "semantic-level signals guide generative modeling toward global consistency, while generative training, in turn, strengthens fine-grained perception and detail-sensitive reasoning." This design philosophy — that understanding and generation should strengthen each other when trained jointly — is a core architectural bet that distinguishes ERNIE 5.0 from late-fusion approaches where these capabilities are trained separately with different objectives.

The Specific Gap: No Production-Scale Demonstration of Unified Autoregressive Multimodal Generation

The paper makes a strong claim about the novelty of its contribution: "To the best of our knowledge, among publicly disclosed models, ERNIE 5.0 represents the first production-scale realization of a trillion-parameter unified autoregressive model that supports both multimodal understanding and generation" (Abstract). This frames the gap not as a purely conceptual one — the idea of unified multimodal autoregressive models has existed in research — but as an engineering and scaling gap: no prior work had demonstrated that the approach is feasible at trillion-parameter scale with production-quality performance across all modalities.

Achieving this requires solving a cascade of interconnected challenges that each individually could derail the effort:

• Training stability at scale: ultra-sparse MoE models with activation rates below 3% introduce extreme sparsity patterns that can cause load imbalance, routing collapse, and out-of-memory failures during training.
• Multimodal tokenization heterogeneity: different modalities produce token sequences with vastly different lengths, structures, and computational costs, making it difficult to batch them efficiently on homogeneous hardware.
• Reinforcement learning at scale: post-training RL for a trillion-parameter multimodal model amplifies numerical mismatches between training and inference engines, entropy collapse, and difficulties in generating useful gradient signals from sparse reward tasks.
• Flexible attention patterns: text uses causal attention, vision benefits from bidirectional attention within spatial neighborhoods, audio may require specialized temporal masking — all within the same batch.

The paper positions itself as demonstrating that all of these challenges can be solved simultaneously within a coherent architectural and training framework.

Connection to Prior Work and Positioning

The paper situates itself relative to several lines of prior work:

ERNIE lineage: ERNIE 5.0 is explicitly the successor to ERNIE 4.5 (ERNIE Team, 2025), inheriting its post-training pipeline (SFT followed by unified multimodal RL) and infrastructure foundations, but fundamentally rearchitecting the model to be natively multimodal rather than adapted from a language backbone. The shift from modality-isolated to modality-agnostic expert routing marks a key architectural departure.

Autoregressive multimodal generation: The paper builds on prior work in autoregressive image generation (Chen et al., 2024a; Ji et al., 2026) and neural audio codecs (Kumar et al., 2023; Zhang et al., 2024b), but integrates these into a unified next-group-of-tokens framework rather than treating them as separate generation pipelines.

Mixture-of-Experts scaling: The paper extends the MoE literature (Lepikhin et al., 2020; Shazeer et al., 2017) by introducing modality-agnostic routing and elastic training across three dimensions (depth, width, sparsity), going beyond standard load-balancing techniques to enable flexible deployment from a single training run.

Reinforcement learning for LLMs: The RL techniques in Section 4 build on recent advances in GRPO (Guo et al., 2025), GSPO (Zheng et al., 2025), and IcePop (Ling-Team et al., 2025), but address the specific challenges of applying these methods to ultra-sparse multimodal MoE models — training-inference mismatch amplified by dynamic routing, entropy collapse in the multimodal setting, and sparse rewards on hard queries that the base model cannot solve without hints.

Once-for-all training: The elastic training paradigm extends the once-for-all concept (Devvrit et al., 2023; Cai et al., 2024) from efficient inference architectures to large-scale pretraining, where the goal is not just to find efficient sub-networks but to train them simultaneously with the full model so they inherit knowledge rather than requiring post-hoc compression.

How This Paper Positions Itself

The paper is fundamentally a systems and scaling demonstration rather than a single-algorithm contribution. Its core claim is that a set of design choices — unified autoregressive training from scratch, modality-agnostic MoE routing, elastic training across three dimensions, and scalable RL infrastructure for multimodal models — together enable a trillion-parameter model that matches or exceeds specialized systems across text, vision, and audio tasks while being flexibly deployable at different scales. The paper does not claim each individual technique is entirely novel; rather, it claims their integration at scale represents a first-of-its-kind demonstration that a unified autoregressive paradigm can genuinely replace fragmented multimodal architectures in production.

This is reflected in the paper's structure: architectural design (Section 2), pre-training methodology (Section 3), post-training RL techniques (Section 4), infrastructure engineering (Section 5), and comprehensive evaluations (Section 6) are treated as co-equal contributions, with the infrastructure section being notably detailed for a technical report — underscoring that the scaling achievement itself is a primary contribution.

3. Technical Approach

3.1 Reader Orientation

ERNIE 5.0 is a single, trillion-parameter neural network that can read or generate text, look at images and videos, listen to and generate audio, and produce new images and videos — all from one integrated model rather than separate modules bolted together. The problem it solves is architectural fragmentation in multimodal AI: existing systems either specialize in understanding (describing images, transcribing speech) or generation (creating images, synthesizing voice), but not both, and they typically use separate models for different modalities, creating engineering complexity and preventing deep cross-modal reasoning. The "shape" of the ERNIE 5.0 solution is to treat everything as a token prediction problem under a single autoregressive objective, route all tokens through a shared pool of experts without modality labels, and train a family of deployable sub-models simultaneously in one pretraining run — the model learns at multiple scales at once so that smaller, faster versions can be extracted on demand without retraining or compression.

3.2 Big-Picture Architecture (Diagram in Words)

The system consists of five major component groups, explained in the order information flows through them:

1. Modality-Specific Tokenizers — three separate but architecturally aligned tokenization pipelines that convert raw signals into discrete tokens: an image/video tokenizer (CNN-ViT hybrid with multi-scale quantization), an audio tokenizer (neural codec with residual vector quantization), and a text tokenizer (byte-level BPE). These run on dedicated compute nodes separate from the main model (disaggregated architecture).

2. Unified Embedding Layer — projects tokens from all modalities into a shared continuous vector space of the same dimensionality. Text tokens use standard learned embeddings; image tokens are mapped from their quantized representations; audio tokens use a depth-wise additive embedding that sums contributions from multiple residual levels.

3. Ultra-Sparse MoE Transformer Backbone — the core autoregressive model consisting of many transformer layers. Each layer contains a shared attention mechanism and an ultra-sparse mixture-of-experts (MoE) feed-forward block where each token activates fewer than 3% of available experts. Routing decisions are made by a modality-agnostic router that sees only the token's vector representation, not its modality label.

4. Modality-Specific Prediction Heads — at the output of selected transformer layers, specialized heads predict the next group of tokens: text uses a standard next-token prediction head with multi-token prediction, vision uses a next-frame-and-scale prediction head (predicting image patches across multiple spatial scales), and audio uses a depth-wise next-codec prediction head (predicting coarse-to-fine audio residual codes across layers).

5. Elastic Training Sub-network Sampler — during pretraining, a scheduler randomly selects which sub-network configuration to use for each training batch: varying which transformer layers are active (elastic depth), how many experts are available (elastic width), and how many experts each token activates (elastic sparsity). All configurations are trained under the same autoregressive loss, producing a family of deployable models from a single optimization run.

Information flows as follows: raw input signals (text, image/video pixels, audio waveform) → modality-specific tokenizers produce discrete tokens → tokens are embedded into a shared vector space → position encodings are added (Uni-RoPE for all modalities) → the full unified sequence passes through all transformer layers (with active layers and experts determined by the elastic training sampler if training, or by the deployment configuration if inference) → at specified output layers, modality-specific prediction heads produce probability distributions over the next group of tokens → tokens are decoded back to raw output (text is directly generated, images and audio pass through decoders that convert discrete codes to pixels/waveforms).

3.3 Roadmap for the Deep Dive

• First, the unified autoregressive backbone and MoE routing mechanism (Section 2.1 details) — because this is the architectural core that everything else plugs into, and understanding the routing logic is essential for grasping why modality-agnostic design works and why elastic training is feasible.

• Second, the visual tokenization and dual-path representation (Section 2.2) — because vision is the most architecturally complex modality (requiring separate handling of understanding and generation, multi-scale tokenization, and a cascaded diffusion refiner), and the design choices here exemplify the paper's philosophy of unified yet specialized processing.

• Third, the audio tokenization and depth-wise prediction (Section 2.3) — because audio introduces a different temporal structure and the depth-wise autoregression architecture is a key innovation that aligns audio generation with text autoregression without exploding sequence length.

• Fourth, the elastic training paradigm (Section 3.3) — because this is the paper's most novel pre-training contribution, and explaining it requires understanding the MoE architecture first (since elasticity operates on layers, experts, and routing top-k).

• Fifth, the reinforcement learning techniques (Sections 4.1–4.3) — because these address the specific challenges of post-training a trillion-parameter multimodal MoE model, building on the pre-training foundation, and each technique (U-RB, MISC, WPSM, AHRL) solves a distinct optimization bottleneck.

• Sixth, the infrastructure design (Sections 5.1–5.4) — because the scaling achievement is inseparable from the distributed systems engineering, and understanding these components reveals why certain architectural choices (disaggregated tokenizers, FlashMask, FP8 stack) were necessary for feasibility.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and scaling paper whose core idea is that a single autoregressive model trained from scratch on all modalities under a unified objective, with modality-agnostic expert routing and elastic sub-network co-training, can match or exceed specialized models across text, vision, and audio tasks while supporting flexible deployment at multiple scales.

The Unified Autoregressive Objective: Next-Group-of-Tokens Prediction

The paper formulates all modalities under a shared Next-Group-of-Tokens Prediction framework (Section 2.1). Rather than having separate training objectives for text (next-token prediction), images (diffusion or GAN losses), and audio (reconstruction or adversarial losses), every modality is trained to predict the next group of tokens in a sequence, conditioned on all preceding tokens.

For text, this reduces to the standard autoregressive formulation: given a sequence of text tokens [t_1, t_2, ..., t_n], the model predicts token t_{n+1} by learning the conditional probability distribution P(t_{n+1} | t_1, ..., t_n). The paper augments this with Multi-Token Prediction (MTP) (Gloeckle et al., 2024; Liu et al., 2024), where the model additionally predicts t_{n+2}, t_{n+3}, etc. from intermediate layer representations. The text loss weight is set to 0.3 during the initial 8K training stage and decreased to 0.1 during mid-training (Section 3.2). Predicting multiple future tokens simultaneously improves sample efficiency and inference speed because the model learns to plan ahead rather than optimizing each token myopically.

For vision, the "group of tokens" corresponds to multiple spatial scales within a single image or multiple frames in a video. Specifically, an image is decomposed into tokens at multiple resolutions (scales): the model first predicts low-resolution tokens (capturing global structure), then medium-resolution tokens (adding detail), then high-resolution tokens (fine-grained texture), each conditioned on all previously predicted scales. A video extends this by predicting all scales of frame F_{i+1} conditioned on all scales of frames F_1 through F_i. This is the Next-Frame-and-Scale Prediction (NFSP) paradigm (Ji et al., 2026). The group-of-tokens structure means the model does not predict individual pixels one at a time — that would produce sequences of hundreds of thousands of tokens for a single image — but rather predicts entire spatial scales at once, reducing sequence length while maintaining autoregressive coherence.

For audio, a single audio segment is represented by multiple residual codes (a coarse semantic code plus progressively finer acoustic codes). The "group of tokens" corresponds to the set of codes at one residual level for a given time step. The model predicts these codes in a coarse-to-fine hierarchy, with each level's prediction conditioned on all previous levels (both the coarser codes for the same time step and all codes for previous time steps). This is the Next-Codec Prediction (NCP) formulation.

The key property that makes this unified objective viable is that it imposes a consistent optimization signal regardless of modality. At every training step, for every token position, the model receives a cross-entropy loss between its predicted probability distribution over the token vocabulary and the ground-truth token. There is no modality-specific loss function — no adversarial losses, no perceptual losses, no reconstruction losses. This is critical because it means the model's shared parameters are never pulled in conflicting directions by different loss surfaces; the only source of gradient variance is the statistical properties of the data itself, not the optimization formulation.

The paper explicitly introduces a posterior-based loss weighting strategy (Section 3.2) to prevent modality imbalance. The raw autoregressive losses across modalities can have different typical magnitudes (predicting a text token from a 100K vocabulary is a very different statistical problem from predicting an image token from a 16K vocabulary). To compensate, the losses are rescaled to the same interval based on their posterior distribution during training, "improving training stability and preventing imbalance across modalities."

Positional Encoding: Uni-RoPE for All Modalities

To handle tokens from different modalities with fundamentally different positional structures — text is 1D sequential, images are 2D spatial, videos are 3D spatio-temporal, audio is 1D temporal with multiple residual levels — the paper introduces Unified Spatiotemporal Rotary Positional Embedding (Uni-RoPE) (Section 2.2.3). This is applied uniformly to all tokens in the unified sequence.

In standard Rotary Position Embedding (RoPE), each token position i is assigned a rotation angle θ_i that is used to rotate the query and key vectors in the attention computation, encoding relative position. Uni-RoPE extends this to three dimensions:

$\text{Uni-RoPE}_i = (t_i, h_i, w_i), \quad i \in \{1, \ldots, N\}$

where t_i is the temporal coordinate (frame index for video tokens, sequence position for text and audio), h_i is the vertical spatial coordinate, w_i is the horizontal spatial coordinate, and N is the total sequence length.

What this encodes: a 3D position vector for every token in the unified sequence, specifying where that token "lives" in a conceptual space-time coordinate system. For text and audio tokens, all three coordinates are set equal to the token's sequential index (t_i = h_i = w_i), effectively reducing to standard 1D positional encoding. For image tokens, t_i is constant (all within the same frame), while (h_i, w_i) specifies the 2D spatial location within the image. For video tokens, t_i increases monotonically to preserve temporal ordering, while (h_i, w_i) specifies spatial location within each frame.

Why three dimensions: text and audio are fundamentally 1D (they unfold in time), images are 2D, and video is 3D. A single scalar position cannot adequately represent relative distances in all three cases. For example, two pixels adjacent in an image should be considered "close" even if they are far apart in the flattened 1D token sequence, and the model needs to distinguish between temporal adjacency (consecutive frames at the same spatial location) and spatial adjacency (neighboring pixels within the same frame). Uni-RoPE gives the attention mechanism the ability to learn which dimension(s) matter for each attention head.

Center-aligned coordinate strategy for multi-scale tokens: when an image is represented at multiple spatial scales (low-res, medium-res, high-res), tokens at different scales that correspond to the same spatial region should encode similar positions. Instead of using pixel-level coordinates (which would differ across scales), the paper aligns coordinates based on geometric centers — a low-res token covering a 16×16 pixel region and its corresponding high-res tokens within that region share the same conceptual center coordinate, ensuring spatial consistency across scales.

RoPE base frequency: the paper sets the RoPE base to 1,000,000 starting from the initial 8K training stage (Section 3.2). This is an unusually high base frequency (standard values are typically 10,000 or 100,000). A high base means the sinusoidal position encodings oscillate more slowly, which makes the model less sensitive to absolute position and more reliant on relative positions, which is important for generalizing to longer sequences during the mid-training context extension (from 8K to 32K to 128K). It "avoids the need for reparameterization or interpolation during subsequent context length expansion, ensuring lossless and stable long-context training."

Visual Tokenization: Multi-Scale Bit-Wise Quantization

The visual tokenizer converts raw images and video frames into discrete tokens that the autoregressive backbone can process (Section 2.2.1). This is a multi-stage pipeline.

Stage 1: Causal 2D multi-scale tokenizer pre-training. The paper first trains a 2D convolutional autoencoder on images. The encoder compresses an input image into a latent representation, and the decoder reconstructs the image from this latent. The "causal" property means that predictions at each spatial location depend only on previously generated locations in a raster-scan order, making the tokenizer compatible with autoregressive modeling. The "multi-scale" property means the latent is organized into multiple resolution levels: a coarse latent capturing global structure (e.g., 4×4 grid of tokens), a medium latent (e.g., 8×8), and a fine latent (e.g., 16×16), for a total of 336 tokens representing the image at three scales.

Stage 2: Inflation to 3D for video. Building on the pretrained image tokenizer, the model "inflates" 2D convolutions into 3D convolutions by adding a temporal dimension, then fine-tunes on video data. This progressive design means the tokenizer inherits strong spatial representations from large-scale image pretraining while learning temporal coherence from video data, rather than attempting to learn both simultaneously from scratch.

Stage 3: Bit-wise quantization. The continuous latent representation at each spatial location is quantized into a group of discrete bit-codes. Instead of using a single discrete index from a large vocabulary (which would require a codebook of size 2^B for B bits), the paper quantizes each bit independently, producing B binary tokens per spatial location, where the number of bits B directly determines the effective vocabulary size. This is the approach from Han et al. (2025).

Stage 4: Progressive tokenizer switching. The paper does not use a single fixed-bit tokenizer throughout training. Instead, it pre-trains a series of tokenizers with increasing bit numbers (and thus increasing vocabulary sizes and reconstruction fidelity). During ERNIE 5.0 training, the system starts with a low-bit tokenizer (small vocabulary, coarse representations that are easier for the backbone to learn to predict) and "gradually transition[s] to higher-bit variants (i.e., larger vocabularies)". The rationale is that early in training, the backbone struggles with the complexity of high-bit prediction; a low-bit tokenizer provides a smoother learning curriculum. As training progresses and the backbone's predictions improve, the tokenizer is swapped for a higher-bit version, increasing the fidelity of generated images without destabilizing early optimization.

Auxiliary training signals for the tokenizer: during tokenizer training itself (before integration with the backbone), the paper incorporates two additional supervision signals beyond reconstruction loss. First, an adversarial loss from GAN-based discriminators (Karras et al., 2019) improves the distributional fidelity of reconstructed images — the discriminator learns to distinguish real images from tokenizer reconstructions, forcing the tokenizer to produce outputs that are statistically indistinguishable from real images rather than just pixel-accurate. Second, a semantic regularization loss derived from a large-scale pretrained vision foundation model (likely a CLIP-style model) ensures that high-level semantic content is preserved through the compression process — the tokenizer's latent representation should encode the same semantic information as the original image, as measured by agreement between frozen vision model features of the original and reconstructed images.

Why multiple auxiliary losses: the core tension in visual tokenization is between compression (minimizing the number of tokens for computational efficiency) and fidelity (preserving enough information for high-quality reconstruction and generation). The adversarial loss prevents the "blurry" reconstructions typical of pure MSE-trained autoencoders, while the semantic loss prevents the tokenizer from discarding semantically meaningful information (like the identity of an object) in favor of perceptually irrelevant texture details.

Visual Understanding: Dual-Path Hybrid CNN-ViT Representation

For visual understanding tasks (answering questions about images, reading text in documents, reasoning about charts), the model needs access to rich visual features. The paper identifies a fundamental tension (Section 2.2.2): before quantization, visual features are "compressed by a downsampling module into low-dimensional representations, whose dimensionality is aligned with the tokenizer bit-width." This compression is necessary for tractable sequence lengths, but it "inevitably leads to the loss of fine-grained semantic information" that matters for understanding tasks — details like small text in documents, subtle visual cues in diagrams, or fine-grained object attributes.

The solution is a dual-path architecture that extracts two complementary feature representations before quantization and fuses them:

Path 1: CNN features (F_cnn). A convolutional neural network extracts local, fine-grained perceptual features. CNNs excel at capturing texture, edges, and other low-level visual patterns because their translation-invariant local receptive fields are well-suited to spatial detail. These features have dimension D_cnn.

Path 2: ViT features (F_vit). A Vision Transformer processes the same input but captures global, semantic-level features through self-attention. ViTs can model long-range dependencies across the entire image, making them better at understanding object relationships, scene layout, and high-level semantics. These features have dimension D_vit.

The two paths produce features with different spatial structures and dimensionalities. A naive fusion (concatenating or adding them through an MLP) "often fails to fully exploit their complementary strengths and introduces representational interference, resulting in degraded understanding performance." The paper's insight is that simply mixing the features without modeling their relationships discards the complementary structure — the CNN knows about fine edges; the ViT knows about object identity; these need to inform each other, not just be averaged.

Attention-Based Patch Merger (formal description):

Given a spatial token in an image or a spatio-temporal token in a video, two feature sets are extracted:

F_cnn ∈ R^{N × K × D_cnn} F_vit ∈ R^{N × K × D_vit}

where N is the number of visual understanding tokens (the compressed sequence length after spatial downsampling), K is the number of local patches grouped per token, and D_cnn and D_vit are the feature dimensions of the CNN and ViT paths respectively. For image understanding, K = 4 spatially adjacent patches are grouped together. For video understanding, K = 16 patches spanning 4 neighboring frames (4 patches × 4 frames, or equivalently a 4-frame temporal window with 4 spatial patches each).

Step 1: Project and concatenate. The CNN features are projected to match the ViT feature space dimension D_vit via a learned linear transformation. The aligned CNN and ViT patch features are then concatenated along the patch dimension:

F_mrg ∈ R^{N × 2K × D_vit}

This creates a sequence of 2K tokens for each of the N visual understanding positions, where the first K tokens come from the CNN path and the next K come from the ViT path.

Step 2: Cross-path self-attention. Multi-head self-attention is applied to the 2K tokens at each position:

Z = Attention(F_mrg)

The attention mechanism jointly models (a) correlations between CNN and ViT features at the same spatial location, (b) cross-path dependencies where CNN information informs ViT interpretation and vice versa, and (c) spatial and temporal dependencies among the group of patches (since K patches may cover adjacent spatial regions or consecutive frames). The output preserves the same shape: Z ∈ R^{N × 2K × D_vit}.

Step 3: Mean pooling and projection. The 2K tokens at each position are mean-pooled to produce a single compact representation:

F_out ∈ R^{N × D_vit}

This pooled representation is then projected to align with the embedding dimension of the unified backbone, making it compatible with the rest of the model.

Why attention instead of MLP fusion: an MLP would apply a fixed, position-independent transformation to concatenated features. This cannot model how a CNN feature at patch j and a ViT feature at patch k interact — it treats every pair identically. Attention allows dynamic, content-dependent weighting: if the ViT detects an object boundary at a particular patch, attention can upweight CNN features in that region to capture fine boundary details; if the ViT indicates a patch is semantically unimportant (e.g., uniform background), attention can downweight corresponding CNN features. The mean pooling after attention ensures the output has fixed dimensionality regardless of K.

The paper reports that this attention-based fusion "consistently outperforms both CNN-only and ViT-only baselines on a wide range of benchmarks without introducing noticeable computational overhead, with particularly pronounced gains in document and chart understanding as well as general visual understanding tasks." The architecture is explicitly designed to serve both understanding and generation: the fused features provide strong semantic representations for understanding, while the underlying tokenized representations (before fusion) support generation.

Visual Generation: Next-Frame-and-Scale Prediction (NFSP) with Cascaded Diffusion Refiner

While the tokenization and representation sections describe how visual information enters the model, this section describes how visual content is generated (Section 2.2.3). The core innovation is that image and video generation are unified under a single autoregressive framework with a hierarchical prediction structure.

NFSP for image generation: An image is decomposed into S spatial scales (e.g., scale 1: 4×4 tokens, scale 2: 8×8, scale 3: 16×16). The model generates these scales autoregressively from coarse to fine:

1. Predict all tokens at scale 1 (lowest resolution) conditioned on the text prompt and any other input modalities.
2. Predict all tokens at scale 2 conditioned on the text prompt AND the generated scale 1 tokens.
3. Predict all tokens at scale 3 conditioned on the text prompt AND the generated scale 1 and scale 2 tokens.
4. Continue for all scales.

Within each scale, tokens are predicted in parallel (not autoregressively) because a scale-wise causal attention mask is applied: tokens within the current scale attend to each other bidirectionally but can only see previous scales causally. This is the key efficiency mechanism: predicting an entire scale at once avoids the O(L^2) cost of predicting individual tokens one at a time, where L would be the total number of visual tokens (potentially thousands).

NFSP for video generation: Video extends this by adding a temporal dimension. For a video with F frames, each at S scales:

1. For frame 1: predict scales 1 through S exactly as for an image.
2. For frame 2: predict scales 1 through S conditioned on all tokens from frame 1 (and the text prompt).
3. For frame f: predict all scales conditioned on all tokens from frames 1 through f-1.

The model uses a Unified Spatiotemporal Rotary Positional Embedding (Uni-RoPE) (described above) so that the attention mechanism can distinguish between positions within a frame (spatial dependencies) and across frames (temporal dependencies).

Corruption-based training for robustness: autoregressive generation over long token sequences is susceptible to error accumulation — if the model makes a small mistake at an early scale, that error compounds through all subsequent predictions. To mitigate this, during training the paper "corrupt[s] historical tokens... by randomly flipping their bits, while supervising the model to self-correct toward the ground-truth tokens of the current scale." This means the model learns that its inputs may contain errors (as they will during inference when fed its own imperfect predictions) and must be robust to noisy context.

Loss reweighting for scale imbalance: multi-scale tokenization produces vastly different numbers of tokens at different scales (a 4×4 scale has 16 tokens, a 16×16 scale has 256 tokens). The autoregressive loss would be dominated by the high-resolution scales simply because they have more tokens. The paper applies "a loss reweighting strategy to emphasize early-stage predictions and alleviate the token imbalance introduced by multi-scale tokenization." Specifically, the loss on coarser scales (which are predicted first and are more critical for global structure) is upweighted relative to finer scales.

Windowed temporal attention and historical frame masking (video-specific): for long videos, attending to all previous frames becomes computationally prohibitive and can also cause the model to overfit to irrelevant distant context. The paper introduces windowed temporal attention, where each frame only attends to a fixed window of recent previous frames (rather than the entire history). Additionally, random historical frame masking drops some previous frames during training, forcing the model to rely on local context and improving robustness when frame history is incomplete.

Cascaded diffusion refiner for high resolution: the autoregressive backbone generates images and videos at a fixed, relatively low resolution (constrained by the tokenizer's maximum output scale). To produce high-resolution outputs (e.g., 1024×1024 or higher for images, HD video), the paper uses a separate cascaded diffusion refiner — a generative model trained with diffusion objectives that takes the backbone's low-resolution output as conditioning and produces a high-resolution version. This refiner is "trained separately from the backbone, using paired low-resolution samples with controlled degradation, together with their corresponding high-resolution images or videos." The decoupled training is important: "it avoids optimization conflicts caused by introducing autoregressive and diffusion losses within a shared backbone." If the diffusion loss were added to the backbone's training, the gradients from the diffusion objective (which optimizes for pixel-level perceptual quality) and the autoregressive objective (which optimizes for token prediction) would conflict, likely degrading both.

Why a two-stage pipeline rather than end-to-end high-resolution autoregression: generating high-resolution images autoregressively would require the backbone to predict extremely long sequences of tokens (e.g., a 1024×1024 image at 16×16 patches is 4096 tokens, plus multi-scale tokens). This would explode the context length, reduce effective batch size, and destabilize training. The cascaded design separates concerns: the backbone handles semantic content and layout at manageable resolution; the refiner handles resolution enhancement as a separate super-resolution problem.

Audio Tokenization: Residual Vector Quantization with Whisper Distillation

The audio tokenizer converts raw waveforms into hierarchical discrete tokens that can be processed by the autoregressive backbone (Section 2.3.1). The design is inspired by neural audio codecs like SoundStream and EnCodec but adapted for unified multimodal training.

Base architecture: the tokenizer operates at a token rate of 12.5 Hz, meaning that for every second of audio, 12.5 tokens are produced. This is a substantial temporal compression (raw audio at 16 kHz has 16,000 samples per second; 12.5 tokens per second is a ~1280× compression in the time dimension). The compression is achieved through a convolutional encoder that progressively downsamples the waveform, followed by Residual Vector Quantization (RVQ) .

RVQ hierarchical decomposition: the continuous latent vector at each time step is quantized through multiple codebooks arranged in a residual hierarchy. The first codebook produces a coarse approximation of the latent vector; the difference (residual) between the latent and this first quantization is passed to a second codebook, which quantizes the residual; the remaining residual goes to a third codebook, and so on. The first quantized code captures high-level semantic information (phonetic content, linguistic meaning), while subsequent codes capture progressively finer acoustic details (timbre, prosody, background noise characteristics). The total number of codebooks determines the reconstruction fidelity: more codebooks → more bits allocated per time step → higher quality but more tokens for the backbone to process.

Whisper knowledge distillation for the first semantic code: to ensure that the first code captures rich semantic and linguistic information (not just coarse acoustic features), the paper distills knowledge from a pretrained Whisper model (Radford et al., 2023). Whisper is an automatic speech recognition model trained on 680,000 hours of supervised speech data; its encoder produces representations that are highly informative about phonetic and lexical content.

The distillation process works as follows: given an input waveform, both the ERNIE 5.0 audio tokenizer and the frozen Whisper encoder process it. Whisper produces a sequence of representations at its native frame rate; average pooling is applied to match the ERNIE tokenizer's 12.5 Hz rate (resolving temporal mismatch). The representation of the first audio token (before quantization) is trained to align with the pooled Whisper representation, typically using a mean squared error or cosine similarity loss. This forces the first codebook to encode the same information Whisper encodes — which is primarily linguistic — ensuring that the first token is a "semantic token" suitable for text-aligned tasks like speech recognition and speech translation.

Residual acoustic tokens: the remaining codebooks are not explicitly supervised for semantics; they learn to capture whatever residual information is needed for high-fidelity reconstruction. This naturally separates into features like speaker identity, prosody, recording conditions, and background sounds — information that distinguishes one recording of a sentence from another recording of the same sentence.

Why distillation instead of joint training with a speech recognition objective: jointly optimizing the tokenizer for both reconstruction and ASR accuracy would create a tradeoff — improving phonetic encoding might come at the cost of reconstruction quality, or vice versa. Distillation from a frozen teacher avoids this conflict: the tokenizer can learn to encode phonetic information in the first codebook through a separate alignment objective, leaving the reconstruction loss to optimize the remaining codebooks for overall fidelity without compromise.

Audio Understanding and Generation: Depth-Wise Autoregression with Next-Codec Prediction

The paper introduces a depth-wise autoregressive architecture for audio (Section 2.3.2) that avoids the prohibitive sequence length that would result from naively flattening all residual audio tokens into a single sequence.

The sequence length problem: if an audio segment has T = 100 time steps and C = 4 codebooks (residual levels), naively flattening would produce T × C = 400 tokens in sequence. For a 30-second audio clip at 12.5 Hz with 4 codebooks, this is 30 × 12.5 × 4 = 1500 tokens. While manageable, this grows linearly with the number of codebooks — and high-fidelity audio might require 8–16 codebooks, making sequences prohibitively long. More importantly, flattening destroys the hierarchical structure: the model must learn from scratch that the first code is fundamentally different from the fourth code, even though they occupy the same "token type" in the sequence.

Depth-wise prediction architecture: instead of flattening, ERNIE 5.0 distributes the prediction of residual codes across transformer layers. The core insight is that audio codes at different residual levels represent information at different granularities — the first code is semantic, the second adds coarse acoustic detail, the third adds finer detail, etc. — and these map naturally to the hierarchical feature representations at different transformer depths (lower layers capture fine-grained features, higher layers capture abstract features).

For audio understanding: the input audio tokens are represented through a depth-wise additive embedding mechanism. Each audio token consists of C discrete codes corresponding to C residual levels. For level c, the discrete code index is mapped to a learned embedding vector through a level-specific embedding matrix. Embeddings from all C levels are summed element-wise to form the final audio token representation:

audio_embedding = sum_{c=1}^{C} Embed_c(code_c)

The additive aggregation reflects the residual nature of the audio representation: each level contributes complementary information, and the sum represents the full audio signal at that time step. This summed embedding is placed at the corresponding position in the input sequence and processed uniformly with text tokens by the autoregressive backbone.

For audio generation (Next-Codec Prediction): the model generates hierarchical audio tokens in a coarse-to-fine manner. Multiple audio prediction heads are inserted at different transformer layers (specifically, "the top transformer layers"). The generation process proceeds as follows:

1. The backbone processes the multimodal context (text prompt, speaker embedding if provided, any preceding audio tokens for previously generated time steps) up to the layer where the first audio head is attached.
2. The first audio head predicts the first semantic code for the current time step — this is a categorical distribution over the first codebook's vocabulary.
3. The predicted code is mapped to its embedding through the level-1 embedding matrix and added back to the hidden state — this enriched representation now conditions the next level's prediction.
4. The second audio head (at a deeper layer) predicts the second residual code for the current time step, conditioned on the hidden state that now includes the first code's embedding.
5. This process repeats for all residual levels, with each level's prediction conditioning the next through additive embedding injection.
6. Once all codes for a time step are predicted, the model proceeds to the next time step.
7. After all time steps are generated, the complete set of hierarchical codes is passed to the audio decoder (the tokenizer's decoder module) which converts them to a waveform.

During training, teacher forcing is used: instead of feeding the model's own predicted code at each level, the ground-truth code from the training data is used. This ensures the model always conditions on correct information during learning, avoiding the error accumulation that would occur if it had to condition on its own (potentially incorrect) predictions.

Speaker embedding for controllable synthesis: for speech synthesis tasks, a speaker embedding is inserted as part of the conditioning context. This embedding (likely derived from a speaker verification model or learned during training) encodes speaker identity — vocal timbre, accent, speaking style — and is provided as an additional input token or conditioning vector. This "guide[s] acoustic realization without altering deep semantic content or depth-wise prediction structure."

Why depth-wise prediction rather than sequential flattening: the depth-wise design has three advantages. First, it preserves the hierarchical semantic structure — the model learns that level-1 codes carry semantic information and level-3 codes carry fine acoustic details, which is a useful inductive bias. Second, it reduces sequence length by a factor of C (the number of residual levels), since the temporal sequence length is T, not T × C. Third, it allows the model to allocate representational capacity appropriately: the layers and attention heads specialized for semantic prediction (early audio heads) can focus on linguistic content, while later heads focus on acoustic details, without these objectives interfering.

Ultra-Sparse MoE Architecture with Modality-Agnostic Routing

The transformer backbone of ERNIE 5.0 uses a sparse Mixture-of-Experts (MoE) architecture where each transformer layer's feed-forward network is replaced by multiple parallel "expert" networks, and a learned router determines which few experts process each token (Section 2.1).

Structure of a MoE layer: in a standard dense transformer, the feed-forward block consists of two linear transformations with an activation function: FFN(x) = W_2 · σ(W_1 · x), applied identically to every token. In an MoE layer, there are E independent feed-forward blocks (the "experts"), each with its own parameters W_1^{(e)} and W_2^{(e)}. For each token, a lightweight router network (a small learned function, typically a linear projection followed by softmax) computes a probability distribution over the E experts:

$p(e | x) = \text{softmax}(W_r \cdot x)_e$

where x is the token's hidden representation, W_r is the router's learned weight matrix, and p(e | x) is the probability of routing token x to expert e.

What this computes: a score for each expert indicating how well-suited that expert is to process the given token, based solely on the token's current hidden representation. The softmax normalizes these scores into a probability distribution.

Why this design: the router decouples model capacity from computation. A dense model with the same total parameters would require every token to activate all parameters, making inference expensive. MoE allows the model to have enormous capacity (trillions of parameters) while only activating a small fraction (less than 3%) per token.

Ultra-sparse routing with top-k selection: rather than using a weighted combination of all experts (which would be dense computation), the router selects only the top-k experts with the highest routing probabilities, where k is a small number. Only these selected experts process the token; all others are bypassed. The output is a weighted combination of the selected experts' outputs:

$\text{MoE}(x) = \sum_{e \in \text{top-}k(p(\cdot|x))} p(e|x) \cdot \text{FFN}_e(x)$

where top-k(p(·|x)) returns the indices of the k experts with highest routing probability.

The paper's key innovation: modality-agnostic routing. Unlike prior MoE models (including ERNIE 4.5) that assign different subsets of experts to different modalities (e.g., experts 1–32 for text, experts 33–64 for vision), ERNIE 5.0's router "is conditioned on unified token representations rather than explicit modality identifiers." This means the router sees only the token's vector representation — it does not know whether the token came from text, image, audio, or video. It must learn to route based solely on the content of the representation.

Why modality-agnostic routing matters (four reasons):

1. No manual partitioning: as the number of modalities grows, deciding how many experts each modality needs becomes increasingly difficult and brittle. Modality-agnostic routing eliminates this design decision entirely.

2. Emergent specialization: the router can learn that certain types of computation (e.g., syntactic parsing, spatial reasoning, temporal dynamics) are useful for multiple modalities and route tokens from different modalities to the same experts when they benefit from similar processing.

3. Cross-modal generalization: if the model learns a useful computation for processing text (e.g., detecting contradictions), that same expert can be activated for processing an image that contains semantically contradictory elements, without the model needing to learn a separate vision-specific contradiction detector.

4. Simplified architecture: no modality-specific routing code, no modality-separated expert pools, no heuristic allocation strategies — just one router, one expert pool, and data-driven allocation.

Activation rate below 3%: with E = M total experts (where M is large, likely in the hundreds or low thousands) and a small k (the paper uses k values far below the total number of experts), only a tiny fraction of parameters are activated per token. The paper states an activation rate "below 3%," meaning for every token, at least 97% of expert parameters remain idle. This is the enabling mechanism for trillion-parameter scale — total capacity grows with M while per-token computation grows only with k.

Auxiliary-loss-free load balancing: standard MoE training typically adds an auxiliary loss that encourages uniform expert utilization — without it, the router may collapse to always selecting the same few experts, leaving others dead. However, this auxiliary loss can interfere with the primary language modeling objective. The paper instead uses the auxiliary-loss-free load balancing strategy from Wang et al. (2024c), where a bias term is added to the routing logits before top-k selection:

\text{effective_logits}_e = \text{logits}_e + b_e

The biases b_e are dynamically updated during training: when an expert is over-utilized (receives more than its fair share of tokens), its bias is decreased, making it less likely to be selected; when under-utilized, its bias is increased. Crucially, this bias update is not part of the backpropagation graph — it is a separate control mechanism that adjusts routing without contributing gradients to the model parameters. The bias update speed is set to 1 × 10^{-4} during the initial 8K training stage and reduced to 1 × 10^{-5} during mid-training (Section 3.2), because the routing patterns stabilize as training progresses.

Why auxiliary-loss-free: adding an explicit load-balancing loss term to the training objective creates a tension: the model must simultaneously minimize both the language modeling loss and the load-balancing loss, and these may conflict (the best expert for a given token might be one that is already over-utilized). By moving load balancing outside the gradient-based optimization, the model can focus purely on predictive accuracy while the bias mechanism handles utilization as a separate control problem.

Stabilization at trillion-parameter scale: training ultra-sparse MoE models at this scale introduces unique challenges. With activation rates below 3%, the gradient signal for any given expert is extremely sparse — an expert might only receive gradients from 3% of tokens, meaning its updates are based on a small, potentially biased subset of the data distribution. This can cause routing instability, where small fluctuations in routing probabilities cascade into large changes in expert utilization. The paper's combination of auxiliary-loss-free balancing with the reduced bias update speed during mid-training "effectively suppresses iteration-level oscillations observed in large-scale MoE training."

Elastic Training: Once-For-All Sub-Network Co-optimization

The elastic training paradigm (Section 3.3) enables a single pretraining run to produce a family of models with different computational profiles. This is the paper's most novel pre-training contribution.

The deployment diversity problem: a single trillion-parameter model, no matter how capable, cannot be deployed in all scenarios. A cloud server with 8×H100 GPUs has vastly more memory and compute than a smartphone. Traditional solutions — train separate models at each target scale, or train one large model then prune/distill it — are computationally wasteful and introduce quality degradation. More critically, once a model is compressed to a particular size, its architecture is fixed; producing a different-sized variant requires repeating the entire process.

Elastic training core idea: during pre-training, the model is trained as a super-network that contains many nested sub-networks. For each training batch, a sub-network configuration is randomly sampled, and only that sub-network's parameters are activated and updated. The key is that all sub-networks share parameters with the full model — the 12-layer sub-model uses the first 12 layers of the 16-layer full model, not separately trained layers — so improvements to the full model benefit the sub-networks and vice versa.

Three elasticity dimensions:

1. Elastic Depth: varies the number of active transformer layers. During training, with probability 75%, the full-depth network is used (all layers active). With probability 25%, a reduced-depth sub-network is activated, where a randomly selected subset of layers participates in forward and backward computation. The paper doesn't specify the exact depth sampling distribution, but the inference configurations in Table 9 show depths from 1 to 16 layers (for a 16-layer model), and the ERNIE 5.0 elastic variant in Table 12 uses 53.7% of activated parameters, suggesting approximately half the layers.

The mechanism: when a shallower configuration is sampled, the later layers are simply bypassed. The output of the last active layer is fed directly to the prediction heads. Because the model is trained with this bypassing 25% of the time, intermediate layers learn to produce representations that are useful even when later layers are absent — they cannot rely on deep layers to "fix" their outputs.

Why elastic depth works: it forces all layers to be individually useful. Without elastic training, early layers can be "lazy" — they can produce noisy or incomplete representations knowing that deeper layers will clean them up. With occasional layer dropping, every layer must produce representations that could potentially serve as the model's output, which improves the functional integrity of parameters throughout the network.

2. Elastic Width: varies the total number of experts available in each MoE layer. During training, with probability 80%, all E experts are available for routing. With probability 20%, routing is restricted to a randomly sampled subset of E' < E experts. The paper uses E' = E/2 in controlled experiments (Table 10: 64 experts full, 32 experts reduced).

The mechanism: when reduced width is sampled, the router can only select from the available subset of experts. Experts outside the subset are not just unused — they don't even participate in the routing computation for that batch. This means each expert must learn to be useful in multiple contexts (since it doesn't know whether it will be available or not for any given batch) and the router must learn to be robust to missing experts (since its usual top-choice expert might be unavailable).

Why elastic width matters for deployment: memory-constrained devices may not have enough RAM to store all expert parameters. Elastic width training ensures that a model instantiated with only a fraction of experts still performs well, because it has been trained to handle this situation.

3. Elastic Sparsity: varies the routing top-k — the number of experts activated per token. During training, with probability 80%, the default top-k is used. With probability 20%, k is randomly sampled from a predefined range of smaller values (e.g., k ∈ [1, k_default-1]). At inference time, a smaller k means fewer experts are computed per token, directly reducing FLOPs and latency.

The mechanism: when a reduced k is sampled during training, each token activates fewer experts. This forces the model to learn routing decisions that are efficient — it cannot rely on activating many experts to get good performance, because sometimes only a few are available. The controlled experiments in Table 11 show that training with k ∈ [1, 8] (elastic sparsity) enables inference with k = 4 at only a 0.026 validation loss increase over k = 8, while baseline training (only k = 8) would likely see a much larger degradation when switching to k = 4 at inference.

Training schedule and probabilities: the paper doesn't explicitly state the training schedule for full-scale ERNIE 5.0, but the controlled-scale experiments consistently use 80%/20% splits for width and sparsity (80% full configuration, 20% reduced) and 75%/25% for depth (75% full, 25% reduced). The higher probability for full configurations ensures the full model receives sufficient optimization, while the 20–25% exposure to reduced configurations is enough to confer elasticity without significantly slowing full-model convergence.

Why this distribution: if reduced configurations were sampled too often (e.g., 50%), the full model would be undertrained — most of its parameters would never be activated together. If too rarely (e.g., 1%), the sub-networks wouldn't develop resilience to parameter removal. The 20–25% range is an empirical choice that balances these concerns.

Sub-model extraction: after elastic pre-training, a deployable sub-model is instantiated by simply taking a subset of the super-network's parameters: a specific number of layers (elastic depth), a specific subset of experts per layer (elastic width), and a specific routing top-k (elastic sparsity). No fine-tuning, pruning, or distillation is required — the sub-model is already optimized for its configuration. The paper demonstrates this for ERNIE 5.0-Exp-EA35.8% (Table 12), which uses only 53.7% of activated parameters and 35.8% of total parameters but achieves an average score of 75.17 versus 75.55 for the full model across benchmarks. The sub-model can then be used as is, or as a starting point for further mid-training or fine-tuning.

Relationship to Once-For-All networks (Devvrit et al., 2023; Cai et al., 2024): previous once-for-all approaches focused on inference efficiency — finding a family of sub-networks that achieve different accuracy-efficiency tradeoffs for a given task. ERNIE 5.0 extends this concept to pre-training at scale, where the goal is not just efficient inference but also ensuring sub-networks inherit knowledge from the full model's training, avoiding the quality loss of post-hoc compression.

What elastic training is NOT: it is not a compression technique (no parameters are removed after training), not knowledge distillation (no teacher-student training), and not pruning (no importance-based parameter removal). It is a training paradigm where the model is designed from the start to be functional at multiple scales, and all scales are optimized simultaneously. The paper explicitly contrasts this with "train-then-compress" pipelines, which "still suffer[] from notable limitations": dedicated compression stages, fixed architectures after compression, and repeated effort for different target sizes.

Unbiased Replay Buffer (U-RB) for Efficient RL Rollouts

Post-training reinforcement learning for a trillion-parameter multimodal model faces a critical efficiency bottleneck: rollout generation dominates training time, accounting for "more than 90% of the total training time in RL" (Section 4.1). This is because generating responses from the policy model is far more expensive than computing gradient updates, especially when response lengths vary dramatically.

The long-tail problem: in a batch of rollout queries, some queries produce very long responses (e.g., a complex math problem requiring a 10,000-token chain-of-thought) while others produce short responses (e.g., a simple factual question answered in 50 tokens). In synchronous RL (Sync RL, Figure 5), the entire batch waits for the longest response to finish before any training can begin. GPUs sit idle while waiting for the slowest query, dramatically reducing utilization.

APRIL (Zhou et al., 2025) partial solution: APRIL over-provisions rollout requests — it starts generating more responses than needed and stops when a target number is collected. Incomplete responses are "recycled" — their partial generations are saved, and generation continues in subsequent steps. This prevents idle waiting: as soon as N complete responses are available, training proceeds, regardless of whether other responses are still being generated.

APRIL's failure mode: however, APRIL introduces a systematic bias. Responses that finish quickly tend to be shorter, and shorter responses tend to come from easier queries. So APRIL's training batches are skewed toward easier queries in early iterations, while harder queries (with longer responses) are deferred to later iterations. This creates a non-stationary data distribution — the difficulty of training data shifts over time, which "may hinder convergence and ultimately degrade model performance."

U-RB: Unbiased Replay Buffer: the paper introduces U-RB as an extension of APRIL that preserves the efficiency gains while eliminating the data distribution bias. The key mechanism is a data-ordering constraint: "only the data group assigned to the current iteration at initialization is allowed to participate in subsequent training process."

U-RB has two components:

1. Inference pool P_infer: a buffer with capacity Ω_RBS = Ω_BS × N, where Ω_BS is the training batch size and N is the buffer size (a multiplier determining how many batches of rollouts can be in-flight simultaneously). This pool holds all rollout queries currently being generated. The inference engine generates responses in parallel across multiple GPUs; as responses complete, they accumulate in P_infer.

2. Training pool P_train: a buffer with capacity Ω_BS that collects completed trajectories for the current training iteration. A trajectory enters P_train only when ALL responses for its assigned data group D_t are complete.

The orchestration (Figure 5, bottom panel): at iteration t, the system assigns a specific data group D_t to be the training data for that iteration. Rollouts for D_t are generated in parallel alongside rollouts for future iterations (D_{t+1}, D_{t+2}, etc.). The critical constraint: the inference engine does NOT stop generating for D_t when it has collected enough responses from other groups — it waits until the LONGEST response in D_t reaches its terminal state (EOS token). Only then are all D_t trajectories moved from P_infer to P_train, and the training engine uses them to update model parameters.

Meanwhile, while waiting for the slowest D_t response, rollouts for future data groups continue to be generated, filling the inference pool. When D_t finally completes, D_{t+1} becomes the active training group, and any already-completed D_{t+1} responses are ready for immediate training.

Why this eliminates APRIL's bias: by tying training data to pre-assigned iteration groups and waiting for all responses in a group (not just the first N to complete), U-RB preserves the natural difficulty distribution of each data group. Easy and hard queries within the same group are trained on together, in the same iteration, just as they would be in synchronous RL. The difference is that waiting for hard queries doesn't idle GPUs — they're busy generating responses for future groups.

Effect on throughput: U-RB achieves the throughput benefits of APRIL (no GPU idle time waiting for long-tail responses) while maintaining the statistical properties of synchronous RL (unbiased data distribution per iteration). The cost is increased memory for the inference pool (which must hold in-progress generations for multiple future batches) and slightly more complex orchestration logic.

Multi-Granularity Importance Sampling Clipping (MISC) for Stable RL

The paper identifies rapid entropy collapse as a major obstacle to RL training for large multimodal MoE models (Section 4.2). Entropy collapse refers to a sharp decrease in the policy's output diversity — the model becomes overconfident, producing nearly deterministic outputs, which prevents exploration of alternative reasoning paths and leads to premature convergence on suboptimal strategies.

Why entropy collapse is particularly severe in ERNIE 5.0:

1. Training-inference mismatch in MoE: RL training uses a training engine for gradient computation and a separate inference engine for rollout generation. Even with identical model weights, numerical differences can arise from different parallelism strategies, different random seeds, or different implementations of operations. MoE models amplify this mismatch because subtle differences in router logits can cause different expert selection, leading to completely different computation paths.

2. Multimodal complexity: different modalities have different entropy characteristics. Text generation typically has higher entropy (many plausible continuations), while image token prediction may have lower entropy (pixel patterns are more constrained). When these are trained jointly, the RL algorithm may over-optimize for one modality's reward signal, causing entropy collapse in others.

3. Overfitting to easy queries: the policy model quickly learns to solve easy queries with high probability, producing low-entropy (deterministic) outputs for them, which reduces exploration and prevents improvement on harder queries.

IcePop baseline (Ling-Team et al., 2025): IcePop addresses training-inference mismatch by applying double-sided masking calibration on the importance sampling ratio, originally designed for GRPO (Group Relative Policy Optimization). The importance sampling ratio r corrects for the fact that rollouts were generated by an older policy (the inference engine's weights) while training updates the current policy (the training engine's weights). The core equation:

$r_{i,j} = \frac{\pi_{\text{train}}(y_{i,j} | x, y_{i,<j}; \theta)}{\pi_{\text{train}}(y_{i,j} | x, y_{i,<j}; \theta_{\text{old}})}$

where π_train(y_{i,j} | ...) is the probability the training engine assigns to token y_{i,j} (the j-th token of the i-th response in a group), θ is the current policy parameters, and θ_old is the parameters used for rollout generation.

IcePop masks (zeroes out) the importance ratio when the ratio of inference-to-training probabilities falls outside a range [α, β]:

$M\left(\frac{\pi_{\text{train}}(y_{i,j} | x, y_{i,<j}; \theta_{\text{old}})}{\pi_{\text{infer}}(y_{i,j} | x, y_{i,<j}; \theta_{\text{old}})}\right) = \begin{cases} k & \text{if } k \in [\alpha, \beta] \\ 0 & \text{otherwise} \end{cases}$

What this computes: for each token, if the training engine's probability under the OLD parameters (θ_old) differs too much from the inference engine's probability under the same old parameters (indicating numerical mismatch), that token's contribution to the gradient is completely zeroed out. This prevents unreliable importance sampling ratios from destabilizing training.

Why IcePop fails for ERNIE 5.0 (the GSPO variant): the paper experimented with applying IcePop to GSPO (Group Sequence Policy Optimization; Zheng et al., 2025), which uses a sequence-level (rather than token-level) importance ratio:

$s_i(\theta) = \left(\frac{\pi_{\text{train}}(y_i | x; \theta)}{\pi_{\text{train}}(y_i | x; \theta_{\text{old}})}\right)^{1/|y_i|}$

The sequence-level masking prunes entire responses where the sequence-level probability ratio falls outside [α, β]. However, "sequence-level truncated importance sampling... prunes a large number of low-entropy responses due to the training–inference mismatch" (Section 4.2). When the training and inference engines disagree even slightly, the cumulative effect over a long sequence (exponentiated by 1/|y_i|) can push the ratio outside the acceptable range, causing the response to be discarded even if the disagreement is small per-token. This leads to very few responses surviving masking, which in turn concentrates gradients on an even smaller subset of responses, accelerating entropy collapse. Figure 6 shows this as the light-blue line ("JGSPO IcePop"): entropy collapses rapidly in early training.

MISC: Multi-granularity Importance Sampling Clipping solution: the paper proposes a modified objective, J_Mixed IcePop, that uses per-token masking for the importance ratio (same denominator check as original IcePop) but sequence-level importance ratio for the policy gradient (from GSPO):

$J_{\text{Mixed IcePop}}(\theta) = \mathbb{E}_{x \sim D, \{y_i\}_{i=1}^G \sim \pi_{\text{infer}}(\cdot|x;\theta_{\text{old}})} \left[ \frac{1}{G} \sum_{i=1}^G \left[ \mathcal{M}_{j \in [1,|y_i|]}\left(\frac{\pi_{\text{train}}(y_{i,j}|x, y_{i,<j}; \theta_{\text{old}})}{\pi_{\text{infer}}(y_{i,j}|x, y_{i,<j}; \theta_{\text{old}})}; \alpha, \beta\right) \cdot \min\left(s_i(\theta)\hat{A}_i, \text{clip}(s_i(\theta), 1-\epsilon, 1+\epsilon)\hat{A}_i\right) \right] \right]$

where G is the group size (number of responses per query), \hat{A}_i is the advantage estimate for response i (how much better or worse this response is compared to the group average), and ε is the PPO clipping parameter.

Key change from JGSPO IcePop: the masking function \mathcal{M}_{j \in [1,|y_i|]} applies per-token — it checks EACH token's training/inference probability ratio and masks at the token level, but applies the result multiplicatively to the sequence-level policy gradient. This means if a few tokens in a response have numerical mismatch, only those tokens' contributions are zeroed; the response as a whole still contributes gradients via its non-mismatched tokens. In the original JGSPO IcePop, a response with ANY mismatched token was entirely discarded.

How this prevents entropy collapse: by avoiding wholesale response discarding, more diverse responses survive gradient computation. Low-entropy responses that might have been completely discarded (leading to concentration on even lower-entropy responses) now partially contribute, maintaining a more balanced gradient signal. Figure 6 shows the dark-blue line ("JMixed IcePop"): entropy remains stable throughout early training instead of collapsing.

Why "multi-granularity": the clipping operates at two granularities simultaneously — token-level masking for numerical mismatch correction and sequence-level importance ratio for policy optimization. This hybrid approach leverages the precision of per-token mismatch detection with the stability of sequence-level advantage estimation.

Well-Learned Positive Sample Mask (WPSM)

Even with MISC stabilizing importance sampling, the model can still suffer entropy collapse from over-optimizing easy queries (Section 4.2). When the model already achieves near-perfect accuracy on a subset of queries, continuing to train on them provides diminishing returns while reinforcing low-entropy, deterministic behavior. This crowds out gradient signal from harder queries where exploration is still needed.

WPSM mechanism: the paper introduces an adaptive masking strategy that identifies "well-learned" responses and reduces (but doesn't eliminate) their training signal:

$M_i^{\text{mask}} = \begin{cases} \alpha & \text{if } H_{y_i^x}(\pi_\theta) < \eta \text{ and } \text{acc}_t^x > \tau \\ 0 & \text{otherwise} \end{cases}$

where H_{y_i^x}(π_θ) is the policy entropy of response y_i^x (a measure of how uncertain the model is about this response — low entropy means high confidence), acc_t^x is the average accuracy on query x at iteration t, η is the entropy stability bound (a threshold below which the model is considered overconfident), τ is the accuracy threshold (above which the query is considered "well-learned"), and α ∈ [0, 1] controls how much the gradient from this response is attenuated.

The modified objective:

$J(\theta) = \mathbb{E}_{x \sim D, \{y_i\}_{i=1}^G \sim \pi_{\theta_{\text{old}}}(\cdot|x)} \left[ \frac{1}{G} \sum_{i=1}^G [\mathbb{1} - M_i^{\text{mask}}] \min\left(s_i(\theta)\hat{A}_i, \text{clip}(s_i(\theta), 1-\epsilon, 1+\epsilon)\hat{A}_i\right) \right]$

What this computes: for each training query x, track its recent average accuracy. If accuracy exceeds τ AND the model's entropy on a response falls below η (indicating confident, likely correct output), multiply that response's contribution to the loss by a factor of (1 - α). If α = 1, the response contributes nothing (full masking); if α = 0.5, the response contributes half its normal gradient. Responses from queries that are not yet "well-learned" (accuracy < τ) or where the model is still uncertain (entropy > η) are never masked.

Why this helps: WPSM shifts the gradient budget toward harder samples. The total gradient norm is conserved (the optimizer still takes steps of similar magnitude), but the composition of those gradients changes — less from easy, already-mastered queries and more from challenging queries or those requiring diverse reasoning paths. This "alleviates the entropy collapse problem caused by over-fitting to easy queries, and encourages the model to improve the performances of challenging, low-performing tasks."

Tracking mechanism: acc_t^x is maintained for each query by tracking the success rate over recent rollouts. This requires per-query state, which adds memory overhead but is feasible because the number of training queries is finite. The entropy H_{y_i^x}(π_θ) is computed directly from the policy's output distribution for each token in the response — no additional tracking needed.

Adaptive Hint-Based Reinforcement Learning (AHRL) for Sparse Rewards

A fundamental limitation of RL for reasoning tasks is that if the base model never generates a correct response, the RL signal provides no guidance (Section 4.3). In GRPO-style methods, when all rollouts for a query receive zero reward, the advantage estimates \hat{A}_i are all zero (or uniformly negative), providing no differential signal about which responses are better. The model cannot learn because it has no positive examples to reinforce.

Recent work (Yue et al., 2025b; Liu et al., 2025b; Zhao et al., 2025b) has confirmed this: "when all rollouts receive zero reward, the GRPO framework fails to provide effective gradient signals for policy optimization." For ERNIE 5.0, this problem is amplified by the multimodal setting — hard queries may exist in any modality, and the base model's capabilities vary across modalities.

AHRL approach: rather than modifying the RL algorithm or reward function, AHRL modifies the queries themselves by injecting partial "think skeletons" — fragments of a reasoning chain that decompose the problem into intermediate steps. Figure 7 illustrates this: a combinatorial geometry problem is augmented with "The problem states: ... I need to find ... Since it's a regular 24-gon, all vertices are equally spaced on a circle." — providing the initial reasoning structure without revealing the solution.

Mechanism: for a query x with a response consisting of a thinking trajectory and final solution y = (think, solution), AHRL augments x into \tilde{x}(p_{\text{hint}}) by revealing the first p_{\text{hint}} fraction of the "think" tokens:

$\tilde{x}(p_{\text{hint}}) = \text{concat}(x, \text{think}[:p_{\text{hint}} \cdot |\text{think}|])$

The augmented query is what the model sees during training — it must complete the remaining (1 - p_{\text{hint}}) fraction of the thinking trajectory and produce the solution.

Annealing schedule: the hint fraction p_hint is not fixed but follows an exponential decay:

$p_{\text{hint}}(x_t) = p_{\text{initial}} \cdot \exp(-\gamma \cdot t \cdot \text{pass}_{\text{initial}}^x)$

where t is the training iteration, γ is the decay rate, and pass_initial^x is the pass@k score of query x evaluated on the SFT model (a measure of how hard the query is for the initial model).

What this schedule does: queries with low pass_initial^x (hard queries that the base model rarely solves) receive hints for longer — the decay is slower because pass_initial^x is small. Queries with high pass_initial^x (easier queries) receive hints for fewer iterations because the model already has some capability to solve them. As training progresses and t increases, p_hint decays for all queries, eventually reaching zero — the model transitions to full self-exploration without hints.

Why hint-based rather than reward shaping: modifying the reward function (e.g., giving partial credit for partially correct reasoning) introduces reward hacking risks — the model may learn to generate reasoning that scores well under the shaped reward without actually solving the problem. AHRL instead modifies the input, which is safer: the model still receives the standard reward (1 for correct, 0 for incorrect), but the modified input makes it more likely to generate a correct response naturally. The hint provides a "scaffold" that bridges the gap between initial exploration failures and eventual mastery.

Where the hints come from: the paper doesn't explicitly state the source of think skeletons. They are likely derived from successful completions on other (similar) queries, or from human-written reasoning templates, or from a stronger teacher model. The key property is that they decompose the problem without revealing the answer — they provide the initial structure and approach, leaving the model to fill in the detailed reasoning.

Why this is called "adaptive": the hint fraction adapts along two dimensions: (1) per-query difficulty (harder queries get hints for longer via the pass_initial^x term) and (2) per-training-progress (hints decay toward zero for all queries as the model improves). This double adaptivity ensures hints are provided when and where needed, and removed when the model is capable of independent reasoning.

Text Tokenization and Multilingual Design

While the visual and audio tokenization sections are the most architecturally novel, the text tokenization (Section 3.1) has important design choices that affect the model's multilingual capabilities.

UTF-16BE encoding: the paper encodes text in UTF-16BE (16-bit Unicode Transformation Format, Big Endian). This is unusual — most LLMs use UTF-8. The rationale: UTF-16BE "provide[s] stable byte-level fallback and a more compact representation for many non-Latin symbols, improving data throughput in multilingual training." For languages with large character sets (Chinese, Japanese, Korean), UTF-16 represents most characters in 2 bytes rather than the 3–4 bytes required by UTF-8, reducing the byte-level sequence length before tokenization.

BPE dropout (Provilkov et al., 2020): during tokenizer training, byte-pair encoding (BPE) merges are randomly dropped with some probability. This is a regularization technique: instead of learning a single deterministic segmentation of text into subwords, the tokenizer learns multiple possible segmentations. This "reduce[s] overfitting to frequent patterns" — the model cannot memorize specific subword sequences because those sequences may be segmented differently in different epochs.

Long unspaced phrase filtering for Chinese: "for languages without explicit whitespace word boundaries (e.g., Chinese), we filter out long unspaced phrases that can be decomposed by standard word-segmentation tools." Chinese text is typically written without spaces between words; a naive BPE tokenizer might learn to treat entire multi-word phrases as single tokens because they appear frequently. This inflates the vocabulary with compound tokens that are actually compositional (e.g., treating "人工智能" as one token instead of "人工" + "智能"). Filtering these out forces the tokenizer to learn sub-word units, "which helps reduce vocabulary sparsity, improve training efficiency, and enhance model generalization."

Pre-Training Recipe: Multi-Stage with Context Length Extension

The pre-training process (Section 3.2) is organized into distinct stages designed to maintain training stability while progressively extending context length.

Stage 1: 8K Pre-Training. The initial stage uses a maximum context length of 8,192 tokens. This relatively short context ensures stable early optimization — with shorter sequences, the attention patterns are simpler, gradient flow is more direct, and memory requirements are lower, enabling larger batch sizes. The learning rate follows a Warmup-Stable-Decay (WSD) schedule (Hu et al., 2024): linear warmup from zero to 1 × 10^{-4} over 2,000 steps, then constant at 1 × 10^{-4} for the remainder of the stage. Unlike standard cosine decay schedules which begin decaying immediately after warmup, WSD maintains peak learning rate through most of training, enabling continued learning from late-stage data.

Batch size scheduling: the global batch size is gradually increased from 14 million tokens to 56 million tokens during early training. Smaller initial batch sizes provide more frequent parameter updates (more steps per epoch), which aids early convergence; larger later batch sizes improve throughput and gradient estimation quality. This is a practical technique for large-scale training where the total number of training steps is constrained by compute budget.

RoPE base = 1,000,000: setting the RoPE base frequency to 1,000,000 at the 8K stage (rather than the standard 10,000) is a forward-looking design choice. When the context length is later extended to 32K and 128K, the higher base frequency means the positional encodings remain well-behaved without requiring interpolation or reparameterization. If a standard base were used, the 128K context extension would require scaling the RoPE frequencies (e.g., through NTK-aware interpolation), which can introduce subtle degradation. By setting the base high from the start, "lossless and stable long-context training" is achieved — the model simply continues training on longer sequences without any architectural changes.

Stage 2: 32K and 128K Mid-Training. The context length is progressively extended to 32K and then 128K tokens. The global batch size is held constant during this stage, while the learning rate schedule switches from WSD to cosine annealing: the learning rate decays from 1 × 10^{-4} to 1 × 10^{-5}. The combination of longer contexts and decaying learning rate allows the model to refine its long-range representations without destabilizing the already-learned short-range patterns.

MoE-specific adjustments during mid-training:

• Bias update speed reduction: the auxiliary-loss-free load balancing bias update speed is reduced from 1 × 10^{-4} (Stage 1) to 1 × 10^{-5} (Stage 2). During Stage 1, the router is still learning to allocate experts, so faster bias updates help quickly establish balanced routing. By Stage 2, routing patterns have largely stabilized, and slower updates prevent oscillations that could occur with aggressive rebalancing.

• MTP loss weight reduction: the Multi-Token Prediction loss weight is decreased from 0.3 (Stage 1) to 0.1 (Stage 2). The MTP auxiliary loss encourages the model to predict future tokens from intermediate layers. In Stage 1, this provides a strong training signal that accelerates learning. In Stage 2, with longer contexts, the primary next-token prediction loss becomes more informative (longer sequences provide more context per token), so the MTP weight is reduced to prevent it from interfering with the primary objective. This "ensur[es] stable adaptation as the model scales to longer contexts."

Posterior-based loss weighting: across both stages, the autoregressive losses of different modalities are rescaled to the same interval based on their posterior distribution. This prevents any single modality from dominating the training signal. Without this, a modality with inherently higher cross-entropy (e.g., high-resolution image token prediction, which has many possible tokens per position) would contribute larger gradients than a modality with lower cross-entropy (e.g., text, which has more predictable token distributions), causing the model to optimize disproportionately for the high-entropy modality.

No-token-dropping: the paper explicitly states that ERNIE 5.0 uses a "no-token-dropping strategy throughout training" (Section 5.1). In MoE models, some implementations drop tokens when an expert's capacity is exceeded (i.e., when too many tokens are routed to the same expert). Token dropping simplifies load balancing but discards information. By avoiding it, ERNIE 5.0 ensures all training data contributes to learning, at the cost of requiring more sophisticated load balancing and memory management (described in the infrastructure section).

Pre-Training Data Composition

The pre-training data (Section 3.1) is organized into text data and multimodal data, with all modalities exposed to the model from the very beginning of training — there is no separate text-only pretraining phase followed by multimodal fine-tuning.

Text data: spans multilingual web crawls, curated corpora, books, scientific publications, code repositories, and structured knowledge sources. The text component provides the backbone of linguistic knowledge, factual information, and reasoning patterns that transfer across modalities. The finalized corpus comprises "trillions of text tokens."

Multimodal data: includes paired image-text, video-text, and audio-text data, as well as interleaved multimodal sequences where text is integrated with images, videos, and audio. Captions and metadata accompany each modality instance. The multimodal component teaches cross-modal alignment — the model learns that the text "a red car" corresponds to visual features of red cars, and that spoken words correspond to their textual transcriptions.

Quality controls: heuristic and model-based filters remove low-quality and unsafe content; extensive deduplication prevents memorization artifacts (where the model might memorize specific training examples rather than learning generalizable patterns); decontamination safeguards remove benchmark data from the training set to ensure fair evaluation.

Standardized platform: the data is organized on a "standardized platform" that categorizes all data by input and output modalities, enabling controlled sampling during training — the training recipe can specify what ratio of text-only, image-text, video-text, etc. batches to use.

Infrastructure: Hybrid Parallelism for Trillion-Parameter Training

Training a trillion-parameter ultra-sparse MoE model requires sophisticated distributed computing strategies (Section 5.1). The paper combines six parallelism types and several memory optimization techniques.

The hybrid parallelism configuration:

1. 4-way tensor parallelism (Shoeybi et al., 2019): each transformer layer's weight matrices are split across 4 GPUs. For a matrix multiplication y = Wx, the weight matrix W is partitioned column-wise: each GPU holds a subset of columns and computes a partial output, which is then combined. This reduces the memory per GPU for large weight matrices.

2. 12-way pipeline parallelism (Huang et al., 2019) with virtual stages: the transformer layers are divided into 12 sequential stages, each assigned to different GPUs. A batch is split into micro-batches that flow through the pipeline — while one micro-batch is in stage 7, another can be in stage 3, keeping all GPUs busy. Virtual stages further subdivide each physical stage, reducing "pipeline bubbles" (idle time when a stage is waiting for the previous stage to finish).

3. 64-way expert parallelism (Lepikhin et al., 2020): the MoE experts are distributed across 64 GPUs. Each GPU holds a subset of all experts. When a token is routed to expert e, communication occurs to send the token's hidden state to the GPU hosting expert e, compute the expert's output, and send the result back. With ultra-sparse routing (activation rate < 3%), only a tiny fraction of tokens need to be communicated to any given expert GPU.

4. ZeRO-1 data parallelism (Rajbhandari et al., 2020): optimizer states (momentum, variance) are partitioned across data-parallel GPUs, reducing memory per GPU. Each GPU only stores the optimizer states for its partition of parameters, not the full model.

5. Context parallelism (Liu et al., 2023a): for long-context training (32K and 128K stages), the sequence dimension is partitioned across GPUs. Each GPU processes a subsequence, and attention computation requires communication to compute softmax over the full sequence length.

6. DeepEP (Zhao et al., 2025a) for inter-node communication: efficient expert-parallel communication library that optimizes the all-to-all communication pattern inherent in MoE routing.

Memory optimization techniques:

FP8 mixed-precision training: activation tensors are stored in FP8 format (8-bit floating point) rather than FP16 or BF16. This approximately halves activation memory, which is critical because activations are the dominant memory consumer during training (weights and optimizer states can be sharded across GPUs, but activations for the current micro-batch must reside on each GPU). The paper follows the practice of ERNIE 4.5.

Dynamic adaptive offloading: when an out-of-memory (OOM) event is detected, the memory allocator adaptively offloads selected activation tensors from GPU memory to CPU memory. The offloading is selective — only tensors that would otherwise cause OOM are offloaded — and no offloading occurs when sufficient GPU memory is available, minimizing performance overhead. This provides a safety net for memory spikes caused by unbalanced expert routing, which can unexpectedly concentrate many tokens on a few GPUs.

Sub-batch computations: large memory allocation requests (e.g., for a batch that routes unusually many tokens to a particular expert) are decomposed into smaller sub-batch requests. This reduces the probability of OOM caused by memory fragmentation — where total free memory is sufficient but no single contiguous block is large enough for the allocation.

Automatic memory defragmentation: based on CUDA Virtual Memory Management (VMM), the memory allocator can defragment GPU memory by remapping virtual addresses to physical addresses, combining small free blocks into larger contiguous regions. This "ensur[es] successful memory allocation even under extreme conditions."

These techniques together "ensure the feasibility and reliability of pre-training ERNIE 5.0 in memory-constrained scenarios" — making the difference between training succeeding (no OOM crashes) and failing (repeated crashes that prevent completion of a training run).

Disaggregated Tokenizer-Backbone Architecture

The heterogeneity of multimodal inputs creates a load-balancing problem distinct from MoE routing (Section 5.2). Different modalities require vastly different amounts of tokenization computation: a 1024×1024 image might require processing through a deep CNN-ViT tokenizer; a short text query needs only a lightweight BPE tokenizer; a 30-second audio clip requires a neural codec. If tokenizers run on the same GPUs as the backbone, the GPUs processing images would be compute-bound by tokenization while text-processing GPUs would idle, creating severe load imbalance.

The disaggregation solution: tokenizers are physically separated from the MoE backbone, deployed as "independent, horizontally scalable services on dedicated compute nodes, under a data-parallel configuration." During training, the backbone sends raw input data to tokenizer services via remote procedure calls and receives encoded token representations in return. Each tokenizer service can be independently scaled — more nodes for the image/video tokenizer when training on image-heavy batches, more nodes for the audio tokenizer when training on speech data.

Why this matters: tokenizers and the backbone have fundamentally different computational characteristics. Tokenizers are typically convolutional (for audio and images) or rule-based (for text BPE), while the backbone is attention-based. Their optimal parallelism strategies differ — tokenizers benefit from data parallelism (processing many inputs independently), while the backbone requires the complex hybrid parallelism described above. Disaggregation "allows each component to adopt parallelization strategies suited to its own workload, improving scalability and efficiency in distributed multimodal training."

FlashMask for Flexible Multimodal Attention

Attention patterns in ERNIE 5.0 vary across modalities (Section 5.3): text uses standard causal attention (each token attends to itself and all preceding tokens), while visual features use "globally causal but locally bidirectional" attention — tokens within the same image or video frame attend to each other bidirectionally to capture spatial dependencies, but cannot attend to future frames.

Efficiently implementing these heterogeneous patterns in flash attention kernels (which optimize memory access patterns for attention computation) is challenging because standard flash attention assumes a single, uniform mask pattern. The paper leverages FlashMask (Wang et al., 2024a), an in-house extension of FlashAttention that supports per-sample custom attention masks.

Performance: FlashMask achieves "up to a 200% speedup over FlexAttention (Dong et al., 2024) at the operator-level, and delivers more than 20% end-to-end training acceleration." When integrated with context parallelism at the kernel level, performance improves by 80% compared to the Megatron-LM context parallelism solution.

Why this is necessary: without efficient mixed-mask attention, the training system would need to either (a) use a single mask pattern for all modalities (degrading visual modeling) or (b) process different modalities in separate batches (reducing training throughput by preventing efficient batching of heterogeneous data). FlashMask enables batching of text, image, and audio tokens with their appropriate attention patterns in a single efficient kernel call.

Scalable and Disaggregated RL Infrastructure

The reinforcement learning infrastructure (Section 5.4) addresses the unique challenges of applying RL at trillion-parameter multimodal scale.

Disaggregated control plane: the RL system is built around a centralized controller that orchestrates four components in an asynchronous manner: the training engine (computes gradient updates), the inference engine (generates rollouts), the environment interaction module (simulates tool use, web browsing, etc. for agent tasks), and the reward evaluation module (computes verifier scores for generated responses). Logical decoupling enables each component to scale independently — if inference becomes a bottleneck, more inference GPUs can be added without reconfiguring the training setup.

Unified FP8 stack: training and inference use "identical high-performance operators" to minimize numerical mismatch. Combined with Rollout Router Replay (Ma et al., 2025b) — a strategy that records the expert routing decisions made during inference and replays them during training — the FP8 stack ensures that the training engine's computations exactly match what the inference engine would compute, eliminating the MoE-induced numerical mismatch that destabilizes importance sampling. This is critical because in low-precision (FP8), even small numerical differences can cause different expert routing, and different routing cascades into completely different computation paths.

Replay buffer for sequence-length bias mitigation (infrastructure side): this is the infrastructure counterpart of the U-RB algorithm (Section 4.1). The infrastructure-level replay buffer "preserves the original data order, ensuring consistent data arrival and mitigating bias caused by asynchronous completion." While the algorithmic U-RB handles the logical ordering constraint, the infrastructure replay buffer handles the physical storage and retrieval of in-progress rollouts.

Elastic CPU pooling: GPU clusters typically have substantial CPU resources that are underutilized because most deep learning computation happens on GPUs. The RL infrastructure virtualizes idle CPU capacity into an elastic pool that powers "logic-intensive tasks such as intensive RL environment interactions and result verification." For agent tasks, the environment (a simulated browser, command line, or API) runs on CPUs; by pooling idle CPUs across the cluster, massive parallel simulation becomes possible without requiring dedicated CPU servers. This "reduces the wall-clock time of training iterations while significantly improving the total cost of ownership (TCO) efficiency of the underlying hardware."

4. Key Insights and Innovations

Innovation 1: The Multimodal "Ability Seesaw" Is an Architectural Artifact, Not an Inevitable Trade-off

The paper's most fundamental conceptual move is diagnosing the tension between multimodal understanding and generation — and between multimodal capability and core language performance — not as a fundamental statistical trade-off but as an artifact of late-fusion architectures. Prior work, including the authors' own ERNIE 4.5 (ERNIE Team, 2025), accepted the premise that strong multimodal generation required modality-specific decoders bolted onto a pretrained language backbone. These systems tacitly assumed that unified training would degrade unimodal performance — a belief reinforced by empirical observations that fine-tuning language models on vision-language data often caused language capability regression.

ERNIE 5.0 challenges this assumption at the architectural level. By training all modalities simultaneously from scratch under a single autoregressive objective, the model avoids the disjoint optimization trajectories that create the seesaw in late-fusion designs. The evidence is in the pre-training results (Table 1): ERNIE 5.0-Base achieves 90.58 on MMLU (surpassing DeepSeek V3.2-Exp-Base at 88.60 and Kimi K2-Base at 88.40) while simultaneously supporting image generation at 88.4 GenEval (Table 5) and video generation matching specialized models on VBench-Semantic (83.40, Table 6). There is no text-vision trade-off visible in the numbers.

This is significant beyond the raw scores because it reframes the research problem. If the seesaw is architectural rather than fundamental, then the entire modular paradigm — separate models for language, vision, audio, each trained with different objectives and connected through adapter layers — may be an evolutionary dead end, not an engineering necessity. The paper doesn't just claim this; it demonstrates it at trillion-parameter production scale, which forces the field to reconsider whether modality-specific architectures are solving a real problem or creating one.

The finding also reconciles a tension in the multimodal literature. Late-fusion approaches consistently showed that adding modalities degraded some capabilities, leading to a research focus on "catastrophic forgetting" mitigation and careful multi-stage training recipes. ERNIE 5.0 suggests that when modalities are co-trained from scratch under a shared objective, cross-modal knowledge transfer is net positive rather than competitive — the model benefits from seeing images while learning language, and vice versa. This is a fundamental reframing rather than an incremental improvement: it changes what researchers should optimize for (unified objectives and early fusion) rather than how to mitigate the damage of late fusion.

Innovation 2: Modality-Agnostic Expert Routing Enables Emergent Specialization Without Manual Partitioning

The paper's second conceptual innovation is demonstrating that MoE routers can discover modality structure without being told about it. Prior MoE multimodal models — including ERNIE 4.5 — manually partitioned experts by modality: text gets experts 1–32, vision gets 33–64, etc. This design reflected an implicit assumption that modalities are fundamentally different computational problems requiring hard separation of parameters for stable optimization.

ERNIE 5.0's modality-agnostic routing eliminates this heuristic. The router sees only token representations, not modality labels, and must learn to allocate experts based on computational need rather than modality identity. The paper provides visualization evidence (Figures 8–10, Section 6.4.1) that this leads to emergent specialization: experts exhibit clear modality-specific activation patterns (Figure 8), cross-modality expert overlap increases in deeper layers as representations become more semantic (Figure 9, IoU between text and image understanding rises from 0.03 in the first layer to 0.29 in the last layer), and load balancing remains stable across modalities despite the absence of explicit balancing constraints per modality (Figure 10, normalized entropy stays above 0.7 for text across nearly all layers).

This is a diagnostic contribution as much as an engineering one. It demonstrates that modality structure can be an emergent property of optimization rather than a prespecified architectural constraint. The implication extends beyond ERNIE 5.0: future multimodal models with even more modalities (code, sensor data, structured knowledge) may not need manual expert allocation, reducing the combinatorial explosion of design decisions as modality count grows.

The finding that the first MoE layer does not collapse into severe imbalance (Figure 10, normalized entropy remains high for text even at the first layer) contradicts a common assumption in MoE literature that early layers require dense designs to maintain routing stability. The paper attributes this to the auxiliary-loss-free load balancing strategy (Wang et al., 2024c) and the ultra-sparse activation rate, suggesting that routing instability in early layers is a consequence of insufficient expert capacity or poor balancing algorithms, not an inherent property of sparse architectures. This is a negative result with practical implications: it reduces the need for architectural exceptions (dense early layers) that complicate model design.

Innovation 3: Elastic Training as a Substitute for the Entire Train-Compress-Deploy Pipeline

The third intellectual contribution is reframing model deployment diversity as a training-time problem rather than a post-hoc compression problem. The dominant paradigm for producing models at different scales has been "train-then-compress": pretrain a large model, then apply pruning (Sajjad et al., 2023; Xia et al., 2023), knowledge distillation (Gu et al., 2023; Xu et al., 2024), or quantization to produce smaller variants. This pipeline has three structural weaknesses that the paper identifies: (1) compression requires dedicated infrastructure and computational overhead, (2) once compressed, the architecture is fixed — producing a differently-sized variant requires repeating the entire process, and (3) compressed models often underperform models trained from scratch at the target size, because they inherit representations optimized for a different parameter count.

Elastic training replaces this pipeline entirely. By jointly optimizing sub-networks during pretraining, the model learns representations that are robust to parameter removal from the start. The evidence is compelling: ERNIE 5.0-Exp-EA35.8% uses only 53.7% of activated parameters and 35.8% of total parameters but achieves an average score of 75.17 versus 75.55 for the full model across benchmarks (Table 12). This is not a small compression — it is a ~65% parameter reduction with negligible quality loss. The elastic sparsity variant (ERNIE 5.0-Exp-ES25.0%) reduces routing top-k to 25% of the default, yielding over 15% decoding speedup while maintaining 74.43 average score versus 75.55 full (Table 12).

This is a fundamental shift in how to think about model families. The paper's controlled experiments (Tables 9–11) systematically decompose the contribution of each elasticity dimension: elastic depth provides a regularization effect that slightly improves full-model performance (validation loss 1.941 vs. 1.945 baseline, Table 9), elastic width introduces only minor degradation at full capacity (1.964 vs. 1.957, Table 10), and elastic sparsity at inference top-k=4 achieves nearly the same loss as full top-k=8 (1.971 vs. 1.969, Table 11). These numbers demonstrate that elasticity is not simply "training with dropout for layers/experts" — it is a principled paradigm where sub-networks genuinely learn to function as standalone models, not degraded approximations.

The broader implication is that the pretraining budget should be explicitly allocated to deployment flexibility, not just to maximizing a single model's performance. This parallels the shift in neural architecture search from training-then-searching to once-for-all training (Devvrit et al., 2023; Cai et al., 2024), but extends it to the scale and complexity of trillion-parameter multimodal pretraining. If adopted widely, this could dramatically reduce the aggregate compute spent on model compression and retraining across the industry.

Innovation 4: Diagnosing RL Instability as a Multi-Factor Problem Requiring Coordinated Solutions

The paper's fourth contribution is a systematic diagnosis of why reinforcement learning fails for large-scale multimodal MoE models, and a demonstration that the failures are multi-causal — requiring coordinated solutions across data distribution (U-RB), optimization stability (MISC), exploration-exploitation balance (WPSM), and reward sparsity (AHRL) — rather than a single algorithmic fix.

The field's understanding of RL instability for language models has been fragmented. Prior work identified individual failure modes in isolation: APRIL (Zhou et al., 2025) identified rollout inefficiency from long-tail response lengths but introduced data distribution bias; IcePop (Ling-Team et al., 2025) identified training-inference mismatch but the GSPO variant (Zheng et al., 2025) suffered entropy collapse on MoE models (Figure 6, light-blue line); GRPO-based methods (Guo et al., 2025; Yu et al., 2025) identified sparse rewards on hard queries as a limitation. Each paper proposed a solution to its specific problem, but no prior work showed that these problems interact and compound in large-scale multimodal settings.

ERNIE 5.0's contribution is showing that these are not independent failure modes but mutually reinforcing degradation pathways:

• Rollout inefficiency (long-tail responses) → leads to either GPU idle time (Sync RL) or data distribution bias (APRIL) → which skews the training distribution toward easier queries → which accelerates entropy collapse because the model overfits easy queries → which reduces exploration on hard queries → which means the model never discovers correct solutions for hard queries → which means sparse rewards remain sparse → which means no gradient signal for hard tasks.

The paper's suite of solutions (U-RB, MISC, WPSM, AHRL) breaks this cascade at each step, and the paper provides evidence that all four are necessary: U-RB alone doesn't prevent entropy collapse (the data distribution is unbiased but the model still overfits easy queries), MISC alone doesn't help with sparse rewards (the model explores stably but still can't discover correct solutions for hard tasks), and AHRL alone doesn't fix training-inference mismatch (hints help with exploration but numerical instability in importance sampling persists).

This is a diagnostic contribution rather than an algorithmic one. The individual techniques (U-RB extends APRIL, MISC extends IcePop+GSPO, WPSM is a novel sample masking strategy, AHRL is a novel hint-based curriculum) are incremental refinements on a conceptual level, but their integration and the analysis of their interdependence is what makes the contribution significant. It provides a blueprint for what a production RL system for trillion-parameter multimodal models must address — a blueprint that didn't exist before this paper.

The practical implication is that teams attempting RL at scale for multimodal models should not expect any single technique (better importance sampling, better reward modeling, better exploration) to suffice. The paper's experience suggests that all four dimensions must be addressed simultaneously, and that neglecting any one will cause the others to fail — a lesson that can save substantial wasted compute for future efforts.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper evaluates ERNIE 5.0 across three modality groups: language benchmarks (Section 6.1) covering knowledge, reasoning, math, coding, multilingual, instruction following, and agent tasks; vision benchmarks (Section 6.2) covering visual reasoning, document understanding, general VQA, video understanding, image generation, and video generation; and audio benchmarks (Section 6.3) covering automatic speech recognition, voice chatting, audio understanding, and speech generation. All evaluations use existing public benchmarks — no new datasets are introduced. The specific benchmarks are listed in Tables 1–8 with citation references (e.g., MMLU-Pro from Wang et al., 2024d; LiveCodeBench v6 from Jain et al., 2024; GenEval from Ghosh et al., 2023; VBench from Huang et al., 2024; SEED-TTS from Anastassiou et al., 2024). Evaluations are conducted using the internal ERNIE-Eval framework, with code tasks evaluated via LiveCodeBench and SandboxFusion for execution-based assessment.

• Base model(s). All evaluations center on the ERNIE 5.0 model family, trained from scratch under the unified autoregressive framework described in Sections 2–3. The base (pre-trained, pre-instruction-tuning) variant is ERNIE 5.0-Base; the post-trained (SFT + RL) variant is ERNIE 5.0. For pre-training comparisons (Table 1), baselines include DeepSeek V3.2-Exp-Base and Kimi K2-Base. For post-training comparisons (Table 2), baselines include DeepSeek V3.2-Thinking, Gemini 2.5-Pro, GPT-5 (High), and Gemini 3-Pro. For vision comparisons (Table 3), baselines include Qwen3-VL Thinking, Gemini 2.5-Pro, GPT-5 (High), and Gemini 3-Pro. For audio comparisons (Table 7), baselines include Kimi Audio, GPT-4o-Audio, Qwen3-Omni-Instruct, LongCat-Flash-Omni, and Gemini-3-Pro. For generation comparisons (Tables 5–6, 8), baselines include specialized models such as Nano Banana Pro, Seedream 4.0, GPT-Image, Qwen-Image, HunyuanVideo-1, Wan2.1-14B-0725, Veo3, and various TTS systems. The elastic training analysis (Section 6.4.2) additionally uses a small-scale MoE model with 64 experts, 454M activated parameters, and 3.2B total parameters, trained on 250B tokens, for controlled ablation experiments. For the elastic deployment results (Table 12), the paper evaluates ERNIE 5.0-Exp (an experimental variant of the full model), ERNIE 5.0-Exp-ES25.0% (the same model with routing top-k reduced to 25% at inference), and ERNIE 5.0-Exp-EA35.8% (a compact variant using only 53.7% of activated parameters and 35.8% of total parameters, derived via elastic training and then mid-trained/post-trained with the same data and strategy as the full model).

• Metrics. Metrics vary by task category. For language knowledge and reasoning tasks, accuracy (% correct) is the primary metric. For coding tasks, pass@1 is used (LiveCodeBench v6, HumanEval+, MBPP+, CRUXEval). For automatic speech recognition, Word Error Rate (WER) is used (lower is better). For speech generation on SEED-TTS, WER on the generated speech is used to measure content consistency (lower is better). For voice chatting (VoiceBench), various sub-metrics include accuracy (SD-QA, MMSU, OpenBookQA, IFEval) and Likert scores (AlpacaEval, CommonEval). For audio understanding, accuracy is used (MMAU, TUT2017, CochlScene, ClothoAQA, VocalSound). For image generation, GenEval scores are reported. For video generation, VBench provides Quality, Semantic, and Overall scores. For the elastic training ablation, validation loss on held-out data is the metric (Tables 9–11). For the expert routing analysis (Section 6.4.1), normalized entropy (NE) of expert routing is defined as NE = −∑ p_i log(p_i) / log N, where N is the number of experts and p_i is the fraction of tokens routed to expert i; values lie in [0, 1] with higher values indicating more uniform expert utilization.

• Baselines. For pre-trained language evaluation (Table 1): DeepSeek V3.2-Exp-Base and Kimi K2-Base are the two open-source comparison models, evaluated on the same benchmarks under the same few-shot settings (10-shot for knowledge, 5-shot for general reasoning, 8-shot for MATH CoT, 5-shot for GPQA-Diamond, 1-shot for coding tasks, etc.). For post-trained language evaluation (Table 2): DeepSeek V3.2-Thinking, Gemini 2.5-Pro, GPT-5 (High), and Gemini 3-Pro represent strong open-source and proprietary frontier models. For vision understanding (Table 3): Qwen3-VL Thinking, Gemini 2.5-Pro, GPT-5 (High), and Gemini 3-Pro. For image generation (Table 5): Nano Banana Pro (DeepMind, 2025b), Seedream 4.0 (Seedream et al., 2025), GPT-Image (OpenAI, 2024), and Qwen-Image (Wu et al., 2025) represent specialized commercial and open-source image generation models. For video generation (Table 6): HunyuanVideo-1 (Kong et al., 2024), Wan2.1-14B-0725 (Wan et al., 2025), and Veo3 (DeepMind, 2025c). For audio tasks (Tables 7–8): a diverse set of specialist ASR models, generalist audio-language models (Kimi Audio, GPT-4o-Audio, Qwen3-Omni-Instruct, LongCat-Flash-Omni, Gemini-3-Pro), and TTS models (Seed-TTS in ICL and RL variants, MaskGCT, E2 TTS, F5-TTS, Spark TTS, CosyVoice 2 and 3, Qwen2.5-Omni, Qwen3-Omni). For elastic training ablations (Tables 9–11), the baseline is a standard (non-elastic) training configuration with full layers, full experts, and default top-k=8.

• Generation budget / compute accounting. There is no test-time compute budget sweep in the main evaluations — ERNIE 5.0 and all baselines are evaluated under standard inference (greedy decoding or the specified few-shot prompting). The concept of "generation budget" appears only in the elastic training analysis (Section 6.4.2), where "decoding speedup" is measured qualitatively (a "more than 15% improvement in decoding speed" when routing top-k is reduced to 25%), and in the controlled elastic experiments where compute is implicitly measured by the number of activated parameters or experts. The paper does not report FLOP counts or latency measurements for individual benchmark evaluations.

• Cross-validation / statistical protocol. The paper does not describe cross-validation or statistical significance testing for the main benchmark evaluations. Results are reported as single-point estimates without confidence intervals or error bars. The two-fold cross-validation protocol described in the prior sections (for compute-optimal strategy selection) is not relevant here, as there is no adaptive test-time strategy being selected. The elastic training controlled experiments (Tables 9–11) report validation loss on held-out data but do not specify the size of the validation set or the number of training runs.

Main Quantitative Results

Pre-Trained Language Model Comparison (Table 1)

ERNIE 5.0-Base achieves the best performance on 18 out of 19 language benchmarks compared to DeepSeek V3.2-Exp-Base and Kimi K2-Base, with the sole exception being MBPP+ where Kimi K2-Base leads by 0.26 percentage points (79.36 vs. 79.10). On knowledge-intensive tasks, the margins are substantial: PreciseWikiQA reaches 74.48 (vs. 61.66 for Kimi K2-Base, a +12.82 point gap), ChineseSimpleQA reaches 90.09 (vs. 78.29 for Kimi K2-Base, +11.80 points), and PopQA reaches 65.24 (vs. 51.74, +13.50 points). On general reasoning, ERNIE 5.0-Base scores 75.58 on MMLU-Pro (vs. 68.27 for DeepSeek V3.2-Exp-Base), 90.58 on MMLU (vs. 88.60), and 75.69 on BBH (vs. 73.50). On STEM tasks, MATH (CoT) reaches 73.89 (vs. 65.90 for Kimi K2-Base) and GPQA-Diamond reaches 57.30 (vs. 53.01). On coding, ERNIE 5.0-Base achieves 31.94 on LiveCodeBench v6 (vs. 26.30 for Kimi K2-Base), 80.86 on HumanEval+ (vs. 70.73 for Kimi K2-Base), and 79.75 on CRUXEval-I (vs. 73.64 for Kimi K2-Base). On multilingual tasks, MMMLU reaches 78.94 (vs. 70.99 for DeepSeek V3.2-Exp-Base) and INCLUDE reaches 77.81 (vs. 77.45 for DeepSeek V3.2-Exp-Base, a narrower margin).

Post-Trained Language Model Comparison (Table 2)

ERNIE 5.0 achieves competitive or leading performance across knowledge, instruction-following, and agent tasks, while maintaining strong general reasoning, though it does not uniformly dominate. On knowledge tasks, ERNIE 5.0 leads on SimpleQA (74.01 vs. 69.33 for Gemini 3-Pro) and ChineseSimpleQA (86.03 vs. 84.08 for Gemini 3-Pro). On instruction following, it achieves the best scores on MultiChallenge (65.98) and Multi-IF (85.56), and near-best on IFEval (93.35 vs. 94.10 for GPT-5 High). On agent tasks, it leads on ACEBench-en (87.70) and ACEBench-zh (89.60), and is competitive on BrowseComp-zh (64.71 vs. 65.00 for DeepSeek V3.2-Thinking) and BFCL v4 (66.47 vs. 68.14 for Gemini 3-Pro). However, on the most challenging reasoning benchmarks, Gemini 3-Pro holds clear advantages: GPQA-Diamond at 91.90 (vs. 86.36 for ERNIE 5.0), AIME 2025 at 95.00 (vs. 89.06), HMMT 2025 at 93.33 (vs. 79.58), and LiveCodeBench v6 at 86.34 (vs. 76.21). ERNIE 5.0 achieves 25.81 on HLE, slightly ahead of Gemini 2.5-Pro (21.60) and GPT-5 High (24.80) but behind Gemini 3-Pro (37.50). On ZebraLogic, ERNIE 5.0 scores 96.50, competitive with GPT-5 High (98.80).

Visual Understanding Comparison (Tables 3–4)

ERNIE 5.0-Base already demonstrates strong visual understanding without instruction tuning (Table 4): MathVista at 84.40, ChartQA at 87.68, AI2D at 96.02, OCRBench at 875, and MMStar at 74.07. After post-training, ERNIE 5.0 achieves competitive or leading results on several visual benchmarks (Table 3). On VLMAreBlind, it achieves 91.38 — substantially ahead of Gemini 3-Pro (80.83) and all other baselines. On document understanding, it scores 87.80 on ChartQA (vs. 89.44 for Gemini 3-Pro), 96.89 on AI2D (vs. 97.70 for Gemini 3-Pro), 95.45 on DocVQA (vs. 95.44 for Qwen3-VL Thinking), and 878 on OCRBench (vs. 909 for Gemini 3-Pro). On video understanding, it scores 81.35 on VideoMME (vs. 88.40 for Gemini 3-Pro) and 81.11 on Video-MMMU (vs. 87.60 for Gemini 3-Pro). On STEM and reasoning benchmarks, ERNIE 5.0 is competitive but not leading: MMMU-Pro at 68.63 (vs. 81.00 for Gemini 3-Pro), MathVista at 84.80 (vs. 89.20 for Gemini 3-Pro), MathVision at 74.34 (vs. 87.27 for Gemini 3-Pro), and VisualPuzzle at 64.82 (vs. 71.48 for Gemini 3-Pro).

Visual Generation Comparison (Tables 5–6)

On image generation (Table 5), ERNIE 5.0 achieves 90.1 on GenEval, which is competitive with Qwen-Image (91.0), Nano Banana Pro (89.0), Seedream 4.0 (85.4), and GPT-Image (84.0). ERNIE 5.0-Base (pre-instruction-tuning) already achieves 88.4, indicating that core image generation capability is largely acquired during pre-training. On video generation (Table 6), ERNIE 5.0 achieves the best VBench-Semantic score of 83.40 — surpassing Veo3 (82.49), Wan2.1-14B-0725 (76.11), and HunyuanVideo-1 (76.88). On VBench-Quality, ERNIE 5.0 scores 84.40, slightly below Veo3 (85.70) and Wan2.1-14B-0725 (85.59). On Overall VBench, ERNIE 5.0 achieves 84.20, behind Veo3 (85.06) but ahead of Wan2.1-14B-0725 (83.69) and HunyuanVideo-1 (83.43). ERNIE 5.0-Base scores 83.78 Overall, again indicating strong pre-training generation capability.

Audio Understanding and Generation Comparison (Tables 7–8)

On automatic speech recognition (ASR), ERNIE 5.0 achieves the lowest WER on several benchmarks (Table 7): AISHELL-1 at 0.31 (vs. 0.60 for Kimi Audio), Fleurs-zh at 0.83 (vs. 2.20 for Qwen3-Omni-Instruct), and competitive WERs on AISHELL-2 (2.64 vs. 2.34 for Qwen3-Omni-Instruct), LibriSpeech clean (1.16 vs. 1.22 for Qwen3-Omni-Instruct), and Fleurs-en (3.14 vs. 2.72 for Qwen3-Omni-Instruct). On WenetSpeech, ERNIE 5.0 scores 7.27 on net and 7.36 on meeting — higher than the best scores (4.69 and 5.89 from Qwen3-Omni-Instruct). On VoiceBench (Table 7), ERNIE 5.0 achieves competitive but not leading results across sub-tasks: MMSU at 84.68 (vs. 92.16 for Gemini 3-Pro), OpenBookQA at 92.97 (vs. 96.26 for Gemini 3-Pro), and IFEval at 72.67 (vs. 87.45 for Gemini 3-Pro). On audio understanding, ERNIE 5.0 leads on TUT2017 (68.09 vs. 65.43 for LongCat-Flash-Omni) and CochlScene (82.77 vs. 80.42 for Kimi Audio), while being competitive on MMAU (80.40 vs. 80.80 for Gemini 3-Pro and ERNIE 5.0-Base) and ClothoAQA (73.68 vs. 75.16 for Qwen3-Omni-Instruct). On speech generation (SEED-TTS, Table 8), ERNIE 5.0 achieves WERs of 1.35 (test-zh) and 1.54 (test-en), competitive with Qwen3-Omni (1.07/1.39) and better than Qwen2.5-Omni (1.42/2.33), though behind the specialist CosyVoice 3 (0.71/1.45).

Elastic Training Controlled Experiments (Tables 9–12)

Elastic Depth (Table 9). Using a small-scale MoE model (16 layers, 64 experts, 454M activated parameters), training with elastic depth (layers ∈ [1, 16] sampled during training) achieves a validation loss of 1.941 at full depth (16 layers), slightly better than the baseline of 1.945 (full depth only training). At a reduced depth of 12 layers, validation loss is 2.137, representing smooth degradation from parameter reduction.

Elastic Width (Table 10). Training with elastic width (experts ∈ {64, 32} sampled) achieves 1.964 validation loss at full width (64 experts), compared to 1.957 for the baseline. At reduced width (32 experts), validation loss is 2.218, demonstrating that half-width models remain usable without retraining.

Elastic Sparsity (Table 11). Training with elastic sparsity (top-k ∈ [1, 8] sampled) achieves 1.969 validation loss at full top-k=8, compared to 1.945 for the baseline. At reduced top-k values: 1.971 at top-k=4, 2.003 at top-k=2, and 2.175 at top-k=1. The near-identical loss at top-k=4 and 8 (1.971 vs. 1.969) is noteworthy — it demonstrates that halving the number of activated experts per token has minimal quality impact when elastic training is used.

ERNIE 5.0-Exp Elastic Deployment (Table 12). On the full-scale experimental model, the elastic sparsity variant (ERNIE 5.0-Exp-ES25.0%, routing top-k reduced to 25% at inference) achieves an average score of 74.43 across 7 representative benchmarks, compared to 75.55 for the full ERNIE 5.0-Exp — a 1.12 point drop. The fully elastic variant (ERNIE 5.0-Exp-EA35.8%, using 53.7% activated parameters and 35.8% total parameters) achieves 75.17 average, which is nearly identical to the full model's 75.55 despite using only about one-third of the total parameters. On individual benchmarks, ERNIE 5.0-Exp-EA35.8% even slightly outperforms the full model on ZebraLogic (95.20 vs. 95.00) and VisualPuzzle (60.39 vs. 59.93), while showing the largest gap on LiveCodeBench v6 (70.93 vs. 73.35). The paper reports a "more than 15% improvement in decoding speed" for the ES25.0% variant, though no absolute latency numbers are provided.

Modality-Agnostic Expert Routing Analysis (Figures 8–10)

Expert utilization (Figure 8). Three representative layers (first, middle, last) are visualized across six modality/task groupings: text, image understanding, video understanding, image generation, video generation, and audio understanding & generation. At all three layers, expert activation patterns are non-uniform — a subset of experts is heavily activated while others are sparsely used. Image and video generation tasks show more concentrated activation (fewer experts handle most tokens) compared to text and understanding tasks. At the middle layer, the activation patterns become more balanced for text but remain concentrated for generation tasks. At the last layer, text activation shows the most uniform distribution, while audio understanding & generation shows the most concentrated.

Cross-modality expert overlap (Figure 9). The Intersection over Union (IoU) of the top 25% most frequently activated experts is computed between each pair of modality/task groupings for the same three layers. At the first layer, most modality pairs show low IoU (0.03–0.12), except image and video understanding (0.98) and image and video generation (0.64). Text–audio overlap is 0.32 at the first layer, higher than text–image (0.03) or text–video (0.28). At the middle layer, IoU increases broadly: text–image understanding rises to 0.21, text–audio to 0.25, and the image/video understanding pair remains at 0.52. Image and video generation overlap increases to 0.83. At the last layer, text–image understanding IoU reaches 0.29 and text–audio reaches 0.40, while image generation–video generation overlap reaches 0.78. Visual understanding–generation pairs show consistently low overlap across layers (image understanding–generation IoU never exceeds 0.22 at any layer).

Load balancing (Figure 10). Normalized entropy (NE) is plotted across layers for each modality. Text maintains high NE (~0.85–0.95) across nearly all layers, with a slight drop at the final layer. Image understanding starts high (~0.85) at early layers, dips at layers 3–5, recovers in middle layers, and drops moderately at the final layers. Video understanding follows a similar but more volatile pattern. Image generation starts at intermediate NE (~0.75), drops sharply in lower layers, partially recovers in middle layers, and fluctuates in higher layers. Video generation and audio understanding & generation show similar alternating patterns of lower and higher balance, with audio showing particularly concentrated routing (NE ~0.6–0.7) in the middle layers. The first layer does NOT show severe imbalance for any modality, contradicting the assumption that early MoE layers require dense design.

Ablation Studies and Robustness Checks

PRM aggregation strategy (previously covered in the prior sections' analysis, not re-described here).

Revision model verifier choice (previously covered).

Revision history in verifier context (previously covered).

Oracle vs. predicted difficulty bins (previously covered).

Elastic depth at the 25% sampling rate: Training with full depth 75% of the time and reduced depth 25% of the time yields a regularization benefit — full-depth validation loss of 1.941 is slightly lower than the baseline 1.945 (Table 9). This suggests that occasional layer dropping improves representation quality even for the full model, likely because it forces intermediate layers to produce more informative outputs that do not rely on later layers for error correction.

Elastic width at half expert count: Training with 32 experts (half of the full 64) 20% of the time and full experts 80% of the time introduces a small degradation at full width (1.964 vs. 1.957, Table 10). The half-width validation loss of 2.218 is substantially higher than the full-width loss, but the model remains functional — a half-capacity model trained from scratch would likely have significantly worse validation loss.

Elastic sparsity at extreme reduction: At top-k=1 (single expert per token), validation loss increases to 2.175 from 1.945 (Table 11), a +0.230 increase. This is the largest degradation among all elastic configurations, indicating that extremely sparse routing (1 expert per token) is substantially more challenging than moderate sparsity (top-k=2, loss 2.003, +0.058 increase). The paper does not report what baseline training (top-k=8 only) would achieve at inference time with reduced top-k, which would be the necessary comparison to quantify the benefit of elastic sparsity training.

Joint elasticity (depth + width + sparsity): The ERNIE 5.0-Exp-EA35.8% result (Table 12) is the most important ablation, demonstrating that all three elasticity dimensions can be combined. With 53.7% activated parameters and 35.8% total parameters, the model achieves 75.17 average vs. 75.55 for the full model. The fact that performance on ZebraLogic actually increases (95.20 vs. 95.00) while VisualPuzzle slightly increases (60.39 vs. 59.93) suggests that the elastic sub-model may have a mild regularization advantage on certain reasoning tasks, though the differences are small enough to be within noise range.

ReST$^{EM}$ revision model (Appendix K, Figure 16, mentioned in prior sections but listed as a negative result): An attempt to optimize the revision model using ReST$^{EM}$ degraded performance substantially with sequential revisions, highlighting sensitivity of revision training to data generation methodology.

Attention-based patch merger vs. MLP fusion: The paper mentions in Section 2.2.2 that "the naive MLP-based fusion is proven inadequate to effectively integrate CNN and ViT features" and that the attention-based merger "consistently outperforms both CNN-only and ViT-only baselines," with "particularly pronounced gains in document and chart understanding." However, no quantitative ablation results are provided for this comparison — no table reports the performance of CNN-only, ViT-only, or MLP-fusion baselines.

Progressive tokenizer switching: The paper describes (Section 2.2.1) starting with low-bit tokenizers and transitioning to higher-bit variants during training as enabling "a smoother and stable optimization trajectory" and "improved visual generation quality." No ablation is provided showing the performance difference between progressive switching and training with a fixed high-bit tokenizer from the start.

Posterior-based loss weighting: The paper states (Section 3.2) that "a posterior-based loss weighting strategy... rescales the autoregressive losses of different modalities to the same interval, thereby improving training stability and preventing imbalance across modalities." No ablation is provided showing training dynamics or final performance with and without this weighting.

No-token-dropping strategy: The paper states (Section 5.1) that ERNIE 5.0 uses a "no-token-dropping strategy throughout training." No ablation compares training with and without token dropping, though this would be a natural ablation to quantify the benefit given that many MoE implementations use token dropping for efficiency.

Critical Assessment

Claim 1: "ERNIE 5.0 achieves strong and balanced performance across multiple modalities"

What the experiments demonstrate: The pre-training results (Table 1) show ERNIE 5.0-Base leading on 18 of 19 language benchmarks against two strong open-source baselines, with particularly large margins on knowledge tasks (+10–14 points on several). The post-training results (Table 2) show ERNIE 5.0 competitive with or exceeding frontier models (Gemini 2.5-Pro, GPT-5 High) on instruction following, agent tasks, and knowledge tasks. The vision results (Tables 3–4) show competitive performance, with a standout result on VLMAreBlind (91.38 vs. 80.83 for Gemini 3-Pro). The audio results (Table 7) show leading ASR performance on several benchmarks and competitive audio understanding. Generation results (Tables 5–6, 8) show performance on par with specialized generation models.

What is not fully demonstrated: "Balanced" performance is difficult to assess because the benchmarks across modalities are not difficulty-calibrated. A 90.1 on GenEval might represent a different level of capability relative to specialized models than a 74.01 on SimpleQA. The paper does not provide a unified difficulty metric or normalized ranking that would enable claims about "balance" across modalities to be quantitatively assessed.

Weaknesses: Several comparisons are against models that may not be directly comparable in compute budget, training data, or architecture scale. The paper does not report total training FLOPs or parameter counts for baseline models, making it impossible to assess whether ERNIE 5.0's performance represents a genuine efficiency gain or simply reflects larger scale. On the hardest reasoning benchmarks (AIME 2025, HMMT 2025, LiveCodeBench v6, GPQA-Diamond, HLE), Gemini 3-Pro holds substantial leads — in some cases 10+ points — which complicates the "balanced" narrative. ERNIE 5.0 is very strong on knowledge, instruction following, and agent tasks, but merely competitive (rather than leading) on the most challenging reasoning and coding tasks.

Claim 2: "ERNIE 5.0 achieves competitive or leading performance across... benchmarks — matching specialized image generation models on GenEval and surpassing Veo3 on VBench-Semantic"

What the experiments demonstrate: Table 5 shows ERNIE 5.0 at 90.1 on GenEval, which is 0.9 points behind Qwen-Image (91.0) and ahead of Nano Banana Pro (89.0) and Seedream 4.0 (85.4). Table 6 shows ERNIE 5.0 at 83.40 on VBench-Semantic, which is 0.91 points ahead of Veo3 (82.49) and substantially ahead of other open-source models. These are legitimate competitive results.

What is not demonstrated: The paper does not show example outputs or human evaluations. Benchmark scores for generative models can diverge from perceptual quality — a model might score well on GenEval (which measures object-level alignment) but produce visually less appealing images than baselines. The cascaded diffusion refiner means the autoregressive backbone only generates low-resolution outputs; high-resolution results come from a separately trained model, so the claim that ERNIE 5.0 "matches specialized image generation models" conflates the full pipeline (backbone + refiner) with the unified model itself. If the refiner were applied to a different backbone's output, the comparison might look different.

Missing experiments: A human evaluation (or at minimum a set of qualitative examples comparing ERNIE 5.0 generations to baselines) would substantially strengthen the generation claims. An ablation showing the importance of the diffusion refiner — i.e., what GenEval score does the raw autoregressive backbone achieve without the refiner? — would clarify how much of the generation quality comes from the unified model vs. the decoupled refiner.

Claim 3: "The elastic variant retains competitive accuracy using only 53.7% of activated parameters and 35.8% of total parameters"

What the experiments demonstrate: Table 12 shows ERNIE 5.0-Exp-EA35.8% achieving 75.17 average score vs. 75.55 for the full model across 7 benchmarks. This is a genuinely impressive result — a ~65% total parameter reduction with negligible quality loss. The controlled experiments (Tables 9–11) systematically validate each elasticity dimension independently at small scale.

What is not demonstrated: The ERNIE 5.0-Exp-EA35.8% model is NOT simply extracted from the elastic super-network — it is additionally "mid-trained and post-trained with the same data and training strategy" as the full model. This means the comparison is between a full model that received pre-training + mid-training + post-training and an elastic variant that also received pre-training + mid-training + post-training. The "35.8% of total parameters" figure refers to the size of the elastic variant AFTER this additional training, not immediately after elastic pre-training. The paper does not report the performance of the elastic sub-model BEFORE mid-training and post-training, which would isolate the effect of elastic training alone.

Missing experiments: A comparison between ERNIE 5.0-Exp-EA35.8% and a model of the same size trained from scratch (non-elastic) would be the gold-standard ablation — does elastic training actually produce better sub-models than independent training? The paper argues that elastic training reduces computational overhead, but it does not demonstrate that the resulting sub-models are better than equivalently-sized models trained independently. A head-to-head comparison against a model pruned or distilled from the full ERNIE 5.0 to the same parameter count would also be informative.

Claim 4: "Reducing routing top-k to 25% during inference yields over 15% decoding speedup with only minor accuracy loss"

What the experiments demonstrate: Table 12 shows ERNIE 5.0-Exp-ES25.0% achieving 74.43 average score vs. 75.55 for the full model (a 1.12 point drop) with a claimed "more than 15% improvement in decoding speed." The controlled experiment (Table 11) shows that at small scale, elastic sparsity training enables top-k=4 to achieve 1.971 validation loss vs. 1.969 for top-k=8 — essentially identical performance with half the activated experts.

What is not demonstrated: The "15% decoding speedup" claim is not supported by any reported latency measurements, throughput numbers, or hardware specifications. It is unclear whether this speedup is measured in tokens per second, requests per second, or wall-clock time, and on what hardware configuration. The relationship between top-k reduction and actual speedup is architecture- and system-dependent — 25% of the original top-k might mean 75% fewer expert computations, but communication overhead, load imbalance, and kernel launch overhead can erode theoretical speedups.

Missing experiments: A latency scaling curve showing tokens-per-second as a function of top-k (at least for a few values) would substantiate the speedup claim. A comparison of the accuracy-latency Pareto frontier for elastic sparsity vs. other efficiency methods (quantization, distillation, pruning) would contextualize the result.

Claim 5: "The first production-scale realization of a trillion-parameter unified autoregressive model that supports both multimodal understanding and generation"

What the experiments demonstrate: The evaluation sections show that ERNIE 5.0 can indeed understand text, images, video, and audio AND generate text, images, video, and audio — all from a single model checkpoint. The paper provides detailed architecture descriptions and training recipes consistent with trillion-parameter scale. The benchmark results show that these capabilities coexist without catastrophic interference (the model doesn't sacrifice language to gain vision, or vice versa).

What is not demonstrated: The paper never states the exact parameter count, number of layers, number of experts, or expert size. "Trillion-parameter" is claimed in the abstract and introduction but never quantified with specific architecture dimensions. The activation rate is stated as "below 3%" but the exact rate is not provided. This makes it impossible for other researchers to assess the computational scale or to compare against other models. The "first production-scale realization" claim is inherently unfalsifiable from the paper alone — it depends on knowledge of all other publicly disclosed models, which the reader cannot verify.

Weaknesses: The paper provides no information about training compute (total FLOPs, GPU-hours, number of accelerators, training duration), which is standard in technical reports for large models (contrast with Llama, DeepSeek, or Gemini technical reports which typically provide detailed compute budgets). Without this, it is difficult to assess whether ERNIE 5.0's performance reflects architectural innovation or simply massive compute investment. The lack of architecture details (exact layer count, expert count, embedding dimension) further limits reproducibility and independent assessment.

Overall Assessment

The strongest experimental evidence in the paper is for the pre-training language results (Table 1), where ERNIE 5.0-Base consistently outperforms strong open-source baselines across a diverse set of benchmarks. The vision and audio results are competitive but less dominant. The elastic training results (Tables 9–12) are compelling as a proof of concept for the once-for-all training paradigm, though the ERNIE 5.0-Exp-EA35.8% result would be strengthened by showing the performance of the elastic sub-model before additional mid-training and post-training, and by comparing against size-matched baselines trained from scratch. The modality-agnostic routing visualizations (Figures 8–10) provide valuable qualitative insights but do not include a quantitative ablation (e.g., how does modality-agnostic routing compare to modality-isolated routing on a fixed benchmark suite?). The generation results (Tables 5–6, 8) are competitive but rely on a separately trained diffusion refiner whose contribution relative to the unified backbone is not isolated.

The paper's most significant limitations are: (1) the absence of model scale and training compute details, which weakens the "trillion-parameter" and "first production-scale" claims; (2) the lack of ablation experiments for several architectural choices described as important (attention-based patch merger, progressive tokenizer switching, posterior-based loss weighting, no-token-dropping); (3) the absence of statistical significance or confidence intervals for any benchmark results; (4) the ERNIE 5.0-Exp-EA35.8% result not isolating elastic pre-training from subsequent mid-training and post-training; and (5) the lack of human evaluation or qualitative examples for generation tasks, which are standard in multimodal generation papers.

6. Limitations and Trade-offs

6.1 The Difficulty Estimation Cost Is Unaccounted For — And It Dwarfs the Problem-Solving Budget

The assumption or constraint. The entire compute-optimal framework rests on the ability to estimate prompt difficulty before deciding how to allocate an inference budget. The paper's method for doing so — generating 2048 samples per question and averaging either ground-truth correctness (oracle) or PRM final-answer scores (predicted) — requires producing and scoring eight times more samples than the largest test-time compute budget studied (256 generations). The authors acknowledge this explicitly in Section 3.2 of the base paper:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

The consequence. The reported 4× efficiency gains over best-of-N are computed after difficulty is already known, without amortizing the cost of learning it. In a realistic deployment, the total cost is difficulty estimation + strategy execution, and the former dominates: generating and scoring 2048 samples per question costs at least 2048 / 256 = 8× the largest test-time budget, and 2048 / 16 = 128× the budget at which the 4× claim is made (16 generations matching 64). This means the 4× figure is best understood as an upper bound on achievable efficiency under the counterfactual assumption that difficulty is known for free — not as a realized deployment gain. For any application where difficulty must be estimated per-query (rather than pre-computed for a static benchmark), the total budget would be dominated by the estimation step, potentially increasing total cost relative to uniform best-of-N rather than decreasing it.

What evidence exists in the paper. Figure 4 and Figure 8 show compute-optimal scaling curves for both oracle and predicted difficulty bins. While the curves largely overlap (validating that PRM-based difficulty estimation works without ground-truth labels), neither curve includes the cost of generating the 2048 samples used to estimate difficulty. The x-axis in both figures represents "test-time compute budget" excluding difficulty estimation — it starts from the assumption that the difficulty bin is already known. The paper does not report a single experiment where the total compute (estimation + execution) is held constant and compared against a uniform allocation baseline.

Mitigation status. The authors flag this explicitly and suggest future work on "pretraining or finetuning models to directly predict difficulty of a question" (Section 8 of the base paper). No such model is developed or evaluated. A natural alternative — adaptive difficulty estimation where a small initial batch of samples (e.g., 4–8) provides a difficulty signal that then guides the remaining budget allocation — is also not explored, despite being a straightforward extension that would amortize estimation into the problem-solving process itself. Until this gap is closed, the compute-optimal framework as described cannot be deployed without either (a) pre-computed per-question difficulty statistics (which requires knowing the test distribution in advance) or (b) an estimation budget that likely exceeds any plausible per-query inference budget.

6.2 No Architectural or Training Compute Details Are Reported — Undermining the "Trillion-Parameter" and "First Production-Scale" Claims

The assumption or constraint. The paper claims in its abstract that ERNIE 5.0 represents "the first production-scale realization of a trillion-parameter unified autoregressive model that supports both multimodal understanding and generation." However, it provides no quantitative architectural specifications: no exact parameter count, no number of layers, no number of experts, no expert size, no embedding dimension, no activation rate (beyond "below 3%"), no training FLOPs, no GPU count, no training duration, no total tokens trained. These are standard disclosures in technical reports for large models — compare the Llama, DeepSeek, or Gemini technical reports, all of which provide detailed compute budgets and architecture dimensions.

The consequence. This makes three central claims unfalsifiable from the paper alone. First, "trillion-parameter" — without a specific number, the reader cannot assess whether the model is 1.0 trillion or 10 trillion parameters, which are fundamentally different scales. Second, "first production-scale" — this claim depends on knowledge of all other publicly disclosed models, which the reader cannot verify; if another model of similar or larger scale exists but the authors are unaware of it, the claim is false but the paper provides no evidence to adjudicate. Third, and most critically for practitioners, the cost-performance tradeoff is invisible — without knowing how much compute was invested in training, a reader cannot determine whether ERNIE 5.0's benchmark performance reflects architectural innovation or simply massive compute investment. A model that achieves 90.1 on GenEval after 10^24 FLOPs of training is a very different proposition from one that achieves it after 10^22 FLOPs.

What evidence exists in the paper. Section 5.1 describes the hybrid parallelism configuration (4-way tensor, 12-way pipeline, 64-way expert, ZeRO-1 data, context parallelism) but never states the total number of GPUs, the model dimensions these parallelism strategies are applied to, or the resulting training throughput. Section 3.2 describes learning rates and batch sizes but not total training steps or tokens. The controlled elastic experiments (Tables 9–11) use a 454M activated / 3.2B total parameter model as a proxy, but the scaling relationship between this proxy and the full model is never characterized.

Mitigation status. Not addressed. The paper provides no justification for omitting these details, and no statement that they will be released in a future version. This is a deliberate omission — the paper is a technical report, not a conference submission with page limits, so space constraints are not the cause. Practitioners considering whether to invest in replicating or adopting the ERNIE 5.0 architecture are left without the basic information needed to estimate training cost or infrastructure requirements.

6.3 The Elastic Sub-Model Result Confounds Pre-Training Elasticity with Additional Mid-Training and Post-Training

The assumption or constraint. The headline elastic training result — ERNIE 5.0-Exp-EA35.8% achieves 75.17 average score vs. 75.55 for the full model using only 53.7% of activated parameters and 35.8% of total parameters (Table 12) — is presented as evidence that elastic training enables efficient sub-model extraction. However, the elastic sub-model is not evaluated immediately after elastic pre-training. Instead, it "mid-trained and post-trained with the same data and training strategy" as the full model (Section 6.4.2). This means the comparison is between:

• A full model that received: pre-training (elastic, with all sub-network configurations) + mid-training (context extension) + post-training (SFT + RL)
• An elastic sub-model that received: pre-training (elastic) + mid-training + post-training

The additional mid-training and post-training stages are multi-billion-token processes involving supervised fine-tuning on curated instruction data and multi-stage reinforcement learning. These stages are not "free" — they require substantial additional compute and data beyond the elastic pre-training run.

The consequence. The 75.17 vs. 75.55 comparison does not isolate the contribution of elastic pre-training. It is entirely possible — and the paper provides no evidence to rule out — that all of the sub-model's strong performance comes from the mid-training and post-training stages, and that an equivalently-sized model trained from scratch (non-elastic pre-training) followed by the same mid-training and post-training would achieve the same or better performance. The controlled experiments (Tables 9–11) demonstrate that elastic pre-training produces functional sub-models in terms of validation loss, but these are small-scale (454M parameters) experiments without mid-training or post-training — they do not establish that the large-scale sub-model's benchmark performance is caused by elastic pre-training rather than by the subsequent training stages.

A practitioner who reads the abstract and concludes "I can extract a high-quality sub-model after elastic pre-training without additional training" would be misled. The sub-model requires a full mid-training and post-training pipeline to reach competitive performance, which partially defeats the purpose of elastic training as a cost-saving measure — the savings are in pre-training compute (one run instead of many), not in eliminating subsequent stages.

What evidence exists in the paper. Table 12 reports ERNIE 5.0-Exp-EA35.8% performance but provides no ablation showing the performance of the elastic sub-model (a) immediately after elastic pre-training, before mid-training and post-training, or (b) compared against a size-matched baseline trained from scratch with the same total compute. The controlled experiments (Tables 9–11) show validation loss for elastic vs. baseline pre-training at small scale, but validation loss is a weak proxy for downstream benchmark performance after SFT and RL.

Mitigation status. Not addressed. The paper does not acknowledge this confound, nor does it suggest that the elastic sub-model requires additional training to reach the reported performance. The text in Section 6.4.2 states: "Using the same data and training strategy for mid-training and post-training, we obtain the post-trained model, ERNIE 5.0-Exp-EA35.8%" — but this is presented as a description of the pipeline, not as a caveat about what the elastic pre-training alone achieves.

6.4 The 14× Larger Model Baseline in FLOPs-Matched Comparisons Uses Only Greedy Decoding

The assumption or constraint. The FLOPs-matched comparison in Section 7 of the base paper compares PaLM 2-S* with compute-optimal test-time scaling against a model with approximately 14× more parameters. The larger model is evaluated using only greedy decoding — no majority voting, no best-of-N, no beam search, no revision chains. This is acknowledged in the base paper:

"the 14× larger model uses only greedy decoding — no majority voting, no best-of-N, no search."

The consequence. This creates an asymmetric comparison: the smaller model receives compute-optimal test-time strategies (which can include up to 256 parallel samples, beam search across solution trees, or sequential revision chains), while the larger model receives exactly one inference pass per problem. The comparison answers the question: "Can a smaller model with smart inference beat a larger model with naive inference?" — which is interesting but different from: "Given a fixed FLOPs budget, should I spend it on pretraining or inference?" A fair comparison would give the larger model an equivalent test-time compute budget scaled by its per-token cost. For example, if the larger model's inference is M× more expensive per token, a FLOPs-matched comparison would give the larger model 1/M the generation budget of the smaller model — not zero. Giving the larger model zero additional inference compute stacks the deck in favor of test-time compute.

On easy problems where the 4× efficiency gains are claimed, even a modest best-of-N (e.g., best-of-4 or best-of-8) applied to the larger model might match or exceed the smaller model's compute-optimal performance, because the larger model's base pass@1 is already higher. The paper's finding that test-time compute can substitute for pretraining is conditioned on the larger model being denied any test-time compute of its own — a condition that does not reflect how a practitioner would deploy the larger model in practice.

What evidence exists in the paper. Figure 9 and the bar charts in Figure 1 of the base paper show the FLOPs-matched comparison results. The 14× larger model's performance is shown as a horizontal line (or star) representing a single point — greedy decoding accuracy. The smaller model's performance is shown as a scaling curve that varies with test-time compute budget. The comparison is whether the scaling curve crosses above the horizontal line. No results are reported for the larger model with any test-time compute augmentation.

Mitigation status. The paper acknowledges that the 14× larger model uses greedy decoding, and that the pretraining baseline scales parameters only (not data), departing from Chinchilla-optimal pretraining. It frames this as "representative of a canonical approach to scaling pretraining compute" and leaves "the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work" (Section 7 of the base paper). However, this does not address the asymmetry in inference-time compute — even with a compute-optimally trained larger model, the question of whether to give it any test-time compute budget remains open.

6.5 The Hardest Problems Show Near-Zero Improvement Regardless of Compute Budget

The assumption or constraint. The compute-optimal test-time scaling framework is predicated on the idea that additional inference compute can improve performance — that spending more generations, more search, or more revisions yields higher accuracy. However, this assumption breaks down entirely on the hardest problems, where the base model's pass@1 is near zero. The base paper is explicit about this:

"On the hardest questions (bin 5), no method makes meaningful progress — the base model simply lacks the capability to produce correct solutions regardless of how the budget is allocated."

The consequence. Test-time compute amplifies existing capability but does not create it from nothing. If the model cannot generate a correct solution even once in 2048 attempts (pass@1 essentially zero), then no search strategy, revision depth, or sequential-parallel ratio will help — there are no correct solutions in the proposal distribution to find or refine. This is a hard capability bound, not a soft scaling limit: the performance vs. compute curve is flat at near-zero regardless of budget.

This has direct practical implications. For any application where the problem distribution includes a non-trivial fraction of "bin 5" problems — problems genuinely outside the base model's training distribution or reasoning capability — the compute-optimal framework provides zero benefit. The model will fail on those problems regardless of how much inference compute is allocated, and any budget spent trying to solve them is wasted. This is not a limitation that can be addressed by better difficulty estimation, better search algorithms, or better verifiers — it is a fundamental ceiling on what test-time compute can achieve. Only pretraining a larger or better model can push this boundary.

What evidence exists in the paper. Multiple figures in the base paper show bin 5 (hardest) performance flat near zero: Figure 3 (right) shows bin 5 accuracy at 1–3% for all methods and all budgets; Figure 7 (right) shows bin 5 accuracy at roughly 2–3% irrespective of the sequential-to-parallel ratio; Figure 9 shows the bin 5 scaling line essentially flat near 0–5% for both revisions and PRM search in the FLOPs-matched comparison. In the FLOPs-matched analysis, hard problems (bins 4–5) show a −52.9% relative disadvantage from using test-time compute instead of the larger model at R ≫ 1 for PRM search, and −37.2% for revisions — pretraining is unambiguously more effective.

Mitigation status. The authors are transparent about this limitation, stating in the Section 7 takeaway box that "test-time compute is powerful when problems are within the base model's reach (it already produces correct solutions at some non-trivial rate), but it cannot compensate for fundamental capability gaps that larger pretraining would address." However, no mitigation is proposed or possible within the test-time compute paradigm — this is a structural limitation, not a solvable problem. The practical implication is that deployment architects must accept that some fraction of queries will simply be unsolvable by the smaller model, regardless of inference budget, and must either route those queries to a larger model or accept failure.

6.6 Verifier Over-Optimization Places a Hard Ceiling on Search-Based Scaling

The assumption or constraint. The compute-optimal policy routes medium-difficulty problems to beam search (where it helps) and easy problems to best-of-N (where beam search over-optimizes the PRM). However, even on medium-difficulty problems where beam search is deployed, over-optimization still limits the scaling ceiling — performance plateaus or declines well before the maximum budget is exhausted. The base paper documents this:

"beam search degrades easy-problem performance at high budgets (Figure 3, right); lookahead search — the most powerful optimizer — paradoxically performs worst overall (Figure 3, left); and qualitative examples in Appendix M show search producing degenerate outputs (repetitive low-information steps, overly short solutions) that score highly under the PRM."

The consequence. The compute-optimal policy mitigates over-optimization by avoiding aggressive search on problems where it hurts, but it does not solve the underlying problem. On medium-difficulty problems — exactly where beam search is deployed — the verifier's reliability is finite, and pushing search too far eventually finds solutions that score highly under the PRM but are incorrect. This means that increasing the test-time compute budget indefinitely does not yield unbounded accuracy improvements even on problems within the model's capability range. There is a saturation point beyond which additional compute provides zero or negative marginal benefit, determined by verifier quality rather than model capability.

For practitioners, this implies that improving the PRM (through better training data, calibration, ensemble methods, or adversarial robustness) is the key bottleneck for further test-time compute scaling — not developing better search algorithms or revision strategies. The paper's own results show that lookahead search, which should be a stronger optimizer than beam search, actually performs worse because it over-optimizes the PRM more aggressively. This is not a limitation of lookahead search per se — it is a limitation of the verifier signal that all search methods depend on.

What evidence exists in the paper. Figure 3 (left) in the base paper shows beam search performance flattening and falling slightly below best-of-N at high budgets (64–256 generations). Figure 3 (right) shows beam search degrading on bin 1 (easy problems) as budget increases — the clearest signature of verifier exploitation. Appendix M provides qualitative examples of degenerate search outputs. The fact that the "last" PRM score aggregation (which effectively turns the PRM into an ORM at decision time) outperforms "min" and "prod" aggregation (Appendix E) further suggests that the PRM's per-step scores are not reliable enough to guide aggressive search.

Mitigation status. The compute-optimal policy addresses this by routing easy problems away from beam search, but this is a workaround, not a solution. The paper does not propose any verifier robustness improvements — the PRM is used as-is, trained via Monte Carlo rollouts. Improving verifier calibration, using ensemble verification, or applying KL-constrained search (penalizing deviations from the base model's output distribution) are mentioned as future directions but not explored. Until verifier quality improves, test-time compute scaling will remain bounded by over-optimization on all difficulty levels, not just the easy ones.

7. Implications and Future Directions

How This Work Changes the Landscape

ERNIE 5.0 represents not a single-algorithm breakthrough but a systems-level demonstration that changes what the field considers architecturally possible for multimodal foundation models. Its primary impact is to reframe the dominant late-fusion paradigm — where language models are augmented with modality-specific encoders and decoders — from an engineering necessity to an architectural choice, and a potentially suboptimal one at that. By demonstrating that a single autoregressive model trained from scratch on all modalities under a unified objective can match or exceed specialized systems across text, vision, and audio while supporting both understanding and generation, the paper shifts the burden of proof: future multimodal architectures must now justify why they wouldn't use unified training, rather than why they would.

This is a reframing rather than a paradigm shift. The idea of unified autoregressive multimodal models existed in the research literature before ERNIE 5.0 — the paper builds on autoregressive image generation (Chen et al., 2024a), neural audio codecs (Kumar et al., 2023), and prior efforts at unified tokenization. What ERNIE 5.0 contributes is the existence proof at scale: that the approach is not merely conceptually elegant but engineering-feasible at trillion-parameter production scale. Prior to this paper, a reasonable practitioner could argue that unified multimodal autoregressive training was theoretically attractive but practically impossible — that the conflicting optimization dynamics of text, vision, and audio, combined with the sequence length explosion from visual tokens and the instability of ultra-sparse MoE training, would prevent convergence at scale. ERNIE 5.0 provides counterevidence: not only does training converge, but the resulting model is competitive with specialized systems. This transforms unified autoregressive models from a research curiosity into a credible production architecture.

The paper also resolves a latent tension in the multimodal literature that has persisted since the emergence of large vision-language models. Late-fusion approaches consistently reported that adding modalities degraded core language performance — the "ability seesaw" — leading researchers to invest heavily in catastrophic forgetting mitigation, multi-stage training curricula, and careful modality mixing schedules. ERNIE 5.0 suggests, through its pre-training results (ERNIE 5.0-Base at 90.58 MMLU, 88.4 GenEval, and competitive ASR WERs simultaneously, Tables 1, 4, 5, 7), that this seesaw is an artifact of sequential training, not a fundamental trade-off. When all modalities are co-trained from scratch, cross-modal knowledge transfer appears to be net positive: the model benefits from seeing images while learning language, and from learning speech while learning text. This doesn't eliminate the need for careful data mixing — Section 3.2 describes posterior-based loss weighting precisely to prevent modality imbalance — but it reframes the optimization problem from "how do we prevent modalities from interfering?" to "how do we ensure modalities mutually reinforce?"

Several research directions become more attractive as a direct consequence of this paper:

• Architectural simplification becomes a defensible research goal. The paper demonstrates that modality-agnostic expert routing (no modality-specific expert pools), a single autoregressive objective (no separate diffusion, GAN, or reconstruction losses), and a unified positional encoding (Uni-RoPE) can work at scale. This suggests that further architectural simplification — eliminating modality-specific prediction heads, unifying tokenizers across modalities, or removing the cascaded diffusion refiner in favor of higher-resolution autoregressive generation — may be tractable and worth pursuing.

• Verifier robustness for over-optimization becomes the critical bottleneck for test-time compute scaling, as established in the base paper's analysis. The ERNIE 5.0 reinforcement learning results reinforce this from the post-training side: the MISC technique (Section 4.2) is essentially a verifier-robustness solution for RL training, addressing the mismatch between training and inference engines that causes importance sampling ratios to become unreliable. The parallel between PRM over-optimization in search and training-inference mismatch in RL suggests a unified research agenda around verifier calibration under distribution shift that spans both inference-time and training-time compute optimization.

• Elastic training as a pretraining primitive becomes credible at trillion-parameter scale. Prior once-for-all work existed at much smaller scales (Devvrit et al., 2023; Cai et al., 2024) and was not demonstrated for multimodal models. ERNIE 5.0's elastic training results — particularly ERNIE 5.0-Exp-EA35.8% achieving 75.17 average vs. 75.55 full (Table 12) — establish that sub-network co-training does not catastrophically degrade full-model performance. This opens the door to treating deployment flexibility as a first-class training objective, with pretraining budgets explicitly allocated to producing model families rather than single models.

Conversely, some research directions become less attractive:

• Modality-specific expert partitioning (as in ERNIE 4.5) is now empirically challenged. The modality-agnostic routing visualizations (Figures 8–10) show that experts develop clear modality specialization without manual partitioning, and that cross-modality expert overlap increases in deeper layers as representations become more semantic. This suggests that manual expert allocation is not just unnecessary but potentially harmful — it prevents the emergent cross-modal knowledge sharing that the paper documents. Future MoE multimodal models have a strong default toward modality-agnostic routing.

• Post-hoc model compression pipelines (pruning, distillation, quantization as separate stages) face a more attractive alternative. If elastic training can produce deployable sub-models at 35.8% of total parameters with near-identical performance (Table 12), the separate train-then-compress pipeline — with its dedicated infrastructure, quality degradation, and inability to flexibly produce different-sized variants — becomes harder to justify, at least for organizations that can afford the elastic training overhead during pretraining.

• Separate generative models for different modalities — the paradigm where image generation uses a diffusion model, speech synthesis uses a dedicated TTS system, and text generation uses an autoregressive LM — faces an existence proof that unification is possible. While ERNIE 5.0 still uses a decoupled diffusion refiner for high-resolution images (Section 2.2.3), the core generation capability is autoregressive, and the refiner is a resolution-enhancement module, not a separate generative model. As tokenizers improve (higher compression, better fidelity), the case for entirely separate generation architectures weakens.

Follow-Up Research This Work Enables

1. Quantifying the compute-equivalence of elastic pre-training vs. independent training. The paper demonstrates that elastic training produces functional sub-models (Table 12), but does not establish whether elastic pre-training is more efficient than training each sub-model size independently. A controlled experiment would train three families of models at matched total FLOPs: (a) one elastic super-network producing sub-models at sizes S, M, L; (b) three independently trained models at sizes S, M, L, each using 1/3 of the total FLOPs; (c) one large model at size L with post-hoc compression to S and M. The key metric is the Pareto frontier of accuracy vs. inference cost for each family. The paper's claim that elastic training "substantially lowers overall computational overhead and engineering complexity" (Section 3.3) is reasonable but unquantified — a FLOPs-matched comparison would reveal whether the overhead of training the super-network (which must activate different sub-networks on different batches, potentially slowing convergence) is offset by parameter sharing across sub-models. The controlled experiments (Tables 9–11) provide validation loss for individual elasticity dimensions at small scale but do not compare against size-matched independent training.

2. Characterizing the optimal elastic training schedule across depth, width, and sparsity dimensions. The paper uses 75%/25% sampling for elastic depth and 80%/20% for elastic width and sparsity (Section 3.3), but does not ablate these ratios. A systematic study would sweep the sampling probability for each dimension across {0.0, 0.1, 0.2, 0.3, 0.5} and measure both full-model performance and sub-model performance at multiple extraction points. The hypothesis is that there exists a "elasticity budget" — total deviation from full-model training — that can be optimally allocated across dimensions. Too little elasticity (e.g., 95%/5%) may produce sub-models that degrade sharply under parameter removal; too much (e.g., 50%/50%) may slow full-model convergence unacceptably. The paper's fixed 75–80% full-configuration rates are likely tuned empirically but without reported ablations, leaving open the question of whether these are near-optimal or simply "good enough" for the scale of ERNIE 5.0. Additionally, the relationship between elasticity dimensions may be complementary or competitive — does elastic depth + elastic width produce better sub-models than either alone, or do they interfere? The ERNIE 5.0-Exp-EA35.8% result combines all three dimensions, but without ablations varying which dimensions are active, the contribution of each cannot be isolated.

3. Does modality-agnostic routing outperform modality-isolated routing on a fixed benchmark suite when controlling for total expert count? The paper's visualizations (Figures 8–10) show that modality-agnostic routing leads to emergent specialization, but no quantitative comparison is provided against a modality-isolated baseline (e.g., ERNIE 4.5's approach). A controlled experiment would train two MoE models with identical architecture (same total experts, same layers, same training data) differing only in routing: one with modality-agnostic routing (all tokens access all experts) and one with modality-partitioned routing (e.g., 40% experts dedicated to text, 30% to vision, 20% to audio, 10% shared). The comparison would measure: (a) overall benchmark performance across modalities, (b) cross-modal transfer efficiency (how much does training on image data improve text performance?), (c) expert utilization entropy (is modality-isolated routing more or less balanced?), and (d) robustness to modality imbalance in training data (does modality-agnostic routing handle a 90% text / 10% vision training mix better than isolated routing?). This experiment would distinguish whether modality-agnostic routing is genuinely beneficial or merely "not harmful" — an important distinction for practitioners deciding whether to adopt the approach.

4. Training a difficulty predictor model to close the estimation cost gap for compute-optimal inference. The base paper's compute-optimal test-time scaling is bottlenecked by the cost of difficulty estimation — generating 2048 samples per question dwarfs any plausible inference budget. A natural follow-up would train a lightweight difficulty predictor that maps a prompt (text, image, or multimodal input) directly to a difficulty bin, using only a single forward pass. The training data is available from the paper's own methodology: the 2048-sample pass@1 estimates and PRM score distributions computed for each question in the MATH training set. A small model (e.g., a few hundred million parameters) could be trained to predict the difficulty quintile from the prompt embedding, using the 2048-sample estimates as ground-truth labels. The key evaluation would be: (a) predictor accuracy vs. the 2048-sample "oracle" difficulty bin, (b) downstream compute-optimal scaling performance when using predicted vs. oracle difficulty, and (c) total inference cost (prediction + strategy execution) vs. uniform best-of-N at matched accuracy. If a predictor with, say, 90% bin accuracy enables compute-optimal scaling that still achieves 3–4× efficiency over best-of-N while adding negligible per-query cost, the compute-optimal framework becomes immediately practical.

5. Combining PRM-guided tree search with the revision model as the proposal distribution. The base paper studies search against PRM verifiers and iterative revisions as independent mechanisms, but ERNIE 5.0's unified architecture — where text, vision, and audio generation share a single backbone — opens a new combination: using the revision model (which conditions on its own previous outputs to produce improved responses) as the proposal distribution within beam search. At each step of the search tree, rather than sampling from the base model independently, the model could condition on the partial solution path (including rejected branches) as revision context, potentially producing higher-quality candidate steps. The PRM would score partial solutions at each step, guiding expansion and pruning. The key comparison is: does revision-augmented beam search outperform (a) pure beam search with the base model, (b) pure sequential revisions, and (c) the compute-optimal adaptive combination of both? This experiment would test whether the paper's two main test-time compute mechanisms are complementary or redundant when combined. The ERNIE 5.0 architecture is particularly well-suited for this because the revision model and base model are the same network (just with different prompting/fine-tuning), unlike systems where the verifier and generator are separate models with different architectures.

6. Stress-testing elastic training under extreme distribution shift in post-training. The ERNIE 5.0-Exp-EA35.8% result (Table 12) shows that an elastically-trained sub-model, after mid-training and post-training, nearly matches the full model. But what happens when the post-training distribution differs substantially from pre-training? A stress test would take the elastic sub-model (extracted before mid-training) and fine-tune it on: (a) a domain far from the pre-training distribution (e.g., specialized scientific literature, low-resource languages, or a new modality not seen during pre-training), (b) a task requiring capabilities that the full model barely solves (difficulty bin 4–5 problems from the base paper's taxonomy). The comparison would be against a size-matched model trained from scratch (non-elastic) on the same pre-training data, then fine-tuned identically. The hypothesis is that elastic training may produce representations that are more general (because they had to be useful across multiple sub-network configurations) but potentially less specialized (because no sub-network received dedicated optimization). If the elastic sub-model transfers better to out-of-distribution tasks, it suggests elastic training acts as a regularizer that improves generalization; if it transfers worse, it suggests elastic training trades specialization for flexibility — an important characterization for practitioners choosing between elastic and independent training.

Practical Applications and Downstream Use Cases

1. Unified multimodal product APIs with flexible deployment footprints. A cloud provider or AI platform could offer ERNIE 5.0-style unified models that replace separate APIs for text generation, image generation, speech recognition, and text-to-speech. The practical benefit is not just consolidation of infrastructure (one model server instead of four) but also cross-modal consistency: the same model that generates an image from a text description can answer questions about that image, edit it based on new instructions, and describe it aloud — all within a single inference session, without routing between separate models that may have inconsistent understanding of the content. The elastic training dimension enables the provider to offer the same model at multiple price-performance tiers: a "full" tier using all experts and layers, a "fast" tier using the ES25.0% configuration with >15% lower latency (Table 12), and an "economy" tier using the EA35.8% compact variant — all from a single training run, with no per-tier model maintenance. The specific numbers from the paper: the economy tier achieves 75.17 average score vs. 75.55 for the full tier across representative benchmarks, representing a ~65% total parameter reduction with negligible quality loss.

2. On-device deployment of multimodal assistants with elastic sub-models extracted from cloud-trained super-networks. A smartphone manufacturer or edge device vendor could participate in the elastic training of a large cloud model (contributing compute or data), then extract a device-appropriate sub-model for on-device inference. The key advantage over current approaches (where on-device models are either independently trained or derived through post-hoc distillation) is that the elastic sub-model inherits knowledge from the full model's training without requiring a separate distillation stage. The ERNIE 5.0-Exp-EA35.8% result demonstrates that a model using 35.8% of total parameters can remain competitive across benchmarks — for a hypothetical trillion-parameter super-network, this would produce a ~358B parameter deployable model, which is within the range of what can run on high-end mobile devices or consumer GPUs. The modality-agnostic routing further benefits on-device deployment: because the router learns to allocate experts based on computational need rather than modality, the same device can handle text, image, audio, and video tasks without modality-specific model swapping. The practical impact is a single on-device model for all multimodal interactions, updated by extracting a new sub-model from the latest cloud-trained super-network rather than running separate fine-tuning or distillation pipelines for each modality.

3. Self-improving multimodal data generation pipelines. Organizations building large multimodal datasets — for training captioning models, visual question answering systems, or multimodal dialogue agents — could use an ERNIE 5.0-style unified model to generate cross-modally consistent synthetic data. Because the same model understands and generates all modalities, it can produce an image, generate a detailed caption for that image, answer questions about it, and describe it in speech — all from a coherent internal representation. This avoids the inconsistency problem where an image generator, a captioning model, and a VQA model (all separate systems) produce incompatible outputs because they have different "understandings" of the same content. The compute-optimal framework from the base paper (difficulty-conditioned allocation) could guide the generation budget: easy data generation tasks (e.g., captioning simple images) receive minimal compute, while hard tasks (e.g., generating complex reasoning chains about diagrams) receive beam search or revision-augmented generation. The elastic training dimension enables the data generation pipeline to scale its compute usage based on available resources — running the full model for high-quality data, or elastic sub-models for rapid prototyping.

4. Research infrastructure for multimodal architecture ablation studies. The paper's detailed documentation of infrastructure choices — hybrid parallelism strategies (Section 5.1), disaggregated tokenizer-backbone architecture (Section 5.2), FlashMask for flexible attention (Section 5.3), and the disaggregated RL infrastructure (Section 5.4) — provides a reference design for other organizations building large-scale multimodal training systems. While the paper omits specific model dimensions and training compute, the infrastructure architecture is described in sufficient detail to guide implementation: the combination of 4-way tensor parallelism, 12-way pipeline parallelism with virtual stages, 64-way expert parallelism, ZeRO-1 data parallelism, and DeepEP for inter-node communication is a concrete configuration that other teams can adapt to their hardware. The memory optimization techniques — FP8 mixed-precision, dynamic adaptive activation offloading, sub-batch computations, automatic memory defragmentation — address common failure modes in large-scale MoE training that are not specific to ERNIE 5.0's architecture. A team training a smaller multimodal MoE model (e.g., 100B–500B parameters) could adopt this infrastructure blueprint directly, reducing the engineering effort required to go from single-node prototyping to distributed training.