Hunyuan-Large: An Open-Source MoE Model with 52 Billion Activated Parameters by Tencent

🎯 Pitch

A 389B-parameter Mixture of Experts model with only 52B activated parameters scores 69.8 on MATH while LLama3.1-405B manages just 53.8—a colossal leap in reasoning for 8× less compute. It also crushes the 405B model on instruction-following benchmarks, proving that clever routing and synthetic data can let a comparatively tiny active footprint punch far above its weight.

1. Executive Summary

This paper introduces Hunyuan-Large, an open-source Mixture of Experts model with 389 billion total parameters and 52 billion activated parameters capable of handling up to 256K tokens. The model is evaluated against LLama3.1-70B and LLama3.1-405B across language understanding, reasoning, mathematics, coding, and long-context benchmarks, outperforming the former and matching the latter despite having significantly fewer activated parameters. Key named mechanisms include high-quality synthetic data (a four-step pipeline producing 1.5T tokens of instruction-response pairs for pre-training) and a mixed expert routing strategy (a shared expert for common knowledge combined with top-1 specialized expert routing, augmented by recycle routing that reassigns tokens from overloaded experts rather than discarding them), alongside KV cache compression (Grouped-Query Attention plus Cross-Layer Attention, reducing cache memory by ~95% compared to standard multi-head attention) and expert-specific learning rate scaling (adjusting per-expert learning rates based on effective batch size differences between shared and specialized experts). Hunyuan-Large's pre-trained model achieves 69.8 on MATH versus LLama3.1-405B's 53.8 (a 16-point absolute gain with ~8× fewer activated parameters), while the instruction-tuned variant posts 81.8 on Arena-Hard compared to LLama3.1-405B-Instruct's 69.3, establishing that MoE architectures with carefully engineered routing, synthetic data, and training recipes can match or exceed dense models substantially larger in activated parameter count on both capability and alignment benchmarks.

2. Context and Motivation

The Core Problem: The Open-Source MoE Gap

The central problem this paper addresses is straightforward but strategically significant: no open-source mixture of experts (MoE) model exists at the scale of the largest dense models, and the community lacks a public blueprint for building one that actually works better than those dense counterparts. The paper's abstract frames this directly — Hunyuan-Large is "currently the largest open-source Transformer-based mixture of experts model," with 389B total parameters and 52B activated. The key phrase is open-source. Models at this scale exist behind closed doors (the paper itself references Tencent's trillion-parameter internal model powering the Yuanbao chatbot since February 2024), but the recipes, the scaling laws, the engineering decisions, and — most critically — the weights are unavailable to researchers and practitioners who want to study, fine-tune, or deploy MoE architectures at scale.

This gap matters for several reasons, some explicit and some implied by the paper's framing:

Dense models dominate the open-source landscape. The LLama series (Touvron et al., 2023; Dubey et al., 2024), Qwen (2024a), and other widely-adopted open models are predominantly dense architectures. While a few open-source MoE models exist — Mixtral-8x22B (Mistral, 2024) at 141B total/39B activated, DeepSeek-V2 (DeepSeek-AI, 2024) at 236B total/21B activated, Jamba (2024) with a hybrid Mamba-transformer design — these are relatively modest in scale compared to the dense frontier. No prior open-source MoE model had pushed into the regime of 50B+ activated parameters, which is where MoE's theoretical advantages over dense architectures are most consequential. If MoE is genuinely more efficient than dense scaling, the evidence should be most visible at the scales where the FLOPs savings are largest — but without a large open-source MoE model, that hypothesis was untestable by the broader community.

The training dynamics of large-scale MoE models are poorly understood. Beyond model availability, the paper identifies a deeper problem: the recipes for training MoE models at scale are under-explored compared to their dense counterparts. While dense models benefit from well-characterized scaling laws (Kaplan et al., 2020; Hoffmann et al., 2022) that guide practitioners in allocating compute to model size versus data volume, the paper's Section 2.3.1 explicitly states that they had to derive MoE-specific scaling laws from scratch — indicating that prior scaling law analyses did not adequately cover the MoE regime with sparse activation, where the compute budget formula differs due to attention complexity interacting with expert sparsity. Similarly, the paper identifies that learning rate scheduling — a solved problem for dense models — requires reconsideration for MoE because different experts see different effective batch sizes. The shared expert processes every token; each specialized expert sees roughly 1/16th of tokens (since only top-1 out of 16 is activated per token). A uniform learning rate across all experts is therefore suboptimal, but no prior work had articulated this principle or provided a formula for the correction (Section 2.2.4).

Synthetic data quality and quantity limits at scale. The paper positions its 1.5T tokens of synthetic pre-training data as "orders larger than in previous literature" (Abstract). This is a specific engineering gap: prior models have used synthetic data to augment pre-training (Dubey et al., 2024; Abdin et al., 2024; Liu et al., 2024), but the scale, quality-control methodology, and domain coverage needed to make synthetic data a substantial fraction of a 7T-token pre-training corpus were not established. The paper's four-step synthesis pipeline (instruction generation → evolution → response generation → filtering) represents a systematization and scaling-up of techniques that previously existed only in smaller-scale or proof-of-concept forms.

KV cache as a deployment bottleneck for large models. The paper's emphasis on KV cache compression (Section 2.2.2) targets a practical deployment problem that becomes acute at the scale of 389B-parameter models with long contexts. Standard multi-head attention (MHA) produces a KV cache of size $4n_h d_h l$ bytes (in bf16), where $n_h$ is the number of attention heads (80 in Hunyuan-Large), $d_h$ is the head dimension, and $l$ is the number of layers (64). For a model of this scale with a 256K-token context, the raw MHA KV cache would be impractically large for most deployment hardware. The paper frames GQA+CLA as achieving ~95% reduction — bringing KV cache memory from $4n_h d_h l$ to $2n_g d_h l$ (where $n_g = 8$ groups, compared to $n_h = 80$ heads) — which is the difference between deployability and infeasibility for many practitioners.

Why This Problem Matters

The importance of closing the open-source MoE gap extends across several dimensions:

Computational efficiency of inference. MoE models activate only a fraction of their total parameters per token. Hunyuan-Large activates 52B out of 389B parameters — roughly 13.4%. If this model matches or exceeds the performance of a 405B-parameter dense model (LLama3.1-405B), it does so using approximately 7.8× fewer activated parameters per forward pass. This translates directly to lower inference latency, reduced memory requirements, and lower serving costs for comparable capability levels. For organizations running LLM inference at scale, this efficiency proposition is economically compelling — but only if the engineering challenges of MoE training and deployment can be solved with publicly available recipes.

Democratizing frontier-scale models. A model with 389B total parameters cannot realistically be served from a single consumer GPU, but with effective sharding and the KV cache compression techniques the paper describes, its deployment footprint becomes substantially more manageable than an equivalent-capability dense model. By releasing weights, code, and detailed methodology, the paper enables academic labs, startups, and research groups without vast compute resources to study, analyze, and build upon a frontier-scale architecture. This is significant not just for direct use but for downstream research: understanding failure modes, biases, and emergent behaviors of large-scale MoE models requires access to the models themselves.

Establishing MoE-specific training laws. The paper's derivation of MoE scaling laws (Section 2.3.1) fills a genuine knowledge gap. Dense model scaling laws (Kaplan et al., 2020; Hoffmann et al., 2022) use $C = 6ND$ as the compute budget formula, where $N$ is total parameters. For MoE models with sparse activation attending to long sequences, this formula is wrong on two counts: (1) only activated parameters contribute to the forward pass computation, and (2) attention complexity grows quadratically with sequence length and becomes non-negligible relative to the feed-forward computation at the long context lengths (8K, 32K, 256K) that Hunyuan-Large targets. The paper's revised formula — $C \approx 9.59ND + 2.3 \times 10^8 D$ , where $N$ is activated parameters — encodes these MoE-specific considerations and provides a framework for future practitioners to reason about compute-optimal MoE training without rediscovering these relationships through expensive trial and error.

Practical guidance on synthetic data at scale. As LLM pre-training increasingly runs into data scarcity constraints, synthetic data generation is becoming a critical capability. The paper's detailed account of how they generated, filtered, and integrated 1.5T tokens of synthetic data (representing over 21% of the total pre-training corpus) provides a blueprint that other teams can adapt. The specific claim that synthetic data targets "the relative capability deficiency merely learned from natural data" (Section 2.1.1) — focusing on mathematics, coding, low-resource domains, and high-educational-value content — encodes a strategic insight: synthetic data is not just a quantity play but a directed intervention to correct the natural distribution's biases.

Where Prior Approaches Fall Short

The paper identifies specific shortcomings in existing approaches, both implicitly through its design choices and explicitly through its comparisons:

Prior open-source MoE models are too small to demonstrate the scaling advantage. Mixtral-8x22B (141B total/39B activated) and DeepSeek-V2 (236B total/21B activated) are the most prominent open-source MoE models available before Hunyuan-Large. While both demonstrated that MoE can be competitive with dense models at their scale, they operate in a regime where the efficiency advantage of MoE is relatively modest. The paper's scaling law analysis (Figure 3) suggests that the optimal number of activated parameters for cost-efficient training is approximately 58.1B — well above the activated parameter counts of either prior model. In other words, the prior models were underscaled relative to the compute-optimal point, making them imperfect testbeds for evaluating MoE's true potential.

Existing routing strategies discard valuable tokens. The paper explicitly identifies a limitation of conventional top-k routing with capacity factors: "tokens of overloaded experts are discarded during training" (Section 2.2.3). The standard approach — used in models like Mixtral (Jiang et al., 2024) — sets a capacity factor that limits the maximum number of tokens each expert can process. Tokens routed to an expert that has reached capacity are simply dropped. This creates a tension: larger capacity factors reduce token loss but decrease training efficiency (since experts sit idle waiting to fill capacity), while smaller capacity factors improve efficiency at the cost of discarding potentially informative tokens. The paper argues this "may cause the loss of crucial information, which in turn negatively impacts training stability" (Section 2.2.3). Prior MoE work had not proposed a mechanism for recovering these dropped tokens — the recycle routing strategy (Figure 2), which randomly reassigns overflow tokens to experts with available capacity, is presented as a new solution to this old problem.

Uniform learning rates ignore expert-level heterogeneity. In a standard MoE training setup, all parameters — shared experts and specialized experts alike — are trained with the same learning rate. The paper argues this is suboptimal because the shared expert processes every token while specialized experts each process approximately $1/16$ th of tokens (in a 1-of-16 setup). The effective batch size differs between expert types by a factor of ~16, and since optimal learning rates depend on batch size (Equation 1, derived from Li et al., 2024a), a uniform learning rate cannot be simultaneously optimal for both shared and specialized experts. The paper quantifies this: the learning rate scaling ratio between shared and specialized experts should be approximately 0.31 (Section 2.2.4). Prior MoE models — including Mixtral and DeepSeek-V2 — did not implement or discuss expert-specific learning rates, meaning they were systematically using suboptimal optimization for one category of expert.

No published MoE scaling laws that account for sparse activation and long sequences. As noted, the compute budget formula for dense models ( $C = 6ND$ ) does not correctly estimate the cost of training MoE models with long-context attention. The paper's derivation of Equation 2 — $C \approx 9.59ND + 2.3 \times 10^8 D$ — accounts for both the reduced forward-pass FLOPs from sparse activation (since only $N$ activated parameters contribute, not the total parameter count) and the attention cost term that becomes significant at long sequence lengths. Without such a formula, practitioners cannot perform the isoFLOPs analyses that guide compute-optimal model sizing. The paper's scaling law exploration (Figures 3 and 4) is presented as filling this gap: training a series of MoE models from 10M to 1B activated parameters on 100B tokens to characterize the relationship between compute budget, activated parameters, and training data volume.

Synthetic data generation at pre-training scale lacks established methodology. Prior work has used synthetic data in pre-training, but the paper suggests that existing approaches fall short of what is needed: either they operate at smaller scale, lack systematic quality control, or fail to target the specific capability gaps that natural data leaves unfilled. The four-step pipeline (Figure 1) with dedicated models for instruction generation, instruction evolution, response generation, and response filtering is presented as a more rigorous and scalable alternative. The paper's emphasis on instruction evolution — making instructions more complex and difficult, not just more numerous — is a specific departure from simpler data augmentation approaches that merely generate more examples at the same difficulty level.

Long-context training recipes are under-explored for MoE. While long-context capability has been studied for dense models (Xiong et al., 2023; Gao et al., 2024), the paper notes that extending these recipes to MoE models — where the interaction between long sequences and expert routing could affect training dynamics — is not straightforward. The paper's long-context pre-training strategy (Section 2.3.3) uses a two-stage approach (32K → 256K) with scaled RoPE base frequency (1 billion for the 256K stage) and a specific data mixture (25% natural long-context data, 75% normal-length data), but prior work had not established whether these choices are appropriate for MoE architectures specifically.

How This Paper Positions Itself

The paper positions Hunyuan-Large not merely as a new model but as a reference implementation and methodological guide for large-scale open-source MoE. This positioning is evident in several aspects of the presentation:

The paper emphasizes reproducibility and community contribution. The abstract foregrounds open-sourcing: "The code and checkpoints of Hunyuan-Large are released to facilitate future innovations and applications." The introduction frames the release as part of contributing to the community "in addition to serving users with the premium models." This is a deliberate positioning against the backdrop of closed-source frontier models — the paper implicitly argues that the community benefits from having not just the model but the complete methodology (data synthesis, routing strategies, scaling laws, training recipes) publicly documented and reproducible.

The paper frames its contributions as a comprehensive system, not a single technique. The introduction lists three categories of innovation — synthetic data, enhanced model structure, and MoE scaling laws — and the body of the paper treats each as essential. This contrasts with papers that focus on a single architectural innovation or training trick. The implicit claim is that building a frontier-scale MoE model requires simultaneous attention to all of these components; no single technique suffices. This is consistent with the paper's origin at Tencent, where the internal trillion-parameter model provided experience that the open-source release systematizes and documents.

The paper directly engages with the scaling laws literature. Section 2.3.1 explicitly builds on Kaplan et al. (2020) and Li et al. (2024a) while adapting their frameworks to the MoE setting. The authors fit their own coefficients ( $N_c = 5.9 \times 10^{-3}$ , $\alpha = 0.5305$ for activated parameters; $D_c = 3.2$ , $\beta = 0.50$ for training tokens) and use these to justify specific design choices — 52B activated parameters (close to the computed optimum of 58.1B) and 7T training tokens (slightly above the computed optimum of 5.6T, chosen to "maximize the use of training data within the optimal cost-performance range"). This is a direct engagement with the compute-optimal paradigm from Hoffmann et al. (2022), adapted to MoE.

The paper positions MoE as a practical path to frontier performance, not just a research curiosity. By comparing against LLama3.1-405B — the largest open-source dense model at the time — the paper makes a specific strategic claim: MoE is not just theoretically interesting but is the most practically viable approach to matching the capabilities of the largest dense models while using fewer resources. The result that Hunyuan-Large outperforms LLama3.1-405B on MMLU (88.4 vs. 85.2) and MATH (69.8 vs. 53.8) — while using only 52B activated parameters versus 405B — is a direct empirical argument for MoE's efficiency advantage at scale.

The paper acknowledges its relationship to Tencent's internal closed-source work. The introduction references the "trillion-parameter flagship LLM" running since February 2024 and notes that the mixed routing strategy (shared + specialized experts) was "first introduced in our closed-source trillion-parameter model concurrently to Deepseek v2" (Section 2.2.3, footnote). This positions Hunyuan-Large as a public, scaled-down instantiation of techniques proven at larger scale internally, lending credibility to the design choices while acknowledging that the open-source model is not the largest model the team has built — just the largest they have released.

Where the paper is deliberately conservative. Despite its scale, the paper does not claim to push architectural boundaries. It uses the "classical Transformer architecture with MoE" (Section 2.2.1), SwiGLU activations, RoPE, and the standard shared-plus-specialized expert design. The innovations are in the training methodology (routing, learning rates, data synthesis, scaling laws) rather than in novel architectural primitives. This conservatism is strategic: it maximizes the transferability of the findings, since the innovations can be applied to any standard MoE Transformer without requiring custom kernels or exotic components. It also means the performance gains are attributable to better engineering of known components rather than architectural novelty, which strengthens the paper's implicit claim that MoE's potential has been underexploited due to insufficient attention to training methodology rather than architectural limitations.

In summary, the paper addresses a multi-faceted gap: the absence of a large-scale open-source MoE model, the lack of MoE-specific training recipes and scaling laws, the engineering challenge of deploying models with huge KV caches, and the methodological vacuum around synthetic data at pre-training scale. Its positioning is that of a comprehensive reference — model plus methodology — that demonstrates MoE's practical competitiveness with the largest dense models while providing the community with both the artifact and the blueprint for building on it.

3. Technical Approach

3.1 Reader Orientation

Hunyuan-Large is a 389-billion-parameter Transformer-based mixture-of-experts language model that activates only 52 billion parameters per token, designed to match or exceed the performance of much larger dense models while being more computationally efficient at inference. The paper solves the problem of "how to build, train, and deploy an open-source MoE model at frontier scale" by addressing four interconnected engineering challenges — data quality at scale, expert routing stability, per-expert optimization, and memory-efficient inference — through a combination of large-scale synthetic data generation, novel routing and learning rate strategies, and aggressive KV cache compression. The solution's shape is a comprehensive training and deployment pipeline: synthetic data is generated through a four-step quality-controlled process, the model architecture balances shared and specialized expert processing with a mechanism to recover otherwise-discarded tokens, optimization adapts learning rates to each expert's effective batch size, and attention is compressed at both the head and layer dimensions to make deployment practical at 256K context length.

3.2 Big-Picture Architecture (Diagram in Words)

The Hunyuan-Large system comprises five major components arranged in a sequential pipeline:

Data Pipeline (Section 2.1.1): Takes raw web pages, code repositories, books, and QA data as inputs. Produces 7T tokens of pre-training data, of which 1.5T tokens are synthetically generated through a four-step process (instruction generation → instruction evolution → response generation → response filtering). The data is categorized with an elaborate label system enabling flexible proportion adjustment.
Tokenizer (Section 2.1.2): Converts the 7T-token text corpus into token IDs using a vocabulary of 128K tokens — 100K from the tiktoken tokenizer plus 28K additional Chinese-optimized tokens. Achieves 3.13 characters per token (versus LLama3.1's 2.78), improving compression for more efficient training.
MoE Transformer Architecture (Section 2.2): The core model — 64 layers, 80 attention heads, 8 key-value heads (GQA), hidden size 6,400, SwiGLU activation, RoPE position embeddings. Each layer contains one shared expert (processes all tokens) and 16 specialized experts (one activated per token via top-1 routing). Total: 389B parameters, 52B activated per token.
KV Cache Compression Subsystem (Section 2.2.2): Operates at two levels: Grouped-Query Attention reduces KV heads from 80 to 8 groups, and Cross-Layer Attention shares KV caches every 2 layers. Together, these achieve approximately 95% KV cache memory reduction relative to standard multi-head attention.
Training Orchestrator (Sections 2.2.3–2.3): Manages the pre-training process across 7T tokens using AdamW with expert-specific learning rates (shared expert gets the full optimal rate; specialized experts get approximately 0.31× that rate), recycle routing (overflow tokens from overloaded experts are randomly reassigned rather than dropped), a three-phase learning rate schedule (warmup → gradual decay → annealing), and a two-stage long-context extension (32K → 256K tokens).
Post-Training Pipeline (Section 3): Takes the pre-trained model through supervised fine-tuning (SFT) on 1M+ instruction examples and Reinforcement Learning from Human Feedback (RLHF) via Direct Preference Optimization (DPO) with both offline and online preference data, producing Hunyuan-Large-Instruct.

Information flows sequentially: raw data → tokenizer → MoE Transformer with KV compression and expert routing → training orchestrator applies learning rate schedule and routing decisions → pre-trained model → SFT → DPO → instruct model. During inference, the post-trained model processes input tokens through the same architecture, with each token activating the shared expert plus one specialized expert, and the KV cache compression reducing memory footprint for long contexts up to 256K tokens.

3.3 Roadmap for the Deep Dive

First, the synthetic data pipeline (Section 3.4: Synthetic Data Generation) — because data is "the fuel of our powerful model" (Section 2.1) and the most upstream component. Understanding the four-step process (instruction generation, evolution, response generation, filtering) is foundational, as the quality and diversity of this data directly drives downstream model capabilities in math, coding, and low-resource domains.
Second, the tokenizer design (Section 3.4: Tokenizer Design and Compression) — a brief but necessary component between data and model. The 128K vocabulary with Chinese optimization determines how efficiently text is encoded, impacting training cost.
Third, the expert routing strategy (Section 3.4: Mixed Expert Routing and Recycle Routing) — the heart of the MoE architecture. This covers the shared/specialized expert design, the top-1 routing mechanism, the capacity overload problem, and the novel recycle routing solution. Understanding this explains how the model achieves balanced expert utilization without discarding tokens.
Fourth, KV cache compression (Section 3.4: KV Cache Compression via GQA and CLA) — a deployment-critical component. Grouped-Query Attention and Cross-Layer Attention together reduce KV cache memory by ~95%, making 256K-context inference practical. This explains the engineering that makes the model deployable.
Fifth, expert-specific learning rate scaling (Section 3.4: Expert-Specific Learning Rate Scaling) — a novel optimization insight. Different expert types have different effective batch sizes, requiring different optimal learning rates. This explains the formula and the 0.31× scaling factor.
Sixth, MoE scaling laws (Section 3.4: MoE Scaling Laws and Compute Budget) — the theoretical framework that guided model sizing. This covers the revised compute budget formula accounting for sparse activation and attention cost, the isoFLOPs analysis, and how it yielded the decision of 52B activated parameters and 7T training tokens.
Seventh, the training schedule and long-context extension (Section 3.4: Training Schedule and Long-Context Pre-Training) — the procedural recipe for executing the pre-training, including the three-phase learning rate schedule and the two-stage context length increase.
Eighth, the post-training pipeline (Section 3.4: Post-Training: SFT and DPO) — how the pre-trained model becomes an instruction-following assistant, including the SFT data pipeline (extraction, generalization, balancing, quality control) and the DPO training strategy with offline-online integration.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an engineering and infrastructure paper whose core idea is that building a frontier-scale open-source MoE model requires simultaneous advances across data quality, routing strategy, optimization, and deployment efficiency — no single technique suffices, and the paper's contribution is the integrated system with publicly released weights, code, and methodology.

Synthetic Data Generation

The synthetic data pipeline produces 1.5T tokens of instruction-response pairs for pre-training — over 21% of the total 7T-token corpus — targeting domains where natural text corpora are deficient: mathematics, coding, low-resource languages, and high-educational-value content (Section 2.1.1). The paper positions this not as mere data augmentation but as a directed intervention: synthetic data specifically addresses "the relative capability deficiency merely learned from natural data." The pipeline operates in four sequential stages, each with dedicated model components.

Step 1: Instruction Generation. The system takes diverse high-quality, knowledge-rich sources — web pages, web-based question-answering data, code repositories, books, and other resources — as seeds. These seeds are fed to instruction generation models prompted with "diverse instruction generation prompts" designed to produce instructions covering various domains with different intended styles and complexity levels. The key design choice is using rich-domain seeds rather than blank-slate generation: by anchoring instructions to real content, the system ensures coverage of factual domains that pure generative approaches might miss or hallucinate. This produces a wide but uneven initial instruction set.

Step 2: Instruction Evolution. The initial instructions are refined along three axes:

Clarity and informativeness enhancement: Instructions are rewritten to be more precise about what is being asked, reducing ambiguity that would lead to poor-quality responses downstream.
Low-resource domain augmentation: For domains where initial instruction coverage is sparse, a self-instruct style process generates additional instructions by prompting models to create variations and extensions of existing instructions in those domains.
Difficulty escalation: Instructions are systematically evolved to increase their difficulty levels — for example, a basic arithmetic instruction might be evolved into a multi-step word problem. The paper states these "evolved high-quality and challenging instructions enable our model to benefit more efficiently from synthetic data to cross the original capability boundaries."

This step is the most methodologically distinctive. Rather than simply generating more instructions at a uniform difficulty level (which would saturate the model's learning), the evolution process creates a curriculum of increasing complexity, allowing the model to learn progressively more sophisticated reasoning patterns from the synthetic data.

Step 3: Response Generation. Specialized models — "several specialized models" of "varying sizes" that are "well-designed" for specific domains — produce answers to the evolved instructions. The paper is deliberately vague about the exact models used (likely proprietary internal models), but the architectural principle is important: using domain-specialized generators rather than a single general-purpose model ensures expert-level response quality across mathematics, coding, and other specialized domains. A single model might produce adequate answers across all domains, but specialized models can produce responses that reflect deeper domain expertise.

Step 4: Response Filtering. Two quality control mechanisms operate on the synthetic instruction-response pairs:

Critique model evaluation: A dedicated critique model (likely similar to the one described later for SFT data quality control, which is based on a 70B Hunyuan dense model) scores the quality of generated responses on dimensions the paper does not enumerate for pre-training data but likely assesses (analogous to the SFT critique model's assessment of "accuracy, relevance, completeness, usefulness, and clarity").
Self-consistency filtering: For tasks with objectively verifiable answers (e.g., math problems, factual QA), the system generates multiple answers to the same instruction and retains only those where the answers are consistent. This eliminates responses where the generator produced a plausible but incorrect answer — a critical filter since synthetic data errors during pre-training would teach the model incorrect reasoning patterns.

Design rationale. The four-step pipeline is designed to maximize three properties the paper identifies as essential: quality (no incorrect or misleading examples in the training data), diversity (coverage across domains, languages, and difficulty levels), and quantity (1.5T tokens requires industrial-scale generation). The explicit separation of instruction generation from response generation and filtering allows each step to be optimized independently, and using specialized models at each stage rather than a single end-to-end generator provides more control points for quality assurance.

Tokenizer Design and Compression

The tokenizer converts raw text into sequences of token IDs that the model can process, and its design directly impacts training and inference efficiency through the compression rate — the number of characters represented per token on average (Section 2.1.2). Higher compression means fewer tokens for the same text, reducing both training FLOPs and inference latency.

Hunyuan-Large uses a vocabulary of 128K tokens, constructed as the union of:

100K tokens from the tiktoken tokenizer (OpenAI, 2023), providing broad coverage of English and code.
28K additional tokens specifically designed for Chinese language support, expanding coverage of Chinese characters, common bigrams/trigrams, and domain-specific terminology.

The paper reports that this tokenizer achieves a compression rate of 3.13 characters per token, compared to LLama3.1's 2.78 characters per token — a 12.6% improvement. For a fixed Chinese-language corpus, this represents approximately 12.6% fewer tokens processed during training and inference, translating directly to proportional FLOPs and latency savings.

Design tradeoff. The paper explicitly acknowledges the tension in tokenizer design: "achieving a high compression rate for efficient training and inference" versus "maintaining an appropriately large vocabulary to ensure adequate learning of each word embedding." A vocabulary that is too small forces common words to be split into multiple subword tokens (hurting efficiency and making it harder for the model to learn word-level semantics), while a vocabulary that is too large means each token embedding receives insufficient training signal because tokens appear too rarely. The 128K size and the hybrid construction (general-purpose tiktoken base plus Chinese-specific extension) represent an empirical compromise between these competing requirements for a bilingual English-Chinese model.

Mixed Expert Routing and Recycle Routing

The mixture-of-experts architecture replaces the standard feed-forward network (FFN) in each Transformer layer with multiple parallel FFN "experts," where each input token is processed by only a subset of these experts (Section 2.2.3). Hunyuan-Large uses a mixed routing strategy comprising one shared expert and sixteen specialized experts, with each token activating the shared expert plus exactly one specialized expert selected via top-1 routing.

Shared vs. specialized expert design. The shared expert is "consumed by all tokens," meaning every token passes through this FFN regardless of routing decisions. Its purpose is to "capture the common knowledge required by all tokens" — the linguistic and world knowledge that is broadly applicable across all inputs, such as grammar, common vocabulary, and general reasoning patterns. The specialized experts "dynamically learn domain-specific knowledge," with each expert presumably specializing in different types of content (mathematical reasoning, code structure, factual knowledge, etc.) through the training process. The routing mechanism assigns each token to the single specialized expert whose parameters are computed to be most appropriate for that token.

Top-1 routing mechanism. For each token at each MoE layer, a learned router (a small linear layer) computes a score for each of the 16 specialized experts. The token is then dispatched to the expert with the highest score. This is top-1 routing: exactly one specialized expert is activated per token, alongside the always-active shared expert. The total activated FFN computation per token is therefore the shared expert plus one specialized expert, rather than all 16 specialized experts — this is the source of MoE's computational efficiency. With hidden size 6,400, the per-token FFN cost is approximately 2 FFNs worth of computation rather than 17, a roughly 8.5× savings in FFN FLOPs relative to activating all experts.

The capacity overload problem. Standard MoE training imposes a capacity factor on each expert — a limit on how many tokens can be routed to a given expert in a single training batch. The capacity factor is defined as:

$\text{capacity} = \text{capacity\_factor} \times \frac{\text{tokens\_per\_batch}}{\text{num\_experts}}$

When more tokens are routed to an expert than its capacity, the excess tokens are traditionally dropped — they bypass the FFN layer entirely, losing the information that would have been processed. This creates a tradeoff the paper explicitly identifies:

Larger capacity factors reduce dropped tokens but decrease training efficiency, because experts sit idle waiting to fill their allocated capacity (and the capacity must be provisioned for the worst-case load imbalance).
Smaller capacity factors improve training throughput but risk discarding important tokens, which "may cause the loss of crucial information, which in turn negatively impacts training stability."

Recycle routing strategy. The paper proposes a new mechanism to resolve this tradeoff. When a token is routed to an expert that has already reached its capacity, instead of being dropped, the token is randomly reassigned to another specialized expert that has not yet exceeded its capacity (Figure 2). Specifically:

The router computes scores and selects the top-1 expert for each token as usual.
Tokens assigned to experts that are below capacity are processed normally.
For tokens assigned to an overloaded expert (capacity reached), the system randomly selects a different specialized expert from the set of experts still under capacity and routes the token there instead.
If all experts are at capacity, the paper does not specify the fallback behavior (presumably tokens would be dropped at that point, but the capacity factor is tuned to make this rare).

The key insight is that while the randomly reassigned expert may not be optimal for that token, it is almost certainly better than dropping the token entirely — and random reassignment avoids the computational cost of computing second-choice expert scores while still preserving the token's information flow through the FFN layer. The paper states this approach "strives to preserve vital information while simultaneously optimizing training efficiency."

Relationship to prior work. The paper notes the mixed routing strategy (shared + specialized experts) was "first introduced in our closed-source trillion-parameter model (training starts from November, 2023) concurrently to Deepseek v2" (footnote, Section 2.2.3). The recycle routing strategy appears to be novel — no prior citation is given, and the paper presents it as a new solution to the capacity overload problem. Standard top-k routing with capacity factors (as used in Mixtral, Jiang et al., 2024, and Switch Transformers, Fedus et al., 2022) drops overflow tokens; recycle routing recovers them.

KV Cache Compression via GQA and CLA

During autoregressive inference, Transformer models cache the key and value tensors from all previous token positions to avoid recomputing them for each new token — this is the KV cache. For a model with 80 attention heads, 64 layers, and a 256K-token context, the KV cache memory can dominate the total inference memory footprint, making deployment on typical hardware infeasible. The paper applies two complementary compression techniques (Section 2.2.2).

Grouped-Query Attention (GQA). Standard Multi-Head Attention (MHA) maintains separate key and value projections for each of the $n_h$ attention heads. The KV cache size for MHA is:

$4 n_h d_h l \text{ bytes}$

where $n_h = 80$ is the number of attention heads, $d_h$ is the dimension per head (hidden_size / n_h = 6,400 / 80 = 80), $l = 64$ is the number of layers, and the factor 4 accounts for 2 bytes per bf16 value times 2 tensors (key and value). Plugging in: $4 \times 80 \times 80 \times 64 = 1,638,400$ bytes per token position — at 256K tokens, this becomes approximately 419 GB just for the KV cache.

GQA reduces the number of key-value head groups to $n_g$ groups, where $n_g < n_h$ . The query heads remain at $n_h = 80$ , but keys and values are shared within groups, so only $n_g = 8$ distinct key-value projections exist. The GQA KV cache becomes:

$4 n_g d_h l \text{ bytes}$

This is a factor of $n_h / n_g = 80 / 8 = 10\times$ reduction from the head dimension — from 419 GB to approximately 41.9 GB at 256K context.

Cross-Layer Attention (CLA). CLA compresses the KV cache from the layer dimension by sharing KV caches between adjacent layers. In Hunyuan-Large, KV caches are shared every 2 layers — layer $i$ and layer $i+1$ use the same key-value tensors, halving the total number of distinct KV caches stored. The CLA KV cache (without GQA, for clarity) would be:

$2 n_h d_h l \text{ bytes}$

since only $l/2 = 32$ distinct KV caches are stored rather than 64. The factor of 2 replaces the factor of 4 in the MHA formula because the cache is shared across layer pairs.

Combined GQA+CLA. Hunyuan-Large uses both techniques simultaneously, reducing the KV cache to:

$2 n_g d_h l \text{ bytes}$

where $n_g = 8$ , $d_h = 80$ , and $l = 64$ . This is $2 \times 8 \times 80 \times 64 = 81,920$ bytes per token — a factor of $4 n_h d_h l / (2 n_g d_h l) = 2 n_h / n_g = 2 \times 80 / 8 = 20\times$ reduction from standard MHA. The paper reports approximately 95% reduction: $1 - 1/20 = 0.95 = 95\%$ . At 256K tokens, the total KV cache is approximately 21 GB rather than 419 GB — a practically meaningful difference for deployment on 8×A100 or 8×H100 nodes.

Design rationale. The paper positions this as addressing a practical deployment necessity: "alleviating memory pressure of KV cache and reducing the cost during inference." The specific choice of $n_g = 8$ (rather than the extreme $n_g = 1$ of Multi-Query Attention) and CLA sharing every 2 layers (rather than every 4 or 8) represents a compromise: the paper states these values are chosen "jointly considering both effectiveness and efficiency." More aggressive compression (MQA with CLA every 4 layers) would further reduce memory but could degrade model quality by limiting the attention mechanism's expressivity. The paper does not report ablation studies on these hyperparameters but implies they were empirically validated against model quality metrics.

Expert-Specific Learning Rate Scaling

In standard LLM training, a single learning rate is applied to all parameters. The paper argues this is suboptimal for MoE models because different types of experts process different numbers of tokens per training step, leading to different effective batch sizes — and optimal learning rate depends on batch size (Section 2.2.4).

The effective batch size problem. In a single training iteration with batch size $B$ (number of tokens in the batch), the shared expert processes all $B$ tokens, so its effective batch size is $B$ . Each specialized expert, by contrast, processes approximately $B/n$ tokens on average, where $n = 16$ is the number of specialized experts and exactly one is activated per token. The specialized expert's effective batch size is roughly $B/16$ . Since the shared expert sees 16× more tokens per iteration, its parameter updates are based on 16× more gradient signal per step.

The optimal learning rate formula. The paper adopts the relationship from Li et al. (2024a) between optimal learning rate $\epsilon_{\text{opt}}$ and batch size $B$ :

$\epsilon_{\text{opt}}(B) = \frac{2\epsilon_{\text{max}}}{\sqrt{\frac{B_{\text{noise}}}{B}} + \sqrt{\frac{B}{B_{\text{noise}}}}}$

where $\epsilon_{\text{max}}$ represents the maximum learning rate achievable (the peak of the learning rate schedule), and $B_{\text{noise}}$ indicates the trade-off point between training speed and data efficiency — the batch size at which the gradient noise scale transitions from being dominated by per-sample variance to being dominated by batch averaging.

What it computes: Given a batch size $B$ , the formula outputs the learning rate that optimally balances training speed (larger steps from larger batches) against data efficiency (noise averaging that reduces the benefit of further batch-size increases). The denominator has two terms under square roots: $\sqrt{B_{\text{noise}}/B}$ dominates when $B$ is small (gradient noise is high, smaller learning rates are needed), and $\sqrt{B/B_{\text{noise}}}$ dominates when $B$ is large (gradients are well-averaged, larger learning rates become beneficial but with diminishing returns).

Why this form: The formula captures a U-shaped relationship: at very small batch sizes, increasing batch size allows higher learning rates (noise reduction benefit); at very large batch sizes, further increases in batch size provide diminishing returns (the $\sqrt{B/B_{\text{noise}}}$ term grows, reducing the optimal learning rate). This is consistent with empirical findings that optimal learning rates do not scale linearly with batch size indefinitely.

Applying the formula to MoE. The shared expert has batch size $B$ , giving optimal learning rate $\epsilon_{\text{opt}}(B)$ . Specialized experts (activated for roughly $1/16$ of tokens) have effective batch size $B/n$ (approximately $B/16$ ), giving optimal learning rate $\epsilon_{\text{opt}}(B/n)$ . The paper computes the ratio between these:

$\text{scaling ratio} = \frac{\epsilon_{\text{opt}}(B/n)}{\epsilon_{\text{opt}}(B)}$

and reports this ratio as approximately 0.31 under their specific setting. The shared expert therefore receives the full optimal learning rate for batch size $B$ , while each specialized expert's learning rate is multiplied by approximately 0.31.

Assumptions and caveats. The paper explicitly notes the approximation used: "Considering the load balance losses, we could safely assume that different specialized experts have approximately similar numbers of effectively trained tokens." In practice, routing is not perfectly uniform — some specialized experts may receive more tokens than others due to data distribution skew. Auxiliary load-balancing losses (standard in MoE training, though not detailed in the paper) push the router toward uniform expert utilization, making the $B/n$ approximation reasonable but not exact. Without this assumption, each expert would need its own individually computed batch size, adding complexity without proportional benefit.

Why this matters. A uniform learning rate across all experts would be simultaneously too fast for specialized experts (causing unstable training from updates based on insufficient gradient averaging) and too slow for the shared expert (underutilizing its well-averaged gradients). The expert-specific scaling addresses this mismatch, which the paper claims "contributes to the overall performance" — though no ablation comparing with/without expert-specific learning rates is provided, the theoretical justification is sound.

MoE Scaling Laws and Compute Budget

Before training the full Hunyuan-Large model, the team conducted a systematic scaling law analysis to determine the optimal model size and data volume for a given compute budget (Section 2.3.1). This analysis adapts the isoFLOPs methodology from Hoffmann et al. (2022) to the MoE setting, accounting for two MoE-specific factors: sparse activation (only activated parameters contribute to forward-pass FLOPs) and attention cost (which becomes non-negligible at the long sequence lengths Hunyuan-Large targets).

The MoE compute budget formula. The standard dense model compute budget is $C = 6ND$ , where $N$ is total parameters and $D$ is training tokens. This formula derives from the fact that each training token requires approximately 6 FLOPs per parameter (2 for the forward pass — one multiply and one add per weight — times roughly 3 to account for the backward pass, which requires approximately twice the forward-pass computation). For MoE models with sparse activation and long sequences, the paper derives a revised formula:

$C \approx 9.59 N D + 2.3 \times 10^8 D$

where $N$ is the number of activated parameters (52B for Hunyuan-Large), and $D$ is the number of training tokens.

What it computes: The total floating-point operations required to train a MoE model on $D$ tokens, given $N$ activated parameters. The first term $9.59 N D$ represents the per-token computation that scales with activated parameters, including both the forward pass through activated experts and the backward pass. The factor $9.59$ (rather than the dense model's $6$ ) accounts for the additional computation from the router, the shared expert being always active, and the interaction between attention and expert computation. The second term $2.3 \times 10^8 D$ is a sequence-length-dependent attention cost that does not scale with $N$ — it represents the quadratic attention complexity that becomes significant at the 8K, 32K, and 256K sequence lengths used during training.

Why this form: Two adjustments over the dense formula are critical. First, using activated rather than total parameters: in a dense model, every parameter participates in every forward pass, so $N$ total = $N$ activated. In MoE, only 52B of 389B parameters are activated per token, so the compute cost scales with 52B, not 389B. Second, adding the $2.3 \times 10^8 D$ term: at short sequence lengths, the attention cost $O(L^2)$ where $L$ is sequence length is negligible compared to the FFN cost $O(N)$ . At the 256K-token context length that Hunyuan-Large targets, the attention cost becomes a non-trivial fraction of total compute. The constant $2.3 \times 10^8$ is empirically fitted from the team's small-scale MoE training runs.

Critical batch size adjustment. The paper further adjusts the compute budget to account for batch size effects. The "critical batch size" $B_{\text{crit}}(L)$ is the batch size that optimizes the trade-off between training time (larger batches train faster by parallelizing gradient computation) and computational efficiency (beyond a certain point, larger batches provide diminishing returns in gradient noise reduction). The minimum compute budget is:

$C_{\text{min}} = \frac{C}{1 + \frac{B}{B_{\text{crit}}(L)}}$

where $B$ is the actual batch size used during training.

What it computes: The effective compute budget after accounting for batch size inefficiency. When $B$ is much smaller than $B_{\text{crit}}$ , the denominator is close to 1, and $C_{\text{min}} \approx C$ (no efficiency loss). When $B$ is larger than $B_{\text{crit}}$ , the denominator grows, meaning the effective compute budget is smaller than the raw FLOPs would suggest — the model is not making efficient use of the extra parallelism.

Why this form: This follows from the observation that gradient noise decreases as $1/\sqrt{B}$ for small $B$ but saturates for large $B$ . Training with batch sizes well above $B_{\text{crit}}$ wastes compute because the model could have learned just as much from fewer, better-averaged gradient steps. The critical batch size formalism captures this saturation, allowing the scaling law analysis to use the effective compute budget rather than the raw FLOPs count.

IsoFLOPs analysis and model sizing. The team trained a series of MoE models with activated parameters ranging from 10M to 1B on 100B tokens of pre-training data, then used these to fit the relationship between compute budget and optimal model size. By fitting:

$N_{\text{opt}} = N_c C_{\text{min}}^{\alpha}$

where $N_{\text{opt}}$ is the optimal number of activated parameters for a given compute budget $C_{\text{min}}$ , they obtained $N_c = 5.9 \times 10^{-3}$ and $\alpha = 0.5305$ (Figure 3). Similarly, fitting:

$D_{\text{opt}} = D_c C_{\text{min}}^{\beta}$

yielded $D_c = 3.2$ and $\beta = 0.50$ (Figure 4).

Numerical results. Plugging the planned compute budget into these formulas gives an optimal activated parameter count of approximately 58.1B. The paper chose 52B instead, citing "the smoothness of the quadratic curve around the optimal value" — meaning the loss function is flat near the optimum, so small deviations from 58.1B incur negligible performance penalties. For training tokens, the formula yields 5.6T optimal, but the team chose 7T, "aiming to maximize the use of training data within the optimal cost-performance range to achieve the best possible model outcomes" — deliberately overshooting the compute-optimal data volume to extract maximum capability from the fixed model size at a small additional compute cost.

Design rationale. The scaling law analysis serves as a "guidebook" for model design decisions, preventing expensive mistakes (training too large or too small a model for the available compute) that would be discovered only after months of GPU-time. The specific choice to slightly undershoot on model size (52B vs. 58.1B optimum) while overshooting on data (7T vs. 5.6T optimum) reflects a practical judgment: larger models are harder to deploy (more memory, more sharding complexity), while more data is relatively cheap if you have the data pipeline infrastructure. The scaling laws provide the quantitative framework for making this tradeoff explicitly rather than through intuition.

Training Schedule and Long-Context Pre-Training

The pre-training process follows a carefully designed schedule across 7T tokens, with three phases of learning rate behavior and a two-stage context length extension (Section 2.3).

Three-phase learning rate schedule. The learning rate follows three sequential phases:

Warmup phase: An initial period where the learning rate increases from near-zero to its peak value. This is standard in LLM training to avoid destabilizing the randomly initialized weights with large gradients. The paper does not specify the warmup duration in tokens or steps.
Prolonged gradual decay phase: The bulk of training, where the learning rate is maintained at a high level initially and then incrementally reduced. The paper explicitly argues for this extended high-learning-rate period: "By sustaining an elevated learning rate during the initial pre-training phase, the model is enabled to efficaciously navigate through diverse regions of the solution space, thereby averting premature convergence to suboptimal local minima." The gradual reduction "ensures a methodical convergence to a more optimal solution." This is conceptually similar to cosine decay schedules used in many LLMs but with a longer high-learning-rate plateau.
Annealing phase: The final 5% of pre-training tokens, where the learning rate is reduced to "one-tenth of its peak value." During this phase, the team switches to "the highest-quality dataset available," which the paper states "plays a pivotal role in augmenting the model's performance in the annealing phase." The low learning rate plus high-quality data allows the model to fine-tune its parameters on the very best examples without overfitting, analogous to the "learning rate cooldown" used in other large-scale training runs (Dubey et al., 2024).

Long-context pre-training strategy. After the annealing phase (i.e., after standard-length pre-training is complete), the model undergoes two additional stages to develop long-context processing capability:

Stage 1 (32K tokens): Training on sequences up to 32K tokens in length.
Stage 2 (256K tokens): Training on sequences up to 256K tokens.

The RoPE base frequency is scaled to 1 billion during the 256K stage, following the approach of Xiong et al. (2023), which demonstrated that increasing the RoPE base frequency extends the effective context length by reducing the decay rate of attention scores with positional distance. A base frequency of 1 billion means the rotary position embeddings rotate much more slowly with token distance, allowing the attention mechanism to maintain meaningful scores between tokens separated by hundreds of thousands of positions.

Long-context data mixture. The corpus for long-context training consists of:

~25% natural long-context data: Sourced from books and code repositories, which naturally contain long contiguous passages.
~75% normal-length pre-training data: Standard-length examples mixed in to prevent catastrophic forgetting of short-context capabilities.

This mixture shares the conclusion observed in Gao et al. (2024) that long-context training does not require an exclusively long-context corpus — mixing standard-length data maintains general capabilities while the long-context data teaches positional generalization.

Efficiency of long-context acquisition. The paper notes a striking empirical finding: "it does not require too much training for LLM to acquire long-context capabilities." Each of the 32K and 256K stages uses only approximately 10 billion tokens — a tiny fraction (0.14%) of the total 7T-token pre-training budget. This suggests that long-context processing is more about learning to use existing knowledge across longer positional spans than about acquiring new knowledge, consistent with the interpretation that RoPE scaling primarily teaches the attention mechanism to generalize to unseen position distances.

Post-Training: SFT and DPO

The post-training pipeline converts the pre-trained Hunyuan-Large into Hunyuan-Large-Instruct, an instruction-following assistant, through supervised fine-tuning followed by reinforcement learning from human feedback (Section 3).

Supervised Fine-Tuning (SFT). The SFT phase fine-tunes the pre-trained model on instruction-response pairs designed to enhance specific capabilities: mathematics, coding, logical reasoning, knowledge-based QA, agent behavior, text generation, NLP comprehension, industrial applications, role-playing, and long-text capabilities. The total SFT data volume exceeds 1 million examples.

The SFT data pipeline mirrors the pre-training synthetic data pipeline in structure but operates at a finer granularity:

Instruction extraction: Specialized extraction models process publicly available data sources (web pages, encyclopedias) to identify naturally occurring instruction-response pairs. These "natural instructions" serve as seeds — they provide diverse, realistic user query patterns that purely synthetic generation might miss.
Instruction generalization: A trained instruction generalization system takes extracted instructions and produces more diverse and complex variants. The system is "trained by synthesizing numerous mappings between simple and complex instructions" — it learns to increase difficulty and complexity while preserving the core task type. A separate instruction taxonomy and classification model tracks the distribution of instruction types, enabling targeted supplementation of underrepresented categories.
Instruction balancing: With over 10 million instructions accumulated, distributional balance becomes critical. Each instruction receives multi-dimensional labels, enabling the team to "more accurately understand and analyze the characteristics of our instruction sets." By enforcing adequate representation across instruction types during training, the system avoids overfitting to common instruction patterns (e.g., simple factual QA) at the expense of rarer but important patterns (e.g., multi-step reasoning).
Data quality control: Three filtering layers operate sequentially:
1. Rule-based filtering: Removes obvious defects like data truncation, duplication, garbled characters, and format errors.
2. Model-based filtering: A critique model (based on a 70B Hunyuan dense model) assigns a four-tier quality score to each instruction sample, evaluating "accuracy, relevance, completeness, usefulness, and clarity of the generated responses, and other possible data quality issues."
3. Human-based filtering: Final human annotation ensures responses "adhere to the desired task-specific response patterns and avoid introducing additional low-quality issues."

SFT training configuration. The model is fine-tuned for 3 epochs on the 1M+ examples. The learning rate decays from $2 \times 10^{-5}$ to $2 \times 10^{-6}$ . To combat overfitting (a particular concern given the small dataset relative to the model's capacity), the paper uses attention dropout 0.1 and hidden dropout 0.2. The paper makes an interesting architectural observation: "compared to the dense models, the MoE architecture of Hunyuan series could benefit more from incorporating suitable dropout rates." This suggests MoE models may be more prone to overfitting during SFT because each expert sees fewer examples than parameters in a dense model of comparable total size, making regularization more important.

Reinforcement Learning from Human Feedback (RLHF). The RLHF phase uses Direct Preference Optimization (DPO) rather than the more common Proximal Policy Optimization (PPO) approach. DPO simplifies RLHF by eliminating the need for a separate reward model — it directly optimizes the policy (the language model) to prefer chosen responses over rejected ones using a preference dataset.

The training strategy has several distinctive features:

Single-stage offline-online integration: The paper uses "a single-stage training strategy that integrates both offline and online training." Offline preference data (pre-compiled examples of good and bad responses) provides controllability — the model learns from carefully curated examples that encode specific desired behaviors. Online data (generated by the current policy model during training, scored by a reward model to select best/worst responses) provides adaptability — the model learns from its own evolving output distribution rather than a static dataset. The paper claims this integration "demonstrates superior controllability and overall performance" compared to offline-only or online-only approaches.
SFT loss on chosen responses: An SFT loss term is added to the DPO objective on the chosen (preferred) responses. The paper states this "helps stabilize DPO training by preventing a decrease in the log probability of chosen responses." Without this term, DPO can sometimes reduce the absolute probability of even the preferred response as long as the relative gap to the rejected response is maintained — the SFT loss counteracts this by directly incentivizing the model to maintain high likelihood on good responses. This technique was also used in Dubey et al. (2024) and Adler et al. (2024).
Exponential moving average (EMA): An EMA of model weights is maintained during training "to mitigate reward hacking and reduce alignment tax." Reward hacking occurs when the model learns to exploit quirks in the preference signal rather than genuinely improving response quality (e.g., producing verbose but vacuous responses that the reward model prefers). The EMA smooths weight updates, preventing the rapid divergence that enables reward hacking, and the paper claims it "ensur[es] a more stable training process across a larger dataset."

Design rationale for DPO over PPO. While the paper does not explicitly justify choosing DPO over PPO-based RLHF, the implicit rationale is practical: DPO requires training only the policy model (no separate reward model training, no PPO's value function, no reward model inference during training), which is simpler and more stable at the scale of a 389B-parameter model. The addition of the SFT loss and EMA addresses DPO's known instability issues — the log-probability drift and reward hacking tendencies that can emerge when optimizing purely on relative preferences — making the approach viable at frontier scale.

4. Key Insights and Innovations

Innovation 1: Recycle Routing Recasts Token Dropping as Unnecessary Information Loss

Prior MoE models — from GShard (Lepikhin et al., 2020) through Switch Transformers (Fedus et al., 2022) to Mixtral (Jiang et al., 2024) — treat token dropping as an unavoidable cost of capacity-constrained expert routing. The standard approach sets a capacity factor on each expert and simply discards tokens that exceed it, accepting that some fraction of training tokens bypass the FFN layer entirely as the price of efficient parallelization. The paper identifies this as a false tradeoff. Recycle routing (Section 2.2.3, Figure 2) introduces the conceptually straightforward but previously unexploited idea that tokens destined for overloaded experts can be randomly reassigned to experts with available capacity rather than dropped. The key intellectual move is recognizing that even a suboptimal expert — one not selected by the router — is better than no expert at all, since it preserves the token's information flow through the FFN and prevents training instability from information loss.

What makes this more than a minor implementation trick is the implicit reframing of expert routing from a hard assignment problem to a soft allocation with a fallback. Classic top-k routing treats the router's expert assignment as canonical: if the chosen expert is full, the token genuinely cannot be processed and must be dropped. Recycle routing weakens this assumption, treating the router's top-1 choice as a preference rather than a requirement. The paper does not provide an ablation isolating recycle routing's contribution (a notable omission), but the qualitative claim — "preserving vital information while simultaneously optimizing training efficiency" — encodes a principle that could generalize beyond the specific reassignment strategy: when capacity constraints force load-balancing tradeoffs, preserving token flow through any expert trumps the fidelity of expert-token matching. This is a diagnostic insight for MoE practitioners that was absent from prior routing literature.

Innovation 2: Expert-Specific Learning Rates Diagnose and Formalize a Hidden Batch-Size Asymmetry

The dominant assumption in MoE training — reflected in all major open-source MoE models before Hunyuan-Large, including Mixtral and DeepSeek-V2 — is that a single learning rate serves all model parameters adequately. The paper's expert-specific learning rate scaling (Section 2.2.4) identifies a specific structural reason this assumption fails: the shared expert processes every token in a batch while each specialized expert processes approximately 1/16th of tokens, creating a roughly 16× difference in effective batch size between expert types. Since optimal learning rates depend on batch size via the relationship from Li et al. (2024a), a uniform learning rate cannot be simultaneously optimal for both shared and specialized experts.

The innovation here is not the learning-rate-to-batch-size formula itself (which is taken directly from Li et al., 2024a), but the diagnostic move: recognizing that the MoE architecture introduces a systematic, architecturally-determined batch-size heterogeneity that requires per-component learning rate correction. This is a shift from viewing learning rate as a global optimizer hyperparameter to viewing it as a per-expert resource that should reflect each expert's gradient averaging regime. The computed 0.31× scaling ratio between specialized and shared expert learning rates is a concrete instantiation of this principle.

This contribution is diagnostic rather than algorithmic — it identifies a problem that was latent in prior MoE training but went unrecognized because practitioners treated model-wide learning rates as standard. Its significance lies in clarifying why MoE optimization may be more brittle than dense model optimization (different components need different optimization dynamics) and providing a formula-driven solution rather than requiring expensive hyperparameter sweeps per expert. The paper does not provide an ablation comparing with/without expert-specific learning rates, which limits the strength of the empirical claim, but the theoretical justification is sound and the identified asymmetry is architecturally fundamental — any MoE model with shared and specialized experts has this batch-size mismatch.

Innovation 3: MoE Scaling Laws Adjust the Compute-Optimal Paradigm for Sparse Activation and Long Sequences

Before this paper, the scaling law framework for language models — established by Kaplan et al. (2020) and Hoffmann et al. (2022) — was developed for and validated on dense architectures using the compute budget formula C = 6ND, where N is total parameters. Extending this to MoE models requires two adjustments that prior work had not systematically addressed: (1) compute cost scales with activated parameters, not total parameters, since only a fraction of experts participate in each forward pass; and (2) at the long sequence lengths that large models increasingly target (8K, 32K, 256K), the O(L²) attention cost becomes non-negligible relative to the FFN cost and must be separately modeled.

The paper's revised compute budget formula (Equation 2, Section 2.3.1) — C ≈ 9.59ND + 2.3×10⁸D, where N is activated parameters — represents the first published attempt to characterize the isoFLOPs relationship for MoE models at scale. The second term (2.3×10⁸D) is the key conceptual addition: it captures the sequence-length-dependent attention cost that the dense scaling law formulation absorbs into the constant factor (since dense models were historically trained at short enough sequences that attention was a negligible fraction of total compute). At 256K tokens, attention cost is no longer negligible, and the paper's fitted constant provides a quantitative handle on this effect.

The scaling law analysis yields specific fitted coefficients: N_opt ∝ C^{0.53} for activated parameters and D_opt ∝ C^{0.50} for training tokens. The exponent near 0.5 for both parameters and data suggests that MoE compute-optimal scaling follows a roughly equal-allocation principle similar to the Chinchilla findings for dense models (Hoffmann et al., 2022, found exponents of approximately 0.5 for both), but now operating on activated rather than total parameters. This is a framing insight: it suggests that the fundamental compute-optimal scaling relationship is invariant to sparsity — what changes is which parameter count (total vs. activated) enters the formula.

The specific design decisions derived from this analysis (52B activated parameters instead of the computed 58.1B optimum; 7T tokens instead of 5.6T) illustrate a practical corollary: the isoFLOPs curve is sufficiently flat near the optimum that practitioners can trade off model size for data volume within a meaningful window without substantial efficiency loss. This is not a new mathematical discovery (Hoffmann et al., 2022, noted similar flatness for dense models), but the paper's explicit demonstration of this flexibility in the MoE regime — and its strategic use of it to slightly undersize the model for deployment convenience while overshooting on data — is a useful case study in applying scaling laws as decision-making tools rather than precise prescriptions. The innovation is primarily empirical and infrastructural: providing the first published MoE-specific scaling law coefficients derived from training runs spanning two orders of magnitude in activated parameter count.

Innovation 4: The Four-Step Synthetic Data Pipeline Codifies Quality Control as a Staged, Multi-Model Process

Synthetic data in LLM pre-training is not new — prior work including Phi-3 (Abdin et al., 2024) and Llama 3 (Dubey et al., 2024) has used synthetic data to augment pre-training corpora, and the broader technique of generating training data from models dates back to self-instruct and related approaches. What distinguishes Hunyuan-Large's synthetic data pipeline (Section 2.1.1, Figure 1) is not any individual step but the explicit architectural separation of data generation into four independently optimized stages with dedicated models at each stage. Instruction generation, instruction evolution, response generation, and response filtering are treated as distinct sub-problems, each with its own specialized models and quality criteria.

The intellectual contribution is in the decomposition itself. Prior work often used a single model to go from seed to instruction to response in an end-to-end fashion, with filtering as a post-hoc cleanup step. Hunyuan-Large's pipeline interposes instruction evolution (Step 2) as an explicit intermediate stage that transforms initial instructions along multiple axes — clarity, domain coverage, difficulty — before responses are generated. This recognizes a subtle but important point: the quality ceiling of synthetic data is determined by instruction quality before response quality, because poor instructions (ambiguous, too simple, domain-redundant) constrain the maximum educational value of even perfect responses. By separating instruction refinement from response generation, the pipeline can invest heavily in instruction quality without being bottlenecked by a generator's ability to simultaneously produce good instructions and good responses.

The difficulty escalation aspect of instruction evolution is particularly notable as a curriculum design insight applied to data synthesis rather than training. Rather than generating more data at a uniform difficulty level (which saturates learning), the pipeline systematically produces harder variants of existing instructions, enabling the pre-training data itself to function as a curriculum — the model encounters progressively more challenging examples in each domain without requiring explicit curriculum scheduling during training.

The pipeline's reported output — 1.5T tokens, over 21% of the total 7T-token corpus — establishes synthetic data as a first-class component of pre-training rather than an auxiliary supplement. This is a scale claim as much as a methodological one: the paper argues (implicitly through the proportion) that synthetic data should be thought of as a primary data source for capability injection in domains where natural data is deficient, not as a marginal data augmentation. The specific targeting of mathematics, coding, low-resource languages, and high-educational-value content operationalizes a principle that prior work gestured at but did not systematize at this scale: synthetic data is most valuable precisely where natural data distributions are scarcest.

Innovation 5: The FLOPs-Matched Efficiency Argument Reframes MoE Deployment Economics

While individual components of Hunyuan-Large — GQA, CLA, shared experts, synthetic data — have precedents in prior work, the paper's most strategically significant contribution may be the aggregate empirical demonstration that MoE can match or exceed the largest dense models while activating 7.8× fewer parameters. The pre-trained model's 69.8 on MATH versus LLama3.1-405B's 53.8 (Table 3) — a 16-point absolute improvement with ~8× fewer activated parameters — and the instruct model's 81.8 on Arena-Hard versus LLama3.1-405B-Instruct's 69.3 (Table 4) — a 12.5-point gap — are not just competitive results. They establish a specific quantitative claim: at the frontier scale of open-source models, MoE is not merely theoretically efficient but practically preferable to dense scaling for deployments where per-token FLOPs or memory footprint constrain serving costs.

This is a framing contribution rather than an architectural one. Prior MoE models (Mixtral-8x22B, DeepSeek-V2) demonstrated competitiveness with dense models at their scale, but the dense models they competed against (LLama3.1-70B for Mixtral) were in the same broad parameter regime. Hunyuan-Large's comparisons are against LLama3.1-405B — a model with 7.8× more activated parameters. The efficiency ratio is large enough to change the economic calculus: if 52B activated parameters can match 405B on language understanding (88.4 vs. 85.2 on MMLU) and substantially exceed 405B on math (69.8 vs. 53.8 on MATH), then for many deployment scenarios the MoE model is the strictly dominant choice — lower memory, lower latency, lower serving cost, equal or better capability.

The KV cache compression results (Table 2) reinforce this economic argument from the inference side. The ~95% reduction in KV cache memory from GQA+CLA relative to MHA means that Hunyuan-Large's 256K-context capability is not just a benchmark claim but a deployment feasibility claim: the model can actually be served at long context lengths on realistic hardware configurations. This connects the scaling-law model sizing (the decision to build at 52B activated) to the inference efficiency engineering — the paper implicitly argues that MoE at this scale, with these compression techniques, hits a sweet spot where capability, efficiency, and deployability align.

The innovation is less in any single technique and more in the integrated demonstration that careful engineering across data synthesis, routing, optimization, and deployment compression can produce an MoE model that is not merely competitive with dense architectures but arguably superior for practical deployment — and that all of this can be done in the open, with released weights, enabling the community to verify, build on, and improve the approach.

5. Experimental Analysis

Evaluation Methodology

Dataset. The pre-trained model is evaluated on a broad collection of established benchmarks spanning English and Chinese: MMLU, MMLU-Pro, BBH, CMMLU, and C-Eval for aggregated tasks; HellaSwag, CommonsenseQA, WinoGrande, and PIQA for commonsense understanding; DROP, C3, and NaturalQuestions for classical NLP; ARC-C and TriviaQA for knowledge-intensive QA; GSM8K, MATH, and CMATH for mathematics; and HumanEval and MBPP for coding (Section 4.1.1). The post-trained model adds GPQA_diamond, AlignBench, MT-Bench, IFEval strict-prompt, Arena-Hard, and AlpacaEval-2.0 for instruction-following and alignment evaluation, plus RULER, LV-Eval, and the proprietary PenguinScrolls for long-context assessment (Sections 4.2.1, 4.3). For SFT data, the paper reports "more than 1 million" examples; for the RLHF preference dataset, no size is specified (Section 3.1.1, 3.2).
Base model(s). Hunyuan-Large is a single model family with no variants — a 389B total / 52B activated parameter MoE Transformer pre-trained on 7T tokens, then instruction-tuned to produce Hunyuan-Large-Instruct (Sections 2.2.1, 3). The scaling law analysis (Section 2.3.1) trained a series of smaller MoE models from 10M to 1B activated parameters, but these are used only for deriving the compute-optimal sizing and are not evaluation baselines.
Metrics. All benchmarks use their standard metrics: accuracy for multiple-choice tasks (MMLU, MMLU-Pro, ARC-C, CommonsenseQA, etc.), exact match or equivalent for generation tasks (MATH, GSM8K, DROP), pass@1 for coding (HumanEval, MBPP), and task-specific scoring for alignment benchmarks (e.g., Arena-Hard uses GPT-4-as-judge with a length-controlled win rate). The paper reports the "best performance among the results that are publicly reported or those reproduced by ourselves for baselines" (Section 4.1.1, 4.2.1), which means baseline numbers may come from original papers, third-party leaderboards, or the authors' own re-evaluation — but the specific provenance of each number is not documented. This introduces ambiguity about whether comparisons are strictly apples-to-apples, since different evaluation pipelines (prompt templates, parsing, number of examples tested) can produce meaningfully different scores on the same benchmark.
Baselines. The pre-trained model comparisons (Table 3) include LLama3.1-405B (Dubey et al., 2024 — the largest open-source dense model), LLama3.1-70B (Dubey et al., 2024 — a similar-activated-parameter-scale dense model), Mixtral-8x22B (Mistral, 2024 — 141B total / 39B activated MoE), and DeepSeek-V2 (DeepSeek-AI, 2024 — 236B total / 21B activated MoE). The post-trained model comparisons (Table 4) include LLama3.1-405B-Instruct, LLama3.1-70B-Instruct, Mixtral-8x22B-Instruct, and DeepSeek-V2.5-Chat. For long-context evaluations (Tables 5, 6), the sole baseline is LLama3.1-70B-Instruct (Dubey et al., 2024), chosen because of its "well-documented strength in processing extended contexts." A notable omission: LLama3.1-405B-Instruct is not included in the long-context comparisons, so we cannot assess whether Hunyuan-Large-Instruct closes the gap to the largest dense model on long-context tasks.
Generation budget / compute accounting. The paper does not conduct any FLOPs-matched comparisons between Hunyuan-Large and baselines during evaluation — the compute budget for training is discussed in Section 2.3.1 (total FLOPs via C ≈ 9.59ND + 2.3×10⁸D), but evaluation-time compute is not accounted for. All comparisons are at the model level: performance-at-given-parameter-scale, not performance-at-given-inference-FLOPs. This matters for the efficiency claims: Hunyuan-Large activates 52B parameters per token versus LLama3.1-405B's 405B, but MoE models incur additional overhead from routing computation and expert parallelism communication that the paper does not quantify in FLOPs terms. The paper claims a 7.8× efficiency advantage (405B / 52B) but this is a parameter-count ratio, not a measured FLOPs ratio.
Cross-validation / statistical protocol. None reported. The paper provides single-point accuracy numbers for each benchmark with no confidence intervals, variance estimates, or statistical significance tests. For the pre-trained model evaluations, most benchmarks are evaluated with a fixed few-shot prompt format (e.g., 5-shot for MMLU, 4-shot for GSM8K), but the paper does not report variance across different prompt templates or random seeds. For the post-trained alignment benchmarks (Arena-Hard, AlpacaEval-2.0, MT-Bench), these inherently involve LLM-as-judge evaluations that can have non-trivial variance, but no error bars or multiple evaluation runs are reported. This makes it impossible to assess whether, for example, Hunyuan-Large's 2.6% advantage over LLama3.1-405B on MMLU (89.9 vs. 87.3) is statistically reliable or within noise. The instruction-following benchmark IFEval reports "strict-prompt" accuracy, but it's unclear whether this is micro-averaged across all prompt categories or a single aggregate.

Main Quantitative Results

Pre-Trained Model Performance Across Standard Benchmarks

The headline result from Table 3 is that Hunyuan-Large achieves the "overall best performance" among all evaluated pre-trained models on the reported benchmarks, including LLama3.1-405B despite activating 7.8× fewer parameters. The specific numbers:

Aggregated benchmarks: On MMLU (5-shot), Hunyuan-Large scores 88.4 versus LLama3.1-405B's 85.2, LLama3.1-70B's 79.3, Mixtral-8x22B's 77.8, and DeepSeek-V2's 78.5 — a 3.2 percentage point lead over the largest dense model. On MMLU-Pro (5-shot), Hunyuan-Large's 60.2 leads LLama3.1-405B's 61.6 by 1.4 points in the opposite direction — this is one of the few benchmarks where it underperforms, though LLama3.1-70B is at 53.8 and Mixtral at 49.5. On BBH (3-shot), Hunyuan-Large's 86.3 edges LLama3.1-405B's 85.9. The pattern is consistent: Hunyuan-Large is competitive with or exceeds the 405B dense model on aggregated understanding while showing larger gaps over the 70B-class models.

Commonsense and reasoning benchmarks: On CommonsenseQA (7-shot), Hunyuan-Large achieves 92.9 versus LLama3.1-405B's 85.8 — a 7.1 point gap, which is the largest absolute advantage reported. On WinoGrande (5-shot), it's 88.7 vs. 86.7. On PIQA (0-shot), 88.3 vs. no value reported for LLama3.1-405B (the paper uses "—" in several cells, meaning either the result was not evaluated or was not publicly available). The paper notes that Hunyuan-Large shows "superior performance in commonsense understanding and reasoning, and classical NLP tasks."

Mathematics benchmarks: This is the most dramatic category. On GSM8K (4-shot), Hunyuan-Large scores 92.8 versus LLama3.1-405B's 89.0 — a 3.8 point gap. On MATH (4-shot), the gap is much larger: 69.8 versus 53.8, a 16.0 absolute percentage point difference (a 29.7% relative improvement). The paper does not isolate why MATH shows such a larger advantage than GSM8K (both are math benchmarks, but MATH involves competition-level problems requiring multi-step reasoning, while GSM8K focuses on grade-school word problems), but the pattern is consistent with the paper's emphasis on synthetic data targeting mathematics as a capability deficiency — if the synthetic data pipeline was particularly effective for generating high-quality competition-math examples, the MATH improvement would naturally outpace GSM8K where natural data already covers the domain adequately. On CMATH (3-shot, Chinese mathematics), Hunyuan-Large scores 91.3 versus the next best (DeepSeek-V2 at 78.7) — a 12.6 point gap.

Coding benchmarks: On HumanEval (0-shot, pass@1), Hunyuan-Large achieves 71.4 versus LLama3.1-405B's 61.0, LLama3.1-70B's 58.5, and Mixtral-8x22B's 53.1. On MBPP (3-shot), the gap is smaller: 72.6 versus LLama3.1-405B's 73.4 — Hunyuan-Large is 0.8 points behind, the other result where it underperforms the 405B dense model.

Chinese-language benchmarks: Hunyuan-Large leads substantially on all three reported Chinese tasks: CMMLU (5-shot, 90.2 vs. next-best DeepSeek-V2 at 84.0), C-Eval (5-shot, 91.9 vs. DeepSeek-V2 at 81.7), and C3 (0-shot, 82.3 vs. DeepSeek-V2 at 77.4). LLama3.1-405B values are not reported for these Chinese benchmarks (marked "—" in Table 3), making it unclear whether the gap reflects genuine Chinese-language superiority or simply the absence of the strongest baseline. The paper's tokenizer design (adding 28K Chinese-specific tokens) and bilingual training data focus would predict strong Chinese performance, but without LLama3.1-405B comparisons, we cannot calibrate the magnitude of the advantage.

Knowledge-intensive QA: On NaturalQuestions (5-shot), Hunyuan-Large scores 52.8 versus LLama3.1-405B's unreported value and DeepSeek-V2's 38.7. On TriviaQA (0-shot), it's 89.2 versus DeepSeek-V2's 79.9. On DROP (3-shot, reading comprehension with discrete reasoning), 88.9 versus LLama3.1-405B's 84.8. These results suggest strong fact-retention capability from pre-training, but the absence of LLama3.1-405B values on NQ and TriviaQA limits the comparison.

A critical observation about Table 3: For several benchmarks, LLama3.1-405B values are missing (marked "—") or are from different evaluation protocols. The paper states it reports "the best performance among the results that are publicly reported or those reproduced by ourselves for baselines" (Section 4.1.1), which means (1) some numbers may come from Llama 3's original paper, (2) some may come from independent re-evaluations, and (3) some may be missing because neither source was available. Without knowing which is which, the comparison is inherently uneven — a number from the Llama 3 paper evaluated with one prompt template is not directly comparable to a number from the Hunyuan team's re-evaluation with a different template. This is a common limitation in LLM leaderboard-style comparisons and does not invalidate the results, but it does mean the quantitative gaps should be interpreted as indicative rather than precise.

Post-Trained Model Performance on Capability and Alignment Benchmarks

Table 4 shifts from pre-trained capabilities to instruction-following performance, where Hunyuan-Large-Instruct is compared against LLama3.1-405B-Instruct and other instruction-tuned models.

Standard capability benchmarks after instruction tuning: On MMLU, Hunyuan-Large-Instruct scores 89.9 versus LLama3.1-405B-Instruct's 87.3 — a 2.6 point gap that is slightly narrower than the 3.2 point pre-trained gap, suggesting both models benefit proportionally from instruction tuning on this benchmark. On MATH, Hunyuan-Large-Instruct scores 77.4 versus LLama3.1-405B-Instruct's 73.8 — a 3.6 point gap that is substantially smaller than the 16.0 point pre-trained gap (69.8 vs. 53.8). This is noteworthy: the pre-trained Hunyuan-Large's enormous MATH advantage largely disappears after instruction tuning. One interpretation is that LLama3.1's instruction tuning process provides disproportionately large math gains (bringing MATH from 53.8 pre-trained to 73.8 post-trained, a 20-point jump not typically seen from SFT/RLHF alone — this likely reflects the use of math-specific instruction data or tool integration that is not present in pre-training evaluations). Another interpretation is that Hunyuan-Large's pre-trained math advantage was partly a benchmark-format artifact that instruction tuning normalizes away. The paper does not discuss this convergence.

On HumanEval, Hunyuan-Large-Instruct achieves 90.0 versus LLama3.1-405B-Instruct's 89.0 — a narrow 1-point gap. On ARC-C, 94.6 versus 96.9, where Hunyuan-Large-Instruct trails by 2.3 points. The pattern across capability benchmarks post-instruction-tuning is mixed: Hunyuan-Large-Instruct leads on MMLU and MATH but trails on ARC-C and is essentially tied on HumanEval, unlike the pre-trained comparisons where it led on nearly every benchmark.

Alignment and instruction-following benchmarks: This is where the paper claims Hunyuan-Large-Instruct's strongest showing. On Arena-Hard (a benchmark that uses GPT-4 to judge response quality and is "frequently updated with new prompts to prevent over-fitting"), Hunyuan-Large-Instruct scores 81.8 versus LLama3.1-405B-Instruct's 69.3 — a 12.5 point gap, the largest reported advantage in either table. On AlpacaEval-2.0 (an automatic evaluation of instruction-following using length-controlled win rates against a reference model), it's 51.8 versus 39.3 — an 12.5 point gap. On MT-Bench (expert-level human preference judgments), it's 9.4 versus 9.1 — a narrow 0.3 point lead. On AlignBench (Chinese alignment evaluation), it's 8.3 versus LLama3.1-405B-Instruct's 6.0 — a 2.3 point gap. On IFEval strict-prompt (evaluating adherence to specific formatting and content constraints in instructions), Hunyuan-Large-Instruct's 85.0 trails LLama3.1-405B-Instruct's 86.0 by 1 point.

The Arena-Hard and AlpacaEval results are striking because these benchmarks are specifically designed to correlate with human preference rankings in chatbot applications, and the 12+ point gaps are substantially larger than the gaps on capability benchmarks. The paper states this "could mainly attribute to its powerful pre-trained model, the high-quality SFT and DPO data with the well-designed four-step data collection and processing that generates this data, and the superior SFT and DPO training strategies." This attribution is broad and non-specific — the paper does not disentangle whether the gains come from better pre-training, better SFT data, or better DPO training, making it an assertion rather than an established causal claim.

A caution on the alignment benchmark comparisons: Arena-Hard and AlpacaEval-2.0 use LLM-as-judge evaluation (GPT-4 for Arena-Hard, likely GPT-4 or similar for AlpacaEval), which introduces systematic biases — models can score higher by producing verbose responses, adopting a particular style the judge model prefers, or exploiting the judge's known weaknesses. The paper's use of length-controlled AlpacaEval-2.0 mitigates the verbosity bias, but other judge biases remain. Without human evaluation or multiple-judge-model cross-validation, the 12.5-point gap on Arena-Hard could partially reflect judge preference alignment rather than genuine response quality improvement. The paper does not report inter-judge agreement or alternative evaluation approaches for these benchmarks.

Long-Context Evaluations

The long-context evaluation (Section 4.3) compares Hunyuan-Large-Instruct against LLama3.1-70B-Instruct across three benchmarks. LLama3.1-405B-Instruct is notably absent from these comparisons.

RULER results (Table 5): RULER evaluates retrieval, multi-hop reasoning, aggregation, and QA across varying context lengths. Hunyuan-Large-Instruct's performance is reported in four length buckets:

0-8K tokens: 94.39 (vs. 95.89 for LLama3.1-70B-Instruct — trailing by 1.5 points)
8K-32K tokens: 94.94 (vs. 95.39 — trailing by 0.45 points)
32K-64K tokens: 93.02 (vs. 94.72 — trailing by 1.7 points)
64K-128K tokens: 89.53 (vs. 86.48 — leading by 3.05 points)

The paper claims that Hunyuan-Large-Instruct "significantly outperforms the baseline model" in the 64K-128K range, which is true for that specific bucket, but at shorter ranges it consistently (though narrowly) trails. The more interesting pattern is the relative stability: Hunyuan-Large-Instruct drops only 4.86 points from the 0-8K bucket to the 64K-128K bucket (94.39 → 89.53), a 5.1% relative decline, while LLama3.1-70B-Instruct drops 9.41 points (95.89 → 86.48), a 9.8% relative decline. The paper highlights this minimal degradation, which is consistent with the RoPE base frequency scaling to 1 billion during the 256K pre-training stage — if the position embeddings maintain better representational quality at long distances, performance should degrade more slowly with context length.

LV-Eval results (Table 5): LV-Eval is described as a "challenging long-context benchmark comprising 11 distinct question-answering datasets" with "confounding facts" designed to test whether models get distracted by irrelevant information. To address high false-negative rates from the original strict metrics, the paper uses "LLM as an evaluator" (the evaluator model is not specified). Results are reported in three length buckets:

0-32K tokens: 81.92 (vs. 75.73 — leading by 6.19 points)
32K-64K tokens: 71.15 (vs. 62.39 — leading by 8.76 points)
64K-128K tokens: 67.87 (vs. 61.57 — leading by 6.30 points)

Hunyuan-Large-Instruct leads across all length ranges, with the largest gap in the middle range. The shift to LLM-as-evaluator from the original strict metric makes direct comparison with other published LV-Eval results invalid, but the within-benchmark comparison against LLama3.1-70B-Instruct under identical evaluation conditions is internally valid.

PenguinScrolls results (Table 6): The proprietary benchmark evaluates four task types: Information Extraction, Information Localization, Qualitative Analysis, and Numerical Reasoning. Hunyuan-Large-Instruct leads on all four:

Information Extraction: 91.14 vs. 82.51 (8.63 point gap)
Information Localization: 89.56 vs. 69.70 (19.86 point gap — the largest single-task gap in any table)
Qualitative Analysis: 92.78 vs. 75.77 (17.01 point gap)
Numerical Reasoning: 67.46 vs. 49.52 (17.94 point gap)
Overall: 85.23 vs. 69.37 (15.86 point gap)

These gaps are substantially larger than those on RULER or LV-Eval, which raises questions about whether PenguinScrolls is measuring the same long-context construct or is inadvertently testing capabilities on which Hunyuan-Large happens to excel (e.g., Chinese-language document processing, which would be consistent with the bilingual training focus). The paper states that "internal user studies corroborate that the improvements on PenguinScrolls strongly correlate with enhancements in actual user experiences," but provides no details on these user studies. Without external validation of PenguinScrolls against established long-context benchmarks, it's difficult to calibrate what a 15.86-point gap on this benchmark means in absolute terms.

A gap in the long-context evaluation: The paper does not report any evaluation at the model's claimed maximum context length of 256K tokens. All benchmarks stop at 128K. The RoPE base frequency scaling to 1 billion was specifically performed for the 256K stage, and the paper claims Hunyuan-Large is "capable of handling up to 256K tokens," but this capability is never directly tested in evaluation. The RULER benchmark would support 256K evaluation; the fact that only up to 128K is reported suggests either that 256K performance was not measured or that it was measured and not reported (potentially because of degradation). This is a significant omission for a model whose headline specification includes 256K context support.

Ablation Studies and Robustness Checks

The paper conducts almost no formal ablation studies. This is the most significant methodological weakness of the experimental analysis. In a paper that introduces multiple named innovations — recycle routing, expert-specific learning rate scaling, the four-step synthetic data pipeline, KV cache compression via GQA+CLA, MoE-specific scaling laws — none of these components are isolated to measure their individual contribution to model performance. The reader cannot determine whether the claimed 16-point MATH advantage over LLama3.1-405B comes from synthetic data, routing, learning rate scaling, or some interaction thereof.

What the paper provides instead are indirect justifications and implicit ablations:

MoE scaling laws (Figures 3, 4): These figures show the fitted isoFLOPs curves that guided model sizing. They are not ablations in the standard sense (they do not test performance with/without the scaling law methodology) but demonstrate that the scaling law analysis produced reasonable fitted coefficients (N_opt ∝ C^{0.53}, D_opt ∝ C^{0.50}). However, the paper does not train a comparison model at the same total compute budget but with different sizing (e.g., 40B activated on 9T tokens vs. 52B activated on 7T tokens) to validate that the scaling laws actually identified a near-optimal point. The scaling laws are presented as a design tool, not as a validated empirical optimization.

KV cache compression comparison (Table 2): Table 2 compares theoretical KV cache memory across attention mechanisms (MHA, GQA, MQA, CLA, GQA+CLA), demonstrating the ~95% reduction from GQA+CLA relative to MHA. This is a memory calculation, not a performance ablation. The paper does not report model quality metrics (e.g., perplexity, downstream accuracy) for alternative attention configurations, so the claim that GQA+CLA achieves compression "without much side effect on model performance" (Section 2.2.2) is asserted without quantitative evidence. Ideally, the paper would show quality metrics for MHA, GQA-only, CLA-only, and GQA+CLA configurations at a fixed parameter count to establish the quality-compression tradeoff. The absence of this makes the compression claim purely a memory calculation.

Expert-specific learning rate scaling (Section 2.2.4): The paper derives the 0.31× scaling ratio theoretically but provides no training run comparing uniform vs. expert-specific learning rates. The claim that this "contributes to the overall performance" is therefore a theoretical assertion, not an empirically validated finding. An informative ablation would train a smaller-scale MoE model (e.g., 1B activated parameters) with and without expert-specific learning rates and measure convergence speed and final loss — but the paper provides neither.

Recycle routing (Section 2.2.3, Figure 2): Perhaps the most straightforward ablation to run — compare training with standard top-1 routing (dropping overflow tokens) versus recycle routing (reassigning overflow tokens) at identical hyperparameters. The paper provides no such comparison. The claim that recycle routing "preserves vital information while simultaneously optimizing training efficiency" is a mechanism-level argument without empirical validation.

Synthetic data contribution: No ablation measures model performance with and without the 1.5T tokens of synthetic data. The paper states that synthetic data targets "the relative capability deficiency merely learned from natural data," but we cannot determine what fraction of Hunyuan-Large's capabilities come from synthetic data versus improved natural data processing or architectural innovations. Given that synthetic data is positioned as one of the paper's three main contributions, the absence of any synthetic data ablation is a major gap.

Long-context pre-training stages (Section 2.3.3): The paper describes a two-stage long-context extension (32K → 256K) using ~10B tokens per stage. No ablation compares one-stage vs. two-stage extension, examines the effect of different RoPE base frequencies (only 1 billion is tested), or measures long-context performance at intermediate context lengths between 0 and 32K to characterize the degradation curve. The claim that long-context capability acquisition is efficient (~10B tokens) is an observation from a single training run, not a controlled experiment.

SFT data quality control (Section 3.1.2): The three-layer filtering (rule-based, model-based, human-based) is not ablated — we don't know what fraction of data each layer removes, whether model-based filtering correlates with downstream performance, or whether human filtering provides gains over model-based filtering alone. The critique model's four-tier scoring system is described but its accuracy (e.g., agreement with human judgments) is not reported.

DPO training components (Section 3.2): The SFT loss term on chosen responses and the exponential moving average strategy are described as stability measures, but no ablation is provided. The paper does not compare DPO vs. PPO-based RLHF, DPO with vs. without SFT loss, DPO with vs. without EMA, or offline-only vs. online-only vs. integrated offline-online preference data. The claim that the integrated approach "demonstrates superior controllability and overall performance" is therefore asserted but not demonstrated.

What minimal ablations would have strengthened the paper: Even at 389B scale, certain ablations are feasible: (1) training a small-scale model (1B activated) with identical architecture but comparing with/without recycle routing and with/without expert-specific learning rates — these are optimizer/routing ablations, not scale-dependent; (2) evaluating the final model on a held-out subset of benchmarks with GQA+CLA ablated to MHA (accepting the higher memory cost for evaluation only); (3) comparing performance on benchmarks that the synthetic data explicitly targeted (MATH, HumanEval) against performance on benchmarks that rely more on natural data (HellaSwag, CommonsenseQA) to infer synthetic data's differential contribution. None of these are provided.

Negative results and failed approaches: The paper reports no negative results — no training configurations that were tried and abandoned, no hyperparameter settings that caused instability, no benchmarks where Hunyuan-Large substantially underperformed expectations. This is atypical for a technical report describing a large-scale training effort, which almost always encounters unexpected behaviors (training instabilities, loss spikes, routing collapse, etc.) that are informative for the community. The paper's silence on failures makes it difficult to assess the robustness of the approach or to identify which components were essential versus incidental.

Critical Assessment

The paper's central claims can be evaluated against the experimental evidence as follows.

The claim that Hunyuan-Large is the "largest open-source Transformer-based mixture of experts model" with "superior performance across various benchmarks" (Abstract): The parameter count claim is factual — 389B total / 52B activated is larger than prior open-source MoE models. The performance claim is supported by Tables 3 and 4, but with important qualifications. Table 3 shows Hunyuan-Large leading on most but not all benchmarks; MMLU-Pro and MBPP show it trailing LLama3.1-405B (60.2 vs. 61.6 and 72.6 vs. 73.4 respectively). The "overall best" language masks these losses. More critically, the comparisons are uneven — some LLama3.1-405B values are missing (NaturalQuestions, TriviaQA, all Chinese benchmarks), some may be from different evaluation protocols, and no statistical significance is reported. The claim of "superior performance" would be more precisely stated as "competitive with or exceeding LLama3.1-405B on most evaluated benchmarks, with particularly large advantages on mathematics and Chinese-language tasks, and with some benchmarks showing mixed or slightly inferior results."

The claim that Hunyuan-Large "outperforms LLama3.1-70B and exhibits comparable performance when compared to the significantly larger LLama3.1-405B model" (Abstract): The outperformance over LLama3.1-70B is unambiguous — Tables 3 and 4 show Hunyuan-Large leading on effectively every benchmark with a reported LLama3.1-70B value. The "comparable performance" claim relative to LLama3.1-405B is underspecified. On some benchmarks (CommonsenseQA, MATH, GSM8K, CMATH, HumanEval), Hunyuan-Large substantially exceeds the 405B model. On others (ARC-C, MMLU-Pro, MBPP), it trails. On the alignment benchmarks, Hunyuan-Large-Instruct leads by wide margins on Arena-Hard and AlpacaEval but trails on IFEval. The aggregate picture is one of consistent competitiveness with specific areas of clear superiority, particularly in mathematics (where the 16-point pre-trained MATH gap is the largest signal in the paper) and Chinese language tasks (where gaps of 6-10 points are typical). The paper would be stronger if it characterized where Hunyuan-Large exceeds the 405B model and where it matches or trails, rather than collapsing to "comparable performance" across the board.

The claim that innovations in synthetic data, routing, KV cache compression, and expert-specific learning rates "contribute to Hunyuan-Large's exceptional performance" (Introduction): This claim is not empirically supported by the experimental results. The paper provides no ablation isolating any of these components. The entire experimental section demonstrates that Hunyuan-Large performs well — it does not demonstrate why. Every component claim (recycle routing preserves information, expert-specific learning rates optimize training, synthetic data enhances capability acquisition, KV cache compression reduces memory without quality loss) is either a theoretical argument or an assertion without controlled comparison against a counterfactual configuration. This does not make the claims false — the components may well contribute as described — but the paper provides no evidence that would allow a reader to determine whether any specific component is essential, beneficial but marginal, or even net-harmful (masked by compensatory gains elsewhere).

The long-context evaluation (Tables 5, 6) partially addresses this by implicitly abating the long-context training strategy: the model demonstrably handles long contexts, so the training recipe worked. But even here, the benchmarks stop at 128K, not the claimed 256K, and the comparison is against LLama3.1-70B-Instruct only (not 405B-Instruct), limiting the strength of the inference.

The claim about MoE scaling laws providing "valuable insights and guidance for future model development and optimization" (Abstract): The scaling law analysis in Section 2.3.1 and Figures 3-4 provides a worked example of applying isoFLOPs methodology to MoE models, which is valuable as a methodological demonstration. However, the fitted coefficients (N_c = 5.9 × 10^{-3}, α = 0.5305, D_c = 3.2, β = 0.50) are validated only internally — they guided Hunyuan-Large's design, and Hunyuan-Large performs well, but this is a single-point validation that doesn't confirm the scaling law's accuracy across the full compute range. A validation would require showing that alternative model sizes chosen against the scaling law's predictions produce worse performance (e.g., training a 30B-activated and an 80B-activated model at the same compute budget and showing they underperform the 52B model). The paper does not do this. The scaling laws are therefore best understood as a design rationale, not as an empirically validated predictive tool.

The claim about KV cache compression enabling practical deployment (Section 2.2.2): Table 2 demonstrates the memory reduction mathematically, and this is straightforward to verify. The missing piece is actual inference benchmarks — throughput, latency, memory usage on specific hardware configurations — that would validate the deployment practicality claim. A memory calculation shows feasibility; deployment benchmarks would show realizability.

What experiments would have strengthened the paper: Four categories of missing experiments stand out:

Component ablations at a tractable scale: Train 1B-activated-parameter MoE models comparing (a) standard top-1 routing with token dropping vs. recycle routing, (b) uniform vs. expert-specific learning rates, (c) with and without synthetic data, (d) MHA vs. GQA vs. GQA+CLA. These would isolate contributions and validate component-level claims without requiring 389B-scale training.
Maximum context length evaluation at 256K: The model claims 256K context support, but no evaluation exceeds 128K. Testing at the claimed limit is essential, even if on a single benchmark like RULER.
LLama3.1-405B-Instruct long-context comparisons: If Hunyuan-Large-Instruct is to be compared against the largest dense model on capability and alignment benchmarks, the same comparison should extend to long-context tasks. The omission of 405B-Instruct from Tables 5 and 6 is unexplained and leaves open whether Hunyuan-Large's long-context advantage is over the 70B model specifically or over dense models generally.
Variance estimation or statistical testing: For the post-trained model especially, LLM-as-judge evaluations (Arena-Hard, AlpacaEval-2.0, MT-Bench) have known variance. Reporting confidence intervals or multiple evaluation runs would allow the reader to assess whether a 12.5-point Arena-Hard gap is practically meaningful or within evaluation noise.

Strengths of the experimental design that deserve recognition: Despite these gaps, the paper's evaluation breadth is genuinely impressive. Covering 20+ benchmarks across English, Chinese, pre-training, post-training, capability, alignment, and long-context domains provides a multi-faceted picture of model performance that exceeds what many technical reports provide. The inclusion of a proprietary benchmark (PenguinScrolls) alongside established benchmarks like RULER is a defensible decision, especially since the paper commits to releasing PenguinScrolls. The reporting of both pre-trained and post-trained results is methodologically valuable — it allows the reader to separate architecture/pre-training effects from instruction-tuning effects, which is how we can observe that the 16-point MATH gap narrows to 3.6 points after instruction tuning, suggesting that post-training data and methods, not just pre-training, drive much of the final instruct model performance.

The paper's strongest empirical contribution is not any single number but the aggregate pattern: a 52B-activated MoE model trained with careful attention to data quality and MoE-specific optimization can, across a wide range of benchmarks, consistently compete with a 405B-parameter dense model. This is a valid and important empirical demonstration, even if the paper cannot attribute the success to specific components. The community benefits from knowing that this outcome is achievable and having access to the weights that produced it, even if the precise recipe for reproducing it remains partially opaque due to missing ablations.

6. Limitations and Trade-offs

No Component Ablations Mean the Contribution of Individual Innovations Is Unknown

The assumption or constraint. The paper introduces four named technical innovations — recycle routing (Section 2.2.3), expert-specific learning rate scaling (Section 2.2.4), the four-step synthetic data pipeline (Section 2.1.1), and KV cache compression via GQA+CLA (Section 2.2.2) — and attributes Hunyuan-Large's performance to these collectively. The paper does not isolate any of these components against a counterfactual configuration. Section 4.3 acknowledges that the favorable outcomes "are largely attributed to our high-quality training data armed with data synthesis, superior model structure, and sophisticated training recipes in both pre-training and post-training," but this attribution is asserted, not demonstrated through controlled comparison.

The consequence. A practitioner reading this paper cannot determine which of the paper's contributions are essential for reproducing Hunyuan-Large's results and which are incidental. It is possible that one or more of the named innovations contributes negligibly — or even negatively — to final performance, with the gains coming entirely from other factors (e.g., the scale of compute, the quality of natural data filtering, or the post-training methodology). This has direct practical implications: a team attempting to build a similar MoE model must either replicate all components (wasting resources on potentially unnecessary complexity) or guess which to prioritize (risking omission of a critical ingredient). The paper's value as a methodological guide is substantially diminished when the reader cannot distinguish signal from noise in the design choices.

What evidence exists in the paper. None. Table 3 and Table 4 demonstrate that Hunyuan-Large performs well; they do not demonstrate why. The scaling law analysis (Section 2.3.1, Figures 3-4) provides validation that the model sizing decision was reasonable, but this validates the sizing methodology, not the routing or optimization innovations. The KV cache compression is validated only as a memory calculation (Table 2), not as a quality-preserving technique — the paper states GQA+CLA achieves compression "without much side effect on model performance" (Section 2.2.2) but provides no perplexity or downstream accuracy comparison against uncompressed attention. Recycle routing, expert-specific learning rates, and the synthetic data pipeline receive no ablations whatsoever.

Mitigation status. The paper does not attempt to address this limitation. It provides no small-scale training runs isolating individual components, no retrospective analysis of which data sources or architectural choices correlated most strongly with benchmark performance, and no discussion of which components were found to be essential versus optional during development. This is the most consequential methodological gap in the paper, and it limits the community's ability to build on the work efficiently.

Synthetic Data Quality Relies on Unspecified Proprietary Models and Processes That Cannot Be Reproduced

The assumption or constraint. The four-step synthetic data pipeline (Section 2.1.1, Figure 1) depends on several models and systems whose details are not disclosed: "several specialized models" of "varying sizes" for response generation (Step 3), a critique model for response filtering (Step 4), an instruction generalization system capable of "generalizing targeted instructions while gradually increasing their difficulty and complexity levels" (Step 2, also Section 3.1.2), and an instruction extraction model for the SFT data pipeline. The paper does not specify the architectures, sizes, training data, or training procedures for these component models. For the SFT critique model, the paper mentions it is "based on a 70B dense model of our Hunyuan series" (Section 3.1.2), but this is a family reference, not a replicable specification. The response generation models are described only as "well-designed specialized models."

The consequence. The synthetic data pipeline — which produces 1.5T tokens of pre-training data (over 21% of the total corpus) and underpins the SFT data quality control — is not independently reproducible. The paper describes the process (four steps, quality criteria, filtering mechanisms) but not the tools (the actual models performing generation, evolution, extraction, and filtering). A practitioner attempting to replicate the pipeline would need to independently develop models matching the unspecified capabilities of Tencent's internal systems, with no way to calibrate whether their implementations match the quality that produced Hunyuan-Large's results. This is particularly significant because the paper positions synthetic data as a first-class pre-training component targeting "the relative capability deficiency merely learned from natural data" — if the synthetic data is the primary driver of the model's strongest results (the 16-point MATH gap over LLama3.1-405B), and the data generation models are unavailable, then the paper's headline performance numbers are not reproducible from the released artifacts (model weights and code) alone. The weights enable inference and fine-tuning; they do not enable retraining with the same data pipeline.

What evidence exists in the paper. The paper provides the four-step framework (Figure 1) and qualitative descriptions of each step's goals. It does not provide: model cards for the component models, training data specifications, quality benchmarks for the generated synthetic data (e.g., human evaluation scores, accuracy of generated math solutions, factuality rates for knowledge-intensive responses), or comparisons between synthetic data generated with different model configurations. The SFT data pipeline (Section 3.1.2) adds detail about the critique model's evaluation dimensions ("accuracy, relevance, completeness, usefulness, and clarity") and the four-tier scoring system, but the critique model's own accuracy (e.g., correlation with human quality judgments, false-positive and false-negative rates) is not reported. The instruction generalization system's training procedure ("synthesizing numerous mappings between simple and complex instructions") is described at a high level without specifics on data construction, model architecture, or validation metrics.

Mitigation status. The paper does not address this limitation. It does not commit to releasing the synthetic data generation models, the critique model, or the instruction extraction/generalization models. It does not provide sample synthetic data that would allow the community to evaluate its quality independently. The paper's framing — releasing "code and checkpoints of Hunyuan-Large" (Abstract) — covers the final trained model but not the infrastructure used to produce its training data. For practitioners interested in the synthetic data methodology specifically, this is a substantial gap: the paper teaches the what (a four-step process) and the why (to address natural data deficiencies in math, coding, and low-resource domains) but not the how (the specific models and quality thresholds that make the process work at 1.5T-token scale).

256K Context Length Is Claimed but Never Evaluated; All Long-Context Benchmarks Stop at 128K

The assumption or constraint. The paper's abstract and introduction prominently claim that Hunyuan-Large is "capable of handling up to 256K tokens." The long-context pre-training procedure (Section 2.3.3) describes a two-stage extension to 32K and then 256K tokens, with RoPE base frequency scaled to 1 billion for the 256K stage. An entire phase of pre-training is dedicated to enabling this capability. The paper implicitly assumes that the 256K training stage successfully transfers to usable 256K inference performance.

The consequence. The paper provides no evidence that the model actually performs usefully at 256K context. All three long-context benchmarks — RULER, LV-Eval, and PenguinScrolls — are evaluated at maximum context lengths of 128K tokens (Tables 5, 6). The RULER results are bucketed up to 128K; the LV-Eval results stop at 128K; PenguinScrolls extends to 128K. The claimed 256K capability is therefore an untested specification, not an empirically validated property of the released model. For practitioners evaluating whether Hunyuan-Large meets their long-context requirements, this is a critical gap: the model might maintain quality to 128K but degrade sharply between 128K and 256K, or it might require specific prompting or retrieval patterns to use the full context effectively, or it might exhibit attention degradation that the benchmarks at 128K do not capture. The paper's RoPE frequency scaling to 1 billion (inspired by Xiong et al., 2023) provides theoretical grounds for extended context, but the gap between positional embedding theory and usable attention quality is well-documented — models can have the capacity to attend over long ranges without actually producing correct outputs at those ranges.

What evidence exists in the paper. None for 256K. The RULER bucket from 64K-128K shows Hunyuan-Large-Instruct at 89.53 versus LLama3.1-70B-Instruct at 86.48 (Table 5), demonstrating that the model does not collapse at 128K. But the performance trend from 32K-64K (93.02) to 64K-128K (89.53) shows a 3.49-point drop across a 2× context length increase. If degradation continued linearly — and attention degradation is often superlinear in context length — performance at 256K could be substantially lower than at 128K. The paper does not report any evaluation at the model's maximum claimed context length.

Mitigation status. The paper does not acknowledge this gap. No explanation is offered for why 256K evaluation is absent despite RULER supporting this length. The paper does not commit to releasing 256K evaluation results in the future. For a model whose headline capability includes 256K context support, the absence of validation at that length is a significant omission that undermines the specification claim.

The Evaluation Protocol Introduces Ambiguity About Cross-Model Comparability

The assumption or constraint. The paper's evaluation methodology (Sections 4.1.1, 4.2.1) states that for baseline models, the authors "report the best performance among the results that are publicly reported or those reproduced by ourselves." This means the numbers in Tables 3 and 4 come from an unspecified mixture of sources: some may be from original model publications (evaluated with their own prompt templates, parsing logic, and hardware), some may be from third-party leaderboards, and some may be from the authors' own re-evaluation (using their own pipeline). The paper does not specify, for any given baseline number, which source it comes from.

The consequence. It is well-established in the LLM evaluation literature that benchmark scores are sensitive to evaluation protocol details: the exact few-shot prompt template, the parsing logic used to extract final answers, the number of evaluation examples (some benchmarks have test sets with slightly different sizes depending on the split), and even the generation temperature and random seed can produce meaningfully different numbers. By mixing numbers from different evaluation pipelines, the paper introduces an unquantified source of variance into the comparisons. A 2-3 point gap between Hunyuan-Large and a baseline could reflect genuine model capability differences, or it could reflect the baseline number coming from an evaluation pipeline that was slightly less optimized for that specific benchmark. This is particularly concerning for the Chinese-language benchmarks (CMMLU, C-Eval, C3) where LLama3.1-405B values are missing entirely (marked "—" in Table 3), making it impossible to assess whether Hunyuan-Large's strong Chinese performance reflects genuine bilingual superiority or simply the absence of the strongest baseline. Similarly, for benchmarks where Hunyuan-Large trails (MMLU-Pro: 60.2 vs. LLama3.1-405B's 61.6; MBPP: 72.6 vs. 73.4), we cannot determine whether the deficit is real or an artifact of evaluation protocol differences.

What evidence exists in the paper. The paper provides the few-shot settings (e.g., "5-shot for MMLU," "4-shot for GSM8K") and the benchmark names, but does not disclose prompt templates, answer extraction code, or the provenance of each baseline number. No inter-evaluation-pipeline variance estimates are reported. The post-trained alignment benchmarks (Arena-Hard, AlpacaEval-2.0, MT-Bench) rely on LLM-as-judge evaluation, which introduces additional variance from the judge model's preferences and prompt sensitivity — the paper does not report multiple evaluation runs, confidence intervals, or judge-model alternatives that would allow the reader to assess whether the 12.5-point Arena-Hard gap (81.8 vs. 69.3) is robust to evaluation noise.

Mitigation status. The paper does not address this limitation. It does not release the exact evaluation prompts and parsing code used for each benchmark (though the model code is released, the evaluation harness configuration is not detailed). It does not provide confidence intervals or statistical tests. The "best performance" criterion for baselines is generous to the baselines (it gives them the benefit of the best available number), but it does not eliminate the fundamental incomparability of numbers from different evaluation pipelines. The most rigorous approach would be to re-evaluate all baseline models under identical conditions using the same evaluation harness, and the paper acknowledges it did this partially ("those reproduced by ourselves") but does not specify which numbers are from this controlled re-evaluation and which are from public reports.

MoE-Specific Inference Overhead Is Not Quantified; Efficiency Claims Are Parameter-Count-Based, Not Measured

The assumption or constraint. The paper's efficiency argument rests on a parameter-count comparison: Hunyuan-Large activates 52B parameters per token versus LLama3.1-405B's 405B, implying an approximately 7.8× reduction in per-token computation. Section 2.3.1 provides a training compute formula (Equation 2: $C \approx 9.59 N D + 2.3 \times 10^8 D$ , where $N$ is activated parameters) that accounts for some MoE overhead — the constant 9.59 (versus 6 for dense models) captures additional computation from routing and the shared expert being always active. The paper implicitly assumes that this training-time overhead ratio also characterizes inference-time efficiency, and that parameter count ratios translate cleanly to FLOPs or latency ratios.

The consequence. MoE inference incurs overhead that dense inference does not, and the paper does not quantify this. The router computation at each MoE layer adds FLOPs that scale with the number of experts (16 specialized experts per layer, 64 layers, 80 attention heads — the router must compute logits over 16 experts for every token at every MoE layer). The expert parallelism communication pattern — where different tokens in a batch are dispatched to different experts potentially residing on different devices — introduces all-to-all communication that does not exist in dense model inference. The KV cache, while compressed by GQA+CLA, stores keys and values for all layers regardless of which expert each token activated; the total KV cache memory is the same for MoE and dense models at the same hidden size and number of layers (assuming identical attention compression). The shared expert processes every token, adding FFN computation that does not exist in a pure top-k MoE without shared experts. These factors mean the 7.8× parameter-count ratio is an upper bound on the actual FLOPs or latency advantage; the realized speedup depends on batch size, hardware topology, expert parallelism strategy, and communication bandwidth in ways the paper does not characterize. For practitioners evaluating deployment costs, the absence of measured inference benchmarks (tokens-per-second, latency-at-various-batch-sizes, memory usage on specific GPU configurations) makes it impossible to translate the parameter efficiency into concrete operational costs.

What evidence exists in the paper. The training compute formula (Equation 2) indicates a 9.59/6 ≈ 1.6× overhead factor relative to dense models on a per-activated-parameter basis, but this is a training FLOPs estimate, not a measured inference cost. The 52B activated parameters × 1.6 relative overhead would imply an effective FLOPs-per-token comparable to a ~83B-parameter dense model (52 × 9.59/6 ≈ 83), which would still be a ~4.9× advantage over 405B, but this is a modeling estimate, not a measurement. The paper provides no inference benchmarks — no throughput numbers, no latency measurements, no memory profiling on specific hardware. The KV cache compression (Table 2) provides a theoretical memory reduction calculation but no measured memory usage during actual inference.

Mitigation status. The paper does not address this limitation. It provides no deployment benchmarks and does not commit to releasing inference performance characterization. The model weights and code are released, so the community can perform these measurements independently, but the paper itself provides no guidance on what inference efficiency to expect. For a paper whose contributions include deployment-focused innovations (KV cache compression) and whose strategic argument rests on MoE's efficiency advantage, the absence of measured inference performance is a significant gap between the theoretical efficiency claims and the practical deployment picture.

No Failure Analysis, Training Instability Discussion, or Negative Results Are Reported

The assumption or constraint. The paper presents Hunyuan-Large as a successful engineering effort with uniformly positive results. It implicitly assumes that the training process was stable, the design choices were correct on the first attempt, and no significant failures or unexpected behaviors were encountered during development. The paper reports no negative results — no configurations that were tried and abandoned, no instability events during the 7T-token training run, no benchmarks where the model substantially underperformed expectations, and no qualitative error analysis showing failure modes.

The consequence. Large-scale LLM training at the 389B-parameter scale is notoriously difficult, with well-documented failure modes including loss spikes, routing collapse (where all tokens are routed to a small subset of experts), training instability from load-balancing loss interactions, and unexpected degradation on specific capability axes. By not reporting any of these challenges or how they were addressed, the paper deprives practitioners of information that is often more valuable than success reports: knowing what doesn't work, what failure signatures to watch for, and what interventions were necessary to stabilize training. For example, the recycle routing strategy (Section 2.2.3) is motivated by the claim that token dropping "may cause the loss of crucial information, which in turn negatively impacts training stability" — this implies that the authors observed training stability issues with standard top-1 routing, but they do not describe these issues, quantify their severity, or show that recycle routing resolved them. The expert-specific learning rate scaling (Section 2.2.4) implies that uniform learning rates are suboptimal, but no training run demonstrating the suboptimality is reported. The post-training section mentions using EMA to "mitigate reward hacking" (Section 3.2), implying reward hacking was observed during DPO, but no examples or quantification are provided. A technical report that omits the failure modes encountered during development is a partially sanitized account; the community learns that the final configuration works but not why alternatives were rejected or what vigilance is needed during training.

What evidence exists in the paper. None. The paper contains no loss curves, no routing entropy plots (showing expert utilization balance over training), no examples of model outputs that illustrate failure modes, and no benchmarks where Hunyuan-Large's performance was unexpectedly poor (all reported numbers are presented as positive results). The closest the paper comes to a negative result is the observation that MMLU-Pro (60.2 vs. 61.6) and MBPP (72.6 vs. 73.4) show Hunyuan-Large trailing LLama3.1-405B, but these are presented matter-of-factly without analysis of why the model underperforms on these specific benchmarks.

Mitigation status. The paper does not acknowledge this as a limitation and does not commit to releasing training logs, loss curves, or failure case analyses. The model weights and code enable the community to discover failure modes post-hoc through evaluation, but the paper misses the opportunity to share the development team's hard-won knowledge about what goes wrong during MoE training at scale and how to prevent it. For a paper that positions itself as providing "valuable insights and guidance for future model development and optimization" (Abstract), the absence of negative results and failure analysis substantially reduces the actionable guidance it provides to practitioners who will encounter the same challenges.

7. Implications and Future Directions

How This Work Changes the Landscape

Hunyuan-Large does not introduce a new architectural paradigm — it uses the "classical Transformer architecture with MoE" (Section 2.2.1), SwiGLU activations, RoPE, and a shared-plus-specialized expert design that the paper notes was introduced concurrently with DeepSeek-V2. What it changes is the perceived ceiling of open-source MoE and the standard of evidence for what a MoE technical report should contain. This is a shift in community expectations rather than a shift in architectural design space.

The primary reframing: MoE as the default path to frontier-scale open models. Before Hunyuan-Large, the dominant open-source models at the largest scales were dense: Llama 3.1-405B, Qwen 2.5-72B, DeepSeek-V2 (236B total but only 21B activated — still modest in activated parameters). The implicit assumption in the community was that building MoE models at the 50B+ activated parameter scale was too complex for open-source efforts, or that the efficiency gains wouldn't materialize at that scale, or that the engineering burden of stable MoE training outweighed the deployment benefits. Hunyuan-Large empirically falsifies each of these implicit assumptions. The pre-trained model matches or exceeds Llama 3.1-405B — the largest open-source dense model — on most evaluated benchmarks while activating 7.8× fewer parameters (52B vs. 405B), and the post-trained model extends this competitiveness to alignment benchmarks like Arena-Hard (81.8 vs. 69.3). The practical implication is clear: for any team contemplating training a frontier-scale open model, MoE is now the demonstrated-efficiency choice, not a risky architectural experiment. The burden of proof has shifted — a team choosing a 400B+ dense model over a 50B-activated MoE must now justify why they're spending 8× more per-token FLOPs for comparable capability, rather than the reverse.

This reframing is amplified by the paper's comprehensive release strategy. By providing weights, code, and detailed methodology (even if incompletely reproducible, as noted in Section 6), the paper lowers the activation energy for other teams to adopt MoE. The scaling laws in Section 2.3.1, even with their limitations, provide a starting-point framework for sizing MoE models that didn't exist publicly before. The four-step synthetic data pipeline, while reliant on unreleased models, establishes a process template that other teams can instantiate with their own models. The specific hyperparameters in Table 1 (64 layers, 80 attention heads, 8 KV heads, 1 shared expert, 16 specialized experts, hidden size 6400) give practitioners a concrete reference configuration that is known to work at scale, reducing the search space for their own designs.

Reconciling prior contradictions. The paper indirectly resolves a tension in the MoE literature between theoretical efficiency and practical stability. Prior open-source MoE models (Mixtral-8x22B, DeepSeek-V2) demonstrated that MoE could be competitive with dense models at modest scale, but they operated below the compute-optimal activated parameter count the paper estimates (~58B, Section 2.3.1). This left open the possibility that MoE efficiency advantages diminished at larger scales — that the routing instability, load-balancing challenges, and expert underutilization problems documented in the Switch Transformer (Fedus et al., 2022) and GShard (Lepikhin et al., 2020) literature would compound as expert count and model depth increased. Hunyuan-Large demonstrates that these challenges are solvable with appropriate training recipes: recycle routing addresses the token-dropping problem, expert-specific learning rates address the gradient-averaging asymmetry, and the scaling-law-guided sizing prevents the undersizing that makes prior models unrepresentative. The paper doesn't claim to have solved MoE training stability permanently — the absence of training logs and instability discussions noted in Section 6 means we don't know what difficulties were encountered — but it demonstrates that stable convergence at 389B total parameters is achievable with publicly describable techniques, which was not previously established.

Research directions that become more attractive. The paper's results make several research bets look stronger. MoE-specific optimization (learning rates, routing strategies, load-balancing losses) moves from a niche concern to a central research area — if expert-count-dependent optimization is essential for stable scaling, there is substantial room for innovation beyond the 0.31× learning rate ratio the paper computes. Synthetic data as directed capability injection becomes more empirically grounded: if the 16-point MATH gap (69.8 vs. 53.8, Table 3) is substantially attributable to the 1.5T tokens of math-targeted synthetic data, then systematic study of how synthetic data composition shapes downstream capabilities becomes a high-leverage research direction. Long-context MoE — the interaction between expert routing and very long sequences (256K tokens) — is largely unexplored; the paper's RoPE scaling approach is adapted from dense model literature (Xiong et al., 2023) and may not be optimal for MoE-specific attention patterns.

Research directions that become less urgent. The paper's conservative architectural choices (standard Transformer, no novel attention mechanisms, no retrieval augmentation, no multi-modal components) suggest that radical architectural departures from the Transformer are unnecessary for frontier performance. The paper achieves its results within the standard MoE-Transformer framework, implying that attention should shift from architecture search to training methodology optimization. Similarly, the paper's success with a relatively simple routing strategy (shared + top-1 specialized) over more complex approaches (top-2, learned routing with auxiliary losses beyond load balancing, token-choice routing) suggests that routing complexity may have diminishing returns compared to data quality and optimization technique — an empirical hypothesis that the paper enables but does not test.

Follow-Up Research This Work Enables

Component attribution through scaled-down ablation studies. The paper's most significant methodological gap is the absence of ablations isolating its four named innovations (recycle routing, expert-specific learning rates, synthetic data pipeline, GQA+CLA). A high-value follow-up would train a series of MoE models at a tractable scale (e.g., 1B-5B activated parameters, trained on 100B-500B tokens) that systematically ablate each component: (a) standard top-1 routing with token dropping vs. recycle routing, (b) uniform learning rate vs. expert-specific scaling at the computed 0.31× ratio, (c) natural-data-only pre-training vs. natural plus synthetic data matching the paper's domain targeting, (d) MHA vs. GQA-only vs. GQA+CLA at matched parameter counts. The key measurements would be: convergence speed (loss vs. FLOPs), downstream benchmark performance (especially on MATH and HumanEval where synthetic data is expected to matter most), expert utilization entropy over training (to detect routing collapse), and final perplexity on held-out data. This would transform Hunyuan-Large from a point demonstration (this combination of techniques works) to a decomposition (each technique contributes X benefit at Y cost), enabling practitioners to prioritize which innovations to adopt. The paper's release of model weights enables an alternative approach: retroactive ablation through activation analysis — measuring expert utilization patterns, gradient statistics, and per-layer behavior in the released model to infer which components were functionally important without retraining.

Direct measurement of synthetic data's capability contribution via data attribution. The paper claims synthetic data specifically targets "the relative capability deficiency merely learned from natural data" in mathematics, coding, and low-resource domains (Section 2.1.1), and the benchmark results show particularly large advantages in math (MATH: 69.8 vs. 53.8, Table 3) and coding (HumanEval: 71.4 vs. 61.0). A critical follow-up question is: what fraction of these gaps is causally attributable to synthetic data versus architectural or optimization improvements? A strong experiment would use influence functions or data attribution methods on the released Hunyuan-Large weights to estimate the marginal contribution of synthetic vs. natural pre-training data to performance on specific MATH and HumanEval test examples. If 80% of the MATH advantage traces to synthetic data, the paper's methodology becomes the primary thing to replicate. If 30% traces to synthetic data and 70% to other factors (optimization, routing, scale), then the synthetic data pipeline may be less essential than the paper implies. This experiment is feasible because the paper releases weights; it requires access to the pre-training data composition (which the paper describes at a high level without releasing actual data), making it partially dependent on the authors releasing data provenance metadata.

Characterizing the long-context degradation curve between 128K and 256K. The paper claims 256K context support but only evaluates to 128K (Tables 5, 6). A straightforward but important follow-up is to evaluate Hunyuan-Large-Instruct on RULER at context lengths from 128K to 256K in increments (e.g., 128K, 160K, 192K, 224K, 256K) to characterize the degradation curve. Key questions: Does performance drop linearly with log(context length), as the RoPE scaling literature would predict, or is there a phase change (e.g., sharp drop after 200K) indicating a failure mode in the attention mechanism? How does the degradation compare to LLama3.1-405B-Instruct at the same lengths (which the paper omitted from long-context comparisons entirely)? Does the degradation vary by task type within RULER (retrieval vs. multi-hop reasoning vs. aggregation)? The paper's release of model weights enables this experiment immediately. If performance degrades gracefully to 256K (e.g., RULER score dropping by <10 points from 128K to 256K), the 256K claim is validated and the RoPE scaling to 1 billion base frequency is further empirically supported. If performance collapses between 128K and 256K, this reveals that the 256K pre-training stage did not successfully transfer to usable inference capability, and the community needs better methods for extreme context length extension in MoE architectures specifically.

Expert specialization analysis: what do the 16 specialized experts actually learn? The paper describes the shared expert as capturing "common knowledge" and specialized experts as learning "domain-specific knowledge" (Section 2.2.3), but provides no empirical characterization of what the experts specialize in. A follow-up study would analyze expert routing patterns on diverse inputs: for a held-out corpus spanning mathematics, code, factual text, dialogue, and multilingual content, measure which specialized experts are activated for which input types. Does expert specialization emerge along the lines the paper intends (e.g., expert 3 activates predominantly on mathematical tokens, expert 7 on code)? Or is routing more syntactic (e.g., experts specialize in sentence positions, punctuation, or token-frequency patterns)? Do the routing patterns change systematically with context length (relevant to the long-context extension, since attention patterns at 256K may route tokens differently than at 8K)? This analysis is feasible with the released weights by running inference on curated input sets and logging router logits. If clear domain specialization emerges, it validates the shared/specialized expert design as producing interpretable, composable expert modules — which opens the door to expert pruning (removing experts irrelevant to a target deployment domain) or expert fine-tuning (further training specific experts on domain data). If routing is essentially stochastic or syntactic, the shared/specialized distinction may be less functionally meaningful than the paper assumes, suggesting that alternative routing architectures (e.g., top-2 routing without a shared expert) might achieve similar results with simpler design.

Interaction between MoE routing and long-context attention. The paper treats long-context pre-training (Section 2.3.3) and expert routing (Section 2.2.3) as independent design decisions, but they may interact in non-obvious ways. At 256K context, tokens span a much wider range of positions, topics, and linguistic structures than at 8K context — does this cause routing entropy to change? Do some experts become overloaded at long contexts because long documents disproportionately contain content from specific domains (e.g., a 256K legal document routes almost entirely to a "formal text" expert)? A follow-up would monitor expert utilization statistics (tokens routed to each expert, router logit entropy, load imbalance metrics) as context length increases from 8K to 256K on a fixed document corpus. If routing remains balanced and entropy stable, the MoE design generalizes to long contexts without modification. If certain experts dominate at long contexts (routing collapse), this suggests that long-context MoE may need length-conditioned routing or capacity factors that adapt to sequence length. The paper's recycle routing mechanism (Figure 2) was designed for the standard-length training regime; its behavior under long-context-induced routing skew is unknown but testable with the released model.

Practical Applications and Downstream Use Cases

Cost-efficient serving of frontier-capability models for multilingual applications. Hunyuan-Large's strong Chinese-language performance (CMMLU: 90.2, C-Eval: 91.9, C3: 82.3; Table 3) combined with competitive English performance and the 7.8× parameter-count advantage over Llama 3.1-405B makes it directly applicable to deployment scenarios requiring bilingual capability with tight serving budgets. Consider a customer support system handling English and Chinese queries at scale: deploying Llama 3.1-405B would require either a multi-node serving setup (for the 405B dense model) or accepting high per-query latency from model parallelism. Hunyuan-Large's 52B activated parameters — roughly the same scale as Llama 3.1-70B's 70B dense parameters — means it can be served on hardware configurations designed for 70B-class models while delivering capability that the paper's benchmarks suggest is competitive with or exceeds the 405B model, particularly on math and reasoning tasks that may arise in technical support contexts. The KV cache compression (~95% reduction, Table 2) further reduces memory pressure for long multi-turn conversations. The concrete benefit: a deployment team could serve Hunyuan-Large on 4×A100-80GB GPUs with tensor parallelism (feasible for a 52B-activated model with KV cache compression) rather than the 8+ GPUs typically needed for 405B dense models, cutting hardware costs approximately in half while maintaining or improving response quality for bilingual users.

Synthetic data generation at scale for domain-specific fine-tuning. The paper's four-step synthetic data pipeline (Figure 1, Section 2.1.1), while not fully reproducible due to unreleased component models, describes a process that organizations can instantiate with their own in-house models. The key takeaway for practitioners is the instruction evolution step (Step 2): systematically increasing the difficulty and complexity of instructions before generating responses, rather than simply generating more instructions at a uniform difficulty level. A concrete application: a financial services company wanting to fine-tune an LLM for financial analysis could follow the pipeline structure: (1) extract seed instructions from financial reports, earnings call transcripts, and regulatory filings; (2) evolve these instructions to cover multi-step reasoning (e.g., escalating from "what was Q3 revenue?" to "what does Q3 revenue imply for full-year guidance given historical Q4 seasonality?"); (3) generate responses using a specialized finance-tuned model; (4) filter using a critique model plus self-consistency on numerical answers. The paper's demonstration that synthetic data can be a substantial fraction of pre-training (21% in Hunyuan-Large's case) and that it specifically augments domains where natural data is scarce (mathematics, low-resource fields) implies that this approach is not a marginal data augmentation tactic but a primary data sourcing strategy for capability injection. The economic benefit: high-quality domain-specific training data that would cost millions to produce via human annotation can be generated at model-inference cost, with the four-step quality control providing systematic protection against the garbage-in-garbage-out failure mode of naive synthetic data generation.

Long-context document processing for legal and academic applications. The paper's RULER results (Table 5) show Hunyuan-Large-Instruct maintaining 89.53 accuracy in the 64K-128K range, with minimal degradation from shorter contexts (94.39 at 0-8K, a 5.1% relative decline), and the PenguinScrolls benchmark (Table 6) demonstrates strong performance on information extraction (91.14), information localization (89.56), and qualitative analysis (92.78) from long documents. For a legal document review workflow — analyzing 100+ page contracts, regulatory filings, or case law — the model's ability to retrieve, reason over, and extract information from documents up to 128K tokens (validated) and potentially 256K (claimed) reduces or eliminates the need for chunking-based retrieval pipelines that can miss cross-document dependencies. A law firm could deploy Hunyuan-Large-Instruct as a single-pass document analyzer: input a full 80,000-word contract, query for specific clause types, obligations, and risk factors, and receive structured extractions without the engineering complexity of maintaining a retrieval corpus and chunking strategy. The concrete advantage over Llama 3.1-70B-Instruct (the baseline in the long-context evaluations) is visible in the Tables 5 and 6 gaps: 3-6 points on RULER/LV-Eval and 15+ points on PenguinScrolls across task types. The absence of Llama 3.1-405B-Instruct from the long-context comparison means we cannot quantify the advantage over the largest dense model, but the parameter efficiency argument applies: if the 52B-activated MoE model matches or exceeds the 70B dense model on long-context tasks, it does so at comparable serving cost with the potential to exceed 405B performance if the pattern from capability benchmarks extends to long-context.

On-device-capable specialization through expert pruning for targeted deployment. While Hunyuan-Large at 389B total parameters is not an on-device model, the paper's shared/specialized expert architecture opens a deployment strategy that is of practical interest: expert pruning for domain-specific deployment. If follow-up analysis (as proposed above) reveals that specific specialized experts handle specific domains (e.g., expert 3 for mathematics, expert 7 for code, expert 12 for Chinese language), a deployment team could prune irrelevant experts for their target application. A coding assistant deployment might retain only the shared expert and the code-specialized expert(s), dropping the remaining specialized experts and reducing the total parameter count from 389B to a much smaller number while preserving coding capability. The shared expert architecture makes this particularly natural — the shared expert captures general language capability that any domain needs, and only domain-specific specialized experts need to be retained. The paper doesn't explore this, but it's a direct architectural consequence of the design, and the paper's release of weights enables the experiment: measure HumanEval and MBPP performance after pruning different subsets of specialized experts to identify which experts are essential for coding. If performance holds with, say, 4 out of 16 specialized experts retained (shared + 4 specialized, roughly 100B total parameters), the effective deployment footprint drops substantially, broadening the range of deployment hardware. This is a speculative application (the paper provides no pruning experiments), but it follows naturally from the architecture and is practically testable with the released artifacts.