Every Activation Boosted: Scaling General Reasoner to 1 Trillion Open Language Foundation

🎯 Pitch

A 1-trillion-parameter MoE model achieves a 7× compute-efficiency leap over dense counterparts, but this leverage only holds when problems fall within the model's reachable test-time compute range. The key is aligning extreme sparsity with reasoning from the start—using chain-of-thought activation during pre-training and an evolutionary RL method that dynamically scales reasoning depth.

1. Executive Summary

Ling 2.0 introduces a series of reasoning-oriented language foundation models scaling from 16B to 1 trillion parameters under a unified Mixture-of-Experts paradigm, demonstrating that high-sparsity, fine-grained MoE with every activation boosted for reasoning—guided by the Ling Scaling Laws (empirical power-law relationships that predict optimal hyperparameters, model-data allocation, and architectural efficiency across compute budgets)—can establish a new Pareto frontier of reasoning accuracy versus computational cost. The series includes three non-thinking instruct models—Ling-mini-2.0, Ling-flash-2.0, and Ling-1T—trained on a reasoning-oriented data pipeline with mid-training reasoning pre-activation via CoT data (introducing chain-of-thought corpora before post-training to raise the reasoning performance ceiling) and post-training Evolutionary Chain-of-Thought (Evo-CoT) (a reinforcement learning paradigm that scales reasoning depth adaptively according to problem complexity, using dynamic length control and sentence-level policy optimization). The models achieve approximately 7× efficiency leverage compared to dense counterparts—Ling-mini-2.0 (1.4B activated) matches dense models below 10B, while Ling-1T (51B activated) surpasses leading open-source models on benchmarks including AIME 2025 and LiveCodeBench—establishing that sparse activation, when aligned with reasoning objectives across architecture, data, and training, enables scalable intelligence only when problems lie within the model's capability range reachable through adaptive test-time compute.

2. Context and Motivation

The Core Problem: Scaling Reasoning Ability Without Scaling Cost Linearly

The fundamental challenge this paper tackles is how to build a trillion-parameter language model that reasons effectively while remaining computationally tractable to train and deploy. This is not merely an engineering exercise in making models bigger—it is a confrontation with the central tension in modern LLM development: as models scale toward hundreds of billions and trillions of parameters, the computational cost of dense architectures becomes prohibitive, yet the performance gains that justify that cost increasingly depend on a model's reasoning capability—the ability to decompose problems, infer hidden relations, and make consistent multi-step deductions.

The paper states this tension directly in its opening:

"As scaling reaches hundreds of billions of parameters, performance gains increasingly depend on a model's ability to reason—to decompose problems, infer hidden relations, and make consistent multi-step deductions."

In other words, simply making models larger is not enough. The value of scale must be extracted through reasoning, and doing so at the trillion-parameter level requires solving two intertwined problems simultaneously: (1) efficient scaling—maintaining computational efficiency, stability, and predictability under extreme scale—and (2) reasoning enhancement—ensuring that expanded capacity leads to more consistent and reliable reasoning rather than just improved pattern matching.

This dual challenge is what distinguishes Ling 2.0 from efforts that optimize for either efficiency or reasoning in isolation. A model that is efficient but cannot reason is of limited use for the complex tasks (mathematical problem-solving, code generation, multi-step logical deduction) that define modern AI capability benchmarks. Conversely, a model that reasons well but costs 7× more to train and run than necessary represents a massive waste of resources, particularly as these models are deployed at scale across thousands or millions of inference requests.

Why This Problem Matters: The Economics and Science of Scale

The importance of this problem has both economic and scientific dimensions.

Economic: The cost wall in dense scaling. Dense transformer architectures scale quadratically in attention cost and linearly in feed-forward cost with parameter count. At the 100B+ parameter scale, training a dense model from scratch requires tens of thousands of GPU-years, putting it out of reach for all but a handful of organizations. At the trillion-parameter scale, a dense model would be essentially untrainable with current hardware. Mixture-of-Experts (MoE) architectures offer a way through this wall—by activating only a subset of parameters per token, they decouple total parameter count (which determines expressive capacity) from active computation (which determines training and inference cost). However, MoE brings its own challenges: load balancing across experts, communication overhead in distributed training, and the risk that sparsity undermines the very reasoning capability that scaling was meant to enhance.

The paper's 7× efficiency leverage claim—that Ling 2.0 models achieve performance comparable to dense models with roughly one-seventh the activated parameters—translates directly to cost. If a trillion-parameter model can be trained and served at the cost of a ~140B dense model, the economic equation for deploying advanced AI systems shifts dramatically. This matters particularly for open-source models, where adoption depends on being practically deployable by downstream users who do not have datacenter-scale resources.

Scientific: Does sparsity harm reasoning? The scientific question lurking beneath the efficiency claims is whether extreme sparsity—Ling 2.0 activates only ~3.5% of its parameters per token—degrades the kind of deep, multi-step computation that reasoning requires. Intuitively, reasoning involves chaining together information across many inference steps, and one might worry that routing each token through different experts would fragment the model's internal representations, making it harder to maintain coherent chains of thought. The paper's positive results—Ling-1T achieving 70.42% on AIME 2025 and 61.68% on LiveCodeBench while activating only 51B parameters—provide evidence that sparsity and reasoning are not in tension, provided the architecture, training data, and training procedure are designed for reasoning from the ground up.

The open-source dimension. The paper explicitly positions Ling 2.0 within the trajectory of open-source LLMs:

"Recent open models highlight this trend. Kimi-K2, an open trillion-scale model, focuses primarily on enhancing agentic capability, while DeepSeek-V3, though smaller at 671B parameters, achieves outstanding reasoning performance under efficient sparse scaling. Ling 2.0 is designed to push beyond: scaling a trillion-parameter reasoning-oriented foundation model that maximizes reasoning accuracy and efficiency under sparse activation, establishing a scalable blueprint for next-generation open intelligent systems."

This positioning matters because open-source models serve as infrastructure for an ecosystem of fine-tuned variants, research experiments, and commercial applications. The existence of an open trillion-parameter model with strong reasoning capabilities changes what downstream practitioners can build, from specialized math tutors to code assistants to scientific reasoning tools.

Where Prior Approaches Fall Short

The paper identifies limitations in prior work along four dimensions: architecture scaling methodology, reasoning-oriented pre-training, post-training reasoning optimization, and infrastructure for sparse models.

1. Scaling laws exist but are piecemeal. The literature on neural scaling laws—from Kaplan et al. (2020) through Chinchilla (Hoffmann et al., 2022) to Clark et al. (2022) for MoE—provides relationships between compute, model size, and data quantity. However, these laws are typically derived for specific architectures and do not provide a unified framework that simultaneously predicts:

Optimal hyperparameters (learning rate, batch size) for MoE models at scale
The tradeoff between model size and data quantity specifically for sparse architectures
How architectural choices (activation ratio, expert granularity) translate to efficiency gains at different compute budgets

The paper's Ling Scaling Laws fill this gap by deriving power-law relationships across all three axes from over a thousand experiments, enabling principled extrapolation from small-scale trials to trillion-parameter design decisions. Without this, trillion-parameter MoE design would be guesswork—expensive guesswork at that.

2. MoE architectural efficiency is poorly characterized at extreme sparsity. Prior work on MoE (Shazeer et al., 2017; Fedus et al., 2022; DeepSeek-AI, 2024) has explored various activation ratios and routing strategies, but the relationship between sparsity and efficiency gains—especially at the very high sparsity levels Ling 2.0 employs (3.5% activation)—was not systematically quantified. The paper's finding that "efficiency leverage" follows a power law with activation ratio, and that this leverage amplifies at larger compute budgets, is a novel empirical contribution that directly informs the trillion-parameter design.

3. Reasoning is typically treated as a post-training concern. Standard LLM training pipelines treat pre-training as a general knowledge acquisition phase and reasoning as something added later through supervised fine-tuning and RLHF. This sequential approach means that the pre-training data mixture, context window, and learning rate schedule are not optimized for reasoning capability. The model develops general language competence, and reasoning capabilities are layered on top—often inefficiently and unstably.

Ling 2.0 breaks from this paradigm by making reasoning a first-class objective throughout the training pipeline:

Pre-training data includes 32–46% reasoning data (math and code), not the typical 5–15%
Mid-training introduces chain-of-thought data to "pre-activate" reasoning before RL ever begins
The WSM scheduler (replacing learning rate decay with checkpoint merging) is designed to preserve reasoning gains through subsequent training stages

The paper's ablation in Figure 12 demonstrates the concrete benefit: DFT models pretrained with CoT data achieve higher ApexEval scores (a measure of RL potential) and faster improvement during evolutionary reasoning RL compared to models without CoT pre-training.

4. Post-training reasoning optimization suffers from granularity mismatch. Prior approaches to reasoning-oriented RL—such as GRPO (Shao et al., 2024) and GSPO (Zheng et al., 2025)—operate at the token or sequence level for policy updates. The paper argues these granularities create a mismatch with the natural structure of reasoning: reasoning proceeds in sentences, with each sentence representing a coherent unit of logical inference. Token-level optimization can disrupt this coherence; sequence-level optimization is too coarse to capture intermediate reasoning quality.

LPO (Linguistic-unit Policy Optimization) addresses this by treating sentences as the fundamental action units for importance sampling and clipping. The empirical result—~10% improvement on complex reasoning benchmarks compared to token-level and sequence-level baselines—is substantial.

5. Infrastructure for trillion-parameter sparse models is underdeveloped. Training a trillion-parameter MoE model in FP8 precision with heterogeneous pipeline stages (dense layers, MoE layers, MTP layers with different computational profiles) poses infrastructure challenges that prior work had not fully solved. The paper identifies specific bottlenecks that existing frameworks (Megatron, DeepEP) do not address: pipeline bubbles from heterogeneous TransformerBlocks, communication overhead from high expert counts, FP8 precision tracking for large-scale training, and the need for cross-platform reproducibility.

The "bitter lesson" in Section 5.6—that computation-communication overlapping techniques like DualPipe yielded limited end-to-end gains due to routing imbalance in shallow MoE layers and gating by slowest-rank in large EP configurations—is a particularly honest acknowledgment that many "obvious" optimizations fail at scale, and that co-design between routing strategy and communication scheduling is necessary.

How This Paper Positions Itself

The paper does not claim to invent Mixture-of-Experts, scaling laws, reasoning-oriented training, or RL-based post-training. Rather, it positions itself as the integrator that brings these components together under a unified principle—"every activation boosts reasoning capability"—and validates that principle at the trillion-parameter scale.

The positioning can be understood through four design philosophies that cut across the paper's architecture, pre-training, post-training, and infrastructure sections:

1. Scaling must be principled, not empirical. The Ling Scaling Laws are not a one-off analysis but a methodology—the "Ling Wind Tunnel"—that enables cheap, reliable extrapolation from models as small as 500M parameters to trillion-parameter designs. This is positioned as a contrast to traditional ablation studies that train a single model at one scale and hope the conclusions transfer. The Wind Tunnel's 35% cost savings and 100× extrapolation factor make it a meta-contribution that could influence how other large-scale LLM projects approach design decisions.

2. Reasoning is a property of the entire training pipeline, not a post-hoc add-on. The introduction of CoT data during mid-training, the proportional increase of reasoning data during pre-training, and the WSM scheduler's preservation of reasoning gains all reflect a philosophy that reasoning capability must be cultivated from the earliest stages of training. This contrasts with models like GPT-4 or Claude, where reasoning emerges primarily through post-training RLHF, and with DeepSeek-V3, which emphasizes efficient sparse scaling but treats reasoning as relatively separable from architectural design.

3. Efficiency and reasoning are not tradeoffs but mutually reinforcing goals. The "high-sparsity, fine-grained" architecture activates 3.5% of parameters per token, which is extreme by current standards. The paper argues that this sparsity, rather than degrading reasoning, actually enables it at scale because it allows the model to have many more total parameters (and thus more specialized expert capacity for different reasoning patterns) within a fixed compute budget. The shared expert design—one expert always activated—provides a common representation backbone, while the 256 routed experts specialize in different reasoning strategies. This is a direct empirical counter to the intuition that sparsity fragments knowledge.

4. Infrastructure is a first-class research contribution, not an afterthought. The paper's detailed discussion of FP8 precision tracking (the "FP8 Training Safeguard System" with real-time monitoring of underflow and distortion metrics), heterogeneous pipeline scheduling (40% throughput improvement from interleaved 1F1B with fine-grained MTP partitioning), and cross-platform reproducibility (loss curves aligning to within 0.1% across GPU architectures) reflects a view that infrastructure innovation is inseparable from model capability. A trillion-parameter model that cannot be trained efficiently or evaluated rapidly is not practically useful, regardless of its theoretical reasoning ability.

The Gap This Paper Fills, Concretely

Before Ling 2.0, the landscape contained:

Dense models (GPT-4, Gemini, Qwen) that reason well but are expensive to scale
Sparse models (DeepSeek-V3, Kimi-K2) that scale efficiently but have not fully integrated reasoning-oriented training across the entire pipeline
Scaling laws for hyperparameters or data allocation, but not a unified framework that simultaneously predicts optimal architecture, training recipe, and efficiency leverage for MoE
Reasoning-oriented training (STaR, ReST $^{EM}$ , GRPO) applied at modest scales or as post-hoc fine-tuning, but not baked into trillion-parameter pre-training and mid-training

Ling 2.0 aims to occupy the intersection: a trillion-parameter model that is (a) trained efficiently under principled scaling laws, (b) reasoning-oriented from pre-training through post-training, (c) supported by infrastructure designed for extreme sparsity and FP8 precision, and (d) fully open-source. Whether it succeeds in all dimensions is an empirical question the evaluation sections address, but the motivation is clear: this intersection is where the next generation of open reasoning models needs to live.

3. Technical Approach

3.1 Reader Orientation

Ling 2.0 is a trillion-parameter Mixture-of-Experts language model family—a neural network where only a small fraction of parameters activate for any given input token, dramatically reducing computation while preserving the representational capacity of a much larger dense model. The system solves the dual problem of how to scale a language model to one trillion parameters without proportional cost increase, while simultaneously ensuring that every activated parameter contributes to reasoning capability—accomplished through a unified pipeline where architectural decisions, data composition, training recipes, and infrastructure are all designed to amplify reasoning from the earliest stages of training through deployment.

3.2 Big-Picture Architecture (Diagram in Words)

The Ling 2.0 system comprises four interconnected subsystems that span the model's entire lifecycle:

Architecture Design Engine (Ling Scaling Laws + Ling Wind Tunnel): A framework of empirically-derived power laws that predicts optimal hyperparameters, model-data allocation, and architectural efficiency for MoE models at any compute budget. This is used before training to design the trillion-parameter architecture without expensive trial-and-error at scale. It determines: how many layers, how many experts, what activation ratio, what learning rate, and what batch size.
Pre-Training and Mid-Training Pipeline: A multi-stage training process that consumes 20T+ tokens of general and reasoning data across three phases—two pre-training stages (20T tokens total, 4K context) where reasoning data increases from 32% to 46% of the mixture, and one mid-training stage (750B tokens, 32K context) that extends the context window to 128K and introduces Chain-of-Thought data to "pre-activate" reasoning capabilities before any fine-tuning occurs. The WSM (Warmup-Stable-Merge) scheduler replaces traditional learning rate decay with checkpoint averaging.
Post-Training Reasoning Stack: A three-stage process that transforms the base model into an instruct model: (i) Decoupled Fine-Tuning (DFT) with dual system prompts (instant-response vs. in-depth reasoning modes) to establish a reasoning-capable initialization; (ii) Evolutionary Chain-of-Thought (Evo-CoT) reinforcement learning using Linguistic-unit Policy Optimization (LPO) that performs sentence-level policy updates with dynamic length control, automatically scaling reasoning depth to problem difficulty; (iii) Group Arena Reward (GAR) for human preference alignment in open-ended tasks, using tournament-style pairwise comparisons rather than absolute scoring.
Distributed Training Infrastructure: A full-stack FP8 training framework with fine-grained quantization, heterogeneous pipeline parallelism that handles the different computational profiles of dense layers, MoE layers, and Multi-Token Prediction (MTP) modules, and distributed evaluation pipelines supporting trillion-parameter inference across multi-node Ray clusters.

Information flows sequentially: the Scaling Laws determine the architecture → the pre-training pipeline builds base reasoning capability → mid-training extends context and pre-activates CoT reasoning → DFT establishes a dual-mode reasoning policy → Evo-CoT + LPO deepens adaptive reasoning → GAR aligns with human preferences. At every stage, the infrastructure layer provides efficiency (FP8, heterogeneous pipelines) and reliability (precision tracking, loss spike monitoring).

3.3 Roadmap for the Deep Dive

First, the Ling Scaling Laws and Wind Tunnel—because all architectural and hyperparameter decisions (number of experts, activation ratio, learning rate, batch size, depth-to-width ratios) flow from these laws. Without understanding how the paper predicts trillion-parameter behavior from 500M-parameter experiments, the design choices are arbitrary.
Second, the MoE architecture itself—the concrete instantiation of "high-sparsity, fine-grained" design with 256 routed experts, 8 active per token, 1 shared expert, MTP, and QKNorm. These are the architectural "knobs" the scaling laws set.
Third, the pre-training and mid-training pipeline—how the data mixture, context window extension, CoT pre-activation, and WSM scheduler work together to build reasoning capability from scratch. The data infrastructure (Data-as-Code, Unified Lakehouse) is covered here as well.
Fourth, the post-training stack (DFT, Evo-CoT, LPO, GAR)—how the base model is converted into a reasoning-oriented instruct model capable of adaptively scaling its computation to problem difficulty.
Fifth, the infrastructure—FP8 training with precision safeguards, heterogeneous pipeline parallelism, the "bitter lesson" about computation-communication overlap, and the evaluation pipeline. This comes last because it supports everything above but is most meaningful once the algorithmic components are understood.

3.4 Detailed, Sentence-Based Technical Breakdown

This is a systems paper whose core idea is that reasoning capability in language models emerges from the coordinated design of architecture, data, training recipe, and infrastructure—with every design choice made to ensure that each activated parameter contributes maximally to reasoning, guided by empirical scaling laws that enable reliable extrapolation from small-scale experiments to trillion-parameter models.

Ling Scaling Laws: Predicting Trillion-Parameter Behavior from Small-Scale Experiments

The Ling Scaling Laws are not a single equation but a family of three power-law relationships derived from over a thousand training experiments across compute budgets up to $3\times 10^{20}$ FLOPs using a Warmup–Stable–Decay (WSD) scheduler on models with 64 experts (4 active) plus 1 shared expert. Their purpose is to answer three questions that must be resolved before training a trillion-parameter model: (1) What learning rate and batch size should we use? (2) How should we allocate compute between model parameters and training tokens? (3) What MoE architectural configuration (activation ratio, expert granularity) maximizes efficiency at our target scale?

Scaling Law 1: Optimal Hyperparameters as a Function of Compute. The paper fits power-law relationships between total compute budget $C$ (measured in FLOPs) and the optimal batch size $B_{\text{opt}}$ and optimal learning rate $\eta_{\text{opt}}$ :

$B_{\text{opt}} \propto C^{a_B}, \quad \eta_{\text{opt}} \propto C^{a_\eta}$

where $a_B$ and $a_\eta$ are fitted exponents derived from experiments where hyperparameters were swept at each compute scale and configurations whose validation loss was within 0.25% of the minimum at that scale were labeled "near-optimal."

What it computes: For a target total training FLOPs budget (e.g., the $C$ for Ling-1T), these equations output the batch size and learning rate that will minimize final validation loss, without requiring hyperparameter sweeps at the target scale. The batch size $B_{\text{opt}}$ determines how many sequences are processed per optimizer step; the learning rate $\eta_{\text{opt}}$ determines the step size for weight updates.

Why this form: The power-law form is standard in scaling laws literature (Kaplan et al., Hoffmann et al.) and captures the empirical observation that optimal hyperparameters change smoothly with compute, not abruptly. A key difference from dense models: MoE models require larger batch sizes at equivalent compute because each expert sees only a fraction of the tokens in each batch (due to sparse routing), so more total tokens per step are needed to ensure stable gradient updates for each expert. The paper states: "at larger compute scales, MoEs tend to use larger batch sizes and relatively lower learning rate. We attribute this phenomenon to MoEs' sparse gradient updates: since only a subset of tokens in each batch contributes to the gradient update for any given expert, a larger batch size is necessary to ensure stable and effective training."

Scaling Law 2: Optimal Model-Data Allocation. The second law addresses the classical scaling tradeoff: given a fixed compute budget $C = M \cdot D$ (where $M$ is model FLOPs-per-token and $D$ is training tokens), what is the optimal balance between model size and data quantity for MoE architectures?

The paper finds that for any given compute budget:

$M_{\text{opt}}^{\text{MoE}} < M_{\text{opt}}^{\text{dense}}, \quad D_{\text{opt}}^{\text{MoE}} > D_{\text{opt}}^{\text{dense}}$

meaning the compute-optimal MoE model has fewer parameters but is trained on more data compared to its compute-optimal dense counterpart. This is shown in Figure 1b.

What it computes: Given a total FLOPs budget, this law tells you how many parameters your MoE model should have (and consequently, how many tokens you need to prepare for training). For the trillion-parameter Ling-1T, this directly determined the 20T+ token pre-training corpus.

Why this form: The result that MoE models should be smaller-but-trained-longer than dense models at equivalent compute follows from MoE's effective capacity: because each token only activates a subset of parameters, the model can process more tokens before overfitting relative to its activated parameter count. This means data, not model size, is the binding constraint for MoE efficiency—a finding that justifies the paper's massive investment in data curation (Section 3.1) and explains why the pre-training corpus is reasoning-heavy (more tokens = more need for high-quality data).

Scaling Law 3: Architectural Efficiency Leverage. This is the most novel of the three laws and directly determines the MoE architecture of all Ling 2.0 models. The metric is Efficiency Leverage (EL), defined as:

$\text{EL} = \frac{\text{Compute required for a dense model to reach loss } L}{\text{Compute required for an MoE model to reach the same loss } L}$

In words: how many times more efficient is an MoE configuration compared to a dense model at achieving the same validation loss. An EL of 7 means the MoE achieves in 1 unit of compute what the dense model needs 7 units for.

The unified EL scaling law integrates three factors:

$\text{EL}(A, G, C) = \hat{A}^{\alpha + \gamma(\log G)^2 + \beta \log G}$

where:

$A$ is the expert activation ratio (fraction of experts activated per token, e.g., 9/257 ≈ 3.5% for Ling 2.0)—the primary driver of efficiency
$G$ is the expert granularity (number of experts activated per token, e.g., 8 for Ling 2.0)—a nonlinear modulator
$\hat{A}$ is a saturating transformation of $A$ from Clark et al. (2022), preventing the exponent from diverging as $A \to 0$
$\alpha = a + d \cdot \log C$ is the compute-dependent scaling exponent, where $d > 0$ is the key finding: efficiency leverage amplifies with compute—larger training budgets make the same MoE configuration relatively more efficient compared to dense
$\beta$ and $\gamma$ define a log-polynomial modulation from expert granularity, capturing the empirical observation that there is an optimal range of ~8–12 activated experts (too few and the model lacks capacity; too many and the efficiency gain diminishes)

What it computes: For any candidate MoE architecture (specified by activation ratio $A$ , expert count and granularity $G$ , and total compute budget $C$ ), this equation predicts the multiplicative efficiency gain over a dense model at the same compute. The prediction is then used to select the architecture that maximizes EL.

Why this form: The power-law relationship between EL and activation ratio is empirically robust across the 300+ models in the study, but using $\hat{A}$ rather than raw $A$ prevents the pathological case where EL → ∞ as activation → 0 (in reality, extremely low activation ratios degrade performance because the model lacks capacity). The log-polynomial term in $G$ captures the non-monotonic effect of expert granularity: increasing from 1 to 8 experts improves EL, but going from 8 to 16 shows diminishing returns (Figure 2a, right). The compute-amplification term $d \cdot \log C$ is the crucial insight that sparsity becomes more valuable at larger scales—the same MoE configuration that gives ~4× EL at a 100B-parameter scale might give ~7× at the trillion-parameter scale because the fixed overhead of routing and the shared expert becomes a smaller fraction of total computation.

Four empirical principles discovered:

Activation ratio is the primary driver. EL is predominantly determined by how sparse the model is, following a robust power law. Even at extremely low activation ratios like 1/128, the relationship remains quantifiable.
Expert granularity acts as a nonlinear modulator. Beyond the activation ratio effect, expert granularity induces a log-polynomial adjustment to EL that is largely independent of compute budget, implying a stable optimal range of 8–12 activated experts. This is empirically identified in Figure 2a.
Compute budget has an amplification effect. EL for a given MoE architecture is not fixed; it scales with training compute following another power law. Larger training runs amplify the efficiency advantage of sparsity.
Secondary architectural factors have minor effects. Arrangements of shared experts or MoE layers admit broadly applicable near-optimal settings and do not require fine-grained tuning.

Concrete outcome for Ling 2.0: Based on these laws, all Ling 2.0 models adopt the same "high-sparsity, fine-grained" configuration: 256 routed experts, 8 activated per token, plus 1 shared expert, yielding a ~3.5% overall activation ratio. The scaling law predicts over 7× efficiency leverage for this configuration, which the paper empirically confirms by showing Ling-mini-2.0 (1.4B activated) matches dense ~8B models and Ling-flash-2.0 (6.1B activated) matches dense ~36–40B models.

The fitting procedure used Huber loss and BFGS optimization (following Hoffmann et al., 2022), and the parameters $(a, d, \beta, \gamma)$ were fit jointly from the 300+ model experiments. The model used for fitting initially fixed 64 experts (4 active) plus 1 shared expert to simplify the search space, and the resulting laws were verified to generalize to different activation ratios.

The Ling Wind Tunnel: A Standardized Experimental Pipeline for 100× Extrapolation

While the scaling laws answer what architecture to build, the Ling Wind Tunnel answers a more practical question: how do we cheaply evaluate whether a novel feature (e.g., a new attention mechanism, a different routing strategy, a modified loss function) will scale favorably to the trillion-parameter regime?

The Wind Tunnel is a standardized experimental protocol consisting of five models ranging from 500M to 8B parameters, distributed according to a power law (Figure 3a, green points). For each model in the tunnel:

Architecture is determined by the Ling Scaling Laws for MoE efficiency (Section 2.3.2)—not arbitrary choices. This ensures the small-scale model is architecturally proportional to the trillion-parameter target.
Training data allocation follows the optimal model-data scaling law (Section 2.3.1, Figure 1b)—each model is trained to the FLOPs count corresponding to its compute-optimal allocation, preventing over- or under-training relative to its size.
Hyperparameters (LR, batch size) are set by the optimal hyperparameter scaling laws (Section 2.3.1, Figure 1a)—ensuring the model is trained near-optimally at its scale.

The result is that a candidate feature (e.g., "should we use MTP depth 1 or 2?") can be tested across all five tunnel models, and its effect on the training loss curve can be observed at multiple compute scales (Figure 3b). Because the scaling behavior is fit to a power law, the trend at small scales can be extrapolated 100× to predict the feature's impact on Ling-1T.

Cost comparison: The five Wind Tunnel experiments collectively consume only 35% of the compute that training a single Ling-mini-2.0 on 400B tokens would cost (the traditional ablation approach, shown as the black point in Figure 3a). More importantly, the multi-scale observations are "significantly more stable and reliable than those derived from a single experimental 'slice'" because they capture the scaling trend rather than a point estimate at one scale.

What this enables: The Wind Tunnel is how the paper validates design choices like:

MTP depth = 1 (tested at multiple scales, shown to consistently improve math and code performance)
MTP loss weight = 0.1 (swept across tunnel models to find the weight that scales best)
First-K-Dense strategy (tested for whether dense initial layers improve routing balance without hurting scaling)
Aux-loss-free load balancing with bias centered around zero (compared against alternatives in the tunnel)

Why this matters beyond Ling 2.0: The Wind Tunnel methodology is a meta-contribution that could influence how other large-scale LLM projects approach design decisions. Instead of the expensive guess-and-check of training multiple 100B+ models with different configurations, projects could adopt a standardized small-scale pipeline for vetting features before committing to full-scale training.

Model Architecture: High-Sparsity MoE with MTP

The Ling 2.0 architecture is a decoder-only transformer with three distinguishing features: (1) an MoE feed-forward network layer where most parameters reside in expert modules, (2) auxiliary-loss-free load balancing to ensure tokens are distributed evenly across experts, and (3) Multi-Token Prediction (MTP) as an auxiliary training objective that predicts future tokens at each position.

Base Transformer Configuration. Every Ling 2.0 model uses:

Grouped-Query Attention (GQA) with 8, 16, or 32 key-value heads depending on scale (Table 1), reducing KV cache size during autoregressive decoding by sharing key-value heads across query heads. Ling-mini-2.0 uses 16 attention heads, Ling-flash-2.0 uses 32, Ling-1T uses 64.
SwiGLU activation in feed-forward layers—the gated variant of GELU that multiplies the output of two linear projections before applying the activation, improving representational capacity.
RMSNorm with pre-normalization—layer normalization applied before attention and feed-forward sublayers (not after), which stabilizes training at large scales.
Partial RoPE (Rotary Position Embedding) applied only to the first 64 dimensions of each attention head. Standard RoPE applies rotation to all head dimensions, but the paper finds that restricting it to the first 64 dimensions improves length extrapolation—the model's ability to handle sequences longer than those seen during training. This is critical for the mid-training context extension from 4K to 128K.
QKNorm (Query-Key Normalization)—an additional LayerNorm applied to the query and key projections before computing attention scores. This is not standard in most transformers. The paper justifies it specifically for FP8 training stability: "we verify [QKNorm] to significantly improve stability under low-precision training" and in Section 5.1, they explain that severe outlier phenomena were observed in both activations and the gradient $\frac{\partial L}{\partial y}$ within the attention.linear_qkv layer, which amplified progressively with layer depth and caused substantial FP8 quantization errors. QKNorm suppresses these outliers, reducing precision loss across all FP8 modules.
BBPE tokenizer with 156K tokens (extended from Ling 1.5's 128K, with targeted additions for multilingual coverage).

Sparse MoE Architecture. The MoE design is what gives Ling 2.0 its efficiency. Each MoE transformer layer contains:

1 shared expert: A feed-forward network that processes every token, regardless of routing decisions. This provides a common representational backbone that all tokens pass through, preventing the fragmentation that could occur if every token were processed by entirely different expert combinations. The shared expert's intermediate size scales with model size (512 for Ling-mini-2.0, 1024 for Ling-flash-2.0, 2048 for Ling-1T).
256 routed experts: Feed-forward networks where only 8 are activated per token. The routing decision is made by a learned gate (a small linear layer) that outputs a score for each of the 256 experts. The top-8 scoring experts are selected, and their outputs are weighted by the softmax-normalized gate scores and summed. The experts are "fine-grained," meaning each is relatively small (expert intermediate size = 512/1024/2048 for the three model scales) compared to having fewer, larger experts.
First-K-Dense strategy: The first $K$ layers of the model are standard dense transformer layers rather than MoE layers. $K = 1$ for Ling-mini-2.0 and Ling-flash-2.0, $K = 4$ for Ling-1T. The paper's justification is that "this approach reduces the total parameter count while maintaining equivalent model performance and improving routing balance." Shallow MoE layers tend to have more imbalanced routing (some experts receive many more tokens than others), which creates pipeline bottlenecks in distributed training (Section 5.6). Making the initial layers dense prevents this imbalance in the earliest stages of processing.
Overall activation ratio ~3.5%: Total activated parameters per token = 8 routed experts + 1 shared expert + attention parameters = 9 expert-equivalent FFNs out of 257 total, plus attention. Since attention parameters are a relatively small fraction of the total, the effective activation is $9/257 \approx 3.5\%$ of FFN parameters.

Auxiliary-Loss-Free Load Balancing. In standard MoE training, an auxiliary loss is added to the training objective to penalize imbalanced routing (some experts receiving many more tokens than others). Ling 2.0 follows DeepSeek-V3's approach of eliminating the auxiliary loss and instead using dynamic expert biases:

$b_i = b_i + u \times \left(\text{sign}(e_i) - \text{mean}(\text{sign}(e))\right)$

where:

$b_i$ is the bias for expert $i$ (added to the gate score before top-8 selection)
$u$ is the update rate (set to 0.001 during pre-training, reduced to 0.0001 after context extension)
$e_i$ is the violation error for expert $i$ —a measure of how much its load deviates from the ideal uniform distribution
$\text{sign}(e_i)$ is +1 if the expert is overloaded, -1 if underloaded
$\text{mean}(\text{sign}(e))$ is the average sign across all experts, subtracted to keep biases centered around zero

What it computes: If an expert is receiving too many tokens ( $e_i > 0$ ), its bias is reduced (making it less likely to be selected). If it is receiving too few tokens ( $e_i < 0$ ), its bias is increased. The bias update is applied after each training step.

Why this form: The auxiliary-loss approach creates a tension between the primary language modeling objective and the load-balancing objective—the model is trying to minimize two losses that may conflict. The bias-based approach separates these concerns: the language modeling loss optimizes routing for prediction quality, while the bias update ensures load balance as a post-hoc adjustment. The subtraction of $\text{mean}(\text{sign}(e))$ is a subtle modification from DeepSeek-V3's original formulation, keeping biases centered around zero to prevent drift over long training runs. The paper also uses dropless routing (tokens are always routed to exactly 8 experts, never dropped due to capacity limits) and group routing (experts are divided into 8 groups; each token is routed within the top-4 scoring groups) to improve training efficiency without performance degradation.

Additional stability measures: router gate scaling with a factor of 2.5 is applied to stabilize the root mean square of the gate outputs before the softmax, preventing the gate from becoming too peaky (which would route most tokens to the same few experts).

Multi-Token Prediction (MTP). MTP is an auxiliary training objective where, in addition to predicting the next token at each position, the model also predicts tokens at future positions (e.g., token $t+2$ , $t+3$ ). The standard language modeling objective is:

$\mathcal{L}_{\text{LM}} = -\sum_{t} \log P(y_t | y_{<t})$

MTP adds additional terms:

$\mathcal{L}_{\text{MTP}} = -\sum_{k=1}^{D} \sum_{t} \log P(y_{t+k} | y_{<t+k})$

where $D$ is the MTP depth (number of future positions to predict). Ling 2.0 uses $D = 1$ (predicting only the immediate next token through the MTP mechanism, in addition to the standard next-token prediction) and an MTP loss weight of 0.1.

What it computes: The MTP module is implemented as an additional transformer layer (or layers, if depth > 1) appended after the main model's final layer. Its input is the main model's hidden states, concatenated with additional embeddings, and it produces its own output logits. The total training loss is:

$\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{LM}} + 0.1 \cdot \mathcal{L}_{\text{MTP}}$

Why this form: MTP serves two purposes. First, it acts as a regularizer that encourages the model to build richer internal representations at each position—to predict what comes two steps ahead, the model must learn more about the current state. Second, it improves inference efficiency through speculative decoding: at inference time, the MTP module can propose candidate future tokens that are verified by the main model, potentially generating multiple tokens per forward pass rather than one. The paper reports that MTP "consistently improves performance on code and math tasks across different model scales" (validated through the Wind Tunnel). The depth of 1 and weight of 0.1 were chosen based on scaling trends observed in the Wind Tunnel, balancing the performance benefit against the additional computational overhead of the MTP module.

The MTP module introduces significant engineering challenges because its computational profile differs from standard MoE layers—approximately 1.7× the computation cost of a standard MoE layer—which the heterogeneous pipeline parallelism in Section 5.2 addresses.

Architectural Scaling Across Model Sizes. Table 1 provides the concrete instantiation for the three released models. Key scaling patterns: hidden size doubles from 2048 (mini) → 4096 (flash) → 8192 (1T); expert intermediate size also doubles; attention heads double; layers roughly double (20 → 32 → 80). The learning rate decreases with scale (3.36e-4 → 2.61e-4 → 1.86e-4), following the Scaling Law 1 prediction that larger models need smaller learning rates. Batch size increases substantially (4400 → 8352 → 18144), also following the scaling law.

All parameters are randomly initialized with standard deviation 0.006.

Pre-Training: Reasoning-Oriented Data and Multi-Stage Training

The pre-training pipeline is designed to build a model that reasons well before any supervised fine-tuning or RL occurs—the hypothesis being that a model with inherent reasoning strengths provides a better foundation for post-training than one where reasoning must be entirely learned through fine-tuning.

Data Composition and Curation. The pre-training corpus is divided into two broad categories: general knowledge data (web pages, books, papers, Wikipedia) and reasoning data (math and code). The overall pre-training consumes 20T tokens with a 4K context window, followed by 750B tokens of mid-training with extended context.

General Knowledge Data Pipeline:

Extraction: Web content is extracted using the trafilatura parser; PDF and HTML parsers are continuously improved for accuracy.
Multi-stage cleaning: Rule-based and model-based pipelines tailored to data type. Sampling-based checks identify common low-quality patterns (ads, embedded URLs, symbol-heavy texts, malformed Markdown and tables).
Automated low-quality detection: An iterative pipeline that (1) recalls suspect samples using classifiers, lightweight LLM scoring, and perplexity (PPL), (2) analyzes issues via LLM categorization, (3) generates cleaning rules, (4) generalizes rules through LLM-driven abstraction of similar cases. Human review validates new rules before integration.
High-quality filtering: Feature models trained per data type (Chinese web, English web, books, papers) to assess quality, education level, knowledge density, and domain. The English web subset is claimed to be "5× larger than FineWeb-Edu and outperforms it on knowledge benchmarks."
Knowledge text rewriting: For complex or rare knowledge in raw text, a recall-rewrite pipeline selects candidate texts by knowledge density, STEM domain, and QA features, then applies semi-synthetic rewriting (Wikipedia-style structuring, QA conversion, concise summaries). Ablations show "consistent gains on MMLU, CMMLU, and CEval benchmarks."

Reasoning Data Pipeline (Ling Code Corpus): The code corpus integrates source code, code-related natural language, and synthetic instructional data from multiple sources:

Raw GitHub source code: Multi-stage curation with language-specific fine-grained cleaning rules, Lint-based syntactic validation (removing files with compilation or structural errors). This yields a corpus covering 660 programming languages.
Quality stratification: Along three dimensions—code style/readability, norm adherence/structure, and complexity/difficulty—producing a 600B token top-quality subset.
Code rephrasing: Generating 300B tokens of augmented code data through rephrasing and paraphrasing techniques.
Commit reconstruction: Replaying event sequences (pull requests, issues, merges) at the repository level from GHArchive to produce commit-level records capturing developer intent, revision rationale, and contextual discussions.
Common Crawl code-related data: Two-stage recall (broad then fine) targeting code-related pages, tutorials, and developer discussions, yielding 700B tokens raw → 140B+ refined.
Programming competition data: Problem statements, user submissions, and discussion threads from diverse platforms.
Synthetic data: Compositional augmentation from programming platforms, library references, and programming concepts to cover broader coding topics.

Validation: A 1B model (Ling-coder-1B) trained from scratch on 2T tokens of code data achieves performance comparable to Qwen2.5-Coder-1.5B-Base on mainstream benchmarks (Figure 4a), providing a lightweight proxy for code data quality before scaling to larger models.

Reasoning Data Pipeline (Ling Math Corpus): The math corpus follows a similar multi-stage philosophy:

Parsing improvement: Iteratively improved PDF and HTML parsers for mathematical content completeness.
Recall: fastText classifiers for high-recall math data identification within candidate pools.
Filtering and refinement: Fine-tuned 4B-parameter LLM-Filter and LLM-Refiner models that identify and refine data containing mathematical knowledge or step-by-step problem-solving processes.
Synthetic generation: Q&A pairs extracted from web and books; a sophisticated question generator producing high-quality, realistic mathematical problems; a large-scale mathematical concept graph (Chen et al., 2025) containing thousands of nodes and millions of edges that extends knowledge boundaries by generating novel problem variations.
Deduplication: MD5 and MinHash applied across sources.

Validation: A 1B model (Ling-math-1B) trained via continual training on the Ling-coder-1B checkpoint with 1.8T math tokens exhibits performance superior to Qwen2.5-Math-1.5B-Base and Qwen3-1.7B-Base on mainstream math benchmarks (Figure 4b). A specific comparison of the Ling math web data against open-source math web datasets (Infi-mm-math, finemath-3plus, megamath, nemotron-cc) shows markedly superior performance (Figure 4c), validating the specialized acquisition and refinement pipeline.

Multilingual Data:

2TB of high-quality multilingual data from open web sources (CulturaX, WanJuan) and parallel corpora (OPUS, MultiUN).
~30 languages covering Romance, Germanic, Slavic, and other language families.
4% of total pre-training data—a proportion empirically determined to significantly improve minor language performance while maintaining Chinese and English capabilities.
Language family sensitivity: Romance and Germanic language data have less negative impact on core language benchmarks; data from some other language families (especially Arabic, Japanese) requires more careful balancing to avoid degrading Chinese/English performance.

Long-Context Data: A retrieve–synthesize–validate pipeline over heterogeneous sources (web pages, books/novels, scientific articles, software docs) with three quality controls:

Linguistic hygiene: Rule checking and model recognition to identify/repair paragraph duplication, language mixing, and content truncation.
Semantic consistency: Model-aided detection and manual observation to detect logical contradictions, optimizing recall/synthesis logic accordingly.
Long-range quality scoring: Elimination of low-quality long text using the PPL gap between long-window and short-window evaluations. The pipeline yields approximately 1.2T high-quality long-text tokens.

Data Infrastructure: Data-as-Code and Unified Lakehouse. Two principles govern the data processing infrastructure:

Data-as-Code: The entire data pipeline is codified and version-controlled (Git), integrated into an automated CI/CD system. A unified AIDataOps library with 50+ data operators across modalities enables transparent, reproducible, and automated feature development. The paper claims this reduced R&D iteration cycles "from months to days."
Unified Data Lakehouse and Wide-Table Architecture: Hundreds of scattered datasets are consolidated into a unified logical wide table for major domains. This supports elastic scalability (new features added without full-table rebuilds) and achieves over 20 TB/hour I/O throughput, positioning data processing as no longer a bottleneck for large-scale training.

Multi-Stage Training Recipe. The pre-training proceeds in three stages, illustrated in Figure 5:

Stage 1: Pre-Training Sub-stage 1 (10T tokens, 4K context):

Data mixture: 68% general data, 32% reasoning data (math + code)
Hyperparameters from Scaling Law 1 (Table 1)
Batch-size ramp for the first ~500B tokens (e.g., from 3,024 to the peak for Ling-1T), then constant
WSM scheduler: linear warmup for first 2,000 steps to peak LR, then constant LR until training ends

Stage 2: Pre-Training Sub-stage 2 (10T tokens, 4K context):

Data mixture shifts to 54% general data, 46% reasoning data—reasoning proportion increased
Corpus quality enhanced and more stringent data decontamination applied
Same context length and hyperparameters

The progressive increase in reasoning data proportion (32% → 46%) is deliberate: early training benefits from broad knowledge acquisition, while later training benefits from more focused reasoning exposure to "activate and enhance the model's reasoning abilities, making Ling a model with inherent strengths in reasoning."

Stage 3: Mid-Training (750B tokens): This stage has two sub-phases:

Long Context Extension (150B tokens): 20% of sequences are 32K-length long-text sequences, while the remaining 80% maintain 4K. The data mixture is similar to Sub-stage 2. This expands the effective context window from 4K to 32K while maintaining short-context benchmark performance (performance on L-Eval and LongBench improves continuously). After this phase, YaRN (Peng et al., 2023) is applied to further extend the context window from 32K to 128K—a method that modifies the RoPE rotation frequencies to support extrapolated sequence lengths without additional training at those lengths.
Reasoning Pre-Activation (600B tokens): A high proportion of reasoning data is maintained and additional high-quality chain-of-thought (CoT) corpora are introduced. Training continues at a higher learning rate (since the data distribution has shifted toward more complex reasoning examples). Robust performance is achieved by merging mid-training checkpoints.

What "reasoning pre-activation" means operationally: The CoT data introduced in this phase consists of problems with explicit step-by-step solution traces. By training on this data during pre-training/mid-training (not just during fine-tuning), the model learns to associate problem statements with structured reasoning patterns at the parameter level—the weights themselves encode reasoning behaviors rather than just factual knowledge. The paper demonstrates (Figure 12, Section 4.5) that models pretrained with CoT data achieve higher ApexEval scores (a measure of RL potential) and faster improvement during subsequent evolutionary reasoning RL compared to models without CoT pre-training. Specifically, after DFT, the CoT-pretrained Ling-mini-2.0 achieved 41.9 on ApexEval vs. 34.5 without CoT; after ERL step 450, AIME 2024 scores were 26.5 vs. 19.7.

The paper also validates (Section 3.3.2, Tables 2-4) that integrating CoT data in mid-training leads to substantial gains on reasoning-intensive benchmarks like MATH (e.g., Ling-mini-2.0-base: 61.96 → 82.52), AIME25 (2.08 → 43.75), and LiveCodeBench (13.71 → 34.47), while maintaining performance on other benchmarks.

WSM (Warmup-Stable-Merge) Scheduler. Traditional LLM pre-training uses learning rate decay (typically cosine or linear) toward the end of training. The WSM scheduler replaces this with checkpoint merging during a final annealing phase. The theoretical justification draws an equivalence between the two approaches.

Theoretical equivalence: Merging $k+1$ consecutive checkpoints $[\theta_n, \theta_{n+1}, \ldots, \theta_{n+k}]$ with weights $\{c_j\}_{j=0}^k$ produces:

$\hat{\theta}_{n+k} = \sum_{j=0}^{k} c_j \theta_{n+j} = \theta_n - \sum_{i=1}^{k} w_i g_{n+i-1}$

where:

$\theta_{n+j}$ is the model parameters at step $n+j$
$c_j$ are the merge weights (non-negative, sum to 1)
$w_i$ are the effective gradient decay coefficients—these determine how much each past gradient contributes to the final merged model, with $w_i$ derived from $\{c_j\}$
$g_{n+i-1}$ is the gradient update at step $n+i-1$

What it computes: The merging operation produces a model $\hat{\theta}_{n+k}$ that is mathematically equivalent to having trained with a learning rate that decayed according to the schedule encoded by $\{w_i\}$ . If $w_i$ monotonically decreases (later gradients contribute less to the final model), this corresponds to a decreasing learning rate schedule. The conversion is invertible: given a desired LR decay schedule $\{w_i\}$ , the corresponding checkpoint merge weights are:

$c_k = w_k, \quad c_j = w_j - w_{j+1} \text{ for } j \in [1, k-1], \quad c_0 = 1 - \sum_{j=1}^k c_j = 1 - w_1$

Why this form: The advantage over standard LR decay is flexibility. With LR decay, you must commit to when decay starts and how long it lasts before training begins. If the model hasn't converged when decay starts, the decaying learning rate may prevent further improvement. With WSM, you train at constant LR for as long as needed, then post-hoc merge checkpoints to simulate any desired decay schedule. The paper shows (Figure 7) that WSM yields average improvements of +1 to +2 points across all benchmark categories compared to WSD (Warmup-Stable-Decay), with the advantage persisting through subsequent supervised fine-tuning.

Practical implementation: In the final stage of mid-training, the top-N checkpoints (selected by validation performance) are averaged. For the final Ling 2.0 models, $N = 32$ . The resulting model is used as the base for post-training.

Post-Training: From Base Model to Reasoning-Oriented Instruct Model

The post-training pipeline transforms the pre-trained base model into an instruct model capable of adaptive reasoning—solving complex problems with deep thought while responding efficiently to simple queries. This is accomplished in three stages.

Stage 1: Decoupled Fine-Tuning (DFT). Standard supervised fine-tuning trains a model on input-output pairs, often with a single system prompt or instruction format. DFT instead trains the model under two distinct system prompts that elicit different response modes (Table 5):

Instant Response mode (detailed think off): The model generates a direct answer {response} with no explicit reasoning trace. This mode is for straightforward queries where chain-of-thought would be wasteful.
In-Depth Reasoning mode (detailed think on): The model generates a structure with thinking tags containing a long chain-of-thought ({long-cot}), followed by response tags with the answer (<answer>{response}</answer>). This mode is for complex queries requiring multi-step reasoning.

What this accomplishes: DFT creates a single model with two behavioral modes that can be toggled by the system prompt, rather than training separate models for reasoning and non-reasoning tasks. More importantly, the in-depth reasoning mode serves as a warm-start for subsequent RL—the model already knows how to produce chain-of-thought when prompted, and RL can then optimize the quality of that reasoning without needing to first teach the model what a reasoning trace looks like.

The SFT data is carefully balanced across three domains:

Reasoning: Mathematical problem solving, STEM and logic reasoning, code generation, operations research, scientific inquiry
General: Creative writing, empathetic dialogue, socio-philosophical discussion
Industrial: Domain-specific tasks in finance, medical/health, production planning, supply chain, transportation optimization

ApexEval for checkpoint selection. Because DFT suppresses explicit chain-of-thought in instant-response mode, standard accuracy metrics may undervalue checkpoints that have high RL potential. ApexEval addresses this by using the highest score of pass@k (not greedy or average pass@k) to estimate the probability of producing at least one correct response in multiple attempts, effectively capturing the model's potential upper bound rather than its mode-seeking behavior. LLM-based judges (MathVerify, XVerify) assess answer validity for math, knowledge, and logic tasks, while test-case execution evaluates coding tasks, reducing misjudgment from format variability.

Stage 2: Evolutionary Chain-of-Thought (Evo-CoT) Reinforcement Learning. Evo-CoT starts from the DFT-initialized policy in instant-response mode (system prompt $s_{\text{instant}}$ ) and progressively deepens its reasoning capability. The key innovation is that the model learns to produce reasoning without explicit "thinking" tags—it is a "reflex-grade non-thinking model" that internally scales its reasoning depth based on problem complexity.

Given a user query $x$ , the policy $\pi$ generates a response $y \sim \pi(\cdot | x_{\text{inst}})$ where $x_{\text{inst}} = (s_{\text{instant}}, x)$ . At each training step $t$ , the policy is optimized via:

$\pi_{t+1} = \arg\max_{\pi} \mathbb{E}_{x \sim \mathcal{D}} \left[ \mathcal{J}\left(R(x, y), \theta\right) - \beta \cdot \text{KL}\left(\pi_\theta(\cdot | x_{\text{inst}}) \| \pi_{\text{ref}}(\cdot | x_{\text{inst}})\right) \right]$

where:

$\mathcal{J}(\cdot)$ is the RL policy update algorithm (LPO, described below)
$R(x, y)$ is a composite reward function
$\beta$ controls the KL divergence penalty from the reference policy $\pi_{\text{ref}}$ (the DFT checkpoint), preventing the model from diverging too far from its supervised initialization
$\text{KL}(\cdot \| \cdot)$ is the Kullback-Leibler divergence

What it computes: For each prompt in the dataset, the current policy samples a response, a reward is computed based on correctness and other criteria, and the policy is updated to increase the probability of high-reward responses while staying close to the reference policy. The KL penalty prevents reward hacking (the model finding degenerate solutions that score highly under the reward function but are not actually good responses).

Why this form: This is a standard KL-regularized policy optimization framework (similar to PPO in RLHF), but applied specifically to reasoning tasks. The KL penalty is critical because the reward function (particularly the correctness reward) is sparse—many reasoning steps may be partially correct but receive zero reward—and without regularization, the policy could collapse to producing only responses it is confident are correct, losing the exploration needed to discover improved reasoning strategies.

Composite Reward Function. The reward $R(x, y)$ comprises:

Correctness $R_{\text{correctness}}$ : +1 if the final answer matches ground truth, 0 otherwise. This is the primary signal.
Dynamic Length Control $R_{\text{length}}$ : This is the mechanism that implements "think more about hard problems, respond quickly to easy ones." The length preference function is:

$\hat{R}_{\text{length}} = \begin{cases} p(l), & \text{if } r_{\text{acc}} = 1 \\ \min\left(p(l), 0\right), & \text{if } r_{\text{acc}} = 0 \end{cases}$

where: $p(l) = 0.5 - \frac{l - \ell_{\min}}{\ell_{\max} - \ell_{\min} + 10^{-9}}$

Here $l$ is the token count of the sampled response, $\ell_{\min} = \min_k l_k$ and $\ell_{\max} = \max_k l_k$ are the minimum and maximum lengths among the $k$ sampled responses to the same input, and $r_{\text{acc}} \in \{0, 1\}$ is correctness.

What it computes: $p(l)$ maps response length to a score in $[-0.5, 0.5]$ —the shortest correct response gets +0.5, the longest gets -0.5, and everything else is linearly interpolated. For incorrect answers, the score is capped at 0 (excessively long incorrect responses are penalized, but short incorrect responses are not rewarded). The final length reward is $R_{\text{length}} = \alpha \cdot \hat{R}_{\text{length}}$ where $\alpha > 0$ is a difficulty-specific coefficient.

Why this form: The difficulty-specific $\alpha$ is the crucial design choice. For easy tasks, $\alpha$ is large—the length penalty strongly discourages long responses, pushing the model toward concise, efficient answers. For hard tasks, $\alpha$ is small—the length penalty is weak, allowing the model to produce long reasoning traces without significant penalty. This creates adaptive reasoning depth: the model learns to allocate its "thinking tokens" proportionally to problem difficulty, because the reward structure makes it optimal to do so. The $p(l)$ normalization by $\ell_{\min}$ and $\ell_{\max}$ within each batch ensures the penalty adapts to the natural length scale of the problem (some tasks inherently require longer answers than others).

Formatting $R_{\text{format}}$ : If explicit reasoning markers " thinking" appear in the output, reward -0.5. This enforces that Evo-CoT trains the model to reason without explicit chain-of-thought tags, producing what the paper calls "reflex-grade" reasoning—internalized, efficient reasoning that doesn't require special prompting.
Task-Specific Rewards $R_{\text{task-specific},k}$ : Optional additional signals for specific domains. For front-end generation tasks, the paper introduces the Visually Augmented Reward (VAR) system: generated code is rendered into a live interface via a headless browser, a multimodal model evaluates the screenshot based on aesthetic and usability criteria, and the resulting score is fed back as a reward signal. This creates a "Syntax–Function–Aesthetic triple-filter positive-feedback loop."

Linguistic-unit Policy Optimization (LPO). LPO is the specific policy gradient algorithm used for the RL updates in Evo-CoT. Its innovation is performing importance sampling and clipping at the sentence level rather than the token level or sequence level.

Given a group of $G$ candidate responses $\{y_i\}_{i=1}^G$ sampled from the old policy $\pi_{\theta_{\text{old}}}$ , the LPO objective is:

$\mathcal{J}_{\text{LPO}}(R, \theta) = \mathbb{E}_{\{y_i\}_{i=1}^G \sim \pi_{\theta_{\text{old}}}(\cdot | x_{\text{inst}})} \left[ \frac{1}{\sum_{i=1}^G |y_i|} \sum_{i=1}^G \sum_{k=1}^{N_{\text{sent}}(y_i)} |s_{i,k}| \cdot \min\left(r_{i,k}(\theta) \hat{A}_i, \text{clip}(r_{i,k}(\theta), 1 - \varepsilon, 1 + \varepsilon) \hat{A}_i\right) \right]$

where:

$N_{\text{sent}}(y_i)$ is the number of sentences in response $y_i$ , where sentences are segmented by common pause punctuation marks after detokenization
$s_{i,k}$ is the $k$ -th sentence in $y_i$
$|s_{i,k}|$ is the token length of that sentence
$|y_i|$ is the total token length of response $y_i$
$r_{i,k}(\theta)$ is the sentence-level importance ratio: $\exp\left(\frac{1}{|s_{i,k}|} \sum_{t \in \text{tokens}(s_{i,k})} \log \frac{\pi_\theta(y_{i,t} | x_{\text{inst}}, y_{i,<t})}{\pi_{\theta_{\text{old}}}(y_{i,t} | x_{\text{inst}}, y_{i,<t})}\right)$
$\hat{A}_i$ is the group-normalized advantage: $\hat{A}_i = \frac{R(x, y_i) - \text{mean}(\{R(x, y_i)\}_{i=1}^G)}{\text{std}(\{R(x, y_i)\}_{i=1}^G)}$
$\varepsilon = 0.03$ is the clipping parameter

What it computes: For each response $y_i$ , compute a per-sentence importance ratio $r_{i,k}(\theta)$ that measures how much more (or less) likely the current policy is to generate sentence $s_{i,k}$ compared to the old policy. Multiply this ratio by the advantage $\hat{A}_i$ (how much better this response is than the average in its group), clip to $[1 - \varepsilon, 1 + \varepsilon]$ for stability, weight by the sentence length $|s_{i,k}|$ , sum over all sentences and responses, and normalize by total token count $\sum_i |y_i|$ for scale invariance. The policy is updated to maximize this objective.

Why this form (sentence-level granularity):

Sentence-level importance sampling: By treating each sentence as an independent action unit with its own importance ratio—but applying the same advantage $\hat{A}_i$ to all sentences in the response—LPO updates the policy based on which sentences contribute to good responses, not just which tokens. A sentence that appears in a high-reward response gets its probability increased even if individual tokens within it are neutral. This aligns the optimization with the natural semantic boundaries of reasoning.
Token-level normalization: Dividing by $\sum_i |y_i|$ ensures that examples with longer responses don't dominate the objective—the contribution is proportional to the fraction of tokens, not the absolute count.
Clipping at the sentence level with $\varepsilon = 0.03$ : This is tighter than typical PPO clipping ( $\varepsilon = 0.2$ ), reflecting the finer granularity—sentence-level ratios are less noisy than token-level ratios (averaging over all tokens in a sentence reduces variance), so a tighter clip can be used without destabilizing training.

Empirical comparison (Figure 10): LPO delivers "smoother reward curves and markedly greater stability than GRPO, GSPO, and the GSPO (Token Mean) variant. It avoids plateaus and collapse, converges faster, and generalizes better." On AIME 2025, LPO-trained models achieve substantially higher accuracy than baselines.

Stage 3: Group Arena Reward (GAR) for Human Preference Alignment. The final post-training stage addresses subjective, open-ended tasks where correctness is not binary (writing quality, dialogue engagement, helpfulness). The problem with standard RLHF for these tasks is that absolute reward scores are noisy—different evaluators (humans or reward models) assign different scores to the same output, creating high-variance advantage estimates that destabilize policy optimization.

GAR addresses this by replacing absolute scoring with relative, tournament-style comparisons. For each prompt:

Multiple responses $\{O_1, O_2, \ldots, O_n\}$ are sampled from the current policy.
A generative reward model acts as a referee, performing pairwise comparisons in a round-robin tournament: for each pair $(O_i, O_j)$ , the referee judges which is better (e.g., $O_i > O_j$ or $O_i \gg O_j$ for a stronger preference).
The cumulative results of these pairwise contests form the final reward for each response.

What it computes: Instead of assigning an absolute score like "7.3/10" to each response independently, GAR produces a relative ranking based on direct comparisons. This is more reliable because comparing two responses is easier for both humans and reward models than assigning absolute scores—the cognitive task shifts from "how good is this on an absolute scale?" to "which of these two is better?", which has lower variance.

Why this form: Relative comparisons are the foundation of ELO-based evaluation systems (Chatbot Arena, LMSYS) and have been shown to produce more stable preference rankings. GAR brings this insight into the RL training loop itself, using the tournament results as the reward signal for policy updates. The round-robin structure ensures all pairs are compared, providing a dense relative reward signal rather than sparse absolute scores.

RubriX: Fine-Grained Multi-Dimensional Reward Guidelines. To complement GAR's comparative structure, RubriX provides the reward model with detailed evaluation rubrics across multiple dimensions: clarity, coherence, creativity, emotional resonance, instruction adherence, and domain-specific accuracy. These rubrics are instantiated for specific task types (writing, translation, long-form QA, emotional dialogue, multi-turn conversation) and guide the reward model's pairwise comparisons toward capturing nuanced aspects of user intent.

The paper reports that GAR + RubriX yields 2–10% higher consistency scores in open-ended evaluations compared to standard absolute scoring approaches.

Infrastructure: Full-Scale FP8 Training and Heterogeneous Pipeline Parallelism

The infrastructure challenge for Ling 2.0 is training a model where 96.5% of parameters are inactive for any given token, using 8-bit floating-point precision, across thousands of GPUs, with three distinct layer types (dense, MoE, MTP) that have different computation and memory profiles. The paper addresses this through coordinated innovations in numerical precision, pipeline scheduling, and distributed communication.

FP8 Training Framework. All Ling 2.0 models are trained entirely in FP8 precision (E4M3 format: 4 exponent bits, 3 mantissa bits). The key to making this work without accuracy degradation is fine-grained block-wise quantization:

Activations and gradients: Quantized in blocks of $[1, 128]$ elements—one row of 128 elements at a time—with per-block FP32 scaling factors. This means each small block gets its own scale, preventing a single outlier from distorting the quantization of an entire tensor.
Weights: Quantized in blocks of $[128, 128]$ elements—128×128 sub-matrices—with per-block scaling.

The quantization process is:

Take a BF16 tensor (the high-precision representation).
For each block, compute the maximum absolute value of elements in that block.
Compute a block-specific scaling factor that maps the block's dynamic range to the representable range of FP8.
Multiply and round, producing an FP8 tensor and a set of FP32 scaling factors.
During forward/backward passes, FP8 GEMM (general matrix multiply) operations compute matrix products in FP8, then dequantize the output back to BF16 using the stored scaling factors.

What this achieves: Computation in FP8 is approximately 2× faster than BF16 on Hopper GPUs (which have dedicated FP8 tensor cores), and memory consumption is halved. The paper reports a ~15% improvement in Model FLOPs Utilization (MFU) for Ling-1T compared to BF16 training, and a memory reduction of over 15%. The validation loss difference between FP8 and BF16 training after 900B tokens is within 0.25%, averaging around 0.1% (Figure 15).

FP8 Training Safeguard System. To ensure FP8 training remains stable at scale—a novel challenge since FP8 training at the trillion-parameter level was not previously demonstrated—the paper introduces a real-time monitoring system tracking two metrics for every operation across all layers:

FP8 Quantization Underflow: The proportion of matrix elements that become zero after quantization (i.e., values too small to represent in FP8). High underflow means information is being lost.
FP8 Quantization Distortion: Measured as the cosine similarity between the original (BF16) tensor and the reconstructed (quantized-then-dequantized) tensor. Values close to 1.0 indicate minimal information loss; lower values indicate distortion.

What this monitors: Figure 16 shows these metrics across layers during Ling-mini-2.0 training. Activations and gradients maintain healthy precision states (low underflow, high distortion/cosine similarity). However, gradient transpose computations for $\frac{\partial L}{\partial W}$ in tail layers show elevated quantization errors. Through joint analysis with high-precision recomputation, the paper concludes these errors have negligible impact because $\frac{\partial L}{\partial W}$ resides at leaf nodes in the backward propagation path—its quantization errors do not accumulate layer by layer (unlike $\frac{\partial L}{\partial x}$ errors, which propagate backward through the network). This is a subtle but critical observation: not all quantization errors are equally harmful, and the safeguard system's value is in distinguishing harmful from benign errors.

FP8 Efficiency Optimizations. Additional optimizations push FP8 training efficiency further:

FP8 On-Demand Transpose Weight: Transformer Engine implementations typically cache both the original and transposed weight matrices to accelerate backpropagation (which requires the transpose). This doubles memory usage for weights. Ling 2.0 introduces a high-performance on-demand transpose kernel that computes the transpose when needed rather than storing it, reducing weight memory footprint by exactly 50%. This is a time-for-space tradeoff that is favorable when memory, not compute, is the bottleneck.
FP8 Padding Routing Map: FP8 GEMM kernels require 16-element alignment for matrix dimensions, which conflicts with dynamic token allocation per expert in MoE—different experts receive different numbers of tokens that are rarely multiples of 16. The standard approach pads tokens to alignment, but this incurs non-negligible CPU overhead. The paper's optimization adjusts the routing map before expert assignment to ensure resulting tensor dimensions satisfy alignment constraints without explicit padding. Modifications are limited to zero-probability routing regions, preserving mathematical equivalence.
CPU overhead reduction: Additional optimizations include removing redundant assert checks and replacing FP8 padding/unpadding layers with the padding routing map approach.

Heterogeneous Fine-Grained Pipeline Parallelism. Pipeline parallelism (PP) splits model layers across GPUs, with each GPU processing a subset of layers. Standard interleaved 1F1B (one-forward-one-backward) scheduling alternates forward and backward passes of micro-batches to reduce idle time (pipeline bubbles). However, Ling 2.0's architecture introduces heterogeneity that breaks standard assumptions:

Embedding and loss layers are computationally light but sit at the pipeline boundaries.
Dense layers (first K) have different computation/memory profiles from MoE layers.
MTP layers are approximately 1.7× the computation cost of standard MoE layers and contain both transformer layers and a loss computation block.

The paper modifies the PP framework with four innovations:

Configurable Transformer Layer Allocation per VPP Stage: Virtual Pipeline Parallelism (VPP) allows multiple pipeline stages per GPU. The paper adds support for flexible allocation of transformer layers per VPP stage, including empty stages, enabling precise load balancing across the heterogeneous layer types.
Scheduling MTP as a Standalone Layer: The MTP layer is no longer grouped with other MoE layers or bound to the loss computation layer during scheduling, preventing it from creating a bottleneck when its computation cost differs significantly from adjacent layers.
Partial Recomputation for MTP: During the backward pass, only the transformer layer portion within MTP is recomputed (a standard memory-saving technique), while the logits computation part is not. This trades additional memory for improved computation speed specific to MTP's structure.
Fine-Grained Partition Strategy for MTP: The MoE layer and the loss computation layer within MTP are partitioned into two separate layers for scheduling (Figure 17). This enables the 1F1B scheduler to interleave these components with other layers, further reducing pipeline bubbles.

Result (Figure 18): These optimizations collectively achieve a 40% relative end-to-end improvement in training throughput for Ling-1T compared to the baseline interleaved 1F1B implementation. For models with balanced routing, increasing virtual pipeline stages from VPP2 to VPP4 yields an additional 5% gain, but this is conditional on routing balance—if a VPP stage contains only a single MoE layer, inter-stage blocking becomes more sensitive to imbalanced routing.

The Bitter Lesson of Computation-Communication Overlapping (Section 5.6). The paper attempted to apply advanced overlapping techniques—DualPipe (DeepSeek-V3) and interleaved 1F1B with All-to-All communication overlap—where expert computation in one micro-batch overlaps with communication for another micro-batch. Despite resolving performance issues like streaming multiprocessor contention and CPU synchronization bottlenecks, the end-to-end acceleration was limited. The paper identifies two root causes:

Overlapping strategies require large EP configurations: With fixed resources and global batch size, larger Expert Parallelism (EP) sizes assign more tokens per expert, improving matrix efficiency but also reducing experts per rank and making the EP group gated by the slowest rank. DeepEP performance is sensitive to routing balance.
Imbalanced routing in shallow MoE layers: Shallow layers tend to have less balanced routing, making PP ranks containing these layers more prone to out-of-memory errors. Addressing this required reconsidering the PP partitioning strategy, which incurred performance penalties.

This is an honest acknowledgment that a promising optimization technique did not deliver at scale, and the paper frames it as a direction for future work: "jointly optimize routing and related components to better realize [computation-communication overlap's] potential benefits."

Other Infrastructure Optimizations.

Intra-node DeepEP: DeepEP (DeepSeek-AI, 2024) was designed for cross-node EP communication. Even though Ling 2.0 does not use cross-node EP, deploying DeepEP for intra-node operations yields a 2% end-to-end speedup from reduced communication redundancy and 13% from operator fusion.
Fused Kernels: RoPE Fusion, Router Fusion, and upgraded GroupGemm reduce memory-bound bottlenecks and CPU overhead, yielding "notable improvement in end-to-end training performance."
Fast Expert Full-Recomputation: Full recomputation saves GPU memory at a 25% compute cost. The paper's insight is that the weighted-sum computation of expert probabilities in MoE layers can be moved to occur within the activation function, eliminating the dependency of linear_fc2 on earlier computation graph results. This allows discarding the recompute flow before linear_fc2, reducing recomputation latency by approximately half at no additional cost. Smaller models see up to 10% end-to-end gains; larger models see ~7% due to pipeline bottlenecks overlapping recomputation gains.
Long-Context Training with MTP: Addressed correctness issues related to loss and gradient misalignment when combining Tensor Parallelism and Context Parallelism with MTP. Resolved NaN issues in cross-sample attention masks during long-sequence training.
Framework Optimization: Distributed checkpoint save time reduced from 269s to 30s (from 2.43% to 0.82% of total training time) by caching metadata that depends only on model architecture. Job startup time reduced by ~30% through a warm-up batch that computes forward and backward passes without storing weights or updating gradients.
Checkpoint Save Interval Optimization: Derived an optimal interval formula $s = \sqrt{2880 \cdot C / F}$ where $C$ is storage overhead per save and $F$ is failover events per day. For Ling-1T, the optimal interval is 48 minutes, computed to minimize the combined impact of checkpointing overhead and failover rollback cost on effective training time.
Loss Spike Handling: Continued methodology from Ling-Team et al. (2025), monitoring training state from both gradient and loss perspectives to prevent spikes.

Cross-Platform Alignment (Section 5.4.3). The paper validates that training on heterogeneous GPU platforms produces consistent results. Loss curves on Hopper vs. non-Hopper GPUs show differences "consistently oscillating around zero" after the first 1,000 steps (where a data switch and not loading optimizer state caused divergence), with mean errors within 0.1%. This is framed as evidence that the 4C principle (Correct, Consistent, Complete, Co-Design) enables reliable cross-platform development.

Evaluation Pipeline (Section 5.5). The evaluation infrastructure is redesigned around OpenCompass to support trillion-parameter model evaluation:

Multi-Node Inference: Extended to support distributed evaluation across Ray Clusters, dynamically allocating multi-node multi-instance resources with SGLang as the inference backend.
Prompt Caching: Prefix-level caching reuses shared prompt embeddings across samples with identical prefixes (e.g., few-shot templates), improving throughput by more than 30%.
Batch Parallelization: Prompt preprocessing, inference, and postprocessing are parallelized and pipelined; small requests are automatically batched into larger groups for GPU saturation.

These optimizations collectively reduce total evaluation time per checkpoint by more than two-thirds compared to the original OpenCompass, enabling checkpoints to be validated "within hours rather than days."

Summary of Design Choices and Their Justifications

High-sparsity, fine-grained MoE (3.5% activation) over dense or coarser MoE: Predicted by Ling Scaling Law to maximize efficiency leverage (>7×), empirically validated at multiple scales. The shared expert provides a common backbone preventing fragmentation.
Ling Scaling Laws + Wind Tunnel over traditional ablation studies: Enables 100× extrapolation from cheap small-scale experiments with only 35% of the cost of a single model ablation, producing more reliable multi-scale trends.
32–46% reasoning data in pre-training over typical 5–15%: Builds inherent reasoning capability from the earliest training stages, providing a higher ceiling for subsequent RL and faster improvement trajectories (Figure 12).
Mid-training CoT pre-activation over reasoning-as-post-training-only: Shifts the model's parameter-level representations toward reasoning patterns before fine-tuning, yielding substantial gains on reasoning benchmarks (MATH: 61.96→82.52; AIME25: 2.08→43.75 for Ling-mini-2.0).
WSM scheduler over LR decay: Provides flexibility (no need to pre-commit to decay schedule), produces +1–2 point average gains across benchmarks, and the advantage persists through post-training.
DFT with dual system prompts over single-mode SFT: Establishes a reasoning-capable initialization while maintaining efficient instant-response behavior, enabling Evo-CoT to start from a policy that already knows how to reason.
Evo-CoT with LPO over GRPO/GSPO: Sentence-level policy updates align optimization with natural reasoning boundaries; dynamic length control with difficulty-specific coefficients creates adaptive reasoning depth; tighter clipping ( $\varepsilon = 0.03$ ) is stable due to reduced variance from sentence-level averaging.
GAR (tournament comparisons) over absolute scoring: Reduces reward noise in subjective tasks; pairwise comparisons are more reliable than absolute scores for both humans and reward models.
Full-scale FP8 training over BF16 or mixed precision: ~15% MFU improvement, >15% memory reduction, with loss difference within 0.25% after 900B tokens, enabled by fine-grained block-wise quantization and QKNorm outlier suppression.
Heterogeneous fine-grained PP over standard interleaved 1F1B: 40% throughput improvement by treating MTP and dense layers as distinct scheduling units with configurable allocation, partial recomputation, and fine-grained MTP partitioning.
First-K-Dense layers over all-MoE: Improves routing balance in shallow layers and overall computational efficiency, at the cost of slightly higher total parameters.
Aux-loss-free load balancing with zero-centered biases over auxiliary loss: Separates load balancing from language modeling objective; centering biases around zero prevents drift over long training runs.

4. Key Insights and Innovations

Innovation 1: Sparsity Amplifies Reasoning, Not Degrades It — A Refutation of the "Fragmentation Hypothesis"

The dominant unspoken assumption in the field before Ling 2.0 was that extreme sparsity and deep reasoning are in tension. The intuition is straightforward: reasoning requires chaining information across multiple inference steps, and if every token activates a different subset of experts, the model's internal representations become fragmented—different experts encode incompatible information, and the coherent "train of thought" necessary for multi-step deduction breaks down. This is not a strawman; it is the implicit reason why previous open reasoning models (Qwen, DeepSeek-V3) either used moderate sparsity or treated reasoning as primarily a post-training concern layered onto a general-purpose sparse backbone.

Ling 2.0 provides the first large-scale empirical refutation of this hypothesis. The model activates only ~3.5% of its parameters per token—256 routed experts with 8 active plus one shared expert—yet achieves 70.42% on AIME 2025 and 61.68% on LiveCodeBench while outperforming dense models with approximately 7× more active parameters. The key conceptual move is not "sparsity doesn't hurt reasoning" but rather "sparsity, when every component of the training pipeline is designed for reasoning, becomes an enabler of reasoning." The shared expert provides a common representational backbone that all tokens pass through, preventing the fragmentation that pure routed-sparsity would create. The massive number of routed experts (256) allows specialization: different reasoning strategies, mathematical domains, or coding patterns can be handled by different expert combinations without interference. The 7× efficiency leverage means the model can afford to have many more total parameters—and thus more specialized reasoning capacity—within the same compute budget as a smaller dense model.

This is a fundamental shift, not an incremental improvement, because it changes the design philosophy for reasoning-oriented LLMs from "build a dense model and add reasoning capabilities through post-training" to "build a sparse model where architecture, pre-training data, mid-training objectives, and post-training are all co-designed for reasoning." Evidence comes from the consistent scaling trajectory across all three model sizes (Tables 2–4, 6–8): the efficiency leverage holds at every scale, and reasoning performance improves with model size following the Ling Scaling Law, demonstrating that the relationship between sparsity and reasoning is not an artifact of a particular scale but a general property of the architecture when properly designed.

Innovation 2: Reasoning as a Pre-Training Objective, Not a Post-Training Add-On

The standard LLM pipeline treats knowledge acquisition (pre-training) and reasoning capability (post-training) as sequential, largely independent phases. Pre-training builds a general language model; supervised fine-tuning and RLHF add instruction-following and reasoning on top. This separation has practical consequences: the pre-training data mixture typically contains 5–15% reasoning data, the learning rate schedule is optimized for general language modeling, and the context window is chosen for training efficiency rather than reasoning structure. Reasoning is something the model learns to do after it has learned to speak.

Ling 2.0's most conceptually distinctive move is to treat reasoning as a first-class objective of pre-training itself. This manifests in three design decisions that collectively invert the standard pipeline:

First: The pre-training data mixture contains 32–46% reasoning data (math and code), not the typical single-digit percentages. This is not merely "more reasoning data" but a deliberate progressive shift: 32% in the first 10T tokens (when the model benefits from broad knowledge acquisition), increasing to 46% in the second 10T tokens (when the model is ready to specialize its representations toward reasoning). The paper's ablation on Ling-coder-1B and Ling-math-1B (Figures 4a–4c, Appendix B.1) demonstrates that this high reasoning proportion is not gratuitous—models trained from scratch on these data mixtures achieve competitive reasoning performance at small scales, validating that the data quality supports building reasoning capability from the ground up.

Second: The introduction of Chain-of-Thought data during mid-training—before any supervised fine-tuning—"pre-activates" reasoning ability. This is a genuinely novel concept: the model learns to associate problem statements with structured reasoning traces at the parameter level during pre-training, so that when RL is applied later, the model is refining existing reasoning behaviors rather than learning them from scratch. The evidence in Figure 12 is striking: Ling-mini-2.0 with CoT pre-training achieves higher ApexEval scores (41.9 vs. 34.5 without CoT) and faster improvement during ERL (AIME 2024: 26.5 vs. 19.7 at step 450). This is not a small tweak; it changes the starting point for post-training from a model that needs to be taught what reasoning looks like to a model that already knows and only needs optimization of how well it reasons.

Third: The WSM scheduler's preservation of reasoning gains through subsequent training stages. While the theoretical equivalence between checkpoint merging and learning rate decay is clever, the practical significance is that WSM produces models whose reasoning advantage persists through supervised fine-tuning, which the paper verifies (Section 3.2.3). This is in contrast to standard LR decay schedules, which optimize for final validation loss but may not preserve the specific parameter configurations that encode reasoning behaviors.

The shift from "reasoning emerges from post-training" to "reasoning is cultivated throughout the entire training pipeline" is fundamental rather than incremental because it implies different resource allocation: invest in high-quality reasoning data for pre-training, modify mid-training to include structured reasoning traces, and choose training schedules that preserve reasoning representations. Prior work (STaR, ReST) focused on the post-training side; Ling 2.0 shows that the pre-training side is equally important and possibly more foundational.

Innovation 3: Adaptive Reasoning Depth via Difficulty-Conditioned Reinforcement Learning

The standard approach to reasoning in LLMs is binary: either the model produces chain-of-thought (via prompting or fine-tuning) or it doesn't. Models like o1 or DeepSeek-R1 use explicit "thinking" tags and produce reasoning traces of roughly uniform verbosity regardless of problem difficulty. This is inefficient: a simple arithmetic problem that the model can solve in one step receives the same reasoning overhead as a complex competition math problem.

Ling 2.0's Evo-CoT paradigm introduces the concept of adaptive reasoning depth: the model learns to scale its internal computation to problem difficulty without explicit "thinking" tags. The mechanism—dynamic length control with difficulty-specific coefficients—is described in Section 3, but the conceptual innovation is the framing of reasoning depth as a resource to be allocated, not a binary mode. The model is trained as a "reflex-grade non-thinking model" that produces answers directly, but the RL process teaches it that longer responses are rewarded for hard problems and penalized for easy ones. This creates an implicit reasoning budget: the model internalizes that difficult problems are worth spending more tokens on, without needing explicit instructions to "think step by step."

Why this is distinctive: Prior approaches to efficient reasoning either (a) use explicit mode-switching (separate "thinking" and "non-thinking" models or system prompts), (b) apply uniform chain-of-thought to all problems, or (c) use length penalties as a coarse regularizer. Evo-CoT is the first approach that learns adaptive depth as an emergent property of the reward structure, not as an architectural or prompting choice. The difficulty-specific coefficient α makes the length penalty contingent on correctness and problem difficulty: easy problems have large α (strong brevity pressure), hard problems have small α (weak brevity pressure). The model discovers that the optimal policy is to spend more tokens on hard problems—not because it is told to, but because the reward function makes it advantageous.

The significance extends beyond benchmark performance. This is a step toward models that self-calibrate their computational effort—a property that is essential for deploying reasoning models in cost-sensitive applications where most queries are easy but a few require deep thought. The Pareto frontier analysis in Figure 13 (AIME-25 accuracy vs. average tokens) demonstrates that Ling-1T achieves higher accuracy at lower token cost than competing models, validating that the adaptive depth is not merely a training trick but produces genuinely more efficient inference.

This is a fundamental conceptual advance—a new way to think about reasoning in language models as a continuum rather than a binary—enabled by an incremental technical mechanism (length-penalty RL). The framing matters: it shifts the design question from "how do we make the model reason better?" to "how do we make the model reason the right amount for each problem?"

Innovation 4: The Ling Scaling Laws as a Methodology, Not Just a Result

Scaling laws for neural networks have existed since Kaplan et al. (2020) and were refined by Chinchilla (Hoffmann et al., 2022) for dense models and Clark et al. (2022) for early MoE architectures. The standard narrative is: scaling laws predict optimal model size and data quantity for a given compute budget, enabling efficient resource allocation.

Ling 2.0's scaling laws are qualitatively different in two ways that elevate them from "another set of power-law fits" to a methodology for model development:

First: The Ling Scaling Laws are actionable at the architectural level, not just at the resource-allocation level. Prior scaling laws answer "how big should my model be and how much data should it see?" The Ling Scaling Laws additionally answer "what activation ratio, expert granularity, learning rate, and batch size maximize efficiency at my target scale?" This is a three-dimensional optimization (hyperparameters, model-data allocation, architectural configuration) rather than the two-dimensional optimization (parameters vs. data) of prior work. The EL formula—incorporating activation ratio, expert granularity, and compute budget in a single equation—is a new kind of scaling law that directly determines architecture rather than just informing it.

Second, and more importantly: The Ling Wind Tunnel transforms scaling laws from a descriptive tool (understanding how models scale) into a prescriptive design methodology (cheaply testing features for trillion-parameter impact). This is the meta-innovation. The Wind Tunnel's five-model protocol, where architecture, data allocation, and hyperparameters are all determined by the scaling laws, creates a standardized "unit test" for features. A new attention mechanism, routing strategy, or loss function can be tested across all five tunnel models, and its scaling trend can be extrapolated 100× to predict trillion-parameter behavior—at only 35% of the cost of training a single model for ablation.

What makes this a fundamental contribution rather than incremental tooling is that it solves the exploration-exploitation problem in large-scale model development. Before the Wind Tunnel, design decisions for trillion-parameter models were made based on either (a) intuition, (b) expensive large-scale ablations, or (c) extrapolation from a single small-scale experiment that may not capture scaling trends. The Wind Tunnel provides a principled, cost-effective way to gather multi-scale evidence, making the design process more scientific and less speculative. The paper's validation that the Wind Tunnel predictions are accurate to within 0.01 in final training loss demonstrates that this is not merely a heuristic but a reliable forecasting tool.

This innovation is methodological rather than algorithmic—it changes how trillion-parameter models are designed, not what they contain. Its significance extends beyond Ling 2.0: any organization building large-scale LLMs could adopt a similar wind tunnel approach, using their own scaling laws to guide small-scale experiments and reliably extrapolating results. The "bitter lesson" in Section 5.6—where advanced computation-communication overlapping failed to deliver at scale—illustrates exactly why this methodology matters: features that work at moderate scale do not always scale favorably, and the Wind Tunnel is designed to catch such failures early.

Innovation 5: Sentence-Level Policy Optimization as a Principles-Based Solution to Granularity Mismatch

Reinforcement learning for language model fine-tuning has converged on two extremes: token-level optimization (PPO, GRPO) and sequence-level optimization (DPO, reward-model-based ranking). Token-level methods treat every token as an independent action, which provides fine granularity but ignores the semantic structure of language—a single incorrect token in an otherwise correct reasoning step receives the same penalty as a fundamentally flawed step. Sequence-level methods treat the entire response as a single action, which preserves semantic coherence but cannot distinguish between partially correct and completely incorrect reasoning—a response that makes a small arithmetic error at the end receives the same reward as a response that is entirely nonsensical.

LPO's conceptual contribution is recognizing that the sentence is the natural action unit for reasoning. A sentence in a reasoning trace corresponds to a discrete logical step—an inference, a calculation, a sub-conclusion. Updating the policy at the sentence level means the model is reinforced for producing coherent reasoning steps, not just correct tokens or entire correct answers. The mechanism—importance sampling with per-sentence importance ratios, normalized by sentence length, with group-based advantage estimation—is detailed in Section 3, but the idea is what matters: align the optimization granularity with the cognitive granularity of the task.

Why this is distinctive from prior work: GRPO and GSPO operate at the token or sequence level because they inherit the action space of language modeling. LPO intentionally breaks from this by segmenting responses at natural linguistic boundaries (pause punctuation after detokenization) and treating each segment as an atomic unit for policy updates. The tighter clipping threshold (ε = 0.03 vs. typical 0.2) is not an arbitrary choice but a consequence of the finer granularity—sentence-level importance ratios have lower variance than token-level ratios (because they average over all tokens in the sentence), so a tighter clip can be applied without destabilizing training. This is a principles-based design: the granularity choice drives the hyperparameter choice, not the other way around.

The empirical results (Figure 10) show that LPO avoids the plateaus and collapse that plague GRPO and GSPO, converges faster, and generalizes better on AIME 2025. But the significance is broader: LPO provides evidence that optimization granularity is a first-class design dimension in reasoning-oriented RL, not merely an implementation detail. The field has spent significant effort on reward design (correctness signals, process rewards, outcome rewards) and policy architectures (PPO variants, DPO, GRPO), but relatively little on what constitutes an action in the policy optimization. LPO demonstrates that getting the action granularity right can produce stability and performance gains that rival improvements in reward design.

This is an incremental technical contribution (a new clipping strategy) with a fundamental conceptual implication (action granularity matters for reasoning RL). It opens a design space—should we align RL action granularity with linguistic units (sentences), logical units (propositions, proof steps), or functional units (subroutines, API calls)?—that prior work had not systematically explored.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper uses a diverse evaluation suite spanning approximately 36 benchmarks for post-trained models and 33 benchmarks for base models, organized into categories: Math, Coding, General Reasoning, Knowledge, Multilingual, Agent, Instruction Following, and Alignment tasks. This is not a single curated testbed but a collection of established benchmarks—ranging from AIME 2025 and LiveCodeBench for reasoning to MMLU-Pro and GPQA-Diamond for knowledge to Arena Hard v2.0 for alignment—enabling assessment across multiple capability dimensions. Unless noted, evaluation uses 0-shot prompting with decontamination, reporting Exact Match (EM), Accuracy (Acc), or Pass@1 as appropriate per benchmark convention. For fill-in-the-blank benchmarks, the paper employs LLM-as-a-Judge to improve evaluation accuracy.
Base model(s). The paper evaluates three models drawn from the Ling 2.0 family, all built on the same "high-sparsity, fine-grained" MoE architecture with 256 routed experts, 8 activated per token plus one shared expert: Ling-mini-2.0 (16B total, 1.4B activated), Ling-flash-2.0 (103B total, 6.1B activated), and Ling-1T (1 trillion total, 51B activated). Each model is evaluated in both base (pre-trained only) and instruct (post-trained) configurations. The base model evaluation is designed to validate the 7× efficiency leverage claim—that these sparse models match the performance of dense models with approximately 7× more activated parameters. The instruct model evaluation assesses the full post-training pipeline's effectiveness at producing a capable reasoning-oriented chat model.
Metrics. Primary metrics vary by benchmark category but center on correctness: Accuracy/EM for math (e.g., MATH, CMATH, CollegeMath) and knowledge benchmarks (e.g., MMLU, C-Eval, GPQA-Diamond); Pass@1 for coding benchmarks (e.g., HumanEval, LiveCodeBench, CodeForces); Pass@k for certain reasoning-intensive benchmarks where a single greedy evaluation would be too noisy (e.g., AIME25 pass@1 for base models, GPQA-Diamond pass@1 with 16 repeats, ARC-AGI-1 and HLE with 4 repeats each). For the pre-training dynamics analysis (Section 3.3.1), metrics are evaluated by their stability over training and consistency with post-training performance (quantified via Kendall's rank correlation). For alignment benchmarks (Arena Hard v2.0, Writing Bench, Creative Writing v3), the metrics shift to style-controlled scores and win rates. For agent tasks (BFCL-V3), function call accuracy is reported.
Baselines. The paper compares against an extensive set of both open-source and proprietary models, selected to cover comparable scales and capability levels:
- For Ling-mini-2.0 (16B total, 1.4B activated): Qwen3-4B, Qwen3-8B (both non-thinking mode), Ernie-4.5-21B-A3B-PT, and gpt-oss-20B-Instruct-2507 (low thinking). For the base model comparison: Hunyuan-7B and Qwen3-8B base models.
- For Ling-flash-2.0 (103B total, 6.1B activated): Qwen3-32B (non-thinking), Hunyuan-A13B, Seed-OSS-36B, GPT-OSS-120B, and GPT-4.1 mini (low think). For the base model comparison: Qwen2.5-72B and Seed-OSS-36B base.
- For Ling-1T (1T total, 51B activated): DeepSeek-V3.1-Terminus, Kimi-K2 (lowthink), GPT-5-main, and Gemini 2.5 Pro (all non-thinking or low-thinking configurations). For the base model comparison: DeepSeek-V3.1 and Kimi-K2 base models.
Critically, the baselines for the reasoning-oriented RL stage (Evo-CoT) also include ablation comparisons against GRPO (Shao et al., 2024), GSPO (Zheng et al., 2025), and a GSPO Token Mean variant, evaluated via reward curves on AIME 2025 training/test sets (Figure 10) rather than through final benchmark scores. For the DFT checkpoint selection analysis (Figure 12), the baseline is the DFT model pretrained without CoT data.
Generation budget / compute accounting. The paper does not use a single unified compute metric across all experiments because the evaluations span pre-trained base models, post-trained instruct models, and RL training dynamics. For the base model comparisons (Tables 2–4), the accounting is done via activated parameters—the efficiency leverage claim is that Ling 2.0 matches dense models with approximately 7× more activated parameters. For the instruct model comparisons (Tables 6–8), compute is implicitly accounted for through model size comparisons. For the Evo-CoT RL analysis (Figure 10), the resource metric is training steps. For the Pareto frontier analysis (Figure 13), the x-axis is average tokens per response, measuring inference-time compute efficiency, plotted against AIME-25 accuracy on the y-axis. The paper does not report FLOPs-matched comparisons for instruct model evaluation—the efficiency claims are based on parameter-to-parameter comparisons rather than total FLOPs.
Cross-validation / statistical protocol. The base model evaluation framework (Section 3.3.1) employs two techniques to mitigate instability: checkpoint merging to smooth parameter-level noise, and Pass@k metrics to reduce evaluation noise from single-sample variance. All benchmark evaluations are conducted using "a unified internal evaluation framework to ensure a fair and consistent comparison" (Section 3.3.2). For the ApexEval-based checkpoint selection (Section 4.5), the protocol uses the highest score of pass@k (rather than greedy or average pass@k) to capture the model's potential upper bound. For GPQA-Diamond, 16 repeats per query are used; for ARC-AGI-1, ZebraLogic, and HLE, 4 repeats each, all reporting Pass@1 as the aggregated metric. The paper does not report confidence intervals or statistical significance tests for the benchmark comparisons—headline numbers are presented as point estimates.

Main Quantitative Results

Base Model Performance: Validating the 7× Efficiency Leverage

The headline finding from the base model evaluation (Tables 2–4) is that Ling 2.0 models achieve performance comparable to or exceeding dense models with approximately 7× more activated parameters, confirming the efficiency leverage predicted by the Ling Scaling Laws. This is evaluated across 33 benchmarks spanning math, coding, general reasoning, knowledge, and multilingual capabilities.

Ling-mini-2.0-base (1.4B activated) vs. dense baselines: Table 2 shows Ling-mini-2.0-base (with CoT data) achieves overall performance comparable to Qwen3-8B-base (~8B activated parameters). Specific comparisons on reasoning benchmarks: on MATH, Ling-mini-2.0-base with CoT achieves 82.52 vs. Qwen3-8B-base at 76.98—a reversal from the without-CoT configuration (61.96 vs. 76.98). On AIME25 pass@1, Ling-mini-2.0-base with CoT achieves 43.75 vs. Qwen3-8B-base at 13.54. On LiveCodeBench pass@1, Ling-mini-2.0-base with CoT achieves 34.47 vs. Qwen3-8B-base at 14.10. These gaps are largest on reasoning-intensive benchmarks, consistent with the claim that the reasoning-oriented pre-training pipeline provides disproportionate gains on structured reasoning tasks.

On knowledge benchmarks, the comparison is less favorable: Ling-mini-2.0-base with CoT achieves 74.26 on MMLU vs. Qwen3-8B-base at 78.62; 47.70 on MMLU-Pro vs. 50.83. This suggests the efficiency leverage is stronger for reasoning than for factual knowledge, though the paper does not explicitly analyze this differential.

Ling-flash-2.0-base (6.1B activated) vs. dense baselines: Table 3 compares against Qwen2.5-72B-base and Seed-OSS-36B-base. Ling-flash-2.0-base with CoT achieves 79.54 on MATH vs. Qwen2.5-72B-base at 76.46 and Seed-OSS-36B-base at 88.64—the latter being a notable exception where Ling-flash underperforms Seed-OSS on MATH despite the efficiency leverage claim. On coding benchmarks, Ling-flash with CoT achieves 89.63 on HumanEval vs. Qwen2.5-72B at 82.32; 83.54 on HumanEval-Plus vs. 73.78; 77.38 on CruxEval vs. 63.10. On CodeForces, Ling-flash with CoT achieves 47.54 vs. Qwen2.5-72B at 17.81—a gap of nearly 30 points. On BigCodeBench, Ling-flash with CoT achieves 52.37 vs. Qwen2.5-72B at 41.18.

Ling-1T-base (51B activated) vs. trillion-scale baselines: Table 4 compares against DeepSeek-V3.1-base (671B total) and Kimi-K2-base (1T total). Ling-1T-base with CoT achieves 82.78 on MATH vs. Kimi-K2 at 76.40 and DeepSeek-V3.1 at 35.64; 62.87 on MinervaMath vs. Kimi-K2 at 55.88; 36.61 on ARC-c vs. Kimi-K2 at 95.59—where Kimi-K2 substantially outperforms. On coding benchmarks, Ling-1T with CoT achieves 89.63 on HumanEval (matching Kimi-K2 at 89.63); 80.88 on CruxEval vs. Kimi-K2 at 78.25; 55.78 on CodeForces vs. Kimi-K2 at 24.79. On knowledge benchmarks, MMLU scores are comparable: Ling-1T with CoT at 86.03 vs. Kimi-K2 at 88.32 vs. DeepSeek-V3.1 at 88.44.

The CoT pre-activation effect: Across all three scales, the comparison between "w/o CoT Data" and "w/ CoT Data" columns in Tables 2–4 provides strong evidence for the mid-training reasoning pre-activation strategy. For Ling-mini-2.0-base, the CoT introduction moves MATH from 61.96 → 82.52, AIME25 from 2.08 → 43.75, and LiveCodeBench from 13.71 → 34.47. For Ling-1T-base, MATH moves from 67.42 → 82.78, CollegeMath from 75.02 → 75.48, and MinervaMath from 50.00 → 62.87. These gains are concentrated on reasoning-intensive benchmarks while non-reasoning benchmarks remain stable—consistent with the claim that CoT pre-training specifically activates reasoning capability rather than providing general performance improvements.

Instruct Model Performance: Post-Training Effectiveness at Each Scale

The instruct model evaluation (Tables 6–8) assesses how the full post-training pipeline transforms base models into capable chat/assistant models, with comparisons against leading open-source and proprietary models at comparable scales.

Ling-mini-2.0-Instruct (Table 6) vs. Qwen3-8B (non-thinking): Ling-mini-2.0 outperforms Qwen3-8B on the majority of reasoning benchmarks despite having ~5.7× fewer activated parameters. Specific comparisons: LiveCodeBench 41.69 vs. 26.10; CodeForces 1410 vs. 624 (ratings); CNMO 2024 72.66 vs. 34.38; AIME24 65.62 vs. 27.97; AIME25 46.72 vs. 24.01; Omni-MATH 60.30 vs. 41.71; KOR-Bench 62.00 vs. 54.40. The gap on reasoning benchmarks is consistent with the 7× efficiency leverage narrative. However, on instruction following (IFEval Prompt Strict), Ling-mini-2.0 scores 77.74 vs. Qwen3-8B at 83.92, and on function calling (BFCL-V3), Ling-mini-2.0 scores 53.71 vs. Qwen3-8B at 59.50—indicating that the reasoning focus may come at a slight cost to some general instruction-following capabilities.

Ling-flash-2.0-Instruct (Table 7) vs. GPT-OSS-120B and Qwen3-32B: Ling-flash-2.0 with 6.1B activated parameters substantially outperforms Qwen3-32B (32B activated) across reasoning tasks: AIME25 55.83 vs. 22.5; Omni-MATH 66.64 vs. 43.81; KOR-Bench 68.80 vs. 56.96; ZebraLogic 86.80 vs. 33.80; ARC-AGI-1 24.56 vs. 3.31. The gap on ARC-AGI-1 is particularly notable—nearly 8× higher—suggesting the reasoning-oriented training provides benefits for abstract visual reasoning even though it is not specifically targeted. Against GPT-OSS-120B (a much larger model at 120B parameters), Ling-flash-2.0 is competitive: AIME25 55.83 vs. 50.83; CodeForces 1600 vs. 1519; Creative Writing v3 85.17 vs. 79.09. On alignment benchmarks (Arena Hard v2.0 Win-Rate), Ling-flash-2.0 scores 61.33 vs. GPT-OSS-120B at 81.19—a substantial gap suggesting the GPT-OSS model has stronger alignment despite weaker reasoning.

Ling-1T-Instruct (Table 8) vs. GPT-5-main and Gemini 2.5 Pro: This is the flagship comparison. Ling-1T (51B activated) matches or exceeds GPT-5-main (a model of unknown but certainly much larger activated parameter count) on most reasoning benchmarks: LiveCodeBench 61.68 vs. 48.57; CodeForces 1901 vs. 1120 (ratings); CNMO 2024 79.25 vs. 63.11; AIME24 80.21 vs. 67.60; AIME25 70.42 vs. 59.43; Omni-MATH 74.46 vs. 61.09; BBEH 47.34 vs. 39.75; ARC-AGI-1 43.81 vs. 14.06; ZebraLogic 90.80 vs. 57.30. Against Gemini 2.5 Pro, Ling-1T is competitive on multiple fronts: AIME25 70.42 vs. 70.10 (essentially tied); Omni-MATH 74.46 vs. 72.02; LiveCodeBench 61.68 vs. 45.43; BBEH 47.34 vs. 29.08. However, on very difficult benchmarks: HLE (Humanity's Last Exam) shows Ling-1T at 7.60 vs. GPT-5-main at 7.33 vs. Gemini 2.5 Pro at 12.07; MedXpertQA shows Ling-1T at 22.33 vs. Gemini 2.5 Pro at 44.82—indicating that on certain specialized knowledge and extremely challenging reasoning tasks, the trillion-scale dense/proprietary models still hold advantages.

On alignment benchmarks, Ling-1T achieves Arena Hard v2.0 style-controlled score of 76.26 vs. GPT-5-main at 68.37 and Gemini 2.5 Pro at 65.37; Writing Bench 89.40 vs. GPT-5-main at 77.07; Creative Writing v3 89.24 vs. GPT-5-main at 80.93—surprising results where Ling-1T substantially outperforms on subjective quality metrics despite its reasoning focus. The paper attributes this to the GAR + RubriX alignment pipeline.

Adaptive Reasoning Efficiency: The Pareto Frontier Analysis

Figure 13 presents what the paper positions as the defining efficiency result: a Pareto frontier plot of AIME-25 accuracy versus average inference tokens, comparing Ling-1T against GPT-5-main, Gemini 2.5 Pro, DeepSeek-V3.1, and Kimi-K2. Ling-1T appears in the upper-left region—higher accuracy at lower token cost—which the paper interprets as demonstrating "efficient thinking and precise reasoning." The claim is that Evo-CoT training produces a model that achieves high accuracy while naturally producing shorter responses than models that use explicit chain-of-thought tokens for every query, because the dynamic length control teaches the model to allocate reasoning tokens proportionally to difficulty.

The paper does not provide the exact numerical coordinates of all data points on this Pareto frontier, making quantitative comparison difficult from the text alone. However, the visual positioning in Figure 13, combined with the benchmark numbers in Table 8, supports the qualitative claim that Ling-1T achieves a favorable accuracy-efficiency tradeoff relative to the compared models on AIME-25 specifically.

Evo-CoT Training Dynamics: LPO Stability and Convergence

Figure 10 provides the empirical comparison between LPO and alternative policy optimization algorithms during RL training for "the latest Ling 2.0 model" (presumably Ling-1T, though the paper is not explicit about which model generated these curves). The left panel shows reward progression on training data: LPO (blue curve) exhibits "smoother growth and greater stability compared to GRPO, GSPO, and the GSPO (Token Mean) baseline, with no severe plateaus or collapses." The right panel shows reward curves on the AIME 2025 test set, where LPO demonstrates "faster convergence and improved generalization." The paper claims this translates to "substantially higher accuracy" on AIME 2025 for LPO-trained models, though specific accuracy numbers for each algorithm variant are not provided in the text of Section 4.2.2—the figure is the primary evidence.

DFT with vs. without CoT Pre-Training: Impact on RL Potential

Figure 12 (Section 4.5) provides the key ablation demonstrating that CoT data during mid-training improves the foundation for subsequent RL. The left panel shows ApexEval scores for DFT models pretrained with and without CoT data across 9 benchmarks (gpqa, aider, LCBench, aime2024, aime2025, minerva math, bbh, bbeh). The CoT-pretrained model scores higher on all benchmarks, with specific bars showing: AIME 2024 41.9 vs. 34.5; AIME 2025 16.0 vs. 14.6; GPQA 33.8 vs. 34.5 (approximately tied, with the paper's Figure 12 showing COT slightly higher but the text's claim of consistent improvement may be overstated on this benchmark). The right panel shows that after applying Evolutionary Reasoning RL (ERL step 450), the CoT-pretrained model achieves AIME 2024 score of 26.5 vs. 19.7 without CoT, and AIME 2025 score of 20.2 vs. 14.2 without CoT—increases of +6.8% and +6.0% respectively.

Ablation Studies and Robustness Checks

The paper's ablation studies are distributed across the architecture (Section 2), pre-training (Section 3.3.2), post-training (Sections 4.5, 4.6), and infrastructure (Section 5) sections rather than consolidated in a single ablation section. I include both explicit ablations and comparative analyses that serve an ablative function.

CoT data in mid-training (base model evaluation, Tables 2–4): All three base model tables include columns for "w/o CoT Data" and "w/ CoT Data," providing a direct ablation of the mid-training reasoning pre-activation strategy. The pattern is consistent across scales: CoT provides large gains on reasoning benchmarks (MATH, AIME25, LiveCodeBench) with minimal impact or slight degradation on some knowledge benchmarks. For Ling-1T, the w/o CoT vs. w/ CoT comparison on MATH shows 67.42 → 82.78 (+15.36 points). On AIME25, the paper only reports the CoT version (43.75 for mini, not separately for 1T w/o CoT in Table 4's AIME25 row), but the Ling-mini-2.0 comparison is 2.08 → 43.75—a 21× improvement that is so dramatic it warrants scrutiny (see Critical Assessment).

DFT with vs. without CoT pre-training (Figure 12): The ApexEval-based checkpoint selection analysis in Section 4.5 functions as an ablation of the CoT mid-training strategy specifically for its impact on RL potential. The finding that CoT-pretrained models achieve higher ApexEval scores (~41.9 vs. 34.5 on AIME 2024) and faster ERL improvement trajectories validates the claim that pre-training reasoning exposure provides a better foundation for RL-based reasoning optimization.

LPO vs. GRPO vs. GSPO (Figure 10): The comparison of policy optimization algorithms during Evo-CoT training serves as the key ablation for the post-training RL methodology. LPO's smoother reward curves and faster convergence on AIME 2025 test set validate the sentence-level optimization approach. The specific comparison to the GSPO Token Mean variant demonstrates that the benefit comes from sentence-level granularity specifically, not just from the group-based advantage estimation that GSPO also provides.

WSM vs. WSD scheduler (Figure 7): The comprehensive comparison in Section 3.2.3, Figure 7 shows WSM (via checkpoint merging) outperforming standard WSD (via LR decay) across five benchmark categories (Overall Average, General Knowledge/Reasoning, Language Understanding, Professional Knowledge, Math, Code) over the course of 500B tokens. The WSM Merge 12 curve consistently lies above the WSD curve, with the "WSM (Before Merge)" curve showing lower performance—demonstrating that the merge operation itself, not just the constant-LR training phase, is responsible for the gain. The advantage is approximately +1 to +2 points across categories. The paper also reports that this advantage persists through 5 epochs of supervised fine-tuning.

FP8 vs. BF16 training precision (Figure 15): The loss difference tracking over 900B tokens on Ling-1T shows the FP8 training loss diverges from BF16 by an average of ~0.1% (within 0.25% maximum), with no systematic drift. This validates that fine-grained block-wise quantization with QKNorm outlier suppression can maintain near-lossless training at the trillion-parameter scale.

QKNorm effectiveness analysis (Figure 16): The per-layer tracking of FP8 quantization underflow and distortion during Ling-mini-2.0 training shows that activations and gradients maintain healthy precision states (underflow near zero, distortion cosine similarity near 1.0), while gradient transpose computations in tail layers show elevated errors (distortion dropping to ~0.90–0.92 in the deepest layers). The paper's analysis attributes this to the leaf-node position of ∂L/∂W in the backward computation graph, meaning these errors do not propagate. This is a non-obvious finding: not all quantization errors are equally damaging, and infrastructure monitoring should distinguish between propagating errors (∂L/∂x) and non-propagating errors (∂L/∂W).

RoPE partial dimension ablation (motivated in Section 2.1, not quantitatively ablated): The paper states that applying RoPE only to the first 64 dimensions of attention heads "bolsters the model's length extrapolation capabilities" and validates this via the NIAH test at 128K context (Figure 6), but does not provide a direct ablation comparing full vs. partial RoPE on length extrapolation benchmarks. The claim rests on the successful long-context evaluation rather than a controlled comparison.

Negative result: Computation-communication overlapping (Section 5.6, "A Bitter Lesson"): The paper honestly reports that advanced overlapping techniques (DualPipe, interleaved 1F1B with A2A overlap) yielded limited end-to-end acceleration despite resolving implementation-level issues. The root causes are routing imbalance in shallow MoE layers and gating by the slowest rank in large EP configurations. This is a genuine negative result that is methodologically important—it demonstrates that theoretically sound optimization techniques can fail at scale due to emergent load-balancing effects.

Negative result: ReST^EM revision model (referenced in Ling-Team, 2025, not in this paper): The paper does not contain revision model ablation as the Ling 2.0 models are "reflex-grade non-thinking" models rather than iterative reasoning models. The Ring series (mentioned in Section 1) handles deep thinking; this paper focuses on the efficiency-oriented non-thinking lineage.

Critical Assessment

Does the 7× Efficiency Leverage Claim Hold?

The efficiency leverage claim—that Ling 2.0 matches dense models with approximately 7× more activated parameters—is the paper's central architectural claim, predicted by the Ling Scaling Laws and validated through the base model comparisons in Tables 2–4. The claim is substantially but not uniformly supported.

The most direct comparison is Ling-mini-2.0-base (1.4B activated) vs. Qwen3-8B-base (~8B activated). On reasoning benchmarks, Ling-mini-2.0 with CoT matches or exceeds Qwen3-8B: MATH (82.52 vs. 76.98), AIME25 (43.75 vs. 13.54), LiveCodeBench (34.47 vs. 14.10). However, on knowledge benchmarks, Qwen3-8B maintains an advantage: MMLU (78.62 vs. 74.26), MMLU-Pro (50.83 vs. 47.70). The efficiency leverage claim thus holds more strongly for reasoning tasks than for knowledge-intensive tasks. This is consistent with the reasoning-oriented training pipeline—the 32–46% reasoning data proportion builds reasoning capability but may underinvest in factual knowledge compared to models with more general-purpose pre-training mixtures. The paper does not explicitly acknowledge this differential, which matters for users who need balanced capability profiles.

For Ling-1T-base (51B activated), finding an appropriate ~350B-parameter dense comparison model is challenging. DeepSeek-V3.1-base (671B total, unknown activated parameters) and Kimi-K2-base (1T total, unknown activated parameters) are the closest comparisons, but both are themselves MoE models. The paper attempts to validate the 7× claim through the aggregate comparisons in Table 4, but without a true ~350B dense baseline, the evidence is indirect. The claim that Ling-1T achieves "performance comparable or superior to other state-of-the-art open-source models of similar scale" is supported by Table 4, but the specific 7× multiplier is validated at smaller scales (mini, flash) and extrapolated to 1T via the Ling Scaling Laws.

Missing experiment: A direct dense baseline at each scale—e.g., a dense ~8B model trained on the same data mixture as Ling-mini-2.0, or a dense ~350B model trained on Ling-1T's data—would provide the cleanest validation of the efficiency leverage. Such experiments are computationally prohibitive at the trillion scale, which is precisely the motivation for the Ling Scaling Laws, but the absence means the efficiency claim rests partially on the validity of the scaling law extrapolation.

Does the Reasoning Pre-Activation via CoT Data Claim Hold?

The claim that mid-training CoT data "pre-activates" reasoning ability and provides a "higher ceiling for reasoning performance" is supported by two pieces of evidence: the base model before/after CoT comparisons (Tables 2–4) and the ApexEval/ERL analysis (Figure 12). The evidence is strong but with one eyebrow-raising result.

For Ling-mini-2.0-base, the AIME25 pass@1 improvement from CoT is from 2.08 to 43.75—a 21× increase. This is an extraordinarily large jump for a change in training data composition, not a change in model architecture or scale. Several factors could contribute: (a) AIME25 is extremely difficult for small models, so small absolute changes translate to large multipliers; (b) the "w/o CoT Data" model may have been evaluated at a disadvantage relative to its true AIME capability due to lack of chain-of-thought formatting familiarity; (c) the 2.08 baseline may be anomalously low due to the evaluation protocol used for base models (0-shot, CoT prompting, which the w/o CoT model has not been trained to produce). The paper would benefit from providing more detail on the evaluation protocol for the w/o CoT baseline—specifically, whether the model without CoT training was prompted to produce chain-of-thought at evaluation time (which would disadvantage it since it never saw such outputs during training) or evaluated in a direct-answer format (which might be a more fair comparison for a model without CoT exposure).

For the ApexEval analysis (Figure 12), the CoT-pretrained model's advantage on AIME 2024 (41.9 vs. 34.5) and AIME 2025 (16.0 vs. 14.6) is clear but more modest. The ~6% improvement after ERL step 450 (26.5 vs. 19.7 on AIME 2024) is a meaningful but not transformative gain. The "faster improvement trajectory" claim is supported by the right panel of Figure 12, but the paper only shows two snapshots (DFT and ERL step 450), not the full training curve, making it difficult to assess whether the CoT model converges faster or just starts from a higher baseline.

Missing analysis: The paper does not report the training curves for the ERL process with vs. without CoT pre-training—only the endpoint values at step 450. Showing the full learning curves would allow readers to distinguish between "CoT provides a better starting point" (higher initial performance with similar slope) and "CoT enables faster learning" (higher slope in the RL phase). The paper's framing suggests the latter, but the presented data only directly supports the former.

Does Evo-CoT Produce Genuinely Adaptive Reasoning Depth?

The claim that Evo-CoT enables the model to "scale its reasoning depth according to problem complexity" is supported by the dynamic length control mechanism design (Section 4.2.1) and the Pareto frontier plot (Figure 13). The evidence is suggestive but incomplete.

The Pareto frontier shows Ling-1T achieving high AIME-25 accuracy with lower average token usage than compared models, which is consistent with adaptive reasoning depth. However, this is an aggregate metric—it doesn't demonstrate that the model actually produces longer responses for harder problems and shorter responses for easier ones within the AIME-25 set or across tasks of varying difficulty. A convincing demonstration would show: (a) on easy MATH problems, Ling-1T produces short, direct solutions; (b) on AIME problems, it produces longer reasoning traces; (c) this variation is not merely a property of the problems themselves (harder problems naturally have longer solutions) but is amplified by the model's learned behavior.

Missing ablation: An experiment where the same model is tested with different α coefficients at inference time (e.g., α=0 for "think as much as you want" vs. α=large for "be concise") would demonstrate that the length control is genuinely controllable rather than just an emergent correlation. Similarly, comparing Evo-CoT against a baseline trained with uniform length penalty (α constant across all difficulty levels) would isolate the difficulty-conditioned aspect of the reward.

The paper also does not provide per-difficulty-level analysis of response lengths for Ling-1T on the MATH benchmark (which has explicit difficulty levels 1–5). Such an analysis would directly validate that the model allocates tokens proportionally to difficulty, which is the operational definition of adaptive reasoning depth.

Does LPO Genuinely Outperform GRPO/GSPO for Reasoning?

Figure 10 shows LPO's reward curves are smoother and show faster convergence on AIME 2025 test set compared to GRPO, GSPO, and GSPO Token Mean. The paper claims this translates to "substantially higher accuracy" but does not report the final AIME 2025 benchmark scores for each algorithm variant in Section 4.2.2. The evidence in Figure 10 is qualitative (smoothness and convergence speed) rather than quantitative (final performance numbers).

The reward curve advantage on training data (left panel) could partially reflect the fact that LPO's sentence-level clipping better fits the reward function used during training, without necessarily translating to better generalization. The test set curves (right panel) partially address this concern by showing evaluation on held-out AIME 2025 problems, but the paper would be stronger with a table of final benchmark scores (AIME 2025, MATH, LiveCodeBench) comparing LPO against each baseline at equivalent training steps.

The claim of "around 10% improvements on complex reasoning benchmarks compared to token-level and sequence-level baselines" (Section 1) is not directly attributed to a specific table or figure in the LPO section, making it difficult to verify the magnitude of the improvement. The 10% figure may refer to relative improvement (e.g., 40% → 44%) or absolute improvement (e.g., +4 percentage points on a 40% baseline), and the specific benchmarks are not named in the LPO section text.

Are the Baselines Fairly Chosen and Evaluated?

The baselines in Tables 2–4 and 6–8 are drawn from major open-source and proprietary models, creating a credible comparison set. However, several fairness concerns arise:

Configuration matching: The paper compares Ling models against baselines in specific "non-thinking" or "low-thinking" configurations, but the AIME 2025 scores for some baselines appear surprisingly low. For example, Qwen3-8B (non-thinking) scores 24.01 on AIME25 (Table 6), while Qwen3-8B in its default configuration might perform differently. The paper does not detail how baseline model configurations were selected—were they chosen to match Ling's "reflex-grade non-thinking" behavior, or were they the standard instruct configurations?

Benchmark coverage gaps: Not all baselines are evaluated on all benchmarks. BFCL-V3 is marked "–" for Ernie-4.5-21B-A3B-PT, some benchmarks in Tables 2–4 are not reported for all models. While this is common in large benchmark tables, it limits the completeness of the comparison.

The Ling model evaluation advantage: The Ling 2.0 models are evaluated using "a unified internal evaluation framework," which may give them a small advantage over baselines whose scores are taken from public leaderboards or run with different evaluation scripts. Prompt formatting, few-shot example selection, and answer parsing can materially affect benchmark scores, and the paper does not detail whether baseline scores were reproduced using the same evaluation framework or sourced from model providers.

Missing strong coding baselines: For the coding benchmark comparisons (especially Table 8), comparing against Qwen2.5-Coder models or DeepSeek-Coder models would provide stronger validation of Ling's coding capabilities than general-purpose models like GPT-5-main or Gemini 2.5 Pro, which are not specialized for code.

Statistical Rigor and Reproducibility Concerns

The paper reports point estimates for all benchmark scores without confidence intervals, standard deviations, or statistical significance tests. For a paper making efficiency claims that involve tradeoffs (Ling is better on reasoning, sometimes worse on knowledge), the absence of uncertainty quantification makes it difficult to assess whether observed differences are statistically meaningful or within evaluation noise.

The test set sizes for specific benchmarks are not explicitly reported, but many are well-known: AIME 2025 has 15 problems (making percentages coarse-grained in 6.67% increments), HLE has a modest number of problems, GPQA-Diamond is a curated set. With small test sets, a difference of a few problems can change rankings, and the paper does not address this variance. The multiple-repeat protocol for some benchmarks (GPQA-Diamond: 16 repeats, ARC-AGI-1: 4 repeats) partially addresses this but is not consistently applied.

The Ling Wind Tunnel methodology is validated by the claim that "training uncertainty can be reduced to within an error of 0.01 in final training loss" (Section 2.3.3), but this validation is for the small-scale experiments, not the full trillion-parameter training run. The extrapolation from the Wind Tunnel to Ling-1T is the critical step that enables the entire design methodology, and while the paper's results are consistent with the scaling laws, the validation is ultimately correlational rather than causal—the laws predicted good performance, and good performance was observed, but this doesn't rule out that other configurations would have performed equally well or better.

The Pareto Frontier Claim: Efficiency and Accuracy

Figure 13 presents Ling-1T as establishing a new Pareto frontier. Without exact coordinates, the reader must rely on the visual presentation. The paper should ideally provide a table with (accuracy, average tokens, model name) for each data point on the frontier, enabling independent verification of the efficiency claim. The qualitative statement that Ling-1T achieves "efficient thinking and precise reasoning" is supported by the combination of Table 8 (high accuracy) and Figure 13 (lower token cost), but the evidence would be stronger with per-benchmark efficiency metrics (e.g., accuracy per 1,000 tokens) across the full benchmark suite, not just AIME-25.

6. Limitations and Trade-offs

The Difficulty Estimation Problem Is Pushed to the Architecture, Not Solved

The assumption or constraint. The Ling 2.0 pipeline produces "reflex-grade non-thinking" models that generate answers directly without explicit chain-of-thought tags. The Evo-CoT training paradigm teaches the model to allocate internal reasoning depth based on problem difficulty through the dynamic length control reward:

$\hat{R}_{\text{length}} = \begin{cases} p(l), & \text{if } r_{\text{acc}} = 1 \\ \min\left(p(l), 0\right), & \text{if } r_{\text{acc}} = 0 \end{cases}$

with the difficulty-specific coefficient $\alpha$ determining how strongly brevity is rewarded. For easy tasks, $\alpha$ is large (strong brevity pressure); for hard tasks, $\alpha$ is small (weak brevity pressure). This creates adaptive reasoning behavior as an emergent property of the reward structure.

The consequence. The model learns to modulate reasoning depth based on problem difficulty during training, but at inference time there is no explicit difficulty estimation mechanism—the model must internally decide how much to "think" without any external difficulty signal. This creates an unresolved calibration problem: if the model misestimates a hard problem as easy, it will produce an insufficiently reasoned answer and likely be wrong; if it overestimates an easy problem as hard, it will waste inference compute. The paper provides no mechanism for users to control this tradeoff at deployment time—you cannot, for example, request "spend up to 2× more tokens on this query because it is high-stakes."

The paper implicitly assumes the model's internal difficulty estimation is well-calibrated, but this is a property of the RL training process that is not directly validated. A misaligned difficulty estimator—one that systematically underestimates certain problem types—would produce confident-sounding but incorrect answers on those problems, which is a more dangerous failure mode than a model that is uniformly verbose or uniformly terse.

What evidence exists in the paper. The Pareto frontier analysis (Figure 13) shows Ling-1T achieving high AIME-25 accuracy with lower average token usage than comparison models, which is consistent with good aggregate calibration. However, this is an average metric—it does not show whether token allocation correlates with problem difficulty within a benchmark, or whether the model's confidence is well-calibrated to its actual correctness. The paper reports no per-difficulty breakdown of response lengths, no calibration curves (reliability diagrams), and no analysis of failure modes where the model produces short, incorrect answers to genuinely hard problems. The ApexEval analysis (Section 4.5) evaluates the model's potential upper bound but does not measure whether the model can reliably identify when to deploy that potential.

Mitigation status. Not addressed. The paper frames the Evo-CoT design as a feature—adaptive reasoning depth—but does not discuss the calibration risk inherent in making reasoning depth an implicit, learned behavior rather than an explicit, controllable parameter. There is no suggested mechanism for users to override or influence the model's internal difficulty assessment at inference time. A practitioner deploying Ling-1T in a high-stakes setting (medical reasoning, financial analysis) would need to know whether the model can be trusted to "think harder" when it matters, and the paper provides no evidence on this question.

Hard Problems Remain Essentially Unsolved — Sparsity Does Not Create Capability

The assumption or constraint. The Ling 2.0 design philosophy—"every activation boosts reasoning capability"—assumes that the base model possesses the fundamental knowledge and reasoning primitives necessary to solve problems, and that the architecture, data, and training pipeline amplify this latent capability. The paper does not claim that sparsity creates new capabilities, but the framing of "scaling general reasoning capability to the trillion-parameter level" suggests that scale alone, with proper efficiency, can push the frontier.

The consequence. For problems genuinely outside the model's knowledge boundary, no amount of architectural efficiency or training optimization helps. This is visible most starkly on Humanity's Last Exam (HLE), where Ling-1T scores 7.60 compared to Gemini 2.5 Pro at 12.07 (Table 8)—both scores are very low in absolute terms. On MedXpertQA, Ling-1T scores 22.33 vs. Gemini 2.5 Pro at 44.82—a gap of over 22 points. On the hardest knowledge benchmarks, Ling-1T's efficiency advantage does not translate to capability parity with the largest proprietary models.

More subtly, the paper's own base model evaluation (Table 4) shows that even with CoT pre-activation, Ling-1T-base scores only 36.61 on ARC-c compared to Kimi-K2-base at 95.59—a massive 59-point gap on a benchmark measuring scientific reasoning with commonsense knowledge. This is not a small difference; it suggests that on certain reasoning-adjacent tasks that require specific knowledge rather than pure logical deduction, Ling's reasoning-oriented training may come at the expense of knowledge coverage.

What evidence exists in the paper. The HLE and MedXpertQA results in Table 8 directly demonstrate the capability ceiling. The base model comparisons in Table 4 show that on knowledge-intensive tasks (MMLU: 86.03 vs. DeepSeek-V3.1 at 88.44; MMLU-Pro: 67.91 vs. 67.75 vs. 67.50—roughly tied), the efficiency advantage is minimal or non-existent. The paper does not analyze which types of problems Ling-1T fails on, making it difficult for practitioners to anticipate failure modes in their own deployment domains.

The paper does not provide a difficulty-stratified analysis of the instruct model benchmark results—unlike some prior work (e.g., the compute-optimal test-time scaling paper that binned MATH problems by difficulty quintile), Ling 2.0 reports only aggregate benchmark scores. This means a practitioner cannot determine whether Ling-1T's strong average math performance is driven by near-perfect accuracy on medium problems while still failing on the hardest tier, or by broad improvement across the difficulty spectrum.

Mitigation status. The paper explicitly acknowledges this limitation in Section 6 (Conclusion):

"the effective reasoning length and depth still have room for enhancement"

and:

"complex instruction following and agentic behaviors remain under development"

However, these acknowledgments frame the limitation as an area for future improvement rather than a fundamental tradeoff: the 32–46% reasoning data proportion that gives Ling its reasoning strength may simultaneously limit its knowledge breadth compared to models with more general-purpose pre-training. The paper does not discuss whether increasing the general knowledge data proportion (at the expense of reasoning data) would improve knowledge benchmark scores, or whether the optimal data mixture might be deployment-dependent—a practitioner building a general-purpose assistant might prefer a different data mixture than one building a math tutor.

Latency Is Not Accounted For in the Efficiency Narrative

The assumption or constraint. The paper's efficiency claims are based on parameter count ratios (7× efficiency leverage vs. dense models) and training throughput (MFU improvements from FP8, heterogeneous pipeline parallelism). The paper measures efficiency in FLOPs and parameters, not wall-clock latency—the time a user waits for a response.

"Ling 2.0 series uniformly adopts a MoE architecture... It integrates aux-loss-free load balancing strategy and Multi-Token Prediction to optimize the training process."

The paper's infrastructure optimizations (Section 5) target training throughput: pipeline bubbles, FP8 utilization, checkpoint saving time. The inference-time characteristics of the models are addressed only through speculative decoding (MTP) and the Pareto frontier analysis (Figure 13), which shows average tokens vs. accuracy but not latency.

The consequence. MoE models with 256 experts and 8-way routing have fundamentally different inference characteristics than dense models with equivalent active parameters. At inference time, all 256 expert weight matrices must be available in memory (or rapidly accessible), even though only 9 are used per token. This creates a memory bandwidth bottleneck: the total parameter count (1T for Ling-1T) determines memory requirements, while the active parameter count (51B) determines computation. A dense ~50B model would fit comfortably on a single high-memory GPU; Ling-1T requires distributed inference across multiple GPUs or sophisticated expert offloading, introducing communication latency that the paper's training-focused infrastructure analysis does not address.

The sequential dependency created by the autoregressive generation process amplifies this: at each token generation step, the router must select 8 of 256 experts, those experts' weight matrices must be accessed, the computation performed, and the results aggregated. If expert parameters are distributed across GPUs (Expert Parallelism), this requires All-to-All communication at every token step. The intra-node DeepEP optimization (Section 5.3.1) is designed for training throughput, not inference latency—training processes many tokens in parallel across micro-batches, while inference generates tokens sequentially, making communication latency directly visible to the user.

The paper's MTP speculative decoding can reduce the number of sequential steps (by generating multiple tokens per forward pass), but does not reduce the latency per step—the expert routing and communication overhead per forward pass remains.

What evidence exists in the paper. The paper does not report inference latency benchmarks for any Ling 2.0 model. There are no measurements of tokens-per-second, time-to-first-token, or inter-token latency for single-user or batched inference scenarios. The evaluation pipeline optimization (Section 5.5) mentions using SGLang for inference serving and achieving throughput improvements through batching and prompt caching, but these are aggregate throughput metrics for evaluation, not latency guarantees for interactive use.

The paper reports that Ling-mini-2.0 delivers "30–60% throughput improvement over LLaMA 3.1 8B and Qwen3 8B when MTP is enabled, and 90–120% throughput improvement without MTP" on "8/16/32 80GB GPUs" (Section 5.1.2). However, this is throughput (tokens processed per second across batches), not latency (time to generate a single response). A 2× throughput improvement with batched inference does not translate to a 2× reduction in per-query wait time for a single user.

Mitigation status. Not addressed. The paper's conclusion mentions that "current grouped-query attention (GQA) architecture constrains efficiency in long-context scenarios; ongoing work explores linear and sparse-attention designs to further improve scalability" (Section 6), but this is about context length scaling, not interactive latency. A practitioner considering deploying Ling-1T as a user-facing assistant would need to know the p50 and p99 latency for typical query lengths, and the paper provides no such data.

The $7\times$ Efficiency Leverage Claim Has No Dense Baseline at the Trillion Scale

The assumption or constraint. The paper's central efficiency claim—that Ling 2.0 models achieve performance comparable to dense models with approximately 7× more activated parameters—is validated at smaller scales (Ling-mini-2.0 vs. ~8B dense models; Ling-flash-2.0 vs. ~36–40B dense models) and extrapolated to the trillion scale via the Ling Scaling Laws. At the trillion-parameter level, there is no direct dense comparison model. The baselines for Ling-1T (DeepSeek-V3.1, Kimi-K2) are themselves MoE models with their own sparsity characteristics.

The Ling Scaling Laws predict 7× efficiency leverage for the architectural configuration (256 experts, 8 active, 1 shared, 3.5% activation ratio), and the Wind Tunnel methodology is designed to validate that features scale predictably from small experiments to trillion-parameter models. However, the extrapolation is 100×—from Wind Tunnel models at 500M–8B parameters to Ling-1T at 1T parameters—and the validation of the extrapolation is ultimately correlational: the scaling laws predicted good performance, and good performance was observed.

The consequence. If the Ling Scaling Law's efficiency leverage prediction is wrong at the trillion scale—even by a modest factor of 2×—the entire architectural justification for the high-sparsity design weakens. If the true efficiency leverage is 3.5× rather than 7×, Ling-1T's performance relative to a hypothetical dense ~350B model would be unremarkable rather than Pareto-frontier-defining. The paper provides no way to distinguish between "the scaling laws are accurate at the trillion scale" and "the model performed well for other reasons (data quality, training recipe) and the architectural efficiency is incidental."

More practically, the lack of a dense comparison makes it impossible to answer the counterfactual: what if the same data, training recipe, and compute budget were applied to a dense architecture instead? The paper's own Scaling Law 2 (Figure 1b) predicts that compute-optimal MoE models have fewer parameters but are trained on more data than dense models at equivalent compute. This means a fair comparison would need to hold total FLOPs constant, not activated parameters constant—the very comparison the paper does not perform. The efficiency leverage metric (EL) compares compute-to-reach-loss, which is a FLOPs-matched comparison, but this is a scaling law prediction, not an experimental result at the trillion scale.

What evidence exists in the paper. The base model comparisons (Tables 2–4) validate the efficiency leverage at the mini and flash scales, but Table 4 for Ling-1T compares only against other MoE models. The instruct model comparisons (Tables 6–8) compare against models of unknown activated parameter counts (GPT-5-main, Gemini 2.5 Pro) or other MoE models (Kimi-K2, DeepSeek-V3.1), making the "7×" claim untestable at the flagship scale. The paper's own text acknowledges:

"the 14× larger model may not be compute-optimally trained"

—this is from the reference paper, not Ling 2.0, but the conceptual point applies: without a dense baseline trained under comparable conditions, the efficiency leverage claim is a scaling law prediction rather than a measured quantity.

The Wind Tunnel methodology (Section 2.3.3) is designed to provide the extrapolation evidence, but the tunnel models are up to 8B parameters—still 125× smaller than Ling-1T. The paper reports that Wind Tunnel predictions are accurate to within 0.01 in final training loss for the small-scale models, but does not provide the analogous measurement for Ling-1T (i.e., how close was the predicted training loss to the actual training loss?). The Figure 3b example shows a loss scaling curve for "a candidate feature," not for the base architecture itself.

Mitigation status. Partially addressed through the Ling Scaling Laws framework, which is the paper's primary mechanism for bridging the extrapolation gap. The framework provides a principled (power-law) basis for extrapolation rather than blind guesswork. However, the validity of the extrapolation depends on the power-law form holding across three orders of magnitude, which is an assumption that the paper validates at small scales but cannot experimentally verify at the target scale without a dense trillion-parameter training run—the very expense the scaling laws are designed to avoid. The paper does not discuss the confidence interval or uncertainty of the extrapolation, nor does it provide sensitivity analysis ("if the EL exponent is off by 0.1, Ling-1T's true efficiency leverage would be X rather than 7×").

Single Model Family, Single Training Paradigm — Generalization of the Design Philosophy Is Untested

The assumption or constraint. All Ling 2.0 models share the same fundamental architecture (high-sparsity MoE with 256 experts, 8 active, shared expert, GQA, SwiGLU, RMSNorm, Partial RoPE, QKNorm), the same training paradigm (reasoning-oriented pre-training with CoT mid-training, WSM scheduler, Evo-CoT RL), and the same infrastructure (FP8 with fine-grained quantization, heterogeneous PP). The paper demonstrates that this specific combination works well, but does not disentangle which components are essential and which are incidental.

The paper states:

"Ling 2.0 demonstrates that large-scale sparse language foundations can advance both reasoning capability and computational efficiency through coordinated innovations in architecture, training, and infrastructure."

The key word is coordinated—the innovations are presented as a package, and the individual contributions are not ablated at the trillion scale (or even at the Ling-mini scale for many combinations).

The consequence. A practitioner who wants to adopt part of the Ling 2.0 design—say, the WSM scheduler for their dense model, or the Evo-CoT training paradigm for their different MoE architecture—has no evidence that these components transfer. Does WSM help dense models as much as MoE models? Does Evo-CoT work with different sparsity ratios? Does the 32–46% reasoning data proportion generalize to models with different architecture or scale? The paper's design is holistic by intent ("every activation boosts reasoning capability"), but this holism makes it difficult to extract portable lessons.

More critically, the paper does not demonstrate that the combination is necessary for the results. Could similar performance be achieved with a simpler architecture (e.g., fewer experts, coarser granularity) if the training data and RL pipeline were the same? Could a dense model trained on the same reasoning-oriented data mixture with the WSM scheduler and Evo-CoT achieve competitive performance at a smaller scale? The Wind Tunnel methodology is designed to answer such questions cheaply, but the paper reports only the final architecture, not the alternatives considered and rejected.

What evidence exists in the paper. The Wind Tunnel experiments (Section 2.3.3) are described as the mechanism for evaluating candidate features, and several specific design choices are attributed to Wind Tunnel results: MTP depth = 1, MTP loss weight = 0.1, the First-K-Dense strategy, the aux-loss-free bias centering. However, the paper does not provide a systematic ablation table showing which features contributed how much to the final performance. The reader cannot determine, for example, whether MTP provides +2% or +5% on coding benchmarks at the trillion scale; whether QKNorm is essential for FP8 stability at all scales or only beyond a certain threshold; whether the shared expert is worth the parameter overhead.

The individual component evaluations that exist are at small scale: the MTP validation was done in the Wind Tunnel; the Ling Code Corpus and Ling Math Corpus were validated on 1B models (Figures 4a–4c); the WSM scheduler was compared against WSD (Figure 7) but only at one scale. The paper does not report Evo-CoT or LPO performance on architectures other than the Ling MoE design, so the interaction between the RL algorithm and the sparse architecture is unknown.

Mitigation status. Not addressed. The paper's framing explicitly celebrates the integration—"coordinated innovations across model architecture, pre-training, post-training, and infrastructure"—and does not discuss the limitations of presenting a tightly coupled system. This is not a flaw of the engineering (tight integration is often necessary for state-of-the-art results) but is a limitation for practitioners who need to make component-level decisions. The open-source release (model weights on HuggingFace, code on GitHub) partially mitigates this by enabling the community to perform ablation studies, but the computational cost of such studies at the trillion scale means most practitioners will not be able to independently verify which components are load-bearing.

The CoT Pre-Activation Claim Has a Potentially Confounded Evaluation

The assumption or constraint. The paper claims that introducing Chain-of-Thought data during mid-training (Reasoning Pre-Activation) substantially improves reasoning performance, as evidenced by the "w/o CoT Data" vs. "w/ CoT Data" columns in Tables 2–4. For Ling-mini-2.0-base, the AIME25 pass@1 improvement is from 2.08 to 43.75—a factor of ~21×.

The consequence. The magnitude of the AIME25 improvement raises a methodological concern: the "w/o CoT Data" model was evaluated on benchmark prompts that may require chain-of-thought formatting to perform well, but the model was never trained to produce such formatting. The paper states that AIME25 is evaluated with "0-shot, CoT" prompting (Section 3.3.2, Math Tasks list). If the "w/o CoT Data" model was prompted to produce chain-of-thought at evaluation time—a format it never encountered during training—its low score (2.08) may reflect format mismatch rather than fundamental reasoning inability. The paper does not clarify whether the baseline evaluation used the same CoT prompt format or a direct-answer format more appropriate for a model without CoT training.

If the evaluation protocol for the "w/o CoT" baseline disadvantaged the model by requiring an output format it was not trained to produce, then the apparent gain from CoT pre-training is partially an evaluation artifact rather than a genuine reasoning improvement—the CoT-trained model is better at producing the format required by the benchmark. The true improvement in underlying reasoning capability would be smaller (though still likely positive, given the gains on other benchmarks where formatting is less critical).

What evidence exists in the paper. The paper's benchmark specification for AIME25 is "0-shot, CoT" (Section 3.3.2). No alternative evaluation protocol (e.g., direct answer extraction, pass@k without CoT prompting) is reported for the "w/o CoT" baseline. The MATH benchmark, which is evaluated with "0-shot, CoT," shows a similar but less extreme pattern: Ling-mini-2.0-base improves from 61.96 to 82.52—a 20.56-point gain that is still large but more plausible than a 21× multiplier. LiveCodeBench shows improvement from 13.71 to 34.47 (2.5×). The variability in gain magnitude across benchmarks is consistent with the hypothesis that the gain depends partially on how strongly the benchmark's standard evaluation protocol depends on chain-of-thought formatting.

The ApexEval analysis (Figure 12) partially addresses this concern by showing that CoT-pretrained models have higher RL potential even when evaluated with ApexEval (which uses the highest pass@k score and LLM-based judges to reduce format sensitivity). The AIME 2024 ApexEval scores are 41.9 (with CoT) vs. 34.5 (without)—a 7.4-point gap that is substantial but far less dramatic than the 21× gap in the base model evaluation. This suggests that format effects are part of the story but not the whole story.

Mitigation status. Not addressed. The paper does not discuss the potential confound between CoT training and CoT evaluation, does not report alternative evaluation formats for the "w/o CoT" baseline, and does not analyze how much of the performance gap is attributable to format familiarity rather than reasoning improvement. Given that this is one of the paper's central methodological claims—that mid-training CoT pre-activation is a novel and effective strategy—the robustness of the supporting evidence is critical. A simple control: evaluate the "w/o CoT" model in a direct-answer format (suitable for a model without CoT training) and measure the gap against the CoT-trained model evaluated with CoT prompting, to estimate the format-independent reasoning gain.

This limitation is particularly important for practitioners who want to decide whether to invest in CoT pre-training for their own models. Without knowing how much of the gain is format-dependent, it is difficult to predict whether the benefit will transfer to deployment scenarios where the model may be prompted differently than during benchmark evaluation.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper establishes that extreme sparsity (~3.5% activation) and strong reasoning capability are not in tension but can be mutually reinforcing when architecture, data, and training are co-designed for reasoning from the earliest stages. This is not an incremental efficiency gain—it is a reframing of the design philosophy for reasoning-oriented LLMs. Before Ling 2.0, the implicit assumption was that reasoning emerges primarily from post-training applied to a general-purpose model, and that sparse architectures trade capability for efficiency. Ling 2.0 inverts both assumptions: reasoning capability is cultivated throughout pre-training, mid-training, and post-training as a unified objective, and sparsity becomes a mechanism for capability—the 7× efficiency leverage allows far more total parameters, and thus more specialized reasoning capacity, within a fixed compute budget.

The methodological contribution is equally significant. The Ling Scaling Laws + Wind Tunnel provides a principled alternative to the expensive guess-and-check that dominates large-scale model development. The ability to vet architectural features at 500M–8B parameters and reliably extrapolate 100× to trillion-parameter decisions, at only 35% of the cost of a single ablation run, changes the economics of model design. This is a meta-contribution that any organization building large-scale LLMs could adopt, not something specific to the Ling architecture. The Wind Tunnel formalizes what was previously intuition-based: multi-scale evidence produces more reliable design decisions than single-scale ablations.

The paper also resolves a latent contradiction in the sparse model literature. DeepSeek-V3 demonstrated that efficient sparse scaling is possible; Kimi-K2 showed that trillion-scale open models can achieve strong agentic capability; various works (GRPO, GSPO) advanced reasoning-oriented RL at moderate scale. However, no prior work demonstrated that all three—trillion-parameter scale, extreme sparsity, and reasoning-oriented training across the entire pipeline—could be combined successfully. The concern that extreme sparsity would fragment the representations needed for coherent multi-step reasoning was reasonable but empirically unfounded, at least when the shared expert provides a common backbone and the training pipeline is reasoning-oriented throughout.

Several research directions become more attractive after this work:

Joint optimization of sparsity and reasoning capability. The paper treats sparsity (activation ratio, expert granularity) as determined by the Ling Scaling Laws independent of the specific reasoning tasks the model will face. A natural next step is to ask: does the optimal sparsity pattern depend on the reasoning domain? Should math-specialized models have different expert granularity than code-specialized models? The Ling Scaling Laws provide the tool (EL formula) to explore this, but the paper only optimizes for general efficiency, not domain-specific reasoning efficiency.
Integration of process rewards with Evo-CoT. The paper's RL approach uses outcome-based correctness rewards with dynamic length control. Process reward models (PRMs), which provide per-step correctness signals, could provide richer training signal for the sentence-level optimization that LPO already performs—since LPO treats sentences as action units, a PRM that scores each sentence's logical validity would align naturally with the optimization granularity. The Ling 2.0 base model, with its strong pre-trained reasoning capability, would be an excellent starting point for PRM training.
Pre-training data mixture as a design variable, not a fixed recipe. The paper shows that 32–46% reasoning data works well, but does not explore the sensitivity—would 60% reasoning data provide further gains on math/code at the cost of general knowledge? Would a curriculum that phases reasoning data differently (early vs. late exposure) outperform the progressive increase used in Ling 2.0? The Wind Tunnel methodology makes such experiments cheap to run at small scale and extrapolate.

Conversely, some directions become less attractive:

Dense architectures for reasoning-oriented models at scale. If the 7× efficiency leverage holds broadly, the case for training a dense 100B+ parameter reasoning model is significantly weakened—the same compute could produce a sparse model with far more total parameters and superior reasoning performance. The burden of proof shifts to proponents of dense architectures to demonstrate that their models possess reasoning capabilities that sparsity fundamentally cannot capture.
Post-training-only reasoning enhancement. The paper's evidence that CoT pre-training provides a higher ceiling for RL (Figure 12) suggests that the "pre-train a general model, then RL for reasoning" paradigm is leaving performance on the table. Models that postpone reasoning to post-training may need to work harder (more RL steps) to achieve what pre-trained reasoning models achieve more efficiently.
Sophisticated search algorithms at inference time for non-thinking models. The paper's Pareto frontier (Figure 13) shows Ling-1T achieving high accuracy with relatively low token usage, suggesting that internalized, adaptive reasoning (Evo-CoT) can substitute for some of the compute that would otherwise be spent on search-based inference strategies (beam search, best-of-N). If models can learn to allocate reasoning depth appropriately during training, the case for complex inference-time search strategies weakens, at least for the class of problems where Ling 2.0 demonstrates strong performance.

Follow-Up Research This Work Enables

Calibration analysis of Evo-CoT adaptive reasoning depth. The paper demonstrates adaptive reasoning depth through aggregate metrics (Pareto frontier, average token efficiency), but does not validate that the model's internal difficulty estimation is well-calibrated. A strong follow-up would: (1) Stratify MATH benchmark problems by difficulty level (1–5) and measure both accuracy and response length for Ling-1T at each level, testing whether token allocation is proportional to actual difficulty. (2) Construct a set of "adversarial" problems that appear easy but require deep reasoning, and measure whether the model systematically under-allocates tokens to these problems. (3) Measure the correlation between the model's implicit confidence (e.g., probability of the generated tokens) and actual correctness, producing reliability diagrams that would reveal whether the model "knows when it doesn't know." The paper's ApexEval methodology (using highest pass@k to measure potential) could be adapted to measure whether the model's token allocation approaches the optimal allocation for a given accuracy target. This follow-up would test the most important unvalidated assumption of the Evo-CoT paradigm: that the learned difficulty-to-depth mapping is well-calibrated rather than merely correlated on average.

Ablation of the reasoning data proportion at the Ling-flash scale. The paper establishes that 32–46% reasoning data works well, but the optimal proportion may be deployment-dependent. A systematic follow-up would train Ling-flash-2.0-scale models (103B total, 6.1B activated) with reasoning data proportions of 20%, 40%, 60%, and 80%, measuring performance on both reasoning benchmarks (MATH, AIME, LiveCodeBench) and general knowledge benchmarks (MMLU, GPQA, C-Eval) to map the tradeoff curve. The Wind Tunnel methodology makes this tractable—five Wind Tunnel experiments at each proportion could predict the Ling-flash behavior, validated by training one or two of the proportions at full scale. The key question is whether the 46% used in Ling 2.0 is near-optimal or merely a point on a broad plateau. This would directly inform practitioners who need to choose a data mixture for their own models, and would test whether the Ling 2.0 design choices are robust to this hyperparameter.

Process reward model training on Ling-1T for scientific and mathematical reasoning. Ling-1T's strong base reasoning performance (82.78 MATH, 62.87 MinervaMath without post-training) makes it a strong candidate for process reward model (PRM) training—a PRM trained on Ling-1T's own reasoning traces could provide per-step correctness signals that would be used to further improve reasoning through search or RL. A concrete experiment: train a PRM on Ling-1T-base's outputs for MATH and AIME problems, using Monte Carlo rollout supervision (similar to Wang et al., 2023), then use the PRM to guide beam search over Ling-1T-instruct's responses. The hypothesis is that the PRM-guided search could push AIME 2025 accuracy beyond 70.42% by enabling deeper exploration of reasoning paths than the Evo-CoT adaptive depth allows. The failure case—if PRM search does not improve over Evo-CoT's adaptive depth—would suggest that the model has already internalized near-optimal reasoning strategies through RL, which would be an important negative result bounding the value of additional inference-time search for well-trained non-thinking models.

Cross-domain validation of the LPO sentence-level optimization. LPO's sentence-level granularity is motivated by the observation that reasoning proceeds in sentences, but this principle may not generalize to all domains. A systematic follow-up would apply LPO to non-reasoning RL tasks—creative writing (where sentences have aesthetic rather than logical coherence), multi-turn dialogue (where conversational turns, not sentences, are the natural action units), and code generation (where lines or statements are the natural units). For each domain, compare LPO against GRPO and GSPO on both training stability and final task performance. Specific experiments: for creative writing, measure whether sentence-level clipping improves coherence or causes the model to produce stilted, disjointed prose; for dialogue, test whether a turn-level variant of LPO (segmenting at speaker changes) outperforms both sentence-level and token-level approaches. The results would map the boundary conditions of LPO's effectiveness, identifying which task structures benefit from which action granularities—transforming LPO from a reasoning-specific technique to a general principle about aligning RL action spaces with task structure.

Inference latency characterization and optimization for trillion-parameter sparse models in interactive settings. The paper reports training throughput but provides no inference latency data. A critical follow-up for deployment viability would measure: (a) time-to-first-token and inter-token latency for Ling-1T on single-GPU, multi-GPU (tensor parallelism), and multi-node (expert parallelism) configurations at batch size 1; (b) the latency impact of the expert routing mechanism—how much time is spent on routing computation vs. expert computation vs. communication; (c) the effectiveness of MTP speculative decoding at reducing perceived latency for typical query lengths. The experiment would benchmark Ling-1T against a dense ~50B model (comparable activated parameters) and against DeepSeek-V3.1 (comparable total parameters) on a standardized set of prompts with controlled output lengths. This would provide the first characterization of whether trillion-parameter sparse models can be practical for interactive applications, or whether the memory bandwidth and communication overheads push them into batch-processing-only deployment scenarios. The paper's intra-node DeepEP optimization and the FP8 on-demand transpose weight technique provide the infrastructure hooks for this analysis, but the measurements themselves are absent.

Wind Tunnel applied to a novel architectural feature: mixture-of-attention. The paper's Wind Tunnel methodology is validated for MoE FFN layers, but attention mechanisms are the other major computational bottleneck. A prospective experiment would use the Wind Tunnel to evaluate a "mixture-of-attention" design—where different attention heads are routed to different subsets of the sequence, creating sparse attention patterns that reduce the quadratic cost of self-attention—using the same 500M–8B model sizes and extrapolating to trillion-parameter behavior. The key metric would be whether the efficiency leverage from sparse attention follows a similar power-law amplification with compute as the MoE FFN EL. A negative result (sparse attention provides fixed efficiency gains that do not amplify with scale) would be equally informative, identifying a fundamental difference between FFN sparsity and attention sparsity in scaling behavior. The paper's existing partial RoPE implementation (first 64 dimensions only) suggests the authors are already thinking about attention efficiency, and the Wind Tunnel provides the experimental framework to explore this systematically before committing to a trillion-parameter architecture change.

Practical Applications and Downstream Use Cases

Cost-efficient mathematical reasoning for educational technology. Ling-mini-2.0 (1.4B activated parameters, 72.66 on CNMO 2024, 65.62 on AIME 2024) can be deployed on a single consumer GPU or even on-device for real-time math tutoring applications. Unlike models that require explicit chain-of-thought prompting (which increases response latency and token cost), Ling-mini-2.0's Evo-CoT training enables it to produce reasoned answers in "reflex-grade" mode—internalizing reasoning without verbose output. For an educational platform handling millions of student queries per day, the combination of small activated parameter count (1.4B vs. ~8B for comparable dense models) and adaptive reasoning depth (shorter responses for simpler problems) translates to roughly a 7× reduction in inference compute cost compared to deploying a dense model of equivalent capability. The paper's benchmark on Omni-MATH (60.30 for Ling-mini-2.0 vs. 41.71 for Qwen3-8B) demonstrates that this efficiency does not come at the cost of math capability—it improves on it.

Code generation for software development workflows with Ling-flash-2.0. Ling-flash-2.0 (6.1B activated, 51.38 LiveCodeBench, 1600 CodeForces rating, 71.24 Aider-Edit accuracy) occupies a sweet spot for integration into developer tools: it substantially outperforms Qwen3-32B on coding benchmarks (LiveCodeBench: 51.38 vs. 31.50; CodeForces: 1600 vs. 696) while using roughly 5× fewer activated parameters, enabling deployment on modest GPU infrastructure (a single 80GB GPU) that development teams can realistically provision. The strong performance on Aider-Edit (71.24 vs. GPT-4.1 mini at 74.44) and FullStack Bench (47.01 vs. GPT-4.1 mini at 56.31) indicates that the model is practical for real software engineering tasks, not just synthetic benchmarks. For a medium-sized engineering organization building an internal code assistant, Ling-flash-2.0 offers a path to self-hosted deployment that avoids sending proprietary code to external API providers while maintaining competitive performance with proprietary models.

Batch inference for large-scale data generation and self-improvement pipelines. Ling-1T's combination of high reasoning accuracy (70.42 AIME 2025, 74.46 Omni-MATH) and FP8 training efficiency (15% MFU improvement, 15% memory reduction) makes it suitable for generating high-quality reasoning traces at scale—a critical component of self-improvement pipelines where a model generates training data for its own fine-tuning. The paper's infrastructure optimizations (heterogeneous pipeline parallelism achieving 40% throughput improvement, distributed evaluation reducing checkpoint validation from days to hours) directly address the throughput bottlenecks that make large-scale data generation with trillion-parameter models impractical. Concretely: a team using Ling-1T to generate 100K CoT reasoning traces for math problems would benefit from both the model's reasoning quality (producing traces that actually reach correct answers) and the infrastructure's throughput (producing those traces in a fraction of the time that a comparably-sized dense model would require). The paper's validation that FP8 training achieves near-lossless accuracy (within 0.25% of BF16) provides confidence that the generated traces are not degraded by precision artifacts.

On-device or edge deployment of reasoning-capable assistants with Ling-mini-2.0. Ling-mini-2.0's 1.4B activated parameters push it into the range where on-device inference is plausible for high-end mobile devices or edge compute nodes, while its reasoning performance (65.62 AIME24, 82.99 MBPP) provides capability that would otherwise require cloud-based models 5–7× larger. For applications requiring offline operation or data privacy (medical reasoning in a hospital setting, legal document analysis in a law firm, field deployment for scientific research), the ability to run a genuinely capable reasoning model without network connectivity is a qualitative change in what is possible. The paper's FP8 training framework and MTP speculative decoding (30–60% throughput improvement on 8 GPU configurations for Ling-mini-2.0) suggest that optimized inference runtimes could achieve interactive latency even on consumer hardware, though the paper does not provide on-device benchmarks. The strong multilingual performance (HumanEvalXL: 65.31 for Ling-mini-2.0-base) further expands the deployment scope beyond English-centric applications.

🎯 Pitch

1. Executive Summary

2. Context and Motivation

The Core Problem: Scaling Reasoning Ability Without Scaling Cost Linearly

Why This Problem Matters: The Economics and Science of Scale

Where Prior Approaches Fall Short

How This Paper Positions Itself

The Gap This Paper Fills, Concretely

3. Technical Approach

3.1 Reader Orientation

3.2 Big-Picture Architecture (Diagram in Words)

3.3 Roadmap for the Deep Dive

3.4 Detailed, Sentence-Based Technical Breakdown

Ling Scaling Laws: Predicting Trillion-Parameter Behavior from Small-Scale Experiments

The Ling Wind Tunnel: A Standardized Experimental Pipeline for 100× Extrapolation

Model Architecture: High-Sparsity MoE with MTP

Pre-Training: Reasoning-Oriented Data and Multi-Stage Training

Post-Training: From Base Model to Reasoning-Oriented Instruct Model

Infrastructure: Full-Scale FP8 Training and Heterogeneous Pipeline Parallelism

Summary of Design Choices and Their Justifications

4. Key Insights and Innovations

Innovation 1: Sparsity Amplifies Reasoning, Not Degrades It — A Refutation of the "Fragmentation Hypothesis"

Innovation 2: Reasoning as a Pre-Training Objective, Not a Post-Training Add-On

Innovation 3: Adaptive Reasoning Depth via Difficulty-Conditioned Reinforcement Learning

Innovation 4: The Ling Scaling Laws as a Methodology, Not Just a Result

Innovation 5: Sentence-Level Policy Optimization as a Principles-Based Solution to Granularity Mismatch

5. Experimental Analysis

Evaluation Methodology

Main Quantitative Results

Base Model Performance: Validating the 7× Efficiency Leverage

Instruct Model Performance: Post-Training Effectiveness at Each Scale

Adaptive Reasoning Efficiency: The Pareto Frontier Analysis

Evo-CoT Training Dynamics: LPO Stability and Convergence

DFT with vs. without CoT Pre-Training: Impact on RL Potential

Ablation Studies and Robustness Checks

Critical Assessment

Does the 7× Efficiency Leverage Claim Hold?

Does the Reasoning Pre-Activation via CoT Data Claim Hold?

Does Evo-CoT Produce Genuinely Adaptive Reasoning Depth?

Does LPO Genuinely Outperform GRPO/GSPO for Reasoning?

Are the Baselines Fairly Chosen and Evaluated?

Statistical Rigor and Reproducibility Concerns

The Pareto Frontier Claim: Efficiency and Accuracy

6. Limitations and Trade-offs

The Difficulty Estimation Problem Is Pushed to the Architecture, Not Solved

Hard Problems Remain Essentially Unsolved — Sparsity Does Not Create Capability

Latency Is Not Accounted For in the Efficiency Narrative

The 7×7\times7× Efficiency Leverage Claim Has No Dense Baseline at the Trillion Scale

Single Model Family, Single Training Paradigm — Generalization of the Design Philosophy Is Untested

The CoT Pre-Activation Claim Has a Potentially Confounded Evaluation

7. Implications and Future Directions

How This Work Changes the Landscape

Follow-Up Research This Work Enables

Practical Applications and Downstream Use Cases

Adjacent in domain

FP8-Flow-MoE: A Casting-Free FP8 Recipe without Double Quantization Error

LongCat-Flash-Omni Technical Report

The $7\times$ Efficiency Leverage Claim Has No Dense Baseline at the Trillion Scale